Workflow succeeds, but encoder node fails - how to "report" that to Web interface?

[crispyjones]

Thanks for the demo and I'd like to contribute as much as I can. Here are the few notes I have so far.

Please add another section on how to deal with the help file. When I open the Force error node, or the example from the wiki, and click on the help ? at the bottom of the processor I get an error about a missing help file. I made my own simple help.html and put it in \Processors\plugin_nodes\custom_nodes\Force Error\help.html, but it definitely doesn't look the same.
The other thing that pops out is the difference in font and size between the title "Hello World, i am a custom processor" and the table section that has input fields, they definitely do not look like the baked in processor nodes table field. Though that may be a choice so that it is obvious to the user they are in a custom node. This is likely some basic CSS tweaking, but I don't deal with the guts of html very much.

One last thing. The Force Error block, if the processor name is left unchanged in the workflow manager, shows up as plugin_htmlgui in the advanced WebUi (sreenshot). If it is modified, the modified name does show up correctly. The name is *not* affected in the result field of the basic webui or the outcome field of the advanced webui.

2020-04-15 15_21_31-FFAStrans Workflow.png (59.26 KiB) Viewed 13076 times

[emcodem]

@crispyjones

wow that really great input! thats exactly what we need to get settled with the custom procs.
I added a section about the help file in the wiki including some styling example, it will be easy for that part...
Finally the design of the processor gui was a huge topic for steinar, he likes the custom procs to reflect all the look and feel from the stock processors.
In the end i created such a processor but it is not yet 100% finished.
All the needed styling was engineered there and i'll use them and create another Example processor. But what i do not plan is to come up with a javascript framework that allows easy building of procs, at least not until a minimum of 10 different users really built their own processors or one user built more than 20 processors.
In the end most is about placing the inputs, thats pretty easy done using margin-left and display:block style.
Besides the GUI building i am very looking forward if you get the processor itself (processor.exe/script) done in your preferred scripting language

[crispyjones]

Attached is my first, meager, attempt at a custom processor. I took the "Force Error" example that @emcodem helpfully wrote and blatantly plagarized it to make my own. This processor is called "Find Owner" and grabs the source file input to the processor and finds the owner of that file using powershell Get-Acl and bit of regex to strip it from the preceding domain. In production I made a user variable %s_username% that I append the message "submitted by %s_username%" to the success or failure messages of later processors so I can see who is submitting files that are causing errors. I have also used it as the input to an email processor that automatically emails %s_username%@mycompany.com on failure of their job.

processor.jpg (52.67 KiB) Viewed 12928 times

I originally accomplished this function using the built in "Command Executor" to run the script, but there were issues with certain long file names that contained emojis and other symbols that were causing issues. This required multiple "Command Executor" and some pass/fail branches, so I made this instead. It works, so far , and makes for a cleaner less complex workflow and I learned a few things.

Please note I am not a programmer, this is 99% copy paste other people's work with a lot of trial and error thrown in. If you're interested in re-writing this one or making your own .... read on.
Somethings I found helpful when building this. Right click workflow you are working on and select properties>maintenance tab>check the "keep jobs until completion" box, otherwise FFAStrans will delete all of the files of interest after every job. Go back to the General tab and note the path in the Work Folder window. Submit some test jobs to your workflow with your custom processor and then go to the workfolder path you found earlier (it will probably look something like c:\.ffastrans_work_root\20200417-2214-1523-1732-8c822b35uda). Once in that folder you can see all the files that are created as a file goes through (or fails) in your workflow. I found that the file ending in ~plugin_htmlgui~stdout.txt contained the windows command run by my processor. I could copy paste that and run it directly and see what errors where happening in the console window.
The files ending with ~plugin_htmlgui.inputs.json and ~plugin_htmlgui.outputs.json contained all the JSON input and output key value pairs. I toyed with using the json sources.current_file value but using the $1 argument directly was easier. Hopefully some smarter person will correct me if that is the wrong assumption.

[emcodem]

@crispyjones

i am so happy that you did this and tell us about your experiences, only this way we get on with this very important feature. Users are asking for different additional processors so often, sometimes just small stuff as you and me did...
I am sorry that i did not yet have the time to write some best practices article about easy developing and debugging as you describe but using your findings, i have an easy time to do so now

It is just perfect and right how you did this, i expect 99% of all processors that other users build will be plagarized from existing stuff, thats how it all is intended by me at least.
You are absolutely right to access the source by using $1 instead of parsing the json, thats why i pass the source file as second argument: to ease your life as a plugin dev!

Of course i downloaded and had a look at your processor, it is working great. Do you mind if i add it to the new wiki page with all processors?
http://ffastrans.com/wiki/doku.php?id=c ... processors

[crispyjones]

Thanks for the kind words and please add it to the wiki so everyone can have access. Do you think that custom processors should get its own sticky or forum?

[emcodem]

Great, thanks... already added to the wiki. Maybe i should add the dev's name in future in the list too.
And yeah, we discussed this topic about giving the custom procs it's platform today too: adding a subforum and all other possibilities. However the feature is deeply buried in the source code and we are not yet 100% certain about it, there is still some work that we do around it and some stuff may lead to existing processors not being backward/forward compatible enough. E.g. today the topic of opening a file browser was raised and a way to control which types of variables are shown in the variable chooser. Then i saw that the monitor sometimes presents a totally different "name" of the running custom node when there are multiple custom nodes installed...

You know, we all are broadcast engineers and therefore not very amused to release totally new technologies without testing them as long as it needs for us to get the feeling the feature is mature enough to be used

Anyway, your help is really extremely welcome, please continue what you are doing now and use the custom node feature in production - but also make sure you have a backup of your workflows, maybe best to do it periodically, especially when altering a workflow that involves multiple custom nodes.

[jan2ooo]

First of all thank you for this great tool! I am still playing around with the nodes, but right now I have a question about the "Force_Error" processor. I move failed files to a "failed_original_files" folder (command executor with "Execute on Error"). So even the errored files will "Success" in status monitor.
Where can I insert the "Force_Error" to receive the "Errored" status in the status monitor? I added the node at the very end. But this will still finish with "Success".



Thank you!

[emcodem]

Hi @jan2ooo,

welcome to the forum and thank you for using FFAStrans!
You should have maybe opened your own thread for this because the way you did it now, you will not get a "notification" if someone answers and does not write @jan2ooo.

So, the position of force_error plugin processor is just fine, the job should be displayed as error in webinterface and in the inbuilt status monitor you should see the error msg as outcome.
Just as a sidenote, instead of using this plugin processor, you can these days just use a populate vars processor and set the variable s_error to some value, it should have the same effect. Maybe you can upload the full_log.json of some job that shows Success but it should not?

[jan2ooo]

Hey @emcodem

Thank you very much! Here is the log file. My input is an EXE file, so this will error for sure.

Does the log help?

[emcodem]

@jan2ooo

yeah, looking at your log, the force error processor fails execution for some reason. You should really toss this processor and instead just use a populate variables, setting "s_error" to your preferred errr msg.