Check if farm node is up when calling an workflow

[veks]

Hi all!
Is it possible to add a check if farm node is up and running to an ffastrans workflow processor, same as all other processors have an option to execute "on success", "on error" and "on both"?

Because, for example, I'd like to have an option to go to the hardware encoding server/s when they are up, or if they are down, go to the another server which do an software encoding.

I know this can be done through bash script, but I think this should be a feature of an FFAStrans too.

Thanks!

[emcodem]

Hey veks,
`
Not sure if i understood your request correctly because of this sentence:

I know this can be done through bash script, but I think this should be a feature of an FFAStrans too.

The thing is, checking if a host is up can be done pretty simple but there is no known to me way to force branches or processors to run on a specific host (group). Because if it was possible to force specific hosts per processor, all you'd need to do is to try first gpu encoding on host group A and on error, execute cpu encoding on host group B...

[veks]

When calling a next workflow
https://prnt.sc/WAIoJ3ypolUY

That you have an output option for an workflow processor so that if workflow fails that it continues as an "error" and goes to the next processor or next workflow processor, same as other processors that have an input or output.

So for example like folder processor works.
https://prnt.sc/8VemSNkvshxO

[emcodem]

Hm i thought host groups of the "sub-workflow" are "ignored" and instead inherited by the main workflow when using this way of starting a sub-wf.
@admin will enlighten us

Anyway, to give you some insight why it is as it is currently: we would need to kind of guess if a workflow "succeeded" or not, the choice is always "success if one or more branches succeeded" or "error if one or more branches errored". Also, same as when you split into branches, ffastrans has currently no way to merge the branches together because of the internal variable handling: each branch carries it's own copy of all the variables and can change that. If branches would be merged, you'd end up with an array of variables (e.g. X times s_source, one for each branch).

There was a lot of internal discussion about all that and i guess it would take a while to address this if @admin wants to.

[admin]

Hi veks,

It is indeed possible in an indirect way to find out if one host is down or not. In this regard the rest service must be enabled on the host in question.
Please import the attached workflow and change the %s_my_host% in the initial populate step. Submit whatever file and take a look at the outcome. If the host is active you will have "success" and of course error if not.

Now, about the way workflows work in regards to end status. It's VERY important to understand that a job can have BOTH a success and a failure outcome. This is because of the way splitting is designed. You can compare it to signal splitting. Once a signal is split into two signals they live independant of each other. Combining the two again means that you either must throw away one of the signals or collect them all. If you want to throw away you need some kind of logic to know what to throw away and not. This cannot be automatically handled by FFAStrans cause it really depends on the intention of the workflow creator. If you collect them you need some way of getting all the different variables set by the different splits. So you see, merging is not as trivial as it might seem.

Regaring the inheritance of farming when initiating a sub workflow; it's optional: You can chose either inherit or not in the "Insert workflow" node

-steinar

[emcodem]

Neverthless it would be a good feature to have an output connector of the sub-workflow processor. But looking at the current implementation of sub-wf, we'd need more something like "start a sub-wf via API and wait for the result", which can be done already today using the HTTP communicate+ plugin processor. Let me know if you need an example for that @veks. Another option is of course to have a branch waiting for the appearance of a status file that the sub-workflow writes on success/error.

also i am asking myself if @veks only uses the sub-workflow in order to simulate the missing "per node host group" functionality.