Workflow control elements Split and Join

[kibi]

At present FFASTrans uses implicit method of workflow control. As far as I understand, it expands complex workflows into set of simple sequential workflows with heavy branching:

24-08-2016 19-11-13.png (32.19 KiB) Viewed 13146 times

It supports split operation and doesn't support joins operation. Every other block uses result of the previous block and there is always only one previous block. It ensures that a particular single source always exists.

But there are a lot of possible cases when JOIN operation is required. For example final step in complex workflow, that should be executed at the very end and only once. Deleting of temporary files or moving of the initial file to another place.

There is another case hard to resolve in a native way - HLS or DASH creation. It is a generation of several streams of different bitrates (the first, parallel, steps) and a final packaging. The packaging could require all the streams completed, as it extracts some metadata from all of them. In terms of FFASTRans packaging is Command Executor Processor, which uses some user variables, defined at previous steps.

Current FFASTrans' paradigm needs a special processor, let's call it "Operations/Join", to implement such command:

24-08-2016 19-15-55.png (33.3 KiB) Viewed 13146 times

The engine should expand all branches as usual except Join processor and other processors after them. JOIN processor should start it's work only when all previous connected processors finish their work.

There is an open question - what to do with Delivery Folders or Filters, if user put it after "Join"? What should be the "source" for processors, which one from previous branches? May be it's reasonable to support only those ones from "Others" and "Operations" groups.

Taking into consideration the above, a proper name for the new processor is not "Join" but "Final".

[admin]

Thanks for your thoughts, Alexander!

I know about this "wait" scenario and it will be looked at. I have been thinking about a "Synchronize"-node which holds the branched jobs until all connected nodes reach that point, just as you describe.

The other thing you mention is a way to join data/assets from previous nodes. I see a "wait"-node and a "join"-node as two separate things. The latter is much more complex to deal with. As you write, how do we know what source data to use if coming from multiple nodes? For FFAStrans, all source data are just variables and so it needs to deal with conflicting variables. Now, I'm very open to suggestions here because I'm not sure how I'm gonna approach this particular "join"-issue

-steinar

[kibi]

I think, that true "Join" in part of video processing steps is impossible to realize, since there is ambiguity what would be the source for the next node. Or how to address to different sources. More over almost any complicated logic of audio/video processing is already possible to implement with Custom AviSynth. It has ternary operator

Code: Select all

b = (a==true) ? 1 : 2

and user defined functions - that's enough. AFAIK you don't want to duplicate whole Avisynth

So there is lack of node that (1) waits for and (2) runs single time only. That's why more suitable name for it is "Final", because there should not be any video processing nodes after it. That makes the trick.

And yes, I can imagine a workflow where "Syncronize" node would be useful. I have several more ideas connected to workflow controlling and will draw them in different threads, to avoid mess here.

[dylanreeve]

I agree some sort of join/wait/synchronize node would be exceptionally helpful.

[admin]

Good things come to those who wait

-steinar

[kibi]

Hi, Steinar!
Could you please tell what's going on with the development of the new nodes? It's still very awaited.

[admin]

The new node was finished a time ago. It has the ability to wait for other file(s) and wait for a specified amount of time. Also, in a branched/forked workflow it can wait for the other sub-jobs to arrive at the same node before continuing.

-steinar