Create Workflow through scripting

[dklooker]

Hi,

Is is possible to create a workflow for ffastrans and submit/start that workflow via an other program?

For example:
-Our scheduling program will run a script when a new client/project has been created
-That script will create folders on our FTP server for submitting files from our client
-Will create projectfolder on our workserver
-Creates a workflow (from maybe a template) for ffastrans where those paths are used for a monitor and deliver node.
-Sumbits and starts that new workflow in ffastrans

Kind regards,
David

[FranceBB]

You can actually use the API to trigger a workflow.
There are a bunch of things that can be done with the API, it's just a matter of how you call FFAStrans from the other program. The same works the other way around as well: FFAStrans can also call another program via the HTTP execution node which calls their API.

Anyway, in your case:

http://<host>:65445/api/json/v2

Open the link, check what you can do and make a workflow that uses the API. I would show you an example but I'm on my mobile now...

[admin]

Hi David,

As FranceBB writes, you can start/stop workflows but currently you cannot create them through the API (which btw. is /api/json/v2 in FFAStrans > 0.9). However, if you are script-savvy you can make yourself a workflow in FFAStrans that acts as a template which you can create copies of and modify. All workflows are just .json files in the "Processors\db\configs\workflows" folder. If you "manually" add new valid workflow .json files in the folder they will be usable by FFAStrans. But modifying .json files is not straightforward and you need to know what you're doing. Remember, this is hacky and you do this at your own risk.

-steinar

[emcodem]

Hehe if we get @momocampo to answer as well, you got attention of the whole team, lucky bastard
What you describe sounds pretty messy to me, you want to work with us trying to design this overall workflow a little less messy?

-Our scheduling program will run a script when a new client/project has been created
-That script will create folders on our FTP server for submitting files from our client
-Will create projectfolder on our workserver

• Base folder names on FTP and on the workserver are equal?

• Do i guess correct that we cannot just have a single ffastrans workflow watching for all subfolders and deliver to the corresponding project folder?

• How many "pending" client uploads to you foresee concurrent, like 10 or more like 1000s?

• Is there anything that qualifies a "complete" upload? If not, how could you determine which of the created "folders" is obsolete?

-Creates a workflow (from maybe a template) for ffastrans where those paths are used for a monitor and deliver node.
-Sumbits and starts that new workflow in ffastrans

Thats the easy part i guess.

[momocampo]

Ok @Emcodem I can answer too
So yeah , in my case I use the api for all kind of things. Schedule a workflow start (execute a command executor node to save files during night or create folder ), submit files manually from different servers through network, specify special rights for a workflow for specific users, etc.
You can even cut a video file from the api !

[dklooker]

Hi Emcodem,

Thanks for the attention! Indeed it's a bit messy and I am just overwhelmed by the possibillities.

--Base folder names on FTP and on the workserver are equal?[

They have a different setup to help steer our clients in the right direction/folder

FTP
CLIENT/TO/Project/Episodes
CLIENT/FROM/Project/Episodes

SERVER
CLIENT/PROJECT/TO/Episodes

But we work also for Clients that work with other facilities who will do the delivery to us so, then the facility is actually the uploading client with their own folder and things won't match up.

--Do i guess correct that we cannot just have a single ffastrans workflow watching for all subfolders and deliver to the corresponding project folder?

Yes you are correct. But It got me thinking that we can setup a workflow that covers 80% percent, the rest we can make custom.

--How many "pending" client uploads to you foresee concurrent, like 10 or more like 1000s?

More like 10.

--Is there anything that qualifies a "complete" upload? If not, how could you determine which of the created "folders" is obsolete?

Not sure what you mean by this.

But I would like to setup a base workflow that checks if the destination folder is equal to the source folder and exists (where we correct the source path string to conform to the destination path structure) IF True than transfer files, otherwise do nothing. Then we basically automate 80% of our uploads.

David

[emcodem]

OK cool, if you keep working with us, i am sure we can design the stuff in a way that it works really well and does not cause too much headache when you face problems later on.

Thanks for the attention! Indeed it's a bit messy and I am just overwhelmed by the possibillities.

Yeah i know it very well, its always a problem when you have lots of possibilities

--Base folder names on FTP and on the workserver are equal?

They have a different setup to help steer our clients in the right direction/folder

FTP
CLIENT/TO/Project/Episodes
CLIENT/FROM/Project/Episodes

SERVER
CLIENT/PROJECT/TO/Episodes

But we work also for Clients that work with other facilities who will do the delivery to us so, then the facility is actually the uploading client with their own folder and things won't match up.


--Do i guess correct that we cannot just have a single ffastrans workflow watching for all subfolders and deliver to the corresponding project folder?

Yes you are correct. But It got me thinking that we can setup a workflow that covers 80% percent, the rest we can make custom.

Nice, this seems to be the key part here.
Currently i am under the impression that ALL Clients folders will always contain FROM and TO, so what would keep you to just have a single watchfolder processor and set the deny folders to "TO"? Of course the calculation of the target path needs to be done in the workflow then but i am sure whatever you need is possible.

--Is there anything that qualifies a "complete" upload? If not, how could you determine which of the created "folders" is obsolete?

Not sure what you mean by this.

But I would like to setup a base workflow that checks if the destination folder is equal to the source folder and exists (where we correct the source path string to conform to the destination path structure) IF True than transfer files, otherwise do nothing. Then we basically automate 80% of our uploads.

OK, this question can be divided into 2 parts:
1) do your clients just upload "single contained files", like mxf op1a, or is it more like they upload video+audio files separate + metadata maybe, or even some kind of playlist or P2 or other complex structure

2) Dynamically creating "watchfolders" is not a big problem as such, but when designing such a business logic, you need to also think about how to STOP or delete the dynamically created watchfolders, e.g. detect that the delivery is complete and then delete the created worfklow.

[dklooker]

Hi Emcodem,

I am almost there (I think). I did some additional scripting. It now checks if the destination project folder is the exact same as the source folder, consideing the different path semantics. When that is succesful is constructs the destination folder to be used in the delivery node. However it seems I can not take the STDOUT of the au3 script to be assigned to a user variable in FFAStrans. Is that true?

I attached my workflow and both au3 scripts. Maybe the construct of the destinationfolder should be done in FFAStrans via a RegExp, but I thought this to be easier...

Kind regards,
David

[dklooker]

Got it. I got show console enabled. I read another post where this was a problem when assigning STDOut to a variable.

Now testdriving and then this should sync our entire ftp with our workserver only when project and client names are exactly the same. The rest we wil do manually, but I can imagine that we do store a reletive links in a xml or so...

After this has been setup I will try to create a program that will create contextual menu in windows for executing workflows.

[emcodem]

Hey David,

nice, i was not aware that a little scripting is an option for you
As long as the target path can be calculated based on the source path i guess it is not neccessary to create "monitor folder" processors in your workflow or even whole workflows dynamically. This makes everything a lot less problematic as you don't need to take care about cleaning up the dynamically created workflows as well.

You might want to know that the monitoring in ffastrans is tuned to support really huge source subfolder structures so it is no problem to have only a single monitor folder processor watching thousands of subfolders. - at least from a performance perspective.

Regarding your scripts, i'd recommend not to "hardcode" paths into your script but pass them as cmd line parameters instead. Also, i guess you already know it, but using autoit, you can compile your stuff easily to an .exe file. This would remove the need to have autoit installed for your workflows and therefore keep your ffastrans installation portable (in case you need to switch machines or go for a farm later on).