REST API P2 Transcoding and destination path

[nknuelle]

Hi everyone,

I am transcoding files with my own external program. For that I want to use REST API. The docs mention that I can submit a new job with a given filepath. I need to transcode p2 files inside a p2 content structure. For this there is a monitor node. But if I use the REST API I can't issue a whole folder path for the transcoding. I think submitting jobs for whole folder structures would be a very nice feature.

I also have the problem, that I need to create folders in the destination folder from time to time. This is because of Avid which has a limit of files inside Mediafolders. For this it would be really nice to also submit an output folder for the transcoding job via REST API. The output folder could then be defined on a per job basis not only on a per workflow basis.

[emcodem]

Hey @nknuelle, nice to hear from you! To be honest i think you should post more often here

But if I use the REST API I can't issue a whole folder path for the transcoding.

Actually using the API you can submit whatever you like to your workflow, so no problem at all to submit a "folder" or any other value. Check out the webinterface job starter for example.
However, your program that submits the API job will always need to replace any functionality that the "monitor nodes" would do. E.g. what the P2 monitor does is just to use bmxtranswrap to create a single file out of the video/audio files from a p2 structure. Of course in case you want to submit P2 stuff via API, you would have to rebuilt all "watching" functionality that checks if a job can be submitted (audio and video is there), and the workflow itself must be able to deal with whatever you submit as source file via API. For example you can submit build a ffmpeg concat file via API and the first thing in the workflow is a commandline processor that uses ffmpeg to wrap your stuff from the concat file.

It would be a massive design issue if the API would transform your files in any form BEFORE it actually starts your workflow. I mean in the end you submit values to a totally generic workflow engine, ffastrans is not bound to transcoding or such stuff... so it would be horror if values or media was transformed internally before they actually reach the workflow. You can build any functionality you need easily into your workflow - and if needed, you can even build

...to also submit an output folder for the transcoding job via REST API.

The API allows you to submit 2 different things, one of them is %s_source% which will be displayed in the status interfaces. Then we have the list of user_variables, you can submit whatever you like there, including output folders or such.

Did you check out this example for example?
viewtopic.php?f=5&t=849&p=3549&hilit=beginners#p3565

Last note on this:
From my perspective, it would be important to explain why you cannot use the inbuilt P2 Monitor and need to use the API for whatever you are trying to do...

[emcodem]

One more thing, you asked once about "how to get part of the development".
I think this should fulfill all your dreams: http://ffastrans.com/wiki/doku.php?id=c ... processors
The thing is, even if the documentation is not finished yet, the feature is already there. And we need someone to build and use custom processors in order to get out possible bugs in the workflow integration (it already works quite well but it needs to be "used" to reveal all the major issues).
Are you interested in building your own processors and helping us finding all the bugs in this new functionality?

[nknuelle]

Hey @encodem

I think it's necessary to make clear what I want to achieve in the end:

I am building a cross platform automatic ingest system based on Python (so it should be more or less platform independent). Because I wanted to "outsource" the watchfolder/monitor mechanism and transcoding to another program I chose FFAStrans, because I think it's very handy to add it to an existing workflow.

My program handles many kind of physical media such as P2, XDCAM, SD Cards, GoPro... It handles mounting and copying, adding workflow configurations for backup and so on.. The main user group are less-technical users like reporters or EB Teams. For this it offers a simple GUI to start the Ingest process. My program also handles generating of AAF files. I need those to check-in Avid Assets into an Avid Interplay Environment.

The problem I have with FFAStrans is, that I need to access "job events" from the outside. E.g. I have to count the number of input and output files to verify that FFAStrans transcoding was succesful and to manage the Check-in functionality. I need to transcode everything into AVC Intra 100 MXF OP-Atom to use the files directly with Avid Media Composer / Avid Interplay.

I thought it is easier to manage all this via REST API because this enables me to access FFAStrans from my external program and make monitoring of the transcoding and file outcome more easy. Currently I am still using the Input Monitor and mounting my external physical drives to those input folders to start transcoding.

The biggest problem: I need to have seperate output folders for every input file. For example:
Transcoding of XDCAM MXF File results in 9 OP-Atom files (8 Audio, 1 Video Stream). I built up a Workflow like this:
Input Monitor for Folder (accepting MXF in recursive folders)->Encoder (AVC Intra, MXF OP Atom)->Deliver to output folder*

* Here I have the problem, that I can't tell FFAStrans to create a new folder for every input file (all 9 output files in case of XDCAM)

Maybe this makes it somehow clearer what features would be nice to have

PS: If it's possible to implement such a workflow with FFAStrans using custom processors let me now and maybe I find some time to implement it

[momocampo]

Hello Nknuelle,

I just answer to your "biggest problem". You can easily create a destination folder for each input file, you just have to for example add a variable at the end of your destination path inside your deliver node like this :

Code: Select all

c:\output\%s_original_name%\

In this example, all Avc Intra Atom files will be delivered in its own folder(called like the input file).

Hope it helps

Benjamin

[nknuelle]

@momocampo thank you for that! I think I missed that feature

[momocampo]

You're welcome
But if you want to use these atom files in Avid(and if your destination workspace is the same for all files) , maybe you should deliver all files in a "AvidMediaFiles\MXF\" hostname\"", delete database with a command executor node, launch avid and finally drag'n drop. mdb file in a bin.
This is just 1 solution when you use Avid. Feel free to ask if you want to create a workflow more specific.
Cheers.
B.

[nknuelle]

Thank you,

but we need to check in these files to our Avid Interplay Environment for our network editing workflow - but yes I am going to deliver these files onto a network drive into avid mediafiles folder.

[davep]

I am building a cross platform automatic ingest system based on Python (so it should be more or less platform independent). Because I wanted to "outsource" the watchfolder/monitor mechanism and transcoding to another program I chose FFAStrans, because I think it's very handy to add it to an existing workflow.

My program handles many kind of physical media such as P2, XDCAM, SD Cards, GoPro... It handles mounting and copying, adding workflow configurations for backup and so on..

nknuelle, this sounds exactly like what I've also been trying to achieve with FFAS! Do you have any further resources or information on your program? I'd love to know more...we're in a similar situation in terms of automating copying, imports and ingesting for video post production.

It sounds like you've got something good up and running?!

[nknuelle]

Hi @davep!,

nice to hear from you.

At the moment the program is still in an experimental, early development phase. If the program is ready to use I will inform you in this thread or by contacting you via PM if you like.

Greetings