Watchfolders work, but submitting via API (or manual in GUI) doesn't

[storm]

Hi,

I've upgraded to FFAStrans 1.3.0.2 because of the Sequence Monitor bug.
We use FFAStrans for transcoding EXR based sequences to delivery formats.

The watchfolders work exactly as they should. However I want to integrate FFAStrans in our pipeline and submit through API.
This way I can just send the path of the EXR sequence to FFAStrans without the need for unneccesary data transfers.

However when I submit a job to the API (or manually by submitting to the node in the GUI) I get the following error: H.264@VTRANS-FFAS-01: Validate: Permission denied.

This is weird, since I am submitting the same path as the watchfolder (which does work). I am accessing the media over UNC (Windows Server 2k19). Also copying the data to the local disk, and submitting via API results in the same error.

The FFAStrans host is running Windows 10 20H2.

Running FFAStrans in Administrator mode doesn't help either.

Any ideas?
TIA.

Tristan

[emcodem]

Aye Tristan, welcome to the forum and thank you for using ffastrans!
My guess is that this is some kind of setup error because if it was really general that manual submits don't work, we wouldnt have released it


please grab and zip the corresponding logs, best if we get the job log from a working vs a non working job.
Processors\db\cache\api_log
Processors\db\cache\jobs\%your_job_guid%
zip them and send them to us, you can use pm if you fear about file paths getting public or such.

[storm]

Hi Emcodem,

Thanks for your quick reply.
I have attached the logs for three submits.

Watchfolder - working correctly
GUI Workflow Manager - throwing error
API - throwing error

I've also attached the API logs.

Thanks again.

[emcodem]

oh sorry, i have overseen the fact that you work with image sequences. Thanks for the logs, they made it easy for me to find a solution for you.

When working via API submit, the "monitor" processor in the workflow is actually useless or better it is kind of bypassed. So all functionality of monitor processors must be done inside your 3rdparty program's logic, e.g. waiting until the image sequence is complete.
What the image sequence monitor also does is to set the most important variable s_source to a single filename of your sequence, e.g.

Code: Select all

%s_source% "F:\29_FFASTRANS\_input\ImageSeq_H264_1080p_8Mbit_25fps\working\o_dpvh2_001_0020_v018.0989.exr"

and also it sets 2 internal variables (which you cannot access) to trigger a recalculation of the source filename in all encoder processors. In the working log, you'll see a line that contains _SetSequenceString().
_SetSequenceString is an internal function that causes a temporary recalculation of the source filename to this:

Code: Select all

F:\29_FFASTRANS\_input\ImageSeq_H264_1080p_8Mbit_25fps\working\o_dpvh2_001_0020_v018.%04d.exr

But, this is only done internally in the encoder processor, actually s_source still represents the vaild filename from above.
This is what ffmpeg needs as input in order to work with the image sequence.

I write this in that great detail because there are 2 ways i see you can deal with the situation:

Option 1) you set the sourcefile in the API call (or manual submit- from input) to be the exact input that ffmpeg needs to work with: F:\29_FFASTRANS\_input\ImageSeq_H264_1080p_8Mbit_25fps\working\o_dpvh2_001_0020_v018.%04d.exr

This is a quick way and it should work for simple workflows. But some builtin variables might be empty because the actual source file is not a real file, so lots of internal steps that are done automatically (mediainfo, ffprobe) fail.

Option 2) you set the sourcefile in the api call to be the first existing file of the image sequence, for example test_0000j.exr but in your workflow, you use a populate processor right before the encoder processor and re-set s_source to be test_%04d.exr. After the encoder, you can optionally set s_source back to s_pickup_file.

Another option might be to use a custom avs script but i don't want to get this too complicated hehe.

Both solutions are not 100% perfect but they should produce some output file. The encoder processors should be able to produce what you like but we still miss the pre-input ffmpeg parameters: -framerate 25 -field_order progressive -start_number 0989
I'm not sure if framerate and field order is neccessary (let's hope ffmpeg doesnt do framerate conversion for you) but the start_number is something you cannot set at all. Except of course you work with a cmd processor instead of the encoder processor (which i would do), in that case anything is possible but the workflow becomes less flexible.

IMHO The correct solution would be if we were able to set the internal variables $i_JOB_SEQ_END And $i_JOB_SEQ_DIGITS along with the API call, maybe this is a feature we see in the future @admin? Maybe this is not needed in case we get the feature to modify the calculated encoding command before it is executed, who knows?