Concatenate

[taner]

Many thanks!
I'll try it and give you feedback.

[emcodem]

Please know that i just updated the Files Find plugin because recently all my processor.exe applications are being detected as virus (i use some exe/script packing method that obviously generates a signature that A/V dont like). So to make your life easy with files find, just re-download it from here:

https://ffastrans.com/wiki/lib/exe/fetc ... s_find.zip

Also, i generated this workflow that works generically for different file formats in the input. because @rlima asked for it

Usage: same as the other workflow, just drop a "link to a folder (where all the files are)" into the watchfolder.

emcodem_concat_different_formats.json

(21.43 KiB) Downloaded 179 times

The strategy is that first each and every file is encoded in the lower branch (into work_dir) and the upper branch waits for them to finish. Then the upper branch executes a simple concat command and delivers the finally resulting concatted file directly to the target folder.

• Set the delivery folder and format by configuring the values in first "populate processor".

• The video and audio format is defined by the encoding node that you use. E.g. just Change the XDCAMHD Encoder to h264 if you want h264.

• The order of files should by by name, this is set in the second files find processor in the top branch.

• It is best if your linked folder contains only the files to be stitched and no xml or other junk files. If there are other files, you have to configure the first files find processor to include only the files of interest.

[emcodem]

So, as the title of this question is pretty straight forward, i decided to use this post as a reference for concat related stuff.

To cover yet another concat usecase, i created this general purpose ffastrans job submitter windows forms application:

https://github.com/emcodem/ffastrans_wi ... /tag/0.0.1

Make sure you do read the readme file here:
https://github.com/emcodem/ffastrans_wi ... bsubmitter

It is designed to run on clients and allow users to compose their source file list, select a ffastrans workflow and submit the job directly to ffastrans API.
Clients must have network access to ffastrans API port for it to work.

jobsubmitter.jpg (89.88 KiB) Viewed 3408 times

The configuration allows some smart stuff like adding multiple workflows to a user dropdown list and also submit variables to the workflow. E.g. i use the local client configuration here to define the final output directory. \\localhost\c$\temp is just used to show off how to enter a valid UNC path. Of course you can just type any other path like D:\transcoding\stitched

tomlconfig.png (110.42 KiB) Viewed 3407 times

Note, as we submit a json array as s_source to the workflow, it has to be prepared for it, e.g. i use s_source in the contained example workflow only for driving the foreach and wait for files processors.


I'll eventually also update the webinterface with the functionality to submit multiple files at once to a single job and post here when it's done.

[Silicon]

Hi emcodem
Thanks for excellent workflow you have created ... coincidentally just on time when I need it
I would like to suggest two improvements (nice-to-have features):
1. add a text field to the "FFAStrans_Job_Submitter" window that allows the user to enter a filename of the concatenated file
2. if possible the timecode of concatenated file should start at value taken from first source file

Both features could be designed as optional:
ad 1) if text field is left empty, use current way of filename construction
ad 2) ideally ad a checkbox or dropbox that allows the user to decide whether the TC of concatenated file will be zero-based or first_source_file-based

Looking forward to see your comments on my suggestions.

[emcodem]

Hi @Silicon

to be honest, i didnt really plan to go anywhere with this windows form as this type of job submission is limited to very smallish scenarios and not really compatible to enterprises needs. You know, alone the requirement to connect directly to ffastrans api using plain http is something that would not stand security requirements for long time.
I'll add all the needed stuff for this to the webinterface (well actually it is just one checkbox "submit all files as single job"). This way, we have much more control and flexibility and futureproof-ness
You think your usecase will work with webint too?

[Silicon]

Hi @emcodem
Thanks for your quick response.
I hope that "submit all files as single job via WebInterface" approach has a potential to fullfill our needs.

[emcodem]

OK @Silicon, so besides a few other major improvements, the functionality to submit "all files to one job" was added in webint now.
Also, we have now more options to control how the job submission "form" looks like on webinterface.

New Webinterface version https://github.com/emcodem/ffastrans_we ... ag/1.3.9.0

To mention is what triggers webinterface to submit all files to a single job instead of creating one job for each file. It is the bare existence of the word "webui_stitch" in the workflow description.

Ready to use example workflow based on the samples before:

Please note you must install the files find processor to use this workflow.
Plugin:
https://ffastrans.com/wiki/lib/exe/fetc ... s_find.zip
Workflow:

emcodem_webuisubmit_concat_different_formats.json

(19.09 KiB) Downloaded 169 times

webint1.png (84.28 KiB) Viewed 3231 times

@Silicon Sorry that i left out the timecode wishes for now, i was hoping you could give it a try and tell me if it is easy/hard to exand this workflow example by the timecode requirement you have. Basically how you can do it is to open the variable editor and change the description of %s_webui_new_select% to your needs. You could for example alter the dropdown values from \\\\localhost... to "-timecode 00:00:00:00" and "" (so nothing in case you want to start with tc of first file) and just use the variable %s_webui_new_select% in the final command executor upper branch, insert after -map 0.

[Silicon]

Hi @emcodem
Thank's a lot for new cool Webinterface features!

I'm testing the webui_stich workflow and I'm facing following issue: whenever the lenght of %s_source% variable / array (containing list of source files) exceeds the 250 chars, the workflow fails.
In this case the %s_source% variable / array start with this string \\?\UNC"\\\\...
I'll PM you complete logs.

As for my "timecode wish" ... thanks for hint. I have implemented it to my workflow (which is an extension of your sample workflow ).

BR,
Silicon

[emcodem]

Hey Silicon,

my bad, i didnt really test with lots of files, only with two. Turns out that ffastrans modifies the value of s_source and adds the UNC stuff string in the case you describe.
That basically turns the whole usage of s_source for submitting arrays of files into it totally unuseable and we'd need to use some user_variable instead. This again makes steinar really happy because he never liked that i use s_source for all kinds of stuff instead of a file actually
Well, i guess we will continue our internal discussions about that and add more support for lists instead of single files natively in ffastrans.

Until then, we can easily workaround the crippling that ffastrans did to our input list and undo it by using a populate processor at workflow start.
But, note that we need to store the restored value (the json array) in another variable than s_source because ffastrans would immediately redo the change.

Code: Select all

$replace('%s_source%','\\?\UNC','[')

In a first test it worked:

emcodem_webuisubmit_concat_different_formats.json

(19.09 KiB) Downloaded 174 times

For now, i will rework the affected workflows here accordingly and just replace the existing download links so users don't download misleading stuff. Also, i'll consider altering the webui and introducing a "fixed name" user_variable where json variables can be retrived from. This way i could write the file count or such into s_source, leading to a better user experience i believe... I'll think about that.

[emcodem]

OK short talk with steinar reveals that issue was already fixed in the most recent source code and it will be part of the upcoming bugfix release ffastrans 1.3.1. The reason for the fix was exactly that usecase, support JSON array in s_source.
Anyway, i am pretty sure you can safely apply my approach about undoing the replacement using

Code: Select all

$replace('%s_source%','\\?\UNC','[')

Because even if the issue is fixed, your workflow would still continue to work and overall this method is very safe as it just replaces \\?\UNC, which will not occur in paths usually anyways, except ffastrans did replace the first character of s_source by \\?\UNC