Files Find excedes character limit

[knk]

Hi guys,
I'm currently struggling with the Files Find processor working with lots of files, usually >50 per job.
When dragged from the web UI, Files Find will look for the complete UNC path + filename, often resulting in error in the command execution.
From my understanding this might be a Windows limitation preventing commands from being > 8191 characters long.
Since Files Find has it's own code, is there any way I can eventually manipulate it to avoid such a long command, for example, ignoring the file path in the json array? Something like that... just so I can work with more files at a time.
Thanks! All the best

[emcodem]

Hi knk,

not sure if i 100% understood your problem (a logfile would be helpful) but from feeling you cannot change the affected part of the code.
If you just want to make sure there are no huge commandlines involved, you can set s_source variable to something short (before executing the files find proc). The findfiles processor will be called from ffastrans like:

Code: Select all

 “script_interpreter” “proc_script” “%temp_path_to_jobjson%” “%s_source%“

As seen here: https://ffastrans.com/wiki/doku.php?id= ... processors

Where script_interpreter and proc script is a fixed value from the processor node.json definition, jobjson path links to a file in ffastrans db and s_source is the only really dynamic value (i regret that i added this for convenience reasons).

[knk]

Hi emcodem, thanks for the reply.
Here's an example from the log:

When pulling, say, 50 files from web UI the Files Find executor fails with:

C:\windows\system32\cmd.exe /c ""\\FFASTrans\processors\plugin_nodes\custom_nodes\Files Find\bin\processor.bat" "\\FFASTrans\Cache\\20221107-0825-2940-6f07-89b799e0f26f\20231031-1134-4714-0a13-142f35fb7b43\1-14-14~231031113452604~15132~20231010-1212-0548-40f2-34939b166129~plugin_htmlgui.inputs.json" "["\\\\QUITE-LONG-UNC-PATH-WITH-SEVERAL-FOLDERS\\PROBABLY-A-LONG-FILENAME-1.MP4","\\\\QUITE-LONG-UNC-PATH-WITH-SEVERAL-FOLDERS\\PROBABLY-A-LONG-FILENAME-2.MP4"]""

etc... x50 files

it fails with:

"Process exited with error code: 1 (The command line is too long.)

[emcodem]

thanks! so it is just as i thought, you need to clean s_source before calling files find processor.
It is my fault, i should have not put s_source on the commandline ^^

If you run 1.3.1 and you like, you can also try this updated version of ffastrans processors.a3x:

processors.a3x

(802.95 KiB) Downloaded 164 times

It should work right away for you i assume.
Just rename the existing FFAStrans\Processors\processors.a3x to .old and copy the new one there, no need to restart anything.

[knk]

Hello emcodem,

Thanks for the new version for the processors, but the result is still the same.

Tried to print to file the s_source so I could try to clean it up and pick it up later in $read, but I can't make it work...

[emcodem]

Hey @knk,

so i just tried it out and i can confirm that my uploaded processors.a3x will not produce "commandline too long" errors while the original one does. How i test is to select hundreds of files in the webui job submitter and start a job with webui_stitch enabled. By default it will end with an error "commandline too long". Then i replace the processors.a3x and resubmit the same job and it will succeed because the list of files is not submitted to the find files processor.

Tried to print to file the s_source so I could try to clean it up and pick it up later in $read, but I can't make it work...

Writing and reading from file might be tricky, you can instead just create a new variable like s_original_source and use a populate processor to set s_original_source to s_source and s_source to something different (but you may never set s_source to empty value "" because it will end the job immediately)

Code: Select all

s_original_source = %s_source%
s_source = "reset"

But this all is kind of tricky, i really think working with the new processors.a3x should do the trick for you.

[knk]

Hi emcodem!
I even deleted my older processes.a3x an re-downloaded the new file you sent, and restarted everything just in case... still it presents the same structure as before “script_interpreter” “proc_script” “%temp_path_to_jobjson%” “%s_source%“
Any idea why I'm getting different results?
Thanks for your help!

image_2023-11-06_140116392.png (37.73 KiB) Viewed 4147 times

[emcodem]

Sorry for the delay @knk maybe i tested wrong and there were 2 places that i needed to change instead of only one.
Here one more try.

processors.a3x

(802.93 KiB) Downloaded 205 times

[knk]

Hi emcodem,
It's working perfectly, you guys are the best!
Thank you so much!
Best regards

[emcodem]

Nice, thanks for letting me know