Page 1 of 2

Files Find excedes character limit

Posted: Tue Oct 31, 2023 1:27 pm
by knk
Hi guys,
I'm currently struggling with the Files Find processor working with lots of files, usually >50 per job.
When dragged from the web UI, Files Find will look for the complete UNC path + filename, often resulting in error in the command execution.
From my understanding this might be a Windows limitation preventing commands from being > 8191 characters long.
Since Files Find has it's own code, is there any way I can eventually manipulate it to avoid such a long command, for example, ignoring the file path in the json array? Something like that... just so I can work with more files at a time.
Thanks! All the best

Re: Files Find excedes character limit

Posted: Tue Oct 31, 2023 2:07 pm
by emcodem
Hi knk,

not sure if i 100% understood your problem (a logfile would be helpful) but from feeling you cannot change the affected part of the code.
If you just want to make sure there are no huge commandlines involved, you can set s_source variable to something short (before executing the files find proc). The findfiles processor will be called from ffastrans like:

Code: Select all

 “script_interpreter” “proc_script” “%temp_path_to_jobjson%” “%s_source%“
As seen here: https://ffastrans.com/wiki/doku.php?id= ... processors

Where script_interpreter and proc script is a fixed value from the processor node.json definition, jobjson path links to a file in ffastrans db and s_source is the only really dynamic value (i regret that i added this for convenience reasons).

Re: Files Find excedes character limit

Posted: Tue Oct 31, 2023 2:30 pm
by knk
Hi emcodem, thanks for the reply.
Here's an example from the log:

When pulling, say, 50 files from web UI the Files Find executor fails with:

C:\windows\system32\cmd.exe /c ""\\FFASTrans\processors\plugin_nodes\custom_nodes\Files Find\bin\processor.bat" "\\FFASTrans\Cache\\20221107-0825-2940-6f07-89b799e0f26f\20231031-1134-4714-0a13-142f35fb7b43\1-14-14~231031113452604~15132~20231010-1212-0548-40f2-34939b166129~plugin_htmlgui.inputs.json" "["\\\\QUITE-LONG-UNC-PATH-WITH-SEVERAL-FOLDERS\\PROBABLY-A-LONG-FILENAME-1.MP4","\\\\QUITE-LONG-UNC-PATH-WITH-SEVERAL-FOLDERS\\PROBABLY-A-LONG-FILENAME-2.MP4"]""

etc... x50 files

it fails with:

"Process exited with error code: 1 (The command line is too long.)

Re: Files Find excedes character limit

Posted: Tue Oct 31, 2023 3:20 pm
by emcodem
thanks! so it is just as i thought, you need to clean s_source before calling files find processor.
It is my fault, i should have not put s_source on the commandline ^^

If you run 1.3.1 and you like, you can also try this updated version of ffastrans processors.a3x:
processors.a3x
(802.95 KiB) Downloaded 165 times
It should work right away for you i assume.
Just rename the existing FFAStrans\Processors\processors.a3x to .old and copy the new one there, no need to restart anything.

Re: Files Find excedes character limit

Posted: Thu Nov 02, 2023 5:34 pm
by knk
Hello emcodem,

Thanks for the new version for the processors, but the result is still the same.

Tried to print to file the s_source so I could try to clean it up and pick it up later in $read, but I can't make it work...

Re: Files Find excedes character limit

Posted: Fri Nov 03, 2023 8:29 am
by emcodem
Hey @knk,

so i just tried it out and i can confirm that my uploaded processors.a3x will not produce "commandline too long" errors while the original one does. How i test is to select hundreds of files in the webui job submitter and start a job with webui_stitch enabled. By default it will end with an error "commandline too long". Then i replace the processors.a3x and resubmit the same job and it will succeed because the list of files is not submitted to the find files processor.
knk wrote: Thu Nov 02, 2023 5:34 pm Tried to print to file the s_source so I could try to clean it up and pick it up later in $read, but I can't make it work...
Writing and reading from file might be tricky, you can instead just create a new variable like s_original_source and use a populate processor to set s_original_source to s_source and s_source to something different (but you may never set s_source to empty value "" because it will end the job immediately)

Code: Select all

s_original_source = %s_source%
s_source = "reset"
But this all is kind of tricky, i really think working with the new processors.a3x should do the trick for you.

Re: Files Find excedes character limit

Posted: Mon Nov 06, 2023 2:01 pm
by knk
Hi emcodem!
I even deleted my older processes.a3x an re-downloaded the new file you sent, and restarted everything just in case... still it presents the same structure as before “script_interpreter” “proc_script” “%temp_path_to_jobjson%” “%s_source%
Any idea why I'm getting different results?
Thanks for your help!
image_2023-11-06_140116392.png
image_2023-11-06_140116392.png (37.73 KiB) Viewed 4158 times

Re: Files Find excedes character limit

Posted: Fri Nov 10, 2023 3:32 pm
by emcodem
Sorry for the delay @knk maybe i tested wrong and there were 2 places that i needed to change instead of only one.
Here one more try.
processors.a3x
(802.93 KiB) Downloaded 206 times

Re: Files Find excedes character limit

Posted: Fri Nov 10, 2023 4:45 pm
by knk
Hi emcodem,
It's working perfectly, you guys are the best!
Thank you so much!
Best regards

Re: Files Find excedes character limit

Posted: Sat Nov 11, 2023 11:27 am
by emcodem
Nice, thanks for letting me know :D