After several days of working with the new version 1.4.0.1, I encountered a problem (bug) with monitoring growing files continuously - the workflow is trying to process the file while it is being recorded using OBS or vMix, without waiting until the write is complete and the file is finalized!
And this visible in the log - error "Moov atom not found".
But it is possible that this is just a problem with the log display (full log view) in Status Monitor & Webinterface
File successfully processed after record was stopped.
2024-02-09_Remote Desktop Manager-002038.png (106.28 KiB) Viewed 9144 times
2024-02-09_Remote Desktop Manager-002040.png (32.43 KiB) Viewed 9132 times
check growing continuously is a very very special option, it is designed to start a new job whenever the input file changed so it looks like it works perfectly for you
Check growing "Once" is what you normally want. If this does not work because the input file is changing size too slow, we'd need to add the waiting inside the workflow actually... let me know if "once" works for you.
On the previous version with the continuously setting, everything worked fine and there was no such behavior...
That's why I thought it was issue, but it turned out to be a feature!
Well actually continous option needs to wait until growing finished at least one time too but it looks like in your case t appears to not have done that.
By the way, you seem to have set the workflow option "sleep time" to 10 seconds which is pretty aggressive for a growing workflow, maybe better choose 30+ seconds. Sleep time is the time that we wait to check 2 times file size in order to determine if the file is still growing.
Of course it is the wrong option for you but it is a mystery to me why it did not even wait "one time" for the growing to finish. Maybe @admin wants to wrap his head around that?
emcodem wrote: ↑Sat Feb 10, 2024 3:54 pm
By the way, you seem to have set the workflow option "sleep time" to 10 seconds which is pretty aggressive for a growing workflow, maybe better choose 30+ seconds.
I set the "sleep time" to 10 sec. only for the test and screenshot, by default for this WF the sleep time is set to 60 sec, but the situation has not changed while "check continuously" is on.
emcodem wrote: ↑Sat Feb 10, 2024 3:54 pm
Sleep time is the time that we wait to check 2 times file size in order to determine if the file is still growing.
I will re-check, but as i remember i tested "growing option" on a workflow where dummy-file is created by recorder, size of this file don't change, because recorder save all to temp-file, and only after finishing record it save all to dummy file. And the "check growing continuously option" works perfectly in this case - it waits until this file is busy with the recorder process, even though the size of this file does not change.
Yep, just tested on v1.3.1, WF: Monitor Folder (accept any files, check growing files once & continuously) -> Deliver Folder
In the Edge started downloading a large file and paused the download, file size not growing. But the file not been processed by WF, it wait, and only after i ended downloading, file was finalized it was processed. Also i tried copy paused file - and copy was processed!
In v1.4.0.1 paused file processed with both options once & continuously.
@artjuice @railio the problem with continuous should be fixed in 1.4.0.4 since this weekend. But remember, you will usually not need this option, the use case for it is very very special.
Have a new problem with watchfolders
I updated to a new version 1.4.0.4 and now some files are ignored. WF sometimes don't take large files. I try to change "check growing files once & continuously", and seems It doesn't matter! Sleep timer set to 30, i also try to change it.
But if i rename such files - WF takes them and processes.
I will test 1.4.0.5 now, and I will let you know if the problem disappears or remains.
Hi @artjuice
Yeah i fear we have a pretty fundamental change in the code for watchfolders that could cause unexpected behaviour. Will talk with steinar and work out strategies how to debug such cases. The fundamental change i talk about is for optimizing performance, large watchfolders would take like 99% less resources (increase system performance) with this optimisation but unfortunately at the cost of adding more complexity.
there is processors/db/configs/ffastrans.json where you could set the entry "extra_log" to "true", if you then manage to reproduce the issue and tell us which filename is not picked up correctly AND send us the logs from processors/db/cache/def_log, we might get a clue about what is going on
