set order in which folders are processed
set order in which folders are processed
I have a workflow with multible folder monitor elements. Is it possible to decide in which order they are processed? I would like that one folder only is processed if there is nothing to do in another folder which is more important.
Re: set order in which folders are processed
I think the best approach would be to duplicate this workflow and separate the two watchfolders.
Then set the priority of the not-so-important one to 0 (lowest) and the one of the very important one to 4 (highest). (For the records, there's another priority, 5, for manually submitted files from the GUI, but we're not gonna talk about it right now as it's a special case). Those numbers (0 and 4) aren't casually chosen 'cause the thing is that while a file with priority 0 is being processed, if a file with priority 4 is picked up and there aren't enough resources available, FFAStrans will pause the job with priority 0, kick off the jobs with priority 4 and then, only when the priority 4 job is over, it will resume the priority 0 one. This is exactly what you want and it will be handled automatically by the FFAStrans priority system.
Then set the priority of the not-so-important one to 0 (lowest) and the one of the very important one to 4 (highest). (For the records, there's another priority, 5, for manually submitted files from the GUI, but we're not gonna talk about it right now as it's a special case). Those numbers (0 and 4) aren't casually chosen 'cause the thing is that while a file with priority 0 is being processed, if a file with priority 4 is picked up and there aren't enough resources available, FFAStrans will pause the job with priority 0, kick off the jobs with priority 4 and then, only when the priority 4 job is over, it will resume the priority 0 one. This is exactly what you want and it will be handled automatically by the FFAStrans priority system.
Re: set order in which folders are processed
The problem with that approach is that I cannot limit the global active jobs, only the jobs per priority class. I would like that there are always just 2 active jobs, which both are supposed to work on the more important folder if there is something to do in it. I don't like if there are paused jobs because sometimes jobs also get stuck for no reason and then I cannot destinguish what's happening.
Re: set order in which folders are processed
Hi Luigi,
My idea would be:
Add a Monitor-Node. In the selected Folder of this monitor-node set a Text-file containig just a 1. This file will be read-out by a "$Read()-Function" in a populate variables-node. When FFAStrans crawls through all your Monitor-Folders it will find this txt-File. Just program a few nodes to test if all other Folders are empty (if exist...). If empty, read out the .txt-file, add 1, write the txt.file. Each iteration of Sleep-Timer of the workflow the number will be increased by 1. If Number of iterations of sleep-timer eyxceeds yout margin e.g.5 (If $Read(.txt) = 5 then...) start a second workflow with the "Insert Workflow"-Node just uunder the "insert Processor"-option.
just my two cents...
My idea would be:
Add a Monitor-Node. In the selected Folder of this monitor-node set a Text-file containig just a 1. This file will be read-out by a "$Read()-Function" in a populate variables-node. When FFAStrans crawls through all your Monitor-Folders it will find this txt-File. Just program a few nodes to test if all other Folders are empty (if exist...). If empty, read out the .txt-file, add 1, write the txt.file. Each iteration of Sleep-Timer of the workflow the number will be increased by 1. If Number of iterations of sleep-timer eyxceeds yout margin e.g.5 (If $Read(.txt) = 5 then...) start a second workflow with the "Insert Workflow"-Node just uunder the "insert Processor"-option.
just my two cents...
Re: set order in which folders are processed
I honestly don't understand any of this. I'm no expert in such things. I just found and use FFAStrans because it's the first option that can automaticly encode new videos that land in folders. But I think even if I understood what you're explaining it wouldn't quite fit to what I'm asking for, because the folder isn't empty after it's done, because I set up that the source files only get deleted after one hour.
Re: set order in which folders are processed
Hi Lugi,
sorry to be so complex. Your approach is not really built into FFAStryns right now. I can understand, that wiggling around with all these possibilities FFAStrans gives you is not easy to understand when you are not in too deep with all that stuff. But let´s try to get a solution!
Maybe it is a good idea to get your Stucked-Workflow-problem under control, so you can sleep well when a job is paused for any reason.
Then it would be great if you can post your existing workflow so that we can have a look especially on the source-files that should be deleted after one hour - maybe you can explain this a little more in depth. Normally I would delete a Sourcefile after the Workflow is done or move it to a "completed"-folder.
So, step-for-step I´m quite sure we can help you.
so long,
Thomas
sorry to be so complex. Your approach is not really built into FFAStryns right now. I can understand, that wiggling around with all these possibilities FFAStrans gives you is not easy to understand when you are not in too deep with all that stuff. But let´s try to get a solution!
Maybe it is a good idea to get your Stucked-Workflow-problem under control, so you can sleep well when a job is paused for any reason.
Then it would be great if you can post your existing workflow so that we can have a look especially on the source-files that should be deleted after one hour - maybe you can explain this a little more in depth. Normally I would delete a Sourcefile after the Workflow is done or move it to a "completed"-folder.
So, step-for-step I´m quite sure we can help you.
so long,
Thomas
Re: set order in which folders are processed
FFAStrans does actively pause jobs? i know we discussed that internally but i doubt that this was implemented, at least i have not tested that ever 
One other way for @Lugi is to just use 2 watchfolders, one with high and one with low priority. In this case, you will end up with 4 running jobs but the 2 high priority jobs will get more CPU resources from Windows because FFAStrans raises their process priority. The priority does however not influence harddisk, network and GPU resources, only CPU resources are managed.
However, high priority jobs can also potentially make the Windows Userinterface very sluggish because they run with higher priority than windows explorer. This is not a problem on a server but if you use the Windows explorer while ffastrans does its jobs then it is a problem. So maybe it is best to go with "normal" prio and "low" prio instead of high and low
And last but not least, @ThomasM is very correct, we don't like hanging jobs at all. How often does that happen for you and do you think it is ffastrans fault that they are hanging or some different reason?
One other way for @Lugi is to just use 2 watchfolders, one with high and one with low priority. In this case, you will end up with 4 running jobs but the 2 high priority jobs will get more CPU resources from Windows because FFAStrans raises their process priority. The priority does however not influence harddisk, network and GPU resources, only CPU resources are managed.
However, high priority jobs can also potentially make the Windows Userinterface very sluggish because they run with higher priority than windows explorer. This is not a problem on a server but if you use the Windows explorer while ffastrans does its jobs then it is a problem. So maybe it is best to go with "normal" prio and "low" prio instead of high and low
And last but not least, @ThomasM is very correct, we don't like hanging jobs at all. How often does that happen for you and do you think it is ffastrans fault that they are hanging or some different reason?
emcodem, wrapping since 2009 you got the rhyme?
Re: set order in which folders are processed
First - thanks to @emcodem for clearing the basics. Very appreciated!
Hi Lugi,
I made some refinements. To go on elabotrating a workflow for your needs, maybe you can add some information:
1. You want to run max two active jobs - right?
2. Are the two Watchfolders - let them call us HighPrio-WF and LowPrio-WF - on the same drive / root?
3. Are the destinations for HighPrio-transcodings and LowPrio-Transcodings the same?
4. Are the Source-Files going to be deleted or ccopied / moved after transcoding? If the latter, to different folders?
This information can help to design a proper workflow for your needs.
Hi Lugi,
I made some refinements. To go on elabotrating a workflow for your needs, maybe you can add some information:
1. You want to run max two active jobs - right?
2. Are the two Watchfolders - let them call us HighPrio-WF and LowPrio-WF - on the same drive / root?
3. Are the destinations for HighPrio-transcodings and LowPrio-Transcodings the same?
4. Are the Source-Files going to be deleted or ccopied / moved after transcoding? If the latter, to different folders?
This information can help to design a proper workflow for your needs.
Re: set order in which folders are processed
Yep, but only if you have a priority 0 job, you finish the available slots and a priority 4 job starts. In all other cases, it won't. So, priority 1 to 4 are just for queued tickets and in which order they're executed so that once a ticket is picked up, there's no way of stopping it until it ends and the next one is picked up. On the other hand, if you have a priority 0 ticket running and you have a priority 4 ticket queued waiting for resources, then FFAStrans can and will pause the priority 0 job, make the priority 4 go through by putting it in running and then once that is over resume the priority 0 job. Keep in mind that while a job is "paused" it will still keep the process open with the RAM allocated and all the files written in the shared cache, however it will not use CPU resources. This was implemented in 2020 by Grandmaster Steinar himself after my farm was about to explode as we were remastering a lot of BetaCAM tapes that would take forever to encode (thanks mainly to QTGMC) and we were receiving files uploaded from everywhere that needed to be processed for Sky News and Sky Sports with very short turnaround.
Yeah yeah I basically begged Grandmaster back then and when he saw my botched attempt (I still suck in AutoIT but I sucked waaaaaaaaaaaaaay more back then) he implemented the whole priority system flawlessly himself. He is Grandmaster for a reason after all. I've been relying on that ever since and I still rely on that today, 4 years later, although lots of things have been offloaded to the cloud (FFAStrans running on AWS EC2) so it's not quite the issue it used to be.
Re: set order in which folders are processed
Guys, FFAStrans auto pausing (in the middle of processing) is turned off by defalut so for "regular" users it's not enabled.
Lugi, we never like stuck jobs but unfortunately we see it from time to time. And depending on your workflow, it's not trivial to pinpoint the cause. So the best you can do for now is to enable full log level (under the "Special" tabs) in the workflows where you get stuck jobs, and wait for a stuck job again to occur. Collect the log and send it.
Regarding your request it's not possible the way you want it with the current design. FFAStrans operates with equal amount of jobs per priority class but it would potentially be possible to go back to the earlier implementation with a global hard limit as an option. Would that help your case? I can also imagine a way to do some kind of folder monitor order but that will take some more design work.
-steinar
Lugi, we never like stuck jobs but unfortunately we see it from time to time. And depending on your workflow, it's not trivial to pinpoint the cause. So the best you can do for now is to enable full log level (under the "Special" tabs) in the workflows where you get stuck jobs, and wait for a stuck job again to occur. Collect the log and send it.
Regarding your request it's not possible the way you want it with the current design. FFAStrans operates with equal amount of jobs per priority class but it would potentially be possible to go back to the earlier implementation with a global hard limit as an option. Would that help your case? I can also imagine a way to do some kind of folder monitor order but that will take some more design work.
-steinar