Limit number of jobs per workflow

[jerrysub]

Hi Team,

May you could integrate a number of job dedicated for each workflow ?

Currently , the number of jobs is set for all workflows even if priority exists , we cannot be sure that at least one job could done per workflow.

Keep working as good as you are !

[Ghtais]

Hi

Need it to if it's possible.
For example, I have a workflow that run a script and If more than one script is running in the same time it crash.
If we can enable only 1 job for this particular workflow, we can submit multiple file at the same time.

[admin]

Thanks for the feedback. I will take a look at it

-steinar

[Ghtais]

Hello

Up for this thread.
Is there any chance to have this very powerful and useful function in the next release ?

thanks.

[emcodem]

Hey @ghtais, @jerrysub

i fear even in the next release 1.2, we will not have such functionality built in to ffastrans. There are new features to controlling priorities and queues but nothing that matches exactly your usecase. Also, from my current perspective, it is a really long way until we get to a point where we can safely add the suggested feature. You know, when we add it, it must be 100% compatible to all existing priority management features which makes it not an easy thing.

Anyway, what you want is already possible when ignoring the currently existing prio features. So if you dont use prio, you can meanwhile probably go with this:
viewtopic.php?f=5&t=875&p=5392&hilit=li ... flow#p5387

[Ghtais]

Hi Emcodem

thank you for your reply and your help.
I understand it is a hard work.
When you have multi users using multi workflows, this feature is extremely useful to avoid overload.

anyway I hope it will be a challenge for the 1.3 or 1.4 release

Bye

[emcodem]

Ok, re-thinking this, what @ghtais wants is not even anyhow connected to what @jerrysub wants.

@ghtais, you are requesting to LIMIT on a per-workflow base how many jobs are running. This is no problem, you can do it today with the provided webinterface based job limiter. I'll happily implement an easy to use interface for it if you try it and confirm that it works for you.

@jerrysub you want to "make sure that at least one job per workflow is running". This seems kind of impossible to do without knowing up in front how much resources each job of the controlled wokflow will require.

For example, you have 2 workflows, workflow A is encoding HEVC, workflow B is encoding MPEG2 (XDCAMHD). One single job of workflow A will more or less use 100% of your 24 core machine while you could easily run like 4-8 paralell jobs of workflow B on the same machine.

@jerrysub could you please give us some more insight in what exactly you want to gain?

[Ghtais]

hé hé, I think it's good point !
I confirm that I want to limit the number of job running at the same time for a workflow.
Example in real life : I have 50 To of video files that I want to encode with a H264 Workflow. It could take as much as time it need. In the same time I want that other users which have more urgent task to do, can do it immediately. With the limit of one job for the H264 workflow, the CPU still available for the others workflows.

[emcodem]

Thanks for the confirmation @Ghtais
So, are you able to try the script i linked before and set up a scheduled job with it in the webinterface (run every minute)?
Just make sure you alter the workflow name in the third line to the workflow that you want to control.

Code: Select all

jobQueue["YOUR WORKFLOW NAME"] = {"max_jobs":1,"active":[],"paused":[]}

As said, if you confirm that it works for you, i'll most likely add an easy to use management interface for it in the webinterface.

[Ghtais]

thank you Emcodem

I must say that I'am not familiar with the webinterface.
I have to install it and learn how to use it first. So it could take a little bit of time