Dual GPU encoding

[llittleton]

Hi,

I have a Dell R7920 with dual nvidia p4000 gpus. I can add a custom ffmpeg and choose which gpu to encode with using -gpu 0 or -gpu 1.

I am trying to figure out how to run 2 workflows with each job using a different gpu. Optimally each workflow would use 4 max jobs at the same time. I need to be able to set max jobs per workflow, not a global max jobs setting.

What happens now is if i set to max job 8, the first workflow runs all 8 on one gpu only. The second workflow does not start as the max jobs is already reached.

Is there another way to use both gpus with one workflow, having it distrubuted across both gpus?

Thanks,

Lawrence

[emcodem]

Hey littleton,

i'd propose to use two user variables, one that holds the current encoding sessions of the first gpu, another one that holds the GPU index for your encoding command. Use a conditional node to set the GPU index for encoding. Use the nvidia-smi tool to check the current encoding sessions of the first GPU in a commandline processor like that:

cmd /C ""C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi.exe" --query-gpu=encoder.stats.sessionCount --format=csv,noheader -i 0"

In the commandline processor, you can set the stdout directly to be written to your first variable %s_sessions_on_first_gpu%. In the conditional node, you can just check if the value is greater than X (e.g. 4) and if yes, then set the variable %s_gpu_index% to 1, else set it to 0.

[llittleton]

Encodem,

Thank you for your detailed reply. This works but now that I am using it I think a better way would to alternate
each gpu for each queued job. If i have only 4 jobs, they all go to gpu 0. It would work better to have 2 on each gpu.
Also, if there is a way to remove the dependency on the nvidia exe it would be helpful.

Thank you for your amazing software,

Lawrence

[Ghtais]

Hi emcodem

would you have a such brilliant trick to do the same thing without GPU ?
To be able to manage CPU resources is really missing in FFAstrans.

thx

[admin]

Hi Ghtais,

Part of the "problem" is that FFAStrans use ffmpeg executable to do its conversions and currently we don't set any CPU restrain on each decoding and encoding. So once you have kicked of this process you cannot constrain it. However, some separate workflow resource management is missing and this is something we are going to address in the future.

-steinar

[Ghtais]

However, some separate workflow resource management is missing and this is something we are going to address in the future.

that is exactly what I would need everyday, Happy to hear that !!
thank you for your work.

Bye

[emcodem]

Encodem,

This works but now that I am using it I think a better way would to alternate
each gpu for each queued job. If i have only 4 jobs, they all go to gpu 0. It would work better to have 2 on each gpu.

Hey, sorry, i have overseen your request. That is easy if you just take the count of running encoding sessions on both GPU's and cacluate the modulo of 2, it will be 0 or 1, so you have automatically the gpu number you want to assign the new job guess.

Also, if there is a way to remove the dependency on the nvidia exe it would be helpful.

Unfortunately i am not aware of any portable way to count the number of running encoding "sessions" but you need the nvidia stuff installed anyway, so why would it matter if you use the smi tools from nvidia? This is very boundto nvidia APIs, i don't think it is possible without using some tool that uses again the nvidia api to get the required count. The nearest alternative would be to use Microsoft WMI in powershell but it would at maximum be possible to get the "gpu utilisation" but pretty sure not the encoding sessions.

[emcodem]

Hey Ghtais,

would you have a such brilliant trick to do the same thing without GPU ?
To be able to manage CPU resources is really missing in FFAstrans.

I am not sure what you mean... in case of CPU encoding, the OS decides which CPU is used, there is no way to influence really; or better no need. In difference to Nvidia's encoding, we don't have a limit of "encoding sessions" on CPU. What is it exactly you want to do, is it like "pause" all other jobs when a "highest priority job comes in" or something like it?
Just to explain and stick to the topic: littleton wants to send one encding job to nvidia board 1 and the other one to nvidia board 2. The best comparison in CPU is when you have multiple Sockets like on a Server Board. Is this maybe what you want to do: send job 1 to socket 1 and job to to socket 2?

that is exactly what I would need everyday, Happy to hear that !!

just out of curiosity... resource management of limited hardware resources is a very expensive task in many ways, sometimes even a very dangerous one (e.g. stuff that could not be delivered in time because of missing hardware resources). Whats the problem about just buying enough hardware to overcome your bottlenecks? It will always be a lot cheaper than operators taking care about resource management or even programmers trying to automate what the operators do...

[Ghtais]

just out of curiosity... resource management of limited hardware resources is a very expensive task in many ways, sometimes even a very dangerous one (e.g. stuff that could not be delivered in time because of missing hardware resources). Whats the problem about just buying enough hardware to overcome your bottlenecks? It will always be a lot cheaper than operators taking care about resource management or even programmers trying to automate what the operators do...

Hi Emcodem

this is my fault, I'am not clear enough. I'am agree with you that there is no need to decide with CPU is in use for each Workflow.
My post was to highlight the idea of "separate workflow resource management"
With many differents users, some task must be done quickly and some task could take a long time. It is very important to be able to manage workflows in that way. To my mind, buy more hardware is not the key. For my workflow, even with 10 computers farms (that is unreasonable in terms of cost, electricity, air conditioning, etc ) I will have bottlenecks. It's more efficient to decide which workflow need maximum power to do the job in a very short time and witch workflow could take a long time.
For example : I have 50 To of rushes for a new show and I have to transcoded all this stuff in many format. This task is not priority and can take one month if it need. I plane that it will take about 7 days. You can say that I just have to add more hardware, but even with 4 more computers it will takes more than one day, and during this day other users will have to wait a long time just for a 5 min H264 encoding.
If I could set : for this low priority workflow take only one (or two) job at the same time and leave free all other resources for others tasks it would be perfect for this case.

Well, it is difficult for me to explain that in english (but it's a good exercise for me ). Hope it is a little bit clearer now

bye

[FranceBB]

Couldn't you just set priorities in the workflows settings, by putting the one that handles new shows as the highest (4) and the one that handles other things to 0 as the lowest? I mean, in the property settings of each workflow, you can easily set the priority. For instance, I have workflows that get things and encode them for our news channel that has the highest priority and the one that handles other thing with a lower priority so that if something happens and there's a breaking news, it's gonna handle that first.
If you want something a bit more advanced, you can set --threads in ffmpeg, however I would strongly advise you against setting it.