GPU USAGE

[igorlima]

Hello!

I've recently setted the cpu roof to 100%, so than the workflow works faster and
efficiently, and i'm able to know if the transcoding is going on or not (based on the performance info at the task manager). However, it would be good if the GPU roof was setable either. Is this possible?

Best,

Igor.

[emcodem]

Hey igorlima,

welcome to the Forum and thank you for using FFAStrans!
So the CPU roof doenst really make anything faster, all it does is to make one job wait for the other in case there is less than the set CPU % available. So it has nothing to do with encoding at all and it does not limit or speed up a single running encoding job.
Regarding GPU, as FFAStrans has nothing to do with GPU at all currently, such a setting would not really be helpful, would it?
What would you expect from such a functionality?

[no_connection]

Lest say you are calling any GAN or similar upscaling you want to wait until GPU resources isn't at 100% to submit another job.
So it makes sense to implement such a thing since ppl are going to want a better way to upscale or process large amount of machine learning based things involving video.
Especially if FFAStrans is used to manage a large render farm to split up the very computationally intensive work.

One thing that makes it somewhat difficult is that GPU usage is the GPU core load and don't take into account it might be downclocked at idle and still be at 70%. But most processing pin it at 100% so a narrower window can be set.

[emcodem]

I see many problems like providing options for "which" gpu's to consider (e.g. intel and nvidia in the same machine) and how to retrieve the actual utilisation values in a general and portable way. Besides that, some might not be interested in the "gpu utilisation" but more in the "encoding slots used" or similar.
Anyway, it's a worlflow engine for this exact reason: workflows can become complex and the engine allows you basically to deal with any problem.

So, let me know if you got a specific usecase and i can post some example workflow that shows how i would solve it.

[no_connection]

The usecase I would be interested in is upscale that uses GPU. But yea I have no good answer how it could or should be done. Esp as resources could be used up by other processes like manual rendering in other apps.

Also with ARC GPU and AV1 for example I would expect that to have a roof as well.

Just thinking out loud here.

[emcodem]

Thats the thing... if FFAStrans had any native feature that uses any GPU stuff it would be kind of straight forward to monitor the needed resources.
E.g. If we introduce a processor that uses a special GPU feature (instead of using the raw processor power) like H.26x encoding and in this processor, you have to actually set the type of GPU it shall use and the codec, then we could easily monitor the stuff too because we know what's used.

After all, FFAStrans can only natively take care about stuff that it natively does. But again, from design it is open for the user to put any logic that he needs into the workflow.

[3dsasha]

I am solving task like this GPU used workflow. I have a workflow tuned to use GPU exactly to 100% by FFAStrans logic nodes. But now I trying to limit that workflow to only one running job instance for every host. Would someone help me to show right way? Now I try parse db\tickets\running\*. json for wf and host IDs - wait if all are true, continue if all are false, but I have a lot of bugs. What is correct way to know that another job of current workflow on current host is running, or limit wf to only one local slot? Thanks!

[emcodem]

... but I have a lot of bugs.

Yeah we need to find a very simple setup that allows fast and easy debugging and testing, otherwise it takes ages until it runs stable.
So, ffastrans does not yet provide any way to influence the job pickup other than the slots and cpu limit settings.

You can potentially just set max jobs to 1 and a very low cpu limit, e.g. 3%. But that also means that your farm is more or less dedicated to do only this workflow.

Also, you can try this approach that i prepared for you. It is showcasing something that i proposed internally a long time ago: allow the user to provide a custom override functionality regarding picking up new tickets.
The idea is to allow the user to override the CPU business check by providing his own number instead. Listen:

For ffastrans version 1.3.1, you can download my modified version of exe_manager.exe:
https://1drv.ms/u/s!AkS-A9Jqq09Fg2CW5nI ... K?e=Jn48ud

Please rename existing Processors/exe_manager.exe to .old and copy my version to the same location.

I added the following functionality:

-) if there is a new job pending execution, all farm nodes will execute hook_business.bat (in same folder as fastrans.exe) BEFORE grabbing the job
-) hook_business.bat (same folder as ffastrans.exe) is YOUR way to indicate if the machine is currently busy. It does that by echo'ing the current business in percent (e.g. 99%). How you get to that number is something i don't care about, you can script around nvsmi tool or whatever. the only thing that counts is that hook_business.bat echos a number in the end.
-) IF hook_business.bat exists and it did output a number, the new number will replace the ffastrans internally calculated CPU business, so you can still use the max. cpu percent setting in ffastrans settings to control everything

The most simple example for hook_business.bat is:

Code: Select all

@echo off
echo 100

In this case, if you did set ffastrans Options to let's say 75% cpu max., no host would pick up any ticket anymore because all think they have 100% cpu usage.

Then another version

Code: Select all

@echo off
echo 0

This would indicate 0% cpu usage and all hosts would pick up tickets again.

Verify your results:
You can check what number your script did output, a file named %hostname%_business.txt will be written in the same folder where ffastrans.exe and hook_business.bat is. Also, the db/cache/logs/exe_log.txt will indicate when a machine refuses to pick up a ticket whenever hook_business.bat did output a higher number than allowed in ffastrans options.

Real life example "%install_dir%\hook_business.bat":

Code: Select all

@echo off
nvidia-smi.exe --query-gpu=encoder.stats.sessionCount --format=csv,noheader -i 0 > C:\temp\smi.txt
set /p SESSIONS=<C:\temp\smi.txt
set /a SESSIONS=%SESSIONS%*100
echo %SESSIONS%

Here also a post that covers some aspects of the topic. E.g. to prevent multiple jobs starting on same machine because it takes a while until GPU is actually used, you must play with abs_min in ffastrans.json.
viewtopic.php?f=5&t=1079&p=6188&hilit=n ... 39ab#p6188

[3dsasha]

Thanks you! That last post link gives me a good idea I didn't think about. It is so simple. In workflow I check for %temp%\%wf_name%.wait exists by Hold node. If file appears - delete that file by Command nodes and send task to GPU. When GPU finish it's job - create that file again. That's all and WF limited to one working instance.

[emcodem]

oha, i thought it is about farm usage for you @3dsasha