Limit the number of cores/threads

[authorleon]

Hello,

At a bit of research, I have found that it is ffmpeg.exe That does all the brunt of the work.

Is there a way to limit how many cores/threads FFAStrans can use.

I do understand that we have the affinity approach is properly fine but the problem is for every job a new ffmpeg.exe Is submitted.

So not quite if I can lock it down. Effectively I have a 12 core processor of which six are hyper- threaded. And I would like to allocate only four cores with four threads ( eight total ). My reasoning for this is that I have other processes which are on the same machine such as image processing. And I would like to balance out the system accordingly.

Thank you very much.

[emcodem]

Yeah... thats why it is named ffastrans, it mainly focusses on driving ffmpeg and it can smartly create avisynth scripts for enhanced filtering which are directly fed into ffmpeg, so it basically combines the worlds best free filtering engine with the worlds best encoder engine.
FFAStrans itself only takes care about the workflow automation around and also it calculates some smart ffmpeg settings when you are using the inbuilt encoder nodes.
BUT the inbuilt encoder nodes currently don't allow you to set the -threads setting of ffmpeg. You can do that using a commandline processor or a custom ffmpeg processor but this way you will be limited to be able to support a certain set of source files (especially audio layouts). For this, you would have to rebuild the ffmpeg command that your encoding node produces by investigate the log files and tear it apart to be able to use it in a customm ffmpeg processor. I can support you on this topic.

Do you have always the same type of input file? Do you have a very limited set of workflows and encodings or is it more a very general request?

By the way, Hyperthreading is of no use at all for transcoding, this helps more for stuff like sending emails or such but the heavy processor stuff like encoding will always use the full physical core so the HT technology does not help. From personal experience i can say it does not make a difference if you turn it off or on but you should not think that "assigning" one virtual core really only uses only this virtual core, it would always use the full physical core. The HT technology does not "split" one core, it is more like "pre-loading pending calculations" in a smart way but this only helps when doing a lot of small operations concurrently.

[authorleon]

Hello,

First of all thank you very much for the explanation. I do appreciate your time and effort.

The issue is that I have a system which has 32 cores ( thread ripper). Use this system to do a variety of tasks and with other applications I am able to lock them down to certain core counts. From image processing, simulation analysis and so on.

I would like to dedicate eight cores to the FFMpeg process. I did some research online and found this:

FFMpeg has an option -threads . You can define there, if it's going to be auto (default) or you can limit the number of used threads (CPU cores). Recommended that you set your number based on the amount of available threads minus 1, or 2. So if you have 8 threads, setting to -threads 6 would be great.

I was wondering if it is possible to write the command in the custom X264 option? Image located below.

Annotation 2020-05-29 153745.png (42.77 KiB) Viewed 12784 times

[emcodem]

Haha good thinking, you are totally correct, the x26... encoders do not neccessarily need to use the ffmpeg -threads option because you can override ffmpeg settings using private codec settings as you point out.
Just the syntax is different, this is what you want:

Code: Select all

threads=8

As the field says, it shall contain the "value for the ffmpeg option -x264-params, this is documented as follows:

Code: Select all

ffmpeg -h encoder=libx264

Code: Select all

-x264-params       <string>     E..V..... Override the x264 configuration using a :-separated list of key=value parameters

it looks like key=value:key=value:key=value
The possible key:value pairs can be seen at the x264 documentation, e.g. here:
http://x264.janhum.alfahosting.org/fullhelp.txt

Note again that all parameters that are listed in the x264 help are not provided with a preceeding "--" but just as key:value when used in the ffmpeg -x264-params field.
I did update the wiki accordingly, i guess currently only ffmpeg experts are able to use this field

[authorleon]

Hello,

Sorry for the delay. Perfect. I will give that a try and let you know.

Thank you once again

[authorleon]

Haha good thinking, you are totally correct, the x26... encoders do not neccessarily need to use the ffmpeg -threads option because you can override ffmpeg settings using private codec settings as you point out.
Just the syntax is different, this is what you want:

Code: Select all

threads=8

As the field says, it shall contain the "value for the ffmpeg option -x264-params, this is documented as follows:

Code: Select all

ffmpeg -h encoder=libx264

Code: Select all

-x264-params       <string>     E..V..... Override the x264 configuration using a :-separated list of key=value parameters

it looks like key=value:key=value:key=value
The possible key:value pairs can be seen at the x264 documentation, e.g. here:
http://x264.janhum.alfahosting.org/fullhelp.txt

Note again that all parameters that are listed in the x264 help are not provided with a preceeding "--" but just as key:value when used in the ffmpeg -x264-params field.
I did update the wiki accordingly, i guess currently only ffmpeg experts are able to use this field

First of all thank you very much for the reply.

Okay first of all I did try this

Code: Select all

threads=8

. However, I could see all the course fluctuating.

Sorry to sound like a complete dance, but do I have to put more than just

Code: Select all

threads=8

in?

Would greatly appreciate an example I will be sending that to those being YouTube channels I hope

[emcodem]

Hmmm, i guess you dont have to input "more" but different, did you e.g. try something like

Code: Select all

threads=1

[authorleon]

Hmmm, i guess you dont have to input "more" but different, did you e.g. try something like

Code: Select all

threads=1

Hello, I did give it a go. However, it seems not to work,

I set it to 4 threds as you can see. I did also try 1 as well as 2 threds.

Any help would be grateley appriciated.

[authorleon]

I have also tired the follows:

-threads 1
-threads 1
threads=1
threads=1

With and without spaces just incase.

[emcodem]

Hey,
no need to test other configurations than exactly this one:

threads.png (70.71 KiB) Viewed 12687 times

Unfortunately i cannot reproduce your problem, on my dev laptop with 4 cores, setting threads=1 limits the encoding speed to 1,5 while threads=8 encodes with 5x speed.
Your screenshot shows that you do lots of different stuff while testing. Can you please try one single encoding isolated and compare the final encoding speed?