Page 1 of 2
Limit the number of cores/threads
Posted: Thu May 28, 2020 1:11 pm
by authorleon
Hello,
At a bit of research, I have found that it is ffmpeg.exe That does all the brunt of the work.
Is there a way to limit how many cores/threads FFAStrans can use.
I do understand that we have the affinity approach is properly fine but the problem is for every job a new ffmpeg.exe Is submitted.
So not quite if I can lock it down. Effectively I have a 12 core processor of which six are hyper- threaded. And I would like to allocate only four cores with four threads ( eight total ). My reasoning for this is that I have other processes which are on the same machine such as image processing. And I would like to balance out the system accordingly.
Thank you very much.
Re: Limit the number of cores/threads
Posted: Thu May 28, 2020 7:02 pm
by emcodem
Yeah... thats why it is named ffastrans, it mainly focusses on driving ffmpeg and it can smartly create avisynth scripts for enhanced filtering which are directly fed into ffmpeg, so it basically combines the worlds best free filtering engine with the worlds best encoder engine.
FFAStrans itself only takes care about the workflow automation around and also it calculates some smart ffmpeg settings when you are using the inbuilt encoder nodes.
BUT the inbuilt encoder nodes currently don't allow you to set the -threads setting of ffmpeg. You can do that using a commandline processor or a custom ffmpeg processor but this way you will be limited to be able to support a certain set of source files (especially audio layouts). For this, you would have to rebuild the ffmpeg command that your encoding node produces by investigate the log files and tear it apart to be able to use it in a customm ffmpeg processor. I can support you on this topic.
Do you have always the same type of input file? Do you have a very limited set of workflows and encodings or is it more a very general request?
By the way, Hyperthreading is of no use at all for transcoding, this helps more for stuff like sending emails or such but the heavy processor stuff like encoding will always use the full physical core so the HT technology does not help. From personal experience i can say it does not make a difference if you turn it off or on but you should not think that "assigning" one virtual core really only uses only this virtual core, it would always use the full physical core. The HT technology does not "split" one core, it is more like "pre-loading pending calculations" in a smart way but this only helps when doing a lot of small operations concurrently.
Re: Limit the number of cores/threads
Posted: Fri May 29, 2020 1:43 pm
by authorleon
Hello,
First of all thank you very much for the explanation. I do appreciate your time and effort.
The issue is that I have a system which has 32 cores ( thread ripper). Use this system to do a variety of tasks and with other applications I am able to lock them down to certain core counts. From image processing, simulation analysis and so on.
I would like to dedicate eight cores to the FFMpeg process. I did some research online and found this:
FFMpeg has an option -threads . You can define there, if it's going to be auto (default) or you can limit the number of used threads (CPU cores). Recommended that you set your number based on the amount of available threads minus 1, or 2. So if you have 8 threads, setting to -threads 6 would be great.
I was wondering if it is possible to write the command in the custom X264 option? Image located below.
- Annotation 2020-05-29 153745.png (42.77 KiB) Viewed 12790 times
Re: Limit the number of cores/threads
Posted: Fri May 29, 2020 3:29 pm
by emcodem
Haha good thinking, you are totally correct, the x26... encoders do not neccessarily need to use the ffmpeg -threads option because you can override ffmpeg settings using private codec settings as you point out.
Just the syntax is different, this is what you want:
As the field says, it shall contain the "value for the ffmpeg option -x264-params, this is documented as follows:
Code: Select all
-x264-params <string> E..V..... Override the x264 configuration using a :-separated list of key=value parameters
it looks like key=value:key=value:key=value
The possible key:value pairs can be seen at the x264 documentation, e.g. here:
http://x264.janhum.alfahosting.org/fullhelp.txt
Note again that all parameters that are listed in the x264 help are not provided with a preceeding "--" but just as key:value when used in the ffmpeg -x264-params field.
I did update the wiki accordingly, i guess currently only ffmpeg experts are able to use this field
Re: Limit the number of cores/threads
Posted: Sun May 31, 2020 11:27 am
by authorleon
Hello,
Sorry for the delay. Perfect. I will give that a try and let you know.
Thank you once again
Re: Limit the number of cores/threads
Posted: Sun May 31, 2020 10:49 pm
by authorleon
emcodem wrote: ↑Fri May 29, 2020 3:29 pm
Haha good thinking, you are totally correct, the x26... encoders do not neccessarily need to use the ffmpeg -threads option because you can override ffmpeg settings using private codec settings as you point out.
Just the syntax is different, this is what you want:
As the field says, it shall contain the "value for the ffmpeg option -x264-params, this is documented as follows:
Code: Select all
-x264-params <string> E..V..... Override the x264 configuration using a :-separated list of key=value parameters
it looks like key=value:key=value:key=value
The possible key:value pairs can be seen at the x264 documentation, e.g. here:
http://x264.janhum.alfahosting.org/fullhelp.txt
Note again that all parameters that are listed in the x264 help are not provided with a preceeding "--" but just as key:value when used in the ffmpeg -x264-params field.
I did update the wiki accordingly, i guess currently only ffmpeg experts are able to use this field
First of all thank you very much for the reply.
Okay first of all I did try this
. However, I could see all the course fluctuating.
Sorry to sound like a complete dance, but do I have to put more than just
in?
Would greatly appreciate an example I will be sending that to those being YouTube channels I hope
Re: Limit the number of cores/threads
Posted: Mon Jun 01, 2020 7:47 am
by emcodem
Hmmm, i guess you dont have to input "more" but different, did you e.g. try something like
Re: Limit the number of cores/threads
Posted: Mon Jun 01, 2020 10:07 pm
by authorleon
emcodem wrote: ↑Mon Jun 01, 2020 7:47 am
Hmmm, i guess you dont have to input "more" but different, did you e.g. try something like
Hello, I did give it a go. However, it seems not to work,
I set it to 4 threds as you can see. I did also try 1 as well as 2 threds.
Any help would be grateley appriciated.
Re: Limit the number of cores/threads
Posted: Mon Jun 01, 2020 10:15 pm
by authorleon
I have also tired the follows:
-threads 1
-threads 1
threads=1
threads=1
With and without spaces just incase.
Re: Limit the number of cores/threads
Posted: Tue Jun 02, 2020 4:20 am
by emcodem
Hey,
no need to test other configurations than exactly this one:
- threads.png (70.71 KiB) Viewed 12693 times
Unfortunately i cannot reproduce your problem, on my dev laptop with 4 cores, setting threads=1 limits the encoding speed to 1,5 while threads=8 encodes with 5x speed.
Your screenshot shows that you do lots of different stuff while testing. Can you please try one single encoding isolated and compare the final encoding speed?