Hi
When ffastrans transcoding few files on same time (in one workflow), then uses only one NUMA node.
Is possible to run one video transcode on numa node 1, and run next video transcode on numa node 2?
Numa nodes
-
PoznanskaPyra
- Posts: 7
- Joined: Mon Jan 04, 2021 3:02 pm
Numa nodes
- Attachments
-
- trcode.PNG (19.78 KiB) Viewed 3012 times
-
- numa2.PNG (14.76 KiB) Viewed 3013 times
-
- numa1.PNG (12.27 KiB) Viewed 3013 times
Re: Numa nodes
Hubert,
you hit me very deep with that question, i just love it (because i know about NUMA nodes but i dont have much experience with it) and i promise to find a solution together with you as long as you keep on delivering all infos we need. Depending on the complexity, hopefully the solution joins the ffastrans source code later on if needed.
At least i know from another transcoder that they imlepmented "something" about it and charge a license for it, so lets get out if its worth any money they charge.
So, we had an internal talk and it looks like steinar alredy has seen full cpu utilisation on a 24 core system (48 core with HT - which hits the 36 core limit for NUMA, at least as far as i know).
Our expectation would be that whenever the application (exe_manager) spawns a new child process, the OS will smartly manage which processor group is assigned to it. Your problem description points us to a direction where this is just not the case and all child processess of exe_manager just share the same processor group as the parent process is assigned to.
You know, as usual: first step in problem solving is to reproduce the problem...
1) Therefore, the first thing we need is your workflow exported and uploaded here. Of course, if you can simplify your workflow to e.g. a single encoding node where you just submit 5 files and you got stuck at utilisation bound to a single NUMA node, it would be tremendious!
2) About your system: i assume it is not a vm Guest but directly on hardware installed windows, right?
3) Just to make sure, you have Hyperthreading enabled, right?
4) what OS version are you running?
5) Why you dont have webinterface installed? (:D)
you hit me very deep with that question, i just love it (because i know about NUMA nodes but i dont have much experience with it) and i promise to find a solution together with you as long as you keep on delivering all infos we need. Depending on the complexity, hopefully the solution joins the ffastrans source code later on if needed.
At least i know from another transcoder that they imlepmented "something" about it and charge a license for it, so lets get out if its worth any money they charge.
So, we had an internal talk and it looks like steinar alredy has seen full cpu utilisation on a 24 core system (48 core with HT - which hits the 36 core limit for NUMA, at least as far as i know).
Our expectation would be that whenever the application (exe_manager) spawns a new child process, the OS will smartly manage which processor group is assigned to it. Your problem description points us to a direction where this is just not the case and all child processess of exe_manager just share the same processor group as the parent process is assigned to.
You know, as usual: first step in problem solving is to reproduce the problem...
1) Therefore, the first thing we need is your workflow exported and uploaded here. Of course, if you can simplify your workflow to e.g. a single encoding node where you just submit 5 files and you got stuck at utilisation bound to a single NUMA node, it would be tremendious!
2) About your system: i assume it is not a vm Guest but directly on hardware installed windows, right?
3) Just to make sure, you have Hyperthreading enabled, right?
4) what OS version are you running?
5) Why you dont have webinterface installed? (:D)
emcodem, wrapping since 2009 you got the rhyme?
Re: Numa nodes
@PoznanskaPyra
luckily, i got some new servers with 32 Cores now and what i found out is that your problem might be less connected to NUMA Nodes but to processor groups instead. In my HP Server DL380 Gen10, i found some NUMA related option that when set to "Cluster", it would force windows to Split my 64 logical cores into 2 groups while i expected to have only 1 group.
So, your goal is always to have only 1 processor group (unless you run with more than 64 cores, in that case let's continue the discussion). You can e.g. simply disable Hyper-Threading as there is no benefit from it for transcoding anyway.
luckily, i got some new servers with 32 Cores now and what i found out is that your problem might be less connected to NUMA Nodes but to processor groups instead. In my HP Server DL380 Gen10, i found some NUMA related option that when set to "Cluster", it would force windows to Split my 64 logical cores into 2 groups while i expected to have only 1 group.
So, your goal is always to have only 1 processor group (unless you run with more than 64 cores, in that case let's continue the discussion). You can e.g. simply disable Hyper-Threading as there is no benefit from it for transcoding anyway.
emcodem, wrapping since 2009 you got the rhyme?
-
PoznanskaPyra
- Posts: 7
- Joined: Mon Jan 04, 2021 3:02 pm
Re: Numa nodes
Sorry. I have lot of jobs and i forgot about this thread :/ At this moment ffastrans is base transcoder, it has 5-6 jobs in one time. CPUs is allways loaded to 100% 