How to enable nvenc encoding

[michael85]

Hi guys,

I want to enable GPU encoding for my workflows. Since I do 99% transcoding material from MXF OP1A to H264 in a low resolution, I think this can speed up the processing (whic is my key factor).
By looking at the forum, I read many times that some of you are using nvenc encoding allready. I have also tried to compile the nvenc headers to ffmpeg by using this guide: https://docs.nvidia.com/video-technolog ... vidia-gpu/
but I get stucked at this point:

Go to the FFmpeg installation folder and run the following command.
./configure --enable-nonfree –disable-shared --enable-cuda-nvcc --enable-libnpp –-toolchain=msvc --extra-cflags=-I../nv_sdk --extra-ldflags=-libpath:../nv_sdk

It leads me to an error with "unknown commands" or something like that.
Anyway, I think these guide is outdated and doesn't fit to the actual ffmpeg versions.

Can anybody please share how you bring nvenc enconding to work with FFAStrans on Windows?

Background: I want to test if GPU Encoding increases the processing speed significant. We are transcoding (in a farm) on old hardware and have to renew them next year. Therefore I would like to know, if it is wisely to invest in a good GPU like NVIDIA RTX A5000 or to go with higer CPU frequency like a AMD EPYC.

Best Regards
Michael

[FranceBB]

Before you try with NVEnc, can you try to see if using OpenCL increases the performance of your workflows?
If you have x264 nodes, you can just write "opencl" in the Custom x264 options (ffmpeg x264 params) and see if it helps.
About NVEnc in general, you can try to see if you can find some builds that have already been compiled with it by using:

ffmpeg.exe -i "my_input.mxf" -c:v h264_nvenc -preset slow -y "my_output.mkv"

and then all the parameter you can configure with it.

The two most common places to find builds for Windows are: https://www.gyan.dev/ffmpeg/builds/ and https://github.com/BtbN/FFmpeg-Builds/releases but I don't know whether they have it enabled at compilation time or not, but just give it a shot, try them and you'll find out.

By the way, I'm not even sure whether this is allowed in the current builds shipped with FFAStrans as I've never tried 'cause I'm not interested in GPU Encoding but only on GPU filtering. Anyway, try to use a custom ffmpeg node in FFAStrans and use:

-c:v h264_nvenc -preset slow -c:a libfdk_aac -b:a 384k

and as container pick mkv, save and see whether it works or not.

[emcodem]

No need to compile ffmpeg on your own, all builds that you usually get from the internet have it, including the one that comes with ffastrans.

Unfortunately we dont yet have a processor for it in ffastrans, so you need to use a commandline or custom ffmpeg.

emcodem_nvenc_lowres_with_tc.json

(5.41 KiB) Downloaded 402 times

I use this productive for exactly your usecase as well, nvenc allows me not really to be faster than x264 but at least i can do more 2 in paralell at a reasonable speed.

[michael85]

Thanks a lot for your help guys.

@FranceBB: I have excatly done what you said and tried OpenCL but it didnt increase the speed of my workflow. I didn't recognize any effect.

@emcodem: I didn't know that FFAStrans comes with a compiled nvenc ffmpeg version. Thanks for letting me know and saving my time with unsuccessfull compiling trials. I have used your workflow and changed some options to get the output as required. The results with nvenc are quite good. The encoding speed is over 50% higher than with using the CPU.

I have used the same clip on a local SSD and had a encoding rate of 110 frames per second on CPU (my old workflow) and 180 frames per second on GPU (your workflow). The GPU (in this testcase a Geforce GTX 1080 Ti) was totaly bored with a usage of 5% for encoding. So I think a Geforce RTX A5000 would be a waste of money for my workflow.

Another thing which is dramaticaly slowing down my workflow is the ethernet connection (1 Gbit).
Normaly I get the files for transcoding from a high bandwith network share (Harmonic MediaGrid) after they were ingested. During the encoding process my network usage is at least for 50% of the job at maximum. This leads into a transcoding rate of ~80 frames per second. No matter whether with CPU or GPU. For the new servers, we will use 10 Gbits network cards to improve that hopefully.
In the meantime I would like to try to cache the files during the ingest localy to avoid the bottleneck of the ethernet connection. Is this even possible?

[emcodem]

Comparing x264 and nvenc encoding is not simple at all, here look at my results, all of them used about 50% of the nvenc chip (in case of h264_nvenc codec) or 100% CPU (in case of x264). Command used was ffmpeg -i INPUT -c:v libx264(/h264_nvenc) -crf 35 -preset PRESET OUTPUTFILE.mp4

x264_vs_nvenc.png (17.7 KiB) Viewed 8108 times

Harware: CoreI9-9900K and/or Nvidia A5000

...180 frames per second ... was totaly bored with a usage of 5% for encoding.

This doesnt work really, except you encode a very small resolution like 320x240 and limit the encoding speed to 180 fps. My RTX5000 uses 50% of the NVENC Chip when i encode fullhd in full speed usually. I never saw 100% NVENC usage for a single file/stream. Maybe you looked at the GPU usage instead of what they call video_engine usage (the NVENC encoder chip)?

So I think a Geforce RTX A5000 would be a waste of money for my workflow.

I think so too... honestly i expected much more from it. Stick to consumer cards if you dont need to do multibitrate live encoding.

Another thing which is dramaticaly slowing down my workflow is the ethernet connection (1 Gbit).
In the meantime I would like to try to cache the files during the ingest localy to avoid the bottleneck of the ethernet connection. Is this even possible?

Why is the network speed only 50% when the network is the bottleneck while encoding, it should be like 90-95%? You got high latency?
Sure thing you can localize. just insert a deliver processor before the encoder and set it to deliver to your local drive, thats all. If you deliver to %s_job_work% for localisation, you dont even need to delete the file once the workflow ends.

[michael85]

Thank you for sharing you results in x264 and nvenc encoding @emcodem.
I guess you are right, comparing both is not easy. It depends heavily on your use case / your workflow.

this doesnt work really, except you encode a very small resolution like 320x240 and limit the encoding speed to 180 fps. My RTX5000 uses 50% of the NVENC Chip when i encode fullhd in full speed usually. I never saw 100% NVENC usage for a single file/stream. Maybe you looked at the GPU usage instead of what they call video_engine usage (the NVENC encoder chip)?

Yes, I'm encoding to a small resolution (640 x 360). Maybe this is the reason why the usage of the GPU is that low. See my Screenshot attached.

screenshot.JPG

Why is the network speed only 50% when the network is the bottleneck while encoding, it should be like 90-95%? You got high latency?

Seems you misunderstood my last post. Network usage is during encoding at nearly 100%. I usualy encode file with a duration of 1 hour. When the encoding starts, I have a very high network usage. After a while the network usage goes down to aprox. 50% without slowing down the encoding. Probably there is enough in the cache!? I dont know how exactly FFAStrans handle (caches) files for processing from a network location.

Thank you for the hint with the deliver processor before the encoder processor. This works very well. Unfortunately my files are to large to have a benefit by copying them to a local HDD.

I have played a little around with x264 and I must revoke my statement regarding x264 from my last post. On my old hardware it speeds up my actuall workflow in encoding processor for +50 frames per second.

[emcodem]

Seems you misunderstood my last post. Network usage is during encoding at nearly 100%. I usualy encode file with a duration of 1 hour. When the encoding starts, I have a very high network usage. After a while the network usage goes down to aprox. 50% without slowing down the encoding. Probably there is enough in the cache!? I dont know how exactly FFAStrans handle (caches) files for processing from a network location.

Ok sorry for misunderstanding.
So FFAStrans itself does not do any caching or such, it is basically just executing ffmpeg for transcoding.
The behaviour you mention only really makes sense in my head if you use the A/V decoder before encoding. In this case, the decoding is done by avisynth instead of ffmpeg and this would first read the whole file for creating a new index (kind of a safety action, for cases when your input file is broke or does not have any index) and only after that, start the encoding.
But i guess you dont use A/V decoder so it is a miracle to me why you see this behaviour.
If i find the time, i'll try a similar encode than you do from 1 Gbit NAS by tomorrow and update you with the results here.

I have played a little around with x264 and I must revoke my statement regarding x264 from my last post. On my old hardware it speeds up my actual workflow in encoding processor for +50 frames per second.

It would be interesting "from where" you measured those +50 frames, was it from your last try with x264 e.g. using default settings or from your fastest try with nvenc or...

[michael85]

Hi emcodem,
I dont use a A/V decoder in my workflow. My small and simple workflow consist of a Monitor:Folder -> Conditional -> H264 encoder -> Deliveries:Folder. Why is this network usage a miracle to you? The processing of the h264 encoding must happens localy on the Server / Client and therefore the file have to be cached from the network (for my understanding). Or am I thinking wrong?
But it would be great if you find the time to make an similar test.

It would be interesting "from where" you measured those +50 frames, was it from your last try with x264 e.g. using default settings or from your fastest try with nvenc or...

Sorry that I did not explain this in detail. I got +50 frames compared to my old workflow. I have only changed the encoding processor from "basic h264" to a custom ffmpeg encoding processor with this values:

Code: Select all

-vf "scale=iw/3:ih/3" -b:v 1500k  -b:a 128k -c:v libx264  -preset ultrafast

The h264 processor used befor had these values:

h264_old.JPG (53.16 KiB) Viewed 8016 times

[emcodem]

What i dont understand from your network utilisation is that it should be stable throughout the whole transocding process. There is no caching involved at all, look at the RAM usage of ffmpeg.exe when you run a transcoding. Well OK, the OS can and does cache data that has been retrieved from NAS but this is only helpful when you read a file more than once, which does not happen in a normal ffmpeg encoding scenario. Anyway, if you underly the OS caching situation, what you see is transcoding progress but at the same time "no network" utilisation at all.

Your situation is the other way around, you see first 100% usage and then it drops down to the actually used bandwith. One might argue that ffmpeg needs to parse through the whole file in order to get the index but this does not cause 100% usage, instead it causes about 20Mbit for about 20 seconds before the transcoding progress even starts (assumed input file 50Mbit and 1 hour, MXF format, frame wrapped XDCAMHD).
The only thing that could cause a short burst of network usage at transcode start would be something like a really long GOP at start, but i don't think that is the case for you. (which format you are decoding anyways?)


Here you see some libx264 -preset veryfast -s 640x480 encodes at 1 G and 10G connection, note that the network utilisation is stable from start until the end of the transcoding process. Note that the 10G Server also has much faster and newer CPU's, and unfortunately i cannot test how it behaves with 1G because i have no 1G storages available.

1g.png (36.83 KiB) Viewed 7992 times

10g.png (20.47 KiB) Viewed 7992 times

By the way, another nice benchmark is "reading only" with ffmpeg, using ffmpeg -i %sourcefile% -codec copy -f null - ...this should read at around 95% of your available Bandwith.
After all as you see, it runs at 450 Mbit for me, so network is not my bottleneck, it is the CPU frequency in fact. On this machine nvenc could really speed up the processing because it is only 2.2Ghz processor

Anyway, you also might want to know that it is not only about fast encoding, as you do rescale, you might also want to look into speeding up the filtering, e.g. discard the bottom field and choose your preferred scale algoritm:

Code: Select all

-vf field=top,scale=iw/3:ih/1.5,setdar=16/9 -sws_flags fast_bilinear