Advanced-FFmpeg encoder plugin

[Aris]

On my recent setup of FFAStrans, I kept things simple with a three-step workflow: watch, sort, deliver.
After that, I added resizing and timecode burning.

The system I’m using is a relatively new AMD 7960X and RTX 5070 Ti.
Most of the time, everything runs automatically without any interaction.
However, one day I was in a hurry to receive a file, and the job took longer than I liked.

That’s when I started investigating how to improve the workflow.
I looked through the forums but couldn’t find anything useful.
I then ran tests outside of FFAStrans and finally discovered something that significantly improved performance: -hwaccel decoding.
It increased speed from around 1.5x to about 8x.

After that, I started looking into how to integrate that or more options into the FFAStrans encoder or command setup.
While researching, I came across this:
viewtopic.php?p=8856&sid=7f6b825f972db1 ... c2a4#p8856

Could it achieve what I needed? Yes. Did I like it? Not really.

So I moved on to generating a text file and using the command executor.

batch.png (48.96 KiB) Viewed 223 times

It worked, but it wasn’t easy to edit due to the FFAStrans UI and AutoIt controls.

At that point, I thought I might as well start developing my own plugin.
This isn’t my first FFAStrans plugin. Last year, I wrote another one for a friend.

I present to you: Advanced-FFmpeg encoder plugin.
https://github.com/Feraris20/Advanced-FFmpeg

More testing is needed. Suggestions and ideas are welcome.
Thank.

[emcodem]

Hey Aris, great addition, thanks a lot for sharing and taking on the project

The idea of such a general ffmpeg node was always in the room but there is one big problem: the hard part of the ffmpeg integration done in ffastrans encoders is the automatic calculation of filters as needed to come from source to target settings. E.g. Create missing audios or videos and map then as required, depending on the source files properties, then add A/V filters as required dynamically. This is why i personally mostly use the s_options trick to first let ffastrans calculate the full cmd line and after that modify it to my needs. This is of course only needed if one wishes to support "all source file flavors" instead of only one but it changes the whole basic design ideas because if the user can add filters AND we have dynamically inserted filters, we need to check and resolve potential conflicts. This leads to A LOT of code logic.

One of my many similar attempts is this (i uploaded this just for you to investigate):
https://github.com/emcodem/ffmpeg_cmd_builder_ui
NOTE: This project is incomplete and will not really generate working ffmpeg cmds.

This project has 2 things in 1:
1) a nodejs script that extracts the capabilities of the provided ffmpeg build, along with settings and help strings of the individual codecs and filters
2) a sample html using the extracted huge json data object, similar to your gui https://emcodem.github.io/ffmpeg_cmd_bu ... ilder.html

It was important for me to only show the capabilities of a specific ffmpeg exe and be able to dynamically extract them. What this misses are general AVContextOpts, but i didnt yet dive into that.

Then we have the execution part which you use powershell for. The script for this is prune to become pretty big, especially if you try to attempt the automatic filter building depending on the source properties.
Honestly, i would recommend to use a proper language instead of powershell. Even if these days AI writes all the code, powershell limits the possibilities. My recommendation is to use python, you can place a copy of portable ("embedded") python in the bin foler to keep everything portable. Python is a little harder to deal with but i am sure if you tell your agent that everything must be 100% portable, it will know where to put the modules (if any) and how to include them from relative path. As the scripts continue growing, you should add testing, lots of automated tests in order to give your agents a way to check if what they did doesnt break existing functionality, also the GUI can be tested using pupetteer and co.

One other question that came up for me is if using prebuilt ffmpeg exe for such doings is still the correct approach. Now as Agents write the code, we coult just use PyAV which might lead to a much more fluent integration - and much more importent it would be MUCH easier to test automated, which supports the Agentic development workflow.

Also, a sidenote, you could ask your Agent to add a github action workflow that creates a zip with the relevant files (ready to deploy for users) and creates a new release or prerelease on your github repo whenever you push a change.

[FranceBB]

I then ran tests outside of FFAStrans and finally discovered something that significantly improved performance: -hwaccel decoding.
It increased speed from around 1.5x to about 8x.

Yep, hardware decoding will always be faster than relying on pure software decoding from libav as it's gonna use the actual dedicated hardware from the GPU which can be considered an ASIC. There's one caveat with that: your GPU must have the hardware decoder for that particular codec. Even more importantly, even if your GPU does have an hardware decoder for a particular codec (let's say H.264), the source you're trying to decode must also be within the supported resolution, level, profile, bit depth, number of reference frames and so on.

A very simple example would be a very normal H.264 1920x1080 4:2:0 yv12 ref 4 keyint 25 Profile High, Level 4.1 at 25fps 8bit planar which your GPU will have absolutely no problem on decoding. On the other hand, if the same stream was encoded with, let's say, ref 16, me esa subme 11 and a very high keyint like 500 (which means that the GOP is potentially like 20 seconds long), then you your GPU wouldn't decode it. Another simple example would be 10bit as the RTX 5xxx series is the first one to support 4:2:2 10bit planar hardware decoding, but it doesn't support 4:4:4 10bit so it would fail decoding there as well.

If we go into other less common codecs like Lagarith, UTVideo, HuffYUV which are lossless but lack any hardware decoder in modern GPU, it would also fail. The point I'm trying to make is that hardware decoding only really works with what your GPU can decode as it relies on the actual hardware and while it might be a quick win in some occasions in which you know for a fact you're gonna be decoding something up to spec, it's not really recommended for workflows in which you have to cater for literally any kind of files. This is also why, in players like MPV, hardware decoding is generally disabled by default and can be turned on if the user wants whenever there's a codec combination supported by the user GPU, while everything else goes via the usual software decoding.

[emcodem]

Code: Select all

It increased speed from around 1.5x to about 8x.

Hm, was this only decode benchmark? Actually i tested a lot with this but usually the filters and encoders were such a huge bottleneck that the Decoding speed itself would not change much. Changing from 1.5 to 8 just by changing the decoder from CPU to hardware smells very fishy to me, there should be something else in play...

[Aris]

Code: Select all

It increased speed from around 1.5x to about 8x.

Hm, was this only decode benchmark? Actually i tested a lot with this but usually the filters and encoders were such a huge bottleneck that the Decoding speed itself would not change much. Changing from 1.5 to 8 just by changing the decoder from CPU to hardware smells very fishy to me, there should be something else in play...

What I described was the whole workflow not just decoding.
For the moment I only need resize (downscale) and timecode burn in.

https://repo.jellyfin.org/archive/jelly ... 0bit%2Emkv
Used this file for testing:

Code: Select all

jellyfish-250-mbps-4k-uhd-hevc-10bit.mkv

Tests run on conversion

tests_0D7MJtdRfI.png (16.12 KiB) Viewed 108 times

I performed 3 tests.
1)Convert to x264.
2)Convert to x264 and resize to 720p.
3)Convert to x264, resize to 720p and burn timecode.

This is with no hardware acceleration.

no hwaccel

hwaccel_off_ayuJIh299C.png (5.01 KiB) Viewed 108 times

System used:
Intel Core i9 14900KF and RTX 5060 Ti

That's all.

Thanks,
Aris

[emcodem]

Ohhh yea sorry i forgot hevc exists, we rarely get one of these where i work so accellerating these is no topic at all. Yea i guess anything compressed higher than h264 will fall in this category where introducing a hardware decoder helps really a lot...

[Aris]

I then ran tests outside of FFAStrans and finally discovered something that significantly improved performance: -hwaccel decoding.
It increased speed from around 1.5x to about 8x.

Yep, hardware decoding will always be faster than relying on pure software decoding from libav as it's gonna use the actual dedicated hardware from the GPU which can be considered an ASIC. There's one caveat with that: your GPU must have the hardware decoder for that particular codec. Even more importantly, even if your GPU does have an hardware decoder for a particular codec (let's say H.264), the source you're trying to decode must also be within the supported resolution, level, profile, bit depth, number of reference frames and so on.

A very simple example would be a very normal H.264 1920x1080 4:2:0 yv12 ref 4 keyint 25 Profile High, Level 4.1 at 25fps 8bit planar which your GPU will have absolutely no problem on decoding. On the other hand, if the same stream was encoded with, let's say, ref 16, me esa subme 11 and a very high keyint like 500 (which means that the GOP is potentially like 20 seconds long), then you your GPU wouldn't decode it. Another simple example would be 10bit as the RTX 5xxx series is the first one to support 4:2:2 10bit planar hardware decoding, but it doesn't support 4:4:4 10bit so it would fail decoding there as well.

If we go into other less common codecs like Lagarith, UTVideo, HuffYUV which are lossless but lack any hardware decoder in modern GPU, it would also fail. The point I'm trying to make is that hardware decoding only really works with what your GPU can decode as it relies on the actual hardware and while it might be a quick win in some occasions in which you know for a fact you're gonna be decoding something up to spec, it's not really recommended for workflows in which you have to cater for literally any kind of files. This is also why, in players like MPV, hardware decoding is generally disabled by default and can be turned on if the user wants whenever there's a codec combination supported by the user GPU, while everything else goes via the usual software decoding.

Thanks for all this information — it’s a great way to share knowledge. Keep it going.
In my case, I could keep it on CMD and edit it whenever needed, but on the other hand, I’d also like to offer to the community.