Best way to accelerate encoding proces

[taurojo]

What is the best way to accelerate the compression process?

We have a NAS where we host the videos and a computer to encode.
Currently we launch the videos from the webinterface. And local temprary files in SSD HD.

The basic process is (from NAS) WEBINTERFACE -> AV/MEDIA -> CHANNEL MAPPER ->WATERMARK- > H264 -> TO FOLDER NAS


It is better to put an intermediate cache unit. That is to say
The basic process is (from NAS) WEBINTERFACE -> FOLDER (CACHE)->AV/MEDIA -> CHANNEL MAPPER ->WATERMARK- > H264 -> FOLDER (CACHE) - >TO FOLDER (NAS)

How is your system set up?
Thanks!!!!

[emcodem]

Hi taurojo,

there is a huge amount of things you can do to speed up.
But looking at your workflow, i don't think you can get A LOT faster by optimizing the cache location. Sure having the cache on the same NAS AND the same Share (but different folder) on the NAS as the final delivery can bring you some seconds: The time that the "TO FOLDER NAS" node takes.
Check out in the webinterface which node takes longest. I am pretty sure it is the h264 node.

The options for speeding this up are huge.

First would be to just set the H264 Node to superfast which results in reduced output quality.

Second would be to try to avoid using all these nodes: "AV/MEDIA -> CHANNEL MAPPER ->WATERMARK" and instead do it with ffmpeg only in a commandline processor.
I can help you with that, but first i must know:
Do your input files always have the same Audio Track/Channel layout?

[taurojo]

The mp4 is indeed the most time-consuming.

Input files can be PAL DV (avi), mp4 CBR hdv 1440x1080 or FHD.
The audio 1,2 tracks or 1,2,4,5.


No work, for my: put "-ac 1" in custom FFMPEG.

My workflow:
Join audio to two tracks audio joining from original (mixing a1+a2 to a1 and a2)

Then png logo with some conditionals, according to size. Insert png 720x576.png or 1440x.png, and add video at the end (insert media)
Finaly encode in mp4 for web and put to te FTP folder

[emcodem]

Code: Select all

"%s_ffmpeg%" -nostdin -hide_banner  -analyzeduration 33554432  -i "%s_source%" -f lavfi -i aevalsrc=0 -f lavfi -i color=color=black:size=%i_width%x%i_height% -shortest -map_metadata -1  -map 0:0  -filter_complex "[0:1]pan=1|c0=c0[a1],[0:1]pan=1|c0=c1[a2],[a1][a2]amerge=2,apad[astr1]" -map "[astr1]" -c:a:0 aac -b:a:0 128k -ar:a:0 48000  -aspect 16:9 -preset veryfast -b:v 2500k -c:v libx264 -r 25 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv -field_order progressive -s %i_width%x%i_height%  -brand mp42  -timecode 00:00:00:00    -max_muxing_queue_size 800 -map_metadata -1 -metadata creation_time=now -y "%s_source%_ffastrans.mp4"

OK, so if you want, we can work this out, i am sure we will find a simple solution for every problem. Biggest issue is to react on the different audio input layouts.
First step, can you please use the EXACT command from above (if you modify, send me plase your modified version back) in a Commandline Executor node?

Workflow: Just commandline executor, submit a file with 1 track, 2 channels (stereo) and check if the encoding process is MUCH faster than your original one.
It will get a little bit slower when we add the logo insertion but not really slower when we add the audio manipulation later on.

Let me know your results please!
ALSO: let me know about your quality requirements. If you MUST encode with "slow" preset in order to get best quality, it might not really pay off to work on this because the result will always be slow.

[Ghtais]

Hi

I would say it depends if you transcode many files at the same time or not. If the answer is yes, I advise to localize file on a very fast SSD Nvme on your station instead of transcoding directly from your NAS. If you use AV / Media your file will be pre-processing before been transcoded and again It will be faster with fast local drive. If you use Watermark node for example you should notice that your workflow doesn't use full power of your CPU so it would be a good idea to transcode maximum file at the same time to hit 100% of CPU.
For the case you have only one file at the same time and you want to process as fast as it possible you should follow emcodem

[emcodem]

Hey @Ghtais!
Good to hear from you!

I would say it depends if you transcode many files at the same time or not. If the answer is yes, I advise to localize file on a very fast SSD Nvme on your station instead of transcoding directly from your NAS. If you use AV / Media your file will be pre-processing before been transcoded and again It will be faster with fast local drive.

In my head, localizing the source file is only helpful in these cases:
1) to support some crappy software in your workflow that does read very small chunks from the source (no such tool comes with ffastrans)
2) to support unstable networks, networks with high ping times or permanently overloaded NAS
Is it the case 2) that you refer here?

[taurojo]

Thank you for your support!!!
I currently have the temporar mounted on a non-nvme SSD, we will acquire a NVME for the temporar.

[emcodem]

Hey taurojo,
i feared that Ghtais comment would confuse this.
Just know that from your current description adding a temp share does not help, its more the other way around.