FFAStrans hardware configuration

[Pogle]

Hello emcodem,
thanks for your explanation! It would be great if you could give a recommendation.
In our archive we have following source formats: SD D10/IMX50 in MXF Op1a, HD AVC Intra 100 in MXF Op1a and in the future probably UHD XAVC Intra 480 and FFV1 in MKV. The main target format is H.264 (deinterlaced, with inserted Logos and Intros / Outros) for SD, HD and H.265 for UHD. But sometimes we also need to encode to the source formats again. Rack space is not an issue. There is enough space for a big workstation. The picture quality I'll get for invested bits is more important than encoding speed. But if AVC-Intra HD-Files could be transcoded to H.264 at a >slower< preset significantly faster than realtime would be appreciated. I can't predict the future workload exactly yet, but I expect only some few (~ 2-4) hours a day.

Thanks for your help!

[momocampo]

Hello Pogle,

Do your production files have almost the same duration?
- If yes, maybe you could create and use just 1 encode setting for all archives with a good value quality/weight (2 or more pass will be better)
- if no, you could create several encode settings according to several duration groups( 0 to 20 min -> setting 1/ 21 to 50 min -> setting 2, etc)
I created one with this kind of workflow :

duration.png (64.09 KiB) Viewed 13493 times

In this way, you always use the better encode setting

Hope it gives you ideas
cheers.

Benjamin

[emcodem]

It would be great if you could give a recommendation.

Hey Pogle,
i fear in your case, the encoding speed (for AVC) will not matter a lot because the filtering that you need to do will most likely slow down the whole processing chain. We will need to open another topic and work on speeding up the filters for you.
The problematic spots in your usecase is:
- Filtering UHD or Filtering in General - most filters only work on a single core and block the chain.
- Encoding HEVC

In fact, for HEVC Encoding, you will not be able to get realtime for high quality/low bitrate on any CPU as far as i know.

Anyway, my recommendation for a typical "general purpose" encoding workstation is clearly to get the newest Intel CPU with the highest Clock speed that you can afford but not everything makes sense. My direct recommendation is: Core i7-8086K

As we see in this table, the Built-in Quicksync ASIC Encoder/Decoder did not really change in the architectures "Kaby Lake, Coffee Lake, Whiskey Lake, Comet Lake" The Core Core i9-9700K would introduce 2 additional cores which could be very interesting for the HEVC Encoding speed on CPU.

This way you have both options, Hardware Encoding and Software Encoding. No need to get an Extra Nvidia card as the current models will not really outperform Quicksync for your usecase.
You say you want to go for quality instead of speed and throughput and ask for the slower preset. But the decision between Hardware and Software encoding currently is about the output bitrate. The Hardware stuff produces a lot more bitrate at the same quality compared to x264 CPU encoding.

New for me is that Nvidia seems to have improved their Quality/Bitrate in their "Turing" chipset - also HEVC B-frame support which is important for the output filesize but i have no personal experience about that:
https://devblogs.nvidia.com/turing-h264 ... d-quality/
I guess the near future will reveal if they also worked on their problems with dark areas and small distant objects....

Some further interesting links:
https://www.intel.com/content/dam/www/p ... -paper.pdf

[Pogle]

@momocampo: Hello and thanks for your proposal! Our input files don't have a predictable duration. It varies from short takes to complete movies. What is the idea behind your procedure? Is it to apply more compression to longer files or to speed up encoding of long files?

@emcodem: To use Quicksync I will have to use different H.264 algorihms than the x.264, won't I? I haven't read (and understood) your attached Intel document completely yet. The Quicksync feature of Intel outperforms the use of AMD Threadrippers? Will the filter algorithms (deinterlacing, resizing, Logo insertion) benefit from a Dual CPU layout? Would you open a new topic for that please? Currently UHD and HEVC is not used in our archive, so only HD / H.264 should be encoded faster than realtime.
Could the budget I got be helpful for a hardware recommendation? (Kinda stupid question.) Fortunately it's quite high. If the workstation price doesn't reach a five digits range, everything is ok. (Hope, this doesn't sound pretentious.)

kind regards
Pogle

[momocampo]

Hello @Pogle,

The idea is to have the best encode setting to reach the same weight. I "aim" a 6-7 Mo file but I can't use the same setting with several duration. So, if I cut my workflow with different duration, I obtain the best encode setting according to the duration and a weight.
It is just an idea, depends of what you looking for, what is your priority : quality or weight? or both ?

Cheers.
B.

[emcodem]

@emcodem: To use Quicksync I will have to use different H.264 algorihms than the x.264, won't I?
I haven't read (and understood) your attached Intel document completely yet.

From the way you ask your question, i see there is a huge confusion in you about Hardware and Software Encoders.
So, some basic information: So we have the ITU Standard H.264, which is using DCT Algorithms. Whatever Codec you choose, no matter if hardware or software encoders, they will always use this algorithm. Your Choice is about the Codec implementation. The different Codecs that you can choose differ in hundreds of internal parameters of the H.264 standard.

-) x264 is an open source project and widely known to be the fastest and at the same time highest quality at lowest bitrate codec. Dont get confused by it's name as it is just yet another H.264 codec, like all others in the list
-) There are different other CPU based Encoders out there like Mainconcepts and such. Even FFMpeg has a second one called H264. All of them will differ in Core Utilisation and quality/bitrate.
-) NVIDIA NVENC is a chip on Nvidia Boards which you can utilize for H.264 encoding.
-) Intel Quicksync is a chip on Intel CPU's (which have a GPU on board), exactly like NVIDIA NVENC but a little different speed/quality/bitrate
-) AMD VCE is a chip on AMD Graphics Boards just as the other 2 above

All Chips on Hardware boards are typically not tuned for the very best output quality/bitrate out of the box and are very limited regarding what you can configure. So no chance to mimic the speed/quality of x264 on any Hardware encoder- they have their own, fixed settings because they are a chip and cannot be changed after they are produced.

The Quicksync feature of Intel outperforms the use of AMD Threadrippers?

This question does not work because Quicksync is a hardware encoder Chip that comes with some Intel CPU's and Threadrippers are bare CPU's.
Quicksync can be compared to AMD's VCE technology which they deliver on some Graphics boards.
But in general, AMD is a sidecase which i never use in professional environments so i cannot tell you too much about it. There was always too much hassle around drivers and installation using AMD so i completely dropeed any experiments with it. AMD is currently not investing into Enterprise environments so i wait until they do.

Will the filter algorithms (deinterlacing, resizing, Logo insertion) benefit from a Dual CPU layout?

Out of the Box, some do, others do not. The best deinterlacing that i know, QTGMC (Avisynth) cannot utilitze more than 1 core, so processing speed of FullHD 50p is as slow as 0.1 realtime and it will only get a little faster when you have higher clock frequency.
As said, we will need to tune your workflow if we recognize the built in filters in ffastrans are slowing down the processing chain for you.

Would you open a new topic for that please?

Currently the questions fit very well into the hardware topic. I trust you open a new topic when you want some specific information about your personal workflow.

Currently UHD and HEVC is not used in our archive, so only HD / H.264 should be encoded faster than realtime.

That should be easy using x264 on CPU..

Could the budget I got be helpful for a hardware recommendation? (Kinda stupid question.) Fortunately it's quite high. If the workstation price doesn't reach a five digits range, everything is ok. (Hope, this doesn't sound pretentious.)

The budget will not change my previous recommendation, it depends only on the usecase. If you can afford it, go for the Core I9 with highest possible clock speed. It has more Cores than the I7.

Most guys here work for the big broadcasters so the budget for hardware is typically not a topic at all. Anyway, from experience we tend to just buy a HP Z8 Workstation which has those low frequency XEON stuff built in - this only helps in case you want multiple streams encoded because the H.264 codecs like x264 were not tuned to utilize Many low-frequency CPUs. From feeling they are tuned to utilize 8 Cores but it depends a lot on your profile and settings... Streaming Companies like Netflix spend multiple Hundred thousands or even Millions to tune their encoders to get out best quality at lowest bitrate.

[Pogle]

Hello!

Sorry for having used the technical terms a bit carelessly and thanks for your explanations! I have thought so far a compression standard only determines the structure of the result of an encoding to guarantee the correct decoding, but the way to get to an encoded video file is not a prescribed procedure.
So my choice is about the different codec implementations and not the algorithm itself.
I appreciate your advice and think we will go for an i9.

regards
Pogle

[Ghtais]

HI guys

just a quick post to share with you my brand new configuration running FFAstrans.
It's a Dell Precision 5820 / Core I9-10980XE @ 3.0Ghz / 32 Go RAM / Flexbay NVME PCIe / 10Gbits NIC
I only use CPU so I have a very chip Quadro 620 GPU to save money.

• 1 x 512 Go NVMe class 40 SSD (M2 port) for system
  1 x 1 To NVMe class 50 SSD (M2 port) for FFAStrans cache

It outperform my old dual xeon (you can find spec from my old computer going to page 1)

H264 1920x1080 2.7 Mbits with XDcam or Dnx120 source : x6
GoPro 1920x1080 to DnxHD : x10
8 x Gopro 1920x1080 to DnxHD at the same time : x2.5 (thank to the NVMe SSD 2, I have seen 2 GBps pike in the windows monitor during transcode)

Very happy with that
Bye

[FranceBB]

I'm glad you're happy with your new configuration.
Having a fast SSD connected via M2 definitely improved things further, especially in the decoding.
As to the rest, x264 is pretty much optimized up to AVX2 and it's kinda lightweight for nowadays CPUs but it has never been parallelized to fully use 2 CPUs, which is probably why your dual xeon config happened to be slower despite having 2 CPUs. (leaving aside the thread/core count and the instruction sets)
(can't say anything about the DNxHD encoder as I very rarely use it. )

[mrazik]

Hi everyone,

I’m struggling with new ffastrans machine setup and would like to get feedback from you.

Most of the transcoding will be done from xdcam to h.264 with a/v media and filter processors (timecode, watermark, change fps). Source and destination clips will be on network storage.

I’m thinking about:
- 2x nvme drive
- 2x 10gbs nic

But have no idea what CPU and if any GPU is needed. Basically at the moment the budget is open, I’m looking for very fast transcoding speed.

Can you please advice?

Many thanks,
mrazik