Camera footage with multi track audio

[Conniver]

Hi, have anyone created a workflow to create h264 proxies from camera files with multi track audio, and keep the same amount of audio channels?

We have several different cameras, and I'd like to create a workflow to make proxies for these with matching number of audio streams.
E.g., most of our Sony cameras have 8 audio track (although most of them are empty).
The number of actual audio streams varies, there is usually always 3 (boom mic and camera stereo mic), but often there is a couple wireless
mics aswell.

I'd like to avoid using ProRes, as they are a bit on the larger size, even with "proxy" setting.

I have tried "same as source" on "Discreet tracks" and "Total channels", but I always get
"Encoding: Unsupported channel layout" 1 channels (FL))"

Data from using ffmpeg in command prompt.

Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:1 -> #0:1 (pcm_s24le (native) -> aac (native))
Stream #0:2 -> #0:2 (pcm_s24le (native) -> aac (native))
Stream #0:3 -> #0:3 (pcm_s24le (native) -> aac (native))
Stream #0:4 -> #0:4 (pcm_s24le (native) -> aac (native))
Stream #0:5 -> #0:5 (pcm_s24le (native) -> aac (native))
Stream #0:6 -> #0:6 (pcm_s24le (native) -> aac (native))
Stream #0:7 -> #0:7 (pcm_s24le (native) -> aac (native))
Stream #0:8 -> #0:8 (pcm_s24le (native) -> aac (native))
Stream #0:9 -> #0:9 (copy)

VLC only shows the first 8 streams (1-8)


I am unsure what the 9th stream is. but it give the error.
"Could not find tag for codec none in stream #9, codec not currently supported in container"

Using "Discreet tracks" 1, and "Total channels" same as source, gives me all the tracks, but the editor (Resolve)
only gives audio on track 1.

TL:DR. Can I make camera footage with multiple audio tracks into a usable h264 (or maybe h265) proxy, for video editing?

[ThomasM]

hmm.... are you sure you mapped the audio-tracks correctly in BMD Resolve? You can do so by right-click on Clip in a bin and select "Clip Attributes". Go to "Audio" and select Track-Layout and Mapping. So you can organize your timeline also - just right-click on a Clip in the Timeline and re-configure the Track-Layout and Mapping in the Clip-Attributes. This is also important with Track-Type in the Timeline. You can have a Mono-Track in the TL confugured with an 8-Track Clip, but only make use of one Track of that Clip by mapping. In a bin you also can select multiple Clips for re-Mapping tracks.

just to be sure...

[emcodem]

Hey Conniver,

My guess why it dont work is that the aac encoder might be aac v2 by default but v2 does not support mono tracks as read here:https://trac.ffmpeg.org/wiki/Encode/AAC
Long story short, ffastrans h264 encoder processor does not allow another audio codec and also it does not support changing the aac profile.

You are right thinking about using another encoder like prores. An alternative would be the xdcamhd 18Mbit encoder.
Anyway, supporting "any" audio layout is not as simple as one would think. Unfortunately i don't have such a workflow in production so i can shoot from the hip here.

My first approach would be using the inbuilt h264 encoder but a separate custom encoding for the audio, then in a final step, wrap and deliver. The reason is mostly because the inbuilt encoders have most of the needed filter logic to come from the input format to the proxy format.


As i don't know any compressed audio codec that supports "any channel layout", i'd go with pcm audio (that might keep away other troubles too).
Then, as mp4 does not support pcm audio usually, we also need to deliver the final file as .mov unfortunately but ok, not a big difference to mp4.

Here is some example workflow, take care to change the delivery directory in the last processor.

emcodem_proxy_copy_audio_layout.json

(4.27 KiB) Downloaded 163 times

Anyway, if you don't need all the fancy inbuilt logic of the h264 processor for the format conversion but instead you want to keep all the video properties of the input file anyway (e.g. resolution, interlacing, framerate etc), you might want to tell us because it might be easier then to work with a totally custom ffmpeg commandline.

[Conniver]

The camera footage is correctly set up, with 8 mono channels. And is showing correctly on the timeline with all 8 tracks (including the empty tracks) when inserted. If I import the proxy as "Full resolution media" I get all channels imported, it just wont work as a proxy.

We also use an online service that stores hires and creates proxy, I know they use FFmpeg, and I just took a look at one of their proxy files,
And I see they have 8 channels AAC, this is proxy made from a Sony camera (FX9) MXF (AVC), so at least I know it's possible.

(Track 8/8)
Audio #8

ID : 9
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Codec ID : mp4a-40-2
Duration : 47 s 640 ms
Source duration : 47 s 661 ms
Source_Duration_LastFrame : -19 ms
Bit rate mode : Variable
Bit rate : 1 526 b/s
Maximum bit rate : 256 kb/s
Channel(s) : 1 channel
Channel layout : C
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 8.87 KiB (0%)
Source stream size : 8.88 KiB (0%)
Default : No
Alternate group : 1
mdhd_Duration : 47640

But I'm guessing this is turning into a pure FFmpeg question, so I'll poke around in a FFmpeg forum.
But if anyone know a way to make a 8 track mp4 similar to the example above.

If I use the h.264 node and use 1 Discreet Track, I get a "surround" track, including LFE.
Audio

ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Codec ID : mp4a-40-2
Duration : 36 s 950 ms
Source duration : 36 s 971 ms
Bit rate mode : Constant
Bit rate : 320 kb/s
Channel(s) : 8 channels
Channel layout : C L R Ls Rs Lw Rw LFE
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 1.41 MiB (5%)
Source stream size : 1.41 MiB (5%)
Default : Yes
Alternate group : 1
Encoded date : UTC 2023-01-18 12:07:52
Tagged date : UTC 2023-01-18 12:07:52
mdhd_Duration : 36949

If I use 8 Discreet Tracks it wont encode, "Unsupported channel layout".
Tried custom mapping as well, unsure if the I use it correct, but I'm guessing the input should be "Track no.1: -> 1, "Track no.2": -> 2... etc
Its a bit confusing when default values is 1 on all.

[Conniver]

Thanks ThomasM and emcodem.

I'll do some more digging in a FFmpeg forum. Will update if I find a solution.
I will also check out your workflow Emcodem, thanks.

[emcodem]

OK so i tested some more and found out that the encoder that we use (ffmpeg native aac encoder in aac-lc config) does support mono for sure, also it supports 16ch track so it will most likely be a good way to go.
It is no problem to encode 16 mono tracks from PCM to aac using just a simple ffmpeg cmd. Anyway ffastrans does insert lots of filters automatically in order to cover all cases of audio maaping. In your case it is too many filters because the "pan" filter seems to insert a channel layout (which is by default obviously unset) which makes the aac filter fail. Might be worth to open a bug for that on the ffmpeg trac

works:

Code: Select all

ffmpeg -f lavfi -i "aevalsrc=exprs=0:s=48000" -t 1 -c:a aac c:\temp\test.mp4

fails:

Code: Select all

ffmpeg -f lavfi -i "aevalsrc=exprs=0:s=48000" -t 1 -c:a aac -filter_complex "[0:a]pan=1|c0=c0[a1]" -map [a1] c:\temp\test.mp4

works (channel layout for pan filter set to 1c instead of 1):

Code: Select all

ffmpeg -f lavfi -i "aevalsrc=exprs=0:s=48000" -t 1 -c:a aac -filter_complex "[0:0]pan=1c|c0=c0[a1]" -map [a1] c:\temp\test.mp4

What ffastrans does is to give "1" as a channel layout specifier to pan filter because the docs say "layout or number of channels". Obviously if the number of channels is 1, pan filter decides to take layout 1FL which is unsupported by the aac encoder. Quickly browsing the ffmpeg source code, this seems to go back to the av_channel_layout_from_string function in libavutil/channel_layout.c BUT this function is huge so i cannot tell if it is intention or a bug that 1FL is chosen for "1" channel.

After all maybe you can use the workflow i proposed and just let it encode aac instead of pcm audio...

[Conniver]

I did some more testing on FFmpeg,
and it seems this command worked, although I have not testet it properly yet (this is just a first test, so ignore video encoding part.)
BMD Resolve at least accepts the mp4 as a proxy with working audio channels.

Code: Select all

ffmpeg -i input.MXF -c:v libx264 -tune animation -crf 35 -x264opts keyint=10 -pix_fmt yuv420p  -map 0:v:0 -map 0:a:0 -map 0:a:1 -map 0:a:2 -map 0:a:3 -map 0:a:4 -map 0:a:5 -map 0:a:6 -map 0:a:7 output.mp4

this command worked with the ffmpeg in FFAStrans\Processors\ffmpeg\x64\

I'll look more into it later today.

edit:
I'm an absolute noob when it comes to FFmpeg, i'm starting from scratch

[Conniver]

Did some testing and tweaking, and found this to be a fairly good setting for editing proxy for our Sony cameras.

Code: Select all

ffmpeg -i input.MXF -c:v libx264 -preset medium -qp 15 -x264-params keyint=20:min-keyint=10 -pix_fmt yuv420p -map 0:v:0 -map 0:a:0 -map 0:a:1 -map 0:a:2 -map 0:a:3 -map 0:a:4 -map 0:a:5 -map 0:a:6 -map 0:a:7 output.mp4

emcodem, I coundn't quite get yours to work om my footage, the top won encoded, but only gave me 1 track.
And the bottom one failed with som filters warning.

I'm off to my next problem in a new thread.

[emcodem]

Code: Select all

ffmpeg -i input.MXF -c:v libx264 -preset medium -qp 15 -x264-params keyint=20:min-keyint=10 -pix_fmt yuv420p -map 0:v:0 -map 0:a:0 -map 0:a:1 -map 0:a:2 -map 0:a:3 -map 0:a:4 -map 0:a:5 -map 0:a:6 -map 0:a:7 output.mp4

That only works for 8 track input, -map 0:a should do virtually the same and work with all tracks/layouts if i'm not wrong. Also the video part will copy the resolution, framerate, interlacing, pixel_formats from input to output (because you don't specify them in your cmd). If possible, you should avoid creating different ffmpeg encoding lines based on the source audio track layout but instead take some time to engineer one line that works with all layouts.

I usually specify the resolution, bitrate, level, profile, pix_fmt in order to avoid ffmpeg doing rubbish.
Example video config for nvidia supported encoding:

Code: Select all

ffmpeg -i "SOURCEFILE"  -map 0:v -map_metadata -1 -sn -s 320x240 -g 12 -b:v 500k  -b:a 128k -preset fast -c:v h264_nvenc -pix_fmt yuv420p -preset fast -level:v 4.1 -profile high "OUTFILE"

But after all, if you go this route, you are trying to recreate functionalities that the inbuilt h264 processor already does for you and because that's a hassle, i did recommend to encode the 2 streams separately.
If i have time later on, i'll probably come up with an example workflow that does a 1 step custom ffmpeg encoding for you.

emcodem, I coundn't quite get yours to work om my footage, the top won encoded, but only gave me 1 track.

You mean the stuff i wrote with -i lavfi above? ..That's just ffmpeg debugging info, i attempted to track down where the problem in ffastrans or better ffmpeg is regarding your problem. In my mind you discovered a little bug, now we can choose if we workaround this in ffastrans or we open a ticket with ffmpeg. But thats more internal stuff, i wrote it here so we have it documented in first place. @admin and the rest of the team will earlier or later discuss this internally.

[Conniver]

Well, you have done it again.

I was looking for something like the -map 0:a but could not find it in my searches and with my minimal FFmpeg knowledge,
that's why I ended up with a messy solution. I've started to read through the FFmpeg documentation to get a basic grip of all the command and options.

Code: Select all

ffmpeg -i input.MXF -map 0:v -map_metadata -1 -sn -s 1920x1080 -g 12 -b:v 6500k -map 0:a -b:a 320k -preset fast -c:v h264_nvenc -pix_fmt yuv420p -preset fast -level:v 4.1 -profile high output.mp4 -y

This worked for both Sony and Arri audio setup.
I'll read about all the different options you have used, and refine it for my use. I'll have to get a better understanding of FFmpeg.

For now, I'll try to make a workflow that filters out footage I know will fail on the h.264 processor node and send them to a custom node.

thank you.