How to handle so called pre-charged frames (XDCAM HD) in FFAStrans

[Silicon]

Hello folks
Do you have any advice how to properly handle (i.e. stripe out) so called pre-charged frames, created by Omneon (Harmonic) videoservers while recording in LongGOP codec (e.g. XDCAM HD), in FFAStrans?
Thanks

[FranceBB]

Stripe out?
I work with Omneon too!
They're the videoserver we use over here.
Omneon puts a metadata which says how many frames do not have to be decoded, which is exactly what you described as "pre-charge" value.
Now, what do you want FFAStrans to do?
Pass this value over? Ignore the value and remove it?
Read the value and trim those frames out then encode without this value?
I'm sorry for asking those question but I didn't actually get what's your goal.

[emcodem]

Aye Silicon,

I guess @FranceBB, the goal here is just to decode the file correctly and conforming to the standard. Also, the Origin Metadata tells what frames cannot be displayed (but MUST BE decoded). The decoder must be in deed feed with the defective frames of the first Open GOP in the file at the start, otherwise it could not decode the actually visible frames. It is something that ffmpeg will not do (at least not in the next 10 years i guess hehe)

Thats kind of an issue because none of our decoders nor ffmpeg supports Precharce (Origin) and Rollout out of the box. In fact, ffmpeg don't support origin AND rollout. That said, if you Transcode (so decode/encode) the file, it is not a MUST to honor origin and rollout because after transcoding the mpeg structure of start and end will not be defective anymore. But of course the duration will differ from the original (up to 15-16 frames longer).

One way to deal with it is working around like this:
1) parse the origin value and duration in frames using the tool on ffastrans.com/share (folder: support/Tools/GetMXFDuration.zip)
2) Note that you have to add the Origin value to the Duration value, %s_parsed_duratoin% = %s_parsed_duratoin%+%s_parsed_origin%
3) use A/V Decoder and after that, use a custom avisynth script , content: m_clip = trim(m_clip,%s_parsed_origin%,%s_parsed_duration%)


Downloads:
1) https://ffastrans.com/share (folder: support/Tools/GetMXFDuration.zip)
Analyzer, a custom tool that can tell you the exact duration that an Omneon Server would recognize when importing the same file and a few more things that usual analyzers don't tell you about MXF files.


2) https://ffastrans.com/share (folder: samples/MXF_TESTFILE_ORIGIN_5_ROLLOUT_1.mxf)
Using this file as a source file for testing, you will be able to visually see if origin and rollout have been processed correctly. All frames that are Origin or Rollout have a special colour and text.

Regex for getting the Duration (use in $regext function in a populate proc): Duration.*? : (\d+)
Regex for getting the Origin (use in $regext function in a populate proc): SPH Origin.*? : (\d+)


The Analyzer tool output will look like this, you only care about the Duration (Omneon decoder) and the SPH Origin value. The Lastsystemitem userdate is buggy, don't count on the value to be what you think (it is the first systemitem time of the last body partition in the file) but all other values are pretty accurate.

Code: Select all

Firstsysitem   Userdate: 10:12:03.13
Lastsystemitem Userdate: 10:12:03.13
Duration  (Omneon decoder) : 12(00:00:00:12)

Materialpackage:
MPH Duration: 18446744073709551615 (error/undefined)
MPH Origin  : 0
MPH StartTC : 10:12:03:15

Sourcepackage:
SPH Duration: 18446744073709551615 (error/undefined)
SPH Origin  : 2
SPH StartTC : 10:12:03:13

Example workflow (notice that we take care about the Origin but not the rollout, lemme know if you need to handle rollout as well):

emcodem_Handle_XDCAM_Origin.json

(5.42 KiB) Downloaded 291 times

Further actions needed:
-) We would also need to reset the stat timecode to the one from Sourcepackage because our analyzers will spit out the one from materialpackage
-) Handle rollout?

[FranceBB]

Totally on spot, as always, emcodem.
Thank you very much indeed, this is gonna be extremely useful!!

[Silicon]

Hi
@FranceBB: I'm sorry, I should have asked the question more clearly.
@emcodem: yes, you have understood my question correctly. The goal is to remove from video essence those frames, which are not valid part of video essence from “human point of view”, even though they are valid part of video essence from GOP structure point of view.

[emcodem]

goal is to remove from video essence those frames, which are not valid part of video essence from “human point of view”, even though they are valid part of video essence from GOP structure point of view

Remove from essence sounds totally different than the solution i provided above, it sounds more like you want to render the first visible frame along with the rest of its GOP to a closed GOP and "copy" the rest of the original essence. Same at Rollout. That would be a lot harder to do...
Is that what u want to Do? Maybe you check out what i provided above and we go from there...

[Silicon]

Hi emcodem
Sory for late reply, I was busy with other stuff.
Thanks a lot for GetMXFDuration tool and workflow provided - I have tested it (using your sample file MXF_TESTFILE_ORIGIN_5_Rollout_1.mxf) and it does what I have been looking for with precharge frames (i.e. they are removed = cut out), but not with rollout frames - it would be nice to have as well

Just to explain the background of my requirement: Currently I'm using Harmonic WFS to process Omneon XDCAM HD422 file ... WFS is able to hnadle correctly both precharge and rollout frames:
- your test: SourcePackage\TimeCode_FirstFrame: 00:09:59:20 / duration: 00:01:00:05 (1505 frames)
- processed by WFS: SourcePackage\TimeCode_FirstFrame: 00:10:00:00 / duration: 00:01:00:00 (1500 frames)

So I need to achieve identical results using FFAstrans if possible. With your GetMXFDuration tool and workflow we are almost there
Thank you.

[emcodem]

OK, well thats a start at least.
i just had an idea ....just needed to subtract -1 from the final calculated duration because in avisynth trim(), the endpoint is "included" in the final duration. As our analyzed duration does NOT include the Rollout Frame, the calculated endpoint should work correctly in any possible case (as long as the analyzing tool spits out the correct duration).

Try this please:

emcodem_Handle_XDCAM_Origin_Rollout.json

(7.78 KiB) Downloaded 291 times

Disclaimer: as the origin and rollout along with all connected startimecode and duration values was not specified clearly in the first version of SMPTE 377 2004, the workflow only works 100% correct with
-) omneon files
-) other files that have NO Origin/Rollout should be fine too
-) BUT files from other Vendors that have Origin/Rollout might work differently than omneon so the duration and starttc could be wrong for those.

[Silicon]

Thanks a lot emcodem!
I'll test and let you know the result.
My plan is to apply this workflow to Omneon recorded files.

[emcodem]

My plan is to apply this workflow to Omneon recorded files.

Haha, sounds like your next question is like how this can be done "while record"