How do I input OP Atom files to create OP1A

[Julica67]

Tried a few scenarios and reading the array to populate variables is not so easy depending on the number of tracks and the path structure. For the proof of concept I think it is a valuable test. I was not able, though to create an OP1A as FFmpeg says unsupported data track although the original Avid file is XDCAM.
I think it would be really great if a new Monitor processor is implemented so, the files from a folder are grouped and a mapping option is available.
I believe this could open the door for something much wider in scope like a timeline rendering, maybe?

[emcodem]

Hehe, timeline rendering is a very huge project, i guess we will leave that to other projects as described in the aaf processing thread I sense we are too less people to work on a project in that size
In the aaf processing thread, my conclusion was that supporting such complex stuff is something that needs huge budget and constant development, so a company must to it. E.g. Pixar Studios: https://github.com/PixarAnimationStudios/OpenTimelineIO ...in case they pop out with something useable in future, we can for sure come up with some simple timeline rendering


To keep modularity i would always keep any "sorting, grouping" and such to separate processors. E.g. the monitor does monitoring, the group processor does grouping etc... After all we have a workflow engine exactly for the reason that we do not want all functionality in a single software component.
For your case, I'd probably come up with a custom processor that executes the createaaf stuff and creates a corresponding ffmpeg line but first we need to proof that it works. As i dont have an avid, we should concentrate on the createaaf tool, it must be proven that it finds all audios to a video.

Can you try with "-dn" option in the ffmpeg command, it ignores data tracks...

EDIT:
for further proof of concept, i created quickly a custom processor for your usecase. What it does is to automatically execute createaaf and spit out the generated input and map list.
https://drive.google.com/open?id=1EfjJi ... HaVLC2l04B
Look at the wiki in the "Plugin Deployment" section how to install it. After copying the files and restarting ffastrans, you should find a new processor in Decoders section. Make sure you work with the latest ffastrans version.
-) Configure %s_source% as "Array of OPAtom Files (yeah, the label is wrong but its just a test)
-) Configure some user variable as "Created ffmpeg line"
-) submit one Audio or video file from your Atom structure to the processor

Here is a workflow where that sucessfully rewraps a file:

Rewrap_OPAtom.json

(2.08 KiB) Downloaded 437 times

As said, the custom processor above will only output -i and -map list, so in a commandline processor, you still need to provide the codec and -dn option like that:

Code: Select all

%comspec% /C ""%s_ffmpeg%" %s_output_of_rewrap_proc% -codec copy -dn c:\temp\rewrap.mxf"

[Julica67]

Hi emcodem,

Thank you for your effort. I have tried the new customer processor and managed to have several clips re-wrapped. I would like to run more tests on some other clips but so far it seems to be going fine, only one clip got too many files - wondering if it is not multi-res...need to check. What field is createaaf --locate using to match the files? It would be possible to extract also additional metadata such as Tape Name, Project when grouping the atom files?
If you could integrate the custom processor with the other processors I could try different transcodes as well.
Thank you for the info about OpenTimeLine - I will get into that thread to keep this one clean.

[emcodem]

Hey Julica!

Sounds promising, if you know whats causing the "too much
Let me answer each question separately:

only one clip got too many files - wondering if it is not multi-res...need to check.

If you find out what it is, it would be very nice if you could get me a very short example file group so i can test a possible fix for it

What field is createaaf --locate using to match the files?

It is the Materialpackage UMID, but i sense it should be the SourcePackage.

It would be possible to extract also additional metadata such as Tape Name, Project when grouping the atom files?

This is already possible using stock functions of ffastrans, read more here "jsonget Usage Example to retrieve values from ffprobe:": http://www.ffastrans.com/wiki/doku.php? ... _functions

If you could integrate the custom processor with the other processors I could try different transcodes as well.

This is very good thinking but unfortunately it requires some engineering. The reason is that all encode nodes are dependent on %s_source% to be a single file. The best bet i have in that direction is to create an avisynth script file that loads the located files. We will need to test if the avisynth ffms2 decode module is able and willing to do that with opatom files.
But if we use avisynth decoding, we cannot use "codec copy" anymore.. hmmm... Well we can use it but the codec that comes out from avisynth is just uncompressed yuv images

[Julica67]

Hey Julica!

Sounds promising, if you know whats causing the "too much
Let me answer each question separately:

only one clip got too many files - wondering if it is not multi-res...need to check.

If you find out what it is, it would be very nice if you could get me a very short example file group so i can test a possible fix for it

where would you like me to upload the sample?

What field is createaaf --locate using to match the files?

It is the Materialpackage UMID, but i sense it should be the SourcePackage.

The source package UMIDs should be unique and the Materialpackage UMID would define a resolution - at least this is my understanding. In this case selecting by MaterialPackage UMID should return a specific resolution. I feel theses files (created by a third party) might be in violation of the uniqueness.
It might be interesting though to expose both options (source package UMID and Materialpackage UMID) in aafcreate so, it can be selected all resolutions for a clip or just a specific one.

It would be possible to extract also additional metadata such as Tape Name, Project when grouping the atom files?

This is already possible using stock functions of ffastrans, read more here "jsonget Usage Example to retrieve values from ffprobe:": http://www.ffastrans.com/wiki/doku.php? ... _functions

I will lock into it. My goal was to either extract or populate some variables for downstream processing.

If you could integrate the custom processor with the other processors I could try different transcodes as well.

This is very good thinking but unfortunately it requires some engineering. The reason is that all encode nodes are dependent on %s_source% to be a single file. The best bet i have in that direction is to create an avisynth script file that loads the located files. We will need to test if the avisynth ffms2 decode module is able and willing to do that with opatom files.
But if we use avisynth decoding, we cannot use "codec copy" anymore.. hmmm... Well we can use it but the codec that comes out from avisynth is just uncompressed yuv images

How is the P2 structure handled? I thought it would be similar as P2 files are OP-Atom as well. It would be great to be able to take the Avid OP-Atom and transcode to an OP1A for distribution outside Avid or even creating another resolution (MXF OP-Atom with Avid project structure) for let's say editing on low res.

[emcodem]

Hey, it is so nice that you keep on discussing this topic! This way i am certain something useful comes out in the end.
Please upload the stuff here https://drive.google.com/open?id=1_JyEN ... IuSobn1S5S

I'll take care about the rest of your queries in a few hours hopefully.

[Julica67]

Hi emcodem, I don't have the permissions to upload the sample to google drive.

[emcodem]

@Julica67 sorry, try this one
https://drive.google.com/drive/folders/ ... sp=sharing

[admin]

Hi Julica67,

The "P2" monitor monitors xml's that are specific for the P2 structure. From the xml it derives what files belongs to what clip and then gather all for concatenation and merging. Thus the structure MUST adhere to P2, so this is far from generic and the current code can't be used for much else.

-steinar

[Julica67]

Hi Julica67,

The "P2" monitor monitors xml's that are specific for the P2 structure. From the xml it derives what files belongs to what clip and then gather all for concatenation and merging. Thus the structure MUST adhere to P2, so this is far from generic and the current code can't be used for much else.

-steinar

Understood. I was hoping a new monitor type would be implemented for getting the Avid OP Atom files form a folder, matched by the MaterialPackage UMID or Source UMID as an option.
Should I better create a Feature Request for this?