Workflow to sort and join AVID and Editcam .MXF-files

[ThomasM]

Hi,

here is what I did for my work to archive old AVID / EDITCAM-projects.

----------------------------------------------------------
WARNING: This Workflows still need improvment. Use at your own risk. Test with copies of your files. It is always a good idea to work with copies!
----------------------------------------------------------

We have a lot of old projects filmed with Ikegami Editcam HD HDN-X10. The PostProduction was done mostly with AVID MediaComposer.

The work with Editcam HD an MC is straight forward as there is no need to ingest or transcode the material. It is ready-to-edit as soon as the FieldPack (the recording-HDD / SSD) is conected to the NLE-Computer running AVID MC.

The drawback is in the details of the managing-logic of AVID MC. When it comes to archiving the source-material there is a way to consolidate the bins to an external archive-hdd. when this is done, all the files are in a file-structure which is not possible to rename or manage by hand easily.

As some projects were carried out over a longer time than others, the archiving of AVID-MC-Projects had some hurdles. Our Archive-LTO-System is not connected directly to the NLE-Machine. So everything was carried out by hand and the files and folders were collected manually. Guess what can happen by doing so........

And in fact. By looking at the archive-folders, it was easy to see that there are LOTS of duplicates and PreComputes. And it is clear - this is not to be solved manually.

The file-structure accepted by AVID MC is

DRIVE:\AVID MediaFiles\MXF\[NUMBER]

where [NUMBER] can be anything, but AVID always starts with "1". as long as the number of files do not exceed a maximum number the files are stored exactly here. I do not know the number, but it´s in the thousands. When the maximum number is reached, the folder will be renamed to "2" and a new folder "1" is created. AVID MC takes care of its files by having databases in the folders, *.mdb.

So we began to archive the original material by directly copying this to a folder like

20161002

where 2016 is the year, 10 is the month and 02 the day of month.

This was our first idea to overcome the AVID-Structure by having the change to completely check the integrity of our material on different storage-systems. AVID-MC-NLE accepts this number.

In some bad cases there were backups made by simply copy the folder-structure from above from the NLE-machine. In the above mentioned "1"-folder is also every PreCompute-File from AVID MC (Title-renders, fades, mattes, FX... even Audio-dissolves and fades). As these files are re-created when a project is opened again, there is no need to archive them. In the end, you cannot even use or view them properly in a third-party-application.

The other drawback by handling Editcam-files is the then used FAT-16-File-system which only handles file-sizes up to 2 GB. Video-files which exceed this limit are named *.0001.mxf, *.0002.mxf [...] *.nnnn.mxf, where * is always the filename in the flavour of NMZTPR0 for example. A complete filename is

NMZTPR0V.MXF

for a video-file. Notice the "V" directly by the following dot.

A complete filepack with video and 2ch audio is

NMZTPR0V.MXF
NMZTPR0A.MXF
NMZTPR0B.MXF

Notice the "A" and "B" for channel 1 and channel 2 audio directly by the following dot.

A complete filepack with longer video and 2ch audio is

NMZTPR0V.MXF
NMZTPR0V.0001.MXF
NMZTPR0V.0002.MXF
NMZTPR0V.0003.MXF
NMZTPR0A.MXF
NMZTPR0B.MXF

Channel 3 and 4 audio would be

NMZTPR0C.MXF
NMZTPR0D.MXF


For archiving-purposes we wanted to have a filepack with

- all originally filmed .mxf in one .zip-file
- based on a day-by-day-structure
- sort out any PreComputes
- sort out any other Company-MXF (Sony, Panasonic...)
- Check for duplicates
- delete duplicates with the same xxHash
- give chance to check duplicates with different xxHash manually
- generate missing files in sequenced VideoFiles (in this portion a hint tells the viewer that this portion of video is missing)
- generate missing Audio-Files (a sinewave will be used then)
- Logging for duplicates with same xxHash
- Logging for duplicates with different xxHash
- create universal-usable Video-Files containing video and audio
- get rid of the sequenced Video-Files

First step is to consolidate ALL *.mxf material (in the archived folder-structure) we can get from the Archive on one big external RAID-System in a single folder called "ARCHIVE". Every sort- and convert-function will be carried out on this RAID. We do have a dedicated machine for running FFAStrans on it to get the job done.

The whole procedure is splitted in eight single workflows, carried out one after another.

1. MXF_01__Sort

This workflow will search the hole "ARCHIVE"-Folder for *.mxf and MOVE it in a folder "MXF-sortiert". The structure of this folder is

[DRIVE]:\
[CAMERA-MANUFACTURER]\
[DATE_YYYYMMDD]\
[V]\ <- in case of Ikegami Editcam
[MULTI_%ORIGINAL_NAME%] <- in Case of sequenced Video-Files
[A1]\ <- in case of Ikegami Editcam
[A2]\ <- in case of Ikegami Editcam
[%ORIGINAL_PATH~1%]\ <- in case of Other Manufacturer than Ikegami

This Workflow also detects the video- and audio-properties for that day and writes this to [DATE_YYYYMMDD]\FormatA.txt and FormatV.txt.

This Workflow also creates xxHash. This is stored in [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD].

This Workflow also detects AVID-MC PreComputes and MOVES them into [DRIVE:\AVID_PreComputes\DATE_YYYYMMDD].

This Workflow also detects duplicates:
In case of matching xxHash the last file will be deleted.
In case of different xxHash the last file will be renamed to [%ORIGINAL_NAME%_GUID()] if several duplicates appear. This duplicates are stored in [DRIVE:\MXF-DUBLETTEN-Diff-Hash].
In both cases there is a LOG storing this actions in [DRIVE:\Dubletten-diff-hash.log] or [DRIVE:\Dubletten.log] respectively.


2. MXF_01a__MOVE .0000

This is a short one. In Case there are segmented Video-Files all these files belonging to one filename are moved to a subsequent folder ( [MXF] ). The original

XXXXXXXV.MXF

will the be renamed to

XXXXXXXV.0000.MXF


3. MXF_02__CheckMissingMultiFiles

Here we have to use the external .BAT-programs. These will check for missing files in the video-sequence. The .BAT have to be saved in the FFAStrans-folder or the path has to be changed.

This workflow needs some improvement. for accurate synchronicity I have to check the filesizes of the different recording-flavours of the Editcam. These are not too much but time-consuming.....

Over all, especially when there is no missing file, this Workflow will work.

4. MXF_02a_Generate_Missing_MultiFiles

In case of a missing video-segment a .MOV-video-segment will be created with [VideoNichtVorhanden.jpg] which has to be saved at the FFAStrans-Folder.

This is a MOV with DNxHD-content to not confusing the system with new .MXF-files.

5. MXF_03_Join Multifiles_for singleWF

This one concatenates the segmented video-files. When this is done, the

XXXXXXXV.0000.MXF is MOVED to [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD\XXXXXXXV.MXF].

All the other MXF involved in this process are also MOVED to [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD\XXXXXXXV.MXF].

The generated .MOV are deleted later on.

6. MXF_04_Join_And_Check_missingFiles_Single

This is a big one. It checks if there are video- or audio-files are missing.

In case of missing video refer to Workflow "4. MXF_02a_Generate_Missing_MultiFiles".

In case of missing audio a sinewave will be created.

The video and audio is then joined into a .MOV saved to

[DRIVE:\MOV\DATE_YYYYMMDD\YYYY-MM-DDTHH;MM;SS--XXXXXXX.mov

Notice that the last letter - A,B,C,D or V is dropped in the name as this is now a joined file.

The original MXF are MOVED to [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD\XXXXXXXV.MXF].

7. MXF_05_ConcatMOV_ZIP_Archive

This is a straight-forward one. It simply concats the joined .MOVs from Workflow 6 into a continuous .MOV. Then a Research-MP4 is created. the involved MXF are ZIPped.
7z has to be in the FFAStrans-Folder.
All these is stored in to [DRIVE:\Archive_Filepacks\DATE_YYYYMMDD].
A JSON-Logfile with xxHashes of each colmpletes the FilePackage.

Result is

[DRIVE]:\
[Archive_Filepacks]
[\DATE_YYYYMMDD]
GUID.ZIP <- Archived original MXF-Files
GUID.MOV <- Joined A/V one-Day NLE-Ready
GUID.MP4 <- Joined A/V one-Day Research-copy
DATE_YYYYMMDD.JSON <- JSON-Log-File


8. n/a

This is only for convenience - it would delete (RD) all empty directories.


Maybe someone can use this Workflows or parts of it.

If someone knows improvements or tips for doing this more convenient - please let me know.

Enjoy.

[momocampo]

Hello Thomas,
My god, it's a huge work !!
But as I know Avid System, I wonder why you need to do such work. I understood you want to keep AVid projects and the original medias but wouldn't it be easier to export your media from MC instead of modify your Atom files? Or I didn't understand (really possible because English isn't my first language...).
Anyway, you export all your master clips files, you save your avid project and if you have to work again on this project, just do a batch-import and voila! All your precomputes can be recreate by avid so no need to keep them.

I may not have understood correctly but please tell me more because it is really the same work I must do often and I'll like to see if there is no other easiest way (even I imagine that you have already searched a lot...)
Thanks

Benjamin

[ThomasM]

Hi Benjamin,

thank you for your reply!

Now, the workflow with an AVID-MC is clear. But what if you do not have an AVID-MC? Or even worse, what if AVID drops support for some reason? What if someone wants to explore the original recorded media 50 years in the future? And for convenience-reasons, nobody really wants to click every single clip...

So we had the idea to put the clips to an order on a day-to-day base. So, everything recorded on January 23rd in 2017 is put in one folder (or Bin). everything recorded on January 27rd in 2017 is put in one folder (or Bin). and so on. To get universal access to this AND to archive the original MXF we would have to
- consolidate every single bin for compatibility with AVID-MC and Archive the original
AND
- create a timeline with all the clips from a bin and play out as ProResHQ or DNxHD for having a continuous researchible file for universal access.

You can imagine that this is no fun on some 100 Projects.

As many people worked here the last 15 Years, all doing different kind of Backup, we are faced with a multitude of File-structures and backup-strategies. Bitter is that.

The AVID-Projects are no problem, as they are all in one folder. We will archive each .avp (AVID-Project-folder for sure) with the original mxf in our MAM-System.

As figured out in my first post, the old File-systems are causing large video-files to split-up, they leave segmented files. So the work to do is to join all the video-segments and to join the accompanying audio-files. All these files are combined to a contimuous file, as said earlyer, on a Day-to-day basis. so we end up with

1. A zip-File with all the original EDITCAM-MXF-Clips. This .ZIP is fine with our MAM-System. This is the Original-Archive-Master. If someone wants to see the original .MXF - here he will find it.

2. A .MOV-File with all the clips of a day as if it was a long Cassette-Tape recorded THAT day. The running-order of the clips in this file is in the right order based on time-of-the-day

3. A .MP4-File derived directly from the .MOV-File from 2. This is for quick research over Internet.

4. A JSON-compatible LOG-File where the xxHash and names of the files from 1. 2. and 3. are stored.

The .ZIP, .MOV and .MP4 get GUID-names as our MAM-System is aware of these.



so long
Thomas

[emcodem]

@ThomasM it is so nice from you that you share your workflow!
Can you explain where you still have problems or if there is anything that you found "missing functionality" in ffastrans while generating those workflows?
Do you check the "Custom Processor Download" section in the wiki from time to time? ...i think the "Files Find" and the "Files Delete" Processor combination could be your friends maybe in future too...

[momocampo]

Hello Thomas,

Thanks to explain me more I understand now your priority and your needs. I especially understand that you have to catch up from archiving
As I said, it was a really big work so bravo!
But I have not the same opinion as you about lifetime of archives. I am surely not working on same kind of products as you but for me, archives don't need to be so "old". I also know archives must be keep for a long time for specific production
My archives are for short programs so really easiest to process. But I think of it now, and can you use aaf embedded(or not) export? You will keep your edit, all your tracks and handles if you want. That's the aaf that I use with a mp4 file for viewing. Maybe a possibility?
Thanks again for your precise explanation
Cheers.

Benjamin

[ThomasM]

@ThomasM it is so nice from you that you share your workflow!
Can you explain where you still have problems or if there is anything that you found "missing functionality" in ffastrans while generating those workflows?
Do you check the "Custom Processor Download" section in the wiki from time to time? ...i think the "Files Find" and the "Files Delete" Processor combination could be your friends maybe in future too...

Hey emcodem,
nice to hear from you! Yes, I like to share my WFs. And the Custom Processors will be used definitely. I have to admit that I was not really aware that there are some already...

What I am missing is a Node for checking for missing files in a sequence. This could be great for checking image-sequence integrity also.

Also, maybe with in conjunction with the find-file-node, a concat-node with Channel-Mapping would be grrrrreat.......

so long,
all the best
tom

[ThomasM]

Hello Thomas,

Thanks to explain me more I understand now your priority and your needs. I especially understand that you have to catch up from archiving
As I said, it was a really big work so bravo!
But I have not the same opinion as you about lifetime of archives. I am surely not working on same kind of products as you but for me, archives don't need to be so "old". I also know archives must be keep for a long time for specific production
My archives are for short programs so really easiest to process. But I think of it now, and can you use aaf embedded(or not) export? You will keep your edit, all your tracks and handles if you want. That's the aaf that I use with a mp4 file for viewing. Maybe a possibility?
Thanks again for your precise explanation
Cheers.

Benjamin

Hey Benjamin,
nice that I could clear some things up - I understand that different producing-techniques do not make use of long-term-archiving. As we do have lots of Time-Witnesses and documentary of now gone industries, we use our material as stock-footage and offer it also to film-makers or television. We are a government-institution and do have to keep track of nearly every recorded item. We do work with old-stock-film also (8, 9,5 16, 35mm and now obsolete video like VCR, video2000, Betamax, 1/2" open reel...), but this is an entirely different story... Nevertheless, when we were on AVID we also cut these old digitised material into our productions. As these are digitized to totally different standards (16 and 35mm-film as 2k / 4k-DPX for example) we do not need the transcoded clips from AVID. so we want to filter them out.
Your idea with AAF with embedded mediafiles works.... as long as you stay on the AVID-MC-platform. We discarded this kind of workflow as we were always looking for different NLEs and it is always hard to tell if an AAF is misinterpreted by a third-party-application. Now we are running some RESOLVE-NLE. AVID is only still on one old machine in Version 8.4 for reference ,-)

all the best
tom

[emcodem]

What I am missing is a Node for checking for missing files in a sequence. This could be great for checking image-sequence integrity also.
Also, maybe with in conjunction with the find-file-node, a concat-node with Channel-Mapping would be grrrrreat.......

Hey Tom,
understood your first point, i think this is possible already using the file count output of the files find node. Basically you need to find a way to "include" only the filenames from this sequence in the files find processor and let it check the count of files with that *name*.
I believe the current files find node will output the files sorted ascending anyway, so you just need to grab the last item of the found file "list" and grab the number in the filename. Then you compare the number with the found items.
Not sure if this is worth a special custom processor because too few people would use it i fear.

I like the idea about a "concat" node, what you mean by audio mapping? You imagine the input files having different audio configs or all have the same audio config and you just want to map channels in the "output" file?

[momocampo]

Hey Thomas,
You are unfortunately right, aaf isn't really compatible with all thid-party application and it is a real problem when you have to share different edit on different editing software. I must admit Resolve is a really good software and it is enough "open" to accept a lot of formats. But, as you know, it is often easier to import into a software than export...New customers are welcome, old customers not -> no real compatibility between softwares.
Anyway, I think we have all our "little issues" that we have to resolve. That's why it is a good thing to share our experiences, it's the way to know that we can help others.
At last, must thanks to @Encodem and obviously @Steinar for their constant help in all my questions and especially for having supported my big gaps ...
Glad you are with us Thomas
Cheers.

Benjamin

[ThomasM]

Hey Benjamin,

thank you for your kind words - same to you

New customers are welcome, old customers not -> no real compatibility between softwares.

so true...

That is why I like free software. "free" in the meaning of "open". It is really great work from @Steinar and not to forget @emcodem. This Software not only safes money, we can talk to the developers and we can help to improve this app. Thanks to all supporters!

all the best
tom