Workflow to sort and join AVID and Editcam .MXF-files
Posted: Wed May 13, 2020 2:56 pm
Hi,
here is what I did for my work to archive old AVID / EDITCAM-projects.
----------------------------------------------------------
WARNING: This Workflows still need improvment. Use at your own risk. Test with copies of your files. It is always a good idea to work with copies!
----------------------------------------------------------
We have a lot of old projects filmed with Ikegami Editcam HD HDN-X10. The PostProduction was done mostly with AVID MediaComposer.
The work with Editcam HD an MC is straight forward as there is no need to ingest or transcode the material. It is ready-to-edit as soon as the FieldPack (the recording-HDD / SSD) is conected to the NLE-Computer running AVID MC.
The drawback is in the details of the managing-logic of AVID MC. When it comes to archiving the source-material there is a way to consolidate the bins to an external archive-hdd. when this is done, all the files are in a file-structure which is not possible to rename or manage by hand easily.
As some projects were carried out over a longer time than others, the archiving of AVID-MC-Projects had some hurdles. Our Archive-LTO-System is not connected directly to the NLE-Machine. So everything was carried out by hand and the files and folders were collected manually. Guess what can happen by doing so........
And in fact. By looking at the archive-folders, it was easy to see that there are LOTS of duplicates and PreComputes. And it is clear - this is not to be solved manually.
The file-structure accepted by AVID MC is
DRIVE:\AVID MediaFiles\MXF\[NUMBER]
where [NUMBER] can be anything, but AVID always starts with "1". as long as the number of files do not exceed a maximum number the files are stored exactly here. I do not know the number, but it´s in the thousands. When the maximum number is reached, the folder will be renamed to "2" and a new folder "1" is created. AVID MC takes care of its files by having databases in the folders, *.mdb.
So we began to archive the original material by directly copying this to a folder like
20161002
where 2016 is the year, 10 is the month and 02 the day of month.
This was our first idea to overcome the AVID-Structure by having the change to completely check the integrity of our material on different storage-systems. AVID-MC-NLE accepts this number.
In some bad cases there were backups made by simply copy the folder-structure from above from the NLE-machine. In the above mentioned "1"-folder is also every PreCompute-File from AVID MC (Title-renders, fades, mattes, FX... even Audio-dissolves and fades). As these files are re-created when a project is opened again, there is no need to archive them. In the end, you cannot even use or view them properly in a third-party-application.
The other drawback by handling Editcam-files is the then used FAT-16-File-system which only handles file-sizes up to 2 GB. Video-files which exceed this limit are named *.0001.mxf, *.0002.mxf [...] *.nnnn.mxf, where * is always the filename in the flavour of NMZTPR0 for example. A complete filename is
NMZTPR0V.MXF
for a video-file. Notice the "V" directly by the following dot.
A complete filepack with video and 2ch audio is
NMZTPR0V.MXF
NMZTPR0A.MXF
NMZTPR0B.MXF
Notice the "A" and "B" for channel 1 and channel 2 audio directly by the following dot.
A complete filepack with longer video and 2ch audio is
NMZTPR0V.MXF
NMZTPR0V.0001.MXF
NMZTPR0V.0002.MXF
NMZTPR0V.0003.MXF
NMZTPR0A.MXF
NMZTPR0B.MXF
Channel 3 and 4 audio would be
NMZTPR0C.MXF
NMZTPR0D.MXF
For archiving-purposes we wanted to have a filepack with
- all originally filmed .mxf in one .zip-file
- based on a day-by-day-structure
- sort out any PreComputes
- sort out any other Company-MXF (Sony, Panasonic...)
- Check for duplicates
- delete duplicates with the same xxHash
- give chance to check duplicates with different xxHash manually
- generate missing files in sequenced VideoFiles (in this portion a hint tells the viewer that this portion of video is missing)
- generate missing Audio-Files (a sinewave will be used then)
- Logging for duplicates with same xxHash
- Logging for duplicates with different xxHash
- create universal-usable Video-Files containing video and audio
- get rid of the sequenced Video-Files
First step is to consolidate ALL *.mxf material (in the archived folder-structure) we can get from the Archive on one big external RAID-System in a single folder called "ARCHIVE". Every sort- and convert-function will be carried out on this RAID. We do have a dedicated machine for running FFAStrans on it to get the job done.
The whole procedure is splitted in eight single workflows, carried out one after another.
1. MXF_01__Sort
This workflow will search the hole "ARCHIVE"-Folder for *.mxf and MOVE it in a folder "MXF-sortiert". The structure of this folder is
[DRIVE]:\
[CAMERA-MANUFACTURER]\
[DATE_YYYYMMDD]\
[V]\ <- in case of Ikegami Editcam
[MULTI_%ORIGINAL_NAME%] <- in Case of sequenced Video-Files
[A1]\ <- in case of Ikegami Editcam
[A2]\ <- in case of Ikegami Editcam
[%ORIGINAL_PATH~1%]\ <- in case of Other Manufacturer than Ikegami
This Workflow also detects the video- and audio-properties for that day and writes this to [DATE_YYYYMMDD]\FormatA.txt and FormatV.txt.
This Workflow also creates xxHash. This is stored in [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD].
This Workflow also detects AVID-MC PreComputes and MOVES them into [DRIVE:\AVID_PreComputes\DATE_YYYYMMDD].
This Workflow also detects duplicates:
In case of matching xxHash the last file will be deleted.
In case of different xxHash the last file will be renamed to [%ORIGINAL_NAME%_GUID()] if several duplicates appear. This duplicates are stored in [DRIVE:\MXF-DUBLETTEN-Diff-Hash].
In both cases there is a LOG storing this actions in [DRIVE:\Dubletten-diff-hash.log] or [DRIVE:\Dubletten.log] respectively.
2. MXF_01a__MOVE .0000
This is a short one. In Case there are segmented Video-Files all these files belonging to one filename are moved to a subsequent folder ( [MXF] ). The original
XXXXXXXV.MXF
will the be renamed to
XXXXXXXV.0000.MXF
3. MXF_02__CheckMissingMultiFiles
Here we have to use the external .BAT-programs. These will check for missing files in the video-sequence. The .BAT have to be saved in the FFAStrans-folder or the path has to be changed.
This workflow needs some improvement. for accurate synchronicity I have to check the filesizes of the different recording-flavours of the Editcam. These are not too much but time-consuming.....
Over all, especially when there is no missing file, this Workflow will work.
4. MXF_02a_Generate_Missing_MultiFiles
In case of a missing video-segment a .MOV-video-segment will be created with [VideoNichtVorhanden.jpg] which has to be saved at the FFAStrans-Folder.
This is a MOV with DNxHD-content to not confusing the system with new .MXF-files.
5. MXF_03_Join Multifiles_for singleWF
This one concatenates the segmented video-files. When this is done, the
XXXXXXXV.0000.MXF is MOVED to [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD\XXXXXXXV.MXF].
All the other MXF involved in this process are also MOVED to [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD\XXXXXXXV.MXF].
The generated .MOV are deleted later on.
6. MXF_04_Join_And_Check_missingFiles_Single
This is a big one. It checks if there are video- or audio-files are missing.
In case of missing video refer to Workflow "4. MXF_02a_Generate_Missing_MultiFiles".
In case of missing audio a sinewave will be created.
The video and audio is then joined into a .MOV saved to
[DRIVE:\MOV\DATE_YYYYMMDD\YYYY-MM-DDTHH;MM;SS--XXXXXXX.mov
Notice that the last letter - A,B,C,D or V is dropped in the name as this is now a joined file.
The original MXF are MOVED to [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD\XXXXXXXV.MXF].
7. MXF_05_ConcatMOV_ZIP_Archive
This is a straight-forward one. It simply concats the joined .MOVs from Workflow 6 into a continuous .MOV. Then a Research-MP4 is created. the involved MXF are ZIPped.
7z has to be in the FFAStrans-Folder.
All these is stored in to [DRIVE:\Archive_Filepacks\DATE_YYYYMMDD].
A JSON-Logfile with xxHashes of each colmpletes the FilePackage.
Result is
[DRIVE]:\
[Archive_Filepacks]
[\DATE_YYYYMMDD]
GUID.ZIP <- Archived original MXF-Files
GUID.MOV <- Joined A/V one-Day NLE-Ready
GUID.MP4 <- Joined A/V one-Day Research-copy
DATE_YYYYMMDD.JSON <- JSON-Log-File
8. n/a
This is only for convenience - it would delete (RD) all empty directories.
Maybe someone can use this Workflows or parts of it.
If someone knows improvements or tips for doing this more convenient - please let me know.
Enjoy.
here is what I did for my work to archive old AVID / EDITCAM-projects.
----------------------------------------------------------
WARNING: This Workflows still need improvment. Use at your own risk. Test with copies of your files. It is always a good idea to work with copies!
----------------------------------------------------------
We have a lot of old projects filmed with Ikegami Editcam HD HDN-X10. The PostProduction was done mostly with AVID MediaComposer.
The work with Editcam HD an MC is straight forward as there is no need to ingest or transcode the material. It is ready-to-edit as soon as the FieldPack (the recording-HDD / SSD) is conected to the NLE-Computer running AVID MC.
The drawback is in the details of the managing-logic of AVID MC. When it comes to archiving the source-material there is a way to consolidate the bins to an external archive-hdd. when this is done, all the files are in a file-structure which is not possible to rename or manage by hand easily.
As some projects were carried out over a longer time than others, the archiving of AVID-MC-Projects had some hurdles. Our Archive-LTO-System is not connected directly to the NLE-Machine. So everything was carried out by hand and the files and folders were collected manually. Guess what can happen by doing so........
And in fact. By looking at the archive-folders, it was easy to see that there are LOTS of duplicates and PreComputes. And it is clear - this is not to be solved manually.
The file-structure accepted by AVID MC is
DRIVE:\AVID MediaFiles\MXF\[NUMBER]
where [NUMBER] can be anything, but AVID always starts with "1". as long as the number of files do not exceed a maximum number the files are stored exactly here. I do not know the number, but it´s in the thousands. When the maximum number is reached, the folder will be renamed to "2" and a new folder "1" is created. AVID MC takes care of its files by having databases in the folders, *.mdb.
So we began to archive the original material by directly copying this to a folder like
20161002
where 2016 is the year, 10 is the month and 02 the day of month.
This was our first idea to overcome the AVID-Structure by having the change to completely check the integrity of our material on different storage-systems. AVID-MC-NLE accepts this number.
In some bad cases there were backups made by simply copy the folder-structure from above from the NLE-machine. In the above mentioned "1"-folder is also every PreCompute-File from AVID MC (Title-renders, fades, mattes, FX... even Audio-dissolves and fades). As these files are re-created when a project is opened again, there is no need to archive them. In the end, you cannot even use or view them properly in a third-party-application.
The other drawback by handling Editcam-files is the then used FAT-16-File-system which only handles file-sizes up to 2 GB. Video-files which exceed this limit are named *.0001.mxf, *.0002.mxf [...] *.nnnn.mxf, where * is always the filename in the flavour of NMZTPR0 for example. A complete filename is
NMZTPR0V.MXF
for a video-file. Notice the "V" directly by the following dot.
A complete filepack with video and 2ch audio is
NMZTPR0V.MXF
NMZTPR0A.MXF
NMZTPR0B.MXF
Notice the "A" and "B" for channel 1 and channel 2 audio directly by the following dot.
A complete filepack with longer video and 2ch audio is
NMZTPR0V.MXF
NMZTPR0V.0001.MXF
NMZTPR0V.0002.MXF
NMZTPR0V.0003.MXF
NMZTPR0A.MXF
NMZTPR0B.MXF
Channel 3 and 4 audio would be
NMZTPR0C.MXF
NMZTPR0D.MXF
For archiving-purposes we wanted to have a filepack with
- all originally filmed .mxf in one .zip-file
- based on a day-by-day-structure
- sort out any PreComputes
- sort out any other Company-MXF (Sony, Panasonic...)
- Check for duplicates
- delete duplicates with the same xxHash
- give chance to check duplicates with different xxHash manually
- generate missing files in sequenced VideoFiles (in this portion a hint tells the viewer that this portion of video is missing)
- generate missing Audio-Files (a sinewave will be used then)
- Logging for duplicates with same xxHash
- Logging for duplicates with different xxHash
- create universal-usable Video-Files containing video and audio
- get rid of the sequenced Video-Files
First step is to consolidate ALL *.mxf material (in the archived folder-structure) we can get from the Archive on one big external RAID-System in a single folder called "ARCHIVE". Every sort- and convert-function will be carried out on this RAID. We do have a dedicated machine for running FFAStrans on it to get the job done.
The whole procedure is splitted in eight single workflows, carried out one after another.
1. MXF_01__Sort
This workflow will search the hole "ARCHIVE"-Folder for *.mxf and MOVE it in a folder "MXF-sortiert". The structure of this folder is
[DRIVE]:\
[CAMERA-MANUFACTURER]\
[DATE_YYYYMMDD]\
[V]\ <- in case of Ikegami Editcam
[MULTI_%ORIGINAL_NAME%] <- in Case of sequenced Video-Files
[A1]\ <- in case of Ikegami Editcam
[A2]\ <- in case of Ikegami Editcam
[%ORIGINAL_PATH~1%]\ <- in case of Other Manufacturer than Ikegami
This Workflow also detects the video- and audio-properties for that day and writes this to [DATE_YYYYMMDD]\FormatA.txt and FormatV.txt.
This Workflow also creates xxHash. This is stored in [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD].
This Workflow also detects AVID-MC PreComputes and MOVES them into [DRIVE:\AVID_PreComputes\DATE_YYYYMMDD].
This Workflow also detects duplicates:
In case of matching xxHash the last file will be deleted.
In case of different xxHash the last file will be renamed to [%ORIGINAL_NAME%_GUID()] if several duplicates appear. This duplicates are stored in [DRIVE:\MXF-DUBLETTEN-Diff-Hash].
In both cases there is a LOG storing this actions in [DRIVE:\Dubletten-diff-hash.log] or [DRIVE:\Dubletten.log] respectively.
2. MXF_01a__MOVE .0000
This is a short one. In Case there are segmented Video-Files all these files belonging to one filename are moved to a subsequent folder ( [MXF] ). The original
XXXXXXXV.MXF
will the be renamed to
XXXXXXXV.0000.MXF
3. MXF_02__CheckMissingMultiFiles
Here we have to use the external .BAT-programs. These will check for missing files in the video-sequence. The .BAT have to be saved in the FFAStrans-folder or the path has to be changed.
This workflow needs some improvement. for accurate synchronicity I have to check the filesizes of the different recording-flavours of the Editcam. These are not too much but time-consuming.....
Over all, especially when there is no missing file, this Workflow will work.
4. MXF_02a_Generate_Missing_MultiFiles
In case of a missing video-segment a .MOV-video-segment will be created with [VideoNichtVorhanden.jpg] which has to be saved at the FFAStrans-Folder.
This is a MOV with DNxHD-content to not confusing the system with new .MXF-files.
5. MXF_03_Join Multifiles_for singleWF
This one concatenates the segmented video-files. When this is done, the
XXXXXXXV.0000.MXF is MOVED to [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD\XXXXXXXV.MXF].
All the other MXF involved in this process are also MOVED to [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD\XXXXXXXV.MXF].
The generated .MOV are deleted later on.
6. MXF_04_Join_And_Check_missingFiles_Single
This is a big one. It checks if there are video- or audio-files are missing.
In case of missing video refer to Workflow "4. MXF_02a_Generate_Missing_MultiFiles".
In case of missing audio a sinewave will be created.
The video and audio is then joined into a .MOV saved to
[DRIVE:\MOV\DATE_YYYYMMDD\YYYY-MM-DDTHH;MM;SS--XXXXXXX.mov
Notice that the last letter - A,B,C,D or V is dropped in the name as this is now a joined file.
The original MXF are MOVED to [DRIVE:\MXF-2-ZIP_Collector\DATE_YYYYMMDD\XXXXXXXV.MXF].
7. MXF_05_ConcatMOV_ZIP_Archive
This is a straight-forward one. It simply concats the joined .MOVs from Workflow 6 into a continuous .MOV. Then a Research-MP4 is created. the involved MXF are ZIPped.
7z has to be in the FFAStrans-Folder.
All these is stored in to [DRIVE:\Archive_Filepacks\DATE_YYYYMMDD].
A JSON-Logfile with xxHashes of each colmpletes the FilePackage.
Result is
[DRIVE]:\
[Archive_Filepacks]
[\DATE_YYYYMMDD]
GUID.ZIP <- Archived original MXF-Files
GUID.MOV <- Joined A/V one-Day NLE-Ready
GUID.MP4 <- Joined A/V one-Day Research-copy
DATE_YYYYMMDD.JSON <- JSON-Log-File
8. n/a
This is only for convenience - it would delete (RD) all empty directories.
Maybe someone can use this Workflows or parts of it.
If someone knows improvements or tips for doing this more convenient - please let me know.
Enjoy.