Archive workflow

[ring4life70]

Hello all,

I would like to create an archive workflow to move folder ( instead of each single files ) and all subfolder and files on it, from one server to another, I need to preserve folder structure of source files, probably I can use the custom processor "Files Find" in order to process folder instead of files, but I need your help to replicate the folder structure in the destination server.

Thank you!

[ThomasM]

Hey,

You can use a CommandExecutor-Node for things like that.

If your Folder-Structure is always the same you can make use of the FileVariables built in into FFAStrans:
%s_original_folder%
%s_original_path%
%s_original_path~n%

The latter is well explained in the Documentation.

So the easyest workflow could look like this:

1. FolderNode to find an appropiate MediaFile
2. Nodes for Transcoding or anything else
3. CustomNode to COPY the tree

If the destination will be filled up over time and at any point there are changes to the FolderStructure, you can make use of UserVariables to define the desired tree-structure. You do so by populating every known folder-path or parts of the path to UserVariables. Then you use a Folder-Node, Set Folder to "Drive:\\Your.Path.From.Variables". The File is then copied to that location.

Combining both you can also copy sidecar-Files without processing to the destination.

Hope this helps...

regards,
Thomas

[ring4life70]

Hi Thomas,
unfortunately the folder structure is not always the same, this is why I think is more simple to use the webinterface and manually submitting to the move node the folder containing multiple source files.
the missing point is how to replicate the folder structure of the source files in the destination server.

[ThomasM]

Hey,

it depends. If there is more effort in programming than MediaFiles and always changing foldernames - then a manual workflow COULD be the solution.

But...

If you can figure out some statics, e.g. "always 'F:\' is the root Drive" or "always 'AlexaDaylies'-Folder" ist the folder containing the folder-structure in question, in this case I would recommend some brain-Power for using automation.

Is your archive workflow just copying files or is there some processing also?

The easiest way to keep the FolderStructure is to use the %s_original_folder% - Variable. Check out the mini-Workflow below.

cheers,
Thomas

[ring4life70]

I am in the first scenario, we have several folders with different names in different paths that that need to be archived, is not convenient in our workflow to create a watch folder to triggers the archiving process, the files are on a protected NAS where I would like to limit access, so I would prefer to use the webinterface
This is the reason why I am looking for a solution to submit a folder with several files in it instead of a single file with the webinterface.

[emcodem]

@ring4life70

It should be simple for you to start a workflow with a folder from webint and the workflow itself doesnt have to do more than "xcopy /E source destination"
Or did i oversee anything here?

[ring4life70]

Thanks emcodem,
please could you provide me the correct string to enter in the command executor ?

[emcodem]

Code: Select all

cmd /C "c:\windows\system32\xcopy.exe /E /I /Y  "%s_source%" c:\temp\archive"

There is a lot to say about this and much depends on your requirements.
First, this command does not work when we omit the path to xcopy which i don't understand. However you can just ignore this for the moment.
Second and much more important, your goal is to archive stuff. Basically if my approach works for you, you would still have to test what happens if xcopy is not able to copy one file, does it throw an error, making the ffastrans workflow to end up in an error as well? (in this case we had an easy time to catch the error)

If you need to "guarantee" that your stuff has been archived, your work just started here. In this case, you should be aware that there is a lot of work you face. E.g. you should compare the md5 of each file that has been copied. Furthermore, it takes a while until one found out how xcopy behaves in different error scenarios, so usually if i have requirements in the "guarantee archival" area, i don't go with builtin windows tools but i do everything in my own program.

You need to know that i am very experienced in the dark art of file copying that is why i am actually able to point out that you should never rely on any tool's error code or opinion regarding it's own work. If you need to guarantee that something has been archived you must make a checksum of source and target, there is no way around that.
On the other hand, the way i target here is easy and cheap. If your IT infrastructure is OK (especially network and storage is stable), the simple xcopy approach will for sure deliver > 99.999% reliability. On the other hand, 99.999% means you can loose 1 of 10000 (or is it 100.000?) which in certain usecases renders to be unreliable as hell.

[ring4life70]

Thanks emcodem I agree with you that a copy verification system needs to be implemented.
In the meantime I copied the string you posted into the command executor of my simple workflow, but it does not work, can you help me ?

[emcodem]

Sorry i was a bit careless.
so as i mentioned, there was something about xcopy which i did not understand. I still don't understand it and i was not able to find out why it works on commandline but not in cmd processor. However, we can switch to a more modern alternative: robocopy.

Robocopy has a shitload of options, i am sure if you are interested you find lots of blogs about it.

Please note that you have to submit a FOLDER to this worklfow which is only possible by manual submit (from input) or from webui (by drag/drop folders from left to right).

emcodem_robocopy.json

(5.42 KiB) Downloaded 193 times

Note that i specify "C:\temp\archive\%s_original_name%" as target dir. This is an attempt to use the base folder name you submitted as subdir on target, e.g. you submit \\server\share\folder1, s_original_name is "folder1" so the target dir is c:\temp\archive\folder1. Without this, robocopy would omit the first folder and just drop the contents of folder1 into c:\temp\archive.

Also this workflow defines a "hidden" webui variable %s_webui_hidden_input%. This is used in order to hide all other user_variables from the worfklow submit page on webui.

Let me know any questions