copy VS move VS delivery folder VS command executor

[dries]

Hi all,

I am a bit confuse in best practices in files moving around...
when a file comes into a monitor folder, how to handle it...

In my setup I use a monitor folder, then copy it local to multiple folders for processing.

What is the best practice to move them around?

1) Use a delivery folder to copy them to numerous processing locations (haven't used that)?
2) Use command executor to copy them to multiple locations and then delete the original file (my setup)?
3) Use first copy command and then move command (seems not to work)?

And, after processing, how safe is it to delete the original? Does the command executor for delete at the end always check the previous nodes?

I am used to the Vantage system, but it feels different with ffastrans.

Dries

[emcodem]

Hey dries,

Sorry if my answer is a little confusing, it covers too many topics... if you got any questions let me know

And, after processing, how safe is it to delete the original? Does the command executor for delete at the end always check the previous nodes?

If you aim 100% accuracy (e.g. Archiving) you would need to add some hash verification (e.g. $xxhash function) to check if source and target matches before deletion. If 99.99% is safe enough for you, you might skip the hashing stuff and just rely on the stuff you use being able to detect and report errors correctly. Of course if you have instable network or frequent network/storage maintenance, things might look different.
How it works in the workflow is that the next processor is executed depending on what you choose at the input pin, e.g. "on success, on error or always". It will depend on the quality of the previous processor if it is able to detect and report errors correctly. E.g. in a cmd processor it depends on the executed cmd, ffastrans does not really have any influence here.

Regarding the difference between commandline and inbuilt processors:
Using a deliver processor for doing the copy instead of a batch would enable to display the progress and retry in different failure scenarios.
Of course a deliver proc can only deliver to a single location, so you need one deliver processor for each destination. The most easy setup for this would be to do one delivery after the other (serial processor connection). This way you can easily handle errors and you don't need to know about "split and merge". You could also deliver dynamically to a list of locations using foreach processor but that's probably not what you want

Regarding split and merge branch stuff:
in ffastrans split and merge is kind of hard to understand, mostly because what you see in the GUI is not what happens in reality. Explained here:
viewtopic.php?f=5&t=849#p3565

Last but not least, if you really have the need to do the deliveries in paralell, you can potentially live with hold-sync option and after that just execute a cmd processor with delete cmd and prepend an "if exists" in the deletion batch. This would lead to the deletion batch being executed multiple times parallel but only one branch would really execute the file deletion.

Very Last but not least, if you really need to split off into many branches and need a way to wait for all branches to finish and go on with "one single branch", there are concepts like this: https://www.ffastrans.com/frm/forum/vie ... t=10#p4666

Finally:

3) Use first copy command and then move command (seems not to work)?

I don't see a reason why first copy then move would not work.