Create .md5-file after a file is delivered?

[stuvmark]

Hi, I'm delivering files to a folder which is picked up by an external company that uses MD5-checksums to check that the download is correct. At the moment I have gotten help to create a powershell-script that creates those files on all files in the folder, and runs this with windows task scheduler each night. The setback with this is that it usually takes three hours to complete this script.

Is there a way to have FFastrans to do this after a file is delivered to a destination? Preferably powershell or command-script (.bat/.cmd-file as my company doesnt like too much of "weird" software)

For reference this is the script I have running;

#| Where-Object { !(Test-Path -Path $($_.Name + ".md5")) }
if(Test-Path -Path "\\domain\server\folder"){
Get-ChildItem -Path "\\domain\server\folder" -Filter "*.mp4" | Get-FileHash -Algorithm MD5 | SELECT Path,Hash | ForEach-Object {New-Item -Path $($_.Path + ".md5") -Force -ItemType "file" -Value $($_.Hash.ToLower() + " " + [System.IO.Path]::GetFileName($_.Path))}
}
else{
Write-Host "Unable to get to \\domain\server\folder"
}

[emcodem]

Hi stuvmark,

welcome to the forum and thank you for using FFAStrans!

thats easy, you can use a command executor after your deliver node and execute this code.

Code: Select all

%comspec% /C "powershell -command "Get-FileHash -Algorithm md5 "%s_source%" | Select-Object -ExpandProperty Hash" "

You need also to set the "Read STDOUT" dropdown to some user variable (create one if you don't have one already).
After that, use some write text file node and let it write the user_variable that now contains the md5 to the corresponding text file. Let us know if you have problems finding out how to set it to write to the correct filename.

By the way, depending on what your IT refers to as "weird" software (even the ps1 script you posted could be seen as to be "weird software"), you could even get around the speed limit of md5 powershell function using this simple tool i wrote once https://github.com/emcodem/fast_md5 :..but i guess using above cmd processor batch code will ease your life far enough.

[stuvmark]

Hmm, I see that I forgot to specify the output. It should be a new file with the same name as the hashed file, with an added -md5 on the end, and the file should look like this (on one line)

Code: Select all

38d9a71e3722281cd61a684814a20d5e  Filename.mp4

I have been writing some batch scripts back in the years, but I'm not fluent enough to do this on my own...

[emcodem]

Ok, no need to write any batch or so. After you stored the md5 in a user variable as described above, e.g. you created a var named %s_md5%, you just take the write file processor and use your md5 variable to construct your filename.
In your case, you specify that your target filename should consist of these parts:

Code: Select all

"MD5" "TWO SPACES" "ORIGINALFILENAME.ORIGINALEXTENSION"

I am under the impression that this is not really your goal But anyway, here some example how you would specify exactly this in the "deliver file" processor's "Text File Name" Input:

Code: Select all

%s_md5%  %s_original_name%.%s_original_ext%

While in the "folder" input of your write text file processor, you would specify:

Here some test workflow for you that does exactly that, just rightclick-submit a file to the first processor

Code: Select all

%s_original_path%

[stuvmark]

Hah, almost done. With the examples you provided in the json-file it was almost there. Just didn't understand why it didn't write the file until I opened my eyes, it wrote it on my pick-up folder;). The file flow is:
1. (folder monitor) Pick up from folder when file is not growing any more.
2. (delivery) deliver picked up file from #1 to server.
3. (command executor with your help)
4. (generate text file) I did understood this. Just needed some changes as I wasn't specific enough when asking for help. Based upon the above workflow I want the new file to land into the destination in #2. I just specified the server address as folder, so I solved it. Is there any way I could get the folder from #2, where I copied the file to?

One thing noticed with the hash, is that it ended up as upper-case, whereas my existing workflow creates the letters in the hash-string in lower-case. I'm checking with our vendor if their import allows this.

The generate of the MD5 took ages, if I create an md5 from the command line it usually takes 2-3 minutes on a 3,6 GB file, the command line executor here uses around 12 minutes or more. After a reboot it was quick - any ideas why?

[emcodem]

Hey! well that took a lotof time hehe... hope you didnt work only on this the last weeks

Is there any way I could get the folder from #2, where I copied the file to?

Not sure if i understand the question. What i do is to use one populate variables processor at the very START of the workflow where i define all stuff like output folders and such... the rest of the workflow then uses the variables from this first populate processor. This way i have only ONE spot where to set and change the variables so i can re-use the workflow.

One thing noticed with the hash, is that it ended up as upper-case, whereas my existing workflow creates the letters in the hash-string in lower-case. I'm checking with our vendor if their import allows this.

Are you working with the powershells script or with the program that i posted above (fast md5)? ...in powershell you could just use ToLower() in order to get lowercase string

The generate of the MD5 took ages, if I create an md5 from the command line it usually takes 2-3 minutes on a 3,6 GB file, the command line executor here uses around 12 minutes or more. After a reboot it was quick - any ideas why?

Same question, did u use powershell or the "fast md5" i posted above? Anyway, this does not really make much sense, there should be no difference between executing the command locally vs. as workflow. But you might want to be aware of storage caches: your test results that you used locally might have been influenced by the storage having the stuff already in the cache. Please be aware the when you benchmark, you should change the filename with each and every try, otherwise you will sometimes get results served from disk cache while other times your result gets served without cache.

[stuvmark]

Have had some time off, and IP I'm posting from has been blocked from posting for almost a week now.

Instead of trying to explain, I'm including my workflow. I have everything working as expected now, except from that the MD5 checksum inside the file is in uppercase.

Getting the delivery path from my second processor is just nice to have...

The speed thing is super-weird - I have it happening with my nigthly script aswell, some days it uses 2-3 minutes per file, some days it uses 10-15 minutes per file. When I check in Task Manager almost no resources is used by powershell, not even network activity.

https://ffastrans.com/frm/forum/downloa ... ew&id=1015

[stuvmark]

I have tried to implement the .ToLower-command in the string

Code: Select all

%comspec% /C "powershell -command "Get-FileHash -Algorithm md5 "%s_source%" | Select-Object -ExpandProperty Hash" "

I'm not able to implement in a way that gives the output as lowercase. Either it fails, it doesn't do anything or I'm able to insert it in a way that takes a part of the command instead of the md5-hash into the output file. This demonstrates my level of knowledge.

[emcodem]

sorry busy Day... I will take a Look for you tomorrow if nobody is faster

[emcodem]

OK, so for the lowercase stuff, please try EXACTLY this in your cmd executor. No %comspec% /C or anything else than exactly this:

Code: Select all

powershell -command "$C = Get-FileHash -Algorithm md5 "%s_source%" | Select-Object -ExpandProperty Hash ; $C.toLower() "

Here a little more advanced version of your workflow that includes the lowercase stuff

NBR_mp4_emcodem.json

(6.12 KiB) Downloaded 203 times