FFAStrans forum

Hi there,

I'm a new user of FFAStrans and I'm really impressed with it so far but I'm falling at one of the last hurdles and it's frustrating me to no end. We've got a bunch of basic watch folders and encode presets but the way we drop files in is, from what I can tell after searching on the forum, a little unique.

Here's the basic idea of our workflow:
Input Folder (watched, users drop files into this) > Encoder (H264) > (1) Output folder // (2) Command executor (moves source file from input folder to "Processed" folder up a couple of levels)

We're using something called LucidLink (https://www.lucidlink.com/) as our primary shared media storage, which is also where the monitored folders are, but the way LL works in terms of file copying is something that I think FFAStrans isn't prepared for.

In a nutshell, when you copy a file within LucidLink (which mounts like a NAS) it'll show the standard operating system copying window but that's just for it to copy into Lucid's local cache and database, both stored on the machine's disk. From there, it's split into lots of tiny files in the background and uploaded bit by bit. The size of the file is updated as this happens, but the rate of this depends on the connection speed of the uploader.

Now, for smaller files, with the standard FFAStrans sleep time, this is absolutely fine and the files are processed without an issue. For larger files, however, they are picked up before the upload has completed, which results in a partial transcode but reporting as successful.

I've tried a combination of the below with no avail:
- Check growing files [once]
- Deny attributes [Read only]
- (Workflow properties) Sleep timer [30] & [60]
- Using the batch script found on this thread: https://ffastrans.com/frm/forum/viewtop ... 114&p=6591

FFAStrans is running on Windows 10 with LucidLink version 1.24 (2943) - which is admittedly a little out of date - with a near-gigabit download speed. The file being copied in (and picked up too soon) is a ProRes 422 HQ MOV about 1.2GB in size. It's being copied from another Windows 10 machine (which has the file cached locally) with an upload speed of ~40 mbps.

How do I stop the file being picked up before it's finished uploading?

Thanks,
Bevan

Well I gotta be fair, the way your system works is a bit peculiar, so a standard SMB share without any tricks would be better.
What happens is that the timeout goes to 0, the monitor doesn't see the file size growing 'cause the transfer is too slow and therefore it picks up the file.
Just to be absolutely sure, does setting "check for growing file size" to "continuosly" make any difference?
If not, then we have to work around the problem in another way...

Hi Bevan93, thank you for using FFAStrans and welcome to the forum!

Have you tried setting the sleep timer to a higher value, like 120? As long as you're not looking at the status monitor for something to happen even setting to 240 should be fine. The higher the better, IF the problem is that it momentarily stops growing due to whatever reason. If not, it might be as FranceBB points out that the way FFAStrans checks for growing files is not compatible with you particular setup.
What you could do is to configure FFAStrans to rename the source file using the command executor. If it fails (which it should do if the file is being used) then use the command executor again to delete the %s_cache_record% which will point to the record file being created for each new file picked up by the monitor.

-steinar

FranceBB wrote: ↑Wed Aug 18, 2021 10:04 pm Well I gotta be fair, the way your system works is a bit peculiar, so a standard SMB share without any tricks would be better.
(...)
Just to be absolutely sure, does setting "check for growing file size" to "continuosly" make any difference?
If not, then we have to work around the problem in another way...

I appreciate our setup is a little outside the norm but it's the best solution we've found to allow people to work remotely or from the office and allow us to bring freelancers onto the system quickly. We used an AFP share connected to a NAS before we were all forced to work from home and FFAStrans would have worked brilliantly with that but it's out of the question now.

I did try changing it to continuously at one point and I think it kept picking the file up, creating the output multiple times, but I'll give it another go and see what happens. That might work if I make it overwrite older files rather than making them unique every time. Thanks for the suggestion.

admin wrote: ↑Thu Aug 19, 2021 6:21 am Hi Bevan93, thank you for using FFAStrans and welcome to the forum!

Have you tried setting the sleep timer to a higher value, like 120? As long as you're not looking at the status monitor for something to happen even setting to 240 should be fine. The higher the better, IF the problem is that it momentarily stops growing due to whatever reason. If not, it might be as FranceBB points out that the way FFAStrans checks for growing files is not compatible with you particular setup.
What you could do is to configure FFAStrans to rename the source file using the command executor. If it fails (which it should do if the file is being used) then use the command executor again to delete the %s_cache_record% which will point to the record file being created for each new file picked up by the monitor.

Thanks for the welcome and response, steinar!

After trying FranceBB's suggestion I'll try changing the sleep timer to something higher. This is supposed to be used by producers to create files quickly so they can send them on without hassling the tech team or their editors. Having a two-minute delay on it may annoy them but if it's the only way then it's the only way.

The thing that kinda confuses me is that the workflow doesn't fail, it just doesn't transcode the entire file. Depending on the source file, it'll either transcode the full duration of the file but not all of the data - only what has already been uploaded - or only the duration that has been uploaded.

Yep, increasing the timeout is the way to go.
About the fact why it doesn't fail, it's easy: some containers can be opened straight away even if they're truncated, mxf is one of those, same goes for .ts for instance.
If you have a file uploaded up to a certain point, it's perfectly valid 'till that very point, so if the file size doesn't grow anymore for a considerable amount of time, FFAStrans will pick it up and encode what has been uploaded.
After all a video file is just an header + a bit stream with its own GOP and as long as the upload is sequential, it will be fine.
For instance, some protocols like P2P split the file into several chunks which are sent separately and not necessarily in order.
If the program was going to pick up something like that, it would have a very big problem: a slice error caused by the GOP, namely there will be times in which we're gonna have some of the partially or bi predicted frames without the Intra they refer to, so motion vectors will have nothing to point towards and you'll see glitches and other issues displayed in the video.
If the files you upload are picked up regularly even if they're incomplete and they're encoded up to that very point it means that even if your system is splitting those files in chunks, all those are progressively uploaded, 'cause otherwise you would have the aforementioned issues when you playback the video.
What is bugging me, though, is that this is happening with a .mov container.
Generally, mov as a container has always been a bit "pretentious" in the sense that from my experience with this container and AppleProRes as video essence inside, it actually wants the whole video to be available from start to end, otherwise it won't open the video in a player nor display the relative info in MediaInfo.
You can notice this when you're exporting a file like that muxed in mov and you can't open it while it's encoding, contrary to many other containers...
But anyway, that might be just one particular muxer and maybe it's not something that "affects" all MOV files...


Anyway, long story short, try increasing the sleep timer and get back to us. Technically the watchfolder should check the file less often and the user will have time to upload another bit and update the info about it (hopefully).

Hi Bevan93
Let me ask monkey question : can't you setup LucidLink SW on your PC to download files with temporary extension and change it to original one whenever the download is finished? If yes, then you would simply use Allow / Deny filters in watchfolder setup

Silicon wrote: ↑Thu Aug 19, 2021 6:49 pm can't you setup LucidLink SW on your PC to download files with temporary extension and change it to original one whenever the download is finished?

Thanks for the response. Unfortunately not, no. LucidLink is designed to stream files, sometimes even as they're uploaded (but mostly only when first opened) so as the file is being uploaded all clients connected and looking at the file see it with the correct file name and extension with a growing file size as it's uploaded.

FranceBB wrote: ↑Thu Aug 19, 2021 5:33 pm What is bugging me, though, is that this is happening with a .mov container.

Yep, that confused me too. I do remember seeing somewhere that LucidLink uploads the start and end of the file (after interpreting it) and then uploads everything else. It's an odd way to do it, for sure, but the performance is incredible so I'm not going to question it!

I changed the timer to 120s and I'm still seeing the same issue when uploading on slower speeds (manually limited to 5MB/s and then 2MB/s on LucidLink to test).

I think my solution here is going to have to be a temporary "transfers" folder that people drop their files into until they're uploaded and then move into the watch folder of whatever preset they want.