Webinterface

[emcodem]

UPD: rebooting the server with FFAStrans dir solved the problem with security permissions on json's

Fantastic report, thanks a lot for the debugging you did.
So there is a recent bug in ffastrans open regarding leaving these files but i guess loosing all the security info on it means basically a bug with the storage or even Microsoft SMB Client (unlikely). We have the same issue at work on netapp sometimes with mxf files but there are so much servers and clients involved that its nearly impossible to debug. So we just inform the storage admins from time to time, they are able to delete the stuff from commandline directly on the storage.

However, this case should be much simpler because the reboot solved it. I guess one of our processes is just still holding the file handle and won't let go of it. But we don't know currently if it is ffastrans itself or webint.
Maybe next time you can apply the procedure to find out which process is holding a file handle to the affected file described here: https://serverfault.com/questions/1966/ ... in-windows

Also, if i remember correctly on Netapp and Isilon we had to opt-in a related feature called "oplocks" (opportunistic file locking), maybe you can check back with the storage documentation if there is some related setting and report back about it?

One thing i want to add for future emcodem here is that webinterface opens the files with "share mode delete" enabled. The documentation of these file open flags is not really good but i believe it means that webint can open the file for reading "while" another one (ffastrans) is allowed to delete the same file - in this case when webint closes the filehandle, the OS should send a delete command to the storage (which it kind of did, why did the permissions get lost otherwise).
Don't get me wrong, i am not sure if the problem is actually anyhow related to webinterface's bare existence, it is absolutely possible that the same error would occur when ffastrans runs alone. We just don't know currently.

file_open_shared.jpg (38.43 KiB) Viewed 66 times

[emcodem]

Hi emcodem
We use for a few days new webinterface_1.4.0.85 and today we see a problem - FFAStrans WebInterface process use almost 30Gb RAM
What logs i can send you for a review?

Dear @artjuice,
it is so nice from you to support us on getting this webint release done, thanks also for this report. The issues are of course related to the latest changes which optimize caching. I already found and mitigated most if not all of them. I'll upload another prerelease as soon as i am happy with the changes and notify you here.
But man, really, you got 30 gig ram usage from job json file caching in a few days? How many jobs you run through every day?

[DCCentR]

However, this case should be much simpler because the reboot solved it. I guess one of our processes is just still holding the file handle and won't let go of it. But we don't know currently if it is ffastrans itself or webint.
Maybe next time you can apply the procedure to find out which process is holding a file handle to the affected file described here: https://serverfault.com/questions/1966/ ... in-windows

The next one didn't take long This time it was the jobs of the same workflow as before, but they was stuck in web and in status monitor too:

Снимок экрана 2024-11-14 110110.png (60.43 KiB) Viewed 50 times

I found these files are locked only by host machine (SRV-CINEGY1) serving the FFAStrans dir and also doing local processing:

FFAStrans dir server3.png (37.77 KiB) Viewed 50 times

Opened by Administrator for reading:

FFAStrans dir server.png (448.44 KiB) Viewed 50 times

Also I managed to start Process Monitor on SRV-CINEGY1 something around 13 minutes before jobs get stuck. it seems that when cmd.exe appeared, the problem started:

FFAStrans dir server2.png (1.72 MiB) Viewed 50 times

Here is the events list from Process Monitor filtered by one of the stuck json (20241114-0613-4367-35a0-e076c917518b~1-0-0.json): https://dropmefiles.com/LHJuA
And full events list without filter, containts all three jsons (20241114-0615-1434-9971-43cca26ef65f~1-0-0.json, 20241114-0613-4367-35a0-e076c917518b~1-0-0.json, 20241114-0616-4487-7178-ca416d100302~1-0-0.json): https://dropmefiles.com/wuW7z

Also, if i remember correctly on Netapp and Isilon we had to opt-in a related feature called "oplocks" (opportunistic file locking), maybe you can check back with the storage documentation if there is some related setting and report back about it?
For the farm we use self-built NAS'es, which are Windows Server servers with hardware raid controllers. In this case, the host machine for FFAStrans dir is a regular Windows 10 Pro.
It looks like Windows has settings regarding Oplocks in the registry

For the farm we use self-built NAS'es and servers, which are Windows Server 2019+ servers with hardware raid controllers. In this case, the host machine for FFAStrans dir is a regular Windows 10 Pro.
It looks like Windows has settings for disabling Oplocks in the registry for SMB1 https://support.storeporter.com/hc/en-u ... 20networks. But it seems that this is no longer relevant as SMB1 is disabled by default in current versions of WIndows.

Don't get me wrong, i am not sure if the problem is actually anyhow related to webinterface's bare existence, it is absolutely possible that the same error would occur when ffastrans runs alone. We just don't know currently.

This time I didn't restart the server to solve the problem. I started stopping the “FFAStrans REST-Service” and “FFAStrans Webinterface” services one by one on all servers involved in the farm.
After stopping all services, the files were not unlocked (I could read them, but could not move them) and were still occupied by System.
I started to restart all services and after starting them on the last server, I noticed that the json files were missing from the monitor directory.
I don't know after which server and service this happened (next time I'll try to wait more after restarting the service), but the last server was the FFAStrans web interface server and it failed to start the “FFAStrans REST-Service” service immediately - it succeeded only from the fourth time.
When starting the service there was an error (I don't remember the number), after which there were warnings in the event log:
Child process [11576 -\\\192.168.100.231\FF_Install\processors\rest_service.exe /ErrorStdOut] finished with 1
and
The “FFAStrans REST-API Service” service was unexpectedly terminated. This occurred (once): 3.

[emcodem]

@DCCentR
sorry for the delay and again thanks for the spot on report and for helping us finally track this nasty thing down. I spent some hours analyzing what you sent and trying to reproduce but no success so far.
The cmd.exe you noticed is not the problem but it wants to be a solution. It is started by ffastrans and it runs in an indefinite loop, trying to delete the file every 5 seconds in case the normal file deletion did not work. Steinar told me that this workaround is in ffastrans since a long time. But obviously the workaround does not work

What catches my interest in your PML and screenshot from locked files is that the path is "V:". To be exact, procmon shows that ffastrans (exe_manager.exe) does access the file sometimes using path "V:" and sometimes using the full path.
To be more exact, within the same millisecond, we see for example exe_manager.exe ReadFile access to these 2 paths:

Code: Select all

\\192.168.24.231\FF_Install\Processors\db\cache\jobs\20241114-0613-4367-35a0-e076c917518b\log\20241114T082655692_9892_9f00e2_sys_SRV-CINEGY1.json
V:\FF_Install\Processors\db\cache\jobs\20241114-0613-4367-35a0-e076c917518b\log\20241114T082655692_9892_9f00e2_sys_SRV-CINEGY1.json

I first thought V: is a mapped network drive but according to my tests, access via mapped network drive letters is still logged in procmon with the full UNC path, at least in my installation.

Do you know what V: is and why every process seems to access both, the full UNC path and V:?

How can we go on with this: we must simplify it and make it easy to reproduce. First i believe that it's a ffastrans problem and webint file access to the json files does not influence in any way, nor does windows defender access disturb. Second, i believe all reports about this problem include workflows "with branches" and also i believe all of them always "hang" after the same processor in the workflow, in your case the RBS worklow after the "custom ffmpeg" processor.
Does every RBS job appear to "hang" after the custom ffmpeg or only some jobs?
Can you please share the workflow?