Distributed Processing Workflow and Monitor

[Noqras]

I just started using the distributed processing feature and its been working very well. There are two things that I have noted however.

Currently I have two servers running ffastrans as a farm, very often now whenever I edit the workflow on one server it will actually disappear from the other server's GUI and then disappear altogether after saving the edited and shared workflow, I then have to import a backup. This is with both server's GUI's open, I don't know if that is the issue.

The second issue is minor, I have noticed for me at least that when I have ffmpeg processors running on both servers, when looking at it in the job monitor one of the jobs often disappears and then reappears a couple seconds later, it doesn't stop or anything, just disappears from the job list and comes back a bit later, doesn't affect the actual processing.

[admin]

Hi Noqras,

Well that's not good! Is the workflow issue behavior more normal than not?

The other issue with momentary disappearing jobs is known. It's related to how jobs are updated. I will try and come up with a fix for it but it's lower priority compared to other more critical bugs. But thanks for notifying

-steinar

[Noqras]

It seems to be pretty normal, the scenario is this in testing:

Both servers open via RDP, both GUIs open, doesn't matter is that particular workflow is open on both or not, I make a change on server 1 to a workflow, save it, it disappears from the other server's GUI, I switch workflows on server 1 and then the saved workflow disappears there as well.
I import a backup, try to edit again, and usually it works that time.

[admin]

Hi Noqras,

I'm struggeling with recreating the issue. You're writing that it usually works after importing a backup. Is all ok then or does it suddenly start removing the workflow again after edit?

-steinar

[Noqras]

Most of the time I can edit and save after that at least once before it disappears again. After that though it is likely to happen again it seems.

If you're not able to reproduce this one though I wouldn't worry about, I might have something else going on with my setup.

[Noqras]

Odd, for me at least in this farm environment, I find as well that if I create new workflow from an existing one, and modify it, save and enable it, submit a file to it, abort the job because its trying to encode past the run time of the video, and go back to the workflow manager, I will find that the workflow isn't there anymore.

It seems to be something weird with the synchronization of workflows between the 2 servers in the farm causing issue for me, working fine on my single local processing ffastrans server.

[emcodem]

Where do you keep the install files in farm mode, what type of storage is it?
Edit: forget this question. Do you find your missed workflow in the directory with corrupt workflows?
FFAStrans\Processors\db\configs\workflows\corrupt

[Noqras]

Hmm, I have nothing in the corrupt folder actually. But, the last time this folder was modified was right around one of the times my workflow disappeared.

And there was a message kind of randomly a few times (I remade the workflow several times trying debug why this happens) while a file was processing before the workflow disappeared that the workflow was corrupted and being moved to the corrupt folder.

As a note as well, I noticed that this happens much more for me if I copy an existing workflow and edit it, and didn't happen right away when I created a new workflow from scratch, though about an hour later it ended up disappearing as well.

As I said, this is only in my farm environment running off a stornext SAN, with both services running an FFAStrans service.

[emcodem]

Thats great information. If you like to go on with this and help us locate the cause of the issue, i would like you to try and run your farm on another storage than stornext.
Easiest would be to create a share on one of the farm machine's local drive and use this share to drive your farm. Can you reproduce the issue this way?

thanks a lot!

[Noqras]

Ok, after extensive testing on this, I can confirm that this is working perfectly as expected when running off of a normal Windows share as you suggested I try.

Before, I was actually running off of a NetApp storage, so Stornext may work fine but there seems to be some issue for me with running this off netapp. That's fine, I will either try a Stornext share or I don't know if its recommended but I will simply make one server a sort of "master" in that it will host ffastrans in a shared network folder on its drive. Do you think that would be fine for the network performance and everything that way?