How to setup farm correctly? Errors when farming

[ThomasM]

Hi emcodem,

thank you for your input.

All These paths must be exactly the same on all Systems and ffastrans must be started on all nodes from S:\NICHT LOESCHEN FFAStrans\
Also, make sure to check the "use global shared media Cache" stuff in Configuration->Host and in Configuraiton->General set it to a Shared Directory like S:\.ffastrans_work_root

In General, make sure that no file at all ever goes to a local drive like C:\, especially not temporary files like encoded files or avs files in Cache Directory.
Can you confirm that?

Yes, only the mentioned network-folders are in use.

I sure can confirm that all machines have the same mapping and credentials to access the network drives. And sure, both machines start FFAStrans from
S:\NICHT LOESCHEN FFAStrans\.

When I let a workflow run on either machine alone (farm environment set to only one machine OR just start FFAStrans on only one machine) the workflow runs through. When switching to both machines, I got these errors. I got a success-rate of apx. 50% to 70%, the rest run through in a second start of a workflow.

And you are right - the farming list looks strange because I was able to do so. Sure I changed it back to only one machine. I just thought that could be one misbehaviour... Is it true that one cannot see all the machines in a List in farming-tab which started ffastrans from one location?

I hope the attached log is good. I will clean-up the logs tomorrow and try to get a clean one wien the error occurs.

regards,

tom

[ThomasM]

sorry,

I also seem th have truoble attaching two or more logs. Here is the one with a failed job.

[emcodem]

Is it true that one cannot see all the machines in a List in farming-tab which started ffastrans from one location?

Yeah, unfortunately you have to enter them manually.

Your logs are perfect, i was able to find out something very strange:
The "error" log Fails at the cmd processor node, executing this command:

Code: Select all

=PROC=> 2019-07-03 11:23:29.631 on FIND _assumefps_ Echo@RENDERER-02, PID: 4608 -> Executing: C:\WINDOWS\system32\cmd.exe /c "FOR /f "tokens=2 delims==" %f in ('findstr "AssumeOriginalFPS" "Z:\WATCHFOLDER\FA_Scanner\SCANNER-STUMM\N8FA10051INV19248_Archivdigitalisat.LOG"') DO ECHO %f > "Z:\WATCHFOLDER\FA_Scanner\SCANNER-STUMM\%s_archnumber%\FPS.fps""

This Fails with message "Path not found". The reason is that the variable %s_archnumber% has not been replaced by it's actual value (look at the very right of above message, %s_archnumber% should not be there, instead it's value should be there...
The strange Thing with that is that the variable should actually have a value. This variable even was used before in the MD WD-DIR processor (which ran on the other Workstation).

The only Option that quickly Comes to my mind how this all can be is in case you did not start ffastrans on both machines from the exact same path or so. But let me think about that quickly I Need to investigate how user_variables in a farm Environment are passed around in the workflow

[ThomasM]

emcodem...
You_are_so_right! Sure, the variable should have been dissolved.

Ffastrans is started from exakt the same share.

So long, thaks for your Patience,
Regards
Tom

[emcodem]

OK, i could probably could use some help here, maybe it is time to summon @admin ... steinar, if you read this, please take a look at 2 posts above (the one with the quote) I am not sure how to go on here besides the next question:

@ThomasM i really enjoy your cooperation on this, thanks a lot! There is (at least) one more thing i would like to ask from you:
In the Workflow settings, "Maintenance", please check "keep jobs work folders on completion" then, please zip and send us the whole log directory from a failed job run. There will be one directory created per job run, looks like this: D:\.ffastrans_work_root\20190618131254\20190703-200522-491-C9F28762477A (but with other numbers )
I'd like to check the path for the user_var_file in the ini file for the cmd_run that fails. The name of the file starts with ~ticket_cmd_run_* but i guess it is better if you just zip and upload all stuff of a failed job
Also don't forget to include the job_log (same as you uploaded before) please!

[admin]

I can think of several reasons why the variable is not populated but they all must be due to a programming error or logical flaw, or system deviation. So as emcodemwrites, the answer is probably in the user variable file that is passed from node to node.
In the ideal world it really should not matter if you are on a farm or not but it might. What I would like you to do, besides posting the full job work folder as described by emcodem, is to set the following registry values to your farming hosts:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters
DirectoryCacheLifetime=0
FileNotFoundCacheLifetime=0
FileInfoCacheLifetime=0

All values are REG_DWORD.

Please do this AFTER you have successfully recreated the error and sent us the job work folder files.

-steinar

[ThomasM]

Hey Steinar,
Hey emcodem,

Thanks for the wonderful help in this unusual case. This is more than anyone can expect. And of course - the reason of putting out errors can probably be found in some weird windows-settings. Here is the status for today:

1. We had a Windows Update on one Renderer (renderer 01). This one now runs on Win 10 (1903). Renderer 02 still runs on Win 10 (1809). The fun fact here is that both renderer ar built from the same components at the same time and were both taken in usage at the same moment.

2. I let a Render-stresstest run on the render-farmm (renderer 01 and 02 in parallel) on thursday (the files 16FA1234TEST2345.AVI; 13 pieces; all at about 600 MB large). The outcome was successful for the first time ever.

3. Now I am working on a real-life scenario with 6 or 7 files at about 250 ... 300 GB large. The outcome i can report on tuesday, as I aM NOT IN THE STUDIUO THE NEXT DAYS.

4. Steinar, I will not change anything in the settings. And the win update started automaticly while restating the machine... No chance to hold this up at restart...

So long,

have a nice weekend,
regards,
tom

[ThomasM]

Hi Steinar,
Hi emcodem,

unfortunally the encoding process with real-life-scenario failed. I attach the Log-Files. I guess it has something to do with file-access. But I cannot find any clue what or where something is blocked. I have cross-access to all mapped drives and via remote-desktop-connection I also can access all drives and network-connections as well.

#Steinar - is it time now to try the settings you suggested?

First I will transcode the files with farming set to only one machine. I have to get them out this week.

Thank you all,
regards,

tom

[emcodem]

Hey Thomas,
the logs are unfotunately not what we were looking for, before enabling the registry changes we would need the logs collected this way:

Code: Select all

In the Workflow settings, "Maintenance", please check "keep jobs work folders on completion" then, please zip and send us the whole log directory from a failed job run. There will be one directory created per job run, looks like this: D:\.ffastrans_work_root\20190618131254\20190703-200522-491-C9F28762477A (but with other numbers ;-))

[ThomasM]

Hi emcodem,

sorry, I thought I had the right ones. Next try:

I removed the video-files, otherwise the .ZIP would have grown to some 17 GB.

Hope I got the right ones now!

Regards,
tom