Farm and global cache problems

[andrezagato]

Hi,

I have 4 pcs running FFastrans on farm, and it went well for a few days, but today I read about the "Move source instead of copy" option, so I changed my cache to global and directed to a ISIS storage, before I had also in a NAS storage but it was set as local. And now I have 2 different issues.

The serious one:
It showed an error during XDCAM transcode, then all of the jobs started the next node - delivery at the same time, no job was done when that happened. And after a while it shows this error. The weird is that somejobs complete! It happened to all farm stations. I stopped the job, switched back to local and apparently its not happening anymore.

The less serious one.
I've enabled the "move source instead of copy", but the filenames started having a weird "~3210" at the end of the filename. I looked into it and saw that this "~1234" thing also appears on the file inside the workfolder in CACHE. I disabled the "move source instead of copy" and it stopped. Any idea why ?

[admin]

Hi andrezagato,
Moving instead of copying should not do anything strange. Both functions is feed with the same output file name.
Can you please export job examples that show all misbehavior? Right click on the history view on a job and select "Open log folder...". Zip the content and send. Please also export your workflow: Manager -> File -> Export workflow(s).

Thanks!

-steinar

[andrezagato]

Hi Steinar,

So, apparently there was some weird stuff happening. I restarted the workflows and FFastrans, updated just to make sure everything was the same between the farms, and it is working. So never mind I guess... but the files are here anyway.
About the first major problem, I don't know why, but It only happened when I uses ISIS as the work root folder. I switched to another NAS and it didn't happen anymore.
Too bad, because I was looking foward to using that MOVE SOURCE INSTEAD OF COPY, but it would only mattter if the work root folder was working in the ISIS share.

Another quick question, I feel like the CREATING AVID COMPILANT takes a long time! like almost as long as the XDCAM encode. Any suggestions to why and if I could improve that ?

[Ghtais]

Another quick question, I feel like the CREATING AVID COMPILANT takes a long time! like almost as long as the XDCAM encode. Any suggestions to why and if I could improve that ?

Hi andrezagato

It is certainly due to the localization of your cache disk. Creating Avid compliant is just a rewrapp of the transcoded file. If your cache is on your ISIS and your FFAstrans station is only 1 Gbits connection it could take a very long time.
to my mind, the best way to deal with farming is to put your cache on a shared SSD NAS with a least a 10 Gbits connection.

Personally I have only one station with a dedicated NVMe SSD Classe 5 for the cache and it rewrapp very very fast : less than 20 sec for 1 hour of Xdcam File.

Bye

[admin]

Hi andrezagato,

The logs you sent, what job was that? The one that created the strange output names which added "~<number>_", or the other with the error during XDCAM-HD transcoding? I cannot see anything reminding of either. But if it's the first then the log clarely states that the output file path handed to the file move function does not contain these added numbers. Here's an example:

Code: Select all

{
		"created": "2021-07-15T20:10:07.704-03.00",
		"node": {
			"id": "20210713-1059-4820-07ad-79d918a97993",
			"type": "dest_folder"
		},
		"split_id": "1-0-0",
		"event": "dest_folder",
		"type": "trace",
		"context": "node execution",
		"host": "PCLOCACAO02",
		"data": {
			"event": "overwrite",
			"move": true,
			"overwrite": false,
			"unique": false,
			"in_file": "W:\\.ffastrans_work_root\\20210715-2004-1939-7e54-3957bf5f6f2d\\20210715-2009-3915-81f5-378e0343203d\\1-0-0~210715200957745~2072~20210713-1055-3568-5ad3-9baea7e2d944~enc_av_xdcamhd~avid~\\D19_0620_VT3B_0405_Capture0000_a11.mxf",
			"out_file": "R:\\FFASTRANS OUT\\D19_0620_VT3B_0540\\D19_0620_VT3B_0540_Capture0000_a11.mxf"
		}
	}

The slow creation of Avid compilant files is correct in the current versions. However, it has been addressed in the future 1.3 release which will come out later this year: It will not take many longer than creating a normal XDCAM-HD file. But you will have to wait. First there will be 1.2.2 then probably 1.3.0.

-steinar

[momocampo]

Hi andrezagato,
In my side, I saw on your workflow that you don't have any other branch about your conditional node. You check the v codec is pro res but if it is not? You should add another node to begin another branch and specify what FFAStrans has to do in case of other video codec. (right click on the green square on the left side of a node and choose "execute on error". You can specify another action, send a mail, etc).
About cache, I have already tried to set a cache storage on avid storage and it isn't reliable. You should have a windows server, it will be much better.
Always about cache, you have 2 ways possible : local or shared. You have 4 hosts and if you choose local cache, each host will have its own local cache. It can be faster than shared if you don't have a 10Gb/s network. With local, each host that begins a workflow will go at the end! With shared cache, hosts can share work inside the same workflow. So depends on your workflows and how you want to work.

About your workflow, I'm wondering why you want to rename your output files if you create avid op atom files. It could be useful especially for op1a files but maybe I missed something?

waiting news about you

B.

[andrezagato]

Hi guys and thanks for all your input.
@steinar , when I sent you the workflow, it was running smootlhy again. I don't know what caused, but apparently just resetting everything was enough to make it work again. Thanks for looking into it anyway.
@steinar and @ghtais
What you said was correct, the problem was with the ISIS share, as soon as I switched to a NAS it performed a lot faster, specially the CREATING AVID COMPLIANT part. I would like to test using local drives to set up for cache folder, but this PCs have 250gbs of NVME storage, when this project is over. I will install another NVME drive with bigger capacity and run some tests to compare. But for now, I'm using a NAS with 10gbps connection, even though it is not SSDs, it does perform quite well.

@momocampo
I have set to dispel job on false outcome. I chose to do that because I have 5 different workflows, each one looking at a specific codec. The medias are all mixed up, this way all the workflows are looking at the same root folder, and this conditionals filter the correct media.
But thinking about it now, I could merge a few of my workflows. Thanks for pointing out! This way I can set up a "failed" condition to see if some file didn't get processed.
About the renaming, I've realized it wasn't necessary after that, I've changed that part in the workflow!

About that, what is more "demanding" for the system, several workflows with less nodes, or less workflows with more nodes ?

Thanks again guys! You are awesome!

[FranceBB]

About cache, I have already tried to set a cache storage on avid storage and it isn't reliable.

Yep, neither cache nor DB on ISIS are reliable.
On NEXIS you can put the cache there, but you still cannot put the DB there.

You should have a windows server, it will be much better.

Yep. Definitely better for FFAStrans.

[emcodem]

DB! (internal joke)

[Ghtais]

You should have a windows server, it will be much better.

Hi momocampo

Do you mean it will run faster with windows server than normal windows ?
With the same hardware ?