Farm and global cache problems

Questions and answers on how to get the most out of FFAStrans
andrezagato
Posts: 43
Joined: Tue Jun 09, 2020 4:07 pm

Farm and global cache problems

Post by andrezagato »

Hi,

I have 4 pcs running FFastrans on farm, and it went well for a few days, but today I read about the "Move source instead of copy" option, so I changed my cache to global and directed to a ISIS storage, before I had also in a NAS storage but it was set as local. And now I have 2 different issues.

The serious one:
It showed an error during XDCAM transcode, then all of the jobs started the next node - delivery at the same time, no job was done when that happened. And after a while it shows this error. The weird is that somejobs complete! It happened to all farm stations. I stopped the job, switched back to local and apparently its not happening anymore.

The less serious one.
I've enabled the "move source instead of copy", but the filenames started having a weird "~3210" at the end of the filename. I looked into it and saw that this "~1234" thing also appears on the file inside the workfolder in CACHE. I disabled the "move source instead of copy" and it stopped. Any idea why ?
Attachments
Deliver folder node
Deliver folder node
delivery folder.PNG (32.41 KiB) Viewed 4896 times
Rename files
Rename files
rename_move_files.jpeg (315.29 KiB) Viewed 4896 times
The first problem
The first problem
encoding error.PNG (396.64 KiB) Viewed 4896 times
admin
Site Admin
Posts: 1658
Joined: Sat Feb 08, 2014 10:39 pm

Re: Farm and global cache problems

Post by admin »

Hi andrezagato,
Moving instead of copying should not do anything strange. Both functions is feed with the same output file name.
Can you please export job examples that show all misbehavior? Right click on the history view on a job and select "Open log folder...". Zip the content and send. Please also export your workflow: Manager -> File -> Export workflow(s).

Thanks!

-steinar
andrezagato
Posts: 43
Joined: Tue Jun 09, 2020 4:07 pm

Re: Farm and global cache problems

Post by andrezagato »

Hi Steinar,

So, apparently there was some weird stuff happening. I restarted the workflows and FFastrans, updated just to make sure everything was the same between the farms, and it is working. So never mind I guess... but the files are here anyway.
About the first major problem, I don't know why, but It only happened when I uses ISIS as the work root folder. I switched to another NAS and it didn't happen anymore.
Too bad, because I was looking foward to using that MOVE SOURCE INSTEAD OF COPY, but it would only mattter if the work root folder was working in the ISIS share.

Another quick question, I feel like the CREATING AVID COMPILANT takes a long time! like almost as long as the XDCAM encode. Any suggestions to why and if I could improve that ?
Attachments
MOVE SOURCE TESTS.json
(8.58 KiB) Downloaded 163 times
LOG_MOVEFOLDER.zip
(14.38 KiB) Downloaded 161 times
Ghtais
Posts: 157
Joined: Thu Jan 19, 2017 11:06 am

Re: Farm and global cache problems

Post by Ghtais »

andrezagato wrote: Thu Jul 15, 2021 11:28 pm
Another quick question, I feel like the CREATING AVID COMPILANT takes a long time! like almost as long as the XDCAM encode. Any suggestions to why and if I could improve that ?
Hi andrezagato

It is certainly due to the localization of your cache disk. Creating Avid compliant is just a rewrapp of the transcoded file. If your cache is on your ISIS and your FFAstrans station is only 1 Gbits connection it could take a very long time.
to my mind, the best way to deal with farming is to put your cache on a shared SSD NAS with a least a 10 Gbits connection.

Personally I have only one station with a dedicated NVMe SSD Classe 5 for the cache and it rewrapp very very fast : less than 20 sec for 1 hour of Xdcam File.

Bye
admin
Site Admin
Posts: 1658
Joined: Sat Feb 08, 2014 10:39 pm

Re: Farm and global cache problems

Post by admin »

Hi andrezagato,

The logs you sent, what job was that? The one that created the strange output names which added "~<number>_", or the other with the error during XDCAM-HD transcoding? I cannot see anything reminding of either. But if it's the first then the log clarely states that the output file path handed to the file move function does not contain these added numbers. Here's an example:

Code: Select all

{
		"created": "2021-07-15T20:10:07.704-03.00",
		"node": {
			"id": "20210713-1059-4820-07ad-79d918a97993",
			"type": "dest_folder"
		},
		"split_id": "1-0-0",
		"event": "dest_folder",
		"type": "trace",
		"context": "node execution",
		"host": "PCLOCACAO02",
		"data": {
			"event": "overwrite",
			"move": true,
			"overwrite": false,
			"unique": false,
			"in_file": "W:\\.ffastrans_work_root\\20210715-2004-1939-7e54-3957bf5f6f2d\\20210715-2009-3915-81f5-378e0343203d\\1-0-0~210715200957745~2072~20210713-1055-3568-5ad3-9baea7e2d944~enc_av_xdcamhd~avid~\\D19_0620_VT3B_0405_Capture0000_a11.mxf",
			"out_file": "R:\\FFASTRANS OUT\\D19_0620_VT3B_0540\\D19_0620_VT3B_0540_Capture0000_a11.mxf"
		}
	}
The slow creation of Avid compilant files is correct in the current versions. However, it has been addressed in the future 1.3 release which will come out later this year: It will not take many longer than creating a normal XDCAM-HD file. But you will have to wait. First there will be 1.2.2 then probably 1.3.0.

-steinar
momocampo
Posts: 592
Joined: Thu Jun 08, 2017 12:36 pm
Location: France-Paris

Re: Farm and global cache problems

Post by momocampo »

Hi andrezagato,
In my side, I saw on your workflow that you don't have any other branch about your conditional node. You check the v codec is pro res but if it is not? You should add another node to begin another branch and specify what FFAStrans has to do in case of other video codec. (right click on the green square on the left side of a node and choose "execute on error". You can specify another action, send a mail, etc).
About cache, I have already tried to set a cache storage on avid storage and it isn't reliable. You should have a windows server, it will be much better.
Always about cache, you have 2 ways possible : local or shared. You have 4 hosts and if you choose local cache, each host will have its own local cache. It can be faster than shared if you don't have a 10Gb/s network. With local, each host that begins a workflow will go at the end! With shared cache, hosts can share work inside the same workflow. So depends on your workflows and how you want to work.

About your workflow, I'm wondering why you want to rename your output files if you create avid op atom files. It could be useful especially for op1a files but maybe I missed something?

waiting news about you ;)

B.
andrezagato
Posts: 43
Joined: Tue Jun 09, 2020 4:07 pm

Re: Farm and global cache problems

Post by andrezagato »

Hi guys and thanks for all your input.
@steinar , when I sent you the workflow, it was running smootlhy again. I don't know what caused, but apparently just resetting everything was enough to make it work again. Thanks for looking into it anyway.
@steinar and @ghtais
What you said was correct, the problem was with the ISIS share, as soon as I switched to a NAS it performed a lot faster, specially the CREATING AVID COMPLIANT part. I would like to test using local drives to set up for cache folder, but this PCs have 250gbs of NVME storage, when this project is over. I will install another NVME drive with bigger capacity and run some tests to compare. But for now, I'm using a NAS with 10gbps connection, even though it is not SSDs, it does perform quite well.

@momocampo
I have set to dispel job on false outcome. I chose to do that because I have 5 different workflows, each one looking at a specific codec. The medias are all mixed up, this way all the workflows are looking at the same root folder, and this conditionals filter the correct media.
But thinking about it now, I could merge a few of my workflows. Thanks for pointing out! This way I can set up a "failed" condition to see if some file didn't get processed.
About the renaming, I've realized it wasn't necessary after that, I've changed that part in the workflow!

About that, what is more "demanding" for the system, several workflows with less nodes, or less workflows with more nodes ?

Thanks again guys! You are awesome!
User avatar
FranceBB
Posts: 230
Joined: Sat Jun 25, 2016 3:43 pm
Contact:

Re: Farm and global cache problems

Post by FranceBB »

momocampo wrote: Fri Jul 16, 2021 10:02 am About cache, I have already tried to set a cache storage on avid storage and it isn't reliable.
Yep, neither cache nor DB on ISIS are reliable.
On NEXIS you can put the cache there, but you still cannot put the DB there.
momocampo wrote: Fri Jul 16, 2021 10:02 am You should have a windows server, it will be much better.
Yep. Definitely better for FFAStrans. ;)
emcodem
Posts: 1631
Joined: Wed Sep 19, 2018 8:11 am

Re: Farm and global cache problems

Post by emcodem »

DB! :D (internal joke)
emcodem, wrapping since 2009 you got the rhyme?
Ghtais
Posts: 157
Joined: Thu Jan 19, 2017 11:06 am

Re: Farm and global cache problems

Post by Ghtais »

momocampo wrote: Fri Jul 16, 2021 10:02 am You should have a windows server, it will be much better.
Hi momocampo

Do you mean it will run faster with windows server than normal windows ?
With the same hardware ?
Post Reply