Farm and global cache problems

Questions and answers on how to get the most out of FFAStrans
momocampo
Posts: 595
Joined: Thu Jun 08, 2017 12:36 pm
Location: France-Paris

Re: Farm and global cache problems

Post by momocampo »

Hello Ghtais,
I don't think a Windows server will be faster than a "simple" windows. It can be interesting to have a server for the network features but about speed, it will be the same thing.
It can be better to have a Win Server to provide a good share for FFAStrans Installation.
;)
B.
User avatar
FranceBB
Posts: 266
Joined: Sat Jun 25, 2016 3:43 pm
Contact:

Re: Farm and global cache problems

Post by FranceBB »

Yep, @Ghtais it's not about where FFAStrans *runs* but rather FROM WHERE it runs.
Some kind of RAID6 storages like AVID NEXIS or Isilon are suited for several big files 'cause they're really meant to store masterfiles like the ones I'm getting constantly like 4K Apple ProRes of 1 TB each or 534 GB XAVC Intra Class 300 and so on.
Now, if you put the FFAStrans installation there (so the database), on NEXIS it will not run, period.
On Isilon it will run but it will have some issues. We're working round the clock to fix those (well, actually, I'm doing nothing right now, Grandmaster Steinar is ehehehehehe), but the main reason it has those issues is that while those kind of solutions are meant to store several big files, FFAStrans works on a ticketing based system which means that every "action" is a ticket.
Monitoring a watchfolder is a ticket.
Several conditionals in a workflow make a ticket each and so on.
All those tickets are then executed (it's the job of the exe_manger to execute them) and process the jobs.
So, for instance, if you have lots of watchfolders, you're gonna have the system creating tickets and the exe_manager executing them being like: "Hey, is there a new file? Hey, is there a new file? Hey, is there a new file? Hey, is there a new file?" over and over and over again.
As you can assume, tickets are relatively tiny files (few KB generally) compared to 1 TB worth of files that those solution generally are meant to store, therefore the storage is gonna be slower in handling those than it is in handling other kind of files.
This means that there might be concurrence issues and other things (but all those have been addressed in our internal development snapshot builds).
The "problem" is that even if those have been addressed, the storage is gonna be slower in creating, reading and deleting thousands of those tiny little files which are crucial for FFAStrans, which is why we recommend a Windows Server 2019 SMB Share for the database. That's gonna be better and faster in writing and reading those tiny little files. :)
emcodem wrote: Mon Jul 19, 2021 4:52 pm DB! :D (internal joke)
Eheheheh I knew you were going to say that xD
Ghtais
Posts: 164
Joined: Thu Jan 19, 2017 11:06 am

Re: Farm and global cache problems

Post by Ghtais »

Hi all

thank you for taking time to explain how FFAstrans really works, I really appreciated.
For the moment I'am happy with my single workstation and I have enough power to ingest and outgest all stuff that we need. We have a majority of HD material so it could explain that.
We are running 20 workflow and on the last 6 month we process 26367 files with only 217 failed (the majority for debug reason during the creation of new workflow).
it is terribly reliable !!

thank you all !
Cheers
User avatar
FranceBB
Posts: 266
Joined: Sat Jun 25, 2016 3:43 pm
Contact:

Re: Farm and global cache problems

Post by FranceBB »

Ghtais wrote: Tue Jul 20, 2021 7:34 pm on the last 6 month we process 26367 files
A few numbers from here at Sky (only one farm, there are 2 farms) ehehehe

Image
Ghtais
Posts: 164
Joined: Thu Jan 19, 2017 11:06 am

Re: Farm and global cache problems

Post by Ghtais »

ok ok you win :-D
Post Reply