Watch/Folder processing speed

[kimble]

Hello again,

I have a simple set-up with the following:

watch folder with AVI > decode > resize > transcode to mp4 > publish to folder

In my watch folder I have a lot of folders with videos in, probably in the region or something like 50k videos, 3-4 folders deep in a hierarchy.

It seems like this structure doesn't work well with the pipeline as the majority of time is spent at very low CPU with just a couple of jobs getting added at a time. This is on a 16cpu server so it should be monstering through. If I manually add some files into the workflow then everything is fine.

So a couple of questions:

(1) Will it eventually speed up naturally after scanning all the folders? It doesn't seem to atm. Even after leaving it overnight.
(2) Is the job limit of 50 in the monitor a hard limit as this seems to prevent me loading up the job pipeline?
(3) Is there a better way to pipe all of this folder/file structure in to get the conversion process at peak throughput? (i.e. piping in a text file or something)

Cheers,

Neil

[admin]

Hi Neil,

What does your "Options"->"FFAStrans configuration" under "Resources" say?

-steipal

[kimble]

I've got it set to 14 of 16 CPUs and 75% CPU. Initially the CPU selector was lower (10 I believe) but I turned it up to try and see if that helped.

[admin]

Can you please try the exact same workflow on a much smaller amount of files? Like 50-100 or so just to see if the behaviour is the same?

And so for you questions:

(1) No, it won't speed up.
(2) The monitor limits does not limit the amount of simultaneous transcoding
(3) Currently there is no "normal" way to give FFAStrans a "list of files".

-steipal

[kimble]

Hi Steipal,

Thank you once again for your help. I've run some tests and the behaviour is much the same with just a folder with about 300 files in. Something definitely doesn't look right.

Code: Select all

[general]
Watcher=rfl_copy_copy
Folder=
Watcher_ID=20160413225341
[variable]
wf_size=105
Start_proc=mon_folder 20160413-141525-800-6291693FB798
[maintainance]
[special]
[user_variables]
[mon_folder 20160413-141525-800-6291693FB798]
execute_on=3800889
Folder=V:\Auto_Clip_Gen_Videos
Accept_Types=*.avi
Deny_Types=
Deny_Paths=
ATTRIB_READ=4
ATTRIB_ARCH=4
ATTRIB_SYS=4
ATTRIB_HIDDEN=4
Recurse=1
Localize=4
Create_Folder=4
DirSize=0
FileLimit=None
proc_x=306
proc_y=137
Outbound=dec_avmedia 20160413-143126-856-D8019B9E5A9A
[dest_folder 20160413-141535-330-D6FECC57D7F1]
execute_on=3800889
Folder=I:\footballondemand
Suffix=%s_original_folder%\
Prefix=
Overwrite=4
Unique=4
Case=None
Drop_name=4
Drop_ext=4
proc_x=936
proc_y=137
Outbound=
[enc_av_mp4 20160413-141542-757-682FBE23A6EA]
execute_on=3800889
V_Codec=x264
V_Bitrate=1500
A_Bitrate=96
Faststart=1
force_16_9=4
Only_Audio=4
restrict=4
A_Samplerate=44100
M4V=4
Level=3.1
Profile=baseline
Tuning=none
QSpeed=medium
ch_layout=stereo
proc_x=778.5
proc_y=137
Outbound=dest_folder 20160413-141535-330-D6FECC57D7F1
[avs_v_resize 20160413-141556-554-FF934180ED51]
execute_on=3800889
Quality=Lanczos
Preset=HD 720 (1280x720)
Interlaced=4
Width=1280
Height=720
proc_x=621
proc_y=137
Outbound=enc_av_mp4 20160413-141542-757-682FBE23A6EA
[dec_avmedia 20160413-143126-856-D8019B9E5A9A]
execute_on=3800889
Video=1
Audio=1
A_Decode=4
V_Decode=4
A_Method=Intelligent
V_Method=Intelligent
proc_x=463.5
proc_y=137
Outbound=avs_v_resize 20160413-141556-554-FF934180ED51
[MATROXDEV2]
PID=6204

[kimble]

From a few more tests it seems that something in the folder input scanning process takes a very long time but it's quite hard for me to get any idea what is going on beyond just looking at some processes in the task manager.

[admin]

Ok, a couple of more questions:

What kind of drive is V:? Is it local, removable, network or SAN?
Are there many other files in the same directory structure you're monitoring? If so, how many (estimate)?
Can you test your workflow on a local drive with some files just to check if show same behavior?

Just to mention; I have seen similar behaviour on a mac-shared network drive with LOTS of files. FFAStrans need to search all files to find match. It's not like asking windows to "give me all *.mxf" files. The problem is it's doing the same operation all the time. So I have for a while thought of redesigning the monitoring to be much faster.

Other than that I can't see any problem with your workflow.

-steipal

[kimble]

Thank you for your help again

It's a NAS box connected over the network. The folders are this structure.

folder/folder/folder/300x .avi
folder/folder/1.txt file

And there are about 100 folders at the top level so all in we are probably looking at around 30k videos and 300 txt files.

I tried doing some tests to try and narrow down the issue but have little success. I tried to repro with my home NAS and movies folder which is about 100 flat folders each with a movie in but that didn't suffer the issue.

Do you think I would get better performance denying the *.txt instead of specifying the *.avi?

I will point out a small bug but the list box that pops up when you manually adds some files doesn't have a vertical scrollbar so can overrun the screen height when you add a lot of files and you can't access the confirm dialog at the bottom. Thankfully you can press Enter on your keyboard and it works.

[admin]

How long does it take between starting the workflow until it start working on the first file? How long does it take to convert one *.avi file (estimate)?
Also, can you enable logging? Workflow -> Properties -> Special: Enable "Log to file". Once you have run through few files with your workflow, please zip the log-folder and send it to me at post@ffastrans.com. I might find some clues there.

I don't think denying *.txt rather than accepting *.avi should make any difference, but you can just test it.

Thank you for reporting the bug. I guess I did not expect the scenario of someone submitting "too many" files for the popup message to handle
I will have a look at it.

-steipal

[kimble]

I'm going to run a few tests today to try and help get you some better information and find the point that it starts to get overloaded. The logs just contained the success/failure that I am getting in the monitor and nothing else but I've turned them of for the day.

Discovery so far:

• Changing the 'watch folder' to just point at the bottom folder with 321 avi's in it started immediately.
  When stopping the slow workflow and quitting the app, the processors.exe(s) just continued to run at the usual very low CPU usage

To Do:

• Slowly increase the number of files/folders to find the point it gets slow
  When hitting that point, copy files to local storage and try again

Couple of other notes:

• As a feature suggestion, being able to drag a folder from explorer onto the workflow and have it process that folder would be amazing.
  It's not totally obvious that to switch between workflows that you have to double click on them.
  I think the warning message when you enable an already enabled workflow is wrong.