Error in def_runner.a3x

[sebird]

Hello,
I just installed the 1.2.2 version, everything seems working well but i have this error message who pop up all the time.
Thank you.

[emcodem]

Aye sebird,

Could you please send all your workflows that have active watchfolders please?

Thanks!

[sebird]

Yes of course, here you are

[emcodem]

OK we found it, it seems to be about "starting" empty workflows.
Steinar thinks it is about the WF EMISSIONS/test but i guess it is better if you just get rid of all empty workflows. (or insert some processor into the empty WF and save)

It also of course will be fixed in next ffastrans release.

By the way, it would be relatively simple to combine most of your move workflows into a single one which gets the target directory for the current file using a mapping list, so you would have all your patterns and target folders in a single configuration file. Is that something you want to work on with me?
This way we reduce some load from the system, you might notice the high load e.g. when starting and stopping the watches hehe

[sebird]

Yes! it was the empty test workflow who bugged. Now everytrhing's fine. Thank's a lot!
I'll be happy to simplify the workflows
Have a good day!

[emcodem]

OK so let's to do all simple moves in a single workflow.

So you have many workflows watching in a single folder but accepting different include patterns and moving to different directories.
The goal is to have only one workflow that watches the same folder but accepting 'all' files and decide later on where to move the files.
For this, i created the powershell script below, it looks for include and exclude patterns in a csv file. The csv looks like this:

Code: Select all

IncludePattern,ExcludePattern,OutputPath
*ASI*,,Z:\ASI\2020
*AOC*,,Z:\MAGAZINE NAME

Above you see that the "ExcludePattern" column is empty. Altough it looks like all your EMISSIONS workflows do not have any exclude pattern, i saw other workflows that also watch in the same directory which actually have exclude patterns, so i thought it might be a good idea to add it.
The patterns work exactly as in ffastrans, you just copy/paste the existing include/exclude patterns.

Workflow:

emcodem_move_patterns.json

(5.46 KiB) Downloaded 310 times

File ( save to c:\temp but move later on to a safe destination)

EMISSIONS.csv

(129 Bytes) Downloaded 323 times

As i cannot upload powershell scripts, you need to save this code to c:\temp\checkpattern.ps1

Code: Select all

$debug = 0 # set to 1 in order to see logdebug messages
$filename = $args[0]
$csvfile = $args[1]
$Outputpath="" # outputpath remains empty unless filename matches the include pattern but does not match exclude pattern

function logdebug($linenum,$what){
#in production we must print the outputpath only, so debug can just be enabled for testing on commandline
    if ($debug){
        Write-Host("csv line $linenum $what")
        
    }
}

$linenumber = 2;
Import-Csv $csvfile | ForEach-Object {
   
    $includeArray = $_.IncludePattern.Split("|")#IncludePattern is a headline in the csv, as well as ExcludePattern and OutputPath
    
    #iterate include patterns
    foreach ($e_ptrn in $includeArray){
        if ($e_ptrn.Trim() -eq ""){
            logdebug $linenumber,"Include Pattern was empty"
            continue
        }
        if ($filename -like $e_ptrn.Trim()){
            logdebug $linenumber,"Filename matches include pattern $pattern"
            $Outputpath = $_.OutputPath
        }
    }

    $excludeArray = $_.ExcludePattern.Split("|")
    #iterate exclude patterns
    foreach ($e_ptrn in $excludeArray){
        if ($e_ptrn.Trim() -eq ""){
            logdebug($linenumber,"Exclude pattern was empty")
            continue
        }
        if ($filename -like $e_ptrn.Trim()){
            $Outputpath = "excluded"
            logdebug($linenumber,"Excluding File because it matches exclude pattern $e_ptrn")
        }
    }
    $linenumber =  $linenumber+1

}

Write-Host $Outputpath #finally write the found path to stdout. if nothing was found this will print just empty

In the end, if everything works for you, make sure to edit the "conditional" node and check the "dispel" checkbox so you get rid of all Jobs that actually did nothing.
Lemme know if this works for you.

[emcodem]

Due to huge interest in this workflow (Benjamin asked a question about it ), here is an updated version.
This Version allows one to deliver one source file that matches a pattern to multiple destinations.

Content of file c:\temp\EMISSIONS.csv (please use Excel or such to modify the csv!):

Code: Select all

IncludePattern,ExcludePattern,OutputPath1,OutputPath2,OutputPath3,OutputPath4,OutputPath5,OutputPath6
*.*,foo,C:\temp\default
*3*|*4*,,C:\temp\3,c:\temp\4
*5* | *6*,,C:\temp\5,c:\temp\6

Explaination: You could add as many OutputPath's as you like, just make sure each one has a unique number, e.g. OutputPath7

Content of file c:\temp\checkpattern.ps1

Code: Select all

$debug = 0 # set to 1 in order to see logdebug messages
$filename = $args[0]
$csvfile = $args[1]
$Outputpath = @() # outputpath remains empty unless filename matches the include pattern but does not match exclude pattern

function logdebug($linenum,$what){
#in production we must print the outputpath only, so debug can just be enabled for testing on commandline
    if ($debug){
        Write-Host("csv line $linenum $what")
        
    }
}

$linenumber = 2;
$havematch = 0
Import-Csv $csvfile | ForEach-Object {
   logdebug $linenumber,":"
    $includeArray = $_.IncludePattern.Split("|")#IncludePattern is a headline in the csv, as well as ExcludePattern and OutputPath
    
    #iterate include patterns
    foreach ($e_ptrn in $includeArray){
        if ($e_ptrn.Trim() -eq ""){
            logdebug $linenumber,"Include Pattern was empty"
            continue
        }
        if ($filename -like $e_ptrn.Trim()){
            if ($havematch -eq 1){
                #reset output path if another line already matched - ensure only last matching line defines output
                $Outputpath = @()
            }
            $havematch = 1 # make sure only the last matched line in the csv counts
            logdebug $linenumber,"Filename matches include pattern $pattern"
            $csvline = $_
            foreach ($property in $csvline.PSObject.Properties){
                if($property.Name -like 'OutputPath*'){
                    if ($property.Value){
                        if ($property.Value.Trim() -ne ""){ #only add if after removing whitespaces, it is not empty
                            $Outputpath += ($property.Value)
                        }
                    }
                }
            }
             
        }
    }

    $excludeArray = $_.ExcludePattern.Split("|")
    #iterate exclude patterns
    foreach ($e_ptrn in $excludeArray){
        if ($e_ptrn.Trim() -eq ""){
            logdebug($linenumber,"Exclude pattern was empty")
            continue
        }
        if ($filename -like $e_ptrn.Trim()){
            if ($havematch -eq 1){
                $Outputpath = @()
                $havematch = 0;
            }
            logdebug($linenumber,"Excluding File because it matches exclude pattern $e_ptrn")
        }
    }
    $linenumber =  $linenumber+1

}

if ($Outputpath.length -eq 1){
    $Outputpath[0] = $Outputpath[0] | ConvertTo-Json
    $Outputpath[0] = '[' + $Outputpath[0] + ']'
     Write-Host $Outputpath #finally write the found path to stdout. if nothing was found this will print just empty
}else{
    $outputjson = $Outputpath | ConvertTo-Json  -Compress  
    Write-Host $outputjson #finally write the found path to stdout. if nothing was found this will print just empty
}

NOTE: to run this workflow, you must download and install the Foreach Processor from here.
https://ffastrans.com/wiki/doku.php?id= ... processors
(sidenote, in future release 1.3, a foreach processor is part of ffastrans but for this workflow you would still need to install the plugin, it has a different nodeid)

Finally, the corresponding Workflow:

emcodem_move_patterns_multipledest.json

(7.25 KiB) Downloaded 294 times

Please make sure you use other paths than C:\temp for the final productive workflow, all you need to do is to change the path of the ps1 and csv file in the command executor processor.

[momocampo]

Thanks a lot my dear Emcodem, really nice of you.
I really think it can be useful when you have 1 monitor folder for many different kind of input files.
thanks!

B.