Custom Processor

[whtghst1]

I am writing a custom processor plugin in C#. What are the current proc_data special inputs and outputs in the latest released version FFAStrans? The proc_data is null in the ticket the processor receives. I am only planning on the processor to localize a FTP URI source submitted by API or the Webiterface and update the source to the localized file. I don't want to localize all FTP content which is what the FTP monitor does. The processor will handle dowloading both ftp and ftps (implicit and explicit) URI's. Eventually it will support sftp URI's as well.

Code: Select all

{
  "job_id": "20231023-1701-2043-303b-9d7b875581d8",
  "workflow": {
    "id": "20231023-1359-0015-02d7-b31eaa475a67",
    "path": "C:\\FFASTrans\\processors\\db\\cache\\jobs\\20231023-1701-2043-303b-9d7b875581d8\\workflows\\20231023-1359-0015-02d7-b31eaa475a67.json",
    "hash": "0xA090E97A90364C470C547645B92BCC7B",
    "general": {},
    "special": {}
  },
  "nodes": {
    "next": {
      "type": "plugin_htmlgui",
      "id": "20231023-1633-1444-18ef-6aff69bf6e88",
      "slots": 1,
      "hosts_group": 0
    },
    "start": {},
    "last": {}
  },
  "submit": {
    "method": "manual",
    "user": "sparker",
    "client": "LBNOC-ENGR-W007",
    "time": "2023-10-23T17:01:20.434-04:00",
    "system": "FFAStrans Workflow Manager"
  },
  "finished": {},
  "sources": {
    "sets": [],
    "current_file": "ftp://testuser:testpass@192.168.20.169/test/test.ts",
    "original_file": "ftp://testuser:testpass@192.168.20.169/test/test.ts",
    "localized_file": "ftp://testuser:testpass@192.168.20.169/test/test.ts",
    "pretty_name": "test.ts"
  },
  "retries": {},
  "split_id": "1-0-0",
  "priority": 5,
  "path": [
    {
      "id": "20231023-1633-1444-18ef-6aff69bf6e88",
      "index": 0,
      "name": "FTP FTPS Submit",
      "type": "plugin_htmlgui",
      "wf_id": "20231023-1359-0015-02d7-b31eaa475a67",
      "split": "1-0-0"
    }
  ],
  "step": 1,
  "proc_data": null,
  "processor_output_filepath": "C:\\FFAStore\\20231023-1359-0015-02d7-b31eaa475a67\\20231023-1701-2043-303b-9d7b875581d8\\1-0-0~231023170121109~19164~20231023-1633-1444-18ef-6aff69bf6e88~plugin_htmlgui.outputs.json",
  "cache_dir": "C:\\FFASTrans\\processors\\db\\cache",
  "ticket_file": "C:\\FFASTrans\\processors\\db\\cache\\tickets\\running\\5~20231023-1701-2043-303b-9d7b875581d8~4bb61e100~1-0-0~20231023-1359-0015-02d7-b31eaa475a67~LBNOC-ENGR-W007~81114F.json"

[emcodem]

Hi whtghst1, welcome to the Forum and thank you for using FFAStrans!
proc_data comes from the processor html GUI. The problem when it is null is that the GUI was not opened and saved at all after the processor was inserted to the workflow. What i do in this case is to end with error and set s_job_error_msg to "Please open the processor GUI and save it".

[emcodem]

btw, writing a native "monitor" plugin is not possible. Monitoring involves the need for an independent 3rdparty process to execute the actual monitoring work and eventually kick off a ffastrans job when needed. Native monitors are handled by "def_runner" process, it's responsibilities are among the following:

-) check if workflows are started/stopped and if another host of the ffastrans farm is executing the monitoring process
-) check if monitor processor config of enabled workflow or the workflow properties (e.g. sleep timer) changed
-) execute the watch cycle itself (e.g. connect to ftp, check readiness, build file list difference)

If i would design a source "plugin", i would not design it inside ffastrans at all but as part of the webinterface. In other words my recommendation is that you completely skip the "custom processor" stuff and write a standalone executeable that does the watching (or use something existing) and only if a new file was found, kicks off a ffastrans job via API and hands over the needed parameters as user variables (ftp server connection details and file to download).
Then, in a second step of course a plugin for downloading the specified file can come in handy but i guess i would prefer some existing command line ftp tool like winscp to do it instead of writing a plugin for it.

As an additional note, what i do for sftp monitoring is to use webint scheduler to start a workflow every 5 minutes, the workflow itself is just calling winscp. Winscp then does it's work and synchronizes/downloads what i configured it to do. The local download folder is then watched by a normal workflow with a local folder monitor processor.

[whtghst1]

Thanks for the feedback. I will use your suggested method on handling the null proc_data.

1. I am not trying to create a monitor plugin. I am trying to write a plugin that will process api or webinterface submitted properly formatted FTP URI's and have FFASTrans download the file to the working directory and pass that file location as the source to the subsequent processor. While it is not a "monitor" it would be the first processor in the workflow. The design would take into account that if the submitted source is not a FTP URI it would pass the original source to the subsequent processor in the workflow.

2. I feel that FFASTrans not being able to localize FTP URI's to the work directory is a shortfall in the product. I am using the custom node to fix this.

3. I don't want to use a command line tool as I want to support proper cancellation and cleanup if the job is cancelled in the middle of downloading. .NET already has FluentFTP and SSH.net nuget packages that implements the underlying protocols. My processor just needs to parse the URI execute the download with progress using the those packages API's and support cancellation of that download if needed.

[emcodem]

OK great, all you say makes sense.
Just because we did not document "everything" around ffastrans, i'll draw how your workflow would work using winscp.

-) start the job via API or similar, provide the needed details to download the ftp file either using s_source or custom user variables
-) write text proc writes script file needed for winscp cmd
-) next proc, cmd proc executes winscp download, it should use work_dir as target so in case of cancel ffastrans deletes unfinished temp files. If cancel, ffastrans will kill the winscp process
-) next proc, move the file from work dir to final destination

Having that said, i look forward to any comment of yours regarding plugin proc implementation, let me know if anything in the wiki is unclear or missing.

[whtghst1]

I am setting the "s_source" outputs data entry to the downloaded file location, but the next processor (for testing a delivery processor to copy to another folder) is not using this value for the source file location. It still uses the original ftp URI source.

Below is the proc_data section of the outputs.json from my processor.

Code: Select all

"proc_data": {
    "inputs": [],
    "outputs": [
      {
        "id": "s_source",
        "value": "",
        "data": "C:\\test\\test.ts"
      }
    ]
  },

Below is from job details on the webinterface for the folder delivery. It still shows the source from the original s_source which would be a ftp URI.

Code: Select all

in_file      ftp://user:pass@192.168.20.169/test/test.ts
out_file     C:\FFAStransOut\test.ts

[whtghst1]

I figured out my problem. For the built-in outputs the structure in the outputs array is different.

This is not very clear in the Wiki.

The custom outputs structure is shown and that is what originally used. It took me digging into the force_error source to see how it was setting up the built-in outputs.

If the built-in outputs can be documented better with examples for each on the wiki that would probably help.

It needs to be like the following:

Code: Select all

"proc_data": {
    "inputs": [],
    "outputs": [
      {
        "value": "s_source",
        "data": "C:\\test\\test.ts"
      }
    ]
  },

Thanks,
whtghst1

[emcodem]

Absolutely, i usually mix up the id and value attributes myself and as you correctly spotted, builtin variables need special attention. I probably should have named the "id" field "gui_id" better but you know how such stuff works... once it is ready and shipped and some plugins exist already, we can never change some stuff anymore because we must be backward compatible

I will definitely add more explaination about it on the wiki.

[whtghst1]

How much time does FFAStrans give a processor to process the CTRL-C sent to it for aborting a job. I want to handle the event to do a proper cancel and cleanup. I have implemented the functionality in my code to handle this, but am getting force killed before I can finish.

It this a configurable item in the FFASTrans configuration or is it hardcoded in your source?

whtghsts1

[emcodem]

Well spotted, it is exactly as you assume it to be. If i understand it correctly, between sending ctrl+c and the kill, it sleeps for

Code: Select all

"current cpu usage * debug.retry_sleep" 

where retry_sleep has a default value of 300 ms.
You should be able to influence the value by adding the field retry_sleep in Processors\db\configs\ffastrans.json under debug section of course.
As far as i see, retry_sleep is also used to wait until ffprobe was finished so there is a chance that when you set it to an extremely high value, every job appears to hang at start and after each filter/encoder.

@admin to correct me if i got anything wrong about it.

However, in case you don't like to modify ffastrans.json advanced settings (there is no GUI to modify the debug settings yet), i see an additional chance for you. In theory it should be possible to kick off another detached child process to do the long running operation, it should not be affected by the child pid killing because the child pids of the running job have already been enumerated at this point in the code and they are not being re-enumerated anymore. But again, the child process must be completely detached in order to prevent that killing the main process also kills the child.