Unified Job Object

[emcodem]

Hey steinar,

as discussed, i open here a thread for dicussing a possible new unified job object (at least in the API). This time i'll not talk about boilerplate or green field but i'll take into account the current design.

Here a list of current job objects,

Queued

Code: Select all

{
		"item": {
			"display": "Sony.mxf",
			"guid": "20190101-170032-841-131F23D311C8",
			"host": "DESKTOP-UIFQT86",
			"localized": "D:\\Sony.mxf",
			"original": "D:\\Sony.mxf",
			"outbounds": ["enc_av_mp4 20181229-100059-462-cb5c7274c481"],
			"source": "D:\\Sony.mxf",
			"watcher_id": 20181220210008
		}
}

Active:

Code: Select all

{
		"job_id": "20190101-165939-894-BED4019C406B",
		"job_start": "2019\/01\/01 16:59:40",
		"file": "%FILEPATH%",
		"wf_name": "sleep600",
		"splits": [{
			"steps": "2 \/ 2",
			"processor": "Hold",
			"status": "%STATUS%",
			"node": "%HOST%",
			"progress": "50"
		},
		{
			"steps": "2 \/ 2",
			"processor": "Hold",
			"status": "Resting workflow... (TTL:540)",
			"node": "DESKTOP-UIFQT86",
			"progress": "50"
		}]
}

History:

Code: Select all

{
		"wf_name": "Loop_Start_Self",
		"job_start": "2018\/12\/31 16:04:52",
		"job_end": "2018\/12\/31 16:05:05",
		"file": "60690",
		"outcome": "Job aborted by: API Service@DESKTOP-UIFQT86",
		"state": 2
}

Now, my idea is to merge all those objects: The problem with that is that the History is basically showing one object for the "splits" in active and queued only contains basic job info but no active job info.
A combined object could look like this:

Unified job object:

Code: Select all

{
        "job_id": "20190101-165939-894-BED4019C406B", //from active, replaces guid from queued read comment!
	"wf_name": "sleep600",                        //from active
	"wf_id": "12345",                        	  //NEW
	"job_start": "2019\/01\/01 16:59:40",         //from active
        "job_end": "2018\/12\/31 16:05:05",           //from history
        "state": 2                                    //from history, but now indicates if job is running, queued, success, cancelled or error
        "display": "Sony.mxf",                        //from queued
        "watcher_id": 20181220210008,                 //from queued
        "outbounds": ["enc_av_mp4 20181229-100059-462-cb5c7274c481"], //from queued
        "file": "%FILEPATH%",                       //from all, replaces "source from queued",could also be moved to splits?
        "user_variables": [...]                     //NEW
        "static_variables":[...]                    //NEW
        "log_file": "%FULLPATH%"                    //NEW, could also be moved to splits, depending on future logging strategy
	"split": {                //from active but not array anymore, there is one job obj per split 
            "split_id" : "combination of split id and job guid",  //NEW
            "outcome": "%OUTCOME%", //from history
	    "steps": "2 \/ 2",      //from active
	    "processor": "Hold",    //from active
	    "status": "%STATUS%",   //from active
	    "node": "%HOST%",       //from active
	    "progress": "50"        //from active            
		}
}

A major problem in my head is how splits work. It is important that the jobid is always unique. Therefore the first split should always used the original job_id that the queued job already had and all other splits that derive a copy of the first job should generate a new job_id object but also indicate what is their original job_id. Therefore the new jobid for e.g. split[1] could look like %original_guid%_splitid. (or it carries another new field called "mother_job_guid"
We could also get rid of the "split" object and move all it's fields to the root object.

Also important: all fields should always be there but have an empty value in case it is not yet known. This makes parsing a lot easier.

Sure this all might not be complete and it is a matter of discussion, also probably it will never be implemented.

cheers,
emcodem

[admin]

Thanks for all your good advice, emcodem! I'm currently working out a new ground structure for the job data model in FFAStrans. This must be in place before figuring out what goes in there but the new model will have a lot more job info which the current model lacks.

-steinar

[emcodem]

Hey, it would be great if you give me some insights... what are you currently looking after, is it if you want to use OOP Extender or similar? Maybe there is a way i can help you with a C++ or C# dll?

cheers,
emcodem

[admin]

The new model is very object oriented, yes. I think I'm good regarding the the internal restructuring but thanks I think after a good code cleanup it's easier to see where and how one can slowly move to a more modern/mature language.

-steinar