Unified Job Object
Posted: Tue Jan 01, 2019 4:39 pm
Hey steinar,
as discussed, i open here a thread for dicussing a possible new unified job object (at least in the API). This time i'll not talk about boilerplate or green field but i'll take into account the current design.
Here a list of current job objects,
Queued
Active:
History:
Now, my idea is to merge all those objects: The problem with that is that the History is basically showing one object for the "splits" in active and queued only contains basic job info but no active job info.
A combined object could look like this:
Unified job object:
A major problem in my head is how splits work. It is important that the jobid is always unique. Therefore the first split should always used the original job_id that the queued job already had and all other splits that derive a copy of the first job should generate a new job_id object but also indicate what is their original job_id. Therefore the new jobid for e.g. split[1] could look like %original_guid%_splitid. (or it carries another new field called "mother_job_guid"
We could also get rid of the "split" object and move all it's fields to the root object.
Also important: all fields should always be there but have an empty value in case it is not yet known. This makes parsing a lot easier.
Sure this all might not be complete and it is a matter of discussion, also probably it will never be implemented.
cheers,
emcodem
as discussed, i open here a thread for dicussing a possible new unified job object (at least in the API). This time i'll not talk about boilerplate or green field but i'll take into account the current design.
Here a list of current job objects,
Queued
Code: Select all
{
"item": {
"display": "Sony.mxf",
"guid": "20190101-170032-841-131F23D311C8",
"host": "DESKTOP-UIFQT86",
"localized": "D:\\Sony.mxf",
"original": "D:\\Sony.mxf",
"outbounds": ["enc_av_mp4 20181229-100059-462-cb5c7274c481"],
"source": "D:\\Sony.mxf",
"watcher_id": 20181220210008
}
}
Code: Select all
{
"job_id": "20190101-165939-894-BED4019C406B",
"job_start": "2019\/01\/01 16:59:40",
"file": "%FILEPATH%",
"wf_name": "sleep600",
"splits": [{
"steps": "2 \/ 2",
"processor": "Hold",
"status": "%STATUS%",
"node": "%HOST%",
"progress": "50"
},
{
"steps": "2 \/ 2",
"processor": "Hold",
"status": "Resting workflow... (TTL:540)",
"node": "DESKTOP-UIFQT86",
"progress": "50"
}]
}Code: Select all
{
"wf_name": "Loop_Start_Self",
"job_start": "2018\/12\/31 16:04:52",
"job_end": "2018\/12\/31 16:05:05",
"file": "60690",
"outcome": "Job aborted by: API Service@DESKTOP-UIFQT86",
"state": 2
}A combined object could look like this:
Unified job object:
Code: Select all
{
"job_id": "20190101-165939-894-BED4019C406B", //from active, replaces guid from queued read comment!
"wf_name": "sleep600", //from active
"wf_id": "12345", //NEW
"job_start": "2019\/01\/01 16:59:40", //from active
"job_end": "2018\/12\/31 16:05:05", //from history
"state": 2 //from history, but now indicates if job is running, queued, success, cancelled or error
"display": "Sony.mxf", //from queued
"watcher_id": 20181220210008, //from queued
"outbounds": ["enc_av_mp4 20181229-100059-462-cb5c7274c481"], //from queued
"file": "%FILEPATH%", //from all, replaces "source from queued",could also be moved to splits?
"user_variables": [...] //NEW
"static_variables":[...] //NEW
"log_file": "%FULLPATH%" //NEW, could also be moved to splits, depending on future logging strategy
"split": { //from active but not array anymore, there is one job obj per split
"split_id" : "combination of split id and job guid", //NEW
"outcome": "%OUTCOME%", //from history
"steps": "2 \/ 2", //from active
"processor": "Hold", //from active
"status": "%STATUS%", //from active
"node": "%HOST%", //from active
"progress": "50" //from active
}
}
We could also get rid of the "split" object and move all it's fields to the root object.
Also important: all fields should always be there but have an empty value in case it is not yet known. This makes parsing a lot easier.
Sure this all might not be complete and it is a matter of discussion, also probably it will never be implemented.
cheers,
emcodem
I'm currently working out a new ground structure for the job data model in FFAStrans. This must be in place before figuring out what goes in there but the new model will have a lot more job info which the current model lacks.