Hey good people! How is everybody doing! (better, if you haven't read or watched any news lately)
Anyway...
I have a few workflows managing processes that are restricted to specific nodes and run a few local commands.
The thing is I'm multiplying these workflows to make sure that all processes are kept in the same machine from the start. So there "MyWF@Node1", "MyWF@Node6" and so on...
Couldn't there be a variable to detect WF was initially picked up (started) by NodeX, so that I could use it on the following (or specific) processors and make sure that any local file usage refers to "this machine"?
Thank you all! Stay safe
Use node variable
Re: Use node variable
Hi @knk
Mr. @admin has for sure more to say about this but i believe @momocampo keeps his workflows on same node with local cache by just unchecking "use global shared media cache". I believe it was like that if you do this, the wf will stay on this node - but @momocampo to confirm...
Mr. @admin has for sure more to say about this but i believe @momocampo keeps his workflows on same node with local cache by just unchecking "use global shared media cache". I believe it was like that if you do this, the wf will stay on this node - but @momocampo to confirm...
emcodem, wrapping since 2009 you got the rhyme?
Re: Use node variable
Hey @emcodem!
This could be good starting point, to have an "opt-out" of global cache, but only if it was a WF related option, and I can't find such option in FFAS (if there is, let me know!).
Even so, starting a local process on NodeX and, for example, later waiting for a specific local file to arrive, we need to make sure that the same it's the same NodeX running those two separated processors in disctinct steps of the WF.
All without removing the Nodes from the Farm and keeping them up and running for all other tasks.
This could be good starting point, to have an "opt-out" of global cache, but only if it was a WF related option, and I can't find such option in FFAS (if there is, let me know!).
Even so, starting a local process on NodeX and, for example, later waiting for a specific local file to arrive, we need to make sure that the same it's the same NodeX running those two separated processors in disctinct steps of the WF.
All without removing the Nodes from the Farm and keeping them up and running for all other tasks.
Re: Use node variable
Yeah if we talk about not yet existing features, a good thing would be to have a "exec on node/group" and or even the "cache folder" setting in all processors.
The exact thing you want to do can currently more or less only be done using the remote executor plugin proc that i came up with for linux/mac process execution but of course it is not the same as you would need to do everything on commandline
The exact thing you want to do can currently more or less only be done using the remote executor plugin proc that i came up with for linux/mac process execution but of course it is not the same as you would need to do everything on commandline
emcodem, wrapping since 2009 you got the rhyme?
Re: Use node variable
@emcodem agree!
(looking at you v1.5....
)
(looking at you v1.5....
)Re: Use node variable
Hi knk,
So let's see if I can paint the picture correct. And let's just for clarity use the term "host" for computers doing the dirty work. In FFAStrans context a "node" is the boxes also known as processors.
Anyway, if I understand properly, you want to be able to configure a processor to lock all subsequent processors to the same host, right? Please correct me if you think I misunderstand your feature request.
PS: Not sure fiddeling with the cache work dir is a good approach for this.
-steinar
So let's see if I can paint the picture correct. And let's just for clarity use the term "host" for computers doing the dirty work. In FFAStrans context a "node" is the boxes also known as processors.
Anyway, if I understand properly, you want to be able to configure a processor to lock all subsequent processors to the same host, right? Please correct me if you think I misunderstand your feature request.
PS: Not sure fiddeling with the cache work dir is a good approach for this.
-steinar
Re: Use node variable
We have 2 different topics regarding cache in this thread, it is not clear which one you refer to.
Option 1 is how momocampo currently uses the checkbox "use global media cache" which i believe, when unchecked leads to the desired behaviour of locking the workflow to one node (might be wrong here)
Option 2 is my suggesition to be able to dynamically set the cache folder. I believe this is a topic we discussed in combination with abillity to write output from encoders directly to target folder instead of first cache, then deliver. Resetting the cache folder for the encoder might be one way to do that but sure there are much better
emcodem, wrapping since 2009 you got the rhyme?
Re: Use node variable
1. With local cache you're not locking workflows to one host. You're locking to the host that initiate the first processor. From then it's the same host doing all subsequent processors. And this is the only way it can work when not using shared cache.
2. My suggestion was to enable any processor at any point in the workflow to lock all subsequent processors to one host. And because processors up to the locking point might NOT be initiated by the the same hosts, suddenly switching cache location would not work. So my objection was where we intitially use shared cache. With this config, switching to local cache per job must be set from the very first processor, which would be far less dynamic than my suggestion.
-steinar
2. My suggestion was to enable any processor at any point in the workflow to lock all subsequent processors to one host. And because processors up to the locking point might NOT be initiated by the the same hosts, suddenly switching cache location would not work. So my objection was where we intitially use shared cache. With this config, switching to local cache per job must be set from the very first processor, which would be far less dynamic than my suggestion.
-steinar
Re: Use node variable
Hey Steinar (@admin), absolutelly right...the local cache is not what I was aiming for, but I think you got the idea. I really need to keep the farm and work on the shared cache, but have some processes running on specific Nodes.
This all started with limiting internet access in some nodes along with some local scripting.
If I have a WF that runs local scripting/internet access only available on Node1 and Node2, I can't have both running a single WF and must duplicate it.
I have no way of grouping these two Nodes and still make sure that IF Node1 is the one that started the process, then it will be Node1 to execute the following processors that require access to the locally generated script outputs.
This all started with limiting internet access in some nodes along with some local scripting.
If I have a WF that runs local scripting/internet access only available on Node1 and Node2, I can't have both running a single WF and must duplicate it.
I have no way of grouping these two Nodes and still make sure that IF Node1 is the one that started the process, then it will be Node1 to execute the following processors that require access to the locally generated script outputs.