nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Deployment Job Queuing

fbuno

I am using OD 5.5.1 (NT) and would like to implement the job deployment queuing. Is anybody already using it and how was it implemented?

Find more posts tagged with

Comments

Adam Stoller

Are you talking about scheduling a specific deployment to run at a certain time, potentially on a recurring basis - or - firing off the same deployment multiple times and having the latest request "queued" until the current invokation is finished?
--------------
You can schedule deployments through either the OpenDeploy Admin UI or via using the CLT iwodschedadd (there are also CLTs for: iwodschedget, iwodscheddelete, and iwodschedactivate)

Deployments can be scheduled to occur once, or multiple times on a recurring basis (hourly, daily, weekly, etc.)
-----------------
For the pending queue - you need to configure the Base Server (odbase.xml) with the initiatorProperties element's pendSessions attribute.

Without this option - if you were to do something like:
iwodstart foo
and the 'foo' deployment takes (for example) 10 minutes to complete - and before it has completed you issue the same command:
iwodstart foo
you will get a failure message for this second invokation.

If you enable this feature - you could issue the command:
iwodstart foo
every minute of the 10 minutes it takes to complete the first invokation - and when that original deployment is completed the *last* invokation that was attempted to be initiated while the first had been running - will then be run.

The pending queue is only recommended for use with deployments that use the sourceFilesystem element - and that do *not* use filelists.
-----------------
Does that help?

--fish
(Interwoven, Curriculum Development)

fbuno

The problem I have is that sometimes a deployment is run while the same deployment is being run hence failing.
I did add the following line <initiatorProperties pendSession="yes" /> to my odbase.xml files and tried to run a deployment twice but it is still not working.
The deployment is a SourceTeamsite type.

Adam Stoller

If you are running OpenDeploy from the command line - you can use the -inst instname flag to have it treat each invokation as distinct entities rather than having them queued (this is useful for filelist deployments in which the contents of the filelist may change between invokations)

Did you stop and [re] start the OpenDeploy Base Server process after having added the < instantiatorProperties pendSessions="yes"/> directive? Also note that the directive is plural not singular as you indicated in your post.

I do not recommend using pendSessions with sourceTeamsite - here's why:

If you start the deployment of differences between "ed_0002" and "ed_0001" and then create a new edition and fire it off again ("ed_0003" vs. "ed_0002") before he first one has completed - you're okay. But if you then publish another edition and invoke that as well ("ed_0004" vs. "ed_0003") also before the first invokation has finished - then this last invokation supercedes the previous one in the queue and you will end up deploying:

1) differences between ed_0002 and ed_0001
2) differences between ed_0004 and ed_0003

but you will not have deployed:
3) differences between ed_0003 and ed_0004

If you want to use pendSessions with sourceTeamsite - you need to wrap your deployments with a script that keeps track of the last successfully deployed edition - and use that as the value for previousArea instead of .../EDITION/IW_PREV

If - the differences between the editions are the same file(s) - then the existing pendSessions queueing will work (because it doesn't matter if you deploy foo/2 over foo/1 or foo/3 over foo/1 - you want to ultimately end up with foo/3 at the target system.

If the difference between the editions involve different files - then you may miss deploying some files.

Does that make sense?

--fish
(Interwoven, Curriculum Development)

fbuno

I managed to make it work. The directive was singular rather than plural as you pointed out. So, when is it ok to use pendSession?

fbuno

also, any idea how to wrap deployments with a script that keeps track of the last successfully deployed edition - and use that as the value for previousArea instead of .../EDITION/IW_PREV?

Adam Stoller

Think of the processes involved:
1) Look for the last-known-good deployed edition for a specific branch in some well-known-locatation/file - set a variable to that value (e.g. "ed_0002")
2) If no such entry exists - use the standard value ("IW_PREV")
3) Capture the latest edition name (CLT: iwlasted)
4) Run the deployment with something like -k "ed=$ed" - where within the config file you have something like:
... previousArea="//IWSERVER/.../EDITIION/$ed^"/>
5) if the deployment is succesful - update the entry in the well-known-location/file for the specific branch with the value captured in step '3'

There may be a few other things to think about - but the above should get you started.

--fish
(Interwoven, Curriculum Development)

Migrateduser

is there a limit to the amount of jobs that can be queued at once?

our internal clients arent satisfied with scheduled deployments and want the ability to push content to the public site as soon as it is approved throughout the company.

gwen1

Our testing indicated that there can be only one deployment in the queue with OpenDeploy 5.5.1 sp2 on Solaris. Also, we did not receive errors when a deployment was dropped from the queue. We had to implement using the -inst ## so that a unique deployment is called each time.

Gwen

Adam Stoller

If you want files deployed as soon as they are reviewed and approved - then you want to do deployments as part of the workflow process I assume you're using for the review and approvals. As mentioned though - when doing it this way, you need to make sure you use the '-inst instname' parameter when invoking the deployment.

Common practice is to use either the jobid or taskid that were passed into the external task script, as unique values for 'instname'.

--fish
(Interwoven Senior Technical Consultant)

Migrateduser

gwen, were you able to get the -inst working with transactional deployments (another business requirement of ours)? when we made out deployments non-transactional it worked finem otherwise we would get doCopy() failed errors on the 2nd deployment.

Migrateduser

-inst should work for transactional deployments.

Did you open a case with Tech Support?

Todd Scallan
Group Product Manager
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com

Migrateduser

Todd,

i assumed from your other posts transactional wouldnt work with -inst...it looks like we got the exact error u were talking about?

In reply to:

(this too is subject to potential issues with regard to transactional deployments - as you might try to clobber the same file.iwnew file....)

So - I'd recommend against using '-inst' with transactional deployments

ERROR: Could not rename (e:\iwov_test\daf.iwnew\images\photos\McCloskey.gif.iwtmp) to (E:\iwov_test\daf.iwnew\images\photos\McCloskey.gif)
ERROR: Could not get file data(.\daf\images\photos\McCloskey.gif) from socket.
ERROR: DoCopying() failed, check target log.
ERROR: Failed deploying local_directory[0] () to ()
ERROR: Failed 1 of 1 local_directories

Edited by arthas76 on 05/29/03 10:53 AM (server time).

Migrateduser

I should clarify. The -inst option should work regardless of whether you are running transactional or non-transactional deployments. An example where you might do this is a file list deployment where the file list is unique for each instance. Another example is when you pass in a variable to indicate which definition to execute and each definition specifies a unique target area.

The problem to avoid is having multiple concurrent deployments writing to the same area. One way to get into this trouble is by running two instances of the same deployment config, but you also have this problem running two different definitions or two different deployments that write to the same target area.

Todd Scallan
Group Product Manager
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com

SuppressEmptyRowSP4.zip

Migrateduser

Todd,

What do you suggest for the scenario when you have multiple same-named filelist transactional deployments that may run at the same time? The filelists may deploy to the same area and may contain duplicate file names between the deployments. With only one getting queued up, we're frequently seeing deployments that fail and drop off the face of the earth (because they never run).

Sebouh

Migrateduser

Sebouh,

Unfortunately, there isn't an elegant out-of-box way to handle colliding filelists. If you can't use differential deployments with the pend session feature (simple queue), perhaps build in some of your own checking into the deployment process.

For instance (this is off the top of my head), have a pre-deployment DNR that checks whether a like-named job or some "active" file list is in progress. If so, schedule the job to run xx minutes from now (where xx is a reasonable amount of time in the future to try again based on how long deployments typically take.)

Fyi, there's a feature request filed for more advanced queuing (#39129) to which I've added your company.

Cheers,

Todd Scallan
Group Product Manager
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com

Migrateduser

Todd,

please add my company to the list for that feature request too...queueing of 1 deployment max isnt useful & all of our workflows will be using the same deplyment scenario Sebouh mentioned.

could you go into more details about the feature request? any ETA?

Migrateduser

arthas76,

No ETA at this time. Your profile doesn't list your company. If you don't want to post it here, then please email it to me. Thanks.

I should also mention there is another open FR (#21765) for preventing file collisions on a target when there are multiple deployments running. This would potentially eliminate the need for queuing at the sender, and also take into account that deployments could be coming from multiple sending systems.

Todd

Todd Scallan
Group Product Manager
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com

RobertPisor

arthas,

I work with Gwen - we don't use transactional deployments at the moment. Sorry I can't provide further insight here.

Robert

Migrateduser

D'oh, sorry!

Company is Federated Investors, please put us down for both feature requests.

Migrateduser

Done.

Todd Scallan
Group Product Manager
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com

Adam Stoller

If you are deploying from workflows using filelist deployments (most likely) - then you should probably be using the '-inst instname' flag to treat every such deployment as unique and not have *any* of them queued.

The only reason to queue a filelist deployment (that I can think of right now) is if you are doing some kind of scheduled release / deployment thing and even then you want to use the '-inst instname' flag - but you have to think carefully about whether or not you want dateDifferent enabled (the file you queued an hour ago to get deployed tomorrow might also get queued in 5 minuted to get deployed in one hour - in which case, the file you queued will now be "older" than the file it will want to replace).....

--fish
(Interwoven Senior Technical Consultant)

Migrateduser

we are using a non-Interwoven workflow product to publish content when a business unit clicks an "approve" button

a custom CGI script calls our deployment using the command line tools

COMMAND: d:/iw-home/INTERW~1/teamsite/iw-perl/bin/iwperl

/iw-home/Interwoven/OpenDeploy/OpenDeployNG/solutions/perl/iwodstart.ipl StgTeamsite_TO_StgPersServer

we can pass parameters in, but we will always be using the same deployment. its a business requirement to publish content on demand, and BUs maybe approving at the same time.

thats why i was hoping the:

iwodstart StgTeamsite_TO_StgPersServer -inst aaa -k "mykey=value1'
iwodstart StgTeamsite_TO_StgPersServer -inst bbb -k "mykey=value2'

would work, but we got the error i mentioned earlier when we tested. and since OD only queues 1 deployment at the moment, thats not acceptable either. what would u suggest?

---------------------
Eric B.
Federated Investors
AOL IM: arthas76

Edited by arthas76 on 06/06/03 07:39 AM (server time).

Adam Stoller

What kind of deployment is this?
sourceTeamsite?
sourceFilesystem?
sourceFilesystem with filelist? (You indicated a workflow-like process being involved).

transactional (I believe you indicated so in an earlier post)

Do you have the option of doing just a sourceFilesystem deployment (directory difference) without the filelist option - or would that not work with your process?

I think the problem stems from the particular combination of options being used and while it probably should work in most combinations, if it isn't right now - the best I can suggest is (a) report the problem, (b) try a different combination of options until a fix can be found.

--fish
(Interwoven Senior Technical Consultant)

Migrateduser

Its a Full OD-DD Sync Deployment using sourceFilesystem (must be transactional)

ive included the config file (its pretty short) to see if you can offer any suggestions on how to invoke this simultanouesly using the -inst

Thanks,
eric

---------------------
Eric B.
Federated Investors
AOL IM: arthas76