Scheduling multiple deployments

Hi,

Current Process: Users cut an edition through a custom GUI (allows users to move content through Pre-Production servers to STAGING to Production servers). They deploy the attached files to the Production Database. The script creates a file list of all the attached and included DCRs and then deploys them one by one to the DB.

Business Problem: Users need an ability to schedule deployments.

Issue: There could be instances where we are scheduling around 1500 DCR deployments (single edition). We cannot schedule those 1500 deployments to run concurrently as that might bring the server down. An alternative discussed is to schedule each deployment instance around 3-5 secs apart but then this will take a lot of time to complete when we run in 1000 odd DCRs.

Is there a viable alternative approach to tackle this business case?

Thanks
Dhruv

Find more posts tagged with

Comments

Adam Stoller

Hi,

Current Process: Users cut an edition through a custom GUI (allows users to move content through Pre-Production servers to STAGING to Production servers). They deploy the attached files to the Production Database. The script creates a file list of all the attached and included DCRs and then deploys them one by one to the DB.

Business Problem: Users need an ability to schedule deployments.

Issue: There could be instances where we are scheduling around 1500 DCR deployments (single edition). We cannot schedule those 1500 deployments to run concurrently as that might bring the server down. An alternative discussed is to schedule each deployment instance around 3-5 secs apart but then this will take a lot of time to complete when we run in 1000 odd DCRs.

Is there a viable alternative approach to tackle this business case?

Thanks
Dhruv

Would they really need to schedule 1500 distinct deployments?
If you're doing this through a custom menu item - you could provide some means for grouping sets of files into time slots (of some granularity) such that you might have 10-20 deployments each with 100+/- files associated.

If they're selecting files from editions, you don't have to worry about the source file changing while waiting for the deployment to occur, but if you allow deployments from staging too (e.g. from workflows) it is possible that one or more files might end up getting reverted when the timed deployment goes off unless you make sure your deployment configuration specifies only to deploy newer files (as opposed to dateDifferent).

infoForDj

Hi,

Appreciate the quick reply.
Reverting content due to conflicting deployments is not a concern.
The deployments will be invoked through a custom GUI.

Scenario: There is a user who creates 2 separate editions for deployment. S/he schedules both of them for lets say midnight. Both jobs/ editions have 100 attached DCRs.
These 200 DCRs will become around 600-700 when we take into account the master-child relationships between DCRs. Each of these 700 DCRs will be deployed individually.

As per your suggestion, we should group these 700 DCRs into a batch of lets say 100 reducing the number of data deployments to 7.
Question is "can we do file list based data deployments? Or do we have to run cron jobs for these batches?"

The scripts are grouped to deploy lets say 7 data types within the same category to 7 different tables. In this cenario, there are 7 definitions. If a file list data deployment is possible, I am assuming, we will have to create different file lists for each definition.

Thanks
Dhruv

Adam Stoller

Hi,

Appreciate the quick reply.
Reverting content due to conflicting deployments is not a concern.
The deployments will be invoked through a custom GUI.

Scenario: There is a user who creates 2 separate editions for deployment. S/he schedules both of them for lets say midnight. Both jobs/ editions have 100 attached DCRs.

Are you talking about two editions each on a different branch? Or two editions on the same branch?

These 200 DCRs will become around 600-700 when we take into account the master-child relationships between DCRs. Each of these 700 DCRs will be deployed individually.

Why do each of the DCRs need to be deployed individually? Are you saying that there is a relationship-chain that is 700 entries long such that the 1stmust be deployed before the 2nd, the 2ndmust be deployed before the 3rd, ... the 699th must be deployed before the 700th??
Or are there N parent-type DCRs and M child-type DCRs such that the latter must be deployed before (or after?) the former?

As per your suggestion, we should group these 700 DCRs into a batch of lets say 100 reducing the number of data deployments to 7.

My original suggestion was assuming scheduled OpenDeploy deployments only - I wasn't taking into account any of this DataDeploy and parent-child relationship information. Thus my suggestion about time-slots might not be as relevant as programmatically determining the groupings and order of deployments based on DCR-type to DCR-type relationship information (the relationship is based on data-type - right? or is it completely arbitrary such that you might get a circular reference?!?)

Question is "can we do file list based data deployments? Or do we have to run cron jobs for these batches?"

Both OpenDeploy and DataDeploy accept filelists (have you read the manuals?)

The scripts are grouped to deploy lets say 7 data types within the same category to 7 different tables. In this cenario, there are 7 definitions. If a file list data deployment is possible, I am assuming, we will have to create different file lists for each definition.

Probably, though I'll admit my DD configing is a bit rusty - you might be able to utilize filters within the configuration such that a single filelist might be able to support multiple deployment specifications within the same overall deployment. And if the parent-child relationship is clear, I think the ordering of the deployment specifications in the configuration is significant, such that you might be able to do the whole thing in one invokation... but I won't swear to that.