Handling Large deployments

Hi having issues with our current TS/OD (6.7.2) setup in that users are deploying a huge number of files both in number and MB which is killing our pipe.

Our ideal scenario (and something we are pursuing) would be to perform an initial filelist deployment to a temp folder then a "no deletes" comparison from the temp to the target environment so that only changes within the scope of the filelist are deployed.

We can't do a straight comparison as the source structure is huge and takes just as long to perform the comparison.

Wondering if anyone else has had to deal with limited bandwidth issues in the context of deploying 1000s of files?

Find more posts tagged with

Comments

Rick Poulin

Wondering if anyone else has had to deal with limited bandwidth issues in the context of deploying 1000s of files?

Do your users frequently need to redeploy that many files? In my experience, almost all production workflows end up being filelist-based nowadays, so the publishing of 1000s is a fairly uncommon occurrence. (And because it's uncommon, the likelihood of having multiple concurrent large deployments is near-zero).

Your issues appear to stem from doing areadiffs at all times, so I'd suggest revisiting whether that's the best approach for your use cases. The only scenario I can think of for this approach, is if you're baking HTML in TeamSite, thus needing a bulk republish whenever you change global elements like a navigation structure. In those cases, you'll pretty much have to either get a bigger pipe, or start using SSI or more dynamic technologies.

nipper

You will also want to look at the packet size, which is an OD configuration variable.

The default is 4K and if you deploy over a big pipe, with a long latency (say cross country or over the ocean) then you will be using a small % of the pipe because of the packet size.

MattDWilliams

Typical scenario is that a user changes a couple of files in folder X and Folder Y then for ease (rather than going into the workflow tab finding the job and adding files from other sibling/parent folders), they opt to deploy the parent folder W.

W = 150.25MB
--X
----PDFs = 100MB
----CHANGED.html = 5k
--Y
----IMAGES = 50MB
----CHANGED.html = 2k
--Z

Hence why my thought would be to differential scoped to the filelist...

bar_spaceIssue_1.rptdesign

sheikh1

In this scenario your editors event not required to select any files but just to run the workflow. In your workflow the first task you need to do is identify the files that has been modified but not submitted and attach it to the workflow job.

MattDWilliams

In this scenario your editors event not required to select any files but just to run the workflow. In your workflow the first task you need to do is identify the files that has been modified but not submitted and attach it to the workflow job.

No they just select the folder rather then specific files, the WF then uses the folders/files to pass to the deployment task as a filelist deployment

sheikh1

No they just select the folder rather then specific files, the WF then uses the folders/files to pass to the deployment task as a filelist deployment

I see that but if your editors wants better performance out of your deployments then you need to ask them to run a workflow.
even BETTER let's suppose your editors don't want to change their habits, either case you will be making changes to include only* modified files before doing that just remove all added folders/files and include modified files this way even editors dont have to learn anything or change their actions

Rick Poulin

I see that but if your editors wants better performance out of your deployments then you need to ask them to run a workflow.
even BETTER let's suppose your editors don't want to change their habits, either case you will be making changes to include only* modified files before doing that just remove all added folders/files and include modified files this way even editors dont have to learn anything or change their actions

Or, in a similar vein, train them to use View > My Modified Files. It's way less arduous than adding files one at a time through the workflow tab, but the result is the same.

Sorry but if a user can't figure out that redeploying thousands of files is going to take longer than deploying a handful, then... er.. you know.. quietly mumble under your breath or something.

No they just select the folder rather then specific files, the WF then uses the folders/files to pass to the deployment task as a filelist deployment

How does this even work anyway? Do they not prematurely end up publishing other people's work in progress?