nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

scaling OpenDeploy

swong74

Hallo,

Is there any document/whitepaper regarding scaling OpenDeploy? We have around 50 receivers now and all are majorly reliant on ONE base server. We've been experiencing among other things a growing number of outages due to out-of-memory (reported in the Java) and 500 errors (may be associated with the memory). Any recommendation to scale and how easy/hard it would be. We would like to plan on it now before the bottom falls off. ;-)

- sw

Ciao ?:-)
S. Wong

Find more posts tagged with

Comments

pflory

S. Wong,

What is the nature of your deployment architecture? Single machine fan-out, multi-tiered? What versions of OD?

Some environment info would be helpful.

Paul

Paul Flory

gzevin

in some cases I would reccomend to use rsync instead of OD (or along with OD). much more reliable - and is pretty fast. It's not as flexible as OD, but does many tricks

Greg Zevin, Ph.D. Comp. Sc.
Independent Interwoven Consultant/Architect
Sydney, AU

Migrateduser

A couple of things to look at:

In the Nov 2003 edition of the OD 5.6 Admin Guide, p. 137, there are examples and recommendations for how to conserve deployment legs when running large fan-out deployments.

Also, one of the things that consumes memory is the iwodstart command line tool, which is implemented in Java. There is a newer tool called iwodcmd that's much more lightweight. So if you are invoking deployments frequently via the command line, choose iwodcmd over iwodstart.

Todd Scallan
Director of Product Management
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com

nipper

Todd, I certainly hope the syntax for iwodcmd is EXACTLY the same as iwodstart.

Somehow I doubt it. While I am happy that it is more efficient, I work on the same project as S. Wong and will be one of the people rewriting our deployment ipls.

You could have made my like *much* easier by making iwodstart work with less memory.

Andy

Migrateduser

Is it the OS that is running out of memory or just the java process? If java can you find the line of code that is spawning the JVM? If so I think there are some parames you can pass that increase the maximum memory size of the JVM.

swong74

Ok, been offline when you guys started replying ...

This outta set it out ...

Our base servers are a mix ... UNIX and NT. I think our UNIX servers are holding up but that's probably because we have one hefty UNIX environment and only used by TeamSite. That's the project that nipper is involved with.

Our NT base server is our non-TeamSite OpenDeploy and being used for a multitude of fan-out deployments. The receivers are a combination of NT/UNIX and some are deployments that are multi-target (replicationFarm), some I have scripted to deploy iteratively (as opposed to OpenDeploy's replicationFarm), some have reverse deployment then chains to a forward deployment ... etc. All the scripted ones use iwodcmd as I've already gone down the discussions about iwodstart and iwodcmd. This is the one I'm concerned with and would like to scale if I could. This sole base targets combination of UNIX/NT receivers (50 to date).

Recently as we ramped up the use as we migrated many of our users to the OpenDeploy, we've been getting more reports of users not being able to get to the UI or use it. One case, people can log on but nothing else ... any tab would generate 500 error. Sometimes, the login just fails. We have created a script that monitors the login and the incidences of these are growing ... not daily but at least once a month.

The reason we are relying on one base server is cost and because we would like to have a central repository for content. But like I said, if there is any recommendation to multi-tier without making it difficult for our users, I'd like to explore it.

I saw a recommendation to rsync. Yes, rsync is a good way ... I've used it extensively for my "old" method of deploying for our company web sites (one I have scripted many years ago and sad to see go but alas, eol is near). But that's the thing, we've implemented OpenDeploy to get away from rsync ... and to have to go back ... that's just funny I think. ;-)

Ciao ?:-)
S. Wong

swong74

Oh, and here's a response for John ...

I think it's the JVM that's running out of memory. Because if it were the server itself, we also monitor memory utilization on that and haven't heard a peep. I've opened a ticket with support but it's still in "researching" status. Which is why I posted here ... slightly differently because I think the solution might be to scale.

I am aware of the limitation on number of legs as Todd pointed out. But then, here's my idea ... if it's at all possible ... to cluster a pool of base servers that share one central doc repository ... and load balance against them eh? :-D That would up my deployment leg limitiations. ... wishful thinking?

- sw

Ciao ?:-)
S. Wong

pflory

Out of curiousity, do you have a trace of memory/io/cpu usage for the processes of interest?

vmstat, iostat or other top-like information would be interesting to review, especially during
those periods where OD stops responding or otherwise poops out.

Paul Flory

Adam Stoller

Todd, I certainly hope the syntax for iwodcmd is EXACTLY the same as iwodstart.

Not EXACTLY - but VERY CLOSE.

Instead of iwodstart configname flags
it becomes: iwodcmd start configname flags

So - you can use a real simple regex (s|iwodstart|iwodcmd start|g;) to fix all your code.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

swong74

Hallo Paul,

Any recommendations on NT equivalents for top and iostat? I believe my colleagues have seen spikes in CPU/memory in the Event Viewer during the problematic times.

Tnx.

Ciao ?:-)
S. Wong

pflory

Performance monitor (in Administrative Tools) should offer equivalent utilities... right?

Paul Flory

Migrateduser

There's a wrapper script that you can call for iwodstart, which in turn calls iwodcmd. It means launching an extra process, but it's not a Java process. Take a look in <odhome>/OpenDeployNG/solutions/clt

Otherwise, if you can adjust your calling code, the commands are pretty similar:

iwodstart <parms>
iwodcmd start <parms>

Todd Scallan
Director of Product Management
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com

swong74

Hi Paul,

Yeah, I got confirmation from my colleague that he sees spikes at the time we're experiencing problems. So far, we've "survived' without problems last week ... but then again, we've been doing multiple system reboots because of patches being applied so we probably don't get to the saturation point.

Anyway, is there any scalability strategy for OpenDeploy? I think we would like to centralize the content repository ... or maybe get arguments pro/con distributed model.

Thanks.

Ciao ?:-)
S. Wong

Migrateduser

Some strategies to consider:

+ Use multi-tiered deployments as a way to cascade fan-outs. For example, deploy to 10 servers which in turn deploy to 10 more. This would give you 100 targets (not counting the intermediate hops) without loading the source host too much. But this approach also requires that you have additional Base Servers installed on the intermediate nodes.

+ Serialize deployment legs within a deployment. This has the advantage of conserving deployment legs, but takes longer than having all the legs run in parallel. (Mentioned earlier... In the Nov 2003 edition of the OD 5.6 Admin Guide, p. 137, there are examples and recommendations for how to conserve deployment legs when running large fan-out deployments.)

Todd Scallan
Director of Product Management
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com