Large TeamSite implementation

We are currently using TeamSite 6.7.1 with Search, Report Center, MetaTagger, OpenDeploy and DataDeploy on Win2k3. Our system usage is expected to grow significantly over the next 6-12 months. We are likely to have 400 branches, 200GB of content, 300+ users. I can't find benchmark information for how the system will scale up.

Does anyone out there have a similar or larger implemention? If so, could you kindly share the hardware (platform, CPU, RAM) information, as well as whether you have single instance of TeamSite or multiple instances, how many backing stores, etc.

Any suggestion is greatly appreciated.

Find more posts tagged with

Comments

nipper

Take information like this with a grain of salt. Everything "depends"

That being said I have done very large implementations (as large or larger than you have specified - with the exception of the # of branches)

One on Windows, the others were on Solaris. All were single servers, however on 2 we used separate Dev servers, which fed to a Test (TS) server, which fed to the Prod TS server. That being said, 4 small to medium servers *may* be cheaper that one big honking box (but I doubt it). You have the luxury of using either.

For hardware specs I would look at the TS install manual, the #s posed there are pretty solid.

A few other pointers, make certain you have a separate box for the search engine if you plan to use it.

With that many branches, I would highly suggest multiple backingstores, probably all 8. That will make you life much easier for backups. I hope you have a really solid architect, setting this up will not be trivial.

HTH

Andy

Mr_Cruise

I agree that implementing multiple stores will help considering the size of the backing store. If many of the assets are media then it may be worthwhile implementing media bin so you do not store the media assets on a different server and spread the load.

You should be able to scale the CPU and memory as these are not often the cause of bottlenecks. I would recommend server that has room to add more CPUs if required so if you run out of steam you just add more! Interwoven have a matrix for CPU recommendation so just ask them. Note. If you have the budgets then go slighter higher than what Interwoven recommends as every bit of power will help.

Since you are going to have a large amount of data you want to manage your editions carefully and rely more on tape backups for restore methods.

I believe the most important areas to focus on are disk IO and workflows. TeamSite is disk IO intensive so you want the fastest storage possible and I would suggest fibre SAN storage. This will also allow you to scale if you require more space. Also TeamSite can only cope with so many active workflows, no matter the hardware, so design your workflows carefully.

I have been working with TeamSite for the last 6 years and I feel that Solaris does scale better, as the IO does seem quicker.

If the implementation is global then you may want to consider multiple instances of TeamSite, but this then has cost and architecture implications.

As nipper said just make sure you have a good architect as the best hardware in world will run slowly with the wrong architecture.

Hope this helps and I’m free from next week ;-)

Lee

Migrateduser

Many thank to Andy and Lee for responding. I have some further questions:

1. I also think that multiple backingstores are better for backup & recovery purposes. What are the issues you've seen with multiple backingstores? (Hard-coded /default/main in legacy code doesn't count)

2. Interwoven manual has hardware specs up to 50GB content store. But it does not provide a formula to get to 200GB content from there.

3. We are currently using SAN with fast network connection. We make clone copies of the content store and then run backup against the clone so that the content store is not frozen for a long time. However, too many old editions does start to bother us in some of the older branches. Do you have a good strategy for removing old editions?

nipper

1. I also think that multiple backingstores are better for backup & recovery purposes. What are the issues you've seen with multiple backingstores? (Hard-coded /default/main in legacy code doesn't count)

I have not really seen any issues with them. With luck the largest issue you will have is /default/main

2. Interwoven manual has hardware specs up to 50GB content store. But it does not provide a formula to get to 200GB content from there.

Hmm, I have not paid that much attention. 50 GB is an awful small backingstore. I do not consider 200 GB overly large.

3. We are currently using SAN with fast network connection. We make clone copies of the content store and then run backup against the clone so that the content store is not frozen for a long time. However, too many old editions does start to bother us in some of the older branches. Do you have a good strategy for removing old editions?

That is the proper way for backups. I certainly hope you have checked out the recovery. People can have the best backup system in the world (on paper) but find out at the wrong time that it was not perfect.

I do not make many editions. Special occations, when we release new dev code, which is once or twice a month. I also have no trouble traversing through hidden editions, so I do not bother with release editions. But you can archive off various stores if they are getting too large.

Ghoti may be able to answer that question better than I

Andy

Migrateduser

1. We've tested recovery. In case the real store corrupts, we can immediate mount the clone SAN drive to replace it. If the clone is also corrupted, then we have to recover from tape, which may take half a day to a day.

2. Andy, if you don't publish editions, how do you delete old file versions?

3. Is it safe to assume that if the level of user activities is about the same, and only backingstore grow from 50GB to 200GB, the system performance should be about the same? In other words, TeamSite performance is more related to the number of active workflows, number of content users, etc?

4. Is there a number for the maxinum number of active workflows that a single instance of TeamSite can handle regardless of hardware specs?

5. We used LoadRunner to load test TeamSite 6.1 a few years ago. When 10 simulated users submited jobs at the exact same time, it crashed TeamSite. And IWOV tech support told us that TeamSite could not be tested that way. So when TeamSite manual talks about the number of concurrent users, I really wonder how IWOV defines concurrent users. Is it more like the number of users who are doing something in TeamSite on the same day? :-)

nipper

1. We've tested recovery. In case the real store corrupts, we can immediate mount the clone SAN drive to replace it. If the clone is also corrupted, then we have to recover from tape, which may take half a day to a day.

Good to hear, it is never a fun meeting when you have to tell a customer that their backups were bad.

2. Andy, if you don't publish editions, how do you delete old file versions?

I don't.

3. Is it safe to assume that if the level of user activities is about the same, and only backingstore grow from 50GB to 200GB, the system performance should be about the same? In other words, TeamSite performance is more related to the number of active workflows, number of content users, etc?

Yes this is true. System performance will be largely unaffected by the backingstore size.

4. Is there a number for the maxinum number of active workflows that a single instance of TeamSite can handle regardless of hardware specs?

There was a thread of someone who was having serious issues because he had ~ 30,000 workflows (most were hung). As they started cleaning up, at about 25K, performance was much better, so I think you could expect the hockey stick graph on delay vs # of WF.

This is where WF design is CRITICAL. I have been on maybe 50 different sites. Invariably I see hundreds or thousands of workflows that are hung for one reason or another. 3 common reasons are reviews by someone who does not know to review, duplicate submits, and external tasks that failed. The last 2 must be addressed in the WF.

5. We used LoadRunner to load test TeamSite 6.1 a few years ago. When 10 simulated users submited jobs at the exact same time, it crashed TeamSite. And IWOV tech support told us that TeamSite could not be tested that way. So when TeamSite manual talks about the number of concurrent users, I really wonder how IWOV defines concurrent users. Is it more like the number of users who are doing something in TeamSite on the same day? :-)

To answer your second question first, yea, that is *probably* how IWOV defines concurrent, certainly not 300 people hitting submit at the same time.

Now to your 1st point, I am stunned that this crashed the system. On a 5.5.2 project, during rollout, the VP insisted on and we did 50 people on a con call, all hit submit at the same time. No problem. I have no idea if it was 6.1, Load Runner, an external task or what, but that should never happen.

I do see issues when I submit several hundred files on my WF, but that is due to the generate_content script (finding and attaching images, building index pages, attaching extended attributes) and not WF related.

HTH

Andy

Migrateduser

Thanks Andy for the comments; It's good to know that I am mostly likely going to be ok to scale up. I do have something to cleanup duplicate submits or tasks that sits there for too long. I probably can make it better, though. Thanks for reminding me of that.

I have not come across a good way of deleting or archiving old file versions or editions. Anyone has suggestions?

gzevin

For hardware specs I would look at the TS install manual, the #s posed there are pretty solid.

are you for real? IMO, there is no real data... These figures get copypasted for years, without any relevance for both Windows and Unix platforms

Mr_Cruise

Jane,

To answer your questions :-

1. I also think that multiple backing stores are better for backup & recovery purposes. What are the issues you've seen with multiple backing stores? (Hard-coded /default/main in legacy code doesn't count)

- Sorry I have not worked with multiple stores but I do remember that you cannot share data between stores For example a DCR in one store cannot, out of the box, browse to another store. However, you could work around this with custom CGI’s. This is where you need to analyse your content to understand how to best store it. Also your Content Delivery Architecture (CDA) will have an influence on this architecture.

2. Interwoven manual has hardware specs up to 50GB content store. But it does not provide a formula to get to 200GB content from there.

- How many users are going to use the system and how many are going to be con-current and how are they split geographically? Ideally you want a separate IO channel for each store to gain the most benefit from each store.

3. We are currently using SAN with fast network connection. We make clone copies of the content store and then run backup against the clone so that the content store is not frozen for a long time. However, too many old editions does start to bother us in some of the older branches. Do you have a good strategy for removing old editions?

TeamSite does have an issue when you have too many editions and I working with a client at the moment that has that exact problem. If you need to keep you editions because of legal reasons then could simply copy the editions to another normal disk drive and remove the edition from TeamSite. Hard disk space is cheap so this provides a cheap and simple solution. If you are using meta-data then you will need to cop this as well. You could then create a recovery script if required to restore the data. Explain to the business that you cannot keep too much history in TeamSite but it can be recovered in 24hrs and agree an appropriate SLA.

Hope this helps

Lee

Adam Stoller

TeamSite does have an issue when you have too many editions and I working with a client at the moment that has that exact problem. If you need to keep you editions because of legal reasons then could simply copy the editions to another normal disk drive and remove the edition from TeamSite. Hard disk space is cheap so this provides a cheap and simple solution. If you are using meta-data then you will need to cop this as well. You could then create a recovery script if required to restore the data. Explain to the business that you cannot keep too much history in TeamSite but it can be recovered in 24hrs and agree an appropriate SLA.

Can you provide more background for the "issue when you have too many editions"?

I was under the impression that the "too many editions" problems pretty much went away with 5.5.2 and were non-existent as of 6.x. If you're saying there are still performance issues - please elaborate.

Or is the issue simply one of disk space and the fact that old editions mean that old versions of files remain and use up disk-space that you'd like to free up?

The best way to archive old editions (IMO) is to

The archived editions can be "restored" or accessed by creating the temporary alternate store and restoring the dump from the removable media into it. The data will include all EAs and historical information.

Migrateduser

Thanks Lee.

1. You are right that for switching to multiple backingstore, we will have data sharing issues. We don't really plan to share DCRs accross branch of backingstore. But from time to time, users want to move content from one branch to another. They can easily do that now, but not if the branches are in different backingstores. Therefore, we will need to develop custom script for moving content. Another thing is how to logically divide content into different stores. Can't divide geographically in our case. That's something I need to discuss with the business owners.

2. We expect to have 300 to 500 total users. As I mentioned before, I can't figure out how IWOV defines "concurrent users". Most of users are in one location in the US. A few others scatter around in other regions of the world.

3. For old file version history, our business users want to be able to pick and choose on each file which versions to keep and which versions to delete. The reason is that some of the versions are created due to slight metadata changes on file (some of our external scripts even change a few extended attributes), and these versions are meaningless for them. I told them that's impossible since IWOV does not allow that. I'm thinking to archive old editions to a different location using the iwmigrate tool and then delete these editions from the active backingstore since we do have extensive metadata. Then if users request old versions, we temporarily activate the archive store to find it. However, there is a bug in the TS6.7.1 iwmigrate tool that caused TeamSite to crash last time I ran it (I had to ship a copy of my backingstore to IWOV for them to find the bug). Anyway, that's a different story.

Migrateduser

Can you provide more background for the "issue when you have too many editions"?

I was under the impression that the "too many editions" problems pretty much went away with 5.5.2 and were non-existent as of 6.x. If you're saying there are still performance issues - please elaborate.

Or is the issue simply one of disk space and the fact that old editions mean that old versions of files remain and use up disk-space that you'd like to free up?

We have not really run into performance issues directly related to "too many editions". But still, yesterday an IWOV tech support person sent me what looks like a KB article titled "TeamSite Performance Factors", item #9 is "large numbers of editons". I will go find out what year what this article created and if it's obsolete.

Yes, someone higher up here is concerned about disk space and thinks that deleting old editions will free up spaces.

But I have doubts: 1. disk space is cheap 2. Deleting editions may not free up a lot of space. I've done experiment with the iwmigrate tool on our staging backingstore a while ago. The store size was 20+GB (it was in TS 6.1 format). After iwmigrate to TS6.7.1 format, the size grew to 50+GB plus crashed TeamSite :-(. I've also tested with deleting a few branches, then run iwfsshrink, it did not free up as much space as I expected.

jed

1. We've tested recovery. In case the real store corrupts, we can immediate mount the clone SAN drive to replace it. If the clone is also corrupted, then we have to recover from tape, which may take half a day to a day.

Heh. I once thought the same thing....

~Four days later, the restore of our 250GB backing store from tape was finally completed. Due to the large number of small files in the backing store, the restore took FOREVER! We backup to disk now (and tape, just in case) and can restore 250GB from disk in ~5 hours.

nipper

Heh. I once thought the same thing....

~Four days later, the restore of out 250GB backing store was finally completed. Due to the large number of small files in the backing store, the restore took FOREVER! We backup to disk now (and tape, just in case) and can restore 250GB from tape in ~5 hours.

You should go buy better disk, I hear EMC has good ones. Smiley Very Happy

Migrateduser

Heh. I once thought the same thing....

~Four days later, the restore of out 250GB backing store was finally completed. Due to the large number of small files in the backing store, the restore took FOREVER! We backup to disk now (and tape, just in case) and can restore 250GB from tape in ~5 hours.

I'm a little confused. What took 4 days? Restore from tape? Then retore from another disk took 5 hours? You have all 250GB in a single backingstore?

jed

I'm a little confused. What took 4 days? Restore from tape? Then retore from another disk took 5 hours? You have all 250GB in a single backingstore?

Whoops! Sorry. Too much multi-tasking.

Tape restore == ~4 days
Disk restore == ~5 hours

Adam Stoller

We have not really run into performance issues directly related to "too many editions". But still, yesterday an IWOV tech support person sent me what looks like a KB article titled "TeamSite Performance Factors", item #9 is "large numbers of editons". I will go find out what year what this article created and if it's obsolete.

Yes, someone higher up here is concerned about disk space and thinks that deleting old editions will free up spaces.

But I have doubts: 1. disk space is cheap 2. Deleting editions may not free up a lot of space. I've done experiment with the iwmigrate tool on our staging backingstore a while ago. The store size was 20+GB (it was in TS 6.1 format). After iwmigrate to TS6.7.1 format, the size grew to 50+GB plus crashed TeamSite :-(. I've also tested with deleting a few branches, then run iwfsshrink, it did not free up as much space as I expected.

iwfsshrink should be run after an iwmigrate and periodically (once a quarter or twice a year).

Deleting editions on a regular basis will reduce disk space usage as, over time, you will eventually delete the last remaining pointer to a specific version of a file and then that version of the file will finally be deleted - it's more like treading water than opening a dam.

Migrateduser

iwfsshrink should be run after an iwmigrate and periodically (once a quarter or twice a year).

Deleting editions on a regular basis will reduce disk space usage as, over time, you will eventually delete the last remaining pointer to a specific version of a file and then that version of the file will finally be deleted - it's more like treading water than opening a dam.

Yes, I'd always run a iwfsshrink after every iwmigrate. Except for the one that I mentioned crashes TeamSite and could not ever complete due to a IWOV bug.

chicka

Don't know whether my poast will help your concerns.

1) For WF..
Go for a schedule tasks, kind of a weekend activity in which existing tasks can be deleted so that Performance from user's end may not be degraded.

2) For backup :
Perform an rsync which will help a lot in completing the backup in less time.

3) Its better to go for Editions in order to maintain TS versions under the control.