nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

24x7 TeamSite availability. When?

System

Hi DevNetters -
I'm in a 'flame on' / 'flame off' mood today. Here goes:

---- Flame on ---------

Given the nature of the TeamSite environment, why can't there be more thought / work put into solidifying the TeamSite infrastucture and minimizing the amount of back end administration necessary to keep it up and running?

I would gladly trade any number of front-end enhancements that have been added to the product set for straightforward methods to perform:
- monitoring,
- backup,
- log file rotation
There are unresolved threads in DevNet (and in Support) on all of these going back years.

Using TeamSite as an enterprise-level application, I imagine there are lots of companies like us with users from several time zones, making it nearly impossible to freeze the backing store for extended periods of time for backups or to bounce the TeamSite process to refresh the log files.

We spend an inordinate amount of time keeping the system running smoothly - because when the system doesn't run smoothly its credibility falls with its users and we also can lose critical business functionality.

So right now, our users are happy with the functions we've implemented for them because we've spent the time to keep it going.

But what if we didn't have to spend this time on the back end? I imagine we'd have a lot more adoption and usage of the product set.

We have a very limited number of people working to support TeamSite. I imagine that's the case with many installations. With these limited resources spending time on the back end, the business-focused functions of researching and 'selling' the great potential of implementing business solutions on top of TeamSite often goes undone.

It's analogous to having a cool MGB (sorry, MG lovers..) that doesn't often get driven because the electrical, etc. systems so often need tweaking.

------ flame off ----------

That all said - what we DO implement on top of TeamSite / OpenDeploy rocks.

Comments? Additional case studies?

Find more posts tagged with

Comments

CRB

in one hand, i agree with most of what you said... specially that our users still use TS 5.5.2, and i have to be in the picture everytime to take care of basic issues and to keep TS running smoothly. but in the other hand, i do have 6.7.1 coming in the way and i can see alot of the basic task taken care of by users and interface.
so looking at how much TS improved from 5.5.2 to 6.7.1, i can see all your wishes coming true, it might not be very soon, but i am sure IWOV will get there one day, specially that they always depend on the users input/requests to improve and design...
so i do agree with you, but i also think IWOV is doing good job...

ps: this was my personal opinion, i dont work for IWOV Smiley Happy

13adam13

The backup problem is, in my opinion, the worst part of it all.

Finding a time, in our growing global economy, where a company would choose to take thier systems off-line is an increasing challenge.

Additionally, it can take DAYS to restore a backing store from tape because of the number of files involved. This makes the idea of disaster recovery almost impossible to achieve. The last time I had to recover a 175 GB backing store, it took almost a complete week (granted there were a few problems in the beginning of the process, but still...).

nipper

Yea I happen to agree with Adam, backup is by far the worst. The freeze is hard with backups that are network based.

I remember working for a company that has a filesystem architecture very similar to TS that had the same issue.

They implemented a snapshot freeze, which placed the DB and backing store in a known good state, when that snapshot is done, then backing store could be thawed, usually a matter of seconds to a couple minutes.

Then work can continue.

It did make restoring a backing store more complex since the system needed to ignore (though IIRC, it archived new changes) any files that were modified after the snapshot.

I have had to, in my current gig, run a perl script to freeze, xcopy to another folder, and thaw, since I cannot predict when the backup will run, or have ti run freeze.

Someday I should try to restore the backup.

Andy

13adam13

I have had to, in my current gig, run a perl script to freeze, xcopy to another folder, and thaw, since I cannot predict when the backup will run, or have it run freeze.

Andy

You should consider using OD, if available. I have found it to be the BEST tool for creatign a local copy of the backing store for the backup system to grab onto. Especially with a larger store. Then you can use the DNR to freeze and unfreeze.

Adam

CRB

i do agree with the backup part is hard and dangerous...and freezing the store or shutting it down, is not effecient for businesses that are running TS 24X7.

it is recommand to check the back up copy from time to time... but who has time for that? it is time consuming and there is always a chance that the next back up wont be good... i just hope that we wont ever want to restore contentstore or at least till IWOV come up with solution such as "revert backing store" Smiley Happy where you click on a button and the store goes back the last known good status.

moonkhanna

Chris,

I feel that is an excellent idea. Why dont you initiate with IWOV and get this as a FR. A similar functionality as windows XP\ VIsta. System Restore kind off. That would an excellent utility to have but I am just wondering with the sizes of the backing stores how would the same be implemented.. anyhow, it would be an excellent to have feature so IWOV can start their research on the same if not yet.

nipper

You should consider using OD, if available. I have found it to be the BEST tool for creatign a local copy of the backing store for the backup system to grab onto. Especially with a larger store. Then you can use the DNR to freeze and unfreeze.

Adam

I may try that, if you are playing poker at GearUp I may pick your brain for a few. Besides some of the others would like you distracted. Smiley Very Happy

Migrateduser

RockNRoll guys -
Good to know I'm not alone.

What we've done with backup is to (on Solaris) run a Solaris 'snapshot' command against the backing store and run the backup against the snapshot while the other 1/2 of the snapshot continues to run. At the end of the backup, we release the snapshot. A poor man's NetApp, I suppose.

However, we've never validated a recovery on this. Living on the edge.

I'm building out a new server with TeamSite 6.7.1 and plan to test this mode (including the recovery) several times.

Wally

ISCBorisB

Chris,

I feel that is an excellent idea. Why dont you initiate with IWOV and get this as a FR. A similar functionality as windows XP\ VIsta. System Restore kind off. That would an excellent utility to have but I am just wondering with the sizes of the backing stores how would the same be implemented.. anyhow, it would be an excellent to have feature so IWOV can start their research on the same if not yet.

IWOV wrote themselves a welfare check on the issue long time ago, see whitepaper here

It's nice to commiserate with our mutual pain just do not expect IWOV to pay attention, never happened before. Why should
they, Administrators are usually not the people who pay IWOV

Adam Stoller

IWOV wrote themselves a welfare check on the issue long time ago, see whitepaper here

It's nice to commiserate with our mutual pain just do not expect IWOV to pay attention, never happened before. Why should
they, Administrators are usually not the people who pay IWOV

Heh - a 2002 KB article referencing a 2001 document

I believe the motivation for the KB article was the removal of the iwbackup CLT oh so many moons ago.

The basics are probably still the same - but it is likely that at least some of the specifics have changed since then ...

ISCBorisB

Heh - a 2002 KB article referencing a 2001 document

I believe the motivation for the KB article was the removal of the iwbackup CLT oh so many moons ago...

I know. In 2001 (or was it 2002?) I once tried to restore Clients' corrupted store with iwbackup-created Files (infamous 5.0, pre-Beta)
Although basics/specifics changed we did not have workable on-line backup then and we do not have it now.
And if I can hazard a guess we won't have it in the near future either

ccprofessional

How do other big Web CMS vendors do in the backup department?
I'm talking about Documentum, Filenet, Stellent ...

jed

You need to throw a lot of money at hardware/infrastructure to get a proper backup solution going. One of my biggest peeves is the fact that operations either abort or silently fail while the store is frozen. This makes it impossible for a 24/7 operation. I would love to see the freeze work in the following manner:
o Flush all updates to disk
o Lock the existing store
o Start writing updates to a trasaction log (memory/alt disk/whatever)
o Sync the state upon unfreeze

If I had a snapshotting filesystem, with this type of freeze I could take snapshots/backup once/hour. That would be cool.

Anyway, for the record, below is the methodology we use for backup. We used to only backup to tape, but we learned the hard way that is not a realistic restore option. We have 2x200 GB stores that we backup.

**Current Backup Strategy**

Hardware:
o 3510 Storage arrays (JBOD)
o 3-way mirror for our backing store (software-based w/ solaris volume manager)

Procedure:
o Freeze the backing store
o Offline 1/slice of the mirror
o Unfreeze the backing store
o Backup to tape and disk
o Rsync the mirror

Pros:
o Since we have a 3-way mirror, we are at a very low risk for corruption due to a disk issue. Even during a backup, we retain a 2-way mirror
o This is on the cheap and an expensive netapp-type device is not needed
o We have a very old archive on tape
o We can recover in ~5 hours from disk if required. Currently, we can maintain ~5 days worth of full backups on disk

Cons:
o Our iwfreeze is short, but during the 30-60 second freeze, operations are aborted and users can get weird warnings
o The mirror rsync takes ~15 hours, so we cannot take more than 1 backup/day
o If one of our mirrors goes offline due to a disk error, we cannot backup safely

We are refreshing our hardware real soon now and finally moving to Solaris 10 since 6.7.1 supports it. ZFS has "instant" snapshotting capabilities at the filesystem level. It is real tempting to switch to this filesystem, but testing uncharted waters is never fun. Is ZFS really ready for production? Is ZFS suited for a workload that TeamSite will give it? Or, I can just spend a lot of money on a snapshotting storage device. ****, maybe I do both and hedge my bet.

The other complication is sharing a repository between a primary and standby machine. You need a SAN or something that does replication between two storage devices. Buh. These things are way too complicated.

Gee, I wonder what I worry about at night...

jed

How do other big Web CMS vendors do in the backup department?
I'm talking about Documentum, Filenet, Stellent ...

Aren't some of them more db-centric for the repository? If/when that is the case, standard oracle/mysql/ tools are used which are a lot more tried and true. Good question, though.

13adam13

Aren't some of them more db-centric for the repository? If/when that is the case, standard oracle/mysql/ tools are used which are a lot more tried and true. Good question, though.

The Interwoven CMS is the only CMS that I have worked with that was not DB based.

Cons:
o The mirror rsync takes ~15 hours, so we cannot take more than 1 backup/day

Unfortunately, this sync time can be pretty significant. Have you experienced any I/O delays as a result? I have experienced I/O delays while backing up one of the stores on the system (of course, that was Windows) and keeping the others running.

I may try that, if you are playing poker at GearUp I may pick your brain for a few. Besides, some of the others would like you distracted.

Can't make it to GearUp this year Smiley Sad

. Without a corporate supporter, the cost is prohibitive (perhaps when it comes back to the East coast).

jed

Unfortunately, this sync time can be pretty significant. Have you experienced any I/O delays as a result? I have experienced I/O delays while backing up one of the stores on the system (of course, that was Windows) and keeping the others running.

The backup and sync will eat up a significant amount of I/O. This will affect "raw" operations on the filesystem. For instance, if performing a copy from partition 1 to partition2, it will be much slower.

Below are the resync stats:

device r/s w/s kr/s kw/s wait actv svc_t %w %b
md1 92.4 91.7 3224.8 3102.9 0.0 1.8 10.0 1 64
md1 146.4 147.4 9367.9 9431.6 0.0 1.0 3.4 0 100
md1 111.0 110.0 7104.5 7040.5 0.0 1.0 4.5 0 100
md1 148.0 148.0 9471.4 9471.4 0.0 1.0 3.4 0 100
md1 137.0 138.0 8768.6 8832.6 0.0 1.0 3.6 0 100
md1 145.0 145.0 9242.1 9242.1 0.0 1.0 3.4 0 100

However, it does not really eat into TeamSite performance. I have never seen TeamSite take more than ~1MB/sec during normal operations since many I/O operations get serialized.

Migrateduser

An update on the log rotation issue:

I opened a ticket with Support asking if there was a way to rotate logs (on Solaris) without stopping TeamSite (iwtrace.log, for example). The answer was no, there's no known current way to do it.

However, at Fish's suggestion, I made sure to request that this be a feature request (FR). It turned out that there is an existing FR for this attached to the 6.7.1 SP1 mid-year release and we got added to it.

If this is of interest to others (and why wouldn't it be?), it may be worth speaking up to Support and asking to also be on this FR list.

Cheers,
Wally Box
Nike, Inc.

CRB

it is certainly worth the call, i ll join the list!

can we get content store rotation too Smiley Very Happy

it will be nice..

Adam Stoller

An update on the log rotation issue:

I opened a ticket with Support asking if there was a way to rotate logs (on Solaris) without stopping TeamSite (iwtrace.log, for example). The answer was no, there's no known current way to do it.

However, at Fish's suggestion, I made sure to request that this be a feature request (FR). It turned out that there is an existing FR for this attached to the 6.7.1 SP1 mid-year release and we got added to it.

If this is of interest to others (and why wouldn't it be?), it may be worth speaking up to Support and asking to also be on this FR list.

Cheers,
Wally Box
Nike, Inc.

Excellent - but do you happen to know the actual FR #? It sometimes helps to reference them by ID ...

org.eclipse.birt.report.engine.emitter.csv_1.0.0.jar.zip

Migrateduser

Yes. The FR is 67296.