Continuous syncing of the backing store?

dppierson

Hello DevNeters,

We're building a disaster recovery (DR) site that includes both our production website and our production content-management system. Our goal is to keep this DR environment in sync with our live production environment as much as possible. We run on RedHat Enterprise Linux AS 4 in both environments.

One idea we're kicking around is the viability of syncing the backing store continuously using a technique built around the "rsync" utility in concert with a Linux kernel module called "inotify". Say your policy is to sync every N minutes or N hours. Rather than requiring rsync to scan all of the backing store’s files with each sync to determine what files have changed, the idea would be to capture the file-update events fired by inotify against the backing-store files during that timeframe, and then rsync just those files. Even then, rsync won’t simply send the changed files over the wire. It communicates with the other end to determine which blocks in the files have changed, and then sends only the changed blocks over the wire.

I have two questions for you:
1. Does anyone have any experiences that might shed some light on the pros and cons of such an approach?
2. Has anyone ever tried copying the backing store while it was not frozen? If so, were there any repurcussions?


Thanks very much for any input,
Doug Pierson
Commonwealth Of Massachusetts

jed

Hello DevNeters, We're building a disaster recovery (DR) site that includes both our production website and our production content-management system. Our goal is to keep this DR environment in sync with our live production environment as much as possible. We run on RedHat Enterprise Linux AS 4 in both environments. One idea we're kicking around is the viability of syncing the backing store continuously using a technique built around the "rsync" utility in concert with a Linux kernel module called "inotify". Say your policy is to sync every N minutes or N hours. Rather than requiring rsync to scan all of the backing store’s files with each sync to determine what files have changed, the idea would be to capture the file-update events fired by inotify against the backing-store files during that timeframe, and then rsync just those files. Even then, rsync won’t simply send the changed files over the wire. It communicates with the other end to determine which blocks in the files have changed, and then sends only the changed blocks over the wire. I have two questions for you: 1. Does anyone have any experiences that might shed some light on the pros and cons of such an approach? 2. Has anyone ever tried copying the backing store while it was not frozen? If so, were there any repurcussions?

Whenever you do anything with the backing store in a non-frozen state, you risk inconsistency. The only way to ensure a valid store on the other end is to transfer the updates of a frozen store. Given that this is a DR-type scenario, it might work and it might be better than nothing, but would you bet your job on this working?

The only way I see replication working in TeamSite is to:
- Get sign-off from exec's that they can lose of to X hours of data
- Freeze store every X every hours
- Create snapshot of store
- Unfreeze
- Send snapshot to DR site

A lot of high-end storage solution will offer both snapshotting and block-level transfers to another node. A snapshot should take less than a second, so end-user impact could be minimal. (Assuming all custom scripts/code are smart enough to detect a frozen store state and re-try the operation. Example: iwcallback)