Replicate a CMS server

Hi guys,
My requirement is something like this.

We want to replicate our CMS server on two machines. Say CMS server is running on machine A and this needs to be replicated as well on another machine B. So that if machine. Machine B will be passive CMS machine and if machine A crashes then we can turn on the machine B so that content is secured.

We can deploy Editions of all the branches to machine B end of day. But problem is, this will be only content. OD wont deploy EA and then at the end of day DCRs will be unusable.
Is there any way out so that content can be replicated on two CMS servers

Find more posts tagged with

Comments

Johnny

Don't deploy the virtualized content, deploy the content store. It is the back end data store containing all files/versions/metadata/workflows.

We do it this way.

1. Freeze TeamSite
2. Deploy content store (iw-store) to a mirror location on the same disk. (faster)
3. Un-Freeze TeamSite
4. Deploy mirror location to back up server.
5. Nightly backups are against the mirror as well.

Adam Stoller

Johnny - I believe you have the right steps here (the frequency of synchronization is a variable that can be played with depending on the systems used for housing the backing store I believe).

Question - these steps will work very easily for Unix. Has anyone done this for Windows where one has to deal with SIDs (e.g. SYSTEM, Administrator, TSIMP_hostname, etc.) that differ between the two servers?

I suppose one could work-in an iwidmap'ing script into the syncronization step, but I'm curious if there are any other solutions - and more importantly - any other issues that might be Windows-specific.

jkm

Thanks Johny and Adams for your replies.

Will these steps across the networks as well ? Like if I want to replicate the complete store between two entirely different networks and my data has to travel across VPN, then OpenDeploy will work or not..
Can you suggest if OD can deploy content across different netwowrks and across firewalls ( obviously OD port is opened in Firewall)

Any particular issues with multiple backing stores and multiple branches while backing store deployments. I believe it should not cause any issues but just want to be sure on this.
As Adam mentioned couple of issues on Windows side, do we forsee any issues on Unix ?

reddyl

You can also use triggers to synchronize the content between multiple TS servers using OpenDeploy. You can also preserver the EA's during the deployment. If the ports are open then OpenDeploy can deploy content to different networks across firewalls.

~ reddy

MichaelP

Question - these steps will work very easily for Unix. Has anyone done this for Windows where one has to deal with SIDs (e.g. SYSTEM, Administrator, TSIMP_hostname, etc.) that differ between the two servers?

Yes we are doing this on Windows. We have two server that we replicate the backing store to another server. We get around the SID issue by using domain users and groups from TeamSite. This leaves on about 3 user/group ids, which we just have a script that iwidmap -c on them.

Will these steps across the networks as well ? Like if I want to replicate the complete store between two entirely different networks and my data has to travel across VPN, then OpenDeploy will work or not..
Can you suggest if OD can deploy content across different netwowrks and across firewalls ( obviously OD port is opened in Firewall)

Yes, it works fine over the network. How we do it is we use OpenDeploy to do a comparision deployment locally first. The a second deployment from the copy to the remote server. This reduces the amount of time the the TeamSite server is out of commission. You could just do a plan copy, but if you use OpenDeploy, since you have already purchased it, then you can just easily only deploy the changes.

-Michael

Valentine

rsync the backing store

You will have to freeze the backing store for that, and the initial copy may take long, but rsync is a good choice for that.

jkm

Thanks guys for your replies
Reddy,
Interesting ..Can you put some more light how to preserve the EA while deployment and triggers to synchronize the content between multiple TS servers. Any example or pointers to manual are appreciated.

nipper

Have you read the OD manual ? The procedure for preserving EAs is disussed. Search for applyExtAttrs="yes"

This is not a way to repicated a complete iw-store, but a way to move a single workarea. So you lose history when you do this (unless you do editions, but then moving iw-store is easier).

BTW, I do not think there is anything documented per the trigger. You need to decide how/when the duplication happens. On a submit ? Do it in a workflow. Every night at midnight ? Schedule it (or wake up and kick it off if you prefer).

Research triggers if you like, but I would suggest NOT using them in this instance.

HTH

Andy

Adam Stoller

OD with EAs is fine for set number of files being deployed from/to TS servers - however the original impetus of this thread was for replicating the entire server, in which case it's the entire backing store that should be deployed - using OD, rsync, whatever.

If using OD to deploy an entire copy of the backing store - you don't have to worry about preserving the EA's on the files being deployed - that data is stored in the backing store and will be transferred anyway.

jkm

Hi Andy,
Thanks for this info. Requirement is extended now and now I need the facilty to replicate single branch to different server and in that case I believe this will be very useful.
I went through the odadmin guide but I am not clear about when we set this attribute to yes then, EAs will be deployed to iw-store by itself.

for example if I need to deploy something like this

/default/main/branch_a/WORKAREA/work on server A to

/EMEA/main/branch_b/WORKAREA/work on server B. Then by just setting only this flag to yes OD will take care by itself to put the extended attributes into iw-store area by itself ?

Any idea if you someone has tried this before.

cheers!!
-Rocky

Have you read the OD manual ? The procedure for preserving EAs is disussed. Search for applyExtAttrs="yes"

This is not a way to repicated a complete iw-store, but a way to move a single workarea. So you lose history when you do this (unless you do editions, but then moving iw-store is easier).

BTW, I do not think there is anything documented per the trigger. You need to decide how/when the duplication happens. On a submit ? Do it in a workflow. Every night at midnight ? Schedule it (or wake up and kick it off if you prefer).

Research triggers if you like, but I would suggest NOT using them in this instance.

HTH

Andy

reddyl

Thanks guys for your replies
Reddy,
Interesting ..Can you put some more light how to preserve the EA while deployment and triggers to synchronize the content between multiple TS servers. Any example or pointers to manual are appreciated.

see page 110 of od.601.admin.pdf for details.

I have used this feature of OpenDeploy to synchronize STAGING and EDITION area between different TeamSite servers using Submit and PublishStagingArea triggers.

This is not a way to repicated a complete iw-store, but a way to move a single workarea. So you lose history when you do this (unless you do editions, but then moving iw-store is easier).

Histroy can be preserved on the target server by sbumiting the files to stanging using iwsubmit CLT. I do agree with Andy that if you want to replicate the full backing store workflows, locks, contents etc.. then triggers will have a huge performance impact on the TeamSite server. An alternative might be to implement a light-weight process that synchronizes the content.

jkm

Reddy,
I have seen that section but it didnt answer my question wheather it will be able to synchronize the content between two different branches.
If yes then does this mean OD will deploy the EAs to iw-store by itself into different backing stores.

Can you post the config file you created when you used OD triggers to synchronize your WA.

Adam Stoller

As has been pointed out (perhaps in a different thread) - the transfer of EAs through OD's feature - will only work if you are doing TeamSite based deployments (i.e. diffing one edition against another, diffing a workarea against staging, etc.) - it will *not* work if you use a filesystem based deployment or regular comparison between source and target areas.

In order to transfer the EAs from one server to another when you are doing a non-TeamSite based deployment, you will need to utilize DNR scripts (one on the source side to capture the EA info into a file - which then has to be deployed to the other side, and one on the target side to read the info from the file and apply it to the deployed assets).

There are a number of such scripts floating around (in the forums, and/or on the support site or perhaps in the devnet library area).

If you're trying to synchronize workareas across servers - and you don't want to take the entire backing store - you will need to utilize the above DNR mechanism.

If you want to synchronize a single branch - but not the workareas - and you want to preserve history and dates - you can make *temporary* usage of TeamSite's MultiStore feature (for long term usage you need to secure a license) - and you can migrate (it's really a copy not a move as the name might suggest) the desired branch into it's own backing store; transfer that backing store to the other server; migrate the transfered copy back into the default store on the target server; remove the temporary second stores on both the source and target machines.

The act of migration does *not* preserve workarea data - only edition data (which is another point - you need to make sure you cut editions on the branch before starting the migration process).

The migration / backing store transfer / migration process will preserve all history.

The deployment from workareas, staging areas, and/or editions will *not* preserve historical information - though it is possible to do so, it would require a significant amount of repetitive deployments with scripts to capture and re-use the historical information (RTFM CLT manual on iwrlog and/or run iwrlog with the '-h' flag for more information) - but the dates of the historical information would not be preserved as such - unless you try something really hackish regarding re-setting the system time on the server before each and every submit operation (and you would probably have to break up the submits to be almost on a per-file basis) ... i.e. this would take a long time to accomplish.

If you want to sync two servers entirely - then the best bet is the original (or near original) suggestion of stopping TS on both servers, transferring a copy of the entire backing store from one server to the other, and then restarting them both. Alternatively, you could use OD or rsync (again, after both servers are stopped) to perform a differential transfer of the entire backing store -- the latter method is quicker but requires that the two backing stores are *always* kept in sync and that no changes are made on the target server only (otherwise the target backing store may have data files that refer to different files which would get overridden by data files from the source backing store and you would most likely end up with a corrupted backing store on the target side - defeating the entire purpose of the exercise).

If work is being performed on both servers and you want to sync them up - it's a much more daunting project. At some point during the process you will have to perform manual diffs/merges of the content on the target side - resolve all conflicts and then submit the files. This would be best done by deploying to a *new* workareas on the target side and then performing the merges between a pre-existing workarea or staging and the new workarea. Besides the amount of time that this takes in general - the main problem is dealing with merges of files for which you are not the authority - so you either need to distribute the merge operations to all the correct authors or someone who can make judgemental decisions on the correct results. Also, depending on the files being merged - the type of file and whether or not it's been formatted differently on the source or target sides can make it nearly impossible to perform the merge without going through additional manual steps.

Example-1: binary files (PDFs, compiled code, etc.) - you cannot merge these unless you have a product specifically designed for such operations.

Example-2: if source server does not have pretty_print enabled for DCRs but target server does - trying to compare / merge the DCRs using just the TeamSite utilities will not work. You would need to ensure that both copies have been identically formatted before attempting the merge operation.

So - before you provide any kind of estimate for completing this project - make sure you have a clear understanding of the requirements and the environment in which you are to implement a solution for those requirements. You could be talking about a job that takes 1-day or 1-year or anywhere in between.

reddyl

As has been pointed out (perhaps in a different thread) - the transfer of EAs through OD's feature - will only work if you are doing TeamSite based deployments (i.e. diffing one edition against another, diffing a workarea against staging, etc.) - it will *not* work if you use a filesystem based deployment or regular comparison between source and target areas.

If work is being performed on both servers and you want to sync them up - it's a much more daunting project.

If you use applyExtAttrs="yes" in the deployment configuration file and target deployment is a TeamSite area OD updates/preserves the extended attributes on the target teamsite server without using any DNR scripts. ( sample od deployment configuration attached ).

If work is being performed on both servers and you want to sync them up - it's a much more daunting project.

In my case work is done on both the servers and synchronizing workarea (includes locks, unlcok, file modifications, new files, directories, workflows ... ) presented with issues performance issues, race around conditions that could not be implemented in the time allocated for the project. So the workarea synchronization was removed from the scope and I designed and implemented the solution to synchronize the STAGING, EDITION areas of different branches to both the servers.

Reddy,
I have seen that section but it didnt answer my question wheather it will be able to synchronize the content between two different branches.
If yes then does this mean OD will deploy the EAs to iw-store by itself into different backing stores.

Can you post the config file you created when you used OD triggers to synchronize your WA.

Attached is the deployment configuration file that is used to sync STAGING areas to multiple teamsite servers.

Johnny

FYI... you can actually set the extended attributes in only one DNR (either source or target - doesn't matter) as the iwextattr command can work across servers.

You could process the manifest and set the EA's respectively.

This sample unix script is an example of how to do it - it's not a DNR, it simply tranverses a directory (workarea or deeper) and set's the EA's that way


#!/usr/bin/sh

this_dir=`pwd`

# $1 - area path     - /default/main/branch/WORKAREA/area
# $2 - get server    - source_server_name # this is the server where you are retrieving the EA's from, if this is the local host you could pass in `hostname`
# $3 - set server and area path //target_server_name/default/main/CWS/WORKAREA/Global

cd $1
find . -type f -print | while read myvar; do

echo "/app/teamsite/iw-home/bin/iwextattr -l -x //$2/$1/${myvar} | /app/teamsite/iw-home/bin/iwextattr -s -x $3/${myvar}"
/app/teamsite/iw-home/bin/iwextattr -l -x //$2/$1/${myvar} | /app/teamsite/iw-home/bin/iwextattr -s -x $3/${myvar}

done
cd $this_dir

jkm

Thanks a lot guys for your detailed replies. I believe applyExtattr will work in my case becasue I need to synchronize content on two TeamSite servers.
If something didnt work then I can always use DNR as suggested by Johny.

One more query, if my DNR fails then can we rollback complete deployment transaction? How do we notify deployment to a failure if any of my DNR script fails.
In my deployment config, I do the following tasks,
Deploy ASCII filelist
Deploy Binary Filelist
On success run DNR script which inserts entries into DB using datadeploy.

I want if datadeploy in DNR script fails then file deployment should also revert.
I can run my DNR before filelist deployment but then if filelist deployment fails then I need to revert database entries. Is there anyway this comeplete thing is in one transaction. I used transaction=yes feature but I believe it doesnt work with DNR.

reddyl

If you want to indicate failure to OpenDeploy server DNR script should print <response code="-2"/>\n. to STDOUT.

<deployment transactional="yes">
<execDeploymentTask useDefinition="ascii_filelist_deployment"/>
<execDeploymentTask useDefinition="binary_filelist_deployment">
<deployNRun>
<dnrDeployment location="source" when="after" triggerPoint="transfer" state="success">
&ltscript cmd="__IW_HOME__/tmp/dd_deployment.ipl">
</dnrDeployment>
<deployNRun>
</execDeploymentTask>
</deployment>

Devnet thread on OD DNR return code.
http://devnet.interwoven.com/forums_vb/showthread.php?t=10565&forumid=12&highlight=DNR+failure

You can find more information on DNR & transaction in OD administrator guide Chapter 5 "Deploy and Run"

jkm

Cool !!
I hope this will solve bunch of problems in my OD life Smiley Happy

jkm

Hi,
I was trying applyExtAttrs attribute to deploy the extended attributes. I am using OD to deploy from one branch to another branch on the same TS server but when I enable applyExtAttrs feature it throws me following exception

ENG: 2006-03-31 18:56:19 BST GMT+01:00 -----------------------------------------------------------
179 ENG: 2006-03-31 18:56:19 BST GMT+01:00 End of deployment config file[/usr/OpenDeployNG/conf/manual_merge.xml]
180
181 ENG: 2006-03-31 18:56:19 BST GMT+01:00 Simulate flag = false
182 ENG: 2006-03-31 18:56:19 BST GMT+01:00 Synchronous flag = true
183 ENG: 2006-03-31 18:56:19 BST GMT+01:00 Start perform internal conversion for: /usr/OpenDeployNG/conf/manual_merge.xml
184 ENG: 2006-03-31 18:56:19 BST GMT+01:00 ***ERROR - Failed converting config file /usr/OpenDeployNG/conf/manual_merge.xml.
185 Error : CONFIG_FILE.
186 Details: applyExtAttr is specified for sourceFileSystem
187
188 ENG: 2006-03-31 18:56:19 BST GMT+01:00 End logfile [/usr/OpenDeployNG/log/src.manual_mergeinst_4.X.Y.0009.log]

Do we need to have different TS server for this?? It doesn't work with if source and target are under same backing store Smiley Sad

Any ideas ??

Adam Stoller

...
 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <deploymentConfiguration>

         <definition name="TEXT">
            <source>
               <sourceFilesystem area="$sourcePath^" filelist="$fileList^">
                     <pathSpecification>
                         <path name="."/>
                     </pathSpecification>
                 </sourceFilesystem>
             </source>

             <target useReplicationFarm="WEB">
                 <targetTeamsite area="$targetPathText^" applyExtAttrs="yes" />
                 <comparisonRules  dateDifferent="yes"/>
                 <transferRules    doDeletes="yes"/>
                 <permissionRules directory="0775" file="0664" group="$targetGroup" user="$targetUser"/>

            </target>
      </definition>

     <deployment transactional="yes">
          <execDeploymentTask useDefinition="TEXT"/>
     </deployment>
  </deploymentConfiguration>
ENG: 2006-03-31 18:56:19 BST GMT+01:00 -----------------------------------------------------------
...
183 ENG: 2006-03-31 18:56:19 BST GMT+01:00 Start perform internal conversion for: /usr/OpenDeployNG/conf/manual_merge.xml
184 ENG: 2006-03-31 18:56:19 BST GMT+01:00 ***ERROR - Failed converting config file /usr/OpenDeployNG/conf/manual_merge.xml.
185 Error : CONFIG_FILE.
186 Details: applyExtAttr is specified for sourceFileSystem
187
188 ENG: 2006-03-31 18:56:19 BST GMT+01:00 End logfile [/usr/OpenDeployNG/log/src.manual_mergeinst_4.X.Y.0009.log]

Do we need to have different TS server for this?? It doesn't work with if source and target are under same backing store

Any ideas ??

The error message seems to imply that it doesn't like using applyExtAttr for sourceFileSystem deployments - the manual seems to indicate this as well:

You can include TeamSite extended attributes (EAs) in a TeamSite comparison deployment to a target TeamSite area.

So if you want to use the built-in feature - you need to use a TeamSite comparison deployment - otherwise you need to use a DNR script (this has been discussed in numerous forum postings - some fairly recently)

jkm

Hi Adams,

I applied applyExtAttrs on the target file system not on source and I am using File Comparision deployment.

I didnt understand what do you mean here.
>>
The error message seems to imply that it doesn't like using applyExtAttr for sourceFileSystem deployments - the manual seems to indicate this as well:

You can include TeamSite extended attributes (EAs) in a TeamSite comparison deployment to a target TeamSite area.
>>

It would be helpful if you can point out in my config file to understand me this in a better way.

Thanks!!

nipper

From the error, it appears you are telling OD to apply EAs on the source filesystem. You cannot do that, they are applied on the destintation filesystem.

jkm

Thats my question Andy and I posted in my first post. I am deploying files from one branch to another branch of same TS machine under same backing stores and trying to set extended attribute in the target branch. Becasue target branch is under the same backing store as of source and that's why its throwing this error??

Does this mean we can not use "applyExtAttrs" feature if we deploy files from one branch to another branch on one TS machine and same backing store. This is not stated anywhere in the OD document.

Adam Stoller

It's not that you're trying to apply the EAs on the sourceFilesystem - it's that you're using the sourceFilesystem directive instead of the sourceTeamsite directive and according to the manual, and the error message - OpenDeploy does not allow the use of applyExtAttrs directive if you are using the sourceFilesystem mode.