nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

URGENT: OD Receiver Service dies intermittently...

talktome_ys

Good afternoon,
We are experiencing intermittent OD receiver service failures on both of our production servers. The OD service stops unexpectedly, causing a great inconvenience to our TeamSite user community and hindering our ability to promote content to the production servers. The most recent cases have happened to a specific production server and after analyzing the logs, we noticed the following errors:

Base log (hostname_odbase.log):

ENG: 2003-10-09 17:20:26 EDT GMT-04:00 Starting job m6 for deployment fileListDeploy with parameters: {WEEKINV=0, STARTTIME=Thu Oct 09 17:20:26 EDT 2003, DAYINV=0, MONTHDAYS=null, HOURINV=0, DEPLOYNAME=fileListDeploy, JOBID=m6, REPAIR=false, STARTDEPLOY=true, DESCRIPTION=, VERIFY=false, NO_RECUR=NO_RECUR, DEPLOYSUFFIX=, ENDTIME=null, OWNER=SYSTEM, WEEKDAY=null, RECURSET=NO_RECUR, PAD6=null, PAD5=null, SYNCH=true, PAD4=null, KEYVALUES=area=Y:/default/main/Intranet/Financial/STAGING
filelist=e:/apps/INTERW~1/iw-home/custom/temp/file_list_for_task_47581
farm=prod
target=E:/data
, REPEAT=false, PAD3=null, PAD2=null, PAD1=null, YEARINV=0, LOGLEVEL=UNSPECIFIED, MINUTEINV=0}

ENG: 2003-10-09 17:20:26 EDT GMT-04:00 Synchronous deployment. MacroDeploy Session[m6,fileListDeploy] started
ENG: 2003-10-09 17:20:26 EDT GMT-04:00 Adding session m6(fileListDeploy)

API: 2003-10-09 17:20:44 EDT GMT-04:00 Requested start of deployment: fileListDeploy
API: 2003-10-09 17:20:44 EDT GMT-04:00 Calling server to start deployment fileListDeploy

ENG: 2003-10-09 17:20:44 EDT GMT-04:00 Received request to start deployment fileListDeploy.
ENG: 2003-10-09 17:20:44 EDT GMT-04:00 ***WARNING - There is already a deployment 'fileListDeploy' running. Request to start it now cannot be done. Queuing deployment to run later.
ENG: 2003-10-09 17:20:44 EDT GMT-04:00 ***WARNING - Failed starting deployment: fileListDeploy

API: 2003-10-09 17:21:04 EDT GMT-04:00 Requested start of deployment: fileListDeploy
API: 2003-10-09 17:21:04 EDT GMT-04:00 Calling server to start deployment fileListDeploy

ENG: 2003-10-09 17:21:04 EDT GMT-04:00 Received request to start deployment fileListDeploy.
ENG: 2003-10-09 17:21:04 EDT GMT-04:00 ***WARNING - There is already a deployment 'fileListDeploy' running. Request to start it now cannot be done. Queuing deployment to run later.
ENG: 2003-10-09 17:21:04 EDT GMT-04:00 ***WARNING - Failed starting deployment: fileListDeploy
ENG: 2003-10-09 17:21:47 EDT GMT-04:00 Removing session m6(fileListDeploy)
ENG: 2003-10-09 17:21:47 EDT GMT-04:00 Pending deployment fileListDeploy detected. Will start up after this one is completed

ENG: 2003-10-09 17:21:47 EDT GMT-04:00 Starting pended deployment [fileListDeploy]

ENG: 2003-10-09 17:21:47 EDT GMT-04:00 Received request to start deployment fileListDeploy.
ENG: 2003-10-09 17:21:47 EDT GMT-04:00 Starting job m7 for deployment fileListDeploy with parameters: {WEEKINV=0, STARTTIME=Thu Oct 09 17:21:47 EDT 2003, PENDSOURCE=iw_clt, DAYINV=0, MONTHDAYS=null, HOURINV=0, DEPLOYNAME=fileListDeploy, JOBID=m7, REPAIR=false, STARTDEPLOY=true, DESCRIPTION=, VERIFY=false, NO_RECUR=NO_RECUR, DEPLOYSUFFIX=, ENDTIME=null, OWNER=SYSTEM, WEEKDAY=null, RECURSET=NO_RECUR, PAD6=null, PAD5=null, SYNCH=true, PAD4=null, KEYVALUES=area=Y:/default/main/Intranet/Financial/STAGING
filelist=e:/apps/INTERW~1/iw-home/custom/temp/file_list_for_task_47605
farm=prod
target=E:/data
, REPEAT=false, PAD3=null, PAD2=null, PAD1=null, YEARINV=0, LOGLEVEL=UNSPECIFIED, MINUTEINV=0}

ENG: 2003-10-09 17:21:47 EDT GMT-04:00 Synchronous deployment. MacroDeploy Session[m7,fileListDeploy] started
ENG: 2003-10-09 17:21:47 EDT GMT-04:00 Adding session m7(fileListDeploy)

We are using the same OD definition for promoting content from different areas of the branch structure, and in doing so, have enabled the queuing option. This would allow OD to queue the requests sequentially, instead of terminating the requests if a deployment of the same definition was already running.

Base log (scr.fileListDeploy.log):

ENG: 2003-10-09 22:52:55 EDT GMT-04:00 Next deployment flag for intraprod01 is false
ENG: 2003-10-09 22:52:55 EDT GMT-04:00 Internal deployment name is .fileListDeploy.fileListDeploy.intraprod01
ENG: 2003-10-09 22:52:55 EDT GMT-04:00 Got internal config file for .fileListDeploy.fileListDeploy.intraprod01
ENG: 2003-10-09 22:52:55 EDT GMT-04:00 Starting internal deployment intraprod02.fileListDeploy, Thread-98
ENG: 2003-10-09 22:52:55 EDT GMT-04:00 Starting internal deployment intraprod01.fileListDeploy, Thread-99
ENG: 2003-10-09 22:52:55 EDT GMT-04:00 Waiting for 2 children to complete phases
ENG: 2003-10-09 22:52:55 EDT GMT-04:00 EXITING [setup] ENTERING [dotask]
ENG: 2003-10-09 22:53:30 EDT GMT-04:00 EXITING [dotask] ENTERING [commit]
ENG: 2003-10-09 22:53:31 EDT GMT-04:00 1 children failed
ENG: 2003-10-09 22:53:31 EDT GMT-04:00 1 children redirected
ENG: 2003-10-09 22:53:31 EDT GMT-04:00 EXITING [commit] ENTERING [rollback]
ENG: 2003-10-09 22:53:33 EDT GMT-04:00 EXITING [rollback] ENTERING [cleanup]
ENG: 2003-10-09 22:5

Entered By: Yama Shalizi 10/10/2003 10:10:41 AM

3:33 EDT GMT-04:00 EXITING [cleanup] ENTERING [task-failed]
ENG: 2003-10-09 22:53:33 EDT GMT-04:00 Signalling completion to MacroDeploy
ENG: 2003-10-09 22:53:33 EDT GMT-04:00 All 2 children completed their phases

Base log (acr.fileListDepoy.fileistDeploy.BASESERVER.to.RECEIVERSERVER.log):

client: Filelist=[E:\apps\INTERW~1\iw-home\custom\temp\file_list_for_task_47875]
client: Sending [.\Financial\Restricted\IncomeStatements\PL03fcst.xls] [reason file_list] client: OK
client: COMMITTING Transactional Deployment
Receiver-ERROR: ERROR: Renaming [E:\data\Financial\Restricted\IncomeStatements\PL03fcst.xls.iwnew] -> [E:\data\Financial\Restricted\IncomeStatements\PL03fcst.xls]
.
ERROR: Failed to deploy directory (.)
client: ROLLBACK Transactional Deployment
ERROR: Could not read next directive.
Receiver-ERROR: .
ERROR: Failed to deploy directory (.)
ERROR: Could not read content size from remote.
ERROR: Could not read server status, client OK.

....

ENG: 2003-10-10 07:31:27 EDT GMT-04:00 Thread-124 exiting initial
ENG: 2003-10-10 07:31:27 EDT GMT-04:00 Thread-124 entering setup
ENG: 2003-10-10 07:31:27 EDT GMT-04:00 Thread-124 Connecting to host=intraprod02 port=20014

ENG: 2003-10-10 07:35:04 EDT GMT-04:00 Thread-124 ***ERROR - Job ID=m28Deployment=fileListDeploy
Base: failed creating socket, got null.
ENG: 2003-10-10 07:35:04 EDT GMT-04:00 Thread-124 exiting setup
ENG: 2003-10-10 07:35:04 EDT GMT-04:00 Thread-124 entering awaiting-cleanup
ENG: 2003-10-10 07:35:04 EDT GMT-04:00 Thread-124 still waiting for 0
ENG: 2003-10-10 07:35:04 EDT GMT-04:00 Thread-124 exiting awaiting-cleanup

.....

We do have two NIC cards on the production servers, however, only one of the NIC cards is active. I did find a KB article related to that error message, but once again, we only have one NIC card active so I'm not sure if the KB applies here.

Once we identified the OD issue, the service(s) had to be restart for us to continue forward.

Considering this issue is affecting our production environment, your most prompted attention/assistance is appreciated!

I will send an email to support.interwoven.com attaching all necessary logs and config files. Time is of the essence and we would appreciated this case be escalated to the highest support level.

Regards,
Yama Shalizi

Find more posts tagged with

Comments

Migrateduser

Yama,

The queuing feature only keeps the last like-named job on the queue. It does not queue *all* the submitted jobs. This is usually ok for differential deployments, but is a problem for file list deployments if the file list changes from one job to the next.

If timing is critical, please work through Tech Support directly to get your case escalated.

Cheers,
Todd

Todd Scallan
Group Product Manager
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com

talktome_ys

Todd,
Thanks for responding to my inquiry and yes, I have communicated this issue with Interwoven Support. Please refer to case 1151194.

"...queuing feature only keeps the last like-named job on the queue. It does not queue *all* the submitted jobs."

I want to make sure I understand you correctly...the queuing feature only keeps a request or multiple requests of the same definition?

Ex:
Assume there is a filelist definition called DEPLOY which is intiated from a workflow called SUBMIT. There are three users called A, B and C. User A manages a workarea in Intranet/HR branch, user B manages a workarea in Internet/Policies branch and user C manages a workarea in the Intranet/Corporate branch. All three user modify content within their respective locations, select the content and instantiate the SUBMIT workflow. User A's request is the first to be process by OD and User B and C have to wait. If the queuing feature was enabled, what should we expect to happen to user B and C requests? Shouldn't their requests be queued and awaiting execution?

We have a definition called fileListDeploy which is universal. We can deploy from any workarea using this definition. We pass multiple parameters to the definition file, such as, sourceFileSystem area and target, farm and which filelist to use. This definition is referred in the deployment script which is called from a task in a workflow.

fileListDeploy.xml

<?xml version="1.0" encoding="UTF-8" ?>
- <deploymentConfiguration>
<localNode host="intrastage01" />
- <replicationFarmSet>
- <replicationFarm name="training">
<nodeRef useNode="MyLocalHost" />
</replicationFarm>
- <replicationFarm name="prod">
<nodeRef useNode="Intraprod01" />
<nodeRef useNode="Intraprod02" />
</replicationFarm>
- <replicationFarm name="Internet">
<nodeRef useNode="pele" />
<nodeRef useNode="pavo" />
<nodeRef useNode="pictor" />
<nodeRef useNode="proteus" />
</replicationFarm>
</replicationFarmSet>
- <definition name="fileListDeploy">
- <source>
- <sourceFilesystem area="$area^" filelist="$filelist^">
- <pathSpecification>
<path name="." />
- <filters>
<excludePath subPath="templatedata" />
</filters>
</pathSpecification>
</sourceFilesystem>
</source>
- <target useReplicationFarm="$farm^">
<comparisonRules dateDifferent="yes" svrTryCount="3" svrTryInterval="10" svrTryDisableOverwrite="yes" rmReadOnly="yes" />
<permissionRules file="0644" directory="0755" />
<targetFilesystem area="$target^" />
</target>
</definition>
- <deployment transactional="yes">
<execDeploymentTask useDefinition="fileListDeploy" />
</deployment>
<logRules maxBytes="32Mb" level="verbose" />
</deploymentConfiguration>

Why would the receiver services fail? What could/would cause that? We have noticed this intermittent behavior on both production servers, however, the last few have occurred on a specific server.

Here's some information on our environment:

Version 5.5.1 Build 8687 REL Interwoven 20020523
OD 551 sp 1 and upgraded to JRE 3.1_5
IIS 5 (W2K)

If there's any specific informaiton you may need, please let me know.

Regards,
Yama Shalizi

Migrateduser

Assuming the same deployment configuration, if A's job is running and then B submits, B is queued and waiting. If C then submits, B is bumped off the queue in favor of C. When A finishes execution, C runs.

For differential deployments this is usually ok, but for file lists that differ between invocations you will miss any files that should have gone in B's deployment.

If you use the -inst flag, then all three instances of the same deployment can run at the same time. (The queuing feature would be completely bypassed.) However, you need to take care that they aren't all writing to the same target area since this can cause file collisions. (There is an open feature request to provide receiver side queuing so that such collisions are automatically avoided.)

As for why the receiver is failing, it could be due to any number of issues. For example, are you running something like an indexing service that locks files during the deployment? Or could there be multiple deployments writing to the same target area at the same time?

You might also consider applying SP2 or upgrading to OD 5.6 SP1 to eliminate the possibility of any known fixed bugs being the cause.

Todd Scallan
Group Product Manager
Interwoven
t: 408-530-7167
e: tscallan@interwoven.com