nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Intermittent deployment Failure

VivekPolaris

Hi All,

My base server deploys to 6 target servers. Since last month I am facing Intermittant deployment failure issue. Series of steps :

1. When deployment fails we check the OD status of base and target servers, it shows :

OpenDeploy server host: localhost, RMI registry port: 9173
OpenDeploy server is alive.
0 deployments are currently active.

2. Then we check the logs related to that workflow in Base Server. We get 2 types of errors :

2.1. Failed creating socket.
2.2 Send attributes to receiver got RC = -1

3. It happens sometime in 2 servers, sometime in 3 servers out of 6 servers. After identification of servers, we get the od restarted and deploymnet happens successfully aterwards.

I checked but could not find any pattern in deploymnet failure. Last week it happend twice, but this week it gave no problem.

Cheked the od.log as well, but could not find any clue.

Please provide some suggestions how to track this. Which steps I should take to handle this intermittent failure.

TS : 6.7.2 Build
OD Base 7.2.0

Thank you !!
---------------- 1st error Details ----------------------------------

ENG: Thread-20 entering setup
LIB: 2014-03-03 09:26:52 LOGFILE=[/export/opt/Interwoven/OD/OpenDeployNG/log/src.cjh_site_UAT_deploy9894671.CITIJPCJHUATWEBENGJP.jutpcsjcmsw1.to.juodgcgjfpvw1.log]
LIB: 2014-03-03 09:26:52 Locale=[English_UnitedStates.US-ASCII@Binary]
ENG: Thread-20 Connecting to host=juodgcgjfpvw1 port=20014

ENG: Thread-20 Sender Side timeout = 0

ENG: Thread-20 Buffer Size 8000
ENG: Thread-20 ***ERROR - Job ID=m2Deployment=cjh_site_UAT_deploy9894671
Base: Exceeded all 3 retries to get a connection with the target OpenDeploy host.
Failed creating socket.
LIB: 2014-03-03 09:26:57 LOGFILE=[/export/opt/Interwoven/OD/OpenDeployNG/log/src.cjh_site_UAT_deploy9894671.CITIJPCJHUATWEBENGJP.jutpcsjcmsw1.to.juodgcgjfpvw1.log]
LIB: 2014-03-03 09:26:57 Locale=[English_UnitedStates.US-ASCII@Binary]
ENG: Thread-20 exiting setup
ENG: Thread-20 entering awaiting-cleanup
ENG: Thread-20 still waiting for 0
ENG: previousPhaseStr:setup
ENG: currentPhaseStr:awaiting-cleanup
ENG: Thread-20 exiting awaiting-cleanup
ENG: Thread-20 entering cleanup
ENG: Thread-20 exiting cleanup
ENG: Thread-20 entering task-failed
ENG: Thread-20 still waiting for 1

-------------------- 2nd error Details ------------------------------

ENG: Thread-1192 Sender Side timeout = 0
ENG: Thread-1192 Buffer Size 8000
ENG: Thread-1192 After Connect RC = 0
ENG: Thread-1192 Sender-side handshake timeout = 120 secs
ENG: Thread-1192 Cipher used: DHE-RSA-AES256-SHA SSLv3 Enc=AES(256)
ENG: Thread-1192 Send version string to receiver, RC = 13
ENG: Thread-1192 ***ERROR - Send attributes to receiver got RC = -1
ENG: Thread-1192 exiting setup
ENG: Thread-1192 entering awaiting-cleanup
ENG: Thread-1192 still waiting for 3
ENG: previousPhaseStr:setup
ENG: currentPhaseStr:awaiting-cleanup
ENG: Thread-1192 exiting awaiting-cleanup
ENG: Thread-1192 entering cleanup
ENG: Thread-1192 exiting cleanup
ENG: Thread-1192 entering task-failed
--------------------------------------------------

Find more posts tagged with

Comments

Rick Poulin

Looks like a network problem more than an OD problem... You should check (or ask your network admins) if there's network security scanners that might be interfering, or perhaps a misconfigured proxy, unstable connectivity, virus scans on the servers, misconfigured load balancer, etc.

What I've had to do in the past is set up a small script that telnets to the receivers on 20014 every few minutes, tracking the results in a log file. Helps isolate the time frames for failures, takes OD out of the equation, and it gives you something to send to the network guys.

Migrateduser

I had this exact same problem after upgrading to OD 7.3.2. According to support it's a known issue with OD 7.3.2 - and there's a patch. Why they don't make it known that there's a patch available for a known issue is maddening, but that's another story. Anyhoo - maybe there's a patch for 7.2 as well. Here's the response from my support case:

What you are seeing is a known issue with OD 732 on Linux. Fix is as follows: File Descriptors leak for Linux in libiwdeploy_hopi.so so... Attached is the patch and README instuctions: README For OD 7.3.2 Linux\QSU\OD-2078 Applying the QSU: 1. stop OpenDeploy server 2. Navigate to /od-home/jre/bin 3. Backup the existing "libiwdeploy_hopi.so" to other location. 4. Copy over the new "libiwdeploy_hopi.so" to /od-home/jre/bin 5. Run "../../install/iwcfgbin libiwdeploy_hopi.so /od-home/" 6. Adjust the owner and permission of the new files to match the orginal files. 7. start OpenDeploy server File attached to case: C__a_Troubleshooting_OD_QSU_FileDescriptorLeak_FileDescriptorLeak_OD-2078-fix.zip

Rick Poulin

News to me.. what's the problem that gets fixed by this? The native lib doesn't properly interact with the sockets or something like that?

Migrateduser

I'm not entirely sure what the main cause is - John (from IW Support) seemed to recognize the problem/solution based on the error messages we were seeing. Started happening as soon as we upgraded to 7.3.2 - twice intermittently the base server crashed. This was in production - we haven't seen it happen in our Dev or QA servers, but there are also not as many deployments happening in those environments. The patch seems to have fixed it - we haven't seen the problem since.

VivekPolaris

Thank you very much to everyone for replying back .

I got in touch with Autonomy Suppport for this Case.
They informed that my base OD version is 7.2 and Taget receiver version is 7.3.1
Which is not compatible to each other.

We need to uninstall the 7.3.1 from receiver and install the OD 7.2 there to match the compatiblity.

This solution is quite difficult and time consuming for us as number of target servers are quite high.

Any other suggestions regading this will be very helpful for us.

Thanks again.

Migrateduser

If that's the solution, then that's the solution. You should have considered that when you installed 7.3.1 on your receivers. Why in the world would you upgrade your receivers and not your base?

Adam Stoller

If that's the solution, then that's the solution. You should have considered that when you installed 7.3.1 on your receivers. Why in the world would you upgrade your receivers and not your base?

Theoretically speaking - it should have worked; at least in the "good ol' days" the OpenDeploy developers spent a fair amount of time making sure that OpenDeploy could handle both up-rev and down-rev deployments.

Technically speaking, it generally makes sense to upgrade the Base first, followed (shortly) by the Receivers. If you're in the midst of a "fresh install upgrade" where you have new TeamSite servers with new versions of the software and old TeamSite servers with old versions of the software, you should hold off upgrading the receivers until you've cut over to the new TeamSite servers - to avoid the up-rev deployments as much as possible.