Teamsite crashes with Bad read (Internal transport error)

We recently upgraded to Teamsite 7.4.1 (Linux) from Teamsite 6.7.2 (Windows).

Teamsite crashes frequently with the below error message. I have raised a case with HP Autonomy and in the meanwhile would like to check if anyone else has seen this error before. Let me know if you want me to upload other log files.

content_center.log:

2014-07-29 12:04:07,209 [http-0.0.0.0-8080-20] ERROR com.interwoven.cssdk.sci.filesys.FileServiceRootSciImpl () - Error returned back is 5000006
2014-07-29 12:04:07,281 [http-0.0.0.0-8080-20] ERROR com.interwoven.ui.teamsite.filesys.context.CSNodeContext () - unknown error
com.interwoven.cssdk.common.CSException: Bad read(Internal transport error)

at com.interwoven.cssdk.sci.exception.ExceptionHandler.handleExceptionSci(ExceptionHandler.java:264)
   at com.interwoven.cssdk.sci.exception.ExceptionHandler.handleException(ExceptionHandler.java:104)
   at com.interwoven.cssdk.sci.filesys.FileServiceRootSciImpl.getFilesByVPaths(FileServiceRootSciImpl.java:437)
   at com.interwoven.cssdk.sci.filesys.FileServiceSciImpl.getFilesByVPaths(FileServiceSciImpl.java:2127)
   at com.interwoven.cssdk.filesys.java.FileServiceAdapterImpl.getFilesByVPaths(FileServiceAdapterImpl.java:315)
   at com.interwoven.cssdk.filesys.impl.CSRootImpl.getFileNodes(CSRootImpl.java:364)
   at com.interwoven.cssdk.filesys.impl.CSRootImpl.lookupByVPaths(CSRootImpl.java:291)
   at com.interwoven.ui.teamsite.filesys.context.CSNodeContext.loadNodes(CSNodeContext.java:87)
   at com.interwoven.ui.teamsite.filesys.context.CSNodeContext.getNodes(CSNodeContext.java:171)
   at com.interwoven.ui.teamsite.filesys.context.CurrentNodeContext.getCurrentNode(CurrentNodeContext.java:59)
   at com.interwoven.ui.teamsite.filesys.FileChooserCommand.splitPath(FileChooserCommand.java:151)
   at com.interwoven.ui.teamsite.filesys.FileChooserCommand.execute(FileChooserCommand.java:106)
   at com.interwoven.ui.base.impl.command.CommandHandler.doExecuteCommand(CommandHandler.java:1041)
   at com.interwoven.ui.base.impl.command.CommandHandler.tryRunCommandDescriptor(CommandHandler.java:864)
   at com.interwoven.ui.base.impl.command.CommandHandler.tryRunCommandID(CommandHandler.java:738)
   at com.interwoven.ui.base.impl.command.CommandHandler.runCommandLoop(CommandHandler.java:330)
   at com.interwoven.ui.base.impl.command.CommandServlet.doGet(CommandServlet.java:163)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at com.interwoven.livesite.spring.hibernate.web.filter.OpenSessionsInViewFilter.doFilterInternal(OpenSessionsInViewFilter.java:108)
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
   at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
   at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at com.interwoven.ui.base.impl.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:203)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at com.interwoven.ui.base.util.SetRequestEncodingFilter.doFilter(SetRequestEncodingFilter.java:106)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
   at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
   at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
   at java.lang.Thread.run(Thread.java:619)

Find more posts tagged with

Comments

Adam Stoller

jagandrk wrote:
We recently upgraded to Teamsite 7.4.1 (Linux) from Teamsite 6.7.2 (Windows).

When you say that - you're not talking about taking the actual backing store from the Windows server and putting it on the Linux server - right?

I just want to make sure - because while I don't know the exact failure messages that would raise, that "concept" has never been supported.

Rick Poulin

Is there anything relevant in TeamSite/local/logs/iwserver.log and/or iwtrace.log? What's the output of iwstat when this happens?

This error means there was a miscommunication between the Java UI and the iwserver.exe process. It's not uncommon during very high server load, but it shouldn't happen often unless there's a problem with your content store.

jagandrk

rpoulin -

You are correct! iwserver process hangs when this error is thrown and only way to bring the service back up is to kill -9 iwserver process and starting it again.

There is relatively high load than usual on the server as users are trying to update content after upgrade and some curious users to see what has changed in new Teamsite. I will run iwfsck today evening and check the output.

ghoti -

We used opendeploy <targetTeamsite with applyExtAttrs="yes" to migrate content from windows to linux. We didn't have any issues in pre-production user testing and load testing.

Adam Stoller

jagandrk wrote:
ghoti -
We used opendeploy <targetTeamsite with applyExtAttrs="yes" to migrate content from windows to linux. We didn't have any issues in pre-production user testing and load testing.

Okay - I just wanted to make sure, since it wasn't clear from your original posting. Glad @rpoulin was able to point you in the right direction.

swati_27

We recently performed store migration from Production to UAT and facing the same error messages in server.log.

This is an intermittent behavior due to which TeamSite UI becomes unavailable for sometime however it comes back again. We are facing this only after store migration. Does it mean that the store is corrupted?

We are able to see/update the latest store content on UAT. Any pointers?

Adam Stoller

swati_27 wrote:
This is an intermittent behavior due to which TeamSite UI becomes unavailable for sometime however it comes back again. We are facing this only after store migration. Does it mean that the store is corrupted?

When in doubt - stop (or iwfreeze) TeamSite and run iwfsck -d on the backing store to determine if the backing store is corrupted.

If problems are found with the backing store - contact HP Support.

jagandrk

An update on this issue, iwfsck output shows no errors in the content store. Then we tried couple of fixes.

1, Updated uvfs module to latest version 2.0.7 as read me file comment says "Fixed performance issue." - No luck here server still crashes with the same error.

2, Decreased cachesize value - again no luck

After sending a number of diagnostic output to HP Autonomy support, they have sent us a patch (iwserver.linux file). We installed the patch on the server and waiting to see whether it copes with the load tomorrow morning. I've asked for more information about the root cause from HP autonomy support to understand why we didn't get this error during load testing in pre-production.

swati_27 - In our case, we have to restart iw.server manually everytime server crashes following this error.

swati_27

Thanks Adam, Jagan for your reply.

I tried performing the store check using iwfsck -d and there were no issues found with the store.

In the server.log we are receiving the attached exceptions intermittently. On hitting the Content Center URL , CCPro logo and the complete header appears but the store content takes time to display. All the directories and files are loaded in like 1 minute or more this operation is taking longer time than expected.

Opening a support ticket now.

Jagan - Let me know if you also found the first error in your server.log?

jagandrk

After applying iwserver.linux patch provided by HP Autonomy support, Teamsite no longer crashes. Apparently, Teamsite was crashing due to mutex deadlock while trying to read user/group cache.

Swati - No, we didn't get this error in logs.

We had similar performance issues when our System Admins moved content store to slow disk.