nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

servletd shows 100% CPU utilization and crashes

JoshProx

I did an upgrade to TeamSite 6.7.2 from TeamSite 6.7.1 on windows 2003.
I noticed that servletd shows 100% CPU utilization when I click on the workflow tab. It also crashes the server and I need to restart TeamSite services. Has anyone encountered similar issue before?

Find more posts tagged with

Comments

JoshProx

This is what I see in servletd logs -

2010-02-15 17:08:10,514 [http-8080-Processor24] ERROR com.interwoven.ui.teamsite.access.UserPreferencesImpl () - Error saving 2 properties (10 byte) to EDB: Transport sequencing error

Root cause:
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$CharsetSD.decode(StringCoding.java:183)
at java.lang.StringCoding.decode(StringCoding.java:228)
at java.lang.String.(String.java:405)
at java.lang.String.(String.java:433)
at com.interwoven.hopi.HopiBuffer.getText(HopiBuffer.java:642)
at com.interwoven.sci.sci_small_string_s.hopiDecode_(sci_small_string_s.java:35)
at com.interwoven.sci.sci_small_string_list_s.hopiDecode_(sci_small_string_list_s.java:58)
at com.interwoven.sci.sci_workflow_event_s.hopiDecode_(sci_workflow_event_s.java:86)
at com.interwoven.sci.sci_workflow_event_list_s.hopiDecode_(sci_workflow_event_list_s.java:58)
at com.interwoven.sci.sci_.GetWorkflowEvents(sci_.java)
at com.interwoven.cssdk.sci.workflow.WorkflowServiceSciImpl.getEvents(WorkflowServiceSciImpl.java:1272)
at com.interwoven.cssdk.workflow.java.WorkflowServiceAdapterImpl.getEvents(WorkflowServiceAdapterImpl.java:110)
at com.interwoven.cssdk.workflow.impl.CSWorkflowImpl.getEvents(CSWorkflowImpl.java:344)
at com.interwoven.ui.teamsite.workflow.job.WFHistoryBuilder.buildHistory(WFHistoryBuilder.java:392)
at com.interwoven.ccpro.workflow.job_005fdetails_jsp._jspService(job_005fdetails_jsp.java:475)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:403)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
at com.interwoven.ui.base.impl.command.CommandServlet.doGet(CommandServlet.java:211)
at com.interwoven.ui.base.impl.command.CommandServlet.doPost(CommandServlet.java:263)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at com.interwoven.ui.base.impl.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:203)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)

Migrateduser

Does it only occur when you login and click the tab, or does it do the same for others?

JoshProx

Its an issue with the workflow tab for all users including me.

Adam Stoller

If you go to a shell / command prompt on the server, and run the command iwgetwfobj (located in IWHOME/bin/) - what do you see?

JoshProx

It returns the xml showing workflow and task ids.

Adam Stoller

It returns the xml showing workflow and task ids.

Well, that would seem to rule-out a corrupted workflow store.
You should probably contact Support.

JoshProx

I analyzed and found out that this issue is seen when workflow store has large size workflow instance data files. The problem seem to be resolved by deleting the current workflow store which in turn removed all those instances. Has anyone seen similar issue before when servletd crashed because of large size workflow instance data files. Let me know your inputs on this.

Adam Stoller

I analyzed and found out that this issue is seen when workflow store has large size workflow instance data files. The problem seem to be resolved by deleting the current workflow store which in turn removed all those instances. Has anyone seen similar issue before when servletd crashed because of large size workflow instance data files. Let me know your inputs on this.

Well - there's a general performance issue known to happen when you have a large number of workflows + tasks in the system. I forget the number - something like 2500 or maybe 25000.

It's not enough to count active jobs, you have to also count all the tasks in all the jobs (essentially what you'd get if you ran iwgetwfobj -r on every active job) to get the actual count.

As has been pointed out many times, stagnant jobs tend to cause problems, which is why all workflows should make use of timeouts on tasks to send email to someone to let them know the task is tardy, and then re-activate that task.

nipper

Yes but it does provide comic relief. One of my all-time favorite threads was the guy who casuallyy asked if his system's poor performance could be due to the 25000 active/hung jobs.

JoshProx

In my case it was only a few jobs (<50) with hung default submit workflow. The only issue was large filelist in some of these jobs. I don't understand why it will make it crash.