nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Teamsite 6.7.2 Performance Issues

parsan99

We are on Teamsite 6.7.2 SP2 (RedHat Linux) and have been facing response issues for the past day on our production system. UI stalls completely for a few seconds and then slowly recovers in a minute or so. And when the users hit reload during these bouts of slowness, it makes things worse. We have looked into the logs - iwtrace.log, iwevents.log, servletd_out.log and have not been able to establish any links to the slowness. CPU and memory usages are well within the norm. We have also tried restarting Teamsite. When the server had a slow response time, I tried iwstat -c and noticed entries such as the following. The wierd part is, I (testuser) have not used the given workarea in a few weeks and I dont believe there are any scripts doing so either. My colleague has also noticed such entries for his login. Any clues if this is contributing to the issue?

0x4073a1 0xc37ffb90 testuser 114.989 default LookupObjids /default/main/HTML/site.com/WORKAREA/site.com

0x4072df 0xe0ceab90 root 214.424 default vfs_create .DS_Store.iwtmp (64,60000005) 5d8bc9/5d8aef

I am noticing the following messages in iwtrace.log

[Tue Jun 7 05:11:27 2011] [default][33] SNAPSHOT[E98F8B90]: enforced, on timeout.
[Tue Jun 7 05:54:23 2011] [default][5] SNAPSHOT[E98F8B90]: enforced, on timeout.
[Tue Jun 7 06:19:26 2011] [default][5] SNAPSHOT[E98F8B90]: enforced, on timeout.

We greatly appreciate any input on this.

Find more posts tagged with

Comments

santosh013

We are on Teamsite 6.7.2 SP2 (RedHat Linux) and have been facing response issues for the past day on our production system. UI stalls completely for a few seconds and then slowly recovers in a minute or so. And when the users hit reload during these bouts of slowness, it makes things worse. We have looked into the logs - iwtrace.log, iwevents.log, servletd_out.log and have not been able to establish any links to the slowness. CPU and memory usages are well within the norm. We have also tried restarting Teamsite. When the server had a slow response time, I tried iwstat -c and noticed entries such as the following. The wierd part is, I (testuser) have not used the given workarea in a few weeks and I dont believe there are any scripts doing so either. My colleague has also noticed such entries for his login. Any clues if this is contributing to the issue?

0x4073a1 0xc37ffb90 testuser 114.989 default LookupObjids /default/main/HTML/site.com/WORKAREA/site.com

0x4072df 0xe0ceab90 root 214.424 default vfs_create .DS_Store.iwtmp (64,60000005) 5d8bc9/5d8aef

I am noticing the following messages in iwtrace.log

[Tue Jun 7 05:11:27 2011] [default][33] SNAPSHOT[E98F8B90]: enforced, on timeout.
[Tue Jun 7 05:54:23 2011] [default][5] SNAPSHOT[E98F8B90]: enforced, on timeout.
[Tue Jun 7 06:19:26 2011] [default][5] SNAPSHOT[E98F8B90]: enforced, on timeout.

We greatly appreciate any input on this.

Hi,
Few pointers:
Have you tried restarting your server?
Is there any other application/applications running on the Server?

Thanks.

Migrateduser

we are having the exact same issues - some of our requests when running iwstat take 2-3 minutes to complete? we have rebooted the box/services/etc - did you find a legitimate solution?

ID Thread User Duration Store Operation 0xdeedd9a 0x610 SYSTEM 0.000 - GetServerStatus 0xdee51d4 0x234 ADMIN\sshandor 87.027 iwadmin LookupObjids //CSCMSADM01/iwadmin/main/workflowModels/STAGING 0xdee1d06 0x2468 ADMIN\ebarker 123.526 iwadmin LookupObjids /iwadmin/main/workflowModels/WORKAREA/iw-wa/Config 0xdee00bd 0x19a8 ADMIN\ebarker 129.557 iwadmin LookupObjids /iwadmin/main/workflowModels/WORKAREA/iw-wa/Config 0xdee0058 0x16e8 ADMIN\ebarker 133.447 iwadmin LookupObjids /iwadmin/main/workflowModels/WORKAREA/iw-wa/Config 0xdedffc5 0x604 ADMIN\ebarker 141.150 iwadmin LookupObjids /iwadmin/main/workflowModels/WORKAREA/iw-wa/Config 0xdedb253 0xf08 SYSTEM 211.553 iwadmin SubmitFSE 0x000021000000000000000154 0xded6910 0x18e0 ADMIN\sshandor 293.237 iwadmin LookupObjids //CSCMSADM01/iwadmin/main/workflowModels/STAGING 0xded1c18 0x19d0 ADMIN\sshandor 327.563 iwadmin LookupObjids //CSCMSADM01/iwadmin/main/workflowModels/STAGING 0xdec250f 0x1a10 SYSTEM 452.089 iwadmin SubmitFSE 0x000021000000000000000154

nipper

we are having the exact same issues - some of our requests when running iwstat take 2-3 minutes to complete? we have rebooted the box/services/etc - did you find a legitimate solution?

ID Thread User Duration Store Operation
0xdeedd9a 0x610 SYSTEM 0.000 - GetServerStatus 0xdee51d4 0x234 ADMIN\sshandor 87.027 iwadmin LookupObjids //CSCMSADM01/iwadmin/main/workflowModels/STAGING
0xdee1d06 0x2468 ADMIN\ebarker 123.526 iwadmin LookupObjids /iwadmin/main/workflowModels/WORKAREA/iw-wa/Config
0xdee00bd 0x19a8 ADMIN\ebarker 129.557 iwadmin LookupObjids /iwadmin/main/workflowModels/WORKAREA/iw-wa/Config
0xdee0058 0x16e8 ADMIN\ebarker 133.447 iwadmin LookupObjids /iwadmin/main/workflowModels/WORKAREA/iw-wa/Config
0xdedffc5 0x604 ADMIN\ebarker 141.150 iwadmin LookupObjids /iwadmin/main/workflowModels/WORKAREA/iw-wa/Config
0xdedb253 0xf08 SYSTEM 211.553 iwadmin SubmitFSE 0x000021000000000000000154
0xded6910 0x18e0 ADMIN\sshandor 293.237 iwadmin LookupObjids //CSCMSADM01/iwadmin/main/workflowModels/STAGING
0xded1c18 0x19d0 ADMIN\sshandor 327.563 iwadmin LookupObjids //CSCMSADM01/iwadmin/main/workflowModels/STAGING
0xdec250f 0x1a10 SYSTEM 452.089 iwadmin SubmitFSE 0x000021000000000000000154

Fixing post.

nipper

Does this happen only with WFM or with all things ?

Have you run iwfsshrink lately ? How about iwfsck ?

What rev of WFM ?

Migrateduser

thanks for fixing the formatting - this happens with all browser based activity in CC pro (switching tabs, etc). like the person who started the post said, its intermittent but when it hits its brutal and really drives users crazy.

Our sysadmin ran both CLTs you mentioned last week, with no benefit.

nipper

OK, so no backingstore errors, that is a good thing.

Can you abort the tasks ?

iwabort

where is the hex # in the 1st column of iwstat 0xdee1d06

How about other submit tasks, or is this only WFM related ?

Migrateduser

I havent tried aborting the tasks as they are valid requests of our users trying to submit workflows or simply browse the GUI. what is accomplished by aborting them? wont another rampant thread just start up? even after rebooting the box the issue still occurs.

we noticed a bunch of these errors in the servletd_out.log which i have NEVER seen before in all my IWOV years...

2012-01-26 06:28:53,540 [Clock Daemon] ERROR org.exolab.jms.util.Clock () - Internal error in clock daemon
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.exolab.jms.util.Clock.run(Unknown Source)
2012-01-26 06:28:53,712 [main] INFO com.interwoven.ui.teamsite.admin.JvmStatLogging () - JvmStatLogging shutting down scheduled logging.
2012-01-26 06:28:53,712 [main] INFO com.interwoven.ui.teamsite.admin.JvmStatLoggingContextListener () - JvmStatLogging destroyed.
2012-01-26 06:28:55,009 [main] ERROR com.interwoven.serverutils100.InstalledLocations () - UtildClientException when calling Utility Daemon in refresh InstalledLocation: encounter an error when borrowing object from pool
com.interwoven.serverutils100.utild.UtildClientException: encounter an error when borrowing object from pool
at com.interwoven.serverutils100.utild.UtildClient.borrowObject(UtildClient.java:90)
at com.interwoven.serverutils100.utild.UtildClient.getServerProperty(UtildClient.java:605)
at com.interwoven.serverutils100.InstalledLocations.reset(InstalledLocations.java:168)
at com.interwoven.serverutils100.InstalledLocations.refresh(InstalledLocations.java:153)
at com.interwoven.serverutils100.InstalledLocations.getInstance(InstalledLocations.java:137)
at com.interwoven.ui.vannotate.storage.ea.TAnnotatedFileAdapterEA.(TAnnotatedFileAdapterEA.java:81)
at sun.misc.Unsafe.ensureClassInitialized(Native Method)
at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:25)
at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:122)
at java.lang.reflect.Field.acquireFieldAccessor(Field.java:917)
at java.lang.reflect.Field.getFieldAccessor(Field.java:898)
at java.lang.reflect.Field.get(Field.java:357)
at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:1641)
at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1524)
at org.apache.catalina.loader.WebappLoader.stop(WebappLoader.java:707)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4557)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:448)
at org.apache.catalina.core.StandardService.stop(StandardService.java:584)
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:744)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:628)
at org.apache.catalina.startup.Catalina.start(Catalina.java:603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Caused by: java.lang.IllegalStateException: Pool not open
at org.apache.commons.pool.BaseObjectPool.assertOpen(BaseObjectPool.java:78)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:781)
at com.interwoven.serverutils100.utild.UtildClient.borrowObject(UtildClient.java:84)
... 28 more

-----------------

2012-01-26 08:43:58,925 [Thread-9] ERROR com.interwoven.modeler.jmsclient.JobFinishedListener () - Unable to delete the instantance model
com.interwoven.cssdk.common.CSException: (Internal transport error)
at com.interwoven.cssdk.sci.exception.ExceptionHandler.handleExceptionSci(ExceptionHandler.java:258)
at com.interwoven.cssdk.sci.exception.ExceptionHandler.handleException(ExceptionHandler.java:102)
at com.interwoven.cssdk.sci.filesys.CSPNOHelper.locateObjectWithType(CSPNOHelper.java:2112)
at com.interwoven.cssdk.sci.filesys.CSWorkarea.getWorkareaData(CSWorkarea.java:929)
at com.interwoven.cssdk.sci.filesys.FileServiceWorkareaSciImpl.getWorkareaData(FileServiceWorkareaSciImpl.java:311)
at com.interwoven.cssdk.sci.filesys.FileServiceSciImpl.getWorkareaData(FileServiceSciImpl.java:3215)
at com.interwoven.cssdk.filesys.java.FileServiceAdapterImpl.getWorkareaData(FileServiceAdapterImpl.java:442)
at com.interwoven.cssdk.filesys.impl.CSWorkareaImpl.refreshData(CSWorkareaImpl.java:123)
at com.interwoven.cssdk.common.impl.CSObject.refresh(CSObject.java:121)
at com.interwoven.cssdk.common.impl.CSObject.forceValid(CSObject.java:98)
at com.interwoven.cssdk.common.impl.CSClientImpl.getWorkarea(CSClientImpl.java:344)
at com.interwoven.modeler.jmsclient.JobFinishedListener.onMessage(JobFinishedListener.java:125)
at org.exolab.jms.client.JmsMessageConsumer.onMessage(Unknown Source)
at org.exolab.jms.client.JmsSession.execute(Unknown Source)
at org.exolab.jms.client.JmsSession.onMessage(Unknown Source)
at org.exolab.jms.client.mipc.IpcJmsMessageListener.onMessage(Unknown Source)
at org.exolab.jms.client.mipc.IpcJmsMessageListener.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

Kapil_sape

Hi,

Can you try to clear up IIS Cache and then do IISRESET.
Once done please recheck the TS performance.

Thanks.

Migrateduser

We have been having the same issue for the last 3 months. We have tried support, professional services and free lancers look at the problem.

Some of the things we have tried that does help the system is
> Make sure the servletd.log file doesnt grow big
> try configuring the cached objects to around 150K objects in iw.cfg
> Restart IW services

-- If those dont help other clean up you can look at are
> Look for zombie files and delete them (mainly in the work flow folders or iwadmin)
> Delete any open unused work flows.

Even after all that, we have to hand paddle the TS servers. Utlimately we are trying to move to 64-bit windows (for more memory) and upgrading to TS 7.2.1 in the next quarter.

Good luck - let us know if you come across any other solutions.

WaterfallGraph.zip

parsan99

Just so you know, ours turned out to be an underlying hardware issue. Once we fixed that, everything was back to normal again. But it would have certainly helped to have better error reporting.

GroupingMultiColumns.zip

Migrateduser

our issue was not related to hardware, we had "zombie" files not be cleaned up properly by TeamSite.

//CSCMSADM01/iwadmin/main/workflowModels/WORKAREA/iw-wa/Instance/ModelInstance/*.ipm

we had over 2500+ instance files, so we wrote a script to clean them out every 48 hours. Our shady sales rep said this was a "known issue".