nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

TeamSite Performance issues in 7.1

chnkumar

Hi,

Sometimes the CPU utilization in TeamSite server goes to 100% and we can't do anything with TeamSite at that time. No other services and softwares running on the TS Server. I have noticed some of the entries in access.log and iwevents.log during the peak times.

OS: Solaris

I see below line 20 to 30 times from the same IP in access.log.

10.196.48.17 - - [01/Apr/2015:06:06:17 -0700] "POST /iw/services/cm/2.0/fileservice HTTP/1.0" 500 761 0 -

I see below entries in iwevents.log

[Wed Apr 1 06:06:17 2015]   daron.herve   Editor   Lock   /default/main/TA/CLUBS/FS/WORKAREA/main   /iwov-resources/images/ATL/photos/2015PHOTOS/DRAFT/150331_DavidCobb_RPF_CP_logo.jpg   0x00001008551becee600c24ee   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:18 2015]   daron.herve   Editor   Lock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_thumb_130_80.jpg   0x0000100854f7f5ac60001203   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:18 2015]   daron.herve   Editor   UnLock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_thumb_130_80.jpg   0x0000100854f7f5ac60001203   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:18 2015]   daron.herve   Editor   UnLock   /default/main/TA/CLUBS/FS/WORKAREA/main   /iwov-resources/images/ATL/photos/2015PHOTOS/DRAFT/150331_DavidCobb_RPF_CP_logo.jpg   0x00001008551becee600c24ee   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:19 2015]   daron.herve   Editor   Lock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_mezz_1280_1024.jpg   0x0000100854f7f5ae60001205   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:20 2015]   daron.herve   Editor   UnLock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_mezz_1280_1024.jpg   0x0000100854f7f5ae60001205   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:21 2015]   daron.herve   Editor   Lock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_small_280_187.jpg   0x0000100854f7f5b060001207   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:21 2015]   daron.herve   Editor   UnLock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_small_280_187.jpg   0x0000100854f7f5b060001207   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:23 2015]   daron.herve   Editor   Lock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_medium_540_360.jpg   0x0000100854f7f5b160001209   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:23 2015]   daron.herve   Editor   UnLock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_medium_540_360.jpg   0x0000100854f7f5b160001209   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:24 2015]   daron.herve   Editor   Lock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_large_580_1000.jpg   0x0000100854f7f5b360001210   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:24 2015]   daron.herve   Editor   UnLock   /default/main/TA/CLUBS/SS/WORKAREA/main   /iwov-resources/images/LAX/01_media--ta_large_580_1000.jpg   0x0000100854f7f5b360001210   daron.herve   Locked by CMS-Web Util
[Wed Apr 1 06:06:34 2015]   daron.herve   Admin   Lock   /default/main/TA/CLUBS/FS/WORKAREA/main   /templatedata/Blog/ARTICLE/data/2015/04-April/0401_DraftSpotlight_RF   0x00001008551ac2c4600c088c   daron.herve   File locked by FormsPublisher.
[Wed Apr 1 06:06:40 2015]   daron.herve   Unknown   TaskAddFile   Publish LiveSite Content   0x1b2e9b   Mark orphan html for deletion   0x1b2ea7   templatedata/Blog/ARTICLE/data/2015/04-April/0401_DraftSpotlight_RF   /default/main/TA/CLUBS/FS/WORKAREA/main
[Wed Apr 1 06:06:40 2015]   daron.herve   Unknown   WorkflowActivate   workflow   Publish LiveSite Content   0x1b2e9b
[Wed Apr 1 06:06:40 2015]   daron.herve   Unknown   TaskActivate   Publish LiveSite Content   0x1b2e9b   Mark orphan html for deletion   0x1b2ea7   /default/main/TA/CLUBS/FS/WORKAREA/main
[Wed Apr 1 06:06:40 2015]   daron.herve   Unknown   TaskGoActive   Publish LiveSite Content   0x1b2e9b   Mark orphan html for deletion   0x1b2ea7   /default/main/TA/CLUBS/FS/WORKAREA/main

Find more posts tagged with

Comments

nipper

Why are you locking/unlocking all the files?

Locked by CMS-Web Util

Is that during a workflow?

You should check with support, I know there were some performance issues with 7.1 (which BTW, is not supported anymore).

Also, when this happens you should run iwstat and start collecting information on what is happening.

chnkumar

Hi Andy,

Yes, it is during the Workflow. (we are not locking/unlocing the files).

I don't find the exact cause from iwstat. By the way, my store is 1TB size and 600GB is available space.

Do you have any idea when below soap service is called?

POST /iw/services/cm/2.0/fileservice

nipper

The locks are likely due to the WF tasks having lock='t" set. Try changing that.

Not certain if the locking is causing the slowness or not, but worth a shot.

nipper

You should also see if you can shrink the store, use

iwfsshrink run

And just let it run for a while. check with

iwfsshrink status

Snap 2017-02-24 at 08.58.32.png

Snap 2017-02-24 at 08.59.52.png

Snap 2017-02-24 at 09.01.08.png

Snap 2017-02-24 at 09.01.56.png

Grouping2Columns.zip

rayinsight

I would suggest not to use iwfsshrink on any TeamSite 7.x version before TS 7.4.1. I cant find the KB article but it was reported to cause store corruptions. Yes, its also correct TS 7.1 is no longer supported unless the customer asks for extended support - though I m not correct person to confirm or validate on the support extension, an account manager/sales rep will be the correct path to follow.

Coming back to your problem, some preliminary diagnostic steps that you can do:

1. So, the CPU seems to be at 100% usage, what about the memory usage of the server - is it also above above the usual limits?

2. Which process takes up most of the CPU usage - is the teamsite server process, or the servletd process or any other process that is taking up most of the CPU? Probably task manager or Process Monitor can be of help here.

3. If the teamsite server is indeed taking up most of the CPU, then I think in most cases we should be able to use iwstat -c -i 5 (run in a loop of 5 secs) to see what iwserver process is doing. Are there any threads that is not exiting and stays for a long time.

Other questions include is whether this problem started recently, does the problem start after workflows are executed - is it any workflow or a very specific workflow that causes the spike.

Thanks,

Akash

HP Support

chnkumar

We run iwfsshrink qurtarly, the last run was in Dec and it didn't gain much space.

Yes, RAM also and iwserver takes up most of the CPU usage. We have a thridy party monitoring tool which calls iwstat and shows the performance in graph.

Not sure how to know what iwserver is doing (logs??, if so, I have already posted some portion of iwevents & access.log)

As per my observation, when more workflows are initiated at the same time, we see that spike. (Of-course we have 65GB RAM)

I still don't understand, why below soap service was called many times from same ip when TS goes peaks state.

POST /iw/services/cm/2.0/fileservice

We have 300 to 400 (non local users) users in TS. I have seen in some of the posts that, if non local users (LDAP users) were not cached, TS always seraches for them, if any actions are done by those users.

I am not sure where to see TS cache users in Solaris.

Please advise.

Rick Poulin

You might want to increase the delay between calls from your third-party monitoring tool, if it's pinging really often. As mentioned earlier, this call does have a performance hit.

iwserver logs to iwserver.log and iwtrace.log, but it only logs errors, not activity.

Your 65GB of RAM is mostly useless. JBoss (the service that handles workflow execution) can't be allocated much more than 2GB of RAM until 7.5+

AFAIK, nothing in the IWOV product stack uses the CSSDK SOAP API. If something is calling that service, then it's custom to your implementation.

Have you identified which service is running hot? Is it JBoss or is it iwserver?

chnkumar

Hi Rpoulin,

Sorry for dealy in my response. Last friday, we have seen the spike reached 100% in disk busy graph and during that time CPU utilization was 35%. We are usning zfs mount definition. We have checked the iwevents.log during that time and found 25 Wfs were initiated in 2 mins. I saw some events by iwui user and it is not a TeamSite User.

Does TS do a lookup for this user always when any events are done by this user?

[Thu Apr 9 13:00:05 2015]   iwui   master   TaskCalledBack   Publish LiveSite Content   0x22fef8
[Thu Apr 9 13:00:05 2015]   iwui   master   ChooseTransition   Publish LiveSite Content   0x22fef8
[Thu Apr 9 13:00:05 2015]   iwui   master   TaskActivate   Publish LiveSite Content   0x22fef8
[Thu Apr 9 13:00:05 2015]   iwui   master   TaskInactivate   Publish LiveSite Content   0x22fef8
[Thu Apr 9 13:00:05 2015]   iwui   master   TaskGoActive   Publish LiveSite Content   0x22fef8

Rick Poulin

iwui is a system account that runs JBoss, and some system tasks in the PLC, so that's not unusual.

What's unusual is that something launched 25 workflows in 2 minutes. That seems excessive. If that's a repeatable issue, then you've got broken custom code somewhere.

That said, you didn't really answer the questions, so I'll ask again -- which process is running hot (either against the processor or the disk)?

chnkumar

I am working on collecting all those details (which process is running hot (either against the processor or the disk)?). Please give me sometime