Intermittent Slow Performance

TS 6.7.2 with OD 6.1.0
Platform: 32 bit Windows Server 2003 Enterprise SP1 with Physical Address Extension
Hardware: Intel Xeon X5355 @ 2.66 Ghz, 32 GB RAM (artificially limited to 4 GB right now so we can take kernel/user memory dumps)

Our issue is that we have slowness where the UI will seem to stop, and mostly, everything seems to stop. Sometimes, it is so bad, you cannot open "My Computer" on the server. It sometimes lasts 10 minutes, but then will go away with no action from us. All my activity on the server (clicking windows, etc). will seemingly be queued up and then processed when the problem goes away.

One symptom that usually coincides is that "iwstat -c -i 5" will be disconnected with an "ERROR"5000059". I am also watching the count of TIME_WAIT for port 8898 (iwserver), but don't really see any huge looking numbers.

On the advice of support, we have lowered the TCP TimedwaitDelay to 30 seconds.

The one memory dump we have sent to MS resulted in MS pointing the finger at iwfsd.sys. They wanted to disable/remove it. I chuckled. We're waiting on the results from another memory dump we took.

Has anybody experienced anything like this? We are at a loss, and IW/Autonomy support seems to be, too.

Ideas are welcomed.

Craig

Find more posts tagged with

Comments

abiondo

Hi Craig:

I want to ask a couple questions to see if it helps.

Are you seeing the UI Freeze when you access TeamSite on the Server? From a Client or Both?
Do you know if your system is running Symantec AV or any other tools like Intrusion Detection monitoring on the server?
Is this with one user accessing the system (you) or with a full load?

Hopefully these will get some other people thinking about this question.

thanks,

Anthony

Migrateduser

Hmmm, I have not tried to access the UI from the server. I'll start checking that. Thanks.

We do run Symantec AV, that has been suspected as part of the issue, but so far no proof has surfaced saying that is related to our issues.

It happens with roughly 100 active users, but there are claims that it has happened with 1-5 users, but I can't confirm that, yet.

Thanks for the help.

Craig

abiondo

I would suggest trying the following. These are all guesses, since I haven't seen the problem myself. They are all simple enough and probably the first couple things I would look at.

Disabling Norton AV. Does the problem go away. I know some other posts suggest excluding Y:\ and the IW_Store.
Doing testing on the server to see if you encounter the issue at the same time your users do. If your users are seeing the issues but you don't see them on the server, you might want to look at networking performance. Are there any errors in the server log.
Running Performance monitoring to look at various aspects of the system.
Also (a longshot). We ran into a problem when we put in Web Sense a while back where it killed the performance of our internal websites. The sites performed fine and then all of a sudden, everything went to a crawl. When you were on the server everything ran fine. We eneded up having to add a command to bypass websense for internal sites. Not sure if this applys, but it is something that matched the behavior.

hope this helps,

Anthony

Migrateduser

Yup, we already exclude our Y: and all store locations from AV scans. No performance monitor attribute, has shed any light on what is going on so far.

We've also been running "iwstat -c" and looking at connections to port 8898 in "TIME_WAIT" state every minute, but neither of those has enlightened us.

It apparently happened yesterday, with only 6-7 users on the system, so that would seem to indicate it may not be load related, but I can't say that for sure.

I'm really struggling with what we SHOULD be looking at. Everything we have been looking at has amounted to nothing.

nipper

We've also been running "iwstat -c" and looking at connections to port 8898 in "TIME_WAIT" state every minute, but neither of those has enlightened us.

This would concern me.

For grins I would check out your networking for the server. Take a look at all host names in iw.cfg (either IP address, system name, localhost, fully qulaified host name, etc). Try pinging everyone, traceroute, etc. They should be immediate.

I have seen cases where there would be a timeout because the route tables were hosed and each connection took 1 minute to even start.

Migrateduser

This would concern me.

Why? From the perspective of adding load to the server?

nipper

Just wondering if there is a networking issue.

I was looking on my system, under load & saw no processes on a TIME_WAIT

Migrateduser

Just wondering if there is a networking issue.

I was looking on my system, under load & saw no processes on a TIME_WAIT

Ahhh....yeah, we lowered the TCPWaitDelay to 30 seconds at the request of support. We sometimes get TIME_WAIT's down to zero, but I have seen them over 1000, but usually hover somewhere closer to 150.

abiondo

Hi Craig:

I agree with Nipper that a networking issue would make sense with the TME_WAIT. Do you have any monitoring tools like HP Openview that can monitor the time it takes to hit the site and return a page and do ping tests of your hosts? If you do you may be able to simulate a login and at least baseline the response time and identify patterns in the slowness.

It would also be cool if you were able to logon the server and try when this happens. This would take the network out of the mix and let us know better if it was a TeamSite Issue or a Network connectivity issue.

Also have you looked through your system log in Windows 2003 to see if there are any TCP/IP or NIC Card failures or warnings?

hope this helps,
Anthony

Migrateduser

Hi Craig:

I agree with Nipper that a networking issue would make sense with the TME_WAIT. Do you have any monitoring tools like HP Openview that can monitor the time it takes to hit the site and return a page and do ping tests of your hosts? If you do you may be able to simulate a login and at least baseline the response time and identify patterns in the slowness.

It would also be cool if you were able to logon the server and try when this happens. This would take the network out of the mix and let us know better if it was a TeamSite Issue or a Network connectivity issue.

Also have you looked through your system log in Windows 2003 to see if there are any TCP/IP or NIC Card failures or warnings?

hope this helps,
Anthony

We have a tool called SiteScope that pulls up the login page. Thus far, no patterns have appeared. Last week, I was able to go onto the server and pull up the interface during the issue, and I'm 95% sure it was slow there, too. However, days are blending for me, so I'll need to verify that.

That last time I looked through the system log, nothing out of the ordinary seemed to be there, but I'll scan it again.

I've got a script going now that runs ping/tracert to external systems like DB and metatagger servers. So, maybe that will show something next time the problem pops up.

nipper

You need to ping/traceroute the server to itself (via IP address, hostname, host+domain, localhost, everything). Make certain you try every one mentioned in iw.cfg.

I saw one case where a server was tied to the wrong DNS server & ping server was fast, but ping server.domain took minutes.

Samir_Friend

1. What is the size of your content store data and the disk(s) it is on?
2. Do the users extensively download static files such as PDFs/Exe(s) etc. stored on Teamsite? What maybe their(static files') approximate memory occupancy?
3. What is your page file limit for the installation drive?
4. You might also want to check the memory logs for your web server and the iwtrace logs for Teamsite server.

RonaldV

I would try to monitor cpu&memory usage AND file usage (using the filemonitor from sysinternals). One of these might give you a hint.

Migrateduser

Our newest tidbit: For the second time Microsoft's memory dump analysis is pointing the finger at iwfsd.sys. This time they added the detail that the driver was not completing an IRP (I/O Request Packet).

Not sure how to tell if this is a symptom or the cause...

Migrateduser

You need to ping/traceroute the server to itself (via IP address, hostname, host+domain, localhost, everything). Make certain you try every one mentioned in iw.cfg.

I saw one case where a server was tied to the wrong DNS server & ping server was fast, but ping server.domain took minutes.

Hmmm...added some of these yesterday to the FQDN and "localhost". So far, nothing. I'll try expanding that as you suggest to see how things look.

Migrateduser

Our newest tidbit: For the second time Microsoft's memory dump analysis is pointing the finger at iwfsd.sys. This time they added the detail that the driver was not completing an IRP (I/O Request Packet).

Not sure how to tell if this is a symptom or the cause...

This was yesterday:

I just witnessed the slow down and have some more information to share.

I’m using the “Process Explorer” app from sysinternals, which is now part of Microsoft.

When the problem just happened, the CPU went to about 15%. The top process was iwserver.exe using 12.5% of the CPU. Using PE, I looked at the threads for iwserver.exe. The top CPU thread was 2696 and it was using all 12.5%. It’s CSwitch Delta (which I assume is context switches?) was around 20. When I clicked on that thread and said “show stack”, I got the following:

ntkrnlpa.exe+0x8cf5e
ntkrnlpa.exe+0x2925c
ntkrnlpa.exe+0x32cee
hal.dll+0x462d
ntdll.dll!RtlFindActivationContextSectionGuid+0x1259
ntdll.dll!RtlFindActivationContextSectionGuid+0x518
ntdll.dll!RtlFindActivationContextSectionGuid+0x1c2
iwserver.exe+0x798dc5
iwserver.exe+0x798de7
iwserver.exe+0x6a2024

When the system was “locked”, it was as stated above. When the system “freed up”, that thread’s CPU dropped to 0, and the CSwitch Delta for the thread went to 0 as well.

I just observed this twice, but I’m not sure if it helps us. Since the observation could still be a symptom, and not the cause.

MS is now looking at the last memory dump again, this time with what I assume are the symbols found in c:\windows\system32\drivers\iwfsd.*

Valentine

Are you using SAN/DAS?

We're experiencing a very similar issue

Migrateduser

Are you using SAN/DAS?

We're experiencing a very similar issue

Nope, all local.

Migrateduser

Latest update:

IW is working on the iwfsd.sys driver. It is apparently, hanging when iwserver.exe doesn't respond fast enough. That's sort of a guess at this point.

We've been providing memory dumps and they've sent us an updated iwfsd.sys driver once already.

Migrateduser

Latest update:

IW says iwfsd.sys is not actually the problem. Instead it waits for iwserver.exe to respond. The question is why???

I've been sending memory dumps of iwserver.exe to them and they feel they are getting close, but I don't have any more detail from them, yet.

Migrateduser

Small update - Autonomy is highly confident the issue is with the AD user/group updates that happen. We are focusing on that, but still no solution.

Migrateduser

Well, this may be coming to an end thanks to a special patch Autonomy has provided us (and apparently three other companies). I think they are calling it patch 9104, and it is only valid on TS 6.7.2 for Windows.

The issue is when the system saves the user and group information to the cache. The patch changes when they lock the information, so they can serialize it to disk. By moving the lock, they lock the info later, which ends up locking it for a shorter period of time.

So, while writing the cache, we would see our "lock up" issue, and as soon as it completes, things "free up". You can track this by doing something along these lines:

tail -f iwtrace.log | find "diskcache"
[Mon Jun 22 09:24:50 2009] Got lock.  Saving to diskcache....
[Mon Jun 22 09:25:10 2009] Finished saving to diskcache.

Before the patch, the duration of the save, would sometimes be many minutes long.

I can't say this is 100% going to solve it, but it does work around it enough to be noticeably better.

So, why doesn't this happen to everybody? What is really the root cause? I don't know yet...