iwserver debugging

Rick Poulin · July 15, 2014

Trying to keep this one short... does anybody know if there's a way to get the iwserver.exe process running in some kind of "debug" or "verbose" mode? (throughput monitoring is not good enough)

Env:

Win2k8 R2 x64, TS 7.3.2. TeamSite Search is turned off, so we can exclude that as a factor.

Context:

Using iwstat, we can see "normal" operations start to queue up until the iwserver.exe process eventually becomes overwhelmed and unresponsive, at which point the process has to be killed and restarted. By "normal ops" I mean not batch jobs like deleted branch cleanups -- more like user ops like "GetBranchById" and "ListFSEsInDir", things that should complete almost instantly. This happens multiple times a day. It started about a month ago and our infrastructure folks claim that nothing relevant has changed. I'm trying to get the iwserver to spit out some kind of error message that would prove to them that there's a problem with the underlying infrastructure.

Ran iwfsck two weeks ago and found nothing. Running it again, but I'm not expecting much.

nipper · July 15, 2014

Not much. I have gotten a special iwserver process from support with debug turned on before,

Did someone do a bunch of branch/edition removals ? I did one (several thousand editions), took 4 weeks to get a clean iwstat.

when was the last iwfsshrink ?

Rick Poulin · July 15, 2014

No branch/area deletions of any kind. If so, I'd have expected long batch jobs that take forever to complete, but there's no such thing in the iwstat output either -- it's only a bunch of normal jobs.

This is a fairly new implementation which has only been in production for 1.5yrs. I ran a iwfsshrink on a backup of the content store about 2 months ago but the savings were so negligible that it wasn't worth applying to the real thing. And the system ran smoothly in all that time up until last month, when all **** broke loose.

I guess I'll have to see if I can get one of those debug-compiled processes from support.

Rick Poulin · July 23, 2014

UPDATE: still working with support. The tech is still "looking into" options for debug output from the iwserver.exe process.

However, should it be useful to anyone else, they had us run a CLT that I haven't seen documented anywhere. It appears to check the integrity of the dependency matrix in a workarea. He made it sound like this is a fairly common cause of what we've experienced (but it returned no errors in our case).

# TeamSite/bin/iwdepcheck -i /path/to/vpaths.txt

where vpaths.txt is a newline-delimited text file containing the vpaths of the workareas you want to check.

Rick Poulin · September 15, 2014

FINAL UPDATE:

We never did get a "debug" version of iwserver.exe from support, but it does appear as though we've found the solution. There were issues with the Dependency Manager in 7.2.0 which can lead to corruption in its "repository" beyond what was already apparent in the UI (broken dependencies would cause all sorts of havoc). Evidently, in upgrading to 7.3.2, we just brought those corrupted items along with us for the ride.

Support has informed us that the bugs that led to the dependency matrix corruption do not exist in 7.3.2, but we still needed to fix our content store. In our case, the fix is this:

* make sure a good backup of the store exists

* detach content store

* delete the DEPENDMGR in the root folder of the store e.g. X:\iw-store\default\DEPENDMGR

* reattach content store

* run iw_migrate_dependencies.ipl -s

Deleting that file effectively discards the existing dependency matrix data, and the IPL recreates them from scratch (the -s means "all branches"). We asked; there's no way to do this on specific branches. It also marks every single file in the store as modified by me, but it does appear to fix the issue.

iwserver debugging

Comments

Categories