TS Search indexing never completes

Rick Poulin · August 26, 2014

TS 7.3.2 with patch TS-18304; Win2k8 x64.

Result of `iwndxstatus -a` shows that many branches are 11 - INDEXED but some are stuck on 12 - PARTIALLY INDEXED. There's nothing special about those branches as far as I can tell. Also, one of the smallest and one of the largest both have this issue, but others in between are fine.

Cranked up debug logging everywhere and the best I can find is that the Java part of the indexing is correctly scanning the affected editions and workareas and finding a whole bunch of files, then submitting index jobs to IDOL. It then waits for those jobs to complete, and I see many of these in the iw.tsindex.log:

[26 Aug 2014 15:41:36,498] DEBUG com.interwoven.hunter.index.IdolIndexHelper (IncrementalIndexer-0) - waitForIndexToComplete() - Query index result for index ID (//MYSERVER/default/main/MyBranchName/EDITION/#2164) and database (TeamSite): expected=4 returned=0 retry=3

Where the retry count goes from 0 to 10 over the course of something like 15 minutes. On the 10th retry, it then barfs out this:

[26 Aug 2014 15:41:36,498] ERROR com.interwoven.hunter.index.IncrementalJobProcessor (IncrementalIndexer-0) - Couldn't complete indexing due to exception
Error Code :-1
com.interwoven.hunter.index.api.IndexException: The index job with index id (//MYSERVER/default/main/MyBranchName/EDITION/#2164) and database (TeamSite) is incomplete.
	at com.interwoven.hunter.index.IdolIndexHelper.waitForIndexToComplete(IdolIndexHelper.java:212)
	at com.interwoven.hunter.index.IndexServiceCFSImpl.indexEditionBased(IndexServiceCFSImpl.java:489)
	at com.interwoven.hunter.index.IndexServiceCFSImpl.indexBranchIncrementally(IndexServiceCFSImpl.java:139)
	at com.interwoven.hunter.index.IncrementalJobProcessor.processNextJob(IncrementalJobProcessor.java:71)
	at com.interwoven.hunter.index.Indexer.run(Indexer.java:157)
	at java.lang.Thread.run(Thread.java:619)

I've been looking over many of the IDOL logs and can't find any corresponding errors on that side. I saw a configuration variable in search.properties that looks like it might extend the amount of time it waits for the index command to complete, but I don't have much hope in that [trying it now anyway].

Any thoughts on what the problem might be, or where else I should be looking?

Thanks!

GoingMyWay · August 26, 2014

Could there be backing store corruption?

-David

Adam Stoller · August 26, 2014

Check with Support - I think I ran into something similar a while back on a project (I think it was last winter) - if I can find any significant information in my notes from the project (over the next couple of days, just got back from a road trip) I'll pass it along...

Rick Poulin · August 27, 2014

Thanks @ghoti. I expect we'll be opening a support ticket shortly because I've run out of ideas.

David, I suspected something like that too, but I'm seeing this behaviour on a few different servers here and the iwfsck all came back clean. Also, we only recently completed our upgrade this summer (using new servers) so the likelihood of bad data seems low. I was rather hoping I'd just gotten a config parameter wrong somewhere...

Adam Stoller · August 27, 2014

DId you get the patches (plural) for Search 7.3.2 from Support?

My notes are from a Solaris installation, but I believe the same information applies to Windows (though hopefully Support would know for sure)

TS-18304 - which gets installed on both TeamSite and Search servers

[Search Server]

Replaces a number of hunter_*.jar files in SEARCH_HOME/lib
Modifications to the AutonomyContent.cfg in SEARCH_HOME/idol

[TeamSite server]

Replaces the livesite.tk.war and search.tk.war in IWHOME/private/lib/content_center
Modifications to the search_results.jsp, a copy of which was placed (and modified) in IWHOME/local/config/lib/content_center/customer_src/web

TS-18853 - which also gets installed on both TeamSite and Search servers

[TeamSite server]

Replaces the iwserver.sol (or whatever is appropriate for your platform) in IWHOME/bin
Replaces the cssdkiface.jar in IWHOME/cssdk
Replaces the cssdkjava.jar, cssdk_sci.jar, and searchiface.jar in IWHOME/cssdk/java
Replaces the base.tk.war, teamsite.tk.war, and search.tk.war in IWHOME/private/lib/content_center
Replaces the cssdk_services.war in ApplicationContainer/server/default/deploy

[Search server]

Replaces (again) the hunter_*.jar files in SEARCH_HOME/lib
Modifications required to both the search.properties and search.private.properties files in SEARCH_HOME/etc
Creation (if it does not exist) of SEARCH_HOME/temp directory
Modifies the TeamSiteConnector.cfg in SEARCH_HOME/connector/TeamSiteConnector
Modifies the ConnectorFramework.cfg in SEARCH_HOME/connector/ConnectorFramework
Modifies the AutonomyContent.cfg in SEARCH_HOME/idol

TS-18853b (supplemental) - this only applies to the Search server (the 'b' designation may have been mine, I don't recall if this was an "official" patch - it was to fix workarea indexing)

Creation (if it does not exist) of SEARCH_HOME/temp directory
Replaces (yet again) the hunter_*.jar files in SEARCH_HOME/lib
Modifies search.properties in SEARCH_HOME/etc

Hope that helps

Rick Poulin · August 27, 2014

Was not aware of a 18853 patch. I'll ask about that in my ticket. Thanks for the tip!

Rick Poulin · August 28, 2014

Update: Looks like our "ConnectorFramework" service is dying on occasion without any explanation anywhere.. After restarting all services in the right order, it appears as though at least one of the problematic branches was able to index to completion.

Now I'm trying to figure out what brings it down in the first place. Circumstantial evidence points to it shutting down sometimes when the "TeamSiteConnector" service is stopped/restarted. I had integrated this into our build process to prevent a bunch of "can't talk to the event subsystem" errors in the logs while JBoss is down, so I might just need to add handling for the status of the ConnectorFramework service as well.

Adam Stoller · August 29, 2014

Have you looked into patch 18853 yet? It would seem that it might be applicable given that it deals with several of the connector/* configuration files and (presumably) associated jar files.

Rick Poulin · August 29, 2014

Submitted the ticket, waiting for a response from support.

TS Search indexing never completes

Comments

Categories