TS Search indexing never completes
TS 7.3.2 with patch TS-18304; Win2k8 x64.
Result of `iwndxstatus -a` shows that many branches are 11 - INDEXED but some are stuck on 12 - PARTIALLY INDEXED. There's nothing special about those branches as far as I can tell. Also, one of the smallest and one of the largest both have this issue, but others in between are fine.
Cranked up debug logging everywhere and the best I can find is that the Java part of the indexing is correctly scanning the affected editions and workareas and finding a whole bunch of files, then submitting index jobs to IDOL. It then waits for those jobs to complete, and I see many of these in the iw.tsindex.log:
[26 Aug 2014 15:41:36,498] DEBUG com.interwoven.hunter.index.IdolIndexHelper (IncrementalIndexer-0) - waitForIndexToComplete() - Query index result for index ID (//MYSERVER/default/main/MyBranchName/EDITION/#2164) and database (TeamSite): expected=4 returned=0 retry=3
Where the retry count goes from 0 to 10 over the course of something like 15 minutes. On the 10th retry, it then barfs out this:
[26 Aug 2014 15:41:36,498] ERROR com.interwoven.hunter.index.IncrementalJobProcessor (IncrementalIndexer-0) - Couldn't complete indexing due to exception
Error Code :-1
com.interwoven.hunter.index.api.IndexException: The index job with index id (//MYSERVER/default/main/MyBranchName/EDITION/#2164) and database (TeamSite) is incomplete.
I've been looking over many of the IDOL logs and can't find any corresponding errors on that side. I saw a configuration variable in search.properties that looks like it might extend the amount of time it waits for the index command to complete, but I don't have much hope in that [trying it now anyway].
Any thoughts on what the problem might be, or where else I should be looking?