TeamSite Search Indexer crashes repeatedly with EXCEPTION_ACCESS_VIOLATION in JRE

Hello dear co-sufferers Smiley Happy

We have some TeamSite 7.2.1 servers (with patch 1) on Windows 2003.

On a few of these, we have occasionally seen the "Interwoven TeamSite Index Service" (iwtsindex) go down, with a log entry saying the JRE has crashed with EXCEPTION_ACCESS_VIOLATION

Sometimes this disappeared by itself, after trying several times to restart that service. On some, it has never gone away, or we didn't care and stopped checking for it.

We have this now again on a server where we want it solved.

Logs are showing:

----- iw.tsindex.log -----
nothing relevant... the last messages are about indexing documents/files

----- iwtsindexwrapper.log -----
An unexpected error has been detected by Java Runtime Environment: EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6dc99925, pid=13412, tid=14156
(...)
An error report file with more information is saved as:
(...)\Search\private\bin\hs_err_pid13412.log

I searched the knowledge database in support website, and found a few hits for EXCEPTION_ACCESS_VIOLATION, and found this exact same thing documented for the Index process in TS 6.5 (and for event subsystem, and for OpenDeploy)

But most solutions do not seem applicable for my version of TeamSite (7.2.1-patch1), like "delete the openjms.log file" etc.

I tried to search devnet, but it refuses to search on EXCEPTION_ACCESS_VIOLATION on the basis that it is too long :-( I have not found anything here, so far.

PS: here are some hits from the support database on keyword EXCEPTION_ACCESS_VIOLATION

https://customers.autonomy.com/support/secure/docs/FAQ/Interwoven/57281.html
https://customers.autonomy.com/support/secure/docs/FAQ/Interwoven/58820.html

Thanks for any pointers
Frederik.

Find more posts tagged with

Comments

Frederik

I'll add some longer excerpts from mentioned logs - for completeness and clarity. :-)

----- iwtsindexwrapper.log -----

INFO   | jvm 5    | 2012/09/10 11:00:05 | 11:00:05.759 [InactivityMonitor WriteCheck] DEBUG o.a.a.transport.InactivityMonitor - 10000 ms elapsed since last write check.
INFO   | jvm 5    | 2012/09/10 11:00:05 | 11:00:05.759 [InactivityMonitor ReadCheck] DEBUG o.a.a.transport.InactivityMonitor - 30000 ms elapsed since last read check.
INFO   | jvm 5    | 2012/09/10 11:00:15 | 11:00:15.759 [InactivityMonitor WriteCheck] DEBUG o.a.a.transport.InactivityMonitor - 10000 ms elapsed since last write check.
INFO   | jvm 5    | 2012/09/10 11:00:22 | #
INFO   | jvm 5    | 2012/09/10 11:00:22 | # An unexpected error has been detected by Java Runtime Environment:
INFO   | jvm 5    | 2012/09/10 11:00:22 | #
INFO   | jvm 5    | 2012/09/10 11:00:22 | #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6dc99925, pid=13412, tid=14156
INFO   | jvm 5    | 2012/09/10 11:00:22 | #
INFO   | jvm 5    | 2012/09/10 11:00:22 | # Java VM: Java HotSpot(TM) Server VM (11.2-b01 mixed mode windows-x86)
INFO   | jvm 5    | 2012/09/10 11:00:22 | # Problematic frame:
INFO   | jvm 5    | 2012/09/10 11:00:22 | # V  [jvm.dll+0x169925]
INFO   | jvm 5    | 2012/09/10 11:00:22 | #
INFO   | jvm 5    | 2012/09/10 11:00:22 | # An error report file with more information is saved as:
INFO   | jvm 5    | 2012/09/10 11:00:22 | # \GWIB\TS\Search\private\bin\hs_err_pid13412.log
INFO   | jvm 5    | 2012/09/10 11:00:22 | #
INFO   | jvm 5    | 2012/09/10 11:00:22 | # If you would like to submit a bug report, please visit:
INFO   | jvm 5    | 2012/09/10 11:00:22 | #   http://java.sun.com/webapps/bugreport/crash.jsp
INFO   | jvm 5    | 2012/09/10 11:00:22 | #

-------- hs_err_pid13412.log --------

#
# An unexpected error has been detected by Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6dc99925, pid=13412, tid=14156
#
# Java VM: Java HotSpot(TM) Server VM (11.2-b01 mixed mode windows-x86)
# Problematic frame:
# V  [jvm.dll+0x169925]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x28bc9400):  JavaThread "BulkIndexer-0" [_thread_in_vm, id=14156, stack(0x29290000,0x292e0000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000008

Registers:
EAX=0x00000000, EBX=0x28bc9400, ECX=0x00000006, EDX=0x28750020
ESP=0x292def84, EBP=0x292df7d8, ESI=0x2a3d2940, EDI=0x28bc9514
EIP=0x6dc99925, EFLAGS=0x00010246

Top of Stack: (sp=0x292def84)
0x292def84:   28bc9400 00000006 00000000 2a3a2db4
0x292def94:   28bc9514 292df7e8 28750020 00000000
0x292defa4:   040e0e60 28bc9400 292df7e8 00000000
0x292defb4:   00000000 00000000 00000000 00000000
0x292defc4:   292df800 292df810 00000000 3a6e490a
0x292defd4:   74733c20 6d616572 754f203e 3c203a74
0x292defe4:   65727473 003e6d61 00000000 00000000
0x292deff4:   00000000 00000000 00000000 00000000 

Instructions: (pc=0x6dc99925)
0x6dc99915:   8b d9 6a 06 53 e8 f1 77 e9 ff 8b 44 24 14 8b 00
0x6dc99925:   8b 48 08 83 c4 08 85 c9 74 3c 8b 54 24 14 85 d2 


Stack: [0x29290000,0x292e0000],  sp=0x292def84,  free space=315k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x169925]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  com.verity.api.filter.Filter.filterStreamToStream([BLjava/lang/String;J[Ljava/lang/Object;IZLjava/lang/String;J)[B
J  com.verity.api.filter.Filter.filterTo(Ljava/io/OutputStreamV
j  com.interwoven.hunter.filter.DocumentFilter.extractContent(Lcom/interwoven/hunter/filter/InputDocument;Ljava/io/OutputStreamLcom/interwoven/hunter/filter/FilteredDocument;+286
j  com.interwoven.hunter.filter.DocumentFilter.filter(Lcom/interwoven/hunter/filter/InputDocument;Ljava/io/OutputStreamLcom/interwoven/hunter/filter/FilteredDocument;+52
j  com.interwoven.hunter.index.IndexerSupport.processFile(Ljava/lang/String;Lcom/interwoven/hunter/common/FileData;Lcom/interwoven/hunter/index/ObjectInfo;Ljava/io/PrintWriter;Ljava/lang/String;Ljava/io/FileJ+904
j  com.interwoven.hunter.index.IndexerSupport.generateIDXContent(Lcom/interwoven/hunter/index/ObjectInfo;Ljava/util/List;Ljava/io/PrintWriter;Ljava/lang/String;Ljava/lang/String;Ljava/io/FileJ+44
j  com.interwoven.hunter.index.AbstractJobProcessor.indexIDXSizedFiles(Lcom/interwoven/hunter/index/ObjectInfo;Lcom/interwoven/hunter/index/BranchIndexInfo;Ljava/util/List;Ljava/lang/StringV+148
j  com.interwoven.hunter.index.AbstractJobProcessor.indexBranch(Lcom/interwoven/hunter/index/ObjectInfo;Lcom/interwoven/hunter/index/BranchIndexInfo;Ljava/lang/String;J)I+321
j  com.interwoven.hunter.index.BulkJobProcessor.processNextJob()I+203
j  com.interwoven.hunter.index.Indexer.run()V+184
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x28a46400 JavaThread "InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@1d709a5" daemon [_thread_blocked, id=10076, stack(0x28700000,0x28750000)]
  0x28155400 JavaThread "Thread-17" daemon [_thread_blocked, id=12356, stack(0x29330000,0x29380000)]
  0x289b5400 JavaThread "IncrementalIndexer-0" [_thread_blocked, id=16252, stack(0x292e0000,0x29330000)]
=>0x28bc9400 JavaThread "BulkIndexer-0" [_thread_in_vm, id=14156, stack(0x29290000,0x292e0000)]
  0x28c96800 JavaThread "Thread-16" daemon [_thread_blocked, id=14628, stack(0x29240000,0x29290000)]
  0x280da400 JavaThread "ActiveMQ Scheduler" daemon [_thread_blocked, id=15728, stack(0x291f0000,0x29240000)]
  0x2815dc00 JavaThread "InactivityMonitor WriteCheck" daemon [_thread_blocked, id=12348, stack(0x291a0000,0x291f0000)]
  0x28ac8400 JavaThread "InactivityMonitor ReadCheck" daemon [_thread_blocked, id=14436, stack(0x29150000,0x291a0000)]
  0x2801c800 JavaThread "ActiveMQ Transport: tcp://localhost/127.0.0.1:3035" [_thread_in_native, id=8544, stack(0x29100000,0x29150000)]
  0x27ea1c00 JavaThread "WrapperSimpleAppMain" [_thread_in_native, id=15188, stack(0x284c0000,0x28510000)]
  0x002a7800 JavaThread "DestroyJavaVM" [_thread_blocked, id=14824, stack(0x005c0000,0x00610000)]
  0x27ff7800 JavaThread "Wrapper-Connection" [_thread_in_native, id=14488, stack(0x283f0000,0x28440000)]
  0x27fa0800 JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_blocked, id=16308, stack(0x283a0000,0x283f0000)]
  0x27a2bc00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=8764, stack(0x27ce0000,0x27d30000)]
  0x27a28800 JavaThread "CompilerThread1" daemon [_thread_blocked, id=13852, stack(0x27c90000,0x27ce0000)]
  0x27a27000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=15308, stack(0x27c40000,0x27c90000)]
  0x27a22400 JavaThread "Attach Listener" daemon [_thread_blocked, id=14516, stack(0x27bf0000,0x27c40000)]
  0x27a21000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=15752, stack(0x27ba0000,0x27bf0000)]
  0x27a11400 JavaThread "Finalizer" daemon [_thread_blocked, id=12780, stack(0x27b50000,0x27ba0000)]
  0x27a0c800 JavaThread "Reference Handler" daemon [_thread_blocked, id=9180, stack(0x27b00000,0x27b50000)]

Other Threads:
  0x27a09400 VMThread [stack: 0x27ab0000,0x27b00000] [id=6308]
  0x27a2dc00 WatcherThread [stack: 0x27d30000,0x27d80000] [id=14804]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 PSYoungGen      total 7104K, used 5304K [0x23df0000, 0x24500000, 0x276d0000)
  eden space 6976K, 75% used [0x23df0000,0x2431a128,0x244c0000)
  from space 128K, 12% used [0x244c0000,0x244c4000,0x244e0000)
  to   space 128K, 0% used [0x244e0000,0x244e0000,0x24500000)
 PSOldGen        total 58304K, used 3914K [0x076d0000, 0x0afc0000, 0x23df0000)
  object space 58304K, 6% used [0x076d0000,0x07aa2828,0x0afc0000)
 PSPermGen       total 16384K, used 10441K [0x036d0000, 0x046d0000, 0x076d0000)
  object space 16384K, 63% used [0x036d0000,0x04102670,0x046d0000)

Dynamic libraries:
0x00400000 - 0x00424000  \GWIB\TS\Search\jre\bin\java.exe
0x7c800000 - 0x7c8c3000  C:\WINDOWS\system32\ntdll.dll
0x77e40000 - 0x77f42000  C:\WINDOWS\system32\kernel32.dll
0x7d1e0000 - 0x7d27c000  C:\WINDOWS\system32\ADVAPI32.dll
0x77c50000 - 0x77cf0000  C:\WINDOWS\system32\RPCRT4.dll
0x76f50000 - 0x76f63000  C:\WINDOWS\system32\Secur32.dll
0x7c340000 - 0x7c396000  \GWIB\TS\Search\jre\bin\msvcr71.dll
0x6db30000 - 0x6def4000  \GWIB\TS\Search\jre\bin\server\jvm.dll
0x77380000 - 0x77411000  C:\WINDOWS\system32\USER32.dll
0x77c00000 - 0x77c49000  C:\WINDOWS\system32\GDI32.dll
0x76aa0000 - 0x76acd000  C:\WINDOWS\system32\WINMM.dll
0x76290000 - 0x762ad000  C:\WINDOWS\system32\IMM32.DLL
0x6d340000 - 0x6d348000  \GWIB\TS\Search\jre\bin\hpi.dll
0x76b70000 - 0x76b7b000  C:\WINDOWS\system32\PSAPI.DLL
0x6d860000 - 0x6d86c000  \GWIB\TS\Search\jre\bin\verify.dll
0x6d3e0000 - 0x6d3ff000  \GWIB\TS\Search\jre\bin\java.dll
0x6d8a0000 - 0x6d8af000  \GWIB\TS\Search\jre\bin\zip.dll
0x28380000 - 0x28392000  \GWIB\TS\Search\lib\wrapper.dll
0x6d6c0000 - 0x6d6d3000  \GWIB\TS\Search\jre\bin\net.dll
0x71c00000 - 0x71c17000  C:\WINDOWS\system32\WS2_32.dll
0x77ba0000 - 0x77bfa000  C:\WINDOWS\system32\msvcrt.dll
0x71bf0000 - 0x71bf8000  C:\WINDOWS\system32\WS2HELP.dll
0x71b20000 - 0x71b61000  C:\WINDOWS\system32\mswsock.dll
0x5f270000 - 0x5f2ca000  C:\WINDOWS\system32\hnetcfg.dll
0x71ae0000 - 0x71ae8000  C:\WINDOWS\System32\wshtcpip.dll
0x76ed0000 - 0x76efa000  C:\WINDOWS\system32\DNSAPI.dll
0x76f70000 - 0x76f77000  C:\WINDOWS\System32\winrnr.dll
0x76f10000 - 0x76f3e000  C:\WINDOWS\system32\WLDAP32.dll
0x76f80000 - 0x76f85000  C:\WINDOWS\system32\rasadhlp.dll
0x68000000 - 0x68035000  C:\WINDOWS\system32\rsaenh.dll
0x7c8d0000 - 0x7d0cf000  C:\WINDOWS\system32\SHELL32.dll
0x7d180000 - 0x7d1d2000  C:\WINDOWS\system32\SHLWAPI.dll
0x77420000 - 0x77523000  C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.4770_x-ww_05FDF087\comctl32.dll
0x2a3a0000 - 0x2a3c2000  \GWIB\TS\Search\idol\filter\lib\nti86\KeyViewFilter.dll
0x78130000 - 0x781cb000  C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.4053_x-ww_E6967989\MSVCR80.dll
0x28510000 - 0x2854b000  \GWIB\TS\Search\idol\filter\lib\nti86\kvfilter.dll
0x28560000 - 0x28628000  \GWIB\TS\Search\idol\filter\lib\nti86\kvutil.dll
0x7c420000 - 0x7c4a7000  C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.4053_x-ww_E6967989\MSVCP80.dll
0x28640000 - 0x286b6000  \GWIB\TS\Search\idol\filter\lib\nti86\kwad.dll
0x286d0000 - 0x286e2000  \GWIB\TS\Search\idol\filter\lib\nti86\kvxtract.dll

VM Arguments:
jvm_args: -Xms64m -Xmx512m -Diw.search.base.dir=D:\GWIB\TS\Search -Djava.library.path=D:\GWIB\TS\Search\lib;D:\GWIB\TS\Search\idol\filter\lib\nti86 -Dwrapper.key=nbJHQD35pSOPqxoX -Dwrapper.port=32000 -Dwrapper.use_system_time=TRUE -Dwrapper.version=3.1.0 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=5 
java_command: org.tanukisoftware.wrapper.WrapperSimpleApp com.interwoven.hunter.index.IndexServer
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=C:\WINDOWS\system32;C:\Program Files\CorporateEngine;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Tools\SysTools;C:\Tools\PubTools;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Program Files\NTRK;C:\Program Files\SupportTools;C:\Program Files\Runtime\;C:\Program Files\Common Files\Interwoven\lib;D:\GWIB\TS\TeamSite\bin;D:\GWIB\TS\Search\idol\filter\lib\nti86
OSTYPE=W3E
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 11, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 15 stepping 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 2097151k(1634488k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot(TM) Server VM (11.2-b01) for windows-x86 JRE (1.6.0_12-b04), built on Jan 17 2009 09:52:33 by "java_re" with MS VC++ 7.1

time: Mon Sep 10 11:00:22 2012
elapsed time: 88 seconds

Adam Stoller

If you haven't already done so, I would strongly suggest opening up a case with Support and providing them all the information you posted here (especially the last two logs) - and possibly more (they'll want to see all sorts of configuration files too).

Frederik

Indeed, a support case is open too. I will post conclusions here, but was hoping the crowd would have a suggestion... I doubt we're the first to be bugged by this...

labbrown

Fredrick,

Did you find out anything from support?