TS Event Subsystem is not stable

fiquebem

We're running TS 6.5 and Search (without any patches) on Solaris 9.

Every now and then the TS Event Subsystem, which is configured to write to Oracle 9i, is crashing on us. Sometimes it stays up for several days.

We enabled additional ES logging in iw-home/eventsubsystem/conf/log4j.xml. We set the "priority" value to DEBUG. (It was ERROR before.) But eventsubd.log shows no sign of distress; the logging merely stops at some point.

The "/teamsite/iw-home/eventsubsystem/../tools/java/jre/bin/java -server -Xmx256m ..." process (see the rest of the command below) dies and we can't connect to port 3035.

We have 2 TS boxes: Dev and Prod. They both have trouble intermittently (independent of each other). Each writes to a separate Oracle events DB server. I talked to the DBA and he assured me that Oracle has been running continuously and flawlessly. I don't think we can blame the instability of the ES on the back-end DB, although maybe there might be some network-related issue (I'm guessing).

In iw-home/eventsubsystem/conf/jmsconfignew.xml, under <DatabaseConfiguration>, I wonder if adding these attributes to <RdbmsDatabaseConfiguration> would help:

retries="5"
timeout="2000"

I can't put a finger on the problem. How can I tell what's causing the crash? Is there some log information I can refer to? Any suggestions?

By the way, when the ES is down, the Search server goes wild -- its CPU usage approaches 100%.

The TS 6.5 SP1 Release Notes mention that these issues have been fixed:

56491 The TeamSite search index server no longer receives events subsystem once the event system goes down or is restarted.
56956 Correct servletd failure due to unsunchronized access to sciface error/message catalog.

Is the ES tolerably stable with TS 6.5 SP1 and/or SP2? I'm leaning towards telling my client that the resolution lies in installing SP2 and praying.

Thanks,

S.

/teamsite/iw-home/eventsubsystem/../tools/java/jre/bin/java -server -Xmx256m -Dopenjms.home=/teamsite/iw-home/eventsubsystem -Djava.security.manager -Djava.security.policy=/teamsite/iw-home/eventsubsystem/conf/openjms.policy -classpath /teamsite/iw-home/eventsubsystem/lib/ant-1.5.3-1.jar:/teamsite/iw-home/eventsubsystem/lib/ant-optional-1.5.3-1.jar:/teamsite/iw-home/eventsubsystem/lib/antlr-2.7.2.jar:/teamsite/iw-home/eventsubsystem/lib/castor-0.9.5.jar:/teamsite/iw-home/eventsubsystem/lib/classes12.zip:/teamsite/iw-home/eventsubsystem/lib/commons-collections-2.1.jar:/teamsite/iw-home/eventsubsystem/lib/commons-dbcp-1.1.jar:/teamsite/iw-home/eventsubsystem/lib/commons-logging-1.0.3.jar:/teamsite/iw-home/eventsubsystem/lib/commons-pool-1.1.jar:/teamsite/iw-home/eventsubsystem/lib/concurrent-1.3.2.jar:/teamsite/iw-home/eventsubsystem/lib/exolabcore-0.3.7.jar:/teamsite/iw-home/eventsubsystem/lib/hsqldb-1.6.1.jar:/teamsite/iw-home/eventsubsystem/lib/jdbc-2.0.jar:/teamsite/iw-home/eventsubsystem/lib/jdbm.jar:/teamsite/iw-home/eventsubsystem/lib/jms-1.0.2a.jar:/teamsite/iw-home/eventsubsystem/lib/jndi-1.2.1.jar:/teamsite/iw-home/eventsubsystem/lib/jta-1.0.1.jar:/teamsite/iw-home/eventsubsystem/lib/junit-3.7.jar:/teamsite/iw-home/eventsubsystem/lib/log4j-1.2.7.jar:/teamsite/iw-home/eventsubsystem/lib/openjms-0.7.6.1.jar:/teamsite/iw-home/eventsubsystem/lib/openjms-client-0.7.6.1.jar:/teamsite/iw-home/eventsubsystem/lib/openjms.war:/teamsite/iw-home/eventsubsystem/lib/oro-2.0.7.jar:/teamsite/iw-home/eventsubsystem/lib/servletapi-2.3.jar:/teamsite/iw-home/eventsubsystem/lib/xerces-2.3.0.jar:/teamsite/iw-home/eventsubsystem/lib/xml-apis-1.0.b2.jar:/teamsite/iw-home/eventsubsystem/conf org.exolab.jms.server.JmsServer -config /teamsite/iw-home/eventsubsystem/conf/jmsconfignew.xml

smenon

Can you check your eventsusbsystem log to see if there are any suspicious warning or error messages?

--Sunil Menon
Sr. Product Manager
Interwoven, Inc.