nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Eventsubsystem down! Production down! help

System

in eventsubd.log...our messages table has over 240k rows and filled up our oracle database! can this table be cleared? archived? please help!

12:16:22.812 ERROR [server-24] - Failed to make message persistent
org.exolab.jms.persistence.PersistenceException:
java.sql.SQLException: ORA-01653: unable to extend table METADATA.MESSAGES by 1024 in tablespace METADATA

at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:289)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:582)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1986)
at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1144)
at oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:2152)
at oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:2035)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2876)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:609)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:233)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:233)
at org.exolab.jms.persistence.Messages.add(Unknown Source)
at org.exolab.jms.persistence.RDBMSAdapter.addMessage(Unknown Source)
at org.exolab.jms.messagemgr.MessageMgr.addPersistentMessage(Unknown Source)
at org.exolab.jms.messagemgr.MessageMgr.add(Unknown Source)
at org.exolab.jms.server.JmsServerSession.sendMessage(Unknown Source)
at org.exolab.jms.server.mipc.IpcJmsSessionConnection.sendMessage(Unknown Source)
at org.exolab.jms.server.mipc.IpcJmsSessionConnection.notify(Unknown Source)
at org.exolab.jms.server.mipc.IpcJmsReceiver.notify(Unknown Source)
at org.exolab.jms.server.mipc.IpcServerChannel.run(Unknown Source)

Find more posts tagged with

Comments

tienle66

Are you on Win or Sol?

When our eventsubsystem went down we were told to remove the following two files from /iw-home/eventsubsystem/

openjms.db openjms.lg

Then restart the eventsubsytem.

nipper

He is on Oracle, so that is not the issue. IIRC, TS should be purging events from Oracle. I assume you called support.

Andy

Migrateduser

yeah the METADATA.MESSAGES table has over 240,000 rows...our temp fix was to have our DBAs add more space. here is what support said *sigh*

Write a SQL script to delete all entries on the eventsubsystem oracle db.

Thank you, we look forward to your response.

Im assuming they mean it is ok to delete everything in the METADATA.MESSAGES table?

More importantly, why is this table growing so large (over 240,000 rows)???

Migrateduser

Hi Arthas -

We've run into the problem of messages that are never purged as well. If all the subscribers that you expect to consume them are in fact consuming them and they are staying around, it's ok to delete them.

In theory, a message can be garbage collected once all registered subscribers (typically search and DAS if you haven't written any of your own) have consumed it. If the message isn't being garbage collected, then some subscriber is never registering that it has consumed it. On one box where we had the problem we hypothesized that this was because search was installed but not actually running. Further testing failed to substantiate that.

Another possibility is the event filter on DAS. By default it is set to ignore messages prior to its last restart. I think that's to keep it from consuming the same message twice if it hasn't been garbage collected.

We've been pursuing this question through an IWOV SE for several weeks now, and we're not any closer to having a definitive answer. The documentation surrounding the eventsubsystem is exceptionally limited and poor, leaving one to make educated guesses based on how OpenJMS is generally used.

nipper

Yea, from what PM has been saying, moving from OpenJMS to Oracle will take care of this issue. Obviously that is not the case. There was a post blaming failure of OpenJMS of deleting events cauing the problem. I guess Oracle is not deleting them either. Must be Oracle's fault. :-)

Migrateduser

it would be nice to get an official response from IW in here about this...i havent heard back about my ticket since the original correspondence.

our DBAs are really anxious to get in a fix for this (see below)

I support the back end database for Content Management. We have noticed that the metadata.messages table has been growing at a much faster rate since about 3 weeks. The app is creating approximately 30 MB everyday but does not seem to be cleaning it up.
When you mention the Event subsystem do you mean just the metadata.messages table. Or, are there other tables which maintain consistency with the messages table that we need to include in the cleanup process. Can you please send any kind of documentation that you may have regarding the cleanup process.
Also is there a way to find out if this is being triggered by some other underlying problem.

nickcolorado

Any updates on this? We are seeing the exact same issue. Our messages table is overwhelmed with messages and we need to clear them out...

Migrateduser

My understanding of the root cause is that the name of the topics that DD expects to subscribe to for DAS changed between version 5.x and version 6.0.2. The clever "solution" was to publish the messages to both topics. Unless you had both versions of DD consuming the messages though, at least some messages will never be consumed.

If you call support and walk through the issue, they should be able to help you set up your jmsconfignew.xml file to only publish to the topics you are actually using.