nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

message_handles_table growing

System

Hi. We're running 6.5 SP1 on Windows with Report Center. I'm trying to understand an issue we're having.

Our message_handles_table continues to grow which I think means that that data is not being "consumed" by anything, and therefore is not getting into our reports database.

I've looked at tsreport.log, tsreport_out.log, eventsubd.log, and eventsubd_err.log but have only seen "java.lang.OutOfMemoryError" in the eventsubd_err.log.

How can I verify our configuration is correct? Where else might some errors be showing up?

Thanks for any help.

Craig

Find more posts tagged with

Comments

Migrateduser

I don't know if this is related, yet. But I started getting lots of these in the tsreport.log file:

23 Feb 2009 13:57:48,830 WARN - Message property type has no matching table column
23 Feb 2009 13:57:54,377 ERROR - Caught CSException in getEvents: com.interwoven.cssdk.common.CSObjectNotFoundException: (Object being looked up was not found)

Faizi

I don't know if this is related, yet. But I started getting lots of these in the tsreport.log file:

23 Feb 2009 13:57:48,830 WARN - Message property type has no matching table column
23 Feb 2009 13:57:54,377 ERROR - Caught CSException in getEvents: com.interwoven.cssdk.common.CSObjectNotFoundException: (Object being looked up was not found)

Hi,
In \\iw-home\eventsubsystem directory there would be openjms.db, this file size might be so big which create problem. So just delete it and create same file (openjms.db).

HTH,

Migrateduser

Hi,
In \\iw-home\eventsubsystem directory there would be openjms.db, this file size might be so big which create problem. So just delete it and create same file (openjms.db).

HTH,

Nope, we don't have that. I think because we are storing them all in a MS SQL Server DB.

Migrateduser

Our latest is that it looks like the Report Center subscriber is processing messages, but simply can't process them fast enough to keep up.

I have a case open and thought I would toss out the text I updated my case with in the hopes that somebody may have been down this road before us.

Last week, we experienced mutiple outages of our system. It would not process deployments and generally stopped processing. We found that restarting the eventsubsystem got it going again. So for a day, we bounced the events subsystem multiple times to limp through.

The next day, I backed up the message_handles_table and deleted all the rows from the original. This "fix" got us through the last 5 days or so. Our message_handles_table is now back to ~350,000 rows, so I'm making plans to repeat the procedure and buy us another week or so.

The interesting part is that the subscriber for Report Center does process rows and delete them from the table, but isn't fast enough to keep up.

Here are the stats so you can see how the count goes up and down:
time,Round Trip Time,Result Column 1,Rows,Content Match,Round Trip Time,Result Column 1,Rows,Content Match
2:35 PM 3/10/09,0.257 sec,1389153,1,n/a,0.257,1389153,1,n/a
6:14 PM 3/10/09,0.182 sec,1389153,1,n/a,0.182,1389153,1,n/a
9:53 PM 3/10/09,0.351 sec,1389153,1,n/a,0.351,1389153,1,n/a
1:32 AM 3/11/09,1.505 sec,1389220,1,n/a,1.505,1389220,1,n/a
5:11 AM 3/11/09,0.187 sec,1389125,1,n/a,0.187,1389125,1,n/a
8:50 AM 3/11/09,0.148 sec,704114,1,n/a,0.148,704114,1,n/a
12:29 PM 3/11/09,0.166 sec,11834,1,no data,0.166,11834,1,no data
4:08 PM 3/11/09,0.265 sec,14505,1,no data,0.265,14505,1,no data
7:47 PM 3/11/09,1.636 sec,10258,1,no data,1.636,10258,1,no data
11:26 PM 3/11/09,0.599 sec,7154,1,no data,0.599,7154,1,no data
3:05 AM 3/12/09,0.152 sec,28975,1,no data,0.152,28975,1,no data
6:44 AM 3/12/09,0.304 sec,80957,1,n/a,0.304,80957,1,n/a
10:23 AM 3/12/09,0.244 sec,114854,1,n/a,0.244,114854,1,n/a
2:02 PM 3/12/09,0.136 sec,145258,1,n/a,0.136,145258,1,n/a
5:41 PM 3/12/09,0.270 sec,157242,1,n/a,0.270,157242,1,n/a
9:20 PM 3/12/09,0.229 sec,157613,1,n/a,0.229,157613,1,n/a
12:59 AM 3/13/09,0.261 sec,156342,1,n/a,0.261,156342,1,n/a
4:38 AM 3/13/09,0.238 sec,155069,1,n/a,0.238,155069,1,n/a
8:17 AM 3/13/09,0.426 sec,190455,1,n/a,0.426,190455,1,n/a
11:56 AM 3/13/09,0.820 sec,240522,1,n/a,0.820,240522,1,n/a
3:35 PM 3/13/09,1.132 sec,278787,1,n/a,1.132,278787,1,n/a
7:14 PM 3/13/09,0.963 sec,279351,1,n/a,0.963,279351,1,n/a
10:53 PM 3/13/09,0.550 sec,279342,1,n/a,0.550,279342,1,n/a
2:32 AM 3/14/09,0.495 sec,279174,1,n/a,0.495,279174,1,n/a
6:11 AM 3/14/09,0.656 sec,278558,1,n/a,0.656,278558,1,n/a
9:50 AM 3/14/09,1.758 sec,277852,1,n/a,1.758,277852,1,n/a
1:29 PM 3/14/09,0.760 sec,276965,1,n/a,0.760,276965,1,n/a
5:08 PM 3/14/09,0.988 sec,275936,1,n/a,0.988,275936,1,n/a
8:47 PM 3/14/09,8.415 sec,275384,1,n/a,8.415,275384,1,n/a
12:26 AM 3/15/09,0.068 sec,275451,1,n/a,0.068,275451,1,n/a
4:05 AM 3/15/09,2.132 sec,275194,1,n/a,2.132,275194,1,n/a
7:44 AM 3/15/09,1.706 sec,274124,1,n/a,1.706,274124,1,n/a
11:23 AM 3/15/09,1.010 sec,273162,1,n/a,1.010,273162,1,n/a
3:02 PM 3/15/09,0.871 sec,272271,1,n/a,0.871,272271,1,n/a
6:41 PM 3/15/09,0.347 sec,271165,1,n/a,0.347,271165,1,n/a
10:20 PM 3/15/09,1.359 sec,270532,1,n/a,1.359,270532,1,n/a
1:59 AM 3/16/09,1.218 sec,270300,1,n/a,1.218,270300,1,n/a
5:38 AM 3/16/09,1.527 sec,271075,1,n/a,1.527,271075,1,n/a
9:17 AM 3/16/09,0.854 sec,307064,1,n/a,0.854,307064,1,n/a
12:56 PM 3/16/09,0.976 sec,350108,1,n/a,0.976,350108,1,n/a
We have added two indices to the Report Center DB which was recommended by IW, and used in the current code (which we will be on in about 2 weeks).

My concern is that it is not keeping up, so our problems will continue.

The events that go to the DB are largely all or nothing, with a few exceptions like the 6 FSE events that can be turned on/off in iw.cfg.

We need options regarding what we can do long term, or in the more short term, what can we do to isolate why the subscriber can't process messages faster. How do we know if it is a CPU/memory problem, or a SQL DB problem, or something else? We need insight into something to guide us to a permanent solution.

pbarua

Craig,
Did you get resolution on this issue?

Thanks,
Pallabi

pbarua

This is an error on reporting side. We saw similar warning messages in tsreport.log:
WARN - Message property duration has no matching table column

After turning debugging on we found out reporting receives duration for all event messages, but for specific events such as ChooseTransition TaskcalledBack it explicitly excludes duration from the SQL insert statement to WFEVENTSSUMMARY, and then complains that there is no matching table column for duration. So, you can ignore this warning.

Also, about the "object not found" error, reporting consumption was probably lagging, so when the message was published, it was a valid object, but when reporting was consuming it was no longer existing.

Migrateduser

Craig,
Did you get resolution on this issue?

Thanks,
Pallabi

Yes and no. We ended up upgrading to TS 6.7.2, and I did get things working with that version, but I essentially don't know what was really wrong before.

FurnaceFace

Thank you so much for posting your updates. We are seeing some of these messages pop up in our logs (such as
the "no matching table column for duration" one) and it's a relief to hear that.

Thanks for sharing,
/FF