nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

DAS failing/Hanging

speck

TS 6.5, OD 6.2

Getting these sorts of errors:

Command dispatch to DD MessageListner
Exception in InvokeDataDeploy
Reason: Cannot communicate with DataDeploy
End MDC deployment

All services are up and running.

We are also seeing out of memory errors in eventsubd_err
java.lang.OutOfMemoryError

We have upped the max Heap size for the jms startup to 2 Gig, and still see these errors.

All ports are up and listening that I can detect, yet these errors persist. The database is up and fine as well.

Any help would be appreciated.

Find more posts tagged with

Comments

thatfish

These are pretty serious and varied issues.

have you just started getting these errors ?

I'd post further details e.g. OS/logs/process that is hogging the memory/etc

speck

Yes, these errors just started.

We don't see any individual process that's hogging memory.

I am wondering if the underlying OS is running out of threads, since Windows OS will throw this OutOfMemory error when it can't allocate a thread, that has nothing to do with actual maximum heap size being reached for the app.. or perhaps we are using up all the permmem.

This is clearly something going on in the event subsystem/data deploy stack, and I am unable to figure out anything more than that.

The only change I can say is we have a lot more workflows running than we have in the past, so I'm wondering if is something like we are running into issue where the OS just cant' allocate enough threads for all the pending workflows/deployments... but I would hope that Teamsite is written in such a way that it has a cap on threads it tries to run and just doesn't open a thread for everything pending all at once.

I'm really chasing my tail at this point.

Adam Stoller

...
The only change I can say is we have a lot more workflows running than we have in the past, so I'm wondering if is something like we are running into issue where the OS just cant' allocate enough threads for all the pending workflows/deployments... but I would hope that Teamsite is written in such a way that it has a cap on threads it tries to run and just doesn't open a thread for everything pending all at once.

I'm really chasing my tail at this point.

If you have a lot of hung jobs, it can cause performance degradation - though I don't think it is likely to cause the errors you posted about. Still - take a look at the output of iwgetwfobj to get an idea of how many jobs you actually have sitting around on your server - if there are a lot of old jobs that are hung, try killing them (iwrmjob) and see if that improves things for you.

cesarecastro

We ran a similar issue over a year ago. One possible solution to the memory errors in eventsubd_err java.lang.OutOfMemoryError is to increase the memory allocation for the JVM. I would recommend talking to IW support.

By the way, DAS became so unreliable that we stopped using it in our environment.

thatfish

When you say you have a lot more workflows - why is this ?

Have you introduced more users , more DCR types, more complex workflows , etc.

In my past experience of DAS it can get memory/CPU hungry with complex DCRs. There were ways to mitigate this. I'd hope this issue is resolved with later releases of OpenDeploy.

speck

We are running conversion workflows as part of things, to convert data, which was why load was higher, but this wasn't root cause.

However the issue was tracked down to corruption in the Event Subsystem.

We deleted the messages, message handles out of the DB and deleted the iwevents files, I believe (will double check and post back here) and restarted services and things went back to normal.

I wish I could claim credit, but TEamsite support actually helped us out, we did delete the messages and handles on our own, in our own attempt to debug, but never deleted the files on the disk. Apparently some sort of corruption happened causing it all to go haywire. SO the messages, handles and files all needed whiped and the services restarted to get everything back in sync.

Thanks for the help.