Runnung TeamSite 7.3.2 with Java 64bit

Hi!

We are running TS 7.3.2 on a 64bit Linux machine. During the last couple of days the JBoss was running out of memory with the following message:

08:49:49,825 ERROR [Naming] Unexpected exception during accept
org.jboss.util.threadpool.ThreadPoolFullException: java.lang.OutOfMemoryError: unable to create new native thread
        at org.jboss.util.threadpool.BasicThreadPool.execute(BasicThreadPool.java:417)
        at org.jboss.util.threadpool.BasicThreadPool.runTaskWrapper(BasicThreadPool.java:192)
        at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:212)
        at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:206)
        at org.jnp.server.Main$AcceptHandler.run(Main.java:485)
        at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
at java.lang.Thread.run(Thread.java:619)

The JBoss is running with the following paramters:
-Xms2048m -Xmx2048m -XX Smiley Tongue ermSize=512m -XX:MaxPermSize=768m

Is it possible to run this TeamSite version with Java 64bit ? (i.e. jdk-7u60-linux-x64)

The configfile “/iw-home/ApplicationContainer/bin/iw_jboss_run.conf" provides an (commented out) option to specify the JAVA location.

Using Java 64bit would let us increase the Java heap size.

I appreciate any help!

Kind regards

Tim

Find more posts tagged with

Comments

Rick Poulin

64bit operation is not supported (at least, not until TeamSite 8). It might work, but you'd be better off finding the actual source of the problem because you're just delaying the inevitable otherwise.

nipper

Agree with Rick here. While loaded servers will consume a pretty decent amount of memory, once you tweak the JBOSS parameters, you are usually pretty solid.

Do you have a lot of custom Java ?

tim_anselmi

Hi,

Thank you for the quick response!

I’ve modified the Java parameter and reduced the MaxPermSize from 768 to 512 to reduce the possibility to reach the maximum of allocable memory (32bit). Additionally I’ve added parameters to get a heap dump which may help me to analyze the root cause.

-Xms2048m -Xmx2048m -XX: PermSize=256m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app/

Do you have other tweaks you could recommend?

Regarding custom Java:

Currently we are using the standard CCPro (no SitePublisher) and FormsPublisher to create/generate pages. But we have implemented a Java submit workflow which does several things when submitting files (generating additional files, dumping databases).

Our TS setup contains a huge number of branches and many users are working with the system every day.

Could be the large amount of branches und users a root cause?

Do you know an approximate date when TS 8 with 64bit will be released?

Tim

Rick Poulin

Depends on what you consider 'large', but I've built implementations with 500 branches and 100 concurrent users and it's been fine (although you do have to intentionally keep your code optimized for performance).

Given that you're not using SitePub, most of my usual pointers wouldn't apply but it also means that the pieces that consume the most memory aren't being used.... so you must be doing something wrong somewhere. Have you analyzed the heap dump to determine what's consuming the most space? Are you holding on to large objects or large collections longer than you have to?

The date I heard back in december was that TS8 should be available in August.. but that'll be a dot-0 version, which is typically not great for production use, so it'll be a while before this is a solution for you.

tim_anselmi

Since my modification of the jvm parameter the jboss 'sadly' did not crash. I'm waiting for the next crash to analyse it.

regarding size/user:

i.e. today 130 users were working with our system and the TS store has > 2000 branches ( grown over the past 5 years)

regarding TS8: you are right, this version should not be used in prod.

I'll get back as soon I've managed to analyze the next heap dump.

Thx!

Rick Poulin

FYI, there's a really useful project for live java webapp monitoring called javamelody. It's extremely easy to integrate, and it introduces very little overhead. Amongst other things, it has a button to generate a heap dump whenever you feel like it.

nipper

Typically the changes I make are due to SP/LS and not TS/WF related.

You will have to ask HP about TS 8, but I would expect to see a major release at the earliest late this year but more likely next year.

From your description, it does not sound like much custom Java, is that correct ?

tim_anselmi

javamelody is nice! Smiley Happy Thank you for the tipp!

Now I've setup jm and I'll observe the memory usage. Generating heap dumps is working fine as well.

tim_anselmi

Hi @all,

We’ve observed the memory usage using javamelody (thx a ton for this tip!).

Finally we found a memory leak in our submit workflow. Our submitworkflow is working with CSSessions which were not closed manually and so the CSSessionCache was increasing until JBoss has reached the memory limit.

Now the memory usage is normal again.

Thank you for your advices!

Kind regards

Tim

Adam Stoller

tim_anselmi wrote:
[...] Our submitworkflow is working with CSSessions which were not closed manually and so the CSSessionCache was increasing until JBoss has reached the memory limit.
Now the memory usage is normal again.[...]

Just to be clear - this submit workflow you were having issues with - this is a custom workflow, or an OOTB workflow?

tim_anselmi

this workflow is a custom implementation. The workflow has been build with TS workflow modeler and the tasks are custom java code.

Just FYI:
When doing something like this:

...

CSSessionCache sc = CSSessionCache.getInstance();
CSSession session = sc.getSession(csContext);

Never forget to release the session!!!:
sc.releaseSession(session);

Otherwise the CSSessionCache increases heavily because it does not release the sessions automatically - seems so.

regards,

Tim

GeoffD

Great tip on JavaMelody, also running into the same problem. Thanks for the helpful information here!

Garth McMillan

Just to answer the original question, although the conversation has moved on: We have run a few different versions of TeamSite: 7.4.1, 7.3.1 and possibly 6.7.2 SP2 (not 100% sure about 6.7.2) on 64-bit JVM without any percevable issues.