servletd using more memory than expected [TS6.5sp2]

Hi folks,

Solaris 9. TS 6.5 SP2 + Patch 1614.

I'm trying to control the memory usage of servletd on our DEV box, where we have limited memory (512M) and equally limited swap space... Teamsite often hangs on this machine, reporting "low on virtual memory, shutting down".

The other day, we changed in IWHOME/servletd/bin/servletd the JAVA_OPTS to say -Xms64m -Xmx256m (down from -Xmx512m).

I'm surprised to see that the process is still using quite a lot of memory... it started low, at about 160M or so, but after about a week, this is now 450M.

$ prstat -s size
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  6580 iwui      448M  269M sleep   29   10   0:13:55 0.0% java/45
  [...]

$ pargs  6580  

6580:   /extra/app/gwts/iw-home/tools/java/bin/java -server -Xms64m -Xmx256m -Djava.lib
argv[0]: /extra/app/gwts/iw-home/tools/java/bin/java
argv[1]: -server
argv[2]: -Xms64m
argv[3]: -Xmx256m
argv[4]: -Djava.library.path=/extra/app/gwts/iw-home/cssdk:/extra/app/gwts/iw-home/lib
argv[5]: -Dcssdk.cfg.path=/extra/app/gwts/iw-home/cssdk/cssdk.cfg
argv[6]: -Djava.endorsed.dirs=/extra/app/gwts/iw-home/servletd/common/endorsed
argv[7]: -classpath
argv[8]: /extra/app/gwts/iw-home/tools/java/lib/tools.jar:/extra/app/gwts/iw-home/tools/java/lib/tools.jar:/extra/app/gwts/iw-home/cssdk/cssdkjni.jar:/extra/app/gwts/iw-home/cssdk/cssdkiface.jar:/extra/app/gwts/iw-home/servletd/bin/bootstrap.jar:/extra/app/gwts/iw-home/servletd/bin/commons-logging-api.jar
argv[9]: -Dcatalina.base=/extra/app/gwts/iw-home/servletd
argv[10]: -Dcatalina.home=/extra/app/gwts/iw-home/servletd
argv[11]: -Djava.io.tmpdir=/extra/app/gwts/iw-home/servletd/temp
argv[12]: org.apache.catalina.startup.Bootstrap
argv[13]: start

A simple jsp page with

<% java.lang.Runtime rt=java.lang.Runtime.getRuntime(); %>
Current heap size: <%= rt.totalMemory() %>

Unused heap: <%= rt.freeMemory() %>

Max memory: <%= rt.maxMemory() %>

... reports that only 66MB is reserved as heap. Also, the -Xmx256m is confirmed.

Current heap size: 66879488
Unused heap: 15363640
Max memory: 265814016

My hands-on Tomcat experience used to be better some years ago, now I haven't played with them so much recently... still though, this strikes me as odd.

ideas?

thx, Fred.

Find more posts tagged with

Comments

chandraiw

still though, this strikes me as odd.

ideas?

thx, Fred.

I think it bites into swap space... isn't virtual address space atleast the physical memory plus swap space?

stefanmaier

Hi,
-Xms64m -Xmx256m configures sets the
initial Java heap size to 64MB and
maximum Java heap size to 256MB.

That doesn't contradict your given memory values, that 66MB are reserved as current heap size...
Note, that these used 66MB are memory consumption of your Java programs (your Java programs within your JavaVM did consume 66MB).
The java process itself, which runs on your Solaris server consumes 448MB. The -Xmx256m parameter doesn't restrict the memory consumption of this java process, at the best it gives a rough direction...

Frederik

Hi,
-Xms64m -Xmx256m configures sets the
initial Java heap size to 64MB and
maximum Java heap size to 256MB.

The java process itself, which runs on your Solaris server consumes 448MB. The -Xmx256m parameter doesn't restrict the memory consumption of this java process, at the best it gives a rough direction...

Ok, I may need to be more clear...
My initial heap is 64MB, and initially the memory taken from the OS by servletd (JVM) was about 160MB
After some days, my heap size is still 64 MB, but in the mean time servletd JVM is using 450MB out of the memory managed by the OS.

The increase from 160MB to 450MB is what I find odd, given that the heap didn't increase.

Frederik

I followed this issue some more: on the dev server, I restarted Teamsite entirely on 29 May. I did not touch teamsite at all, did not visit the web interface (the file IWLOGS/iwui/access_log is totally empty for this period). I did track the memory usage by servletd (the java process owned by iwui)... See attached image.

I saw the virtual memory size of the process grow from 150M to nearly 670M, at which time Teamsite gave up because:

Your system has run low on virtual memory (99 Mb is available)
[Tue Jun  6 10:57:01 2006] All stores are being deactivated

This excessive memory usage is bizarre, although I did read some scattered comments on the web about poor memory behaviour of Sun JVM on Solaris!? Does anybody else see similar memory usage of servletd (TS65) on Solaris?

--Fred.

stefanmaier

Hi,
the memory consumption of a non-touched servletd is really strage.
I suggest that you open a support case, perhaps they can bring some light on this...

Stefan

user_job_settings.JPG

jed

512M! Yikes! We just ditched a dev server because it only had 4Gigs of ram. You could always ad a swap file which is rather easy to do. Worst case scenario, add this as cron job Smiley Happy


#!/bin/sh

ALIVE=`PATH-TO-IWHOME/bin/iwstat | grep IWTRACE`
if [ "$ALIVE" = "" ]; then
  echo "TeamSite is alive"
else
umount -f /.iwmnt/default
umount -f /iwmnt/default
/etc/init.d/iw.server stop
/etc/init.d/iw.server start
   echo "TeamSite is dead"
fi

fiquebem

We're experiencing the same problem on our TS 6.5 SP2, Solaris 8 servers: servletd's memory usage keeps growing and growing. If servletd's size becomes larger than the amount of swap space left, one can't open a (FormsPublisher) form anymore.

We are also running Interwoven Search. You weren't, were you?

So, what happened in your case? Was Interwoven Support able to help you? What's the root cause of the memory leak?

We have a case open with Support ourselves. We asked them whether passing the following parameters to servletd would help:

-Xincgc: enable incremental garbage collection
-Xss: set java thread stack size

Here's what we were told by the technician who is handling our ticket:

"I don't think the setting will be helpful here as it seems like the
memory leak is with the native c++ code (Called from Java) and not with
Java Code. If it was with the java code we would have seen out of memory
error message in the servletd log and also the mem footage of servlet
will not grow more than the heap size specified."

Many thanks,

S.

John742

Have you looked at the logs to see what servletd may be doing? Is event subsystem enabled?

iportal_schedule.JPG

priv_setting_iserver.JPG

notification_tab.JPG

notification_tab_2.JPG

fiquebem

Thanks for your reply!

>Have you looked at the logs to see what servletd may be doing?

Servletd.log doesn't say much, except that it has messages such as:

2006-12-22 11:39:27,847 [main] WARN com.interwoven.ui.base.config.XMLMerger () - iwov-insert-before tag found but the element null id="iw.ccpro.file.versions.link" it referenced was not found

When the size of the servletd process exceeds the amount of free swap space and one attempts to open a form, servletd.log shows:

2006-12-22 10:56:11,354 [http-8080-Processor22] ERROR com.interwoven.ui.formspub.datacapture.AbstractOpenDCRCommand (sk73712 commandID iw.formspub.view_form) - A problem was encountered trying to read the datacapture.cfg file for type Product. Please contact your system administrator.
com.interwoven.ui.formspub.datacapture.dct.InlineCallout$InlineCalloutException: Not enough space

> Is event subsystem enabled?

Nope. Incidentally, servletd/logs/iweventproxy.log is logging the following error message every 5 minutes:

2006-12-22 13:38:21,494 [JMSBridge] ERROR com.interwoven.bridge100 () - Unable connect to IWOV JNDI provider.
2006-12-22 13:38:21,495 [JMSBridge] ERROR com.interwoven.bridge100 () - Msg: Failed to connect to JNDI provider

We asked Interwoven Support how to stop that and we were told the following: "Move the eventsubsystem.war file from iw-home/httpd/webapps to
another location and delete eventsubsystem directory." Seems too draconian to me! What do you think?

Which other logs should we look at?

Thanks,

S.