Frameset servlet very slow

By looking around the Internet, I have seen this topic before, but nothing seems to offer a specific answer. I am running birt-viewer along with a web app on two machines: A test machine and a live machine. I'm using Tomcat as the app server. On the test machine, the viewer runs as expected - I just access the frameset link and I get the little animated "in progress" display for a few seconds and then the paginated report pops up. On the live server, I get the start of the page ("BIRT Report Viewer" heading and the green bar with the six icons on the left hand side) but no report for a long time - several minutes. No progress indicator at all.
 
The machines look to be identical:
 
Tomcat version: 7.0.55
Java version: 1.8.0_31-b13
birt-runtime version: 4.4.1
Linux: Mint 17 (kernel 3.13.0-24-generic - 64-bit) (Mint is a Ubuntu derivative)
 
The live server (that doesn't work) is standalone and has 8 gig of RAM. The test machine (that runs fine) is actually running in a VMware partition with 4gig of RAM.
 
The resource stats from the Tomcat Complete Server Status page show that the resources available on the live machine are higher than the resources on the test machine.
 
While looking around for others with the same problem, I found a note that said that if you replace "frameset" with "run" it would work right away. I tried that, and sure enough -- it does. But, I need to use frameset for both pagination as well as exporting options (not to mention the fact that I want to understand why apparently identical environments behave differently).
 
I saw other notes that said their problem was related to a Java version that was too old (Java 6.something, I believe). That is clearly not my problem. I was running Java 7u65 or so before and when it didn't work, I just upgraded to the latest rev from Oracle.
 
I have no idea where to look. The viewer appears to be acting like something that has to wait for a timeout before it proceeds. Has someone seen and solved this problem? Is there some way to see some important vitals within Tomcat or birt-viewer so that I can get to the bottom of this?
 
thx,
Vince

Find more posts tagged with

Comments

mwilliams

If you go to <a data-ipb='nomediaparse' href='http://hostinfo/birt'>http://hostinfo/birt</a>do you get the welcome screen? If you click on the test report from there, does that work? Both environments are exactly as you have listed above and only one works? Using the same browser? Do you get any information in the Tomcat console on the one that's not working? Are they pulling the same data or is the live server accessing different or much larger data?

vguarna

yes, same browser (Firefox version 36.0 running on a Mac). I am actually accessing the live server over the Internet in one tab and my test system locally (a different machine on my same subnet) in another tab. BTW, browser doesn't seem to matter. IE and Safari also have the same problem.
 
Same database. Maybe off by a few records here and there, but extremely close if not identical. Also very small. In one case, i try to run a report that has only about 100 lines in it. And that's 100 out of 100 records, not 100 out of a large number. The delay seems to be data-independent.
 
The welcome screen appeared to work. I get this screen (right away):
 
 
<a data-ipb='nomediaparse' href='http://www.eclipse.org/'><img height="50" src="http://74.92.188.138:8080/birt-viewer/webcontent/birt/images/EclipseBannerPic.jpg" width="115" alt="EclipseBannerPic.jpg"></a> <img height="50" src="http://74.92.188.138:8080/birt-viewer/webcontent/birt/images/gradient.jpg" width="300" alt="gradient.jpg"><a data-ipb='nomediaparse' href='http://www.eclipse.org/birt'>BIRT </a> BIRT viewer has been installed.
 
<img height="86" src="http://74.92.188.138:8080/birt-viewer/webcontent/birt/images/Idea.jpg" width="120" alt="Idea.jpg">
Thank you for your choosing BIRT (Business Intelligence Reporting Tool).
Viewer Version : 4.4.1
Engine Version: 4.4.1
JRE version: 1.8.0_31
<a data-ipb='nomediaparse' href='http://74.92.188.138:8080/birt-viewer/frameset?__report=test.rptdesign&sample=my+parameter'>View Example</a>
 
 
When I click on "View Example" it hangs. Actually, just stalls for a while, just like the other reports.While I was typing this (about 5 minutes) I got this:
 
<div>Title</div>
<div>Congratulations! 
If you can see this report, it means that the BIRT viewer is installed correctly. 
</div>
<div>Sample Parameter:</div>
<div>my parameter</div>
<div>Mar 5, 2015, 9:27 AM</div>

vguarna

By the way, for what it's worth, I don't think it always did this. When I first stood up the birt-viewer server, I think things worked OK. Then, it seemed to be intermittent (sometimes it would display reports right away, sometimes it would not.) My recollection (which could be faulty) is that when it was stalled and then finally did show a report, I could generate another report and it would display right away. Then, some other time...say an hour later...I would get a stall again.
 
Now it appears to reliably stall every time. So, it seems like it has been deteriorating over time.
 
I'm tempted to delete the viewer and reinstall. But, I was hoping to understand what is wrong before I take a crowbar to it.
 
I have also gone into WEB-INF in the birt-viewer folder under Tomcat web apps to make sure none of the config files (like web.xml) are different between the working and non-working system. They are the same.

mwilliams

What are you JVM settings on both? Are they the same? Have you restarted the app server on the live setup?

mwilliams

You also might try clearing the work folder in your Tomcat and let it recompile your applications when you restart the app server.

vguarna

Yes, Tomcat gets restarted pretty regularly (I do a complete shutdown and reboot of TC every time I load a new version of the app which is a couple of times per week.
 
Here's the JVM info from Tomcat on the broken (live) machine:
 
 

JVM
Free memory: 207.93 MB Total memory: 609.00 MB Max memory: 1774.50 MB

Memory Pool

Type

Initial

Total

Maximum

Used

PS Eden Space

Heap memory

32.00 MB

227.00 MB

632.00 MB

139.93 MB (22%)

PS Old Gen

Heap memory

84.00 MB

365.00 MB

1331.00 MB

253.70 MB (19%)

PS Survivor Space

Heap memory

5.00 MB

17.00 MB

17.00 MB

7.42 MB (43%)

Code Cache

Non-heap memory

2.43 MB

67.93 MB

240.00 MB

67.48 MB (28%)

Compressed Class Space

Non-heap memory

0.00 MB

16.75 MB

1024.00 MB

14.30 MB (1%)

Metaspace

Non-heap memory

0.00 MB

137.75 MB

-0.00 MB

128.27 MB
 
 
Here's the info from the working machine:

JVM
Free memory: 199.87 MB Total memory: 417.97 MB Max memory: 955.12 MB

Memory Pool

Type

Initial

Total

Maximum

Used

Eden Space

Heap memory

16.50 MB

115.43 MB

263.56 MB

13.01 MB (4%)

Survivor Space

Heap memory

2.06 MB

14.37 MB

32.87 MB

14.37 MB (43%)

Tenured Gen

Heap memory

41.37 MB

288.16 MB

658.68 MB

190.71 MB (28%)

Code Cache

Non-heap memory

2.43 MB

23.68 MB

240.00 MB

23.21 MB (9%)

Compressed Class Space

Non-heap memory

0.00 MB

10.37 MB

1024.00 MB

9.60 MB (0%)

Metaspace

Non-heap memory

0.00 MB

84.87 MB

-0.00 MB

82.31 MB
 
 
It also seems curious that the live machine shows the memory pool items as "PS Eden Space, PS..." whereas the test machine shows just as "Eden Space, Survivor....". But, the Server information block shows that it is the identical information: (Hostname left out)
 
 
Stalling Machine:

Server Information

Tomcat Version

JVM Version

JVM Vendor

OS Name

OS Version

OS Architecture

Hostname

IP Address

Apache Tomcat/7.0.55

1.8.0_31-b13

Oracle Corporation

Linux

3.13.0-24-generic

amd64

(deleted)

127.0.1.1
 
 
Good Machine:
 

Server Information

Tomcat Version

JVM Version

JVM Vendor

OS Name

OS Version

OS Architecture

Hostname

IP Address

Apache Tomcat/7.0.55

1.8.0_31-b13

Oracle Corporation

Linux

3.13.0-24-generic

amd64

(deleted)

127.0.1.1
 
 
I will try deleting the birt-viewer app and redeploying from a known good war file. I will also delete the work folder as you suggested
 
(I assume that means I should delete ...tomcat/work/Catalina/localhost/birt-viewer ?)

mwilliams

Yes, that's what I mean for the work folder. Though, you could delete the entire directory (just don't delete the webapps

). It should rebuild when you restart Tomcat. Any log info or info in the Tomcat console on the slow/non-working setup?

vguarna

Actually, I get a number of SEVERE message about birt-viewer on both machines. But, it does seem that I have more on the slow machine. I have tended to ignore them based on sniffing around forums on the Internet (and the fact that the test machine works). Bad practice I know. Here's an example:
 
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/birt-viewer] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/birt-viewer] registered the JDBC driver [org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager.WrappedDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/birt-viewer] appears to have started a thread named [Thread-10] but has failed to stop it. This is very likely to create a memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/birt-viewer] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/birt-viewer] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@29f795fd]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [org.eclipse.birt.report.context.BirtContext@4e6d6f91]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1b0cea4a]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [com.ibm.icu.impl.ResourceBundleWrapper@4032abe4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@55ff9662]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@29f795fd]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [org.eclipse.birt.report.context.BirtContext@70cb6aab]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@55ff9662]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@1c46b4fb]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@4fb30d1e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1b0cea4a]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [com.ibm.icu.impl.ResourceBundleWrapper@4032abe4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@29f795fd]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [org.eclipse.birt.report.context.BirtContext@7f37057]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@55ff9662]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@1c46b4fb]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@6d6e63ac]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1b0cea4a]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [com.ibm.icu.impl.ResourceBundleWrapper@4032abe4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@29f795fd]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [org.eclipse.birt.report.context.BirtContext@44a6f396]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1b0cea4a]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [com.ibm.icu.impl.ResourceBundleWrapper@4032abe4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@55ff9662]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

vguarna

No luck. I wiped out the app in Tomcat and also the work folder. Restarted Tomcat and it is still the same. It's stalled right now - waiting for the first report to display....

mwilliams

Very odd. You said this installation of BIRT has been up for a while, right? Has anything changed on your server at all around the time that BIRT started experiencing problems?

vguarna

Nothing that I'm aware of. The only thing that changes is the app, but that's separate from birt-viewer. And, it only connects with birt-viewer via the URL connection (<a data-ipb='nomediaparse' href='http://hostinfo/birt-viewer/frameset.....etc'>http://hostinfo/birt-viewer/frameset.....etc</a>
 
The only change I can think of is the Java change I just made. But, that was in response to the problem, not the other way around.
 
Are there any debugging modes that can be turned on to see where the viewer is spending its time? When I do the "complete server status" from Tomcat, I can see that the failing machine is spending hundreds of seconds in processing time (the /frameset, /run servlet section). The "working" system has only a few seconds. Looks like a busy loop running somewhere that finally gets satisfied (because the report eventually comes out) but I have no idea what keeps it spinning.

vguarna

I reckon the next thing I'll try is to tear down Tomcat and bring that up fresh.
 
Grasping at straws....

mwilliams

If everything is exactly the same between the VM and the standalone server, I couldn't think of anything other than a glitch somewhere. Seems very odd. I will say, I haven't tried Java 8 again after having seemingly similar issues to what you're seeing, now. I use 1.7.0_60. You might try downgrading to that to see if it works for you. As for diagnostics, you can change the logging level in the viewer web.xml. That might help you out some. You could also write script in a report to write out to the console and run it to test where it's getting hung up.

vguarna

BIRT is working now on the production system. After waking up this morning ready for some more experimentation, I started by trying to back rev Tomcat as you suggested. I went to the Oracle site and picked up 1.7.0_75 (I don't know how to get older versions like 7_60 as you suggested). I shutdown Tomcat changed the tomcat7/bin/setenv.sh script to look at the new JDK tree, and restarted Tomcat. Appeared to start at first so I tried to log into my app. Kaboom. outofmemory PermGen space. Try to restart Tomcat again. Won't start because ports are in use. Reboot the system. Restart Tomcat again. Same thing. PermGen first, then ports in use.
 
I was about to delete app trees for both birt-viewer and my app as well as work folders and try a couple of fresh deploys but decided that I first wanted to make sure that things were at least as good as when I woke up this morning. So, reboot one more time. Change JAVA_HOME back to point to 1.8.0_31, and start Tomcat again. The app was working again and this time, reports are working, too - they process right away (!).
 
I mentioned above that I restart Tomcat several times each week. But, I rarely reboot the underlying Linux system....could easily be months. So, right now I don't know if that's related or not. I figured that restarting Tomcat would be a fresh start with respect to system resources but maybe there is something else going on. That is certainly one difference between the production system and the test system. The test system, running in a VMware partition, gets a fresh reboot almost every time I use it (that is, a Linux reboot as well as a fresh Tomcat start).
 
So, I'll keep an eye on it now. I don't expect it will last but I'll enjoy it while I can. If I see a stalled report again, the first thing I will do is reboot Linux again to see whether that fixes the problem. Still seems very odd because even when the reports are slow, the app runs fine.
 
Thanks for the help, Michael.

mwilliams

Great to hear! Yeah, with Java 7, you'd need to go back and set the MaxPermSpace to something like 512m. With Java 8, there is no longer a perm gen space, so you don't have to worry about this. Let me know if you run into issues again.

AndroK0n

If on SERVER birt viewer is very slow respond to "frameset" request, it may be helpfull to disable (set to OFF) BIRT_VIEWER_PRINT_SERVERSIDE in WEB-INF/web.xml of birt viewer serlet. Or it may be possible to install CUPS and CUPS-PDF printer on server.