Frameset servlet very slow

By looking around the Internet, I have seen this topic before, but nothing seems to offer a specific answer. I am running birt-viewer along with a web app on two machines: A test machine and a live machine. I'm using Tomcat as the app server. On the test machine, the viewer runs as expected - I just access the frameset link and I get the little animated "in progress" display for a few seconds and then the paginated report pops up. On the live server, I get the start of the page ("BIRT Report Viewer" heading and the green bar with the six icons on the left hand side) but no report for a long time - several minutes. No progress indicator at all.


 


The machines look to be identical:


 


Tomcat version: 7.0.55


Java version:  1.8.0_31-b13


birt-runtime version: 4.4.1


Linux: Mint 17 (kernel 3.13.0-24-generic - 64-bit) (Mint is a Ubuntu derivative)


 


The live server (that doesn't work) is standalone and has 8 gig of RAM. The test machine (that runs fine) is actually running in a VMware partition with 4gig of RAM.


 


The resource stats from the Tomcat Complete Server Status page show that the resources available on the live machine are  higher than the resources on the test machine.


 


While looking around for others with the same problem, I found a note that said that if you replace "frameset" with "run" it would work right away. I tried that, and sure enough -- it does. But, I need to use frameset for both pagination as well as exporting options (not to mention the fact that I want to understand why apparently identical environments behave differently).


 


I saw other notes that said their problem was related to a Java version that was too old (Java 6.something, I believe). That is clearly not my problem. I was running Java 7u65 or so before and when it didn't work, I just upgraded to the latest rev from Oracle.


 


I have no idea where to look. The viewer appears to be acting like something that has to wait for a timeout before it proceeds. Has someone seen and solved this problem? Is there some way to see some important vitals within Tomcat or birt-viewer so that I can get to the bottom of this?


 


thx,


Vince


Comments

  • mwilliamsmwilliams BIRT Guru
    If you go to http://hostinfo/birtdo you get the welcome screen? If you click on the test report from there, does that work?

    Both environments are exactly as you have listed above and only one works? Using the same browser? Do you get any information in the Tomcat console on the one that's not working? Are they pulling the same data or is the live server accessing different or much larger data?

    Regards,

    Michael Williams

    eSignLive Evangelism & Community Manager | eSignLive by VASCO
     
    Find me on:
    Twitter
    Facebook
    Blog

    LinkedIn

    eSignLive Developer Community
     
    Email me:
    Google: [email protected]

  • edited March 2015

    yes, same browser (Firefox version 36.0 running on a Mac). I am actually accessing the live server over the Internet in one tab and my test system locally (a different machine on my same subnet) in another tab. BTW, browser doesn't seem to matter. IE and Safari also have the same problem.


     


    Same database. Maybe off by a few records here and there, but extremely close if not identical. Also very small. In one case, i try to run a report that has only about 100 lines in it. And that's 100 out of 100 records, not 100 out of a large number. The delay seems to be data-independent.


     


    The welcome screen appeared to work. I get this screen (right away):


     


     


    EclipseBannerPic.jpg gradient.jpgBIRT BIRT viewer has been installed.


     


    Idea.jpg


    Thank you for your choosing BIRT (Business Intelligence Reporting Tool).


    Viewer Version : 4.4.1


    Engine Version: 4.4.1


    JRE version: 1.8.0_31


    View Example


     


     


    When I click on "View Example" it hangs. Actually, just stalls for a while, just like the other reports.While I was typing this (about 5 minutes) I got this:


     


    Title

    Congratulations!


    If you can see this report, it means that the BIRT viewer is installed correctly.

     

    Sample Parameter:

    my parameter

    Mar 5, 2015, 9:27 AM

  • By the way, for what it's worth, I don't think it always did this. When I first stood up the birt-viewer server, I think things worked OK. Then, it seemed to be intermittent (sometimes it would display reports right away, sometimes it would not.) My recollection (which could be faulty)  is that when it was stalled and then finally did show a report, I could generate another report and it would display right away. Then, some other time...say an hour later...I would get a stall again.


     


    Now it appears to reliably stall every time. So, it seems like it has been deteriorating over time.


     


    I'm tempted to delete the viewer and reinstall. But, I was hoping to understand what is wrong before I take a crowbar to it.


     


    I have also gone into WEB-INF in the birt-viewer folder under Tomcat web apps to make sure none of the config files (like web.xml) are different between the working and non-working system. They are the same.


  • mwilliamsmwilliams BIRT Guru
    What are you JVM settings on both? Are they the same? Have you restarted the app server on the live setup?

    Regards,

    Michael Williams

    eSignLive Evangelism & Community Manager | eSignLive by VASCO
     
    Find me on:
    Twitter
    Facebook
    Blog

    LinkedIn

    eSignLive Developer Community
     
    Email me:
    Google: [email protected]

  • mwilliamsmwilliams BIRT Guru
    You also might try clearing the work folder in your Tomcat and let it recompile your applications when you restart the app server.

    Regards,

    Michael Williams

    eSignLive Evangelism & Community Manager | eSignLive by VASCO
     
    Find me on:
    Twitter
    Facebook
    Blog

    LinkedIn

    eSignLive Developer Community
     
    Email me:
    Google: [email protected]

  • Yes, Tomcat gets restarted pretty regularly (I do a complete shutdown and reboot of TC every time I load a new version of the app which is a couple of times per week.


     


    Here's the JVM info from Tomcat on the broken (live) machine:


     


     



    JVM

    Free memory: 207.93 MB Total memory: 609.00 MB Max memory: 1774.50 MB



    Memory Pool

    Type

    Initial

    Total

    Maximum

    Used

    PS Eden Space

    Heap memory

    32.00 MB

    227.00 MB

    632.00 MB

    139.93 MB (22%)

    PS Old Gen

    Heap memory

    84.00 MB

    365.00 MB

    1331.00 MB

    253.70 MB (19%)

    PS Survivor Space

    Heap memory

    5.00 MB

    17.00 MB

    17.00 MB

    7.42 MB (43%)

    Code Cache

    Non-heap memory

    2.43 MB

    67.93 MB

    240.00 MB

    67.48 MB (28%)

    Compressed Class Space

    Non-heap memory

    0.00 MB

    16.75 MB

    1024.00 MB

    14.30 MB (1%)

    Metaspace

    Non-heap memory

    0.00 MB

    137.75 MB

    -0.00 MB

    128.27 MB

     


     


    Here's the info from the working machine:



    JVM

    Free memory: 199.87 MB Total memory: 417.97 MB Max memory: 955.12 MB



    Memory Pool

    Type

    Initial

    Total

    Maximum

    Used

    Eden Space

    Heap memory

    16.50 MB

    115.43 MB

    263.56 MB

    13.01 MB (4%)

    Survivor Space

    Heap memory

    2.06 MB

    14.37 MB

    32.87 MB

    14.37 MB (43%)

    Tenured Gen

    Heap memory

    41.37 MB

    288.16 MB

    658.68 MB

    190.71 MB (28%)

    Code Cache

    Non-heap memory

    2.43 MB

    23.68 MB

    240.00 MB

    23.21 MB (9%)

    Compressed Class Space

    Non-heap memory

    0.00 MB

    10.37 MB

    1024.00 MB

    9.60 MB (0%)

    Metaspace

    Non-heap memory

    0.00 MB

    84.87 MB

    -0.00 MB

    82.31 MB

     


     


    It also seems curious that the live machine shows the memory pool items as "PS Eden Space, PS..." whereas the test machine shows just as "Eden Space, Survivor....". But, the Server information block shows that it is the identical information: (Hostname left out)


     


     


    Stalling Machine:



    Server Information

    Tomcat Version

    JVM Version

    JVM Vendor

    OS Name

    OS Version

    OS Architecture

    Hostname

    IP Address

    Apache Tomcat/7.0.55

    1.8.0_31-b13

    Oracle Corporation

    Linux

    3.13.0-24-generic

    amd64

    (deleted)

    127.0.1.1

     


     


    Good Machine:


     



    Server Information

    Tomcat Version

    JVM Version

    JVM Vendor

    OS Name

    OS Version

    OS Architecture

    Hostname

    IP Address

    Apache Tomcat/7.0.55

    1.8.0_31-b13

    Oracle Corporation

    Linux

    3.13.0-24-generic

    amd64

    (deleted)

    127.0.1.1

     


     


    I will try deleting the birt-viewer app and redeploying from a known good war file. I will also delete the work folder as you suggested


     


    (I assume that means I should delete ...tomcat/work/Catalina/localhost/birt-viewer ?)


  • mwilliamsmwilliams BIRT Guru
    Yes, that's what I mean for the work folder. Though, you could delete the entire directory (just don't delete the webapps ;) ). It should rebuild when you restart Tomcat.

    Any log info or info in the Tomcat console on the slow/non-working setup?

    Regards,

    Michael Williams

    eSignLive Evangelism & Community Manager | eSignLive by VASCO
     
    Find me on:
    Twitter
    Facebook
    Blog

    LinkedIn

    eSignLive Developer Community
     
    Email me:
    Google: [email protected]

  • Actually, I get a number of SEVERE message about birt-viewer on both machines. But, it does seem that I have more on the slow machine. I have tended to ignore them based on sniffing around forums on the Internet (and the fact that the test machine works). Bad practice I know. Here's an example:


     


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc


    SEVERE: The web application [/birt-viewer] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc


    SEVERE: The web application [/birt-viewer] registered the JDBC driver [org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager.WrappedDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads


    SEVERE: The web application [/birt-viewer] appears to have started a thread named [Thread-10] but has failed to stop it. This is very likely to create a memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads


    SEVERE: The web application [/birt-viewer] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads


    SEVERE: The web application [/birt-viewer] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [[email protected]fb]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [[email protected]fb]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [[email protected]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


    Mar 05, 2015 12:26:14 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks


    SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [[email protected]]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


  • No luck. I wiped out the app in Tomcat and also the work folder. Restarted Tomcat and it is still the same. It's stalled right now - waiting for the first report to display....


  • mwilliamsmwilliams BIRT Guru
    Very odd. You said this installation of BIRT has been up for a while, right? Has anything changed on your server at all around the time that BIRT started experiencing problems?

    Regards,

    Michael Williams

    eSignLive Evangelism & Community Manager | eSignLive by VASCO
     
    Find me on:
    Twitter
    Facebook
    Blog

    LinkedIn

    eSignLive Developer Community
     
    Email me:
    Google: [email protected]

  • Nothing that I'm aware of. The only thing that changes is the app, but that's separate from birt-viewer. And, it only connects with birt-viewer via the URL connection (http://hostinfo/birt-viewer/frameset.....etc


     


    The only change I can think of is the Java change I just made. But, that was in response to the problem, not the other way around.


     


    Are there any debugging modes that can be turned on to see where the viewer is spending its time? When I do the "complete server status" from Tomcat, I can see that the failing machine is spending hundreds of seconds in processing time (the /frameset, /run servlet section). The "working" system has only a few seconds. Looks like a busy loop running somewhere that finally gets satisfied (because the report eventually comes out) but I have no idea what keeps it spinning.


  • I reckon the next thing I'll try is to tear down Tomcat and bring that up fresh.


     


    Grasping at straws....


  • mwilliamsmwilliams BIRT Guru
    If everything is exactly the same between the VM and the standalone server, I couldn't think of anything other than a glitch somewhere. Seems very odd. I will say, I haven't tried Java 8 again after having seemingly similar issues to what you're seeing, now. I use 1.7.0_60. You might try downgrading to that to see if it works for you.

    As for diagnostics, you can change the logging level in the viewer web.xml. That might help you out some. You could also write script in a report to write out to the console and run it to test where it's getting hung up.

    Regards,

    Michael Williams

    eSignLive Evangelism & Community Manager | eSignLive by VASCO
     
    Find me on:
    Twitter
    Facebook
    Blog

    LinkedIn

    eSignLive Developer Community
     
    Email me:
    Google: [email protected]

  • BIRT is working now on the production system. After waking up this morning ready for some more experimentation, I started by trying to back rev Tomcat as you suggested. I went to the Oracle site and picked up 1.7.0_75 (I don't know how to get older versions like 7_60 as you suggested).  I shutdown Tomcat changed the tomcat7/bin/setenv.sh script to look at the new JDK tree, and restarted Tomcat. Appeared to start at first so I tried to log into my app. Kaboom. outofmemory PermGen space. Try to restart Tomcat again. Won't start because ports are in use. Reboot the system. Restart Tomcat again. Same thing. PermGen first, then ports in use.


     


    I was about to delete app trees for both birt-viewer and my app as well as work folders and try a couple of fresh deploys but decided that I first wanted to make sure that things were at least as good as when I woke up this morning. So, reboot one more time. Change JAVA_HOME back to point to 1.8.0_31, and start Tomcat again. The app was working again and this time, reports are working, too - they process right away (!).


     


    I mentioned above that I restart Tomcat several times each week. But, I rarely reboot the underlying Linux system....could easily be months. So, right now I don't know if that's related or not. I figured that restarting Tomcat would be a fresh start with respect to system resources but maybe there is something else going on. That is certainly one difference between the production system and the test system. The test system, running in a VMware partition, gets a fresh reboot almost every time I use it (that is, a Linux reboot as well as a fresh Tomcat start).


     


    So, I'll keep an eye on it now. I don't expect it will last but I'll enjoy it while I can. If I see a stalled report again, the first thing I will do is reboot Linux again to see whether that fixes the problem. Still seems very odd because even when the reports are slow, the app runs fine.


     


    Thanks for the help, Michael.

  • mwilliamsmwilliams BIRT Guru
    Great to hear!

    Yeah, with Java 7, you'd need to go back and set the MaxPermSpace to something like 512m. With Java 8, there is no longer a perm gen space, so you don't have to worry about this.

    Let me know if you run into issues again.

    Regards,

    Michael Williams

    eSignLive Evangelism & Community Manager | eSignLive by VASCO
     
    Find me on:
    Twitter
    Facebook
    Blog

    LinkedIn

    eSignLive Developer Community
     
    Email me:
    Google: [email protected]

  • If on SERVER birt viewer is very slow respond to "frameset" request, it may be helpfull to disable (set to OFF) BIRT_VIEWER_PRINT_SERVERSIDE in WEB-INF/web.xml of birt viewer serlet. Or it may be possible to install CUPS and CUPS-PDF printer on server.

Sign In or Register to comment.