Home
Analytics
OutOfMemory Nested Tables Using Filters
elbeau
Our product allows customers to add BIRT report design files and run them in-process in our system. One such file is causing OutOfMemory errors and it's hard to understand why. I've already solved the problem by creating the tables another way, but I'd really like to understand why this didn't work in the first place so I can give good advise to customers wanting to do similar things in the future.
The report runs off of a flat-file data source with multiple CSV tables. The report had one table nested inside another, and the outer table would set a BIRT variable that the inner table would use in a filter to get just the rows that the inner table wanted. BIRT's memory limit is set to 2048. The data files the report ran off of were relatively small, there were three tables, and the largest on was 2,200 rows, but when you ran the report in the BIRT designer the CPU would just spin and the memory would keep building up and up until it got the OutOfMemory error. The report ran fine on a smaller (1/7 size) data set.
Do nested tables using filters not get cleaned up properly or something? What am I missing here? Simply deleting the nested table made the report run, so it definitely has something to do with the nested table's filter that uses the variable set by the outer table.
Any ideas would be greatly appreciated.
Thanks
Find more posts tagged with
Comments
mwilliams
Hi elbeau,
This is probably because BIRT is bringing in the entire dataSet each time the nested table is called and filtering it down. This takes up a lot of memory and processing time. If you were using SQL and could take advantage of the dataSet parameter binding, this probably wouldn't have been an issue. The newest versions may help with this issue because of better caching. What BIRT version are you using?
elbeau
<blockquote class='ipsBlockquote' data-author="'mwilliams'" data-cid="71349" data-time="1292345635" data-date="14 December 2010 - 09:53 AM"><p>
Hi elbeau,<br />
<br />
This is probably because BIRT is bringing in the entire dataSet each time the nested table is called and filtering it down. This takes up a lot of memory and processing time. If you were using SQL and could take advantage of the dataSet parameter binding, this probably wouldn't have been an issue. The newest versions may help with this issue because of better caching. What BIRT version are you using?<br /></p></blockquote>
<br />
Thanks for the quick reply.<br />
<br />
I'm using 2.6.1. I figured it was bringing in the whole data set each time the nested dataset is run, but I still don't think that accounts for the large spike in memory. All the flat files together are less than 350k, but it dies while trying to use more than 2g of memory, so if I figure my JVM is already using 512m, then the memory footprint of the running report is more than 4200 times the size of the raw csv files. My guess would be that some reference to the data is being maintained by the nested table that is keeping the data from being garbage collected after each iteration of the nested table. If that is the case, and I am running the nested table about 700 times (which is accurate for what I'm doing) and the data set the nested table filters is about 307k (as raw csv), then a 1.5g (2g - 512 for the jvm) footprint is about 7 times the size of the raw data being utilized...which adds up a lot better with what I'm seeing.<br />
<br />
I know these numbers are far from scientific, but this sure looks like a memory leak. What do you think?
mwilliams
Sounds like it. You could log a bug for this at
http://www.birt-exchange.org/org/resources/bug-reporting/
and see what they say. You might search the bugs to see if there is something similar already reported first.
shahamar
<p>I have Very similar observation. I know this thread is old. We are currently on birt 4.3.2 and there seems to be quite a buit improvement on the way BIRT fetches data set for nested table.</p>
<p> </p>
<p>As per this thread, data set is cached and reused. However I still see Outofmemory quite often with this use case.</p>
<p>Our use case</p>
<p>1. Multisection report which has 4 different layout - Main List with 4 tabels each represent one section (i.e one layout)</p>
<p>2. All List including Dataset is bind to same resultset. List filters data with RecordType='Summary', just to get each item for which 4 tables needs to be shown.</p>
<p>3. All tables in list has filters set with RecordType='Detail' and few more columns to filter data only for this table.</p>
<p> </p>
<p>We run our webapp on 1G weblogic instance and see OutofMemory for report arround 8K rows.</p>
<p> </p>
<p>Do we know if there was any memory leak identified using Filters in Nested Table?</p>