nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Temporary data to disc

HenkT

Hi, 
 
I have created a report to compare 2 datasets from 2 different sources. 
This report is not working when the 2 datasets have a large number of rows returned. 
 
I think the 2 datasets are stored in memory and there is not enough memory available to store both datasets. 
 
Is there a solution to use with BIRT in order to store both datasets on disk instead of memory. 
 
Thanks 
Henk

Find more posts tagged with

Comments

kclark

Is this happening when you deploy it in tomcat or when you preview the report? How many rows of data will it work with until it stops working?

HenkT

It happens on Tomcat, but also within the BIRT RCP.
When both datasets have about 100K records it fails.
I don't know the exact number which still succeeds, but the 100K records will grow in time, and it would be very nice if I could use this report to compare both datasets. The report is working fine for datasets with about 1000 records from both sources.

kclark

Are you getting any errors you can post? What do you have your maxpermgen set to?

CBR

In addition to kclark: 
There was a change with BIRT 3.7. Before 3.7 BIRT was swaping data to disc if a dataset was consuming more then 20MB of RAM. This has been changed to unlimited starting with BIRT 3.7 causing outofmemory exceptions if you have very large datasets. There is a parameter on the report engine that can be set to enable birt to swap data to disc if it exceeds a certain amount of RAM. 
Additionaly most jdbc driver do not stream the resultset by default which means that the driver first loads all records into memory before even processing the first one. 
 
I did some tests and if you enable swapping and resultset streaming for the driver BIRT can process tons of data in a reasonable timeframe. Just taking care of jdbc streaming or BIRT memory limit will not help and will cause you to run into memory issues again on very large datasets. 
 
It makes sense to first have a look at your memory settings including heap size. In most cases it is sufficient to increase the java heap size to be able to process more data. But again you will run into issues because of BIRT 3.7 or jdbc drivers default behaviour.