Generate report with big CSV

Hello,

If I write down not in correct forum - sorry.

I try generate report by runtime components and I have some CSV file with size of 137,3Mb and 1 959 892 strings.
In my log file I see some error:

11.12.2012 0:02:49 org.eclipse.birt.report.engine.api.impl.EngineTask handleFatalExceptions
SEVERE: There is insufficient memory to execute this report.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.lang.AbstractStringBuilder.substring(Unknown Source)
at java.lang.StringBuffer.substring(Unknown Source)
...

If I correctly understand and see some answer in internet - I must use options "-Xms1024m -Xmx2048m -XX:MaxPermSize=512m", but I can't understand where and how I must use this options in "genReport.bat"... maybe some people solved this problem? Now I try generate with this batch file:

===cut===
set BIRT_HOME=D:\BIRT\runtime

cd /d "C:\Program Files (x86)\Java\jre6\bin"

%BIRT_HOME%\ReportEngine\genReport.bat -f PDF -o

\BIRT\Reports\report_Daily_%DATE%.pdf

\BIRT\workspace\Daily.rptdesign
===cut===

Find more posts tagged with

Comments

kclark

Hubba -- I know that I've added the maxpermsize flags to eclipse.ini in the past. It looks like your just running the birt engine. What version of birt are you running?

hubba900

I use BIRT RCP 4.2.1 for create and test report. For automaticly generate report - I use BIRT runtime 4.2.1. I try add in the past of BIRT.ini parameters "-Xms256m-Xmx512m-XX:MaxPermSize=512m" - but is not solved my issue. Maybe CSV-file very big or I some do it wrong? How I can use this parameters in BIRT runtime?

gauglerb

<blockquote class='ipsBlockquote' data-author="'hubba900'" data-cid="112303" data-time="1355204438" data-date="10 December 2012 - 10:40 PM">
I use BIRT RCP 4.2.1 for create and test report. For automaticly generate report - I use BIRT runtime 4.2.1. I try add in the past of BIRT.ini parameters "-Xms256m-Xmx512m-XX:MaxPermSize=512m" - but is not solved my issue. Maybe CSV-file very big or I some do it wrong? How I can use this parameters in BIRT runtime? </blockquote>
 
How are you generating a CSV ? Thanks

hubba900

My CSV file is data export from SIEM platform. He has structure:

"Date/Time","SourceAddress","SourceHostName","SourcePort","DestinationAddress","DestinationPort","sum(Bytes)"
"2012-12-10 23","190.7.60.10","firewall.farmar.com.ar",1358,"10.0.0.1",25,294
....

Sometimes this file have less 400 000 strings (~ 20 Mb) and report generate successfully. But now I'd like generate report for more big time period - one week. This CSV file have ~ 2 000 000 strinags and have size 140 Mb.

I'd understand - I can use input file with 2 000 000 strings (~ 140 Mb) or is very more for BIRT?

Clement Wong

<blockquote class='ipsBlockquote' data-author="'hubba900'" data-cid="112292" data-time="1355170216" data-date="10 December 2012 - 01:10 PM">
... 
 
If I correctly understand and see some answer in internet - I must use options "-Xms1024m -Xmx2048m -XX:MaxPermSize=512m", but I can't understand where and how I must use this options in "genReport.bat"... maybe some people solved this problem? Now I try generate with this batch file: 
 </blockquote>
 
 
genReport.bat is the actual batch file where it calls the "java" command with the JVM option to run the ReportEngine. This file is where you would want to modify the Java options to increase the memory. 
 
Assuming those memory values are correct, you'll see these two lines towards the end of the genReport.bat file: 
<pre class='_prettyXprint _lang-auto _linenums:0'>JAVACMD='java';
$JAVACMD -cp "$BIRTCLASSPATH" -DBIRT_HOME="$BIRT_HOME/ReportEngine" -Dorg.eclipse.datatools_workspacepath="$org_eclipse_datatools_workspacepath" org.eclipse.birt.report.engine.api.ReportRunner ${1+"$@}
</pre>
 
And you'll want to add the additional memory options right before the -cp: 
<pre class='_prettyXprint _lang-auto _linenums:0'>JAVACMD='java';
$JAVACMD -Xms1024m -Xmx2048m -XX:MaxPermSize=512m -cp "$BIRTCLASSPATH" -DBIRT_HOME="$BIRT_HOME/ReportEngine" -Dorg.eclipse.datatools_workspacepath="$org_eclipse_datatools_workspacepath" org.eclipse.birt.report.engine.api.ReportRunner ${1+"$@}
</pre>
 
If you want to clean the script up a bit, you can add an options variable: 
<pre class='_prettyXprint _lang-auto _linenums:0'>JAVACMD='java';
JAVA_OPTS='-Xms1024m -Xmx2048m -XX:MaxPermSize=512m';
$JAVACMD $JAVA_OPTS -cp "$BIRTCLASSPATH" -DBIRT_HOME="$BIRT_HOME/ReportEngine" -Dorg.eclipse.datatools_workspacepath="$org_eclipse_datatools_workspacepath" org.eclipse.birt.report.engine.api.ReportRunner ${1+"$@}
</pre>
 
 
BTW, does the .rptdesign run in your designer with the large set of XML data? 
 
 
Regards, 
Clement Wong 
Actuate Corporation

hubba900

Yes! This that that is necessary. I think issue is solved. But at my laptop with 6GB RAM and Windows 7 64 bit I can use max "-Xms512m -Xmx768m -XX:MaxPermSize=512m" and with this options report not generate. I try exec with options "-Xms256m -Xmx1024m -XX:MaxPermSize=512m" ("-Xmx1024m" is max what I can use, else I have error message) at my server with 8Gb RAM and Windows 2003 R2 64 bit and after 27 minutes report was be generated.

When I try generate report in BIRT RCP - I can't do it with 400 000 strings, I use priview with limit first 5000 string only for testing structure, tables and charts.