Hi,<br />
<br />
I need to write a test case for my <a class='bbc_url' href='
http://www.spudsoft.co.uk/2011/10/the-spudsoft-birt-excel-emitters/'>Excel Emitter</a> that output a huge report (>256 columns, > 65000 rows).<br />
Because it's just a test case it needs to be quick and self contained.<br />
<br />
I started with a scripted data set and the performance was awful (25 minutes until <a class='bbc_url' href='
https://bitbucket.org/yaytay/spudsoft-birt-excel-emitters/issue/19/change-xlsx-emitter-to-use-sxssf-to-avoid'>my emitter crashed</a>), I've tried changing to a flat file data source and BIRT falls over claiming I have duplicate column names (I don't, they are Col1 - Col500) and the designer is still glacially slow (as few minutes delay every time anything is changed).<br />
This is on an 8 core, 8GB RAM, Vista machine, not a Commodore 64.<br />
<br />
Both the flat file and the scripted dataset seem to loading into memory completely and that is causing a huge number of GCs and using a vast amount of memory (the flat file is only 36MB).<br />
<br />
Is there any way I can get a self-contained, quick, report of this kind of size?<br />
<br />
Thanks.<br />
<br />
Jim<br />
<br />
Update: Just found that reading the flat file takes up to 0.1ms per byte! And for 36MB that's a silly amount of time. Why is it reading the file one byte at a time??<br />
<br />
Update2: Even if I change it to a JDBC connection it tries to read all the data in before sending it to the emitter. Surely there must be some way to turn this off, otherwise BIRT is severely limited for large reports.<br />
<br />
Update3: It seems that oda-jdbc is semi-hard-coded to get query metadata for MySQL queries by running the whole query and caching the results (it's in a config.xml file contained with oda-jdbc.jar). Which means that BIRT <em class='bbc'>is</em> severely limited for large reports from MySQL (and nearly all other JDBC drivers)
.
