Problem with processing huge flat files(too slow)

alankar

Hi,

I am using the Birt 2.3 release to create a summary report which uses a pipe seperated flat file as the data source.
The report contains four tables with groups and a lot of aggerations and my flat file is huge approx 8 gb.
My problem is the that it takes a lot of time(more then 15Hr) for the generation of report. Also there are many files created in the temp foldar, Its creating 14Mb file starting from data_0_1 to data_0_919 and then data_1_0 which keeps on growing to Gbs.

I like to know is there a better way to do it, how can i optimize the report design so that it can generate report faster.

Please find the design file attached.

Thanks,

Alankar

mwilliams

Hi Alankar,

You could possibly speed up the processing of the data by doing a scripted data set that steps through the flat file. This would also allow you to handle the temp files differently. I haven't looked at the report design as of yet to see if I see anything you can do in it to decrease the time. If I see anything, I'll let you know.

Hope this helps.