CSV Size limit

Hey guys,

Is there a limit to the size a csv data source can be?

I have a report which uses a csv data source and processes it into a few cross tabs. Whilst running some test data (which is about 1/20th the size of what i want the final product to be using) i get an error when running the report which states that the file is broken beyond repair.

However, if i delete a big chunk of the data and run it again, it works perfectly.

I know that the data that is in there is correct as i can run both halves of the test data individually and it runs fine.

I have pinned it down to the file "breaking" around 430Kb, or 16,000 lines

Thanks

Find more posts tagged with

Comments

mwilliams

Hi Adam,

Let me check into this and get back to you. I have not run into this problem myself, so I have never tested it out. What is your BIRT version in case things are different for different versions?

thuston

Can you also post the Error message, and if possible strip any confidential data and recreate and post your failing CSV and Design files?
I can 't think that there would be any built in limitation (only practical limitations like available Heap space for loading) and 16k rows is not very many.

Is the CSV on the machine running the BIRT report?
It seems that something else must be wrong in how you access the file.

AdamUK88

I've attached the csv data and the design file. My birt version is 2.3.1. also yes, the csv is on the same computer that is running the report.

If i run the report as a pdf (the desired output) i get a message saying "the file is damaged and could not be repaired" but if i run it in web viewer i get a "java.lang.NullPointerException" error. Running in HTML half creates the report, but stops half way through a cross tab to put an error in one of the cells

Thanks for your help guys

thuston

I also see the problem in Actuate BIRT 10Sp1 (2.3.2).<br />
As you said, if you reduce the data it finishes (thought the CrossTabs were empty for me).<br />
<br />
I have not found any quick solution.<br />
If the others don't see anything and your company has an agreement, please open a Support case with Actuate Support

AdamUK88

Yeh i was expecting the majority of the cross tabs to be empty, there was only 3 lines of data which flag as report worthy in that test data.

Thanks for checking for me. See if anyone else fancies a stab at this and if not we'll raise the issue.

Has anyone ever used a csv file bigger than this in a report?

mwilliams

Adam,

Try making a report without a crosstab. Just a simple table attached to your dataSet. Do you still get the error? I just created a 115000+ line csv and displayed it in a table with no issues. Over 2800 pages in the web viewer.

AdamUK88

Ok i tried that, and went up to a file with half a million lines (about the size of file the final product will be using). . . and everything went fine. I then put this into crosstabs and tried again and again it worked fine (although it started taking it's sweet time

). Unfortunately this means that something is broken in my original report

If any of you can make sense of the error reports i'd be grateful if you could try and pinpoint the error for me. failing that i'll try making the report from scratch and see where it breaks.

Thanks

mwilliams

Adam,

So, the flat file and report above produce the error you're talking about? If so, I'll take a look.