Thread: Bizarre Issue with Scanner Failing to "notice" rest of file

I'm working on a rather irritating problem, and was hoping someone might have gone through something similar.

I'm trying to convert a massive amount of microsoft excel data into a format easy to read for Flash AS3, with
some manipulation required. So I figured I'd write a very simple parser in Java taking in a .csv file, make the
necessary little tweaks on the way and I'd be done.

What a fool I was! After almost 4 hours of errors I've run into I'm wondering if it wouldn't be faster to do it manually.
It's essentially working now, thank heavens but I've hit one massive problem that has got me completely perplexed,
and wondering if there's something strange going on with my data or I'm not realizing how to use the Scanner properly.

After about cell 40, the scanner completely fails to recognize any more of the many hundreds of cells after that line. I
thought it might be because it has a limited buffer or something so I cut down the file size but after that line it's as
though the file is invisible- the scanner opens fine, no IO exception but the second you call readLine command I get
a NoSuchElementException, informing me that the file I provided must be completely empty. I do this and open up
the file with thousands more lines to read and scratch my head in confusion.

I won't worry with a code sample because it really seems more like a data problem- really it's just opening the file
and reading it using a scanner, which we all know how to do. Has anyone ever tried parsing large files with a scanner
and hit this problem, and if so can you suggest a solution?

Thanks a lot for reading, and I'd be very appreciative of a response. There's no mad rush to solve the problem, I can
use the 40 lines I have now to demonstrate the core of the program is working, but I will need those other lines
eventually and I really don't want to manipulate them by hand, especially if there's more coming!

NitrogenFingers

Well, why don't you try converting your file date to byteArray and then start parsing it?

For some help I need to see the line with the problem and how do your scanner work? If it is possible...

For your interest RmSc I've included a SSCCE, though the code works fine for a portion of the file:

 
Scanner reader;
String filename = "";
try {
    filename = args[0] + ".csv";
    reader = new Scanner(new File(filename));
} catch (FileNotFoundException fnfe) {
     System.err.println("File not found : " + filename);
     System.exit(1);
} catch (IOException ioe) {
     System.err.println("Cannot write- IO exception occurred");
     System.exit(1);
}
 
//Clears the first line, the headers in the excel spreadsheet
reader.nextLine();
//Error occurs at this line of the code

Mr 777, thanks for your reply, I had a go at using a FileReader and then a BufferedReader to get
a ByteBuffer, but once again I just get an empty buffer when calling the reader.read(cb) method.
Or have I misinterpreted?

Well, the problem may be the lack of the memory but JVM must throw an exception about OutofMemory.
Well, JVM is memory limited so i guess there are different memory limits for 32 bit and 64 bit operating system. You are trying to run it on 32 bit OS?
Well, there can be a soltuion to your problem and that is; parse the file without reading it to the memory.
You can use Runtime Environment using System commands, that will be efficient, and might solve your problem too.

I try this with a file .cvs and work well... Maybe I need to see your file .cvs or you can print the line with the problem...

Thanks for your response Mr. 777. I'm running this on a 64 bit machine, and a fairly powerful one at that. Changing memory allocation to stack and heap
in Java through command line options hasn't changed my results.

A memory issue was my first suspicion, which is why I reduced the file size to a subset of the data I already had- in this instance the same thing happened,
suggesting that the particular data I was looking at was somehow less visible to Java than the rest of it. Disappointingly, this makes up most of the data I have.

Nevertheless I will try a non-buffered read- will post again if that works or I solve the issue some other way, but for now off to bed.

Thanks again,
Nitrogen Fingers

Sure, it doesn't seem to be a memory issue as no such error generated by JVM, well, you might give it a try without buffering into the memory and check the results. Good Luck.

Where exactly are you actually READING the lines of the file with the Scanner? I only see a nextLine() call...

nextLine() returns a String that you aren't capturing. You commented that it was skipping over the header line... ok great. Now where is the reading code?

Originally Posted by bgroenks96

Where exactly are you actually READING the lines of the file with the Scanner? I only see a nextLine() call...

nextLine() returns a String that you aren't capturing. You commented that it was skipping over the header line... ok great. Now where is the reading code?

reader.nextLine();

Here.... and i hope you read the whole post......