Thread: Does the Scanner class use an internal buffer?

Your jdk folder should contain src.zip. Check that out, find the Scanner class, and you can see for yourself exactly what it does. I'd be curious to see what you find.

It depends on how your system is setup, but for example my JDK folder is:

C:\Program Files\Java\jdk1.7.0_07

In that directory, I have a src.zip, and inside that I have a Scanner.java file (inside java/util within the zip).

But I will say that the short answer to your question, from reading the source, is that Scanner uses a CharBuffer internally.

Originally Posted by Fazan

Yeah, I checked my Java folder, but all it has is JRE folders for Java 6 and 7 (I'm still using 6, by the way, legacy stuff). No JDK. As far as Windows 7 can be trusted, I ran a search just in case I wasn't aware of where my Java folders were, but no src.zip turned up.

Strange. What jdk are you using?

Either way, you can download the source either as part of the full JDK or standalone from this page: Java SE Downloads

Originally Posted by Fazan

At the risk of revealing that I'm not entirely certain I know what a Java Development Kit is, I use Eclipse for code-writing purposes and that's about the extent of it. Beyond that, my only other Java-related software are the JRE packages, and even then I only have 6 and 7.

Either way, thank you for the link. I'll ook into it.

Okay, gotcha. I assume that eclipse has its own JDK tucked away somewhere, but a JDK is simply the set of tools that compile your java code into bytecode (namely the javac tool). Compare that to the JRE, which is the set of tools that run compiled bytecode. Eclipse might hide it, but behind the scenes it has to be using a JDK, even if it's not installed where you'd expect. Most people install a JDK like I have above so that they can compile with the command prompt. The source comes with the "real" (non-eclipse) JDK, which is another reason it's a good thing to have.

As far as I know the Scanner class is a convenient Regex-wrapper for any incoming stream. What stream you pass to it will determine if the file is buffered in memory or not.

Scanner buffed_reader = new Scanner(new BufferedReader(new FileReader("file.txt"))); // file gets buffered into memory, usually faster
Scanner unbuffed_reader = new Scanner(new FileReader("file.txt")); // file isn't buffered into memory

That being said, I don't know what the behavior is if you pass a file to the Scanner object directly (though it sounds like it isn't).

It's also possible that it's the Regex side which is slowing your application down. If all you need is basic read-line stuff Scanners may be overkill. Scanners work best when you're parsing the data while your reading it in (such as reading in a file of numbers).

Originally Posted by helloworld922

As far as I know the Scanner class is a convenient Regex-wrapper for any incoming stream. What stream you pass to it will determine if the file is buffered in memory or not.

Scanner buffed_reader = new Scanner(new BufferedReader(new FileReader("file.txt"))); // file gets buffered into memory, usually faster
Scanner unbuffed_reader = new Scanner(new FileReader("file.txt")); // file isn't buffered into memory

That being said, I don't know what the behavior is if you pass a file to the Scanner object directly (though it sounds like it isn't).

It's also possible that it's the Regex side which is slowing your application down. If all you need is basic read-line stuff Scanners may be overkill. Scanners work best when you're parsing the data while your reading it in (such as reading in a file of numbers).

From the source, the Scanner(File) constructor wraps the File in a FileInputStream:

    public Scanner(File source) throws FileNotFoundException {
        this((ReadableByteChannel)(new FileInputStream(source).getChannel()));
    }

Which is converted into a Readable:

    public Scanner(ReadableByteChannel source) {
        this(makeReadable(Objects.requireNonNull(source, "source")),
             WHITESPACE_PATTERN);
    }

And finally passed into the main constructor:

    private Scanner(Readable source, Pattern pattern) {
        assert source != null : "source should not be null";
        assert pattern != null : "pattern should not be null";
        this.source = source;
        delimPattern = pattern;
        buf = CharBuffer.allocate(BUFFER_SIZE);
        buf.limit(0);
        matcher = delimPattern.matcher(buf);
        matcher.useTransparentBounds(true);
        matcher.useAnchoringBounds(false);
        useLocale(Locale.getDefault(Locale.Category.FORMAT));
    }

...which creates the CharBuffer. The CharBuffer is used in the base method for reading input:

    private void readInput() {
        if (buf.limit() == buf.capacity())
            makeSpace();
 
        // Prepare to receive data
        int p = buf.position();
        buf.position(buf.limit());
        buf.limit(buf.capacity());
 
        int n = 0;
        try {
            n = source.read(buf);
        } catch (IOException ioe) {
            lastException = ioe;
            n = -1;
        }
 
        if (n == -1) {
            sourceClosed = true;
            needInput = false;
        }
 
        if (n > 0)
            needInput = false;
 
        // Restore current position and limit for reading
        buf.limit(buf.position());
        buf.position(p);
    }

I suppose I should have performed benchmarks before making assumptions

import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.PrintStream;
import java.util.Scanner;
 
public class StreamTest
{
	public static void gen_file(String file_name, int lines) throws FileNotFoundException
	{
		PrintStream out = new PrintStream(new BufferedOutputStream(new FileOutputStream(file_name)));
		for (int i = 0; i < lines; ++i)
		{
			out.print("header,");
			for (int j = 0; j < 5; ++j)
			{
				out.print(i * 0.33f + j + ",");
			}
			out.println();
		}
		out.close();
	}
 
	public static void main(String[] args) throws IOException
	{
		String file_name = "data.txt";
		gen_file(file_name, 100000);
		Scanner scan;
		BufferedReader read;
 
		int times = 10;
		long scan_times[] = new long[times];
		long buff_times[] = new long[times];
		System.out.println("scanner test");
		for (int i = 0; i < times; ++i)
		{
			scan = new Scanner(new BufferedReader(new FileReader(file_name)));
			scan.useDelimiter(",");
			long start_time = System.currentTimeMillis();
			while (scan.hasNextLine())
			{
				// skip header
				scan.next();
				// read 5 doubles
				scan.nextDouble();
				scan.nextDouble();
				scan.nextDouble();
				scan.nextDouble();
				scan.nextDouble();
				// finish line
				scan.nextLine();
			}
			long end_time = System.currentTimeMillis();
			scan_times[i] = end_time - start_time;
			System.out.println(scan_times[i]);
			scan.close();
		}
 
		System.out.println("buffered reader test");
		for (int i = 0; i < times; ++i)
		{
			read = new BufferedReader(new FileReader(file_name));
			long start_time = System.currentTimeMillis();
			String line;
			while ((line = read.readLine()) != null)
			{
				String[] split = line.split(",");
				// convert items to doubles
				Double.parseDouble(split[1]);
				Double.parseDouble(split[2]);
				Double.parseDouble(split[3]);
				Double.parseDouble(split[4]);
				Double.parseDouble(split[5]);
			}
			long end_time = System.currentTimeMillis();
			buff_times[i] = end_time - start_time;
			System.out.println(buff_times[i]);
			read.close();
		}
	}
}

This code does creates a reader stream, reads in a line which consists of a header and 5 doubles, all separated by commas. It also generates a file to run the test with.

Results:

 

scanner test

3162

2855

2787

2800

2799

2797

2760

2755

2779

2756

buffered reader test

209

128

112

108

110

108

108

108

107

107

So it is looks like it is indeed true that Scanner's are significantly slower than BufferedReaders, even after buffering.