Thread: BufferedWriter writing garbage after a close()

I've been getting garbage repeated data in my output files for a few days now and have been troubleshooting my code to no avail. Finally, I tracked it down to my file-writing method. For simplicity's sake, the method looks like this:

public static void writeFile(ArrayList<String> data, Path fileName)
	{
		try
		{
			BufferedWriter writer = Files.newBufferedWriter(fileName, StandardOpenOption.CREATE);
			for(int index = 0; index < data.size(); index++)
			{
				writer.append(data.get(index));
				writer.newLine();
			}
			writer.close();
		}
		catch(IOException exception)
		{
			System.err.println(fileName + " cannot be written to.");
			return;
		}
	}

Take an ArrayList of text data, print it to a file line-by-line. Simple enough - in theory anyway. In practice, it prints a bunch of garbage data - seemingly from its own buffer - past the end of the list. I'm looking at the debug values right now. The list "data" is of size 4843, we're on index == 4842, element number 4842 is "99901 100000 1 -3", paused at writer.newLine(). The very next command will take me out of the loop, call Close, then take me out of the method so I would presume that the last thing I appended to the BufferedWriter object would be the last thing I'd see in the file, with possibly an empty line at the end. That's not what I'm seeing. Instead, what I see in the file is:

...
99801	99900	1	-1
99901	100000	1	-3
4	-2
480101	480200	3	-4
480201	480300	2	-1
...

In total, an extra 40 lines are printed to the file, replicated from earlier in the same file. The line "4 -2" is clearly the third and fourth column from somewhere that I can't determine and the rest of the lines seem to be taken from random places in the file, though oddly sorted in a way I can't fully track. I have no idea why this is happening and I'm not entirely sure of a good way to dump the BufferedWriter's buffer to check. The Eclipse debug viewer shows me the buffer as an array (which is what it is) but I can't recognise the data in there as relating to what I'm printing.

I've been using BufferedWriter for years, and this is the first time it's ever done this to me. I'm literally creating a BufferedWriter, appending its buffer, then printing to the hard drive and getting 40 extra lines that I didn't append after everything else is said and done. And I don't know why... Does anyone have any idea? Anything else I can share - files, additional code, additional information? I'm seriously stumped here.

Sounds like it's appending to the file when you expected it to overwrite instead. If that's the case, did you try using BufferedWriter.write() instead of append()?

Originally Posted by BinaryDigit09

Sounds like it's appending to the file when you expected it to overwrite instead. If that's the case, did you try using BufferedWriter.write() instead of append()?

From experimentation and reading about their implementation, both write() and append() work the same way except the former takes a String instance while the other takes a CharSequence instance, which String implements anyway. I did try, however, and the problem persisted.

Originally Posted by Norm

Have you tried debugging the code by having a print statement that prints out the value of data.get(index) before it is used in the append method?

I have, yes. The list contains only the data I would expect it to have. The information in the OP is cited from a debug paused right at the end of the cycle so I know for a fact what the last String instance was which was appended to the BufferedWriter. Nevertheless, when closing it it still printed an extra 40 lines.

Originally Posted by Norm

Also add a statement that writes an easily found line to the output just before the call to the close. See if anything is written to the file after that line.

I tried doing that, but I couldn't find any way to access the writer's buffer to print lines from it. I wanted to see if the writer indeed held the "garbage" data in its buffer prior to closing, or if that data was coming from somewhere else - some kind of index overflow or whatever else might be causing it to read from earlier in its own buffer. The buffer array appears to be private and the BufferedWriter class didn't seem to offer any methods for reading it, though I may have missed something.

Originally Posted by Norm

Can you copy and paste here the "garbage" that you are seeing?

All of it? Sure, though I don't think it'll help. It's just lines from previously in the file:

4	-2		
480101	480200	3	-4
480201	480300	2	-1
480301	480400	5	-1
480401	480500	1	-1
480501	480600	7	-1
480601	480700	5	-133
480701	480800	3	-118
480801	480900	0	-5
480901	481000	1	-7
481001	481100	1	-7
481101	481200	5	-6
481201	481300	0	-18
481301	481400	0	-12
481401	481500	0	-13
481501	481600	3	-17
481601	481700	4	-16
481701	481800	7	-34
481801	481900	6	-8
481901	482000	13	-28
482001	482100	4	-1
482101	482200	2	-4
482201	482300	1	-1
482301	482400	1	-4
482401	482500	0	-1
482501	482600	3	-2
482601	482700	2	0
482701	482800	2	0
482801	482900	2	-3
482901	483000	4	-5
483001	483100	2	0
483101	483200	2	-3
483201	483300	0	-1
483301	483400	2	-3
483401	483500	18	-1
483501	483600	14	-8
483601	483700	9	0
483701	483800	7	-15
483801	483900	0	0
483901	484000	4	0
484001	484100	2	0
484101	484138	0	0

---

In the day between posting this and today, though, I did some extra investigation and experimentation on my own. Crucially, I tried creating the BufferedWriter through its own constructor. I'd not been doing that as I don't want to use java.io, but I tried it. Doing so caused it to behave as normal, writing only what I'd explicitly written to its buffer. I went looking for code snippets and found someone suggesting I create a BufferedReader via Files.newBufferedReader() but supply it with both a CREATE and a WRITE instance of OpenOption. I hadn't realised that that method was set up with variable arguments for OpenOption and would take multiple of them. Doing so seems to have fixed the issue. In other words, my writeFile() method now looks like this:

public static void writeFile(ArrayList<String> data, Path fileName)
	{
		try
		{
			BufferedWriter writer = Files.newBufferedWriter(fileName, StandardOpenOption.WRITE, StandardOpenOption.CREATE);
			//BufferedWriter writer = new BufferedWriter(new FileWriter(fileName.toFile()));
			for(int index = 0; index < data.size(); index++)
			{
				writer.append(data.get(index));
				writer.newLine();
			}
			writer.close();
		}
		catch(IOException exception)
		{
			System.err.println(fileName + " cannot be written to.");
			return;
		}
	}

The above seems to work without issue, though I have not the slightest idea why. I'm additionally unable to replicate the issue from the OP any more, even if I restore the readFile() method to its original implementation by removing the WRITE OpenOption. I worry that this might have been Eclipse playing tricks on me. I recently switched to a new version and installed the AppEngine plugin which changes a LOT of my settings which I've been restoring as I run into them. I did get an Eclipse update last night at some point so maybe that's done SOMETHING? I don't know what to tell you, but the issue which was persistent across multiple runs is now no longer reproducible. "It just works."

Which brings me to a question - if my code now works and I don't know why it was broken or what I did to fix it... Do I tag the thread as "Solved" so it doesn't waste people's time any more? Is this worth pursuing any further?