Thread: Reading In Then Writing Out Massive Files

Ok, so I have a MASSIVE text file (2,148,327 characters). I need to read in the text file, edit some of the data, then write it back out (not necessarily in the same order that it was read in).

This procedure will eventually be implemented in a web program that I have on a free host server. For the time being, I am figuring out how to do this on my local machine.

At the moment, I am reading in data, storing the data I need to edit in an object vector and storing the rest of the data in a String, then writing the data back out later. The problem I am having is that I am running into memory walls while creating the massive String (thanks to the Arrays.copy() method that the String class uses to resize Strings). In an ideal world, I could use a database for this but, as far as I know, my host does not support applets to communicate with databases. Increasing the heap stack size will not be a solution because the same would need to be done on the server (or clients perhaps), which I cannot do. So, I need to figure out a better way to read and write the data.

Due to it being web based, I also have to communicate with the files via Streams. Theoretically, I could backup the input file, erase the original file, read input from the backup, store the data I need to edit in my object vector and append the rest of the data to the original file, and then append the edited data to the file at the end. That way I wouldn't be storing the excess data in memory. The problem with that is I don't think I can append onto Streams or, if I can, I have no idea how.

Does anyone have any suggestions as to what I should do here? I've got no clue what to do.

In order to append, does it read in the data the file already contains? Because if it does, I'll have the same problem.

EDIT: This actually seems to work fantastically on my computer. In a perfect world I would be able to add to the beginning to the file instead of the end, but that is more of a convenience instead of a requirement. We'll see how it works on the server when I try it on there in a few days.