Thread: Loading file into memory is taking up TOO much memory!

Hi, I am attempting to load a small file (Excel format for now) into memory. The excel file is roughly 80 KB in size and only contains about 20 or 30 columns and 80 rows. I've set my JVM parameters to 2 GB. Considering its only an 80 KB file I wouldn't except it to take up more than a MB of memory but for some reason it seems to expand my JVM by 40 MB once loaded!

Upon startup of my program 40 MB is used according to the task manager, which I think is fine. But for it to increase form 40 to 80 MB on an 80 KB file is too much because I plan on loading much larger files and if memory requirements grow linear then its simply not going to be possible.

Some classes

/*
 TableTabInfo Class
 *
 * This Class is a container for all the information that needs to be
 * segregated between multiple loaded tables
 */
package dataStorageClasses;
 
import java.util.ArrayList;
import javax.swing.JTable;
 
public class [B]TableTabInfo [/B]{
 
    public JTable jDataTable;
    public ArrayList<String> headers;
    public RowData[] rData;
 
    public TableTabInfo() {
        this.headers = new ArrayList<String>();
    }
 
    public void setSizeOfRowData(int size) {
        this.rData = new RowData[size];
 
        for (int i = 0; i < size; i++) {
            this.rData[i] = new RowData();
        }
 
    }
 
    public void addCell(Object cell, int rowNumber) {
        this.rData[rowNumber].addToMyList(cell);
    }
 
    public Object getCell(int row, int col) {
        return this.rData[row].myList.get(col);
    }
 
    public void addToHeaderList(String header) {
        this.headers.add(header);
    }
 
    public void emptyHeaderList() {
        this.headers.clear();
    }
 
    public int sizeOfRowData() {
        return this.rData.length;
    }
}
 
package dataStorageClasses;
import java.util.ArrayList;
 
public class [B]RowData[/B]<T> {
 
    public ArrayList<T> myList = new ArrayList<T>();
 
    public RowData()
    {
    }
 
    public T getFromMyList(int index) {
        return this.myList.get(index);
    }
 
    public void addToMyList(T e) {
        this.myList.add(e);
    }
}

Loading the file

if (getExtension.contains("xls") || getExtension.contains("lsx")) {
                        try {
                            File inputFile = new File(chooser.getSelectedFile().getAbsolutePath());
                            FileInputStream inputStream = new FileInputStream(inputFile);
 
                            Workbook wb = null;
                            try { 
                                wb = WorkbookFactory.create(inputStream);
                            } catch (InvalidFormatException ex) {
                                System.out.println(ex.toString());
                            }
 
                            Sheet sheet = wb.getSheetAt(0);
 
                            //Add a new tab info object into our vector
                            TableTabInfo newTab = new TableTabInfo();
                            newTab.setSizeOfRowData(sheet.getLastRowNum());
 
                            vTabInfo.add(newTab);
 
                            Boolean boolIterOnce = false; 
                            int maxSize = 0; // the variable that is used as the pseudo max
 
                            // Iterate over each row in the sheet
                            Iterator<Row> rowsIterator = sheet.rowIterator();
                            while (rowsIterator.hasNext()) {
                                Row row = rowsIterator.next();
 
                                if (!boolIterOnce) //only 1 time for headers
                                {
                                    Iterator<Cell> cellsIterator = row.cellIterator();
                                    ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addToHeaderList("Row");
 
                                    while (cellsIterator.hasNext()) {
                                        Cell cell = cellsIterator.next();
 
                                        switch (cell.getCellType()) {
 
                                            case Cell.CELL_TYPE_STRING:
                                                if (row.getRowNum() == 0) { //In the file there is no row number column so I am just inserting one manually
                                                    ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addToHeaderList(cell.getStringCellValue());
                                                }
                                                break;
 
                                            default:
                                                break;
                                        }
 
                                    } 
                                }
                                else if (boolIterOnce) // if not traversing the headers
                                {                      
                                    for (int i = 0; i < maxSize; i++) {
                                        if (i == 0) {
                                            //First column is used for row numbers, convert to string so they are left aligned by default
                                            ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(Integer.toString(row.getRowNum() - 1), row.getRowNum() - 1);
                                        } else {
                                            Cell cell = row.getCell(i - 1, Row.CREATE_NULL_AS_BLANK);
                                            switch (cell.getCellType()) {
                                                case Cell.CELL_TYPE_NUMERIC: // if the cell is a number
                                                    if (row.getRowNum() == 0) {
                                                    } else {
                                                        if (DateUtil.isCellDateFormatted(cell)) {//speical case when its a date
                                                            ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getDateCellValue(), row.getRowNum() - 1);
                                                        } else {
                                                            ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getNumericCellValue(), row.getRowNum() - 1);
                                                        }
                                                    }
                                                    break;
 
                                                case Cell.CELL_TYPE_STRING:
                                                    if (row.getRowNum() == 0) {
                                                    } else {
                                                        ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getStringCellValue(), row.getRowNum() - 1);
                                                    }
                                                    break;
                                                case Cell.CELL_TYPE_BLANK:
                                                    ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell("NO DATA", row.getRowNum() - 1);
                                                    break;
                                                case Cell.CELL_TYPE_BOOLEAN: // if the cell contains a boolean value                                       
                                                    ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getBooleanCellValue(), row.getRowNum() - 1);
                                                    break;
                                                case Cell.CELL_TYPE_FORMULA: // if the cell contaions a formula (Shouldn't be called)
                                                    ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getCellFormula(), row.getRowNum() - 1);
                                                    break;
                                                default: 
                                                    System.out.println("unsuported Cell type Row: " + row.getRowNum() + " Column: " + cell.getColumnIndex());
                                                    break;
                                            }
                                        }
                                    }
                                }
                                maxSize = ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).headers.size();
                                boolIterOnce = true;
                            } 
                        } catch (IOException ex) {
                            ex.printStackTrace();
                            outputTextArea.setText("ERROR: FILE COULD NOT BE OPENED!");
                        }
                    }

I am using vectors to store each new instance of a TableTabInfo class, which shouldn't be an issue because only a few exist at any given time. I thought it was the data inside that was taking up a lot of space (rData variable perhaps as thats where the raw data is actually stored (in addition to them being loaded into a JTable also in that class. But I set rData to null after loading the file and java.exe remained at 80 MB usage and I understand that setting the object to null essentially deletes it and frees the memory but that did not seem to be the case.

Any suggestions would be great.

The problem is occuring when loading .txt files with the same data as well. A 100 MB file takes 1.5 GB in memory using the above code.

Originally Posted by copeg

Memory management in java isn't as simple as watching memory usage in the task manager. Memory is managed by a garbage collector - its pretty smart but is not constantly monitoring object references and immediately acting accordingly. Thus setting an object to null (and presuming no more references exist to the object) does not mean it is immediately removed from memory. If you truly wish to inspect memory usage, use a java profiler (for instance VisualVM) to inspect the runtime of your program - from there you can try to nail down what is consuming the most memory, whether you need it, and/or whether you are holding onto references to object you no longer need.

Thanks. Here are the results from the profiler. They don't seem to add up as the largest classes which are just double and chars show only usage of around 7-8 MB each. In the task manager java.exe is showing as 312 MB used.

http://i.imgur.com/pZkHoQr.png

Originally Posted by copeg

Memory management in java isn't as simple as watching memory usage in the task manager. Memory is managed by a garbage collector - its pretty smart but is not constantly monitoring object references and immediately acting accordingly. Thus setting an object to null (and presuming no more references exist to the object) does not mean it is immediately removed from memory. If you truly wish to inspect memory usage, use a java profiler (for instance VisualVM) to inspect the runtime of your program - from there you can try to nail down what is consuming the most memory, whether you need it, and/or whether you are holding onto references to object you no longer need.

Thanks, I tried Visual VM and here are the results from the sampler.

http://i.imgur.com/QhaJSmO.png

It looks like its showing basic objects - which is hard to tell where exactly they come from.

These are results from a 27 MB text file. Takes up 650 MB at the start, then drops for some reason.

See here http://i.imgur.com/cJw2xyU.png

I really have no clue whats going on, looks like high usage and then a manageable amount. Then it climbs back up.