Thread: Reading CSV file to take min, max, average of various stats

Hi all,

I want to write a Java class that will take a CSV file and take the min, max, average, etc... of various statistics. Please see below for a sample of this file. For example, I want to get the stats for "Opening login page", "Logging in as lead user", etc..

I have started by reading TestName and Time into a Map:

static Map<String, String> data = new TreeMap<String, String>();
...
		for (String line : file) {
			if (line.contains("Test Name")){
				//Get next line, this is the first one.
				continue;
			}
			String TestName = (line.substring(0,line.indexOf(","))).trim();
			String Time = (line.substring(line.lastIndexOf(",")+2, line.length())).trim();
			//System.out.println(TestName + " " + Time);
			data.put(TestName, Time);			
		}

This works fine. However, I'm not sure of the best way to go about getting all unique values and running the calculation. I don't need to use a Map, it was what just popped into my head. Does anyone know how to do something like this?


CSV file sample:
=============

Test Name, Thread No, Run No, Time(s)
Opening login page, 0, 0, 1.8869998455047607
Opening login page, 1, 0, 2.246999979019165
Opening login page, 2, 0, 2.1710000038146973
Logging in as regular user, 1, 0, 22.009999990463257
Logging in as regular user, 1, 0, 22.009999990463257
Logging in as regular user, 1, 0, 22.009999990463257
Logging in as lead user, 0, 0, 23.616000175476074
Opening login page, 13, 0, 2.125999927520752
Opening login page, 15, 0, 1.8939998149871826
Logging in as lead user, 3, 0, 20.244999885559082
Logging in as lead user, 2, 0, 23.039999961853027

Let's see if I understand what you're asking:

There a few different tests which can be run on different threads (or different run numbers), and you want to find the mean/min/max/etc. time that task was being run?

For example, in your above code, for the test "Opening login page", you'd perform your statistical analysis (min, max, mean, etc.) on the following items:

Opening login page, 0, 0, 1.8869998455047607
Opening login page, 1, 0, 2.246999979019165
Opening login page, 2, 0, 2.1710000038146973
Opening login page, 13, 0, 2.125999927520752
Opening login page, 15, 0, 1.8939998149871826

Similarly, you'd do the same for "Logging in as regular user", etc. etc.

If that's the case, using a map would work, however there is one modification that will need to be made: you need to keep track all the times, not just the last time to be read. So instead of creating a map of strings to strings, map strings to a list.

// note: this is untested code, but should demonstrate the idea
static Map<String, ArrayList<String>> data = new TreeMap<String, ArrayList<String>>();
...
		for (String line : file) {
			if (line.contains("Test Name")){
				//Get next line, this is the first one.
				continue;
			}
			String TestName = (line.substring(0,line.indexOf(","))).trim();
			String Time = (line.substring(line.lastIndexOf(",")+2, line.length())).trim();
			if(!data.containsKey(TestName))
			{
				// put a new test and list of times
				data.put(TestName, new ArrayList<String>());
			}
			// add the time to the correct list
			data.get(TestName).add(Time);
		}

As a side note, if you could even consider using a HashMap instead of a TreeMap as a HashMap theoretically has O(1) access (note: depending on how Java implements the HashMap, it's likely to have O(n) access if the given key doesn't exist, or for various other reasons), while the TreeMap will be ~O(log(n)) for most operations.

Another alternative is to use a Radix Tree (aka PATRICIA tree), or a Trie. These provide O(k) access (where k is the length of the key, not the size of the dataset), which is almost always considerably faster/reliable than a TreeMap or HashMap. However, unless you really need quick computations on a large data set, it's probably not worth trying to implement them (I don't think Java has either of these data structures built-in).