Thread: Neural Network Programming - Wrong result after training

I am attempting to write a Java implementation of a NeuralNetwork. This consists of:

A "Network" class
An abstract "Neuron" class
An "Hidden Neuron" class, which extends Neuron
An "Output Neuron" class, which extends Neuron
A "Layer" class, which holds HiddenNeuron objects in an Array
A "Link" class, which creates the link between two neurons
A "RunNet" class used to run the network

THe main Network classes essentially has methods to a) train and b) run the network based on a series of inputs/outputs.

My code:

Network.java

 import java.util.ArrayList;
 
 
    public class Network {
 
 
	OutputNeuron outputNeuron = new OutputNeuron();
 
	ArrayList<Layer> networkLayers = new ArrayList<Layer>();
 
	public void AddLayer(Layer layer) {
 
		networkLayers.add(layer);
 
	}
 
	public void BridgeNetwork() {
 
		for(int i = 0; i+1 < networkLayers.size(); i++) {
 
			Layer currentLayer = networkLayers.get(i);
			Layer nextLayer = networkLayers.get(i+1);
 
			for(HiddenNeuron currentLayerNeuron : currentLayer.getLayerNeurons()) {
 
				for(HiddenNeuron nextLayerNeuron : nextLayer.getLayerNeurons()) {
 
					Link link = new Link(currentLayerNeuron, nextLayerNeuron);
 
					currentLayerNeuron.addForwardLink(link);
					nextLayerNeuron.addBackLink(link);
 
				}
 
			}
 
			//Now taking care of linking last hidden layer to output neuron
			ArrayList<HiddenNeuron> lastLayer = (networkLayers.get(networkLayers.size()-1)).getLayerNeurons();
 
			for(HiddenNeuron currentNeuron : lastLayer) {
 
				Link link = new Link(currentNeuron, outputNeuron);
 
				currentNeuron.addForwardLink(link);
				outputNeuron.addBackLink(link);
 
			}
 
		}
 
	}
 
	public boolean SetupInputs(ArrayList<Double> inputs) {
 
		//Setting up inputs		
		ArrayList<HiddenNeuron> firstLayerNeurons = (networkLayers.get(0)).getLayerNeurons();
 
		if(firstLayerNeurons.size() != inputs.size()) {
 
			System.out.println("ERROR! Number of inputs must equal number of neurons in first layer!");
			return false;
 
		}
 
		for(int i = 0; i < firstLayerNeurons.size(); i++) {
 
			(firstLayerNeurons.get(i)).SetOutput(inputs.get(i));
 
 
		}
 
		return true;
 
	}
 
	public double FeedForward() {
 
 
		//Feeding forward!
		for(Layer currentLayer : networkLayers) {
 
			for(HiddenNeuron currentNeuron : currentLayer.getLayerNeurons()) {
 
				currentNeuron.FeedForward();
 
			}
 
		}
 
		//Alright! We got to the output neuron!
		outputNeuron.ComputeOutput();
 
		return outputNeuron.getOutput();
 
 
	}
 
	public void BackPropagate() {
 
		outputNeuron.BackPropagate();
 
		for(int i = networkLayers.size()-1; i >= 0; i--) {
 
			ArrayList<HiddenNeuron> currentLayerNeurons = (networkLayers.get(i)).getLayerNeurons();
 
			for(HiddenNeuron currentNeuron : currentLayerNeurons) {
 
				currentNeuron.BackPropagate();
 
			}
 
		}
 
	}
 
	public void TrainNet(ArrayList<ArrayList<Double>> myTemplates, int epochs) {
 
		for(int a = 0; a < myTemplates.size(); a++) {
 
			System.out.println("Training throught testSet " + a);
 
			ArrayList<Double> currentTemplate = myTemplates.get(a);
 
			outputNeuron.setDesired(currentTemplate.get(0));
			currentTemplate.remove(0);
			SetupInputs(currentTemplate);
 
			for(int i = 0; i < epochs; i++) {
				System.out.println("Epoch " + i + " of test testSet " + a);
 
				FeedForward();
				BackPropagate();
 
			}
 
 
		}
 
	}
 
	public double RunNet(ArrayList<Double> inputs) {
		System.out.println("Executing net");
		SetupInputs(inputs);
		return FeedForward();
 
	}
 
    }

Layer.java

import java.util.ArrayList;
 
 
public class Layer {
 
	ArrayList<HiddenNeuron> layerNeurons = new ArrayList<HiddenNeuron>();
 
	public void addNeuron(HiddenNeuron neuron) {
 
		layerNeurons.add(neuron);
 
	}
 
	public ArrayList<HiddenNeuron> getLayerNeurons() {
 
		return layerNeurons;
 
	}
 
    }

Link.java

public class Link {
 
	HiddenNeuron neuronA = null;
	Neuron neuronB = null;
 
	double weight = 0;
 
	public Link(HiddenNeuron neuronA, Neuron neuronB) {
 
		this.neuronA = neuronA;
		this.neuronB = neuronB;
 
		double weight = Math.random();
 
		if(weight > 0.5)
			this.weight = weight;
		else
			this.weight = -1*weight;
 
	}
 
	protected void BackPropagate(double error) {
 
		weight += error;
		neuronA.receiveError(error*weight);
 
	}
 
	protected void FeedForward(double input) {
 
		neuronB.ReceiveInput(weight*input);
 
	}
 
    }

Neuron.java

import java.util.ArrayList;
 
 
    public abstract class Neuron {
 
	double output = 0;
	double derivativeOutput = 0;
	double error = 0;
 
	ArrayList<Double> inputs = new ArrayList<Double>();
	ArrayList<Link> backLinks = new ArrayList<Link>();
 
	public void addBackLink(Link link) {
 
		backLinks.add(link);
 
	}
 
	protected void ComputeOutput() {
 
		double partialSum = 0;
 
		//Summing all inputs
		for(double currentInput : inputs) {
 
			partialSum += currentInput;
 
		}
 
		//Setting output to be result of sigmoid function taking sum of inputs as parameter
		output = 1/(1+Math.pow(Math.E, (-1*partialSum)));
 
		ComputeDerivative();
 
	}
 
	private void ComputeDerivative() {
 
		derivativeOutput = output*(1-output);
 
	}
 
	protected void ReceiveInput(double input) {
 
		inputs.add(input);
 
	}
 
	protected void BackPropagate() {
 
		ComputeError();
 
		for(Link currentLink : backLinks) {
 
			currentLink.BackPropagate(error);
 
		}
 
	}
 
	protected abstract void ComputeError();
 
 
 
    }

HiddenNeuron.java

 import java.util.ArrayList;
 
 
    public class HiddenNeuron extends Neuron {
 
	ArrayList<Double> myErrors = new ArrayList<Double>();
	ArrayList<Link> forwardLinks = new ArrayList<Link>();
 
	public void addForwardLink(Link link) {
 
		forwardLinks.add(link);
 
	}
 
	public void SetOutput(double output) {
 
		super.output = output;
 
	}
 
	protected void receiveError(double error) {
 
		myErrors.add(error);
 
	}
 
	protected void ComputeError() {
 
		double partialSum = 0;
 
		for(double currentError : myErrors) {
 
			partialSum += currentError;
 
		}
 
		super.error = partialSum*super.derivativeOutput;
 
	}
 
	protected void FeedForward() {
 
		super.ComputeOutput();
 
		for(Link currentLink : forwardLinks) {
 
			currentLink.FeedForward(super.output);
 
		}
 
	}
 
    }

OutputNeuron.java

public class OutputNeuron extends Neuron {
 
	double desired = 0.0;
 
	public double getOutput() {
 
		return super.output;
 
	}
 
	public void setDesired(double desired) {
 
		this.desired = desired;
 
	}
 
	protected void ComputeError() {
 
		super.ComputeOutput();
 
		super.error = (desired-super.output)*super.derivativeOutput;
 
	}
 
    }

RunNet.java (used to train the network for an "OR" gate)

import java.util.ArrayList;
 
 
    public class RunNet {
 
	public static void main(String[] args){
 
		ArrayList<Double> testA = new ArrayList<Double>();
		testA.add(1.0);
		testA.add(1.0);
		testA.add(1.0);
 
		ArrayList<Double> testB = new ArrayList<Double>();
		testB.add(1.0);
		testB.add(0.0);
		testB.add(1.0);
 
		ArrayList<Double> testC = new ArrayList<Double>();
		testC.add(0.0);
		testC.add(1.0);
		testC.add(1.0);
 
		ArrayList<Double> testD = new ArrayList<Double>();
		testD.add(0.0);
		testD.add(0.0);
		testD.add(0.0);
 
		ArrayList<ArrayList<Double>> myTemplates = new ArrayList<ArrayList<Double>>();
		myTemplates.add(testA);
		myTemplates.add(testB);
		myTemplates.add(testC);
		myTemplates.add(testD);
 
		Network myNet = new Network();
 
		Layer first = new Layer();
		Layer second = new Layer();
 
		first.addNeuron(new HiddenNeuron());
		first.addNeuron(new HiddenNeuron());
 
		second.addNeuron(new HiddenNeuron());
		second.addNeuron(new HiddenNeuron());
 
		myNet.AddLayer(first);
		myNet.AddLayer(second);
 
		myNet.BridgeNetwork();
 
		ArrayList<Double> testE = new ArrayList<Double>();
		testE.add(0.0);
		testE.add(0.0);
 
		myNet.TrainNet(myTemplates, 10000);
		System.out.println(myNet.RunNet(testD));
 
	}
 
    }

However, when I then run the network with both A and B of the expression A OR B the network returns true... Any suggestions for improvement? :S

Thanks

Originally Posted by copeg

I'm not sure I get the training data: don't testA and testC conflict with each other (presuming the first index is output and latter 2 the input)? What is the input layer?

Hi copeg,

You are right, testC appeared to have been wrong. I have fixed it to:

ArrayList<Double> testC = new ArrayList<Double>();
		testC.add(1.0);
		testC.add(1.0);
		testC.add(0.0);

Which is basically saying for A = true and B = false return true. However, when trained the network is still returning true for A = false and B = false.

There are no proper input neurons, what happens is inputs ae set as values of neurons in the first layer are set as outpu the various input values:

public boolean SetupInputs(ArrayList<Double> inputs) {
 
		//Setting up inputs		
		ArrayList<HiddenNeuron> firstLayerNeurons = (networkLayers.get(0)).getLayerNeurons();
 
		if(firstLayerNeurons.size() != inputs.size()) {
 
			System.out.println("ERROR! Number of inputs must equal number of neurons in first layer!");
			return false;
 
		}
 
		for(int i = 0; i < firstLayerNeurons.size(); i++) {
 
			(firstLayerNeurons.get(i)).SetOutput(inputs.get(i));
 
 
		}
 
		return true;
 
	}

Perhaps this isn't the most wise decision?

I advice you to prototype all your machine learning soft on languages like Matlab or Octave and only then start programming in Java. They are well-tuned for this staff and, hence, you can see errors much easily.

Originally Posted by angstrem

I advice you to prototype all your machine learning soft on languages like Matlab or Octave and only then start programming in Java. They are well-tuned for this staff and, hence, you can see errors much easily.

Hi angstream, in what ways are they better? I'm not trying to sound skeptical, I'm just not very familiar with them.

They natively support "heavy math": linear algebra, statistics, operations with functions (like minimization) - all the staff necessary for machine learning. Hence, a thing of that kind that would take hundreds of lines of code in Java or C++ (or deep search of appropriate (and efficient) math libraries) can be easily done just in several lines of code in Octave or Matlab. This leads to simplicity of implementation of machine learning algorithms. You can spend minutes for realization your algorithm in ML, see whether it works or not, and then spend hours to implement it in Java or C++.

Ho copeg,

Yes, this is simply a challenge I set myself. I would like to stress that it is not an academic work that will earn me a grade, just something I decided to do. This said, I've been reading for the past days about neural nets basics, but I can't seem to understand where I'm going wrong. Would you have any suggestions?

Hi Copeg,

Admittedly I was forgetting to reset lists of inputs and outputs. I added the following:

public void ResetNetwork() {
 
		for(int i=0; i < networkLayers.size(); i++) {
 
			ArrayList<HiddenNeuron> currentLayerNeurons = networkLayers.get(i).getLayerNeurons();
 
			for(HiddenNeuron currentNeuron : currentLayerNeurons) {
 
				if(i > 0)
					currentNeuron.ResetInputs();
 
				currentNeuron.ResetErrors();
 
			}
 
		}
 
	}

Which is called at each step of training after BackPropagate(). Still though, I see no improvement. Can you suggest some visual tool? I'm using Eclipse debugger but with hundreds of cycles it isn't very useful...