MLP Neural Network: Handwriting Recognition

The following demo uses a feed forward multi-layer perceptron with 40 hidden units, trained for about 700 epochs using the back-propagation algorithm, with weight momentum.

Draw a roman numeral digit 0 – 9, and it will try and find which digit are you writing!

NOTE: If it doesn’t work, try writing it in another style, or try making your digit fatter or thinner or taller or shorter.
Numbers must span about 90% of the height of the box, and 80% of the width of the box!


Example Digit

Example digit

Example digit

Artificial Neural Networks

The Brain

The core component of a biological brain is a neural network. Naturally occurring neural networks are extremely complex arrangements of massive numbers of neurons. The basic unit that makes these neural networks is a cell called a neuron.

Artificial Neural Networks (ANNs) are the result of computer scientists taking a reverse engineering approach to artificial intelligence. This allows us to some extent, to make computers do what computers are bad at doing but we are good at doing, by vaguely emulating what a brain does. One of the things that the brain does well that computers do not, is cognition; we can easily recognise people by their faces, easily know the difference between a tree and a book, easily tell whether it is a car or a tree given a picture. Computer systems that require solving this problem can make use of ANNs.

Diagram of a biological neuron

Diagram of a biological neuron. Image from http://en.wikipedia.org/wiki/Neuron

A neuron is a specialised cell that takes some inputs from some neurons, does some sort of processing, and gives some outputs to some other neurons. The neuron communicates through an electrochemical process. Neurons have extensions: Dendrites that bring information in (the inputs) and axons, which pass the outputs to other cells. Dendrites are mostly local: they surround the cell. Axons can connect to distant cells: sometimes even more than a meter away.
Inputs cause neurons to fire; they send a pulse down their axon to more neurons. For example, touching a liquid with your finger causes neurons in your finger’s nerve system to fire, which may cause the neurons they are connected to also fire, and so on. This message hopefully transmits down the line into your brain, which may cause neurons in your brain firing, which may eventually do down the network and cause the feeling of wetness.

Perceptron: A neuron simplified

A perceptron is an extremely simplified model of the neuron. It takes in a set of (positive or negative) numerical input values x = { x1,…,xn}, multiplies each input xk with its respective weight value wk from w = { w1,…,wn}. It takes the sum of all n weighted inputs → o, and “fires” (outputs 1) if it is over 0, and does not fire otherwise (outputs -1).

Diagram of perceptron model

Diagram of perceptron model. Image from COMP9844 09s2 lecture notes by A/Prof. Achim Hoffmann.

The limitation of a perceptron is that it effectively splits the input into two categories by a hyperplane (a). For example, if we have 2 inputs x and y, and we plot points corresponding to our input data, a perceptron will draw a straight line through the data, and return 1 for inputs one side of the line, and -1 for inputs on the other side. Therefore, we cannot represent something like an XOR function (b), as no matter how we draw the straight line, we will never be able to split with a single line between the positive and negative outputs. Therefore, perceptrons can only ever represent linearly separable functions.

Perception limitations.

Perception limitations. Image from COMP9844 09s2 lecture notes by A/Prof. Achim Hoffmann.

Perceptron Training

To train a perceptron so that it outputs the correct values for our set of testing input data, we get its set of outputs o, subtract o from our set of target values t, and we get the sign of this difference and multiply it by the input itself, and add this to our weights. (t and o are both either 1 or -1.) In other words,

  • If the output was correct, t – o = 0, we don’t do anything at all.
  • If the output is -1 but the correct output should be 1, then we add our inputs to the perceptron’s weights, (that is, wi=wi+xi for i = 1…n)
  • If the output is 1 but the correct output should be -1, then we subtract
    our inputs from the perceptron’s weights, (that is, wi=wi-xi for i = 1…n)

Source Code Downloads

handwriting_MLP_network.nwk – Network Weights Data File
evaluate.cpp – Evaluation Program – Main Source
evaluate.exe – Evaluation Program – Win32 x86 Binary
neural.h – Neural Network MLP Class
train.h – Training Management Functions
neural.cpp – Training Program – Main Source
neural.exe – Training Program – Win32 x86 Binary
optdigits.tra – Training Data Set
optdigits.tes – Testing Data Set

Data set taken from
UCI Maching Learning Repository
, and originally came from E Alpaydin, C. Kaynak.

Share Button

Leave a Reply