### Introduction

Single-layer neural networks can be trained using various learning algorithms. The best-known algorithms are the Adaline, Perceptron and Backpropagation algorithms for supervised learning. The first two are specific to single-layer neural networks while the third can be generalized to multi-layer perceptrons.

### Credits

The applet was written by Olivier Michel. This page written by Alix Herrmann.
The optimal perceptron was added by Tobias Denninger.

### Presentation

Let's consider a single-layer neural network with b inputs and c outputs:

• Wij = weight from input i to unit j in output layer; Wj is the vector of all the weights of the j-th neuron in the output layer.
• Ip = input vector (pattern p) = (I1p, I2p, ..., Ibp).
• Tp = target output vector (pattern p) = (T1p, T2p, ..., Tcp).
• Ap = Actual output vector (pattern p) = (A1p, A2p, ..., Acp).
• g() = sigmoid activation function: g(a ) = [1 + exp (-a)]-1

### Theory

Click on each topic to learn more.  Then scroll down to the applet.

### Applet

This applet allows you to compare the different learning algorithms.  The network implemented here has two inputs and a single output neuron.  In this tutorial, you will train it to classify 2-dimensional data points into two categories.

Click here to see the instructions.  You may find it helpful to open a separate browser window for the instructions, so you can view them at the same time as the applet window.

### Questions

1. Ideal case:   place 10  red points (class 1) and 10 blue points (0) in two similar, distinct, and linearly separable clusters.
• Compare the speed of convergence of the four algorithms. Which one is the fastest?
• Which values of the learning rate provide the best results ?
2. Different cluster dispersions:  Place 20 red points (1) in a very narrow cluster (strongly correlated points) and 5 blue points (0) in a very wide cluster in such a way that the classes are linearly separable.
• Compare the performance of the four algorithms on this problem. Which one is the best?
• Which values of the learning rate provide the best results ?

•
3. Imperfectly separable case:  Place 10 red points to (1) and 10 blue points (0) in two similar,  linearly separable clusters. Then, place an additional blue point inside the red cluster.
• Compare the behavior of the perceptron with the behavior of the pocket algorithm.
• Which values for the learning rate give the best results ?
4. For which kind of problem is the Adaline algorithm the best ?
5. For which kind of problem is the Backpropagation algorithm the best ?
6. For which kind of problem is the Perceptron algorithm the best ?
7. For which kind of problem is the Pocket algorithm the best ?