Artificial neural networks Day 01

I. Artificial neural networks

Limitations

1. Hard to explain the solution (explanation capabilities)
2. Have to adjust (fine-tune) many parameters (e.g. training dataset, validation dataset, test dataset, network architecture (feedforward, recurrent/feedback networks), number of layers, number of hidden neurons, learning rate, initial weights, number of epochs, accuracy (error tolerance))
3. Require a lot of resources (resource-intensive): time consuming, fast computers

Benefits

1. Solve many problems: predictions, classifications, clustering
2. Enable paralleled processing, speeding up certain computations

II. Perceptron model for linear data analysis

Single layer perceptron for linear data analysis (e.g. pattern recognition)

Linearly separable classification problems (linearly separable patterns) vs non-linear pattern recognition problems

III. Neural networks for non-linear processing

The following neural networks can be used for non-linear processing (non-linear data analysis)

1. Multilayer perceptron (MLP) networks
2. Radial basis function (RBF)
3. Polynomial nets
4. Generalized regression neural networks (GRNN)
5. Generalize neural networks (GNN)

Training algorithms in MLP networks

1. Backpropagation learning/ training algorithm (i.e. The error is propagated back from the output to adjust the weights.). The learning challenge is to find weights that result in the minimum error for the whole training dataset.

A multilayer perceptron network is considered as a "universal approximator".

There is no theory yet on how many units in the hidden layer that can approximate any given function.

Multicategory classification problem (i.e. multi-output perceptron)

References

Grace Rumantir, Monash University FIT5167 Lecture Notes, 2012.

Glossary

Linear or non-linear neurons refer to the linear or non-linear activation function.
Neuron: a nerve cell in a biological nervous system.
Axon: outgoing terminals from a biological neuron.
Dendrites: the incoming connections to biological neurons.
Synapse: the area of electrochemical contact between two biological neurons.
Biological neural networks are assumed to comprise the human's brain. The brain is composed of millions of networks each having several thousands of highly interconnected neurons. Networks are integrated with other networks (but not with all of them).
An artificial neural network is a computerized system which structure and operation take the metaphor of those of a biological network.