Jaringan Syaraf Tiruan

Computational Intelligence S2

1
What are Neural Networks ?
1. Neural Networks (NNs) are networks of neurons, for example, as found
in real (i.e. biological) brains.
2. Artificial Neurons are crude approximations of the neurons found in
brains. They may be physical devices, or purely mathematical constructs.
3. Artificial Neural Networks (ANNs) are networks of Artificial Neurons,
and hence constitute crude approximations to parts of real brains. They
may be physical devices, or simulated on conventional computers.
4. From a practical point of view, an ANN is just a parallel computational
system consisting of many simple processing elements connected
together in a specific way in order to perform a particular task.
5. One should never lose sight of how crude the approximations are, and
how over-simplified our ANNs are compared to real brains.

2
Why are Artificial Neural Networks
worth studying?
1. They are extremely powerful computational devices (Turing
equivalent, universal computers).
2. Massive parallelism makes them very efficient.
3. They can learn and generalize from training data so there is no
need for enormous feats of programming.
4. They are particularly fault tolerant this is equivalent to the
graceful degradation found in biological systems.
5. They are very noise tolerant so they can cope with situations
where normal symbolic systems would have difficulty.
6. In principle, they can do anything a symbolic/logic system can do,
and more. (In practice, getting them to do it can be rather difficult)

3
What are Artificial Neural
Networks used for?
As with the field of AI in general, there are two basic goals for neural network research:

Brain modelling : The scientific goal of building models of how real brains work.
This can potentially help us understand the nature of human intelligence, formulate better teaching
strategies, or better remedial actions for brain damaged patients.

Artificial System Building : The engineering goal of building efficient systems for real
world applications. This may make machines more powerful, relieve humans of tedious
tasks, and may even improve upon human performance.

These should not be thought of as competing goals. We often use exactly the same
networks and techniques for both. Frequently progress is made when the two
approaches are allowed to feed into each other. There are fundamental differences
though, e.g. The need for biological plausibility in brain modelling, and the need for
computational efficiency in artificial system building.

4
Learning in Neural Networks
There are many forms of neural networks. Most operate by passing
neural activations through a network of connected neurons.
One of the most powerful features of neural networks is their
ability to learn and generalize from a set of training data. They
adapt the strengths/weights of the
connections between neurons so that the final output activations
are correct.
There are three broad types of learning:
1. Supervised Learning (i.e. learning with a teacher)
2. Reinforcement learning (i.e. learning with limited feedback)
3. Unsupervised learning (i.e. learning with no help)

5
A Brief History of the Field
1943 McCulloch and Pitts proposed the McCulloch-Pitts neuron model
1949 Hebb published his book The Organization of Behavior, in which the
Hebbian learning rule was proposed.
1958 Rosenblatt introduced the simple single layer networks now called
Perceptrons.
1969 Minsky and Paperts book Perceptrons demonstrated the limitation of
single layer perceptrons, and almost the whole field went into hibernation.
1982 Hopfield published a series of papers on Hopfield networks.
1982 Kohonen developed the Self-Organising Maps that now bear his name.
1986 The Back-Propagation learning algorithm for Multi-Layer Perceptrons
was rediscovered and the whole field took off again.
1990s The sub-field of Radial Basis Function Networks was developed.
2000s The power of Ensembles of Neural Networks and Support Vector
Machines becomes apparent.
6
Some Current Artificial Neural
Network Applications
Brain modelling
Models of human development help children with developmental problems
Simulations of adult performance aid our understanding of how the brain works
Neuropsychological models suggest remedial actions for brain damaged patients
Real world applications
Financial modelling predicting stocks, shares, currency exchange rates
Other time series prediction climate, weather, airline marketing tactician
Computer games intelligent agents, backgammon, first person shooters
Control systems autonomous adaptable robots, microwave controllers
Pattern recognition speech recognition, hand-writing recognition, sonar signals
Data analysis data compression, data mining, PCA, GTM
Noise reduction function approximation, ECG noise reduction
Bioinformatics protein secondary structure, DNA sequencing

7
 The Nervous System
The human nervous system can be broken down into three stages that may be
represented in block diagram form as:

The receptors collect information from the environment e.g. photons on the
retina.
The effectors generate interactions with the environment e.g. activate
muscles.
The flow of information/activation is represented by arrows feedforward and
feedback.
Naturally, in this module we will be primarily concerned with the neural
network in themiddle.
8
 Levels of Brain Organization
The brain contains both large scale and small scale anatomical structures and
different functions take place at higher and lower levels.

There is a hierarchy of interwoven levels of organization:

1. Molecules and Ions
2. Synapses
3. Neuronal microcircuits
4. Dendritic trees
5. Neurons
6. Local circuits
7. Inter-regional circuits
8. Central nervous system

The ANNs we study in this module are crude approximations to levels 5 and 6.

9
Brains versus Computers : Some
numbers
1. There are approximately 10 billion neurons in the human cortex,
compared with 10 of thousands of processors in the most powerful
parallel computers.
2. Each biological neuron is connected to several thousands of other
neurons, similar to the connectivity in powerful parallel computers.
3. Lack of processing units can be compensated by speed. The typical
operating speeds of biological neurons is measured in milliseconds (10-
3 s), while a silicon chip can operate in nanoseconds (10-9 s).
4. The human brain is extremely energy efficient, using approximately 10-
16 joules per operation per second, whereas the best computers today
use around 10-6 joules per operation per second.
5. Brains have been evolving for tens of millions of years, computers have
been evolving for tens of decades.
10
Structure of Human Brain

11
 Basic Components of Biological
Neurons
1. The majority of neurons encode their activations or outputs as a
series of brief electrical pulses (i.e. spikes or action potentials).
2. The neurons cell body (soma) processes the incoming
activations and converts them into output activations.
3. The neurons nucleus contains the genetic material in the form of
DNA. This exists in most types of cells, not just neurons.
4. Dendrites are fibres which emanate from the cell body and
provide the receptive zones that receive activation from other
neurons.
5. Axons are fibres acting as transmission lines that send activation
to other neurons.
6. The junctions that allow signal transmission between the axons
12
and dendrites are called synapses. The process of transmission is
Schematic Diagram of a
Biological Neuron

13
 The McCulloch-Pitts Neuron
This vastly simplified model of real neurons is also known as a Threshold
Logic Unit :

1. A set of synapses (i.e. connections) brings in activations from other neurons.

2. 2. A processing unit sums the inputs, and then applies a non-linear activation
function (i.e. squashing/transfer/threshold function).
3. 3. An output line transmits the result to other neurons.

14
 Some Useful Functions
A function y = f(x) describes a relationship (input-output mapping) from x
to y.

Example 1 The threshold or sign function (bipolar) sgn(x) is defined as

Example 2 The logistic or sigmoid function Sigmoid(x) is defined as

This is a smoothed (differentiable) form of the threshold function.

15
 The McCulloch-Pitts Neuron Equation
Using
the above notation, we can now write down a simple equation for
the output out of a McCulloch-Pitts neuron as a function of its n inputs :

where is the neurons activation threshold.

We can easily see that:

Note that the McCulloch-Pitts neuron is an extremely simplified model of

real biological neurons. Some of its missing features include: non-binary
inputs and outputs, non-linear summation, smooth thresholding,
stochasticity, and temporal information processing.
Nevertheless, McCulloch-Pitts neurons are computationally very powerful.
One can show that assemblies of such neurons are capable of universal
computation.
16
 Networks of McCulloch-Pitts Neurons
One
neuron cant do much on its own. Usually we will have
many neurons labelled by indices and activation flows
between them via synapses with strengths and

17
 The Perceptron
We can connect any number of McCulloch-Pitts neurons together in any way we
like.
An arrangement of one input layer of McCulloch-Pitts neurons feeding forward to
one output layer of McCulloch-Pitts neurons is known as a Perceptron.

Already this is a powerful computational device. Later we shall see variations that
make it even more powerful.

18
 Implementing Logic Gates with
M-P Neurons
We can use McCulloch-Pitts neurons to implement the basic logic
gates.
All we need to do is find the appropriate connection weights and
neuron thresholds to produce the right outputs for each set of inputs.
We shall see explicitly how one can construct simple networks that
perform NOT, AND, and OR.
It is then a well known result from logic that we can construct any
logical function from these three operations.
The resulting networks, however, will usually have a much more
complex architecture than a simple Perceptron.
We generally want to avoid decomposing complex problems into
simple logic gates, by finding the weights and thresholds that work
directly in a Perceptron architecture.
19
 Implementation of Logical NOT,
AND, and OR
In
andeach case we have inputs and outputs , and need to determine the weights
thresholds. It is easy to find solutions by inspection:

20
 ANN
Architectures/Structures/Topolo
gies
Mathematically, ANNs can be represented as weighted directed
graphs. For our purposes, we can simply think in terms of
activation flowing between processing units via one-way
connections. Three common ANN architectures are:
Single-Layer Feed-forward NNs One input layer and one output
layer of processing units. No feed-back connections. (For example, a
simple Perceptron.)
Multi-Layer Feed-forward NNs One input layer, one output layer,
and one or more hidden layers of processing units. No feed-back
connections. The hidden layers sit in between the input and output
layers, and are thus hidden from the outside world. (For example, a
Multi-Layer Perceptron.)
Recurrent NNs Any network with at least one feed-back connection. It
may, or may not, have hidden units. (For example, a Simple Recurrent
Network.) 21
Examples of Network
Architectures

22
 Other Types of
Activation/Transfer Function
Sigmoid Functions These are smooth (differentiable) and monotonically increasing.

The logistic function (sigmoid biner)

Hyperbolic tangent (sigmoid bipolar)

sigmoid_bipolar

Piecewise-Linear Functions Approximations of a sigmoid functions.

23
 Other Types of
Activation/Transfer Function
Symetric saturating linear

Threshold-bipolar

Symetric Hard-Limit

24
 Learning Process
Pada otak manusia, informasi yang dilewatkan dari satu neuron
ke neuron yang lainnya berbentuk rangsangan listrik melalui
dendrit.
Jika rangsangan tersebut diterima oleh suatu neuron, maka neuron
tersebut akan membangkitkan output ke semua neuron yang berhubungan
dengannya sampai informasi tersebut sampai ke tujuannya yaitu
terjadinya suatu reaksi.
Jika rangsangan yang diterima terlalu halus, maka output yang
dibangkitkan oleh neuron tersebut tidak akan direspon.
Tentu saja sangatlah sulit untuk memahami bagaimana otak
manusia bisa belajar. Selama proses pembelajaran, terjadi
perubahan yang cukup berarti pada bobot-bobot yang
menghubungkan antar neuron.
Apabila ada rangsangan yang sama dengan rangsangan yang telah
diterima oleh neuron, maka neuron akan memberikan reaksi dengan 25
 Learning Process
Jaringan syaraf akan mencoba untuk mensimulasikan kemampuan
otak manusia untuk belajar. Jaringan syaraf tiruan juga tersusun
atas neuron-neuron dan dendrit.
Tidak seperti model biologis, jaringan syaraf memiliki struktur yang
tidak dapat diubah, dibangun oleh sejumlah neuron, dan memiliki
nilai tertentu yang menunjukkan seberapa besar koneksi antara
neuron (yang dikenal dengan nama bobot).
Perubahan yang terjadi selama proses pembelajaran adalah
perubahan nilai bobot.
Nilai bobot akan bertambah, jika informasi yang diberikan oleh neuron
yang bersangkutan tersampaikan.
Sebaliknya jika informasi tidak disampaikan oleh suatu neuron ke
neuron yang lain, maka nilai bobot yang menghubungkan keduanya
akan dikurangi.
Pada saat pembelajaran dilakukan pada input yang berbeda, maka 26
 Supervised
Learning
Unsupervis
Paradigma
ed
Learning
Learning

Hybrid

Proses
Learning Error
Correction

Boltzmann
Jenis Learning
Aturan
Learning Hebian
Learning

Competitiv
e Learning
27
 Supervised Learning
Metode pembelajaran pada jaringan syaraf disebut terawasi jika output
yang diharapkan telah diketahui sebelumnya.
Contoh: andaikan kita memiliki jaringan syaraf yang akan digunakan untuk
mengenali pasangan pola, misalkan pada operasi AND:

Pada proses pembelajaran, satu pola input akan diberikan ke satu neuron
pada lapisan input. Pola ini akan dirambatkan di sepanjang jaringan syaraf
hingga sampai ke neuron pada lapisan output. Lapisan output ini akan
membangkitkan pola output yang nantinya akan dicocokkan dengan pola
output targetnya.
Apabila terjadi perbedaan antara pola output hasil pembelajaran dengan
pola target, maka disini akan muncul error. Apabila nilai error ini masih
cukup besar, mengindikasikan bahwa masih perlu dilakukan lebih banyak 28
 Supervised Learning (Hebb
Hebb

Rule)
rule adalah metode pembelajaran yang paling sederhana.
Pada metode ini pembelajaran dilakukan dengan cara memperbaiki nilai
bobot sedemikian rupa sehingga jika ada 2 neuron yang terhubung, dan
keduanya pada kondisi hidup (on) pada saat yang sama, maka bobot
antara keduanya dinaikkan.
Apabila data direpresentasikan secara bipolar, maka perbaikan bobotnya
adalah:

= bobot sinapsis yang menghubungkan neuron i dan neuron j

= nilai neuron input i
= nilai neuron output j
= learning rate
29
Supervised Learning (Algorithm
of Hebb Rule)

30
 Supervised Learning (Example 1
of Hebb Rule)
Misalkan kita ingin membuat jaringan syaraf untuk melakukan
pembelajaran terhadap fungsi OR dengan input dan target
bipolar sebagai berikut:

31
Supervised Learning (Example
of Hebb Rule)

32
 Supervised Learning (Example 2
of Hebb Rule)
Misalkan kita ingin membuat jaringan syaraf untuk melakukan
pembelajaran terhadap fungsi AND dengan 2 input (-1 dan 1) dan
target bipolar sesuai aturan operator AND serta bobot awal dan
bobot bias adalah 0.

33
 Supervised Learning (Example 3
of Hebb Rule)
Misalkan kita ingin membuat jaringan syaraf untuk melakukan
pembelajaran terhadap fungsi OR dengan 3 input (setiap input
mempunyai kemungkinan nilai 0 dan 1) dan bobot awal dan
bobot bias adalah 0 (tentukan fungsi yang sesuai).

34
Perceptron Learning Procedure
with Error Correction Rule

35
Perceptron Rule Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: wi = ct z)xi
Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output W

001 1 0 0000

36
Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: wi = ct z)xi
Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output W

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000

37
Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: wi = ct z)xi
Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output W

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111

38
Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: wi = ct z)xi
Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output W

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111 3 1 0 0 0 0
011 1 0 1111 39
Example
Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Assume a learning rate c of 1 and initial weights all 0: wi = ct z)xi
Training set 0 0 1 -> 0
11 1 -> 1
10 1 -> 1
01 1 -> 0

Pattern Target Weight Vector Net Output W

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111 3 1 0 0 0 0
011 1 0 1111 3 1 0 -1 -1 -1
001 1 0 1000
40
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
Example Assume a learning rate c of 1 and initial weights all 0: w = ct z)x
i i

Training set 0 0 1 -> 0

1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output W

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111 3 1 0 0 0 0
011 1 0 1111 3 1 0 -1 -1 -1
001 1 0 1000 0 0 0 0 0 0
111 1 1 1000 1 1 0 0 0 0
101 1 1 1000 1 1 0 0 0 0
011 1 0 1000 0 0 0 0 0 0
41