Neural Network

Output

Input
Module 5 Neural Networks: Gradient-Based Operations
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then Neurons
passes the sum through the activation function to the other neurons.

Drawing by Santiago Ramón y Cajal (1899) of neurons in the pigeon cerebellum

https://en.wikipedia.org/wiki/Neuroscience
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Neuron or node

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Input Signal
Neuron or node

2
Input Signal

Input Signal

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Input Signal
Neuron or node

2 2 Output
Input Signal

Input Signal

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Input Signal
Neuron or node

2 2 Output Signal or Value

Input Signal

Input Signal

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Input Signal
Neuron or node

2 2 Output Signal or Value

Input Signal

Input Signal
Synapse

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Input Signal 1
These are independent Neuron or node
variable

2 2 Output Signal or Value

Input Signal 2
These are independent
variable

Input Signal 3
These are independent Synapse
variable

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Input Signal 1
These are independent Neuron or node
variable

2 2 Output Signal or Value

Input Signal 2
These are independent
variable

Input Signal 3
These are independent
variable

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Input Signal 1
These are independent Neuron or node Output Signal or Value
variable
Categorical
2 Continues
2
Input Signal 2 Binary
These are independent
variable

Input Signal 3
These are independent
variable

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Output 1
2
Input Signal 1
These are independent Neuron or node
variable
Categorical
2 Continues Output Signal or Value
2
Input Signal 2 Binary
These are independent Output 2
variable

Input Signal 3 2 Output 3

These are independent
variable

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Output 1
2
Input Signal 1
These are independent Neuron or node
variable
Categorical
2 Continues Output Signal or Value
2
Input Signal 2 Binary
These are independent Output 2
variable

Input Signal 3 2 Output 3

These are independent
variable

Same Obervations

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Input Signal 1
w1
These are independent Neuron or node
variable

w2 2 2 Output Signal or Value

Input Signal 2
These are independent W
variable …

Input Signal 3 Weight

These are independent
variable

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Structure of Neurons
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Neuron or node
Input Signal 1
These are independent w1
variable

1st Step
w2
Input Signal 2 2 Output Signal or Value
These are independent W
variable …

Input Signal 3 Weight

These are independent
variable

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Activation Function
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Neuron or node
Input Signal 1
These are independent w1
variable 1st Step
2nd Step
w2
Input Signal 2 1rd Step 2 Output Signal or Value
These are independent W
variable …

Input Signal 3 Weight

These are independent
variable

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Activation Function
Neural network is a set of neurons organized in layers. Each neuron is a
mathematical operation that takes it’s input, multiplies it by it’s weights and then
passes the sum through the activation function to the other neurons.

Neuron or node
Input Signal 1
These are independent w1
variable 1st Step
2nd Step
w2
Input Signal 2 1rd Step 2 Output Signal or Value
These are independent W
variable …

Input Signal 3 Weight

These are independent
variable

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Threshold Activation Function
A binary step function is a threshold-based activation function. If the input value
is above or below a certain threshold, the neuron is activated and sends exactly
the same signal to the next layer.

0 x

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Sigmoid Activation Function
Smooth gradient, preventing “jumps” in output values.
Output values bound between 0 and 1, normalizing the output of each neuron.
Clear predictions—For X above 2 or below -2, tends to bring the Y value (the
prediction) to the edge of the curve, very close to 1 or 0. This enables clear
predictions.

0 x

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
ReLU Activation Function
•Computationally efficient—allows the network to converge very
quickly
•Non-linear—although it looks like a linear function, ReLU has a
derivative function and allows for back propagation

0 x

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Hyperbolic Tangent Activation Function
•Zero centered—making it easier to model inputs that have
strongly negative, neutral, and strongly positive values.
•Otherwise like the Sigmoid function.

0 x

https://commons.wikimedia.org/wiki/File:Neuron.jpg
Module 5 Neural Networks: Gradient-Based Operations
Activation Function

Input 1 x1

Input 2 x1
Output
layer
Input n x

Input
layer Hidden
layer
Module 5 Neural Networks: Gradient-Based Operations
Working of Neural Network

Input 1
w1 Y / output

w2
Input 2

Wn
Input 3 Output Layer

Input Layer
Module 5 Neural Networks: Gradient-Based Operations
Working of Neural Network

Area x1

y
Bedroom x1

Distance x3

Output Layer
Age x4

Hidden Layer

Input Layer
Module 5 Neural Networks: Gradient-Based Operations
Working of Neural Network

Area x1

y
Bedroom x1

Distance x3

Output Layer
Age x4

Hidden Layer

Input Layer