u
Convulotional neural net
Data statistics
Outline
▪ Brief recap of neural networks
• Application: autoencoders
▪ Convolutional neural networks
3
Introduction Linear regression Logistic regression
Feature engineering Data statistics Naive Bayes
KNN Clustering Dimensionality reduction
Neural networks Convolutional neural Decision-trees
networks
Last time
“3-layer Neural Net”, or
“2-hidden-layer Neural Net”
Additional resource: https://www.deeplearningbook.org/
Example application of Neural Networks
Autoencoder
Recall - dimensionality reduction
PCA
Autoencoder
Autoencoder vs. PCA
Top: Some examples of the original MNIST
test samples
Middle: Reconstructed output from an auto-
encoder with a latent space of 8 dimensions
This auto-encoder uses convolutional layers, and
was trained on the MNIST training set
Bottom: Reconstructed output from PCA with
8 latent dimensions
Image credit: F. Fleuret, Deep Learning (EPFL)
Autoencoder
Autoencoder vs. PCA
Training for NN
Goal of optimization in ML:
N (i) (i) (i)
Minimize cost over batch: ∑ L =L(y , ŷ ) of the i
i=1
-th training example of batch
Want the optimization to:
• Converge quickly
• Find a good local minima (or even global minima)
Gradient descent (and variants) is the preferred way to
optimize neural networks
Choice of optimizer and hyper-parameters affect speed of
convergence and kind of local minima found
A. Amini et al. Spatial Uncertainty Sampling for
End-to-End Control, 2019
Gradient descent variants
(Vanilla / Batch) Gradient descent (GD):
1 N
▪ J= N
∑i=1 L (i)
▪ Weights updated after calculating the gradient over the entire dataset
• slow
• requires large memory
Stochastic gradient descent (SGD):
▪ J = L (i)
▪ Weights updated after calculating the gradient of a single example
• requires much less memory than GD
• high variance in parameter updates
Mini-batch Gradient descent
1 Nb
▪ J= Nb
∑i=1 L (i)
▪ Weights updated after calculating the gradient over the entire dataset
• Faster than SGD
• Reduces variance of gradient estimation
Optimization
Learning rate
Image credit: Jeremy Jordan (https://www.jeremyjordan.me/nn-learning-rate/)
Optimization
Optimizers
Variants of gradient descent are
commonly used in practice to speed-up
and improve convergence:
▪ Momentum update
▪ Nesterov Accelerated Gradient (NAG)
▪ Adam
▪ and more…
Convolutional Neural Networks
Real-World Problem
Detecting and Classifying Pavement Distress
! ! ! !
! !
Why? On-time preventive maintenance
Lack of on-time maintenance
• x3 the maintenance cost
• traffic delay
• more fuel consumption
• accidents
• …
Automatic pavement distress monitoring
!
Pavement
distress
Convolutional Neural Networks (CNN)
Intro - Handling images with fully-connected NN
3072x1 input
3x32x32 image
Flatten
height (32)
By flattening, spatial structure gets lost!
width (32)
depth (3)
→ for the 3 color channels: R, G, B
Convolutional Neural Networks
Intro - Handling images with fully-connected NN
A fully-connected neural net:
▪ Requires flattening the image
→ spatial structure gets lost Flatten
▪ Doesn’t scale well to large images
• e.g. 1024x1024x3 image results in
3’145’728 weights for each neuron of first
hidden layer
How to efficiently model correlation between neighboring pixels?
=> Convolutional Neural Networks
Convolution definition
Convolution of a signal with a filter
Convolution - what to do at boundaries
Convolution to get features
Filters can approximate what happens in neighborhood with a few numbers:
give average value of signal in a neighborhood
sharpen the signal
blur the signal
approximate derivative of a signal in a neighborhood:
approximate second derivative of a signal in a neighborhood
Convolution with a filter is a linear function
Convolution extensions
Example of image convolution
-1 -1 -1
-1 8 -1
-1 -1 -1
https://muthu.co/basics-of-image-convolution/
Convolutional
2D Convolution computation example
To show how the convolution operation is computed, let’s use a simpler example:
5x5 input, 3x3 filter
Input (5x5) Filter (3x3)
1 0 3 0 2 1 -1 0
0 3 4 0 2 0 2 -3
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0
Convolutional
Convolution computation example
1 0 3 0 2 1 -1 0 2
0 3 4 0 2
⋅ 0 2 -2 =
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 1x1 + 0x(-1) + 3x0 + 0x0 + 3x2 +
4x(-2) + 1x(-1) + 0x0 + 2x2 + 0
=2
Convolutional
Convolution computation example
1 0 3 0 2 1 -1 0 2 5
0 3 4 0 2
⋅ 0 2 -2 =
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 0x1 + 3x(-1) + 0x0 + 3x0 + 4x2 +
0x(-2) + 0x(-1) + 2x0 + 0x2 + 0 b
ye
↓
=5
I
Convolutional
Convolution computation example
1 0 3 0 2 1 -1 0 2 5 -1
0 3 4 0 2
⋅ 0 2 -2 =
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 3x1 + 0x(-1) + 2x0 + 4x0 + 0x2 +
2x(-2) + 2x(-1) + 0x0 + 1x2 + 0
= -1
Convolutional
Convolution computation example
1 0 3 0 2 1 -1 0 2 5 -1
0 3 4 0 2
⋅ 0 2 -2 = -15
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 0x1 + 3x(-1) + 4x0 + 1x0 + 0x2 +
2x(-2) + 8x(-1) + 12x0 + 0x2 + 0
= -15
Convolutional
Convolution computation example
1 0 3 0 2 1 -1 0 2 5 -1
0 3 4 0 2
⋅ 0 2 -2 = -15 -7
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 3x1 + 4x(-1) + 0x0 + 0x0 + 2x2 +
0x(-2) + 12x(-1) + 0x0 + 1x2 + 0
= -7
Convolutional
Convolution computation example
1 0 3 0 2 1 -1 0 2 5 -1
0 3 4 0 2
⋅ 0 2 -2 = -15 -7 2
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 4x1 + 0x(-1) + 2x0 + 2x0 + 0x2 +
1x(-2) + 0x(-1) + 1x0 + 0x2 + 0
=2
Convolutional
Convolution computation example
1 0 3 0 2 1 -1 0 2 5 -1
0 3 4 0 2
⋅ 0 2 -2 = -15 -7 2
1 0 2 0 1 -1 0 2 31
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 1x1 + 0x(-1) + 2x0 + 8x0 + 12x2
+ 0x(-2) + 0x(-1) + 6x0 + 3x2 + 0
= 31
Convolutional
Convolution computation example
1 0 3 0 2 1 -1 0 2 5 -1
0 3 4 0 2
⋅ 0 2 -2 = -15 -7 2
1 0 2 0 1 -1 0 2 31 -8
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 0x1 + 2x(-1) + 0x0 + 12x0 + 0x2 +
1x(-2) + 6x(-1) + 3x0 + 2x2 + 0
= -8
Convolutional Neural Networks
Convolution computation example
1 0 3 0 2 1 -1 0 2 5 -1
0 3 4 0 2
⋅ 0 2 -2 = -15 -7 2
1 0 2 0 1 -1 0 2 31 -8 1
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 2x1 + 0x(-1) + 1x0 + 0x0 + 1x2 +
0x(-2) + 3x(-1) + 2x0 + 0x2 + 0
=1
Convolution example
Example from ML4Engineers book.
Convolutional Neural Networks (CNN)
Convolution Layer
3x32x32 image
In PyTorch, images are represented as
(CxHxW)
• C: number of channels (depth)
• H: height
• W: width
height (32)
A pixel can be represented by a vector of 3 color (R, G, B)
intensities
width (32) I(c, h, w)
depth (3)
Convolutional Neural Networks
Convolution Layer
3x32x32 image Filters always extend the full
depth of the input volume
3x5x5 filter
Convolve the filter with the image
32 i.e. “slide over the image spatially,
computing dot products”
32
3 Note:
Filters are sometimes referred to as kernels
Convolutional Neural Networks
Convolution Layer
3x32x32 image
3x5x5 filter (w)
32
1 number:
32 ▪ The result of taking a dot product between the
filter and a small 3x5x5 chunk of the image
3 • (i.e. 5x5x3=75-dimensional dot product + bias)
Convolutional Neural Networks
Convolution Layer
3x32x32 image activation map
3x5x5 filter (w)
28
32 Convolve (slide) over all
spatial locations
32 28
3 1
Convolutional Neural Networks
Convolution Layer
3x32x32 image activation map
28
32 Convolve (slide) over all
spatial locations
32 28
3 1
Convolutional Neural Networks
Convolution Layer
3x32x32 image activation map
28
32 Convolve (slide) over all
spatial locations
32 28
3 1
Convolutional Neural Networks
Convolution Layer
3x32x32 image activation map
28
32 Convolve (slide) over all
spatial locations
32 28
3 1
Convolutional Neural Networks
Convolution Layer
3x32x32 image activation map
28
32 Convolve (slide) over all
spatial locations
32 28
3 1
Convolutional Neural Networks
Convolution Layer
3x32x32 image activation map
28
32 Convolve (slide) over all
spatial locations
32 28
3 1
Convolutional Neural Networks
Convolution Layer
3x32x32 image activation map
28
32 Convolve (slide) over all
spatial locations
32 28
3 1
Convolutional Neural Networks
Convolution Layer
3x32x32 image activation maps
3x5x5 filter
28
32
32 28
Consider a second filter
3 Perform same convolution operation with 1
this filter to get a second activation map
Pooling
Pooling example
Max-pooling, stride of 2
Input (1x6x6) Output (1x3x3)
3 0 1 0 2 4
0 1 8 12 0 0
3 12 4
4 0 0 3 2 2
4 3 2
2 0 1 0 1 1
3 6 9
3 2 0 6 0 5
1 0 6 0 0 9
Pooling example
Example from ML4Engineers book.
of stude a
max
pooky
a
ppked to the
image
(after Laplacian Kernel
applied to
image
Convolutional Neural Networks
Pooling layer
CNNs may include pooling layers to reduce the spatial size of the representation
Pooling layers require two hyper-parameters: their spatial extent F and their stride S
▪ Most common layer uses 2x2 filters of stride 2 (F = 2, S = 2)
Convolutional applied with a stride
We can also apply convolution with a stride of 2
Input (1x5x5) Filter (1x3x3)
1 0 3 0 2 1 -1 0
0 3 4 0 2 0 2 -3
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0
Convolutional Neural Networks
Changing the stride
Back to our simple example, but change to stride of 2
Input (1x5x5) Filter (1x3x3) Output (1x2x2)
1 0 3 0 2 1 -1 0 2
0 3 4 0 2
⋅ 0 2 -2 =
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 1x1 + 0x(-1) + 3x0 + 0x0 + 3x2 +
4x(-2) + 1x(-1) + 0x0 + 2x2 + 0
=2
Convolutional Neural Networks
Changing the stride
Back to our simple example, but change to stride of 2
Input (1x5x5) Filter (1x3x3) Output (1x2x2)
1 0 3 0 2 1 -1 0 2 -1
0 3 4 0 2
⋅ 0 2 -2 =
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 3x1 + 0x(-1) + 2x0 + 4x0 + 0x2 +
2x(-2) + 2x(-1) + 0x0 + 1x2 + 0
= -1
Convolutional Neural Networks
Changing the stride
Back to our simple example, but change to stride of 2
Input (1x5x5) Filter (1x3x3) Output (1x2x2)
1 0 3 0 2 1 -1 0 2 -1
0 3 4 0 2
⋅ 0 2 -2 = 31
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 1x1 + 0x(-1) + 2x0 + 8x0 + 12x2
+ 0x(-2) + 0x(-1) + 6x0 + 3x2 + 0
= 31
Convolutional Neural Networks
Changing the stride
Back to our simple example, but change to stride of 2
Input (1x5x5) Filter (1x3x3) Output (1x2x2)
1 0 3 0 2 1 -1 0 2 -1
0 3 4 0 2
⋅ 0 2 -2 = 31 1
1 0 2 0 1 -1 0 2
8 12 0 1 0 Bias: b = 0
0 6 3 2 0 2x1 + 0x(-1) + 1x0 + 0x0 + 1x2 +
0x(-2) + 3x(-1) + 2x0 + 0x2 + 0
=1
Convolutional Neural Networks
Zero-padding
Height and width shrink quite quickly due to the repeated convolutions
To avoid this, we can add zero-padding:
Zero-padded input (1x7x7)
Input (1x5x5) 0 0 0 0 0 0 0
1 0 3 0 2 0 1 0 3 0 2 0
0 3 4 0 2 0 0 3 4 0 2 0
1 0 2 0 1 Zero-padding = 1 0 1 0 2 0 1 0
8 12 0 1 0 0 8 12 0 1 0 0
0 6 3 2 0 0 0 6 3 2 0 0
0 0 0 0 0 0 0
If we use a 3x3 filter with a stride of 1 on the padded input, we get a 5x5 output
→ same size as input
Convolutional Neural Networks
Convolution layer summary
The convolution layer:
▪ Accepts a volume of size Cin × H1 × W1
▪ Requires four hyper-parameters:
• Number of filters K
• Spatial extent of filters F
• Stride S
• Amount of zero padding P
▪ Produces of a volume of size Cout × H2 × W2 where:
• Cout = K
• H2 = (H1 − F + 2P)/S + 1
• W2 = (W1 − F + 2P)/S + 1
Convolutional Neural Networks
Convolution layer summary
The convolution layer:
▪ Accepts a volume of size Cin × H1 × W1
▪ Requires four hyper-parameters:
• Number of filters K
• Spatial extent of filters F
• Stride S
• Amount of zero padding P
▪ Produces of a volume of size Cout × H2 × W2 where: Note:
• Cout = K There are F ⋅ F ⋅ Cin weights per filter,
for a total of (F ⋅ F ⋅ Cin) ⋅ K weights
• H2 = (H1 − F + 2P)/S + 1
and K biases per layer
• W2 = (W1 − F + 2P)/S + 1
Convolutional neural net
From: Machine Learning for Engineers book
Convolutional Neural Networks
Perception tasks
Convolutional Neural Networks
Optional
Popular architectures
LeNet-5
LeCun et al. ,1998
----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Conv2d-1 [-1, 6, 28, 28] 156
-1 in output shape represents
ReLU-2 [-1, 6, 28, 28] 0 the mini-batch dimension
MaxPool2d-3 [-1, 6, 14, 14] 0
Conv2d-4 [-1, 16, 10, 10] 2,416
ReLU-5 [-1, 16, 10, 10] 0
MaxPool2d-6 [-1, 16, 5, 5] 0
Linear-7 [-1, 120] 48,120
ReLU-8 [-1, 120] 0
Linear-9 [-1, 84] 10,164
ReLU-10 [-1, 84] 0
Linear-11 [-1, 10] 850
Softmax-12 [-1, 10] 0
================================================================
Total params: 61,706
Convolutional Neural Networks
Optional
Popular architectures
AlexNet
Krizhevsky et al., 2012
Winner of ImageNet Competition 2012
Convolutional Neural Networks
Optional
Popular architectures ----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Conv2d-1 [-1, 64, 224, 224] 1,792
ReLU-2 [-1, 64, 224, 224] 0
Conv2d-3 [-1, 64, 224, 224] 36,928
ReLU-4 [-1, 64, 224, 224] 0
MaxPool2d-5 [-1, 64, 112, 112] 0
Conv2d-6 [-1, 128, 112, 112] 73,856
ReLU-7 [-1, 128, 112, 112] 0
Conv2d-8 [-1, 128, 112, 112] 147,584
ReLU-9 [-1, 128, 112, 112] 0
MaxPool2d-10 [-1, 128, 56, 56] 0
Conv2d-11 [-1, 256, 56, 56] 295,168
ReLU-12 [-1, 256, 56, 56] 0
Conv2d-13 [-1, 256, 56, 56] 590,080
ReLU-14 [-1, 256, 56, 56] 0
Conv2d-15 [-1, 256, 56, 56] 590,080
ReLU-16 [-1, 256, 56, 56] 0
MaxPool2d-17 [-1, 256, 28, 28] 0
Conv2d-18 [-1, 512, 28, 28] 1,180,160
ReLU-19 [-1, 512, 28, 28] 0
Conv2d-20 [-1, 512, 28, 28] 2,359,808
ReLU-21 [-1, 512, 28, 28] 0
Conv2d-22 [-1, 512, 28, 28] 2,359,808
ReLU-23 [-1, 512, 28, 28] 0
MaxPool2d-24 [-1, 512, 14, 14] 0
Conv2d-25 [-1, 512, 14, 14] 2,359,808
ReLU-26 [-1, 512, 14, 14] 0
Conv2d-27 [-1, 512, 14, 14] 2,359,808
ReLU-28 [-1, 512, 14, 14] 0
Conv2d-29 [-1, 512, 14, 14] 2,359,808
ReLU-30 [-1, 512, 14, 14] 0
MaxPool2d-31 [-1, 512, 7, 7] 0
VGG16 Linear-32
ReLU-33
[-1, 4096]
[-1, 4096]
102,764,544
0
Simonian & Zisserman, 2014
Dropout-34 [-1, 4096] 0
Linear-35 [-1, 4096] 16,781,312
ReLU-36 [-1, 4096] 0
Dropout-37 [-1, 4096] 0
Linear-38 [-1, 1000] 4,097,000
Softmax-39 [-1, 1000] 0
================================================================
Total params: 138,357,544
Convolutional Neural Networks
Optional
Popular architectures
GoogLeNet (Inception v1)
Szegedy et al., 2014
ConvNets have been getting deeper and deeper
e.g. ResNet-152 (He et al, 2015) → 152 layers
Example application: transfer learning
Train network for a task
Example: image classification
Requires large number of training data, and training resources
Modify the trained network for a different task (transfer learning)
Why? Can address limited data and time/resources for training
Case study 6.5 from Machine Learning for Engineers book: “Finding volcanos on
Venus with pre-fit models”
Summary - exercises
