Deep Learning Tutorial
Brains, Minds, and Machines Summer Course 2018
TA: Eugenio Piasini & Yen-Ling Kuo
Roadmap
● Supervised Learning with Neural Nets
● Convolutional Neural Networks for Object Recognition
● Recurrent Neural Network
● Other Deep Learning Models
Supervised Learning
with Neural Nets
General references:
Hertz, Krogh, Palmer 1991
Goodfellow, Bengio, Courville 2016
Supervised learning
Given example input-output pairs (X,Y),
learn to predict output Y from input X
Logistic regression, support vector machines, decision trees, neural networks...
Binary classification:
simple perceptron
(McCulloch & Pitts 1943)
Perceptron learning rule
(Rosenblatt 1962)
g is a nonlinear activation function, in this case
Linear separability
Simple perceptrons can only learn to solve linearly separable problems (Minsky and Papert 1969).
We can solve more complex problems by composing many units in multiple layers.
Multilayer perceptron (MLP)
(“forward propagation”)
MLPs are universal function approximators (Cybenko 1989; Hornik 1989).
(under some assumptions… exercise: show that if g is linear, this architecture reduces to a simple perceptron)
Deep vs shallow
Universality: “shallow” MLPs with one hidden layer can represent any continuous function to arbitrary
precision, given a large enough number of units. But:
● No guarantee that the number of required units is reasonably small (expressivity).
● No guarantee that the desired MLP can actually be found with our chosen learning method
(learnability).
Two motivations for using deep nets instead (see Goodfellow et al 2016, section 6.4.1):
● Statistical: deep nets are compositional, and naturally well suited to representing hierarchical
structures where simpler patterns are composed and reused to form more complex ones
recursively. It can be argued that many interesting structures in real world data are like this.
● Computational: under certain conditions, it can be proved that deep architectures are more
expressive than shallow ones, i.e. they can learn more patterns for a given total size of the network.
Problem: compute all
Key insights: the loss depends
● on the weights w of a unit only through that unit’s
Backpropagation
activation h
● on a unit’s activation h only through the activation of
those units that are downstream from h.
(Rumelhart, Hinton, Williams 1986)
The “errors” being
backpropagated
These give the gradient of the loss with respect to the weights,
which you can then use with your favorite gradient descent method.
Backpropagation - example
(exercise: derive gradient wrt bias terms b)
The Navy revealed the embryo of The perceptron has shown itself
an electronic computer today that it worthy of study despite (and even
expects will be able to walk, talk, because of!) its severe limitations. It
see, write, reproduce itself and has many features to attract
be conscious of its existence […] attention: its linearity; its intriguing
Dr. Frank Rosenblatt, a research learning theorem; its clear
psychologist at the Cornell paradigmatic simplicity as a kind of
Aeronautical Laboratory, Buffalo, parallel computation. There is no
said Perceptrons might be fired to reason to suppose that any of
the planets as mechanical space these virtues carry over to the
explorers. many-layered version.
Nevertheless, we consider it to be
The New York Times an important research problem to
July 8th, 1958 elucidate (or reject) our intuitive
judgement that the extension to
multilayer systems is sterile.
Minsky and Papert 1969
(section 13.2)
Convolutional Neural Networks
for Object Recognition
General (excellent!) reference:
“Convolutional Networks for Visual Recognition”, Stanford university
http://cs231n.stanford.edu/
Traditional Object Detection/Recognition Idea
● Match low-level
vision features
(e.g. edge, HOG,
SIFT, etc)
● Parts-based
models (Lowe 2004)
Learning the features - inspiration from neuroscience
Hubel and Wiesel:
● Topographic organization of
connections
● Hierarchical organization of
simple/complex cells
(Hubel and Wiesel 1962)
(Fukushima 1980)
“Canonical” CNN structure
INPUT -> [[CONV -> RELU]*K -> POOL?]*L -> [FC -> RELU]*M -> FC
Credit: cs231n.github.io
Four basic operations:
1. Convolution
2. Nonlinearity (ReLU)
3. Pooling
4. Fully connected layers
(LeCun et al 1998)
2D Convolution
Example: blurring an image
Replacing each pixel with an
average of its neighbors
2D Convolution
kernel / filter
Input image Output image
2D Convolution
kernel / filter
Input image Output image
2D Convolution
kernel / filter
Input image Output image
2D Convolution
kernel / filter
Input image Output image
2D Convolution
kernel / filter
Input image Output image
2D Convolution
If N=input size, K=filter size, S=stride
(stride is the size of the step you take
on the input every time you move by
one on the output)
Output size = (N-K)/S + 1
kernel / filter
Input image Output image
N=32, K=5, S=1 →(N-K)/S + 1 = 28
More on convolution sizing
32 5x5x3 filter 32 28
1 1
32 32 28
1
3
3
Input depth = # of channels in previous layer
(often 3 for input layer (RGB); can be arbitrary Output depth = # of filters
for deeper layers) (feature maps)
Convolve with Different Filters
Ⓧ =
Convolution (with learned filters)
● Dependencies are local input
● Filter has few parameters to learn Multiple filters ...
○ Share the same parameters
across different locations
Feature map
Fully Connected vs. Locally Connected
Credit: Ranzato’s
CVPR 2014 tutorial
output
Non-linearity
input
● Rectified linear function (ReLU)
○ Applied per-pixel, output = max(0, input)
Input feature map Output feature map
Pooling
● Reduce size of representation in following layers
● Introduce some invariance to small translation
Image credit: http://cs231n.github.io/convolutional-networks/
Learning
LeNet - LeCun et al 1998
Backpropagation, gradient descent
Key evolutionary steps
Neocognitron - Fukushima 1980
Inspired by Hubel and Wiesel
“Convolutional” structure,
alternating “pooling” layers
AlexNet - Krizhevsky et al 2012
Larger, deeper network (~10^7 params), much more data (ImageNet -
~10^6 images), more compute (incl. GPUs), better regularization (Dropout)
Image classification Image retrieval
test
image
smallest Euclidian distance to test image
But also object detection, image segmentation, captioning...
Recurrent Neural Network
Handling Sequential Information
● Natural language processing: sentences, translations
● Speech / Audio: signal processing, speech recognition
● Video: action recognition, captioning
● Sequential decision making / Planning
● Time-series data
● Biology / Chemistry: protein sequences, molecule structures
● ...
Dynamic System / Hidden Markov Model
Classical form of a dynamic system Hidden Markov Model
With an external signal x
Recurrent Network / RNN
● A general form to process a sequence.
y ○ Applying a recurrence formula at each time step
● The state consists of a vector h.
It summarizes input up to time t.
RNN h
New state Old state Input at time t
A function with
x parameter W
Processing a Sequence: Unrolling in Time
I like this course
RNN RNN RNN RNN
predict predict predict predict
prediction prediction prediction prediction
Training: Backpropagation Through Time
I like this course
RNN RNN RNN RNN
predict predict predict predict
prediction prediction prediction prediction
loss loss loss loss
PRP VBP DT NN
Total loss
Parameter Sharing Across Time
● The parameters are shared and derivatives are accumulated.
● Make it possible to generalize to sequences of different lengths.
t
Vanishing Gradient
X f f f Loss
● expanded quickly!
○ |.| > 1, gradient explodes → clipping gradients
○ |.| < 1, gradient vanishes → introducing memory via LSTMs, GRUs
● Have problem in learning long-term dependency.
Long Short Term Memory (LSTM)
● Introducing gates to
optionally let information
flow through.
○ An LSTM cell has three
gates to protect and
control the cell state.
Forget the Selected update Output certain
irrelevant part of cell state values parts of the
previous state cell state Image credit: http://harinisuresh.com/2016/10/09/lstms/
Flexibility of RNNs
Image Sentiment Machine POS
Captioning Classification Translation Tagging
Image credit: Andrej Karpathy
Other Deep Learning Models
Auto-encoder
● Learning representations
○ a good representation should keep the information well
Encoder Decoder
Original input Learned Reconstructed
representation image
○ → objective: minimize reconstruction error
[LeCun, 1987]
latent variables:
color, shape, position, ...
Generative Models
● What are the learned representations?
observed data
○ One view: latent variables to generate the observed data
● Goal of learning a generative model: to recover p(x) from data
Desirable properties Problem
Sampling new data Directly computing
Evaluating likelihood of data
Extracting latent features
is intractable!
Adapt from IJCAI 2018 deep generative model tutorial
Variational Autoencoder (VAE)
p(z) p(x|z)
● Idea: approximate p(z|x)
with a simpler, tractable q(z|x)
z Decoder x
● Learning objective
Reconstruction error
z Encoder x
q(z|x) Measure how close q is to p
[Kingma et al., 2013]
Generative Adversarial Network (GAN)
● An implicit generative model, formulated as a minimax game.
○ The discriminator is trying to distinguish real and fake samples.
○ The generator is trying to generate fake samples to fool the discriminator.
[Goodfellow et al., 2014]
● Link to the slides
Thanks & Questions? ○ https://goo.gl/pUXdc1
● Hands-on session on
Monday!
Yen-Ling Kuo (
[email protected])
