Material for The Mathematical Engineering of Deep Learning. See the actual book content on deeplearningmath.org or (when it is out) purchase the book from CRC press.
This repository contains general supporting material for the book.
Below is a detailed list of the source code used for creating figures and tables in the book. We use Julia, Python, or R and the code is sometimes in stand alone files, sometimes in Jupyter notebooks, sometimes as R Markdown, and sometimes in Google Colab. Many of our static illustrations were created using TikZ by Ajay Hemanth and Vishnu Prasath with the source of their illustrations also available so you can adapt it for purposes.
| Figure | Topic | Source Code |
|---|---|---|
| 1.1 | Fast.ai example | Python Google Colab |
| 1.3 | Architectures | TikZ(a), TikZ(b), TikZ(c), TikZ(d), TikZ(e), TikZ(f) |
| 1.4 | Neurons | TikZ(b), TikZ(d) |
| 1.5 | Data on earth | Julia |
| Figure | Topic | Source Code |
|---|---|---|
| 2.1 | Supervised Learning | TikZ |
| 2.2 | Unsupervised Learning | TikZ |
| 2.3 | Simple regression | R |
| 2.4 | Breast Cancer ROC curves | R |
| 2.5 | Least Squares | TikZ |
| 2.6 | Loss functions | Julia |
| Table 2.1 | Linear MNIST classification | Julia |
| 2.7 | Gradient Descent Learning Rate | Python |
| 2.8 | Loss Landscape | R |
| 2.9 | Generalization and Training | TikZ or Julia |
| 2.10 | Polynomial fit | R |
| 2.11 | K-fold cross validation | TikZ |
| 2.12 | K-means clustering | R |
| 2.13 | K-means image segmentation | R |
| 2.14 | Breast Cancer PCA | R |
| 2.15 | SVD Compression | Julia |
| Figure | Topic | Source Code |
|---|---|---|
| 3.1 | Logistic regression model curves | R(a) R(b) |
| 3.2 | Linear decision boundary for logistic regression | R |
| 3.3 | Components of an artificial neuron | TikZ |
| 3.4 | Loss landscape of MSE vs. CE on logistic regression | Python |
| 3.5 | Evolution of gradient descent learning in logistic regression | R |
| 3.6 | Multinomial regression for classification | R |
| 3.7 | Shallow multi-output neural network with softmax | TikZ |
| 3.8 | Feature engineering in simple logistic regression | R |
| 3.9 | Non-linear classification decision boundaries with feature engineering in logistic regression | R |
| 3.10 | Non-linear classification decision boundaries with feature engineering in multinomial regression | R |
| 3.11 | Single hidden layer autoencoder | TikZ |
| 3.12 | Autoencoder projections of MNIST including using PCA | R |
| 3.13 | Reconstruction of MNIST using autoencoders | R |
| 3.14 | Denoising autoencoder | TikZ |
| 3.15 | Denoising autoencoder used for MNIST | R |
| Figure | Topic | Source Code |
|---|---|---|
| 4.1 | Convexity and local/global extrema | Python |
| 4.2 | Saddle points | Python |
| 4.3 | Various parameters of gradient descent | Python |
| 4.4 | Stochastic gradient descent | Python |
| 4.5 | Early stopping in deep learning | Python |
| 4.6 | Momentum and RMS Prop enhancing gradient descent | Python |
| 4.7 | Nesterov and other variations of gradient descent | Python |
| 4.8 | Performance of second order methods | |
| 4.9 | Computational graph for automatic differentiation | TikZ |
| Figure | Topic | Source Code |
|---|---|---|
| 5.1 | Fully Connected Feedforward Neural Networks | TikZ(a), TikZ(b) |
| 5.2 | Arbitrary function approximation with neural nets | Julia |
| 5.3 | Binary classification with increasing depth | R |
| 5.4 | A continuous multiplication gate with 4 hidden units | TikZ |
| 5.5 | Several common scalar activation functions | Julia |
| 5.6 | Flow of information in general back propagation | TikZ |
| 5.7 | Simple neural network hypothetical example | TikZ |
| 5.8 | Flow of information in standard neural network back propagation | TikZ |
| 5.9 | Computational graph for batch normalization | TikZ |
| 5.10 | The effect of dropout | TikZ |
| Figure | Topic | Source Code |
|---|---|---|
| 6.1 | TBD |
| Figure | Topic | Source Code |
|---|---|---|
| 7.1 | TBD |
| Figure | Topic | Source Code |
|---|---|---|
| 8.1 | TBD |
| Figure | Topic | Source Code |
|---|---|---|
| 9.1 | TBD |
| Figure | Topic | Source Code |
|---|---|---|
| 10.1 | TBD |