Applied Machine Learning for Engineers
FS 2020 - B. Vennemann
Artificial Neural Networks
In [1]:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
np.random.seed(0)
Install Tensorflow
conda install tensorflow
Update Numpy
conda update numpy
OSX Error workaround
conda install nomkl
Check installation
In [2]:
import tensorflow as tf
from tensorflow import keras
print("Tensorflow version:", tf.__version__)
print("Keras version:", keras.__version__)
Tensorflow version: 2.0.0
Keras version: 2.2.4-tf
Load the MNIST dataset
In [3]:
mnist = keras.datasets.mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()
print("Training set size :", X_train.shape)
print("Test set size :", X_test.shape)
Training set size : (60000, 28, 28)
Test set size : (10000, 28, 28)
Display the first 20 images
In [4]:
plt.figure(figsize=(18, 10))
for i in range(20):
plt.subplot(4, 5, i+1)
plt.imshow(X_train[i].reshape((28, 28)), cmap='gray')
plt.text(24, 3, str(y_train[i]), fontsize=12, color='white')
plt.xticks([])
plt.yticks([])
plt.show()
Preprocess data
In [5]:
print("Max:", X_train.max())
print("Min:", X_train.min())
Max: 255
Min: 0
In [6]:
X_train = X_train / 255.0
X_test = X_test / 255.0
Create validation set to monitor training
In [7]:
X_train, X_val = X_train[:50000], X_train[50000:]
y_train, y_val = y_train[:50000], y_train[50000:]
print("Training set size :", X_train.shape)
print("Validation set size :", X_val.shape)
print("Test set size :", X_test.shape)
Training set size : (50000, 28, 28)
Validation set size : (10000, 28, 28)
Test set size : (10000, 28, 28)
Create the ANN using the Keras Sequential API
In [8]:
model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28, 28])) # turns (batch_size x 28 x 28) into
(batch_size x 784)
model.add(keras.layers.Dense(300, activation="relu")) # hidden layer
model.add(keras.layers.Dense(10, activation="softmax")) # output layer
or creating the ANN in one go
In [9]:
model = keras.models.Sequential([
keras.layers.Flatten(input_shape=[28, 28]),
keras.layers.Dense(300, activation="relu"),
keras.layers.Dense(10, activation="softmax")
])
Display the model
In [10]:
model.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten_1 (Flatten) (None, 784) 0
_________________________________________________________________
dense_2 (Dense) (None, 300) 235500
_________________________________________________________________
dense_3 (Dense) (None, 10) 3010
=================================================================
Total params: 238,510
Trainable params: 238,510
Non-trainable params: 0
_________________________________________________________________
Compile the model
In [11]:
model.compile(loss="sparse_categorical_crossentropy",
optimizer="sgd",
metrics=["accuracy"])
Train the model
In [12]:
history = model.fit(X_train, y_train, epochs=20, batch_size=32,
validation_data=(X_val, y_val))
Train on 50000 samples, validate on 10000 samples
Epoch 1/20
50000/50000 [==============================] - 26s 514us/sample - loss: 0.667
8 - accuracy: 0.8361 - val_loss: 0.3628 - val_accuracy: 0.9034
Epoch 2/20
50000/50000 [==============================] - 20s 400us/sample - loss: 0.352
0 - accuracy: 0.9039 - val_loss: 0.2929 - val_accuracy: 0.9203
Epoch 3/20
50000/50000 [==============================] - 18s 351us/sample - loss: 0.301
3 - accuracy: 0.9161 - val_loss: 0.2614 - val_accuracy: 0.9288
Epoch 4/20
50000/50000 [==============================] - 16s 321us/sample - loss: 0.269
9 - accuracy: 0.9253 - val_loss: 0.2410 - val_accuracy: 0.9340
Epoch 5/20
50000/50000 [==============================] - 16s 328us/sample - loss: 0.246
6 - accuracy: 0.9314 - val_loss: 0.2224 - val_accuracy: 0.9398
Epoch 6/20
50000/50000 [==============================] - 18s 351us/sample - loss: 0.226
9 - accuracy: 0.9367 - val_loss: 0.2068 - val_accuracy: 0.9435
Epoch 7/20
50000/50000 [==============================] - 20s 400us/sample - loss: 0.210
7 - accuracy: 0.9415 - val_loss: 0.1923 - val_accuracy: 0.9469
Epoch 8/20
50000/50000 [==============================] - 19s 379us/sample - loss: 0.196
4 - accuracy: 0.9448 - val_loss: 0.1827 - val_accuracy: 0.9505
Epoch 9/20
50000/50000 [==============================] - 15s 301us/sample - loss: 0.184
0 - accuracy: 0.9487 - val_loss: 0.1732 - val_accuracy: 0.9540
Epoch 10/20
50000/50000 [==============================] - 16s 310us/sample - loss: 0.172
9 - accuracy: 0.9518 - val_loss: 0.1655 - val_accuracy: 0.9556
Epoch 11/20
50000/50000 [==============================] - 17s 343us/sample - loss: 0.163
3 - accuracy: 0.9542 - val_loss: 0.1568 - val_accuracy: 0.9579
Epoch 12/20
50000/50000 [==============================] - 20s 404us/sample - loss: 0.154
7 - accuracy: 0.9566 - val_loss: 0.1501 - val_accuracy: 0.9593
Epoch 13/20
50000/50000 [==============================] - 17s 350us/sample - loss: 0.146
5 - accuracy: 0.9588 - val_loss: 0.1456 - val_accuracy: 0.9604
Epoch 14/20
50000/50000 [==============================] - 17s 335us/sample - loss: 0.139
6 - accuracy: 0.9612 - val_loss: 0.1391 - val_accuracy: 0.9630
Epoch 15/20
50000/50000 [==============================] - 18s 361us/sample - loss: 0.133
0 - accuracy: 0.9632 - val_loss: 0.1354 - val_accuracy: 0.9645
Epoch 16/20
50000/50000 [==============================] - 19s 374us/sample - loss: 0.127
1 - accuracy: 0.9646 - val_loss: 0.1325 - val_accuracy: 0.9650
Epoch 17/20
50000/50000 [==============================] - 16s 328us/sample - loss: 0.121
5 - accuracy: 0.9661 - val_loss: 0.1274 - val_accuracy: 0.9660
Epoch 18/20
50000/50000 [==============================] - 11s 222us/sample - loss: 0.116
4 - accuracy: 0.9680 - val_loss: 0.1237 - val_accuracy: 0.9662
Epoch 19/20
50000/50000 [==============================] - 11s 215us/sample - loss: 0.111
7 - accuracy: 0.9695 - val_loss: 0.1195 - val_accuracy: 0.9672
Epoch 20/20
50000/50000 [==============================] - 11s 218us/sample - loss: 0.107
5 - accuracy: 0.9704 - val_loss: 0.1163 - val_accuracy: 0.9670
Plot the learning curves
In [13]:
loss = history.history['loss']
val_loss = history.history['val_loss']
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
epochs = np.arange(1, len(loss)+1)
plt.figure(figsize=(16, 5))
plt.subplot(1,2,1)
plt.plot(epochs, loss, label='training loss')
plt.plot(epochs, val_loss, label='validation loss')
plt.xlabel('epoch')
plt.ylabel('loss')
plt.legend()
plt.grid(True)
plt.subplot(1,2,2)
plt.plot(epochs, acc, label='training accuracy')
plt.plot(epochs, val_acc, label='validation accuracy')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.legend()
plt.grid(True)
plt.plot()
plt.show()
Plotting training history using Pandas
In [14]:
import pandas as pd
df = pd.DataFrame(history.history)
df.plot(figsize=(8, 5))
plt.grid(True)
plt.show()
Making a prediction using the trained model
In [16]:
i = 15
y_pred = model.predict(X_test[i].reshape(1, 28, 28))
print(y_pred)
print(y_pred.sum())
[[3.62362363e-04 7.51586573e-04 3.04447574e-04 2.38896813e-02
1.27459425e-04 9.70107079e-01 1.51503045e-04 1.00383026e-04
4.13918169e-03 6.63491373e-05]]
1.0
In [17]:
y_pred = model.predict_classes(X_test[i].reshape(1, 28, 28))
print("Predictied class : ", y_pred[0])
print("Actual class : ", y_test[i])
Predictied class : 5
Actual class : 5
Evaluate test set performance
In [18]:
test_loss, test_acc = model.evaluate(X_test, y_test, batch_size=64, verbose=0)
print("Test set loss : ", test_loss)
print("Test set acc. : ", test_acc)
Test set loss : 0.11765392680168152
Test set acc. : 0.9668
In [ ]:
