Skip to content

Commit 4493d1f

Browse files
lavanyashuklalavanyashukla
authored
sweeps notebook for oreilly class
1 parent c89043b commit 4493d1f

File tree

1 file changed

+382
-0
lines changed

1 file changed

+382
-0
lines changed

‎examples/keras-fashion/sweeps.ipynb

Lines changed: 382 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,382 @@
1+
{
2+
"nbformat": 4,
3+
"nbformat_minor": 0,
4+
"metadata": {
5+
"kernelspec": {
6+
"name": "python3",
7+
"display_name": "Python 3"
8+
},
9+
"language_info": {
10+
"pygments_lexer": "ipython3",
11+
"nbconvert_exporter": "python",
12+
"version": "3.6.4",
13+
"file_extension": ".py",
14+
"codemirror_mode": {
15+
"name": "ipython",
16+
"version": 3
17+
},
18+
"name": "python",
19+
"mimetype": "text/x-python"
20+
},
21+
"colab": {
22+
"name": "Hyperparameter Sweeps with W&B.ipynb",
23+
"provenance": [],
24+
"private_outputs": true,
25+
"collapsed_sections": []
26+
},
27+
"accelerator": "GPU"
28+
},
29+
"cells": [
30+
{
31+
"cell_type": "markdown",
32+
"metadata": {
33+
"id": "QJc0KMZlzdtO",
34+
"colab_type": "text"
35+
},
36+
"source": [
37+
"# Introduction to Hyperparameter Sweeps\n",
38+
"\n",
39+
"Searching through high dimensional hyperparameter spaces to find the most performant model can get unwieldy very fast. Hyperparameter sweeps provide an organized and efficient way to conduct a battle royale of models and pick the most accurate model. They enable this by automatically searching through combinations of hyperparameter values (e.g. learning rate, batch size, number of hidden layers, optimizer type) to find the most optimal values.\n",
40+
"\n",
41+
"In this tutorial we'll see how you can run sophisticated hyperparameter sweeps in 3 easy steps using Weights and Biases.\n",
42+
"\n",
43+
"## Sweeps: An Overview\n",
44+
"\n",
45+
"Running a hyperparameter sweep with Weights & Biases is very easy. There are just 3 simple steps:\n",
46+
"\n",
47+
"1. **Define the sweep:** we do this by creating a dictionary or a [YAML file](https://docs.wandb.com/library/sweeps/configuration) that specifies the parameters to search through, the search strategy, the optimization metric et all.\n",
48+
"\n",
49+
"2. **Initialize the sweep:** with one line of code we initialize the sweep and pass in the dictionary of sweep configurations:\n",
50+
"`sweep_id = wandb.sweep(sweep_config)`\n",
51+
"\n",
52+
"3. **Run the sweep agent:** also accomplished with one line of code, we call wandb.agent() and pass the sweep_id to run, along with a function that defines your model architecture and trains it:\n",
53+
"`wandb.agent(sweep_id, function=train)`\n",
54+
"\n",
55+
"And voila! That's all there is to running a hyperparameter sweep! In the notebook below, we'll walk through these 3 steps in more detail.\n",
56+
"\n",
57+
"\n",
58+
"We highly encourage you to fork this notebook, tweak the parameters, or try the model with your own dataset!\n",
59+
"\n",
60+
"## Resources\n",
61+
"- [Sweeps docs →](https://docs.wandb.com/library/sweeps)\n",
62+
"- [Launching from the command line →](https://www.wandb.com/articles/hyperparameter-tuning-as-easy-as-1-2-3)\n"
63+
]
64+
},
65+
{
66+
"cell_type": "markdown",
67+
"metadata": {
68+
"id": "ITmB8ev90U_t",
69+
"colab_type": "text"
70+
},
71+
"source": [
72+
"# Setup\n",
73+
"Start out by installing the experiment tracking library and setting up your free W&B account:\n",
74+
"\n",
75+
"\n",
76+
"* **pip install wandb** – Install the W&B library\n",
77+
"* **import wandb** – Import the wandb library\n"
78+
]
79+
},
80+
{
81+
"cell_type": "code",
82+
"metadata": {
83+
"trusted": true,
84+
"id": "0m3yWmVgaJ5Y",
85+
"colab_type": "code",
86+
"colab": {}
87+
},
88+
"source": [
89+
"# WandB – Install the W&B library\n",
90+
"%pip install wandb -q\n",
91+
"import wandb\n",
92+
"from wandb.keras import WandbCallback"
93+
],
94+
"execution_count": 0,
95+
"outputs": []
96+
},
97+
{
98+
"cell_type": "code",
99+
"metadata": {
100+
"trusted": true,
101+
"colab_type": "code",
102+
"id": "3rv3mo4IuJn-",
103+
"colab": {}
104+
},
105+
"source": [
106+
"!pip install wandb -qq\n",
107+
"from keras.datasets import fashion_mnist\n",
108+
"from keras.models import Sequential\n",
109+
"from keras.layers import Conv2D, MaxPooling2D, Dropout, Dense, Flatten\n",
110+
"from keras.utils import np_utils\n",
111+
"from keras.optimizers import SGD\n",
112+
"from keras.optimizers import RMSprop, SGD, Adam, Nadam\n",
113+
"from keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, Callback, EarlyStopping\n",
114+
"\n",
115+
"import os\n",
116+
"os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
117+
"import tensorflow as tf\n",
118+
"tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)\n",
119+
"\n",
120+
"import wandb\n",
121+
"from wandb.keras import WandbCallback\n",
122+
"\n",
123+
"(X_train, y_train), (X_test, y_test) = fashion_mnist.load_data()\n",
124+
"labels=[\"T-shirt/top\",\"Trouser\",\"Pullover\",\"Dress\",\"Coat\",\n",
125+
" \"Sandal\",\"Shirt\",\"Sneaker\",\"Bag\",\"Ankle boot\"]\n",
126+
"\n",
127+
"img_width=28\n",
128+
"img_height=28\n",
129+
"\n",
130+
"X_train = X_train.astype('float32') / 255.\n",
131+
"X_test = X_test.astype('float32') / 255.\n",
132+
"\n",
133+
"# reshape input data\n",
134+
"X_train = X_train.reshape(X_train.shape[0], img_width, img_height, 1)\n",
135+
"X_test = X_test.reshape(X_test.shape[0], img_width, img_height, 1)\n",
136+
"\n",
137+
"# one hot encode outputs\n",
138+
"y_train = np_utils.to_categorical(y_train)\n",
139+
"y_test = np_utils.to_categorical(y_test)\n",
140+
"num_classes = y_test.shape[1]"
141+
],
142+
"execution_count": 0,
143+
"outputs": []
144+
},
145+
{
146+
"cell_type": "markdown",
147+
"metadata": {
148+
"id": "AXcXCVlDU4ME",
149+
"colab_type": "text"
150+
},
151+
"source": [
152+
"## 1. Define the Sweep\n",
153+
"\n",
154+
"Weights & Biases sweeps give you powerful levers to configure your sweeps exactly how you want them, with just a few lines of code. The sweeps config can be defined as a dictionary or a [YAML file](https://docs.wandb.com/library/sweeps).\n",
155+
"\n",
156+
"Let's walk through some of them together:\n",
157+
"* **Metric** – This is the metric the sweeps are attempting to optimize. Metrics can take a `name` (this metric should be logged by your training script) and a `goal` (maximize or minimize). \n",
158+
"* **Search Strategy** – Specified using the 'method' variable. We support several different search strategies with sweeps. \n",
159+
" * **Grid Search** – Iterates over every combination of hyperparameter values.\n",
160+
" * **Random Search** – Iterates over randomly chosen combinations of hyperparameter values.\n",
161+
" * **Bayesian Search** – Creates a probabilistic model that maps hyperparameters to probability of a metric score, and chooses parameters with high probability of improving the metric. The objective of Bayesian optimization is to spend more time in picking the hyperparameter values, but in doing so trying out fewer hyperparameter values.\n",
162+
"* **Stopping Criteria** – The strategy for determining when to kill off poorly peforming runs, and try more combinations faster. We offer several custom scheduling algorithms like [HyperBand](https://arxiv.org/pdf/1603.06560.pdf) and Envelope.\n",
163+
"* **Parameters** – A dictionary containing the hyperparameter names, and discreet values, max and min values or distributions from which to pull their values to sweep over.\n",
164+
"\n",
165+
"You can find a list of all configuration options [here](https://docs.wandb.com/library/sweeps/configuration)."
166+
]
167+
},
168+
{
169+
"cell_type": "code",
170+
"metadata": {
171+
"id": "EdaHO-3M8ly3",
172+
"colab_type": "code",
173+
"colab": {}
174+
},
175+
"source": [
176+
"# Configure the sweep – specify the parameters to search through, the search strategy, the optimization metric et all.\n",
177+
"sweep_config = {\n",
178+
" 'method': 'random', #grid, random\n",
179+
" 'metric': {\n",
180+
" 'name': 'accuracy',\n",
181+
" 'goal': 'maximize' \n",
182+
" },\n",
183+
" 'parameters': {\n",
184+
" 'epochs': {\n",
185+
" 'values': [2, 5, 10]\n",
186+
" },\n",
187+
" 'batch_size': {\n",
188+
" 'values': [256, 128, 64, 32]\n",
189+
" },\n",
190+
" 'dropout': {\n",
191+
" 'values': [0.3, 0.4, 0.5]\n",
192+
" },\n",
193+
" 'conv_layer_size': {\n",
194+
" 'values': [16, 32, 64]\n",
195+
" },\n",
196+
" 'weight_decay': {\n",
197+
" 'values': [0.0005, 0.005, 0.05]\n",
198+
" },\n",
199+
" 'learning_rate': {\n",
200+
" 'values': [1e-2, 1e-3, 1e-4, 3e-4, 3e-5, 1e-5]\n",
201+
" },\n",
202+
" 'optimizer': {\n",
203+
" 'values': ['adam', 'nadam', 'sgd', 'rmsprop']\n",
204+
" },\n",
205+
" 'activation': {\n",
206+
" 'values': ['relu', 'elu', 'selu', 'softmax']\n",
207+
" }\n",
208+
" }\n",
209+
"}"
210+
],
211+
"execution_count": 0,
212+
"outputs": []
213+
},
214+
{
215+
"cell_type": "markdown",
216+
"metadata": {
217+
"id": "Ll_zFhpaU6cu",
218+
"colab_type": "text"
219+
},
220+
"source": [
221+
"## 2. Initialize the Sweep"
222+
]
223+
},
224+
{
225+
"cell_type": "code",
226+
"metadata": {
227+
"id": "kdc7RBBaU0F3",
228+
"colab_type": "code",
229+
"colab": {}
230+
},
231+
"source": [
232+
"# Initialize a new sweep\n",
233+
"# Arguments:\n",
234+
"# – sweep_config: the sweep config dictionary defined above\n",
235+
"# – entity: Set the username for the sweep\n",
236+
"# – project: Set the project name for the sweep\n",
237+
"sweep_id = wandb.sweep(sweep_config, entity=\"sweep\", project=\"sweeps-tutorial\")"
238+
],
239+
"execution_count": 0,
240+
"outputs": []
241+
},
242+
{
243+
"cell_type": "markdown",
244+
"metadata": {
245+
"id": "FT4dKp1xVeeT",
246+
"colab_type": "text"
247+
},
248+
"source": [
249+
"### Define Your Neural Network\n",
250+
"Before we can run the sweep, let's define a function that creates and trains our neural network.\n",
251+
"\n",
252+
"In the function below, we define a simplified version of a VGG19 model in Keras, and add the following lines of code to log models metrics, visualize performance and output and track our experiments easily:\n",
253+
"* **wandb.init()** – Initialize a new W&B run. Each run is single execution of the training script.\n",
254+
"* **wandb.config** – Save all your hyperparameters in a config object. This lets you use our app to sort and compare your runs by hyperparameter values.\n",
255+
"* **callbacks=[WandbCallback()]** – Fetch all layer dimensions, model parameters and log them automatically to your W&B dashboard.\n",
256+
"* **wandb.log()** – Logs custom objects – these can be images, videos, audio files, HTML, plots, point clouds etc. Here we use wandb.log to log images of Simpson characters overlaid with actual and predicted labels."
257+
]
258+
},
259+
{
260+
"cell_type": "code",
261+
"metadata": {
262+
"trusted": true,
263+
"id": "aIhxl7glaJ5k",
264+
"colab_type": "code",
265+
"colab": {}
266+
},
267+
"source": [
268+
"# The sweep calls this function with each set of hyperparameters\n",
269+
"def train():\n",
270+
" # Default values for hyper-parameters we're going to sweep over\n",
271+
" config_defaults = {\n",
272+
" 'epochs': 5,\n",
273+
" 'batch_size': 128,\n",
274+
" 'weight_decay': 0.0005,\n",
275+
" 'learning_rate': 1e-3,\n",
276+
" 'activation': 'relu',\n",
277+
" 'optimizer': 'nadam',\n",
278+
" 'hidden_layer_size': 128,\n",
279+
" 'conv_layer_size': 16,\n",
280+
" 'dropout': 0.5,\n",
281+
" 'momentum': 0.9,\n",
282+
" 'seed': 42\n",
283+
" }\n",
284+
"\n",
285+
" # Initialize a new wandb run\n",
286+
" wandb.init(config=config_defaults)\n",
287+
" \n",
288+
" # Config is a variable that holds and saves hyperparameters and inputs\n",
289+
" config = wandb.config\n",
290+
" \n",
291+
" # Define the model architecture - This is a simplified version of the VGG19 architecture\n",
292+
" model = Sequential()\n",
293+
" \n",
294+
" # Set of Conv2D, Conv2D, MaxPooling2D layers with 32 and 64 filters\n",
295+
" model.add(Conv2D(filters = config.conv_layer_size, kernel_size = (3, 3), padding = 'same', \n",
296+
" activation ='relu', input_shape=(img_width, img_height,1)))\n",
297+
" model.add(Dropout(config.dropout))\n",
298+
"\n",
299+
" model.add(Conv2D(filters = config.conv_layer_size, kernel_size = (3, 3),\n",
300+
" padding = 'same', activation ='relu'))\n",
301+
" model.add(MaxPooling2D(pool_size=(2, 2)))\n",
302+
"\n",
303+
" model.add(Flatten())\n",
304+
" model.add(Dense(config.hidden_layer_size, activation ='relu'))\n",
305+
"\n",
306+
" model.add(Dense(num_classes, activation = \"softmax\"))\n",
307+
"\n",
308+
" # Define the optimizer\n",
309+
" if config.optimizer=='sgd':\n",
310+
" optimizer = SGD(lr=config.learning_rate, decay=1e-5, momentum=config.momentum, nesterov=True)\n",
311+
" elif config.optimizer=='rmsprop':\n",
312+
" optimizer = RMSprop(lr=config.learning_rate, decay=1e-5)\n",
313+
" elif config.optimizer=='adam':\n",
314+
" optimizer = Adam(lr=config.learning_rate, beta_1=0.9, beta_2=0.999, clipnorm=1.0)\n",
315+
" elif config.optimizer=='nadam':\n",
316+
" optimizer = Nadam(lr=config.learning_rate, beta_1=0.9, beta_2=0.999, clipnorm=1.0)\n",
317+
"\n",
318+
" model.compile(loss = \"categorical_crossentropy\", optimizer = optimizer, metrics=['accuracy'])\n",
319+
"\n",
320+
" model.fit(X_train, y_train, batch_size=config.batch_size,\n",
321+
" epochs=config.epochs,\n",
322+
" validation_data=(X_test, y_test),\n",
323+
" callbacks=[WandbCallback(data_type=\"image\", validation_data=(X_test, y_test), labels=labels),\n",
324+
" EarlyStopping(patience=10, restore_best_weights=True)])"
325+
],
326+
"execution_count": 0,
327+
"outputs": []
328+
},
329+
{
330+
"cell_type": "markdown",
331+
"metadata": {
332+
"id": "qVxXIXXTyOLC",
333+
"colab_type": "text"
334+
},
335+
"source": [
336+
"## 3. Run the sweep agent"
337+
]
338+
},
339+
{
340+
"cell_type": "code",
341+
"metadata": {
342+
"id": "1gD9qhA9yOYs",
343+
"colab_type": "code",
344+
"colab": {}
345+
},
346+
"source": [
347+
"# Initialize a new sweep\n",
348+
"# Arguments:\n",
349+
"# – sweep_id: the sweep_id to run - this was returned above by wandb.sweep()\n",
350+
"# – function: function that defines your model architecture and trains it\n",
351+
"wandb.agent(sweep_id, train)"
352+
],
353+
"execution_count": 0,
354+
"outputs": []
355+
},
356+
{
357+
"cell_type": "markdown",
358+
"metadata": {
359+
"id": "lWE4BXssxsDJ",
360+
"colab_type": "text"
361+
},
362+
"source": [
363+
"# Visualize Sweeps Results\n",
364+
"\n",
365+
"## Parallel coordinates plot\n",
366+
"This plot maps hyperparameter values to model metrics. It’s useful for honing in on combinations of hyperparameters that led to the best model performance.\n",
367+
"\n",
368+
"![](https://assets.website-files.com/5ac6b7f2924c652fd013a891/5e190366778ad831455f9af2_s_194708415DEC35F74A7691FF6810D3B14703D1EFE1672ED29000BA98171242A5_1578695138341_image.png)\n",
369+
"\n",
370+
"## Hyperparameter Importance Plot\n",
371+
"The hyperparameter importance plot surfaces which hyperparameters were the best predictors of, and highly correlated to desirable values for your metrics.\n",
372+
"\n",
373+
"![](https://assets.website-files.com/5ac6b7f2924c652fd013a891/5e190367778ad820b35f9af5_s_194708415DEC35F74A7691FF6810D3B14703D1EFE1672ED29000BA98171242A5_1578695757573_image.png)\n",
374+
"\n",
375+
"These visualizations can help you save both time and resources running expensive hyperparameter optimizations by honing in on the parameters (and value ranges) that are the most important, and thereby worthy of further exploration.\n",
376+
"\n",
377+
"# Next step - Get your hands dirty with sweeps\n",
378+
"We created a simple training script and [a few flavors of sweep configs](https://github.com/wandb/examples/tree/master/keras-cnn-fashion) for you to play with. We highly encourage you to give these a try. This repo also has examples to help you try more advanced sweep features like [Bayesian Hyperband](https://app.wandb.ai/wandb/examples-keras-cnn-fashion/sweeps/us0ifmrf?workspace=user-lavanyashukla), and [Hyperopt](https://app.wandb.ai/wandb/examples-keras-cnn-fashion/sweeps/xbs2wm5e?workspace=user-lavanyashukla)."
379+
]
380+
}
381+
]
382+
}

0 commit comments

Comments
 (0)