Jupyter Notebooks

In this tutorial, we will use Jupyhai, to run a Valohai execution from your local Jupyter notebook.

A short recap on Notebooks

  • Jupyhai is a Jupyter notebook extension developed and maintained by Valohai.

  • Use the Run remote button, instead of the Run cell button to run your notebook on Valohai.

  • The Jupyhai addon will generate a valohai.yaml - Config File file for each execution based on the contents of the Notebook. You don’t need to create the YAML file yourself.

  • Notebook executions use a custom Docker image called valohai/pypermill. Make sure to use it as a base for your custom Docker images.

Installing Jupyhai to on your machine

pip install --upgrade pip
pip install notebook
pip install jupyhai
jupyhai install

Start notebook

Create a new folder valohai-notebook on your desktop and launch Jupyter in that directory.

mkdir valohai-notebook
cd valohai-notebook
jupyter notebook
  • Create a new Python 3 notebook

  • Open the new Notebook

  • Click on Settings on the toolbar

  • Login with your Valohai credentials

  • Update the Docker image to drazend/jupyhai:tf.2.4.1

  • Close the settings tab

  • Add print('Hello Valohai Notebooks!') to the first cell

  • Click on Run remote to run the notebook remotely

  • View the logs from the execution by clicking on the gizmo on the right side of the page (e.g. #1)

Jupyhai Settings

Click on Settings on the toolbar to:

  • Connect your Notebook to a Valohai project

  • Choose the environment you want to run on

  • Define the Docker image you’d like to use

Open a Notebook from a previous execution

Each of the colored gizmos on the right side of the page signify a single Valohai execution. You can click on any of the completed executions and select Notebook to inspect the Notebook version that was used to run the execution.

MNIST sample

Prerequirements

Download the sample MNIST dataset and place it in the same folder as the notebook.

Replace your print('Hello Valohai Notebooks!') with the sample code from below.

This is a simple MNIST sample that loads up the data from a local file, trains a model and saves the trained model on the local machine.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import tensorflow as tf
import numpy

myinput = 'mnist.npz'

with numpy.load(myinput, allow_pickle=True) as f:
   x_train, y_train = f['x_train'], f['y_train']
   x_test, y_test = f['x_test'], f['y_test']

x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])

predictions = model(x_train[:1]).numpy()
predictions

tf.nn.softmax(predictions).numpy()

loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

loss_fn(y_train[:1], predictions).numpy()

model.compile(optimizer='adam',
            loss=loss_fn,
            metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)

model.save('model.h5')

Add parameters

Parameterizing a notebook happens through cell tags. Tags are a standard Jupyter feature.

  • Create a new cell and define a new variable epoch_val in it.

  • Show cell tags by going to View->Cell Toolbar->Tags.

  • Add a new tag parameters to the first cell.

epoch_val = 6

Then update model.fit to set the epochs value from the variable.

28
29
30
31
32
33
34
model.compile(optimizer='adam',
            loss=loss_fn,
            metrics=['accuracy'])

model.fit(x_train, y_train, epochs=epoch_val)

model.save('model.h5')

Add inputs

You can easily download data to your notebook either from a public location (HTTP/HTTPS) or a private cloud storage.

  • Create a new cell at the top of your notebook

  • Add an inputs tag to the new cell

  • In the new cell define a variable mydata and paste the address of your dataset

mydata = 's3://onboard-sample/tf-sample/mnist.npz'

Tip

Valohai will download the input data to /valohai/inputs/<name>/<file>.

Update the myinput in your sample code to point to /valohai/inputs/mydata/mnist.npz

1
2
3
4
5
6
7
8
import tensorflow as tf
import numpy

myinput = '/valohai/inputs/mydata/mnist.npz'

with numpy.load(myinput, allow_pickle=True) as f:
   x_train, y_train = f['x_train'], f['y_train']
   x_test, y_test = f['x_test'], f['y_test']

Save a trained model

Finally you’d need to save the trained model as a Valohai output.

34
model.save('/valohai/outputs/model.h5')

View in Valohai

  • Open your project on app.valohai.com

  • Open the latest execution

  • The details type contains information about the executions

  • The logs tab contains all logs from the execution

  • You can click on the blue Notebooks button, to view an executed Notebook

See also

  • Using Tasks for hyperparameter tuning

🐞 Give feedback about this page