Quick Start - Jupyter Notebooks (MNIST)

In this tutorial, we will use the Valohai Jupyter Notebook Extension to build a machine learning model using the MNIST dataset.

You can also

Installation

Install Valohai’s Jupyhai package to run notebooks from your local machine on Valohai.

pip install jupyhai
jupyhai install

Start the notebook server

Navigate to the folder where your notebooks are located and start notebooks.

jupyter notebook

Run your own code and notebooks

  • This mounts your current working directory into the container.
  • All the files in your current working directory will be available within the container
  • All changes within the mounted folder will persist after shutting down
  • You can run install additional libraries to the environment by adding a new cell at the top of your notebook and running !pip install mylibrary

Open a notebook

Creating new notebook

Create new notebook folder or choose an existing one.

Sign in

Login to Valohai

Press the Valohai button in the toolbar and login using your Valohai credentials.

You’ll be automatically logged in, if you’ve already logged in valohai-cli.

Select the following:

  • Project: Valohai project where the executions will be version controlled
  • Environment: Environment type for the cloud executions (E.g. AWS p2.xlarge)
  • Docker Image: Docker image that provides the required libraries (E.g. TensorFlow)
    • Use valohai/pypermill as the base when building your custom Docker image to be used as a part of Valohai Notebook executions.

These are the same settings you would choose when using Valohai website, CLI or valohai.yaml.

Once you are happy with your selections. Press save.

Create an execution & watch the results

Running Valohai execution

Press the Run remote button on the toolbar

The gizmo for the new execution will appear to the right.

You can click the #1 > Notebook button to download the finished notebook back to your local machine.

Open a Notebook from a previous execution

Each of the colored gizmos on the right side of the page signify a single Valohai execution. You can click on any of the completed executions and select Notebook to launch the Notebook version that was used to run the execution.

Parameterize the notebook

Adding hyperparameter

Parameterizing a notebook happens through cell tags. Tags are a standard Jupyter feature.

Here we mark the first cell with a parameters tag, which means all variables are considered to be Valohai parameters, just like you would define in the valohai.yaml.

Download the inputs

Adding parameterized input

Here we marked the first cell with inputs tag and ran it in Valohai.

All the variables in this cell will be considered as Valohai input URIs for the execution, just like in the valohai.yaml.

Reusing the parameterized notebook

Adding hyperparameter

Now you can run notebook based experiments without a notebook!

Because the learning_rate is parameterized, you can change it via Valohai web interface and run additional experiments without even opening the notebook.

FAQ

When I try to download my outputs back from a finished execution, I get 404: Not Found

Always use the notebook server through http://127.0.0.1:8888 instead of http://localhost:8888.