Table of Contents:

Building Docker Images

In this guide, we’ll build a Docker image to be used as a runtime environment on Valohai.

Building images is frequently optional, but is recommended especially when…

  1. … your runtime setup takes longer than a couple of minutes (apt-get install, etc.)
  2. … you want to use Valohai backend for iterative machine learning development where you start an execution with vh exec run and it’ll start running under a second

If you don’t wish to maintain your own Docker images, you can find images for the most popular machine learning frameworks on Docker Hub.

1. Requirements

For this tutorial you will need:

  • Docker CE (Community Edition)

How to install Docker CE:

2. Choose a base image

First, we need to choose a base image to build the Docker image upon.

It’s wise to choose a Docker image that contains the core libraries you are using; for example:

Also make effort to check out what kind of image variants do the repository host under the Tags tab. For example, nvidia/cuda tags include various CUDNN versions and Ubuntu versions. Docker Hub image repository Overview tab usually contains information what different tags mean.

If you choose a machine learning framework Docker base image such as tensorflow/tensorflow, make sure that variant includes GPU support if you plan on using GPUs, like tensorflow/tensorflow:1.12.0-gpu-py3 where the gpu part tells that it has been built on top of nvidia/cuda, enabling GPU access.

By the end of this step, you should end up with a base Docker image that closest resembles your project stack e.g. nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04 or tensorflow/tensorflow:1.12.0-gpu-py3.

3. Write build instructions - Dockerfile

As an example, we’ll be creating a Docker image that utilizes GPUs with TensorFlow. In the real world, you would want to use tensorflow/tensorflow in a simple situation like this.

Docker images are build with Dockerfiles that specify the steps how to build the image. More information about Dockerfile syntax at Dockerfile reference.

Write the following into a file called Dockerfile (without extension).

# Our base image
FROM nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04

# Some common environmenta variables that Python uses
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

# Install lower level dependencies
RUN apt-get update --fix-missing && \
    apt-get install -y curl python3 python3-pip && \
    update-alternatives --install /usr/bin/python python /usr/bin/python3 10 && \
    update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 10 && \
    apt-get clean && \
    apt-get autoremove && \
    rm -rf /var/lib/apt/lists/*

# Install a specific version of TensorFlow
# You may also install anything else from pip like this
RUN pip install --no-cache-dir tensorflow-gpu==1.12.0

The run the following commands to name and build the image.

docker build -t my-name/my-image:1.12.0 .

# now you can run commands in your brand new Docker image to try it out
docker run --rm -i -t my-name/my-image:1.12.0 python --version  # => Python 3.5.2

Note

If you are using nvidia/cuda base image, you might be required to use Linux with kernel version >3.10 to work with the images. You need to have nvidia-docker installed.

Now you have your own Docker image! Next we’ll host it somewhere for later use.

4. Host the image

We recommend hosting your images on Docker Hub if there is nothing secret about your dependencies. Just create an account and login using the Docker Desktop app or the command-line client.

On Docker Hub, create repository using the Create Repository button on the dashboard. Give it a descriptive name like my-image like we have been using in this example ;)

# in-case you need to rename your image at this point...
docker tag my-name/my-image:1.12.0 actual-account/my-image:1.12.0
docker rmi my-name/my-image:1.12.0

# and finally push the Docker image to the repository
docker push actual-account/my-image:1.12.0

And now you finally have a publicly available Docker image actual-account/my-image:1.12.0 you can use on Valohai or anywhere else!