Example: Docker Image Builder

This example shows you how to create a custom library steps that take in a Dockerfile, use it to build a docker image and finally push the image to a private registry. The files used in the example are available in the public valohai-toolkit Github repository. There are three different steps for different image registries (AWS ECR, GCP Artifact Registry and Docker Hub).

Why use this?

No local Docker setup: Write a Dockerfile, run an execution, get a pushed image. No Docker daemon, no local builds.

For teams without Docker experience: Data scientists focus on writing Dockerfiles. Valohai handles authentication and pushing.

For CI/CD pipelines: Build images as part of your workflow without managing Docker infrastructure.

Available registry options

The valohai-toolkit library includes builders for:

• AWS ECR: docker-image-aws

• GCP Artifact Registry: docker-image-gcp

• Docker Hub: docker-image-dockerhub

Source: github.com/valohai/valohai-toolkit

Requirements

• A Git repository containing a valohai.yaml with at least one step and all the other required code files.

• A Dockerfile (provided as an input file or a parameter)

• Docker registry credentials (environment variables). This example covers three use cases for private registries:

  • Docker Hub login with credentials

  • AWS ECR login with machine identity

  • GCP Artifact Registry login with service account key

• Valohai environment configured with --privileged flag

The --privileged flag is required to run Docker-in-Docker and it will give the containers root capabilities. Contact Valohai support ([email protected]) if you need this enabled.

Connect the library

1. Click Hi, <username> (top-right) → Manage <organization>

2. Open the Libraries tab

3. Click Add a new Library

4. Configure:

   • Name: valohai-toolkit

   • URL: git+https://github.com/valohai/valohai-toolkit.git

5. Click Add Library

6. Click Update to fetch commits

The steps defined in the valohai.yaml file are now available under the Create Execution page of all your projects!

Build and push an image to a Docker registry

After you’ve connected the repository as a Library, you can follow the steps below to build and push an image to your Docker registry.

• Click on the Create Execution button on the Executions tab of your project and choose the step you want to run from the custom library.

• Configure the Dockerfile (input or parameter) and adjust other parameters.

• Include the needed environmental variables for authentication as explained below.

Define your Dockerfile

You can provide the Dockerfile in two ways:

Option 1: As an input file

Upload your Dockerfile to your data store and reference it as an input when creating the execution.

Option 2: As a parameter

Write the Dockerfile directly in the Valohai UI using the dockerfile parameter.

Default Dockerfile (if not specified):

FROM python:3.9
RUN pip install valohai-utils tensorflow==2.6.0

This creates a basic image with Python 3.9, valohai-utils, and TensorFlow.

Set parameters

Configure these parameters when creating the execution:

• repository: Repository name, e.g., username/myimage or mycompany.azurecr.io/ml-training

• docker-tag: Image tag, e.g., v1.0 or latest

Example:

• repository=mycompany/ml-image

• docker-tag=v2.0

This builds and pushes mycompany/ml-image:v2.0.

Environment variables and authentication

Depending on which registry you are using, you will need to provide certain environment variables. You can define these either on the execution, project or organization level (admin users only) depending on the use case.

You will also need to make sure the Valohai worker running the job can authenticate to the Docker registry for pushing the image.

AWS ECR

Define the following environment variables:

• AWS_ECR_ACCOUNT_ID: Your AWS account ID

• AWS_ECR_REGION: AWS region (e.g., us-east-1)

For authentication, it is possible to use the machine identity to provide access to AWS ECR. This way you do not need to save the credentials in environment variables.

We recommend, that you create a separate role for in your AWS account for the environment type that is allowed to push imaged to your ECR, e.g. ValohaiWorkerRoleECR.

Required IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListImagesInRepository",
            "Effect": "Allow",
            "Action": [
                "ecr:ListImages"
            ],
            "Resource": "arn:aws:ecr:<REGION>:<ACCOUNT-ID>:repository/<REPOSITORY>*"
        },
        {
            "Sid": "ManageRepositoryContents",
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:PutImage",
                "ecr:UntagResource",
                "ecr:CreateRepository",
                "ecr:TagResource"
            ],
            "Resource": "arn:aws:ecr:<REGION>:<ACCOUNT-ID>:repository/<REPOSITORY>*"
        },
        {
            "Effect": "Allow",
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*"
        }
    ]
}

Add this policy to ValohaiWorkerRoleECR . You should also attach the existing ValohaiWorkerPolicy to the new role.

If you created the new ValohaiWorkerRoleECR , make sure to add it to the ValohaiMaster role in your AWS Account.

     {
       "Sid" : "4",
       "Effect" : "Allow",
       "Action" : [
         "iam:PassRole",
         "iam:GetRole"
       ],
       "Resource" : [
          "arn:aws:iam::<ACCOUNT-ID>:role/ValohaiWorkerRole",
          "arn:aws:iam::<ACCOUNT-ID>:role/ValohaiWorkerRoleECR"
       ]
     }

GCP Artifact Registry

Define the following environment variables:

• GCP_PROJECT_ID: Your GCP project ID

• GCP_REGION: GCP region (e.g., us-central1)

• GCP_NEW_REPOSITORY: (Optional) Set to create a new repository

In GCP you can use service accounts to push Docker images to the Artifact Registry. You can read more about the permissions from GCP documentation.

• Create a new service account under your Google Cloud project that contains the registry.

• Add Artifact Registry Repository Administrator role so it can push Docker images to your private repository and also create new repositories.

Finally, inform Valohai Customer Team ([email protected]) about the new role name so that they can set up the environment for you.

Docker Hub

Define the following environment variables:

• DHUSERNAME: Docker Hub username

• DHPASSWORD: Docker Hub password or access token (mark as a secret)

The environment variables are are used for authentication and no additional steps are needed.

Run the build

1. Open your project

2. Click Create Execution

3. Expand valohai-toolkit → Select your registry step:

   • docker-image-aws for AWS ECR

   • docker-image-gcp for GCP Artifact Registry

   • docker-image-dockerhub for Docker Hub

4. Configure:

   • Dockerfile: Provide as an input or as a parameter

   • repository: Image repository name

   • docker-tag: Image tag

5. Add environment variables (if not already set project-wide)

6. Click Create Execution

The execution builds the image and pushes it to your registry.

Use the built image

After the build succeeds, reference the image in your valohai.yaml:

- step:
    name: train
    image: mycompany.azurecr.io/ml-training:v2.0
    command:
      - python train.py

If using a private registry, ensure you've configured registry authentication.