Add a valohai.yaml configuration file

Valohai is completely technology agnostic. You can develop in notebooks or scripts in a language and framework of your choice.

valohai.yaml

Valohai expects that you have a valohai.yaml config file in the root of your repository. The file defines what kind of jobs can be executed inside your project, and the different properties of each job type (these are called steps in Valohai).

You can either write the valohai.yaml by hand or if you’re using Python you can use the valohai-utils Toolkit to define the configuration file in your code.

Let’s start by bringing just a single job type to Valohai.

Open your existing Python file and define a Valohai step inside it:

import valohai

# Prepares a new step called train in valohai.yaml
# Execute this Python file to run every time the train step is launched in Valohai
valohai.prepare(step="train")

You can now generate a valohai.yaml configuration file that includes the train step by running:

vh yaml step myfilename.py

This will create:

  • A valohai.yaml file with the train step that will run:

    • pip install -r requirements.txt

    • your Python script.

  • A requirements.txt with valohai-utils

    • If you have an existing requirements.txt file the valohai-utils line will be just added there.

By default, your step will use a standard Python Docker image and install everything from requirements.txt during runtime.

You could also specify a different Docker image by passing the image name to prepare.

import valohai

# Prepares a new step called train in valohai.yaml
# Execute this Python file to run every time the train step is launched in Valohai
# inside a GPU enabled Tensorflow 2.6.1 image.
valohai.prepare(step="train", image="tensorflow/tensorflow:2.6.1-gpu")

and then rerun vh yaml step to update your valohai.yaml.

vh yaml step myfilename.py

Create a new file called valohai.yaml and define a Step inside it.

In our example below we’re defining two commands to be ran every time the step called train is ran. You can remove the pip install command, if you don’t have a need for it.

- step:
    name: train
    image: tensorflow/tensorflow:2.6.1-gpu
    command:
      - pip install -r requirements.txt
      - python myfilename.py

You can now upload your local code to Valohai and run the train step inside the Docker container you just defined.

vh execution run --adhoc train

Using –adhoc

The --adhoc flag allows you to easily upload your local changes to Valohai without going through Git. Ideally, you’d always upload your changes to Git but sometimes it’s just easier to test small changes using --adhoc

The other option would be to go through git:

git add .
git commit -m "Added Valohai valohai.yaml"
git push

# Fetch new commits to Valohai
vh project fetch
# Run a new execution based on the latest Git commit
vh exec run train