Valohai YAML

Each Valohai-enabled git repository has a valohai.yaml configuration file that defines the runs that can be executed and what kind of REST endpoints can be automatically generated by the project. We’ll cover these execution types or “steps”, as we call them, in the next section.

A single configuration file can be used by multiple projects by various users, as long as the user has access to the git repository.

Example of a valohai.yaml file:

# This step downloads a set of images
# and runs preprocess.py
- step:
    name: Preprocess dataset (MNIST)
    image: tensorflow/tensorflow:1.13.1-py3
    command: python preprocess.py
    inputs:
      - name: training-set-images
        default: https://valohaidemo.blob.core.windows.net/mnist/imgages.tar.gz

# This step downloads preprocessed-data
# and runs train.py with two parameters (max_steps and learning_rate)
- step:
    name: Train model
    image: tensorflow/tensorflow:1.13.1-py3
    command: python train.py {parameters}
    parameters:
      - name: max_steps
        type: integer
        default: 300
      - name: learning_rate
        type: float
        default: 0.001
    inputs:
      - name: preprocessed-data
        default: https://valohaidemo.blob.core.windows.net/mnist/data.tar.gz

# This step downloads a trained model file and images,
# installs libraries listed in requirements.txt
# and runs batch_inference.py
- step:
    name: Batch inference
    image: tensorflow/tensorflow:1.13.1-py3
    command:
      - pip install -r requirements.txt
      - python batch_inference.py
    inputs:
      - name: model
      - name: images

Tip

Python users can use valohai-utils to define Valohai steps in their code, and then run vh yaml step myfile.py to generate the YAML file.

Large YAML files

Each Project can have one valohai.yaml file. As your project grows, you might start having repetitive content in your YAML file.

We recommend using YAML anchors, aliases, and extensions to define and re-use repeatitive content.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
- definitions:
    my-inputs: &my_inputs
        - name: test-set
          keep-directories: suffix
          default: s3://onboard-sample/test/*
        - name: train-set
          keep-directories: suffix
          default: s3://onboard-sample/train/*

- step:
    name: train-model
    image: tensorflow/tensorflow:2.0.1
    command:
      - pip install valohai-utils numpy
      - python ./train.py {parameters}
    parameters:
      - name: epoch
        default: 5
        type: integer
    inputs: *my_inputs