Define Valohai parameters

See also

This how-to is a part of our Bring your existing projects to Valohai series.

A short introduction parameters

  • Defining parameters allows you to easily rerun, sort, and keep track of executions based on the parameter values used to run them.

  • You can easily create (or copy) an execution and change the parameters in the UI, without changing your code.

  • Defining parameters allows you to start creating Tasks where you run multiple parallel executions with different parameter combinations.

  • Each step has it’s own parameter configuration in valohai.yaml.

  • A parameter can be type of a string, integer, float, and flag (=boolean).

  • Parameters are passed as command-line arguments to your code.

    • Edit how parameters are passed to your code using pass-as.

    • You can also parse parameter values from YAML and JSON files inside your execution. See file-based configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import valohai

# Define inputs available for this step and their default location
# The default location can be overriden when you create a new execution (UI, API or CLI)
default_inputs = {
    'myinput': 's3://bucket/mydata.csv'
}

# Define parameters in a dictionary
default_parameters = {
    'iterations': 10,
}

# Open the CSV file from Valohai inputs
with open(valohai.inputs("myinput").path()) as csv_file:
    reader = csv.reader(csv_file, delimiter=',')

# Create a step 'train' in valohai.yaml with a set of parameters
valohai.prepare(step="train", image="tensorflow/tensorflow:2.6.1-gpu", default_inputs=default_inputs, default_parameters=default_parameters)

# Access the parameters in your code
for i in range(valohai.parameters('iterations').value):
    print("Iteration %s" % i)

Generate or update your existing YAML file by running

vh yaml step myfile.py

The generated valohai.yaml configuration file looks like:

import argparse

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--iterations', type=int, default=10)
    return parser.parse_args()

args = parse_args()

print(args.iterations)

Create a valohai.yaml configuration file and define your step in it:

library(optparse)

option_list <- list(
    make_option("--iterations", default = 10),
)

parser <- OptionParser(option_list = option_list)
args <- parse_args(parser)

iterations <- args$iterations
write(iterations, stdout())

Create a valohai.yaml configuration file and define your step in it:

- step:
    name: Train model
    image: tensorflow/tensorflow:1.13.1
    command: python myfile.py {parameters}
    parameters:
      - name: iterations
        type: integer
        default: 100