Pipeline parameters

In certain scenarios, you may have parameters that multiple nodes within the pipeline share. Instead of manually updating each node individually when parameter values change, you can define pipeline parameters.

These parameters supersede the default values specified in valohai.yaml, even when the default value is left empty.

This approach streamlines parameter management and updates across multiple nodes.

Define pipeline parameters in valohai.yaml

Similarly to step parameters, pipeline parameters are added in the valohai.yaml file. You will need to specify the targets for them, i.e. the step parameters that they will override.

Example valohai.yaml

• This example extends the example in the previous chapter Introduction to pipelines, with additional modifications highlighted
• The syntax for defining pipeline parameter targets is <node-name>.parameters.<step-parameter-name>.
• The pipeline parameter id gets two targets: preprocess.parameters.exec_id and train.parameters.exec_id. This means that the pipeline parameter value overrides the exec_id value in both preprocess and train nodes.
• The exec_id for the train_in_task node is not affected by the pipeline parameter id but gets the default value from the step definition.
• Multi-value parameters can also be defined as pipeline parameters. In this example a new filters parameter is added to the preprocess-dataset step, which resolves as preprocess.parameters.filters for the pipeline node

Keep in mind that you can’t directly access the pipeline parameter’s value; you’ll need to read it from the execution, like reading any other parameter from the parameter config file, parsing the command-line argument or using valohai-utils valohai.parameters("exec_id").value.

- step:
    name: preprocess-dataset
    image: python:3.9
    command:
      - pip install numpy valohai-utils
      - python ./preprocess_dataset.py
    parameters: 
      - name: filters
        type: string
        default: [ "low-pass" ]
    inputs:
      - name: dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz

- pipeline:
    name: Example for Pipeline Parameters
    parameters:
      - name: id
        targets:
          - preprocess.parameters.exec_id
          - train.parameters.exec_id
      - name: training_parameter
        targets:
          - train.parameters.train_param
        default: 3
      - name: task_parameter
        target: train_in_task.parameters.train_param
        default: 123
      - name: filters
        target: preprocess.parameters.filters
        default: [ "remove-outliers" ]
    nodes:
      - name: preprocess
        step: pre_step
        type: execution
      - name: train
        step: train_step
        type: execution
      - name: train_in_task
        step: train_step
        type: task
    edges:
      - [preprocess.output.preprocessed_mnist.npz, train.input.dataset]
      - [preprocess.output.preprocessed_mnist.npz, train_in_task.input.dataset]

In the web app

In the image below, you can find the node parameters and the pipeline parameters.

• Under the Shared Parameters section in the user interface, you’ll find pipeline parameters and which node parameters they’ll replace. You can also find the
• In the example below the parameter exec_id will be overwritten by the pipeline parameter id, and its value cannot be modified in the node’s parameter section.
• In the image, the pipeline parameter id is empty because there was no default value for it in valohai.yaml.
• The parameter pre_param hasn’t been designated as a target for any pipeline parameter, so you can edit its value under the node parameters.

You’ll also notice that train_in_task node has been set as a Task node, allowing you to specify multiple values for the pipeline parameter. If you define 10 parameter values, you’ll end up with 10 executions in this node, each having its unique parameter value.