Parallel pipelines

With parallel pipelines you can easily run the same pipeline blueprint multiple times with different parameter values.

For example:

• You’re working with images from various factories. You want to seperate each factory to it’s own pipeline and easily launch one pipeline per factory.
• I need to finetune my model for different use cases. For example, we have a model specific to a ship type (cruiser) and we need to finetune the model for new types (e.g. cargo ships).

What’s needed?

1. You’ll need to make sure your pipeline has at least one pipeline parameter defined in it.
2. Your pipeline should be pushed to a remote Git repository and fetched to Valohai.

Example pipeline

- step:
    name: pre_step
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./preprocess.py
    parameters:
      - name: exec_id
        type: string
        default: "id_123"
      - name: pre_param
        type: integer
        default: 5
    inputs:
      - name: pre_dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz

- step:
    name: train_step
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./train.py
    parameters:
      - name: exec_id
        type: string
        default: "id_123"
      - name: train_param
        type: integer
        default: 5
    inputs:
      - name: dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz
        optional: true

- pipeline:
    name: Example for Pipeline Task
    parameters:
      - name: id
        targets:
          - preprocess.parameters.exec_id
          - train.parameters.exec_id
        default: "id_123"
    nodes:
      - name: preprocess
        step: pre_step
        type: execution
      - name: train
        step: train_step
        type: execution
        override:
          inputs:
            - name: dataset
    edges:
      - [preprocess.output.preprocessed_mnist.npz, train.input.dataset]

Launch from the user interface

Getting started with your pipeline is a breeze. Just follow these steps:

• Create Your Pipeline:
  • Click on “Create Pipeline.”
  • Choose the commit and pipeline you want to work with.
• Configure Pipeline Parameters:
  • Scroll down to “Pipeline Parameters.”
  • Change “Single Pipeline” to “Pipeline Task” (this lets you set up multiple pipelines).
  • Make sure to set the ‘Variant’ to on.
• Define Your Parameters:
  • Enter the different values you want for your pipeline parameters.
• Create Your Pipeline:
  • Hit “Create a Pipeline.”
  • Voila! You’ll now see a Task with multiple pipelines, each using the parameter values you set.

In the screenshot we’ve defined three values for id. This means that we’ll get 3 pipelines, each of them with their own id.