Customize node environment

Each node of a pipeline can run in a different type of an environment, like a different type of a cloud instance or an on-premise machine You have several options to select the environments for your pipeline steps in Valohai:

Using the web app

• Create a new pipeline.
• Click on the specific step in the pipeline graph.
• Choose the desired environment from the dropdown menu in the “Runtime” section.

Define Environments in valohai.yaml

• In your valohai.yaml, use the environment property in the step definition.
• The value should be the environment slug, which you can obtain by running vh environments in your CLI.
• If this property is not defined, the default environment set for your organization or project will be used.

- step:
    name: preprocess-dataset
    image: python:3.9
    environment: aws-eu-west-1-g3s-xlarge
    command:
      - pip install numpy valohai-utils
      - python ./preprocess_dataset.py
    inputs:
      - name: dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz

Override Environment in valohai.yaml

In some cases, you may want to change the environment if the execution/Task is running inside a pipeline. You can do this with pipeline overrides in valohai.yaml.

- pipeline:
    name: Training Pipeline
    nodes:
      - name: preprocess
        type: execution
        step: preprocess-dataset
        override:
          environment: aws-eu-west-1-g3s-xlarge 
      - name: train
        type: execution
        step: train-model
      - name: evaluate
        type: execution
        step: batch-inference
    edges:
      - [preprocess.output.preprocessed_mnist.npz, train.input.dataset]
      - [train.output.model*, evaluate.input.model]