Each node of a pipeline can run in a different type of an environment, like a different type of a cloud instance or an on-premise machine You have several options to select the environments for your pipeline steps in Valohai:
Using the web app
- Create a new pipeline.
- Click on the specific step in the pipeline graph.
- Choose the desired environment from the dropdown menu in the “Runtime” section.
Define Environments in valohai.yaml
- In your
valohai.yaml, use the
environmentproperty in the step definition.
- The value should be the environment slug, which you can obtain by running
vh environmentsin your CLI.
- If this property is not defined, the default environment set for your organization or project will be used.
- step: name: preprocess-dataset image: python:3.9 environment: aws-eu-west-1-g3s-xlarge command: - pip install numpy valohai-utils - python ./preprocess_dataset.py inputs: - name: dataset default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz
Override Environment in valohai.yaml
In some cases, you may want to change the environment if the execution/Task is running inside a pipeline. You can do this with pipeline overrides in
- pipeline: name: Training Pipeline nodes: - name: preprocess type: execution step: preprocess-dataset override: environment: aws-eu-west-1-g3s-xlarge - name: train type: execution step: train-model - name: evaluate type: execution step: batch-inference edges: - [preprocess.output.preprocessed_mnist.npz, train.input.dataset] - [train.output.model*, evaluate.input.model]