Each node of a pipeline can run in a different type of an environment, like a different type of a cloud instance or an on-premise machine You have several options to select the environments for your pipeline steps in Valohai:
Using the web app
- Create a new pipeline.
- Click on the specific step in the pipeline graph.
- Choose the desired environment from the dropdown menu in the “Runtime” section.
Define Environments in valohai.yaml
- In your
valohai.yaml
, use theenvironment
property in the step definition. - The value should be the environment slug, which you can obtain by running
vh environments
in your CLI. - If this property is not defined, the default environment set for your organization or project will be used.
- step:
name: preprocess-dataset
image: python:3.9
environment: aws-eu-west-1-g3s-xlarge
command:
- pip install numpy valohai-utils
- python ./preprocess_dataset.py
inputs:
- name: dataset
default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz
Override Environment in valohai.yaml
In some cases, you may want to change the environment if the execution/Task is running inside a pipeline. You can do this with pipeline overrides in valohai.yaml
.
- pipeline:
name: Training Pipeline
nodes:
- name: preprocess
type: execution
step: preprocess-dataset
override:
environment: aws-eu-west-1-g3s-xlarge
- name: train
type: execution
step: train-model
- name: evaluate
type: execution
step: batch-inference
edges:
- [preprocess.output.preprocessed_mnist.npz, train.input.dataset]
- [train.output.model*, evaluate.input.model]