Step

Each step in Valohai defines a specific workload type, such as data anonymization, data generation, feature extraction, training, model evaluation, and batch inference.

To execute a step, you create an execution, which effectively runs the step. Executions are rigorously version-controlled, ensuring that re-running any previous workloads remains feasible as long as the Docker image and inputs are still available.

Valohai provides users with maximum flexibility in constructing their own data science pipelines, recognizing the diversity of machine learning projects. Typically, steps are defined for various purposes, including:

1. Preprocessing files and uploading them for use by other steps.
2. Integrating with database services to establish version-controlled snapshots of training data.
3. Executing Python scripts or C code, such as training predictive models.
4. Validating the suitability of a trained model for production.
5. Deploying trained models to staging or production environments.
6. Building application binaries for use in subsequent steps.

Steps are specified in the project’s valohai.yaml configuration file.

Example steps in valohai.yaml

---
- step:
    name: preprocess-dataset
    image: python:3.9
    command:
      - pip install numpy valohai-utils
      - python ./preprocess_dataset.py
    inputs:
      - name: dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz

- step:
    name: train-model
    image: tensorflow/tensorflow:2.6.0
    command:
      - pip install valohai-utils
      - python ./train_model.py {parameters}
    parameters:
      - name: epochs
        default: 5
        type: integer
      - name: learning_rate
        default: 0.001
        type: float
    inputs:
      - name: dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/preprocessed_mnist.npz