# Introduction Valohai is a modular MLOps platform that orchestrates your ML workflows through configuration, without any invasive SDKs touching your code. Run experiments on any infrastructure like AWS, Azure, GCP, Oracle, Snowflake, on-premises, Kubernetes, Slurm. Start where it hurts most (data versioning, pipelines, compute efficiency) and expand from there. *** ## Already Running ML Somewhere? **Valohai fits into your existing stack.** We don't require rip-and-replace. If you're using MLflow, SageMaker, custom scripts, or Kubeflow—keep them. Valohai adds orchestration, reproducibility, and compute efficiency on top of what you already have. * [Migrate your ML jobs to Valohai](/migration-strategy) — step-by-step guide * [See how others migrated](/project-gallery) — YOLO, Mistral, MMDetection3D, and more *** ## Start Building ### New to Valohai? Get your first execution running in 10 minutes: * [Quickstart: Hello World](/getting-started/quickstart) * [Tutorial: Computer Vision Pipeline](/project-gallery/computer-vision) * [Why YAML over SDK?](/readme/philosophy/yaml-over-sdk) ### Common Tasks * [Run an execution](/executions/run-basic-execution) * [Build a data pipeline](/getting-started/intro/pipelines) * [Track experiments and metrics](/experiment-tracking) * [Serving your models](/serving-your-models) *** ## Example Projects See Valohai in action with production-ready templates: **Computer Vision** * [YOLO Object Detection](/project-gallery/computer-vision/yolo-example) — train and deploy YOLOv8 * [MMDetection3D](/project-gallery/computer-vision/mmdetection3d-on-valohai) — 3D object detection pipelines **NLP & LLM** * [Mistral Fine-Tuning](/project-gallery/nlp-and-llm/mistral-example) — fine-tune LLMs with distributed training * [RAG Documentation Assistant](/project-gallery/nlp-and-llm/rag-doc-example) — build a retrieval-augmented chatbot **Audio & Data Engineering** * [NVIDIA NeMo ASR Pipeline](/project-gallery/audio-and-speech/nvidia-nemo-valohai) * [Snowflake ML with Snowpark](/project-gallery/data-engineering-and-etl/snowpark-example) [Browse all examples →](/project-gallery) *** ## Learn the Platform ### Core Concepts Understand how Valohai works and why it's built this way: * [Docker in Valohai](/docker-in-valohai) — bring your own images * [Data Versioning](/models/artifacts-and-versioning) — automatic lineage and caching * [Pipelines](/pipelines) — chain jobs with dependency graphs * [Reproducibility by Default](/readme/philosophy/reproducibility-by-default) — every run is traceable ### How-To Guides Task-focused instructions for specific workflows: * [Configure execution environments](/executions) * [Use spot instances](/executions/advanced-features/spot-instances) * [Debug failed pipelines](/pipelines/debug-failed-pipeline) * [Set up private Docker registries](/docker-in-valohai/private-docker-registries) *** ## Get Help * **Need support?** [support.valohai.com](https://support.valohai.com) * **Want training?** 🎓 [Valohai Academy](https://learn.valohai.academy) * **See what's new?** 🆕 [Changelog](/changelog) # Academy **Master MLOps through hands-on learning paths—earn your certification.** Valohai Academy takes you from first execution to production-ready pipelines through structured modules, real-world exercises, and knowledge checks. *** ## Why Academy vs. Documentation? **Use Academy when:** * You're new to Valohai and want guided, progressive learning * You need to onboard a team with consistent training * You want to earn a certification for your LinkedIn profile **Use Docs when:** * You need a quick answer or specific how-to guide * You're looking up syntax, references, or troubleshooting steps * You want to explore concepts in any order Think of Academy as your **structured curriculum**, and Docs as your **on-demand reference library**. *** ## Who Should Enroll ### Data Scientists & ML Engineers Learn to version experiments, orchestrate pipelines, and scale compute—without touching infrastructure. ### Technical Decision Makers Evaluate platform capabilities through practical workflows. See how Valohai standardizes ML operations across teams. *** ## What You'll Learn ### Core Workflows * Run versioned experiments with automatic lineage tracking * Build multi-step pipelines with data dependencies * Debug executions using logs, metadata, and reproducibility tools ### Production Best Practices * Structure projects for team collaboration and reproducibility * Manage datasets across cloud storage providers * Optimize compute costs with spot instances and autoscaling ### Platform Integration * Connect Git repositories for code versioning * Configure Docker environments for custom dependencies * Integrate cloud storage (S3, Azure Blob, GCS) *** ## How Academy Works Each module follows a proven learning pattern: 1. **Conceptual foundation** — understand the "why" behind each workflow 2. **Hands-on exercises** — apply concepts in a real Valohai environment 3. **Knowledge checks** — validate understanding before moving forward 4. **Code examples** — see both YAML and Python (`valohai-utils`) approaches Complete all modules and pass knowledge checks to **earn your Certificate of Completion**—shareable on LinkedIn. *** ## Prerequisites * Basic Python knowledge * Familiarity with Git * Access to a Valohai organization ([start a free trial](https://valohai.com/signup)) No DevOps or YAML experience required. We'll guide you through everything. *** ## Get Started [**🎓 Enroll in Valohai Academy**](https://learn.valohai.academy) Already enrolled? [Continue your progress →](https://learn.valohai.academy/login) *** > 💡 **Tip:** Academy is included with all Valohai plans, including free trials. Some advanced platform features (like private Docker registries) require a paid plan. # Philosophy # YAML Over SDK Valohai is built on a simple principle: **your ML code should not be entangled with your MLOps platform**. Many tools ask you to decorate functions, subclass Tracker objects, or wrap training loops in platform-specific APIs. We don't. ## Why We Avoid SDK Integration Valohai treats `valohai.yaml` as the single source of truth for your ML workflows. This architectural choice brings immediate benefits: * **No vendor lock-in** — Your ML code stays agnostic, portable, and unmodified * **Zero hidden orchestration** — What you see in YAML is what runs * **No tight coupling** — Platform configuration lives separately from business logic ## Benefits of YAML-First Architecture ### Portability Without Pain Delete Valohai tomorrow and your code still runs. No refactoring, no API removal, no broken imports. ### Git-Native Configuration Your entire pipeline configuration lives in version control: * Track changes with standard Git workflows * Review pipelines in pull requests * Roll back configurations instantly ### Team-Friendly Reviews Platform engineers can audit pipelines without understanding your ML framework. Data scientists can focus on algorithms without learning orchestration APIs. ## The Philosophy in Practice When you run an execution, Valohai: 1. Reads your `valohai.yaml` configuration 2. Provisions the compute environment 3. Injects parameters command line arguments or through JSON/YAML files 4. Runs your code exactly as written Your training script doesn't need to know it's running on Valohai. That's the point. # Reproducibility by Default Most teams treat reproducibility as an afterthought — a fragile checklist of manual logs and naming conventions that break the moment someone forgets to follow them. Valohai flips this model. Every execution becomes an immutable, reproducible unit by default. No conventions needed, no extra code required. ## What Gets Tracked (Without You Lifting a Finger) Every execution automatically captures: * **Git commit hash** — the exact code that ran * **Docker image digest** — the precise environment * **Input file hashes** — data versions locked in * **Parameter values** — all configuration frozen * **Execution command** — the complete invocation This isn't metadata you log. It's infrastructure-level tracking that happens whether you remember or not. ## Why Infrastructure Beats Convention Traditional approaches fail because they rely on human discipline: * "Remember to log your hyperparameters" * "Follow our naming convention for experiments" * "Don't forget to track your data versions" Valohai's approach succeeds because it removes human error from the equation. Your ML engineer having a bad Monday? Doesn't matter. The execution is still fully reproducible. ## The Payoff: Reproduce Any Run, Anytime Six months later, when your model starts drifting in production, you can: 1. Find the original execution in your history 2. Click "Create execution from this" 3. Get identical results — same code, same data, same environment No detective work. No "wait, which dataset version was this?" No begging your colleague for their random seed. > 💡 *This reproducibility extends to your entire pipeline. If you chain executions together, every step maintains its audit trail.* *** *Your experiments are reproducible by default. You don't build it. We guarantee it.* # Unifying Your ML Infra Most ML teams build Frankenstein stacks: Airflow for orchestration, MLFlow for tracking, S3 for storage, Kubernetes for compute. Each tool solves one problem well—until you need them to work together. Valohai replaces fragmented MLOps tooling with a unified platform that handles orchestration, tracking, storage, and compute without glue code. ## The Cost of Fragmentation When you stitch together multiple tools, you inherit their collective problems: **Pipeline failures cascade mysteriously** * Airflow DAGs fail without propagating context to downstream tools * Error messages reference internal task IDs instead of ML concepts * Debugging requires SSH access across multiple systems **Data lineage evaporates between tools** * Training outputs land in S3 with no metadata * Model artifacts lose connection to their training runs * Reproducing results means archaeology through logs **Infrastructure becomes everyone's problem** * Data scientists debug Kubernetes networking * ML engineers maintain Airflow workers * Platform teams juggle incompatible tool versions ## The Unified Alternative Valohai connects every piece of the ML workflow through a single abstraction layer: **Executions replace scattered jobs** * Each run tracks inputs, outputs, logs, and metadata automatically * Failed steps show exactly which data and parameters were used * Re-running experiments preserves complete lineage **Pipelines orchestrate without overhead** * Define DAGs in YAML that version with your code * Pass outputs between steps without manual wiring * Monitor progress through one interface, not five dashboards **Infrastructure adapts to workloads** * Specify compute requirements per step (GPU type, memory, region) * Scale from laptops to cloud clusters with the same code * Pay only for what you use—no idle Kubernetes nodes ## When Unification Matters Most This approach pays dividends when: * Your team spends more time on infrastructure than ML * Reproducing old results requires tribal knowledge * Onboarding new team members takes weeks of tool training * Compliance audits demand end-to-end traceability # Let Data Scientists Be Scientists Data scientists shouldn't wrestle with cloud permissions, network configurations, or storage mounts. That's infrastructure work, not science. Valohai inverts the traditional model: your ops team owns the environments, your data scientists own the experiments. Whether you're on AWS, Azure, GCP, or on-premises, Valohai abstracts these details away. ## The Problem with Traditional Platforms Most ML platforms force data scientists to become part-time DevOps engineers. They demand cloud credentials, network configurations, and security policies before you can run a single experiment. This approach breaks down because: * Data scientists waste time on infrastructure instead of model improvement * Security risks multiply when everyone needs cloud access * Onboarding takes weeks instead of hours ## How Valohai Works Differently ### Zero Cloud Credentials for Data Scientists Your data science team never touches: * Cloud provider CLIs or authentication tokens * Virtual networks, subnets, or firewall rules * Identity management or permission policies * Storage bucket configurations or access keys Instead, they select pre-configured environments and run experiments. ### Clear Separation of Concerns **Infrastructure Team Handles:** * Environment setup and maintenance * Cloud resource provisioning * Security policies and access controls * Cost optimization and monitoring **Data Science Team Focuses On:** * Experiment design and execution * Model architecture and hyperparameters * Data preprocessing and feature engineering * Results analysis and iteration ## The Outcome This separation delivers concrete benefits: * **Faster onboarding**: New team members run experiments on day one * **Better security**: Cloud credentials stay with the ops team * **Higher productivity**: Data scientists spend 100% of their time on ML problems * **Controlled costs**: Centralized environment management prevents resource sprawl ## Implementation in Practice Here's how a data scientist runs an experiment: ```shell vh execution run --environment production-gpu ``` Behind that simple command, Valohai handles: * Provisioning the right compute instance * Mounting data stores with proper credentials * Injecting secrets and configuration * Setting up monitoring and logging The data scientist sees none of this complexity. They get results, not infrastructure headaches. ## When to Use This Pattern This approach works best when: * Your team has dedicated infrastructure/platform engineers * Security and compliance matter * You want to scale beyond a handful of researchers * Cloud costs need active management # Getting Started Valohai is an MLOps platform that handles infrastructure complexity while you build production ML systems. Train models, run experiments, and deploy to production, all without DevOps overhead. ### Core Capabilities #### Full experiment tracking and lineage Every run becomes reproducible and auditable. * **Automatic versioning** — Code, data, parameters, and environments captured on every run * **Metric comparison** — Compare runs, spot regressions, track model drift * **Dataset versioning** — Link datasets to experiments without storage duplication #### Infrastructure abstraction Run ML workloads on any compute with one command. * **Multi-cloud execution** — AWS, GCP, Azure, Oracle Cloud Infrastructure, Scaleway, OVH, Slurm, Kubernetes, or on-premises hardware * **Elastic scaling** — Same code runs on 1 GPU or 100 GPUs * **Production deployment** — Batch inference, REST APIs, or streaming endpoints with built-in monitoring #### Framework agnostic Your code, your tools, zero lock-in. * **Any ML framework** — PyTorch, TensorFlow, JAX, Hugging Face, or custom stacks * **Simple integration** — Add a `valohai.yaml` to any project * **API-first design** — REST API and webhooks for CI/CD pipelines ### Who uses Valohai? **Data Scientists & ML Engineers** — Focus on model development instead of cloud configurations\ **MLOps Teams** — Standardize workflows across projects without forcing tool changes\ **Enterprise ML Teams** — Meet compliance requirements with full audit trails and data lineage ### Start Building | Resource | Description | | ------------------------------------------------------------ | ------------------------------------------------- | | [Quickstart](/getting-started/quickstart) | Run your first execution in 5 minutes | | [🎓 Valohai Academy](https://learn.valohai.academy/) | Interactive learning path with hands-on exercises | | [Example: Computer Vision](/project-gallery/computer-vision) | Import a working computer vision pipeline | | [Example: Fine-tune LLMs](/project-gallery/nlp-and-llm) | Adapt language models to your domain | > 💡 **First time with MLOps?** Start with [Valohai Academy](https://learn.valohai.academy/) for guided tutorials that build from basics to advanced workflows. # Hello World Start building reproducible ML experiments in minutes. This guide walks you through creating and running your first Valohai execution with the help of the `valohai-utils` library. ## Prerequisites * Python 3.9 or later * A Valohai account ([create one free](https://app.valohai.com)) ## Install the CLI Get the Valohai CLI and utilities for experiment tracking: ```shell pip install valohai-cli ``` > **Tip:** Use `pipx install valohai-cli` to avoid dependency conflicts. ## Login ```shell vh login ```
Using SSO or self-hosted Valohai? **SSO Users (Azure, Google, SAML)** Generate a personal access token: 1. Go to **Hi, \[username]** → **My Profile** → **Authentication** 2. Click **Manage Tokens** → **Create a new personal token** 3. Save the token securely (shown only once) ```shell vh login --token ``` **Self-Hosted Installations** ```shell vh login --host https://your-company.valohai.io ``` > 💡 Combine options: `vh login --host https://your-company.valohai.io --token `
## Create Your First Project Set up a project directory and connect it to Valohai: ```shell mkdir my-ml-project cd my-ml-project vh project create --name my-ml-project ``` This links your local directory to Valohai for experiment tracking. > **Already have a project?** Link to it with `vh project link` and select from the list. ## Write Your Training Script Save this as `train.py`: ```python import valohai import json # Define hyperparameters with automatic tracking params = { "learning_rate": valohai.parameters("learning_rate").value, "epochs": valohai.parameters("epochs").value, } print(f"Training with lr={params['learning_rate']} for {params['epochs']} epochs") # Simulate training (replace with your model code) for epoch in range(params["epochs"]): accuracy = 0.85 + (epoch * 0.02) # Mock improving accuracy valohai.log_metric("accuracy", accuracy, step=epoch) print(f"Epoch {epoch}: accuracy={accuracy:.3f}") # Save model or results output_path = valohai.outputs().path("model_results.json") with open(output_path, "w") as f: json.dump( { "final_accuracy": accuracy, "parameters": params, "model_version": "1.0", }, f, ) print(f"Results saved to {output_path}") print("Training complete!") ``` ## Configure Your Execution Environment Create `valohai.yaml` to define how your code runs: ```yaml - step: name: train image: python:3.9 command: - pip install valohai-utils - python train.py parameters: - name: learning_rate type: float default: 0.001 - name: epochs type: integer default: 10 ``` ### About Docker Images The `image` field specifies your execution environment. Think of it as a clean environment with only the software you specify. For example: * **Python**: `python:3.9`, `python:3.11` * **TensorFlow**: `tensorflow/tensorflow:2.13.0` * **PyTorch**: `pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime` * **R**: `r-base:4.3.0` * **Custom**: Your own Docker images with specific dependencies > **Note:** The Docker image can different from your local Python version. Valohai runs your code in this isolated environment for reproducibility. ## Run Your First Execution Submit the job and watch logs in real-time: ```shell vh execution run train --adhoc --watch ``` What happens: * `--adhoc` uploads your current code without pushing to Git (great for testing) * `--watch` streams logs to your terminal * Valohai tracks all inputs, outputs, and parameters automatically ## View Results in the UI Open your execution in the browser: ```shell vh execution open ``` The UI shows: * Real-time logs and metrics charts * Parameter values and configuration * Output files (downloadable) * Full reproducibility information ## What's Next? You've successfully run your first Valohai execution! Here's where to go next: **🎓** [**Valohai Academy**](https://learn.valohai.academy)\ Free courses on MLOps best practices and advanced Valohai features [**Working with Data**](/data)\ Upload datasets and connect them to executions [**Building Pipelines**](/getting-started/intro/pipelines)\ Chain multiple steps for end-to-end ML workflows [**Using Git Integration**](/git-integration)\ Connect repositories for production-ready experiments *** ### Troubleshooting
Command 'vh' not found The CLI isn't in your system PATH. Fix it: **macOS/Linux:** ```bash echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc source ~/.bashrc ``` **Windows:** Add the Python Scripts folder to your PATH environment variable. **Alternative:** Run with `python -m valohai_cli` instead of `vh`.
Project not linked Run `vh project link` to connect your directory to an existing Valohai project.
Import error for valohai module Install the Valohai utilities as a part of your step in `valohai.yaml`: ```shell pip install valohai-utils ```
# Tutorial - Computer Vision Learn how to build production-ready ML workflows on Valohai through a practical example. This series takes you beyond "Hello World" to master the features that make ML workflows reproducible, scalable, and collaborative. ## Why This Tutorial Series? You've run your first Valohai execution. Now what? This series teaches you how to transform any ML project into a production-ready workflow. We use YOLOv8 (a computer vision model) as our example, but **you don't need any computer vision knowledge**. The patterns you'll learn apply to any ML project—from CV, geospatial data, NLP to time series forecasting. By the end, you'll have built a complete ML pipeline that: * Runs reproducibly on any infrastructure * Tracks all experiments automatically * Passes data between training and inference steps * Can be scheduled, monitored, and iterated on by your team ## What You'll Build Starting with a simple training script, you'll progressively add Valohai features to create a full pipeline: ### 1. [Defining Jobs](/getting-started/intro/jobs) Transform a Python script into a Valohai execution. You'll run YOLOv8 training on remote machines while Valohai automatically versions every output file and creates aliases for your trained models. ### 2. [Using Parameters](/getting-started/intro/parameters) Make your training configurable without touching code. Control hyperparameters from the UI, compare different model configurations, and track which settings produced each model. ### 3. [Logging Metrics](/getting-started/intro/metrics) Track training progress in real-time. Print JSON from your scripts to automatically log metrics, then visualize and compare performance across experiments. ### 4. [Building Pipelines](/getting-started/intro/pipelines) Connect multiple jobs into automated workflows. Build a train-and-predict pipeline where outputs flow seamlessly between steps—no manual file handling needed. ### 5. [Working with Data](/getting-started/intro/data) Master data management. Load datasets from cloud storage, chain outputs from previous jobs, and override data sources at runtime for maximum flexibility. ## Before You Start **Required:** * Completed the [Quickstart guide](/getting-started/quickstart) * A Valohai project connected to a Git repository * Basic Python knowledge **Not Required:** * YOLOv8 or computer vision experience * DevOps or infrastructure knowledge ## How to Use This Series While each tutorial can stand alone, we recommend following them in order. Each builds on concepts from the previous one, gradually introducing more powerful features. > 💡 **About YOLOv8:** We use this object detection model because it generates multiple output files during training (logs, checkpoints, and a `best.onnx` model file). This makes it perfect for demonstrating Valohai's file handling and versioning capabilities. The concepts apply to any ML framework. Ready? Let's start by [defining your first job](/getting-started/intro/jobs) → # Defining Jobs > 💡 **About this tutorial:** We use YOLOv8 as a practical example to demonstrate Valohai's features. You don't need computer vision knowledge—the patterns you learn here apply to any ML framework. This tutorial focuses on defining jobs (=executions) and saving their output files while ensuring proper versioning and tracking of your ML workflows. ### Prerequisites * Python 3.8 or later * A Valohai account ([create one free](https://app.valohai.com)) ### Install the CLI Get the Valohai CLI and utilities for experiment tracking: ```shell pip install valohai-cli ``` > **Tip:** Use `pipx install valohai-cli` to avoid dependency conflicts. ### Login ```shell vh login ```
Using SSO or self-hosted Valohai? **SSO Users (Azure, Google, SAML)** Generate a personal access token: 1. Go to **Hi, \[username]** → **My Profile** → **Authentication** 2. Click **Manage Tokens** → **Create a new personal token** 3. Save the token securely (shown only once) ```shell vh login --token ``` **Self-Hosted Installations** ```shell vh login --host https://your-company.valohai.io ``` > 💡 Combine options: `vh login --host https://your-company.valohai.io --token `
### Create Your First Project Set up a project directory and connect it to Valohai: ```shell mkdir my-ml-project cd my-ml-project vh project create --name my-ml-project ``` This links your local directory to Valohai for experiment tracking. ### Write Your Training Script Save this as `train.py`. Half of it is the standard model training from Yolo and the other half is Valohai specific where we: * Copy the trained model files to `/valohai/outputs/` direction, from Valohai will version and upload them * The yolo training script will develop several files, including a file called `best.onnx` * We'll want this create an Valohai alias called `latest-model` that we can reference, and it'll always point to the newest file generated by this job. (that we'll use for inferencing later) * Generate a JSON file called `best.onnx.metadata.json` where we put in some JSON, to define that we want this the alias `latest-model` to point to this newly generated file. Read more about aliases in our alias docs. ```python import shutil from ultralytics import YOLO import json # Load a pretrained model (recommended for training) model = YOLO("yolov8n.pt") # Train the model model.train(data="coco128.yaml", epochs=1, verbose=False) # Export the model to ONNX format path = model.export(format="onnx") # Valohai parts start here # Copy the exported model to the Valohai outputs directory shutil.copy(path, "/valohai/outputs/") # Define a JSON dictionary containing a friendly name # You can then reference this file with datum://latest-model file_metadata = { "valohai.alias": "latest-model", } # Attach the metadata to the file with open("/valohai/outputs/best.onnx.metadata.json", "w") as f: f.write(file_metadata) ``` ### Configure Your Execution Environment Create `valohai.yaml` to define how your code runs: Update the environment field with the GPU machine you want to run the job on. You can see a list of available GPU machines by running vh environments --gpu. Use the slug name provided in the output. ```yaml - step: name: yolo image: docker.io/ultralytics/ultralytics:8.0.180-python command: python train.py environment: aws-eu-west-1-p3-2xlarge ``` #### About Docker Images The `image` field specifies your execution environment. Think of it as a clean environment with only the software you specify. For example: * **Python**: `python:3.9`, `python:3.11` * **TensorFlow**: `tensorflow/tensorflow:2.13.0` * **PyTorch**: `pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime` * **R**: `r-base:4.3.0` * **Custom**: Your own Docker images with specific dependencies > **Note:** The Docker image can different from your local Python version. Valohai runs your code in this isolated environment for reproducibility. ### Run Your First Execution Submit the job and watch logs in real-time: ```shell vh execution run yolo --adhoc --watch ``` What happens: * `--adhoc` uploads your current code without pushing to Git (great for testing) * `--watch` streams logs to your terminal * Valohai tracks all inputs, outputs, and parameters automatically ### View Results in the UI Open your execution in the browser: ```shell vh execution open ``` The UI shows: * Real-time logs and metrics charts * Parameter values and configuration * Output files (downloadable) * Full reproducibility information *** #### Troubleshooting
Command 'vh' not found The CLI isn't in your system PATH. Fix it: **macOS/Linux:** ```bash echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc source ~/.bashrc ``` **Windows:** Add the Python Scripts folder to your PATH environment variable. **Alternative:** Run with `python -m valohai_cli` instead of `vh`.
Project not linked Run `vh project link` to connect your directory to an existing Valohai project.
Import error for valohai module Install the Valohai utilities as a part of your step in `valohai.yaml`: ```shell pip install valohai-utils ```
# Using Parameters > 💡 **About this tutorial:** We use YOLOv8 as a practical example to demonstrate Valohai's features. You don't need computer vision knowledge—the patterns you learn here apply to any ML framework. This tutorial focuses on defining parameters and configuration values while ensuring proper versioning and tracking of your ML workflows. ### Define parameters in valohai.yaml Tell Valohai which arguments your training script accepts. Parameters become command-line arguments automatically, no boilerplate needed. ```yaml - step: name: yolo image: docker.io/ultralytics/ultralytics:8.0.180-python command: python train.py {parameters} environment: aws-eu-west-1-p3-2xlarge parameters: - name: epochs type: integer default: 3 - name: verbose type: flag default: False pass-true-as: --verbose=True pass-false-as: --verbose=False ``` **What's happening here:** * `{parameters}` placeholder → Valohai injects `--epochs=3 --verbose=True` * `environment` → GPU instance type (find available options: `vh environments --gpu`) * Flag parameters need explicit pass-true/false values for boolean handling ### Parse parameters in Python Use Python's standard `argparse` to receive these values. No special Valohai SDK required. ```python import shutil from ultralytics import YOLO import argparse import json def parse_args(): parser = argparse.ArgumentParser() parser.add_argument("--epochs", type=int, default=3) parser.add_argument("--verbose", type=bool, default=False) return parser.parse_args() args = parse_args() # Load a model model = YOLO("yolov8n.pt") # load a pretrained model (recommended for training) # Use the model model.train(data="coco128.yaml", epochs=args.epochs, verbose=args.verbose) # train the model path = model.export(format="onnx") # export the model to ONNX format # Copy the exported model to the Valohai outputs directory shutil.copy(path, "/valohai/outputs/") # Define a JSON dictionary containing a friendly name # You can then reference this file with datum://latest-model file_metadata = { "valohai.alias": "latest-model", } # Attach the metadata to the file with open("/valohai/outputs/best.onnx.metadata.json", "w") as f: f.write(file_metadata) ``` ### Run with custom parameters Override defaults directly from the CLI: ```shell vh execution run yolo --adhoc --open-browser --epochs=1 ``` Your execution details page now shows: * **Parameters section**: Displays `epochs: 1` and other values * **Logs**: Confirms YOLOv8 ran exactly 1 epoch ### Compare experiments at a glance Navigate to the **Executions** tab to see all your runs in a table with parameter columns. No more guessing which job used which configuration. *** #### Related content * [Add hyperparameter sweeps](/notebook-executions/tune-from-notebook) to test multiple configurations automatically # Logging Metrics > 💡 **About this tutorial:** We use YOLOv8 as a practical example to demonstrate Valohai's features. You don't need computer vision knowledge—the patterns you learn here apply to any ML framework. This tutorial focuses on collecting performance metrics while ensuring proper versioning and tracking of your ML workflows. Valohai automatically tracks metrics whenever your script prints JSON. No SDK needed, just `print()` your metrics as JSON and we'll handle the rest. ### Why metrics matter * **Sort executions** by metric values in your dashboard * **Compare performance** across multiple training runs * **Set early stopping** rules to halt underperforming jobs * **Control pipelines** by advancing stages based on metric thresholds ### Print final metrics YOLOv8 already outputs metrics in JSON format. Here's how to capture them: ```python import shutil from ultralytics import YOLO import argparse import json def parse_args(): parser = argparse.ArgumentParser() parser.add_argument("--epochs", type=int, default=3) parser.add_argument("--verbose", type=bool, default=False) return parser.parse_args() args = parse_args() # Load a model model = YOLO("yolov8n.pt") # Load a pretrained model (recommended for training) # Use the model model.train(data="coco128.yaml", epochs=args.epochs, verbose=args.verbose) # Train the model path = model.export(format="onnx") # Export the model to ONNX format metadata = {} # Loop through the metrics for metric in model.metrics.results_dict: # Some metrics have a 'metrics/' prefix (e.g., metrics/precision) # We split it to isolate the actual metric name. metric_name = metric.split("metrics/")[-1] metric_value = model.metrics.results_dict[metric] metadata[metric_name] = metric_value # Print the JSON dictionary to register metrics and their values in Valohai print(json.dumps(metadata)) # Copy the exported model to the Valohai outputs directory shutil.copy(path, "/valohai/outputs/") # Define a JSON dictionary containing a friendly name # You can then reference this file with datum://latest-model file_metadata = { "valohai.alias": "latest-model", } # Attach the metadata to the file with open("/valohai/outputs/best.onnx.metadata.json", "w") as f: f.write(file_metadata) ``` ### Run and view metrics Launch your training and watch the metrics appear in your execution logs: ```shell vh execution run yolo --adhoc --open-browser ``` Navigate to the **Executions** tab to see your metrics in table format: > 💡 **Customize your view:** Click the controls above the table (marked as #2) to choose which columns to display and set decimal precision. ### Track metrics during training Print metrics after each epoch to monitor training progress in real-time. YOLOv8 uses callbacks for this: ```python import shutil from ultralytics import YOLO import argparse import json def parse_args(): parser = argparse.ArgumentParser() parser.add_argument("--epochs", type=int, default=2) parser.add_argument("--verbose", type=bool, default=False) return parser.parse_args() args = parse_args() def print_valohai_metrics(trainer): metadata = { "epoch": trainer.epoch, } # Loop through the metrics for metric in trainer.metrics: metric_name = metric.split("metrics/")[-1] metric_value = trainer.metrics[metric] metadata[metric_name] = metric_value print(json.dumps(metadata)) # Load a model model = YOLO("yolov8n.pt") # Load a pretrained model (recommended for training) model.add_callback("on_train_epoch_end", print_valohai_metrics) # Use the model model.train(data="coco128.yaml", epochs=args.epochs, verbose=args.verbose) # Train the model path = model.export(format="onnx") # Export the model to ONNX format metadata = {} # Loop through the metrics for metric in model.metrics.results_dict: # Some metrics have a 'metrics/' prefix # Splitting it yields the actual metric name metric_name = metric.split("metrics/")[-1] metric_value = model.metrics.results_dict[metric] metadata[metric_name] = metric_value # Valohai metrics are collected as JSON key:value pairs print(json.dumps(metadata)) # Copy the exported model to the Valohai outputs directory shutil.copy(path, "/valohai/outputs/") # Define a JSON dictionary containing a friendly name # You can reference this file with datum://latest-model file_metadata = { "valohai.alias": "latest-model", } # Attach the metadata to the file with open("/valohai/outputs/best.onnx.metadata.json", "w") as f: f.write(json.dumps(file_metadata)) ``` Run with more epochs to see live metric updates: ```shell vh execution run yolo --epochs=10 --adhoc --open-browser ``` Open the **Metadata** tab and select: * **X-axis:** epoch * **Y-axis:** any metric you want to track Watch your training curves update in real-time: ### Compare executions Need to compare hyperparameter experiments? 1. Go to the **Executions** tab 2. Check the boxes next to executions you want to compare 3. Click **Compare** above the table You'll see all metrics side-by-side for easy analysis. *** #### Related topics * [Experiment Tracking & Visualizations](/experiment-tracking) * [Early stopping with metrics](/tasks/early-stopping) * [Pipeline conditions](/pipelines/dynamic-conditions) # Working with Data > 💡 **About this tutorial:** We use YOLOv8 as a practical example to demonstrate Valohai's features. You don't need computer vision knowledge—the patterns you learn here apply to any ML framework. This tutorial focuses on ingesting data, running inference, and saving resulting files while ensuring proper versioning and tracking of your ML workflows. ### Define Inputs in valohai.yaml Add input groups to any step in your `valohai.yaml`: ```yaml - step: name: inference image: docker.io/ultralytics/ultralytics:8.0.180-python command: python inference.py inputs: - name: model default: datum://latest-model # References data from your catalog filename: best.onnx # Renames the file when downloaded - name: images default: https://ultralytics.com/images/bus.jpg ``` #### Input Configuration Options * **name**: Identifier for accessing files at `/valohai/inputs/{name}/` * **default**: Pre-filled source (URL, S3 path, or `datum://` reference) * **filename**: Rename downloaded files for consistent access > 💡 *Use `datum://` to reference any file in your project's data catalog, including outputs from previous executions.* ### Access Inputs in Your Code All inputs download to `/valohai/inputs/{input-name}/` before your code runs: ```python from ultralytics import YOLO import os # Inputs are always at predictable paths path_to_model = "/valohai/inputs/model/best.onnx" path_to_images = "/valohai/inputs/images/" model = YOLO(path_to_model) # Process all files in the images input for image in os.listdir(path_to_images): image_path = os.path.join(path_to_images, image) if os.path.isfile(image_path): results = model.predict(image_path, save=True, project="/valohai/outputs", name="predictions") for r in results: print(r.boxes) ``` ### Run with Inputs Execute your step and Valohai handles the downloads: ```shell vh execution run inference --adhoc --open-browser ``` On the execution page, the **Inputs** section shows: * Source locations (URLs, S3 paths, or datum links) * File previews for supported formats * Download status and file sizes ### Override Inputs at Runtime #### From the UI 1. Copy any execution using the "Copy" button 2. Modify inputs in the form: * Add URLs directly * Browse and select from your data catalog * Reference outputs from other executions 3. Create the new execution #### From the CLI Override default inputs without modifying `valohai.yaml`: ```shell # Single file vh execution run inference --adhoc --model=s3://my-bucket/model.onnx # Multiple files for one input vh execution run inference --adhoc --images=https://example.com/img1.jpg --images=https://example.com/img2.jpg ``` ### Input Sources Valohai accepts inputs from: * **Direct URLs**: Any publicly accessible HTTP/HTTPS endpoint * **Cloud storage**: S3, Azure Blob, GCS (with configured credentials) * **Data catalog**: Files uploaded to your project or outputs from executions * **Version control**: Files committed to your repository #### Connecting Executions Use outputs from one job as inputs to another: 1. Train a model → saves to `/valohai/outputs/` 2. Reference it with `datum://` in your next step 3. Valohai tracks the lineage automatically This creates traceable pipelines where you can see exactly which model version produced which predictions. #### 💬 Best Practices * Use descriptive input names that indicate the expected file type * Set sensible defaults in `valohai.yaml` for common use cases * Override inputs at runtime for experimentation * Check file existence before processing to handle optional inputs gracefully # Building Pipelines > 💡 **About this tutorial:** We use YOLOv8 as a practical example to demonstrate Valohai's features. You don't need computer vision knowledge—the patterns you learn here apply to any ML framework. This tutorial focuses on chaining multiple steps into a pipeline while ensuring proper versioning and tracking of your ML workflows. Connect your execution steps into automated workflows. Pipelines let you chain training, evaluation, and inference jobs with automatic data handoffs between them. ### Core Concepts **Nodes** — Individual execution steps in your pipeline\ **Edges** — Data connections between nodes (outputs → inputs, metrics → parameters, parameters → parameters) Your pipeline is fully defined in `valohai.yaml`. No separate orchestration needed. ### Define a Pipeline Add this pipeline definition to your `valohai.yaml`: ```yaml - pipeline: name: train-and-predict nodes: - name: train step: yolo type: execution - name: inference step: inference type: execution override: inputs: - name: model - name: images default: https://ultralytics.com/images/bus.jpg edges: - [train.output.best.onnx, inference.input.model] ``` #### What's happening here? 1. **Two nodes**: `train` runs your YOLO training step, `inference` runs predictions 2. **One edge**: Connects the trained model output to the inference input 3. **Override**: Sets default values for the inference step > 💡 *The edge syntax `[source.output.filename, target.input.name]` automatically passes files between steps.* ### Run Your Pipeline Execute the pipeline from your terminal: ```shell vh pipeline run train-and-predict --adhoc ``` The `--adhoc` flag runs using your local `valohai.yaml` without committing to Git first. ### Monitor Pipeline Progress Open the Valohai UI to track your pipeline: 1. **Click nodes** to view individual job details, logs, and metrics 2. **Check outputs** in the top-right corner for pipeline-level results When the training job completes, inference automatically starts with the trained model. ### Related Content * [Add conditional logic to pipelines](/pipelines/dynamic-conditions) * [Debug failed pipeline nodes](https://github.com/valohai/dokuhai/tree/main/how-to/debugging/pipeline-failures/README.md) # Examples & Templates *** ## Categories ### Computer Vision * [Object Detection with YOLO](/project-gallery/computer-vision/yolo-example) * [3D Object Detection with MMDetection3D](/project-gallery/computer-vision/mmdetection3d-on-valohai) * [Medical Imaging Segmentation with NVIDIA MONAI](/project-gallery/computer-vision/nvidia-monai-valohai) * [Object Detection with NVIDIA TAO Toolkit](/project-gallery/computer-vision/nvidia-tao-valohai) ### NLP & LLM * [Fine-Tuning Mistral 7B LLM](/project-gallery/nlp-and-llm/mistral-example) * [End to end RAG pipeline with Documentation](/project-gallery/nlp-and-llm/rag-doc-example) ### Audio & Speech * [Automatic Speech Recognition with NVIDIA NeMo](/project-gallery/audio-and-speech/nvidia-nemo-valohai) ### Data Engineering & ETL * [ML Pipeline with Snowpark & Snowflake](/project-gallery/data-engineering-and-etl/snowpark-example) * [Apache Spark examples on AWS & Valohai](/project-gallery/data-engineering-and-etl/spark-examples) ### Monitoring & Drift Detection * [Drift Detection with WhyLabs](/project-gallery/monitoring-and-drift-detection/drift-detection-example) *** Each example comes complete with: * An overview of the use case * Step and pipeline definitions * How to run via Valohai UI or CLI * Dataset references and expected outputs # Computer Vision # Object Detection with YOLO *** ### Overview This project demonstrates: * Training YOLOv5 and YOLOv5-Seg models * Validating trained models * Running YOLOv8 for inference and ONNX export * Using datasets stored in S3 with Valohai inputs and outputs *** ### Steps {% stepper %} {% step %} **Data Preparation** Store datasets in Amazon S3 and configure Valohai inputs for retrieval and processing in training. {% endstep %} {% step %} **Configuration in Valohai** Set up Valohai pipelines to automate the training, validation, and inference stages. {% endstep %} {% step %} **Training Execution** Execute training runs on YOLOv5 and YOLOv5-Seg with predefined configurations to start building robust models for object detection. {% endstep %} {% step %} **Validation Process** Conduct validation on trained models to compare predicted results against benchmark datasets, refining model performance accordingly. {% endstep %} {% step %} **Inference Implementation** Deploy YOLOv8 for inference tasks, focusing on optimizing speed and accuracy. Transition the model to ONNX format when necessary for enhanced compatibility. {% endstep %} {% endstepper %} *** ### GitHub repository The repository walks you through how to go through the above steps: {% embed url="" %} # 3D Object Detection with MMDetection3D *** ### Overview This project shows how to: * Prepare the KITTI dataset for 3D detection * Train a model using MMDetection3D * Evaluate the trained model * Automate the process using a Valohai pipeline *** ### Steps {% stepper %} {% step %} **Dataset Preparation** Set up the KITTI dataset for 3D object detection tasks. {% endstep %} {% step %} **Model Training** Use MMDetection3D to train a model on the prepared dataset. {% endstep %} {% step %} **Model Evaluation** Assess the performance of the trained model using appropriate metrics. {% endstep %} {% step %} **Pipeline Automation** Implement the automation of these steps using a Valohai pipeline. {% endstep %} {% endstepper %} *** ### GitHub Repository The repository walks you through how to go through the above steps: {% embed url="" %} # Medical Imaging Segmentation with NVIDIA MONAI *** ### Overview This example demonstrates how to: * Preprocess medical imaging data * Train a MONAI U-Net model * Evaluate segmentation performance * Run inference on new images *** ### Steps {% stepper %} {% step %} **Preprocess Data** Load and normalize medical imaging data. {% endstep %} {% step %} **Train Model** Initialize and train a MONAI U-Net model using your dataset. {% endstep %} {% step %} **Evaluate Performance** Assess the model's segmentation accuracy with test data. {% endstep %} {% step %} **Run Inference** Use the trained model to predict and segment new images. {% endstep %} {% endstepper %} *** ### GitHub repository The repository walks you through how to go through the above steps: {% embed url="" %} # Object Detection with NVIDIA TAO Toolkit *** ### Overview This project shows how to: * Preprocess and convert KITTI data into TFRecords * Train a DetectNet\_v2 model using TAO Toolkit * Evaluate and visualize model performance *** ### Steps {% stepper %} {% step %} **Data Preparation** Preprocess the KITTI dataset and convert it to TFRecords for compatibility with the training pipeline. {% endstep %} {% step %} **Environment Setup** Set up the TAO Toolkit environment to allow for seamless model training and evaluation. {% endstep %} {% step %} **Training Execution** Train the DetectNet\_v2 model using the TAO Toolkit to build a robust model for object detection. {% endstep %} {% step %} **Validation Process** Evaluate the trained model's performance on the validation dataset to ensure accuracy and reliability. {% endstep %} {% step %} **Visualization and Analysis** Visualize the model's predictions and results to assess performance and make necessary adjustments. {% endstep %} {% endstepper %} *** ### GitHub Repository The repository walks you through the steps above: {% embed url="" %} # NLP & LLM # Fine-Tuning Mistral 7B LLM *** ### Overview This example demonstrates how to: * Preprocess datasets for Mistral fine-tuning * Fine-tune a large language model using Valohai * Run inference with the fine-tuned checkpoint * Deploy an endpoint for inference *** ### Steps {% stepper %} {% step %} **Data Preprocessing** Fetch data from an S3 bucket and automatically store it in Valohai for preprocessing and tokenization using Mistral. {% endstep %} {% step %} **Model Fine-Tuning** A base-model is loaded and is fine-tuned using the "PEFT" method to better understand video gaming texts. {% endstep %} {% step %} **Model Inference** The fine-tuned LLM is used to generate texts based on a prompt. {% endstep %} {% endstepper %} *** ### GitHub Repository The repository walks you through how to go through the above steps: {% embed url="" %} # End to end RAG pipeline with Documentation ## Overview This example shows how to build your own RAG Doctor :woman\_health\_worker:: * Build an embeddings database from documentation CSVs * Query the database with LLMs * Serve the RAG model as an API endpoint ### Steps {% stepper %} {% step %} **Data Preparation** Compile CSV documentation and create an embeddings database, ensuring data is organized for efficient retrieval. {% endstep %} {% step %} **Embedding Database** Utilize Large Language Models (LLMs) to build and query the embeddings database effectively. {% endstep %} {% step %} **API Deployment** Configure and deploy the RAG model as an API endpoint to facilitate seamless interaction and integration. {% endstep %} {% endstepper %} ### GitHub Repository The repository walks you through how to go through the above steps: {% embed url="" %} # Audio & Speech # Automatic Speech Recognition with NVIDIA NeMo *** ## Overview This example demonstrates: * Preparing and preprocessing LibriSpeech data * Fine-tuning QuartzNet ASR models * Evaluating Word Error Rate (WER) *** ### Steps {% stepper %} {% step %} **Data Preparation** Prepare and preprocess the LibriSpeech dataset to ensure it is ready for training. Convert the data into the required format compatible with the QuartzNet model. {% endstep %} {% step %} **Environment Setup** Configure the environment for the QuartzNet ASR model to enable efficient fine-tuning and evaluation. Ensure all dependencies and tools are installed. {% endstep %} {% step %} **Model Fine-tuning** Fine-tune the QuartzNet ASR model on the prepared LibriSpeech data to enhance its transcription capabilities. {% endstep %} {% step %} **Evaluation Process** Assess the model's performance by calculating the Word Error Rate (WER) on a test dataset to determine its accuracy. {% endstep %} {% step %} **Results Analysis** Analyze the model's predictions and WER results to determine areas of improvement and refine the model if necessary. {% endstep %} {% endstepper %} *** ### GitHub repository The repository walks you through how to go through the above steps: {% embed url="" %} # Data Engineering & ETL # ML Pipeline with Snowpark & Snowflake *** ## Overview This example demonstrates: * Running Snowpark jobs from Valohai * Training and deploying models within Snowflake * Visualizing predictions in Snowpark Container Service *** ### Steps {% stepper %} {% step %} **Data Loading** Load the data from a CSV to a Snowflake table called `SOURCE_OF_TRUTH`. {% endstep %} {% step %} **Model Training** Use the loaded data to train and test a model to predict insurance charges. {% endstep %} {% step %} **Mock Streaming Data** Use the some of the data in `INCOMING_DATA_SOURCE` and insert it to `LANDING_TABLE` to mock the process of data coming in. {% endstep %} {% step %} **Running Inference** Use the model to run inference on incoming data and save results. {% endstep %} {% step %} **Deploy a Streamlit app to Snowpark Container Service** Trigger a long running service in Snowpark Container Service to host the Streamlit app. {% endstep %} {% endstepper %} *** ### GitHub Repository The repository walks you through the steps above: {% embed url="" %} # Apache Spark examples on AWS & Valohai *** ## Overview This example demonstrates: * Launching AWS EMR clusters from Valohai * Running Spark batch jobs remotely * Managing EMR configuration via Valohai parameters *** ### Steps {% stepper %} {% step %} **Setup AWS IAM users** Create a new IAM role to access EMR and S3. {% endstep %} {% step %} **Import and Run the examples on Valohai** Start with running the `run-debug-with-minimal-configuration` example step. {% endstep %} {% step %} **Running your own Spark applications** The `valohai.yaml` in the project includes examples for a minimal configuration as well as maximal example. The maximal example should cover most of the configuration options. {% endstep %} {% endstepper %} *** ### GitHub Repository The repository walks you through the steps above: {% embed url="" %} # Monitoring & Drift Detection # Drift Detection with WhyLabs *** ## Overview This example shows how to: * Detect input data drift using WhyLabs * Automate retraining pipelines upon drift detection * Require human approval for retraining actions {% hint style="info" %} The example shows how to do drift detection with WhyLabs, but the same approach can be used with other drift detection tools. {% endhint %} *** ### Steps {% stepper %} {% step %} **Setup a Training pipeline** Data is loaded from S3, preprocessed and saved as a Valohai dataset. Then using YOLO a model is trained and saved to Valohai {% endstep %} {% step %} **Setup a Drift Detection pipeline** Setup the pipeline for inference and log data to WhyLabs in order to detect drift. {% endstep %} {% step %} **Configure notifications for Approvals** As part of the drift detection pipeline you can include a notification and approval step to start a retraining of the model. {% endstep %} {% endstepper %} *** ### GitHub repository The repository walks you through the steps above: {% embed url="" %} # Migrate Your ML Jobs Your existing ML code can run on Valohai with minimal changes. This guide walks you through migrating your workflows in under 5 hours, keeping your code intact while gaining versioning, reproducibility, and scalability. This page covers an overview for the steps. More detailed instructions for each of them can be found in the other sections. > 💡 **Using an AI coding agent?** We have open-source [Agent Skills](/migration-strategy/migrate-with-ai-skills) for Claude Code, Copilot, Cursor, Zencoder, and others that can handle these migration steps for you. Install the skills, point your agent at your ML project, and let it generate the `valohai.yaml`, migrate your code, and run your first execution. ### Migration Timeline * **Step 1:** Define dependencies * **Step 2:** Create valohai.yaml (30 minutes) * **Step 3:** Add parameters and metrics (1-2 hours) * **Step 4:** Configure outputs (30 minutes) * **Step 5:** Update data access (1-2 hours) *** ### Step 1: Define Your Dependencies Identify the Python packages your code needs. You have two options: **Option A: Install at runtime** ```shell pip install -r requirements.txt conda install pandas=0.13.1 ``` **Option B: Use a Docker image with pre-installed dependencies** ```yaml image: tensorflow/tensorflow:2.6.0 ``` > 💡 **Tip:** Include version numbers to ensure reproducible environments across all executions. *** ### Step 2: Write Your valohai.yaml (30 minutes) Create a `valohai.yaml` file in your repository root. Start simple—your existing code runs as-is: ```yaml - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - pip install -r requirements.txt - python train_model.py ``` That's it. Your job now runs on Valohai without touching your Python code. *** ### Step 3: Add Parameters and Metrics (1-2 hours) #### Parameters If your code uses argparse or similar, this takes minutes. Define parameters in `valohai.yaml` : ```yaml - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - python train_model.py {parameters} parameters: - name: iterations type: integer default: 10 - name: learningrate type: float default: 0.01 ``` #### Metrics Log metrics by printing JSON from your Python code, e.g. ```python print( json.dumps( { "precision": 0.8125, "recall": 0.8667, "f1_score": 0.8387, }, ), ) ``` Valohai automatically captures and visualizes these metrics. *** ### Step 4: Save Output Artifacts (30 minutes) Save models, CSVs, or any outputs to `/valohai/outputs/` directory: ```python # Before: local save model.save("model.h5") # After: Valohai versioned output model.save("/valohai/outputs/model.h5") ``` Valohai automatically versions and uploads all outputs to your cloud storage. *** ### Step 5: Update Data Access (1-2 hours) Valohai handles all the complexity of cloud storage—authentication, access control, downloading, and caching. Your code just reads from local paths while Valohai manages everything behind the scenes. #### Define Your Data Sources Specify inputs in your YAML configuration: ```yaml - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - python train_model.py inputs: - name: images keep-directories: suffix default: - s3://mybucket/factories/images/*.png - azure://myblobstorage/factories/images/*.png - gs://mybucket/factories/images/*.png ``` #### Simplify Your Code Remove all cloud authentication and data management code: ```python # Before: Complex cloud operations s3_client = boto3.client("s3", aws_access_key_id=KEY, aws_secret_access_key=SECRET) download_from_s3("mybucket/factories/images/") handle_caching_logic() # After: Just read local files images = "/valohai/inputs/images/" # All files are already there, downloaded and cached by Valohai ``` Valohai automatically: * Authenticates with your cloud storage * Downloads files to the execution environment * Caches the input data for faster access * Works identically across AWS, Azure, GCP, and on-premises storage > 💡 **Advanced data management:** Use Valohai datasets and aliases to version your data without hardcoding storage paths. Reference data as `dataset://my-training-data` or models as `model://cats-v2` for better tracking and reproducibility. *** ### You're Done! 🎉 Your ML jobs now run on Valohai with: * Automatic versioning of code, data, and outputs * Experiment tracking and comparison * Scalability across cloud and on-premises infrastructure * No vendor lock-in—your code remains portable #### Next Steps * [Run your first execution](/executions/run-basic-execution) * [Build multi-step pipelines](/migration-strategy/migrate-pipeline-yaml) * [Schedule recurring training runs](/automation-overview/triggers) # Migrate with AI Coding Agent Skills Valohai is built on standard, well-understood components: YAML configuration, JSON for metrics, plain file I/O for data, and argparse for parameters. No proprietary SDKs, no framework entanglement, no magic. That design choice pays off in a big way when it comes to migration. Every step of a Valohai migration follows a clear, repeatable pattern: 1. Read your existing code 2. Add a small configuration or path change 3. Validate with `vh lint` Because these steps are predictable and pattern-based, they're a perfect fit for AI coding agents. We've built a set of open-source **Agent Skills** that teach your AI assistant how to migrate ML projects to Valohai, step by step. ## What Are Agent Skills? Agent Skills are knowledge modules that plug into AI coding agents. They give your agent deep context about Valohai's conventions, file paths, YAML syntax, and best practices — so it handles Valohai-specific patterns correctly instead of guessing. Each skill covers one part of the migration: | Skill | What It Does | | ---------------------------- | ------------------------------------------------------------------ | | `valohai-yaml-step` | Creates `valohai.yaml` step definitions from your existing scripts | | `valohai-migrate-parameters` | Converts hardcoded values to Valohai-managed parameters | | `valohai-migrate-metrics` | Adds experiment tracking via JSON output | | `valohai-migrate-data` | Migrates data loading to Valohai's input/output system | | `valohai-design-pipelines` | Designs multi-step pipelines from your workflow | | `valohai-project-run` | Sets up projects and runs executions via the CLI | ## Supported AI Coding Agents The skills work with all major AI coding agents: * **Claude Code** — Anthropic's CLI agent * **GitHub Copilot** — GitHub's coding assistant * **Cursor** — AI-first code editor * **Zencoder** — AI coding agent * **Windsurf** — Codeium's AI editor * **Gemini CLI** — Google's CLI agent * **Codex CLI** — OpenAI's CLI agent And 30+ other agents that follow the open [Agent Skills specification](https://agentskills.io). ## Install the Skills ```bash npx skills add valohai/valohai-skills --all ``` This auto-detects which agents you have installed and configures skills for each of them. To install for a specific agent: ```bash npx skills add valohai/valohai-skills --all --agent claude-code npx skills add valohai/valohai-skills --all --agent cursor npx skills add valohai/valohai-skills --all --agent copilot ``` ## Migrate Your Project Once installed, you can tell your agent to use the Valohai skills, to ensure they get loaded. Here's what a typical migration session looks like. ### Create Your Steps and Pipeline ``` "Look at my ML project and create a valohai.yaml with steps and a pipeline using the Valohai skills" ``` The agent scans your scripts, identifies frameworks and dependencies, picks appropriate Docker images, generates step definitions with parameters, inputs, outputs, and metrics — and wires it all together into a pipeline. It runs `vh lint` to validate the result. ### Run Your First Execution ``` "Set up a Valohai project and run my training step" ``` The agent walks you through `vh project create`, links your directory, and fires off your first execution with `vh execution run train-model --adhoc --watch`. ### Debug Failed Executions When something breaks, the skills also cover debugging: ``` "Check why the latest execution failed" ``` ``` "Why did execution #12 fail?" ``` The agent pulls the logs, identifies the error, and suggests a fix — whether it's a missing dependency, a wrong file path, or a YAML misconfiguration. ## Why This Works Valohai migrations are mechanical, not creative. Each step follows a well-defined pattern: * **Parameters** — Find hardcoded values, add argparse, declare in YAML * **Metrics** — Find where metrics are computed, print as JSON * **Data** — Find cloud SDK calls, replace with `/valohai/inputs/` paths * **Outputs** — Find save calls, redirect to `/valohai/outputs/` These patterns are the same whether you're migrating a PyTorch training script, a TensorFlow preprocessing pipeline, or a scikit-learn evaluation job. The agent applies the same rules each time, consistently. No SDK to learn. No framework-specific integration to configure. Your code stays portable — the configuration lives in `valohai.yaml` where it belongs. ## Next Steps * [Migration overview](/migration-strategy) — The full migration strategy * [Define your job types](/migration-strategy/migrate-job-yaml) — Manual guide to writing `valohai.yaml` * [Why migrate to Valohai?](/migration-strategy/why-migrate-to-valohai) — Business case and phased adoption # Define Your Job Types Migrating to Valohai is simpler than you think. You can run your existing code with just a YAML file — no rewrites, no vendor lock-in. This guide shows you how to define your ML jobs and run them on Valohai's infrastructure while keeping your code portable. > 💡 **Already using MLflow, W\&B, or other tools?** Keep using them. Valohai runs your code as-is. ## Step 0: Keep Your Code Portable The best code for Valohai is code that runs anywhere. Before defining your jobs: * **Remove vendor-specific decorators** (they'll still work, but why lock yourself in?) * **Use standard Python/R code** that runs locally * **Keep dependencies explicit** in `requirements.txt` or similar Your code stays yours — portable in and out of Valohai. > 💡 Want to use some other language than Python or R? No problem! Valohai jobs run inside Docker containers so you just need to provide a suitable image in your `valohai.yaml`. ## Step 1: Create valohai.yaml Add a `valohai.yaml` file to your project root. This tells Valohai what job types exist in your project. A minimal example: ```yaml - step: name: train-model image: docker.io/python:3.10 command: - pip install -r requirements.txt - python train.py ``` That's it. Your existing `train.py` runs unchanged. ### What's in a Step? * **name**: How you'll reference this job type (e.g., `preprocessing`, `training`, `evaluation`) * **image**: A Docker image with your base dependencies (Python, TensorFlow, etc.) * **command**: Exactly what you'd run locally > 💡 **No Docker experience?** Start with official images like `python:3.10` or `tensorflow/tensorflow:2.6.0` from [Docker Hub](https://hub.docker.com/). ## Step 2: Run Your First Execution An "execution" is just Valohai's term for running your job once. ### Quick Test with Local Code ```shell # Install Valohai CLI pip install valohai-cli # Login and create project vh login vh project create # Run your job vh execution run train-model --adhoc ``` Your code runs on Valohai's infrastructure, but behaves exactly like it does locally. ### Production Runs from Git Once you're happy, push to Git and run from there: ```shell git add valohai.yaml git commit -m "Add Valohai job definitions" git push # Fetch and run vh project fetch vh execution run train-model ``` ## Common Patterns ### Multiple Job Types Define all your workflow steps: ```yaml - step: name: preprocess image: docker.io/python:3.10 command: - python preprocess_data.py - step: name: train image: tensorflow/tensorflow:2.6.0 command: - python train_model.py - step: name: evaluate image: tensorflow/tensorflow:2.6.0 command: - python evaluate.py ``` ### Using Private Registries Have custom Docker images? After [connecting a private Docker registry to Valohai](/docker-in-valohai/private-docker-registries), you can use images from there in your executions: ```yaml - step: name: train image: myregistry.azurecr.io/ml-base:latest command: - python train.py ``` ### Non-pip Dependencies Valohai doesn't restrict what you can run inside your jobs. Instead of running `pip install` you can also install packages using for example `conda` or `apt-get install -y`. ```yaml - step: name: train image: continuumio/miniconda3 command: - conda install pytorch -c pytorch -y - python train.py ``` ## What About My Existing Tools? Keep using them. Valohai runs your code as-is: * **MLflow tracking?** Works * **Weights & Biases?** Works * **TensorBoard?** Works * **Custom logging?** Works You can migrate gradually — or not at all. Your choice. ## Next Steps 1. **Try one job** — Start with your simplest script 2. **Add parameters** — Make jobs configurable (covered in the next guide) 3. **Handle data** — Connect to your data sources 4. **Build pipelines** — Chain jobs together > 💡 **Want the full picture?** Check out [Valohai Academy](https://learn.valohai.academy/paths) for comprehensive tutorials. ## Quick Reference ### Minimal valohai.yaml ```yaml - step: name: my-job image: docker.io/python:3.10 command: - python my_script.py ``` ### CLI Commands ```shell vh login # One-time setup vh project create # New project vh execution run my-job --adhoc # Run with local code vh execution run my-job # Run from Git ``` ### Common Docker Images * `python:3.10` — Standard Python * `tensorflow/tensorflow:2.6.0` — TensorFlow CPU only * `tensorflow/tensorflow:2.6.0-gpu` — TensorFlow with GPU * `pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime` — PyTorch * `rocker/r-base:4.3.0` — R language *** **Bottom line:** If your code runs locally, it runs on Valohai. No rewrites needed. # Outputs: Save Models and Files When your execution ends, Valohai needs to know which files to keep. This guide shows how to save your trained models, datasets, and other files. > 💡 **Already saving files locally?** Just change the path to `/valohai/outputs/` and Valohai handles the rest. ## How Outputs Work 1. **Save to a special directory**: `/valohai/outputs/` 2. **Valohai uploads automatically** to your cloud storage (AWS S3, Azure Blob, Google Cloud Storage, MinIO, etc.) 3. **Every file is versioned**, no accidental overwrites The `/valohai/outputs/` directory already exists in every execution. Just save your files there. ## Update Your Code ### Before (Local) ```python # Saving locally model.save("model.h5") df.to_csv("results.csv") ``` ### After (Valohai) ```python # Save to Valohai outputs model.save("/valohai/outputs/model.h5") df.to_csv("/valohai/outputs/results.csv") ``` That's the only change needed. ## Common Patterns ### Save Multiple Files ```python # Models model.save("/valohai/outputs/model.h5") # Metrics with open("/valohai/outputs/metrics.json", "w") as f: json.dump(metrics, f) # Plots plt.savefig("/valohai/outputs/loss_curve.png") ``` ### Preserve Directory Structure Your folder structure is maintained: ```python # This structure... """ /valohai/outputs/ ├── models/ │ └── best_model.h5 ├── logs/ │ └── training.log └── visualizations/ ├── loss.png └── accuracy.png """ # ...stays exactly the same in cloud storage ``` ### Save During Training Don't wait until the end, save checkpoints as you go. Files that are marked as read only under the `/valohai/outputs/` directory will be uploaded immediately to the data store. ```python import os from stat import S_IREAD, S_IRGRP, S_IROTH # Some code for epoch in range(epochs): # Training code... if epoch % 10 == 0: filename = f"/valohai/outputs/checkpoint_epoch_{epoch}.h5" model.save(filename) os.chmod(filename, S_IREAD | S_IRGRP | S_IROTH) ```
Optional: Use the valohai-utils Python helper tool The `valohai-utils` library offers convenience methods: ```python import valohai # Generate output path output_path = valohai.outputs().path("model.h5") model.save(output_path) # Live logging valohai.outputs().live_upload("training.log") ``` This is optional, direct paths work just fine.
## File Types and Sizes Save any file type your code can create: * **Models**: `.h5`, `.pkl`, `.pt`, `.onnx`, `.joblib` * **Data**: `.csv`, `.parquet`, `.json`, `.npz` * **Images**: `.png`, `.jpg`, `.pdf` * **Archives**: `.tar`, `.zip`, `.gz` * **R files**: `.rdata`, `.rds` No size limits, save what you need. ## Using Aliases (Human-Friendly Names) Each job is isolated and versioned on its own. This means that if job #1 outputs a file called `model.h5` it will be versioned on its own, and when job #2 outputs a `model.h5` Valohai won't override the first file but create a separate file for it. We recommend looking into the [Valohai Aliases](/data/data-versioning/metadata-overview/aliases) if you're looking for friendly names for a specific version of a file: * `latest-model-project-b` * `production-model-project-a` Now you can reference this model as `datum://best-model` in future jobs instead of a long cloud storage URL. ## Where Do Outputs Go? Valohai automatically uploads your outputs to: * **Cloud**: Your configured AWS S3, Azure Blob Storage, Google Cloud Storage, OCI * **On-premise**: MinIO, NetApp, or other S3-compatible storage You don't need to write upload code. Save locally, Valohai handles the rest. ## Quick Reference ### Essential Pattern ```python # Just prepend /valohai/outputs/ to your existing save paths model.save("/valohai/outputs/model.h5") ``` ### Supported Storage * AWS S3 * Azure Blob Storage * Google Cloud Storage * MinIO * NetApp * Any S3-compatible storage ### Good to Know * Directory structure is preserved * Every execution's outputs are versioned separately * No file size or count limits * Files appear in UI immediately after execution *** # Parameters: Make Your Jobs Configurable Turn your hardcoded values into parameters that you can change without touching code. This guide shows how to expose configuration values, hyperparameters, and other settings to Valohai. > 💡 **Already using argparse?** You're 90% done. Just list your parameters in `valohai.yaml`. ### Why Use Parameters? * **Change values without code changes**, Adjust hyperparameters from the UI * **Track experiments automatically**, Sort and filter by parameter values * **Run parallel experiments**, Test multiple configurations at once * **Reproduce results**, Every execution saves its exact parameters ### Quick Example Turn hardcoded values into command-line arguments: #### Before ```python # Hardcoded values iterations = 10 learning_rate = 0.01 ``` #### After ```python # Parse from command line import argparse parser = argparse.ArgumentParser() parser.add_argument("--iterations", type=int, default=10) parser.add_argument("--learning_rate", type=float, default=0.01) args = parser.parse_args() iterations = args.iterations learning_rate = args.learning_rate ``` Then declare them in `valohai.yaml`: ```yaml - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - python train_model.py parameters: - name: iterations type: integer default: 10 - name: learning_rate type: float default: 0.01 ``` ### How Parameters Work 1. **Define in `valohai.yaml`**, List parameters with types and defaults 2. **Parse in your code**, Use argparse (or read from config files) 3. **Override at runtime**, Change values via UI, CLI, or API Valohai passes parameters as command-line arguments: ```shell python train_model.py --iterations=50 --learning_rate=0.001 ``` ### Common Patterns #### Multiple Parameter Types ```yaml parameters: - name: batch_size type: integer default: 32 - name: dropout_rate type: float default: 0.5 - name: optimizer type: string default: "adam" - name: use_augmentation type: flag default: false ``` > 💡 Flags are not exactly booleans. When a flag gets the value `false`, nothing is passed to the execution. If you want the flags to behave more like actual booleans, i.e. to pass `true` or `false` to the execution, you can use the `pass-true-as` / `pass-false-as` in your `valohai.yaml`: > > ``` > - name: verbose > type: flag > default: False > pass-true-as: --verbose=True > pass-false-as: --verbose=False > ``` #### Parse in Python ```python import argparse parser = argparse.ArgumentParser() parser.add_argument("--batch_size", type=int, default=32) parser.add_argument("--dropout_rate", type=float, default=0.5) parser.add_argument("--optimizer", type=str, default="adam") parser.add_argument("--use_augmentation", action="store_true") args = parser.parse_args() ``` #### Override When Running ```shell # Via CLI vh execution run train-model --batch_size=64 --optimizer=sgd # Parameters appear in the UI for easy changes # Also available via API for automation ``` ### Alternative: Read from Config Files Don't use command-line arguments? Valohai creates config files you can read: ```python import json # Read from JSON with open("/valohai/config/parameters.json") as f: params = json.load(f) learning_rate = params["learning_rate"] # Or read from YAML import yaml with open("/valohai/config/parameters.yaml") as f: params = yaml.safe_load(f) ``` These files are read-only and created automatically by Valohai so you can't change the parameter values in them while the job is running.
Optional: Use the valohai-utils Python helper tool The `valohai-utils` helper library offers a simpler syntax: ```python import valohai # No argparse needed batch_size = valohai.parameters("batch_size").value learning_rate = valohai.parameters("learning_rate").value ```
### Parameter Types Reference | Type | YAML | Python (argparse) | Example | | ------- | --------------- | --------------------- | ---------------- | | Integer | `type: integer` | `type=int` | `--epochs=100` | | Float | `type: float` | `type=float` | `--lr=0.001` | | String | `type: string` | `type=str` | `--model=resnet` | | Flag | `type: flag` | `action='store_true'` | `--augment` | ### What's Next? With parameters defined, you can: * **Compare experiments**, Filter executions by parameter values * **Run hyperparameter sweeps**, Test multiple values in parallel * **Create reproducible pipelines**, Lock in successful parameter sets *** # Inputs: Access Your Data Valohai handles secure access to your files in object storages like for example AWS S3, Azure Blob Storage, Google Cloud Storage, and more. > 💡 **Your data stays where it is.** Valohai downloads files only when needed and manages caching automatically. ### How Inputs Work 1. **Configure data store access** once (project or organization level) 2. **Define inputs** in `valohai.yaml` with cloud URLs 3. **Access files locally** at `/valohai/inputs/`, no download code needed Valohai handles authentication, parallel downloads, and caching behind the scenes. ### Quick Example #### Define in valohai.yaml ```yaml - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - python train_model.py inputs: - name: images default: - s3://mybucket/factories/images/*.png keep-directories: suffix - name: model default: datum://production-latest filename: model.pkl ``` The inputs will be downloaded to `/valohai/inputs/`. In the above case, you'll find your: * `images` in the directory `/valohai/inputs/images/` with the folder structure from your object data stores intact. * `model` will be downloaded to the directory `/valohai/inputs/model/` and the file will always be renamed to `model.pkl` #### Use in Python In Python you'll access these files like any other file, as they'll be available locally on the machine. ```python import os from PIL import Image # Path to downloaded model model_path = "/valohai/inputs/model/model.pkl" # Path to the directory where the images are downloaded images_directory = "/valohai/inputs/images/" # Loop through images in the directory for root, dirs, files in os.walk(images_directory): for filename in files: image_path = os.path.join(root, filename) image_name = os.path.basename(image_path) image = Image.open(image_path) image.load() ``` That's it. No boto3, no credentials, no download loops.
Optional: Use the valohai-utils Python helper tool The `valohai-utils` helper library offers a simpler syntax: ```python import valohai import os from PIL import Image # Path to downloaded model model_path = valohai.inputs("model").path() # Path to the directory where the images are downloaded images_directory = valohai.inputs("images").dir_path() # Loop through images in the directory for root, dirs, files in os.walk(images_directory): for filename in files: image_path = os.path.join(root, filename) image_name = os.path.basename(image_path) image = Image.open(image_path) image.load() ```
### Common Patterns #### Multiple Files with Wildcards ```yaml inputs: - name: images default: - s3://mybucket/train/images/*.jpg - s3://mybucket/train/images/*.png ``` All matching files download to `/valohai/inputs/images/`. #### Multiple Cloud Sources ```yaml inputs: - name: data default: - s3://aws-bucket/data/*.parquet - azure://mycontainer/data/*.parquet - gs://gcs-bucket/data/*.parquet ``` Mix and match storage providers in one input. All the files will be downloaded under `/valohai/inputs/data/`. > 💡 Files defined under the same input are downloaded to the same directory. If their names are not unique, they will override each other and only one of them will be available in the execution. #### Single File with Rename ```yaml inputs: - name: pretrained default: s3://models/bert-base.h5 filename: model.h5 # Always save as this name ``` Access at `/valohai/inputs/pretrained/model.h5`. #### Keep Directory Structure `keep-directories` is used to define what folder structure should Valohai use in the inputs folder. * **none:** (default) all files are downloaded to `/valohai/inputs/myinput` * **full:** keeps the full path from the storage root. For example `s3://special-bucket/foo/bar/**.jpg` could end up as `/valohai/inputs/myinput/foo/bar/dataset1/a.jpg` * **suffix:** keeps the suffix from the "wildcard root". For example `s3://special-bucket/foo/bar/*` the special-bucket/foo/bar/ would be removed, but any relative path after it would be kept, and you might end up with `/valohai/inputs/myinput/dataset1/a.jpg` ```yaml inputs: - name: dataset default: s3://bucket/project/**/*.json keep-directories: suffix # Preserves folder structure ``` ### Override Inputs at Runtime Default inputs are just starting points. Override them when running: ```shell # CLI vh execution run train-model --dataset=s3://different-bucket/experiment-data/*.csv # Or use the web UI to browse and select different files # Or pass different URLs via API ``` ### Quick Reference #### Define inputs in `valohai.yaml` ```yaml inputs: - name: mydata default: s3://bucket/*.csv # Access at: /valohai/inputs/mydata/file.csv ``` #### Use as local files Inputs are available under `/valohai/inputs/{input-name}/` : ```python import pandas as pd import glob # Files are already downloaded to /valohai/inputs/mydata/ csv_files = glob.glob("/valohai/inputs/mydata/*.csv") ``` #### Dynamic File Selection Don't hardcode paths in YAML. Pass them at runtime: ```shell vh execution run train --images=s3://bucket/client-xyz/images/*.jpg ``` #### Inputs can be overridden at runtime ```shell vh execution run train-model --dataset=s3://different-bucket/experiment-data/*.csv # Or in the UI / with the API ``` #### Options * `filename: newname.ext` — Rename single input file on download * `keep-directories: suffix` — Preserve folder structure *** # Metadata: Track and Compare Your Metrics Print your metrics as JSON and Valohai automatically tracks, visualizes, and lets you compare them across experiments. No special libraries required. Moreover, Valohai doesn't restrict what metrics you can track, let it be `accuracy`, `loss`, `f1_score`, `ROC`, `AUC`, etc. The metrics are not restricted to numeric values. > 💡 **Already logging metrics?** Just wrap them in JSON format and print. Valohai does the rest. ### How It Works 1. **Print JSON to stdout** — Any valid JSON key value pairs 2. **Valohai captures automatically** — Real-time collection during training 3. **View and compare** — Sort experiments, plot graphs, set alerts ```python import json # That's it - just print JSON print(json.dumps({"accuracy": 0.92, "loss": 0.08})) ``` ### Quick Example #### Training Loop with Metrics ```python import json for epoch in range(epochs): # Your training code... train_loss = train_one_epoch() val_acc = validate() # Track metrics - just print as JSON print( json.dumps( { "epoch": epoch, "train_loss": train_loss, "val_accuracy": val_acc, }, ), ) ``` Now you can: * **Sort experiments** by the metric values * **Plot timeseries graphs** across multiple runs * **Set early-stopping rules** based on the metrics * E.g. "stop execution when accuracy is over 0.9"
Optional: Use the valohai-utils Python helper tool The `valohai-utils` helper library offers a simpler syntax: ```python import valohai # Log incrementally for epoch in range(epochs): with valohai.logger() as log: log.log("accuracy", accuracy) log.log("loss", loss) ```
### Common Patterns #### Multiple Metrics ```python import json # Track everything you care about metrics = { "accuracy": 0.94, "precision": 0.92, "recall": 0.95, "f1_score": 0.93, "auc_roc": 0.97, "inference_time_ms": 23.5, } print(json.dumps(metrics)) ``` #### Metrics Over Time ```python import json for step in range(training_steps): # Training... if step % log_interval == 0: print( json.dumps( { "step": step, "loss": current_loss, "learning_rate": scheduler.get_lr()[0], "gradient_norm": grad_norm, }, ), ) ``` ### How to Use the Metrics in Valohai #### Execution Table See latest values for each execution: * Sort by any metric * Filter by thresholds * Compare at a glance #### Automatic Visualizations * Time series plots per execution * Compare multiple executions * Download data as CSV ### Best Practices #### Log Regularly ```python # Log every N batches, not just epochs if batch_idx % 100 == 0: print( json.dumps( { "batch": batch_idx, "loss": loss.item(), }, ), ) ``` #### Include Context ```python # Add helpful context print( json.dumps( { "epoch": epoch, "phase": "validation", # training/validation/test "accuracy": acc, "samples_seen": epoch * len(train_loader), }, ), ) ``` ### Quick Reference #### Basic Pattern ```python import json print(json.dumps({"metric_name": value})) ``` #### What Works * Any value * Multiple metrics in one print #### What to Track * Model performance (accuracy, loss, F1) * Training dynamics (learning rate, gradients) * Any metadata you want log from executions *** # Pipelines: Chain Your Jobs Connect your existing jobs into automated workflows. Define how data flows between steps and let Valohai handle the orchestration. > 💡 **Already have working steps?** You're ready to build pipelines. Just define how outputs connect to inputs. ### How Pipelines Work A pipeline is a recipe for connecting jobs: * **Nodes** = Your jobs (preprocessing, training, evaluation, etc.) * **Edges** = Data / information flow (e.g. which outputs become which inputs) When you run a pipeline, Valohai automatically: * Executes jobs in the right order * Passes outputs between steps into defined inputs * Handles parallel execution where possible * Tracks the complete lineage ### Quick Example Connect three existing steps into a pipeline: ```yaml - pipeline: name: ml-workflow nodes: - name: preprocess type: execution step: preprocess-data - name: train type: execution step: train-model - name: evaluate type: execution step: evaluate-model edges: # Connect outputs → inputs - [preprocess.outputs.*, train.inputs.dataset] - [train.outputs.model*, evaluate.inputs.model] - [preprocess.outputs.*test*, evaluate.inputs.test-data] ``` Run it: ```shell vh pipeline run ml-workflow --adhoc ``` > 💡 If you have pushed the valohai.yaml to Git and fetched the commit to your Valohai project, you can omit the `--adhoc` flag. ### Complete Example Let's build a real pipeline with three steps: #### 1. Define Your Steps (if not already done) ```yaml - step: name: preprocess-data image: python:3.10 command: - pip install -r requirements.txt - python data-preprocess.py - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - python train_model.py inputs: - name: train-data default: dataset://images/latest-train parameters: - name: epochs default: 100 - step: name: evaluate-model image: tensorflow/tensorflow:2.6.0 command: - python evaluate_model.py inputs: - name: test-data default: dataset://images/latest-test - name: model default: datum://production-latest ``` #### 2. Connect as Pipeline ```yaml - pipeline: name: training-pipeline nodes: - name: prep-node type: execution step: preprocess-data - name: train-node type: execution step: train-model override: parameters: - name: epochs value: 200 # Override default - name: eval-node type: execution step: evaluate-model edges: # Output wildcards → specific inputs - [prep-node.outputs.*, train-node.inputs.train-data] - [prep-node.outputs.*test*, eval-node.inputs.test-data] - [train-node.outputs.*.h5, eval-node.inputs.model] ```
Optional: Use the valohai-utils Python helper tool Using `valohai-utils`, define pipelines in Python: ```python # pipeline.py from valohai import Pipeline def main(config) -> Pipeline: # Create a pipeline called "utilspipeline". pipe = Pipeline(name="train-inference-pipeline", config=config) # Define the pipeline nodes. preprocess = pipe.execution("preprocess") train = pipe.execution("train-model") inference = pipe.execution("batch-inference") # Configure the pipeline, i.e. define the edges. preprocess.output("*").to(train.input("data")) preprocess.output("*.pkl").to(inference.input("model")) return pipe ``` Generate YAML: ```shell vh yaml pipeline pipeline.py ```
### Edge Patterns #### Basic Output → Input ```yaml # All outputs from A go to B's 'data' input - [nodeA.outputs.*, nodeB.inputs.data] ``` #### Wildcard Matching ```yaml # Only .pkl files go to model input - [train.outputs.*.pkl, inference.inputs.model] # Multiple patterns - [prep.outputs.*train*, train.inputs.train-data] - [prep.outputs.*val*, train.inputs.val-data] ``` #### Pass parameters and metrics between nodes In addition to defining the edges via outputs and inputs, they can be also used to pass parameters between nodes. ```yaml # Pass parameter values between nodes - [train-model.parameter.user-id, test-model.parameter.user-id] # Pass metrics into parameters - [train.metadata.best_lr, finetune.parameter.learning_rate] ``` #### Multiple Targets ```yaml edges: # One output goes to multiple nodes - [preprocess.outputs.*, train.inputs.data] - [preprocess.outputs.*, evaluate.inputs.data] ``` ### Advanced Features #### Conditional Execution You can define specific conditions for pipeline nodes. **When**: Actions trigger when certain events occur during pipeline execution. The available options include: * `node-starting`: When a node is about to start. * `node-complete`: When a node successfully completes. * `node-error`: When a node encounters an error. **If Condition**: The condition to trigger the action can be based on either metric or a parameter value. **Then**: Depending on the condition being met, you can take one of the following actions: * `stop-pipeline`: Halts the entire pipeline. * `require-approval`: Pauses the pipeline until a user manually approves the previous results. ```yaml - pipeline: name: conditional-workflow nodes: - name: train type: execution step: train-model actions: - when: node-complete if: metadata.foo >= 0.8 then: stop-pipeline - name: test-model type: execution step: test-model actions: - when: node-starting then: require-approval ``` #### Parallel Execution In the example below the nodes `train-model-a` and `train-model-b` will run in parallel. The `ensemble` node will only start once both of them are finished. ```yaml nodes: - name: train-model-a type: execution step: train - name: train-model-b type: execution step: train - name: ensemble type: execution step: combine-models edges: # Both training nodes run in parallel - [train-model-a.outputs.*, ensemble.inputs.model-a] - [train-model-b.outputs.*, ensemble.inputs.model-b] ``` It is also possible to run Task nodes inside pipelines: ```yaml nodes: # Other nodes omitted - name: train type: task step: Train model (MNIST) on-error: stop-all override: inputs: - name: training-set-images - name: training-set-labels - name: test-set-images - name: test-set-labels edges: # Other edges omitted - [preprocess.output.*train-images*, train.input.training-set-images] - [preprocess.output.*train-labels*, train.input.training-set-labels] - [preprocess.output.*test-images*, train.input.test-set-images] - [preprocess.output.*test-labels*, train.input.test-set-labels] ``` #### Deployments In addition to execution and Task nodes, it is possible to create deployments from pipelines. 
nodes:
  # Other nodes omitted
  - name: deploy
    type: deployment
    deployment: deployment-test
    actions:
      - when: node-starting
        then: require-approval
    endpoints:
      - predict-digit

edges:
    # Other edges omitted
    - [find-best-model.output.model*, deploy.file.predict-digit.model]
It is possible to create pipeline nodes after a deployment node. This can be used to for example check the endpoint once it has been created or clean old endpoints within the pipeline. ```yaml edges: - [deploy.deployment.id, cleanup.parameters.deployment_id] - [deploy.deployment.version_id, cleanup.parameters.version_id] ``` ### Running Pipelines #### From CLI ```shell # Run with local code vh pipeline run training-pipeline --adhoc # Run from Git vh pipeline run training-pipeline ``` ### Quick Reference #### Minimal Pipeline ```yaml - pipeline: name: my-pipeline nodes: - name: step1 type: execution step: my-step-1 - name: step2 type: execution step: my-step-2 edges: - [step1.outputs.*, step2.inputs.data] ``` #### Edge Syntax Sources: * `node-name.output.*` — All outputs * `node-name.output.*.csv` — Only CSV files * `node-name.output.name*` — Starts with "name" * `node-name.metadata.accuracy` — Metadata value * `node-name.parameter.learning_rate` — Parameter value * `deploy.deployment.id` / `deploy.deployment.version_id` — Deployment / deployment version id Targets: * `node-name.input.input-name` — Any input available on the node * `node-name.parameter.learning_rate` — Parameter value * `deploy.file.predict-digit.model` — File for deployment nodes #### Node Types * `execution` — Run a step * `task` — Run parameter sweep * `deployment` — Create endpoint *** **Bottom line:** If your steps work individually, connecting them into a pipeline takes just a few lines of YAML. # Why Migrate to Valohai? **TL;DR:** Migrate to Valohai in phases starting with one high-impact project. Most organizations see 50% faster experimentation within weeks and achieve full adoption in 3-6 months. Zero vendor lock-in and your code stays yours. ## Why Teams Migrate to Valohai Your ML teams are likely battling at least one of these challenges: ### Infrastructure Overhead Is Killing Productivity * Data scientists spending 40% of time on DevOps tasks * "Works on my machine" blocking production deployments * Each team maintaining their own MLOps stack * Cloud costs spiraling due to idle resources ### Experiments Aren't Reproducible * Can't recreate that model from 6 months ago * Missing dependency versions breaking reruns * Audit requirements forcing manual documentation * Teams re-running identical experiments unknowingly ### Scaling Hits a Wall * Single-machine limits blocking larger experiments * Manual provisioning creating bottlenecks * No resource sharing between teams * Every new project starts from zero ### Teams Can't Collaborate Effectively * Models trapped in individual laptops * No standard deployment pipeline * Knowledge lost when people leave * Integration nightmares between team tools ## The Strategic Migration Path
Phase 1: Prove Value Fast (Weeks 1-4) Start with one team's biggest pain point. Get a working project in hours, not weeks. **Choose Your Pilot:** * High-impact project with clear metrics * 3-5 person team eager for better tools * Existing code that runs today **Success Looks Like:** * First execution running within 2 hours * 50% reduction in experiment setup time * Zero infrastructure debugging by data scientists * Team asking "can we migrate more projects?" **Deliverables:** * Working project with automated tracking * Before/after metrics showing time saved * Initial cost analysis * Team testimonial for internal buy-in
Phase 2: Standardize Your Workflows (Months 2-3) Scale what works. Move from individual wins to team transformation. **Expand Scope:** * Full project lifecycle (data → training → deployment) * Advanced features (pipelines, hyperparameter optimization) * Team-wide best practices **Success Metrics:** * All team projects using Valohai * 70% reduction in "plumbing" work * Models deploying in hours, not weeks * 30% infrastructure cost reduction **Deliverables:** * Reusable templates for common workflows * Automated CI/CD pipelines * Team playbook documented * Quarterly cost savings report
Phase 3: Scale Across the Organization (Months 4-6) Transform ML from cost center to innovation engine. **Organization-Wide Impact:** * All ML teams onboarded * Cross-team model sharing * Enterprise governance active * Executive visibility enabled **Success Metrics:** * 60% faster model delivery to production * 90% experiment reproducibility * Full compliance audit trail * 50% reduction in total ML infrastructure costs
## Common Migration Scenarios
"We Need to Replace Our Existing Platform" Your current platform promised the world but delivered complexity. **Migration Strategy:** * Run Valohai parallel to existing platform * Migrate your most painful workflows first * Compare metrics side-by-side * Sunset old platform once value proven **Timeline:** 2-3 months for complete transition
"We're Drowning in Tool Sprawl" Different teams, different tools, zero standardization. **Migration Strategy:** * Map current tool landscape and overlaps * Identify common workflows across teams * Replace tool-by-tool with unified platform * Calculate maintenance hours saved **Timeline:** 3-4 months to consolidate
"We Can't Get Models to Production" Research breakthroughs dying in deployment purgatory. **Migration Strategy:** * Start with research workflow * Add deployment in same platform * No handoffs between teams * Measure time-to-production improvement **Timeline:** 1-2 months for first production model
"We're Starting Our ML Journey" Green field opportunity to build it right. **Migration Strategy:** * Implement best practices from day one * Avoid accumulating technical debt * Scale gradually as team grows * Learn from others' mistakes **Timeline:** Immediate value, scales with growth
## Addressing Executive Concerns
"What About Lock-In?" **Reality Check:** * Your code runs unchanged—no Valohai SDK required * Standard Docker, Git, and YAML throughout * Full API access to export everything * Migration is so pattern-based that [AI coding agents can do it](/migration-strategy/migrate-with-ai-skills) Your migration path out is as easy as your path in.
"How Do We Justify the Investment?" **Measurable Returns:** **Month 1:** * 50% reduction in experiment setup time * Infrastructure debugging eliminated * First cost optimizations visible **Months 2-3:** * 30-50% infrastructure cost reduction * 2x faster development cycles * Tool consolidation savings **Months 4+:** * 60% faster time-to-market * 5x improvement in asset reuse * Complete compliance coverage
"Will Our Teams Actually Adopt This?" **Why Teams Love Valohai:** * Keep using familiar tools (Python, notebooks, Git) * No new languages or frameworks * Less time on plumbing, more on ML * [AI coding agent skills](/migration-strategy/migrate-with-ai-skills) handle the migration steps automatically * Success spreads organically Early adopters become internal champions.
"What If Something Goes Wrong?" **Risk Mitigation Built In:** * Enterprise SLAs with 24/7 support (by separate agreement) * Gradual migration reduces risk * Your code remains portable * Professional services available
## Making the Decision ### Key Decision Factors **Cost of Delay:** Every month without proper ML infrastructure costs you: * Lost innovation opportunities * Accumulated technical debt * Growing compliance risk * Widening competitive gap **Why Valohai Over Alternatives:** * Only true bring-your-own-code platform * Managed service = zero maintenance * Proven with enterprises like yours * Scales from startup to enterprise ### Next Steps 1. **Technical Validation** * Review the [technical migration guide](/migration-strategy) * Run proof-of-concept with pilot team * Benchmark against current setup 2. **Business Case Development** * Calculate current infrastructure + maintenance costs * Estimate productivity gains from metrics above * Factor in compliance and risk reduction 3. **Get Expert Input** * Schedule migration assessment with Valohai * Connect with similar organizations who've migrated * Review security and compliance requirements *** **Ready to transform your ML operations?** Start with our [technical migration guide](/migration-strategy) or contact our team at for a migration assessment. # Docker in Valohai Valohai uses Docker images to define your runtime environment. This means you can run any code: Python, R, Julia, C++, libraries, or custom binaries, as long as it runs inside a container. {% hint style="info" %} Depending on the environment, you might leverage a different container platform like Singularity. {% endhint %} ## Do I need a container? **Yes, but you don't need to build one yourself.** Every execution in Valohai requires a Docker image. The good news: you can start with public images that already exist. **Common starting points:** ```yaml # TensorFlow image: tensorflow/tensorflow:2.13.0-gpu ``` ```yaml # PyTorch image: pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime ``` ```yaml # Python (any framework) image: python:3.11 ``` ```yaml # R image: r-base:4.3.0 ``` You reference an image by its full name. Valohai downloads it from the registry (Docker Hub, AWS ECR, etc.), caches it on the worker machine, and uses it for your execution. ## What goes inside a Docker image? A Docker image should contain your **runtime dependencies**: * System libraries (CUDA, OpenCV, FFmpeg) * Python/R packages (TensorFlow, PyTorch, scikit-learn) **Don't include:** * Your ML code (comes from Git) * Your data (comes from data stores) * Secrets or credentials (use environment variables) ## Do I need to rebuild every time I do changes to my code? No. When experimenting, you can easily install additional packages as a part of your step. ### Install additional packages without building a new Docker image Install packages directly as a part of your step: ```yaml command: - pip install transformers==4.30.0 - python train.py ``` This is **fast for iteration**. You change your `step.command` in `valohai.yaml` and run. No Docker build needed. ### When to build a custom image Build a custom image when: * **You use the same dependencies often:** baking them into an image saves time on every execution, as you don't have the wait for the installs to complete. * **You're building pipelines:** avoids reinstalling packages at each pipeline node * **You're running production workloads:** faster startup, better versioning, more reproducible * **You need system-level dependencies** e.g., CUDA libraries, compiled binaries, specific OS packages **Key point:** Build images for **speed and reproducibility**, not just because you added one package. ## Does it need Valohai-specific components? **No.** Your Docker image should be generic and reusable. Valohai doesn't require anything special in your image. The platform handles data mounting, code injection, logging, etc. to the container based on your custom image. Your image just needs to run your code. That's it. ## Where can I host Docker images? You can use any Docker registry: **Public registries:** * Docker Hub (default for most public images) * GitHub Container Registry **Private registries:** * AWS Elastic Container Registry (ECR) * Google Cloud Artifact Registry * Azure Container Registry * JFrog Artifactory * Self-hosted registries Valohai supports authentication for all major private registries. See [Private Docker Registries](/docker-in-valohai/private-docker-registries) for setup instructions. ## Common questions ### Can I use root inside the container? Yes. By default, Valohai runs containers as root, but your custom Docker image can override this using the `USER` directive in your Dockerfile. ### What about custom ENTRYPOINT or CMD? Valohai explicitly overrides `ENTRYPOINT` and replaces `CMD` with your step commands. This is by design, only your step commands execute inside the container. If you need specific startup behavior, include it in your step commands or use an init script. ## Next steps **Starting out?** Use a public image like `python:3.11` or `tensorflow/tensorflow:2.13.0-gpu`. Install packages in your script as needed. **Ready to optimize?** Learn how to [build custom images](/docker-in-valohai/building-images) and follow our [best practices](/docker-in-valohai/image-best-practices). **Using private images?** Set up authentication with your [private registry](/docker-in-valohai/private-docker-registries). # Build Custom Images Build custom Docker images when you need speed, reproducibility, or system-level dependencies. ## When to build custom images **Build a custom image when:** * You use the same dependencies across many executions * You're building production pipelines * You need system libraries (CUDA, OpenCV, FFmpeg) * Startup time matters (avoid reinstalling packages) * You want reproducible environments with pinned versions **Don't build yet if:** * You're still experimenting and dependencies change frequently * Installing packages in your script is fast enough * You only need pure Python packages Start simple, optimize later. ## Two ways to build We recommend building Docker images with your existing workflow, locally or through CI/CD. If you need to build images directly in Valohai, add the [**valohai-toolkit**](/reusable-step-libraries/build-your-own-library/docker-image-builder) library to your organization. It provides a pre-configured step definition for building and pushing images to your registry. ### Option 1: Build locally (traditional) Build images on your machine and push to a registry. **Pros:** * Full control over the build process * Fast iteration during development * Works with any Docker tooling **Cons:** * Requires Docker installed locally * Need to manage registry authentication * Manual process (build, tag, push) ### Option 2: Build on Valohai Use the [Docker Image Builder](/reusable-step-libraries/build-your-own-library/docker-image-builder) library step. **Pros:** * No Docker installation needed * Handles authentication to private registry using organization credentials * Reproducible builds tracked in Valohai * Perfect for teams without Docker build experience **Cons:** * Requires `--privileged` environment (contact Valohai support) * Slightly slower than local builds for rapid iteration *** ## Building locally ### 1. Write your Dockerfile Start with a base image and add your dependencies: ```dockerfile FROM python:3.11-slim # Install system dependencies RUN apt-get update && apt-get install -y \ git \ libgl1-mesa-glx \ && rm -rf /var/lib/apt/lists/* # Copy and install Python packages COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Set working directory WORKDIR /workspace ``` **Don't include:** * Your code (comes from Git) * Your data (comes from data stores) * Secrets or tokens ### 2. Build the image ```shell docker build -t myusername/ml-image:v1.0 . ``` Use a descriptive tag that includes a version number. Avoid `latest` for production. ### 3. Test locally Run a container to verify everything works: ```shell docker run --rm -it myusername/ml-image:v1.0 bash ``` Inside the container: ```shell python --version pip list # Test imports python -c "import tensorflow; print(tensorflow.__version__)" ``` ### 4. Push to your registry **Docker Hub:** ```shell docker login docker push myusername/ml-image:v1.0 ``` **AWS ECR:** ```shell aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin .dkr.ecr.us-east-1.amazonaws.com docker tag myusername/ml-image:v1.0 .dkr.ecr.us-east-1.amazonaws.com/ml-image:v1.0 docker push .dkr.ecr.us-east-1.amazonaws.com/ml-image:v1.0 ``` **GCP Artifact Registry:** ```shell gcloud auth configure-docker us-central1-docker.pkg.dev docker tag myusername/ml-image:v1.0 us-central1-docker.pkg.dev//ml-images/ml-image:v1.0 docker push us-central1-docker.pkg.dev//ml-images/ml-image:v1.0 ``` **Azure Container Registry:** ```shell az acr login --name myregistry docker tag myusername/ml-image:v1.0 myregistry.azurecr.io/ml-image:v1.0 docker push myregistry.azurecr.io/ml-image:v1.0 ``` ### 5. Use in Valohai If using a private registry, configure [authentication](/docker-in-valohai/private-docker-registries) first. Then reference your image in `valohai.yaml`: ```yaml - step: name: train image: myusername/ml-image:v1.0 command: - python train.py ``` *** ## Building on Valohai ### 1. Write your Dockerfile Same as local builds. Create a Dockerfile with your dependencies: ```dockerfile FROM python:3.11-slim COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt WORKDIR /workspace ``` ### 2. Add the Docker Image Builder library See [Docker Image Builder](/reusable-step-libraries/build-your-own-library/docker-image-builder) for setup instructions. ### 3. Create a build execution 1. Open your project 2. Click **Create Execution** 3. Select the appropriate builder step: * `docker-image-aws` for AWS ECR * `docker-image-gcp` for GCP Artifact Registry * `docker-image-dockerhub` for Docker Hub 4. Provide your Dockerfile (as input or parameter) 5. Set parameters: * **repository**: Your image name * **docker-tag**: Version tag 6. Set registry environment variables 7. Click **Create Execution** ### 4. Use the built image After the build completes, reference it in your project: ```yaml - step: name: train image: myregistry.azurecr.io/ml-image:v1.0 command: - python train.py ``` *** ## Common patterns ### GPU workloads Start with NVIDIA's official CUDA images: ```dockerfile FROM nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04 RUN apt-get update && apt-get install -y python3 python3-pip RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 ``` ### Enable GPU access in your images Regardless of your base image, it's good practice to add these environment variables to your Dockerfile so Valohai executions can use GPUs: ```dockerfile ENV NVIDIA_VISIBLE_DEVICES=all ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility ``` Without these, your execution won't detect available GPUs even when running on GPU-enabled machines. ### R workloads ```dockerfile FROM r-base:4.3.0 RUN R -e "install.packages(c('tidyverse', 'caret', 'randomForest'), repos='https://cran.rstudio.com/')" WORKDIR /workspace ``` ### Julia workloads ```dockerfile FROM julia:1.9 RUN julia -e 'using Pkg; Pkg.add(["DataFrames", "CSV", "Plots"])' WORKDIR /workspace ``` *** ## Troubleshooting ### Build fails with "no space left on device" Your Docker daemon is out of disk space. Clean up: ```shell docker system prune -a ``` ### Image is too large **Check layer sizes:** ```shell docker history myimage:v1.0 ``` **Reduce size:** * Use `-slim` or Alpine base images * Clean up in the same `RUN` command * Use multi-stage builds * Remove unnecessary files (docs, tests) ### Push fails with authentication error **Docker Hub:** ```shell docker login ``` **Other registries:** Check your registry's authentication documentation. For AWS ECR, GCP, and Azure, see [Private Docker Registries](/docker-in-valohai/private-docker-registries). *** ## Next steps **Optimize your images:** Follow [Best Practices](/docker-in-valohai/image-best-practices) for faster builds and smaller images. **Use private registries:** Set up [authentication](/docker-in-valohai/private-docker-registries) for your organization. # Image Best Practices Follow these guidelines to build efficient, reproducible Docker images for Valohai. ## Use specific version tags Always pin versions for reproducibility. **Good:** ```dockerfile FROM python:3.11.4 RUN pip install tensorflow==2.13.0 ``` **Avoid:** ```dockerfile FROM python:latest RUN pip install tensorflow ``` Why? `latest` tags change over time. Six months from now, `latest` might be Python 3.13 with breaking changes. Pinned versions ensure your executions stay reproducible. ## Start with minimal base images Smaller images download faster and use less disk space. **Good choices:** * `python:3.11-slim` (smaller than `python:3.11`) * `nvidia/cuda:12.1.0-base-ubuntu22.04` (only CUDA runtime, not full SDK) * `alpine` variants when compatible **Compare sizes:** * `python:3.11` → 1.0 GB * `python:3.11-slim` → 130 MB For GPU workloads, use NVIDIA's official base images to ensure CUDA compatibility. ## Leverage Docker layer caching Docker builds images in layers. Each instruction in your Dockerfile creates a layer that can be cached. **Order matters:** ```dockerfile FROM python:3.11-slim # 1. Install system dependencies (changes rarely) RUN apt-get update && apt-get install -y \ git \ libgl1-mesa-glx \ && rm -rf /var/lib/apt/lists/* # 2. Copy requirements first (changes occasionally) COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # 3. Copy code last (changes frequently) COPY . /workspace WORKDIR /workspace ``` When you change your code, only the last layer rebuilds. Requirements and system packages stay cached. ## Don't include code or data in the image Your Docker image should only contain the environment, not your code or data. **Your image:** * Python runtime * System libraries * Python packages **Not your image:** * Training scripts (comes from Git) * Datasets (comes from data stores) * Model files (generated during execution) Why? Separating code from environment makes images reusable and keeps them small. ## Pin Python package versions Use a `requirements.txt` with exact versions: ``` tensorflow==2.13.0 transformers==4.30.2 numpy==1.24.3 ``` Avoid version ranges like `tensorflow>=2.0` in production images. Ranges are fine for experimentation, but pinned versions ensure reproducibility. ## Cache control in Valohai Valohai caches Docker images on worker machines by default. This means the first execution downloads the image, and subsequent executions reuse the cached version. ### Force a fresh image pull If you've updated an image in your registry (using the same tag), force Valohai to pull the latest version: Set the environment variable **`VH_NO_IMAGE_CACHE=1`** on your execution. This ignores the cached image and pulls fresh from the registry. ### Clear all caches To clear both image and data caches from a worker machine: Set **`VH_CLEAN=1`** on your execution. This forcibly removes all Docker images and cached data before and after execution. Use sparingly—it adds significant time. ### When to use cache controls * `VH_NO_IMAGE_CACHE=1` → You pushed a new version with the same tag (not recommended, but sometimes necessary) * `VH_CLEAN=1` → Debugging disk space issues or testing fresh environments For normal workflows, let Valohai's default caching work. It's fast and efficient. ## Speed up image downloads with a pull-through cache If you frequently build or pull large Docker images, a pull-through cache can significantly reduce download times. ### When to use this Consider a pull-through cache if: * You build or update Docker images frequently * Download speeds are slow or you hit timeouts * You want to reduce bandwidth costs ### How it works Valohai sets up a caching server in your VPC. When workers pull images, they first check the cache. If the image exists, it's served locally (fast). If not, it's fetched once and cached for future use. ### Setup This requires a dedicated machine in your VPC and network configuration to route traffic through the cache. Contact `support@valohai.com` to set up a pull-through cache for your organization. ## Custom container runtime options Valohai controls the `docker run` command and its arguments. This ensures executions work consistently across environments. **You cannot:** * Pass custom `docker run` flags * Override the entrypoint Valohai sets * Modify container networking or volume mounts **You can:** * Use any Docker image * Pass parameters to your code * Set environment variables * See [system environment variables](/executions/system-environment-variables) * Mount data from your data stores This design keeps infrastructure management outside your containers, so you focus on code, not configuration. If you have a use case requiring custom `docker run` arguments, contact our support team to discuss alternatives. ## Building images without Docker installed You don't need Docker installed locally to build images. Use the [Docker Image Builder](/reusable-step-libraries/build-your-own-library/docker-image-builder) from Valohai's [Reusable Step Libraries.](/reusable-step-libraries) This library step: * Takes your Dockerfile as input * Builds the image on Valohai infrastructure * Pushes to your registry Perfect for teams without Docker experience or for CI/CD pipelines. ## Summary **Fastest path:** * Start with `python:3.11-slim` or similar * Pin all versions * Install packages in your code while iterating * Build a custom image once dependencies stabilize **Production-ready:** * Use multi-stage builds * Clean up in the same layer * Leverage layer caching * Never include code or data in the image **Troubleshooting:** * When launching an execution * Use `VH_NO_IMAGE_CACHE=1` to pull fresh images * Use `VH_CLEAN=1` to clear all caches (rarely needed) * Contact support for pull-through cache setup # Private Docker Registries Valohai supports pulling images from private Docker registries. Once configured, your organization's projects can use private images just like public ones. ## Supported registries Valohai supports authentication with: * [AWS Elastic Container Registry (ECR)](/docker-in-valohai/private-docker-registries/aws-ecr) * [Docker Hub](/docker-in-valohai/private-docker-registries/docker-hub) * [Google Cloud Artifact Registry](/docker-in-valohai/private-docker-registries/gcp) * [Azure Container Registry](/docker-in-valohai/private-docker-registries/acr) * [JFrog Artifactory](/docker-in-valohai/private-docker-registries/jfrog) * Any registry using standard Docker authentication ## How it works 1. **Organization admins** configure registry credentials in Valohai 2. **Match patterns** determine which images use which credentials 3. **All projects** in your organization can pull matching private images 4. **Workers** automatically authenticate when pulling images ## Who can configure registries? Only **organization admins** can add and manage private registry credentials. Personal projects cannot access organization-level private registries. If you need private images, make sure your project is owned by an organization. ## Setup process Each registry type has specific authentication requirements: * **AWS ECR**: IAM user or instance role * **Docker Hub**: Access token (not password) * **GCP Artifact Registry**: Service account with Reader role * **Azure Container Registry**: Service principal * **JFrog**: Access token Follow the guide for your registry to get started. ## Using private images After configuration, reference private images in your `valohai.yaml` using their full name: ```yaml - step: name: train-model image: myregistry.azurecr.io/ml-training:v2.0 command: - python train.py ``` No changes to your code needed, Valohai handles authentication automatically. ## Troubleshooting image pulls ### Error: "unauthorized" or "access denied" **Check:** 1. Registry credentials are correctly configured in organization settings 2. Match pattern covers your image (e.g., `myregistry.azurecr.io/*`) 3. Project is owned by an organization (personal projects can't access private registries) 4. Credentials have read permissions for the repository ### Image pull takes too long or times out If pulls are consistently slow: * Your registry might be rate-limiting requests * Network connectivity between workers and registry might be constrained * Consider setting up a [pull-through cache](/docker-in-valohai/image-best-practices#speed-up-image-downloads-with-a-pull-through-cache) ### Need to verify which image was used? Check the execution logs. Valohai logs the full image name and digest when pulling: ``` Pulling image: myregistry.azurecr.io/ml-training:v2.0 Image digest: sha256:abc123... ``` # AWS ECR Configure Valohai to pull private images from AWS Elastic Container Registry. > Private registries are restricted to organization projects. Personal projects don't have access to private repositories. ## Authentication options Choose one: 1. **IAM User** (recommended for multi-cloud setups) 2. **Instance Role** (recommended for AWS-only deployments) *** ## Option 1: IAM User with ECR permissions ### Create the IAM user 1. Log in to your AWS Management Console 2. Create a new User with **Programmatic access** 3. Create a policy named `valohai-ecr-policy` with the JSON below Replace ``, ``, and `` with your values: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ReadECRRepository", "Effect": "Allow", "Action": [ "ecr:DescribeImageScanFindings", "ecr:GetLifecyclePolicyPreview", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:DescribeImages", "ecr:DescribeRepositories", "ecr:ListTagsForResource", "ecr:ListImages", "ecr:BatchCheckLayerAvailability", "ecr:GetRepositoryPolicy", "ecr:GetLifecyclePolicy" ], "Resource": "arn:aws:ecr:::repository/" }, { "Sid": "GetECRAuthToken", "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" } ] } ``` 4. Attach the policy to your new IAM user 5. Save the **Access Key ID** and **Secret Access Key** ### Add credentials to Valohai 1. Navigate to `Hi, ` (top-right menu) → `Manage ` 2. Go to **Registries** under organization controls 3. Click **Add a new entry** 4. Configure: * **Match pattern**: `.dkr.ecr..amazonaws.com/*` * Example: `123456789012.dkr.ecr.us-east-1.amazonaws.com/*` * Or narrow it: `123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo/*` * **Registry type**: AWS ECR (IAM credentials) * **Access Key ID**: Your IAM user's access key * **Secret Access Key**: Your IAM user's secret key * **Region**: Your ECR region (e.g., `us-east-1`) 5. Save *** ## Option 2: Instance Role Instead of creating a separate IAM user, attach ECR permissions directly to your worker role. ### Update the worker role Add the same policy from Option 1 to your `ValohaiWorkerRole` (or equivalent worker IAM role). If you need to restrict access to specific environments, create a dedicated role and contact Valohai support (`support@valohai.com`) for configuration. ### Add credentials to Valohai 1. Navigate to `Hi, ` (top-right menu) → `Manage ` 2. Go to **Registries** under organization controls 3. Click **Add a new entry** 4. Configure: * **Match pattern**: `.dkr.ecr..amazonaws.com/*` * **Registry type**: AWS ECR (Instance Role) * **Role name**: Name of your worker IAM role (e.g., `ValohaiWorkerRole`) * **Region**: Your ECR region 5. Save *** ## Use the private image Reference your ECR image in `valohai.yaml`: ```yaml - step: name: train image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-ml-image:v1.0 command: - python train.py ``` Valohai handles authentication automatically. # Docker Hub Configure Valohai to pull private images from Docker Hub. > Private registries are restricted to organization projects. Personal projects don't have access to private repositories. ## Create an access token 1. Log in to [hub.docker.com](https://hub.docker.com) 2. Click your username (top-right) → **Account Settings** 3. Select **Security** → **New Access Token** 4. Add a description (e.g., "Valohai access token") 5. Set permissions (Read-only is sufficient for pulling images) 6. Click **Generate** 7. **Copy the token immediately**—Docker Hub won't show it again Read more: [Docker Hub access tokens](https://docs.docker.com/docker-hub/access-tokens/) ## Add credentials to Valohai 1. Navigate to `Hi, ` (top-right menu) → `Manage ` 2. Go to **Registries** under organization controls 3. Click **Add a new entry** 4. Configure: * **Match pattern**: `docker.io//*` * Example: `docker.io/mycompany/*` * Matches all repositories under your username/organization * **Registry type**: Docker (username and password authentication) * **Username**: Your Docker Hub username * **Password**: The access token you created (not your account password) 5. Save ## Use the private image Reference your Docker Hub image in `valohai.yaml`: ```yaml - step: name: train image: docker.io/mycompany/ml-training:v2.0 command: - python train.py ``` You can also use the short form (Valohai expands it automatically): ```yaml - step: name: train image: mycompany/ml-training:v2.0 command: - python train.py ``` Valohai handles authentication automatically. # GCP Artifact Registry Configure Valohai to pull private images from Google Cloud Artifact Registry. > Private registries are restricted to organization projects. Personal projects don't have access to private repositories. ## Create a service account 1. Navigate to your Google Cloud project that hosts the Artifact Registry 2. Go to **IAM & Admin** → **Service Accounts** 3. Click **Create Service Account** 4. Add roles: * **Service Account Token Creator** (allows self-token creation) * **Artifact Registry Reader** (allows pulling images) 5. Create and download the **JSON key file** Read more: [GCP Artifact Registry access control](https://cloud.google.com/artifact-registry/docs/access-control#permissions) ## Add credentials to Valohai 1. Navigate to `Hi, ` (top-right menu) → `Manage ` 2. Go to **Registries** under organization controls 3. Click **Add a new entry** 4. Configure: * **Match pattern**: `-docker.pkg.dev//*` * Example: `us-central1-docker.pkg.dev/my-project/*` * Or narrow it: `us-central1-docker.pkg.dev/my-project/ml-images/*` * **Registry type**: GCP Artifact Registry * **Service Account JSON**: Paste the entire contents of your JSON key file 5. Save ## Use the private image Reference your Artifact Registry image in `valohai.yaml`: ```yaml - step: name: train image: us-central1-docker.pkg.dev/my-project/ml-images/training:v1.0 command: - python train.py ``` Valohai handles authentication automatically. # Azure Container Registry Configure Valohai to pull private images from Azure Container Registry (ACR). > Private registries are restricted to organization projects. Personal projects don't have access to private repositories. ## Create a service principal Follow Microsoft's guide: [Create and use service principal credentials](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-service-principal) ### Quick steps 1. Create a service principal with read access to your ACR: ```shell az ad sp create-for-rbac \ --name valohai-acr-reader \ --role acrpull \ --scopes /subscriptions//resourceGroups//providers/Microsoft.ContainerRegistry/registries/ ``` 2. Save the output: * `appId` (your username) * `password` (your password) ## Add credentials to Valohai 1. Navigate to `Hi, ` (top-right menu) → `Manage ` 2. Go to **Registries** under organization controls 3. Click **Add a new entry** 4. Configure: * **Match pattern**: `.azurecr.io/*` * Example: `mycompany.azurecr.io/*` * Or narrow it: `mycompany.azurecr.io/ml-images/*` * **Registry type**: Docker (username and password authentication) * **Username**: Service principal's `appId` * **Password**: Service principal's `password` 5. Save ## Use the private image Reference your ACR image in `valohai.yaml`: ```yaml - step: name: train image: mycompany.azurecr.io/ml-training:v1.0 command: - python train.py ``` Valohai handles authentication automatically. # JFrog Artifactory Configure Valohai to pull private images from JFrog Artifactory. > Private registries are restricted to organization projects. Personal projects don't have access to private repositories. ## Generate an access token 1. Log in to your JFrog Cloud Platform 2. Navigate to **User Management** → **Access Tokens** 3. Click **Generate Token** 4. Configure the token: * **Description**: Valohai Docker Registry Credentials * **Token Scope**: User * **Service**: Artifactory * **Expiration Time**: Set according to your security policy 5. Save the token > When the token expires, you'll need to generate a new one and update it in Valohai. ## Add credentials to Valohai 1. Navigate to `Hi, ` (top-right menu) → `Manage ` 2. Go to **Registries** under organization controls 3. Click **Add a new entry** 4. Configure: * **Match pattern**: `//*` * Example: `mycompany.jfrog.io/docker-images/*` * Matches all images in the specified repository * **Registry type**: Docker (username and password authentication) * **Username**: Your JFrog username * **Password**: The access token you generated 5. Save ## Use the private image Reference your JFrog image in `valohai.yaml`: ```yaml - step: name: train image: mycompany.jfrog.io/docker-images/ml-training:v1.0 command: - python train.py ``` Valohai handles authentication automatically. # valohai.yaml Overview The `valohai.yaml` file serves as a configuration blueprint for your machine learning experiments, allowing you to define all the necessary jobs and their parameters and dependencies in a structured manner. This file is typically stored in your project’s repository, making it easy to version and reproduce experiments. > 💡 **Tip:** Instead of writing YAML by hand, Python users can use valohai-utils to define Valohai steps in their code. Check the [Generate YAML with valohai-utils](/valohai.yaml-overview/generate-from-python) section for more information. Here’s a brief overview of what a `valohai.yaml` file typically contains: 1. **Name**: A user-friendly label for a step, pipeline, or deployment. Users trigger specific steps by mentioning their names. 2. **Environment**: Specifies the default execution environment for jobs. It can refer to a set of cloud-based or on-premises machines, or a combination of both. 3. **Image**: The Docker image that serves as the foundational environment for your step. This image contains essential software, libraries, and packages needed to run your code. For example, it might include Python 3.9, PyTorch, and various Python packages. The Docker image shouldn’t contain your own code or data. 4. **Command**: You can specify the command to execute within the chosen environment. Typically, this command runs your machine learning training script or other tasks. You can define one or multiple commands (e.g., `python train.py`, `mkdir test`, `apt-get install`, `pip install`, etc.). 5. **Inputs**: Describes the necessary inputs for your experiment, such as datasets, models, or other files. Valohai downloads and caches these inputs, making them accessible in your code as if they were local files. 6. **Parameters**: Enables you to set hyperparameters and other configurable settings for your experiment. These parameters can be easily adjusted during experimentation, allowing you to modify job configurations or perform hyperparameter tuning. ### Example valohai.yaml ```yaml - step: name: preprocess-dataset image: python:3.9 command: - pip install numpy valohai-utils - python ./preprocess_dataset.py inputs: - name: dataset default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - pip install valohai-utils - python ./train_model.py {parameters} parameters: - name: epochs default: 5 type: integer - name: learning_rate default: 0.001 type: float inputs: - name: dataset default: https://valohaidemo.blob.core.windows.net/mnist/preprocessed_mnist.npz - step: name: batch-inference image: tensorflow/tensorflow:2.6.0 command: - pip install pillow valohai-utils - python ./batch_inference.py inputs: - name: model - name: images default: - https://valohaidemo.blob.core.windows.net/mnist/four-inverted.png - https://valohaidemo.blob.core.windows.net/mnist/five-inverted.png - https://valohaidemo.blob.core.windows.net/mnist/five-normal.jpg - pipeline: name: training-pipeline nodes: - name: preprocess type: execution step: preprocess-dataset - name: train type: execution step: train-model override: inputs: - name: dataset - name: evaluate type: execution step: batch-inference edges: - [preprocess.output.preprocessed_mnist.npz, train.input.dataset] - [train.output.model*, evaluate.input.model] ``` # Generate YAML with valohai-utils If you prefer defining ML workflows in Python instead of writing YAML by hand, the Python helper tool `valohai-utils` lets you generate `valohai.yaml` from your code. This is optional. Many users write YAML directly to keep their code free of Valohai dependencies. *** ## Why Generate YAML from Python? **Familiar syntax**: If you're more comfortable with Python than YAML, this approach feels more natural. **Type safety**: Python editors provide autocomplete and type checking, catching errors before execution. **Programmatic generation**: Build YAML dynamically based on conditions, loops, or external configs. *** ## How It Works Install `valohai-utils`: ```shell pip install valohai-utils ``` Define a step in your Python script: ```python # train.py import valohai # Define parameters params = { "epochs": 10, "learning_rate": 0.001, } # Define inputs inputs = { "dataset": "s3://my-bucket/train.csv", } valohai.prepare(step="train", image="python:3.12", default_parameters=params, default_inputs=inputs) # Your training code print(f"Training with lr={lr} for {epochs} epochs") print(f"Dataset: {dataset}") ``` Generate the YAML: ```shell vh yaml step train.py ``` This creates the following `valohai.yaml` file: ```yaml - step: name: train image: python:3.12 command: python train.py {parameters} parameters: - name: learning_rate default: 0.001 type: float - name: epochs default: 10 type: integer inputs: - name: dataset default: s3://my-bucket/train.csv ``` *** ## When to Use This Approach **You're Python-first**: Your team is more comfortable with Python than YAML syntax. **Dynamic workflows**: You need to generate steps programmatically based on runtime conditions. **Rapid prototyping**: You want to define and test steps quickly without switching between files. *** ## When NOT to Use This Approach **Keep code clean**: If you want your ML code to remain framework-agnostic, write YAML by hand. **Team collaboration**: Non-Python users may find YAML easier to read and edit. **Complex pipelines**: Large multi-step pipelines are often clearer in YAML than generated from Python. *** ## What's Next? * [Validate your YAML](/valohai.yaml-overview/lint) with the linter * [Multiple YAML files](/valohai.yaml-overview/multiple-files) for monorepo projects * [Manage large YAML files](/valohai.yaml-overview/large-yaml) with anchors # Validate Your YAML (Linter) The Valohai linter catches syntax errors, indentation issues, and configuration mistakes in your `valohai.yaml` before you commit or launch a job. It's faster to fix a typo locally than to wait for a job to fail in the cloud. *** ## Why Use the Linter? **Catch errors early**: Find syntax mistakes, missing required fields, and invalid configurations before execution. **Save time and money**: Avoid spinning up expensive GPU instances only to have them crash due to a YAML typo. **Learn YAML syntax**: The linter explains what's wrong and where, helping you understand YAML structure. *** ## Run the Linter Make sure you have the [Valohai CLI installed](/cli-commands) and you're in a directory linked to a Valohai project. Then run: ```shell vh lint ``` The linter will scan your `valohai.yaml` and report any issues. *** ## Common Errors the Linter Catches ### Indentation errors YAML is whitespace-sensitive. The linter catches misaligned blocks: ```yaml # ❌ Incorrect indentation - step: name: train image: python:3.9 # This line should be indented 4 spaces ``` ```yaml # ✅ Correct indentation - step: name: train image: python:3.9 ``` ### Missing required fields Steps must have `name`, `image`, and `command`: ```yaml # ❌ Missing image - step: name: train command: python train.py ``` ```yaml # ✅ Complete step - step: name: train image: python:3.9 command: python train.py ``` ### Invalid parameter types Parameters must specify a valid type: ```yaml # ❌ Invalid type parameters: - name: learning_rate default: 0.001 type: number # Should be 'float' ``` ```yaml # ✅ Valid type parameters: - name: learning_rate default: 0.001 type: float ``` ### Syntax mistakes Forgotten colons, quotes, or brackets: ```yaml # ❌ Missing colon after 'command' - step: name: train image: python:3.13 command - python train.py ``` ```yaml # ✅ Correct syntax - step: name: train image: python:3.13 command: - python train.py ``` *** ## Example Linter Output When you run `vh lint`, for a valid `valohai.yaml` you'll see output like this: ```shell $ vh lint Success! /Users/myuser/Projects/ml-project/valohai.yaml: No errors ``` *** ## What the Linter Doesn't Catch The linter validates YAML structure and Valohai-specific syntax, but it can't catch: * **Logic errors**: If your command references a file that doesn't exist, the linter won't know * **Runtime issues**: Missing Python packages or incorrect paths will only surface during execution * **Input availability**: The linter doesn't check if the input URLs actually exist Think of it as a spell-checker, not a code reviewer. *** ## Best Practices **Lint before every commit**: Make it part of your workflow. Run `vh lint` before pushing to Git. **Fix warnings, not just errors**: Warnings often point to configuration issues that will cause problems later. *** ## What's Next? * [Generate YAML with valohai-utils](/valohai.yaml-overview/generate-from-python) to skip writing YAML by hand (Python users) * [Multiple YAML files](/valohai.yaml-overview/multiple-files) for monorepo projects * [Managing large YAML files](/valohai.yaml-overview/large-yaml) with anchors and aliases # Multiple YAML Files & Monorepos Your repository can contain multiple `valohai.yaml` files. This is useful when different teams or services share one repository but need separate ML configurations. Each Valohai project connects to one `valohai.yaml` file, which can live anywhere in your repository. *** ## Why Multiple YAML Files? **Monorepo management**: Different teams (data engineering, ML research, inference) maintain their own configurations without conflicts. **Service separation**: Each microservice or model has its own isolated workflow definition. **Environment isolation**: Dev, staging, and production pipelines use different YAML files with different resource requirements. *** ## How It Works By default, Valohai looks for `valohai.yaml` in your repository root: ``` my-repo/ ├── valohai.yaml # Default location ├── train.py └── preprocess.py ``` But you can point Valohai to any subfolder: ``` my-repo/ ├── data-engineering/ │ └── valohai.yaml # ETL pipelines ├── model-training/ │ └── valohai.yaml # ML training jobs ├── inference/ │ └── valohai.yaml # Batch prediction └── shared/ └── utils.py ``` *** ## Configure Custom YAML Path Go to **Project Settings > Repository** and set the YAML path: ``` data-engineering/valohai.yaml ``` Valohai will now use that file instead of the root `valohai.yaml`. *** ## Execution Default Working Directory Even if your `valohai.yaml` is in a subfolder, Valohai clones your entire repository during execution. The default working directory will be the root of the Git repository, not the one where `valohai.yaml` is placed. You can reference code from anywhere: ```yaml # File: model-training/valohai.yaml - step: name: train image: python:3.9 command: - python ./shared/utils.py # Access code from parent directory - python train.py ``` The full Git commit is available, so imports and relative paths work as expected. *** ## Launch Jobs from CLI with Custom YAML When running jobs from the command line, specify the YAML path. ### Adhoc Execution For adhoc jobs (no Git commit), use the `--yaml` flag: ```shell vh exec run train-model --adhoc --yaml model-training/valohai.yaml ``` ### From Git Commit For jobs based on a Git commit, first set the YAML path in the web UI (as shown above). Then run: ```shell vh exec run train-model --commit abc123 ``` Or use project mode to launch from the latest fetched commit: ```shell vh --project-mode remote --project exec run train-model ``` *** ## Example: Monorepo Structure Here's a real-world monorepo setup: ``` ml-platform/ ├── data-ingestion/ │ ├── valohai.yaml # ETL and data validation │ └── ingest.py ├── training/ │ ├── valohai.yaml # Model training │ ├── train.py │ └── evaluate.py ├── inference/ │ ├── valohai.yaml # Batch and real-time inference │ └── predict.py └── shared/ ├── preprocessing.py └── metrics.py ``` Create three Valohai projects: 1. **Data Ingestion Project** → points to `data-ingestion/valohai.yaml` 2. **Training Project** → points to `training/valohai.yaml` 3. **Inference Project** → points to `inference/valohai.yaml` Each team works independently but shares the `shared/` utilities. *** ## Best Practices **Use descriptive paths**: Name folders by function (`training/`, `inference/`) not by team (`team-a/`, `team-b/`). **Share common code**: Put reusable utilities in a `shared/` or `common/` directory accessible to all projects. **Coordinate dependencies**: If one YAML depends on outputs from another, use [dataset versioning](https://github.com/valohai/dokuhai/tree/main/concepts/datasets.md) to pass data between projects. **Keep YAML close to code**: Place `valohai.yaml` in the directory where the relevant Python scripts live for easier navigation. *** ## What's Next? * [Generate YAML with valohai-utils](/valohai.yaml-overview/generate-from-python) to skip writing YAML by hand (Python users) * [Validate your YAML](/valohai.yaml-overview/lint) with the linter * [Manage large YAML files](/valohai.yaml-overview/large-yaml) with anchors # Managing Large YAML Files As your project grows to 30+ steps, your `valohai.yaml` can become repetitive and hard to maintain. YAML anchors and aliases let you define reusable blocks once and reference them everywhere, keeping your config clean and consistent. *** ## Why This Matters **Reduce duplication**: Define common inputs, parameters, or commands once instead of copying them across dozens of steps. **Easier updates**: Change a dataset path in one place, and it updates everywhere that references it. **Better readability**: A 500-line YAML with anchors is easier to scan than a 2000-line file with repetition. *** ## YAML Anchors & Aliases: The Basics ### Define a reusable block with `&anchor` ```yaml - definitions: my-common-inputs: &common_inputs # <- Anchor named "common_inputs" - name: dataset default: s3://my-bucket/train.csv - name: config default: s3://my-bucket/config.yaml - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: python train.py inputs: *common_inputs # Uses the block defined above - step: name: evaluate-model image: tensorflow/tensorflow:2.6.0 command: python evaluate.py inputs: *common_inputs # Same inputs, no repetition ``` Both steps now share the same input definitions. Update `&common_inputs` once, and both steps inherit the change. *** ## Common Use Cases ### Shared input datasets ```yaml - definitions: standard-datasets: &datasets - name: train-set default: s3://data/train/* - name: test-set default: s3://data/test/* - step: name: model-a image: python:3.13 command: python train_a.py inputs: *datasets - step: name: model-b image: python:3.13 command: python train_b.py inputs: *datasets ``` ### Repeated parameters ```yaml - definitions: tuning-params: - &learning_rate_param name: learning_rate default: 0.001 type: float - &weight_decay_param name: weight_decay default: 0.0001 type: float - step: name: train-cnn image: python:3.13 command: python train_cnn.py {parameters} parameters: - *learning_rate_param - *weight_decay_param - step: name: train-transformer image: python:3.13 command: python train_transformer.py {parameters} parameters: ## Reference an anchor but also override certain properties (e.g. default value) - <<: *learning_rate_param default: 0.002 - *weight_decay_param ``` ### Standard commands ```yaml - definitions: setup-commands: &preprocess_cmd - apt-get update - pip install -r requirements.txt - pip install valohai-utils - python preprocess.py long-commands: - &setup_env export PYTHONPATH=/app && mkdir -p /valohai/outputs/checkpoints && mkdir -p /valohai/outputs/logs ## Reuse the whole command section - step: name: preprocess image: python:3.13 command: *preprocess_cmd ## Reuse only some of the long commands and add some more - step: name: train image: python:3.13 command: - *setup_env - python train.py ``` *** ## Merge and Override with `<<: *anchor` :heavy\_check\_mark: If the anchor references an object (a single parameter, a single input ... ), you can merge it with additional properties or override the existing ones. ```yaml - definitions: tuning-params: - &learning_rate_param name: learning_rate default: 0.001 type: float - &weight_decay_param name: weight_decay default: 0.0001 type: float - step: name: train-transformer image: python:3.13 command: python train_transformer.py {parameters} parameters: ## learning_rate_param is referencing an object - <<: *learning_rate_param default: 0.002 ## overrides the originally set default value optional: true ## adds a new property - *weight_decay_param ``` :x: If the anchor references a list (such as a list of parameters or commands), you cannot merge it with additional elements; it can only populate the entire property. ```yaml - definitions: setup-commands: &setup - apt-get update - pip install -r requirements.txt - pip install valohai-utils - step: name: preprocess image: python:3.13 command: ## The "setup" anchor references a list of strings, resulting in a command section ## with the following structure: [[string, string, string], string] ## This will cause a lint error as the command is expected to be a list of strings. - *setup - python preprocess.py ``` The next syntax might appear valid, and the linter will not report any errors ... ```yaml - definitions: base-params: &base - name: epochs default: 10 type: integer - name: dataset_name type: string default: small_set - step: name: quick-test image: python:3.13 command: echo {parameters} parameters: - <<: *base # Merge base parameters - name: debug_mode # Add a new parameter default: true type: flag ``` But the output of the `quick-test` step will be: > \--epochs=10 --debug\_mode Which indicates that only the first parameter (`epochs`) is taken from the `base` anchor, confirming that merging lists with anchors and `<<` operator is not possible. *** ## Tips for Large YAML Files **Define anchors at the top**: Keep all reusable blocks in a `definitions` section at the start of your file for easy reference. ```yaml # Anchor definitions - definitions: common-inputs: &inputs - name: dataset default: s3://bucket/data.csv training-params: ¶ms - name: epochs default: 10 type: integer # Steps - step: name: train image: python:3.13 command: python train.py {parameters} inputs: *inputs parameters: *params ``` **Use descriptive anchor names**: `&training_params` is clearer than `¶ms1`. **Don't over-anchor**: If a block is only used once, don't create an anchor. They're for repeated content. **Lint regularly**: Run `vh lint` after editing anchors to catch syntax mistakes *** ## What's Next? * [Generate YAML with valohai-utils](/valohai.yaml-overview/generate-from-python) to skip writing YAML by hand (Python users) * [Validate YAML with the linter](/valohai.yaml-overview/lint) to catch anchor syntax errors * [Multiple YAML files](/valohai.yaml-overview/multiple-files) for monorepo organization # Executions An **execution** is how Valohai runs your machine learning code on remote infrastructure. If you've used other ML platforms, think of executions as "jobs" or "experiments" but more focused on being composable building blocks within your larger workflow. ## What is an execution? An execution runs one or more commands on a remote server with a specific configuration. Each execution is tied to a **step** (defined in your `valohai.yaml`), but you can run the same step multiple times with different: * Parameters * Input files * Hardware (GPU/CPU types) * Docker images > 💡 *Every training run, data preprocessing task, or evaluation script becomes an execution in Valohai.* ## Three components shape your execution ### 1. Environment The compute infrastructure where your code runs. * **Machine type**: GPU instances for training, high-memory instances for data processing * **Cloud provider or on-premise**: AWS, Azure, GCP, Kubernetes, or on-premise * **Example**: Train neural networks on `p3.8xlarge` (4x V100 GPUs), run feature extraction on `m5.24xlarge` (384GB RAM) ### 2. Docker image The software environment containing your dependencies. You can use: * Pre-built images for TensorFlow, PyTorch, scikit-learn * Custom images with your specific library versions * Use images from Docker Hub, AWS ECR, or private registries ### 3. Repository code **How it works:** * Valohai clones your Git repository at a specific commit * Code is available at `/valohai/repository` (your working directory) * Same commit = same code = full reproducibility #### **Running local code with `--adhoc`:** During development, you often want to test changes without committing to Git. Use the `--adhoc` flag to run your local code directly: ```shell vh execution run --adhoc ``` This packages your local changes, uploads them to your data store, and downloads them on the worker for the execution. Everything stays fully reproducible, Valohai tracks the exact code snapshot used. ## Creating executions You have three ways to launch executions: **1. Define steps in `valohai.yaml`** ```yaml - step: name: train-model image: tensorflow/tensorflow:2.13.0-gpu command: python train.py --epochs {parameter:epochs} ``` **2. Use 🐍 valohai-utils (Python)** ```python import valohai # Auto-generates the YAML configuration valohai.prepare(step="train-model", image="tensorflow/tensorflow:2.13.0-gpu") ``` **3. Launch via:** * **Web UI**: Point-and-click parameter selection * **CLI**: `vh execution run train-model --adhoc` * **API**: Programmatic execution management # Steps A **step** is a reusable blueprint that defines a specific ML workload in your project. Think of it as a template that describes what should happen when you want to run a particular job. When you actually run a step, you create an **execution,** a versioned snapshot of that step run with specific inputs and parameters. > 💡 *One step can generate thousands of executions. Each execution is version-controlled and reproducible.* ## Why steps matter Steps give you reproducible, scalable ML workflows. Instead of manually running scripts with different parameters each time, you define the work once and run it as many times as needed. Common step types include: * Data preprocessing and feature engineering * Model training with hyperparameter sweeps * Model validation and testing * Batch inference and predictions * Model deployment to staging/production ## Step vs. Execution | **Step** | **Execution** | | ------------------------- | ----------------------------------------- | | Blueprint/template | Versioned snapshot of the step being run | | Defined in `valohai.yaml` | Created when you run a step | | Reusable | Specific run with exact inputs/parameters | | Static definition | Has logs, outputs, duration, and lineage | ## Anatomy of a step Steps are defined in your project's `valohai.yaml` file and specify: * **Docker image** — the environment your code runs in * **Commands** — what gets executed * **Inputs** — data files your step needs * **Parameters** — configurable values (learning rate, epochs, etc.) * **Environment** — compute requirements ## Example: Simple training pipeline ```yaml --- - step: name: preprocess-dataset image: python:3.14 command: - pip install numpy valohai-utils - python ./preprocess_dataset.py inputs: - name: dataset default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - pip install valohai-utils - python ./train_model.py {parameters} parameters: - name: epochs default: 5 type: integer - name: learning_rate default: 0.001 type: float inputs: - name: dataset default: https://valohaidemo.blob.core.windows.net/mnist/preprocessed_mnist.npz - step: name: evaluate-model image: tensorflow/tensorflow:2.6.0 command: - pip install valohai-utils scikit-learn - python ./evaluate_model.py inputs: - name: model - name: test_data ``` ## Pipeline examples For production workflows, you might chain steps together: **Data Pipeline**: `fetch-data` → `clean-data` → `feature-engineering` **Training Pipeline**: `train-model` → `validate-model` → `register-model` **Deployment Pipeline**: `build-inference-service` → `deploy-staging` → `deploy-production` Each step runs independently but can use outputs from previous steps as inputs. ## Next steps Ready to create your first step? Check out: * [Quickstart: Run your first job](/getting-started/quickstart) — hands-on tutorial for beginners * [Pipeline quickstart](/pipelines) — connect multiple steps together * [valohai.yaml reference](/valohai-yaml) — complete configuration options # Run Basic Execution Executions are Valohai's core compute unit, think of them as containerized ML jobs that run your code with specific parameters and inputs. This guide shows you how to create and run executions using both the web interface and command line. ## What You Need Before running an execution: * A project with a `valohai.yaml` file containing at least one step * Your code committed to the linked repository * Sufficient compute quota in your organization ## Method 1: Web Interface Navigate to your project and click **Create Execution**.
**Configure your execution:** 1. Select the step you want to run 2. Set parameters (or use defaults) 3. Add input files if your step requires them 4. Choose your compute environment 5. Click **Create Execution** The execution will queue automatically and start when resources are available. ## Method 2: Command Line ### Basic Execution Run a step with default settings: ```shell vh exec run step-name ``` ### With Parameters Override the default parameter values: ```shell vh exec run step-name --epochs 100 --learning-rate 0.001 ``` ### With Inputs Override the default input files or datasets: ```shell vh exec run step-name --dataset=https://example.com/data.zip ``` ### Combined Example ```shell vh exec run train-model \ --dataset=s3://my-bucket/training-data.zip \ --epochs 50 \ --batch-size 32 ``` > 💡 *Use `vh exec run --help` to see all available options.* ### **Running local code with `--adhoc`:** During development, you often want to test changes without committing to Git. Use the `--adhoc` flag to run your local code directly: ```shell vh execution run --adhoc ``` This packages your local changes, uploads them to your data store, and downloads them on the worker for the execution. Everything stays fully reproducible, Valohai tracks the exact code snapshot used. ## Execution Lifecycle Your execution moves through these states: | State | Description | | ------------ | --------------------------------------- | | **created** | Waiting for quota or queuing | | **queued** | Waiting for available compute resources | | **started** | Currently running your code | | **stopping** | Graceful shutdown in progress | | **complete** | Finished successfully | | **error** | Failed with an error | | **stopped** | Manually cancelled | Only the **started** state runs your actual code. ## Monitor Your Execution ### Real-time Logs **Web UI:** Click on any running execution to view live logs. **CLI:** Stream logs to your terminal: ```shell vh exec logs --stream ``` **Get execution status:** ```shell vh exec list --count 5 ``` ## Execution Environment Each execution runs in an isolated container with: * **Your code** at `/valohai/repository/` (working directory) * **Input files** at `/valohai/inputs/{input-name}/` * **Output directory** at `/valohai/outputs/` (write your results here) * **Docker image** with your specified tools and libraries ## Common Issues ### Files Not Found **Problem:** `FileNotFoundError: /valohai/inputs/data/file.csv` **Solution:** Check your input paths. Inputs are directories, not files: ```python # Wrong data = pd.read_csv("/valohai/inputs/dataset.csv") # Correct data = pd.read_csv("/valohai/inputs/dataset/data.csv") ``` ### Outputs Not Saved **Problem:** Generated files don't appear in the web UI. **Solutions:** * Write files to `/valohai/outputs/` only * Ensure files exist when your script completes ### Permission Errors **Problem:** `PermissionError: [Errno 13] Permission denied` **Solution:** Don't write to `/valohai/inputs/` it's read-only. Use `/valohai/outputs/` instead. ## Next Steps * **Monitor progress** in the web UI or with `vh exec logs` * **Download outputs** once execution completes * **Compare results** across multiple executions * **Scale up** by running multiple executions with different parameters 🎓 **Want a visual walkthrough?** [Complete Module 3 on Valohai Academy](https://learn.valohai.academy/training/629c55ce-5c77-11ee-b550-02e4e072a081/overview?certification_id=ac3f8064-5c29-11ee-8f31-06c8c79519b9) # Configure Placeholders Parameter placeholders give you precise control over how parameters are passed to your training scripts and commands. Instead of manually crafting command-line arguments, you define them once in your `valohai.yaml` and let Valohai handle the parameter injection. This approach makes hyperparameter optimization seamless and ensures all parameter values are tracked for reproducibility. ## When to Use Parameter Placeholders Use parameter placeholders when you need to: * **Track parameters automatically** for hyperparameter optimization * **Avoid hardcoding values** in your training scripts * **Pass different parameter formats** to various tools (e.g., `--lr=0.001` vs `lr 0.001`) * **Handle multiple parameter values** in a single execution ## Core Placeholder Types ### {parameters} - Inject All Parameters The `{parameters}` placeholder injects all defined parameters into your command at once. **YAML Configuration:** ```yaml - step: name: train-model image: python:3.11 command: python train.py {parameters} parameters: - name: max-steps type: integer description: Number of training steps default: 300 - name: learning-rate type: float pass-as: --lr={v} description: Initial learning rate default: 0.001 - name: architecture type: string pass-as: --arch={v} default: resnet50 optional: true ``` **Generated Command:** ```shell python train.py --max-steps=300 --learning-rate=0.001 --architecture=resnet50 ``` > 💡 *Parameters only appear if they have values. Optional parameters with no value are skipped.* ### {parameter:} - Use Specific Parameters Target individual parameters using `{parameter:}` syntax. This is useful for multi-step commands or when you need parameters in specific positions. **YAML Configuration:** ```yaml - step: name: preprocess-and-train image: python:3.11 command: - python preprocess.py {parameter:train-split} - python train.py {parameter:learning-rate} parameters: - name: train-split type: integer pass-as: --split={v} default: 80 - name: learning-rate type: float pass-as: --lr={v} default: 0.001 ``` **Generated Commands:** ```shell python preprocess.py --split=80 python train.py --lr=0.001 ``` ### {parameter-value:} - Raw Values Only Use `{parameter-value:}` to pass only the parameter value, ignoring any `pass-as` formatting. **YAML Configuration:** ```yaml - step: name: preprocess image: python:3.11 command: python preprocess.py {parameter-value:train-split} {parameter-value:style} parameters: - name: train-split type: integer default: 80 - name: style type: string pass-as: --style={v} # This gets ignored default: nested ``` **Generated Command:** ```shell python preprocess.py 80 nested ``` ## Advanced Use Cases ### Parameters in Input Paths Parameter placeholders work in input definitions, enabling dynamic data loading based on parameter values. ```yaml - step: name: train-model image: python:3.11 command: python train.py parameters: - name: device-id type: integer default: 455 - name: experiment type: string default: "baseline" inputs: - name: dataset default: s3://ml-data/{parameter:device-id}/train/* - name: model-config default: datum://config-{parameter:experiment} - name: sensor-data default: dataset://sensors-{parameter:experiment}/{parameter:device-id} ``` ### Multiple Value Parameters Handle lists of values with `multiple` parameter configuration. **Comma-Separated Values:** ```yaml - step: name: train-ensemble image: python:3.11 command: python train.py {parameters} parameters: - name: seed-values type: integer multiple: separate multiple-separator: "," default: [455, 922, 1344] ``` **Generated Command:** ```shell python train.py --seed-values=455,922,1344 ``` **Repeated Arguments:** ```yaml - step: name: train-ensemble image: python:3.11 command: python train.py {parameters} parameters: - name: seed-values type: integer multiple: repeat default: [455, 922, 1344] ``` **Generated Command:** ```shell python train.py --seed-values=455 --seed-values=922 --seed-values=1344 ``` ## Alternative: Configuration Files For complex parameter structures, read parameters from JSON or YAML configuration files instead of command placeholders. **Python (Standard Library):** ```python import json with open("/valohai/config/parameters.json") as f: params = json.load(f) print(params["seed_values"]) # [455, 922, 1344] ``` **Python (valohai-utils):** ```python import valohai # Note: Don't use {parameters} placeholders if reading from files epochs = valohai.parameters("epochs").value learning_rate = valohai.parameters("learning-rate").value ``` > 💡 *Learn more about available configuration files in* [*System Configuration Files*](https://github.com/valohai/dokuhai/tree/main/reference/executions/system-configuration-files.md)*.* ## Best Practices **Combine placeholder types freely:** ```yaml command: - python setup.py {parameter:config-file} - python train.py {parameters} - python evaluate.py --model-path={parameter-value:output-dir}/model.pkl ``` **Use descriptive parameter names** that match your script's argument parser. **Set reasonable defaults** so executions work without manual parameter tuning. **Use `optional: true`** for parameters that shouldn't appear when undefined. Parameter placeholders make your ML workflows more flexible and your experiments more reproducible. They're essential for automated hyperparameter optimization and tracking parameter impact on model performance. # System Environment Variables Valohai automatically sets environment variables in every execution to provide context about your job and control agent behavior. ## Execution Context Variables These variables give you information about the current execution: ```shell # Execution paths VH_CONFIG_DIR=/valohai/config # Configuration files VH_INPUTS_DIR=/valohai/inputs # Downloaded input files VH_OUTPUTS_DIR=/valohai/outputs # Files to upload VH_REPOSITORY_DIR=/valohai/repository # Git repository code (working directory) # Execution identifiers VH_JOB_ID=exec-016eb6ec-50cb-0031-3f48-d556e47b1c78 # Job UUID VH_EXECUTION_ID=016eb6ec-50cb-0031-3f48-d556e47b1c78 # Execution ID VH_PROJECT_ID=04a37c09-dbe1-4c01-b715-0a3223c50188 # Project ID # Pipeline context (when applicable) VH_TASK_ID=f9c97759-513e-44a1-9666-97cf198cde80 # Task ID VH_PIPELINE_ID=f403603b-ad11-4cc4-a90d-3118f51c8dcd # Pipeline ID VH_PIPELINE_NODE_ID=972834e2-23b5-429a-9f6d-80b8c4a75c8a # Pipeline node ID # TPU VH_TPU= # Contains the GRPC endpoint(s) of the allocated Cloud TPU(s), separated by spaces (when TPUs are available) # Distributed tasks context VH_DIST_MEMBER_ID=abc123dfg # The name of this execution in the distributed execution group VH_DIST_MEMBER_INDEX=1 # Zero-based index of this execution in the distributed execution group VH_DIST_MEMBER_COUNT=3 # Number of executions in the distributed execution group # Cloud instance IPs VH_PUBLIC_IP=13.53.65.91 # Public IP (falls back to private if unavailable) VH_LOCAL_IP=172.16.0.1 # Local/private IP address of cloud instance ``` > 💡 *Use these paths to read inputs and write outputs in your code instead of hardcoding locations.* ## Agent Behavior Controls Set these variables to modify how the Valohai agent handles your execution. > :exclamation: These must be defined in your **step definition** to take effect. > > Environment variables set via `export` in `commands` section only work inside the container and wont' affect agent behavior. ### **Interactive Terminal** #### VH\_INTERACTIVE Enables the use of [Interactive Terminal](/development-and-debugging/interactive-terminal). ```shellscript VH_INTERACTIVE=1 # true values: 1, yes, true ``` ### **Data** #### **VH\_NO\_DATA\_CACHE** Ignore cached input data and re-download from source. ```bash VH_NO_DATA_CACHE=1 # true values: 1, yes, true ``` Useful when you've reused URLs but the underlying data has changed. #### **VH\_NO\_OUTPUT\_CACHE** Prevent caching of produced output files. ```shellscript VH_NO_OUTPUT_CACHE=1 # true values: 1, yes, true ``` Prevents caching when your execution produces large outputs that won't be reused.\ This saves disk space for future executions. #### **VH\_CLEAN** Remove all Docker images and cached data before and after execution ```bash VH_CLEAN=true # true values: 1, yes, true ``` This increases execution time but ensures a clean environment. #### VH\_NO\_INPUT\_HASHING Whether to skip hashing input files. This can speed up initialization of the execution, at the expense of\ not having the data hashed for integrity checking. ```shellscript VH_NO_INPUT_HASHING=1 # true values: 1, yes, true ``` #### VH\_NO\_OUTPUT\_HASHING Whether to skip hashing output files. This can speed up finalization of the execution, at the expense of\ not having the data hashed for integrity checking. ```shellscript VH_NO_OUTPUT_HASHING=1 # true values: 1, yes, true ``` #### VH\_RENAME\_DUPLICATES Whether to rename input files with duplicate filenames to avoid conflicts. Clashing filenames will be suffixed with an underscore and a counting number like `file.txt`, `file_2.txt`, etc. ```shellscript VH_RENAME_DUPLICATES=1 # true values: 1, yes, true ``` #### VH\_ENABLE\_DATASET\_VERSION\_PACKAGING Whether to enable the dataset version packaging feature. Default to false. ```shellscript VH_ENABLE_DATASET_VERSION_PACKAGING=1 # true values: 1, yes, true ``` ### **Configuration Files** Each [configuration file](/executions/system-configuration-files) will be written in both .**json** and .**yaml** format.\ In case you don't need the .yaml version of these files (their advantage is being a bit more human readable but the machine won't mind json either), you can disable their writing with: ```shellscript VH_YAML_CONFIG_FILES=0 # false values: 0, no, false ``` > :bulb: `/valohai/config/inputs.yaml` will contain details about each requested file. In case your execution requests a large amount of files (>20k), generating and writing the .yaml version of this configuration file might take a while (even a few minutes).\ > If your execution does not rely on the .yaml version of this file, feel free to disable it and speed up the execution startup. ### **Logging** #### VH\_INPUT\_LOGGING During the input download phase, Valohai will log status of each requested file (downloaded/found-in-cache/on-demand). These messages could clutter the logs and make it harder to inspect the rest of the logs.\ There are three possible options (thus values for this environment variable): * **`VH_INPUT_LOGGING=enable`** This is the default value and it allows logging status of each input file. * **`VH_INPUT_LOGGING=disable`** Suppress all input processing logs * **`VH_INPUT_LOGGING=file`** Write logs to `/tmp/peon/runs/exec-/input_processing.log` and show minimal logs in the UI (occasionally number of downloaded files and errors). :exclamation: Note that this file is written on the path on the machine, and not inside the execution's container, therefore it's not accessible from within the execution. ### **Shared Cache** > :exclamation: **Currently applicable only in Kubernetes environments** When additional (shared) cache layers are used, it's expected that these will be remote/network file systems.\ When files are found in local cache, the ones that are requested by the execution, Valohai will expose to the execution container by creating **hard-links**. This is not possible when files are found on remote/network file system.\ \ There are two possible behaviors, controlled by the next environment variables: * **Copying** each file from remote file system - Execution will access actual files but this may take longer than the regular download (depending on the amount of files) * Creating **sym-links** - Execution will start sooner but will have to access data via sym-links. Even though this is generally "safe", it may cause misbehavior of some programs. #### VH\_ALLOW\_INPUT\_SYMLINK By setting this environment variable to a truthy value, you instruct the Valohai agent to create sym-links pointing to the files on the remote file system. ```shellscript VH_ALLOW_INPUT_SYMLINK=true # true values: 1, yes, true ``` > :bulb:This behavior is **disabled** by default. #### VH\_ALLOW\_INPUT\_COPY Copying files is default behavior (when **VH\_ALLOW\_INPUT\_SYMLINK** is disabled) but also used as a fallback behavior in case sym-linking of a file fails (when **VH\_ALLOW\_INPUT\_SYMLINK** is enabled). ```shellscript VH_ALLOW_INPUT_COPY=1 # true values: 1, yes, true ``` > :bulb:This behavior is **enabled** by default. Execution will be stopped and marked as errored in case: * **VH\_ALLOW\_INPUT\_SYMLINK = 0 and VH\_ALLOW\_INPUT\_COPY=0 -** Neither sym-linking nor copying is allowed * **VH\_ALLOW\_INPUT\_SYMLINK=1 and VH\_ALLOW\_INPUT\_COPY=0 and \[sym-linking of a file fails]** In such case, copying would be used as a fallback, but since it's disabled, execution will be stopped. ### **Resource Limits** #### VH\_CPU\_LIMIT Limit CPU cores available to the job. ```bash VH_CPU_LIMIT=2 # Use 2 cores VH_CPU_LIMIT=0,2,4 # Use specific core indices ``` #### VH\_MEMORY\_LIMIT Set memory usage limit. ```bash VH_MEMORY_LIMIT=500M # 500 megabytes VH_MEMORY_LIMIT=2G # 2 gigabytes ``` #### VH\_SHM\_SIZE Increase shared memory directory size. ```bash VH_SHM_SIZE=16G # 16 gigabytes ``` Useful for applications that need large shared memory (e.g., PyTorch DataLoader). ### **Docker** #### VH\_DOCKER\_NETWORK Name of the network in which to create the execution container. ```shellscript VH_DOCKER_NETWORK="training-net" ``` #### VH\_EXPOSE\_PORTS Comma-separated list of port mappings (\:\) to expose on the execution container. ```shellscript VH_EXPOSE_PORTS='8080:80, 22:22' ``` #### VH\_INIT Whether to use an `init`-like daemon in the Docker container. Defaults to **true**, since most workloads don't properly handle signals. In case a workload does handle signals properly, this can be set to a falsy value. ```shellscript VH_INIT=0 # false values: 0, no, false ``` #### VH\_ALLOW\_ENTRYPOINT Whether to use the `ENTRYPOINT` defined in the Docker image. ```shellscript VH_ALLOW_ENTRYPOINT=1 # true values: 1, yes, true ``` #### **VH\_NO\_IMAGE\_CACHE** Force Docker image re-pull, ignoring local cache. ```bash VH_NO_IMAGE_CACHE=true # true values: 1, yes, true ``` ### **Storage Behavior** #### VH\_TMPFS Control whether `/tmp` writes to memory or disk. ```bash VH_TMPFS=0 # false values: 0, no, false ``` By default, `/tmp` is a memory filesystem. Setting this to false writes to disk instead, which is slower but avoids out-of-memory errors for large temporary files. ### GPU #### VH\_RESET\_GPU Whether to issue an Nvidia GPU reset command before running the container. ```shellscript VH_RESET_GPU=1 # true values: 1, yes, true ``` #### VH\_XORG Whether to start a Xorg server for the execution's duration. ```shellscript VH_XORG=1 # true values: 1, yes, true ``` ### CODE #### VH\_CHOWN\_REPOSITORY Whether to try to `chown` the repository directory to the user running the container. ```shellscript VH_CHOWN_REPOSITORY=1 # true values: 1, yes, true ``` # Environment Variables Environment variables are key-value pairs that your executions can read and write. Valohai gives you flexible options for defining them at different scopes. You can set environment variables in four ways: * **Inline commands** — Quick `export MY_VAR=value` in your step commands * **Step definitions** — Explicit variables in `valohai.yaml` * **Project level** — Shared across all project executions * **Organization level** — Grouped variables across multiple projects ## Execution Environment Variables ### Inline Commands Set variables directly in your step commands: ```yaml - step: name: train-model image: python:3.11 command: - export MODE="1" - export POST="clip" - python train.py ``` This approach works but won't be tracked in Valohai's interface. ### Step Definitions Define expected variables explicitly in your `valohai.yaml`: ```yaml - step: name: train-model image: python:3.11 command: python train.py environment-variables: - name: MODE default: "1" - name: POST default: "clip" ``` **Variable properties:** * `name` — Variable name passed to the execution * `description` — (optional) Human-readable explanation * `default` — (optional) Default value if not specified * `optional` — (optional) Set to `false` to make required > 💡 *Step-defined variables appear in the UI when creating executions and can be modified via CLI or API.* ## Project Environment Variables Project variables work well for API keys, database connections, or shared configuration across executions. **To create project variables:** 1. Navigate to Project Settings 2. Open "Environment Variables" tab 3. Enter variable name and value 4. Check "Secret" to hide the value in the UI 5. Click "Add" then "Save" **To use project variables:** Check **Inherit project's environment variables and secrets** when creating executions. > 💡 *Secret variables are hidden in the UI but can still be printed in execution logs by users with project access.* ## Organization Environment Variables Organization admins can create variable groups shared across multiple projects. This centralizes management and simplifies tasks like key rotation. ### Creating Variable Groups 1. Click your username in the top-right corner 2. Select "Manage \" 3. Open "Environment Variables" tab 4. Click "Create new environment variable group" 5. Name your group and select target projects 6. Add variables with names and values 7. Mark sensitive values as "Secret" 8. Click "Save" ### Example Setup The screenshot below shows two environment variable groups: * **Production** — Used by "fun-tensorflow" and "productionproject" projects * **Staging** — Used exclusively by "staging" project Both groups contain public and secret variables for their respective environments.
### Using Variable Groups Organization variable groups appear under each project's "Env Variables" tab. Non-admin users can view group names and public values but cannot modify them.
### Transient Environment Variables Transient environment variables allow you to inject short-lived values into executions without persisting them in your project configuration. Unlike step-defined, project-level, or organization-level variables, transient variables are **execution-scoped**. They are injected at runtime and are not stored in: * `valohai.yaml` * Project environment variables * Organization variable groups This enables you to pass temporary or sensitive values without committing them to version control or storing them in shared settings. #### Use Cases Transient environment variables are useful when you need to: * Inject API keys or access tokens for external services that rotate frequently * Pass experiment-specific credentials that should not be shared across executions * Manage temporary authentication tokens for CI/CD pipelines triggering Valohai jobs * Provide user-specific or session-specific variables when creating executions programmatically * Handle sensitive configuration that should not be stored in project YAML or visible in the UI #### Using Transient Variables via API When creating an execution through the Valohai API, include transient environment variables in the execution creation payload. These variables are injected into the runtime environment but are not stored in the project configuration. Refer to the API documentation for the execution creation endpoint: See the `ExecutionCreate` operation for the exact payload structure and details on how to provide environment variables in the request body. #### Managing Transient Variables in the UI You can define transient environment variables in both user and organization settings. **User-Level Transient Variables** To create user-scoped transient variables: 1. Click your username in the top-right corner 2. Select **My Profile** 3. Open the **Environment Variables** tab 4. Navigate to **Transient Environment Variables** 5. Add your variable name and value These variables apply to executions triggered by you.
**Organization-Level Transient Variables** Organization admins can define transient variables available across projects. To create organization-scoped transient variables: 1. Click your username in the top-right corner 2. Select **Manage \** 3. Open the **Environment Variables** tab 4. Navigate to **Transient Environment Variables** 5. Add variables and mark sensitive values as **Secret** if needed
# System Configuration Files During execution, Valohai provides ten configuration files at `/valohai/config/` containing execution metadata, parameters, inputs, and runtime information. These files are available in both JSON and YAML formats. ### Available Files
FileDescription
execution.json
execution.yaml		Execution metadata (project, creator, environment)
inputs.json
inputs.yaml		Downloaded input files and metadata
parameters.json
parameters.yaml		Parameter values for the execution
runtime.json
runtime.yaml		Runtime environment information
api.json
api.yaml		Available API endpoints for the execution
### execution.json / execution.yaml Contains execution-specific metadata including project information, creator details, environment configuration, and execution status. #### Basic Execution Data **JSON:** ```json { "valohai.commit-identifier": "2db0b9da830749e462f0226a481559e0cf47292c", "valohai.creator-email": "drazen@valohai.com", "valohai.creator-id": 3323, "valohai.creator-name": "drazen", "valohai.environment-id": "0167d05d-a1d7-cc02-8256-6455a6ecfa56", "valohai.environment-name": "Microsoft Azure F2s v2 (No GPU)", "valohai.environment-slug": "azure-westeurope-f2sv2", "valohai.execution-counter": 6, "valohai.execution-ctime": "2025-06-07T18:49:43.984292+00:00", "valohai.execution-duration": null, "valohai.execution-id": "01974bba-352f-7c8c-d111-d8595a8649f2", "valohai.execution-image": "tensorflow/tensorflow:2.6.0", "valohai.execution-qtime": null, "valohai.execution-status": "created", "valohai.execution-step": "train-model", "valohai.execution-tags": [], "valohai.execution-title": null, "valohai.project-id": "0196c52a-528b-8d35-dd35-737aa5ae672b", "valohai.project-name": "Project/lovely-tensorflow" } ``` **YAML:** ```yaml valohai.commit-identifier: 2db0b9da830749e462f0226a481559e0cf47292c valohai.creator-email: drazen@valohai.com valohai.creator-id: 3323 valohai.creator-name: drazen valohai.environment-id: 0167d05d-a1d7-cc02-8256-6455a6ecfa56 valohai.environment-name: Microsoft Azure F2s v2 (No GPU) valohai.environment-slug: azure-westeurope-f2sv2 valohai.execution-counter: 6 valohai.execution-ctime: '2025-06-07T18:49:43.984292+00:00' valohai.execution-duration: null valohai.execution-id: 01974bba-352f-7c8c-d111-d8595a8649f2 valohai.execution-image: tensorflow/tensorflow:2.6.0 valohai.execution-qtime: null valohai.execution-status: created valohai.execution-step: train-model valohai.execution-tags: [] valohai.execution-title: null valohai.project-id: 0196c52a-528b-8d35-dd35-737aa5ae672b valohai.project-name: Project/lovely-tensorflow ``` #### Extended Execution Data When the execution is part of a task, pipeline, or triggered workflow, additional metadata is included: **JSON:** ```json { "valohai.task-id": "0192e1a3-c409-0021-c074-87370badadee", "valohai.task-counter": 124, "valohai.pipeline-id": "0192e1a4-33e3-6184-d308-0d87ab016111", "valohai.pipeline-counter": 1952, "valohai.pipeline-title": "Pathway to Production", "valohai.pipeline-node-id": "0192e1a7-fb19-e8c2-420c-bb53a1c0fc5d", "valohai.pipeline-tags": [ "high-performance" ], "valohai.trigger-id": "0192e1a4-9332-6aaf-96ed-d78617a66ab2", "valohai.trigger-run-id": "0192e1a9-49a4-3efa-b8c0-a612a1e1456f", "valohai.trigger-title": "Webhook Automation" } ``` **YAML:** ```yaml valohai.task-id: 0192e1a3-c409-0021-c074-87370badadee valohai.task-counter: 124 valohai.pipeline-id: 0192e1a4-33e3-6184-d308-0d87ab016111 valohai.pipeline-counter: 1952 valohai.pipeline-title: Pathway to Production valohai.pipeline-node-id: 0192e1a7-fb19-e8c2-420c-bb53a1c0fc5d valohai.pipeline-tags: ["high-performance"] valohai.trigger-id: 0192e1a4-9332-6aaf-96ed-d78617a66ab2 valohai.trigger-run-id: 0192e1a9-49a4-3efa-b8c0-a612a1e1456f valohai.trigger-title: Webhook Automation ``` ### inputs.json / inputs.yaml Lists all downloaded input files with their metadata, checksums, and file paths. **JSON:** ```json { "dataset": { "files": [ { ## If file is selected using the cloud store URL ## e.g: azure://valohaidemo/mnist/preprocessed_mnist.npz "checksums": { "file_size": 12, "md5": "d3f9231518971c43c3e0fcf11666e88d", "sha1": "0d3d38ff5e5c2d0ead2486b769059b7cf895bf50", "sha256": "f55c2f6d1c89cdc26d5a51779deffcba8665531a0a83b557bc8fe43dae3cd118" }, "download_intent": "always", "input_id": "01974bb9-016c-267c-61a6-8782103ecf2f", "name": "preprocessed_mnist.npz", "path": "/valohai/inputs/dataset/preprocessed_mnist.npz", "size": 22181444, "storage_uri": "azure://valohaidemo/mnist/preprocessed_mnist.npz", "uri": "azure://valohaidemo/mnist/preprocessed_mnist.npz" }, { ## If file is selected using the datum ID, datum query or manually selected ## e.g: datum://019c0503-ad47-ba1e-9167-583aa21e8ef6 "checksums": { "file_size": 13, "md5": "d3f9231518971c43c3e0fcf11666e88d", "sha1": "0d3d38ff5e5c2d0ead2486b769059b7cf895bf50", "sha256": "f55c2f6d1c89cdc26d5a51779deffcba8665531a0a83b557bc8fe43dae3cd118" }, "download_intent": "always", "input_id": "01974bb9-016c-267c-61a6-8782103ecf2f", "datum_id": "019c0503-ad47-ba1e-9167-583aa21e8ef6", "name": "preprocessed_mnist_datum.npz", "path": "/valohai/inputs/dataset/preprocessed_mnist_datum.npz", "size": 22181222, "storage_uri": "azure://valohaidemo/mnist/preprocessed_mnist.npz", "uri": "datum://019c0503-ad47-ba1e-9167-583aa21e8ef6" } ], "input_id": "01974bb9-016c-267c-61a6-8782103ecf2f" } } ``` **YAML:** ```yaml dataset: files: ## If file is selected using the cloud store URL ## e.g: azure://valohaidemo/mnist/preprocessed_mnist.npz - checksums: file_size: 12 md5: d3f9231518971c43c3e0fcf11666e88d sha1: 0d3d38ff5e5c2d0ead2486b769059b7cf895bf50 sha256: f55c2f6d1c89cdc26d5a51779deffcba8665531a0a83b557bc8fe43dae3cd118 download_intent: always input_id: 01974bb9-016c-267c-61a6-8782103ecf2f name: preprocessed_mnist.npz path: /valohai/inputs/dataset/preprocessed_mnist.npz size: 22181444 storage_uri: azure://valohaidemo/mnist/preprocessed_mnist.npz uri: azure://valohaidemo/mnist/preprocessed_mnist.npz ## If file is selected using the datum ID, datum query or manually selected ## e.g: datum://019c0503-ad47-ba1e-9167-583aa21e8ef6 - checksums: file_size: 13 md5: d3f9231518971c43c3e0fcf11666e88d sha1: 0d3d38ff5e5c2d0ead2486b769059b7cf895bf50 sha256: f55c2f6d1c89cdc26d5a51779deffcba8665531a0a83b557bc8fe43dae3cd118 download_intent: always input_id: 01974bb9-016c-267c-61a6-8782103ecf2f datum_id: 019c0503-ad47-ba1e-9167-583aa21e8ef6 name: preprocessed_mnist_datum.npz path: /valohai/inputs/dataset/preprocessed_mnist_datum.npz size: 22181222 storage_uri: azure://valohaidemo/mnist/preprocessed_mnist_datum.npz uri: datum://019c0503-ad47-ba1e-9167-583aa21e8ef6 input_id: 01974bb9-016c-267c-61a6-8782103ecf2f ``` ### parameters.json / parameters.yaml Contains parameter values defined for the execution. **JSON:** ```json { "epochs": 5, "learning_rate": 0.001, "batch_size": 32, "model_architecture": "resnet50" } ``` **YAML:** ```yaml epochs: 5 learning_rate: 0.001 batch_size: 32 model_architecture: resnet50 ``` ### runtime.json / runtime.yaml Provides runtime environment information including runner identity and node details. **JSON:** ```json { "runner.identity": "azwesteuropef2sv2-hsygefxy", "runner.nodename": "azwesteuropef2sv2-hsygefxy" } ``` **YAML:** ```yaml runner.identity: azwesteuropef2sv2-hsygefxy runner.nodename: azwesteuropef2sv2-hsygefxy ``` ### api.json / api.yaml Contains available API endpoints for the execution with authentication headers. **JSON:** ```json { "set_status_detail": { "method": "POST", "url": "https://app.valohai.com/api/v0/executions/018aa9d9-f899-8e4b-80f9-464ea1c855eb/set_status_detail/", "headers": { "Authorization": "Execution-Token {...}" } }, "trigger_catalog": { "method": "POST", "url": "https://app.valohai.com/api/v0/worker-trigger-catalog/018aa9d9-f899-8e4b-80f9-464ea1c855eb/", "headers": { "Authorization": "Execution-Token {...}" } } } ``` **YAML:** ```yaml set_status_detail: method: POST url: https://app.valohai.com/api/v0/executions/018aa9d9-f899-8e4b-80f9-464ea1c855eb/set_status_detail/ headers: Authorization: Execution-Token {...} trigger_catalog: method: POST url: https://app.valohai.com/api/v0/worker-trigger-catalog/018aa9d9-f899-8e4b-80f9-464ea1c855eb/ headers: Authorization: Execution-Token {...} ``` ### File Access Examples **Python (JSON):** ```python import json # Read parameters with open("/valohai/config/parameters.json") as f: params = json.load(f) # Read execution metadata with open("/valohai/config/execution.json") as f: execution = json.load(f) # Read input file information with open("/valohai/config/inputs.json") as f: inputs = json.load(f) ``` **Python (YAML):** ```python import yaml # Read parameters with open("/valohai/config/parameters.yaml") as f: params = yaml.safe_load(f) # Read execution metadata with open("/valohai/config/execution.yaml") as f: execution = yaml.safe_load(f) ``` # Advanced Features # Queue Priority Queue priority lets you run urgent executions immediately instead of waiting for the entire queue to clear. Use this when you need to: * Deploy a model update to production right away * Re-run a failed critical experiment * Process time-sensitive inference requests Without priority, all executions run first-in, first-out (FIFO). With priority enabled, you create two separate queues: high priority executions run before normal priority ones. *** ## How It Works When you create an execution with high priority: 1. It's placed at the **end of the high-priority queue** 2. All high-priority executions run before any normal-priority executions 3. Within each priority level, executions still run FIFO **Important:** Running executions are never interrupted. If normal-priority jobs are already running when you queue a high-priority execution, those running jobs will complete first. *** ## Enable Priority at Creation ### Web App Toggle **High Priority** next to the **Create Execution** button when starting a new execution.
### Command Line Add the `--priority` flag: ```shell # Run the Step named train-model with high priority vh exec run train-model --priority ``` *** ## Change Priority While Queued You can upgrade (or downgrade) an execution's priority while it's waiting in the queue. When you change priority: * The execution is **removed from its current position** * It's **rescheduled at the end** of the new priority queue This is useful when priorities shift — for example, if a stakeholder flags an experiment as urgent after it's already been queued. *** ## Availability The queue priority feature has to be enabled on the environment setup level. Please contact Valohai support for help with this. Queue priority works with: * Cloud virtual machine environments * On-premises execution environments Not currently supported: * Kubernetes environments * Slurm environments If you try to set priority on an unsupported environment via CLI or API, you'll receive an error response. The priority toggle in the web app only appears for supported environments. *** ## When to Use Priority ### ✅ Best For **Dedicated, fixed-capacity environments** where you have: * A constant pool of workers (e.g., 10 GPUs always running) * Mixed workloads (urgent production jobs + background experiments) * Predictable queue times Priority helps you process critical work immediately while batching lower-priority jobs during off-hours. ### ⚠️ Less For **Auto-scaling environments** that spin up capacity on-demand. These environments already handle bursts by adding workers automatically, so the entire queue gets processed quickly regardless of priority. You can still use priority for organizational clarity, but it won't dramatically change execution times. *** ## Performance Considerations Priority queueing requires the queue service to do more work when scheduling executions. For most use cases, this overhead is negligible. However, **very high-capacity environments** (hundreds of simultaneous executions) may experience slightly slower queue scheduling or need additional resources allocated to queue services. This is a theoretical concern for most deployments, contact Valohai support if you're running at extreme scale and notice queue performance issues. # Spot Instances Spot instances are unused virtual machines that cloud providers offer at steep discounts, often 60-90% cheaper than standard on-demand instances. The tradeoff? Your job can be interrupted when the provider needs that capacity back. This makes spot instances perfect for fault-tolerant ML workloads like training experiments, hyperparameter sweeps, and batch inference. ## Why Use Spot Instances? **Cost savings without infrastructure complexity.** Valohai handles interruptions gracefully, so you can focus on your models instead of managing cloud infrastructure. Spot instances work identically to standard environments in Valohai. Select a spot machine type from the environment dropdown when launching your execution. ## How Spot Interruptions Work When a cloud provider reclaims a spot instance: 1. Your code receives a `KeyboardInterrupt` signal 2. You have 2-3 minutes to save checkpoints and wrap up 3. Valohai uploads files from `/valohai/outputs/` 4. The machine terminates and removes its disk **Critical:** Use [Live Outputs](/data/data-versioning/save-files-from-jobs#live-uploads) to save checkpoints continuously during training. Waiting until the end risks losing large files when interruptions happen. ### Automatic Restart Enable auto-restart to requeue interrupted jobs automatically. Your new execution will include a special `_restart` input containing all outputs from the interrupted run. Your code should check for checkpoints in the `_restart` input and resume from the latest one: ```python import valohai import os # Check if this is a restarted execution restart_path = "/valohai/inputs/_restart" if os.path.exists(restart_path) and os.listdir(restart_path): # Load latest checkpoint checkpoint_files = sorted(os.listdir(restart_path)) latest_checkpoint = os.path.join(restart_path, checkpoint_files[-1]) model.load_state_dict(torch.load(latest_checkpoint)) print(f"Resuming from checkpoint: {latest_checkpoint}") ``` > **The disk is removed when a spot instance terminates.** Restarted executions begin from a clean slate. Your code must explicitly load checkpoints from the `_restart` input to continue training. ## Selecting Spot Environments Choose spot environments the same way you select any other machine type: 1. **Show only spot types** using the filter in the environment dropdown 2. **Note the environment slug** (e.g., `aws-eu-west-1-g4dn-xlarge-spot`) for CLI/API usage 3. **Enable auto-restart** in the execution settings if you want automatic requeuing ### CLI Example ```shell vh exec run train \ --adhoc \ --environment aws-eu-west-1-g4dn-xlarge-spot \ --autorestart ``` ### API Example To use the autorestart feature, add the following in your API call payload. Make sure the environment you have defined is a spot instance. ```json "runtime_config": { "autorestart": True }, ``` ## Managing Outputs During Interruption When Valohai receives a shutdown signal, it immediately starts uploading everything in `/valohai/outputs/`. **For small files** (< 1GB): Most files upload successfully before termination. **For large files** (models, datasets): Upload continuously with Live Outputs instead of waiting until the end. ```python import shutil # Save checkpoint to outputs immediately checkpoint_path = f"/valohai/outputs/checkpoint_epoch_{epoch}.pt" torch.save(model.state_dict(), checkpoint_path) os.chmod(checkpoint_path, S_IREAD | S_IRGRP | S_IROTH) # Valohai uploads this file immediately as a Live Output ``` > You cannot overwrite or delete files in `/valohai/outputs`. Save files with unique names (e.g., timestamped checkpoints). ## Testing Interruption Handling Verify your code handles interruptions correctly using cloud provider tools: ### AWS: Fault Injection Simulator Use [AWS FIS](https://aws.amazon.com/fis/) to create an experiment that terminates your spot instance mid-execution. ### GCP: Simulate Maintenance Event ```shell gcloud auth login gcloud compute instances simulate-maintenance-event --zone ``` Your code should catch the `KeyboardInterrupt`, save critical state, and exit cleanly. ## Pricing and Quotas Spot pricing varies by provider. Understanding these differences helps you choose the right cloud for your workload. ### AWS Spot Instances Pricing adjusts dynamically based on supply and demand. Each AWS environment in Valohai has a "max price" setting (default: on-demand price). AWS limits the number of running and requested spot instances per region. [Request quota increases](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-limits.html) if you hit these limits during large sweeps. **Reference:** [AWS Spot Instance documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html) ### Google Cloud Spot VMs GCP uses fixed pricing that changes at most once per month. When planning capacity, consider CPU, disk, and GPU quotas. Request **preemptible quotas** separately from standard quotas to prevent spot jobs from consuming your regular allocation. **Reference:** [GCP Spot VM documentation](https://cloud.google.com/spot-vms) ### Azure Spot Virtual Machines Pricing varies by region and machine type. Azure distinguishes between vCPU quotas for spot and standard VMs. **Reference:** * [Azure Spot VM documentation](https://docs.microsoft.com/en-us/azure/virtual-machines/spot-vms) * [Azure vCPU quota management](https://docs.microsoft.com/en-us/azure/quotas/per-vm-quota-requests) # Time Limits Time limits help you control costs and prevent runaway executions. Valohai provides two timeout mechanisms: * **Time Limit** — Maximum total execution duration * **No Output Timeout** — Stop if execution produces no logs for a period Both can be configured in the web UI or in your `valohai.yaml`. ## Time Limit Set a maximum duration for your execution. When the time limit is reached, Valohai terminates the execution. **Use cases:** * Prevent forgotten executions from running indefinitely * Enforce budget constraints on expensive GPU instances * Ensure batch jobs complete within a scheduling window ### Set in Web UI 1. Create a new execution 2. Scroll to the **Runtime** section 3. Check **Set a Time Limit** 4. Enter the maximum duration in hours and minutes > Not setting a time limit allows the execution to run indefinitely. This is the default behavior. ### Set in valohai.yaml Define a default time limit for a step: ```yaml - step: name: train-model image: tensorflow/tensorflow:2.13.0-gpu time-limit: 1h # 1 hour command: - python train.py {parameters} ``` The `time-limit` value supports human-readable formats like `1h 30m 5s`, or you can specify seconds as an integer (e.g., `3600`). *** ## No Output Timeout Stop executions that become unresponsive. If your execution produces no logs or output for the specified duration, Valohai terminates it. **Use cases:** * Detect and stop hung processes * Catch infinite loops that produce no output * Identify network or I/O blocking issues ### Set in Web UI 1. Create a new execution 2. Scroll to the **Runtime** section 3. Check **Set a No Output Timeout** 4. Enter the timeout duration in hours and minutes > Not setting this will default to about 8 hours. ### Set in valohai.yaml Define a default no-output timeout for a step: ```yaml - step: name: train-model image: tensorflow/tensorflow:2.13.0-gpu no-output-timeout: 30m # 30 minutes command: - python train.py {parameters} ``` The `no-output-timeout` value supports human-readable formats like `1h 30m 5s`, or you can specify seconds as an integer (e.g., `1800`). *** ## Example: Complete Step Configuration Combine time limits with other step settings: ```yaml - step: name: train-model image: tensorflow/tensorflow:2.13.0-gpu time-limit: 4h # 4 hours max no-output-timeout: 30m # 30 min no-output timeout command: - pip install -r requirements.txt - python train.py {parameters} parameters: - name: epochs default: 100 type: integer ``` *** ## Best Practices **Set reasonable defaults in YAML.** Define time limits in your `valohai.yaml` so all team members use consistent settings. Override in the UI when needed. **Use no-output timeout to catch hangs.** Long-running jobs should periodically log progress. If your training loop runs for hours without output, it may be stuck. **Account for setup time.** Time limits include dependency installation, data download, and model initialization. Give enough buffer for these steps. **Combine with early stopping.** For training jobs, consider using [early stopping](/tasks/early-stopping) based on metrics in addition to time limits. *** ## Troubleshooting ### Execution stopped unexpectedly Check the execution logs for timeout messages. Common causes: * **Time limit reached** — Increase the limit or optimize your code * **No output timeout** — Add periodic logging to your training loop *** ## Related * [Early Stopping](/tasks/early-stopping) — Stop based on metadata conditions * [Spot Instances](/executions/advanced-features/spot-instances) — Handle interruptions for cost savings * [Run Basic Execution](/executions/run-basic-execution) — Creating and running executions # Dynamic GPU Allocation Split GPU resources on multi-GPU machines to run more jobs concurrently. Instead of dedicating entire machines to single executions, allocate only the GPUs each job needs. This feature is particularly effective on on-premises servers with multiple GPUs. It's not enabled by default, your organization administrator must configure it first. ## When to Use Dynamic Allocation **On-premises multi-GPU servers:** Run multiple 1-GPU jobs simultaneously on an 8-GPU machine instead of queuing them sequentially. **When NOT needed:** * **Cloud auto-scaling:** Select instance types that match your needs exactly (e.g., `p3.2xlarge` for 1 GPU, `p3.8xlarge` for 4 GPUs) * **Kubernetes environments:** Resource allocation is handled through runtime configuration Dynamic allocation is only available for **Virtual Machine (Dispatch)** environments running dispatch mode workers. ## Configure GPU Allocation Set the `VH_GPUS` environment variable to specify how many GPUs your execution needs: ```shell VH_GPUS=2 ``` Your execution will wait in the queue until 2 GPUs become available on any machine in the environment. ### Set via Web UI Add the environment variable in the execution configuration:
### Set via valohai.yaml ```yaml - step: name: distributed-training image: pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime command: - python train_distributed.py environment-variables: - name: VH_GPUS default: 4 ``` > **Be careful with GPU requests.** If you request more GPUs than any single machine has, your execution will remain queued indefinitely. ## How GPU Scheduling Works Valohai uses a **first-come, first-served** queue with intelligent prioritization: ### Priority Rules 1. **Small jobs run first:** If two executions are queued, the one requesting fewer GPUs gets priority 2. **Escalation after 1 hour:** Executions waiting longer than 1 hour get elevated priority, preventing indefinite starvation of large multi-GPU jobs ### GPU Assignment GPUs are allocated in device index order — the same order tools like `nvidia-smi` display them: ```shell # Your 2-GPU execution gets devices 0 and 1 nvidia-smi # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.60.13 Driver Version: 525.60.13 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | 0 Tesla V100-SXM2 On | 00000000:00:1E.0 Off | 0 | ← First # | 1 Tesla V100-SXM2 On | 00000000:00:1F.0 Off | 0 | ← Second # | 2 Tesla V100-SXM2 On | 00000000:00:20.0 Off | 0 | # | 3 Tesla V100-SXM2 On | 00000000:00:21.0 Off | 0 | ``` ## Example Use Cases ### Single-GPU Training on Multi-GPU Server Run 8 experiments simultaneously on an 8-GPU machine: ```yaml - step: name: single-gpu-experiment image: tensorflow/tensorflow:latest-gpu command: - python train.py environment-variables: - name: VH_GPUS default: 1 ``` Launch 8 executions — they'll all run in parallel instead of queuing. ### Multi-GPU Distributed Training Reserve 4 GPUs for a single distributed training job: ```yaml - step: name: distributed-training image: pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime command: - torchrun --nproc_per_node=4 train_distributed.py environment-variables: - name: VH_GPUS default: 4 ``` ### Mixed Workload Scheduling Queue both small and large jobs efficiently: ```shell # Small job (1 GPU) - runs immediately vh exec run quick-test --adhoc VH_GPUS=1 # Large job (4 GPUs) - waits for 4 GPUs to free up vh exec run full-training --adhoc VH_GPUS=4 # Another small job (1 GPU) - runs before large job if submitted within 1 hour vh exec run another-test --adhoc VH_GPUS=1 ``` After 1 hour, the 4-GPU job escalates in priority and will run next, even if more 1-GPU jobs are queued. ## Monitoring GPU Utilization Track how effectively you're using GPU resources: * [Hardware Statistics](/observability/resource-monitoring/hardware-statistics) — Real-time GPU utilization during execution * [Visualize Utilization](/observability/resource-monitoring/visualize-utilization) — Historical GPU usage patterns * [Track Underutilization](/observability/resource-monitoring/track-underutilization) — Identify over-allocated GPUs ## Related Topics * [Tasks & Parallel Execution](/tasks) — Run hyperparameter sweeps with dynamic GPU allocation * [Distributed Training](/distributed-training) — Coordinate multi-GPU training across executions * [Team Quotas](/user-and-organization-management/environments-and-access-control/team-quotas) — Limit concurrent GPU usage per team # Custom Execution Status Track progress and communicate execution state beyond Valohai's default statuses (queued, started, completed, error). Custom statuses help ML teams monitor long-running jobs, debug failures faster, and integrate with external monitoring systems. ## Why Use Custom Status **For ML Engineers:** Get real-time visibility into multi-hour training jobs without SSH access or log diving. **For Teams:** Share meaningful progress updates like "Loading 50GB dataset" or "Epoch 45/100" that stakeholders actually understand. **For Integration:** Feed execution state into monitoring dashboards, Slack notifications, or CI/CD pipelines via API. ## Set Status with Direct API This method works without additional dependencies and gives you full control over status updates. ```python import json import requests # Read Valohai execution config with open("/valohai/config/api.json", "r") as json_file: data = json.load(json_file) headers = data["set_status_detail"]["headers"] # Set your custom status response = requests.post( data["set_status_detail"]["url"], headers=headers, json={"status_detail": "Processing epoch 23/100"}, ) print(f"Status updated: {response.status_code}") ``` > 💡 *The status overwrites previous custom statuses. Set it at key milestones for best visibility.* ## Set Status with valohai-utils If you're already using valohai-utils in your project, this one-liner handles the API call: ```python import valohai valohai.set_status_detail("Processing epoch 23/100") ``` Remember to add `valohai-utils` to your requirements.txt. ## Common Status Patterns ### Milestone Updates Set status at major workflow transitions: ```python # Data loading phase requests.post(url, headers=headers, json={"status_detail": "Loading training data..."}) # Training phase requests.post(url, headers=headers, json={"status_detail": "Training model - epoch 1/50"}) # Validation phase requests.post(url, headers=headers, json={"status_detail": "Running validation..."}) ``` ### Progress Tracking Update status during long-running operations: ```python for epoch in range(total_epochs): if epoch % 5 == 0: # Update every 5 epochs status = f"Training: epoch {epoch}/{total_epochs}" requests.post(url, headers=headers, json={"status_detail": status}) ``` ### Error Context Provide debugging context when things go wrong: ```python try: load_model() except Exception as e: error_status = f"Failed loading model: {str(e)[:100]}" requests.post(url, headers=headers, json={"status_detail": error_status}) raise ``` ## Rich Status Content Add visual elements like progress bars and charts to your status updates. Each rich element must be valid JSON on a single line. ### Text with Color ```python status_data = '{"type": "text", "text": "Model converged!", "color": "good"}' requests.post(url, headers=headers, json={"status_detail": status_data}) ``` Available colors: `good`, `bad`, `warn`, or any CSS color value. ### Progress Gauge ```python # Simple percentage gauge = '{"type": "gauge", "value": 0.75, "label": "Training Progress"}' # With custom range gauge = '{"type": "gauge", "value": 45, "min": 0, "max": 100, "label": "Epoch 45/100", "color": "good"}' requests.post(url, headers=headers, json={"status_detail": gauge}) ``` ### Sparkline Charts ```python # Loss values over time loss_chart = '{"type": "sparkline", "values": [0.8, 0.6, 0.4, 0.3, 0.25], "color": "good"}' requests.post(url, headers=headers, json={"status_detail": loss_chart}) ``` ## Access Status via API Retrieve custom status from external services or local scripts: ```python import requests import os # Use your Valohai API token auth_token = os.environ["VH_API_TOKEN"] headers = {"Authorization": f"Token {auth_token}"} # Get execution details url = "https://app.valohai.com/api/v0/executions/{execution_id}/" response = requests.get(url, headers=headers) # Extract custom status custom_status = response.json()["status_detail"] print(f"Current status: {custom_status}") ``` > 🔐 *Keep your API token secure and out of version control. Use environment variables or secret management.* ## Best Practices **Update Strategically:** Don't spam status updates. Set them at meaningful milestones or every N iterations. **Keep It Readable:** Status shows in the UI and API responses. Make it human-friendly. **Handle Failures Gracefully:** Wrap status updates in try/catch blocks so they don't crash your execution. **Use Rich Content Sparingly:** Gauges and charts are great for long processes, but plain text is often clearer for distinct operations. # Private PyPI Repositories Use JFrog Artifactory to host private Python packages and share them securely across your organization with fine-grained access control. This guide shows how to authenticate Valohai executions with your private PyPI repository. The same pattern works for other private PyPI hosts, adjust the authentication method as needed. ## Generate Artifactory Credentials 1. Log into your JFrog Cloud Platform 2. Navigate to **Repositories** 3. Select **Set Up Client/CI Tool** for your PyPI repository 4. Copy the full `index-url` value (starts with `http://` or `https://`) The URL includes embedded credentials and looks like: ``` https://username:password@yourcompany.jfrog.io/artifactory/api/pypi/pypi-local/simple ``` ## Store Credentials in Valohai Save your PyPI credentials securely as environment variables: ### Organization-Level (Recommended) Share credentials across all projects: 1. Navigate to **Hi, \** → **Manage \** 2. Open the **Environment Variables** tab 3. Click **Create new environment variable group** 4. Add variable: * Name: `PRIVATE_PYPI_INDEX_URL` * Value: Paste the full index URL from JFrog * Check **Secret** (click the key icon) 5. Select which projects can access this group 6. Click **Save** See [Environment Variables & Secrets](/user-and-organization-management/getting-started/environment-variables) for detailed instructions. ### Project-Level For project-specific repositories: 1. Open your project 2. Go to **Settings** → **Environment Variables** 3. Add the `PRIVATE_PYPI_INDEX_URL` variable as a secret ## Use in Executions Configure pip to use your private repository before installing packages: ### In valohai.yaml ```yaml - step: name: train-with-private-package image: python:3.12 command: - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL - pip install my-private-package - pip install scikit-learn # Still works for public packages - python train.py ``` The `pip config` command tells pip to check your private repository first. It falls back to public PyPI for packages not in your private registry. ### Install Multiple Private Packages ```yaml - step: name: install-multiple-private image: python:3.12 command: - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL - pip install my-ml-library my-data-utils my-viz-tools - python run_pipeline.py ``` ### Mix Private and Public Packages Pip checks your private repository first, then falls back to PyPI: ```yaml command: - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL - pip install -r requirements.txt ``` **requirements.txt:** ``` my-private-models==1.2.3 # From your Artifactory torch==2.0.0 # From public PyPI transformers==4.30.0 # From public PyPI my-company-utils>=2.0.0 # From your Artifactory ``` ## Alternative: Use Extra Index URL If you want to keep public PyPI as primary and add your private repo as supplementary: ```yaml command: - pip install --extra-index-url $PRIVATE_PYPI_INDEX_URL my-private-package - python train.py ``` This checks public PyPI first, then your private registry. Useful when most packages are public and only a few are private. ## Verify Authentication Test that authentication works: ```yaml - step: name: test-private-repo image: python:3.12 command: - echo "Testing private PyPI access..." - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL - pip install --dry-run my-private-package - echo "Authentication successful" ``` If authentication fails, you'll see 401 or 403 errors in the logs. ## Multiple Private Repositories If you have multiple private PyPI repositories: ```yaml - step: name: multi-repo-install image: python:3.12 command: - pip config set global.index-url $PRIMARY_PYPI_URL - pip config set global.extra-index-url "$SECONDARY_PYPI_URL" - pip install package-from-primary package-from-secondary - python train.py environment-variables: - name: PRIMARY_PYPI_URL default: https://user:pass@repo1.jfrog.io/... - name: SECONDARY_PYPI_URL default: https://user:pass@repo2.jfrog.io/... ``` > Mark repository URLs as secrets to avoid exposing credentials in logs or UI. ## Security Best Practices **Never commit credentials:** Don't put credentials directly in `valohai.yaml` or code. Always use environment variables marked as secrets. **Rotate credentials regularly:** Update the `PRIVATE_PYPI_INDEX_URL` value when rotating JFrog tokens or passwords. **Use scoped tokens:** Configure JFrog to issue read-only tokens for CI/CD access rather than using admin credentials. **Audit access:** Use [Audit Log](https://github.com/valohai/dokuhai/tree/main/user-management/fundamentals/audit-log/README.md) to track which executions accessed private packages. ## Troubleshooting ### 401 Unauthorized **Cause:** Invalid credentials in the index URL. **Fix:** Regenerate credentials in JFrog and update the `PRIVATE_PYPI_INDEX_URL` environment variable. ### 404 Not Found **Cause:** Package doesn't exist in your private repository. **Fix:** Verify the package name and version. Check that it's published to your Artifactory instance. ### Connection Timeout **Cause:** Network connectivity issues or incorrect URL. **Fix:** Test the URL manually: ```shell curl -I $PRIVATE_PYPI_INDEX_URL ``` ### Mixed Public/Private Failures **Cause:** Using `--index-url` completely replaces PyPI, causing public packages to fail. **Fix:** Use `--extra-index-url` to keep both registries: ```shell pip install --extra-index-url $PRIVATE_PYPI_INDEX_URL package-name ``` ## Related Topics * [Environment Variables & Secrets](/user-and-organization-management/getting-started/environment-variables) — Manage credentials organization-wide * [Private Docker Registries](/docker-in-valohai/private-docker-registries) — Pull private Docker images from JFrog * [Custom Docker Images](/docker-in-valohai/building-images) — Bake private packages into images * [JFrog Artifactory Documentation](https://jfrog.com/help/r/jfrog-artifactory-documentation/use-a-pip-configuration-file) — Official pip authentication guide # Service Buttons Service buttons allow you to create additional tabs in your application's interface by writing specially formatted messages to stderr. These tabs can either link to external URLs or embed content directly. ### Basic Usage To display a service button, print a message to stderr in the following format: ``` ::show-service-button::{options} ``` #### Required Options * `url` (string): The URL the button should link to or embed * `name` (string): The display name for the button #### Optional Options * `embed` (boolean): When set to `true`, embeds the URL content inline instead of showing a button. Default: `false` * `style` (string): Alternative way to set embed mode using `style: "embed"` ### Examples #### Basic Button Tab Create a tab that links to an external URL: ```python import sys print('::show-service-button::{"url":"http://example.com/","name":"Example"}', file=sys.stderr) ```
#### Embedded Content Tab Create a tab with embedded content displayed directly: ```python import sys print('::show-service-button::{"url":"https://dashboard.example.com/","name":"Dashboard",embed:true}', file=sys.stderr) ``` Or using the `style` parameter: ```python import sys print('::show-service-button::{"url":"https://dashboard.example.com/","name":"Dashboard",style:"embed"}', file=sys.stderr) ```
### Important Notes * Service button commands **must** be written to **stderr**, not stdout * Each service button creates a new **tab** in the interface * The JSON options should be valid JSON format (use double quotes for strings) * Tabs with `embed: true` will display the content inline, while regular tabs will link to external URLs # Reusable Step Libraries Reusable Step Libraries let you define steps once and use them across all projects in your organization. ## How they differ from project steps **Project steps** are defined in each project's `valohai.yaml`: * Specific to one project * Tied to your codebase * Modified by anyone with project access **Library steps** are defined in separate Git repositories: * Shared across your entire organization * Managed centrally by admins * Reusable without duplication ## When to use libraries Create library steps for workflows you run repeatedly across projects: * Database queries (Redshift, BigQuery, Snowflake) * Image preprocessing * Model evaluation scripts * Docker image builds * Data validation checks * Report generation If you find yourself copying the same step definition into multiple projects, it belongs in a library. ## Two types of libraries ### Valohai Ecosystem Libraries Pre-built steps maintained by Valohai: * Database connectors (AWS Redshift, GCP BigQuery, Snowflake) * Common ML workflows * Infrastructure utilities (Docker Image Builder) See [Using Valohai Ecosystem Libraries](/reusable-step-libraries/ecosystem-libraries) to get started. ### Custom Libraries Your own reusable steps: * Organization-specific workflows * Custom integrations * Internal tooling See [Build Your Own Library](/reusable-step-libraries/build-your-own-library) to create one. ## How it works 1. **Admins** connect Git repositories as libraries 2. **Library YAML** defines steps with metadata (category, icon, description) 3. **All users** see library steps when creating executions 4. **Steps run** just like project steps—same parameters, same environments Library steps appear in the execution creation UI alongside your project steps, organized by category. ## What makes a good library step? **Good candidates:** * Generic and reusable across projects * Well-documented with clear parameters * Stable and rarely changes * Solves a common problem **Poor candidates:** * Highly project-specific logic * Frequently changing implementations * Tightly coupled to one codebase ## Next steps **Use existing libraries:** Start with [Valohai Ecosystem Libraries](/reusable-step-libraries/ecosystem-libraries) for database connectors and common utilities. **Build your own:** Follow [Build Your Own Library](/reusable-step-libraries/build-your-own-library) to create custom steps for your organization. # Using Valohai Ecosystem Libraries Valohai maintains a collection of pre-built library steps for common workflows. These are production-ready and available to all organizations. ## Available libraries ### Database Connectors Run SQL queries and save results to your data store: * [BigQuery Connector](/reusable-step-libraries/ecosystem-libraries/bigquery) * [AWS Redshift Connector](/reusable-step-libraries/ecosystem-libraries/redshift) * [Snowflake Connector](/reusable-step-libraries/ecosystem-libraries/snowflake) All database connectors: * Accept SQL queries as parameters * Save results as CSV outputs * Support both credential-based and machine identity authentication * Automatically version query results in your data store ### Docker Image Builder Build and push Docker images without installing Docker locally: * [Docker Image Builder](/reusable-step-libraries/build-your-own-library/docker-image-builder) Supports AWS ECR, GCP Artifact Registry, and Docker Hub. ## How to use ecosystem libraries Ecosystem libraries are automatically available in your organization. No setup required. ### Run a library step 1. Open your project 2. Click **Create Execution** under the Executions tab 3. Expand the step library by clicking the **+** next to **valohai-ecosystem** in the left panel 4. Select a library step (e.g., `bigquery-query`) 5. Configure parameters and environment variables 6. Click **Create Execution** Library steps run like any other execution—same logs, same outputs, same metadata tracking. ## Example: Query BigQuery Let's run a BigQuery query and save the results: ### Add environment variables Under your project Settings or as an organization-wide environment variable group: * `GCP_PROJECT` — Your GCP project ID * `GCP_IAM` — Set to `1` to use machine identity, or `0` for keyfile auth * `GCP_KEYFILE_CONTENTS_JSON` — (If using keyfile) Service account JSON ### Create the execution 1. Select the `bigquery-query` step from **valohai-ecosystem** 2. Write your SQL query in the **query** parameter: ```sql SELECT user_id, COUNT(*) as events FROM `my-project.analytics.events` WHERE date >= '2025-01-01' GROUP BY user_id ORDER BY events DESC LIMIT 100 ``` 3. (Optional) Set an output path like `top_users.csv` 4. (Optional) Add a datum alias like `latest-user-stats` for easy reference 5. Click **Create Execution** The query runs on BigQuery, and results are saved to your data store. Use the output in other executions with `datum://latest-user-stats`. ## Why use ecosystem libraries? **No setup needed:** No YAML to write, no Git repository to manage. Just run. **Battle-tested:** These steps are maintained by Valohai and used across hundreds of organizations. **Versioned results:** Query outputs are automatically tracked and versioned in your data store. **Consistent patterns:** All connectors work the same way—write a query, get a CSV. Easy to learn once and reuse everywhere. ## Next steps **Database connectors:** * [BigQuery](/reusable-step-libraries/ecosystem-libraries/bigquery) * [AWS Redshift](/reusable-step-libraries/ecosystem-libraries/redshift) * [Snowflake](/reusable-step-libraries/ecosystem-libraries/snowflake) **Build custom images:** * [Docker Image Builder](/reusable-step-libraries/build-your-own-library/docker-image-builder) **Create your own:** * [Build Your Own Library](/reusable-step-libraries/build-your-own-library) # BigQuery Connector Run SQL queries on Google Cloud BigQuery and save results to your data store. ## Why use this connector? **Query directly from Valohai:** No need to export data manually. Write SQL, run execution, get CSV output. **Version your queries:** Every query is saved with the execution. Reproduce results months later by checking which query ran when. **Feed downstream jobs:** Query outputs get datum URLs. Use them as inputs in other executions or pipelines. ## Requirements * BigQuery workspace on your GCP account with data. * GCP Service Account with **BigQuery Data Viewer** and **BigQuery User** roles. * Authentication via keyfile or machine identity. ## Authentication options ### Option 1: Machine identity (recommended) If your Valohai workers run on GCP with a service account attached: 1. Attach the service account to your workers (contact Valohai support if needed) 2. Set environment variables: * `GCP_PROJECT`: Your GCP project ID * `GCP_IAM`: Set to `1` ### Option 2: Service account keyfile If not using machine identity: 1. Create a service account with required roles 2. Download the JSON keyfile 3. Set environment variables: * `GCP_PROJECT`: Your GCP project ID * `GCP_IAM`: Set to `0` * `GCP_KEYFILE_CONTENTS_JSON`: Paste entire JSON keyfile (**mark as secret**) ## Add environment variables Environment variables can be added: * **Project-wide**: Project Settings → Environment Variables * **Organization-wide**: Admin users can create environment variable groups that can be passed to several projects. * **Per-execution**: Set when creating the execution We recommend project or organization settings for credentials. ## Run a query 1. Open your project 2. Click **Create Execution** 3. Expand **valohai-ecosystem** → Select `bigquery-query` 4. Configure parameters: * **query**: Your SQL query * **output-path**: (Optional) Output filename, default is `results.csv` * **datum-alias**: (Optional) Alias for easy reference, e.g., `latest-sales-data` 5. Verify environment variables are set 6. Click **Create Execution** ### Example query ```sql SELECT user_id, DATE(event_timestamp) as date, COUNT(*) as events, SUM(revenue) as total_revenue FROM `my-project.analytics.events` WHERE DATE(event_timestamp) >= '2025-01-01' GROUP BY user_id, date ORDER BY total_revenue DESC LIMIT 1000 ``` Results are saved as `results.csv` (or your custom output path) and uploaded to your data store. ## Use query results The output of the execution gets a datum URL. Reference it in other executions by the URL directly or by using the datum alias shown in the example below: ```yaml - step: name: train-model image: python:3.11 command: - python train.py inputs: - name: training-data default: datum://latest-sales-data ``` Or use it in a pipeline by passing the execution output to the next node. ## Troubleshooting ### Authentication fails **Check:** * `GCP_PROJECT` matches your BigQuery project ID * Service account has **BigQuery Data Viewer** and **BigQuery User** roles * If using keyfile: `GCP_KEYFILE_CONTENTS_JSON` contains the full JSON (not just the path) * If using machine identity: Worker service account is properly attached ### Query returns no results BigQuery queries run successfully even if they return zero rows. Check your `WHERE` clauses and table names. ### Timeout on large queries BigQuery has query execution limits. For very large datasets, consider: * Adding `LIMIT` clauses for testing * Breaking queries into smaller chunks * Using BigQuery's table partitioning ## Next steps **Other database connectors:** * [AWS Redshift](/reusable-step-libraries/ecosystem-libraries/redshift) * [Snowflake](/reusable-step-libraries/ecosystem-libraries/snowflake) **Build your own:** * [Create custom library steps](/reusable-step-libraries/build-your-own-library) # AWS Redshift Connector Run SQL queries on AWS Redshift and save results to your data store. ## Why use this connector? **Query directly from Valohai:** No need to export data manually. Write SQL, run execution, get CSV output. **Version your queries:** Every query is saved with the execution. Reproduce results months later by checking which query ran when. **Feed downstream jobs:** Query outputs get datum URLs. Use them as inputs in other executions or pipelines. ## Requirements * Redshift cluster on your AWS account * Cluster security group allows connections from `valohai-sg-workers` * Authentication via IAM role or username/password ## Authentication options ### Option 1: IAM role (recommended) If your Valohai workers run on AWS with IAM roles: 1. Attach the policy below to `ValohaiWorkerRole` (or your worker role): ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "GetRedshiftCredentials", "Effect": "Allow", "Action": "redshift:GetClusterCredentials", "Resource": "*" } ] } ``` 2. Set environment variables: * `RSCLUSTERIDENTIFIER`: Redshift cluster identifier * `RSDATABASE`: Database name * `RSHOST`: Cluster endpoint (e.g., `my-cluster.abc123.us-east-1.redshift.amazonaws.com`) * `RSREGION`: AWS region (e.g., `us-east-1`) * `RSIAM`: Set to `1` * `RSPORT`: (Optional) Default is `5439` ### Option 2: Username and password If not using IAM roles: 1. Set environment variables: * `RSCLUSTERIDENTIFIER`: Redshift cluster identifier * `RSDATABASE`: Database name * `RSHOST`: Cluster endpoint * `RSREGION`: AWS region * `RSIAM`: Set to `0` * `RSUSER`: Redshift username * `RSPASSWORD`: Redshift password (**mark as secret**) * `RSPORT`: (Optional) Default is `5439` ## Add environment variables Environment variables can be added: * **Project-wide**: Project Settings → Environment Variables * **Organization-wide**: Admin users can create environment variable groups that can be passed to several projects. * **Per-execution**: Set when creating the execution We recommend project or organization settings for credentials. ## Run a query 1. Open your project 2. Click **Create Execution** 3. Expand **valohai-ecosystem** → Select `redshift-query` 4. Configure parameters: * **query**: Your SQL query * **output-path**: (Optional) Output filename, default is `results.csv` * **datum-alias**: (Optional) Alias for easy reference, e.g., `latest-orders` 5. Verify environment variables are set 6. Click **Create Execution** ### Example query ```sql SELECT customer_id, product_category, SUM(order_total) as total_spent, COUNT(*) as order_count FROM orders WHERE order_date >= '2025-01-01' GROUP BY customer_id, product_category ORDER BY total_spent DESC LIMIT 1000 ``` Results are saved as `results.csv` (or your custom output path) and uploaded to your data store. ## Use query results The output of the execution gets a datum URL. Reference it in other executions by the URL directly or by using the datum alias shown in the example below: ```yaml - step: name: train-model image: python:3.11 command: - python train.py inputs: - name: training-data default: datum://latest-orders ``` Or use it in a pipeline by passing the execution output to the next node. ## Troubleshooting ### Connection refused **Check:** * Redshift cluster security group allows connections from `valohai-sg-workers` * `RSHOST` includes the full cluster endpoint (not just the identifier) * `RSPORT` is correct (default: `5439`) ### Authentication fails **If using IAM (`RSIAM=1`):** * Verify `ValohaiWorkerRole` has `redshift:GetClusterCredentials` permission * Check that worker role is properly attached to your workers **If using username/password (`RSIAM=0`):** * Verify `RSUSER` and `RSPASSWORD` are correct * Ensure password is marked as a secret in Valohai ### Query returns no results Redshift queries run successfully even if they return zero rows. Check your `WHERE` clauses and table names. ## Next steps **Other database connectors:** * [BigQuery](/reusable-step-libraries/ecosystem-libraries/bigquery) * [Snowflake](/reusable-step-libraries/ecosystem-libraries/snowflake) **Build your own:** * [Create custom library steps](/reusable-step-libraries/build-your-own-library) # Snowflake Connector Run SQL queries on Snowflake and save results to your data store. ## Why use this connector? **Query directly from Valohai:** No need to export data manually. Write SQL, run execution, get CSV output. **Version your queries:** Every query is saved with the execution. Reproduce results months later by checking which query ran when. **Feed downstream jobs:** Query outputs get datum URLs. Use them as inputs in other executions or pipelines. ## Requirements * Organization and account in Snowflake * Existing database with data * Firewall settings allow Valohai workers to access the database * User with permissions to query the database ## Authentication Snowflake uses username and password authentication. Set these environment variables: * `SNOWSQL_ACCOUNT`: Your Snowflake account in format `-` * `SNOWSQL_WAREHOUSE`: Warehouse name * `SNOWSQL_DATABASE`: Database name * `SNOWSQL_SCHEMA`: (Optional) Schema name, default is `PUBLIC` * `SNOWSQL_USER`: Snowflake username * `SNOWSQL_PWD`: Snowflake password (**mark as a secret**) ## Add environment variables Environment variables can be added: * **Project-wide**: Project Settings → Environment Variables * **Organization-wide**: Admin users can create environment variable groups that can be passed to several projects. * **Per-execution**: Set when creating the execution We recommend project or organization settings for credentials. ## Run a query 1. Open your project 2. Click **Create Execution** 3. Expand **valohai-ecosystem** → Select `snowflake-query` 4. Configure parameters: * **query**: Your SQL query * **output-path**: (Optional) Output filename, default is `results.csv` * **datum-alias**: (Optional) Alias for easy reference, e.g., `latest-sales` 5. Verify environment variables are set 6. Click **Create Execution** ### Example query ```sql SELECT region, product_line, SUM(revenue) as total_revenue, AVG(profit_margin) as avg_margin FROM sales_data WHERE sale_date >= '2025-01-01' GROUP BY region, product_line ORDER BY total_revenue DESC LIMIT 500 ``` Results are saved as `results.csv` (or your custom output path) and uploaded to your data store. ## Use query results The output gets a datum URL. Reference it in other executions: ```yaml - step: name: analyze-sales image: python:3.11 command: - python analyze.py inputs: - name: sales-data default: datum://latest-sales ``` Or use it in a pipeline by passing the execution output to the next node. ## Troubleshooting ### Connection fails **Check:** * `SNOWSQL_ACCOUNT` format is correct: `-` * Firewall rules allow Valohai workers to connect * Warehouse is running (not suspended) ### Authentication fails **Check:** * `SNOWSQL_USER` and `SNOWSQL_PWD` are correct * User has access to the specified database and warehouse * Password is marked as a secret in Valohai ### Query fails with permission error **Check:** * User has `SELECT` permission on the tables * Schema name is correct (default is `PUBLIC`) * Database and warehouse names match your Snowflake setup ## Next steps **Other database connectors:** * [BigQuery](/reusable-step-libraries/ecosystem-libraries/bigquery) * [AWS Redshift](/reusable-step-libraries/ecosystem-libraries/redshift) **Build your own:** * [Create custom library steps](/reusable-step-libraries/build-your-own-library) # Build Your Own Library Library steps refer to customizable and reusable executions that can be incorporated into Valohai projects. These steps can be either predefined by Valohai’s Ecosystem or custom ones created by users. Unlike project-specific steps defined in each project’s own `valohai.yaml` file, library steps are shared across an entire organization. This articles shows you how to add custom library steps from your public and private Git repositories. For instructions on how to build the library step, see the [Docker Image builder from the valohai-toolkit repository](https://github.com/valohai/valohai-toolkit/tree/master/docker_image_builder). ### Requirements * A Git repository containing a `valohai.yaml` with at least one step and all the other required code files. * Admin account in your Valohai organization. ### Add a Git-repository with library steps * Click on `Hi, !` in the top-right corner. * Select `Manage `. * Open the Libraries tab. * Click on the Add a new Library button. * Name your library. Note that the name can only consist of letters, numbers, underscores or hyphens.
#### Public Repositories * Add **HTTPS URL of the public repository** in the respective field. * The URL looks like `git+https://github.com//.git` #### Private Repositories * Add SSH URL of the private repository in the respective field. * The URL looks like `git@github.com:/.git` * Add the SSH private key in the field. * You can either use the Generate new SSH key button in Valohai or generate the key on your command line with the following command ```shell ssh-keygen -t rsa -b 4096 -f my-project-deploy-key ``` * Click on the **Add Library** button. * Click on the **Update** button to fetch the commits. You will need to add the public key as a deploy key for your repository, see the instructions for different hosting services under the [Git integration](/git-integration) section.
# Example: Docker Image Builder This example shows you how to create a custom library steps that take in a Dockerfile, use it to build a docker image and finally push the image to a private registry. The files used in the example are available in the public [valohai-toolkit Github repository](https://github.com/valohai/valohai-toolkit). There are three different steps for different image registries (AWS ECR, GCP Artifact Registry and Docker Hub). ## Why use this? **No local Docker setup:** Write a Dockerfile, run an execution, get a pushed image. No Docker daemon, no local builds. **For teams without Docker experience:** Data scientists focus on writing Dockerfiles. Valohai handles authentication and pushing. **For CI/CD pipelines:** Build images as part of your workflow without managing Docker infrastructure. ## Available registry options The `valohai-toolkit` library includes builders for: * **AWS ECR**: `docker-image-aws` * **GCP Artifact Registry**: `docker-image-gcp` * **Docker Hub**: `docker-image-dockerhub` Source: [github.com/valohai/valohai-toolkit](https://github.com/valohai/valohai-toolkit) ## Requirements * A Git repository containing a valohai.yaml with at least one step and all the other required code files. * A Dockerfile (provided as an input file or a parameter) * Docker registry credentials (environment variables). This example covers three use cases for private registries: * Docker Hub login with credentials * AWS ECR login with machine identity * GCP Artifact Registry login with service account key * Valohai environment configured with `--privileged` flag > The `--privileged` flag is required to run Docker-in-Docker and it will give the containers root capabilities. Contact Valohai support (`support@valohai.com`) if you need this enabled. ## Connect the library 1. Click `Hi, ` (top-right) → `Manage ` 2. Open the **Libraries** tab 3. Click **Add a new Library** 4. Configure: * **Name**: `valohai-toolkit` * **URL**: `git+https://github.com/valohai/valohai-toolkit.git` 5. Click **Add Library** 6. Click **Update** to fetch commits The steps defined in the `valohai.yaml` file are now available under the Create Execution page of all your projects! ## Build and push an image to a Docker registry After you’ve connected the repository as a Library, you can follow the steps below to build and push an image to your Docker registry. * Click on the Create Execution button on the Executions tab of your project and choose the step you want to run from the custom library. * Configure the Dockerfile (input or parameter) and adjust other parameters. * Include the needed environmental variables for authentication as explained below.
### Define your Dockerfile You can provide the Dockerfile in two ways: #### Option 1: As an input file Upload your Dockerfile to your data store and reference it as an input when creating the execution. #### Option 2: As a parameter Write the Dockerfile directly in the Valohai UI using the `dockerfile` parameter. **Default Dockerfile** (if not specified): ```dockerfile FROM python:3.9 RUN pip install valohai-utils tensorflow==2.6.0 ``` This creates a basic image with Python 3.9, valohai-utils, and TensorFlow. ## Set parameters Configure these parameters when creating the execution: * **repository**: Repository name, e.g., `username/myimage` or `mycompany.azurecr.io/ml-training` * **docker-tag**: Image tag, e.g., `v1.0` or `latest` Example: * `repository=mycompany/ml-image` * `docker-tag=v2.0` This builds and pushes `mycompany/ml-image:v2.0`. ## Environment variables and authentication Depending on which registry you are using, you will need to provide certain environment variables. You can define these either on the execution, project or organization level (admin users only) depending on the use case. You will also need to make sure the Valohai worker running the job can authenticate to the Docker registry for pushing the image. ### AWS ECR Define the following environment variables: * `AWS_ECR_ACCOUNT_ID`: Your AWS account ID * `AWS_ECR_REGION`: AWS region (e.g., `us-east-1`) For authentication, it is possible to use the machine identity to provide access to AWS ECR. This way you do not need to save the credentials in environment variables. We recommend, that you create a separate role for in your AWS account for the environment type that is allowed to push imaged to your ECR, e.g. `ValohaiWorkerRoleECR`. **Required IAM policy:** ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ListImagesInRepository", "Effect": "Allow", "Action": [ "ecr:ListImages" ], "Resource": "arn:aws:ecr:::repository/*" }, { "Sid": "ManageRepositoryContents", "Effect": "Allow", "Action": [ "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:GetRepositoryPolicy", "ecr:DescribeRepositories", "ecr:ListImages", "ecr:DescribeImages", "ecr:BatchGetImage", "ecr:InitiateLayerUpload", "ecr:UploadLayerPart", "ecr:CompleteLayerUpload", "ecr:PutImage", "ecr:UntagResource", "ecr:CreateRepository", "ecr:TagResource" ], "Resource": "arn:aws:ecr:::repository/*" }, { "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" } ] } ``` Add this policy to `ValohaiWorkerRoleECR` . You should also attach the existing `ValohaiWorkerPolicy` to the new role. If you created the new `ValohaiWorkerRoleECR` , make sure to add it to the `ValohaiMaster` role in your AWS Account. ```json { "Sid": "4", "Effect": "Allow", "Action": [ "iam:PassRole", "iam:GetRole" ], "Resource": [ "arn:aws:iam:::role/ValohaiWorkerRole", "arn:aws:iam:::role/ValohaiWorkerRoleECR" ] } ``` ### GCP Artifact Registry Define the following environment variables: * `GCP_PROJECT_ID`: Your GCP project ID * `GCP_REGION`: GCP region (e.g., `us-central1`) * `GCP_NEW_REPOSITORY`: (Optional) Set to create a new repository In GCP you can use service accounts to push Docker images to the Artifact Registry. You can read more about the permissions from [GCP documentation](https://cloud.google.com/artifact-registry/docs/access-control#permissions). * Create a new service account under your Google Cloud project that contains the registry. * Add `Artifact Registry Repository Administrator` role so it can push Docker images to your private repository and also create new repositories. Finally, inform Valohai Customer Team () about the new role name so that they can set up the environment for you. ### Docker Hub Define the following environment variables: * `DHUSERNAME`: Docker Hub username * `DHPASSWORD`: Docker Hub password or access token (**mark as a secret**) The environment variables are are used for authentication and no additional steps are needed. ## Run the build 1. Open your project 2. Click **Create Execution** 3. Expand **valohai-toolkit** → Select your registry step: * `docker-image-aws` for AWS ECR * `docker-image-gcp` for GCP Artifact Registry * `docker-image-dockerhub` for Docker Hub 4. Configure: * **Dockerfile**: Provide as an input or as a parameter * **repository**: Image repository name * **docker-tag**: Image tag 5. Add environment variables (if not already set project-wide) 6. Click **Create Execution** The execution builds the image and pushes it to your registry. ## Use the built image After the build succeeds, reference the image in your `valohai.yaml`: ```yaml - step: name: train image: mycompany.azurecr.io/ml-training:v2.0 command: - python train.py ``` If using a private registry, ensure you've configured [registry authentication](/docker-in-valohai/private-docker-registries). # Pipelines Valohai pipelines transform complex ML workflows into modular, reusable components. Instead of running monolithic scripts, break your work into steps that can be versioned, reused, and optimized independently. ### Why use pipelines? #### Automatic checkpointing between steps Each pipeline step runs as a separate execution, creating natural checkpoints. When something fails, you don't lose hours of computation, just restart from the last successful step. #### Efficient resource allocation Different steps need different resources. Your data preprocessing might need high CPU and memory, while training needs GPUs. Pipelines let you specify exact requirements per step, freeing up expensive resources when they're not needed. #### Reuse previous work Made a code change to step 4 of a 6-step pipeline? Use the "reuse nodes" capability to skip steps 1-3 and start directly from your fix. No more waiting for preprocessing to finish again. #### Built for experimentation and production Pipelines aren't just for production workflows. During experimentation: * Benchmark multiple models against different datasets in parallel * Add conditional logic to explore different paths based on results * Pause for human approval before expensive training steps * Run hyperparameter tuning as a pipeline after data processing has been completed ### Core concepts #### Nodes Individual jobs within your pipeline: * **Executions**: Standard Valohai executions running your code * **Tasks**: Collections of executions with the same code but different parameters/data (perfect for hyperparameter tuning or benchmarking models/datasets) * **Deployments**: Create new model endpoints as part of your workflow #### Edges Connections that pass data between nodes: * **Output → Input**: Files produced by one node become inputs for the next * **Input → Input**: Share the same input files across multiple nodes * **Parameters → Parameters**: Pass parameter values between nodes * **Metadata → Parameters**: Use runtime-generated values (like optimal hyperparameters) as parameters to downstream nodes ### When to use pipelines Pipelines excel when you have: * Multi-step workflows where each step has different resource requirements * Long-running processes where failure recovery matters * Workflows you'll run repeatedly (with small variations) * Complex dependencies between different processing stages * Need for conditional execution or human approval steps ### Common patterns #### Training pipeline 1. **Preprocess**: Clean and transform raw data (CPU-intensive) 2. **Train**: Train your model (GPU-intensive) 3. **Evaluate**: Test model performance 4. **Deploy**: Create endpoint if metrics pass threshold #### Experimentation pipeline 1. **Prepare datasets**: Create train/validation/test splits 2. **Hyperparameter search**: Run parallel training jobs with different parameters 3. **Compare results**: Analyze performance across experiments 4. **Select best model**: Automatically identify top performer #### Production pipeline (scheduled) 1. **Fetch new data**: Pull latest data from your warehouse 2. **Validate quality**: Check data integrity and distributions 3. **Retrain model**: Update model with new data 4. **A/B test**: Deploy to staging for comparison 5. **Promote**: Move to production after approval # Run a Pipeline This guide covers the three ways to create and run pipelines in Valohai: through the web interface, command-line, or API. ### Prerequisites Before creating a pipeline, ensure you have: * Individual steps defined and tested in your `valohai.yaml` ### Define a pipeline in valohai.yaml Pipelines are defined in your `valohai.yaml` configuration file. Here's a complete example: ```yaml # Define individual steps first - step: name: preprocess-dataset image: python:3.9 command: - pip install numpy valohai-utils - python ./preprocess_dataset.py inputs: - name: dataset default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - pip install valohai-utils - python ./train_model.py {parameters} parameters: - name: epochs default: 5 type: integer - name: learning_rate default: 0.001 type: float inputs: - name: dataset default: https://valohaidemo.blob.core.windows.net/mnist/preprocessed_mnist.npz - step: name: batch-inference image: tensorflow/tensorflow:2.6.0 command: - pip install pillow valohai-utils - python ./batch_inference.py inputs: - name: model - name: images default: - https://valohaidemo.blob.core.windows.net/mnist/four-inverted.png - https://valohaidemo.blob.core.windows.net/mnist/five-inverted.png - https://valohaidemo.blob.core.windows.net/mnist/five-normal.jpg # Define the pipeline structure - pipeline: name: training-pipeline nodes: - name: preprocess type: execution step: preprocess-dataset - name: train type: execution step: train-model override: inputs: - name: dataset # Replace default inputs with values from the edge - name: evaluate type: execution step: batch-inference edges: - [preprocess.output.preprocessed_mnist.npz, train.input.dataset] - [train.output.model*, evaluate.input.model] ``` #### Key configuration elements **Nodes**: Reference your existing steps and give them names within the pipeline context. **Edges**: Define data flow between nodes using the format: * `[source_node.output.filename, target_node.input.input_name]` * Use wildcards (`*`) to pass multiple files matching a pattern **Override**: Remove default values from inputs that will receive data via edges. ### Create via web interface Once your `valohai.yaml` with the pipeline definition is committed and Valohai has fetched the latest changes, you can create a pipeline run. 1. Navigate to your project 2. Click the **Pipelines** tab 3. Click **Create pipeline** 4. Select your pipeline blueprint from the dropdown (populated from `valohai.yaml`) 5. Review and modify the configuration: * Adjust parameters for any node * Change input URLs if needed * Select different environments or Docker images 6. Click **Create pipeline** to launch > 💡 *The web interface pre-fills configurations from your YAML but allows runtime overrides without changing code.* ### Create via command-line Run a pipeline using the Valohai CLI: ```shell # Run with current directory's valohai.yaml vh pipeline run training-pipeline --adhoc # Run with latest fetched commit in Valohai vh pipeline run training-pipeline ``` #### CLI options * `--adhoc`: Use local uncommitted changes * `--commit`: Specify a Git commit/branch/tag ### Create via API For programmatic pipeline creation, send a POST request to the pipelines endpoint. #### Basic example ```shell curl -X POST https://app.valohai.com/api/v0/pipelines/ \ -H "Authorization: Token YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d @pipeline.json ``` #### Pipeline configuration format The API expects a JSON payload with complete pipeline specification: ```json { "project": "YOUR_PROJECT_ID", "title": "training-pipeline", "nodes": [ { "name": "preprocess", "type": "execution", "template": { "environment": "ENVIRONMENT_ID", "commit": "main", "step": "preprocess-dataset" } }, { "name": "train", "type": "execution", "template": { "commit": "main", "step": "train-model", "parameters": { "epochs": 10, "learning_rate": 0.001 } }, "on_error": "stop-all" } ], "edges": [ { "source_node": "preprocess", "source_key": "preprocessed_mnist.npz", "source_type": "output", "target_node": "train", "target_type": "input", "target_key": "dataset" } ] } ``` > 💡 *Find your project ID and environment IDs via the web interface or API endpoints.* ### Pipeline execution behavior Once created, pipelines execute automatically: * Nodes start when all their input edges are satisfied * Parallel execution occurs when dependencies allow * Failed nodes can be configured to stop the entire pipeline or allow others to continue ### Next steps * Learn about [reusing nodes](/pipelines/reuse-nodes) to avoid re-running successful steps * Explore [conditional execution](/pipelines/dynamic-conditions) for dynamic workflows * Set up [scheduled pipelines](/automation-overview/triggers/scheduled-triggers) for production automation # Configure Resources Per Pipeline Node Optimize your pipeline's performance and cost by allocating appropriate compute resources to each step. Data preprocessing might need high memory, training requires GPUs, and inference runs efficiently on CPUs. ### Why customize resources per node? Consider a typical ML pipeline: 1. **Data preprocessing**: Needs 32GB RAM, multiple CPUs, no GPU 2. **Model training**: Requires 4 GPUs, moderate memory 3. **Model evaluation**: Runs fine on 2 CPUs, minimal memory Running everything on GPU instances wastes money. Running everything on CPU instances makes training impossibly slow. The solution: configure each node's environment independently. ### Resource configuration methods #### 1. Configure in valohai.yaml (recommended) Define environments at the step level for consistent, version-controlled configuration: ```yaml - step: name: preprocess-dataset image: python:3.9 environment: aws-eu-west-1-m5-4xlarge # 16 vCPUs, 64GB RAM, no GPU command: - pip install pandas numpy valohai-utils - python preprocess.py - step: name: train-model image: tensorflow/tensorflow:2.6.0-gpu environment: aws-eu-west-1-p3-8xlarge # 4x V100 GPUs command: - python train.py {parameters} - step: name: evaluate-model image: python:3.9 environment: aws-eu-west-1-t3-medium # 2 vCPUs, 4GB RAM, cost-efficient command: - python evaluate.py ``` > 💡 *Find environment slugs available in your project with `vh environments` in the CLI.* #### 2. Override in pipeline definition When the same step needs different resources in different contexts: ```yaml - pipeline: name: efficient-training nodes: - name: prepare-small type: execution step: preprocess-dataset override: environment: aws-eu-west-1-t3-large # Small dataset = small instance parameters: - name: dataset_size default: "sample" - name: prepare-full type: execution step: preprocess-dataset override: environment: aws-eu-west-1-m5-4xlarge # Full dataset = more memory parameters: - name: dataset_size default: "full" ``` #### 3. Web interface selection For ad-hoc adjustments: 1. Create your pipeline 2. Click on any node in the graph 3. Select environment from the "Runtime" dropdown
### Best practices #### 1. Profile before optimizing Run steps individually to understand resource needs: ```shell # Test with different instance types vh exec run preprocess-dataset --environment aws-eu-west-1-m5-large vh exec run preprocess-dataset --environment aws-eu-west-1-m5-2xlarge ``` Monitor resource usage in the execution details view. The graphs at the top show CPU, memory, and GPU utilization, helping you identify the optimal resource requirements. #### 2. Consider spot/preemptible instances For non-critical steps, use cheaper spot instances: ```yaml environment: aws-eu-west-1-p3-2xlarge-spot # Up to 70% cheaper ``` #### 3. Document resource requirements ```yaml - step: name: train-model # Requirements: 4x V100 GPUs, 32GB+ system RAM, CUDA 11.2+ environment: aws-eu-west-1-p3-8xlarge ``` ### Cost optimization strategies #### Right-size your resources * **Data prep**: High CPU/memory, no GPU * **Training**: GPU instances only for actual training * **Evaluation**: Minimal resources * **Deployment prep**: Standard instances #### Example cost comparison ``` Pipeline with uniform p3.8xlarge (4x V100): ~$12.24/hour Optimized pipeline: - Preprocess on m5.xlarge: ~$0.19/hour - Train on p3.8xlarge: ~$12.24/hour - Evaluate on t3.medium: ~$0.04/hour Total savings: ~40% for typical 3-hour pipeline ``` ### Troubleshooting #### Node fails with SIGKILL (9) or "out of memory" Upgrade to instance with more RAM: ```yaml # Before environment: aws-eu-west-1-m5-large # 8GB RAM ``` ```yaml # After environment: aws-eu-west-1-m5-xlarge # 16GB RAM ``` #### GPU not detected Ensure: 1. Environment has GPUs: Check with `vh environments --details` 2. Docker image supports GPU: Use images that have GPU capabilities 3. Code checks for GPU: `torch.cuda.is_available()` 4. Run `nvidia-smi` as a part of your execution to see if the job has access to a GPU #### Environment not found ```shell # List available environments vh environments # Use exact slug from the list environment: "azure-north-europe-standard-nc6" ``` # Pipeline Parameters Pipeline parameters let you control multiple nodes with a single value. Instead of updating each node individually, define shared parameters that automatically propagate to their targets. ## When to use pipeline parameters Use pipeline parameters when: * Multiple nodes need the same configuration value (e.g., batch size, data version) * You want to experiment with different settings across an entire pipeline * You need centralized control over distributed processing parameters ## Configure in valohai.yaml Define pipeline parameters with their target nodes in your `valohai.yaml`: ```yaml - step: name: preprocess-dataset image: python:3.9 command: - pip install numpy valohai-utils - python ./preprocess_dataset.py parameters: - name: exec_id type: string - name: filters type: string default: ["low-pass"] inputs: - name: dataset default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: - pip install valohai-utils - python ./train_model.py {parameters} parameters: - name: exec_id type: string - name: train_param type: integer default: 5 - pipeline: name: shared-parameters-example parameters: - name: id targets: - preprocess.parameters.exec_id - train.parameters.exec_id - name: training_parameter targets: - train.parameters.train_param default: 3 - name: filters target: preprocess.parameters.filters default: ["remove-outliers", "normalize"] nodes: - name: preprocess step: preprocess-dataset type: execution - name: train step: train-model type: execution - name: train_in_task step: train-model type: task edges: - [preprocess.output.preprocessed_mnist.npz, train.input.dataset] ``` ### Key concepts **Target syntax**: `.parameters.` **Multiple targets**: One pipeline parameter can override multiple node parameters: ```yaml - name: id targets: - preprocess.parameters.exec_id - train.parameters.exec_id ``` **Selective targeting**: Nodes without targets keep their default values. In the example, `train_in_task` uses the step's default `exec_id`, not the pipeline parameter. ## Access in your code Pipeline parameters work exactly like regular parameters in your code: ### Command-line parsing ```python import argparse parser = argparse.ArgumentParser() parser.add_argument("--exec_id", type=str) parser.add_argument("--train_param", type=int, default=5) args = parser.parse_args() ``` ### Python with valohai-utils ```python import valohai # Access the parameter value - doesn't matter if it's pipeline or node parameter exec_id = valohai.parameters("exec_id").value filters = valohai.parameters("filters").value ``` > 💡 *Your code doesn't need to know whether a parameter comes from the pipeline or node level.* ## Multi-value parameters for tasks Pipeline parameters can distribute multiple values across task nodes: ```yaml - pipeline: name: parallel-processing parameters: - name: task_configs target: process_batch.parameters.config_id default: [100, 200, 300, 400] nodes: - name: process_batch step: batch-processor type: task # Creates 4 parallel executions ``` Each value creates a separate execution: * Execution 1: `config_id=100` * Execution 2: `config_id=200` * Execution 3: `config_id=300` * Execution 4: `config_id=400` ## Web interface behavior
In the web interface: 1. **Shared Parameters section** shows all pipeline parameters and their targets 2. **Overridden parameters** appear grayed out in node configurations 3. **Non-targeted parameters** remain editable at the node level 4. **Task nodes** allow multiple value entry for parallel execution ## Common patterns ### Experiment tracking Share a unique ID across all nodes for unified logging: ```yaml - name: experiment_id targets: - preprocess.parameters.exp_id - train.parameters.exp_id - evaluate.parameters.exp_id ``` ### Resource scaling Adjust compute resources uniformly: ```yaml - name: batch_size targets: - preprocess.parameters.batch_size - train.parameters.batch_size default: 32 ``` ## Next steps * Learn about [conditional execution](/pipelines/dynamic-conditions) based on parameter values * Set up [parameter sweeps](/notebook-executions/tune-from-notebook) for hyperparameter optimization # Pass Data Between Pipeline Nodes Pipeline edges define how data flows between nodes. This guide covers the three types of connections: file outputs, parameters, and metadata. ### File outputs to inputs The most common pipeline pattern: one node produces files, the next consumes them. #### Basic file passing ```yaml - step: name: train-model image: tensorflow/tensorflow:2.4.1 command: python train.py - step: name: test-model image: tensorflow/tensorflow:2.4.1 command: python test.py inputs: - name: model - pipeline: name: training-pipeline nodes: - name: train-model step: train-model type: execution - name: test-model step: test-model type: execution edges: - [train-model.output.model.pkl, test-model.input.model] ``` The `train-model` step saves `model.pkl` to `/valohai/outputs/`. The edge passes this file to `test-model` as its `model` input where it'll be available at `/valohai/inputs/model/model.pkl` You can also use files from subdirectories in the edges. For example, if you have a subdirectory called `results` and a file `model.pkl` in it, i.e. path `/valohai/outputs/results/model.pkl`, the edge would look like this: ``` edges: - [train-model.output.results/model.pkl, test-model.input.model] ``` > 💡 *Use wildcards to pass multiple files: `train-model.output.*.pkl` passes all pickle files.. For files under a subdirectory called `results`: `train-model.output.results/*.pkl`* #### Edge merge modes Control how edges interact with default inputs using `edge-merge-mode`: ```yaml - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: python train.py inputs: - name: preprocessed_dataset default: s3://mybucket/preprocessed_data/* - pipeline: name: merge-example nodes: - name: preprocess step: preprocess type: execution - name: train-model step: train-model type: execution edge-merge-mode: append # Add edge files to defaults edges: - [preprocess.output.*, train-model.input.preprocessed_dataset] ``` **Merge modes:** * `replace` (default): Edge data replaces default inputs * `append`: Edge data adds to default inputs ### Parameter passing Share configuration values between nodes without modifying code. #### Static parameter passing Pass a parameter value from one node to another: ```yaml - step: name: train-model image: tensorflow/tensorflow:2.4.1 command: python train.py {parameters} parameters: - name: user-id default: 345345 type: integer - step: name: test-model image: tensorflow/tensorflow:2.4.1 command: python test.py {parameters} parameters: - name: user-id default: 3 type: integer - pipeline: name: parameter-pipeline nodes: - name: train-model step: train-model type: execution - name: test-model step: test-model type: execution edges: - [train-model.parameter.user-id, test-model.parameter.user-id] ``` The `test-model` node inherits the `user-id` value from `train-model`. ### Metadata to parameters Use runtime-generated values to configure downstream nodes. #### Single value metadata Generate metadata in your code: ```python import json # Find optimal learning rate during training optimal_lr = 0.0003 print(json.dumps({"learning_rate": optimal_lr})) ``` Pass it to the next node: ```yaml - pipeline: name: dynamic-parameters nodes: - name: hyperparameter-search step: find-best-params type: execution - name: final-training step: train-model type: execution edges: - [hyperparameter-search.metadata.learning_rate, final-training.parameter.lr] ``` #### Multi-value metadata for tasks Generate multiple values to create parallel task executions: ```python # Generate user IDs for parallel processing user_ids = [463, 674, 888, 233] print(json.dumps({"user": user_ids})) ``` ```yaml - pipeline: name: parallel-processing nodes: - name: generate-ids step: prepare-data type: execution - name: process-users step: process-user type: task # Creates parallel executions edges: - [generate-ids.metadata.user, process-users.parameter.user-id] ``` > 💡 *Tasks create one execution per value in the metadata list. With 4 user IDs, you get 4 parallel executions.* For multi-dimensional parameters: ```python # Each sublist becomes one execution's parameters configs = [[204, 302], [593, 120]] print(json.dumps({"params": configs})) ``` ### Common issues and fixes
Parameter not passed **Symptom:** Downstream node uses default value instead of edge value **Fix:** Verify parameter names match exactly between edge definition and step parameters: ```yaml # Wrong - names don't match edges: - [node1.parameter.userId, node2.parameter.user-id] ``` ```yaml # Correct - exact match edges: - [node1.parameter.user-id, node2.parameter.user-id] ```
Outputs not available **Error:** `FileNotFoundError: /valohai/inputs/model/result.csv` **Fix:** Ensure the upstream node saves files to `/valohai/outputs/`: ```python # Wrong - local directory model.save("output/model.pkl") # Correct - Valohai outputs model.save("/valohai/outputs/model.pkl") ```
Conditional output handling **Issue:** Optional outputs cause downstream failures **Fix:** Add existence checks in consuming nodes: ```python import os model_path = "/valohai/inputs/model/model.pkl" if os.path.exists(model_path): model = load_model(model_path) else: print("No model provided, using defaults") model = create_default_model() ```
Metadata not recognized **Symptom:** Metadata edge doesn't populate parameter **Fix:** Ensure metadata is valid JSON printed to stdout: ```python # Wrong - not JSON print(f"Best accuracy: {accuracy}") # Correct - proper JSON print(json.dumps({"accuracy": accuracy})) ```
### Best practices 1. **Name outputs descriptively**: Use `model.pkl` instead of `output.pkl` 2. **Validate inputs exist**: Always check for file existence in consuming nodes 3. **Log metadata early**: Print metadata as soon as values are determined 4. **Use type-specific nodes**: * `execution` for single runs * `task` for parallel processing with metadata lists ### Next steps * Learn about [conditional execution](/pipelines/dynamic-conditions) based on metadata values * Explore [pipeline scheduling](/automation-overview/triggers/scheduled-triggers) for automated workflows * Set up [pipeline notifications](/automation-overview/triggers/notification-triggers) for status updates # Execution Reuse and Caching Skip redundant computations by reusing results from previous executions. When Valohai detects an identical step configuration, it uses cached results instead of running the step again. ### How execution reuse saves time Consider this scenario: You're iterating on a model architecture, but your 3-hour data preprocessing step hasn't changed. With execution reuse: 1. **First run**: All steps execute normally 2. **Second run** (after model code changes): Preprocessing is skipped, saving 3 hours 3. **Result**: Iterate on model development 5x faster ### When executions are reused Valohai reuses an execution when ALL of these match exactly: * **Source code**: Same Git commit or file contents * **Parameters**: Identical parameter values * **Input data**: Same files (verified by checksums) * **Docker image**: Same container environment * **Step name**: Same step definition If any element differs, the step runs fresh to ensure reproducibility. ### Enable execution reuse #### Method 1: Pipeline-wide in valohai.yaml Enable for all runs of a pipeline: ```yaml - pipeline: name: model-training reuse-executions: true # Enable caching nodes: - name: preprocess type: execution step: preprocess-dataset - name: train type: execution step: train-model - name: evaluate type: execution step: evaluate-model edges: - [preprocess.output.*, train.input.dataset] - [train.output.model, evaluate.input.model] ``` #### Method 2: Per-run in the web interface Toggle reuse for individual pipeline runs:
> 💡 *Use the web interface to temporarily disable reuse when you need fresh results despite unchanged inputs.* ### Practical examples #### Data science iteration workflow ```yaml - pipeline: name: experiment-pipeline reuse-executions: true nodes: # This rarely changes - perfect for reuse - name: fetch-and-clean type: execution step: data-preparation # This might change - but reuse when it doesn't - name: feature-engineering type: execution step: create-features # This changes frequently - but benefits from upstream reuse - name: train-experiment type: execution step: train-model ``` **Reuse pattern**: * Data preparation: Reused 95% of the time * Feature engineering: Reused 70% of the time * Model training: Runs fresh but starts immediately with cached inputs ### Understanding cache behavior #### What triggers a fresh run? Any change to: ```yaml # Parameters parameters: - name: batch_size default: 32 # Changing to 64 = fresh run # Inputs inputs: - name: dataset default: s3://bucket/v1/*.csv # New files = fresh run # Code command: python train.py # Different commit = fresh run # Environment environment: aws-p3-2xlarge # Different instance = fresh run ``` ### Best practices #### 1. Structure pipelines for maximum reuse ```yaml # Good: Separate volatile and stable steps nodes: - name: stable-preprocessing # Changes monthly - name: volatile-training # Changes daily ``` ```yaml # Bad: Combining volatile and stable logic nodes: - name: preprocess-and-train # Any change reruns everything ``` #### 2. Use deterministic operations ```python # Good: Deterministic preprocessing def preprocess(data): return data.sort_values("id").reset_index(drop=True) # Bad: Non-deterministic operations def preprocess(data): return data.sample(frac=0.8) # Random sampling = no reuse ``` #### 3. Version your data explicitly ```yaml inputs: - name: dataset # Good: Versioned data default: s3://bucket/data/v2.1/train.parquet # Bad: Mutable references # default: s3://bucket/data/latest/train.parquet ``` #### 4. Monitor reuse effectiveness In the pipeline view, reused executions show a special indicator. Track reuse rates to optimize pipeline structure. ### Manual execution reuse Besides automatic reuse, you can manually select specific past executions to use as pipeline nodes. This is useful when: * You have a perfect execution from last week you want to reuse * You're building a pipeline incrementally, testing one node at a time * You want to skip expensive steps during development #### Reuse via web interface 1. Click on the Reuse nodes button 2. Select from the **Pipeline from which to reuse** 3. Click the checkboxes on what nodes you want to reuse 4. The node will use that execution's outputs without running again
#### Reuse via API For programmatic pipeline creation, use `reuse_execution_id` instead of a template: ```python import requests import os pipeline_config = { "project": "PROJECT_ID", "title": "experiment-with-reuse", "nodes": [ { "name": "preprocess", "type": "execution", "reuse_execution_id": "exec_123456", # Reuse past execution }, { "name": "train", "type": "execution", "template": { # Run fresh "step": "train-model", "environment": "aws-p3-2xlarge", "commit": "main", }, }, ], "edges": [ ["preprocess.output.*", "train.input.dataset"], ], } response = requests.post( "https://app.valohai.com/api/v0/pipelines/", json=pipeline_config, headers={ "Authorization": f"Token {os.getenv('VH_TOKEN')}", "Content-Type": "application/json", }, ) ``` #### Manual vs automatic reuse | Aspect | Automatic Reuse | Manual Reuse | | --------------- | ---------------------------------------- | --------------------------------------------- | | **When to use** | Iterative development with small changes | Building pipelines with known good executions | | **Selection** | System finds matching execution | You choose specific execution | | **Flexibility** | Based on exact parameter/input match | Use any compatible execution | | **Use case** | "Run this again if nothing changed" | "Use that great run from Tuesday" | # Pipeline Error Handling Control how your pipeline responds to failures. By default, any node failure stops the entire pipeline but you can customize this behavior for more resilient workflows. ### Why customize error handling? Default behavior works for critical paths where every step must succeed. But consider these scenarios: * **Parallel model training**: If 9 out of 10 hyperparameter combinations succeed, you want the best model, not a failed pipeline * **Data quality checks**: Optional validation that shouldn't block core processing * **A/B testing**: One model variant failing shouldn't prevent evaluating others * **Batch processing**: A few failed items shouldn't stop processing thousands of others ### Error handling strategies #### `stop-all` (default) Any failure stops the entire pipeline immediately. ```yaml - pipeline: name: critical-pipeline nodes: - name: validate-data type: execution step: data-validation # on-error: stop-all # implicit ``` Use when: Every step is critical to the final output. #### `continue` Node completes regardless of failures. Downstream nodes still run. ```yaml - name: hyperparameter-search type: task on-error: continue step: train-model ``` Use when: You expect some failures and want to collect all successful results. #### `stop-next` Failed node blocks its dependents but allows parallel branches to continue. ```yaml - name: optional-preprocessing type: execution on-error: stop-next step: enhance-data ``` Use when: This branch is optional, but if it runs, subsequent steps need its output. ### Task node considerations Error handling is especially important for task nodes running parallel executions: ```yaml - pipeline: name: parallel-training parameters: - name: learning_rates target: train-models.parameters.lr default: [0.001, 0.01, 0.1, 1.0] # 1.0 will likely fail nodes: - name: train-models type: task on-error: continue # Don't let one bad LR stop everything step: train-model - name: select-best type: execution step: compare-models edges: - [train-models.output.model*, select-best.input.models] ``` With `on-error: continue`: * 3 models train successfully with reasonable learning rates * 1 fails with `lr=1.0` * `select-best` receives 3 models and picks the best * Pipeline succeeds overall ### Practical example Here's a pipeline that handles failures gracefully: ```yaml - step: name: train-model image: tensorflow/tensorflow:2.6.0 command: python train.py {parameters} parameters: - name: architecture type: string - step: name: evaluate-model image: tensorflow/tensorflow:2.6.0 command: python evaluate.py inputs: - name: model - pipeline: name: model-comparison parameters: - name: architectures target: experimental-models.parameters.architecture default: ["resnet", "efficientnet", "experimental-v1"] nodes: # Prepare data - critical step - name: prepare-data type: execution step: preprocess-dataset # Default on-error: stop-all - data is required # Try proven models - expect success - name: proven-models type: task on-error: stop-next # If these fail, something's wrong step: train-model override: parameters: - name: architecture default: ["resnet", "efficientnet"] # Try experimental model - might fail - name: experimental-models type: task on-error: continue # Don't block pipeline if experimental fails step: train-model # Evaluate all successful models - name: compare-all type: execution step: compare-models edges: - [prepare-data.output.*, proven-models.input.dataset] - [prepare-data.output.*, experimental-models.input.dataset] - [proven-models.output.model*, compare-all.input.proven-models] - [experimental-models.output.model*, compare-all.input.experimental-models] ``` ### Debugging failed pipelines #### View execution logs Check individual execution logs to understand failures: 1. Click on the failed node in the pipeline graph 2. Select the failed execution 3. Review logs for error messages # Debug Pipeline Failures When pipelines fail, quickly identify whether the issue is at the node level (execution failure) or pipeline level (configuration error). This guide covers both types of failures and debugging strategies. ### Understanding pipeline logs Pipelines have two types of logs: #### Node logs Individual execution logs for each step: 1. Click on any node in the pipeline graph 2. View the execution details and logs 3. Check **Logs** tab for error messages #### Pipeline logs System-level logs for the pipeline orchestration: 1. Navigate to the pipeline view 2. Click the **Logs** tab 3. Look for configuration or dependency errors
### Common failure patterns #### Node execution failures **Symptom**: Node shows as "Failed" in red **How to debug**: In the UI: 1. Click the failed node 2. Select "View execution" 3. Check the Logs tab **Common causes**: * Code errors (Python exceptions, import failures) * Out of memory or disk space * Missing dependencies in Docker image * Incorrect file paths #### Pipeline configuration errors **Symptom**: Pipeline fails to start or shows "Crashed" **Pipeline log messages and solutions**: ``` Node "train" transitioned to crashed ``` **Cause**: Execution failed within the node\ **Fix**: Check node's execution logs for the actual error ``` Stopping due to 1 incompletable edges ``` **Cause**: Required inputs missing\ **Fix**: Verify all edges are correctly defined and source nodes produce expected outputs ``` No valid environment found for node ``` **Cause**: Specified environment doesn't exist or user lacks access\ **Fix**: Check environment slug with `vh environments` and verify permissions ### Step-by-step debugging process #### 1. Examine pipeline logs first Look for orchestration issues: * Missing edges * Invalid node references * Parameter mismatches * Environment problems #### 2. Check individual node logs For execution failures: **Via web interface**: 1. Click on the failed node (red) in the pipeline graph 2. Click "View execution" in the popup 3. Navigate to the "Logs" tab 4. Use the log filters to show/hide stdout, stderr, or system logs **Via CLI**: ```shell # View specific node logs vh execution logs EXECUTION_ID # Or download full logs vh execution logs EXECUTION_ID > debug_logs.txt ``` #### 3. Verify data flow Ensure outputs exist and match expected names: ```python # In your code, add debug output import os print("Files in output:", os.listdir("/valohai/outputs")) ``` ### Preventing silent failures #### Problem: Step fails but shows "Completed" By default, Valohai runs all commands even if one fails: ```yaml # Problematic configuration command: - python preprocess.py # Fails - python train.py # Still runs! - python evaluate.py # Also runs ``` #### Solution: Add error handling ```yaml # Fail fast on any error command: - set -e # Exit on first error - python preprocess.py - python train.py - python evaluate.py ``` Or use Python-specific error handling: ```yaml command: - python -u preprocess.py || exit 1 - python -u train.py || exit 1 - python -u evaluate.py || exit 1 ``` ### YAML configuration debugging #### Lint before committing Always validate your `valohai.yaml`: ```shell vh lint # Example error: # error: PipelineParameter.__init__() missing 'targets' ``` #### Common YAML issues **Missing targets**: ```yaml # Wrong parameters: - name: batch_size default: 32 ``` ```yaml # Correct parameters: - name: batch_size targets: - train.parameters.batch_size default: 32 ``` **Incorrect indentation**: ```yaml # Wrong (3 spaces) nodes: - name: train ``` ```yaml # Correct (2 spaces) nodes: - name: train ``` ### Advanced debugging techniques #### 1. Add debug nodes Insert lightweight debug nodes between steps: ```yaml - name: debug-features type: execution step: debug-step command: - ls -la /valohai/inputs/ - head -n 5 /valohai/inputs/features/* - echo "File count: $(ls /valohai/inputs/features | wc -l)" ``` #### 2. Use conditional debugging Add debug output based on parameters: ```python import valohai debug_mode = valohai.parameters("debug").value if debug_mode: print("=== DEBUG: Input shapes ===") print(f"Training data: {X_train.shape}") print(f"First 5 samples:\n{X_train[:5]}") ``` #### 3. Implement checkpoint logging Log progress at key points: ```python import json def log_checkpoint(stage, metrics): checkpoint = { "stage": stage, "timestamp": datetime.now().isoformat(), "metrics": metrics, } print(json.dumps(checkpoint)) # Usage log_checkpoint("preprocessing_complete", {"samples": len(data)}) log_checkpoint("training_started", {"epochs": epochs}) ``` ### Quick reference: Error messages | Error | Location | Likely Cause | Solution | | --------------------------- | ------------- | ---------------------- | ----------------------------------------- | | "No such file or directory" | Node logs | Missing input file | Check edge definitions and output names | | "Out of memory" | Node logs | Insufficient resources | Use larger environment | | "incompletable edges" | Pipeline logs | Missing node outputs | Verify source node completed successfully | | "Module not found" | Node logs | Missing dependency | Add to Docker image or pip install | | "Permission denied" | Node logs | File access issue | Check file permissions in outputs | ### Best practices 1. **Always use `set -e`** in multi-command steps 2. **Validate YAML** before committing with `vh lint` 3. **Log liberally** during development 4. **Name outputs clearly** to avoid edge mismatches 5. **Test nodes individually** before pipeline integration 6. **Use version control** for configurations # Pipeline Conditions and Actions Create intelligent pipelines that adapt to runtime conditions. Stop execution when models underperform, require human approval before deployment, or branch logic based on metrics. ### Why use conditions? Without conditions, pipelines run every step regardless of results. With conditions, you can: * **Prevent bad deployments**: Stop if model accuracy drops below threshold * **Save compute costs**: Skip expensive steps when unnecessary * **Add safety checks**: Require human review before critical operations * **Create dynamic workflows**: Different paths based on data characteristics ### Action structure Every action has three parts: ```yaml actions: - when: node-complete # Trigger event if: metadata.accuracy >= 0.95 # Condition (optional) then: stop-pipeline # Action to take ``` #### When: Trigger events * `node-starting`: Before a node begins execution * `node-complete`: After successful completion * `node-error`: When a node fails #### If: Conditions Conditions compare values using operators: * **Comparison**: `>`, `>=`, `<`, `<=`, `==`, `!=` * **Sources**: `metadata.key`, `parameter.name` * **Values**: Numbers, strings, booleans #### Then: Actions * `stop-pipeline`: Halt entire pipeline execution * `require-approval`: Pause until human approves ### Common patterns #### Quality gates Stop pipeline if model doesn't meet standards: ```yaml - pipeline: name: model-training-with-gates nodes: - name: train type: execution step: train-model actions: - when: node-complete if: metadata.val_accuracy < 0.90 then: stop-pipeline - name: deploy type: execution step: deploy-model ``` The model only deploys if validation accuracy exceeds 90%. #### Human-in-the-loop approval Require manual review before critical operations: ```yaml - pipeline: name: production-deployment nodes: - name: staging-tests type: execution step: run-integration-tests - name: production-deploy type: execution step: deploy-to-production actions: - when: node-starting then: require-approval ``` #### Conditional processing Different actions based on data characteristics: ```yaml - pipeline: name: adaptive-processing nodes: - name: analyze-data type: execution step: data-analysis actions: - when: node-complete if: metadata.sample_count < 1000 then: stop-pipeline # Too few samples - name: train-complex-model type: execution step: train-deep-model ``` #### Multi-condition example Combine multiple conditions for complex logic: ```yaml - pipeline: name: comprehensive-ml-pipeline nodes: - name: preprocess type: execution step: prepare-data actions: - when: node-complete if: metadata.missing_data_pct > 0.3 then: stop-pipeline # Too much missing data - name: train type: execution step: train-model actions: - when: node-complete if: metadata.f1_score < 0.85 then: stop-pipeline # Performance too low - name: validate type: execution step: validate-model actions: - when: node-complete if: metadata.bias_detected == true then: require-approval # Human review for bias - name: deploy type: execution step: deploy-model actions: - when: node-starting then: require-approval # Always approve production deploys ``` ### Working with metadata Generate metadata in your code for conditions: ```python import json # Training script accuracy = model.evaluate(X_test, y_test) print( json.dumps( { "val_accuracy": accuracy, "model_size_mb": model_size / 1024 / 1024, "training_time_minutes": training_time / 60, }, ), ) ``` Use in conditions: ```yaml actions: - when: node-complete if: metadata.model_size_mb > 100 then: require-approval # Review large models ``` ### Handling approvals When a pipeline requires approval: 1. **Email notification** sent to project members 2. **Pipeline pauses** at the approval point 3. **Review interface** shows: * Node outputs and logs * Metrics that triggered approval * Approve/Reject buttons 4. **Decision logged** with timestamp and user ### Best practices #### 1. Log decision context ```python # Help reviewers understand the approval request if requires_manual_review: print("=== APPROVAL REQUIRED ===") print(f"Accuracy: {accuracy:.3f} (threshold: 0.95)") print(f"False positive rate: {fp_rate:.3f}") print(f"Dataset: {dataset_version}") ``` #### 2. Use descriptive metadata keys ```python # Unclear print(json.dumps({"val": 0.87})) # Self-documenting print(json.dumps({"validation_auc_score": 0.87})) ``` ### Common issues #### Condition never triggers **Check metadata is valid JSON**: ```python # Wrong - not JSON print(f"Accuracy: {acc}") # Correct - valid JSON print(json.dumps({"accuracy": acc})) ``` #### Approval emails not sent Ensure project members have notification settings enabled in their profile and the project settings have been configured to send notifications. #### Pipeline stops unexpectedly Add logging before metadata output: ```python metrics = {"accuracy": acc} print(f"DEBUG: Outputting metrics: {metrics}") print(json.dumps(metrics)) ``` # Parallel Runs in a Pipeline # Run multiple pipeline instances in parallel Launch multiple instances of your pipeline, each with different parameter configurations. This pattern scales your ML workflows across multiple contexts: different factories, regions, customers, or any dimension that requires isolated pipeline runs. ### When to use parallel pipelines Parallel pipeline runs solve the multi-context problem: when you need the same workflow executed independently for different scenarios. **Common scenarios:** * **Multi-site deployments**: Train site-specific models (factories, stores, regions) * **Customer-specific models**: Fine-tune base models for different clients * **A/B testing pipelines**: Run competing pipeline configurations side-by-side * **Multi-domain training**: Apply the same pipeline to different data domains > 💡 **Parallel pipelines vs. task nodes**: Use parallel pipelines when each context needs its own complete pipeline. Use [task nodes](/pipelines/parallel-runs-in-a-pipeline/parallel-executions-in-pipeline) when you want parallel execution within a single pipeline. ### Prerequisites Before creating parallel pipeline runs: 1. Define at least one [pipeline parameter](/pipelines/pipeline-parameters) in your `valohai.yaml` 2. Push your pipeline configuration to a Git repository 3. Fetch the latest commit in your Valohai project ### Example: Multi-factory pipeline Here's a pipeline that processes factory-specific data. The `factory_id` parameter creates separate pipeline instances: ```yaml - step: name: preprocess_factory_data image: python:3.10 command: - pip install valohai-utils - python ./preprocess.py parameters: - name: factory_id type: string default: "factory_001" - name: quality_threshold type: float default: 0.95 inputs: - name: raw_data default: s3://data/factories/{parameter:factory_id}/raw/ - step: name: train_quality_model image: python:3.10 command: - pip install valohai-utils - python ./train.py parameters: - name: factory_id type: string default: "factory_001" - name: model_type type: string default: "quality_inspector" inputs: - name: training_data optional: true - pipeline: name: Factory Quality Pipeline parameters: - name: factory_identifier targets: - preprocess.parameters.factory_id - train.parameters.factory_id default: "factory_001" nodes: - name: preprocess step: preprocess_factory_data type: execution - name: train step: train_quality_model type: execution override: inputs: - name: training_data edges: - [preprocess.output.processed_data*, train.input.training_data] ``` ### Create parallel pipelines in the UI Transform your single pipeline into multiple parallel runs with different configurations: #### Step-by-step setup 1. **Navigate to pipelines** * Open your project * Click the **Pipelines** tab * Click **Create Pipeline** 2. **Select your blueprint** * Choose your commit * Select the pipeline with parameters 3. **Configure for parallel execution** * Scroll to **Pipeline Parameters** * Change dropdown from "Single Pipeline" to "Pipeline Task" * Toggle **Variant** to "on" 4. **Define parameter variations** * Enter each parameter value on a new line * Each value creates a separate pipeline instance 5. **Launch the pipelines** * Click **Create Pipeline** * View your Pipeline Task containing all instances #### Configuration example For example, three factory IDs create three independent pipelines: * Pipeline 1: `factory_berlin` * Pipeline 2: `factory_munich` * Pipeline 3: `factory_hamburg` Each pipeline runs the complete workflow with its specific factory context. ### Parameter configuration patterns #### Simple list (one parameter) ``` factory_001 factory_002 factory_003 ``` #### Multiple parameters When you have multiple pipeline parameters, Valohai creates pipelines for every combination: * Parameter 1: `region` → europe, asia * Parameter 2: `model_size` → small, large **Result**: 4 pipelines (europe-small, europe-large, asia-small, asia-large) #### Dynamic parameter usage Use parameters in your steps to create context-specific behavior: ```yaml - step: name: train_step image: python:3.10 command: - pip install valohai-utils - python ./train.py parameters: - name: factory_id type: string default: "factory_001" inputs: - name: dataset default: dataset://{parameter:factory_id}/latest # dataset://factory_001/latest optional: true ``` ### Combine with task nodes You can use both patterns together: parallel pipelines where each pipeline contains task nodes for hyperparameter tuning: ```yaml - pipeline: name: Factory ML Pipeline parameters: - name: factory_id targets: - train.parameters.factory_id nodes: - name: train type: task # Each factory runs hyperparameter tuning step: train_model ``` This creates multiple factory-specific pipelines, each running hyperparameter optimization. # Run parallel executions within a pipeline Use task nodes to run multiple executions in parallel within your pipeline. Perfect for hyperparameter tuning, parameter sweeps, evaluating multiple datasets/models in parallel, or any scenario where you need to explore multiple configurations as part of your ML workflow. ### When to use task nodes Task nodes solve a common ML workflow pattern: running the same step multiple times with different parameters. Instead of creating separate pipelines or manually orchestrating parallel runs, a single task node handles it all. **Common scenarios:** * **Hyperparameter optimization**: Test multiple learning rates, batch sizes, or model architectures * **Cross-validation**: Run k-fold validation as part of your pipeline * **Multi-configuration training**: Train models with different preprocessing options > 💡 **Task node vs. parallel pipelines**: Use task nodes when you want parallel execution *within* a pipeline. Use [parallel pipeline runs](/pipelines/parallel-runs-in-a-pipeline/parallel-pipeline-runs) when you need to run entire pipelines with different configurations. ### How task nodes work A task node spawns multiple executions of the same step, each potentially with different parameter values. All outputs from these executions flow to the next pipeline node. ```yaml - pipeline: name: Training Pipeline nodes: - name: preprocess type: execution step: Preprocess dataset (MNIST) - name: train type: task # This node runs multiple executions step: Train model (MNIST) override: inputs: - name: training-set-images - name: training-set-labels - name: test-set-images - name: test-set-labels - name: evaluate type: execution step: Batch inference (MNIST) edges: - [preprocess.output.*train-images*, train.input.training-set-images] - [preprocess.output.*train-labels*, train.input.training-set-labels] - [preprocess.output.*test-images*, train.input.test-set-images] - [preprocess.output.*test-labels*, train.input.test-set-labels] - [train.output.model*, evaluate.input.model] ``` ### Handle task failures Task nodes need clear rules for handling failures since multiple executions run in parallel. Configure the `on-error` property to control pipeline behavior when executions fail. #### Error handling options | Option | Behavior | Use when | | ----------- | ------------------------------------------------------------ | ---------------------------------------------- | | `stop-all` | Stop the entire pipeline if any execution fails (default) | Every execution must succeed | | `continue` | Continue despite failures if at least one execution succeeds | You expect some parameter combinations to fail | | `stop-next` | Stop downstream nodes but let parallel branches continue | You have independent pipeline branches | #### Example: Robust hyperparameter search This pipeline runs two parallel hyperparameter searches with different failure strategies: ```yaml - pipeline: name: Dual Training Pipeline nodes: - name: preprocess type: execution step: preprocess-dataset - name: train_conservative type: task on-error: stop-next # Fail fast for critical path step: train-model override: inputs: - name: dataset - name: evaluate_conservative type: execution step: batch-inference - name: train_experimental type: task on-error: continue # Allow experimental configs to fail step: train-model override: inputs: - name: dataset - name: evaluate_experimental type: execution step: batch-inference edges: - [preprocess.output.preprocessed_mnist.npz, train_conservative.input.dataset] - [preprocess.output.preprocessed_mnist.npz, train_experimental.input.dataset] - [train_conservative.output.model*, evaluate_conservative.input.model] - [train_experimental.output.model*, evaluate_experimental.input.model] ``` **Result**: Conservative training stops the evaluation if any execution fails. Experimental training continues to evaluation as long as one configuration succeeds. ### Handle task outputs Task nodes pass all outputs to downstream nodes, but there's a critical limitation to understand. **Warning:** If multiple executions produce outputs with identical filenames, only one file (chosen randomly) passes to the next node. #### Best practices for output naming **Avoid:** All executions saving `model.pkl`\ **Better:** Include parameters in filename: `model_lr0.01_batch32.pkl` **Python example with valohai-utils:** ```python import valohai # Get parameter values lr = valohai.parameters("learning_rate").value batch_size = valohai.parameters("batch_size").value # Create unique output filename model_filename = f"model_lr{lr}_batch{batch_size}.pkl" valohai.outputs().live(model_filename) ``` ### Create task nodes in the UI Convert any execution node with parameters into a task node directly in the pipeline builder: 1. Open your project's **Pipelines** tab 2. Click **Create Pipeline** 3. Select your pipeline blueprint 4. Click on any node that has parameters 5. Click **Convert to task** below the graph 6. Configure your parameter grid in the **Parameters** section 7. Click **Create pipeline** # Data Valohai treats data as a first-class citizen. Every file you load or save gets versioned, tracked, and linked to the code that created it — automatically. No more "which dataset did I use for this model?" or "where did this file come from?" Your data has a complete audit trail from raw input to final output. *** ## How Valohai Handles Data ### Everything is Versioned Every file saved from an execution gets a unique `datum://` link that points to an immutable version. Use these links as inputs in future jobs, and Valohai guarantees you'll always get the exact same file. **Example:** ```yaml inputs: - name: training-data default: datum://01234567-89ab-cdef-0123-456789abcdef ``` This single link captures: * The exact file content * When it was created * Which execution produced it * What code and parameters were used ### Cloud Storage Without the Complexity Connect your [S3](/data/configure-data-stores/amazon-s3), [Azure Blob Storage](/data/configure-data-stores/azure-blob-storage), [GCS](/data/configure-data-stores/google-bucket), [OVH Object Storage](/data/configure-data-stores/ovh-object-storage) or [Oracle](/data/configure-data-stores/oracle-bucket-storage) bucket once, after that, files appear as local paths in your code—no boto3, no authentication logic, just `pd.read_csv('/valohai/inputs/dataset/data.csv')`. Valohai handles: * Authentication and credential rotation * Cross-region, cross-cloud transfers and caching * Access control between projects * Metadata and lineage tracking You control: * Where data lives (your cloud account) * Bucket policies and compliance * Cost and retention > :warning: **Platform setup required:** Data scientists can start using data immediately, but platform teams need to [configure data stores](/data/configure-data-stores) first. ### From Files to Datasets Individual files work great for single models or CSVs. But when you're managing hundreds of images or train/validation splits, datasets keep everything organized. **Datasets group related files into versioned collections:** ```yaml # Instead of manually adding 1000 individual files inputs: - name: training-images default: dataset://imagenet/train-v3 # One reference, 1000 files ``` Update all files together, track changes between versions, and use aliases like `production` or `staging` to promote datasets through your workflow. *** ## Working with Data in Valohai ### Save Files from Your Code Write files to `/valohai/outputs/` and Valohai uploads them automatically: ```python # Save any file to outputs import pandas as pd df.to_csv("/valohai/outputs/processed_data.csv") model.save("/valohai/outputs/model.pkl") ``` That's it. No upload logic, no authentication. Files appear in your project's Data tab with full lineage. ### Add Context with Metadata Attach tags, aliases, and custom properties to files for search, filtering, and audit trails: ```python import json metadata = { "valohai.tags": ["validated", "production"], "valohai.alias": "latest-model", "accuracy": 0.95, "training_date": "2024-01-15", } # Save metadata alongside your output with open("/valohai/outputs/model.pkl.metadata.json", "w") as f: json.dump(metadata, f) ``` Now search by tag, reference by alias, or query custom properties through the API. ### Load Files in Your Code Reference data in your YAML, and Valohai downloads it before your code runs: ```yaml - step: name: train image: python:3.9 command: python train.py inputs: - name: dataset default: s3://mybucket/mydata/project-a/data.csv ``` Access it like a local file: ```python import pandas as pd df = pd.read_csv("/valohai/inputs/dataset/data.csv") ``` Supports S3, Azure, GCS, OVH, Oracle or public URLs, `datum://` links from previous executions and Valohai datasets and models (`dataset://` and `model://` links). ### Browse data #### Directory tree Valohai automatically recognizes the directory structure in `/valohai/outputs` and lets you browse the produced data in supported directories. > :bulb: To achieve the structure shown in the screenshot below, execution outputs would have to be saved this way: > > ``` > /valohai/outputs/logs/errors/log_part.1.bin > /valohai/outputs/logs/errors/log_part.2.bin > ... > /valohai/outputs/logs/errors/log_part.12.bin > ``` A collapsible directory tree lets you select the directory to inspect.
In the example above, per directory filtering is applied to the execution outputs but such filter is also available for [Dataset version](/data/datasets) as well as when browsing data within a project. #### Searching Using the search bar, filter datums where the search term matches: * Datum name * One of the tags assigned to the datum * Title of the execution that produced the datum * One of the tags assigned to the execution that produced the datum * Aliases referencing the datum * Datum URI or ID \ By default titles are searched starting with the search term, but an asterisk (\*) or underscored (\_) can be used to turn it into a wildcard expression. * Asterisk (\*) - matches any sequence of characters (including none) * Underscore(\_) - matches exactly one character Using the **datum alias.**
Using the **datum URI.**
Using the **datum name wildcard.**
#### Comparing dataset versions While previewing the dataset, select two versions and then click on **Compare** button to check the differences between the two.
By choosing the comparison mode, you control which datums will be shown.
In the mode selected in the screenshot above, all datums (from both versions) are shown.\ The ones that are removed in later version are highlighted in red while the ones added only in the later version will be highlighted in green (no such datums in the given example). *** ## Data Storage Options ### Cloud Object Storage (Recommended) Configure [S3](/data/configure-data-stores/amazon-s3), [Azure Blob](/data/configure-data-stores/azure-blob-storage), [GCS](/data/configure-data-stores/google-bucket), [OVH ](/data/configure-data-stores/ovh-object-storage)or [Oracle Bucket Storage](/data/configure-data-stores/oracle-bucket-storage) as your primary data store. Best for: * Versioned inputs and outputs * Reproducible pipelines * Audit trails and compliance * Multi-region access [Set up your data store →](/data/configure-data-stores) ### Network Storage (For Shared Data) Mount AWS EFS, Google Filestore, or on-premises NFS when you need: * Shared scratch space across executions * Access to existing on-prem datasets * Shared cache layer for large files or large amount of files (tens or hundreds of thousands) **Trade-off:** Network mounts are fast but could include data that's not tracked by Valohai. Be careful, because use of such data could break reproducibility. [Mount network storage →](/data/data-nfs) ### Databases (For Structured Queries) Query BigQuery, Redshift, or Snowflake directly from executions. Best for: * Pulling training data from data warehouses * Running feature engineering on SQL tables * Joining external datasets [Query databases →](/data/data-databases) *** ## Data Organization Patterns ### For ML Experiments Use[ datasets with aliases](/data/datasets/creating-datasets#dataset-aliases) for environment promotion: ```yaml inputs: - name: training-data default: dataset://customer-churn/production - name: validation-data default: dataset://customer-churn/staging ``` Update aliases when promoting datasets—no code changes needed. ### For Production Pipelines Use `datum://` links with [aliases](/data/data-versioning/metadata-overview/aliases) for immutable references: ```yaml inputs: - name: model-weights default: datum://abc123... # Exact version, always reproducible ``` ### For Large-Scale Data Package files into tar archives before creating datasets: ```python import tarfile with tarfile.open("/valohai/outputs/images.tar", "w") as tar: tar.add("/valohai/outputs/images/", arcname="images") ``` Downloading one 10GB tar is much faster than 100,000 individual files. > :bulb: If packaging is not an option, and you do require a large mount of data, checkout available [caching strategies](/data/data-nfs) that could speed up the access to your data. *** ### When to Use What | Need | Solution | Why | | ---------------------------- | ------------------------------- | ------------------------------- | | Single model file | `datum://` link | Immutable, versioned, traceable | | Training/validation split | Dataset with versions | Files versioned together | | Image classification folders | Dataset with `keep-directories` | Preserves folder structure | | Existing on-prem data | Network mount | Data stays on premises | | Query data warehouse | Database connector | No data movement needed | | Environment promotion | Dataset aliases | Update alias, not code | *** ## Next Steps **For Data Scientists:** 1. [Save file from your code](/data/data-versioning/save-files-from-jobs) to outputs 2. [Load files in your code](/data/data-versioning/load-files-in-jobs) using inputs 3. [Create datasets](/data/datasets/creating-datasets) for multi-file workflows 4. [Add metadata](/data/data-versioning/metadata-overview) for searchability **For Platform Teams:** 1. [Configure cloud storage](/data/configure-data-stores) as your data store 2. [Connect databases](/data/data-databases) for SQL access 3. [Set up network mounts](/data/data-nfs) if needed > 💡 **Platform setup required:** Before data scientists can save files, platform teams must [configure at least one data store](/data/configure-data-stores). # Configure Data Stores To start using Valohai, you will have to configure at least one data store. \\ This store will be used for keeping: * Data, used or produced by the executions * Code that's been used in the executions * Logs produced by the executions Valohai currently supports next stores: * [Amazon S3](/data/configure-data-stores/amazon-s3) * [Azure Blob Storage](/data/configure-data-stores/azure-blob-storage) * [Google Buckets](/data/configure-data-stores/google-bucket) * [OVH Object Storage](/data/configure-data-stores/ovh-object-storage) * [Oracle Bucket Storage](/data/configure-data-stores/oracle-bucket-storage) # Amazon S3 ## Amazon S3 Requirements * an AWS S3 Bucket, with “Block all public access” enabled. * a Valohai organization or project to link the S3 bucket to ### Bucket CORS Settings If you wish to be able to upload files to the store using the **app.valohai.com** web UI, you will need to add a CORS policy document to the S3 bucket.
1. First, you navigate to the AWS S3 bucket you created. 2. Then you go to the Permissions tab and scroll down to Cross-origin resource sharing (CORS). 3. Click Edit to add the rules below: ```json [ { "AllowedHeaders": [ "Authorization" ], "AllowedMethods": [ "GET" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [], "MaxAgeSeconds": 3000 }, { "AllowedHeaders": [ "Authorization" ], "AllowedMethods": [ "POST" ], "AllowedOrigins": [ "https://app.valohai.com" ], "ExposeHeaders": [], "MaxAgeSeconds": 3000 } ] ``` Now your bucket allows POSTs for your user on **** website. ### Create an IAM user Using the AWS console, start creating a new IAM user with programmatic access credentials (access key ID / secret access key).
1. Username can be anything, try to be descriptive. 2. Skip the permission configuration. We will add permissions later; you can skip to the next step. 3. After creating the user, navigate to the security tab and click on “Create Access Key”.
Save your keys Download the CSV or copy-paste the Access key ID and Secret access key somewhere safe.
### Allow the IAM user to access the bucket Now we have a user without any permissions; let’s allow the user to access our new bucket. 1. Find and open the user you created in the previous section. 2. Add a new inline policy. You can use any other AWS IAM policy definition methods just as well. Inline policies are the easiest get started.
The user needs to have full access to the S3 bucket; an example of a suitable access policy document is below. Make sure to change the resource name `my-valohai-bucket`! ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::my-valohai-bucket", "arn:aws:s3:::my-valohai-bucket/*" ] } ] } ```
Give your policy a descriptive name, and we are done with the mandatory AWS setup! ### Large file upload If executions need to upload outputs larger than 5 GB, an additional setup is needed. > :bulb: This is optional and only required for large outputs. To upload large outputs using Amazon’s multipart upload API, a temporary AWS IAM role will be dispensed to the worker machines when required. Be sure to replace the following placeholders in the following policy examples! * `my-valohai-bucket` – the target S3 bucket * `ARN` - The ARN of the IAM user you created above You can find the username and account number by going to IAM -> Users and selecting the user you just created. 1. Copy the ARN of the IAM user you created above 2. Select the Roles tab and create a new AWS IAM Role. 3. Select Custom trust policy.
The Custom trust policy document should look like this: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "ARN" }, "Action": "sts:AssumeRole" } ] } ``` Replace the `ARN` with your own User ARN that you copied previously. Click next. #### Create policy for the role A new tab will open. Select `JSON`. The policy JSON should look like this: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "MultipartAccess", "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:ListBucketVersions", "s3:ListMultipartUploadParts", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::my-valohai-bucket", "arn:aws:s3:::my-valohai-bucket/*" ] } ] } ``` Make sure to change the resource name `my-valohai-bucket` to your own bucket name. * Click Next: Tags * Next: Review. * Name your policy `ValohaiMultipartRole` and click Create policy. * Go back to the tab that has the Role creating open and refresh the page. * Select the policy that you just created by clicking the checkbox in front of it. * Click next from the bottom of the page. * Give your role a descriptive name and click Create role from the bottom of the page. * Take note of the role’s AWS ARN (`arn:aws:...`), that will be configured to your Valohai project. ### Link the store to Valohai You can connect this data store either to a single project or create it on the organization level. The recommended way is to create it under the organization. All the Data Stores added under the organization will be available for all projects. #### Link to a Valohai organization 1. Login at 2. Navigate to `Hi, ` (the top-right menu) > `Manage `. 3. Open the Data Stores tab. 4. Click on Amazon S3 to add a new S3 Data Store The data store can be shared with everyone in the organization, or you can expose the data store only to certain team(s). * **Name:** The name of the Store in Valohai. This can be same as the Bucket name. * **Bucket Name:** Name of the S3 bucket * **IAM Access Key:** and **IAM Secret Access Key:** are the credentials of the IAM user you created * **Region:** The AWS Region you used (e.g. `us-east-1`) * **Multipart Upload IAM Role ARN:** Add the ARN of your `ValohaiMultipartRole` When you create the store, the credentials provided will be checked by creating a small test file in the bucket. The test file is automatically removed after the connection is verified. #### Organization or project level This data store is now available to every project in the organization.\ You can also define project-level data stores by configuring the Data Store under each Project settings. > :bulb: ListObjects error > > One common error that can be seen is the ListObjects error, which happens when the credentials can’t be used to list objects from the S3. \\ > > Check your user’s policy document that is correct - it has to have two items, s3://name and s3://name/\*. > > ``` > ... > "Resource": [ > "arn:aws:s3:::my-valohai-bucket", > "arn:aws:s3:::my-valohai-bucket/*" > ] > ... > ``` # Azure Blob Storage ## Azure Blob Storage On Azure, you create storage accounts that have multiple services attached. One of those services is blob container, which is Valohai’s main interface on Azure-based installations. Requirements * A Microsoft Azure subscription you can administer * A Valohai project which to link the Azure Blob Storage to ### Storage Account and Container Using an existing Azure Blob storage You can skip this part and go directly to the next section if you’re using an existing Storage container. 1. Create an Azure Storage Account in your Microsoft Azure subscription. 2. Select storage account name and location. Create the storage account in the location you’ll be running your work to reduce transfer costs. 3. Click on Containers on the navigation bar on the right side. 4. Click on + Container. 5. Give the container a name like valohai-sample and keep the public access level as Private. 6. Click Create. ### CORS Settings What is CORS? CORS is an HTTP feature that enables a web application running under one domain (**app.valohai.com** for example) to access resources in another domain (your storage). Read more at [Microsoft Docs](https://docs.microsoft.com/en-us/azure/storage/common/storage-cors-support). If you wish to be able to upload files to the store using the **app.valohai.com** web UI, you will need to add a CORS policy document to the blob container. 1. Click on CORS on the navigation bar on the right side. 2. Make sure the Blob service tab is selected. 3. Add the following 2 lines of configuration:
OriginsMethodsAllowed HeadersExposed HeadersMax Age
*GET,OPTIONScontent-type,x-ms-*x-ms-meta-*3000
https://app.valohai.comPOST,PUTcontent-type,x-ms-*x-ms-meta-*3000
Now your blob container allows uploads through **** web application. ### Access Key Using the Azure portal, find and save the access key under the storage account Access keys tab. This will be added to Valohai in the next step. ### Add the store to Valohai You can connect this data store either to a single project, or create it on the organization level. #### Link to a Valohai organization 1. Login at 2. Navigate to Hi, `` (the top-right menu) > Manage ``. 3. Open the Data Stores tab and add your store’s details. 4. The data store can be shared with everyone in the organization, or you can expose the data store to only certain team(s). #### Link the store to a Valohai project Data Stores can be either configured on the project level or shared across your organization. #### Default data store for project 1. Open a project in the web application. 2. Click on the **Data Store** table. 3. Click on **Add Azure Blob Storage Store**. 4. Add in the details you created in the Azure portal. 5. The store name is the name that will be visible on Valohai. Make it something you recognize (e.g. `-store`). 6. (optional) Click on Make project default store to make it the default upload location for this project. 7. When you create the store, the provided access key will be validated. # Google Bucket ### Requirements 1. A Google Cloud Platform project that you can administer. 2. A Valohai project or organization to link Google Storage to. 3. A Google Bucket ### Create a Service Account Next, we’ll create a new service account using the GCP console. The service account is “an account” that Valohai workers use to access this particular GCP bucket.
Navigate to IAM & admin > Service accounts > Create service account.
Name your service account so that you can later remember what it’s meant for (here we are using “my-valohai-bucket-admin”) and press “Create.”
On the next screen, you don’t need to add any roles as we will configure more limited access rights later. Just press “Continue.”
Press the “Create Key” button and select JSON format. This will automatically download a JSON file that we’ll be using later. The resulting JSON file will look something like this: ```json { "type": "...", "project_id": "...", "private_key_id": "...", "private_key": "...", "client_email": "my-valohai-bucket-admin@chubby.iam.gserviceaccount.com", "client_id": "...", "auth_uri": "...", "token_uri": "...", "auth_provider_x509_cert_url": "...", "client_x509_cert_url": "..." } ``` Also, take note of the `client_email` value; we’ll be using that later. You can later find the service account email in the Service Accounts listing:
### Permissions Next, we permit the new service account to access files in the bucket.
Navigate to Storage > Browse > “your-bucket” > Permissions > Add member.
* New members: Copy-and-paste the service account email into the field; it will validate it. We got the service account email in the previous section. * Role: Select “Storage Object Admin,” this allows download and uploading files. * Press the “Save” button. ### CORS * Click on “Activate Google Cloud Shell” in the top-right corner. * Create a new CORS configuration file: ```sh echo '[{"origin": ["*"],"responseHeader": ["Content-Type", "x-ms-*"],"method": ["GET", "HEAD", "OPTIONS"],"maxAgeSeconds": 3600}, {"origin": ["https://app.valohai.com"],"responseHeader": ["Content-Type", "x-ms-*"],"method": ["POST", "PUT"],"maxAgeSeconds": 3600}]' > cors-config.json ``` * Update the CORS settings for your bucket: ```sh gsutil cors set cors-config.json gs:// ``` * Check the CORS settings: ```sh gsutil cors get gs:// ``` ### Link the Store to Valohai You can connect this data store either to a single project or create it on the organization level. #### Link to a Valohai Organization * Navigate to `Hi, ` (the top-right menu) > `Manage ` * Open the “Data Stores” tab and add your store’s details. The data store can be shared with everyone in the organization, or you can expose the data store only to certain teams. #### Link the Store to a Valohai Project 1. Navigate to **Project** -> **Settings** -> **Data Stores** > **Add Google Storage** 2. **Name**: usually makes sense to use the same name as the bucket name. 3. **Bucket**: the bucket name; “my-valohai-bucket” in this example. 4. **Service Account JSON**: copy-and-paste the contents of the JSON file we downloaded earlier. 5. When you create the store, the credentials provided will be checked by creating a small test file in the bucket. If the creation succeeds, you are good to go. 6. Once the data store is linked, you can set it as your project’s default upload store under **Settings** -> **General** -> **Default upload store.** This ensures that uploaded outputs will be stored in this particular GCP bucket by default. # OVH Object Storage ### Requirements 1. An OVH Cloud project that you can administer. 2. A Valohai project or organization to link OVH object Storage to. ### Create OVH Object Storage To set up new object storage within the OVH console, follow these steps: 1. Navigate to your OVH project and select **Object storage** under the **Storage** section.
2. Click on **Create an object container**. 3. Choose **S3 API** as the solution. 4. Select any preferred region.
5. At the **Link User** step, click **Create User**. Be sure to save the credentials generated during this step. 6. Enter a name for your container and then create it.
### Permissions Now, let’s grant the new S3 user access to the files in the bucket: 1. Create a JSON file with the necessary policy using your terminal: ```sh echo '{"Statement":[{"Action":["s3:*"],"Effect":"Allow","Resource":["*"],"Sid":"FullAccess"}]}' > my_policy.json ``` 1. Navigate to the **S3 Users** tab under **Object Storage**, select your user, and choose **Import S3 Policy**.
2. Select the newly created `my_policy.json` file to import the policy. Once the policy is imported we are ready to link this bucket to Valohai. ### Link the Store to Valohai You can connect this data store either to a single project or create it on the organization level. #### Link to a Valohai Organization * Navigate to `Hi, ` (the top-right menu) -> `Manage ` * Open the “Data Stores” tab and add your store’s details. The data store can be shared with everyone in the organization, or you can expose the data store only to certain teams. #### Link the Store to a Valohai Project 1. Navigate to **Project** -> **Settings** -> **Data Stores** -> **Add Amazon S3**. 2. Name: usually makes sense to use the same name as the bucket name. 3. **Bucket**: the bucket name; “ovh-bucket-test” in this example; use the name that you set when creating the object storage in OVH. 4. **Access Keys**: provide the `Access Key` and `Secret Access Key` associated with your S3 user. 5. Set the **region**: check the region in OVH Object Storage, use lowercase, for example if ‘DE’ type ‘de’. 6. Check the box labeled `Use put_object for uploads`. 7. When you create the store, the credentials provided will be checked by creating a small test file in the bucket. If the creation succeeds, you are good to go. 8. Once the data store is linked, you can set it as your project’s default upload store under **Settings** -> **General** -> **Default upload store**. This ensures that uploaded outputs will be stored in this particular OVH storage by default. # Oracle Bucket Storage ### Requirements 1. An Oracle Cloud project that you can administer. 2. A Valohai project or organization to link with the Oracle Bucket. ### Create an Oracle Bucket To set up a new bucket within the Oracle Cloud console, follow this [official Oracle guide](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/managingbuckets_topic-To_create_a_bucket.htm). Once you have a bucket created, make a note of the namespace, this will be used as part of the endpoint URL of the bucket itself. ### Generate access key * Navigate to your Profile (click the top right person icon). * Click My Profile. * Navigate to customer secret keys. * Click generate a secret key. * Insert any name (make it memorable, as you will need to find the IAM ID later). * Click generate a secret key. * Copy the generated key. ### Oracle S3 Endpoint To point to your Oracle bucket, you will need the following information to build the correct endpoint: * Namespace of your bucket. * Region of your bucket. The endpoint will look something like the following:\ `https://NAMESPACE.compat.objectstorage.REGION.oraclecloud.com` ### Link the Store to a Valohai Project 1. Navigate to: **Project** -> **Settings** -> **Data Stores** -> **Add Amazon S3**. 2. **Store Name**: use the same name as the Oracle bucket. 3. **Bucket Name**: use the same name as the Oracle bucket. 4. **IAM Secret Access Key**: paste in the previously generated secret key from Oracle. 5. **IAM Access Key ID**: in Oracle, under `Customer secret keys` find the key you just created. Under the column `Access key`, click the access key ID. Copy and paste this into Valohai. 6. Check the box labeled `Use put_object for uploads`. 7. When you create the store, the credentials provided will be checked by creating a small test file in the bucket. If the creation succeeds, you are good to go. 8. Once the data store is linked, you can set it as your project’s default upload store under **Settings** -> **General** -> **Default upload store**. This ensures that uploaded outputs will be stored in this particular Oracle S3 storage by default. # Data Management Valohai automatically versions every file in your ML workflow, from raw datasets to trained models, ensuring complete reproducibility and traceability without manual intervention. ### Why automatic versioning matters Machine learning teams face three critical data challenges: **Reproducing experiments**: Without proper versioning, recreating a model from 6 months ago becomes impossible. Which exact dataset version was used? What preprocessing was applied? **Tracking data lineage**: Understanding how your production model was created requires tracing through multiple data transformations, from raw images to augmented training sets to the final model artifacts. **Managing dataset iterations**: Image datasets with millions of files evolve constantly. Teams add new samples, fix labels, and create subsets. Manual tracking quickly becomes unmanageable. Valohai solves these challenges by automatically versioning every file that passes through the platform. ### How versioning works Every file in Valohai is immutable and permanently stored. When an execution creates a new output: * Files never overwrite existing versions * Each file gets a unique identifier * Files remain accessible unless explicitly purged This happens automatically when you use Valohai-supported data stores like AWS S3, Azure Blob Storage, Google Cloud Storage, OCI Object Storage or OpenStack Swift. #### Handling massive datasets For projects with millions of files, Valohai offers: * [**Dataset packaging**](/data/datasets/package-datasets): Bundle thousands of files into a single archive for faster job starts * [**On-demand inputs**](/data/data-versioning/dynamic-inputs): Start processing immediately without waiting for all files to download * [**Additional caching layers**](/data/data-nfs)**:** Allow multiple machines to access already downloaded data ### Execution tracking Every execution in Valohai captures complete context for reproducibility. Alongside used **Environment, Code, Parameters and produced Metadata**, Valohai will keep track of every file used or produced by the execution. ### Lineage visualization Trace any file backward and forward through your pipeline to understand: * Which execution created this model? * What datasets were used for training? * Which deployments are using this model?
The trace view handles millions of files efficiently, showing summarized dataset views with drill-down capabilities. ### Organizing with tags and aliases #### Tags Group related models, files, executions for easy filtering: * `experiment-phase-1` * `production-candidate` * `quarterly-report` Tags help teams navigate hundreds of experiments without losing important runs.
#### Aliases Create human-readable pointers to specific file versions: * `production-config` → points to the latest production-ready configuration file * `clean-dataset-v2` → references your latest preprocessed data Aliases automatically version themselves—when you update `production-config`, Valohai keeps the complete history.
### Datasets for complex collections Datasets group related files into versioned collections—perfect for managing training data that evolves over time. **Common use cases:** * Image classification datasets with thousands of photos * Multi-modal data (images + labels + metadata) * Train/validation/test splits #### Smart versioning without duplication When you modify a dataset, adding new files, removing outdated ones, or replacing specific items, Valohai creates a new version without duplicating unchanged files. The platform only stores references to existing files plus any new additions. This means: * Adding 1,000 images to a million-image dataset doesn't duplicate the million * Removing mislabeled samples creates a new clean version without copying data * Multiple dataset versions can share the same underlying files Each dataset modification creates a new version. ### Real-world scenarios #### Rolling back a production model ```shell # Reference a specific model version by alias vh execution run train.py --input model=datum://cat-model-20251005 ``` #### Managing dataset iterations 1. Create initial dataset: `animal-photos-v1` 2. Add new images → automatically becomes `animal-photos-v1.1` 3. Fix mislabeled data → `animal-photos-v1.2` 4. Create subset for experiments → `animal-photos-subset-v1` Each version remains accessible for comparison and rollback. ### Next steps * [Configure your data store](/data/configure-data-stores) to enable automatic versioning * [Create your first dataset](/data/datasets/creating-datasets) for organizing training data * [Work with tags and aliases](/data/data-versioning/metadata-overview) for better organization * [Upload and version files](/data/data-versioning/save-files-from-jobs) programmatically # Save Files from Jobs Valohai will automatically upload files produced by the execution to the desired store, assign them datum IDs and start tracking them. This will then allow you to use these files as inputs of successive executions. *** In order to make Valohai upload the files and start tracking them as datums, all you have to do is move them to the `/valohai/outputs/` directory. Once the execution is completed, everything found in the directory tree starting on this path will be uploaded and tracked. ## Upload file [Create a step](/executions/steps), that will run a python script containing only the code bellow: ```python with open("/valohai/outputs/data.txt", "w") as f: f.write("Hello Valohai") ``` This script will create a file on path `/valohai/outputs/data.txt` that will, once the execution is completed, be uploaded to the desired store. You can inspect this file by navigating to `Outputs` tab of the execution.
### Generate output file path Valohai provides multiple way to generate the path for your output files, without having to remember where they should be saved. #### Using valohai-utils ```python import valohai, json data = {"name": "training_stat", "accuracy": 0} output_path = valohai.outputs().path("stats.json") with open(output_path, "w") as f: json.dump(data, f) ``` If the value passed to the `path()` method contains not only the file name (as shown in the example above) but also one or more subdirectories, `valohai-utils` will make sure to create all those directories and preserve the directory structure once the file is uploaded. ```python import valohai, json data = {"name": "training_stat", "accuracy": 0} output_path = valohai.outputs().path("run_1/epoch_3/stats.json") with open(output_path, "w") as f: json.dump(data, f) ``` Code above will produce such output file:
> :bulb:When using this file as an [input to your execution, ](/data/data-versioning/load-files-in-jobs)you can control if you would like to [preserve this directory structure ](https://github.com/valohai/dokuhai/tree/main/docs/data/data-versioning/broken-reference/README.md)or just download all files in a single directory. #### Using system environment variables If you don't like importing unnecessary libraries, you can just use the environment variables provided by Valohai in every execution environment: ```python import os, json data = {"name": "training_stat", "accuracy": 0} ## In case this code is executed outside of the Valohai environment (e.g on your local ## machine), VH_OUTPUTS_DIR won't be populated and the outputs will be saved in the ## outputs directory in your current directory, e.g 'current_dir/outputs/stats.json' outputs_dir = os.getenv("VH_OUTPUTS_DIR", default="./outputs") output_path = os.path.join(outputs_dir, "stats.json") with open(output_path, "w") as f: json.dump(data, f) ``` > :bulb:Checkout the other [environment variables](/executions/system-environment-variables) and configuration files available in each execution. #### Technology agnostic Most of the examples on this page will be in `Python` but you absolutely can use whatever programming language you like. Here is an example in `R` : ```r # Get the location of Valohai outputs directory vh_outputs_path <- Sys.getenv("VH_OUTPUTS_DIR", unset = "./outputs") # Define a filepath in Valohai outputs directory # e.g. /valohai/outputs/>filename.ext> out_path <- file.path(vh_outputs_path, "mydata.csv") write.csv(output, file = out_path) ``` Or just a one-liner in the shell: ```shell echo '{"name": "training_stat", "accuracy": 0}' > $VH_OUTPUTS_DIR/stats.json ``` ### Destination store If not specified, every execution will inherit a default output store, defined on either Organization or Project level - depending under whose ownership execution was created. Changing the upload store for an execution can be done either through UI when copying the execution:
or by specifying the store ID through `valohai.yaml` when defining step, using the `upload-store` property: ```yaml - step: name: datum_uploader environment: ec2-instance image: python:3.10 upload-store: "0199971a-c953-7bbe-407e-87a1d46e4d5e" command: - python writer.py ``` > :bulb: To obtain the store ID, you can use this [StoreList](https://app.valohai.com/api/docs/#operation/StoreList) API endpoint. > > Instructions and examples on how to use the API can be found on [this page](https://github.com/valohai/dokuhai/tree/main/docs/data/data-versioning/broken-reference/README.md). ### Live uploads If a file, found under the `/valohai/outputs` directory, is marked as read-only, it will be automatically uploaded even before the execution is completed. This allows you to inspect outputs of a long running execution, even before their completion. This would allow you, in case you are not satisfied with the results, to stop the long running execution and start another one with slightly different parameters, that would yield better results. > :warning: After the upload is completed, these files will be deleted from the machine on which the execution is run. Be sure that you code won't have to use these files after they are created! To mark a file as read-only, you can use standard python libraries: ```python import os os.chmod("/valohai/outputs/file_to_upload", 0o444) ``` or `valohai-utils` library: ```python import valohai valohai.outputs().live_upload("output_file.json") ``` or any other method, including basic shell call: ```shell ## These two are equivalent chmod a-w /valohai/outputs/file_to_upload chmod 444 /valohai/outputs/file_to_upload ``` # Load Files in Jobs Valohai inputs are data files, or collections of files, accessible during an execution, fetched from your cloud storage or public sources (HTTP/HTTPS) Supported sources include AWS S3, Azure Storage, GCP Cloud Storage, Oracle buckets, on-prem S3, or any public links. Valohai simplifies data handling: * Manages authentication with your cloud storage. * Handles downloading, unpacking and caching. * Eliminates the need to manage keys or authentication in your code. ## Define an input In your `valohai.yaml`, each step can have one or multiple inputs, each resulting in one or more files. You can set default values for inputs, which can be overridden every time you create an execution.\ For instance, you can change the set of images for batch inference. ```yaml - step: name: image_processor environment: ec2-instance image: python:3.10 command: - python process_image.py inputs: ## Results in one file ## e.g /valohai/inputs/image/ - name: image default: datum://01234567-89ab-cdef-0123-456789abcdef ## Results in a group of files with .jpg extension ## e.g ## /valohai/inputs/image_set/image_1.jpg ## /valohai/inputs/image_set/image_2.jpg ## ... - name: image_set default: s3://mybucket/unprocessed/*.jpg ## As is, will result in no files being download ## Later on you can override this input and make it resolve to one or more files - name: additional_data optional: true ``` ### Select single file To select single file as an input you can use: * **Object store URL** * **Amazon S3**: `s3://{bucket}/{key}` * **Azure Blob Storage**: `azure://{account_name}/{container_name}/{blob_name}` * **Google Storage**: `gs://{bucket}/{key}` * **OpenStack Swift**: `swift://{project}/{container}/{key}` * **Datum URI** * datum://01234567-89ab-cdef-0123-456789abcdef * **Any public http/https URL** * ### Select collection of files #### Wildcards with object store URLs * `s3://my-bucket/dataset/images/*.jpg` for all .jpg (JPEG) files * `s3://my-bucket/dataset/image-sets/**.jpg` for recursing subdirectories for all .jpg (JPEG) files For example, expression above will match: * `s3://my-bucket/dataset/image-sets/cats/big/cat-1.jpg` * `s3://my-bucket/dataset/image-sets/cats/small/cat-9.jpg` * `s3://my-bucket/dataset/image-sets/dog/dog-5.jpg` * `...` > :bulb: **Parameter interpolation** > > You can also interpolate [execution parameters ](/migration-strategy/migrate-job-parameters)into input URIs: > > `s3://my-bucket/dataset/images/{parameter:user-id}/*.jpeg` would replace `{parameter:user-id}` with the value of the parameter `user-id` during an execution. #### Datasets Since [datasets ](/data/datasets)are actually just pointers to the collections of files, using dataset version URL as an input will result in every file included in that version to be downloaded. An example of dataset version URL: `dataset://boats/semi-processed` > :bulb: **Latest version** > > You can always use `latest` as a version, for any dataset, to point to the latest dataset version. > > e.g: `dataset://boats/latest` #### Datum queries Using the UI you can select the files to be downloaded by executing a simple query on datum [properties](/data/data-versioning/metadata-overview/custom-properties).
Query defined in the example above will match all datums that: * have `index` property greater than 10 **and** have `processed_date` property assigned (no matter of its value) **and** have property `type` equals to "dog". * `Limit` field determines the maximum number of files that will be returned by the query (it's optional) **Supported operators**: * `>`, `>=`, `<`, `<=`: Intended for use with numerals. * `==`, `!=`: Applicable to all data types. * `contains`: Used for strings. * `exists`: Checks for property existence, e.g., `property_name exists true` or `property_name exists false`. Numerical values (e.g., 2, 2.1, 1.332) are treated as **numbers**.\ Values: `true`, `false`, `True`, and `False` are treated as **booleans.**\ Everything else is treated as a **string**. > :exclamation: If a property value of a datum is set to `"False"` (or `"True"` - boolean passed as a string) > > ``` > e.g. valohai.metadata.jsonl > {..., "metadata": { "property_one": "False", ... }} > ``` > > such datum will not be matched with the query: `property_one == False` - value passed in the query will be treated as a **boolean** while the one that's actually attached to the datum is of the **string** type. Click `Add metadata filter block` to add multiple metadata query blocks. The final results will be the union of each block's results. > :warning: **Non reproducible** > > Be aware that the results of these queries will be calculated each time you create an execution.\ > If new files are added, with the matching properties, in between the executions, query results **will not be the same** - the more recent one will include newly added files as well! #### Directory structure Without any additional configuration, examples above will result in all files being downloaded in a single directory under `/valohai/inputs//`. From their original path (where they are stored in a store e.g `s3://bucket-name/dir/subdir/file.jpg`) only the file name (basename) will be kept (i.e. `file.jpg`). To preserve the directory structure found in the store, you can use `keep-directories` input property. **keep-directories** can take next values: * **none** (default) all files are downloaded to `/valohai/inputs/myinput` , only the file name is kept from the actual storage path * **full** * When selecting files using a datum URL or dataset version URL, the entire directory structure shown under the `Data` tab for each datum will be maintained. This structure is derived either from subdirectories in `/valohai/outputs/` at upload time (for[ execution output files](/data/data-versioning/save-files-from-jobs)) or from the `Upload path` if the file is [manually uploaded](/data/data-versioning/upload-files-via-web-ui). * When files are selected using store URL or URL wildcards, full path from the storage root is kept and recreated in `/valohai/inputs/`.\ For example `s3://special-bucket/foo/bar/**.jpg` will end up as\ `/valohai/inputs//foo/bar/dataset1/a.jpg` * **suffix** * When selecting files using a datum URL or dataset version URL, behavior is the same as with the **full** option. * When files are selected using store URL, only the filename is preserved.\ For example: `s3://valohai-data/data/output-1389/subdir/file.txt` will result in\ `/valohai/inputs//file.txt` * When files are selected using URL wildcards, directory structure after the wildcard character will be preserved.\ For example `s3://special-bucket/foo/bar/*` the `special-bucket/foo/bar/` would be removed, but any relative path after it would be kept, and you might end up with\ `/valohai/inputs/myinput/dataset1/a.jpg` Example: ```yaml - step: name: image_processor environment: ec2-instance image: python:3.10 command: - python process_image.py inputs: - name: image_set keep-directories: suffix default: s3://mybucket/unprocessed/* ``` ## **Access files** ### Download location All requested data is downloaded to a local directory `/valohai/inputs` on the machine used for execution - no matter if that's a cloud instance or a physical on-prem machine, behavior is the same. Each input has its own directory structure within this location. From the [example above,](#define-an-input) three directories will be created: * `/valohai/inputs/image/` * `/valohai/inputs/image_set/` * `/valohai/inputs/additional_data/`\ :bulb: Only if some value is provided when the execution is created, otherwise, the directory for this input will not be created, as it's marked as `optional` ### Using valohai-utils `valohai-utils` library provides a convenient way of traversing the downloaded files just by specifying the `input` name, e.g: ```python import valohai input_name = "training_data" for file_path in valohai.inputs(input_name).paths(): with open(file_path, "r") as f: content = f.read() ## Do some processing ``` ### Using system environment variables To avoid the use of additional libraries, `VH_INPUTS_DIR` environment variable will be provided in every execution environment, pointing to `/valohai/inputs` . You can then use this path in such way: ```python import os input_name = "training_data" inputs_dir = os.getenv("VH_OUTPUTS_DIR", default="./inputs") inputs_path = os.path.join(inputs_dir, input_name) for file_path in os.listdir(inputs_path): with open(file_path, "r") as f: content = f.read() ## Do some processing ``` ## Inspect selected files Open the **Inputs** tab on any past or running execution to see which datums each input resolved to.
You can also see how each file was selected: * From a [dataset](/data/datasets) (`big_data`) * Through a [datum alias](/data/data-versioning/metadata-overview/aliases) (`config`) * As a single datum (`data_package`) * From a URL (`dist_config`) * From URL [wildcard expansion](#id-4-wildcards) (`external_samples`) * With a [datum query](#datum-queries) (`indexed`) In the example above, `from #114 Pre-process` next to `big-file-4` and `big-file-11` shows the execution counter and the title of the execution that produced those files. Similarly, `latest_config` is the datum alias used to reference that file. # Dynamic Inputs Skip the wait. Start processing terabytes of data while it's still downloading. Instead of downloading your entire dataset before execution starts, on-demand inputs let you stream files during runtime. Perfect for scenarios like processing a 500GB dataset where you only need to analyze the first 10% to determine next steps. > **Prerequisites:** Understand [data concepts](/data) and [getting started with inputs](/migration-strategy/migrate-job-inputs) first. ## When to Use On-Demand Inputs **Use on-demand inputs when:** * Your datasets are hundreds of GBs or larger * You need to start processing immediately without waiting for downloads * You only need partial data from large input sets * You want to stream data analysis in real-time **Skip on-demand inputs for:** * Small datasets (under 1GB) where download time isn't a concern * Workflows that need all data available before starting ## Configuration Mark inputs with `download: on-demand` to enable streaming: ```yaml - step: name: stream-processing image: python:3.12 command: - python ./process_stream.py inputs: # Download of this input will be delayed until the data is explicitly requested - name: large_dataset download: on-demand default: dataset://big-data/latest # This input will be downloaded before the execution starts - name: config default: datum://9122f6fc-2aff-5366-6abd-ffbd7461f1aa ``` Your execution starts immediately! \ Inputs marked as `on-demand` are not yet downloaded, but the metadata describing them is included in the `/valohai/config/inputs.json` file (as well as `/valohai/config/inputs.yaml`). \\ These inputs (marker as `on-demand`) will be download only once (if ever) execution explicitly request them. ## Quick Option: valohai-utils If you're using Python and want automatic download handling, valohai-utils provides a simple interface: ```python import valohai # Downloads all files in the input automatically for filepath in valohai.inputs("large_dataset").paths(): process_file(filepath) ``` **Limitations:** * Downloads every file in the input (no selective downloading) * No custom retry logic * Requires Python environment ## Advanced option: Manual Download API For maximum control over what gets downloaded and when, use the input request API directly. > 💡 Use the manual API for production workflows with large datasets where you need selective downloading. ### 1. Get API Configuration Read `/valohai/config/api.json` to find your execution's unique API endpoint: ```json { "input_request": { "method": "POST", "url": "https://app.valohai.com/request-input-data/?execution={unique-execution-id}", "headers": { "Authorization": "Execution-Token {unique-execution-token}" } } } ``` This token is valid only during your execution's lifetime. ### 2. Find Input IDs Get input metadata from `/valohai/config/inputs.json`: ```json5 { "large_dataset": { // <- input name "input_id": "5c83f6fb-1cb2-4fc3-bda6-2cbd7461f16f", "files": [ // <- list of files this input resolves to { "datum_id": "019a17bd-e035-e6a6-b76e-4f4ea313b46a", "download_intent": "on-demand", "input_id": "5c83f6fb-1cb2-4fc3-bda6-2cbd7461f16f", "metadata": [ // <- custom properties, tags and dataset versions this datum belongs to { "index": 4, "prefix": "linking", "valohai.dataset-versions": ["dataset://big-data/latest"] } ], "name": "dogs/000201.jpg", "path": "/valohai/inputs/large_dataset/dogs/000201.jpg", "size": 1828388, "storage_uri": "https://{store-url}/{bucket-name}/{path-in-bucket}?{aws-signing-specific-properties}", "uri": "datum://019a17bd-e035-e6a6-b76e-4f4ea313b46a" }, // ... <- one object of this kind for each file input resolves to ] } } ``` ### 3. Request Download URLs Make a POST request to get pre-signed download URLs. Filter by input ID to get only what you need: ```shell curl -X POST \ "https://app.valohai.com/request-input-data/?execution={execution-id}&input={input-id}" \ -H "Authorization: Execution-Token {unique-execution-token}" ``` **Response:** ```json [ { "name": "large_dataset", "files": [ { "filename": "dogs/000201.jpg", "original_uri": "datum://019a17bd-e035-e6a6-b76e-4f4ea313b46a", "url": "https://{store-url}/{bucket-name}/{path-in-bucket}?{aws-signing-specific-properties}", "input_id": "5c83f6fb-1cb2-4fc3-bda6-2cbd7461f16f", "metadata": { "...": "..." }, "download_intent": "on-demand" } ] } ] ``` > :bulb: Note that each file description in `/valohai/config/inputs.json` already has a signed URL assigned as `storage_uri` field. You can use this URI to download the file, but keep in mind that these URIs will expire after the set period (configured on the bucket/store level) and will have to be renewed. ### 4. Download Files Selectively Use the `url` field to download only the files you need: ```python import requests, json def get_header(): with open("/valohai/config/api.json", "r") as f: return json.load(f)["input_request"]["headers"] def get_input_id(input_name: str): with open("/valohai/config/inputs.json", "r") as f: return json.load(f)[input_name]["input_id"] def get_url(): with open("/valohai/config/api.json", "r") as f: return json.load(f)["input_request"]["url"] headers = get_header() input_id = get_input_id("large_dataset") url = f"{get_url()}&input={input_id}" response = requests.post(url, headers=headers) ## TODO handle failure in post request response_data = response.json() # Download specific files based on your processing logic for file in response_data[0]["files"]: if should_process_file(file["filename"]): download_file(file["url"], file["filename"]) ``` You control exactly what gets downloaded and when. > :bulb: Pre-signed URLs will eventually expire (exact duration depends on the store configuration). If per-file processing takes a lot of time, it could happen that URLs for still non-processed files will expire. If you start getting such errors when trying to download file, you will have to request pre-signed url again (by using the same URL). ### Authentication Methods **Pre-signed URLs (Default):** Download URLs work immediately without additional authentication. Most secure and convenient for most use cases. **Machine Roles:** For security-conscious environments, configure Valohai to use IAM Instance Roles, Service Accounts, or other machine-based authentication instead of pre-signed URLs. Contact support to configure this option. > :warning: If you are expecting to have more than \~100k input files, **Machine Roles** will be required method of authentication. ### Error Handling Implement your own retry logic for download failures: ```python import time import requests def download_with_retry(url, max_retries=3): for attempt in range(max_retries): try: response = requests.get(url, stream=True) response.raise_for_status() return response except requests.RequestException as e: if attempt == max_retries - 1: raise e time.sleep(2**attempt) # Exponential backoff ``` Network interruptions and temporary failures are common with large file downloads. ## File Lifecycle Valohai will take care of all the data downloaded using it's internal mechanism, this includes: * Keeping downloaded data in cache too speed up successive executions * Removing all unused data * Performing necessary clearing if machine is running out of disk space Limitation of the manual approach of downloading inputs would be that you would have to do the above yourselves. :warning: **Things worth keeping in mind with manual inputs download**: * Take care of the available disk space and, if necessary, remove unused files * Files you manually download will be located in the execution container. If you would like to persist them for the next execution, consider [mounting an external directory](/data/data-nfs) from the host machine or shared network storage. # Add Context to Your Files Your output files shouldn't exist in isolation. Attach experiment details, quality metrics, and production context directly to your files so your team can find, understand, and trust your data. *** ### The Problem Without metadata, files become black boxes: * Which experiment produced this model? * What was the validation accuracy? * Is this the production-ready version? * What preprocessing was applied to this dataset? Tracking this information in spreadsheets, wikis, or README files breaks down as projects scale. Valohai solves this by collecting experiment and lineage metadata automatically, and letting you attach additional context directly to files. *** ### Three Types of Metadata Valohai supports three types of metadata, from simple to sophisticated: #### 1. Tags — Simple Labels Organize and filter files with text labels. **Use for:** Categorization, status tracking, quick filtering **Example:** `["validated", "production", "experiment-42"]` **Learn more:** [Organize Files with Tags](/data/data-versioning/metadata-overview/tags) *** #### 2. Aliases — Stable Pointers Create human-readable shortcuts to specific files that can be updated over time. **Use for:** Production references, "latest" pointers, team coordination **Example:** `datum://model-prod` always points to current production model **Learn more:** [Create File Shortcuts with Aliases](/data/data-versioning/metadata-overview/aliases) *** #### 3. Custom Properties — Rich Data Store any structured data in JSON format. **Use for:** Experiment tracking, quality metrics, production metadata **Example:** `{"accuracy": 0.95, "factory": "EU", "stage": "release"}` **Learn more:** [Track Custom Metadata](/data/data-versioning/metadata-overview/custom-properties) *** ### Quick Comparison | Type | Format | Mutable | Example Use Case | | -------------- | --------------------- | --------------------- | ------------------------------------------------ | | **Tags** | List of strings | Yes | Mark files as "validated" or "production-ready" | | **Aliases** | Single string pointer | Yes (pointer updates) | Point "model-prod" to latest approved model | | **Properties** | Any JSON | Yes | Store `{"accuracy": 0.95, "hyperparams": {...}}` | > :bulb: Tags and aliases are actually special property keys (`valohai.tags` and `valohai.alias`). You can combine all three in the same metadata file. *** ### How to Add Metadata You have four options for adding metadata to your files. Choose based on when you want to add it and how many files you're processing. #### Decision Tree ``` ┌─ Saving 1-2 files? │ └─→ Use sidecar files (.metadata.json) │ ┌─ Saving 3+ files? │ └─→ Use single metadata file (valohai.metadata.jsonl) ← RECOMMENDED │ └─ After execution completes? ├─→ Many files, or automated? Use API └─→ A few files, by hand? Use the web UI ``` *** ### Method 1: Sidecar Files (1-2 Files) Save a `.metadata.json` file alongside each output file. #### Naming Rules (Critical!) The metadata file must have the **exact same name** as your output file, plus `.metadata.json`: ``` Correct: model.pkl → model.pkl.metadata.json data.csv → data.csv.metadata.json results.json → results.json.metadata.json Wrong: model.pkl → model.metadata.json (missing .pkl) model.pkl → metadata.json (missing full filename) data.csv → data.csv.meta.json (wrong extension) ``` #### Python Example ```python import json # Your metadata (tags, alias, and custom properties) metadata = { "valohai.tags": ["validated", "production"], "valohai.alias": "model-prod", "accuracy": 0.95, "epochs": 100, } # Save your output file save_path = "/valohai/outputs/model.pkl" model.save(save_path) # Save metadata file metadata_path = f"{save_path}.metadata.json" with open(metadata_path, "w") as f: json.dump(metadata, f) ``` *** ### Method 2: Single Metadata File (3+ Files) — RECOMMENDED When processing many files, creating individual `.metadata.json` files is tedious. Use one `valohai.metadata.jsonl` file instead. #### Why This Is Better **Without JSONL (tedious):** ``` 100 output files = 200 total files /valohai/outputs/image_001.jpg /valohai/outputs/image_001.jpg.metadata.json /valohai/outputs/image_002.jpg /valohai/outputs/image_002.jpg.metadata.json ... (98 more pairs) ``` **With JSONL (clean):** ``` 100 output files = 101 total files /valohai/outputs/image_001.jpg /valohai/outputs/image_002.jpg ... (98 more images) /valohai/outputs/valohai.metadata.jsonl ← One file for all metadata ``` #### Format Requirements **Filename:** Must be exactly `valohai.metadata.jsonl` **Location:** `/valohai/outputs/valohai.metadata.jsonl` **Format:** JSON Lines (JSONL) — one JSON object per line, newline-separated Each line must have this structure: ```json { "file": "output_filename.ext", "metadata": { "your": "property", "another_property": "value" } } ``` > ⚠️ **Important:** JSONL requires a newline (`\n`) after each JSON object. Missing newlines will cause parsing errors. > ⚠️ If output file is not saved under `/valohai/inputs/file.txt` but instead under one or more subdirectories (e.g `/valohai/inputs/subdir/subdir_2/file.txt`) those have to be included in the value of `file` field inside `valohai.metadata.jsonl as well.` > > e.g. For such file: `/valohai/inputs/subdir/subdir_2/file.txt`\ > Value of `file` should be: `subdir/subdir_2/file.txt` #### Python Example ```python import json # Process many files for i in range(100): # Save output file image_path = f"/valohai/outputs/image_{i:03d}.jpg" processed_image.save(image_path) # Create single metadata file for all outputs metadata_path = "/valohai/outputs/valohai.metadata.jsonl" with open(metadata_path, "w") as f: for i in range(100): metadata_entry = { "file": f"image_{i:03d}.jpg", "metadata": { "quality_score": scores[i], "processing_time": times[i], "valohai.tags": ["processed", "batch-2024-Q1"], }, } json.dump(metadata_entry, f) f.write("\n") # Critical: newline after each entry ``` > :bulb:The `metadata_entry` in the example above is not a single line like in the general example above. Why does this still work? In the example we are using `json.dump` which will actually produce just one line. Note that we are still adding the newline with `f.write("\n")` after each object as `json.dump` will not take care of that. #### Common JSONL Mistakes ```python # Wrong: Missing newlines with open("/valohai/outputs/valohai.metadata.jsonl", "w") as f: json.dump({"file": "file1.jpg", "metadata": {...}}, f) json.dump({"file": "file2.jpg", "metadata": {...}}, f) # No \n! # Correct: Newline after each object with open("/valohai/outputs/valohai.metadata.jsonl", "w") as f: json.dump({"file": "file1.jpg", "metadata": {...}}, f) f.write("\n") json.dump({"file": "file2.jpg", "metadata": {...}}, f) f.write("\n") ``` #### Helper Function Create a reusable helper for your projects: ```python import json def save_metadata_jsonl(file_metadata_dict, output_dir="/valohai/outputs"): """ Save metadata for multiple files in JSONL format. Args: file_metadata_dict: Dict mapping filenames to metadata dicts e.g., {"model.pkl": {"accuracy": 0.95}} """ metadata_path = f"{output_dir}/valohai.metadata.jsonl" with open(metadata_path, "w") as f: for filename, metadata in file_metadata_dict.items(): json.dump({"file": filename, "metadata": metadata}, f) f.write("\n") # Usage file_metadata = { "model.pkl": {"accuracy": 0.95, "valohai.alias": "model-prod"}, "data.csv": {"rows": 10000, "valohai.tags": ["validated"]}, "results.json": {"experiments": 42}, } save_metadata_jsonl(file_metadata) ``` #### With valohai-utils The `valohai-utils` package provides built-in helpers: ```python import valohai with valohai.output_properties() as properties: for i in range(100): filename = f"image_{i:03d}.jpg" # Save output file image.save(valohai.outputs().path(filename)) # Add metadata properties.add( file=filename, properties={ "quality_score": scores[i], "valohai.tags": ["processed"], }, ) ``` *** ### Method 3: API (After Execution) Add or update metadata after execution completes using the Valohai API. Useful for validation workflows, quality gates, or manual approval steps. #### Three API Endpoints | Endpoint | Use When | What It Does | | ---------------------------------- | ---------------------------------- | ---------------------------------------- | | `/api/v0/data/{id}/metadata/` | One file, one metadata set | Apply properties to single datum | | `/api/v0/data/metadata/apply/` | Multiple files, different metadata | Apply different properties to each datum | | `/api/v0/data/metadata/apply-all/` | Multiple files, same metadata | Apply same properties to all datums | #### Quick Example ```python import os import requests properties = { "validation_score": 0.98, "approved_by": "data-team", "valohai.tags": ["validated"], } datum_id = "01234567-89ab-cdef-0123-456789abcdef" response = requests.post( f"https://app.valohai.com/api/v0/data/{datum_id}/metadata/", json=properties, headers={ "Authorization": "Token " + os.getenv("VH_TOKEN"), "Content-Type": "application/json", }, ) ``` *** ### Method 4: The Web UI (After Execution) You can edit properties in the UI by clicking on the `Edit Properties` button (under the listed properties).
Modal dialog with a form that contains four properties, their values and types, and an Add property button
*** ### Reserved Metadata Keys A few keys have special meaning in Valohai:
KeyTypePurposeDetails
valohai.tagsList of stringsCreates tagsTags page
valohai.aliasStringCreates/updates aliasAliases page
valohai.dataset-versionsList of dataset version URLsIncludes this datum in the dataset versionCreate dataset
valohai.model-versionsList of model version URLsIncludes this datum in the model versionCreate and manage Models
contains, icontains, has_key–Reserved names used in property queries
*__*–Internal query separator __ not allowed in property names
*gt, *gte, *lt, *lte–Internal lookup suffixes not allowed in property names
constructor, prototype–Reserved names used in lookups
All other keys are your custom properties. #### Example Combining All Three ```python metadata = { # Reserved Valohai keys "valohai.tags": ["validated", "production", "resnet50"], "valohai.alias": "model-prod", "valohai.dataset-versions": ["dataset://big-data/processed"], # Your custom properties "accuracy": 0.95, "precision": 0.93, "recall": 0.97, "epochs": 100, "learning_rate": 0.001, "dataset_version": "v2.3", "training_duration_minutes": 145, "experiment_id": "exp-042", } ``` *** ### Common Issues & Fixes #### Metadata Not Appearing **For sidecar files:** * Wrong filename → Must be `output.ext.metadata.json` (exact match plus suffix) * Not saved to `/valohai/outputs/` → Save in same directory as output * Invalid JSON → Validate syntax (commas, quotes, brackets) **For JSONL file:** * Wrong filename → Must be exactly `valohai.metadata.jsonl` * Missing newlines → Add `f.write('\n')` after each `json.dump()` * Wrong structure → Each line must have `{"file": "...", "metadata": {...}}` #### "Do I Need .metadata.json for Every File?" **No!** This is the most common confusion. Here's the comparison: **Sidecar approach (1-2 files):** ```python # Good for small number of outputs model.save("/valohai/outputs/model.pkl") with open("/valohai/outputs/model.pkl.metadata.json", "w") as f: json.dump(metadata, f) ``` **JSONL approach (3+ files):** ```python # Much better for many outputs for i in range(100): image.save(f"/valohai/outputs/image_{i}.jpg") # One metadata file for all 100 images with open("/valohai/outputs/valohai.metadata.jsonl", "w") as f: for i in range(100): json.dump({"file": f"image_{i}.jpg", "metadata": {...}}, f) f.write("\n") ``` **Result:** * Sidecar: 100 images = 200 files 😱 * JSONL: 100 images = 101 files 😊 *** ### Next Steps Now that you understand the metadata system, dive into specific use cases: * [**Organize Files with Tags**](/data/data-versioning/metadata-overview/tags) — Label files for filtering and discovery * [**Create File Shortcuts with Aliases**](/data/data-versioning/metadata-overview/aliases) — Set up production references and team coordination * [**Track Custom Metadata**](/data/data-versioning/metadata-overview/custom-properties) — Store experiment results, quality metrics, and more *** ### Related Pages * [Save Files](/data/data-versioning/save-files-from-jobs) — Save outputs that can have metadata * [Versioning and Lineage](/data/data-versioning) — Track file dependencies * [Load Data in Jobs](/data/data-versioning/load-files-in-jobs) — Use files with metadata as inputs --- [Next Page](/llms-full.txt/1)