# Run multiple pipeline instances in parallel

Launch multiple instances of your pipeline, each with different parameter configurations. This pattern scales your ML workflows across multiple contexts: different factories, regions, customers, or any dimension that requires isolated pipeline runs.

### When to use parallel pipelines

Parallel pipeline runs solve the multi-context problem: when you need the same workflow executed independently for different scenarios.

**Common scenarios:**

* **Multi-site deployments**: Train site-specific models (factories, stores, regions)
* **Customer-specific models**: Fine-tune base models for different clients
* **A/B testing pipelines**: Run competing pipeline configurations side-by-side
* **Multi-domain training**: Apply the same pipeline to different data domains

> 💡 **Parallel pipelines vs. task nodes**: Use parallel pipelines when each context needs its own complete pipeline. Use [task nodes](https://docs.valohai.com/pipelines/parallel-runs-in-a-pipeline/parallel-executions-in-pipeline) when you want parallel execution within a single pipeline.

### Prerequisites

Before creating parallel pipeline runs:

1. Define at least one [pipeline parameter](https://docs.valohai.com/pipelines/pipeline-parameters) in your `valohai.yaml`
2. Push your pipeline configuration to a Git repository
3. Fetch the latest commit in your Valohai project

### Example: Multi-factory pipeline

Here's a pipeline that processes factory-specific data. The `factory_id` parameter creates separate pipeline instances:

```yaml
- step:
    name: preprocess_factory_data
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./preprocess.py
    parameters:
      - name: factory_id
        type: string
        default: "factory_001"
      - name: quality_threshold
        type: float
        default: 0.95
    inputs:
      - name: raw_data
        default: s3://data/factories/{parameter:factory_id}/raw/

- step:
    name: train_quality_model
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./train.py
    parameters:
      - name: factory_id
        type: string
        default: "factory_001"
      - name: model_type
        type: string
        default: "quality_inspector"
    inputs:
      - name: training_data
        optional: true

- pipeline:
    name: Factory Quality Pipeline
    parameters:
      - name: factory_identifier
        targets:
          - preprocess.parameters.factory_id
          - train.parameters.factory_id
        default: "factory_001"
    nodes:
      - name: preprocess
        step: preprocess_factory_data
        type: execution
      - name: train
        step: train_quality_model
        type: execution
        override:
          inputs:
            - name: training_data
    edges:
      - [preprocess.output.processed_data*, train.input.training_data]
```

### Create parallel pipelines in the UI

Transform your single pipeline into multiple parallel runs with different configurations:

#### Step-by-step setup

1. **Navigate to pipelines**
   * Open your project
   * Click the **Pipelines** tab
   * Click **Create Pipeline**
2. **Select your blueprint**
   * Choose your commit
   * Select the pipeline with parameters
3. **Configure for parallel execution**
   * Scroll to **Pipeline Parameters**
   * Change dropdown from "Single Pipeline" to "Pipeline Task"
   * Toggle **Variant** to "on"
4. **Define parameter variations**
   * Enter each parameter value on a new line
   * Each value creates a separate pipeline instance
5. **Launch the pipelines**
   * Click **Create Pipeline**
   * View your Pipeline Task containing all instances

#### Configuration example

For example, three factory IDs create three independent pipelines:

* Pipeline 1: `factory_berlin`
* Pipeline 2: `factory_munich`
* Pipeline 3: `factory_hamburg`

Each pipeline runs the complete workflow with its specific factory context.

### Parameter configuration patterns

#### Simple list (one parameter)

```
factory_001
factory_002
factory_003
```

#### Multiple parameters

When you have multiple pipeline parameters, Valohai creates pipelines for every combination:

* Parameter 1: `region` → europe, asia
* Parameter 2: `model_size` → small, large

**Result**: 4 pipelines (europe-small, europe-large, asia-small, asia-large)

#### Dynamic parameter usage

Use parameters in your steps to create context-specific behavior:

```yaml
- step:
    name: train_step
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./train.py
    parameters:
      - name: factory_id
        type: string
        default: "factory_001"
    inputs:
      - name: dataset
        default: dataset://{parameter:factory_id}/latest # dataset://factory_001/latest
        optional: true
```

### Combine with task nodes

You can use both patterns together: parallel pipelines where each pipeline contains task nodes for hyperparameter tuning:

```yaml
- pipeline:
    name: Factory ML Pipeline
    parameters:
      - name: factory_id
        targets:
          - train.parameters.factory_id
    nodes:
      - name: train
        type: task  # Each factory runs hyperparameter tuning
        step: train_model
```

This creates multiple factory-specific pipelines, each running hyperparameter optimization.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.valohai.com/pipelines/parallel-runs-in-a-pipeline/parallel-pipeline-runs.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.