# Run multiple pipeline instances in parallel

Launch multiple instances of your pipeline, each with different parameter configurations. This pattern scales your ML workflows across multiple contexts: different factories, regions, customers, or any dimension that requires isolated pipeline runs.

### When to use parallel pipelines

Parallel pipeline runs solve the multi-context problem: when you need the same workflow executed independently for different scenarios.

**Common scenarios:**

* **Multi-site deployments**: Train site-specific models (factories, stores, regions)
* **Customer-specific models**: Fine-tune base models for different clients
* **A/B testing pipelines**: Run competing pipeline configurations side-by-side
* **Multi-domain training**: Apply the same pipeline to different data domains

> 💡 **Parallel pipelines vs. task nodes**: Use parallel pipelines when each context needs its own complete pipeline. Use [task nodes](https://docs.valohai.com/pipelines/parallel-runs-in-a-pipeline/parallel-executions-in-pipeline) when you want parallel execution within a single pipeline.

### Prerequisites

Before creating parallel pipeline runs:

1. Define at least one [pipeline parameter](https://docs.valohai.com/pipelines/pipeline-parameters) in your `valohai.yaml`
2. Push your pipeline configuration to a Git repository
3. Fetch the latest commit in your Valohai project

### Example: Multi-factory pipeline

Here's a pipeline that processes factory-specific data. The `factory_id` parameter creates separate pipeline instances:

```yaml
- step:
    name: preprocess_factory_data
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./preprocess.py
    parameters:
      - name: factory_id
        type: string
        default: "factory_001"
      - name: quality_threshold
        type: float
        default: 0.95
    inputs:
      - name: raw_data
        default: s3://data/factories/{parameter:factory_id}/raw/

- step:
    name: train_quality_model
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./train.py
    parameters:
      - name: factory_id
        type: string
        default: "factory_001"
      - name: model_type
        type: string
        default: "quality_inspector"
    inputs:
      - name: training_data
        optional: true

- pipeline:
    name: Factory Quality Pipeline
    parameters:
      - name: factory_identifier
        targets:
          - preprocess.parameters.factory_id
          - train.parameters.factory_id
        default: "factory_001"
    nodes:
      - name: preprocess
        step: preprocess_factory_data
        type: execution
      - name: train
        step: train_quality_model
        type: execution
        override:
          inputs:
            - name: training_data
    edges:
      - [preprocess.output.processed_data*, train.input.training_data]
```

### Create parallel pipelines in the UI

Transform your single pipeline into multiple parallel runs with different configurations:

#### Step-by-step setup

1. **Navigate to pipelines**
   * Open your project
   * Click the **Pipelines** tab
   * Click **Create Pipeline**
2. **Select your blueprint**
   * Choose your commit
   * Select the pipeline with parameters
3. **Configure for parallel execution**
   * Scroll to **Pipeline Parameters**
   * Change dropdown from "Single Pipeline" to "Pipeline Task"
   * Toggle **Variant** to "on"
4. **Define parameter variations**
   * Enter each parameter value on a new line
   * Each value creates a separate pipeline instance
5. **Launch the pipelines**
   * Click **Create Pipeline**
   * View your Pipeline Task containing all instances

#### Configuration example

For example, three factory IDs create three independent pipelines:

* Pipeline 1: `factory_berlin`
* Pipeline 2: `factory_munich`
* Pipeline 3: `factory_hamburg`

Each pipeline runs the complete workflow with its specific factory context.

### Parameter configuration patterns

#### Simple list (one parameter)

```
factory_001
factory_002
factory_003
```

#### Multiple parameters

When you have multiple pipeline parameters, Valohai creates pipelines for every combination:

* Parameter 1: `region` → europe, asia
* Parameter 2: `model_size` → small, large

**Result**: 4 pipelines (europe-small, europe-large, asia-small, asia-large)

#### Dynamic parameter usage

Use parameters in your steps to create context-specific behavior:

```yaml
- step:
    name: train_step
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./train.py
    parameters:
      - name: factory_id
        type: string
        default: "factory_001"
    inputs:
      - name: dataset
        default: dataset://{parameter:factory_id}/latest # dataset://factory_001/latest
        optional: true
```

### Combine with task nodes

You can use both patterns together: parallel pipelines where each pipeline contains task nodes for hyperparameter tuning:

```yaml
- pipeline:
    name: Factory ML Pipeline
    parameters:
      - name: factory_id
        targets:
          - train.parameters.factory_id
    nodes:
      - name: train
        type: task  # Each factory runs hyperparameter tuning
        step: train_model
```

This creates multiple factory-specific pipelines, each running hyperparameter optimization.