> For the complete documentation index, see [llms.txt](https://docs.valohai.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.valohai.com/pipelines/configure-failure-handling.md).

# Pipeline Error Handling

Control how your pipeline responds to failures. By default, any node failure stops the entire pipeline but you can customize this behavior for more resilient workflows.

### Why customize error handling?

Default behavior works for critical paths where every step must succeed. But consider these scenarios:

* **Parallel model training**: If 9 out of 10 hyperparameter combinations succeed, you want the best model, not a failed pipeline
* **Data quality checks**: Optional validation that shouldn't block core processing
* **A/B testing**: One model variant failing shouldn't prevent evaluating others
* **Batch processing**: A few failed items shouldn't stop processing thousands of others

### Error handling strategies

#### `stop-all` (default)

Any failure stops the entire pipeline immediately.

```yaml
- pipeline:
    name: critical-pipeline
    nodes:
      - name: validate-data
        type: execution
        step: data-validation
        # on-error: stop-all # implicit
```

Use when: Every step is critical to the final output.

#### `continue`

Node completes regardless of failures. Downstream nodes still run.

```yaml
- name: hyperparameter-search
  type: task
  on-error: continue
  step: train-model
```

Use when: You expect some failures and want to collect all successful results.

#### `stop-next`

Failed node blocks its dependents but allows parallel branches to continue.

```yaml
- name: optional-preprocessing
  type: execution
  on-error: stop-next
  step: enhance-data
```

Use when: This branch is optional, but if it runs, subsequent steps need its output.

### Task node considerations

Error handling is especially important for task nodes running parallel executions:

```yaml
- pipeline:
    name: parallel-training
    parameters:
      - name: learning_rates
        target: train-models.parameters.lr
        default: [0.001, 0.01, 0.1, 1.0]  # 1.0 will likely fail
    nodes:
      - name: train-models
        type: task
        on-error: continue  # Don't let one bad LR stop everything
        step: train-model
      - name: select-best
        type: execution
        step: compare-models
    edges:
      - [train-models.output.model*, select-best.input.models]
```

With `on-error: continue`:

* 3 models train successfully with reasonable learning rates
* 1 fails with `lr=1.0`
* `select-best` receives 3 models and picks the best
* Pipeline succeeds overall

### Practical example

Here's a pipeline that handles failures gracefully:

```yaml
- step:
    name: train-model
    image: tensorflow/tensorflow:2.6.0
    command: python train.py {parameters}
    parameters:
      - name: architecture
        type: string

- step:
    name: evaluate-model
    image: tensorflow/tensorflow:2.6.0
    command: python evaluate.py
    inputs:
      - name: model

- pipeline:
    name: model-comparison
    parameters:
      - name: architectures
        target: experimental-models.parameters.architecture
        default: ["resnet", "efficientnet", "experimental-v1"]
    nodes:
      # Prepare data - critical step
      - name: prepare-data
        type: execution
        step: preprocess-dataset
        # Default on-error: stop-all - data is required

      # Try proven models - expect success
      - name: proven-models
        type: task
        on-error: stop-next  # If these fail, something's wrong
        step: train-model
        override:
          parameters:
            - name: architecture
              default: ["resnet", "efficientnet"]

      # Try experimental model - might fail
      - name: experimental-models
        type: task
        on-error: continue  # Don't block pipeline if experimental fails
        step: train-model

      # Evaluate all successful models
      - name: compare-all
        type: execution
        step: compare-models

    edges:
      - [prepare-data.output.*, proven-models.input.dataset]
      - [prepare-data.output.*, experimental-models.input.dataset]
      - [proven-models.output.model*, compare-all.input.proven-models]
      - [experimental-models.output.model*, compare-all.input.experimental-models]
```

### Debugging failed pipelines

#### View execution logs

Check individual execution logs to understand failures:

1. Click on the failed node in the pipeline graph
2. Select the failed execution
3. Review logs for error messages


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.valohai.com/pipelines/configure-failure-handling.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.