Launch Pipelines with Webhooks

This quickstart guide shows you how to create a webhook trigger that launches a Valohai pipeline. You'll learn how to parse webhook payloads, pass data between pipeline nodes, and secure your webhook with authentication.

Prerequisites: Basic understanding of Python, valohai.yaml configuration, and pipelines.

---

What You'll Build

A webhook-triggered pipeline that:

1. Receives a webhook POST request with form data

2. Extracts a prompt parameter from the webhook payload

3. Passes the prompt to a second pipeline node

4. Processes the prompt (demonstration only)

This pattern can be adapted for:

• Triggering inference from external applications

• Processing uploaded files referenced in webhooks

• Responding to events from third-party services

• Building custom integrations

---

Step 1: Create the Pipeline Scripts

Webhook Entry Script

This script receives and parses the webhook payload.

Create webhook_entry.py:

import json
import valohai

from urllib.parse import parse_qs

valohai.prepare(
    step="Webhook Entry",
)

with open(valohai.inputs("webhook-payload").path()) as f:
    post_parameters = {k: v[0] for k, v in parse_qs(f.read()).items()}

print("Received these parameters from the webhook:")
for k, v in post_parameters.items():
    print(f"{k}: {v}")
if "prompt" in post_parameters:
    # Submit metadata
    print(json.dumps({"parsed_prompt": post_parameters["prompt"]}))

Metadata pattern: Printing JSON with specific keys creates metadata that can be connected to downstream parameters via pipeline edges.

---

Followup Processing Script

This script receives data parsed from the webhook.

Create webhook_followup.py:

import valohai

valohai.prepare(
    step="Webhook Followup",
)

prompt = valohai.parameters("prompt").value
print("Received prompt:", prompt)
print("---")
print("This is just a demonstration to pass values around in a pipeline, so not doing anything useful with it.")

---

Step 2: Configure Pipeline in YAML

Add these steps and pipeline to your valohai.yaml:

- step:
    name: Webhook Entry
    image: python:3.11
    command:
    - pip install valohai-utils
    - python ./webhook_entry.py
    inputs:
    - name: webhook-payload
      optional: false
- step:
    name: Webhook Followup
    image: python:3.11
    command:
    - pip install valohai-utils
    - python ./webhook_followup.py {parameters}
    parameters:
    - name: prompt
      optional: false
      type: string

- pipeline:
    name: Webhook Pipeline
    edges:
    - configuration: {}
      source: entry.metadata.parsed_prompt
      target: followup.parameter.prompt
    nodes:
    - name: entry
      on-error: stop-all
      step: Webhook Entry
      type: execution
    - name: followup
      on-error: stop-all
      step: Webhook Followup
      type: execution

How the Pipeline Works

Entry node:

• Receives webhook payload as input

• Parses form-encoded data

• Outputs parsed_prompt as metadata

Pipeline edge:

• Connects entry.metadata.parsed_prompt

• To followup.parameter.prompt

Followup node:

• Receives prompt via parameter

• Processes it (or does your actual work)

---

Step 3: Generate a Webhook Secret

Generate a strong secret for authentication. Your password manager can do this, or use:

openssl rand -base64 32

Store this securely, you'll need it for both the trigger configuration and when calling the webhook.

Keep secrets secure: Never commit secrets to version control. Store them in environment variables or secret managers.

---

Step 4: Create the Webhook Trigger

1. Go to Project Settings → Triggers → Create Trigger

2. Trigger Type: Webhook

Add Authentication Condition

Click + Add in Conditions column → Web Request Authentication

Configure:

• Auth Type: Static Secret Token

• Token Lookup Namespace: HTTP Header

• Token Lookup Key: Authorization

• Secret Key: Paste your webhook secret

• Value Prefix: secret=

Why a prefix? The secret= prefix lets you put the token after a standard header prefix. The Authorization header will be secret=YOUR_SECRET, and Valohai strips the prefix before comparing.

---

Add Run Pipeline Action

Click + Add in Actions column → Run Pipeline

Configure:

• Source Commit Reference: main (or your branch)

• Pipeline Name: Webhook Pipeline

• Payload Input Name: entry.webhook-payload

---

Save and Get Webhook URL

1. Click Save Trigger

2. Edit the trigger again (... menu → Edit)

3. Copy the webhook URL shown (looks like: https://app.valohai.com/api/v0/launch/<id>/)

---

Step 5: Test the Webhook

Using curl

Replace <webhook-url> and <webhook-secret> with your values:

curl -X POST <webhook-url> \
  -H "Authorization: secret=<webhook-secret>" \
  -d "hello=valohai&prompt=draw+me+a+shark"

---

Using Python (urllib)

import urllib.request
import urllib.parse

urllib.request.urlopen(
    urllib.request.Request(
        "<paste the webhook URL here>",
        data=urllib.parse.urlencode(
            {
                "hello": "valohai",
                "prompt": "draw me a shark",
            },
        ).encode(),
        headers={
            "Authorization": "secret=<paste the Webhook Secret here>",
        },
    ),
)

---

Using Python (requests)

Install requests: pip install requests

import requests

requests.post(
    "<paste the webhook URL here>",
    data={"hello": "valohai", "prompt": "draw me a shark"},
    headers={"Authorization": "secret=<paste the Webhook Secret here>"},
)

More examples: Check the Webhook Triggers Reference for additional authentication methods and Python examples.

---

Step 6: Verify Pipeline Launched

1. Go to your project's Pipelines tab

2. You should see a new pipeline run

3. Open it and check both executions:

Entry execution logs:

Received these parameters from the webhook:
hello: valohai
prompt: draw me a shark

Followup execution logs:

Received prompt: draw me a shark
---
This is just a demonstration to pass values around in a pipeline...

Success! Your webhook trigger is working.

---

Troubleshooting

400 Bad Request

Cause: Authentication failed

Solutions:

• Verify secret matches exactly

• Check Authorization header format

• Ensure secret= prefix is correct

• Look at trigger logs for detailed error

---

Pipeline Launches but Fails

Cause: Payload parsing error

Solutions:

• Check entry execution logs for Python errors

• Verify payload format matches parsing code

• Print raw payload to debug: print(f.read())

• Test entry script manually with sample payload

---

No Data Passed to Followup

Cause: Edge misconfigured

Solutions:

• Verify metadata key matches: parsed_prompt

• Check parameter name matches: prompt

• Ensure edge in YAML is correct

• Test entry execution manually and check metadata

---

Webhook URL Not Working

Cause: Trigger not saved or disabled

Solutions:

• Ensure trigger is saved

• Check trigger is enabled

• Verify URL copied correctly (entire URL including /)

• Test with curl first before custom code

---

Security Best Practices

Always Use Authentication

Never create webhooks without authentication:

• Minimum: Static secret token

• Better: HMAC with timestamp validation

• Best: JWT with short expiration

Use HTTPS Only

Valohai enforces HTTPS. Your webhook clients should too.

Rotate Secrets Regularly

Change webhook secrets periodically:

1. Generate new secret

2. Update trigger configuration

3. Update webhook senders

4. Verify old secret stops working

Monitor and Rate Limit

• Set up rate limiting conditions

• Monitor trigger logs for unusual activity

• Alert on repeated authentication failures

• Review launched pipelines regularly

---

Next Steps

• More authentication methods: Read the Webhook Triggers Reference

• Real integrations: Check out Slack, GitHub, and V7 examples

• Advanced validation: Learn about Webhook Run Info to track launched pipelines

• Sending data to webhooks: See Webhooks over API Requests