Launch Pipelines with Webhooks

Quickstart guide for webhook triggers that launch pipeline runs

This quickstart guide shows you how to create a webhook trigger that launches a Valohai pipeline. You'll learn how to parse webhook payloads, pass data between pipeline nodes, and secure your webhook with authentication.

Prerequisites: Basic understanding of Python, valohai.yaml configuration, and pipelines.

What You'll Build

A webhook-triggered pipeline that:

Receives a webhook POST request with form data
Extracts a prompt parameter from the webhook payload
Passes the prompt to a second pipeline node
Processes the prompt (demonstration only)

This pattern can be adapted for:

Triggering inference from external applications
Processing uploaded files referenced in webhooks
Responding to events from third-party services
Building custom integrations

Step 1: Create the Pipeline Scripts

Webhook Entry Script

This script receives and parses the webhook payload.

Create webhook_entry.py:

import json
import valohai

from urllib.parse import parse_qs

valohai.prepare(
    step="Webhook Entry",
)

with open(valohai.inputs('webhook-payload').path()) as f:
    post_parameters = {k: v[0] for k, v in parse_qs(f.read()).items()}

print("Received these parameters from the webhook:")
for k, v in post_parameters.items():
    print(f"{k}: {v}")
if "prompt" in post_parameters:
    # Submit metadata
    print(json.dumps({'parsed_prompt': post_parameters['prompt']}))

Metadata pattern: Printing JSON with specific keys creates metadata that can be connected to downstream parameters via pipeline edges.

Followup Processing Script

This script receives data parsed from the webhook.

Create webhook_followup.py:

import valohai

valohai.prepare(
    step="Webhook Followup",
)

prompt = valohai.parameters("prompt").value
print("Received prompt:", prompt)
print("---")
print("This is just a demonstration to pass values around in a pipeline, so not doing anything useful with it.")

Step 2: Configure Pipeline in YAML

Add these steps and pipeline to your valohai.yaml:

- step:
    name: Webhook Entry
    image: python:3.11
    command:
    - pip install valohai-utils
    - python ./webhook_entry.py
    inputs:
    - name: webhook-payload
      optional: false
- step:
    name: Webhook Followup
    image: python:3.11
    command:
    - pip install valohai-utils
    - python ./webhook_followup.py {parameters}
    parameters:
    - name: prompt
      optional: false
      type: string

- pipeline:
    name: Webhook Pipeline
    edges:
    - configuration: {}
      source: entry.metadata.parsed_prompt
      target: followup.parameter.prompt
    nodes:
    - name: entry
      on-error: stop-all
      step: Webhook Entry
      type: execution
    - name: followup
      on-error: stop-all
      step: Webhook Followup
      type: execution

How the Pipeline Works

Entry node:

Receives webhook payload as input
Parses form-encoded data
Outputs parsed_prompt as metadata

Pipeline edge:

Connects entry.metadata.parsed_prompt
To followup.parameter.prompt

Followup node:

Receives prompt via parameter
Processes it (or does your actual work)

Step 3: Generate a Webhook Secret

Generate a strong secret for authentication. Your password manager can do this, or use:

openssl rand -base64 32

Store this securely, you'll need it for both the trigger configuration and when calling the webhook.

Keep secrets secure: Never commit secrets to version control. Store them in environment variables or secret managers.

Step 4: Create the Webhook Trigger

Go to Project Settings → Triggers → Create Trigger
Trigger Type: Webhook

Add Authentication Condition

Click + Add in Conditions column → Web Request Authentication

Configure:

Auth Type: Static Secret Token
Token Lookup Namespace: HTTP Header
Token Lookup Key: Authorization
Secret Key: Paste your webhook secret
Value Prefix: secret=

Why a prefix? The secret= prefix lets you put the token after a standard header prefix. The Authorization header will be secret=YOUR_SECRET, and Valohai strips the prefix before comparing.

Add Run Pipeline Action

Click + Add in Actions column → Run Pipeline

Configure:

Source Commit Reference: main (or your branch)
Pipeline Name: Webhook Pipeline
Payload Input Name: entry.webhook-payload

Save and Get Webhook URL

Click Save Trigger
Edit the trigger again (... menu → Edit)
Copy the webhook URL shown (looks like: https://app.valohai.com/api/v0/launch/<id>/)

Step 5: Test the Webhook

Using curl

Replace <webhook-url> and <webhook-secret> with your values:

curl -X POST <webhook-url> \
  -H "Authorization: secret=<webhook-secret>" \
  -d "hello=valohai&prompt=draw+me+a+shark"

Using Python (urllib)

import urllib.request
import urllib.parse

urllib.request.urlopen(urllib.request.Request(
   '<paste the webhook URL here>', 
   data=urllib.parse.urlencode({
      'hello': 'valohai',
      'prompt': 'draw me a shark'
   }).encode(), 
   headers={
      'Authorization': 'secret=<paste the Webhook Secret here>'
   }
))

Using Python (requests)

Install requests: pip install requests

import requests

requests.post(
   '<paste the webhook URL here>', 
   data={'hello': 'valohai', 'prompt': 'draw me a shark'}, 
   headers={'Authorization': 'secret=<paste the Webhook Secret here>'}
)

More examples: Check the Webhook Triggers Reference for additional authentication methods and Python examples.

Step 6: Verify Pipeline Launched

Go to your project's Pipelines tab
You should see a new pipeline run
Open it and check both executions:

Entry execution logs:

Received these parameters from the webhook:
hello: valohai
prompt: draw me a shark

Followup execution logs:

Received prompt: draw me a shark
---
This is just a demonstration to pass values around in a pipeline...

Success! Your webhook trigger is working.

Troubleshooting

400 Bad Request

Cause: Authentication failed

Solutions:

Verify secret matches exactly
Check Authorization header format
Ensure secret= prefix is correct
Look at trigger logs for detailed error

Pipeline Launches but Fails

Cause: Payload parsing error

Solutions:

Check entry execution logs for Python errors
Verify payload format matches parsing code
Print raw payload to debug: print(f.read())
Test entry script manually with sample payload

No Data Passed to Followup

Cause: Edge misconfigured

Solutions:

Verify metadata key matches: parsed_prompt
Check parameter name matches: prompt
Ensure edge in YAML is correct
Test entry execution manually and check metadata

Webhook URL Not Working

Cause: Trigger not saved or disabled

Solutions:

Ensure trigger is saved
Check trigger is enabled
Verify URL copied correctly (entire URL including /)
Test with curl first before custom code

Security Best Practices

Always Use Authentication

Never create webhooks without authentication:

Minimum: Static secret token
Better: HMAC with timestamp validation
Best: JWT with short expiration

Use HTTPS Only

Valohai enforces HTTPS. Your webhook clients should too.

Rotate Secrets Regularly

Change webhook secrets periodically:

Generate new secret
Update trigger configuration
Update webhook senders
Verify old secret stops working

Monitor and Rate Limit

Set up rate limiting conditions
Monitor trigger logs for unusual activity
Alert on repeated authentication failures
Review launched pipelines regularly

Next Steps

More authentication methods: Read the Webhook Triggers Reference
Real integrations: Check out Slack, GitHub, and V7 examples
Advanced validation: Learn about Webhook Run Info to track launched pipelines
Sending data to webhooks: See Webhooks over API Requests

PreviousWebhook Triggers NextWebhook Triggers Reference

Last updated 4 hours ago

Was this helpful?