Webhook Triggers Reference

Develop custom Valohai webhook triggers to launch pipeline runs

This reference guide covers everything you need to build custom webhook triggers. Learn about authentication methods, conditions, and how to launch webhooks programmatically from Python.

Launching webhook triggers with Python

While webhooks can be set up from outside services, you can also create custom workflows by launching webhook triggers yourself. This reference is focused on webhook triggers on Valohai, but you can also use these tools to launch webhooks at other services on the Internet that accept them.

While it is not strictly necessary to add authorization headers to secure your webhooks, it is very much recommended. All examples provided here implement authorization conditions.

You can use the demonstration trigger from the Launch Pipelines with Webhooks chapter as a base to experiment with.

Secrets Management

Authorization requires managing secrets to verify requests with. The following examples assume we have a secret value set in the project environment variables called WEBHOOK_TOKEN. Create it as follows:

1. Generate a reasonably long secret passphrase or random text. You can use your password manager to do this.

2. On the Valohai app, go to Project settings → Env variables

3. On the bottom of the list, enter WEBHOOK_TOKEN as variable name, and the secret created in step (1) to the variable value.

4. Check the key icon and save.

Token as an environment variable: If you are running the examples on a local machine or otherwise outside Valohai, you should add the token to your running environment for these examples to work. On a Valohai execution within your project, the environment variable is available automatically.

You can add an environment variable to your shell with export WEBHOOK_TOKEN=<secret>. When you start another shell, you will need to do it again.

Creating Webhook Triggers

Webhook triggers can be created in the project settings, under Triggers. Click "Create Trigger" and select "Webhook" as the trigger type.

Trigger behaviour can be defined with conditions that validate the webhook request and actions that can launch specific executions and pipelines in response to a valid request.

Webhook conditions reference

As with Valohai triggers in general, all conditions must simultaneously be satisfied for the trigger to launch.

Failed conditions result in a 400 Bad Request response code to the webhook, and a log entry may be recorded on the trigger log.

Web Authentication condition

This is the main tool to authenticate webhook requests with several flexible options.

Auth type and Secret key: The token's internal format and its verification data.

• Static secret token: Secret key is a sequence of characters the token is compared to.

• Hash-based Message Authentication Code (HMAC): Secret Key is a shared secret with the sender that is used to generate the HMAC signature.

• JSON Web Token (JWT): Secret key is either a symmetric key or the asymmetric public key that verifies the given JWT-format token.

Verifying the request body: HMAC can only use Header and Querystring lookup namespaces to get the HMAC signature to compare against, as it cannot generate a signature from data that contains the signature itself.

The secret key can use (secret) project environment variables using {env:MY_ENV_VAR}. This is very useful if you have several webhooks from the same provider reusing the same verification secret.

HMAC Basestring Format: HMAC only. This describes the structure of the HMAC signature's source data, which can use the request body content as well as select headers from the request. You can include the entire request body with {body}. You can include request header values with {header:My-Header}. If the named header is not found in the request, the condition will always fail. This field is typically used to represent a static prefix, punctuation or timestamp value in the HMAC signature.

The remaining options describe how the token to compare against is acquired.

Token Lookup Namespace: Where to look for the authentication token?

• HTTP Header: Supplied within the request headers. Key is the header name in standard X-Http-Header format.

• Querystring: Supplied as part of the URL after the question mark: https://app.valohai.com/api/v0/launch/.../?search=sharks&size=3 would have the querystring key search with the value sharks, and the key size with the value 3.

• URL Encoded Body: Supplied as part of the request body content, in the same format as querystring

• JSON Body: Supplied as part of the request body content, using JSON format. Note: Only top level keys are supported.

Token Lookup Key: What part of the namespace contains the authentication token?

Value Prefix: The token often has a static prefix that is not part of the token. This field can be used to cut the prefix away prior to verifying the remaining token.

Static token authorization example

Static token authorization is handled with a Web Request Authentication trigger condition. This is the simplest authorization method as a shared, static secret is passed by the webhook sender within the request and compared with the stored secret on the Valohai project.

Add a static token authentication condition with these steps:

1. For Auth Type, select "Static Secret Token"

2. For Token Lookup Namespace, select "HTTP Header"

3. For Token Lookup Key, type Authorization

4. For Secret Key, enter {env:WEBHOOK_TOKEN}

5. Leave Value prefix empty.

Launch a webhook using urllib (no install required):

import os
import urllib.request
import urllib.parse

urllib.request.urlopen(urllib.request.Request(
   '<paste the webhook URL here>', 
   data=urllib.parse.urlencode({
      'hello': 'valohai',
      'prompt': 'draw me a shark'
   }).encode(), 
   headers={
      'Authorization': os.getenv("WEBHOOK_TOKEN")
   }
))

Launch a webhook with requests (install it with pip install requests or add requests to your requirements.txt):

import os
import requests

requests.post(
   '<paste the webhook URL here>', 
   data={'hello': 'valohai', 'prompt': 'draw me a shark'}, 
   headers={'Authorization': os.getenv("WEBHOOK_TOKEN")}
)

Requests and Urllib: Read more about how these libraries work on their respective documentation pages: Requests and Urllib

Hash-based Message Authentication Code (HMAC) authentication example

Hash-based Message Authentication Code (HMAC) uses the shared secret to create a check value derived from the request's contents, which is not a secret. The check value is then generated again by the condition and compared with the value sent with the request to validate it. HMAC is more secure than a simple shared token, as the token is never transmitted within the request.

Add a HMAC authentication condition with these steps:

1. For Auth Type, select "Hash-based Message Authentication Code"

2. For Token Lookup Namespace, select "Header"

3. For Token Lookup Key, type Authorization

4. For Secret Key, enter {env:WEBHOOK_TOKEN}

5. Leave Value prefix empty.

6. For Algorithm, select SHA256

Launch the trigger with this example Python code:

import hmac
import os
import urllib.request
import urllib.parse

data = urllib.parse.urlencode({
  'hello': 'valohai',
  'prompt': 'draw me a shark'
}).encode()

calculated_hmac = hmac.new(
    key=os.getenv("WEBHOOK_TOKEN").encode(),
    msg=data,
    digestmod='sha256',
).hexdigest()

urllib.request.urlopen(urllib.request.Request(
   '<paste the webhook URL here>', 
   data=data, 
   headers={
      'Authorization': calculated_hmac
   }
))

JSON Web Token (JWT) authentication example

JSON Web Tokens (JWT) use a shared or asymmetric secret to sign a token authenticating the request. A new, quickly expiring token can be generated for each webhook request. Like HMAC, the shared secret is not transmitted with the request.

Asymmetric algorithms (e.g. RS256) can be used to avoid sharing secrets. To do so, create an RSA keypair (private and public key). Encode the JWT with the private key and set the WEBHOOK_TOKEN to the corresponding public key. The public key only allows verifying that the token was created by the private key. This example uses a symmetric algorithm.

Add a JWT authentication condition with these steps:

1. For Auth Type, select "JSON Web Token"

2. For Token Lookup Namespace, select "HTTP Header"

3. For Token Lookup Key, type Authorization

4. For Secret Key, enter {env:WEBHOOK_TOKEN}

5. Leave Value prefix empty.

6. For Algorithm, select HS256

Launch the trigger with this example Python code. You will need PyJWT installed to run this, which you can get with e.g. pip install pyjwt.

import jwt
import os
import time
import urllib.request
import urllib.parse

data = urllib.parse.urlencode({
  'hello': 'valohai',
  'prompt': 'draw me a shark'
}).encode()

token = jwt.encode(
    {'ts': int(time.time())}, 
    key=os.getenv("WEBHOOK_TOKEN"), 
    headers={'exp': int(time.time() + 60)}
)

urllib.request.urlopen(urllib.request.Request(
   '<paste the webhook URL here>', 
   data=data, 
   headers={
      'Authorization': token
   }
))

More Web Authentication condition examples

Here are some more examples to help match webhook authorization headers with Valohai trigger conditions. You can inspect how webhook requests look like by configuring them to deliver to a custom Web server or a service like Smee. Valohai does not provide direct access to Webhook request contents as they may contain sensitive information.

Incoming header: Authorization: JWT ey...(JWT)

• Auth type: JWT

• Token lookup namespace: HTTP Header

• Token lookup key: Authorization

• Value prefix: JWT . Note: Remember the trailing space. The prefix is cut from the beginning of the supplied value exactly.

• Secret key: Secret key that verifies the JWT

Incoming header: X-Hmac-Signature: v0=u2jhlpq...(HMAC signature)

The HMAC comparison is specified to use the static string "hmac", a value from another header "X-Hmac-Timestamp" and the request body, delimited with the colon character ":".

• Auth type: HMAC

• Token lookup namespace: HTTP Header

• Token lookup key: X-Hmac-Signature

• Value prefix: v0=

• HMAC Basestring Format: hmac:{header:X-Hmac-Timestamp}:{body}

• Secret key: Shared secret with the sender

Incoming request content: token=pat_842717_...(static token)&event_type=new_data&...

• Auth type: Static token

• Token lookup namespace: URL Encoded body

• Token lookup key: "token"

• Secret key: "pat_842717_...(static token)"

Web Timestamp condition

Timestamp conditions are typically used combined with HMAC authentication conditions to make HMAC-signed requests expire after a short period, which protects against replay attacks where a valid request is recorded and sent again later.

Timestamp lookup namespace and lookup key work the same way as web authentication.

If the timestamp cannot be found, or is found but is more than Drift tolerance seconds away from current time, the condition fails.

The timestamp must be a UNIX timestamp as seconds.

Web Timestamp with HMAC authentication example

Add a Web Request Authentication condition using HMAC with these steps:

1. For Auth Type, select "Hash-based Message Authentication Code"

2. For Token Lookup Namespace, select "HTTP Header"

3. For Token Lookup Key, type Authorization

4. For Secret Key, enter {env:WEBHOOK_TOKEN}

5. For HMAC Basestring Format, type prefix.{body}.{header:Request-Timestamp}

6. Leave Value prefix empty.

7. For Algorithm, select "SHA256"

Add a Web Request Timestamp condition with these steps:

1. For Token Lookup Namespace, select "HTTP Header"

2. For Token Lookup Key, type Request-Timestamp

3. For Tolerance, type 60 (this is in seconds)

Launch this trigger with this example Python code:

import time
import hmac
import os
import urllib.request
import urllib.parse

data = urllib.parse.urlencode({
  'hello': 'valohai',
  'prompt': 'draw me a shark'
}).encode()

timestamp = int(time.time())

calculated_hmac = hmac.new(
    key=os.getenv("WEBHOOK_TOKEN").encode(),
    msg=f"prefix.{data.decode()}.{timestamp}".encode(),
    digestmod='sha256',
).hexdigest()

urllib.request.urlopen(urllib.request.Request(
    '<paste the webhook URL here>', 
    data=data, 
    headers={
        'Authorization': calculated_hmac,
        'Request-Timestamp': str(timestamp), 
    }
))

Rate Limit condition

Rate limit conditions can prevent triggers from launching more often than you intend. You should set a reasonable rate limit to protect against accidentally launching too many triggered workloads.

Set a time Period (in seconds) during which at most Quota of triggers can be launched. This condition fails if the trigger being launched would exceed the quota.

Several rate limits can be applied at once to allow e.g. a large daily quota, but also a smaller hourly quota that you can't use the whole day's quota at once.

Webhook Actions Reference

Web Response Text & JSON

These actions can customize successful webhook responses. They can only show static content. Failed response content cannot be customized. The default response is ok (text/plain). Only the first web response action will be used.

For Response Text, the content type will be text/plain.

For Response JSON, the content type will be application/json.

You can use the {run_id} placeholder to include a reference to the trigger run that the webhook has launched. This reference can be used to request information about the trigger's results, such as created executions using the webhook run info endpoint.

Run Execution

Webhook-specific parameter Payload input name allows setting the webhook request body as an input on the execution being launched. The execution step must have a matching declared input in the YAML.

The execution will receive the untouched request body into that input when launched through the webhook, and is responsible for parsing and processing it.

Run Pipeline

Webhook-specific parameter Payload input name allows setting the webhook request body as an input on one of the executions in the pipeline being launched. The format is nodename.input_name. The node of the specified name must be an execution node and the node's execution step must have matching a declared input in the YAML.

Multiple nodes with webhook input? While only one execution node can directly receive the webhook request body as an input, the pipeline can define input-input edges to distribute it to any other execution nodes as needed.