Valohai webhooks over API requests

Overview

Integrate your applications with Valohai using the Valohai API to pass data to executions or pipelines and retrieve results efficiently. This guide emphasizes the advantages of using Valohai webhooks, which provide a seamless method to automate and manage data transfers and pipeline executions.

Advantages of Using Valohai Webhooks

• Automation: Automatically trigger pipeline executions based on specific events or conditions.
• Efficiency: Reduce manual intervention and streamline the data transfer process.
• Flexibility: Handle various data types and formats, ensuring your pipelines receive the necessary input.
• Scalability: Easily manage and scale your data workflows with minimal setup.

Using Valohai Webhooks

Valohai webhooks enable automatic triggering of pipelines and other actions like starting executions, copying pipelines, and fetching repositories. This guide focuses on setting up a webhook to run a pipeline.

Follow the steps below to configure the webhook and handle the payload in Valohai.

1. Create a trigger in Valohai

1. Go to your project settings.
2. Navigate to the “Triggers” section.
3. Create a new trigger and set the Trigger Type to Webhook.
4. Set no conditions.
5. Choose the action you want to perform, such as running a pipeline, form the dropdown menu and click on the “Add” button.
6. Configure the pipeline details (e.g., select a pipeline to run).
7. Set the “Payload input name” to enable sending a file to the webhook, which will be used as an input for one of your pipeline nodes.
8. Go to your pipeline configuration either in the UI or valohai.yaml, identify the node name where you want to pass the file, and set the input name for this node (e.g., “node_name.input_name”).
9. Press “Create trigger”.
10. Generate the Webhook URL by editing the trigger (click on the three dots in the trigger table) and copy the generated webhook URL for making POST requests as shown in the next section.

2. Send a file via POST request

Use a tool like Python, Postman, or any other HTTP client to send the file.

Example Python code to send a file:

import requests

url = "YOUR_WEBHOOK_URL" # generated on step 10 in previous section (e.g https://app.valohai.com/api/v0/launch/<webhook-id>/)
file_path = "path/to/your/file.ext"

with open(file_path, 'rb') as file:
    files = {'file': file}
    response = requests.post(url, files=files)

print(response.status_code)
print(response.text)

3. Check the pipeline execution

• Verify that a new pipeline execution is created.
• Check the step you defined in the trigger settings to ensure the file is received in the chosen input.

4. Extract content from payload

Webhook payloads are generally passed through to your server unmodified. However, client programs like Python’s requests library may wrap the payload in multipart form-data metadata. This wrapping is a feature of how the client structures the POST request, not an intrinsic characteristic of the webhook.

To extract content from a wrapped payload and save it as a separate file, follow the example below:

Extracting JSON Content Using sed

If your payload contains text-based files, you can use the sed command to extract the content. Add the following command to your execution setup (either through the UI or in your valohai.yaml configuration). This example uses sed, but you can adapt it to other extraction methods as needed:

input_file="/valohai/inputs/dataset/payload.txt"
output_file="/valohai/repository/dataset/processed_data.ext"
sed -n '/^{/,/^}/p' "$input_file" > "$output_file"

• input_file is the path to the file containing the payload.
• output_file is the path where you want to save the extracted content.

Sending a File Directly in a POST Request

If you prefer to avoid multipart form-data wrapping, you can send the file as raw data directly. Here’s how to do it with curl:

To send a file as raw data with curl, use the --data-binary option:

curl -X POST https://app.valohai.com/webhook-example --data-binary @yourfile.txt

• https://app.valohai.com/webhook-example is the URL of the server endpoint.
• yourfile.txt is the path to the file being sent.

By sending the file directly, you bypass the multipart form-data wrapping, making the extraction process simpler.