Triggering an execution after a push to a code repository

Not using GitHub? Scroll down to see a example of the Valohai API call you’d need to trigger from your own code repository (e.g. WebHooks on GitLab or BitBucket)

GitHub Actions allow you to automate development workflows right from your repository. In this tutorial we’ll use GitHub Actions to trigger a new execution on Valohai every time a new push is made to our repository.

➡️ Check out GitHub Actions Documentation to learn more about Actions.

Pre-requisites

Create your GitHub Action workflow

Workflows are used to define an automated process for your repository. In our case we’ll define a workflow that triggers a new action every time there is a push to the repository.

A workflow needs to know when it should be triggered, what are the jobs to run and in which environment should it run these jobs.

  • In your existing project, create a new folder .github/

  • Add a new folder to keep our workflows .github/workflows/

  • Create a new file called .github/workflows/main.yml and define the workflow as:

    name: Workflow for Create Valohai execution on push
on: push
jobs:
build:
    name: Create Valohai execution on push
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - uses: ./valohai-execution-action

Define the action that should be triggered

GitHub Actions can run directly on the machine (JavaScript) on in a Docker container.

Our action will run a Python script inside a Docker container. That script will use the Valohai APIs to trigger a new Valohai execution based on the new commit.

  • Start by creating a new folder in the root of your directory valohai-execution-action/.

  • Create a new file valohai-execution-action/runExecution.py. In there create a new call to the Valohai APIs.

    • You’ll need an authentication token and know your project ID to continue. You can find your project’s ID under its Settings-tab.

    import requests
import json
import os

# Authenticate yourself with the token.
# Remember to follow your organization's security standards when handling the token.
auth_token = os.environ['VH_API_TOKEN']
headers = {'Authorization': 'Token %s' % auth_token}
project_id = '<your-project-id>'
step_name = '<your-step-name-from-valohai.yaml>'

# Fetch all new changes from the repository
# https://app.valohai.com/api/docs/#projects-fetch
# This will fetch changes from all the branches that you've defined on the Project->Settings->Repository tab
fetchResponse = requests.post(('https://app.valohai.com/api/v0/projects/{0}/fetch/').format(project_id), data={'id': project_id}, headers=headers)
fetchResponse.raise_for_status()


# Define the payload for a new execution
# https://app.valohai.com/api/docs/#executions-create
#
# GitHub Actions creates an environment variable on the Docker container
# Called GITHUB_SHA that stores the identifier of the commit that was created
new_exec_payload = {
"project": project_id,
"commit": os.getenv('GITHUB_SHA'),
"step": step_name,
}

# Send a POST request to create a new execution
createExecutionResponse = requests.post('https://app.valohai.com/api/v0/executions/', data=new_exec_payload, headers=headers)
createExecutionResponse.raise_for_status()

# Print the response you've received back
print('# API Response:\n')
print(json.dumps(createExecutionResponse.json(), indent=4))
  • Next create a valohai-execution-action/Dockerfile that will be used to build your container
    # We'll use a slim python image as a base
FROM python:3.8.2-slim-buster

# Our code will need requests, so we can install them on the image with pip
RUN pip install requests

# Add our file and run it
ADD runExecution.py /runExecution.py
CMD ["/runExecution.py"]
ENTRYPOINT ["python"]
  • Finally we’ll need to create valohai-execution-action/action.yml to describe our action for GitHub
    name: "Create Valohai execution on push"
description: "Run a new execution on app.valohai.com on each new push to repository"
author: "<your-email>"

runs:
  using: "docker"
  image: "Dockerfile"

Push a new commit

✨ Ta-da! That’s it! Now you can commit your code and the push changes.

  • git add .github/ *

  • git commit -m "added github action to trigger vh executions"`

  • git push

Navigate to your GitHub repository’s Actions tab to see the new action…in action 😂

You can test it out by making some changes to your step’s code (e.g. train.py) and push the changes to GitHub. You should see the Action trigger and create a new execution on Valohai with that new commit.