Trigger an execution after a push to a code repository

Not using GitHub? Scroll down to see a example of the Valohai API call you’d need to trigger from your own code repository (e.g. WebHooks on GitLab or BitBucket)

GitHub Actions allow you to automate development workflows right from your repository. In this tutorial we’ll use GitHub Actions to trigger a new execution on Valohai every time a new push is made to our repository.

➡️ Check out GitHub Actions Documentation to learn more about Actions.

Pre-requisites

• Existing Valohai project with a valohai.yaml file and at least one step. You can find a sample valohai.yaml and train.py from the sample repository of this tutorial.

• Your project is connected to a GitHub repository (either public or private)

Create your GitHub Action workflow

Workflows are used to define an automated process for your repository. In our case we’ll define a workflow that triggers a new action every time there is a push to the repository.

A workflow needs to know when it should be triggered, what are the jobs to run and in which environment should it run these jobs.

• In your existing project, create a new folder .github/

• Add a new folder to keep our workflows .github/workflows/

• Create a new file called .github/workflows/main.yml and define the workflow as:

  name: Workflow for Create Valohai execution on push
  on: push
  jobs:
  build:
      name: Create Valohai execution on push
      runs-on: ubuntu-latest
      steps:
      - uses: actions/checkout@v1
      - uses: ./valohai-execution-action
  

Define the action that should be triggered

GitHub Actions can run directly on the machine (JavaScript) on in a Docker container.

Our action will run a Python script inside a Docker container. That script will use the Valohai APIs to trigger a new Valohai execution based on the new commit.

• Start by creating a new folder in the root of your directory valohai-execution-action/.

• Create a new file valohai-execution-action/runExecution.py. In there create a new call to the Valohai APIs.

  • You’ll need an authentication token and know your project ID to continue. You can find your project’s ID under its Settings-tab.

  import requests
  import json
  import os
  
  # Authenticate yourself with the token
  auth_token = '<your-auth-token>'
  headers = {'Authorization': 'Token %s' % auth_token}
  project_id = '<your-project-id>'
  step_name = '<your-step-name-from-valohai.yaml>'
  
  # Fetch all new changes from the repository
  # https://app.valohai.com/api/docs/#projects-fetch
  # This will fetch changes from all the branches that you've defined on the Project->Settings->Repository tab
  fetchResponse = requests.post(('https://app.valohai.com/api/v0/projects/{0}/fetch/').format(project_id), data={'id': project_id}, headers=headers)
  fetchResponse.raise_for_status()
  
  
  # Define the payload for a new execution
  # https://app.valohai.com/api/docs/#executions-create
  #
  # GitHub Actions creates an environment variable on the Docker container
  # Called GITHUB_SHA that stores the identifier of the commit that was created
  new_exec_payload = {
  "project": project_id,
  "commit": os.getenv('GITHUB_SHA'),
  "step": step_name,
  }
  
  # Send a POST request to create a new execution
  createExecutionResponse = requests.post('https://app.valohai.com/api/v0/executions/', data=new_exec_payload, headers=headers)
  createExecutionResponse.raise_for_status()
  
  # Print the response you've received back
  print('# API Response:\n')
  print(json.dumps(createExecutionResponse.json(), indent=4))
  

• Next create a valohai-execution-action/Dockerfile that will be used to build your container

  # We'll use a slim python image as a base
  FROM python:3.8.2-slim-buster
  
  # Our code will need requests, so we can install them on the image with pip
  RUN pip install requests
  
  # Add our file and run it
  ADD runExecution.py /runExecution.py
  CMD ["/runExecution.py"]
  ENTRYPOINT ["python"]
  

• Finally we’ll need to create valohai-execution-action/action.yml to describe our action for GitHub

  name: "Create Valohai execution on push"
  description: "Run a new execution on app.valohai.com on each new push to repository"
  author: "<your-email>"
  
  runs:
    using: "docker"
    image: "Dockerfile"
  

Push a new commit

✨ Ta-da! That’s it! Now you can commit your code and the push changes.

• git add .github/ *

• git commit -m "added github action to trigger vh executions"`

• git push

Navigate to your GitHub repository’s Actions tab to see the new action…in action 😂

You can test it out by making some changes to your step’s code (e.g. train.py) and push the changes to GitHub. You should see the Action trigger and create a new execution on Valohai with that new commit.