Update the Valohai Agent

This guide covers the process of updating the Valohai agent (Peon) running on your on-premises infrastructure.

Prerequisites

Before beginning the update process, ensure you have:

• Root or sudo access to the server

• The download URL for the new Peon version

• Knowledge of your Peon installation paths

Step 1: Enable Inhibit Mode

Inhibit mode allows Peon to complete any currently running jobs while preventing it from picking up new ones. This ensures no job interruptions during the update.

Locate Your Peon Executable

First, identify where your Peon executable is installed by checking the systemd service file:

cat /etc/systemd/system/peon.service

Look for the executable path, which typically appears as something like /usr/local/bin/valohai-peon or /opt/valohai/peon-venv/bin/valohai-peon

Activate Inhibit Mode

Run the inhibit command using the path you identified:

/path/to/peon/valohai-peon inhibit

For example:

/opt/valohai/peon-venv/bin/valohai-peon inhibit

This command blocks Peon from accepting new jobs while allowing currently running jobs to complete.

Wait for Running Jobs to Complete

Monitor your running jobs and wait until all have finished before proceeding with the update.

Step 2: Update Peon

Once all jobs have completed, you can proceed with the actual update.

Download and Extract the New Version

wget -O peon.tar <URL>
tar -C peon/ -xvf peon.tar

Replace <URL> with the download link for your specific Peon version. You can get it from your Valohai contact.

Install the New Version

Install the new Peon wheel file using pip. The exact path depends on your original installation location:

sudo /opt/valohai/peon-venv/bin/pip install peon/peon-<version>.whl

Replace <version> with the actual version number you're installing, and adjust /opt/valohai/peon-venv/bin/pip if your virtual environment is installed elsewhere.

Restart the Peon Service

sudo systemctl restart peon

Verify the Update

Check that Peon is running correctly:

sudo systemctl status peon

The status should show the service as active and running.

Step 3: Clear Inhibit Mode

Once the update is complete and you've verified that Peon is running properly, clear the inhibit mode to allow Peon to resume accepting new jobs:

/path/to/peon/valohai-peon uninhibit

For example:

/opt/valohai/peon-venv/bin/valohai-peon uninhibit

Important Notes

• No Running Jobs: The service restart in Step 2 is recommended only when there are no jobs running. The inhibit mode process ensures this condition is met.

• Custom Installation Paths: The pip installation path (/opt/valohai/peon-venv/bin/pip) may differ depending on where Peon was originally installed in your environment. Verify your installation path before running the command.

• Maintenance Window: Plan your update during a maintenance window or low-activity period to minimize impact on your workflows.

Troubleshooting

If you encounter issues after the update:

1. Check the Peon logs for error messages

2. Verify that the correct version was installed: /path/to/peon/valohai-peon --version

3. Ensure all file permissions are correct

4. If needed, roll back to the previous version and contact Valohai support