Linux Workers

Deploy Valohai workers on your on-premises Linux servers to run machine learning jobs on your own hardware.

Overview

The Compute and Data Layer of Valohai can be deployed to your on-premise environment. This enables you to:

• Use your own on-premises machines to run machine learning jobs

• Use your own cloud storage for storing training artifacts (trained models, preprocessed datasets, visualizations)

• Mount local data to your on-premises workers

• Access databases and data warehouses directly from workers inside your network

Valohai doesn't have direct access to on-premises machines that execute ML jobs. Instead, it communicates with a separate static virtual machine in your on-premise environment that's responsible for storing the job queue, job states, and short-term logs.

Prerequisites

Hardware requirements:

• Linux server (Ubuntu 24.04 recommended)

• Python 3.10+ installed

• For GPU workloads: NVIDIA drivers and NVIDIA Container Toolkit installed

From Valohai:

Contact [email protected] to receive:

• queue-name - Name for this worker or group of workers

• queue-address - Address of your job queue

• redis-password - Password for the queue

• url - Download URL for the Valohai worker installer

❗Let us know if you wish to use this queue/worker in a Dispatch mode, as it requires special configuration.

Network requirements:

• Worker can connect to the queue machine on port 63790

• Worker can access your object storage (S3, Azure Blob, GCS)

• Optional: Outbound internet access for pulling Docker images

Understanding Queue Names

The queue name identifies this worker or group of workers in Valohai.

Examples:

• myorg-onprem-1

• myorg-onprem-machine-name

• myorg-onprem-gpus

• myorg-onprem-gpus-prod

Each machine can have its own queue, but we recommend using the same queue name on all machines that have the same configuration and are used for the same purpose.

Installation Methods

Choose your installation method based on your operating system and preferences.

Ubuntu Installer (Recommended)

Automated installer for Ubuntu systems.

What it installs:

• Valohai agent (Peon)

• Docker (if not already installed)

• NVIDIA Container Toolkit (if needed for GPU workloads)

• System service configuration

Warning: Only use on fresh, dedicated machines. This will reinstall Docker and NVIDIA Container Toolkit, breaking any existing container workloads. Follow the manual installation steps if you want more control.

Installation:

sudo su
apt-get update -y && apt-get install -y python3 python3-distutils

TEMPDIR=$(mktemp -d)
pushd $TEMPDIR

export NAME=<queue-name>
export QUEUE_ADDRESS=<queue-address>
export PASSWORD=<redis-password>
export URL=<bup-url>

curl $URL --output bup.pex
chmod u+x bup.pex
env "CLOUD=none" "ALLOW_MOUNTS=true" "INSTALLATION_TYPE=private-worker" "REDIS_URL=rediss://:$PASSWORD@$QUEUE_ADDRESS:63790" "PEON_WARDEN_ENABLED=true" 'PEON_EXTRA_CONFIG={"ALLOW_MOUNTS":"true"}' "QUEUES=$NAME" ./bup.pex

popd

Replace the placeholder values with the information from Valohai.

After installation, the Valohai agent will start automatically and begin pulling jobs from the queue.

💡Setting the "PEON_WARDEN_ENABLED=true" enables monitoring the status of the agent in the Valohai UI. If you want this disabled, you can set the value to false.

Manual Installation

For non-Ubuntu systems or custom configurations.

Manual Installation Steps

Step 1: Install Dependencies

Python 3.10+

Verify Python is installed:

python3 --version

Docker

Install Docker for your Linux distribution. Visit the Docker installation guide and select your distribution.

NVIDIA Drivers (GPU only)

If using GPUs, install NVIDIA drivers appropriate for your GPU model.

Verify installation:

nvidia-smi

NVIDIA Container Toolkit (GPU only)

Install NVIDIA Container Toolkit to enable GPU access in containers.

Follow the NVIDIA documentation for your distribution.

Verify it works:

docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

Step 2: Download and Install Peon

Download the Peon agent using the URL provided by Valohai. We recommend installing the agent inside a virtual environment.

sudo mkdir -p /opt/valohai
sudo chown -R valohai /opt/valohai
python -m venv /opt/valohai/peon-venv
pushd $(mktemp -d)
wget <URL>
tar xvf peon.tar
/opt/valohai/peon-venv/bin/python -m pip install *.whl
popd

Replace <URL> with the download URL from Valohai.

⚠️ The distutils module has been deprecated on Ubuntu 24.04 and Python 3.12. As some of the Valohai components depend on this, you will need to install for example setuptools inside the virtual environment to ensure everything works as expected.

/opt/valohai/peon-venv/bin/python -m pip install setuptools

Step 3: Configure Peon

Create the configuration file /etc/peon.config:

CLOUD=none
INSTALLATION_TYPE=private-worker
QUEUES=<queue-name>
REDIS_URL=rediss://:<redis-password>@<queue-address>:63790
ALLOW_MOUNTS=true

Configuration values:

Replace these placeholders:

• <queue-name> - Your queue name from Valohai

• <redis-password> - Redis password from Valohai (stored in your cloud Secret Manager)

• <queue-address> - Queue address from Valohai

Step 4: Create Systemd Service

Create the service file /etc/systemd/system/peon.service:

[Unit]
Description=Valohai Peon Service
After=network.target

[Service]
Environment=LC_ALL=C.UTF-8 LANG=C.UTF-8
EnvironmentFile=/etc/peon.config
ExecStart=/opt/valohai/peon-venv/bin/valohai-peon
User=valohai
Group=valohai
Restart=on-failure

[Install]
WantedBy=multi-user.target

Important: Update these values:

• ExecStart - Path to valohai-peon binary (use sudo find / -iname "valohai-peon" -print to find it)

  • If you installed Peon inside a virtual environment, the binary should be there

  • Common locations for global installation: /home/valohai/.local/bin/valohai-peon or /usr/local/bin/valohai-peon

• User - Linux user that will run the service

• Group - Linux group for the user

Step 5: Create Cleanup Service

Create /etc/systemd/system/peon-clean.service:

[Unit]
Description=Valohai Peon Cleanup
After=network.target

[Service]
Type=oneshot
Environment=LC_ALL=C.UTF-8 LANG=C.UTF-8
EnvironmentFile=/etc/peon.config
ExecStart=/opt/valohai/peon-venv/bin/valohai-peon clean
User=valohai
Group=valohai

[Install]
WantedBy=multi-user.target

Update ExecStart, User, and Group as needed.

Step 6: Create Cleanup Timer

Create /etc/systemd/system/peon-clean.timer:

[Unit]
Description=Valohai Peon Cleanup Timer
Requires=peon-clean.service

[Timer]
# Every 10 minutes
OnCalendar=*:0/10
Persistent=true

[Install]
WantedBy=timers.target

This runs the cleanup service every 10 minutes to remove stale caches and Docker images.

Step 7: Create Warden Service

Warden is a component that allows monitoring the Peon status in the Valohai UI.

Create /etc/systemd/system/peon-warden.service:

[Unit]
Description=Valohai Peon Warden Service
After=network.target

[Service]
Environment=LC_ALL=C.UTF-8 LANG=C.UTF-8
EnvironmentFile=-/etc/peon.config
ExecStart=/opt/valohai/peon-venv/bin/valohai-peon-warden
User=valohai
Group=valohai
Restart=on-failure
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Step 8: Grant Docker Permissions

The user running Peon needs permissions to control Docker:

sudo usermod -aG docker <User>

Replace <User> with the user from your service files (e.g., valohai).

Step 9: Start Services

Reload systemd to recognize the new service files:

systemctl daemon-reload

Start the Peon service:

systemctl start peon
systemctl start peon-clean
systemctl start peon-clean.timer
systemctl start peon-warden

Check that services are running:

systemctl status peon
systemctl status peon-clean.timer
systemctl status peon-warden

Step 10: Enable Auto-Start

Enable services to start automatically on boot:

systemctl enable peon
systemctl enable peon-clean
systemctl enable peon-warden

Troubleshooting Service Start

For a global installation, if services fail to start, try using the full Python module path in ExecStart:

ExecStart=/usr/bin/env python3 -m peon.cli

Use this in both peon.service and peon-clean.service files if needed.

Multi-GPU Configuration

If your server has multiple GPUs, you can configure Peon in such way so that it allows you, for each execution, to specify how many GPUs it should use.

If this value is less than the number of available GPUs it will allow you running multiple executions at the same time on the same machine and therefore having much better resource utilization.

This kind of Peon configuration is called Dispatch mode. For example, considering a single machine with 4 GPUs, you could achieve next utilization:

• 4 executions, each using 1 GPU

• 1 execution using 1 GPU, 1 execution using 3 GPUs

• 1 execution using 4 GPUs

Configure Dispatch mode

Prerequisites:

• Valohai worker already installed (Ubuntu installer or manual installation)

• Multiple GPUs available on the server

Steps:

1. Stop the running Peon service

💡 Take a look at inhibition mode, that will allow you to safely stop the running Peon service without the loss of any information or data.

sudo systemctl stop peon

1. Add DISPATCH_MODE=true to the peon.config, created above, making it look like this

CLOUD=none
INSTALLATION_TYPE=private-worker
QUEUES=<queue-name>
REDIS_URL=rediss://:<redis-password>@<queue-address>:63790
ALLOW_MOUNTS=true
DISPATCH_MODE=true

1. You can start the Peon service again

sudo systemctl start peon

❗When contacting the Valohai support (per Prerequisites) don't forget to mention that the queue you wish to create will be used in Dispatch mode

Create an execution that uses Dispatch mode

When creating an execution, using VH_GPUS environment variable, you can specify how many GPUs does that execution requires.

💡Take a look at how you can add an environment variable to your execution.

VH_GPUS takes an integer value greater than 0 and if not specified, it's value will be 1 - execution will occupy 1 GPU. Be careful to not assign a value that's greater than the number of available GPUs on any of the machines, in that case, execution will stay in the queue indefinitely.

Troubleshooting

Worker Not Connecting

Check Peon service status:

sudo systemctl status peon

View Peon logs:

sudo journalctl -u peon -f

Common issues:

• Incorrect Redis password

• Queue address unreachable

• Network firewall blocking port 63790

• Missing environment variables in configuration

Docker Permission Errors

If you see "permission denied" errors when running Docker:

# Add user to docker group
sudo usermod -aG docker <username>

# Log out and back in, or run:
newgrp docker

# Verify
docker ps

NVIDIA GPU Issues

Verify NVIDIA drivers:

nvidia-smi

Test GPU access in Docker:

docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

Check NVIDIA Container Toolkit installation:

docker info | grep -i nvidia

If GPU access doesn't work, verify that the NVIDIA Container Toolkit is properly installed and configured (see Manual Installation Step 1).

Jobs Not Starting

Check logs:

sudo journalctl -u peon -r

Look for errors related to:

• Docker image pull failures

• Network connectivity issues

• Storage access problems

Verify Redis connection:

# Install redis-tools if needed
apt-get install redis-tools

# Test connection (replace with your values)
redis-cli -h <queue-address> -p 63790 --tls PING

No Jobs Running or Service Stuck

Restart the Peon service:

sudo systemctl restart peon

Check for recent logs:

sudo journalctl --all --since "1 hour ago" -u peon

High Disk Usage

The Peon cleanup service should automatically remove old caches and Docker images.

Verify cleanup timer is running:

systemctl status peon-clean.timer

Manually trigger cleanup:

sudo systemctl start peon-clean

Check Docker disk usage:

docker system df

Manual cleanup:

# Remove unused Docker images
docker image prune -a

# Remove unused volumes
docker volume prune

# Full cleanup (careful - removes all unused resources)
docker system prune -a

Collecting Logs for Support

If you need to contact Valohai support, collect logs:

# Restart service to get fresh logs
sudo systemctl restart peon

# Collect last hour of logs
sudo journalctl --all --since "1 hour ago" -u peon > peon-logs.txt

Send peon-logs.txt to [email protected] with:

• Description of the issue

• Queue name

• Server specifications (CPU, RAM, GPU)

• When the issue started

Getting Help

Valohai Support: [email protected]

Include in support requests:

• Operating system and version

• Python version

• Docker version

• GPU model (if applicable)

• Peon logs (see "Collecting Logs for Support" above)

• Description of the issue and when it started