Enable multiple peons on a single GPU machine

If your on-premises server has more than one GPU, it is possible set Valohai to use all of them for one job or run jobs in parallel. In the latter case each job will have access to one GPU and will take up as much memory/CPU as it needs. Note that the limitation is that you can only choose one of these options at a time.

Follow the steps below to set up multiple Valohai agents on your on-premises server. Note that you will have to first install the agent either with the Ubuntu installer or manually.

1. Stop the original Valohai agent with sudo systemctl stop peon and sudo systemctl disable peon
2. Rename the /etc/systemd/system/peon.service file to /etc/systemd/system/peon@.service.
3. Open the service file and add a new line in the [Service] section: Environment='EXTRA_ENVIRONMENT_VARIABLES={"NVIDIA_VISIBLE_DEVICES": "%I"}' You can place this, for example, after the EnvironmentFile=-/etc/peon.config line.
4. Add also the following line in the [Service] section: Environment="IDENTITY=UUID.%i". Replace the UUID by a UUID you have generated (you can use, for example, this generator).
5. Run sudo systemctl daemon-reload to read the updated service file.
6. Enable multiple Valohai agents by running the following commands. The example below is for a case when you have 4 GPUs. Adjust the number depending on your own server.

sudo systemctl enable --now peon@0
sudo systemctl enable --now peon@1
sudo systemctl enable --now peon@2
sudo systemctl enable --now peon@3