Go to Valohai AppBook a Demo

Hybrid Deployment

Deploy Valohai's compute and data layer to your GCP project using Terraform or manual setup

Deploy Valohai workers and storage to your GCP project while Valohai manages the application layer at app.valohai.com.

What Gets Deployed

The Compute and Data Layer of Valohai can be deployed to your GCP project. This enables you to:

  • Use your own Virtual Machine instances to run machine learning jobs

  • Use your own Google Storage Bucket for storing training artifacts (trained models, preprocessed datasets, visualizations)

  • Access databases and data warehouses directly from workers inside your network

Valohai doesn't have direct access to the virtual machine instances that execute machine learning jobs. Instead, it communicates with a static virtual machine in your GCP project that's responsible for storing the job queue, job states, and short-term logs.

Important: Make sure you have enough quota for both vCPUs and GPUs on your GCP account. Read more about GCP quotas.

Prerequisites

From Valohai:

Contact [email protected] to receive:

  • valohai_email - Email of the Valohai service account

  • queue_address - DNS name assigned to your queue

From your GCP account:

  • GCP project with admin access

  • Region and zone selected

  • Sufficient vCPU and GPU quotas

Installation Methods

Terraform (Recommended)

Deploy using Terraform for infrastructure as code.

Repository: github.com/valohai/gcp-hybrid-workers-terraform

Follow the instructions in the repository to deploy Valohai resources to your GCP project.

Manual Setup

Follow the manual deployment steps below for complete control over the installation.

Manual Deployment

Step 1: Configure IAM Resources

Create a custom role and two service accounts that Valohai needs to manage resources.

Create Custom Role

Navigate to GCP Project IAM & Admin → Roles.

Click Create Role with the following configuration:

Property
Value

Title

ValohaiMaster

Description

A role used by app.valohai.com to manage Valohai related resources

ID

ValohaiMaster

Role launch stage

General Availability

Permissions

Add the following permissions:

Permissions:

  • compute.disks.create

  • compute.disks.delete

  • compute.disks.setLabels

  • compute.instances.create

  • compute.instances.delete

  • compute.instances.list

  • compute.instances.setLabels

  • compute.instances.setMetadata

  • compute.instances.setServiceAccount

  • compute.instances.setTags

  • compute.subnetworks.use

  • compute.subnetworks.useExternalIp

  • compute.zones.list

Create Service Account: valohai-sa-master

Navigate to IAM & Admin → Service Accounts.

Click Create Service Account with the following configuration:

Property
Value

Name

valohai-sa-master

Description

Used to manage Valohai related VM resources in the project

Roles to assign:

Add these roles to the service account:

1. Compute Viewer

  • No conditions

2. Service Account User

  • No conditions

3. Secret Manager Secret Accessor

  • Title: Only Valohai secrets

  • Condition Editor:

resource.name.extract('/secrets/{name}/versions/') == "valohai_redis_password" || resource.name.extract('/secrets/{name}/versions/') == "valohai_master_sa"

4. ValohaiMaster (custom role)

  • Title: Only Valohai managed VMs

  • Condition Editor:

resource.name.extract("instances/{name}").startsWith("valohai") || resource.name.extract("disks/{name}").startsWith("valohai") || resource.name.extract("subnetworks/{name}").startsWith("valohai")

Grant access:

In the service account configuration, grant the valohai_email (provided by Valohai) the Service Account Token Creator role.

Create Service Account: valohai-sa-queue

Create a second service account for the queue virtual machine.

Property
Value

Name

valohai-sa-queue

Description

Service account used by the Valohai queue virtual machine

Roles to assign:

1. Service Account User

  • No conditions

2. Secret Manager Secret Accessor

  • Title: Only the Valohai redis password

  • Condition Editor:

resource.name.extract('/secrets/{name}/versions/') == "valohai_redis_password"

Step 2: Create Secret Manager Secrets

Upload two secrets that Valohai needs to operate.

Create valohai_redis_password Secret

Navigate to Security → Secret Manager.

Click Create Secret:

Property
Value

Name

valohai_redis_password

Secret value

Generate a random password with lowercase, uppercase letters, and numbers (no special characters)

Create valohai_master_sa Secret

1. Navigate to IAM & Admin → Service Accounts.

Open valohai-sa-master and go to the KEYS tab.

Click Add Key → Create new key → JSON.

The key file will be downloaded to your workstation.

2. Navigate to Security → Secret Manager.

Click Create Secret:

Property
Value

Name

valohai_master_sa

Secret value

Paste the entire JSON contents of the downloaded key file

Step 3: Create VPC

Navigate to VPC Network → VPC networks.

Click Create VPC Network:

Property
Value

Name

valohai-vpc

Subnet creation mode

Automatic

This will create subnets automatically across all regions.

Step 4: Create Firewall Rules

Navigate to VPC Network → Firewall.

Create two firewall rules for Valohai traffic.

Rule 1: valohai-fr-queue-redis

Property
Value

Name

valohai-fr-queue-redis

Description

Allows connection to the queue from Valohai services and valohai workers from this project

Network

valohai-vpc

Direction

Ingress

Action on match

Allow

Target tags

valohai-queue

Source IP Ranges

34.248.245.191/32, 63.34.156.112/32

Second source filter

Source tags: valohai-worker

Specified protocols and ports

TCP on port 63790

Rule 2: valohai-fr-queue-http

Property
Value

Name

valohai-fr-queue-http

Description

Allows connections on port 80 for the Let's Encrypt HTTP challenge

Network

valohai-vpc

Direction

Ingress

Action on match

Allow

Target tags

valohai-queue

Source IP Ranges

0.0.0.0/0

Specified protocols and ports

TCP on port 80

Step 5: Create Queue Virtual Machine

Navigate to Compute Engine → VM instances.

Click Create Instance:

Basic configuration:

  • Name: valohai-queue

  • Region: Choose your region

  • Zone: Choose your zone

  • Machine type: e2-medium

Boot disk:

  • Operating system: Ubuntu

  • Version: Ubuntu 20.04 LTS

  • Boot disk type: SSD persistent disk

  • Size: 16 GB

Identity and API access:

  • Service Account: valohai-sa-queue

Networking:

  • Network tags: valohai-queue

  • Network interface: valohai-vpc

  • External IP: Click Create IP Address → Name: valohai-ip-queue

Management:

Under the Management tab, add this startup script.

Important: Replace <queue_address> with your actual queue address from Valohai.

export QUEUE=<queue_address>
export PASSWORD=`gcloud secrets versions access 1 --secret="valohai_redis_password"`
curl https://raw.githubusercontent.com/valohai/worker-queue/main/host/setup.sh | sudo QUEUE_ADDRESS=$QUEUE REDIS_PASSWORD=$PASSWORD bash
unset PASSWORD

Click Create to launch the instance.

Step 6: Collect Information

Gather the following information to send to Valohai:

Project Details:

  • Project ID: ____________

Queue Instance:

  • External IP: ____________

Next Steps

Send the collected information to your Valohai contact at [email protected].

After Valohai confirms your environment is configured:

1. Verify the setup

  • Log in to app.valohai.com

  • Check that GCP environments appear in your organization

  • Create a test project

  • Run a simple execution to verify workers launch correctly

2. Configure additional resources

  • Add existing GCS buckets as data stores

  • Set up private Docker registries

  • Configure access to Cloud SQL databases

Getting Help

Valohai Support: [email protected]

Include in support requests:

  • Project ID

  • Region and zone

  • Error messages or logs

  • Steps already attempted

PreviousGCPNextKubernetes

Last updated

Was this helpful?