# Hybrid Deployment

Deploy Valohai workers and storage to your GCP project while Valohai manages the application layer at app.valohai.com.

## What Gets Deployed

The Compute and Data Layer of Valohai can be deployed to your GCP project. This enables you to:

* Use your own Virtual Machine instances to run machine learning jobs
* Use your own Google Storage Bucket for storing training artifacts (trained models, preprocessed datasets, visualizations)
* Access databases and data warehouses directly from workers inside your network

Valohai doesn't have direct access to the virtual machine instances that execute machine learning jobs. Instead, it communicates with a static virtual machine in your GCP project that's responsible for storing the job queue, job states, and short-term logs.

> **Important:** Make sure you have enough quota for both vCPUs and GPUs on your GCP account. [Read more about GCP quotas](https://cloud.google.com/compute/resource-usage).

## Prerequisites

**From Valohai:**

Contact **<support@valohai.com>** to receive:

* `valohai_email` - Email of the Valohai service account
* `queue_address` - DNS name assigned to your queue

**From your GCP account:**

* GCP project with admin access
* Region and zone selected
* Sufficient vCPU and GPU quotas

## Installation Methods

### Terraform (Recommended)

Deploy using Terraform for infrastructure as code.

**Repository:** [github.com/valohai/gcp-hybrid-workers-terraform](https://github.com/valohai/gcp-hybrid-workers-terraform)

Follow the instructions in the repository to deploy Valohai resources to your GCP project.

### Manual Setup

Follow the manual deployment steps below for complete control over the installation.

## Manual Deployment

### Step 1: Configure IAM Resources

Create a custom role and two service accounts that Valohai needs to manage resources.

#### Create Custom Role

Navigate to **GCP Project IAM & Admin → Roles**.

Click **Create Role** with the following configuration:

<table><thead><tr><th width="180.40234375">Property</th><th>Value</th></tr></thead><tbody><tr><td>Title</td><td><code>ValohaiMaster</code></td></tr><tr><td>Description</td><td>A role used by app.valohai.com to manage Valohai related resources</td></tr><tr><td>ID</td><td><code>ValohaiMaster</code></td></tr><tr><td>Role launch stage</td><td>General Availability</td></tr><tr><td>Permissions</td><td>Add the following permissions:</td></tr></tbody></table>

**Permissions:**

* `compute.disks.create`
* `compute.disks.delete`
* `compute.disks.setLabels`
* `compute.instances.create`
* `compute.instances.delete`
* `compute.instances.list`
* `compute.instances.setLabels`
* `compute.instances.setMetadata`
* `compute.instances.setServiceAccount`
* `compute.instances.setTags`
* `compute.subnetworks.use`
* `compute.subnetworks.useExternalIp`
* `compute.zones.list`

#### Create Service Account: valohai-sa-master

Navigate to **IAM & Admin → Service Accounts**.

Click **Create Service Account** with the following configuration:

<table><thead><tr><th width="168.82421875">Property</th><th>Value</th></tr></thead><tbody><tr><td>Name</td><td><code>valohai-sa-master</code></td></tr><tr><td>Description</td><td>Used to manage Valohai related VM resources in the project</td></tr></tbody></table>

**Roles to assign:**

Add these roles to the service account:

**1. Compute Viewer**

* No conditions

**2. Service Account User**

* No conditions

**3. Secret Manager Secret Accessor**

* Title: Only Valohai secrets
* Condition Editor:

```
resource.name.extract('/secrets/{name}/versions/') == "valohai_redis_password" || resource.name.extract('/secrets/{name}/versions/') == "valohai_master_sa"
```

**4. ValohaiMaster (custom role)**

* Title: Only Valohai managed VMs
* Condition Editor:

```
resource.name.extract("instances/{name}").startsWith("valohai") || resource.name.extract("disks/{name}").startsWith("valohai") || resource.name.extract("subnetworks/{name}").startsWith("valohai")
```

**Grant access:**

In the service account configuration, grant the `valohai_email` (provided by Valohai) the **Service Account Token Creator** role.

#### Create Service Account: valohai-sa-queue

Create a second service account for the queue virtual machine.

<table><thead><tr><th width="154.828125">Property</th><th>Value</th></tr></thead><tbody><tr><td>Name</td><td><code>valohai-sa-queue</code></td></tr><tr><td>Description</td><td>Service account used by the Valohai queue virtual machine</td></tr></tbody></table>

**Roles to assign:**

**1. Service Account User**

* No conditions

**2. Secret Manager Secret Accessor**

* Title: Only the Valohai redis password
* Condition Editor:

```
resource.name.extract('/secrets/{name}/versions/') == "valohai_redis_password"
```

### Step 2: Create Secret Manager Secrets

Upload two secrets that Valohai needs to operate.

#### Create valohai\_redis\_password Secret

Navigate to **Security → Secret Manager**.

Click **Create Secret**:

<table><thead><tr><th width="141.9296875">Property</th><th>Value</th></tr></thead><tbody><tr><td>Name</td><td><code>valohai_redis_password</code></td></tr><tr><td>Secret value</td><td>Generate a random password with lowercase, uppercase letters, and numbers (no special characters)</td></tr></tbody></table>

#### Create valohai\_master\_sa Secret

**1.** Navigate to **IAM & Admin → Service Accounts**.

Open `valohai-sa-master` and go to the **KEYS** tab.

Click **Add Key → Create new key → JSON**.

The key file will be downloaded to your workstation.

**2.** Navigate to **Security → Secret Manager**.

Click **Create Secret**:

<table><thead><tr><th width="145.73046875">Property</th><th>Value</th></tr></thead><tbody><tr><td>Name</td><td><code>valohai_master_sa</code></td></tr><tr><td>Secret value</td><td>Paste the entire JSON contents of the downloaded key file</td></tr></tbody></table>

### Step 3: Create VPC

Navigate to **VPC Network → VPC networks**.

Click **Create VPC Network**:

<table><thead><tr><th width="208.77734375">Property</th><th>Value</th></tr></thead><tbody><tr><td>Name</td><td><code>valohai-vpc</code></td></tr><tr><td>Subnet creation mode</td><td>Automatic</td></tr></tbody></table>

This will create subnets automatically across all regions.

### Step 4: Create Firewall Rules

Navigate to **VPC Network → Firewall**.

Create two firewall rules for Valohai traffic.

#### Rule 1: valohai-fr-queue-redis

<table><thead><tr><th width="262.37890625">Property</th><th>Value</th></tr></thead><tbody><tr><td>Name</td><td><code>valohai-fr-queue-redis</code></td></tr><tr><td>Description</td><td>Allows connection to the queue from Valohai services and valohai workers from this project</td></tr><tr><td>Network</td><td><code>valohai-vpc</code></td></tr><tr><td>Direction</td><td>Ingress</td></tr><tr><td>Action on match</td><td>Allow</td></tr><tr><td>Target tags</td><td><code>valohai-queue</code></td></tr><tr><td>Source IP Ranges</td><td><code>34.248.245.191/32</code>, <code>63.34.156.112/32</code></td></tr><tr><td>Second source filter</td><td>Source tags: <code>valohai-worker</code></td></tr><tr><td>Specified protocols and ports</td><td>TCP on port <code>63790</code></td></tr></tbody></table>

#### Rule 2: valohai-fr-queue-http

<table><thead><tr><th width="264">Property</th><th>Value</th></tr></thead><tbody><tr><td>Name</td><td><code>valohai-fr-queue-http</code></td></tr><tr><td>Description</td><td>Allows connections on port 80 for the Let's Encrypt HTTP challenge</td></tr><tr><td>Network</td><td><code>valohai-vpc</code></td></tr><tr><td>Direction</td><td>Ingress</td></tr><tr><td>Action on match</td><td>Allow</td></tr><tr><td>Target tags</td><td><code>valohai-queue</code></td></tr><tr><td>Source IP Ranges</td><td><code>0.0.0.0/0</code></td></tr><tr><td>Specified protocols and ports</td><td>TCP on port <code>80</code></td></tr></tbody></table>

### Step 5: Create Queue Virtual Machine

Navigate to **Compute Engine → VM instances**.

Click **Create Instance**:

**Basic configuration:**

* Name: `valohai-queue`
* Region: Choose your region
* Zone: Choose your zone
* Machine type: `e2-medium`

**Boot disk:**

* Operating system: Ubuntu
* Version: Ubuntu 20.04 LTS
* Boot disk type: SSD persistent disk
* Size: 16 GB

**Identity and API access:**

* Service Account: `valohai-sa-queue`

**Networking:**

* Network tags: `valohai-queue`
* Network interface: `valohai-vpc`
* External IP: Click **Create IP Address** → Name: `valohai-ip-queue`

**Management:**

Under the **Management** tab, add this startup script.

**Important:** Replace `<queue_address>` with your actual queue address from Valohai.

```bash
export QUEUE=<queue_address>
export PASSWORD=`gcloud secrets versions access 1 --secret="valohai_redis_password"`
curl https://raw.githubusercontent.com/valohai/worker-queue/main/host/setup.sh | sudo QUEUE_ADDRESS=$QUEUE REDIS_PASSWORD=$PASSWORD bash
unset PASSWORD
```

Click **Create** to launch the instance.

### Step 6: Collect Information

Gather the following information to send to Valohai:

**Project Details:**

* Project ID: `____________`

**Queue Instance:**

* External IP: `____________`

## Next Steps

Send the collected information to your Valohai contact at **<support@valohai.com>**.

After Valohai confirms your environment is configured:

**1. Verify the setup**

* Log in to app.valohai.com
* Check that GCP environments appear in your organization
* Create a test project
* Run a simple execution to verify workers launch correctly

**2. Configure additional resources**

* Add existing GCS buckets as data stores
* Set up private Docker registries
* Configure access to Cloud SQL databases

## Getting Help

**Valohai Support:** <support@valohai.com>

**Include in support requests:**

* Project ID
* Region and zone
* Error messages or logs
* Steps already attempted


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.valohai.com/installation-and-setup/gcp/hybrid.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.