Linking Google Cloud Storage

In this guide, we’ll link a private Google Cloud Platform bucket to a Valohai project.

1. Requirements

For this tutorial you will need:

  • a Google Cloud Platform (GCP) project you can administer
  • a Valohai project which to link the bucket to

2. Create the bucket

Skip this step if you already have a bucket.

Path to GCP bucket creation

Create a bucket through Google Cloud Platform web console (https://console.cloud.google.com/storage/browser).

Main steps of the GCP bucket creation

Recommended configuration for the bucket:

  • Name: can be anything valid for GCP, here are using my-valohai-bucket as an example
  • Region: pick the region that hosts majority of the workers you’ll be using to minimize transfer
  • Storage Class: use Standard if you have no further preference
  • Access Control: Uniform (allow only bucket-level permissions)
  • Encryption: Google-managed key
  • Retention Policy: none
  • Labels: none

Keep pressing the Continue until you’ve created the bucket.

An empty GCP bucket for models and datasets

Now you have an empty bucket that you can use for your data; e.g. training datasets and models.

3. Create a service account

Next, we’ll create a new service account using the GCP console. The service account is effectively “an account” that Valohai workers use to access this particular GCP bucket.

Path to service account creation

Navigate to IAM & admin > Service accounts > Create service account

The first step of service account creation

Name your service account so that you can later remember what it’s meant for (here we are using my-valohai-bucket-admin) and press Create.

The second step of service account creation

On the next screen, you don’t need to add any roles as we will configure more limited access rights later. Just press Continue.

The last step of service account creation

Press the Create Key button and select JSON format, this will automatically download a JSON file that we’ll be using later.

The resulting JSON file will look something like this:

{
  "type": "...",
  "project_id": "...",
  "private_key_id": "...",
  "private_key": "...",
  "client_email": "my-valohai-bucket-admin@chubby.iam.gserviceaccount.com",
  "client_id": "...",
  "auth_uri": "...",
  "token_uri": "...",
  "auth_provider_x509_cert_url": "...",
  "client_x509_cert_url": "..."
}

Also, take a note of the client_email value, we’ll be using that later.

You can later find the service account email in the Service Accounts listing:

GCP console service account listing including emails

4. Allow access for the new service account

Next, we permit the new service account to access files in the bucket.

Path to bucket member management

Navigate to Storage > Browse > “your-bucket” > Permissions > Add member

Adding members to a GCP bucket
  1. New members: Copy-and-paste the service account email to the field, it will validate it. We got the service account email in the previous section.
  2. Role: Select Storage Object Admin, this allows download and uploading files.
  3. Press the Save button.