Add Google Cloud Storage

In this guide, we’ll link a private Google Cloud Platform bucket to a Valohai project.


  • Google Cloud Platform project that you can administer

  • A Valohai project which to link the Goolge Storage to

Create the bucket

Using an existing Google Cloud Store

You can skip this part and go directly to the next section, if you’re using an existing Cloud Storage

Create a bucket through Google Cloud Platform web console.

Recommended configuration for the bucket:

  • Name: can be anything valid for GCP, here are using sample-valohai-bucket as an example

  • Region: pick the region that hosts majority of the workers you’ll be using to minimize transfer.

  • Storage Class: use Standard if you have no further preference

  • Access Control: Uniform (allow only bucket-level permissions)

  • Encryption: Google-managed key

  • Retention Policy: none

  • Labels: none

Keep pressing the Continue until you’ve created the bucket.

Path to GCP bucket creation

Now you have an empty bucket that you can use for your data; e.g. training datasets and models.

Create a service account

Next, we’ll create a new service account using the GCP console. The service account is effectively “an account” that Valohai workers use to access this particular GCP bucket.

Path to service account creation

Navigate to IAM & admin > Service accounts > Create service account

The first step of service account creation

Name your service account so that you can later remember what it’s meant for (here we are using my-valohai-bucket-admin) and press Create.

The second step of service account creation

On the next screen, you don’t need to add any roles as we will configure more limited access rights later. Just press Continue.

The last step of service account creation

Press the Create Key button and select JSON format, this will automatically download a JSON file that we’ll be using later.

The resulting JSON file will look something like this:

  "type": "...",
  "project_id": "...",
  "private_key_id": "...",
  "private_key": "...",
  "client_email": "",
  "client_id": "...",
  "auth_uri": "...",
  "token_uri": "...",
  "auth_provider_x509_cert_url": "...",
  "client_x509_cert_url": "..."

Also, take a note of the client_email value, we’ll be using that later.

You can later find the service account email in the Service Accounts listing:

GCP console service account listing including emails

Allow access for the new service account

Next, we permit the new service account to access files in the bucket.

Path to bucket member management

Navigate to Storage > Browse > “your-bucket” > Permissions > Add member

Adding members to a GCP bucket
  1. New members: Copy-and-paste the service account email to the field, it will validate it. We got the service account email in the previous section.

  2. Role: Select Storage Object Admin, this allows download and uploading files.

  3. Press the Save button.

Set CORS settings for your bucket

Click on “Activate Google Cloud Shell” in the upper right corner.

  • Create a new CORS configuration file
    • echo '[{"origin": ["*"],"responseHeader": ["Content-Type", "x-ms-*"],"method": ["GET", "HEAD", "OPTIONS"],"maxAgeSeconds": 3600}, {"origin": [""],"responseHeader": ["Content-Type", "x-ms-*"],"method": ["POST", "PUT"],"maxAgeSeconds": 3600}]' > cors-config.json

  • Update the CORS settings for your bucket
    • gsutil cors set cors-config.json gs://<your-bucket-name>

  • Check the CORS settings
    • gsutil cors get gs://<your-bucket-name>