Linking Google Cloud Storage
In this guide, we’ll link a private Google Cloud Platform bucket to a Valohai project.
For this tutorial you will need:
- a Google Cloud Platform (GCP) project you can administer
- a Valohai project which to link the bucket to
2. Create the bucket
Skip this step if you already have a bucket.
Create a bucket through Google Cloud Platform web console (https://console.cloud.google.com/storage/browser).
Recommended configuration for the bucket:
- Name: can be anything valid for GCP, here are using
my-valohai-bucket as an example
- Region: pick the region that hosts majority of the workers you’ll be using to minimize transfer
- Storage Class: use Standard if you have no further preference
- Access Control: Uniform (allow only bucket-level permissions)
- Encryption: Google-managed key
- Retention Policy: none
- Labels: none
Keep pressing the Continue until you’ve created the bucket.
Now you have an empty bucket that you can use for your data; e.g. training datasets and models.
3. Create a service account
Next, we’ll create a new service account using the GCP console.
The service account is effectively “an account” that Valohai workers use to access this particular GCP bucket.
Navigate to IAM & admin > Service accounts > Create service account
Name your service account so that you can later remember what it’s meant for
(here we are using
my-valohai-bucket-admin) and press Create.
On the next screen, you don’t need to add any roles as we will configure more limited access rights later. Just press Continue.
Press the Create Key button and select JSON format, this will automatically download a JSON file that we’ll be using later.
The resulting JSON file will look something like this:
Also, take a note of the
client_email value, we’ll be using that later.
You can later find the service account email in the Service Accounts listing:
4. Allow access for the new service account
Next, we permit the new service account to access files in the bucket.
Navigate to Storage > Browse > “your-bucket” > Permissions > Add member
- New members: Copy-and-paste the service account email to the field, it will validate it. We got the service account email in the previous section.
- Role: Select Storage Object Admin, this allows download and uploading files.
- Press the Save button.
5. Link the store to Valohai
Navigate to Project > Settings > Data Stores > Add Google Storage
- Name: usually makes sense to use the same name as the bucket name.
- Bucket: the bucket name;
my-valohai-bucket in this example.
- Service Account JSON: copy-and-paste the contents of the JSON file we downloaded earlier.
When you create the store, the credentials provided will be checked by creating a small test file in the bucket.
If the creation succeeds, you are good to go.
Once the data store is linked, you can set it as your project’s default upload store under Settings > General > Default upload store.
This ensures that uploaded outputs will be stored in this particular GCP bucket by default.