Private OpenStack Swift Data Store

In this guide, we’ll link a private OpenStack Swift container to a Valohai project.

1. Requirements

For this tutorial you will need:

  • an OpenStack account you can administer
  • a Valohai project which to link the Swift container to
  • any modern version of Python

2. Install OpenStack clients

First we need to install OpenStack and Swift command-line clients.

pip install python-openstackclient python-swiftclient

Then we need to authenticate to your OpenStack installation. The easiest way to authenticate is using an OpenStack RC File if your provider allows generating those. More authentication methods in OpenStack documentation.

For example:

An example how to get OpenStack RC File on Pouta OpenStack installation.

Run the OpenStack RC File:

source project-1234-openrc.sh
# it will ask your password but you can optionally edit the .sh file
# to include your password automatically by editing the OS_PASSWORD variable

Now your OpenStack client is ready for use.

3. Create a container

Create a Swift container for your Valohai project.

openstack container create my-project-data
+----------------+-----------------+------------+
| account        | container       | x-trans-id |
+----------------+-----------------+------------+
| AUTH_123456789 | my-project-data | 1234567890 |
+----------------+-----------------+------------+

openstack container show my-project-data
+--------------+--------------------------------+
| Field        | Value                          |
+--------------+--------------------------------+
| account      | AUTH_123456789                 |
| bytes_used   | 0                              |
| container    | my-project-data                |
| object_count | 0                              |
+--------------+--------------------------------+

3.1 Configure Temporary URL key

Temporary URL keys are required to give secure download privileges for Valohai workers.

# generate random string from somewhere to use as Temporary URL key, e.g.
cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1
# set the key to the Swift cluster
swift post -m "Temp-URL-Key:randomlygeneratedkey"

3.2 Configure CORS

If you wish to be able to upload files to the store using the Valohai web user interface, you will need to add a CORS (Cross-Origin Resource Sharing) policy to the Swift container.

openstack container set \
    --property Access-Control-Allow-Origin='https://app.valohai.com' \
    --property Access-Control-Max-Age='3000' \
    my-project-data

swift stat -v my-project-data
# ...
Meta Access-Control-Allow-Origin: https://app.valohai.com
Meta Access-Control-Max-Age: 3000
# ...

Now your container allows uploads from https://app.valohai.com website