Connecting to OpenStack Swift

In this guide, we’ll link a private OpenStack Swift container to a Valohai project.

Data Stores can be either configured on the project level, or shared across your organization.

This guide will show you how to configure a Data Store for a single project, but you can follow the same steps to configure a shared Data Store under your settings.

You can also promote a previously created Data Store from a project owned by your organization to a shared store.

1. Requirements

For this tutorial you will need:

  • an OpenStack account you can administer
  • a Valohai project which to link the Swift container to
  • any modern version of Python

2. Install OpenStack clients

First we need to install OpenStack and Swift command-line clients.

pip install python-openstackclient python-swiftclient

Then we need to authenticate to your OpenStack installation. The easiest way to authenticate is using an OpenStack RC File if your provider allows generating those. More authentication methods in OpenStack documentation.

For example:

An example how to get OpenStack RC File on Pouta OpenStack installation.

Run the OpenStack RC File:

# it will ask your password but you can optionally edit the .sh file
# to include your password automatically by editing the OS_PASSWORD variable

Now your OpenStack client is ready for use.

3. Create a container

Create a Swift container for your Valohai project.

openstack container create my-project-data
| account        | container       | x-trans-id |
| AUTH_123456789 | my-project-data | 1234567890 |

openstack container show my-project-data
| Field        | Value                          |
| account      | AUTH_123456789                 |
| bytes_used   | 0                              |
| container    | my-project-data                |
| object_count | 0                              |

3.1 Configure Temporary URL key

Temporary URL keys are required to give secure download privileges for Valohai workers.

# generate random string from somewhere to use as Temporary URL key, e.g.
cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1
# set the key to the Swift cluster
swift post -m "Temp-URL-Key:randomlygeneratedkey"

3.2 Configure CORS

If you wish to be able to upload files to the store using the Valohai web user interface, you will need to add a CORS (Cross-Origin Resource Sharing) policy to the Swift container.

openstack container set \
    --property Access-Control-Allow-Origin='' \
    --property Access-Control-Max-Age='3000' \

swift stat -v my-project-data
# ...
Meta Access-Control-Allow-Origin:
Meta Access-Control-Max-Age: 3000
# ...

Now your container allows uploads from website