Table of Contents:

Connecting to AWS S3Ā¶

In this guide, weā€™ll link a private AWS S3 bucket to a Valohai project.

Data Stores can be either configured on the project level, or shared across your organization.

This guide will show you how to configure a Data Store for a single project, but you can follow the same steps to configure a shared Data Store under your settings.

You can also promote a previously created Data Store from a project owned by your organization to a shared store.

1. RequirementsĀ¶

For this tutorial you will need:

  • an Amazon Web Services (AWS) account you can administer
  • a Valohai project which to link the S3 bucket to

2. Create the S3 bucketĀ¶

Create an S3 bucket through AWS console (https://s3.console.aws.amazon.com/s3/home).

2.1 Select bucket name and regionĀ¶

S3 bucket creation, page 1
  1. Throughout this guide, we will assume the name of the bucket is my-valohai-bucket; be sure to replace this with the actual name of your bucket when copying in any example configuration!
  2. Create the bucket in the region youā€™ll be running your training to minimize data transfer costs. If you donā€™t have a preference, we recommend using Ireland (eu-west-1) as most of our computation resides there.

2.2 Use default bucket properties & permissionsĀ¶

S3 bucket creation, page 2

Default bucket properties are fine, but double check that your bucket is not public. You can of course edit the default settings based on your needs.

Create a new bucket.

3. Configure CORS for the S3 bucketĀ¶

If you wish to be able to upload files to the store using the app.valohai.com web UI, you will need to add a CORS policy document to the S3 bucket.

S3 bucket with the CORS setting

First you navigate to the AWS S3 bucket you created.

Then you go to the Permissions tab and scroll down to Cross-origin resource sharing (CORS).

Click Edit add the rules below.

[
   {
      "AllowedHeaders": [
            "Authorization"
      ],
      "AllowedMethods": [
            "GET"
      ],
      "AllowedOrigins": [
            "*"
      ],
      "ExposeHeaders": [],
      "MaxAgeSeconds": 3000
   },
   {
      "AllowedHeaders": [
            "Authorization"
      ],
      "AllowedMethods": [
            "POST"
      ],
      "AllowedOrigins": [
            "https://app.valohai.com"
      ],
      "ExposeHeaders": [],
      "MaxAgeSeconds": 3000
   }
]

Now your bucket allows POSTs for your user on https://app.valohai.com website

4. Create an IAM userĀ¶

IAM home page

Using the AWS console, start creating a new IAM user with programmatic access credentials (access key ID / secret access key).

4.1 Select name for your programmatic userĀ¶

IAM user creation, page 1
  1. User name can be anything, try to be descriptive.
  2. Double check that programmatic access is turned on.

4.2 Skip the permission configurationĀ¶

We will add permissions later, you can skip to the next step.

4.3 Save access key ID and secret for later usageĀ¶

IAM user creation, page 3

Download the CSV or copy-paste the ā€œAccess key IDā€ and ā€œSecret access keyā€ somewhere safe.

Tip

If you lose these credentials, you can generate new ones though IAM > Select user > Security credentials > Create access key.

5. Allow the IAM user to access the bucketĀ¶

Now we have a user without any permissions, letā€™s allow the user to access our new bucket.

IAM user listing

Find and open the user you created in the previous section.

IAM user inline policy location

Add a new inline policy. You can use any other AWS IAM policy definition methods just as well. Inline policies are the easiest get started.

IAM user inline policy definition

The user needs to have full access to the S3 bucket; an example of a suitable access policy document is below. Make sure to change the resource name my-valohai-bucket!

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::my-valohai-bucket",
                "arn:aws:s3:::my-valohai-bucket/*"
            ]
        }
    ]
}
IAM user policy creation review page

Give your policy a descriptive name and we are done with the mandatory AWS setup!

5. Large file uploads (optional)Ā¶

If executions need to upload outputs larger than 5 GB, additional setup is needed. This is optional and only required for large outputs.

To upload large outputs using Amazonā€™s multi-part upload API, a temporary AWS IAM role will be dispensed to the worker machines when required.

Be sure to replace the following placeholders in the following policy examples!

  • BUCKET ā€“ the target S3 bucket
  • ACCOUNTNUMBER ā€“ your AWS account number
  • USERNAME ā€“ the username liked to the access keys that are being used with the store

Create a new AWS IAM Role. The role policy document should look like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "MultipartAccess",
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions",
                "s3:ListMultipartUploadParts",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET",
                "arn:aws:s3:::BUCKET/*"
            ]
        }
    ]
}

The trust relationship document should look like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNTNUMBER:user/USERNAME"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Take note of the roleā€™s AWS ARN (arn:aws:...), that will be configured to your Valohai project.