AWS Elastic File System

Mounting provides direct access to network file systems (e.g., AWS EFS or GCP Filestore) without the need to download files.

Valohai does not version or track changes made inside the mounted file system. Files created, edited, or deleted are not versioned in Valohai executions.

You can mount a filesystem to access a large dataset, perform preprocessing, and save the processed dataset in Valohai Outputs. This ensures that a snapshot of the preprocessed data is versioned.

AWS EFS Requirements

1. Use an existing AWS EFS or create a new one.
2. Ensure your EFS is in the same VPC as Valohai resources or set up VPC peering between them.
3. Configure the EFS security group to allow inbound traffic from Valohai workers (sg-valohai-workers).
4. Ensure DNS hostnames and DNS resolution are enabled in the VPC if connecting via DNS name.

valohai.yaml

- step:
    name: mount-sample
    image: python:3.8
    command:
        - ls -la /my-data
    mounts:
        - destination: /my-data
          source: fs-1234aa62.efs.eu-west-1.amazonaws.com:/
          type: nfs
          readonly: false

REST API

You can define mounts when starting executions with an API call. You do not need to define the mount in the valohai.yaml in this case.

{
  "mounts": [
    {
      "destination": "/my-data",
      "source": "fs-1234aa62.efs.eu-west-1.amazonaws.com:/",
      "type": "nfs",
      "readonly": false
    }
  ]
}