Google Cloud Filestore

Mounting provides direct access to network file systems (e.g., AWS EFS or GCP Filestore) without the need to download files.

Valohai does not version or track changes made inside the mounted file system. Files created, edited, or deleted are not versioned in Valohai executions.

You can mount a filesystem to access a large dataset, perform preprocessing, and save the processed dataset in Valohai Outputs. This ensures that a snapshot of the preprocessed data is versioned.

Google Cloud Filestore Requirements

1. You can either use an existing or create a new GCP Filestore.
2. Create your Filestore in the same VPC where all Valohai resources are in or set up VPC peering between the two VPCs

valohai.yaml

- step:
    name: mount-sample
    image: python:3.8
    command:
        - ls -la /my-data
    mounts:
        - destination: /my-data
          source: IP:/mystore
          type: nfs
          readonly: false

Replace the <IP> with your Filestore IP address, for example

REST API

You can define mounts when starting executions with an API call. You do not need to define the mount in the valohai.yaml in this case.

{
  "mounts": [
    {
      "destination": "/my-data",
      "source": "IP:/mystore",
      "type": "nfs",
      "readonly": false
    }
  ]
}