Mounting provides direct access to network file systems (e.g., AWS EFS or GCP Filestore) without the need to download files.
Valohai does not version or track changes made inside the mounted file system. Files created, edited, or deleted are not versioned in Valohai executions.
You can mount a filesystem to access a large dataset, perform preprocessing, and save the processed dataset in Valohai Outputs. This ensures that a snapshot of the preprocessed data is versioned.
NFS mounts are not version-controlled.
We strongly recommend using Valohai’s inputs and outputs system, which versions data as part of the execution process.
AWS EFS Requirements
- Use an existing AWS EFS or create a new one.
- Ensure your EFS is in the same VPC as Valohai resources or set up VPC peering between them.
- Configure the EFS security group to allow inbound traffic from Valohai workers (sg-valohai-workers).
- Ensure DNS hostnames and DNS resolution are enabled in the VPC if connecting via DNS name.
valohai.yaml
- step:
name: mount-sample
image: python:3.8
command:
- ls -la /my-data
mounts:
- destination: /my-data
source: fs-1234aa62.efs.eu-west-1.amazonaws.com:/
type: nfs
readonly: false
REST API
You can define mounts when starting executions with an API call. You do not need to define the mount in the valohai.yaml in this case.
{
"mounts": [
{
"destination": "/my-data",
"source": "fs-1234aa62.efs.eu-west-1.amazonaws.com:/",
"type": "nfs",
"readonly": false
}
]
}
The information about the mount is not currently preserved in the execution details when launched from an API. This means that copying the execution from the Valohai web app does not include the mounting when the execution has been started with an REST API call.