Mounting a network file system (NFS)¶
This feature is available for customers who are using their own cloud or on-premises workers.
Valohai does not version or keep track of the changes made inside the mounted file system. The files that are created, edited, or deleted will not be versioned as part of a Valohai execution.
NFS mounts are not version controlled
We strongly recommend using the Valohai inputs and outputs system, as they are versioned as a part of the execution.
You can mount a filesystem to access a large dataset, run preprocessing operations, and output the processed dataset into Valohai Outputs before it’s used further in the pipeline. This way the snapshot of the preprocessed data will be versioned.
Define a mount in YAML¶
Mounts are defined for each environment seperately. Contact Valohai, if you see an error
mounts were ignored due to environment configuration. in your execution logs.
valohai.yaml specify a new mount:
You can either use an existing, or create a new EFS.
Create your EFS in the same VPC where all Valohai resources are in or setup VPC peering between the two VPCs
Make sure the security group of your EFS has an inbound rule to accept traffic from the Valohai workers (
Valohai will connect to the EFS over DNS name or IP address. Make sure the VPC has DNS hostnames and DNS resolution enabled, if you’re connecting over DNS name.
1 2 3 4 5 6 7 8 9
- step: name: mount-sample image: python:3.7 command: - ls -la /my-data mounts: - destination: /my-data source: fs-1234aa62.efs.eu-west-1.amazonaws.com:/ type: nfs
1 2 3 4 5 6 7 8 9
- step: name: mount-sample image: python:3.7 command: - ls -la /my-data mounts: - destination: /my-data source: <IP>:/mystore type: nfs
1 2 3 4 5 6 7 8
- step: name: mount-sample image: python:3.7 command: - ls -la /my-data mounts: - destination: /my-data source: /path/to/directory/outside/container