Google Cloud Filestore

Mount Google Cloud Filestore to access shared network storage directly from Valohai executions.

---

Overview

Google Cloud Filestore provides managed NFS storage that you can mount in Valohai executions. Use Filestore to:

• Access large datasets without downloading

• Share preprocessed data across multiple executions

• Cache intermediate results on fast shared storage

• Process data in place and save versioned outputs

⚠️ Important: Files on Filestore mounts are NOT versioned by Valohai. Always save final results to /valohai/outputs/ for reproducibility.

---

Prerequisites

Before mounting Filestore in Valohai:

1. Existing Filestore instance — Use an existing Filestore or create a new one in GCP Console

2. Same VPC or VPC peering — Filestore must be in the same VPC as Valohai resources, or set up VPC peering between VPCs

3. Network access — Filestore automatically allows access from VMs in the same VPC (no additional firewall rules needed)

---

Setup: Configure Filestore Access

Step 1: Create or Find Filestore Instance

In GCP Console:

1. Go to Filestore → Instances

2. Find your instance or click "Create Instance"

3. Note the IP address (e.g., 10.0.0.5)

4. Note the File share name (e.g., share1)

---

Step 2: Get Filestore IP Address

Via GCP Console:

1. Go to Filestore → Instances

2. Click on your instance

3. Copy the IP address from instance details

Via gcloud command:

# List all Filestore instances
gcloud filestore instances list

# Get specific instance details
gcloud filestore instances describe INSTANCE_NAME \
    --location=ZONE \
    --format="value(networks.ipAddresses[0])"

Example output:

10.0.0.5

---

Step 3: Verify VPC Configuration

Ensure Filestore and Valohai VMs are in the same VPC:

1. In GCP Console, go to Filestore → Instances

2. Check the Network column for your instance

3. Verify it matches the VPC where Valohai resources are deployed

4. If different VPCs, set up VPC peering

---

Mount Filestore in Execution

Basic Mount Configuration

valohai.yaml:

- step:
    name: process-with-filestore
    image: python:3.9
    command:
      - python process_data.py
    mounts:
      - destination: /mnt/filestore-data
        source: <ip-address>:/share1
        type: nfs
        readonly: true

Parameters:

• destination — Mount point inside container (e.g., /mnt/filestore-data)

• source — Filestore IP and share name (format: <ip-address>:/<share-name>)

• type — Always nfs for Filestore

• readonly — true (recommended) or false

---

Mount Specific Filestore Directory

mounts:
  - destination: /mnt/training-data
    source: <ip-address>:/share1/ml-datasets/training
    type: nfs
    readonly: true

Mounts only the /ml-datasets/training directory from Filestore share.

---

Complete Workflow Example

Mount → Process → Save Pattern

Scenario: Process large video dataset stored on Filestore, extract features, save to Valohai outputs.

valohai.yaml:

- step:
    name: extract-video-features
    image: python:3.9
    command:
      - pip install opencv-python numpy pandas
      - python extract_features.py
    mounts:
      - destination: /mnt/raw-videos
        source: <ip-address>:/video-storage/raw
        type: nfs
        readonly: true
    environment-variables:
      - name: SAMPLE_RATE
        default: "1"  # Process every Nth frame

extract_features.py:

import os
import cv2
import numpy as np
import pandas as pd
import json

# Configuration
FILESTORE_PATH = "/mnt/raw-videos/"
OUTPUT_PATH = "/valohai/outputs/"
SAMPLE_RATE = int(os.getenv("SAMPLE_RATE", "1"))

# 1. Scan Filestore for videos (NOT versioned)
print(f"Scanning Filestore mount: {FILESTORE_PATH}")
video_files = [f for f in os.listdir(FILESTORE_PATH) if f.endswith((".mp4", ".avi"))]
print(f"Found {len(video_files)} videos on Filestore")

# 2. Extract features from videos
features_list = []

for i, filename in enumerate(video_files):
    video_path = os.path.join(FILESTORE_PATH, filename)

    # Read video from Filestore
    cap = cv2.VideoCapture(video_path)

    frame_count = 0
    video_features = []

    while True:
        ret, frame = cap.read()
        if not ret:
            break

        # Sample frames based on SAMPLE_RATE
        if frame_count % SAMPLE_RATE == 0:
            # Extract features (e.g., histogram)
            hist = cv2.calcHist([frame], [0, 1, 2], None, [8, 8, 8], [0, 256, 0, 256, 0, 256])
            hist = hist.flatten()
            video_features.append(hist)

        frame_count += 1

    cap.release()

    # Aggregate features for this video
    if video_features:
        avg_features = np.mean(video_features, axis=0)

        features_list.append(
            {
                "video_filename": filename,
                "frame_count": frame_count,
                "sampled_frames": len(video_features),
                "features": avg_features.tolist(),
            },
        )

    if (i + 1) % 10 == 0:
        print(f"Processed {i + 1}/{len(video_files)} videos...")

# 3. Save features to Valohai outputs (VERSIONED ✅)
print(f"\nSaving {len(features_list)} video features...")

# Save as CSV for analysis
df = pd.DataFrame(
    [
        {
            "video_filename": item["video_filename"],
            "frame_count": item["frame_count"],
            "sampled_frames": item["sampled_frames"],
        }
        for item in features_list
    ],
)
df.to_csv(os.path.join(OUTPUT_PATH, "video_metadata.csv"), index=False)

# Save features as numpy array
features_array = np.array([item["features"] for item in features_list])
np.save(os.path.join(OUTPUT_PATH, "video_features.npy"), features_array)

# 4. Create dataset version
dataset_metadata = {
    "video_metadata.csv": {
        "valohai.dataset-versions": [
            {
                "uri": "dataset://video-features/batch-001",
            },
        ],
        "valohai.tags": ["video-analysis", "features", "extracted"],
    },
    "video_features.npy": {
        "valohai.dataset-versions": [
            {
                "uri": "dataset://video-features/batch-001",
            },
        ],
    },
}

metadata_path = os.path.join(OUTPUT_PATH, "valohai.metadata.jsonl")
with open(metadata_path, "w") as f:
    for filename, file_meta in dataset_metadata.items():
        json.dump({"file": filename, "metadata": file_meta}, f)
        f.write("\n")

print(f"\nFeature extraction complete:")
print(f"  - Processed {len(features_list)} videos")
print(f"  - Extracted features shape: {features_array.shape}")
print(f"  - Created dataset: dataset://video-features/batch-001")

Result:

• ✅ Raw videos accessed from Filestore (no download time for 100GB+ videos)

• ✅ Extracted features saved to /valohai/outputs/ (versioned)

• ✅ Dataset created for reproducible model training

• ✅ Can train on dataset://video-features/batch-001 anytime

---

Best Practices

Use Readonly for Input Data

# ✅ Good: Readonly prevents accidental modifications
mounts:
  - destination: /mnt/training-data
    readonly: true

# ⚠️ Avoid: Writeable unless necessary
mounts:
  - destination: /mnt/training-data
    readonly: false

---

Always Version Final Results

# ❌ Bad: Only use Filestore, nothing versioned
results = process_data("/mnt/filestore-data/")
results.save("/mnt/filestore-output/results.pkl")  # NOT versioned

# ✅ Good: Filestore for input, outputs for results
results = process_data("/mnt/filestore-data/")
results.save("/valohai/outputs/results.pkl")  # VERSIONED

---

Organize Your Filestore Structure

/share1/
├── raw-data/
│   ├── images/
│   ├── videos/
│   └── text/
├── cache/
│   └── preprocessing/
├── team-workspace/
│   ├── experiments/
│   └── notebooks/
└── temp/

Clear organization makes mounting and access control easier.

---

Handle Mount Errors

import os
import sys

FILESTORE_PATH = "/mnt/filestore-data/"

# Verify mount is accessible
if not os.path.exists(FILESTORE_PATH):
    print(f"ERROR: Filestore mount {FILESTORE_PATH} not accessible")
    print("Possible causes:")
    print("  - Wrong IP address in mount configuration")
    print("  - VPC connectivity issue")
    print("  - Filestore instance not running")
    sys.exit(1)

# Verify expected data exists
expected_dir = os.path.join(FILESTORE_PATH, "datasets")
if not os.path.exists(expected_dir):
    print(f"WARNING: Expected directory not found: {expected_dir}")
    print(f"Available: {os.listdir(FILESTORE_PATH)}")

print(f"Filestore mount verified: {FILESTORE_PATH}")

---

Maintaining Reproducibility

⚠️ Critical: Filestore data can change between executions. Always save processed results to /valohai/outputs/ for versioning.

The problem:

# Today: Process data from Filestore
data = load_from_filestore("/mnt/filestore-data/")
model = train(data)

# Next week: Someone updates Filestore data
# Retraining gives different results
# Can't reproduce original model

The solution:

# Load from Filestore (current state)
data = load_from_filestore("/mnt/filestore-data/")

# Save snapshot to versioned outputs
data.to_csv("/valohai/outputs/training_snapshot.csv")

# Create dataset version
metadata = {
    "training_snapshot.csv": {
        "valohai.dataset-versions": [
            {
                "uri": "dataset://training-data/2024-01-15",
            },
        ],
    },
}

# Train on versioned snapshot in next execution
# Can reproduce exactly anytime

See: Access Network Storage for complete patterns.

---

Related Pages

• Access Network Storage — Overview and when to use NFS

• AWS Elastic File System — AWS equivalent

• On-Premises NFS — Mount on-prem storage

• Load Data in Jobs — Alternative: Valohai's versioned inputs

---

Next Steps

• Set up Filestore in your GCP project (or use existing)

• Get Filestore IP address using gcloud or Console

• Create test execution mounting Filestore

• Build pipeline: mount → process → save to outputs

• Monitor Filestore performance in GCP Console