Cache Volumes

A cache volume is a persistent volume claim that already exists in your cluster and that Valohai uses as an extra lookup location for execution inputs. When an execution downloads an input, the file is copied onto the cache volume. When a later execution asks for the same input, it is read from the volume instead of being downloaded again.

Cache volumes apply to executions on Kubernetes environments.

When you need one

• The same inputs are used by many executions, for example a training dataset that every run reads.

• Downloads from object storage are a noticeable part of the runtime of your executions.

• Egress costs from repeated downloads are a concern.

Before you start

Create the persistent volume claim in the namespace your Kubernetes environment uses, or ask your cluster administrator to create it. Give it a storage class your cluster offers and a size that fits the data you expect to cache.

Use a ReadWriteMany claim. Several executions can then use the cache at the same time, including executions running on different nodes. A ReadWriteOnce claim can only be mounted by one node at a time, which limits how many executions can use it in parallel.

Add a cache volume to an execution

1. Start creating an execution and choose a Kubernetes environment.

2. Open the Kubernetes Resource Settings section.

3. Find the Cache Volumes row.

4. Pick a claim from the list, which shows the claims available in the namespace of the environment. If there's no volume available, contact your cluster admin or Valohai support for further assistance.

5. Add more claims if you want several cache locations.

Volumes that Valohai provisions for a single execution or endpoint replica are not offered in the list, because they disappear with the workload.

Each cache volume is mounted at /mnt/valohai/<claim-name> inside the execution. You do not read or write that path from your code. Valohai manages what goes there and your inputs stay where they always are, under /valohai/inputs.

Define them in the valohai.yaml

You can define the cache volume within valohai.yaml by adding a cache-volumes configuration containing the claim you wish to use.

- step:
    name: cache-volume-example
    image: python:3.13
    command:
      - echo "hello"
    inputs:
      - name: dataset
        optional: false
        default: s3://mydata/dataset/*
    cache-volumes:
      - valohai-cache-efs

Through the API

Cache volumes are part of the Kubernetes runtime configuration.

{
  "runtime_config": {
    "kubernetes": {
      "cache_volumes": [
        {"pvc_name": "shared-input-cache"}
      ]
    }
  }
}

In a runtime config preset

A runtime config preset stores cache volumes together with the rest of the Kubernetes resource settings, so a team can pick a named preset instead of selecting the same claims for every run. Presets belong to an environment and are managed in the organization settings of that environment.

What to expect

The cache is best effort. Valohai removes old files from a cache volume when it is running low on space, so a file that was cached yesterday is not guaranteed to be there today. An execution whose cached file is gone downloads it again, which is slower but never wrong.

Nothing you write yourself belongs on a cache volume. Use outputs for results, and a workspace volume when the run needs more working space.

Troubleshooting

• Executions still download the same inputs on every run.

  • Check that the claim is ReadWriteMany if runs land on different nodes, and that the claim is large enough to hold the data. A volume that is constantly low on space is cleaned constantly.

• An execution stays pending.

  • The claim may not exist in the namespace of the environment, or it may be mounted by another workload in a mode that does not allow sharing. Check the claim name and its access mode.