> For the complete documentation index, see [llms.txt](https://docs.valohai.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.valohai.com/data/kubernetes-volumes/workspace-volumes.md). # Workspace Volumes A workspace volume gives one workload a disk of its own. Valohai asks Kubernetes for a volume of the size and storage class you specify, mounts it for that execution or endpoint replica, and removes it afterwards. Without one, the workload writes to the disk of the node and competes for that space with everything else scheduled there. A workspace volume always starts empty. It is there for capacity and disk speed, not for keeping data. ### When you need one * Your inputs, checkpoints or intermediate files are larger than the free disk on a node. * Executions fail with out of space errors, or pods get evicted because the node is under disk pressure. * Your job is heavy on disk I/O and you want it on faster storage than the node offers. * Your endpoint downloads a model or builds a cache at startup and needs room for it. If you do not set one, nothing changes. The workload keeps using the disk of the node. ### Storage classes The storage class decides what kind of disk you get, for example a local NVMe disk, a network SSD or cheap bulk storage. The names come from your own Kubernetes cluster, so ask your cluster administrator which ones are available. A name that does not exist in the cluster means the volume cannot be provisioned and the workload waits instead of starting. Sizes are given in gibibytes and the smallest volume you can ask for is 1 GiB. ### Workspace volumes for executions For an execution, the workspace volume backs /valohai, the directory that holds the repository, the inputs and the outputs of the run. #### Set one for a single execution 1. Start creating an execution and choose a Kubernetes environment. 2. Open the Kubernetes Resource Settings section. 3. Find the Workspace Volume row. 4. Enter the size in gibibytes, for example 200. 5. Enter the storage class offered by your cluster, for example gp3 or fast-ssd. 6. Create the execution. Size and storage class go together. Set both or leave both empty. If you fill in only one, the form tells you to complete the pair or clear it. When an execution has finished, its workspace volume settings are shown on the execution details page with the rest of the Kubernetes configuration. #### Reuse the same settings with a runtime config preset Typing a size and a storage class for every run does not scale across a team. A runtime config preset stores the workspace volume together with the CPU, memory, device and cache volume settings, and appears as a named option when you create an execution. Presets belong to an environment and are managed in the organization settings of that environment. The preset listing shows the workspace volume size and storage class in its summary, so you can tell presets apart without opening them. A preset can also be selected for pipeline nodes and task nodes, and pinned in `valohai.yaml` for a step. ```yaml - step: name: train image: python:3.13 command: python train.py environment: my-kubernetes-environment runtime-config-preset: large-workspace ``` An environment can also carry default runtime configuration, which applies a workspace volume to every execution on that environment unless someone overrides it. Contact Valohai support if you want defaults set for an environment. #### Set one through the API Include the workspace volume in the runtime configuration of the execution creation payload. ```json { "project": "", "commit": "", "step": "train", "environment": "", "runtime_config": { "kubernetes": { "workspace_volume": { "size": 200, "storage_class_name": "fast-ssd" } } } } ``` The size is an integer in gibibytes and the storage class is the name of a storage class in your cluster. Alternatively pass `runtime_config_preset` with the slug or ID of a preset, together with an explicit `environment`. #### What happens to the data The directories of the run are cleared before your code starts, so files written by an earlier execution are never visible to a later one. To move data between runs, save it as an output and use it as an input, or mount a cache volume for input data that is downloaded repeatedly. You do not need to create, resize or delete anything in Kubernetes yourself. Valohai handles the lifecycle of the volume. ### Workspace volumes for deployment endpoints For a deployment endpoint, you choose where the volume is mounted, and each replica gets its own volume. #### Add one when creating a deployment version 1. Create a new deployment version. 2. Open the resource configuration of the endpoint. 3. Find the Workspace Volume row and select Add workspace volume. 4. Enter the size in gibibytes. 5. Enter the mount path, the absolute path inside the container where the disk appears, for example `/my/cache`. 6. Optionally enter a storage class. Leave it empty to use the default storage class of the cluster. Size and mount path are both required. The mount path cannot be the same as the mount path of a shared volume on the same endpoint. Remove a workspace volume with the X button on the row. The configured volume is listed on the endpoint details page after the version is created. #### Define one in valohai.yaml Keeping the definition in version control means you do not retype it for every version. Valohai pre-fills the deployment version form from the YAML whenever the request does not set the field itself. ```yaml - endpoint: name: predict image: python:3.13 server-command: python predict.py workspace-volume: size: 10 mount-path: /my/cache storage-class-name: fast-ssd ``` `storage-class-name` is optional and falls back to the default storage class of the cluster. #### Sizing and lifetime Every replica of the endpoint gets a separate volume, so the storage the endpoint reserves is the number of replicas multiplied by the size. An endpoint with 3 replicas and a 10 GiB workspace volume uses 30 GiB. The volume lives as long as the replica pod. Its contents do not survive a restart of the replica, re-enabling the endpoint or a new deployment version. That suits a model downloaded at startup, a package or model cache, or large temporary files written while serving requests. When the data has to outlive the replica, or be identical for every replica, use a shared volume instead. Endpoint workspace volumes require a deployment target that supports persistent storage. They are not available on public deployment targets. ### Requirements and limits * Kubernetes environments and Kubernetes deployment targets only. * Minimum size is 1 GiB. Sizes are given in gibibytes. * Storage class names must exist in your cluster and follow Kubernetes naming rules. * For executions, size and storage class are both required when a workspace volume is used. * For endpoints, size and mount path are required, and the storage class is optional. * Endpoint workspace volumes are not available on public deployment targets. ### Troubleshooting * An execution or endpoint stays pending and never starts. * The most common cause is a storage class name that does not exist in the cluster, or a cluster that cannot satisfy the requested size. Check the storage class name against the list your cluster administrator provides. * The endpoint cannot write to the mounted path. * Valohai deployment images run as the valohai user, and the mounted volume is given to that user's group. If you build a custom image that runs as a different user, make sure that user can write to the mount path. * Files written by a previous run are missing. * This is expected. A workspace volume is cleared for every execution and every replica start. Use outputs and inputs, or a cache volume, when data needs to outlive the run. * Saving a deployment version fails with a mount path conflict. * The workspace volume and a shared volume are using the same mount path. Give the workspace volume a different path. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.valohai.com/data/kubernetes-volumes/workspace-volumes.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.