> For the complete documentation index, see [llms.txt](https://docs.valohai.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.valohai.com/executions/execution-alerts.md). # Execution Alerts Valohai automatically inspects every execution and records alerts when it spots something that may have gone unnoticed: an oversized worker, duplicate input files, a dropped runtime setting. Alerts help you catch wasted compute and misconfigurations without reading through logs. {% hint style="info" %} Alerts are **recorded automatically** on the execution itself. There is nothing to subscribe to and nothing to configure. If you want to be actively told when something happens, [see Outgoing Webhook Notifications](/automation-overview/outgoing-webhooks.md). {% endhint %} *** ### What Are Alerts? An alert is an advisory note attached to a single execution. It does not change the execution's status, an execution with alerts can still complete successfully. Alerts point at things worth reviewing. Usually cost, data correctness, or a setting that did not apply the way you expected. #### **Common use cases** * Spot executions running on machines far larger than the job needs * Catch input files that silently overwrite each other * Understand why a runtime setting like automatic restart did not take effect * Find out why an output dataset version was not created * See when log output was too fast to be stored in full Alerts are never sent anywhere on their own and cannot be dismissed or deleted. They stay on the execution as part of its record. *** ### Where to Find Alerts #### Alerts Tab on an Execution Open any execution and select the **Alerts** tab. The tab shows a count badge, and lists each alert with its time, type, and message. The tab is disabled when the execution has no alerts. #### Alerts Column in the Executions Table The executions table has an **Alerts** column that is **hidden by default**. To enable it: 1. Open your project's **Executions** list 2. Open the column settings 3. Enable the **Alerts** column Executions with alerts then show an orange count and a warning icon. Hover it for a preview of the messages, or click it to jump straight to the execution's Alerts tab. #### Inputs Tab Alerts that concern a specific input are also shown on that input's row on the **Inputs** tab, so you can see which input is affected without cross-referencing. #### Logs Tab If log output was rate limited, a notice appears directly in the execution's log view, where you would notice the missing lines. #### Productivity Dashboard The [productivity dashboard](/observability/productivity-dashboard.md) reports GPU utilization across your organization together with a count of utilization alerts for the selected period, which is useful for spotting waste at the organization level rather than per execution. *** ### Alert Reference
AlertWhat it means
CPU was underutilizedPeak CPU use stayed below half of what the worker provided
Memory was underutilizedPeak memory use stayed below half of what the worker provided
GPU processor was underutilizedPeak GPU use stayed below half of what the worker provided
GPU memory was underutilizedPeak GPU memory use stayed below half of what the worker provided
Single input has files with same namesOne input expanded to several files sharing a filename
Dataset version creation failedAn output dataset version could not be created
Automatic restart is not available on this environmentautorestart was requested but dropped, because the environment is not a spot environment
Log rate limit exceededThe execution produced log output faster than it could be stored
*** ### Resource Underutilization These four alerts are the most common, and the most directly tied to cost. Valohai compares the peak resource use recorded during the execution against what the worker actually provided. An underutilization alert is recorded when: * The execution ran for **at least 60 seconds**, and * Peak use of a resource stayed **below 50%** of what was available The message names the resource and the peak figure, for example: ``` The worker GPU memory was underutilized with 12% peak use. ``` Resources the worker did not have are skipped, so a CPU-only execution never reports GPU alerts. #### **How to act on it** * switch the step to a smaller environment * increase batch size or parallelism so the machine you are paying for is actually used. A single alert on a short experiment is rarely worth chasing; the same alert repeating across a task or a scheduled pipeline usually is. {% hint style="warning" %} The check runs once, when the execution finishes. Executions shorter than a minute are never evaluated, because peak figures from a very short run are not meaningful. {% endhint %} #### Getting Notified About Underutilization Underutilization is the one alert family that can also reach you as a notification. These notifications are **enabled by default** for new projects as in-app notifications. To review or change them: 1. Open **My Notifications** from your user settings 2. Find the events **worker CPU was underutilized**, **worker memory was underutilized**, **worker GPU processor was underutilized**, and **worker GPU memory was underutilized** 3. Enable or disable the targets you want (in-app, email) To send them to a shared channel instead: 1. Open your project settings and navigate to the **Notifications** tab 2. Create a channel under **Channels** if you do not have one yet 3. Under **Project Notifications**, create a routing for the underutilization events to that channel If one execution triggers several underutilization alerts, they are combined into a single notification listing each one, linking to the execution's Alerts tab. *** ### Single Input Has Files With Same Names Raised when one input expands to two or more files that share a filename, for example a wildcard URL matching `data/january/results.csv` and `data/february/results.csv`. Because inputs are downloaded into one directory per input, only one of them survives. The message names the input and the filename, and states the consequence: ``` Input 'training-data' has multiple files with the same name 'results.csv'. Only the first one will be used. ``` #### **How to act on it** * split the files into separate inputs * rename them so each filename is unique within the input This check runs shortly after the execution is created, so the alert can appear before the execution has even started. Inputs referenced as datums are not affected. *** ### Dataset Version Creation Failed Raised when an execution finished but an output dataset version described in its metadata could not be created. The execution itself is unaffected: the outputs are stored as normal, only the dataset version is missing. #### Common causes include * A dataset version name that is already in use * An invalid version name, or the reserved name `latest` * Targeting a dataset version that is already published * Conflicting `from` or `start_fresh` values for the same version * A version alias that points at a different dataset than the version * Model version information that could not be parsed Each problem is recorded as its own alert with the specific reason, so the Alerts tab tells you exactly what to fix in your dataset version metadata. *** ### Automatic Restart Is Not Available on This Environment Raised when an execution requested `autorestart` but the setting was dropped, because automatic restart only applies to spot (pre-emptible) environments. On a regular on-demand instance there is no interruption to restart from, so the setting has no meaning and is removed. The message names the environment: ``` 'autorestart' ignored on AWS eu-west-1 g4dn.xlarge: only available on spot environments. ``` #### **How to act on it** * if you want automatic restarts, choose a spot environment for the step * if you intended to run on an on-demand instance, no action is needed, the alert just confirms the setting was not silently kept. The same explanation is also written to the execution's event stream, but the alert makes it easier to notice. *** ### Log Rate Limit Exceeded Raised when an execution produced log output faster than Valohai could store it, meaning some log lines were dropped. This is most common with very chatty training loops or debug logging inside tight loops. The limit is enforced by the worker running the job, so the exact rate depends on the worker your execution landed on. Alongside the Alerts tab, this alert is shown as a notice in the execution's log view, so you can see that the logs you are reading are incomplete. #### **How to act on it** * reduce log verbosity * log progress at intervals rather than every step * write detailed logs to an output file instead of standard output. Metrics are better recorded as execution metadata than printed as log lines. *** ### Reading Alerts via the API You can fetch an execution's alerts programmatically: ```shell curl -H "Authorization: Token $VH_TOKEN" \ https://app.valohai.com/api/v0/executions//alerts/ ``` The response is a list of alerts: ```json [ { "id": "01H8Z3QK7B9YV2N4T6M0XA1CDE", "execution": "0178f482-8686-fb28-8af1-3d36af38079d", "type": "underutilized_gpu_memory", "message": "The worker GPU memory was underutilized with 12% peak use.", "ctime": "2026-09-10T09:14:22.481000+00:00" } ] ``` #### Key Fields * `type`**:** Machine-readable alert type — one of `underutilized_cpu`, `underutilized_memory`, `underutilized_gpu_proc`, `underutilized_gpu_memory`, `input_has_duplicates`, `dataset_version_creation`, `autorestart_unavailable`, `log_rate_limit` * `message`**:** Human-readable explanation, including the specific resource, input, or environment involved * `ctime`**:** When the alert was recorded * `execution`**:** ID of the execution the alert belongs to An execution's alert count is also available as `n_alerts` on the execution itself, so you can find executions worth inspecting without fetching every alert. #### **Common use cases** * Report weekly on underutilized executions per project or per team * Flag executions whose inputs contained duplicate filenames before trusting their results * Build a cost dashboard that ranks steps by how often they over-provision *** ### Next Steps * **Sending events to Slack or your own service?** See [Outgoing Webhook Notifications](/automation-overview/outgoing-webhooks.md) * **Choosing the right machine size?** See [Environments](/user-and-organization-management/environments-and-access-control/configure-environments.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.valohai.com/executions/execution-alerts.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.