Ctrlk
Go to Valohai AppBook a Demo
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Execution Alerts

Valohai automatically inspects every execution and records alerts when it spots something that may have gone unnoticed: an oversized worker, duplicate input files, a dropped runtime setting. Alerts help you catch wasted compute and misconfigurations without reading through logs.

Alerts are recorded automatically on the execution itself. There is nothing to subscribe to and nothing to configure. If you want to be actively told when something happens, see Outgoing Webhook Notifications.

What Are Alerts?

An alert is an advisory note attached to a single execution. It does not change the execution's status, an execution with alerts can still complete successfully. Alerts point at things worth reviewing. Usually cost, data correctness, or a setting that did not apply the way you expected.

Common use cases

  • Spot executions running on machines far larger than the job needs

  • Catch input files that silently overwrite each other

  • Understand why a runtime setting like automatic restart did not take effect

  • Find out why an output dataset version was not created

  • See when log output was too fast to be stored in full

Alerts are never sent anywhere on their own and cannot be dismissed or deleted. They stay on the execution as part of its record.

Where to Find Alerts

Alerts Tab on an Execution

Open any execution and select the Alerts tab. The tab shows a count badge, and lists each alert with its time, type, and message. The tab is disabled when the execution has no alerts.

Alerts Column in the Executions Table

The executions table has an Alerts column that is hidden by default. To enable it:

  1. Open your project's Executions list

  2. Open the column settings

  3. Enable the Alerts column

Executions with alerts then show an orange count and a warning icon. Hover it for a preview of the messages, or click it to jump straight to the execution's Alerts tab.

Inputs Tab

Alerts that concern a specific input are also shown on that input's row on the Inputs tab, so you can see which input is affected without cross-referencing.

Logs Tab

If log output was rate limited, a notice appears directly in the execution's log view, where you would notice the missing lines.

Productivity Dashboard

The productivity dashboard reports GPU utilization across your organization together with a count of utilization alerts for the selected period, which is useful for spotting waste at the organization level rather than per execution.

Alert Reference

Alert
What it means

CPU was underutilized

Peak CPU use stayed below half of what the worker provided

Memory was underutilized

Peak memory use stayed below half of what the worker provided

GPU processor was underutilized

Peak GPU use stayed below half of what the worker provided

GPU memory was underutilized

Peak GPU memory use stayed below half of what the worker provided

Single input has files with same names

One input expanded to several files sharing a filename

Dataset version creation failed

An output dataset version could not be created

Automatic restart is not available on this environment

autorestart was requested but dropped, because the environment is not a spot environment

Log rate limit exceeded

The execution produced log output faster than it could be stored

Resource Underutilization

These four alerts are the most common, and the most directly tied to cost. Valohai compares the peak resource use recorded during the execution against what the worker actually provided.

An underutilization alert is recorded when:

  • The execution ran for at least 60 seconds, and

  • Peak use of a resource stayed below 50% of what was available

The message names the resource and the peak figure, for example:

Resources the worker did not have are skipped, so a CPU-only execution never reports GPU alerts.

How to act on it

  • switch the step to a smaller environment

  • increase batch size or parallelism so the machine you are paying for is actually used.

A single alert on a short experiment is rarely worth chasing; the same alert repeating across a task or a scheduled pipeline usually is.

The check runs once, when the execution finishes. Executions shorter than a minute are never evaluated, because peak figures from a very short run are not meaningful.

Getting Notified About Underutilization

Underutilization is the one alert family that can also reach you as a notification. These notifications are enabled by default for new projects as in-app notifications.

To review or change them:

  1. Open My Notifications from your user settings

  2. Find the events worker CPU was underutilized, worker memory was underutilized, worker GPU processor was underutilized, and worker GPU memory was underutilized

  3. Enable or disable the targets you want (in-app, email)

To send them to a shared channel instead:

  1. Open your project settings and navigate to the Notifications tab

  2. Create a channel under Channels if you do not have one yet

  3. Under Project Notifications, create a routing for the underutilization events to that channel

If one execution triggers several underutilization alerts, they are combined into a single notification listing each one, linking to the execution's Alerts tab.

Single Input Has Files With Same Names

Raised when one input expands to two or more files that share a filename, for example a wildcard URL matching data/january/results.csv and data/february/results.csv. Because inputs are downloaded into one directory per input, only one of them survives.

The message names the input and the filename, and states the consequence:

How to act on it

  • split the files into separate inputs

  • rename them so each filename is unique within the input

This check runs shortly after the execution is created, so the alert can appear before the execution has even started. Inputs referenced as datums are not affected.

Dataset Version Creation Failed

Raised when an execution finished but an output dataset version described in its metadata could not be created. The execution itself is unaffected: the outputs are stored as normal, only the dataset version is missing.

Common causes include

  • A dataset version name that is already in use

  • An invalid version name, or the reserved name latest

  • Targeting a dataset version that is already published

  • Conflicting from or start_fresh values for the same version

  • A version alias that points at a different dataset than the version

  • Model version information that could not be parsed

Each problem is recorded as its own alert with the specific reason, so the Alerts tab tells you exactly what to fix in your dataset version metadata.

Automatic Restart Is Not Available on This Environment

Raised when an execution requested autorestart but the setting was dropped, because automatic restart only applies to spot (pre-emptible) environments. On a regular on-demand instance there is no interruption to restart from, so the setting has no meaning and is removed.

The message names the environment:

How to act on it

  • if you want automatic restarts, choose a spot environment for the step

  • if you intended to run on an on-demand instance, no action is needed, the alert just confirms the setting was not silently kept.

The same explanation is also written to the execution's event stream, but the alert makes it easier to notice.

Log Rate Limit Exceeded

Raised when an execution produced log output faster than Valohai could store it, meaning some log lines were dropped. This is most common with very chatty training loops or debug logging inside tight loops.

The limit is enforced by the worker running the job, so the exact rate depends on the worker your execution landed on.

Alongside the Alerts tab, this alert is shown as a notice in the execution's log view, so you can see that the logs you are reading are incomplete.

How to act on it

  • reduce log verbosity

  • log progress at intervals rather than every step

  • write detailed logs to an output file instead of standard output.

Metrics are better recorded as execution metadata than printed as log lines.

Reading Alerts via the API

You can fetch an execution's alerts programmatically:

The response is a list of alerts:

Key Fields

  • type: Machine-readable alert type — one of underutilized_cpu, underutilized_memory, underutilized_gpu_proc, underutilized_gpu_memory, input_has_duplicates, dataset_version_creation, autorestart_unavailable, log_rate_limit

  • message: Human-readable explanation, including the specific resource, input, or environment involved

  • ctime: When the alert was recorded

  • execution: ID of the execution the alert belongs to

An execution's alert count is also available as n_alerts on the execution itself, so you can find executions worth inspecting without fetching every alert.

Common use cases

  • Report weekly on underutilized executions per project or per team

  • Flag executions whose inputs contained duplicate filenames before trusting their results

  • Build a cost dashboard that ranks steps by how often they over-provision

Next Steps

PreviousSystem Configuration Files
NextAdvanced Features

Last updated

Was this helpful?