# Track Underutilization

## Why this matters

If your executions consistently have low peak utilization, you're paying for hardware you're not using.

A model training on an 8-GPU instance but only using 40% of GPU capacity could run on a 4-GPU instance at half the cost. Underutilization tracking helps you spot these opportunities automatically.

## How it works

Valohai automatically highlights executions with less than **50% peak utilization** of any resource.

This threshold is currently fixed. You can specify which resources you want to track under **Project > Notifications** settings.

Valohai only reports on executions that took more than one minute to complete, filtering out short setup tasks where utilization isn't meaningful.

> 💡 *If you need a different threshold, contact your Valohai representative to help us gather requirements for upcoming features.*

## Underutilization indicators

### Alerts

Lightweight messages displayed on the execution page.

Alerts can't be turned off but can be hidden on execution lists.

![Runtime alerts on the execution view](/files/LhBBYPtw5R5sv0yeeXfh)

*Alerts as shown on the execution details page.*

![Alerts on the execution list](/files/Ak0UPoVCTQDINYWS7Nbc)

*Turn on the Alerts column in execution listings to see warnings at a glance.*

1. **Column settings:** Change Alerts column visibility here.
2. **Alerts column:** Click to navigate to the full list of alerts for an execution.

### Notifications

Receive alerts via in-app messages, email, or Slack.

* **Personal notifications:** Sent to the user who configured the notification.
* **Project notifications:** Sent to the target channel (e.g., Slack or webhook).

Configure both types in the project settings under the Notifications tab.

![Personal notifications in the project settings](/files/wrj5NejkrU0DaElJmryB)

1. **Project settings page:** Navigate here from your project.
2. **Notifications tab:** Access personal and project notifications.
3. **Personal notifications view:** Configure your own notification preferences.
4. **Toggle in-app messages:** Turn on/off in-app notifications for underutilization.
5. **Toggle email notifications:** Turn on/off email notifications for underutilization.
6. **Change scope:** Choose whether to receive notifications for your executions or all executions in the project.

![Project notifications in the project settings](/files/7c4zsvFZQbCgK8gpK7Ys)

*Project notifications are shared with all users, unlike personal notifications.*

> 💡 *Setting up project notifications requires configuring a channel as the receiver. This can be done under the **Channels** tab in the project notifications settings.*

## What to do when you get an alert

### For GPU underutilization

* **Check batch size:** Increase batch size to better utilize GPU memory and processor.
* **Switch to smaller instance:** If utilization is consistently low, use a cheaper GPU type.
* **Enable mixed precision:** Use automatic mixed precision training to increase throughput.

### For CPU underutilization

* **Reduce CPU allocation:** Switch to an instance with fewer cores.
* **Increase data loader workers:** If GPU is waiting on CPU for data loading, add more workers.

### For memory underutilization

* **Switch to instance with less RAM:** Save costs by using an instance with appropriate memory.
* **Increase batch size:** If GPU memory allows, increase batch size to better utilize system memory.

## What's next

[**Visualize utilization**](/observability/resource-monitoring/visualize-utilization.md) to understand detailed resource usage charts and identify specific bottlenecks.

[**Hardware statistics overview**](/observability/resource-monitoring/hardware-statistics.md) to learn what metrics Valohai tracks and how data is collected.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.valohai.com/observability/resource-monitoring/track-underutilization.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.