# Hardware Statistics

## Why hardware statistics matter

GPU hours are expensive. If you're requesting 8 GPUs but only using 40% of their capacity, you're burning money on idle hardware.

Hardware statistics give you real-time visibility into whether you're actually using the resources you're requesting. This lets data scientists optimize their compute configurations and helps platform engineers identify systemic inefficiencies.

## What Valohai tracks

Valohai records the following metrics out-of-the-box for every execution:

* **CPU:** Utilization of the CPU processor.
* **Memory:** Utilization of system memory.
* **GPU Processor:** Utilization of GPU computational resources.
* **GPU Memory:** Utilization of GPU memory resources.

The availability of these metrics depends on the runtime environment. For example, executions on CPU-only machines won't have GPU metrics.

![Workload resource utilization as shown on the execution page. Click the ▼ to expand.](/files/7FsCuXYj8QBuNzBsHXBl)

## How statistics are collected

Usage metrics are collected from the host machine where the execution runs.

**Default collection interval:** 2 minutes.

Each statistics entry includes average, maximum, and minimum values for each metric over that interval.

**Real-time statistics** are collected at finer granularity for live monitoring during execution.

### Statistics data structure

```json
{
  "version": 2,
  "start_time": 1715682773.4884605,
  "end_time": 1715682894.42783,
  "n_entries": 61,
  "min": {
    "cpu_usage": 1.407035175879397e-05,
    "num_cpus": 20,
    "memory_usage_kb": 9632,
    "memory_total_kb": 65272740,
    "network_rx_kb": 1330,
    "network_tx_kb": 27,
    "num_gpus": 1,
    "gpu_usage": 0.10007844033914715,
    "gpu_memory_usage_kb": 808730,
    "gpu_memory_total_kb": 8072192
  },
  "max": {
    "cpu_usage": 6.872157655381506e-05,
    "num_cpus": 20,
    "memory_usage_kb": 9632,
    "memory_total_kb": 65272740,
    "network_rx_kb": 1332,
    "network_tx_kb": 27,
    "num_gpus": 1,
    "gpu_usage": 0.8991820020962256,
    "gpu_memory_usage_kb": 7260324,
    "gpu_memory_total_kb": 8072192
  },
  "avg": {
    "cpu_usage": 4.246493545678705e-05,
    "num_cpus": 20,
    "memory_usage_kb": 9632,
    "memory_total_kb": 65272740,
    "network_rx_kb": 1331,
    "network_tx_kb": 27,
    "num_gpus": 1,
    "gpu_usage": 0.4994439844256729,
    "gpu_memory_usage_kb": 3971050.295081967,
    "gpu_memory_total_kb": 8072192
  }
}
```

## Features

### Real-time monitoring

Track your CPU, memory, and GPU utilization in real-time while executions are running.

### Automatic visualizations

Visualize your resource utilization directly in the Valohai UI without additional configuration.

### Historical data access

Access and analyze historical data to identify trends and optimize resource allocation over time.

### Alerts and notifications

Receive notifications when resources are underutilized:

* **Alerts** are displayed on the execution page and are less intrusive.
* **Notifications** are more configurable in-app, email, and Slack messages.

## What's next

Now that you understand what Valohai tracks, learn how to:

[**Visualize utilization**](/observability/resource-monitoring/visualize-utilization.md) to interpret resource usage charts.

[**Track underutilization**](/observability/resource-monitoring/track-underutilization.md) to identify and fix inefficient resource allocation.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.valohai.com/observability/resource-monitoring/hardware-statistics.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.