Introduce to metrics & visualisations

You can easily track key metrics and sort executions by employing custom metrics, and any JSON data printed from your code automatically becomes Valohai metadata.

You can visualize and compare metrics through various options like time series, scatter plots, or confusion matrices.

Additionally, you have the ability to establish early stopping rules using metadata values and download metadata in CSV or JSON format directly from the executions metadata tab.

Creating visualisations

You can create visualisations by either generating images and saving them in /valohai/outputs/ or using the Valohai metadata system for interactive graphs in the web application.

Python

import json

print()
print(json.dumps({
    "step": epoch,
    "accuracy": accuracy,
    "loss": loss
}))

R

library(jsonlite)

metadataEvent <- list()
metadataEvent[["step"]] <- epoch
metadataEvent[["accuracy"]] <- accuracy
metadataEvent[["loss"]] <- loss

write(toJSON(metadataEvent, auto_unbox=TRUE), stdout())

Python with the valohai-utils helper library

import valohai

with valohai.metadata.logger() as logger:
    logger.log("step", epoch)
    logger.log("accuracy", accuracy)
    logger.log("loss", loss)

Each metadata point includes an implicit _time value in UTC (e.g., 2017-04-04T15:03:39.321000).

Real-Time Charts

You can generate real-time charts based on metadata, which helps with monitoring long runs so you can stop them if training doesn’t converge well. Optionally, you can use any metadata value as the Horizontal axis value by selecting it in the corresponding dropdown menu (number 1 in the picture below).

You can add and remove graphs for different metadata by clicking on “Add/remove”. Once added, you can smooth the graph and choose the position of the vertical axis for each metadata.

You can download metadata as a CSV or JSON file from one or multiple executions directly from metadata tab (number 2 in the picture below).

You can sort executions by metadata values in the web interface under the “Executions“ tab, which is useful for e.g. finding training executions with the highest prediction accuracy.

The latest or last value of each key, such as accuracy, can be used for the sorting hyperparameter optimization results.