Track metrics – Valohai Docs

This is just a short recap

We strongly recommend completing the Mastering Valohai learning path on Valohai Academy.

This guide will provide you with a overview and a “cheatsheet” for migrating projects. It won’t explain the concepts and all the options in detail.

Easily collect and track key metrics from your executions by exposing them as Valohai metadata.

Valohai doesn’t restrict what metrics you can track, let it be accuracy, loss, f1_score, ROC, AUC, etc.

Collecting metrics, is simple. You’ll just need to print the metrics you collect as JSON to the standard out (logs). For example:

import json
import random

for i in range(10)
    print(json.dumps({
        "accuracy": random.uniform(0.6, 0.98),
        "loss": random.random(0, 0.1),
    }))

You can now:

See the latest printed values in the execution table, so you can sort by those metrics values.
Plot timeseries graphs when comparing executions or inside each execution`s metadata tab.
Set early-stopping rules based on the metrics (e.g. “stop execution when accuracy is over 0.9”)

valohai-utils

valohai-utils users can also collect metadata using the valohai.metadata.logger:

import valohai
import random

for i in range(10)
     with valohai.logger() as logger:
        logger.log('accuracy', random.uniform(0.6, 0.98))
        logger.log('loss', random.random(0, 0.1))