Valohai 101 (Python)

Throughout this tutorial, we’ll walk you through running creating a new project on Valohai and running an sample Python script in Valohai.

The tutorial will start with simply getting a Python script to run on Valohai, and then continue with adding parameters, fetching input data and collecting key metrics.

Prerequirements

# Create a new directory for our sample project
mkdir valohai-quickstart
cd valohai-quickstart

# Install Valohai tools
pip install --upgrade valohai-cli valohai-utils

# Login with your Valohai username and password
vh login

# Create a new project and link the current directory to project
vh project create

A sample training script

  • Create a new file valohai-quickstart/train.py and paste the following example script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
import tensorflow as tf
import numpy

mnist = tf.keras.datasets.mnist

mnist_file_path = 'mnist.npz'

with numpy.load(mnist_file_path, allow_pickle=True) as f:
    x_train, y_train = f['x_train'], f['y_train']
    x_test, y_test = f['x_test'], f['y_test']

x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])

predictions = model(x_train[:1]).numpy()
predictions

tf.nn.softmax(predictions).numpy()

loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

loss_fn(y_train[:1], predictions).numpy()

model.compile(optimizer='adam',
            loss=loss_fn,
            metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)

save_path = 'model.h5'
model.save(save_path)

Import Valohai

Now we’ll import valohai and define a new Step.

  • Create a new the step called train-model

  • Define a default Docker image for this step. The Docker image should contain the packages we need to run our code, like Tensorflow.

1
2
3
4
5
import tensorflow as tf
import numpy
import valohai

valohai.prepare(step='train-model', image='tensorflow/tensorflow:2.4.1')

Save trained model

Finally we’ll update the save_path to a Valohai output path.

37
38
39
40
model.fit(x_train, y_train, epochs=5)

save_path = valohai.outputs().path('model.h5')
model.save(save_path)

Run in Valohai

Finally on the command line run:

vh yaml step train.py
vh exec run train-model --adhoc
  • vh yaml step uses the command-line tools to generate a valohai.yaml - Config File and a requirements.txt that contains valohai-utils (which is need to run the Python script).

  • vh exec run <step-name> creates a new Valohai execution with the step that we defined.

  • --adhoc tells Valohai that the code for this execution is coming from our local machine instead of our code repository. Valohai will package the local directory and upload it for an execution.

View in Valohai

  • Open your project on app.valohai.com

  • Open the latest execution

  • The details type contains information about the executions

  • The logs tab contains all logs from the execution

🐞 Give feedback about this page