Save files

With Valohai, you can save various types of files as outputs, such as trained models, CSV files, images, and more.

Output Storage

Valohai automatically stores outputs in the project’s default data store, which includes AWS S3, Azure Storage, GCP Cloud Storage, and OpenStack Swift.

You can customise this per execution or step, by changing the output storage.

By default, all outputs are saved to your data store at the end of an execution.

Save files mid-execution

You can use Live outputs to save files during an execution rather than waiting until the end. Using live outputs sets the file as read-only, which signals to the Valohai platform that this file can be uploaded immediately.

Python

import os
from stat import S_IREAD, S_IRGRP, S_IROTH

filename = "path/to/file"
os.chmod(filename, S_IREAD|S_IRGRP|S_IROTH)

Python with valohai-utils

# Query an output path for the filename "resized_image.png"
out_path = valohai.outputs().path("resized_image.png")

# Save the file to the output path
new_image.save(out_path)

# Request for an immediate upload
valohai.outputs().live_upload("resized_image.png")

# Carry on doing something else...

Unique Links

Each output is assigned a unique datum:// link, allowing easy file access for downloading or use in another execution.

File Saving Location

To save and upload files to your cloud storage, simply write your files to the /valohai/outputs/ folder. Your saved files will appear under the “Outputs” tab of your execution and in the project’s “Data” tab.

Code examples

Once you’ve saved a file to outputs and it’s been uploaded you can use the copied datum:// link as an input in another execution. For example, use a link to a trained model file in a new inference execution.

Python

# Define a filepath in Valohai outputs directory
# e.g. /valohai/outputs/filename.ext
out_path = '/valohai/outputs/mydata.csv'
df.to_csv(out_path)

R

# Get the location of Valohai outputs directory
vh_outputs_path <- Sys.getenv("VH_OUTPUTS_DIR", unset = ".outputs")

# Define a filepath in Valohai outputs directory
# e.g. /valohai/outputs/>filename.ext>
out_path <- file.path(vh_outputs_path, "mydata.csv")
write.csv(output, file = out_path)

Python with valohai-utils

import valohai
# Define a path in Valohai outputs for "myfile.csv"
out_path = valohai.outputs().path('mydata.csv')
df.to_csv(out_path)