Attach tags and metadata to your files

Quick recap

  • Tagging files allows you to easily find and sort data files

  • Use can use tag names when searcing for an execution input file, or any data from your outputs

  • In addition to tags, you can send arbitrary JSON metadata to be stored with your files.

Tag a file

You can add tags to the files either in the UI or during an execution.

Add tags in the Data tab of the project, or in executions outputs.

You can save tags to your outputs by saving an additional JSON file <filename.ext>.metadata.json for each of your execution’s output files.

Everything that you store in the valohai.tags section will be used as the file’s tags in Valohai.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import valohai
import json

metadata = {
    "valohai.tags": ["production", "lemonade"]
}

save_path = valohai.outputs().path('model.h5')
model.save(save_path)

metadata_path = valohai.outputs().path('model.h5.metadata.json')
with open(metadata_path, 'w') as outfile:
    json.dump(metadata, outfile)

Find a file using tags

Once a file has been tagged you can easily find it in the UI.

  1. Create a new execution

  2. Add a new data file

  3. Search by tag name

Tip

You can also search for files by tags using the Valohai APIs (e.g. https://app.valohai.com/api/v0/data/?project=<project-id>&tag=production)

Learn more about the DatumList API.

Store arbitrary metadata with your files

You can attach additional metadata to each file you output from an execution by saving a .metadata.json-file. The file should contain, in JSON format, all the metadata you’re looking to save with the file.

Valohai will store the contents of the <filename.ext>.metadata.json-file with the output file.

You can access the files metadata using Valohai DatumList API or DatumRetrieve API.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import valohai
import json

metadata = {
    "valohai.tags": ["prod", "lemonade"], #creates Valohai tags for the file
    "key": "value",
    "sample": "metadata"
}

save_path = valohai.outputs().path('model.h5')
model.save(save_path)

metadata_path = valohai.outputs().path('model.h5.metadata.json')
with open(metadata_path, 'w') as outfile:
    json.dump(metadata, outfile)
🐞 Give feedback about this page