Dataset properties

Datasets and dataset versions support structured key-value properties. A similar metadata system is already available for individual datums and is now extended to both datasets and dataset versions. Use properties to describe your datasets with rich, structured metadata instead of working around the limitation by overloading tags or naming conventions.

---

Datums vs. Datasets vs. Dataset Versions

Properties exist at three levels, each describing a different scope:

• Datum properties describe a single file (a specific image, model artifact, CSV, etc.). See the related documentation page for the file-level metadata system.

• Dataset properties describe stable attributes of a dataset that hold across all of its versions.

• Dataset version properties describe metadata specific to one version of a dataset.

A property is a key-value pair where the value can be a string, number, boolean, or nested JSON object — the same shape as datum metadata.

---

When to Use Dataset vs. Dataset Version Properties

Use dataset properties for attributes that don't change between versions: the subject domain, the source equipment, the species being studied, the project owner. If you'd answer the same way regardless of which version someone is looking at, it belongs on the dataset.

{
  "domain": "manufacturing",
  "machine": "press-04",
  "material": "stainless-steel-304",
  "project_owner": "[e-mail]"
}

Use dataset version properties for attributes that are specific to one snapshot: when the data was captured, how many samples it contains, which preprocessing pipeline produced it, what the validation results were.

{
  "capture_date": "2026-01-15",
  "sample_count": 4700,
  "preprocessing_version": "3.2",
  "shift": "night",
  "defect_rate": 0.002
}

---

How to Add Properties

Similarly to the datum properties, it is possible to add them to datasets and dataset versions programmatically during executions or via the Valohai API. In addition, it is also possible to manage the properties in the web UI. Editing properties is possible via the API and in the web UI.

💡 Some properties are managed automatically by the platform and use the valohai. prefix. For example, valohai.last-used records when a dataset was last referenced which is useful for identifying stale or unused datasets.

In the Web Application

Open any dataset or dataset version detail page to see its Properties section. You can add, edit, and remove properties directly in the UI by clicking on the Edit properties button. Each entry is a key-value pair, and values accept strings, numbers, or booleans. Additionally, you can also edit the entries as JSON.

During Execution

Properties for dataset versions can be set from your execution code through the valohai.metadata.jsonl sidecar file, the same mechanism used for datum metadata. Properties written this way are staged during the execution and applied automatically when the dataset version finishes building, so metadata isn't lost if the version completes while a write is still in progress.

# Define the properties for dataset
# Make sure to replace <dataset-name> with the actual name
dataset_metadata = {
    "dataset://<dataset-name>": {
      "domain": "manufacturing",
      "machine": "press-04",
      "material": "stainless-steel-304",
      "project_owner": "[e-mail]"
    }
}

# Define the properties for dataset version
# Make sure to replace <dataset-name> and <dataset-version-name> with the actual names
dataset_version_metadata = {
    "dataset://<dataset-name>/<dataset-version-name>": {
        "capture_date": "2026-01-15",
        "sample_count": 4700,
        "preprocessing_version": "3.2",
        "shift": "night",
        "defect_rate": 0.002
    }
}

# Add the information to the valohai.metadata.jsonl
metadata_path = "/valohai/outputs/valohai.metadata.jsonl"
with open(metadata_path, "w") as outfile:
    for dataset_name, ds_metadata in dataset_metadata.items():
        json.dump({"dataset": dataset_name, "metadata": ds_metadata}, outfile)
        outfile.write("\n")
    for dataset_version_name, dsv_metadata in dataset_version_metadata.items():
        json.dump({"dataset-version": dataset_version_name, "metadata": dsv_metadata}, outfile)
        outfile.write("\n")

💡 For details on the sidecar file format, see Custom Properties.

Via the API

Use the properties API endpoints to set, update, or remove properties on existing datasets and dataset versions. This is useful for backfilling metadata, editing properties outside an execution, or integrating with external systems.

To add or edit dataset properties, send a POST request to /api/v0/datasets/<dataset-id>/metadata/ API endpoint. The properties are defined in the payload:

{
  "domain": "manufacturing",
  "machine": "press-04",
  "material": "stainless-steel-304",
  "project_owner": "[e-mail]"
}

Similarly, to add or edit the dataset version properties, you can send a POST request to /api/v0/dataset-versions/<dataset-version-id>/metadata/ .

💡 Made a mistake and want to remove the properties? You can do this by setting the value to null in the payload. The following would delete all the four properties from the dataset:

{
  "domain": null,
  "machine": null,
  "material": null,
  "project_owner": null
}

---

Filtering and Search

Because properties are structured, you can filter and search datasets by property values rather than relying on naming conventions or tag matching. This makes it practical to find all datasets matching a specific attribute, or all versions whose metadata meets a numeric threshold.

💡 Querying is not available for dataset versions at the moment. If you would need this feature discuss it with your Valohai contact.

---

Properties vs. Tags

Tags are flat labels — fine for simple categorization, but awkward once you need typed values, distinct keys, or filtering by specific fields. If you've been encoding metadata into dataset names or piling it into tags, properties are the structured replacement.

---

Related Pages

• Custom Properties — Properties on individual datums (files)

• Add Context to Your Data Files — Overview of the metadata system

• Organize Files with Tags — How to label datums with tags