Load Files in Jobs

Valohai inputs are data files, or collections of files, accessible during an execution, fetched from your cloud storage or public sources (HTTP/HTTPS)

Supported sources include AWS S3, Azure Storage, GCP Cloud Storage, Oracle buckets, on-prem S3, or any public links.

Valohai simplifies data handling:

• Manages authentication with your cloud storage.

• Handles downloading, unpacking and caching.

• Eliminates the need to manage keys or authentication in your code.

Define an input

In your valohai.yaml, each step can have one or multiple inputs, each resulting in one or more files.

You can set default values for inputs, which can be overridden every time you create an execution. For instance, you can change the set of images for batch inference.

- step:
    name: image_processor
    environment: ec2-instance
    image: python:3.10
    command:
      - python process_image.py
    inputs:
      ## Results in one file
      ## e.g /valohai/inputs/image/<image-name>
      - name: image
        default: datum://01234567-89ab-cdef-0123-456789abcdef

      ## Results in a group of files with .jpg extension
      ## e.g
      ## /valohai/inputs/image_set/image_1.jpg
      ## /valohai/inputs/image_set/image_2.jpg
      ## ...
      - name: image_set
        default: s3://mybucket/unprocessed/*.jpg

      ## As is, will result in no files being download
      ## Later on you can override this input and make it resolve to one or more files
      - name: additional_data
        optional: true

Select single file

To select single file as an input you can use:

• Object store URL

  • Amazon S3: s3://{bucket}/{key}

  • Azure Blob Storage: azure://{account_name}/{container_name}/{blob_name}

  • Google Storage: gs://{bucket}/{key}

  • OpenStack Swift: swift://{project}/{container}/{key}

• Datum URI

  • datum://01234567-89ab-cdef-0123-456789abcdef

• Any public http/https URL

  • https://somewebsite.com/some_image

Select collection of files

Wildcards with object store URLs

• s3://my-bucket/dataset/images/*.jpg for all .jpg (JPEG) files

• s3://my-bucket/dataset/image-sets/**.jpg for recursing subdirectories for all .jpg (JPEG) files

  For example, expression above will match:

  • s3://my-bucket/dataset/image-sets/cats/big/cat-1.jpg

  • s3://my-bucket/dataset/image-sets/cats/small/cat-9.jpg

  • s3://my-bucket/dataset/image-sets/dog/dog-5.jpg

  • ...

💡 Parameter interpolation

You can also interpolate execution parameters into input URIs:

s3://my-bucket/dataset/images/{parameter:user-id}/*.jpeg would replace {parameter:user-id} with the value of the parameter user-id during an execution.

Datasets

Since datasets are actually just pointers to the collections of files, using dataset version URL as an input will result in every file included in that version to be downloaded.

An example of dataset version URL: dataset://boats/semi-processed

💡 Latest version

You can always use latest as a version, for any dataset, to point to the latest dataset version.

e.g: dataset://boats/latest

Datum queries

Using the UI you can select the files to be downloaded by executing a simple query on datum properties.

Query defined in the example above will match all datums that:

• have index property greater than 10 and have processed_date property assigned (no matter of its value) and have property type equals to "dog".

• Limit field determines the maximum number of files that will be returned by the query (it's optional)

Supported operators:

• >, >=, <, <=: Intended for use with numerals.

• ==, !=: Applicable to all data types.

• contains: Used for strings.

• exists: Checks for property existence, e.g., property_name exists true or property_name exists false.

Numerical values (e.g., 2, 2.1, 1.332) are treated as numbers. Values: true, false, True, and False are treated as booleans. Everything else is treated as a string.

❗ If a property value of a datum is set to "False" (or "True" - boolean passed as a string)

e.g. valohai.metadata.jsonl
{..., "metadata": { "property_one": "False", ... }}

such datum will not be matched with the query: property_one == False - value passed in the query will be treated as a boolean while the one that's actually attached to the datum is of the string type.

Click Add metadata filter block to add multiple metadata query blocks. The final results will be the union of each block's results.

⚠️ Non reproducible

Be aware that the results of these queries will be calculated each time you create an execution. If new files are added, with the matching properties, in between the executions, query results will not be the same - the more recent one will include newly added files as well!

Directory structure

Without any additional configuration, examples above will result in all files being downloaded in a single directory under /valohai/inputs/<input-name>/. From their original path (where they are stored in a store e.g s3://bucket-name/dir/subdir/file.jpg) only the file name (basename) will be kept (i.e. file.jpg).

To preserve the directory structure found in the store, you can use keep-directories input property.

keep-directories can take next values:

• none (default) all files are downloaded to /valohai/inputs/myinput , only the file name is kept from the actual storage path

• full

  • When selecting files using a datum URL or dataset version URL, the entire directory structure shown under the Data tab for each datum will be maintained. This structure is derived either from subdirectories in /valohai/outputs/ at upload time (for execution output files) or from the Upload path if the file is manually uploaded.

  • When files are selected using store URL or URL wildcards, full path from the storage root is kept and recreated in /valohai/inputs/<input-name>. For example s3://special-bucket/foo/bar/**.jpg will end up as /valohai/inputs/<input-name>/foo/bar/dataset1/a.jpg

• suffix

  • When selecting files using a datum URL or dataset version URL, behavior is the same as with the full option.

  • When files are selected using store URL, only the filename is preserved. For example: s3://valohai-data/data/output-1389/subdir/file.txt will result in /valohai/inputs/<input-name>/file.txt

  • When files are selected using URL wildcards, directory structure after the wildcard character will be preserved. For example s3://special-bucket/foo/bar/* the special-bucket/foo/bar/ would be removed, but any relative path after it would be kept, and you might end up with /valohai/inputs/myinput/dataset1/a.jpg

Example:

- step:
    name: image_processor
    environment: ec2-instance
    image: python:3.10
    command:
      - python process_image.py
    inputs:
      - name: image_set
        keep-directories: suffix
        default: s3://mybucket/unprocessed/*

Access files

Download location

All requested data is downloaded to a local directory /valohai/inputs on the machine used for execution - no matter if that's a cloud instance or a physical on-prem machine, behavior is the same.

Each input has its own directory structure within this location. From the example above, three directories will be created:

• /valohai/inputs/image/

• /valohai/inputs/image_set/

• /valohai/inputs/additional_data/ 💡 Only if some value is provided when the execution is created, otherwise, the directory for this input will not be created, as it's marked as optional

Using valohai-utils

valohai-utils library provides a convenient way of traversing the downloaded files just by specifying the input name, e.g:

import valohai

input_name = "training_data"

for file_path in valohai.inputs(input_name).paths():
    with open(file_path, "r") as f:
        content = f.read()
        ## Do some processing

Using system environment variables

To avoid the use of additional libraries, VH_INPUTS_DIR environment variable will be provided in every execution environment, pointing to /valohai/inputs . You can then use this path in such way:

import os

input_name = "training_data"
inputs_dir = os.getenv("VH_OUTPUTS_DIR", default="./inputs")

inputs_path = os.path.join(inputs_dir, input_name)
for file_path in os.listdir(inputs_path):
    with open(file_path, "r") as f:
        content = f.read()
        ## Do some processing

Inspect selected files

Open the Inputs tab on any past or running execution to see which datums each input resolved to.

You can also see how each file was selected:

• From a dataset (big_data)

• Through a datum alias (config)

• As a single datum (data_package)

• From a URL (dist_config)

• From URL wildcard expansion (external_samples)

• With a datum query (indexed)

In the example above, from #114 Pre-process next to big-file-4 and big-file-11 shows the execution counter and the title of the execution that produced those files.

Similarly, latest_config is the datum alias used to reference that file.