Add existing files to data catalog – Valohai Docs

In general, only files that go through execution or are uploaded into Valohai will get a datum URL. However, in some cases, it might be beneficial to create datum URLs for existing files in your S3 storage without needing to push them through Valohai separately.

To do this, it is possible to adopt storage URLs into real, Valohai-tracked datums. This gives all the benefits related to datum URLs, including traceability and setting aliases.

It is possible to adopt storage URLs into Valohai datums either in the web UI or by using the Valohai APIs.

Web application

Open your project
Open the Data tab
Open the Adopt tab
Select the Destination store from the dropdown menu
Add the URLs to adopt (one per line). Note that they have to be supported by the destination store.
Click Adopt selected files

Using APIs

Generate yourself an API token
Send a POST request to https://app.valohai.com/api/v0/stores/{id}/adopt/
- You can get the IDs to replace the placeholder {id} for all your stores by sending an API request to https://app.valohai.com/api/v0/stores/
In the request body, send a JSON consisting of the URL to adopt and the project ID:
- You can get the project ID from under the Settings tab in the Valohai UI

{
    "url":"s3://project-name/samplefile.txt",
    "project":"project ID"
}

If the datum adoption was successful, you should get the following response.

{
    "ok": true,
    "created": {
        "s3://my-bucket/my-file.txt": "017a515f-30a4-d0f1-d37a-53ffc38e90c7"
        }
}

In case the datum already exists, this will be indicated in the response.

{
    {
        "message": "s3://my-bucket/my-file.txt already exists in my-bucket",
        "code": null
    }
}

Finally, if the file was not found, you will be notified of it as well.

{
    {
        "message": "Not found in my-bucket: 's3://my-bucket/my-file.txt'",
        "code": "adoptable_file_not_found"
    }
}