Using Valohai Ecosystem Libraries
Last updated
Was this helpful?
Valohai maintains a collection of pre-built library steps for common workflows. These are production-ready and available to all organizations.
Run SQL queries and save results to your data store:
All database connectors:
Accept SQL queries as parameters
Save results as CSV outputs
Support both credential-based and machine identity authentication
Automatically version query results in your data store
Build and push Docker images without installing Docker locally:
Supports AWS ECR, GCP Artifact Registry, and Docker Hub.
Ecosystem libraries are automatically available in your organization. No setup required.
Open your project
Click Create Execution under the Executions tab
Expand the step library by clicking the + next to valohai-ecosystem in the left panel
Select a library step (e.g., bigquery-query)
Configure parameters and environment variables
Click Create Execution
Library steps run like any other execution—same logs, same outputs, same metadata tracking.
Let's run a BigQuery query and save the results:
Under your project Settings or as an organization-wide environment variable group:
GCP_PROJECT — Your GCP project ID
GCP_IAM — Set to 1 to use machine identity, or 0 for keyfile auth
GCP_KEYFILE_CONTENTS_JSON — (If using keyfile) Service account JSON
Select the bigquery-query step from valohai-ecosystem
Write your SQL query in the query parameter:
(Optional) Set an output path like top_users.csv
(Optional) Add a datum alias like latest-user-stats for easy reference
Click Create Execution
The query runs on BigQuery, and results are saved to your data store. Use the output in other executions with datum://latest-user-stats.
No setup needed: No YAML to write, no Git repository to manage. Just run.
Battle-tested: These steps are maintained by Valohai and used across hundreds of organizations.
Versioned results: Query outputs are automatically tracked and versioned in your data store.
Consistent patterns: All connectors work the same way—write a query, get a CSV. Easy to learn once and reuse everywhere.
Database connectors:
Build custom images:
Create your own:
Last updated
Was this helpful?
Was this helpful?
SELECT user_id, COUNT(*) as events
FROM `my-project.analytics.events`
WHERE date >= '2025-01-01'
GROUP BY user_id
ORDER BY events DESC
LIMIT 100