Environment Variables & Secrets

Organization-level environment variables let you share credentials and configuration across multiple projects. Define database passwords, API keys, or cloud credentials once, and grant access to specific projects.

This centralizes secret management — rotate a credential in one place instead of updating every project individually.

When to Use Organization Variables

• Shared credentials: Database connection strings, API tokens, cloud access keys used across multiple projects.

• Environment-specific config: Different settings for development, staging, and production projects.

• Team-scoped secrets: Variables accessible only to specific teams or projects.

• Simplified rotation: Update one variable group instead of editing dozens of projects.

Create Variable Groups

Variable groups bundle related environment variables and control which projects can access them.

1. Click Hi, <username> in the top-right corner

2. Select Manage <organization>

3. Open the Environment Variables tab

4. Click Add new environment variable group

5. Name your group (e.g., "production-db", "staging-credentials")

6. Select which projects can access this group

7. Add variables with the Add button

8. Mark sensitive values as Secret (click the key icon)

9. Click Save

Example: Production Credentials

Create a "production" group for live infrastructure:

Group: production Projects: production_db Variables:

• PRODUCTION_ENVVAR(public):production

• PRODUCTION_SECRET(secret):hidden

Example: Staging Environment

Create a separate "staging" group for development:

Group: staging Projects: staging_jobs Variables:

• STAGING_ENVVAR(public):staging

• STAGING_SECRET(secret):hidden

Using Variables in Projects

Variables from assigned groups appear under each project's Environment Variables tab:

For regular users:

• Variables are visible but read-only

• Secret values are hidden

• Variables are automatically available in all executions

For admins:

• Can modify values in the organization settings

• Changes apply to all projects using that group

Access in Executions

Organization variables work identically to project-level variables:

import os

# Access any organization or project variable
db_url = os.environ["DATABASE_URL"]
api_key = os.environ["API_KEY"]
environment = os.environ["ENVIRONMENT"]

print(f"Running in {environment} environment")

- step:
    name: connect-to-database
    image: python:3.12
    command:
      - pip install psycopg2-binary
      - python run_query.py
    # DATABASE_URL automatically available from org group

Security Best Practices

Mark Credentials as Secrets

Always mark sensitive values as secrets:

✅ DATABASE_URL = postgresql://user:pass@host/db (secret)
✅ PRIVATE_API_KEY = sk_live_abc123 (secret)
❌ DATABASE_URL = postgresql://user:pass@host/db (not secret - visible in UI)

Secrets are still accessible to users. Marking a variable as secret hides it in the UI, but users can still access the value in their code (e.g., print(os.environ['SECRET'])). Secrets prevent accidental exposure, not malicious access.

Separate Environments

Create distinct groups for production vs staging:

production-secrets → projects: production-ml, production-api
staging-secrets → projects: dev-experiments, staging-api

This prevents staging code from accidentally accessing production credentials.

Rotate Credentials Regularly

Update secrets in one place:

1. Generate new credentials in your service (database, API provider)

2. Update the variable value in the organization group

3. Click Save

4. All projects using that group immediately use the new credentials

Audit Secret Access

Use Audit Log to track:

• When variables were created or modified

• Which projects access which variable groups

• Login attempts and user activity

Managing Multiple Groups

Projects can access multiple variable groups simultaneously:

Example: Shared infrastructure + project-specific config

Group 1: "shared-infrastructure" (all projects)

• DOCKER_REGISTRY_URL

• SHARED_S3_BUCKET

Group 2: "ml-training-secrets" (only ML projects)

• WANDB_API_KEY

• MLFLOW_TRACKING_URI

Group 3: "production-db" (only production projects)

• DATABASE_URL

• DATABASE_PASSWORD

Assign groups based on project needs. A production ML project might use all three groups, while a staging project only gets "shared-infrastructure" and "ml-training-secrets."

Override Variables in Projects

Project-level variables override organization variables with the same name:

Organization variable:

API_ENDPOINT = https://api.production.example.com

Project override:

API_ENDPOINT = https://api.staging.example.com

The project value takes precedence, letting you test against different endpoints without modifying the organization group.

Common Patterns

Database Connections

# Organization group: "production-db"
DATABASE_URL = postgresql://user:[email protected]/mldata (secret)
DB_POOL_SIZE = 10

import os
import psycopg2

# Connection string from organization variable
conn = psycopg2.connect(os.environ["DATABASE_URL"])

API Authentication

# Organization group: "external-apis"
OPENAI_API_KEY = sk-abc123... (secret)
HUGGINGFACE_TOKEN = hf_xyz789... (secret)

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

Cloud Credentials

# Organization group: "aws-credentials"
AWS_ACCESS_KEY_ID = AKIA... (secret)
AWS_SECRET_ACCESS_KEY = abc123... (secret)
AWS_DEFAULT_REGION = us-east-1

import os
import boto3

# Credentials loaded automatically from environment
s3 = boto3.client("s3")

Private Package Repositories

# Organization group: "private-repos"
PRIVATE_PYPI_INDEX_URL = https://user:[email protected]/... (secret)
NPM_TOKEN = npm_abc123... (secret)

- step:
    name: install-private-packages
    command:
      - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
      - pip install my-private-ml-library

See Private PyPI Repositories for more details.

Troubleshooting

Variable Not Available in Execution

Cause: Variable group not assigned to the project.

Fix:

1. Go to organization Environment Variables

2. Edit the variable group

3. Add your project to the Projects list

4. Save changes

Wrong Variable Value

Cause: Project-level variable overrides organization variable.

Fix: Check project Settings → Environment Variables for conflicting names.

Secret Value Needed for Debugging

Cause: Secret values are hidden in the UI.

Fix: Temporarily print the value in an execution (delete the execution afterward):

import os

print(f"Debug: {os.environ['SECRET_VAR']}")

Related Topics

• Custom Environment Variables — Project-level variables

• Private PyPI Repositories — Use organization variables for package auth

• Audit Log — Track variable access and changes

• System Variables — Built-in Valohai variables