Private PyPI Repositories

Use JFrog Artifactory to host private Python packages and share them securely across your organization with fine-grained access control.

This guide shows how to authenticate Valohai executions with your private PyPI repository. The same pattern works for other private PyPI hosts, adjust the authentication method as needed.

Generate Artifactory Credentials

1. Log into your JFrog Cloud Platform

2. Navigate to Repositories

3. Select Set Up Client/CI Tool for your PyPI repository

4. Copy the full index-url value (starts with http:// or https://)

The URL includes embedded credentials and looks like:

https://username:[email protected]/artifactory/api/pypi/pypi-local/simple

Store Credentials in Valohai

Save your PyPI credentials securely as environment variables:

Organization-Level (Recommended)

Share credentials across all projects:

1. Navigate to Hi, <name> → Manage <organization>

2. Open the Environment Variables tab

3. Click Create new environment variable group

4. Add variable:

   • Name: PRIVATE_PYPI_INDEX_URL

   • Value: Paste the full index URL from JFrog

   • Check Secret (click the key icon)

5. Select which projects can access this group

6. Click Save

See Environment Variables & Secrets for detailed instructions.

Project-Level

For project-specific repositories:

1. Open your project

2. Go to Settings → Environment Variables

3. Add the PRIVATE_PYPI_INDEX_URL variable as a secret

Use in Executions

Configure pip to use your private repository before installing packages:

In valohai.yaml

- step:
    name: train-with-private-package
    image: python:3.12
    command:
      - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
      - pip install my-private-package
      - pip install scikit-learn  # Still works for public packages
      - python train.py

The pip config command tells pip to check your private repository first. It falls back to public PyPI for packages not in your private registry.

Install Multiple Private Packages

- step:
    name: install-multiple-private
    image: python:3.12
    command:
      - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
      - pip install my-ml-library my-data-utils my-viz-tools
      - python run_pipeline.py

Mix Private and Public Packages

Pip checks your private repository first, then falls back to PyPI:

command:
  - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
  - pip install -r requirements.txt

requirements.txt:

my-private-models==1.2.3      # From your Artifactory
torch==2.0.0                   # From public PyPI
transformers==4.30.0           # From public PyPI
my-company-utils>=2.0.0        # From your Artifactory

Alternative: Use Extra Index URL

If you want to keep public PyPI as primary and add your private repo as supplementary:

command:
  - pip install --extra-index-url $PRIVATE_PYPI_INDEX_URL my-private-package
  - python train.py

This checks public PyPI first, then your private registry. Useful when most packages are public and only a few are private.

Verify Authentication

Test that authentication works:

- step:
    name: test-private-repo
    image: python:3.12
    command:
      - echo "Testing private PyPI access..."
      - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
      - pip install --dry-run my-private-package
      - echo "Authentication successful"

If authentication fails, you'll see 401 or 403 errors in the logs.

Multiple Private Repositories

If you have multiple private PyPI repositories:

- step:
    name: multi-repo-install
    image: python:3.12
    command:
      - pip config set global.index-url $PRIMARY_PYPI_URL
      - pip config set global.extra-index-url "$SECONDARY_PYPI_URL"
      - pip install package-from-primary package-from-secondary
      - python train.py
    environment-variables:
      - name: PRIMARY_PYPI_URL
        default: https://user:[email protected]/...
      - name: SECONDARY_PYPI_URL
        default: https://user:[email protected]/...

Mark repository URLs as secrets to avoid exposing credentials in logs or UI.

Security Best Practices

Never commit credentials: Don't put credentials directly in valohai.yaml or code. Always use environment variables marked as secrets.

Rotate credentials regularly: Update the PRIVATE_PYPI_INDEX_URL value when rotating JFrog tokens or passwords.

Use scoped tokens: Configure JFrog to issue read-only tokens for CI/CD access rather than using admin credentials.

Audit access: Use Audit Log to track which executions accessed private packages.

Troubleshooting

401 Unauthorized

Cause: Invalid credentials in the index URL.

Fix: Regenerate credentials in JFrog and update the PRIVATE_PYPI_INDEX_URL environment variable.

404 Not Found

Cause: Package doesn't exist in your private repository.

Fix: Verify the package name and version. Check that it's published to your Artifactory instance.

Connection Timeout

Cause: Network connectivity issues or incorrect URL.

Fix: Test the URL manually:

curl -I $PRIVATE_PYPI_INDEX_URL

Mixed Public/Private Failures

Cause: Using --index-url completely replaces PyPI, causing public packages to fail.

Fix: Use --extra-index-url to keep both registries:

pip install --extra-index-url $PRIVATE_PYPI_INDEX_URL package-name

Related Topics

• Environment Variables & Secrets — Manage credentials organization-wide

• Private Docker Registries — Pull private Docker images from JFrog

• Custom Docker Images — Bake private packages into images

• JFrog Artifactory Documentation — Official pip authentication guide