# Private PyPI Repositories

Use JFrog Artifactory to host private Python packages and share them securely across your organization with fine-grained access control.

This guide shows how to authenticate Valohai executions with your private PyPI repository. The same pattern works for other private PyPI hosts, adjust the authentication method as needed.

## Generate Artifactory Credentials

1. Log into your JFrog Cloud Platform
2. Navigate to **Repositories**
3. Select **Set Up Client/CI Tool** for your PyPI repository
4. Copy the full `index-url` value (starts with `http://` or `https://`)

The URL includes embedded credentials and looks like:

```
https://username:password@yourcompany.jfrog.io/artifactory/api/pypi/pypi-local/simple
```

## Store Credentials in Valohai

Save your PyPI credentials securely as environment variables:

### Organization-Level (Recommended)

Share credentials across all projects:

1. Navigate to **Hi, \<name>** → **Manage \<organization>**
2. Open the **Environment Variables** tab
3. Click **Create new environment variable group**
4. Add variable:
   * Name: `PRIVATE_PYPI_INDEX_URL`
   * Value: Paste the full index URL from JFrog
   * Check **Secret** (click the key icon)
5. Select which projects can access this group
6. Click **Save**

See [Environment Variables & Secrets](https://docs.valohai.com/user-and-organization-management/getting-started/environment-variables) for detailed instructions.

### Project-Level

For project-specific repositories:

1. Open your project
2. Go to **Settings** → **Environment Variables**
3. Add the `PRIVATE_PYPI_INDEX_URL` variable as a secret

## Use in Executions

Configure pip to use your private repository before installing packages:

### In valohai.yaml

```yaml
- step:
    name: train-with-private-package
    image: python:3.12
    command:
      - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
      - pip install my-private-package
      - pip install scikit-learn  # Still works for public packages
      - python train.py
```

The `pip config` command tells pip to check your private repository first. It falls back to public PyPI for packages not in your private registry.

### Install Multiple Private Packages

```yaml
- step:
    name: install-multiple-private
    image: python:3.12
    command:
      - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
      - pip install my-ml-library my-data-utils my-viz-tools
      - python run_pipeline.py
```

### Mix Private and Public Packages

Pip checks your private repository first, then falls back to PyPI:

```yaml
command:
  - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
  - pip install -r requirements.txt
```

**requirements.txt:**

```
my-private-models==1.2.3      # From your Artifactory
torch==2.0.0                   # From public PyPI
transformers==4.30.0           # From public PyPI
my-company-utils>=2.0.0        # From your Artifactory
```

## Alternative: Use Extra Index URL

If you want to keep public PyPI as primary and add your private repo as supplementary:

```yaml
command:
  - pip install --extra-index-url $PRIVATE_PYPI_INDEX_URL my-private-package
  - python train.py
```

This checks public PyPI first, then your private registry. Useful when most packages are public and only a few are private.

## Verify Authentication

Test that authentication works:

```yaml
- step:
    name: test-private-repo
    image: python:3.12
    command:
      - echo "Testing private PyPI access..."
      - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
      - pip install --dry-run my-private-package
      - echo "Authentication successful"
```

If authentication fails, you'll see 401 or 403 errors in the logs.

## Multiple Private Repositories

If you have multiple private PyPI repositories:

```yaml
- step:
    name: multi-repo-install
    image: python:3.12
    command:
      - pip config set global.index-url $PRIMARY_PYPI_URL
      - pip config set global.extra-index-url "$SECONDARY_PYPI_URL"
      - pip install package-from-primary package-from-secondary
      - python train.py
    environment-variables:
      - name: PRIMARY_PYPI_URL
        default: https://user:pass@repo1.jfrog.io/...
      - name: SECONDARY_PYPI_URL
        default: https://user:pass@repo2.jfrog.io/...
```

> Mark repository URLs as secrets to avoid exposing credentials in logs or UI.

## Security Best Practices

**Never commit credentials:** Don't put credentials directly in `valohai.yaml` or code. Always use environment variables marked as secrets.

**Rotate credentials regularly:** Update the `PRIVATE_PYPI_INDEX_URL` value when rotating JFrog tokens or passwords.

**Use scoped tokens:** Configure JFrog to issue read-only tokens for CI/CD access rather than using admin credentials.

**Audit access:** Use [Audit Log](https://github.com/valohai/dokuhai/blob/main/user-management/fundamentals/audit-log/README.md) to track which executions accessed private packages.

## Troubleshooting

### 401 Unauthorized

**Cause:** Invalid credentials in the index URL.

**Fix:** Regenerate credentials in JFrog and update the `PRIVATE_PYPI_INDEX_URL` environment variable.

### 404 Not Found

**Cause:** Package doesn't exist in your private repository.

**Fix:** Verify the package name and version. Check that it's published to your Artifactory instance.

### Connection Timeout

**Cause:** Network connectivity issues or incorrect URL.

**Fix:** Test the URL manually:

```shell
curl -I $PRIVATE_PYPI_INDEX_URL
```

### Mixed Public/Private Failures

**Cause:** Using `--index-url` completely replaces PyPI, causing public packages to fail.

**Fix:** Use `--extra-index-url` to keep both registries:

```shell
pip install --extra-index-url $PRIVATE_PYPI_INDEX_URL package-name
```

## Related Topics

* [Environment Variables & Secrets](https://docs.valohai.com/user-and-organization-management/getting-started/environment-variables) — Manage credentials organization-wide
* [Private Docker Registries](https://docs.valohai.com/docker-in-valohai/private-docker-registries) — Pull private Docker images from JFrog
* [Custom Docker Images](https://docs.valohai.com/docker-in-valohai/building-images) — Bake private packages into images
* [JFrog Artifactory Documentation](https://jfrog.com/help/r/jfrog-artifactory-documentation/use-a-pip-configuration-file) — Official pip authentication guide


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.valohai.com/executions/advanced-features/private-pypi-jfrog.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.