# Configure SSH Access

This guide walks administrators through the one-time setup required to enable SSH debugging for Valohai executions.

## Who needs this guide?

**Platform administrators** responsible for:

* Managing cloud infrastructure (AWS, Azure, GCP)
* Configuring firewall rules and security groups
* Setting up Valohai worker environments

**Data scientists and ML engineers** should use the [SSH Overview](/development-and-debugging/ssh-overview.md) guide after administrators complete this setup.

## What you'll configure

1. **Default SSH port** - Set the port users will connect to (above 1023)
2. **Firewall rules** - Open the SSH port for inbound connections
3. **Optional:** Configure special network setups like NAT gateways ([separate guide](/installation-and-setup/advanced-topics/workers-behind-nat.md))

## Step 1: Set the default SSH port

Configure the default port that Valohai workers will use for SSH connections:

1. Log in to Valohai and click **Hi, \<name>** in the top right
2. Select **Manage \<organization>**
3. Go to **Settings**
4. Find **Default Debug Port**
5. Enter a port number above 1023 (e.g., `2222`)
6. Click **Save**

This port becomes the default for all SSH-enabled executions. Users can override it when starting individual executions.

> Ports below 1024 are reserved and cannot be used for SSH debugging.

## Step 2: Configure firewall rules

Open the SSH port in your cloud provider's firewall to allow inbound connections to Valohai worker machines.

### AWS Security Groups

1. Open the **AWS Management Console**
2. Navigate to **EC2** → **Security Groups**
3. Find the security group named `valohai-sg-workers`
4. Click **Edit Inbound Rules**
5. Add a new rule:
   * **Type:** Custom TCP
   * **Port Range:** `2222` (or your chosen port)
   * **Source:** Choose based on your security requirements:
     * `0.0.0.0/0` - Allow from anywhere (users still need SSH keys)
     * Specific IP ranges - Restrict to office networks or VPN
     * Source security group - Limit to specific AWS resources
   * **Description:** `Valohai SSH debugging access`
6. Click **Save Rules**

### Google Cloud Firewall

1. Open **Google Cloud Console**
2. Navigate to **VPC Network** → **Firewall**
3. Click **Create Firewall Rule**
4. Configure the rule:
   * **Name:** `valohai-fr-worker-ssh`
   * **Description:** `Valohai SSH debugging access`
   * **Network:** Your Valohai VPC (e.g., `valohai-vpc`)
   * **Direction:** Ingress
   * **Action:** Allow
   * **Targets:** Specified target tags: `valohai-worker`
   * **Source filter:** Choose based on security requirements:
     * IP ranges: `0.0.0.0/0` (allow from anywhere)
     * Or specific IP ranges (office networks, VPN)
   * **Protocols and ports:** TCP: `2222` (or your chosen port)
5. Click **Create**

### Azure Network Security Groups

1. Open **Azure Portal**
2. Navigate to **Network Security Groups**
3. Select the security group used by Valohai workers
4. Click **Inbound Security Rules**
5. Click **Add** and configure:
   * **Source:** Any (or specific IP ranges)
   * **Source port ranges:** `*`
   * **Destination:** Any
   * **Destination port ranges:** `2222` (or your chosen port)
   * **Protocol:** TCP
   * **Action:** Allow
   * **Priority:** Pick an available priority (e.g., `1000`)
   * **Name:** `Valohai-SSH-Debug`
   * **Description:** `Valohai SSH debugging access`
6. Click **Add**

## Step 3: Verify the configuration

After configuring firewalls, verify that SSH access works:

1. Start a test execution with SSH enabled:

   ```shell
   vh exec run --adhoc --debug-port=2222 <your-step>
   ```
2. Generate an SSH key if you don't have one:

   ```shell
   ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/valohai-test-key
   ```
3. Provide the public key (`valohai-test-key.pub`) when starting the execution
4. Wait for the execution logs to show the worker IP address
5. Test the SSH connection:

   ```shell
   ssh -i ~/.ssh/valohai-test-key <IP-ADDRESS> -p 2222 -t /bin/bash
   ```

If the connection succeeds, SSH debugging is configured correctly.

## Security considerations

### Network access scope

**Public access (0.0.0.0/0):**

* Users can connect from anywhere
* Still requires valid SSH private key for authentication
* Simplest setup for distributed teams

**Restricted access (specific IP ranges):**

* Limit connections to office networks or VPN
* Adds network-level protection
* Requires updating rules when IP ranges change

**Source security groups (AWS only):**

* Allow connections only from specific AWS resources
* Useful for bastion hosts or VPN gateways

### SSH key management

**Rotate keys regularly**: Establish a policy for generating new SSH key pairs (e.g., every 90 days).

**Separate keys by purpose**: Don't reuse SSH keys between different systems (GitHub, Valohai, servers).

**Audit key usage**: Track which team members have access to SSH private keys.

**Revoke compromised keys**: If a private key is exposed, immediately generate new keys and update team members.

### Monitoring and logging

**Enable CloudWatch/Stackdriver logs**: Monitor SSH connection attempts to worker machines.

**Set up alerts**: Get notified about unusual connection patterns or failed authentication attempts.

**Review audit logs**: Regularly check Valohai's audit log for SSH-enabled executions.

## Special network configurations

### Workers behind NAT

If your worker instances don't have public IP addresses, you'll need additional infrastructure to enable SSH access:

→ [Workers Behind NAT Setup Guide](/installation-and-setup/advanced-topics/workers-behind-nat.md)

This setup uses a jump host with a reverse proxy to provide SSH access through a NAT gateway.

### Multiple environments

If you have separate Valohai environments (dev, staging, production), configure SSH access for each:

1. Set different default ports for each environment (e.g., `2222` for dev, `2223` for staging)
2. Create separate firewall rules for each environment's worker security group
3. Use descriptive rule names to avoid confusion

### VPN-only access

For maximum security, require VPN connections before allowing SSH access:

1. Set up a VPN that provides access to your Valohai VPC
2. Configure firewall rules to only allow connections from the VPN IP range
3. Document VPN setup for users in your internal documentation

## Troubleshooting

**Users report "Connection refused"**: Verify the firewall rules are applied to the correct security group and include the SSH port.

**"Permission denied" errors**: This is typically a client-side issue with SSH keys, not firewall configuration.

**Can't find valohai-sg-workers**: Check if Valohai workers use a different security group name in your setup.

**Firewall changes not taking effect**: Security group rule changes are usually instant, but verify the rules are attached to active worker instances.

## Update worker configuration (self-managed workers only)

If you manage Valohai workers manually (not using Valohai's automation), update the worker configuration:

1. SSH into your worker machine
2. Edit `/etc/peon.config`
3. Add or update this line:

   ```shell
   DEBUG_PORT=2222
   ```
4. Restart the Valohai worker service:

   ```shell
   sudo systemctl restart peon
   ```

For workers managed by Valohai's automation, this configuration is handled automatically.

## Next steps

After completing this setup:

1. **Inform your team**: Let data scientists and ML engineers know that SSH debugging is now available
2. **Share user documentation**: Point them to the [SSH Overview](/installation-and-setup/advanced-topics/configure-ssh-access.md) guide
3. **Document your specifics**: Note any custom IP ranges, ports, or VPN requirements in your internal docs
4. **Set up monitoring**: Configure alerts for SSH connection attempts and security events

Users can now start debugging executions by following:

* [SSH Overview](/installation-and-setup/advanced-topics/configure-ssh-access.md) - General SSH debugging guide
* [VS Code Remote Debugging](/development-and-debugging/pycharm-remote-debugging.md) - VS Code IDE setup
* [PyCharm Remote Debugging](https://github.com/valohai/dokuhai/blob/main/ssh/pycharm.md) - PyCharm IDE setup


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.valohai.com/installation-and-setup/advanced-topics/configure-ssh-access.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.