# Remote Access (SSH)

SSH access lets you connect directly to running Valohai executions for interactive debugging, IDE attachment, and real-time inspection.

**Why use SSH debugging:**

* Attach your local IDE (VS Code, PyCharm) to debug code running on cloud infrastructure
* Inspect execution state interactively without stopping the job
* Tunnel to services like TensorBoard for real-time monitoring
* Test code changes directly in the execution environment

### Requirements <a href="#requirements" id="requirements"></a>

SSH access requires:

1. **Enterprise plan** with on-premises, AWS, Azure, or GCP environments
2. **Firewall configuration** by your organization administrator
3. **SSH key pair** for authentication

> *SSH connects to the Docker container running your code, not the underlying VM. This means Valohai internals and the host OS are not accessible.*

### Quick Start <a href="#quick-start" id="quick-start"></a>

#### For Developers <a href="#for-developers" id="for-developers"></a>

1. Generate SSH keys (or use auto-generated keys)
2. Start execution with "Run with SSH" enabled
3. Wait for IP address in logs
4. Connect using SSH or your IDE

**Choose your debugging approach:**

* [VS Code Remote Debugging](/development-and-debugging/vs-code-remote-debugging.md) - debugpy integration
* [PyCharm Remote Debugging](/development-and-debugging/pycharm-remote-debugging.md) - PyCharm Professional debugger
* Direct SSH connection (terminal access)

**Alternative for Kubernetes users:**

* [Kubernetes Shell Access](/development-and-debugging/kubernetes-shell-access.md) - kubectl-based debugging (no SSH keys required)

#### For Administrators <a href="#for-administrators" id="for-administrators"></a>

Before developers can use SSH, complete the one-time setup below.

### Administrator Setup <a href="#administrator-setup" id="administrator-setup"></a>

<details>

<summary><strong>Step 1: Set Default SSH Port</strong></summary>

1. Navigate to **Hi, \[name]** → **Manage \[organization]**
2. Go to **Settings**
3. Set **Default Debug Port** (must be above 1023, e.g., `2222`)

</details>

<details>

<summary><strong>Step 2: Configure Firewall Rules</strong></summary>

**AWS**

1. Open the Security Group named `valohai-sg-workers`
2. Click **Edit Inbound Rules** → **Add Rule**
3. Configure:
   * **Type:** Custom TCP
   * **Port Range:** Your debug port (e.g., `2222`)
   * **Source:** `0.0.0.0/0` (or whitelist specific IPs/CIDR blocks)
   * **Description:** "Allows SSH to Valohai executions"

> 💡 *Setting source to `0.0.0.0/0` allows connections from anywhere, but users still need the SSH private key to authenticate.*

**Google Cloud**

1. Create a firewall rule:
   * **Name:** `valohai-fr-worker-ssh`
   * **Description:** "Allows SSH to Valohai executions"
   * **Network:** Your Valohai VPC (e.g., `valohai-vpc`)
   * **Direction:** Ingress
   * **Targets:** Specified target tags: `valohai-worker`
   * **Source:** `0.0.0.0/0` (or whitelist specific IPs/CIDR blocks)
   * **Protocols and ports:** TCP: `2222` (your debug port)

**Azure**

1. Open the Network Security Group associated with Valohai workers
2. Add an **Inbound security rule**:
   * **Source:** Any (or specific IP ranges)
   * **Source port ranges:** \*
   * **Destination:** Any
   * **Destination port ranges:** `2222` (your debug port)
   * **Protocol:** TCP
   * **Action:** Allow
   * **Priority:** 1000 (or appropriate for your NSG)
   * **Name:** `AllowValohaiSSH`

</details>

### Generate SSH Keys <a href="#generate-ssh-keys" id="generate-ssh-keys"></a>

You can auto-generate keys in the Valohai UI or create them manually.

<details>

<summary><strong>Option A: Auto-generate in UI (Recommended)</strong></summary>

When starting an execution with "Run with SSH" enabled:

1. Click **Generate new SSH key**
2. **Download and securely store the private key**
3. Start the execution

⚠️ **The private key is shown only once. Store it securely and never commit it to version control.**

</details>

<details>

<summary><strong>Option B: Manual Key Generation</strong></summary>

Generate a 4096-bit RSA key pair:

```shell
ssh-keygen -t rsa -b 4096 -N '' -f valohai-debug-key
```

This creates:

* `valohai-debug-key.pub` - Paste into Valohai UI before starting execution
* `valohai-debug-key` - Use to connect (keep this secure)

⚠️ **Never commit SSH keys to version control. Anyone with the private key can access your execution.**

Regenerate keys periodically according to your organization's security policies.

</details>

***

### Start an Execution with SSH <a href="#start-an-execution-with-ssh" id="start-an-execution-with-ssh"></a>

#### From Web UI <a href="#from-web-ui" id="from-web-ui"></a>

1. Create or open an execution
2. Enable **Run with SSH**
3. Paste your public key (or auto-generate)
4. Adjust TCP/IP port if needed (default uses organization setting)
5. Click **Create Execution**

<figure><img src="/files/zYN8ooflXvE09nFtnWSQ" alt=""><figcaption></figcaption></figure>

#### From Command Line <a href="#from-command-line" id="from-command-line"></a>

```shell
vh exec run --adhoc --debug-key-file=/path/to/your-key.pub --debug-port 2222 train
```

***

### Connect to Your Execution <a href="#connect-to-your-execution" id="connect-to-your-execution"></a>

#### 1. Wait for IP Address <a href="#id-1-wait-for-ip-address" id="id-1-wait-for-ip-address"></a>

After starting the execution, watch the logs for the connection details:

<figure><img src="/files/geAvnMHPZLjap3wOzj91" alt=""><figcaption></figcaption></figure>

The log will show:

```
SSH connection available at: 52.214.159.193:2222
```

> 💡 *If you don't see the IP, ensure SSH was enabled when starting the execution.*

#### 2. Keep Execution Running <a href="#id-2-keep-execution-running" id="id-2-keep-execution-running"></a>

**Important:** Executions shut down when the command finishes. Add a sleep to keep it alive:

```yaml
- step:
    name: train
    command:
      - python train.py {parameters}
      - sleep 1h  # Keeps execution alive for debugging
```

**Why 1 hour?** Setting a reasonable timeout prevents costly mistakes from infinite runtimes.

> For IDE debugging, use debugger wait patterns (see [VS Code](https://untitled+.vscode-resource.vscode-cdn.net/vs-code-remote-debugging.md) or [PyCharm](https://untitled+.vscode-resource.vscode-cdn.net/pycharm-remote-debugging.md) guides) instead of sleep.

#### 3. Choose Connection Method <a href="#id-3-choose-connection-method" id="id-3-choose-connection-method"></a>

**Interactive Shell**

```shell
ssh -i /path/to/private-key <IP> -p 2222 -t /bin/bash
```

This opens a bash shell session inside your execution container.

**Run Single Command**

```shell
ssh -i /path/to/private-key <IP> -p 2222 -t ps aux
```

Returns command output to your terminal.

**SSH Tunnel (for services like TensorBoard)**

```shell
ssh -i /path/to/private-key <IP> -p 2222 -L 5678:127.0.0.1:5678
```

Forwards port 5678 from the execution to your local machine.

***

### Next Steps <a href="#next-steps" id="next-steps"></a>

**Debug with your IDE:**

* [VS Code Remote Debugging](/development-and-debugging/vs-code-remote-debugging.md)
* [PyCharm Remote Debugging](/development-and-debugging/pycharm-remote-debugging.md)

**Alternative for Kubernetes:**

* [Kubernetes Shell Access](/development-and-debugging/kubernetes-shell-access.md) - No SSH keys needed

***

### Common Issues <a href="#common-issues" id="common-issues"></a>

**No IP address in logs?**

* Verify SSH was enabled when starting the execution
* Check that you pasted the public key (not private key)

**Connection refused?**

* Confirm firewall rules allow traffic on your debug port
* Verify the port matches your organization settings

**Authentication failed?**

* Ensure you're using the private key (not `.pub` file)
* Check file permissions: `chmod 600 /path/to/private-key`

**Execution shuts down too quickly?**

* Add `sleep 1h` at the end of your command
* For IDE debugging, use debugger wait patterns


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.valohai.com/development-and-debugging/ssh-overview.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.