# Kubernetes Shell Access

For Valohai executions running on Kubernetes, you can open debug shells using `kubectl exec` instead of SSH. This method uses your existing Kubernetes credentials and doesn't require SSH key setup.

**When to use this:**

* Your Valohai environment runs on Kubernetes
* You need quick shell access without SSH configuration
* You're a platform engineer with cluster access

**Differences from SSH debugging:**

* No SSH keys or firewall rules required
* Uses Kubernetes RBAC for authentication
* Direct `kubectl` access to pods
* Cannot attach IDE debuggers (use SSH methods for that)

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

You need cluster-level access to use `kubectl`:

#### 1. Kubernetes Cluster Access <a href="#id-1-kubernetes-cluster-access" id="id-1-kubernetes-cluster-access"></a>

Configure `kubectl` to reach your cluster's control plane. This typically means:

* Client certificate authentication
* Service account with access token
* Kubeconfig file properly configured

Verify access:

```shell
kubectl get pods -n valohai-workers
```

#### 2. Required RBAC Permissions <a href="#id-2-required-rbac-permissions" id="id-2-required-rbac-permissions"></a>

Your Kubernetes user needs permissions in the `valohai-workers` namespace:

```yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: valohai-workers
  name: debug-executions
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
```

**What this enables:**

* `get pods` - List and view execution pods
* `create pods/exec` - Open shell sessions

> 💡 *Contact your Kubernetes administrator if you don't have these permissions.*

### Open Shell to Execution <a href="#open-shell-to-execution" id="open-shell-to-execution"></a>

#### 1. Start a Valohai Execution <a href="#id-1-start-a-valohai-execution" id="id-1-start-a-valohai-execution"></a>

Launch any execution using your Kubernetes environment. No special flags needed - SSH is not required for this method.

#### 2. Get Pod Name from Logs <a href="#id-2-get-pod-name-from-logs" id="id-2-get-pod-name-from-logs"></a>

Find the unique pod name in the execution logs:

<figure><img src="/files/k6ziylM8J8lf6dKKQTzA" alt=""><figcaption></figcaption></figure>

The logs will show something like:

```
Pod name: valohai-exec-a1b2c3d4e5f6
Namespace: valohai-workers
Container: workload
```

**Copy the pod name** - you'll use it in the next step.

#### 3. Execute Shell Command <a href="#id-3-execute-shell-command" id="id-3-execute-shell-command"></a>

```shell
kubectl exec -it <POD-NAME> -c workload --namespace valohai-workers -- bash
```

**Example:**

```shell
kubectl exec -it valohai-exec-a1b2c3d4e5f6 -c workload --namespace valohai-workers -- bash
```

**Command breakdown:**

* `-it` - Interactive terminal
* `<POD-NAME>` - Your execution's unique pod name
* `-c workload` - Container name (always `workload` for Valohai)
* `--namespace valohai-workers` - Namespace (always `valohai-workers`)
* `bash` - Shell to open (can use `sh` if `bash` unavailable)

### What You Can Do <a href="#what-you-can-do" id="what-you-can-do"></a>

Once connected, you have shell access inside the execution container:

**Inspect execution state:**

```shell
# View running processes
ps aux

# Check Python packages
pip list

# Examine logs
cat /valohai/logs/execution.log

# View mounted data
ls /valohai/inputs
```

**Debug code:**

```shell
# Navigate to your repository
cd /valohai/repository

# Run Python interactively
python

# Test imports
python -c "import your_module"
```

**Monitor resources:**

```shell
# Memory usage
free -h

# Disk usage
df -h

# GPU status (if applicable)
nvidia-smi
```

### Limitations <a href="#limitations" id="limitations"></a>

**What you cannot do with kubectl exec:**

* Attach IDE debuggers (use [VS Code](/development-and-debugging/vs-code-remote-debugging.md) or [PyCharm](/development-and-debugging/pycharm-remote-debugging.md) with SSH)
* Access multiple pods simultaneously from one command
* Open tunnels to services (use SSH tunneling for this)

**Comparison to SSH:**

| Feature              | kubectl exec                     | SSH                      |
| -------------------- | -------------------------------- | ------------------------ |
| Authentication       | Kubernetes RBAC                  | SSH keys                 |
| Setup complexity     | Low (if you have cluster access) | Medium (firewall + keys) |
| IDE debugger support | ❌ No                             | ✅ Yes                    |
| Port forwarding      | ❌ No                             | ✅ Yes                    |
| Use case             | Quick inspection                 | Interactive debugging    |

### Common Issues <a href="#common-issues" id="common-issues"></a>

**Permission denied?**

* Verify you have `get pods` and `create pods/exec` permissions
* Check you're using the correct namespace: `valohai-workers`

**Pod not found?**

* Ensure execution is still running (not completed/failed)
* Verify pod name copied correctly from logs
* Check you're connected to the correct cluster

**Container 'workload' not found?**

* This is rare - contact Valohai support if you see this
* Container name should always be `workload`

**Execution exits too quickly?**

* Add `sleep 1h` to your execution command to keep it alive:

```yaml
  command:
    - python train.py
    - sleep 1h
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.valohai.com/development-and-debugging/kubernetes-shell-access.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
