Go to Valohai AppBook a Demo

Configure SSH Access

One-time setup guide for administrators to enable SSH debugging for Valohai workers

This guide walks administrators through the one-time setup required to enable SSH debugging for Valohai executions.

Who needs this guide?

Platform administrators responsible for:

  • Managing cloud infrastructure (AWS, Azure, GCP)

  • Configuring firewall rules and security groups

  • Setting up Valohai worker environments

Data scientists and ML engineers should use the SSH Overview guide after administrators complete this setup.

What you'll configure

  1. Default SSH port - Set the port users will connect to (above 1023)

  2. Firewall rules - Open the SSH port for inbound connections

  3. Optional: Configure special network setups like NAT gateways (separate guide)

Step 1: Set the default SSH port

Configure the default port that Valohai workers will use for SSH connections:

  1. Log in to Valohai and click Hi, <name> in the top right

  2. Select Manage <organization>

  3. Go to Settings

  4. Find Default Debug Port

  5. Enter a port number above 1023 (e.g., 2222)

  6. Click Save

This port becomes the default for all SSH-enabled executions. Users can override it when starting individual executions.

Ports below 1024 are reserved and cannot be used for SSH debugging.

Step 2: Configure firewall rules

Open the SSH port in your cloud provider's firewall to allow inbound connections to Valohai worker machines.

AWS Security Groups

  1. Open the AWS Management Console

  2. Navigate to EC2 → Security Groups

  3. Find the security group named valohai-sg-workers

  4. Click Edit Inbound Rules

  5. Add a new rule:

    • Type: Custom TCP

    • Port Range: 2222 (or your chosen port)

    • Source: Choose based on your security requirements:

      • 0.0.0.0/0 - Allow from anywhere (users still need SSH keys)

      • Specific IP ranges - Restrict to office networks or VPN

      • Source security group - Limit to specific AWS resources

    • Description: Valohai SSH debugging access

  6. Click Save Rules

Google Cloud Firewall

  1. Open Google Cloud Console

  2. Navigate to VPC Network → Firewall

  3. Click Create Firewall Rule

  4. Configure the rule:

    • Name: valohai-fr-worker-ssh

    • Description: Valohai SSH debugging access

    • Network: Your Valohai VPC (e.g., valohai-vpc)

    • Direction: Ingress

    • Action: Allow

    • Targets: Specified target tags: valohai-worker

    • Source filter: Choose based on security requirements:

      • IP ranges: 0.0.0.0/0 (allow from anywhere)

      • Or specific IP ranges (office networks, VPN)

    • Protocols and ports: TCP: 2222 (or your chosen port)

  5. Click Create

Azure Network Security Groups

  1. Open Azure Portal

  2. Navigate to Network Security Groups

  3. Select the security group used by Valohai workers

  4. Click Inbound Security Rules

  5. Click Add and configure:

    • Source: Any (or specific IP ranges)

    • Source port ranges: *

    • Destination: Any

    • Destination port ranges: 2222 (or your chosen port)

    • Protocol: TCP

    • Action: Allow

    • Priority: Pick an available priority (e.g., 1000)

    • Name: Valohai-SSH-Debug

    • Description: Valohai SSH debugging access

  6. Click Add

Step 3: Verify the configuration

After configuring firewalls, verify that SSH access works:

  1. Start a test execution with SSH enabled:

    vh exec run --adhoc --debug-port=2222 <your-step>

  2. Generate an SSH key if you don't have one:

    ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/valohai-test-key

  3. Provide the public key (valohai-test-key.pub) when starting the execution

  4. Wait for the execution logs to show the worker IP address

  5. Test the SSH connection:

    ssh -i ~/.ssh/valohai-test-key <IP-ADDRESS> -p 2222 -t /bin/bash

If the connection succeeds, SSH debugging is configured correctly.

Security considerations

Network access scope

Public access (0.0.0.0/0):

  • Users can connect from anywhere

  • Still requires valid SSH private key for authentication

  • Simplest setup for distributed teams

Restricted access (specific IP ranges):

  • Limit connections to office networks or VPN

  • Adds network-level protection

  • Requires updating rules when IP ranges change

Source security groups (AWS only):

  • Allow connections only from specific AWS resources

  • Useful for bastion hosts or VPN gateways

SSH key management

Rotate keys regularly: Establish a policy for generating new SSH key pairs (e.g., every 90 days).

Separate keys by purpose: Don't reuse SSH keys between different systems (GitHub, Valohai, servers).

Audit key usage: Track which team members have access to SSH private keys.

Revoke compromised keys: If a private key is exposed, immediately generate new keys and update team members.

Monitoring and logging

Enable CloudWatch/Stackdriver logs: Monitor SSH connection attempts to worker machines.

Set up alerts: Get notified about unusual connection patterns or failed authentication attempts.

Review audit logs: Regularly check Valohai's audit log for SSH-enabled executions.

Special network configurations

Workers behind NAT

If your worker instances don't have public IP addresses, you'll need additional infrastructure to enable SSH access:

→ Workers Behind NAT Setup Guide

This setup uses a jump host with a reverse proxy to provide SSH access through a NAT gateway.

Multiple environments

If you have separate Valohai environments (dev, staging, production), configure SSH access for each:

  1. Set different default ports for each environment (e.g., 2222 for dev, 2223 for staging)

  2. Create separate firewall rules for each environment's worker security group

  3. Use descriptive rule names to avoid confusion

VPN-only access

For maximum security, require VPN connections before allowing SSH access:

  1. Set up a VPN that provides access to your Valohai VPC

  2. Configure firewall rules to only allow connections from the VPN IP range

  3. Document VPN setup for users in your internal documentation

Troubleshooting

Users report "Connection refused": Verify the firewall rules are applied to the correct security group and include the SSH port.

"Permission denied" errors: This is typically a client-side issue with SSH keys, not firewall configuration.

Can't find valohai-sg-workers: Check if Valohai workers use a different security group name in your setup.

Firewall changes not taking effect: Security group rule changes are usually instant, but verify the rules are attached to active worker instances.

Update worker configuration (self-managed workers only)

If you manage Valohai workers manually (not using Valohai's automation), update the worker configuration:

  1. SSH into your worker machine

  2. Edit /etc/peon.config

  3. Add or update this line:

    DEBUG_PORT=2222

  4. Restart the Valohai worker service:

    sudo systemctl restart peon

For workers managed by Valohai's automation, this configuration is handled automatically.

Next steps

After completing this setup:

  1. Inform your team: Let data scientists and ML engineers know that SSH debugging is now available

  2. Share user documentation: Point them to the SSH Overview guide

  3. Document your specifics: Note any custom IP ranges, ports, or VPN requirements in your internal docs

  4. Set up monitoring: Configure alerts for SSH connection attempts and security events

Users can now start debugging executions by following:

PreviousShared CacheNextWorkers Behind NAT

Last updated

Was this helpful?