Self-Host RustDesk Server with Docker Compose: Private, Encrypted Remote Desktop Alternative to TeamViewer

Remote desktop access is a cornerstone of modern IT operations, system administration, and distributed engineering. For years, organizations and homelab practitioners relied on commercial remote desktop platforms such as TeamViewer, AnyDesk, or Splashtop. However, commercial vendors have increasingly shifted toward restrictive licensing, aggressive commercial-use detection algorithms, sudden session terminations, and centralized relay architectures that route encrypted video streams through third-party infrastructure.

For organizations prioritizing data sovereignty, security, and low latency, RustDesk represents the premier open-source remote desktop solution. Written in Rust for maximum memory safety and raw performance, RustDesk supports end-to-end encryption (E2EE), multi-monitor control, bi-directional clipboard sync, file transfers, and remote terminal sessions. By self-hosting the RustDesk Server infrastructure—comprising the signal server (hbbs) and relay server (hbbr)—via Docker Compose, you establish an independent remote management environment free from external bandwidth throttles or third-party tracking.

Architecture: How RustDesk Handles Signaling and Encrypted Relaying

To successfully deploy and troubleshoot a self-hosted RustDesk instance, it is vital to understand the underlying connection lifecycle between the controlling client and the remote host. RustDesk divides responsibilities across two specialized daemons:

+-------------------------------------------------------------------------------+
|                            Controlling Client (Admin)                         |
+---------------------------------------+---------------------------------------+
                                        |
                 1. Initial Handshake   |  2. Direct UDP Hole Punching (NAT-PMP)
                    & ID Resolution     v
+---------------------------------------+---------------------------------------+
|                    RustDesk Signal Server (hbbs)                              |
|           Port 21115 (TCP), 21116 (TCP/UDP), 21118 (TCP Web)                  |
| - Registers Client IDs       - Resolves Target IP      - Initiates P2P Punch  |
+---------------------------------------+---------------------------------------+
                                        |
                 Direct P2P Success?    +--------------------+
                 (Yes: Encrypted Direct Stream)              | No (Symmetric NAT)
                                                             v
+-------------------------------------------------------------------------------+
|                    RustDesk Relay Server (hbbr)                               |
|                     Port 21117 (TCP), 21119 (TCP Web)                         |
|   - Multiplexes Encrypted TCP Data Streams Between Restricted Endpoints       |
+---------------------------------------+---------------------------------------+
                                        |
                                        v
+-------------------------------------------------------------------------------+
|                               Remote Target Client                            |
+-------------------------------------------------------------------------------+

• hbbs (Rendezvous / Signal Server): Listens for incoming client registrations, maintains active heartbeat status, verifies client public keys, and orchestrates peer-to-peer (P2P) NAT hole punching over UDP.
• hbbr (Relay Server): Acts as an encrypted fallback proxy when direct peer-to-peer UDP connections fail due to restrictive enterprise firewalls, carrier-grade NAT (CGNAT), or symmetric NAT routing.
• Security Model: All sessions utilize modern asymmetric cryptography (Ed25519) for identity authentication and ChaCha20-Poly1305 / AES-256 for symmetric payload encryption. Keys never leave the local client machines.

Network Requirements and Port Allocations

RustDesk utilizes distinct network ports across TCP and UDP protocols. If running behind a firewall (such as UFW, iptables, or cloud security groups), ensure the following ports are open and forwarded to the Docker host:

Port	Protocol	Service	Description
21115	TCP	hbbs	NAT traversal test and connection initialization
21116	TCP / UDP	hbbs	UDP for heartbeat and hole punching; TCP for rendezvous signaling
21117	TCP	hbbr	Encrypted remote session relaying services
21118	TCP	hbbs	Web client WebSocket connection for signaling (optional)
21119	TCP	hbbr	Web client WebSocket connection for relaying (optional)

Crucial Rule: Port 21116 must be reachable via both TCP and UDP. If UDP 21116 is blocked, clients can register their ID but cannot complete NAT discovery, forcing all traffic through the relay.

Step 1: Host Directory Setup and Volume Initialization

Create a dedicated directory structure on your server (e.g., Ubuntu 24.04 or Debian 12). Dedicated persistent storage ensures generated encryption keys and client databases survive container restarts:

sudo mkdir -p /opt/rustdesk-server/{hbbs,hbbr}
cd /opt/rustdesk-server
sudo chown -R 1000:1000 /opt/rustdesk-server

Step 2: Defining the Docker Compose Deployment

Create the /opt/rustdesk-server/docker-compose.yml file. We enforce cryptographic verification by launching hbbs with the -k _ flag. This mandatory security setting ensures that only clients presenting the server’s public key can initiate sessions, preventing unauthorized public abuse of your relay bandwidth:

services:
  hbbs:
    image: rustdesk/rustdesk-server:latest
    container_name: rustdesk_hbbs
    restart: unless-stopped
    command: hbbs -r rustdesk.yourdomain.com:21117 -k _
    volumes:
      - ./hbbs:/root
    ports:
      - "21115:21115/tcp"
      - "21116:21116/tcp"
      - "21116:21116/udp"
      - "21118:21118/tcp"
    networks:
      - rustdesk_net
    healthcheck:
      test: ["CMD-SHELL", "pgrep hbbs || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3

  hbbr:
    image: rustdesk/rustdesk-server:latest
    container_name: rustdesk_hbbr
    restart: unless-stopped
    command: hbbr -k _
    volumes:
      - ./hbbr:/root
    ports:
      - "21117:21117/tcp"
      - "21119:21119/tcp"
    networks:
      - rustdesk_net
    healthcheck:
      test: ["CMD-SHELL", "pgrep hbbr || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3

networks:
  rustdesk_net:
    driver: bridge

Note on Parameters:

• -r rustdesk.yourdomain.com:21117: Instructs hbbs to advertise your public relay address and port to connecting clients whenever direct P2P cannot be negotiated. Replace this domain with your public IP address or fully qualified domain name (FQDN).
• -k _: Forces both hbbs and hbbr to enforce public-key authentication. The server automatically generates an Ed25519 keypair on first launch and rejects unauthenticated connections.

Step 3: Launching the Stack and Extracting the Public Key

Bring up the containers in detached mode:

cd /opt/rustdesk-server
docker compose up -d

Inspect the running containers to verify that both daemons are operational:

docker compose ps
docker compose logs -f hbbs

On initial startup, hbbs generates an asymmetric Ed25519 key pair inside the ./hbbs directory: id_ed25519 (the private key) and id_ed25519.pub (the public key). Because hbbr also enforces key validation via -k _, copy this key pair into the hbbr directory so both services share the identical cryptographic identity:

# Copy public and private keys to the relay server directory
sudo cp /opt/rustdesk-server/hbbs/id_ed25519* /opt/rustdesk-server/hbbr/

# Restart hbbr to load the key pair
docker compose restart hbbr

# Display the public key (save this string for client configuration)
cat /opt/rustdesk-server/hbbs/id_ed25519.pub

The output is a base64-encoded string (for example: 7X9mK...w8L0=). This string is your server’s public key fingerprint.

Step 4: Configuring the RustDesk Client Software

Install the official RustDesk client on your controlling workstation (Windows, macOS, or Linux) and remote machines. By default, the client points to public community relay servers. To bind the client to your private self-hosted infrastructure:

1. Launch the RustDesk desktop client.
2. Click the Menu (three vertical dots) next to your ID and navigate to Network > Unlock Network Settings.
3. Configure the following parameters:
   • ID Server: rustdesk.yourdomain.com (or your public IP)
   • Relay Server: rustdesk.yourdomain.com:21117
   • API Server: Leave blank (unless running the RustDesk Pro web console)
   • Key: Paste the exact string from id_ed25519.pub obtained in Step 3.
4. Click Apply.

Once applied, check the bottom status bar of the RustDesk interface. It should transition from “Connecting…” to “Ready” with a green indicator dot.

Pro Tip: Zero-Touch Automated Client Configuration via Filename

For mass deployment across multiple homelab nodes or family machines, RustDesk supports embedding configuration parameters directly within the executable filename. Rename the installer executable before distribution:

rustdesk-host=rustdesk.yourdomain.com,key=YOUR_PUBLIC_KEY_STRING.exe

Upon launch, the installer parses its own filename and automatically configures the network settings without requiring any manual entry from the user.

Security Hardening & Production Best Practices

1. Firewall Hardening via UFW

Prevent unauthorized port scans and restrict service discovery to the exact required protocols using Uncomplicated Firewall (UFW):

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh

# Allow RustDesk Signal & Relay ports
sudo ufw allow 21115/tcp comment "RustDesk NAT test"
sudo ufw allow 21116/tcp comment "RustDesk signaling TCP"
sudo ufw allow 21116/udp comment "RustDesk heartbeat UDP"
sudo ufw allow 21117/tcp comment "RustDesk relay TCP"

sudo ufw enable
sudo ufw status verbose

2. Automated Backup of Cryptographic Identity

If your id_ed25519 private key is lost or deleted, the server will generate a new keypair upon reboot. Consequently, every configured client in your fleet will immediately fail to authenticate with a “Key mismatch” error. Automate backups of the key files:

sudo tar -czvf /opt/rustdesk-keys-backup-$(date +%F).tar.gz /opt/rustdesk-server/hbbs/id_ed25519*
sudo chmod 600 /opt/rustdesk-keys-backup-*.tar.gz

Troubleshooting Common RustDesk Server Issues

1. Client Displays “Connecting to the RustDesk Network…” Indefinitely

Symptom: The desktop client fails to achieve a “Ready” state and remains trapped in a connection loop.
Root Cause: UDP port 21116 is blocked or unroutable between the client and the server. Many cloud VPS providers (such as AWS, Oracle Cloud, or Hetzner) require separate security group rules for UDP traffic.
Solution: Verify UDP accessibility from an external machine using nc (netcat):

# Test UDP packet transmission from client terminal
nc -zvu rustdesk.yourdomain.com 21116

If the port test fails, adjust your cloud ingress firewall to explicitly permit 0.0.0.0/0 protocol UDP port 21116.

2. “Key Mismatch” or “Failed to connect to relay server”

Symptom: The client connects to hbbs, but initiating a remote session aborts immediately with a key verification failure.
Root Cause: hbbr was started with -k _ but did not have access to the identical id_ed25519.pub and id_ed25519 files generated by hbbs, causing the relay daemon to generate its own separate keypair.
Solution: Ensure both volumes share the exact same key pair:

sudo cp /opt/rustdesk-server/hbbs/id_ed25519* /opt/rustdesk-server/hbbr/
docker compose restart hbbr

3. Sluggish Performance and High Frame Drop over LAN Connections

Symptom: Machines situated on the same local subnet experience latency and low framerates during screen streaming.
Root Cause: Direct P2P negotiation failed due to NAT Hairpinning (NAT Loopback) issues on the local router, forcing traffic to route out to the public relay and back.
Solution: Enable Direct IP Access in RustDesk settings (Settings > Network > Direct IP Access). On local subnets, clients can connect directly to internal IP addresses (e.g., 192.168.1.150) bypassing the external relay entirely for uncompressed LAN speeds.

Summary and Key Takeaways

By containerizing RustDesk Server with Docker Compose, you retain complete sovereignty over your remote administration pipeline. The decoupled dual-daemon design (hbbs and hbbr) provides optimized peer-to-peer connectivity whenever feasible while retaining a low-overhead, encrypted fallback relay when navigating complex NAT topologies.

Enforcing mandatory Ed25519 cryptographic key authentication (-k _) guarantees that your server remains strictly private, guarding your compute and network bandwidth against unauthorized usage. Pair this infrastructure with automated client filename deployment for a seamless, secure alternative to proprietary remote support tools.