Setting Up RustDesk Self-Hosted Server with HTTPS on Docker

A self-hosted RustDesk server gives you full control over remote desktop infrastructure — no third-party relay, no telemetry leakage, and HTTPS-protected web access from anywhere.

Overview & Architecture

What RustDesk Does

RustDesk is an open-source remote desktop application (VNC alternative). When you run the official clients, they fall back to RustDesk's public rendezvous servers if no custom server is configured. Hosting your own servers means:

• All traffic stays on your network or through your relay instead of leaking through public infrastructure.
• Web desktop access over HTTPS from any browser — useful when the native client can't be installed.
• Zero data retention on third parties.

Core Components

Trade-offs

Self-hosting RustDesk is appropriate when you want control over connection paths or need encrypted web desktop access. If you only need occasional remote access inside your home network, the default public server mode is simpler and requires no ongoing maintenance.

The main overhead is keeping two containers updated and maintaining a public IP or DDNS target for inbound connections.

System Requirements & Directory Structure

Hardware Recommendations

• CPU: Any modern 64-bit CPU. RustDesk servers are lightweight (~10–30 MB RAM idle).
• RAM: 256 MB minimum per container; 512 MB total recommended.
• Storage: ~100 MB for binaries + persistent volumes for keys and config (~50 MB).
• Network: A public IPv4 address or working IPv6 with port forwarding on 21115–21119 and 443.

Directory Layout

mkdir -p ~/homelab/rustdesk/{data,key}
cd ~/homelab/rustdesk

The data directory holds persistent identity keys. The key directory stores TLS certificates generated by Caddy.

Docker Compose Configuration

docker-compose.yml

services:
  rustdesk-hbbs:
    image: rustdesk/rustdesk-server:latest
    container_name: rustdesk-hbbs
    command: hbbs -r rust://rd.example.com:443
    restart: unless-stopped
    ports:
      - "21115:21115"     # TCP + UDP registration
      - "21116:21116"     # TCP relay handshakes
      - "21116:21116/udp" # UDP relay handshakes
      - "21117:21117"     # TCP relay data
      - "21118:21118"     # TCP extra relay ports
      - "21119:21119"     # TCP extra relay ports
    volumes:
      - ./data:/root
    networks:
      - rustdesk-net
    depends_on:
      - rustdesk-hbbr

  rustdesk-hbbr:
    image: rustdesk/rustdesk-server:latest
    container_name: rustdesk-hbbr
    command: hbbr
    restart: unless-stopped
    ports:
      - "21117:21117"     # Relay data port
    volumes:
      - ./data:/root
    networks:
      - rustdesk-net

  caddy:
    image: caddy:2-alpine
    container_name: caddy
    restart: unless-stopped
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./key:/data
      - ./data/caddy_config:/config
    networks:
      - rustdesk-net

networks:
  rustdesk-net:
    driver: bridge

Caddyfile

Place this in ~/homelab/rustdesk/Caddyfile:

rd.example.com {
    encode gzip

    # Serve the RustDesk web desktop
    reverse_proxy rustdesk-hbbs:21116

    # Let's Encrypt HTTPS — change email for notifications
    tls admin@example.com
}

Replace rd.example.com with your actual domain and admin@example.com with your contact email.

.env File

Create ~/homelab/rustdesk/.env:

# Domain used by RustDesk web client and Caddy
RUSTDESK_DOMAIN=rd.example.com

# Email for Let's Encrypt certificate notifications
LETS_ENCRYPT_EMAIL=admin@example.com

# Optional: custom relay domain (different from web desktop domain)
# RUSTDESK_RELAY_DOMAIN=relay.example.com

Step-by-Step Deployment

1. Generate Identity Keys

On first launch, hbbs generates host key pairs stored in /root/. Run the container once to create them:

cd ~/homelab/rustdesk
docker compose up -d rustdesk-hbbs rustdesk-hbbr
docker compose logs -f rustdesk-hbbs

Look for output containing id_ed25519 pub — copy that public key line. You will need it when configuring clients.

Stop the containers after noting the key:

docker compose stop

2. Start All Services

docker compose up -d
docker compose ps

Expected output shows all three containers as healthy (or at least running).

3. Verify Service Health

# Check that hbbs and hbbr are running
docker compose logs --tail=20 rustdesk-hbbs
docker compose logs --tail=20 rustdesk-hbbr

# Confirm Caddy obtained a certificate
docker compose logs --tail=20 caddy

# Test the web desktop endpoint
curl -kI https://localhost

You should see an HTTP 200 or a redirect to the RustDesk login page.

4. Port Forwarding

Forward these ports on your router to the host machine:

Configuring Clients

On each RustDesk client, go to ID/Server → Set and enter:

• ID Relay Servers: rd.example.com:21117
• ID Server: rd.example.com:21115
• Key: Paste the id_ed25519 pub line you saved earlier

The key enables end-to-end encrypted peer discovery. Without it, connections still work but lack host-key verification.

Common Pitfalls & Troubleshooting

Permission Denied on Data Volume

If the container fails to start and logs show permission denied on /root:

chown -R 1000:1000 ~/homelab/rustdesk/data
docker compose down && docker compose up -d

(RustDesk server containers typically run as UID 1000.)

Port Conflict

Another service may already occupy 21115–21119 or 443:

# Check what's listening on suspect ports
sudo ss -tlnp | grep -E '2111[5-9]|443|80'

# Stop the conflicting service or remap ports in docker-compose.yml

To remap local ports without changing the internal ports:

ports:
  - "12115:21115"
  - "12116:21116"

Then update client configuration to use the new external ports.

Certificate Not Issued

Caddy won't issue a certificate if:

1. Port 80 is not publicly reachable (Let's Encrypt must validate via HTTP-01 challenge).
2. DNS does not resolve to your public IP.

Check resolution and connectivity:

# Verify DNS points to your IP
dig +short rd.example.com

# Test HTTP reachability from outside
curl -I http://rd.example.com

Web Desktop Shows Blank Page

The RustDesk web client expects connections to arrive on the same port the server advertises. Ensure Caddy's reverse_proxy target matches the actual container name in the Docker network:

reverse_proxy rustdesk-hbbs:21116

Do not replace rustdesk-hbbs with localhost — Docker networking resolves by service name.

Relay Dropouts Under Load

If multiple users connect simultaneously and the relay becomes a bottleneck, increase available relay ports and assign more to the hbbr container:

# In docker-compose.yml, add additional mappings
ports:
  - "21117:21117"
  - "21118:21118"
  - "21119:2119"

Also ensure your host's ulimit allows sufficient file descriptors:

# Add to /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536

Backup & Maintenance

Backup Persistent Data

The critical files are in ./data/ — they contain identity keys and peer registration state:

# Full backup of RustDesk data
tar -czf ~/backups/rustdesk-$(date +%Y%m%d).tar.gz \
  ~/homelab/rustdesk/data/

# Also back up the Caddy certificate store
tar -azf ~/backups/rustdesk-certs-$(date +%Y%m%d).tar.gz \
  ~/homelab/rustdesk/key/

Update RustDesk Server

cd ~/homelab/rustdesk
docker compose pull
docker compose up -d --force-recreate

Always check the upstream changelog before upgrading in production.

Rotate Keys

To regenerate host keys (forces all clients to re-accept the new key):

cd ~/homelab/rustdesk
cp -r data data.bak.$(date +%s)
docker compose stop
rm -rf data/*
docker compose up -d rustdesk-hbbs
# Copy the new pub key and distribute to all clients
docker compose logs rustdesk-hbbs | grep 'id_ed25519 pub'

Official Resources & Upstream Links

• RustDesk Server Source: https://github.com/rustdesk/rustdesk-server
• Docker Image (ghcr.io): https://github.com/rustdesk/rustdesk-server/wiki
• RustDesk Client: https://github.com/rustdesk/rustdesk
• Caddy Server Docs: https://caddyserver.com/docs/
• Docker Compose Spec: https://docs.docker.com/compose/compose-file/