Self-Hosted AI Gateway with OpenRouter: A Production Deployment Runbook for 2026

1. Executive Summary & Architecture Overview

In 2026, the landscape of large language model (LLM) access is fragmented. Proprietary vendors each offer their own API endpoints, rate limits, and billing structures. OpenRouter solves this by aggregating hundreds of models—from open-weight giants like Llama 3.1 to frontier commercial models—behind a single, unified API. However, relying on OpenRouter's cloud service means your prompts, API keys, and usage patterns traverse third-party infrastructure. For privacy-conscious homelab operators, the solution is to deploy a self-hosted AI gateway that fronts OpenRouter, giving you a local control plane for traffic routing, prompt logging, and key management.

This guide details the production deployment of a self-hosted AI gateway—using the open-source tool LiteLLM (Latest Stable 2026, released 2026-09-05)—which acts as a transparent proxy to OpenRouter. By self-hosting this gateway, you achieve:

• Data Ownership: Prompts and responses are logged (if enabled) on your own hardware, not on a third-party SaaS dashboard.
• Centralized Key Management: Your OpenRouter API key is stored server-side, never exposed to client applications. You can issue virtual keys with budgets and rate limits.
• Cost Control: Set spend limits per virtual key or user, preventing runaway costs from rogue scripts.
• Model Governance: Enforce which models are accessible, and route traffic to specific models based on your own rules.

Architecture Flow

[Client Apps (Home Assistant, custom scripts, etc.)]
        |
        |  HTTP/HTTPS (Local or via Tailscale)
        v
[Reverse Proxy (Caddy/NPM) - TLS termination]
        |
        v
[Self-Hosted AI Gateway (LiteLLM container)]
        |  - Virtual key validation
        |  - Budget & rate limiting
        |  - Logging & analytics
        |  - Model routing rules
        v
[OpenRouter API (outbound)]
        |
        v
[Various LLM providers]

2. Hardware, OS & Network Requirements

LiteLLM is a lightweight Python service; it does not inference models locally. Thus, you do not need a GPU. The requirements below focus on the gateway itself, plus optional monitoring.

Ports:

• Inbound (local network or via VPN): TCP 4000 (LiteLLM default) and TCP 443 (if using reverse proxy).
• Outbound: TCP 443 to openrouter.ai and api.openrouter.ai.

3. Step 1: Host Preparation & Directory Layout

We'll install to /opt/litellm. Create the directory structure and set permissions for a non-root user (e.g., homelab).

# Create user if needed, and add to docker group
sudo useradd -m homelab
sudo usermod -aG docker homelab

# Create directories
sudo mkdir -p /opt/litellm/config
sudo mkdir -p /opt/litellm/data
sudo chown -R homelab:homelab /opt/litellm

Set environment variables for PUID/PGID (we'll use the homelab user's UID/GID).

echo "PUID=$(id -u homelab)" >> /opt/litellm/.env
echo "PGID=$(id -g homelab)" >> /opt/litellm/.env

4. Step 2: Production-Grade docker compose.yaml

Create /opt/litellm/docker-compose.yaml with the modern Compose Spec (no version key). We use the ghcr.io/berriai/litellm:main-stable image (latest stable 2026).

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    container_name: litellm
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
      - DATABASE_URL=${DATABASE_URL}
      - STORE_MODEL_IN_DB=True
      - LITELLM_LOG=INFO
    volumes:
      - ./config:/app/config
      - ./data:/app/data
    ports:
      - "4000:4000"
    networks:
      - ai_net
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4000/health/liveliness"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s

networks:
  ai_net:
    driver: bridge

.env.example

Create a .env file in /opt/litellm with the following content. Generate secure keys with openssl rand -hex 32.

# /opt/litellm/.env

# UID/GID of the user running the container
PUID=1000
PGID=1000

# Master key for admin UI and API. Generate with: openssl rand -hex 32
LITELLM_MASTER_KEY=sk-1234...

# Salt key for encrypting virtual keys. Generate with: openssl rand -hex 32
LITELLM_SALT_KEY=sk-5678...

# OpenRouter API key. Obtain from https://openrouter.ai/keys
OPENROUTER_API_KEY=sk-or-v1-...

# SQLite database URL (for simplicity). For production, use Postgres.
DATABASE_URL=sqlite:////app/data/litellm.db

5. Step 3: Deployment & Health Verification

Pull and start the stack:

cd /opt/litellm
docker compose up -d

Check container status and logs:

docker compose ps
docker compose logs -f

Verify health endpoint:

curl http://localhost:4000/health/liveliness

Access the admin UI at http://<host-ip>:4000/ui. Log in with your master key. On first login, you'll be prompted to create a new admin user—do so, and store credentials in a password manager.

6. Step 4: Reverse Proxy, Domain & SSL Hardening

For secure remote access, we'll use Caddy (or Nginx Proxy Manager). Here's a Caddyfile example:

ai.example.com {
    reverse_proxy litellm:4000
    encode gzip
    header {
        X-Forwarded-For {remote_host}
        X-Forwarded-Proto {scheme}
        Upgrade {http.request.header.Upgrade}
        Connection {http.request.header.Connection}
    }
}

If using Nginx Proxy Manager, add these custom locations:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

Enable HSTS by adding the header Strict-Transport-Security: max-age=31536000; includeSubDomains in your proxy config.

7. Step 5: Zero-Trust Remote Access with Tailscale

Install Tailscale on the host:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

Then, access the LiteLLM UI via the Tailscale IP (e.g., http://100.x.x.x:4000/ui). To restrict access, use Tailscale ACLs. No public ports are opened.

8. Step 6: Automated Backup & Disaster Recovery

Create a backup script that stops the container, copies the SQLite database, and compresses it with tar. Store backups on a separate disk or remote location.

#!/bin/bash
# /opt/litellm/backup.sh

BACKUP_DIR="/mnt/backups/litellm"
TIMESTAMP=$(date +%Y%m%d%H%M%S)
mkdir -p $BACKUP_DIR

# Stop container to ensure consistent DB
docker compose -f /opt/litellm/docker-compose.yaml stop

# Backup database and config
tar -czf "$BACKUP_DIR/litellm-$TIMESTAMP.tar.gz" -C /opt/litellm data/litellm.db config/

# Restart container
docker compose -f /opt/litellm/docker-compose.yaml start

# Prune old backups (keep last 14 days)
find $BACKUP_DIR -name "litellm-*.tar.gz" -mtime +14 -delete

Make executable and schedule:

chmod +x /opt/litellm/backup.sh
crontab -e
# Add: 0 2 * * * /opt/litellm/backup.sh

9. Step 7: Deep Troubleshooting Matrix

10. Step 8: Frequently Asked Questions (FAQ)

Q1: How do I set spend limits per user or virtual key?

In the LiteLLM UI, navigate to Virtual Keys and create a new key with a max_budget (e.g., 10 USD). This limit is enforced server-side. For per-user limits, configure user_max_budget in the key settings.

Q2: Should I use Watchtower for automatic updates or pin my image tag?

For production, pin to a specific tag (e.g., main-stable) and update manually after reviewing release notes. Watchtower can introduce breaking changes without notice. If you automate, use Watchtower with --revive-stack and --cleanup, but always test in a staging environment first.

Q3: Can I migrate from SQLite to PostgreSQL without downtime?

Yes. Use LiteLLM's built-in database migration tool. Stop the container, export SQLite data to a dump, then import into a Postgres instance. Update DATABASE_URL and restart. Test thoroughly in a staging environment.

Q4: How do I enable request/response logging for audit purposes?

Set environment variable LITELLM_LOG=DEBUG for verbose logs, but for production, use the Spend Logs feature in the UI. This logs all requests, tokens, and costs without storing full prompts unless you set LOG_RAW_REQUEST_RESPONSE=True (be cautious of sensitive data).

Q5: What is the best practice for storing the master key and salt key?

Never hard-code them in docker-compose.yaml. Use a .env file with chmod 600, or better, use Docker secrets or a vault like HashiCorp Vault. For homelab, a .env file with strict permissions is acceptable.

This guide is part of the Self-Hosted AI category at Play5afe.com. Always verify the latest image tags and documentation from the official LiteLLM repository.