Run Local LLMs: Complete Ollama Install Guide with Docker Compose (2026)

Introduction

The appeal of running Large Language Models (LLMs) locally is no longer just a niche hobby. With growing concerns over data privacy, API costs, and the need for offline capabilities, self-hosting an LLM has become a practical infrastructure decision for many developers and homelab operators. Ollama simplifies this process dramatically by packaging model management, inference, and an OpenAI-compatible API into a single, efficient daemon.

This guide provides a production-oriented path to deploying Ollama using Docker Compose. We will move beyond the basic docker run example found in most quickstarts. You will learn how to set up a persistent, secure, and maintainable installation that can serve as the foundation for applications like Open WebUI, Continue.dev, or custom scripts. We will cover hardware prerequisites, complete Compose files, environment variable management, and integration with a reverse proxy for remote access.

The focus is on reproducibility and avoiding common pitfalls. By following the steps below, you will avoid the typical errors related to GPU passthrough, permission mismatches, and version pinning. We will also address critical security aspects, ensuring your local AI endpoint is not accidentally exposed to the public internet without authentication and TLS.

Finally, this guide is written for a technical audience. We assume you are comfortable with the command line and basic Docker concepts. While we provide copy-paste commands, we also explain the rationale behind each configuration block, empowering you to adapt the setup to your specific homelab architecture.

Prerequisites

Before starting, ensure your host meets the minimum requirements. The performance of your LLM will vary significantly based on the model size and your hardware. The numbers below are typical ranges for running 7B to 13B parameter models.

Note on Performance: Inference speed (tokens/second) is highly dependent on the model quantization, context window size, and hardware. Do not expect real-time performance on CPU-only hosts for models larger than 7B. Check the official Ollama GitHub README for the latest compatibility matrix for GPUs.

Installation Steps

Step 1: Create Directory Structure and Environment File

First, create a dedicated directory for Ollama and its data. We will store the .env file here, which contains secrets and configurable parameters. This file must never be committed to version control.

mkdir -p ~/ollama && cd ~/ollama

Now, create the .env file. This file holds the variables referenced in the docker-compose.yml. Replace the placeholder values with strong, unique passwords.

cat <<'EOF' > .env
# Ollama Configuration
OLLAMA_VERSION=v0.33.2
OLLAMA_MODELS=/root/.ollama/models

# API Authentication (Basic Auth for reverse proxy)
PROXY_USER=admin
PROXY_PASSWORD_CHANGE_ME=your_strong_password_here

# Timezone
TZ=UTC
EOF

Security Warning: The .env file contains your proxy password. Ensure file permissions are restricted to your user only. Run chmod 600 .env and add .env to your .gitignore file if you are using a git repository.

Step 2: Create the Docker Compose File

Create a docker-compose.yml file in the same directory. This configuration pins the exact version we verified (v0.33.2), sets up persistent storage, and prepares the environment for GPU passthrough if needed.

services:
  ollama:
    image: ollama/ollama:${OLLAMA_VERSION:-latest}
    container_name: ollama
    restart: unless-stopped
    environment:
      - OLLAMA_MODELS=${OLLAMA_MODELS}
      - TZ=${TZ}
    volumes:
      - ./ollama_data:/root/.ollama
    ports:
      - "127.0.0.1:11434:11434"
    # Uncomment the following lines for NVIDIA GPU support
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

Version Pinning: We use ${OLLAMA_VERSION:-latest}. The default is set to v0.33.2 in our .env file. If you omit the .env variable, it will pull latest. Check the official GitHub releases page before pinning a version — the version above may be outdated by now.

Network Binding: We bind to 127.0.0.1:11434 only. This prevents direct access from the network. We will expose this via a reverse proxy in a later step. Do not change this to 0.0.0.0 unless you have a specific, firewalled reason.

Step 3: Pull the Image and Start the Container

Start the container. This will pull the image and create the data directory.

docker compose up -d

Verify the container is running and healthy.

docker compose ps

Step 4: Verify Basic Functionality

Check that the API is responding locally. We will send a request to the API to list models. It should return an empty list or an error if no models are pulled yet.

curl http://127.0.0.1:11434/api/tags

If the container fails to start, check the logs for errors. Common issues include permission problems on the mounted volume or missing GPU drivers.

docker compose logs ollama

Step 5: Pull Your First Model

Now, pull a model. For this guide, we will use llama3.2:1b, a small model suitable for testing. This will take a few minutes depending on your internet connection.

docker exec -it ollama ollama pull llama3.2:1b

After pulling, test a generation request.

curl http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.2:1b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

Step 6: Configure GPU Passthrough (Optional but Recommended)

If you have an NVIDIA GPU, you must install the NVIDIA Container Toolkit on the host. The Compose file above has the deploy section commented out. Uncomment it and restart the container.

First, install the toolkit by following the official NVIDIA documentation for your distribution. Then, restart the stack.

docker compose down && docker compose up -d

Verify the GPU is visible inside the container.

docker exec -it ollama nvidia-smi

You should see the GPU status. If you see an error, ensure the nvidia-container-runtime is configured as the default runtime in /etc/docker/daemon.json.

Step 7: Set Up Reverse Proxy with SSL and Basic Auth

To access Ollama securely from your LAN or the internet, we will use Caddy as a reverse proxy. Caddy automatically obtains and renews SSL certificates. We will add basic authentication to protect the endpoint.

Create a Caddyfile in the same directory. Replace ollama.example.com with your domain.

ollama.example.com {
    reverse_proxy ollama:11434
    basicauth {
        user {$PROXY_USER} {$PROXY_PASSWORD_CHANGE_ME}
    }
}

Add the Caddy service to your docker-compose.yml. We will append it to the existing file.

  caddy:
    image: caddy:2-alpine
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    environment:
      - PROXY_USER=${PROXY_USER}
      - PROXY_PASSWORD_CHANGE_ME=${PROXY_PASSWORD_CHANGE_ME}
    depends_on:
      - ollama

volumes:
  caddy_data:
  caddy_config:

Important: The environment variables for Caddy are passed from the .env file. Caddy will hash the password at runtime and use it for Basic Auth. Restart the stack to apply the changes.

docker compose down && docker compose up -d

Now, access https://ollama.example.com from your browser. You should be prompted for a username and password. After logging in, you will see the Ollama API root page (404 error is expected for /).

Step 8: Configure Backups

Your models are stored in the ./ollama_data directory. Backing up this directory is sufficient. However, models can be re-pulled easily. The critical data is your custom configurations and any custom modelfiles.

Create a simple backup script. This script uses tar to create a compressed archive with a timestamp.

#!/bin/bash
BACKUP_DIR=~/backups/ollama
mkdir -p $BACKUP_DIR
DATE=$(date +%Y%m%d_%H%M%S)
tar -czvf ${BACKUP_DIR}/ollama_backup_${DATE}.tar.gz -C ~/ollama ollama_data

Save this script as backup.sh, make it executable, and add it to your crontab for daily execution.

chmod +x backup.sh && ./backup.sh

Advanced Configuration & Optimization

Model Management

Ollama supports custom model files (Modelfile). You can create a custom model that adjusts parameters like temperature or system prompts. For example, create a file called Modelfile:

FROM llama3.2:1b
SYSTEM "You are a helpful assistant that speaks only in JSON."

Build the custom model with:

docker exec -it ollama ollama create my-json-assistant -f ./Modelfile

Performance Tuning

Performance is primarily limited by hardware. However, you can control the context window size via the OLLAMA_CONTEXT_LENGTH environment variable. A larger context uses more RAM/VRAM. Set this in your .env file and add it to the environment section of the Compose file.

    environment:
      - OLLAMA_CONTEXT_LENGTH=4096

Monitoring

Use docker stats to monitor resource usage. For more advanced monitoring, consider integrating with Prometheus and Grafana using the cAdvisor container, but this is beyond the scope of this guide.

Optional Hardening

Warning: The following hardening measures are specific to the Ollama image. Applying these blindly to other containers will likely break them. Test in a staging environment first.

You can run the container with a read-only root filesystem and drop all Linux capabilities. This reduces the attack surface. However, Ollama needs to write to /root/.ollama. We must mount that directory as a writable volume.

    security_opt:
      - no-new-privileges:true
    read_only: true
    tmpfs:
      - /tmp
    cap_drop:
      - ALL

Note: With read_only: true, the only writable layer is the mounted volume. If Ollama needs to write to other paths (e.g., /usr/local/bin for updates), this will fail. Test thoroughly.

Troubleshooting

Conclusion

You now have a fully functional, secure Ollama installation running via Docker Compose. We have covered the initial setup, model management, GPU acceleration, and secure remote access via Caddy. This foundation is stable and ready for integration with frontends like Open WebUI.

The key takeaway is the separation of concerns: the docker-compose.yml defines the infrastructure, the .env file holds the configuration and secrets, and the data lives in a persistent volume. This structure allows for easy upgrades and rollbacks. Remember to check for new Ollama versions regularly, but always test before upgrading your production instance.

FAQ

Q1: How do I update Ollama to a new version?

Update the OLLAMA_VERSION variable in your .env file to the new release tag. Then run docker compose pull ollama && docker compose up -d. This will recreate the container with the new image. Always check the release notes for breaking changes.

Q2: Can I use a different reverse proxy like Nginx or Traefik?

Yes. The principle is the same: proxy requests to http://ollama:11434 on the internal Docker network. For Nginx, you would use proxy_pass http://ollama:11434;. For Traefik, use the traefik.http.services.ollama.loadbalancer.server.port=11434 label. Ensure you implement TLS and authentication at the proxy level.

Q3: How do I remove a model to free up space?

Use the CLI inside the container. Run docker exec -it ollama ollama rm <model_name>. This will delete the model weights from the ollama_data directory. You can list installed models with docker exec -it ollama ollama list.

Q4: Is it safe to expose Ollama directly to the internet without a reverse proxy?

No. Ollama has no built-in authentication or TLS. Exposing port 11434 to the internet allows anyone to execute models and access your hardware. Always use a reverse proxy with Basic Auth or better, OAuth2/OIDC. The Compose file in this guide binds to 127.0.0.1 precisely to prevent this.

Q5: Why is my GPU not being used?

Check if the deploy section is uncommented in your Compose file. Verify the NVIDIA Container Toolkit is installed on the host. Run docker exec -it ollama nvidia-smi inside the container. If it fails, the runtime is not configured. Ensure /etc/docker/daemon.json has "default-runtime": "nvidia" and restart Docker.