Deploy AI Models with Ollama and Docker

Learn how to run large language models locally using Ollama and Docker. This guide walks through setting up a local AI development environment for experimentation and engineering workflows.

Prerequisites

Before we begin, make sure you have:

Docker installed (Get Docker )
Basic familiarity with command line
At least 8GB RAM (16GB+ recommended for larger models)

What is Ollama?

Ollama is a tool that makes it easy to run large language models locally. It handles:

Model downloading and management
Optimal resource utilization
A simple API for inference

Step 1: Pull the Ollama Docker Image


docker pull ollama/ollama

Step 2: Run Ollama Container


docker run -d \
  --name ollama \
  -p 11434:11434 \
  -v ollama:/root/.ollama \
  ollama/ollama

This command:

Runs Ollama in detached mode (-d)
Maps port 11434 for API access
Creates a volume to persist downloaded models

Step 3: Download a Model


docker exec -it ollama ollama pull llama2

You can replace llama2 with other models like:

mistral - Fast and efficient
codellama - Optimized for code
llama2:13b - Larger, more capable

Step 4: Test the Model


docker exec -it ollama ollama run llama2 "Hello, how are you?"

Step 5: Use the API

Ollama exposes a REST API on port 11434:


curl http://localhost:11434/api/generate -d '{
  "model": "llama2",
  "prompt": "Explain Docker in simple terms"
}'

Python Example


import requests
 
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama2",
        "prompt": "Write a haiku about coding",
        "stream": False
    }
)
 
print(response.json()["response"])

Docker Compose Setup

For a more robust local setup, use Docker Compose:

docker-compose.yml


version: '3.8'
 
services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
 
volumes:
  ollama_data:

Note: The GPU configuration is optional and requires NVIDIA Container Toolkit.

Tips and Best Practices

1. Model Selection

Choose the right model for your use case:

Chat/General: llama2, mistral
Code: codellama, deepseek-coder
Small/Fast: phi, tinyllama

2. Resource Management


# Check running models
docker exec ollama ollama list
 
# Stop a model to free memory
docker exec ollama ollama stop llama2

3. GPU Acceleration

For faster inference, use GPU passthrough:


docker run -d --gpus all ollama/ollama

Troubleshooting

Out of Memory

Try a smaller model (e.g., llama2:7b instead of llama2:13b)
Increase Docker memory limits
Close other applications

Connection Refused

Ensure the container is running: docker ps
Check port mapping: docker port ollama

Slow Performance

Enable GPU acceleration
Use quantized models (e.g., llama2:7b-q4_0)

Next Steps

Now that you have Ollama running locally, you can:

Build a chatbot - Create a web interface with Next.js
RAG applications - Combine with vector databases
Fine-tuning - Customize models for your use case

Resources

Found this helpful? Share it with colleagues who are building local AI workflows.

Have questions? Reach out on GitHub or LinkedIn .