Ollama: Run AI Models Locally for Free (Complete Setup Guide)

On this page
  1. Installation
  2. Popular Models
  3. Essential Commands
  4. Hardware Requirements
  5. Using the API (Python)
  6. Connect Ollama to OpenClaw (Free Local AI Agent)
  7. Performance Tips
  8. Local Ollama vs Cloud APIs
  9. Which Model Should You Start With?
  10. Frequently Asked Questions
  11. Get Started
Quick answer: Ollama is a free, open-source tool that runs LLMs — Llama 3, DeepSeek, Qwen, Gemma, Mistral and more — locally on your own machine. There's no account, no API key, and $0 per token: install it, run ollama run llama3.1, and you're chatting with a local model in 30 seconds. It exposes an OpenAI-compatible API at localhost:11434, so existing code works unchanged.

Ollama makes local AI deployment as simple as a single command — full data privacy, zero-cost inference, and offline use, with no cloud accounts or per-token fees. This guide covers install, the popular models, hardware needs, the API, and how local stacks up against cloud.

Installation

# macOS (or download from ollama.com/download, needs macOS 11+)
brew install ollama

# Windows (or the installer at ollama.com/download, needs Windows 10+)
winget install Ollama.Ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Docker
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Then run a model — Ollama auto-downloads it (~4.7 GB for Llama 3.1 8B) and drops you into an interactive chat:

ollama run llama3.1
ModelSizesBest ForDownload (Q4)
Llama 3.1 (Meta)8B, 70B, 405BGeneral purpose~4.7 GB (8B)
DeepSeek-R11.5B - 671BReasoning, math, coding~4.7 GB (7B)
Qwen 3 (Alibaba)0.6B - 235BMultilingual, 128K context~4.7 GB (7B)
Gemma 3 (Google)1B - 27BBalanced performance~2.5 GB (4B)
Mistral7BGeneral purpose~4.1 GB
Phi-3 (Microsoft)3B, 14BLightweight, coding~2.2 GB (3B)
Gemma 4 (Google)2B - 31BLatest gen, MoE~1.5 GB (2B)

Essential Commands

ollama pull llama3.1                              # download a model
ollama run llama3.1                               # interactive chat
ollama run llama3.1 "Explain Docker in a line"    # one-shot prompt
ollama list                                       # list downloaded models
ollama ps                                         # running models + memory
ollama rm llama3.1                                # delete a model

Hardware Requirements

Model size determines the RAM/VRAM you need:

Model SizeMinimum RAMRecommended RAMGPU VRAM (Q4)
1B - 3B4 GB8 GB2-4 GB
7B - 8B8 GB16 GB5-6 GB
13B - 14B16 GB32 GB8-10 GB
27B - 32B32 GB48 GB16-20 GB
70B64 GB96 GB38-45 GB
  • No GPU? Ollama runs on CPU, just 5-10x slower.
  • Apple Silicon uses unified memory — a 16 GB M1/M2 runs 7B-8B models well.
  • NVIDIA GPUs with 6+ GB VRAM give significant speedups; AMD/Intel GPUs work via the Vulkan backend.

Using the API (Python)

Ollama serves a fully OpenAI-compatible API at http://localhost:11434/v1 — any OpenAI SDK code works by just changing the base URL.

pip install openai
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required by SDK but not used
)

response = client.chat.completions.create(
    model="llama3.1",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is machine learning?"}
    ]
)

print(response.choices[0].message.content)

Streaming and the native ollama library are both supported:

# Streaming (OpenAI SDK)
stream = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Write a short poem about coding"}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

# Native Ollama library (pip install ollama)
import ollama
response = ollama.chat(
    model="llama3.1",
    messages=[{"role": "user", "content": "Explain Docker in one paragraph"}]
)
print(response["message"]["content"])

Connect Ollama to OpenClaw (Free Local AI Agent)

Pair Ollama with OpenClaw for a fully local, zero-cost AI agent — shell commands, file operations, browser control, scheduled tasks, and chat on WhatsApp or Telegram, all on your own hardware. OpenClaw auto-discovers your local models:

# 1. Make sure Ollama is running with a model
ollama pull llama3.1

# 2. Install and configure OpenClaw
npm install -g openclaw@latest
openclaw onboard
# Select "Ollama" from the provider list

Performance Tips

Quantization compresses models to fit less memory with minimal quality loss:

LevelQuality LossVRAM per 1B ParamsWhen to Use
Q4_K_M~1-3%~0.6 GBBest balance (default)
Q5_K_MMinimal~0.75 GBWhen quality matters more
Q8_0Near-zero~1.0 GBWhen accuracy is critical
FP16None~2.0 GBResearch only
  • Reduce KV cache memory: OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
  • Keep models loaded: OLLAMA_KEEP_ALIVE=-1 prevents unloading between requests
  • Concurrent requests: OLLAMA_NUM_PARALLEL=4 for multiple sessions
  • A 14B Q4 model on GPU beats a 70B model on CPU — smaller can be faster.

Local Ollama vs Cloud APIs

FactorOllama (Local)Cloud APIs
Cost$0 per tokenPay per token or monthly
Privacy100% local, no data leaves your machineData sent to servers
Latency10-50ms first token (typical range, not measured here)200-800ms
Model QualityGood for 80% of tasksFrontier models still superior
OfflineWorks without internetInternet required
ScalabilityLimited by hardwareInfinite

Pro tip: Most developers in 2026 run a hybrid — Ollama for high-volume and sensitive tasks, cloud APIs for complex reasoning — which typically cuts cloud costs by 60-80%.

Which Model Should You Start With?

  • 8 GB RAM, no GPU: Gemma 3 4B or Phi-3 Mini 3B
  • 16 GB RAM or 8 GB VRAM: Llama 3.1 8B or Qwen 3 8B
  • 32 GB RAM or 16 GB VRAM: DeepSeek-R1 14B or Gemma 3 27B
  • 64+ GB RAM or 24+ GB VRAM: Llama 3.1 70B or Qwen 3 32B

Frequently Asked Questions

Is Ollama really free?

Yes — it's open-source and runs on your own hardware, so there's no account, no API key, and no per-token fee. Your only cost is the machine it runs on.

Does Ollama work without a GPU?

Yes. If no GPU is detected it runs on CPU, roughly 5-10x slower. Apple Silicon Macs use unified memory and run 7B-8B models well on 16 GB.

Can I use my existing OpenAI code with Ollama?

Yes. Ollama exposes a fully OpenAI-compatible API at http://localhost:11434/v1 — point the SDK's base URL there and pass any placeholder api_key.

Get Started

ollama.com | Model Library | GitHub