← Cheatsheets

Tags: ollama, llm, local-ai, inference, quantization, gguf, llama.cpp, beginner Last updated: 2026-07-02

Ollama Cheatsheet

What Is Ollama?

Ollama is the easiest way to run large language models locally. It wraps llama.cpp under the hood and provides a simple CLI and REST API — no CUDA setup, no Python environments, no manual model downloads.

Installation

Linux

curl -fsSL https://ollama.com/install.sh | sh

macOS

# Download from ollama.com or use Homebrew
brew install ollama

Windows

Download the installer from ollama.com — native Windows build, no WSL required.

Docker

docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

Post-install: The Ollama service starts automatically. On Linux it runs as a systemd service; on macOS/Windows it runs as a background app.

Quick Start

# Download and run a model in one command
ollama run llama3.2

# Download only (no chat)
ollama pull llama3.2

# List downloaded models
ollama list

# Remove a model
ollama rm llama3.2

# Stop a running model
ollama stop llama3.2

The first time you run a model, Ollama downloads it. After that it loads from cache. Type /help in the chat interface to see slash commands.

🚨 Pro Tip: Model Sizes & VRAM Requirements

This is the single most important thing to understand before using Ollama:

Rule of thumb:

• A Q4_K_M quantized model needs roughly 0.5 GB per billion parameters of VRAM.
• A 70B model at Q4_K_M (~35GB) will NOT fit on a single RTX 3090 (24GB) — you need dual GPUs or a 48GB A6000.
• Ollama can offload to system RAM when VRAM runs out, but inference becomes 10–50x slower. Avoid this if possible.
• Check your VRAM before pulling a large model. On Linux: nvidia-smi. On macOS: Activity Monitor.
• When in doubt, start small. A 7B or 8B model (like Llama 3.1 8B or Mistral 7B) at Q4_K_M runs comfortably on 8GB VRAM and is remarkably capable.

The Ollama CLI

Environment Variables

Modelfile Reference

A Modelfile lets you customise how a model behaves. Create a file called Modelfile (no extension):

FROM llama3.2

# System prompt sets the model's persona
SYSTEM "You are a senior Python developer. Be concise and use type hints."

# Override default generation parameters
PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER top_k 40
PARAMETER num_ctx 4096
PARAMETER repeat_penalty 1.1

# Load custom adapters (LoRA)
ADAPTER ./my-lora.safetensors

# Set the licence (for custom models)
LICENSE MIT

Then build it:

ollama create my-custom-model -f ./Modelfile
ollama run my-custom-model

Key Parameters

REST API

Ollama exposes a full REST API on http://localhost:11434. No auth by default (keep it local or use a reverse proxy).

Generate (one-shot)

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

Chat (conversation)

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"}
  ],
  "stream": false
}'

List models

curl http://localhost:11434/api/tags

Pull a model

curl http://localhost:11434/api/pull -d '{
  "model": "llama3.2"
}'

Python example

import requests

response = requests.post("http://localhost:11434/api/chat", json={
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Say hello in 5 languages."}],
    "stream": False
})
print(response.json()["message"]["content"])

Popular Models

Using Ollama with Tools

Open WebUI (ChatGPT-like frontend)

docker run -d \
  -p 3000:8080 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000 — add Ollama as the backend.

Continue.dev (VS Code / Cursor)

1. Install the Continue extension in VS Code.
2. Set the model provider to Ollama.
3. Select a downloaded model.
4. Get autocomplete, chat, and inline editing powered by your local model.

AnythingLLM (RAG with your documents)

1. Download AnythingLLM from anythingllm.com.
2. Select Ollama as the LLM provider.
3. Point it at your documents for local RAG.

LangChain / LlamaIndex

from langchain_ollama import OllamaLLM

llm = OllamaLLM(model="llama3.2", temperature=0.3)
result = llm.invoke("Write a haiku about servers")
print(result)

Troubleshooting

Tips & Advice

• Start with a 7–8B model (Llama 3.1 8B or Mistral 7B) at Q4_K_M. Runs on any GPU with 6–8GB VRAM.
• Use num_ctx to balance context vs VRAM. 4096 tokens is the sweet spot.
• Set OLLAMA_KEEP_ALIVE=0 when experimenting with different models.
• Don't expose Ollama to the internet without a reverse proxy with auth.
• Keep Ollama updated — re-run the install script: curl -fsSL https://ollama.com/install.sh | sh
• Use Modelfiles to lock in your preferred parameters.
• Ollama can run multiple models concurrently if your VRAM allows.
• For RAG workflows, use a smaller embedder model (like nomic-embed-text) alongside your main LLM.
• Monitor your VRAM with ollama ps.
• If you have an AMD GPU, ensure ROCm is installed. Windows AMD support has limitations.