Running LLMs Locally: GGUF, Quantization, and Memory Planning

The single most common reason a local model fails is memory: the model weights fit, but the KV cache spills over, inference crawls, or the process gets killed. Running models locally is a memory planning exercise first and a performance exercise second. This guide covers the GGUF format, how quantization actually trades quality for footprint, and the exact math for fitting a model onto your hardware.

Why GGUF Won

GGUF (GPT-Generated Unified Format) is the file format introduced by llama.cpp, and it became the standard for local inference because it packages everything into one portable binary:

• Weights at a chosen quantization
• Tokenizer configuration (so the model cannot be run with the wrong vocab)
• Architecture metadata and quantization parameters

One file in, one model out. No dependency on PyTorch checkpoints, no transformers version pinning. If you have browsed Hugging Face for local models, you have seen GGUF everywhere — it is what Ollama's registry, LM Studio, and llama.cpp all consume.

The Quantization Ladder

Quantization converts FP16/FP32 weights into lower-precision integers. The naming scheme (Q4_K_M) encodes the format: 4 bits per weight, K-means calibration, M (medium) or S (small) granularity of the scaling tensors.

Quant	Bits/weight	Size of 70B model	Typical use
Q2_K	~2.6	~18GB	Absolute floor, visible quality loss
Q3_K_M	~3.9	~28GB	Small models on weak hardware
Q4_K_M	~4.5	~43GB	Ollama default, best size/quality balance
Q5_K_M	~5.5	~49GB	Quality-sensitive local use
Q6_K	~6.6	~54GB	Near-lossless on consumer hardware
Q8_0	~8.5	~70GB	Best local quality, high memory cost
FP16	16	~140GB	Reference precision, datacenter only

Ollama defaults to Q4_K_M, which is roughly 4.5 bits per weight. The rule that matters: models under ~7B lose the most quality at 4-bit, so a larger model at Q4 almost always beats a tiny model at Q4. If you have the memory headroom, step code models up to Q8_0.

Quantization and Coding Quality

Quality loss is not uniform across tasks. Exact-output tasks like code generation degrade first: a Q4 model produces more subtle off-by-one errors and API-misuse bugs than the same model at Q8_0. For a concrete example, qwen3-coder:30b ships at:

Quant	Download size	When to use
Q4_K_M	19GB	24GB GPU or 32GB Mac, fits with context
Q8_0	32GB	48GB+ hardware, best local quality
FP16	61GB	Reference precision, multi-GPU

The Memory Math

Total memory needed is never just the weights:

Total = Model_Weights + KV_Cache + Runtime_Overhead

KV_Cache ≈ 2 * num_layers * max_tokens * num_kv_heads * head_dim * dtype_bytes

The factor of 2 accounts for the separate K and V tensors stored per layer. The KV cache scales linearly with context length, which is why a model that fits at 4K context can OOM at 128K.

Practical rule of thumb from the Ollama catalog:

• A 70B model at default context adds roughly 14GB of KV cache at 32K tokens and over 40GB at 128K.
• At default (shorter) context, expect 2-6GB of KV cache on top of your weights.

That is why every model card says "fits in 24GB" and your 24GB GPU still runs out of memory: the card is quoting weights, and you are paying for context too.

Apple Silicon: Unified Memory Changes Everything

On Apple Silicon, CPU and GPU share one pool of RAM. There is no separate VRAM budget: a 32GB Mac M-series can hold a model that needs ~28GB total, which would never fit a 24GB NVIDIA GPU. This is why Macs punch above their weight for local inference.

Two backends matter on Mac:

• Metal (via llama.cpp/Ollama): GPU offload with -ngl (number of GPU layers). On Apple Silicon, offloading everything to the Metal GPU is usually fastest.
• MLX: Apple's array framework, used by LM Studio and available in Ollama's MLX engine. In Ollama 0.32.6+, the MLX engine automatically uses a model's MTP head for speculative decoding, which accelerates generation on Apple GPUs.

Apple's unified memory also has a catch: bandwidth is shared. Generation speed on a Mac is dominated by memory bandwidth, and the large context you can afford on 64GB will still feel slower than a mid-range NVIDIA GPU on long generations.

Tuning Ollama

# Halve KV cache memory with minimal quality loss (per-request, run before serving)
OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

# Cap context so the KV cache cannot blow past your RAM
ollama run qwen3-coder:30b --keep-alive 5m
# In a Modelfile, set num_ctx:

# Modelfile
FROM qwen3-coder:30b
PARAMETER num_ctx 32768
PARAMETER temperature 0.2
PARAMETER num_predict 2048

ollama create code-mac -f Modelfile

Context is the silent memory killer. num_ctx 32768 on a 30B model consumes a meaningful chunk of RAM that you could otherwise spend on a higher quant. Match context to the actual job: agentic coding needs 32K+, autocomplete needs 8-16K.

Tuning llama.cpp

llama-server -m qwen3-coder-30b-q4_k_m.gguf \
  -ngl 99 \        # offload all layers to GPU
  -c 32768 \       # context size
  -b 512 \         # batch size (smaller = less memory spikes on prefill)
  -fa on           # Flash Attention where supported

-ngl is the lever that trades CPU RAM against GPU memory: drop it to spill layers to system RAM and avoid GPU OOM, at the cost of slower inference and PCIe transfer overhead.

Memory Cheat Sheet by Hardware

Memory	Hardware	Best coding model	Best general model
8GB	RTX 3060, M2 Air	qwen2.5-coder:7b (4.7GB)	gemma4:e2b-it-qat (4.3GB)
12GB	RTX 3060 12GB	deepseek-coder-v2:16b (8.9GB)	gemma4:12b / qwen3.5:9b
16GB	RTX 4060 Ti, 16GB laptop	qwen2.5-coder:14b (9GB)	gpt-oss:20b (14GB)
24-32GB	RTX 4090, 32GB Mac	qwen3-coder:30b (19GB) / qwen3.6:27b	deepseek-r1:32b
48GB+	2x RTX 3090, 64GB Mac	qwen2.5-coder:32b (q8_0)	llama3.3:70b (43GB)
80GB	H100, A100 80GB	qwen3-coder:30b (fp16)	gpt-oss:120b

Sizes quoted are the Q4_K_M download; remember the KV cache on top.

Implementation Checklist

• Pick the quant by memory budget first: Q4_K_M default, Q8_0 for code when it fits
• Compute weights + KV cache + overhead, never weights alone
• Set num_ctx explicitly; never run at the model's max context by default
• On Mac, offload fully to Metal and prefer MLX/MTP tags for generation speed
• Use OLLAMA_KV_CACHE_TYPE=q8_0 to halve KV cache when context-heavy
• For models under 7B, prefer a larger model at Q4 over a tiny model at Q4
• Verify with ollama ps / GPU tools that weights AND cache fit before benchmarking

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub