Deploy GPU ↗
VRAM & MEMORY GUIDE5 min read • Updated 2026-10-22

How Much VRAM Does KV Cache Really Use? The Exact Formula & Sizing Guide

Demystifying token memory footprint: How context length, attention architectures, and quantization impact GPU memory.

A
Alex Vance
Inference Systems Engineer
When running local LLMs, model weights are static, but KV cache memory scales linearly with context length and concurrency. If your model works fine for short prompts but suddenly crashes with an Out-of-Memory (OOM) error on long documents, your KV cache is the culprit. Here is the exact math to size it accurately.

⚡ Key Architectural Takeaways

  • KV cache memory formula: 2 × Layers × KV Heads × Head Dimension × Context Length × Bytes Per Precision.
  • Grouped-Query Attention (GQA) reduces KV cache by 4x to 8x compared to legacy Multi-Head Attention (MHA).
  • Multi-Head Latent Attention (MLA), introduced by DeepSeek, compresses KV cache by over 90% into low-rank latent vectors.
  • Switching from FP16 KV cache (2 bytes) to Q8_0 (1 byte) or Q4_0 (0.5 bytes) saves 50% to 75% of cache memory with zero loss in generation logic.

1. The Universal KV Cache Formula

For any transformer architecture, the memory consumed by the key-value cache for a single sequence is given by:

$$\text{KV Cache (Bytes)} = 2 \times N_{\text{layers}} \times N_{\text{kv\_heads}} \times d_{\text{head}} \times L_{\text{context}} \times B_{\text{precision}}$$

Where:

  • 2 accounts for both the Key and Value matrices stored at each layer.
  • N_layers is the number of transformer decoder layers (e.g., 64 in Qwen 3.8 27B, 80 in Llama 3.3 70B).
  • N_kv_heads is the number of Key/Value heads. In modern models with Grouped-Query Attention (GQA), this is typically 8, whereas in old MHA it was equal to the query heads (32–64).
  • d_head is the dimension per attention head (typically 128).
  • L_context is the active sequence length in tokens (e.g., 32,768).
  • B_precision is the byte size per element (2 for FP16, 1 for FP8 / Q8_0, 0.5 for Q4_0).

2. Memory Footprint by Context Window

Notice how KV cache scales for Llama 3.3 70B (80 layers, 8 KV heads, 128 head dim) across various sequence lengths:

Context Length FP16 KV Cache (2 Bytes) Q8_0 KV Cache (1 Byte) Q4_0 KV Cache (0.5 Bytes)
8,192 (8K) 2.62 GB 1.31 GB 0.65 GB
32,768 (32K) 10.48 GB 5.24 GB 2.62 GB
65,536 (64K) 20.97 GB 10.48 GB 5.24 GB
131,072 (128K) 41.94 GB 20.97 GB 10.48 GB

At 128K context, storing an unquantized FP16 KV cache consumes 42 GB of VRAM on its own—more than the model weights themselves! This is why enabling 8-bit or 4-bit KV cache quantization is essential for document ingestion and codebase indexing.

INTERACTIVE BENCHMARK TOOL

Calculate Exact VRAM & Tokens/Sec for Your Rig

Adjust context lengths, KV cache quantizations, and concurrency models directly.

Explore 24GB VRAM Limits in Calculator →
ON-DEMAND RUNPOD CLOUD

Test 70B & MoE Models from $0.45/hr

Boot 1-click vLLM and Ollama instances on RTX 4090, RTX 5090, or 8x H100 with zero upfront hardware costs.

Deploy on RunPod →
VAST.AI GPU MARKETPLACE

Rent Dual 3090 (48GB) from $0.78/hr

Cheapest way to run full Q4_K_M 70B models in unconstrained high-VRAM cloud memory.

Rent on Vast.ai ($0.78/hr) →