Explainers: Memory

KV Cache Memory: Why a Loaded Model Can Still Run Out

Model weights are only part of the memory budget. Longer sequences can add a substantial KV cache.