Skip to content
The methodOpen data
Back to the collection

Efficient Memory Management for Large Language Model Serving with PagedAttention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang

Show all 9 authors

Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica

2023 · arXiv

Language modelsComputer systemsAlgorithms
Read the paper
HN score
0
Mentions
4
Readers
3
Months
4
What’s in a score?

From the abstract

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4$\times$ with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

jbarrow
Neutral mention · rule
The whole thing feels AI written, generated from the codebase. For instance, it goes into (nano)vLLM internals and doesn’t mention PagedAttention once (one of the core ideas that vLLM is based on)[1]. Also mentions that Part 2 will cover dense vs MoE’s, which is weird because nanovllm hardcodes a dense Qwen3 into the source. Here are better (imo) explainers about how vLLM works: - https://hamzaelshafie.bearblog.dev/paged-attention-from-firs... - https://www.aleksagordic.com/blog/vllm - https://huggingface.co/blog/continuous_batching Aleksa’s blog is a bit in the weeds for my taste but it’s really worth working through. A lot of the magic of vLLM happens in the PagedAttention kernels, which are really succinctly implanted in nanovllm. And the codebase is great and readable by itself! — 1. https://arxiv.org/abs/2309.06180
Original comment Nano-vLLM: How a vLLM-style inference engine works
jmorgan
Neutral mention · rule
Sorry this isn't more obvious. Ideally VRAM usage for the context window (the KV cache) becomes dynamic, starting small and growing with token usage, whereas right now Ollama defaults to a size of 2K which can be overridden at runtime. A great example of this is vLLM's PagedAttention implementation [1] or Microsoft's vAttention [2] which is CUDA-specific (and there are quite a few others). 1M tokens will definitely require a lot of KV cache memory. One way to reduce the memory footprint is to use KV cache quantization, which has recently been added behind a flag [3] and will 1/4 the memory footprint if 4-bit KV cache quantization is used (OLLAMA_KV_CACHE_TYPE=q4_0 ollama serve) [1] https://arxiv.org/pdf/2309.06180 [2] https://github.com/microsoft/vattention [3] https://smcleod.net/2024/12/bringing-k/v-context-quantisatio...
Original comment Qwen2.5-1M: Deploy your own Qwen with context length up to 1M tokens
AaronFriel
Neutral mention · rule
The PagedAttention paper is a good starting point as it represents the first major open source inference engine that had "pretty good" batch performance for transformers. https://arxiv.org/pdf/2309.06180
Original comment Gemini Flash
AaronFriel
Neutral mention · rule
It indeed is. An attention mechanism's key and value matrices grow linearly with context length. With PagedAttention[1], we could imagine an external service providing context. The hard part is the how, of course. We can't load our entire database in every conversation, and I suspect there are also challenges around training (perhaps addressed via LandmarkAttention[2]) and building a service efficiently retrieve additional key-value matrices. The external service vector database may require tight timings necessary to avoid stalling LLMs. To manage 20-50 tokens/sec, answers must arrive within 50-20ms. And we cannot do this in real-time, pausing the transformer when a layer…
Read the full comment
produces a query vector stalls the batch, so we need a way to predict queries (or embeddings) several tokens ahead of where they'd be useful and inject the context in when it's needed, and to know when to page it out. [1] https://arxiv.org/abs/2309.06180 [2] https://arxiv.org/abs/2305.16300
Original comment LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

Keep following the thread.

More in Language models

More from the same topics.