Training a large language model dominates research papers and benchmark leaderboards. Serving it is where engineering reality begins. A model that achieves state-of-the-art scores in evaluation is worthless in production if it takes five seconds to respond to a user, exhausts GPU memory after twenty concurrent requests, or costs ten times more per query than the business can support.
The common misunderstanding is that serving is just "running the model." In practice, inference engines are complex systems that interleave memory management, batching strategy, hardware utilization, and scheduling. Those decisions can produce a two-to-four times difference in throughput on identical hardware, serving the same model.
TLDR
- An inference engine tokenizes a prompt, runs it through the model layer by layer, and streams output tokens back. The hard part is doing this efficiently for many simultaneous users.
- The KV cache stores attention key-value pairs per token so they are never recomputed, trading GPU memory for speed.
- Continuous batching slots new requests into running batches the moment a sequence finishes, instead of waiting for a whole batch.
- PagedAttention, speculative decoding, and quantization stack on top to cut memory waste and per-token latency.
What an inference engine actually solves#
The core challenge of LLM serving is the mismatch between how GPUs work best and how user requests arrive. GPUs are maximally efficient when computing on large batches of uniform work. User requests arrive sporadically, have variable-length prompts, generate variable-length outputs, and need responses in milliseconds.
A naive serving approach (wait for a batch of requests, run the full batch together, return all results) works poorly. Short requests finish early but wait for slow ones. Long prompts consume more memory. GPU utilization drops when requests are not aligned. Every token of output must be generated one step at a time, so a 500-token response requires 500 sequential forward passes.
Modern inference engines solve this through a combination of:
| Problem | Solution |
|---|---|
| Recomputing attention over all previous tokens | KV cache |
| Variable request arrival times | Continuous batching |
| KV cache memory fragmentation | PagedAttention |
| Single-token generation bottleneck | Speculative decoding |
| Underutilized GPU for short prompts | Chunked prefill and dynamic batching |
Each of these techniques targets a specific inefficiency. Together they determine whether a serving system can handle real traffic at acceptable cost.
Prefill and decode are two different phases#
LLM inference has two distinct phases, and understanding them is the foundation for everything else.
Prefill is the phase where the entire input prompt is processed at once. All prompt tokens are passed through the model in parallel, and the model computes attention over all of them simultaneously. This is relatively fast and GPU-efficient because you can compute many tokens at once.
Decode is the autoregressive phase where the model generates one output token at a time. Each new token is appended to the context, and the model runs another forward pass to predict the next one. This continues until the model generates a stop token or hits the maximum output length.
The decode phase is slower and less GPU-efficient than prefill because each step only produces one new token, but the forward pass must still run through all model layers. This is why "time to first token" (TTFT) and "time per output token" (TPOT) are the two primary latency metrics in LLM serving. They measure prefill latency and decode latency respectively.
The full serving path runs left to right, with the decode step looping until a stop token appears.
What happens inside the engine#
A production inference engine receives an HTTP request, processes it through several stages, and streams tokens back to the caller. The scheduler is what lets one batched GPU pass serve many clients at once.
Receive and tokenize
The request arrives with a text prompt. The engine tokenizes it using the model's vocabulary, converting text into a sequence of integer token IDs.
Schedule
The scheduler decides when and how to include the request in a batch. It considers available GPU memory for the KV cache, current batch state, and request priority.
Prefill
The full prompt is passed through all model layers. Attention runs over all input tokens in parallel. Key and value vectors for each token and each layer are computed and stored in the KV cache.
Decode loop
The model generates one token per step. At each step, attention over previous tokens uses cached K and V vectors. Only the new token's K and V need to be computed and appended to the cache.
Stream output
Each generated token is decoded back to text and streamed to the client. Most modern APIs stream token by token rather than waiting for the full response.
Finalize
When generation ends, the KV cache memory for this request is freed and the slot is made available for new requests.
The KV cache trades memory for speed#
The KV (key-value) cache is the single most important memory structure in LLM inference. Without it, every decode step would require recomputing attention over all previous tokens from scratch, an O(n²) cost per step that would make long responses prohibitively slow.
With the KV cache, each layer stores the key and value vectors for every token it has processed. On each new decode step, only the newest token's K and V are computed and appended. Attention can then attend to the full cached history without recomputation.
The cost is memory. For a model with L layers, H attention heads, and a head dimension of d_k, the KV cache for a single request with n tokens requires:
KV cache size = 2 × L × H × d_k × n × dtype_bytes
For a large model at bfloat16 precision, a single request with a few thousand tokens can require gigabytes of GPU memory. Scale that to hundreds of concurrent users and the KV cache becomes the primary memory bottleneck, not the model weights themselves.
KV cache is the real memory constraint
In typical production serving, GPU memory is divided between model weights (fixed) and KV cache (dynamic, per-request). As context lengths grow and concurrent users increase, KV cache pressure determines the maximum batch size and directly affects throughput.
PagedAttention manages the KV cache like virtual memory#
The original KV cache implementation allocated a contiguous block of GPU memory for each request upfront, sized for the maximum possible output length. This caused two problems: internal fragmentation (unused space within allocated blocks) and reserved-but-unused memory preventing other requests from being served.
Woosuk Kwon et al. introduced PagedAttention in the paper "Efficient Memory Management for Large Language Model Serving with PagedAttention" (SOSP 2023). The key insight was to apply OS virtual memory concepts to KV cache management.
Instead of contiguous allocation, PagedAttention divides the KV cache into fixed-size blocks (pages). Each page holds KV vectors for a fixed number of tokens. Pages are allocated on demand as generation proceeds, and are freed immediately when a request completes. Non-contiguous pages are tracked through a page table, similar to how an OS manages virtual memory.
The paper found that existing systems wasted 60–80% of allocated KV cache memory through fragmentation. PagedAttention reduced this waste to near zero. The result, implemented in vLLM, was a 2–4x throughput improvement over prior state-of-the-art serving systems at the same latency level.
An additional benefit: KV cache pages for a shared prefix (such as a system prompt used across all requests) can be physically shared between requests, reducing memory duplication when many users share the same preamble.
Continuous batching fills GPU slots without waiting#
Static batching (collect N requests, run them together, return all results) causes GPU underutilization when requests finish at different times. Short requests complete early but hold their memory slot until the entire batch is done.
Continuous batching (also called in-flight batching or iteration-level scheduling) solves this by treating each generation step as the scheduling unit rather than the entire request. When any request in the current batch completes a decode step and generates a stop token, its slot is immediately freed. A new waiting request can be inserted into the next forward pass.
Anyscale published a detailed explanation of continuous batching for LLM inference, demonstrating how it reduces time-to-first-token for queued requests and improves GPU utilization. The key benefit is that GPU utilization stops depending on request length variance. Short and long requests coexist in the same batch without blocking each other.
vLLM implements continuous batching as its default scheduler. The vLLM documentation describes how the scheduler interleaves prefill and decode operations across requests to maximize throughput while respecting memory constraints. This is the same scheduler shown in the sequence diagram above.
Speculative decoding drafts ahead to cut latency#
Speculative decoding addresses the decode bottleneck from a different angle. Instead of running the large target model for every token, a smaller draft model generates several tokens quickly, and the large model verifies them all in a single parallel pass.
If the large model agrees with all k draft tokens, k tokens are accepted in one step, at the same cost as a single-token step. If the large model rejects a draft token at position i, tokens 1 through i-1 are kept and the large model generates the correct token at position i, then the process restarts.
NVIDIA's TensorRT-LLM team has documented speculative decoding with TensorRT-LLM, achieving throughput speedups of up to 3.55x on Llama 3.3 70B with appropriate draft models. The speedup is highest when the draft model and target model agree frequently, which depends on the task and the quality of the draft model.
The tradeoff is added complexity: the system must run two models, manage their KV caches separately, and handle token rejection and rollback. For latency-critical applications where TPOT is the key metric, speculative decoding is a powerful tool.
Quantization shrinks the memory footprint#
Model weights are typically stored in float32, float16, or bfloat16. Quantization reduces precision to int8 or int4, shrinking the memory footprint and enabling faster matrix multiplications on hardware that has integer compute units.
A 70B parameter model at bfloat16 requires roughly 140 GB of GPU memory for weights alone, more than two A100 80GB GPUs. At int4 quantization, this drops to approximately 35 GB, fitting on a single high-end GPU. The tradeoff is some degradation in model quality, which varies by quantization method and task.
NVIDIA's Mastering LLM Techniques: Inference Optimization technical blog covers quantization methods including GPTQ, AWQ, and FP8, each with different quality-performance tradeoffs. TensorRT-LLM applies these optimizations automatically for supported architectures.
| Precision | Memory (70B model) | Quality impact | Hardware support |
|---|---|---|---|
| bfloat16 | ~140 GB | Baseline | H100, A100, most modern GPUs |
| int8 | ~70 GB | Minimal | H100, A100 with int8 units |
| int4 (GPTQ/AWQ) | ~35 GB | Small to moderate | H100, A100, consumer GPUs |
| FP8 | ~70 GB | Very small | H100, H200 |
Example: serving 100 concurrent users#
Consider a deployment serving a 13B parameter chat model with typical prompts of 500–2000 tokens and responses of 100–400 tokens.
Without optimization, a naive server might handle 5–10 concurrent requests before exhausting GPU memory. With a full serving stack:
- PagedAttention divides KV cache into 16-token pages, allocated on demand. Memory fragmentation drops to near zero.
- Continuous batching keeps the GPU fully utilized. As fast requests finish, new ones join mid-batch. Throughput scales with request volume rather than batch configuration.
- Chunked prefill breaks long prompts into chunks and interleaves them with decode steps, preventing a single long-prompt request from monopolizing the GPU during prefill.
- int8 quantization on weights reduces memory pressure, allowing a larger batch size within the same GPU budget.
- KV cache prefix sharing stores the common system prompt once, shared across all requests that use it.
The result is a system that can serve tens to hundreds of concurrent users on hardware that a naive approach would saturate at single digits.
Tradeoffs and failure modes#
Understanding the failure modes of inference systems helps prevent production surprises.
KV cache exhaustion. If too many long-context requests arrive simultaneously, the KV cache fills and new requests must wait. Systems handle this differently: vLLM preempts low-priority requests, some systems return errors. Monitoring KV cache utilization is a critical operational metric.
Prefill dominance. A burst of requests with very long prompts can saturate the GPU with prefill work, spiking time-to-first-token for all queued requests. Chunked prefill mitigates this by interleaving prefill chunks with decode steps.
Draft model mismatch in speculative decoding. If the draft model diverges from the target model (due to domain shift or temperature settings), rejection rates increase and speculative decoding can be slower than standard decoding due to the overhead of running two models.
Quantization quality regression. Aggressive quantization (int4 or lower) can degrade quality on reasoning tasks more than on generation tasks. Evaluate quantized models specifically on your use case before deploying.
Memory bandwidth vs. compute bottleneck. At small batch sizes, LLM inference is memory-bandwidth-bound, not compute-bound. More arithmetic units do not help if the GPU cannot feed them data fast enough. This is why high-bandwidth memory (HBM) capacity is as important as TFLOP counts when selecting inference hardware.
When serving system choices matter#
The serving stack matters most in these scenarios:
- High-concurrency deployments where many users share a single model instance. Continuous batching and PagedAttention provide the biggest returns here.
- Long-context applications such as document analysis or multi-turn conversations with many prior turns. KV cache size and management become the binding constraint.
- Latency-sensitive products like real-time chat or voice interfaces. Speculative decoding and efficient prefill scheduling reduce TPOT.
- Cost-constrained environments where maximizing requests per GPU-hour directly affects unit economics. Throughput optimization matters most.
For most teams starting with LLM integration, a managed inference API abstracts all of this. But when you hit cost walls, latency ceilings, or need custom control, understanding the engine beneath the API is what makes the difference. For a broader view of how these systems fit into larger AI architectures, see system design for AI products and how GPUs run AI workloads.
Key Takeaways#
LLM inference is not just "running the model." It is a carefully orchestrated system of memory management, scheduling, and hardware utilization. The KV cache eliminates redundant attention computation at the cost of memory. PagedAttention manages that memory without fragmentation. Continuous batching keeps GPU utilization high across variable request patterns. Speculative decoding reduces per-token latency by parallelizing verification. Quantization trades precision for memory and throughput.
The result of combining these techniques is a 2–10x improvement in throughput and utilization compared to naive serving. Understanding them lets you make informed choices about serving infrastructure, debug capacity problems, and evaluate vendor claims about inference efficiency.
FAQ#
What is the difference between latency and throughput in inference?
Latency measures how long a single user waits, typically broken into time to first token (TTFT, or prefill latency) and time per output token (TPOT, or decode latency). Throughput measures how many tokens or requests the system can process per second across all users. Techniques that improve throughput (like larger batches) often increase latency for individual requests, and vice versa.
Why can't you just use more GPUs to solve all serving problems?
More GPUs help with model parallelism and total capacity, but some bottlenecks are architectural. The autoregressive decode loop is inherently sequential, since each token depends on the previous one. More GPUs do not parallelize this within a single request. The KV cache memory constraint also exists per-GPU unless distributed KV cache is implemented. Scaling out helps with total throughput but does not fix single-request latency.
What is the relationship between context window length and serving cost?
Longer contexts increase KV cache size quadratically in computation (attention) but linearly in memory (stored K and V vectors). In practice, serving cost scales roughly linearly with context length per request in terms of memory, but longer prompts also spend more time in prefill, which temporarily reduces throughput. Very long contexts can limit the number of requests that can be served concurrently due to memory pressure.
How does vLLM compare to TensorRT-LLM?
vLLM is an open-source Python-first serving framework focused on ease of deployment and broad model support. It pioneered PagedAttention and continuous batching. TensorRT-LLM is NVIDIA's inference optimization library, focused on maximum hardware utilization through custom CUDA kernels, quantization, and speculative decoding, with more complex setup and NVIDIA-only hardware requirements. Many production deployments combine both: TensorRT-LLM for engine optimization, and a serving layer on top for scheduling and request management.
When should I use a managed inference API vs. self-hosting?
Use managed APIs when you are still validating product-market fit, when traffic is unpredictable, or when serving complexity exceeds your team's operational capacity. Self-host when you need control over data residency, when per-token costs at scale exceed API pricing, when you need custom model versions, or when latency requirements demand dedicated hardware. The serving concepts in this article apply to both. Managed APIs implement them for you, while self-hosting requires you to configure and tune them directly.
