You rent a GPU to run a model and it finishes in twelve seconds. You rent a CPU cluster with eight times as many cores and the same job takes four minutes. The CPU has more cores. The CPU clock is faster. But the GPU wins by an order of magnitude.
This confuses people who think of a GPU as a "faster CPU." It is not. It is a fundamentally different kind of processor, built around a different assumption about what work looks like. CPUs are optimized for a few threads doing complex, branchy, latency-sensitive work. GPUs are optimized for thousands of threads doing simple, predictable, throughput-sensitive work simultaneously.
Neural network training and inference happen to be nearly perfect fits for the GPU model. Understanding why (and where that fit breaks down) is the foundation for making good infrastructure decisions for AI systems.
TLDR
- A GPU runs thousands of simple threads in parallel, and AI workloads like the matrix multiplications inside transformer layers map naturally onto that model.
- Dedicated hardware units called Tensor Cores handle matrix multiplication at much higher throughput than general-purpose CUDA cores.
- The dominant performance constraint is usually not compute but memory bandwidth: how fast weights and activations move between GPU memory and the compute units.
What a GPU is built to solve#
A CPU is designed for sequential, low-latency tasks. It has a small number of powerful cores (typically 4–64 in a workstation), large caches, and sophisticated out-of-order execution hardware. This design makes a CPU fast at executing a single thread of work that branches unpredictably, calls system APIs, and touches arbitrary memory addresses.
A GPU has thousands of simpler cores with smaller caches and a simpler execution model. An NVIDIA H100 GPU contains 16,896 CUDA cores and 528 Tensor Cores, organized into 132 Streaming Multiprocessors (SMs), as documented in NVIDIA's Hopper architecture technical blog. The GPU's design assumption is that the application will launch many thousands of threads that all execute the same code on different data simultaneously.
Neural networks are essentially a long chain of tensor operations: matrix multiplications, element-wise activations, layer normalizations, attention score computations. Each of these operations applies the same math to millions of numbers. That is exactly the workload the GPU was built for.
Mental model: factory floor vs. custom workshop#
Imagine a CPU as a custom workshop with three expert craftspeople. Each person can handle almost any job: complicated, unique, requiring judgment. They work fast on individual items but can only handle a few at a time.
Imagine a GPU as a factory floor with ten thousand assembly workers. Each worker performs one specific task: they take a component, apply a single operation, and pass it on. Individually, each worker is slower than the expert craftsperson. But when the factory runs at full capacity, the collective throughput dwarfs the workshop.
AI workloads are factory-floor work: compute the dot product of these two 4096-dimensional vectors, apply this activation function to these 32 million values, multiply these two 8192×8192 matrices. The same operation applies to millions of data points with no branching, no decisions, and perfect predictability.
| CPU | GPU |
|---|---|
| 4–64 powerful cores | Thousands of simpler cores |
| Large L3 cache (tens of MB) | Large HBM memory (tens of GB) |
| Low latency per thread | High throughput across threads |
| Suited for sequential, branchy code | Suited for parallel, predictable code |
| General-purpose | Optimized for throughput-dominated workloads |
How CUDA organizes GPU work#
NVIDIA's CUDA programming model is the software layer that maps computational work onto GPU hardware. The CUDA Programming Guide describes three key abstractions: thread groups, shared memories, and barrier synchronization.
In CUDA, work is organized into a hierarchy:
- Thread: The smallest unit of work. Executes one instance of the kernel function.
- Warp: A group of 32 threads that execute in lockstep on the GPU hardware. The CUDA Programming Guide describes this as the SIMT (Single Instruction Multiple Threads) model: all 32 threads execute the same instruction simultaneously, each on different data.
- Block: A group of threads (up to 1024) that can communicate through shared memory and synchronize with each other.
- Grid: The collection of all blocks launched for a given kernel invocation.
The warp is the key hardware concept. When you launch 8,192 threads to compute a matrix row in parallel, those threads are scheduled in groups of 32 warps. Each warp executes in one clock cycle across 32 CUDA cores. If threads within a warp diverge, meaning they take different branches due to conditional logic, the hardware must execute both paths sequentially with some threads masked off, halving throughput. This is called warp divergence, and it is one of the main reasons GPU code is written to avoid branching.
# PyTorch example: this operation launches thousands of CUDA threads
import torch
a = torch.randn(8192, 8192, device='cuda')
b = torch.randn(8192, 8192, device='cuda')
# This matrix multiply maps onto thousands of parallel thread groups
c = torch.matmul(a, b)
When PyTorch calls torch.matmul, it dispatches a CUDA kernel that organizes the work into blocks and grids, maps it onto the GPU's SMs, and schedules warps across the available CUDA cores. The programmer does not manage threads manually for standard operations. Frameworks like PyTorch, JAX, and TensorFlow handle kernel dispatch.
Tensor Cores: dedicated matrix math hardware#
CUDA cores perform general floating-point arithmetic: one multiply-add per cycle per core. For matrix multiplication (the core operation of transformer attention, fully connected layers, and convolutions) this is not fast enough.
NVIDIA introduced Tensor Cores starting with the Volta architecture. Tensor Cores are specialized matrix multiply-accumulate units that operate on small matrix tiles (such as 16×16 or 8×8 blocks) in a single instruction, producing far higher throughput than general CUDA cores for this specific operation.
The H100 GPU contains 528 fourth-generation Tensor Cores, four per SM across 132 SMs. NVIDIA's documentation describes them as handling the MMA (Matrix Multiply-Accumulate) operations that are fundamental to neural network computation. For the FP8 data type, the H100's Tensor Cores reach 2,000 TFLOPS of peak throughput (4,000 with sparsity enabled), compared to roughly 333 TFLOPS for CUDA cores in FP32. That is approximately a 6x advantage on equivalent operations.
Tensor Cores operate on mixed precision: the inputs can be FP16 or FP8, while the accumulator remains FP32. This is called mixed-precision training. It preserves numerical accuracy in the accumulator while allowing the input data to use half the memory bandwidth, effectively doubling throughput for memory-bound workloads.
Tensor Cores are the reason matrix size matters
Tensor Cores operate on matrix tiles of specific dimensions. When matrix dimensions are not multiples of 8 or 16 (depending on the precision), the operation falls back to CUDA cores or pads the computation, wasting cycles. AI frameworks automatically handle alignment for standard layer sizes, but custom layers with odd dimensions can silently underuse Tensor Core throughput.
Memory bandwidth: the real bottleneck#
Raw compute (FLOPS) tells you how many floating-point operations per second the hardware can perform in theory. Memory bandwidth tells you how fast data can move from GPU memory to the compute units. For most AI workloads, memory bandwidth is the binding constraint, not compute.
The intuition is simple: a Tensor Core that can perform a matrix multiply in one cycle needs the matrix data to arrive in time. If memory cannot deliver the values fast enough, the compute units idle while waiting. This is called being memory-bound. The data path below shows where that bottleneck lives, from host memory all the way down to the compute units.
The H100 SXM5 uses HBM3 (High Bandwidth Memory, third generation) providing approximately 3 TB/sec of memory bandwidth, according to NVIDIA's H100 product page and the Hopper architecture deep dive. HBM stacks the memory dies on top of the GPU die and connects them with thousands of wires, which is why GPU memory bandwidth is an order of magnitude higher than what a CPU can achieve through DDR5.
The compute-to-memory-bandwidth ratio determines whether a workload is compute-bound or memory-bound. Large batch sizes spread the cost of loading weights across many tokens, making training typically compute-bound. Small batch inference (serving one request at a time) must reload weights for each forward pass without amortizing the cost, making it memory-bound.
| Workload | Typical bottleneck | Why |
|---|---|---|
| Large-batch training | Compute (Tensor Cores) | Weights loaded once, reused many times |
| Small-batch inference | Memory bandwidth | Weights loaded for each forward pass |
| Token generation (LLM) | Memory bandwidth | Autoregressive: one token at a time |
| Large-batch inference | Compute | Enough parallelism to keep Tensor Cores busy |
This is why inference optimization techniques like quantization (reducing weight precision from FP16 to INT8 or INT4) and continuous batching (grouping concurrent requests) directly target the memory-bound nature of inference. Smaller weights move faster; more concurrent requests amortize the load cost.
How a forward pass runs on a GPU#
When a transformer model processes a batch of tokens, here is the rough execution sequence from software to silicon:
Framework dispatch
PyTorch or JAX decomposes the forward pass into a sequence of operations: embedding lookup, attention QKV projection, softmax, FFN layers, and so on.
CUDA kernel launch
For each operation, the framework launches one or more CUDA kernels, which are compiled GPU programs. A fused kernel (common in optimized inference engines like vLLM or TensorRT-LLM) combines multiple operations into one kernel to reduce memory round-trips.
SM scheduling
The GPU's work scheduler assigns thread blocks to available SMs. Each SM processes its block independently, using local registers and a small shared memory scratchpad.
Tensor Core execution
Inside an SM, the matrix multiply operation is dispatched to the four Tensor Cores. The CUDA cores handle surrounding work: loading data into registers, applying activation functions (like GELU or SiLU), writing results back to HBM.
Memory access
Values not in registers or shared memory must be fetched from HBM. L1 cache (per SM) and L2 cache (shared across the GPU) reduce HBM reads for reused data. Cache misses stall the compute pipeline.
Kernel completion
Results are written to HBM, and the next kernel begins. The GPU handles thousands of such kernel-level pipeline stages across a forward pass.
Key components and their responsibilities#
| Component | Role |
|---|---|
| CUDA core | General floating-point arithmetic (FP32, FP64, INT) |
| Tensor Core | High-throughput matrix multiply-accumulate (MMA) |
| Streaming Multiprocessor (SM) | Contains CUDA cores, Tensor Cores, shared memory, schedulers |
| HBM (High Bandwidth Memory) | Main GPU memory: stores weights, activations, KV cache |
| L1 / L2 Cache | Reduce HBM latency for reused values |
| NVLink | High-bandwidth GPU-to-GPU interconnect for multi-GPU jobs |
| CUDA runtime | Software layer mapping operations to hardware |
Where GPU workloads go wrong#
Small batch sizes at inference. A single-request inference load barely touches Tensor Core throughput. The GPU spends most of its time loading weights rather than computing. Continuous batching and request queuing systems like vLLM address this by grouping concurrent requests.
Warp divergence. Custom operators with conditional branches produce divergent warps and effectively serialize execution within a warp. Standard layers in well-maintained frameworks avoid this; custom CUDA kernels can introduce it accidentally.
Memory fragmentation in KV cache. Autoregressive LLM generation grows the key-value cache dynamically. Naive allocation fragments GPU memory and limits maximum concurrent requests. Paged attention (used in vLLM) addresses this by managing the KV cache in fixed-size pages.
PCIe data transfer bottleneck. Moving large tensors between CPU RAM and GPU HBM over PCIe is slow compared to HBM bandwidth. Workloads that repeatedly move data across this boundary instead of keeping it resident on the GPU pay a large transfer penalty.
Underutilized precision. Using FP32 throughout when FP16 or BF16 is sufficient doubles memory usage and halves Tensor Core throughput. Mixed-precision training with automatic loss scaling is now standard practice in PyTorch via torch.cuda.amp.
How to reason about GPU performance#
Three ratios are useful for reasoning about whether a workload is GPU-friendly:
Arithmetic intensity: the ratio of FLOPs to bytes of memory loaded. High arithmetic intensity (compute-bound) means Tensor Cores are the constraint. Low arithmetic intensity (memory-bound) means HBM bandwidth is the constraint. Matrix multiplications have high arithmetic intensity at large batch sizes; token generation has low arithmetic intensity.
GPU utilization (MFU, Model FLOP Utilization): what fraction of peak theoretical FLOPS the model actually achieves end to end. Well-optimized transformer training achieves 30 to 60% MFU. Poorly batched inference can fall below 5%. This metric surfaces whether the bottleneck is compute, memory, or operator overhead.
Memory occupancy: how much of the GPU's HBM is occupied by the model weights, activations, and KV cache. A 70B parameter model in FP16 requires roughly 140 GB just for weights, which exceeds a single H100's 80 GB and requires model parallelism across multiple GPUs.
# Check GPU memory usage in PyTorch
import torch
print(f"Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
print(f"Reserved: {torch.cuda.memory_reserved() / 1e9:.2f} GB")
print(f"Max allocated: {torch.cuda.max_memory_allocated() / 1e9:.2f} GB")
# Profile a model forward pass
with torch.profiler.profile(
activities=[torch.profiler.ProfilerActivity.CUDA],
record_shapes=True
) as prof:
output = model(input_tensor)
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
Practical examples#
Training a transformer on a GPU cluster#
Distributed training uses multiple GPUs connected by NVLink or InfiniBand. Each GPU holds a shard of the model (tensor parallelism) or a different batch partition (data parallelism). NVLink on the H100 supports up to 57.6 TB/sec of aggregate GPU-to-GPU bandwidth across 256 GPUs using NVLink Switch, according to the Hopper architecture documentation. Gradient synchronization between GPUs is the main multi-GPU communication cost.
LLM inference with continuous batching#
A language model serving system like vLLM accepts multiple concurrent user requests and groups them into a single GPU forward pass where possible. This amortizes the weight-loading cost across requests, moving the workload from memory-bound toward compute-bound. The inference engine serves the initial prompt phase (prefill) and the token generation phase (decode) differently because their arithmetic intensity differs significantly. See how inference engines serve models for a detailed look at this architecture.
Quantization for memory reduction#
INT8 and INT4 quantization reduce model weight size by 2x or 4x relative to FP16. This allows a larger model to fit into GPU HBM, or the same model to run on cheaper hardware. Tensor Cores on H100 support INT8 natively, maintaining high throughput while reducing memory bandwidth demand. The tradeoff is a small accuracy loss, which must be evaluated per model and task.
Key Takeaways#
GPUs run AI workloads efficiently because neural networks are essentially large, regular tensor operations: the kind of parallel, predictable math that the GPU's thousands of cores and dedicated Tensor Core units were built for. CPUs are faster per thread. GPUs win on aggregate throughput across thousands of simultaneous operations.
Tensor Cores provide the highest throughput for matrix multiplications, the dominant operation in transformer attention and feedforward layers. Memory bandwidth, not compute, is the binding constraint for most inference workloads. Techniques like quantization, batching, and fused kernels are all strategies for reducing how much data must move between HBM and compute units.
The H100 with HBM3 delivers approximately 3 TB/sec of memory bandwidth and 2,000 TFLOPS of FP8 Tensor Core throughput. Understanding the arithmetic intensity of your specific workload tells you which of those two numbers is the one to optimize around.
FAQ#
Why can't I just use more CPUs instead of a GPU?
CPUs are designed for latency-sensitive sequential work, not throughput-sensitive parallel work. A CPU's cores are powerful and handle complex branching code efficiently, but even a 64-core server CPU cannot match the parallelism of a GPU's thousands of simpler cores for matrix-heavy workloads. The arithmetic structure of matrix multiplication specifically maps onto GPU Tensor Cores in a way that has no CPU equivalent.
What is the difference between CUDA cores and Tensor Cores?
CUDA cores are general-purpose floating-point arithmetic units that perform one multiply-add per cycle on individual numbers. Tensor Cores are specialized hardware for matrix multiply-accumulate operations. They operate on small matrix tiles in one instruction and deliver much higher throughput for that specific operation. Most modern AI workloads should be Tensor Core-bound at large batch sizes, and CUDA cores handle everything else.
Why does batch size affect GPU performance so much?
At large batch sizes, model weights are loaded from HBM once and reused across many inputs, making the workload compute-bound (Tensor Core-limited). At small batch sizes, weights must be loaded for each pass without amortization, making the workload memory-bound. This is why inference throughput increases roughly linearly with batch size until compute saturation is reached.
What does "memory bandwidth" mean in GPU context?
Memory bandwidth is the rate at which data can flow between GPU HBM (the main memory chip on the GPU) and the compute units (SMs, Tensor Cores). It is measured in TB/sec. When a computation needs data faster than memory can deliver it, the compute units idle and throughput drops. The H100 SXM5 delivers approximately 3 TB/sec of HBM3 bandwidth, which is why it is preferred for memory-bandwidth-sensitive inference.
What is mixed-precision training?
Mixed-precision training uses FP16 or BF16 for most computations (activations, weight updates) while keeping FP32 for the loss accumulator where numerical precision matters most. This halves the memory footprint and memory bandwidth demand relative to FP32-only training, while allowing Tensor Cores to run at higher throughput. PyTorch supports this with torch.cuda.amp.autocast() and automatic gradient scaling to handle underflow.
