GPU memory hierarchies & kernel profiling

A toy kernel: an N×N matrix multiply. Slide N and watch two clocks race — one for moving the matrices through HBM (High Bandwidth Memory, the GPU's main memory), one for the arithmetic itself. Whichever is longer is the kernel's actual runtime; the other one is free, because it overlaps.

An N×N matmul reads two N×N operands and writes one N×N result — 3 arrays of $N^2$ numbers, 4 bytes each — and performs the standard $2N^3$ floating-point operations (one multiply and one add per output element, summed over N):

• $N$ — the matrix size: an N×N by N×N matmul.
• $\text{bytes}(N)$ — total bytes moved through memory for this kernel.
• $\text{flops}(N)$ — total floating-point operations this kernel performs.

Each quantity turns into a time by dividing by the rate at which it can happen — bandwidth for bytes, throughput for FLOPs — and the kernel's actual runtime is whichever time is larger, since a well-written kernel overlaps the two:

• $B$ — HBM bandwidth, in bytes per unit time.
• $R$ — compute throughput, in FLOPs per unit time.
• $T(N)$ — the kernel's actual runtime, the roofline model's prediction.

1. Bytes grow as N², FLOPs grow as N³

   Every extra unit of N adds proportionally less memory traffic than compute. That single fact is the whole story: it guarantees the two curves cross exactly once.

2. Below the crossover: memory-bound

   Small N means little arithmetic relative to how much data has to move — the GPU sits idle waiting on HBM. This is where flash-attention-style tiling and kernel fusion earn their keep.

3. Above the crossover: compute-bound

   Large N means the arithmetic dominates — moving the data was cheap by comparison. Here, the number that matters is how many FLOPs/sec the hardware can sustain, not how fast memory is.

The same two curves, now on a fixed scale so their crossing point is visible directly: transfer time climbing gently as $N^2$, compute time climbing steeply as $N^3$, meeting at exactly one N along the way.

1. At N = 8: bytes and FLOPs

   • $\text{bytes}(8) = 3 \cdot 4 \cdot 64 = 768$
   • $\text{flops}(8) = 2 \cdot 512 = 1024$
2. Two times, exactly equal

   With $B = 3$ and $R = 4$:

   • $T_{\text{transfer}}(8) = 768 / 3 = 256$
   • $T_{\text{compute}}(8) = 1024 / 4 = 256$

   Same number — N = 8 is exactly the crossover for these constants.

3. One step either side

   • At $N=3$: $\text{bytes}(3)=3\cdot4\cdot9=108$, $T_{\text{transfer}}=108/3=36$. $\text{flops}(3)=2\cdot27=54$, $T_{\text{compute}}=54/4=13.5$ — memory-bound, transfer dominates.
   • At $N=10$: $\text{bytes}(10)=3\cdot4\cdot100=1200$, $T_{\text{transfer}}=1200/3=400$. $\text{flops}(10)=2\cdot1000=2000$, $T_{\text{compute}}=2000/4=500$ — compute-bound, the arithmetic now dominates instead.