Caching
The latency of access to RAM, particularly on a remote NUMA node, can be quite high. So we need ways to bring the data closer to the CPU, which is where cache comes into play.
Caches take advantage of the fact that programs usually operate on data that is in the same memory region. This spatial and temporal locality of data access presents an opportunity for the CPU to hide the latency of access to RAM.
A typical cache hierarchy on a modern multi-core CPU looks like this. L1 is split into a data cache (L1D) and an instruction cache (L1I). L2 and L3 mix instructions and data freely, as does main memory. L2 is private to a core; L3 is the last-level cache (LLC), shared across cores on the socket.
Typical capacities vary by vendor. Line size is the unit of transfer between levels: the number mechanical sympathy has to respect.
| Vendor | Cache line size | Typical L1 size | Typical L2 size | Typical L3 / SLC size |
|---|---|---|---|---|
| Intel (Arrow Lake / Core Ultra 200) | 64 B | P-core: ~64 KB I + 192 KB D E-core: 64 KB I + 32 KB D | P-core: 3 MB private E-cluster: 4 MB shared | Up to 36 MB shared |
| AMD (Zen 5) | 64 B | 32 KB I + 48 KB D per core | 1 MB private per core | 32 MB per CCD (8 cores) Up to 96–128 MB with 3D V-Cache |
| Apple Silicon (M4 family) | 128 B | P-core: 192 KB I + 128 KB D E-core: 128 KB I + 64 KB D | P-cluster: 16 MB shared E-cluster: 4 MB shared | 8 MB (base) Up to 24–48 MB (Pro/Max) 96 MB (Ultra) |
| ARM Cortex / Neoverse (standard) | 64 B (most modern) | 32–64 KB I + 32–64 KB D (configurable) | 256 KB – 2 MB private (configurable) | Optional shared, often 8–32+ MB |
| Qualcomm (Oryon / Snapdragon X) | 64 B | 192 KB I + 96 KB D | 12 MB shared per 4-core cluster | 6 MB SLC |
| IBM POWER | 128 B | Typically 32–64 KB range | Hundreds of KB to 1+ MB | Tens to hundreds of MB |
Things to talk about caching:
- cache warmup
- the architecture of the cache (sizes etc..), mechanical sympathy
- talk about cache misses, cache storm, etc.
The L3 cache functions more like a global mempool, while L1 and L2 are dedicated to each core. The L3 cache functions as a victim cache, where any evicted cache line from L1 and L2 will be sent to L3 before being sent to RAM. L3 is typically in the refill oath of a CPU cache, storing all the blocks evicted from that level.
A cache is organized as:
Sets: groups of storage locations Ways: the number of cache lines inside each set
Associativity = number of ways per set
Low associativity → higher chance of conflict misses (two pieces of data want the same set and keep kicking each other out). Higher associativity → fewer conflict misses, but the cache has to compare more tags and the circuitry becomes larger and more power-hungry.
| Associativity | Hit latency | Conflict misses | Hardware cost | Typical use |
|---|---|---|---|---|
| Low (4–8 way) | Faster | Higher | Lower | L1 |
| Medium (8–16 way) | Medium | Medium | Medium | L2 |
| High (12–16+ way) | Slower | Lower | Higher | L3 / SLC |
| Aspect | High associativity | Low associativity |
|---|---|---|
| Hit latency | Higher (slower) | Lower (faster) |
| Conflict misses | Fewer | More |
| Hit rate | Better | Worse (in conflict cases) |
| Hardware cost / power | Higher | Lower |
| Typical use | L2 / L3 | L1 |
Hit latency
This is how fast the cache can answer when the data is already inside it.
With low associativity (e.g. 4-way), the hardware only has to compare the address against 4 tags. Simple, short critical path: faster, lower latency.
With high associativity (e.g. 16-way), the hardware has to compare against 16 tags and then select the correct one. More complex logic: slower, higher latency.
So low associativity wins on the pure speed of a hit.
Hit rate
This is the percentage of accesses that find the data in the cache. The main problem with low associativity is conflict misses.
Imagine a 4-way cache and a program that repeatedly uses 5 different memory locations that all map to the same set. Only 4 of them can stay in that set at the same time. The 5th one will constantly kick one of the others out. Even though the cache still has plenty of free space in other sets, these 5 lines keep fighting and causing misses.
With higher associativity (say 16-way), all 5 lines can live in the same set: no conflict misses, higher hit rate.
NUMA
In NUMA architectures, there are memory controllers, with memory being phisically connected to a socket. Each CPU socket has its mempool phisically connected to it.
Cache coherence protocol
It’s possible that the CPU will have cacehd an old version of data ontained in another CPU’s memory. To solve these cases, we need cache coherence protocols. These protocols enable a CPU to determine whether it uniquely owns, shares, or has a locally modified version of a particular memory region, and share information with other CPUs if they try to access the exact memory location.
HFT applications are written to rarely touch the coherence protocols, because the cost of synching the ownership is high.