TENSOR ATLAS / AN INTERACTIVE COMPUTE STUDY
Inside the inference.
From a token to a tensor. From silicon to a sentence.
Follow the operations of an LLM through an NVIDIA-style GPU.
01 / THE MACHINE
Silicon, in motion.
Loading the hardware view…
FOLLOW ONE OUTPUT / ALL CELLS UPDATE TOGETHER
How this maps to NVIDIA Tensor Core execution
Real kernels stage matrix tiles in SM-local storage and issue cooperative matrix instructions. CUDA’s WMMA interface distributes opaque matrix fragments across a warp’s 32 threads; Hopper also supports warp-group asynchronous MMA with different operand placement. A tile instruction performs many products and accumulates their sums. The visible 4 × 4 cells and k steps explain this arithmetic; they do not depict physical multipliers, fragment-to-thread assignments, or instruction timing.
Inputs commonly use lower precision while the accumulator may use higher precision, depending on the instruction. This example uses JavaScript numbers and does not simulate FP16/BF16/FP8 rounding. Completed accumulators can feed later operations or be written back to memory.
NVIDIA / Warp matrix functions ↗NVIDIA / Hopper tuning guide ↗03 / THE MATH 01
Token embeddings
LIVE TENSORS / DETERMINISTIC TOY MODEL
See the numbers move.
CONTEXT / FOUR PARALLEL HEADS
What does “on” look at?
Each head uses its own learned projections. Select a head to inspect it above.
These are attention weights from untrained toy parameters, not explanations of a trained model’s reasoning.
VOCABULARY / A DIFFERENT SOFTMAX
One more token.
Synthetic eight-token vocabulary. Bars show the final distribution; this explorer does not run a trained LLM.
READING THE ATLAS
Many operations.
The same silicon.
Attention heads and model layers are software structures. GPU kernels schedule their work across reusable streaming multiprocessors. A head does not own a dedicated chip region.
Conceptual steps, real arithmetic.
The animation separates operations for learning. Real kernels overlap data transfer and compute, and may fuse several steps. FlashAttention computes attention in tiles and can avoid storing the full score matrix in HBM. The full matrix here makes the math inspectable.
Prefill vs. decode.
Prefill computes four query rows. Decode displays the last token’s query against three cached keys and values plus its new key and value. The toy recomputes a reference block to show identical values; it does not benchmark a KV-cache implementation.
A schematic, not a performance monitor.
The 24 visible SM tiles and six HBM stacks are representative geometry, not an exact H100 floorplan or enabled-unit count. Brightness and particle speeds illustrate dataflow. They do not report utilization, throughput, or hardware measurements. Norm scale is one; weights are deterministic and untrained; dropout and biases are omitted.