Build From Scratch
A running max and a running sum replace attention's score matrix
A from-scratch NumPy version of FlashAttention's online-softmax core reproduces exact attention to float32 precision, without ever holding the full score matrix that costs the naive version 256 mebibytes by sequence length 8192.
Two rotated vectors remember only the distance between them
Rotary position embeddings turn a token's index into a rotation angle, the mechanism behind most open-weight models' sense of where each token sits and behind the single-constant edits that stretch a model past its training length.
A rejection rule keeps speculative decoding's output exact
Two hand-built character models run two hundred thousand times confirm the accept-or-resample test reproduces the target's exact probabilities, and show how many draft tokens survive it as the draft improves.
July 2026Byte-pair encoding hides the letters inside every common word a language model reads
A merge-and-count tokenizer trained from scratch on 617 bytes of text reproduces the same blind spot as GPT-2's real tokenizer, and turns the integer 2249 into four digits while leaving 2250 in three.