transformers-from-scratch
collection · 2 published
transformers-from-scratch
Tokens and Embeddings: Turning Text into Vectors
Lesson one of building a GPT from scratch: how a string of text becomes the grid of numbers a transformer can actually read.
transformers-from-scratch
Self-Attention by Hand: Queries, Keys, and Values
Lesson two builds a single attention head from a plain average of past tokens, deriving queries, keys, values, the 1/sqrt(d_k) scaling, and the causal mask in short, shape-annotated PyTorch.