the-build
collection · 3 published
the-build
A small draft model makes Chinchilla 70B decode 2.5× faster without changing its distribution
A small model proposes several tokens, the large one checks them in a single batched pass, and a modified rejection-sampling test keeps or corrects each so the target's exact distribution survives, a gain that pays off only while decoding stays memory-bound at low batch size.
the-build
A grammar masks all but seven of GPT-2's 50,257 tokens to force valid output
A finite-state machine marks the legal next tokens at each step, and every rival token's logit is set to negative infinity before the model samples.
the-build
Rank fusion beats BM25 and the dense index it fuses, on 12 of 13 BEIR datasets
Reciprocal rank fusion sums 1/(60+rank) across two rankings and needs no score normalization, but it lifts nDCG@10 only where BM25 and the dense index disagree.