Blog

Notes from the lab.

Long-form writeups on language models, training systems, architecture choices, and what broke along the way.

Building a 25M Parameter GPT from Scratch

Architecture choices, tokenizer mistakes, training issues, and lessons from training LiteGPT on FineWeb and TinyStories.