Stories by smaddrellmander

The annotated PyTorch training loop

Mae vs. MSE: more than just the mean vs. median debate

DiffusionGemma: Discrete diffusion in a large language model

Heaven knows I'm perplexed now

Reading MAI's efficiency gain. How to pick architectures like serious people

MAI-Thinking-1: Building a Hill-Climbing Machine [pdf]

Are contrastive losses just cross entropy all along?

Every token, everywhere, all at once

The cut in the Mixture of Experts compute graph

DeepSeek V4 from the Inside

Softmax, can you derive the Jacobian? And should you care?

Gemma 4 is not your standard transformer