Stories by gpjt

Writing an LLM from scratch, part 32g – Interventions: weight tying

Writing an LLM from scratch, part 32f – Interventions: weight decay

Writing an LLM from scratch, part 32e – Interventions: the learning rate

Writing an LLM from scratch, part 32d – Interventions: adding attention bias

Writing an LLM from scratch, part 32c – Interventions: removing dropout

Writing an LLM from scratch, part 32B – Interventions: gradient clipping

Writing an LLM from scratch, part 32a – Interventions: training a baseline model

Getting a Custom PyTorch LLM onto the Hugging Face Hub

Writing an LLM from scratch, part 31 – the models are now on Hugging Face

Writing an LLM from scratch, part 30 – digging into the LLM-as-a-judge results

LLM from scratch, part 29 – using DDP to train a base model in the cloud

LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

Writing an LLM from scratch, part 27 – what's left, and what's next?

Writing an LLM from scratch, part 26 – evaluating the fine-tuned model

Writing an LLM from scratch, part 25 – instruction fine-tuning

Writing an LLM from scratch, part 24 – the transcript hack

Retro Language Models: Rebuilding Karpathy's RNN in PyTorch

Writing an LLM from scratch, part 23 – fine-tuning for classification

Writing an LLM from scratch, part 22 – training our LLM

Revisiting Karpathy's 'Unreasonable Effectiveness of Recurrent Neural Networks'

1 2