Backprop to Frontier

Skip this unless matrix calculus feels rusty. You need enough linear algebra to read a shape mismatch and enough calculus to trust the chain rule. Nothing more.

• 3Blue1Brown: Essence of Linear Algebra15 short videos. Watch at 1.25x.4h
• 3Blue1Brown: Essence of CalculusChapters 1–4 are the load-bearing ones.3h
• MML book, Ch. 2, 5, 6Linear algebra, vector calculus, probability. Free PDF.10h

The single best bridge from classical ML to modern AI. You write backprop, then a tokenizer, then a GPT, in bare Python and PyTorch. Retype every line. Watching this does nothing.

• micrograd, backprop from scratchAn autograd engine in ~100 lines. repo6h
• makemore, parts 1–3Bigrams → MLP → activations, gradients, BatchNorm.12h
• makemore, parts 4–5Manual backprop through the whole net, then WaveNet.8h
• Let's build GPT: from scratch, in codeThe centerpiece. Self-attention, blocks, the full transformer.8h
• Let's build the GPT TokenizerBPE. Explains half the weird LLM failure modes you've seen.4h
• Train nanoGPT yourself, end to endRent one GPU. Reproduce a small run. repo10h

The opposite pedagogy from Stage 1: working model in lesson one, theory later. Doing both is the point: you get intuition for what actually moves a metric, not just what the math says.

• Part 1, lessons 1–4Vision, deployment, SGD, NLP with transformers.16h
• Part 1, lessons 5–9Tabular, collaborative filtering, convnets, ethics.20h
• Ship one model of your ownYour data, deployed publicly. This is the item that gets skipped. Don't.12h
• Part 2: Stable Diffusion from scratchRebuilds a diffusion model from the paper up.30h

The nitty-gritty you asked for. Lectures and assignments are public. Students build a tokenizer, a transformer, a training loop, kernels, distributed training and alignment, with nothing imported that could be written. Budget 15h/week and expect it to hurt.

• Assignment 1: BasicsBPE tokenizer, transformer, optimizer, training loop. From zero.35h
• Assignment 2: SystemsTriton kernels, FlashAttention, distributed data parallel.30h
• Assignment 3: Scaling lawsFit a scaling law, then spend a compute budget with it.15h
• Assignment 4: DataFiltering, dedup, quality classifiers. Where most real gains live.20h
• Assignment 5: Alignment & RLSFT, then policy-gradient post-training.25h

By now you can read any of these at speed. Choose by need, not completeness. Reinforcement learning is the one most worth doing even if it isn't obviously relevant, because post-training runs on it.

• OpenAI: Spinning Up in Deep RLBest free RL on-ramp, with implementations. spinningup.openai.com20h
• Stanford CS224N: NLP with Deep LearningLectures on YouTube. course site25h
• Stanford CS231N: Computer VisionStill the reference convnet course. cs231n.stanford.edu25h
• MIT 6.S191: Intro to Deep LearningRe-shot every January, so it stays current. introtodeeplearning.com12h

Courses lag the field by a year or more. This stage never finishes: it's the habit that keeps the rest from going stale. One paper a week, read properly, beats fifty skimmed.

• Hugging Face LLM CourseTransformers, fine-tuning, datasets. huggingface.co/learn20h
• Hugging Face Agents CourseTool use, planning loops, agent evaluation.15h
• Work through a provider cookbookAnthropic or OpenAI . The practical layer courses skip.10h
• Build an eval suite for something you shippedThe skill that separates people who ship LLM features from people who demo them.15h
• Read the eight foundational papersAttention Is All You Need, GPT-2/3, InstructGPT, Chinchilla, LoRA, DPO, and one recent frontier model card.20h
• Establish the weekly paper habitSame slot every week for four weeks straight. Then it sticks.4wk