Post-Training 101: From Base Model to Assistant
genAI Fundamentals Part 5: instruction tuning (SFT), RLHF vs DPO, and the RLVR/GRPO reasoning era — how a base model learns to follow instructions and which answers people actually prefer.
Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:
This is part of the genAI Fundamentals series. Each post picks one building block of modern LLMs and explains it from first principles, with code.
Here's a fact that is surprising to almost everyone new to LLMs: the giant model that comes out of pretraining isn't a chatbot at all. As we saw at the end of Pretraining 101, you get a base model — not an assistant. After trillions of tokens and millions of dollars, what comes out the other end can complete text beautifully and won't, on its own, actually answer you.
Ask a raw base model "What is the capital of France?" and it might not say "Paris." A very plausible continuation is another question — "What is the capital of Germany? What is the capital of Italy?" — because on the open web, that line usually appears in a list of quiz questions, not in a helpful reply. The base model isn't broken. It's doing exactly what it was trained to do: continue the pattern.
So the knowledge is in there. Paris is in there, along with grammar, code, and a startling amount of the internet. What's missing is the behavior of being a helpful assistant — answering instead of continuing, following instructions, staying on task, knowing when to refuse. That behavior gets installed in a second phase, and it's a much shorter and cheaper one. It's called post-training, and it's where ChatGPT, Claude, and DeepSeek-R1 actually get made.
By the end of this post, you'll understand the whole modern playbook: instruction tuning (SFT), the reward-model route (RLHF), the shortcut that took over (DPO), its unpaired cousin (KTO), and the new reasoning layer (RLVR with GRPO) that powers models like DeepSeek-R1 — with the actual math worked through on real numbers, plus a simple guide to which one to reach for and when.
This is where the whole series culminates. How does everything work all together to create an assistant you can talk with?
The mental model I'd start with: a base model is a brilliant, impossibly well-read intern who has read every book in the library but has never once been taught how to answer a question or hold a conversation. Pretraining filled their head. Post-training teaches them manners.
Let's dive in.
1. The Two Jobs of Post-Training
Post-training has two distinct jobs.
The first job is behavior: teach the model to act like an assistant — to answer the question instead of extending it, follow the instruction it's given, speak in the chat format, and stop when it's actually done. This is about form.
The second job is preference: teach the model which answer is better. For almost any real prompt, there are many valid responses, and they're not equally good — one is more helpful, more honest, less likely to make something up, better-toned. Pretraining never taught the model to rank; it only taught it to predict what comes next. This is about taste.
Those two jobs map onto the modern post-training stack:
Instruction tuning (SFT) handles behavior.
Preference tuning (RLHF or DPO) handles taste.
An optional RLVR layer — Reinforcement Learning from Verifiable Rewards (GRPO and friends) — adds genuine reasoning on tasks where you can check the answer.
Two things are worth holding onto before we open each box. First, all of this is cheap compared to pretraining. Where pretraining burned months of compute across thousands of GPUs, post-training runs on human-curated data that's tiny by comparison — thousands to low millions of examples, not trillions of tokens. The base model did the expensive work of learning the world; post-training just shapes how it behaves.
Second, there's no single recipe anymore. A few years ago you could say "pretrain, then RLHF" and be roughly right. In 2026 that's about as complete as describing a car as "the thing with an engine." Every major model — DeepSeek-R1, the Llama and Qwen families, the frontier closed models — uses a different mix of these layers. Post-training has become a modular stack, and the interesting work is in how you combine the pieces. So let's understand the pieces.
