Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

This is part of the genAI Fundamentals series. Each post picks one building block of modern LLMs and explains it from first principles, with code.

Here's a fact that is surprising to almost everyone new to LLMs: the giant model that comes out of pretraining isn't a chatbot at all. As we saw at the end of Pretraining 101, you get a base model — not an assistant. After trillions of tokens and millions of dollars, what comes out the other end can complete text beautifully and won't, on its own, actually answer you.

Ask a raw base model "What is the capital of France?" and it might not say "Paris." A very plausible continuation is another question — "What is the capital of Germany? What is the capital of Italy?" — because on the open web, that line usually appears in a list of quiz questions, not in a helpful reply. The base model isn't broken. It's doing exactly what it was trained to do: continue the pattern.

So the knowledge is in there. Paris is in there, along with grammar, code, and a startling amount of the internet. What's missing is the behavior of being a helpful assistant — answering instead of continuing, following instructions, staying on task, knowing when to refuse. That behavior gets installed in a second phase, and it's a much shorter and cheaper one. It's called post-training, and it's where ChatGPT, Claude, and DeepSeek-R1 actually get made.

By the end of this post, you'll understand the whole modern playbook: instruction tuning (SFT), the reward-model route (RLHF), the shortcut that took over (DPO), its unpaired cousin (KTO), and the new reasoning layer (RLVR with GRPO) that powers models like DeepSeek-R1 — with the actual math worked through on real numbers, plus a simple guide to which one to reach for and when.

This is where the whole series culminates. How does everything work all together to create an assistant you can talk with?

The mental model I'd start with: a base model is a brilliant, impossibly well-read intern who has read every book in the library but has never once been taught how to answer a question or hold a conversation. Pretraining filled their head. Post-training teaches them manners.

Let's dive in.

1. The Two Jobs of Post-Training

Post-training has two distinct jobs.

The first job is behavior: teach the model to act like an assistant — to answer the question instead of extending it, follow the instruction it's given, speak in the chat format, and stop when it's actually done. This is about form.

The second job is preference: teach the model which answer is better. For almost any real prompt, there are many valid responses, and they're not equally good — one is more helpful, more honest, less likely to make something up, better-toned. Pretraining never taught the model to rank; it only taught it to predict what comes next. This is about taste.

Those two jobs map onto the modern post-training stack:

• Instruction tuning (SFT) handles behavior.

• Preference tuning (RLHF or DPO) handles taste.

• An optional RLVR layer — Reinforcement Learning from Verifiable Rewards (GRPO and friends) — adds genuine reasoning on tasks where you can check the answer.

Two things are worth holding onto before we open each box. First, all of this is cheap compared to pretraining. Where pretraining burned months of compute across thousands of GPUs, post-training runs on human-curated data that's tiny by comparison — thousands to low millions of examples, not trillions of tokens. The base model did the expensive work of learning the world; post-training just shapes how it behaves.

Second, there's no single recipe anymore. A few years ago you could say "pretrain, then RLHF" and be roughly right. In 2026 that's about as complete as describing a car as "the thing with an engine." Every major model — DeepSeek-R1, the Llama and Qwen families, the frontier closed models — uses a different mix of these layers. Post-training has become a modular stack, and the interesting work is in how you combine the pieces. So let's understand the pieces.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

This is part of the genAI Fundamentals series. Each post picks one building block of modern LLMs and explains it from first principles, with code.

In the last post, we ended on a tidy one-liner: we said an LLM is just a function: token IDs in, next-token probabilities out. It’s a clean, satisfying way to put it — but I quietly skipped the single most important word in that sentence.

Trained.

When you first build that function, every weight in it is random — the embeddings, the transformer layers, the output head, all of it meaningless static. Feed a brand-new, freshly-initialized model the prompt “The capital of France is” and it won’t say “ Paris” — it’ll pick a token essentially at random, maybe “ marmalade,” maybe “ 7,” maybe “ the.” Every token in the vocabulary is roughly equally likely, because the model has never seen a single sentence in its life.

So here’s the question this whole post answers: how do we get from a pile of random numbers to a model that completes “The capital of France is” with “ Paris” — and, along the way, picks up grammar, facts, a bit of arithmetic, and enough code to autocomplete your Python?

The answer is pretraining. And the beautiful thing is that it comes down to one loss function, computed over a few trillion tokens, a few trillion times. There are no human labels and no clever supervision anywhere in it — the model just guesses the next token, checks the answer, nudges its weights, and does it all over again.

By the end of this post you’ll understand four things, which also happen to be the four things that decide whether a pretrained model is any good:

1. The loss function — what cross-entropy actually measures, walked through one training step.

2. The data — where 15 trillion tokens come from and why the pipeline matters more than the model code.

3. The compute — the simple FLOP math behind a training run, and why these runs cost tens of millions of dollars.

4. The scaling laws — why Chinchilla proved that a smaller model trained on more data beats a bigger model trained on less.

And finally, what you actually get when the run finishes — and why it isn’t ChatGPT.

Follow along in code. There’s a companion notebook that pretrains a tiny GPT from scratch on a real public-domain book (pulled from Project Gutenberg) — it runs on a laptop CPU in a few minutes, and every concept below maps to a cell. Grab it on Kaggle.

Let’s dive in.

1. From Random Noise to a Base Model

Pretraining is the first, longest, and most expensive phase of building an LLM. It has exactly one job: take a model full of random weights and teach it to predict the next token across a giant, generic pile of text — a single objective, repeated at a scale that’s genuinely hard to picture.

The clever part — the thing that makes the whole modern LLM era possible — is that this learning is self-supervised. Here’s why that matters.

In a normal supervised setup, you need labeled examples: a photo and a human-written “cat,” a transaction and a human-written “fraud.” Labels are expensive: a human has to make each one, which puts a hard ceiling on how much data you can learn from.

Pretraining sidesteps the ceiling entirely. The trick: any piece of text is already its own answer key. Take a sentence:

“The cat sat on the mat.”

Hide the last word. Now you have a training example for free: the input is “The cat sat on the,” and the correct answer is “mat.” No human had to label it; the text supplied the answer itself. And you don’t just get one example per sentence — you get one for every position:

• Given “The” → predict “cat”

• Given “The cat” → predict “sat”

• Given “The cat sat” → predict “on”

• Given “The cat sat on” → predict “the”

• Given “The cat sat on the” → predict “mat”

One short sentence becomes five labeled examples, a single web page becomes thousands, and the entire internet becomes a near-infinite supply of free, self-labeling training data.

That’s the whole idea. Everything else in this post is detail on top of it: what loss we use to measure a wrong guess, where the text comes from, how much compute it takes, and how big the model should be.

One last bit of framing before we open up the machinery. Pretraining is only the first of three stages people constantly mix up — pretraining, post-training, and fine-tuning. They sound interchangeable, but they’re not. Let’s put them side by side first, then spend the rest of the post inside pretraining.

2. Pretraining, Post-Training, and Fine-Tuning: Who Does What

If you take one thing from this section: pretraining, post-training, and fine-tuning are three different jobs, done by different people, at wildly different costs. They are not synonyms, and mixing them up is the most common confusion I see when people talk about “training” a model. Here’s the whole lineage, start to finish:

Pretraining is what this whole post is about. Start from random weights, predict the next token over trillions of tokens of generic text, and end up with a base model. It’s self-supervised, so there are no human labels; it costs millions of dollars and months of compute; and only a handful of labs can afford to do it. This single stage is where roughly 99% of the model’s raw knowledge comes from.

Post-training takes that base model and teaches it behavior. A base model can complete text but won’t reliably answer a question or follow an instruction (I’ll show you exactly why at the end of the post). Post-training fixes that in two moves: instruction tuning — also called supervised fine-tuning — where you show it lots of instruction → good-response pairs, and preference tuning — RLHF, DPO, or GRPO — where you teach it which answers people actually prefer. The data is human-curated, there’s far less of it, and it’s comparatively cheap. The output is the instruct (chat) model you actually talk to, and model creators do this before they ship.

Fine-tuning is the part you do. You take an existing model (base or instruct) and adapt it to your own domain or task — whether that’s your support tickets, your medical notes, or your house code style. It’s supervised on your own labeled data, and these days it’s usually parameter-efficient (LoRA/PEFT), so it runs in hours on one or two GPUs for a handful of dollars. The output is a specialist.

One honest note on terminology: “post-training” and “fine-tuning” overlap, because instruction tuning literally is a kind of fine-tuning. The distinction that actually matters in practice is who and why: post-training is the model creator turning a raw base model into a general-purpose assistant; fine-tuning is you adapting a finished model to a narrow job. The rest of this post lives entirely in that first box — pretraining — but it helps to know what comes after.

That first stage does the bulk of the learning and burns almost all of the money; the later two are comparatively cheap. Keep that imbalance in mind — it explains a lot about why the industry looks the way it does. Now let’s open up that first box.

3. The Loss Function: Cross-Entropy, One Step at a Time

We keep saying “nudge the weights when the guess is wrong.” Time to make that precise, because the entire training run is just this one step, repeated billions of times.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

I love keeping track of everything week to week — here’s what happened this week. Enjoy this free weekly post! For those who want to dive deeper into any of these topics, that’s what my paid posts are for.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

This is the third part of the genAI Fundamentals series. Each post picks one building block of modern LLMs and explains it from first principles, with code.

Have you ever noticed how many screenshots float around LinkedIn of simple problems LLMs get wrong? “How many r’s in strawberry?” Wrong. “Reverse the word lollipop.” Wrong. “What’s 9.11 vs 9.9, which is bigger?” Confidently wrong.

People share these as “gotcha” moments. Proof that LLMs are dumb. Proof that we’re overhyping AI.

But here’s what most of those posts miss: these failures aren’t random. They’re predictable. And once you understand how an LLM actually processes text, you can tell in advance which tasks will trip it up and why.

The strawberry problem? The model never sees individual letters. It sees tokens, chunks like “str”, “aw”, “berry”. By the time any “reasoning” starts, the letters are already gone.

When I started working with LLMs in production, I had the same confusion everyone does. I couldn’t answer basic questions: Why doesn’t the model just process one character at a time? If it did, the strawberry problem would be trivial. What happens between typing a prompt and getting a response? Why does the model sometimes just... stop mid-sentence?

Once I understood the pipeline end-to-end, everything clicked. And those LinkedIn gotcha screenshots? They stopped being surprising. You look at one and think, “yeah, obviously it fails at that.”

This post covers the full picture: how text becomes tokens, how tokens become vectors, what happens inside the model, and how vectors become text again. So, let’s start.

1. What is an LLM, really?

Here’s the shortest accurate description:

That’s the entire interface. The integers represent tokens (subwords). The output is one probability per possible token in the vocabulary. The model picks one, appends it to the list, and runs the function again. ChatGPT, Claude, Llama, Gemini: they all do exactly this. Everything else, the chat formatting, the system prompts, is built on top of this one operation.

In PyTorch pseudo-code:

logits = model(input_ids)           # list of ints → raw scores
probs = softmax(logits[-1])         # last position → probabilities over vocab
next_token = sample(probs)          # pick one token

Three lines. That’s the core loop. The rest of this post explains what each piece does and where the numbers come from.

2. Tokenization: how text becomes numbers

A neural network processes numbers. Text is not numbers. So the first step is converting text into a sequence of integers. This conversion is tokenization, and it’s one of those things that sounds boring until you realize it’s responsible for half the weird behavior you’ve seen from LLMs.

The question is: what should each integer represent?

You have three options.

A. Characters. Each character gets its own ID. “cat” becomes three IDs. The vocabulary stays small. The problem is length: a 4,000-word document is around 20,000 characters, so that’s 20,000 positions for the attention mechanism to chew through, and attention cost grows quadratically with sequence length.

B. Words. Each whole word gets an ID. The vocabulary balloons to 500,000+ for English alone, and you still can’t handle typos (”caat”), brand-new words (”ChatGPT” didn’t exist in older vocabularies), or variants (”running”, “ran”, “runs” all become unrelated entries). Every unseen word is a dead end.

C. Subwords. Common words stay whole, rare words split into known pieces. “unfamiliarize” becomes [”un”, “familiar”, “ize”]. The vocabulary lands in a comfortable middle (32K to 256K). This is what every modern LLM uses.

The rest of this section is about how subword tokenizers actually get built. But first, two concepts on which everything depends.

Unicode and bytes

Unicode is one giant table that assigns a number to every character in every writing system. “A” is 65. “é” is 233. The Chinese character “中” is 20,013. The emoji “😀” is 128,512. About 150,000 characters are defined today, and the table keeps growing. Each number is called a code point.

Bytes are how those numbers actually get stored on a computer. A byte is just a value from 0 to 255 (8 bits, hence “byte”). Everything on your machine, including text, is ultimately a stream of these byte values.

The catch is that Unicode has ~150,000 code points, but a byte only goes up to 255. So you can’t fit “中” (code point 20,013) into a single byte. You need a scheme for packing big code points into sequences of small bytes. That scheme is UTF-8, the encoding that essentially all text uses today.

UTF-8 is variable-length: common characters get fewer bytes, rare ones get more.

• Plain English (the original ASCII set, code points 0-127) takes 1 byte. “A” is just byte 65.

• Accented Latin, Greek, Cyrillic, Hebrew, Arabic take 2 bytes.

• Most Chinese, Japanese, and Korean characters take 3 bytes.

• Emoji and rarer symbols take 4 bytes.

So “中” is not stored as the single number 20,013. UTF-8 packs that code point into three bytes: 228, 184, 173. If you wrote “中” to a file, those are the three values actually on disk. The string “A中” would be four bytes total: [65, 228, 184, 173].

This is the punchline for tokenization: no matter what language or symbol you throw at it, every piece of text is, at the lowest level, a sequence of byte values between 0 and 255. There are only ever 256 possible building blocks.

Hold onto this distinction (characters versus bytes), because the single most important design choice in a tokenizer is which of these it starts from.

BPE: the core algorithm

Byte Pair Encoding (BPE) started life as a data-compression trick in 1994. Sennrich et al. adapted it for NLP in 2016, and it has been the dominant tokenization algorithm ever since. GPT-2, GPT-4, Llama, Qwen, Mistral, DeepSeek: nearly all of them run BPE.

The algorithm is short. I’ll walk through it, because once you see it the rest falls into place.

Step 1: Start with a base set of atomic units. Say there are 256 of them (I’ll explain what they are in the next section). Every piece of text is now a sequence of these units.

Step 2: Scan the entire training corpus. Find the pair of adjacent units that appears most often. Maybe “e” followed by “r” shows up 50 million times.

Step 3: Merge that pair into one new unit, “er”. Add it to the vocabulary (now 257). Replace every “e” + “r” in the corpus with “er”.

Step 4: Repeat. The next most frequent pair might be “t” + “h” → “th”, then “th” + “e” → “the”. Merge, add, repeat.

Run 128,000 merges and you end up with ~128,256 tokens (the 256 base units plus 128K learned merges). That’s right in the range where a lot of modern models have landed.

Merge order matters. Early merges grab the most common letter combos (”th”, “in”, “er”), which combine into “the”, “ing”, “tion”, and eventually whole common words. Rare words stay split into pieces. The ordered list of merges is saved with the model and replayed at inference time.

Characters or bytes? The choice that actually matters

BPE merges pairs, but Step 1 has to start from something. This is the real decision, and it’s where the two main flavors of BPE split apart.

Character-level BPE starts from Unicode characters. The catch: there are ~150,000 of them. You either spend vocabulary slots on thousands of rare characters, or you hit characters you’ve never seen at inference time. Older tokenizers had a special <UNK> (”unknown”) token for exactly this, and it caused real failures on emoji, rare scripts, and even unusual typos.

Byte-level BPE starts from the 256-byte values instead. This is GPT-2’s key trick, and it’s why the base vocabulary is 256. Since every possible text in every language is just a sequence of bytes, there is no such thing as an unknown token. A brand-new emoji is simply four known byte-tokens stitched together. Chinese, Arabic, source code, corrupted text: all representable. Rare text just splits into more tokens. Nothing ever breaks.

This is why essentially all modern tokenizers are byte-level. When you saw “256 base units” in the algorithm above, those 256 are the byte values.

SentencePiece and tiktoken

BPE is the algorithm. SentencePiece and tiktoken are libraries that run it. Moving from one library to the other is not moving away from BPE. This trips up a lot of people who read that some model “switched from SentencePiece to tiktoken” and assume the algorithm changed. It didn’t. Both run BPE.

SentencePiece (Kudo & Richardson, 2018) is Google’s tokenizer library. Its real contribution is handling raw text without language-specific rules. Most tokenizers first split text on spaces, then run BPE per word, which breaks for Chinese, Japanese, and Thai, which don’t put spaces between words. SentencePiece skips the space-splitting step. It treats the space itself as a character (the “▁” symbol) and runs the merge algorithm on the raw stream. The early Llama models, Mistral, and the Gemma family all used SentencePiece, typically with a 32,000-token vocabulary.

SentencePiece is still widely used. The Gemma models use it, for example. Here it is, tokenizing a sentence (notice the “▁” marking each space):

# SentencePiece example
import sentencepiece as spm

sp = spm.SentencePieceProcessor()
sp.Load(”tokenizer.model”)

text = “Hello, how are you?”
tokens = sp.EncodeAsPieces(text)
# [’▁Hello’, ‘,’, ‘▁how’, ‘▁are’, ‘▁you’, ‘?’]
# The ▁ marks the start of a word (space replaced with ▁)

Notice the “▁” prefix. SentencePiece uses this to mark where spaces were in the original text. When you decode, it converts “▁” back to a space. This is how the model knows that “Hello” starts a new word.

tiktoken: OpenAI’s fast byte-level BPE

tiktoken is OpenAI’s tokenizer library. It runs byte-level BPE, it’s written in Rust (with Python bindings), and it’s significantly faster than older Python implementations. It’s the tokenizer behind GPT-3.5, GPT-4, and GPT-4o, and many newer open models (the later Llama releases, Qwen, DeepSeek) adopted tiktoken-style tokenizers too.

tiktoken ships with several pre-trained encodings:

• cl100k_base: used by GPT-4 and GPT-3.5 Turbo. 100,277 tokens.

• o200k_base: used by GPT-4o. ~200,019 tokens.

• gpt2: the original GPT-2 encoding. 50,257 tokens.

A lot of newer models moved from SentencePiece to tiktoken-style tokenizers, and it’s worth being clear that this isn’t “BPE stopped working.” Both run BPE. The migration is mostly about speed: tiktoken’s Rust encoder is much faster at turning text into tokens, which matters when you’re tokenizing trillions of training tokens and every inference request.

Let’s see tiktoken in action:

import tiktoken

# GPT-4’s tokenizer
enc = tiktoken.get_encoding(”cl100k_base”)

# Simple English
text = “The quick brown fox jumps over the lazy dog”
tokens = enc.encode(text)
print(f”Text: {text}”)
print(f”Tokens: {tokens}”)
print(f”Count: {len(tokens)} tokens”)
print(f”Decoded: {[enc.decode([t]) for t in tokens]}”)
# Tokens: [791, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679]
# Count: 9 tokens
# Decoded: [’The’, ‘ quick’, ‘ brown’, ‘ fox’, ‘ jumps’, ‘ over’, ‘ the’, ‘ lazy’, ‘ dog’]

# Now try something interesting
code = “def fibonacci(n):\n    return n if n < 2 else fibonacci(n-1) + fibonacci(n-2)”
code_tokens = enc.encode(code)
print(f”\nCode: {code}”)
print(f”Count: {len(code_tokens)} tokens”)
print(f”Decoded: {[enc.decode([t]) for t in code_tokens]}”)
# Common programming patterns are single tokens: “def”, “return”, “fibonacci”
# Indentation is a single token too

# The strawberry problem
word = “strawberry”
word_tokens = enc.encode(word)
print(f”\n’{word}’ = {[enc.decode([t]) for t in word_tokens]}”)
# ‘strawberry’ -> [’str’, ‘aw’, ‘berry’] (3 tokens)
# The model literally never sees the individual letters

9 tokens for 9 English words. That’s efficient. Code tokenizes well, too, because the BPE merges were trained on a corpus that included a lot of code. But look at “strawberry”: it splits as “str” + “aw” + “berry”, not into individual letters. This is why the model can’t count letters. It never sees them.

Who uses what: tokenizers across the major models

Here’s how tokenization evolved across the major model families. This is worth studying because the tokenizer choice tells you a lot about a model’s design priorities.

The “Algorithm” column tells you the flavor of BPE and the library used. Every model below runs BPE; what changes is whether it’s byte-level and which library implements it.

A few patterns jump out:

2019-2022: Small vocabulary, English-first. GPT-2 and GPT-3 used 50K tokens. Llama 1/2 and Mistral used 32K. These vocabularies were heavily English-biased. The same content in Hindi could need several times more tokens than in English.

2023: The 100K jump. GPT-4 doubled the vocabulary to 100K. This was the first big move toward multilingual efficiency: more tokens dedicated to non-English scripts means fewer tokens per sentence in those languages.

2024-2025: The 128K-256K era. Everyone expanded. Llama 3 and 4 went to 128K then 202K. Qwen settled on 152K. Gemma pushed to 256K, then 262K. The reasoning is consistent: bigger vocab = fewer tokens per text = faster inference, lower cost, longer effective context. The tradeoff is a bigger embedding table, but at 8B+ model sizes, that table is a small fraction of total parameters.

The library choice is mostly about speed, not capability. SentencePiece (Gemma, older Llama, Mistral) and tiktoken (GPT, newer Llama, Qwen) both run BPE. The migration toward tiktoken in newer models is largely because its Rust implementation encodes and decodes faster. The algorithm underneath is the same.

Qwen’s CJK advantage. Qwen’s 152K vocabulary was built with heavy Chinese, Japanese, and Korean coverage from the start. That’s why Qwen models tend to do well on CJK benchmarks relative to their size: the tokenizer compresses CJK text efficiently, giving the model more room per context window.

The rest of this post is for paid subscribers. So far, we’ve covered what an LLM is at its simplest and the full tokenization story: Unicode, bytes, UTF-8, BPE, byte-level vs character-level, SentencePiece vs tiktoken, and the cross-model tokenizer table.

Behind the paywall: how token IDs become vectors (the embedding table and what those vectors mean), how the model turns vectors back into words (the LM head, softmax, weight tying with a worked numeric example, and temperature), the full end-to-end pipeline with real numbers, autoregressive generation and the KV cache, and why tokenization quietly decides what a model can and can’t do.

Subscribe to keep reading.

Subscribe now

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Read more

Subscribe now

Over the coming weeks, I’ll be writing more about GenAI, including topics like pre-training and post-training. This post is the second one of the foundational pieces meant to set up that series.

Transformers run most of modern NLP, but they’re still surprisingly hard to internalize from a diagram alone. In my last post, I walked through how they work — the encoder, decoder, and the data flow between them.

This post is where we stop reading and start building: an end-to-end English-to-German translator in PyTorch, written from scratch with a Transformer at its core. Because the fastest way to actually understand something is to implement it.

Task Description

We want to create a translator that uses transformers to convert English to German. So, if we look at it as a black box, our network takes as input an English sentence and returns a German sentence.

Data Preprocessing

To train our English-German translation Model, we will need translated sentence pairs between English and German.

Fortunately, there is a pretty standard way to get these with the OPUS-100 dataset (English-German subset), a curated multilingual translation corpus we can access via HuggingFace datasets.

Also, before we really get into the whole coding part, let us understand what we need as input and output to the model while training. We will actually need two matrices to be input to our Network:

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Over the coming weeks, I’ll be writing more about GenAI, including topics like pre-training and post-training. This post is one of the foundational pieces meant to set up that series.

Transformers have become the de facto standard for almost everything. Though the architecture was introduced for NLP, it now powers computer vision, recommender systems, and—most importantly—the entire wave of modern LLMs.

Yet for all their ubiquity, transformers remain as hard to understand as ever.

It has taken me multiple readings through the Google research paper that first introduced transformers, along with just so many blog posts, to really understand how a transformer works.

So, I thought of putting the whole idea down in as simple words as possible, and with some very basic Math and some puns, as I am a proponent of having some fun while learning. I will try to keep both the jargon and the technicality to a minimum, yet it is such a topic that I could only do so much. And my goal is to make the reader understand even the most gory details of Transformer by the end of this post.

Also, this is officially my longest post, both in terms of time taken to write it as well as the length of the post. Hence, I will advise you to Grab A Coffee. ☕️

Before we dive in, here’s the path we’ll walk together: we’ll start with the big picture of what a transformer even does, then crack open the encoder stack (attention, feed-forward, positional encodings, and those mysterious “Add & Norm” boxes). From there, we’ll move to the decoder stack and the masking trick that makes it tick, bolt on an output head to actually get our German words, and finish with how the whole thing is trained and how it makes predictions at test time. Long road, but I promise the view is worth it. Onwards.

Q: So, why should I even understand Transformer?

In the past, the LSTM and GRU architecture(as explained here in my past post on NLP), along with the attention mechanism, used to be the State of the Art Approach for Language modeling problems (put very simply, predict the next word) and Translation systems. But the main problem with these architectures is that they are recurrent in nature, and the runtime increases as the sequence length increases. That is, these architectures take a sentence and process each word in a sequential way, and hence, with the increase in sentence length, the whole runtime increases.

Transformer, a model architecture first explained in the paper Attention is all you need, lets go of this recurrence and instead relies entirely on an attention mechanism to draw global dependencies between input and output. And that makes it FAST.

This is the picture of the full transformer as taken from the paper. And, it surely is intimidating. So, I will aim to demystify it in this post by going through each piece. So read ahead.

The Big Picture

Q: That sounds interesting. So, what does a transformer do exactly?

Essentially, a transformer can perform almost any NLP task. It can be used for language modeling, Translation, or Classification as required, and it does it fast by removing the sequential nature of the problem. So, the transformer in a machine translation application would convert one language to another, or for a classification problem will provide the class probability using an appropriate output layer.

It all will depend on the final output layer for the network; the Transformer basic structure will remain quite the same for any task. For this particular post, I will be continuing with the machine translation example.

So, from a very high place, this is how the transformer looks for a translation task. It takes as input an English sentence and returns a German sentence.

The Building Blocks

Q: That was too basic. 😎 Can you expand on it?

Okay, just remember in the end, you asked for it. Let’s go a little deeper and try to understand what a transformer is composed of.

So, a transformer is essentially composed of a stack of encoder and decoder layers. The role of an encoder layer is to encode the English sentence into a numerical form using the attention mechanism, while the decoder aims to use the encoded information from the encoder layers to give the German translation for the particular English sentence.

In the figure below, the transformer is given an English sentence as input, which gets encoded using 6 encoder layers. The output from the final encoder layer then goes to each decoder layer to translate English to German.

1. Encoder Architecture

Q: That’s alright, but how does an encoder stack encode an English sentence exactly?

Patience, I am getting to it. So, as I said, the encoder stack contains six encoder layers on top of each other(As given in the paper, but the future versions of transformers use even more layers). And each encoder in the stack has essentially two main layers:

• a multi-head self-attention Layer, and

• a position-wise fully connected feed-forward network

They are a mouthful. Right? Don’t lose me yet as I will explain both of them in the coming sections. Right now, just remember that the encoder layer incorporates attention and a position-wise feed-forward network.

Q: But, how does this layer expect its inputs to be?

This layer expects its inputs to be of the shape SxD (as shown in the figure below) where S is the source sentence(English Sentence) length, and D is the dimension of the embedding whose weights can be trained with the network. In this post, we will be using D as 512 by default throughout. While S will be the maximum length of a sentence in a batch. So it normally changes with batches.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

This is Part 9d of the RecSys for MLEs series. Part 9c explained why HSTU works: the softmax-to-SiLU switch that preserves engagement intensity, relative attention bias that gives the model a sense of time, and M-FALCON for cheap candidate scoring. This post is the hands-on follow-up. We’re going to train an HSTU from scratch, on the MovieLens-1M dataset, and benchmark it against the rectools library’s reference HSTU implementation, so you have a real number to compare against.

Last week, I published the conceptual deep dive on HSTU. Within 24 hours, the most common question in my replies was the same: “OK, but how do I actually train one?”

Fair. So today we build it from scratch — the fused (item, action) input layer, all three HSTU sub-layers, the multi-task retrieval + rating heads, and the M-FALCON inference cache. We’ll train it on MovieLens-1M, then benchmark head-to-head against rectools’ reference HSTU implementation, so you have a real number to compare against.

We’ll use PyTorch 2.x on a single GPU (a free Colab T4 works fine). I’m intentionally keeping the model small (D=64, 2 layers) so it trains in few hours on free hardware. And to make sure I wasn’t fooling myself with vanity numbers, I trained the rectools library’s HSTU on the same data and split, then ran my from-scratch model with a similar training config. That way, if my numbers come out worse, I know exactly where to look.

Here’s what we’ll cover:

• The dataset: how MovieLens ratings map to the (item, action, time) triples HSTU consumes

• The fused input layer: item embedding + action embedding + fusion MLP

• The HSTU block: all three sub-layers as a single PyTorch module — SiLU attention, RAB, gated transformation

• Multi-task heads: retrieval (sampled softmax with cosine similarity + temperature) and rating prediction

• Results: HR@10 and NDCG@10 against rectools HSTU and SASRec, plus the SiLU vs softmax ablation

• Example predictions: what the model actually recommends for specific MovieLens users

• Retrieval and ranking demos: brute-force, FAISS, and ranking by retrieval score + predicted rating

• M-FALCON inference: the K/V caching trick that makes serving feasible

1. The dataset: MovieLens-1M ratings as (item, action, time) triples

One of the most basic but important questions that we need to answer is how the data is structured. We’re using MovieLens-1M from GroupLens which contains ~1M ratings across 6,040 users and 3,706 movies.

Each row is (user, movie, rating, timestamp) where rating is 1-5 stars. SASRec would treat every rating as one positive interaction. HSTU’s input can be richer — it can fuse the action type alongside the item ID — so I want to give the model the rating sentiment, not just the fact that the user watched the movie. This is how I’m incorporating an action signal in this model. Honestly, every setup can have a different action vocabulary — add-to-cart vs. purchase on an e-commerce store, click vs. like vs. share on a feed, watch-25% vs. watch-90% vs. skip on a video platform. The point I want to make here is that HSTU lets you encode whichever signal actually matters for your domain, not just “the user clicked this item.”

For our case, we create three behavioral signals, derived from the rating value:

• POSITIVE (rating ≥ 4): the user liked the movie

• NEUTRAL (rating = 3): the user was indifferent

• NEGATIVE (rating < 3): the user disliked the movie

The actual rating value (1-5) is kept separately as the label for the rating-prediction head.

import pandas as pd
import numpy as np

ratings = pd.read_csv(”ml-1m/ratings.dat”, sep=”::”, header=None,
                      names=[”user”, “item”, “rating”, “ts”], engine=”python”)

def rating_to_action(r):
    if r >= 4: return “POSITIVE”
    if r == 3: return “NEUTRAL”
    return “NEGATIVE”

events = ratings.copy()
events[”action”] = events[”rating”].apply(rating_to_action)
events[”value”]  = events[”rating”].astype(np.float32)

print(events.action.value_counts())
# POSITIVE    575281
# NEUTRAL     261197
# NEGATIVE    163731

Train/test split: leave-one-out. For each user, the last interaction goes to test, and everything before that goes to training. During training, the last item of the training sequence is held out as the validation target.

def split_seq(seq, max_len=200):
    n = len(seq[”items”])
    if n < 3: return None
    history_slice = slice(max(0, n - 1 - max_len), n - 1)
    history = {k: seq[k][history_slice].tolist() for k in seq}
    test_target = (
        int(seq[”items”][n - 1]),
        int(seq[”actions”][n - 1]),
        int(seq[”times”][n - 1]),
        float(seq[”values”][n - 1]),
    )
    return history, test_target

splits = [s for s in (split_seq(seq) for seq in sequences) if s is not None]
# Train/test sequences: 6,040

max_len=200 is the cap. With an average length of 165, most users will fit. Long-tail users get truncated to their most recent 200 events.

2. The fused input: item embedding + action embedding + fusion MLP

Here’s the actual code.

import torch
import torch.nn as nn
import torch.nn.functional as F

class FusedInputEmbedding(nn.Module):
    “”“Item embedding + action embedding, fused with an MLP into a single D-dim vector.”“”
    def __init__(self, num_items, num_actions, dim):
        super().__init__()
        self.item_emb   = nn.Embedding(num_items + 1, dim, padding_idx=0)
        self.action_emb = nn.Embedding(num_actions + 1, dim, padding_idx=0)
        self.fuse = nn.Sequential(
            nn.Linear(2 * dim, dim),
            nn.SiLU(),
            nn.Linear(dim, dim),
        )

    def forward(self, item_ids, action_ids):
        i = self.item_emb(item_ids)      # (B, T, D)
        a = self.action_emb(action_ids)  # (B, T, D)
        return self.fuse(torch.cat([i, a], dim=-1))  # (B, T, D)

What’s happening here is that each item ID looks up a D-dim vector in item_emb (the “what is this item” signal) table, and each action ID looks up another D-dim vector in action_emb (the “how did the user engage with it” signal).

The forward pass then concatenates them into a 2D-dim vector per token, then the fuse MLP projects back to D dims.

The MLP matters. It’s what lets the model learn that “watched Toy Story with rating 5” represents something different from “watched Toy Story with rating 1”. A simple sum or concatenation wouldn’t give the model the capacity to learn that interaction.

3. The HSTU block in PyTorch

Now, it is time to write the actual HSTU block. If you want to refresh on what each piece does, Part 9c walks through the three sub-layers conceptually. Here I’ll just translate that directly into code.

First, the relative attention bias (RAB) module. Two learnable tables: one for relative position offset, one for log-spaced time buckets.

class RelativeAttentionBias(nn.Module):
    “”“Learnable position + time biases, added to QK^T before SiLU.”“”
    TIME_BUCKETS = [
        (0, 3600),                  # 0-1 hour
        (3600, 86400),              # 1-24 hours
        (86400, 86400 * 7),         # 1-7 days
        (86400 * 7, 86400 * 30),    # 7-30 days
        (86400 * 30, float(”inf”)), # 30+ days
    ]

    def __init__(self, max_seq_len):
        super().__init__()
        self.max_seq_len = max_seq_len
        self.pos_bias  = nn.Embedding(2 * max_seq_len - 1, 1)
        self.time_bias = nn.Embedding(len(self.TIME_BUCKETS), 1)
        nn.init.zeros_(self.pos_bias.weight)
        nn.init.zeros_(self.time_bias.weight)

    def _bucket(self, time_deltas):
        out = torch.zeros_like(time_deltas, dtype=torch.long)
        for k, (lo, hi) in enumerate(self.TIME_BUCKETS):
            mask = (time_deltas >= lo) & (time_deltas < hi)
            out = torch.where(mask, torch.full_like(out, k), out)
        return out

    def forward(self, times):
        B, T = times.shape
        idx = torch.arange(T, device=times.device)
        rel_pos = (idx.view(T, 1) - idx.view(1, T)) + (self.max_seq_len - 1)
        pos_b = self.pos_bias(rel_pos).squeeze(-1)
        time_deltas = (times.unsqueeze(2) - times.unsqueeze(1)).abs()
        time_b = self.time_bias(self._bucket(time_deltas)).squeeze(-1)
        return pos_b.unsqueeze(0) + time_b

Initialize both bias tables to zeros. That way, the first forward pass behaves like a vanilla attention block, and the time/position biases learn from gradient flow.

Now the HSTU block itself:

class HSTUBlock(nn.Module):
    def __init__(self, dim, max_seq_len, dropout=0.2):
        super().__init__()
        self.linear_in  = nn.Linear(dim, 4 * dim)
        self.rab        = RelativeAttentionBias(max_seq_len)
        self.norm       = nn.LayerNorm(dim)
        self.linear_out = nn.Linear(dim, dim)
        self.dropout    = nn.Dropout(dropout)

    def forward(self, x, times, attn_mask):
        B, T, D = x.shape

        # --- Sub-layer 1: pointwise projection ---
        proj = F.silu(self.linear_in(x))
        K, Q, V, U = proj.chunk(4, dim=-1)

        # --- Sub-layer 2: spatial aggregation (SiLU attention + RAB + causal/pad mask) ---
        scores = torch.matmul(Q, K.transpose(-2, -1)) + self.rab(times)
        causal = torch.tril(torch.ones(T, T, device=x.device))
        scores = scores * causal
        pad_mask = attn_mask.unsqueeze(1).float()
        scores = scores * pad_mask
        activated = F.silu(scores)                     # NOT softmax — pointwise SiLU
        attn_out  = torch.matmul(activated, V) / T     # 1/T normalization

        # --- Sub-layer 3: gated transformation + residual ---
        gated  = self.norm(attn_out) * U
        output = self.dropout(self.linear_out(gated))
        return output + x

Stacking blocks is one line:

class HSTUEncoder(nn.Module):
    def __init__(self, num_items, num_actions, dim, num_layers, max_seq_len, dropout=0.2):
        super().__init__()
        self.embed  = FusedInputEmbedding(num_items, num_actions, dim)
        self.blocks = nn.ModuleList([
            HSTUBlock(dim, max_seq_len, dropout=dropout) for _ in range(num_layers)
        ])

    def forward(self, item_ids, action_ids, times, attn_mask):
        x = self.embed(item_ids, action_ids)
        for block in self.blocks:
            x = block(x, times, attn_mask)
        return x

That’s the encoder.

The rest of this post covers the multi-task heads (retrieval + rating), the full results table comparing our from-scratch HSTU to rectools HSTU and SASRec, the SiLU vs softmax ablation, the learned time-bias curve, M-FALCON inference benchmark (210× speedup), and a production field guide.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

This is Part 9c of the RecSys for MLEs series. In Part 9a, we built GRU4Rec and SASRec from scratch on the Steam Games dataset and got our hands dirty with sequential recommenders. In Part 9b, we covered Semantic IDs and TIGER, Google’s clever approach to generative retrieval, where item IDs become a learned token vocabulary. Now we close the arc with the architecture that’s actually running at Meta scale: HSTU.

Late 2022. You’re an ML engineer on a recommendation team, and you’ve just shipped a beefed-up SASRec model. The metrics look great. Your director walks over and says: “What if we just scaled this thing? More layers, bigger embeddings, a hundred billion parameters. Like the LLM folks are doing.”

So you do. And for a while, the model gets better. Then it stops getting better. Then you throw more compute at it and nothing. No improvement. A bigger, slower, more expensive model that performs about the same.

Frustrating, because over in NLP-land the scaling laws are clean: double the compute, get a predictably better model. GPT-3 had proven that. LLaMA was about to prove it again. DLRMs (Deep Learning Recommendation Models)? They kept plateauing. Something was fundamentally broken, and nobody could quite articulate what.

Meta’s answer was HSTU (Hierarchical Sequential Transduction Units).

Here’s what we’ll work through in this post:

• The three structural issues that quietly sabotage standard Transformers when you point them at recommendation data

• What HSTU actually consumes: the fused (item, action) input format, plus the actual training data schema (impression table + history table)

• Inside the HSTU block: three sub-layers, with the math, the code, the intuition, and the diagrams

• M-FALCON: the caching optimization that lets you score 10,000 candidates without recomputing 8,000 history events 10,000 times

By the end, you should understand exactly why HSTU works, how it works, and why every big tech RecSys team is suddenly building one.

1. Three reasons standard Transformers fail for RecSys

Three structural mismatches between language data and recommendation data make standard Transformers fail at scale. These are fundamental, not edge cases you can patch with a clever hack.

If you’d like a refresher on how standard Transformer attention works before going further, that post will get you up to speed. From here on, I’ll assume you’re comfortable with Q, K, V, softmax, and self-attention.

Problem 01: non-stationary vocabularies

In NLP, your vocabulary is fixed. “The” is always token 42. “Recommendation” is always token 18,973. The model trains with a stable set of possible next tokens, and that set doesn’t really change between training and serving.

In recommendations, your “vocabulary” is the entire item catalog, and that catalog is constantly changing. New videos go live every second on Reels. A trending creator didn’t exist yesterday. The model you trained last Tuesday has never seen most of the items it’s being asked to score on a Friday.

Softmax was designed for a fixed vocabulary with stable class boundaries. It’s defined over a set of mutually exclusive classes that sum to 1. When the set of possible “next tokens” is a moving target, when items appear and disappear from the catalog every minute, the softmax assumption starts to silently misbehave. The probability mass keeps getting redistributed among the items the model happens to have seen, which is not the same set as the items it needs to recommend.

This is a much bigger deal than it sounds. Almost every metric you care about (CTR, watch time, retention) depends on the model surfacing new items that the user will love. If your normalization assumes a stable world, you’re starting from a broken assumption.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

I love keeping track of everything week to week — here's what happened this week. Enjoy this free weekly post! For those who want to dive deeper into any of these topics, that's what my paid posts are for.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

At the end of Part 1, I left you with a question: What if items had semantic identifiers that captured their content?

That question sounds innocent enough, but it’s actually the hinge point for the entire generative recommender revolution we are going to be talking about in this post. And it is very interesting to say the least.

So, we built GRU4Rec and SASRec on the Steam Games dataset in our previous post. But both those models treat items as arbitrary integers. The model sees “Item 4,271 → Item 8,903 → Item 2,156” and learns some sort of statistical patterns between these numbers.

The thing we need to note is that everything about what these games actually are — their genre, their developer, their visual style, the reason a player moves from one to the next — lives entirely outside the model. That is a pretty big opportunity to work on.

Now imagine if every item carried an ID that meant something. Similar games naturally share ID prefixes. A brand-new title gets a meaningful ID the moment it’s published — before anyone plays it. The recommender can reason about games it’s never seen just from their ID, just like a human would by seeing a game’s description or title.

That’s Semantic IDs.

And these semantic IDs unlock something bigger →instead of scoring candidates from a retrieved shortlist(which is how SASRec and GRU4Rec work), a model can now generate the next item token by token, the way GPT generates words.

In this post, we will create a complete pipeline to do exactly that on the same Steam dataset, every game compressed into meaningful tokens using Residual Quantized VAE, and a generative recommender(TIGER) trained from scratch.

Let’s dive in!

1. The Problem with Random Item IDs

Just think about how SASRec works for a second - A user plays [Counter-Strike, Portal, Half-Life 2]. The model looks up the embedding for each game, runs self-attention, and predicts the next game. So far, so good.

But what does the model actually know about Counter-Strike? Nothing. It’s just item ID 4,271. The model has no idea it’s a first-person shooter made by Valve in 2004.

This creates three problems that compound in production:

1. No knowledge sharing. If 10,000 users play Counter-Strike → Team Fortress 2, the model learns that specific transition. But it learns nothing about why they’re related. A new Valve FPS arrives tomorrow, and the model has zero signal for it — even though any human could tell you “people who like Counter-Strike would probably like this.”

2. Cold-start is brutal. New items have brand-new IDs with randomly initialized embeddings. As I covered in my post on the cold-start problem, this means they need thousands of interactions before the model can meaningfully recommend them. In fast-moving catalogs — think news, short videos, new game releases — items can go stale before the model even learns to recommend them.

3. No generalization. The model memorizes specific ID→ID transitions. It can’t reason about categories of items, properties of items, or relationships between items.

And that’s where semantic IDs could help us.

2. Semantic IDs: Making Item IDs mean Something

Ok, so now we understand why Semantic IDs might be beneficial, let’s see how they are made. The idea behind Semantic IDs was introduced in TIGER (Transformer Index for Generative Recommenders) at NeurIPS 2023, and it completely changed how we think about item representation in recommender systems.

The pipeline works in three steps. Here’s what happens to a single game:

In step 1, we start by getting a content-based embedding of an item using a powerful text encoder.

Once we have that, we run the RQVAE algorithm in step 2 to generate Semantic IDs of items. (Don’t worry, we will talk about this in this post. For now, just understand that it gives you some sort of discrete token-based vector given your item embeddings)

In Step 3, we generate the next item’s Semantic ID token by token, the same way GPT writes one word at a time. It’s not scoring a fixed list; it’s constructing an answer.

Now think about what this structure gives you. Counter-Strike and Half-Life might both get the prefix [42, 187, ...] because they’re both Valve FPS games. A brand-new Valve FPS that launched this morning might then be assigned a similar prefix based purely on its content — and the model already knows what to do with items that start with [42, 187, ...], even though it’s never seen a single player interact with this game.

Remember how in the cold start post I wrote about how new items have nothing but their metadata to work with? With Semantic IDs, the metadata becomes their ID. The model doesn’t need thousands of interactions to figure out what a new game is — it already knows, just from reading the ID.

And notice what else changed: we replaced the entire retrieve→rank→rerank pipeline with a single model that generates recommendations directly. At inference time, the decoder uses beam search: instead of greedily committing to one token at each level and hoping for the best, it keeps the top B candidates at each step and extends them all in parallel. You end up with a ranked list of complete Semantic IDs, each mapping to a real item. That means you don’t have to worry about maintaining an ANN index, a candidate retrieval stage, and a separate ranker stage. You can get all of this from a single model.

Now, the magic of this pipeline lives entirely in Step 2 — the RQVAE. That is how we create these semantic IDs. And if the quantization is bad, the Semantic IDs are meaningless, and we’re back to square one. So let’s understand exactly how it works.

3. How RQVAE Works — The Engine Behind Semantic IDs

Residual Quantized VAE sounds intimidating, but the intuition is straightforward. It’s progressive compression — like describing a location with increasing precision.

“North America” tells you the continent. “California” narrows it down. “San Francisco” pins it to a city. Each level adds detail that the previous level didn’t capture. RQVAE does exactly this, but for item embeddings.

Here’s the algorithm.

Step 1: Encode. Take the 1024-dim item embedding and compress it through an encoder network (1024 → 512 → 256 → 128 → 32). It’s easier to follow with concrete numbers, so let’s walk through a simplified example — we will use 4 dimensions instead of 32, but the math is identical.

Encoded: [0.8, -0.3, 0.5, 0.1]

Step 2: Level 1 Quantization. You have a codebook — a table of 256 learned vectors. Find the one closest to your latent vector:

Read more

Story of the Week: The Great Unbundling

The Microsoft–OpenAI partnership came apart Sunday night. Revenue sharing, the AGI clause, and exclusivity all dissolved in a single restructuring.

Twenty-four hours later, Sam Altman and AWS CEO Matt Garman were on Stratechery announcing OpenAI models on Amazon Bedrock. The speed tells you the deal was pre-negotiated; only the Microsoft contract was holding it back.

The practical change for ML teams: GPT models now run natively on Bedrock alongside Claude, Llama, and Mistral. If you standardized on AWS but routed around for OpenAI access, that workaround is gone. If you picked Azure specifically for OpenAI, you have a real alternative for the first time. Multi-cloud routing for frontier models is table stakes now — expect inference prices to drop and cross-cloud model-parity to start mattering in vendor evaluations.

Then both labs made a stranger move. Anthropic and OpenAI launched enterprise consulting arms on the same day, aimed at different mar…

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Read more

PyTorch has sort of became one of the de facto standards for creating Neural Networks now, and I love its interface. Yet, it is somehow a little difficult for beginners to get a hold of.

I remember picking PyTorch up only after some extensive experimentation a couple of years back. To tell you the truth, it took me a lot of time to pick it up but am I glad that I moved from Keras to PyTorch . With its high customizability and pythonic syntax,PyTorch is just a joy to work with, and I would recommend it to anyone who wants to do some heavy lifting with Deep Learning.

So, in this PyTorch guide, I will try to ease some of the pain with PyTorch for starters and go through some of the most important classes and modules that you will require while creating any Neural Network with Pytorch.

But, that is not to say that this is aimed at beginners only as I will also talk about the high customizability PyTorch provides and will talk about custom Layers, Datasets, Dataloaders, and Loss functions.

So let’s get some coffee ☕ ️and start it up.

Tensors

Tensors are the basic building blocks in PyTorch and put very simply, they are NumPy arrays but on GPU. In this part, I will list down some of the most used operations we can use while working with Tensors. This is by no means an exhaustive list of operations you can do with Tensors, but it is helpful to understand what tensors are before going towards the more exciting parts.

1. Create a Tensor

We can create a PyTorch tensor in multiple ways. This includes converting to tensor from a NumPy array. Below is just a small gist with some examples to start with, but you can do a whole lot of more things with tensors just like you can do with NumPy arrays.

# Using torch.Tensor
t = torch.Tensor([[1,2,3],[3,4,5]])
print(f"Created Tensor Using torch.Tensor:\n{t}")

# Using torch.randn
t = torch.randn(3, 5)
print(f"Created Tensor Using torch.randn:\n{t}")

# using torch.[ones|zeros](*size)
t = torch.ones(3, 5)
print(f"Created Tensor Using torch.ones:\n{t}")
t = torch.zeros(3, 5)
print(f"Created Tensor Using torch.zeros:\n{t}")

# using torch.randint - a tensor of size 4,5 with entries between 0 and 10(excluded)
t = torch.randint(low = 0,high = 10,size = (4,5))
print(f"Created Tensor Using torch.randint:\n{t}")

# Using from_numpy to convert from Numpy Array to Tensor
a = np.array([[1,2,3],[3,4,5]])
t = torch.from_numpy(a)
print(f"Convert to Tensor From Numpy Array:\n{t}")

# Using .numpy() to convert from Tensor to Numpy array
t = t.numpy()
print(f"Convert to Numpy Array From Tensor:\n{t}")

2. Tensor Operations

Again, there are a lot of operations you can do on these tensors. The full list of functions can be found here .

A = torch.randn(3,4)
W = torch.randn(4,2)
# Multiply Matrix A and W
t = A.mm(W)
print(f"Created Tensor t by Multiplying A and W:\n{t}")
# Transpose Tensor t
t = t.t()
print(f"Transpose of Tensor t:\n{t}")
# Square each element of t
t = t**2
print(f"Square each element of Tensor t:\n{t}")
# return the size of a tensor
print(f"Size of Tensor t using .size():\n{t.size()}")

Note: What are PyTorch Variables? In the previous versions of Pytorch, Tensor and Variables used to be different and provided different functionality, but now the Variable API is deprecated , and all methods for variables work with Tensors. So, if you don’t know about them, it’s fine as they re not needed, and if you know them, you can forget about them.

The nn.Module

Here comes the fun part as we are now going to talk about some of the most used constructs in Pytorch while creating deep learning projects. nn.Module lets you create your Deep Learning models as a class. You can inherit from nn.Moduleto define any model as a class. Every model class necessarily contains an __init__ procedure block and a block for the forward pass.

• In the __init__ part, the user can define all the layers the network is going to have but doesn’t yet define how those layers would be connected to each other.

• In the forward pass block, the user defines how data flows from one layer to another inside the network.

So, put simply, any network we define will look like:

class myNeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        # Define all Layers Here
        self.lin1 = nn.Linear(784, 30)
        self.lin2 = nn.Linear(30, 10)
    def forward(self, x):
        # Connect the layer Outputs here to define the forward pass
        x = self.lin1(x)
        x = self.lin2(x)
        return x

Here we have defined a very simple Network that takes an input of size 784 and passes it through two linear layers in a sequential manner. But the thing to note is that we can define any sort of calculation while defining the forward pass, and that makes PyTorch highly customizable for research purposes. For example, in our crazy experimentation mode, we might have used the below network where we arbitrarily attach our layers. Here we send back the output from the second linear layer back again to the first one after adding the input to it(skip connection) back again(I honestly don’t know what that will do).

class myCrazyNeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        # Define all Layers Here
        self.lin1 = nn.Linear(784, 30)
        self.lin2 = nn.Linear(30, 784)
        self.lin3 = nn.Linear(30, 10)

    def forward(self, x):
        # Connect the layer Outputs here to define the forward pass
        x_lin1 = self.lin1(x)
        x_lin2 = x + self.lin2(x_lin1)
        x_lin2 = self.lin1(x_lin2)
        x = self.lin3(x_lin2)
        return x

We can also check if the neural network forward pass works. I usually do that by first creating some random input and just passing that through the network I have created.

x = torch.randn((100,784))
model = myCrazyNeuralNet()
model(x).size()
--------------------------
torch.Size([100, 10])

A word about Layers

Pytorch is pretty powerful, and you can actually create any new experimental layer by yourself using nn.Module. For example, rather than using the predefined Linear Layer nn.Linear from Pytorch above, we could have created our custom linear layer.

class myCustomLinearLayer(nn.Module):
    def __init__(self,in_size,out_size):
        super().__init__()
        self.weights = nn.Parameter(torch.randn(in_size, out_size))
        self.bias = nn.Parameter(torch.zeros(out_size))
    def forward(self, x):
        return x.mm(self.weights) + self.bias

You can see how we wrap our weights tensor in nn.Parameter. This is done to make the tensor to be considered as a model parameter. From PyTorch docs :

Parameters are <code>*Tensor*</code> subclasses, that have a very special property when used with Module - when they’re assigned as Module attributes they are automatically added to the list of its parameters, and will appear in parameters() iterator

As you will later see, the model.parameters() iterator will be an input to the optimizer. But more on that later.

Right now, we can now use this custom layer in any PyTorch network, just like any other layer.

class myCustomNeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        # Define all Layers Here
        self.lin1 = myCustomLinearLayer(784,10)

    def forward(self, x):
        # Connect the layer Outputs here to define the forward pass
        x = self.lin1(x)
        return x
x = torch.randn((100,784))
model = myCustomNeuralNet()
model(x).size()
------------------------------------------
torch.Size([100, 10])

But then again, Pytorch would not be so widely used if it didn’t provide a lot of ready to made layers used very frequently in wide varieties of Neural Network architectures. Some examples are: nn.Linear , nn.Conv2d , nn.MaxPool2d , nn.ReLU , nn.BatchNorm2d , nn.Dropout , nn.Embedding , nn.GRU / nn.LSTM , nn.Softmax , nn.LogSoftmax , nn.MultiheadAttention , nn.TransformerEncoder , nn.TransformerDecoder

I have linked all the layers to their source where you could read all about them, but to show how I usually try to understand a layer and read the docs, I would try to look at a very simple convolutional layer here.

So, a Conv2d Layer needs as input an Image of height H and width W, with Cin channels. Now, for the first layer in a convnet, the number of in_channels would be 3(RGB), and the number of out_channels can be defined by the user. The kernel_size mostly used is 3x3, and the stride normally used is 1.

To check a new layer which I don’t know much about, I usually try to see the input as well as output for the layer like below where I would first initialize the layer:

conv_layer = nn.Conv2d(in_channels = 3, out_channels = 64, kernel_size = (3,3), stride = 1, padding=1)

And then pass some random input through it. Here 100 is the batch size.

x = torch.randn((100,3,24,24))
conv_layer(x).size()
--------------------------------
torch.Size([100, 64, 24, 24])

So, we get the output from the convolution operation as required, and I have sufficient information on how to use this layer in any Neural Network I design.

Datasets and DataLoaders

How would we pass data to our Neural nets while training or while testing? We can definitely pass tensors as we have done above, but Pytorch also provides us with pre-built Datasets to make it easier for us to pass data to our neural nets. You can check out the complete list of datasets provided at torchvision.datasets and torchtext.datasets . But, to give a concrete example for datasets, let’s say we had to pass images to an Image Neural net using a folder which has images in this structure:

data
    train
        sailboat
        kayak
        .
        .

We can use torchvision.datasets.ImageFolder dataset to get an example image like below:

from torchvision import transforms
from torchvision.datasets import ImageFolder
traindir = "data/train/"
t = transforms.Compose([
        transforms.Resize(size=256),
    transforms.CenterCrop(size=224),
        transforms.ToTensor()])
train_dataset = ImageFolder(root=traindir,transform=t)
print("Num Images in Dataset:", len(train_dataset))
print("Example Image and Label:", train_dataset[2])

This dataset has 847 images, and we can get an image and its label using an index. Now we can pass images one by one to any image neural network using a for loop:

for i in range(0,len(train_dataset)):
    image ,label = train_dataset[i]
    pred = model(image)

But that is not optimal. We want to do batching. We can actually write some more code to append images and labels in a batch and then pass it to the Neural network. But Pytorch provides us with a utility iterator torch.utils.data.DataLoader to do precisely that. Now we can simply wrap our train_dataset in the Dataloader, and we will get batches instead of individual examples.

train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=True, num_workers=10)

We can simply iterate with batches using:

for image_batch, label_batch in train_dataloader:
    print(image_batch.size(),label_batch.size())
    break
-------------------------------------------------
torch.Size([64, 3, 224, 224]) torch.Size([64])

So actually, the whole process of using datasets and Dataloaders becomes:

t = transforms.Compose([
        transforms.Resize(size=256),
    transforms.CenterCrop(size=224),
        transforms.ToTensor()])

train_dataset = torchvision.datasets.ImageFolder(root=traindir,transform=t)
train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=True, num_workers=10)

for image_batch, label_batch in train_dataloader:
    pred = myImageNeuralNet(image_batch)

You can look at this particular example in action in my previous blogpost on Image classification using Deep Learning here .

This is great, and Pytorch does provide a lot of functionality out of the box. But the main power of Pytorch comes with its immense customization. We can also create our own custom datasets if the datasets provided by PyTorch don’t fit our use case.

Understanding Custom Datasets

To write our custom datasets, we can make use of the abstract class torch.utils.data.Dataset provided by Pytorch. We need to inherit this Dataset class and need to define two methods to create a custom Dataset.

• __len__ : a function that returns the size of the dataset. This one is pretty simple to write in most cases.

• __getitem__: a function that takes as input an index i and returns the sample at index i.

For example, we can create a simple custom dataset that returns an image and a label from a folder. See that most of the tasks are happening in __init__ part where we use glob.glob to get image names and do some general preprocessing.

from glob import glob
from PIL import Image
from torch.utils.data import Dataset

class customImageFolderDataset(Dataset):
    """Custom Image Loader dataset."""
    def __init__(self, root, transform=None):
        """
        Args:
            root (string): Path to the images organized in a particular folder structure.
            transform: Any Pytorch transform to be applied
        """
        # Get all image paths from a directory
        self.image_paths = glob(f"{root}/*/*")
        # Get the labels from the image paths
        self.labels = [x.split("/")[-2] for x in self.image_paths]
        # Create a dictionary mapping each label to a index from 0 to len(classes).
        self.label_to_idx = {x:i for i,x in enumerate(set(self.labels))}
        self.transform = transform

    def __len__(self):
        # return length of dataset
        return len(self.image_paths)

    def __getitem__(self, idx):
        # open and send one image and label
        img_name = self.image_paths[idx]
        label = self.labels[idx]
        image = Image.open(img_name)
        if self.transform:
            image = self.transform(image)
        return image,self.label_to_idx[label]

Also, note that we open our images one at a time in the __getitem__ method and not while initializing. This is not done in __init__ because we don’t want to load all our images in the memory and just need to load the required ones.

We can now use this dataset with the utility Dataloader just like before. It works just like the previous dataset provided by PyTorch but without some utility functions.

t = transforms.Compose([
        transforms.Resize(size=256),
    transforms.CenterCrop(size=224),
        transforms.ToTensor()])

train_dataset = customImageFolderDataset(root=traindir,transform=t)
train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=True, num_workers=10)

for image_batch, label_batch in train_dataloader:
    pred = myImageNeuralNet(image_batch)

Understanding Custom DataLoaders

This particular section is a little advanced and can be skipped going through this post as it will not be needed in a lot of situations. But I am adding it for completeness here.

So let’s say you are looking to provide batches to a network that processes text input, and the network could take sequences with any sequence size as long as the size remains constant in the batch. For example, we can have a BiLSTM network that can process sequences of any length. It’s alright if you don’t understand the layers used in it right now; just know that it can process sequences with variable sizes.

class BiLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden_size = 64
        drp = 0.1
        max_features, embed_size = 10000,300
        self.embedding = nn.Embedding(max_features, embed_size)
        self.lstm = nn.LSTM(embed_size, self.hidden_size, bidirectional=True, batch_first=True)
        self.linear = nn.Linear(self.hidden_size*4 , 64)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(drp)
        self.out = nn.Linear(64, 1)


    def forward(self, x):
        h_embedding = self.embedding(x)
        h_embedding = torch.squeeze(torch.unsqueeze(h_embedding, 0))

        h_lstm, _ = self.lstm(h_embedding)
        avg_pool = torch.mean(h_lstm, 1)
        max_pool, _ = torch.max(h_lstm, 1)
        conc = torch.cat(( avg_pool, max_pool), 1)
        conc = self.relu(self.linear(conc))
        conc = self.dropout(conc)
        out = self.out(conc)
        return out

This network expects its input to be of shape (batch_size, seq_length) and works with any seq_length. We can check this by passing our model two random batches with different sequence lengths(10 and 25).

model = BiLSTM()
input_batch_1 = torch.randint(low = 0,high = 10000, size = (100,**10**))
input_batch_2 = torch.randint(low = 0,high = 10000, size = (100,**25**))
print(model(input_batch_1).size())
print(model(input_batch_2).size())
------------------------------------------------------------------
torch.Size([100, 1])
torch.Size([100, 1])

Now, we want to provide tight batches to this model, such that each batch has the same sequence length based on the max sequence length in the batch to minimize padding. This has an added benefit of making the neural net run faster. It was, in fact, one of the methods used in the winning submission of the Quora Insincere challenge in Kaggle, where running time was of utmost importance.

So, how do we do this? Let’s write a very simple custom dataset class first.

class CustomTextDataset(Dataset):
    '''
    Simple Dataset initializes with X and y vectors
    We start by sorting our X and y vectors by sequence lengths
    '''
    def __init__(self,X,y=None):
        self.data = list(zip(X,y))
        # Sort by length of first element in tuple
        self.data = sorted(self.data, key=lambda x: len(x[0]))

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]

Also, let’s generate some random data which we will use with this custom Dataset.

import numpy as np
train_data_size = 1024
sizes = np.random.randint(low=50,high=300,size=(train_data_size,))
X = [np.random.randint(0,10000, (sizes[i])) for i in range(train_data_size)]
y = np.random.rand(train_data_size).round()
#checking one example in dataset
print((X[0],y[0]))

Example of one random sequence and label. Each integer in the sequence corresponds to a word in the sentence.

We can use the custom dataset now using:

train_dataset = CustomTextDataset(X,y)

If we now try to use the Dataloader on this dataset with batch_size>1, we will get an error. Why is that?

train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=False, num_workers=10)
for xb,yb in train_dataloader:
    print(xb.size(),yb.size())

This happens because the sequences have different lengths, and our data loader expects our sequences of the same length. Remember that in the previous image example, we resized all images to size 224 using the transforms, so we didn’t face this error.

So, how do we iterate through this dataset so that each batch has sequences with the same length, but different batches may have different sequence lengths?

We can use collate_fn parameter in the DataLoader that lets us define how to stack sequences in a particular batch. To use this, we need to define a function that takes as input a batch and returns (x_batch, y_batch ) with padded sequence lengths based on max_sequence_length in the batch. The functions I have used in the below function are simple NumPy operations. Also, the function is properly commented so you can understand what is happening.

def collate_text(batch):
    # get text sequences in batch
    data = [item[0] for item in batch]
    # get labels in batch
    target = [item[1] for item in batch]
    # get max_seq_length in batch
    max_seq_len = max([len(x) for x in data])
    # pad text sequences based on max_seq_len
    data = [np.pad(p, (0, max_seq_len - len(p)), 'constant') for p in data]
    # convert data and target to tensor
    data = torch.LongTensor(data)
    target = torch.LongTensor(target)
    return [data, target]

We can now use this collate_fn with our Dataloader as:

train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=False, num_workers=10,collate_fn = collate_text)

for xb,yb in train_dataloader:
    print(xb.size(),yb.size())

It will work this time as we have provided a custom collate_fn. And see that the batches have different sequence lengths now. Thus we would be able to train our BiLSTM using variable input sizes just like we wanted.

Training a Neural Network

We know how to create a neural network using nn.Module. But how to train it? Any neural network that has to be trained will have a training loop that will look something similar to below:

num_epochs = 5
for epoch in range(num_epochs):
    # Set model to train mode
    model.train()
    for x_batch,y_batch in train_dataloader:
        # Clear gradients
        optimizer.zero_grad()
        # Forward pass - Predicted outputs
        pred = model(x_batch)
        # Find Loss and backpropagation of gradients
        loss = loss_criterion(pred, y_batch)
        loss.backward()
        # Update the parameters
        optimizer.step()
    model.eval()
    for x_batch,y_batch in valid_dataloader:
        pred = model(x_batch)
        val_loss = loss_criterion(pred, y_batch)

In the above code, we are running five epochs and in each epoch:

1. We iterate through the dataset using a data loader.

2. In each iteration, we do a forward pass using model(x_batch)

3. We calculate the Loss using a loss_criterion

4. We back-propagate that loss using loss.backward() call. We don’t have to worry about the calculation of the gradients at all, as this simple call does it all for us.

5. Take an optimizer step to change the weights in the whole network using optimizer.step(). This is where weights of the network get modified using the gradients calculated in loss.backward() call.

6. We go through the validation data loader to check the validation score/metrics. Before doing validation, we set the model to eval mode using model.eval().Please note we don’t back-propagate losses in eval mode.

Till now, we have talked about how to use nn.Module to create networks and how to use Custom Datasets and Dataloaders with Pytorch. So let’s talk about the various options available for Loss Functions and Optimizers.

Loss functions

Pytorch provides us with a variety of loss functions for our most common tasks, like Classification and Regression. Some most used examples are nn.CrossEntropyLoss , nn.NLLLoss , nn.KLDivLoss and nn.MSELoss . You can read the documentation of each loss function, but to explain how to use these loss functions, I will go through the example of nn.NLLLoss

The documentation for NLLLoss is pretty succinct. As in, this loss function is used for Multiclass classification, and based on the documentation:

• the input expected needs to be of size (batch_size x Num_Classes ) — These are the predictions from the Neural Network we have created.

• We need to have the log-probabilities of each class in the input — To get log-probabilities from a Neural Network, we can add a LogSoftmax Layer as the last layer of our network.

• The target needs to be a tensor of classes with class numbers in the range(0, C-1) where C is the number of classes.

So, we can try to use this Loss function for a simple classification network. Please note the LogSoftmax layer after the final linear layer. If you don’t want to use this LogSoftmax layer, you could have just used <code>nn.CrossEntropyLoss</code>

class myClassificationNet(nn.Module):
    def __init__(self):
        super().__init__()
        # Define all Layers Here
        self.lin = nn.Linear(784, 10)
        self.logsoftmax = nn.LogSoftmax(dim=1)
    def forward(self, x):
        # Connect the layer Outputs here to define the forward pass
        x = self.lin(x)
        x = self.logsoftmax(x)
        return x

Let’s define a random input to pass to our network to test it:

# some random input:

X = torch.randn(100,784)
y = torch.randint(low = 0,high = 10,size = (100,))

And pass it through the model to get predictions:

model = myClassificationNet()
preds = model(X)

We can now get the loss as:

criterion = nn.NLLLoss()
loss = criterion(preds,y)
loss
------------------------------------------
tensor(2.4852, grad_fn=<NllLossBackward>)

Custom Loss Function

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Let me admit something upfront → there is always a small, mildly annoying blocker before picking up a new tool.

That little <do I really want to learn yet another thing?> feeling — even when half of LinkedIn is screaming that you must.

I felt it with Claude Code. I had it installed on my machine for two full weeks before I actually sat down to use it. And when I finally did, the first hour was a lot of reading docs, clicking “yes” to prompts I did not understand, and wondering if this was genuinely going to pay off or if I had just given another AI tool access to my filesystem.

Here is the thing about Claude Code, though. The hello-world is easy.

You npm install, you type claude, you ask it to fix a bug, it fixes the bug. You feel clever. Then you look at the docs and you feel lost. Are you making the most out of it?

This is so confusing. Which of this actually matters? And which of it can you safely ignore for now?

I’ve been using Claude Code daily for months since that slow start, and in the first week I made every setup mistake I could. I clicked “yes” to permission prompts a thousand times before I realized acceptEdits and dangerously-skip-permissions existed. Racked up a $200 API bill before I discovered the Max plan can be used to login as well. Ended up starting forty-something terminal claude sessions which got lost as I didn’t know claude -c was a thing.

This post is the shortcut I wish someone had handed me in week one — an opinionated setup that gets you from “I installed it” to “I actually use this daily” in a weekend. The 20% of the surface area that delivers 80% of the value. Plus honest opinions on which features are worth your time and which ones you can skip.

Let’s dive in.

1. What Claude Code Actually Is (And What It Isn’t)

Before we install anything, let’s get the mental model right. This is the single most common reason people bounce off Claude Code in the first hour.

Claude Code is not a VS Code extension that autocompletes your code. It is not a chat sidebar. It is not Cursor or Copilot. Yes, it has a VS Code integration, but the integration is a thin window over a CLI tool that runs in your terminal.

What Claude Code actually is: an agentic CLI. You run claude in your terminal, you tell it what you want in plain English, and it then reads files, writes files, runs bash commands, executes your tests, browses the web, calls APIs, and keeps iterating until the task is done. You watch it work, you can interrupt at any point, you can steer.

Think of it less like autocomplete and more like pair-programming with a junior developer who types fast, never gets tired, sometimes goes off the rails, and occasionally needs a hard “no, do not do that.”

That loop above — shown in the diagram at the start of this section — is the whole product. Read, think, act, observe, repeat. Your job is to give it good context up front and steer when it drifts.

If this sounds familiar, it should — I wrote about my first dance with vibe coding using Claude Pro a while back. Claude Code is what happens when that same loop moves out of a chat window and into your actual terminal, with access to your actual files and your actual tests.

And it is not a small thing. As of February 2026, roughly 4% of all public commits on GitHub were authored by Claude Code — about 135,000 commits per day. Anthropic itself says 90% of their internal code is now AI-written. ServiceNow has 29,000 daily users on it. This is not a Twitter fad. This is real and this is here to stay.

The question is no longer “is this useful.” The question is how you go from “I installed it” to “I actually use it well.” Which is the rest of this post.

2. Install and Get Logged In

Installation is genuinely a one-liner now.

npm install -g @anthropic-ai/claude-code

You need Node 18 or newer. That is the whole prerequisite list.

Open a terminal and Run claude in any directory. The first time, it walks you through login. You get two choices: a claude.ai account (Pro or Max plan) or an API key from the Anthropic Console. Pick the first one if you are a human writing code daily. Pick the second one only if you have a specific reason — Bedrock, Vertex, headless CI, Corporate use and that kind of thing.

Pricing reality check: Max plan is $100 a month for individuals, $200 a month for teams. A typical 30-to-60-minute Claude Code session costs roughly $0.50 to $3.00 on the API. Do that math for yourself. If you are coding even a few hours a day, Max is the cheaper option, and it gives you Opus access without rate-limit anxiety.

If you want my full breakdown of how the AI subscriptions compare across coding, research, and general use, I broke it down here.

Once you are in, run /status to see which settings are active and how many tokens you have used up. We will come back to that command when things get weird.

3. CLAUDE.md — The One File You Have to Write

If you only learn one thing from this entire post, learn this.

CLAUDE.md is a markdown file at the root of your project. Every time you start Claude Code in that directory, the contents of this file get loaded into the conversation automatically. Think of this file as your project’s onboarding document — except the onboardee shows up every single session and reads it cover to cover.

This is where you tell Claude things like → what the project does, which command runs the tests, what the build tool is, conventions the team follows, paths it should never touch, and the gotchas.

Here is a real-world CLAUDE.md for a Python data science project, slightly trimmed:

Read more

🏆 Story of the Week: OpenAI Just Bought 10% of Cerebras

For two years, I’ve been hearing the same question in every infrastructure conversation: how does the Nvidia monopoly end?

This week, we got the answer. And it’s not what anyone predicted.

It wasn’t a DOJ antitrust case. It was a procurement contract — but a procurement contract structured like nothing we’ve ever seen in this industry.

Cerebras filed its S-1 on Friday. But buried inside was a deal between OpenAI and Cerebras that is pretty hard to believe.

Here’s what OpenAI committed to:

• $20+ billion in chip spending through 2028

• 750 MW of capacity, with an option to expand to 2 GW

• A $1 billion loan to OpenAI from Cerebras at 6% interest

In exchange, OpenAI got about 10% of Cerebras post-IPO.

Read that again. The customer got equity in the supplier. The customer also got a billion-dollar loan from the supplier.

OpenAI is now Cerebras’s biggest customer, biggest creditor, and one of its biggest shareholders. People are calling it “circular…

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Every tech revolution follows the same pattern. First, we solve the problem one way. Then, we realize we’ve been solving the wrong problem.

Natural language processing: we spent a decade on classification (sentiment, NER, QA as pick-the-right-answer).

Then GPT said: generation subsumes classification. Just generate the output.

Recommendation systems are having their moment right now. We spent years building the pipeline that would retrieve 10K candidates with Two-Tower, score 1K with a ranker, re-rank the top 100. But what if the recommender could just generate the next item directly? That’s where this series is headed.

But you can’t understand the generative revolution without understanding what came before it.

In this two-part post, we’ll trace the full evolution: how RNNs first cracked sequential recommendation, how Transformers took over, and ultimately how generative models are rewriting the rules entirely.

This is Part 1 — covering

• GRU4Rec (2016),

• SASRec (2018),

• the BERT4Rec controversy, and

• production deployment patterns.

Part 2 will cover Semantic IDs, TIGER, HSTU, and who’s deploying generative recommenders in production today.

Let’s dive in!

1. The Sequential Problem: Why Order Matters

Let’s say that you just finished watching Inception. Netflix recommends Interstellar. You watch it. Next up: Arrival.

As you can see this watching order is not random. This is not just “you like sci-fi” and so are watching sci-fi movies.

There’s a trajectory here. The recommender is following your path through Christopher Nolan’s mind-bending sci-fi catalog — what you watched second might change what you should see third.

In the first post of this series, we covered collaborative filtering, which treats user history as an unordered matrix — a bag of items. So, if you watched [Inception, Interstellar, Arrival], traditional CF treats that the same as [Arrival, Inception, Interstellar]. But the order you watched them in tells you something completely different about what to recommend next.

Sequential models fixed that. They learned to predict not just “what you might like” but “what comes next.”

Formalising the Problem

Given a sequence of items a user has interacted with:

[i₁, i₂, i₃, ..., iₙ]

Predict the next item: iₙ₊₁

This formulation applies across domains:

• E-commerce: product browsing → purchase prediction

• Streaming: watch history → next video

• Music: listening sequence → next song

• News: reading pattern → next article

The Benchmark: Steam Games Dataset

For this post, we’ll use the Steam Games dataset — a rich gaming interaction dataset from UCSD’s repository for building our models:

• 67,287 users (raw) → 56,808 users (after 5-core filtering)

• 32,133 games (raw) → 6,382 games (after 5-core filtering)

• 2,235,453 interactions (playtime > 1 hour)

• Average sequence length: 39.4 games (median: 26)

5-core filtering is a technique that removes all users and items with fewer than 5 interactions, applied iteratively until every remaining user and item has at least 5. It’s a standard preprocessing step in RecSys research to eliminate extreme cold-start cases (users who tried one game, games nobody played) that add noise without enough signal to learn from.

Now let’s see how different architectures tackle sequential prediction.

2. GRU4Rec — When RNNs Met Recommendations (2016)

In 2016, Gravity R&D published “Session-based Recommendations with Recurrent Neural Networks” at ICLR. First major work applying RNNs to sequential recommendation. It dominated the field for nearly two years.

Read more