Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

🏆 Story of the Week: The $35 Billion Deal That Turned AI Chips Into Toll Roads

Apollo, Blackstone, and a consortium of global banks closed the largest private financing in history to back Broadcom’s new AI XPV Platform. The target: 20+ gigawatts of AI compute capacity through 2028, with Anthropic and OpenAI as anchor customers.

Here’s why this matters more than anything released this week.

Until now, the AI compute buildout was constrained by four balance sheets: Google, Microsoft, Meta, and Amazon. The pace of AI infrastructure was gated by how fast those companies could allocate capital. Apollo just took their moat away.

Pension funds, insurance companies, and sovereign wealth can now finance AI compute the way they finance power plants and highways.

The Broadcom angle is the second story inside this deal. Broadcom CEO Hock Tan is running the VMware playbook: control the infrastructure layer, make it financeable, let private capital scale the deployment.

Money is no longer the binding constraint.

I think this is the week AI infrastructure crossed from a technology story to a financial one. Your cost of compute will probably drop as private capital floods in. Your cost of power and compliance will rise.

🤖 Models That Dropped This Week

Gemma 4 12B (Google) — The first encoder-free open multimodal model that runs on a laptop. No separate vision encoder, no CLIP adapter. Raw image patches flow directly into the transformer alongside text tokens. At 12B parameters with quantization, it fits on consumer GPUs. Download and test it. I was able to get it working with Ollama without a GPU, but was not able to get it working with Claude. Let me know if you are able to run Claude's code with this.

MiniMax M3 (MiniMax) — A Shanghai lab claiming 59.0% on SWE-Bench Pro (edging past GPT-5.5’s 58.6%), a 1M-token context window, and pricing at $0.60/$2.40 per 1M tokens. That’s 15x cheaper than Claude Opus on input. The MiniMax Sparse Attention architecture delivers 9x prefill speedup at 1M tokens. Benchmarks are vendor-reported and need verification.

MAI-Code-1-Flash (Microsoft) — Microsoft’s first in-house coding model, built without OpenAI data. 137B total / 5B active params via sparse MoE, 256K context. Claims +16 points over Claude Haiku 4.5 on SWE-Bench Pro. Priced at $0.75/M input tokens. Already rolling out in GitHub Copilot. The clearest signal yet of Microsoft-OpenAI decoupling.

🧠 Papers That Matter

Mult-DPO: Multinomial Direct Preference Optimization for Recommender Systems (Netflix) — I keep telling teams that LLM alignment techniques will reshape RecSys. Netflix just proved it. DPO works on pairwise preferences (”A is better than B”), but recommendation data is set-wise: multiple positive items, multiple negatives, no meaningful ordering among the positives. Forcing set-wise data into pairwise comparisons loses information on every training step.

Mult-DPO extends DPO to a multinomial formulation that handles sets directly. The model learns “all items in set S+ should outrank all items in S−” without imposing artificial order within the positives. If you’re training recommendation models with any flavor of pairwise loss, you’re leaving performance on the table. Benchmark against this.

📝 Some Good Reads

“When AI Builds Itself” (Anthropic Institute) — Anthropic published detailed research on recursive self-improvement mid-IPO, showing they’re already delegating a growing share of AI development to AI systems.

“Coding Is No Longer the Constraint” (Spotify Engineering) — Niklas Gustavsson at Spotify published the numbers: 99% of engineers use AI tools weekly, 650+ agent-generated PRs merged per month, 90% migration time reduction. The thesis: years of platform investment in CI/CD, testing, and docs now compound with AI agents. Companies that underinvest in dev platforms won’t benefit from AI coding.

⚡ Quick Hits

• Google is paying SpaceX $920M/month to rent GPUs — the company that builds TPUs can’t build capacity fast enough. $11B/year to a competitor.

• Anthropic’s Glasswing expanded to 200 organizations across 15+ countries. Mythos found 10,000+ high/critical vulnerabilities since April.

• SpaceX/xAI prices its IPO Thursday at a $1.75T target valuation — Totally unethical. Just to put SpaceX in the Nasdaq 100. I think markets are in “greed mode.”

• TurboVec hit #1 on GitHub trending — a Rust vector index fitting 10M documents in 4GB (vs. 31GB float32), beating FAISS by 12-20% on ARM, with no training step and filter-at-search-time. Need to look into this myself.

Subscribe now

Over the coming weeks, I’ll be writing more about GenAI, including topics like pre-training and post-training. This post is the second one of the foundational pieces meant to set up that series.

Transformers run most of modern NLP, but they’re still surprisingly hard to internalize from a diagram alone. In my last post, I walked through how they work — the encoder, decoder, and the data flow between them.

This post is where we stop reading and start building: an end-to-end English-to-German translator in PyTorch, written from scratch with a Transformer at its core. Because the fastest way to actually understand something is to implement it.

Task Description

We want to create a translator that uses transformers to convert English to German. So, if we look at it as a black box, our network takes as input an English sentence and returns a German sentence.

Data Preprocessing

To train our English-German translation Model, we will need translated sentence pairs between English and German.

Fortunately, there is a pretty standard way to get these with the OPUS-100 dataset (English-German subset), a curated multilingual translation corpus we can access via HuggingFace datasets.

Also, before we really get into the whole coding part, let us understand what we need as input and output to the model while training. We will actually need two matrices to be input to our Network:

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Over the coming weeks, I’ll be writing more about GenAI, including topics like pre-training and post-training. This post is one of the foundational pieces meant to set up that series.

Transformers have become the de facto standard for almost everything. Though the architecture was introduced for NLP, it now powers computer vision, recommender systems, and—most importantly—the entire wave of modern LLMs.

Yet for all their ubiquity, transformers remain as hard to understand as ever.

It has taken me multiple readings through the Google research paper that first introduced transformers, along with just so many blog posts, to really understand how a transformer works.

So, I thought of putting the whole idea down in as simple words as possible, and with some very basic Math and some puns, as I am a proponent of having some fun while learning. I will try to keep both the jargon and the technicality to a minimum, yet it is such a topic that I could only do so much. And my goal is to make the reader understand even the most gory details of Transformer by the end of this post.

Also, this is officially my longest post, both in terms of time taken to write it as well as the length of the post. Hence, I will advise you to Grab A Coffee. ☕️

Before we dive in, here’s the path we’ll walk together: we’ll start with the big picture of what a transformer even does, then crack open the encoder stack (attention, feed-forward, positional encodings, and those mysterious “Add & Norm” boxes). From there, we’ll move to the decoder stack and the masking trick that makes it tick, bolt on an output head to actually get our German words, and finish with how the whole thing is trained and how it makes predictions at test time. Long road, but I promise the view is worth it. Onwards.

Q: So, why should I even understand Transformer?

In the past, the LSTM and GRU architecture(as explained here in my past post on NLP), along with the attention mechanism, used to be the State of the Art Approach for Language modeling problems (put very simply, predict the next word) and Translation systems. But the main problem with these architectures is that they are recurrent in nature, and the runtime increases as the sequence length increases. That is, these architectures take a sentence and process each word in a sequential way, and hence, with the increase in sentence length, the whole runtime increases.

Transformer, a model architecture first explained in the paper Attention is all you need, lets go of this recurrence and instead relies entirely on an attention mechanism to draw global dependencies between input and output. And that makes it FAST.

This is the picture of the full transformer as taken from the paper. And, it surely is intimidating. So, I will aim to demystify it in this post by going through each piece. So read ahead.

The Big Picture

Q: That sounds interesting. So, what does a transformer do exactly?

Essentially, a transformer can perform almost any NLP task. It can be used for language modeling, Translation, or Classification as required, and it does it fast by removing the sequential nature of the problem. So, the transformer in a machine translation application would convert one language to another, or for a classification problem will provide the class probability using an appropriate output layer.

It all will depend on the final output layer for the network; the Transformer basic structure will remain quite the same for any task. For this particular post, I will be continuing with the machine translation example.

So, from a very high place, this is how the transformer looks for a translation task. It takes as input an English sentence and returns a German sentence.

The Building Blocks

Q: That was too basic. 😎 Can you expand on it?

Okay, just remember in the end, you asked for it. Let’s go a little deeper and try to understand what a transformer is composed of.

So, a transformer is essentially composed of a stack of encoder and decoder layers. The role of an encoder layer is to encode the English sentence into a numerical form using the attention mechanism, while the decoder aims to use the encoded information from the encoder layers to give the German translation for the particular English sentence.

In the figure below, the transformer is given an English sentence as input, which gets encoded using 6 encoder layers. The output from the final encoder layer then goes to each decoder layer to translate English to German.

1. Encoder Architecture

Q: That’s alright, but how does an encoder stack encode an English sentence exactly?

Patience, I am getting to it. So, as I said, the encoder stack contains six encoder layers on top of each other(As given in the paper, but the future versions of transformers use even more layers). And each encoder in the stack has essentially two main layers:

• a multi-head self-attention Layer, and

• a position-wise fully connected feed-forward network

They are a mouthful. Right? Don’t lose me yet as I will explain both of them in the coming sections. Right now, just remember that the encoder layer incorporates attention and a position-wise feed-forward network.

Q: But, how does this layer expect its inputs to be?

This layer expects its inputs to be of the shape SxD (as shown in the figure below) where S is the source sentence(English Sentence) length, and D is the dimension of the embedding whose weights can be trained with the network. In this post, we will be using D as 512 by default throughout. While S will be the maximum length of a sentence in a batch. So it normally changes with batches.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

This is Part 9d of the RecSys for MLEs series. Part 9c explained why HSTU works: the softmax-to-SiLU switch that preserves engagement intensity, relative attention bias that gives the model a sense of time, and M-FALCON for cheap candidate scoring. This post is the hands-on follow-up. We’re going to train an HSTU from scratch, on the MovieLens-1M dataset, and benchmark it against the rectools library’s reference HSTU implementation, so you have a real number to compare against.

Last week, I published the conceptual deep dive on HSTU. Within 24 hours, the most common question in my replies was the same: “OK, but how do I actually train one?”

Fair. So today we build it from scratch — the fused (item, action) input layer, all three HSTU sub-layers, the multi-task retrieval + rating heads, and the M-FALCON inference cache. We’ll train it on MovieLens-1M, then benchmark head-to-head against rectools’ reference HSTU implementation, so you have a real number to compare against.

We’ll use PyTorch 2.x on a single GPU (a free Colab T4 works fine). I’m intentionally keeping the model small (D=64, 2 layers) so it trains in few hours on free hardware. And to make sure I wasn’t fooling myself with vanity numbers, I trained the rectools library’s HSTU on the same data and split, then ran my from-scratch model with a similar training config. That way, if my numbers come out worse, I know exactly where to look.

Here’s what we’ll cover:

• The dataset: how MovieLens ratings map to the (item, action, time) triples HSTU consumes

• The fused input layer: item embedding + action embedding + fusion MLP

• The HSTU block: all three sub-layers as a single PyTorch module — SiLU attention, RAB, gated transformation

• Multi-task heads: retrieval (sampled softmax with cosine similarity + temperature) and rating prediction

• Results: HR@10 and NDCG@10 against rectools HSTU and SASRec, plus the SiLU vs softmax ablation

• Example predictions: what the model actually recommends for specific MovieLens users

• Retrieval and ranking demos: brute-force, FAISS, and ranking by retrieval score + predicted rating

• M-FALCON inference: the K/V caching trick that makes serving feasible

1. The dataset: MovieLens-1M ratings as (item, action, time) triples

One of the most basic but important questions that we need to answer is how the data is structured. We’re using MovieLens-1M from GroupLens which contains ~1M ratings across 6,040 users and 3,706 movies.

Each row is (user, movie, rating, timestamp) where rating is 1-5 stars. SASRec would treat every rating as one positive interaction. HSTU’s input can be richer — it can fuse the action type alongside the item ID — so I want to give the model the rating sentiment, not just the fact that the user watched the movie. This is how I’m incorporating an action signal in this model. Honestly, every setup can have a different action vocabulary — add-to-cart vs. purchase on an e-commerce store, click vs. like vs. share on a feed, watch-25% vs. watch-90% vs. skip on a video platform. The point I want to make here is that HSTU lets you encode whichever signal actually matters for your domain, not just “the user clicked this item.”

For our case, we create three behavioral signals, derived from the rating value:

• POSITIVE (rating ≥ 4): the user liked the movie

• NEUTRAL (rating = 3): the user was indifferent

• NEGATIVE (rating < 3): the user disliked the movie

The actual rating value (1-5) is kept separately as the label for the rating-prediction head.

import pandas as pd
import numpy as np

ratings = pd.read_csv(”ml-1m/ratings.dat”, sep=”::”, header=None,
                      names=[”user”, “item”, “rating”, “ts”], engine=”python”)

def rating_to_action(r):
    if r >= 4: return “POSITIVE”
    if r == 3: return “NEUTRAL”
    return “NEGATIVE”

events = ratings.copy()
events[”action”] = events[”rating”].apply(rating_to_action)
events[”value”]  = events[”rating”].astype(np.float32)

print(events.action.value_counts())
# POSITIVE    575281
# NEUTRAL     261197
# NEGATIVE    163731

Train/test split: leave-one-out. For each user, the last interaction goes to test, and everything before that goes to training. During training, the last item of the training sequence is held out as the validation target.

def split_seq(seq, max_len=200):
    n = len(seq[”items”])
    if n < 3: return None
    history_slice = slice(max(0, n - 1 - max_len), n - 1)
    history = {k: seq[k][history_slice].tolist() for k in seq}
    test_target = (
        int(seq[”items”][n - 1]),
        int(seq[”actions”][n - 1]),
        int(seq[”times”][n - 1]),
        float(seq[”values”][n - 1]),
    )
    return history, test_target

splits = [s for s in (split_seq(seq) for seq in sequences) if s is not None]
# Train/test sequences: 6,040

max_len=200 is the cap. With an average length of 165, most users will fit. Long-tail users get truncated to their most recent 200 events.

2. The fused input: item embedding + action embedding + fusion MLP

Here’s the actual code.

import torch
import torch.nn as nn
import torch.nn.functional as F

class FusedInputEmbedding(nn.Module):
    “”“Item embedding + action embedding, fused with an MLP into a single D-dim vector.”“”
    def __init__(self, num_items, num_actions, dim):
        super().__init__()
        self.item_emb   = nn.Embedding(num_items + 1, dim, padding_idx=0)
        self.action_emb = nn.Embedding(num_actions + 1, dim, padding_idx=0)
        self.fuse = nn.Sequential(
            nn.Linear(2 * dim, dim),
            nn.SiLU(),
            nn.Linear(dim, dim),
        )

    def forward(self, item_ids, action_ids):
        i = self.item_emb(item_ids)      # (B, T, D)
        a = self.action_emb(action_ids)  # (B, T, D)
        return self.fuse(torch.cat([i, a], dim=-1))  # (B, T, D)

What’s happening here is that each item ID looks up a D-dim vector in item_emb (the “what is this item” signal) table, and each action ID looks up another D-dim vector in action_emb (the “how did the user engage with it” signal).

The forward pass then concatenates them into a 2D-dim vector per token, then the fuse MLP projects back to D dims.

The MLP matters. It’s what lets the model learn that “watched Toy Story with rating 5” represents something different from “watched Toy Story with rating 1”. A simple sum or concatenation wouldn’t give the model the capacity to learn that interaction.

3. The HSTU block in PyTorch

Now, it is time to write the actual HSTU block. If you want to refresh on what each piece does, Part 9c walks through the three sub-layers conceptually. Here I’ll just translate that directly into code.

First, the relative attention bias (RAB) module. Two learnable tables: one for relative position offset, one for log-spaced time buckets.

class RelativeAttentionBias(nn.Module):
    “”“Learnable position + time biases, added to QK^T before SiLU.”“”
    TIME_BUCKETS = [
        (0, 3600),                  # 0-1 hour
        (3600, 86400),              # 1-24 hours
        (86400, 86400 * 7),         # 1-7 days
        (86400 * 7, 86400 * 30),    # 7-30 days
        (86400 * 30, float(”inf”)), # 30+ days
    ]

    def __init__(self, max_seq_len):
        super().__init__()
        self.max_seq_len = max_seq_len
        self.pos_bias  = nn.Embedding(2 * max_seq_len - 1, 1)
        self.time_bias = nn.Embedding(len(self.TIME_BUCKETS), 1)
        nn.init.zeros_(self.pos_bias.weight)
        nn.init.zeros_(self.time_bias.weight)

    def _bucket(self, time_deltas):
        out = torch.zeros_like(time_deltas, dtype=torch.long)
        for k, (lo, hi) in enumerate(self.TIME_BUCKETS):
            mask = (time_deltas >= lo) & (time_deltas < hi)
            out = torch.where(mask, torch.full_like(out, k), out)
        return out

    def forward(self, times):
        B, T = times.shape
        idx = torch.arange(T, device=times.device)
        rel_pos = (idx.view(T, 1) - idx.view(1, T)) + (self.max_seq_len - 1)
        pos_b = self.pos_bias(rel_pos).squeeze(-1)
        time_deltas = (times.unsqueeze(2) - times.unsqueeze(1)).abs()
        time_b = self.time_bias(self._bucket(time_deltas)).squeeze(-1)
        return pos_b.unsqueeze(0) + time_b

Initialize both bias tables to zeros. That way, the first forward pass behaves like a vanilla attention block, and the time/position biases learn from gradient flow.

Now the HSTU block itself:

class HSTUBlock(nn.Module):
    def __init__(self, dim, max_seq_len, dropout=0.2):
        super().__init__()
        self.linear_in  = nn.Linear(dim, 4 * dim)
        self.rab        = RelativeAttentionBias(max_seq_len)
        self.norm       = nn.LayerNorm(dim)
        self.linear_out = nn.Linear(dim, dim)
        self.dropout    = nn.Dropout(dropout)

    def forward(self, x, times, attn_mask):
        B, T, D = x.shape

        # --- Sub-layer 1: pointwise projection ---
        proj = F.silu(self.linear_in(x))
        K, Q, V, U = proj.chunk(4, dim=-1)

        # --- Sub-layer 2: spatial aggregation (SiLU attention + RAB + causal/pad mask) ---
        scores = torch.matmul(Q, K.transpose(-2, -1)) + self.rab(times)
        causal = torch.tril(torch.ones(T, T, device=x.device))
        scores = scores * causal
        pad_mask = attn_mask.unsqueeze(1).float()
        scores = scores * pad_mask
        activated = F.silu(scores)                     # NOT softmax — pointwise SiLU
        attn_out  = torch.matmul(activated, V) / T     # 1/T normalization

        # --- Sub-layer 3: gated transformation + residual ---
        gated  = self.norm(attn_out) * U
        output = self.dropout(self.linear_out(gated))
        return output + x

Stacking blocks is one line:

class HSTUEncoder(nn.Module):
    def __init__(self, num_items, num_actions, dim, num_layers, max_seq_len, dropout=0.2):
        super().__init__()
        self.embed  = FusedInputEmbedding(num_items, num_actions, dim)
        self.blocks = nn.ModuleList([
            HSTUBlock(dim, max_seq_len, dropout=dropout) for _ in range(num_layers)
        ])

    def forward(self, item_ids, action_ids, times, attn_mask):
        x = self.embed(item_ids, action_ids)
        for block in self.blocks:
            x = block(x, times, attn_mask)
        return x

That’s the encoder.

The rest of this post covers the multi-task heads (retrieval + rating), the full results table comparing our from-scratch HSTU to rectools HSTU and SASRec, the SiLU vs softmax ablation, the learned time-bias curve, M-FALCON inference benchmark (210× speedup), and a production field guide.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

This is Part 9c of the RecSys for MLEs series. In Part 9a, we built GRU4Rec and SASRec from scratch on the Steam Games dataset and got our hands dirty with sequential recommenders. In Part 9b, we covered Semantic IDs and TIGER, Google’s clever approach to generative retrieval, where item IDs become a learned token vocabulary. Now we close the arc with the architecture that’s actually running at Meta scale: HSTU.

Late 2022. You’re an ML engineer on a recommendation team, and you’ve just shipped a beefed-up SASRec model. The metrics look great. Your director walks over and says: “What if we just scaled this thing? More layers, bigger embeddings, a hundred billion parameters. Like the LLM folks are doing.”

So you do. And for a while, the model gets better. Then it stops getting better. Then you throw more compute at it and nothing. No improvement. A bigger, slower, more expensive model that performs about the same.

Frustrating, because over in NLP-land the scaling laws are clean: double the compute, get a predictably better model. GPT-3 had proven that. LLaMA was about to prove it again. DLRMs (Deep Learning Recommendation Models)? They kept plateauing. Something was fundamentally broken, and nobody could quite articulate what.

Meta’s answer was HSTU (Hierarchical Sequential Transduction Units).

Here’s what we’ll work through in this post:

• The three structural issues that quietly sabotage standard Transformers when you point them at recommendation data

• What HSTU actually consumes: the fused (item, action) input format, plus the actual training data schema (impression table + history table)

• Inside the HSTU block: three sub-layers, with the math, the code, the intuition, and the diagrams

• M-FALCON: the caching optimization that lets you score 10,000 candidates without recomputing 8,000 history events 10,000 times

By the end, you should understand exactly why HSTU works, how it works, and why every big tech RecSys team is suddenly building one.

1. Three reasons standard Transformers fail for RecSys

Three structural mismatches between language data and recommendation data make standard Transformers fail at scale. These are fundamental, not edge cases you can patch with a clever hack.

If you’d like a refresher on how standard Transformer attention works before going further, that post will get you up to speed. From here on, I’ll assume you’re comfortable with Q, K, V, softmax, and self-attention.

Problem 01: non-stationary vocabularies

In NLP, your vocabulary is fixed. “The” is always token 42. “Recommendation” is always token 18,973. The model trains with a stable set of possible next tokens, and that set doesn’t really change between training and serving.

In recommendations, your “vocabulary” is the entire item catalog, and that catalog is constantly changing. New videos go live every second on Reels. A trending creator didn’t exist yesterday. The model you trained last Tuesday has never seen most of the items it’s being asked to score on a Friday.

Softmax was designed for a fixed vocabulary with stable class boundaries. It’s defined over a set of mutually exclusive classes that sum to 1. When the set of possible “next tokens” is a moving target, when items appear and disappear from the catalog every minute, the softmax assumption starts to silently misbehave. The probability mass keeps getting redistributed among the items the model happens to have seen, which is not the same set as the items it needs to recommend.

This is a much bigger deal than it sounds. Almost every metric you care about (CTR, watch time, retention) depends on the model surfacing new items that the user will love. If your normalization assumes a stable world, you’re starting from a broken assumption.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

I love keeping track of everything week to week — here's what happened this week. Enjoy this free weekly post! For those who want to dive deeper into any of these topics, that's what my paid posts are for.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

At the end of Part 1, I left you with a question: What if items had semantic identifiers that captured their content?

That question sounds innocent enough, but it’s actually the hinge point for the entire generative recommender revolution we are going to be talking about in this post. And it is very interesting to say the least.

So, we built GRU4Rec and SASRec on the Steam Games dataset in our previous post. But both those models treat items as arbitrary integers. The model sees “Item 4,271 → Item 8,903 → Item 2,156” and learns some sort of statistical patterns between these numbers.

The thing we need to note is that everything about what these games actually are — their genre, their developer, their visual style, the reason a player moves from one to the next — lives entirely outside the model. That is a pretty big opportunity to work on.

Now imagine if every item carried an ID that meant something. Similar games naturally share ID prefixes. A brand-new title gets a meaningful ID the moment it’s published — before anyone plays it. The recommender can reason about games it’s never seen just from their ID, just like a human would by seeing a game’s description or title.

That’s Semantic IDs.

And these semantic IDs unlock something bigger →instead of scoring candidates from a retrieved shortlist(which is how SASRec and GRU4Rec work), a model can now generate the next item token by token, the way GPT generates words.

In this post, we will create a complete pipeline to do exactly that on the same Steam dataset, every game compressed into meaningful tokens using Residual Quantized VAE, and a generative recommender(TIGER) trained from scratch.

Let’s dive in!

1. The Problem with Random Item IDs

Just think about how SASRec works for a second - A user plays [Counter-Strike, Portal, Half-Life 2]. The model looks up the embedding for each game, runs self-attention, and predicts the next game. So far, so good.

But what does the model actually know about Counter-Strike? Nothing. It’s just item ID 4,271. The model has no idea it’s a first-person shooter made by Valve in 2004.

This creates three problems that compound in production:

1. No knowledge sharing. If 10,000 users play Counter-Strike → Team Fortress 2, the model learns that specific transition. But it learns nothing about why they’re related. A new Valve FPS arrives tomorrow, and the model has zero signal for it — even though any human could tell you “people who like Counter-Strike would probably like this.”

2. Cold-start is brutal. New items have brand-new IDs with randomly initialized embeddings. As I covered in my post on the cold-start problem, this means they need thousands of interactions before the model can meaningfully recommend them. In fast-moving catalogs — think news, short videos, new game releases — items can go stale before the model even learns to recommend them.

3. No generalization. The model memorizes specific ID→ID transitions. It can’t reason about categories of items, properties of items, or relationships between items.

And that’s where semantic IDs could help us.

2. Semantic IDs: Making Item IDs mean Something

Ok, so now we understand why Semantic IDs might be beneficial, let’s see how they are made. The idea behind Semantic IDs was introduced in TIGER (Transformer Index for Generative Recommenders) at NeurIPS 2023, and it completely changed how we think about item representation in recommender systems.

The pipeline works in three steps. Here’s what happens to a single game:

In step 1, we start by getting a content-based embedding of an item using a powerful text encoder.

Once we have that, we run the RQVAE algorithm in step 2 to generate Semantic IDs of items. (Don’t worry, we will talk about this in this post. For now, just understand that it gives you some sort of discrete token-based vector given your item embeddings)

In Step 3, we generate the next item’s Semantic ID token by token, the same way GPT writes one word at a time. It’s not scoring a fixed list; it’s constructing an answer.

Now think about what this structure gives you. Counter-Strike and Half-Life might both get the prefix [42, 187, ...] because they’re both Valve FPS games. A brand-new Valve FPS that launched this morning might then be assigned a similar prefix based purely on its content — and the model already knows what to do with items that start with [42, 187, ...], even though it’s never seen a single player interact with this game.

Remember how in the cold start post I wrote about how new items have nothing but their metadata to work with? With Semantic IDs, the metadata becomes their ID. The model doesn’t need thousands of interactions to figure out what a new game is — it already knows, just from reading the ID.

And notice what else changed: we replaced the entire retrieve→rank→rerank pipeline with a single model that generates recommendations directly. At inference time, the decoder uses beam search: instead of greedily committing to one token at each level and hoping for the best, it keeps the top B candidates at each step and extends them all in parallel. You end up with a ranked list of complete Semantic IDs, each mapping to a real item. That means you don’t have to worry about maintaining an ANN index, a candidate retrieval stage, and a separate ranker stage. You can get all of this from a single model.

Now, the magic of this pipeline lives entirely in Step 2 — the RQVAE. That is how we create these semantic IDs. And if the quantization is bad, the Semantic IDs are meaningless, and we’re back to square one. So let’s understand exactly how it works.

3. How RQVAE Works — The Engine Behind Semantic IDs

Residual Quantized VAE sounds intimidating, but the intuition is straightforward. It’s progressive compression — like describing a location with increasing precision.

“North America” tells you the continent. “California” narrows it down. “San Francisco” pins it to a city. Each level adds detail that the previous level didn’t capture. RQVAE does exactly this, but for item embeddings.

Here’s the algorithm.

Step 1: Encode. Take the 1024-dim item embedding and compress it through an encoder network (1024 → 512 → 256 → 128 → 32). It’s easier to follow with concrete numbers, so let’s walk through a simplified example — we will use 4 dimensions instead of 32, but the math is identical.

Encoded: [0.8, -0.3, 0.5, 0.1]

Step 2: Level 1 Quantization. You have a codebook — a table of 256 learned vectors. Find the one closest to your latent vector:

Read more

Story of the Week: The Great Unbundling

The Microsoft–OpenAI partnership came apart Sunday night. Revenue sharing, the AGI clause, and exclusivity all dissolved in a single restructuring.

Twenty-four hours later, Sam Altman and AWS CEO Matt Garman were on Stratechery announcing OpenAI models on Amazon Bedrock. The speed tells you the deal was pre-negotiated; only the Microsoft contract was holding it back.

The practical change for ML teams: GPT models now run natively on Bedrock alongside Claude, Llama, and Mistral. If you standardized on AWS but routed around for OpenAI access, that workaround is gone. If you picked Azure specifically for OpenAI, you have a real alternative for the first time. Multi-cloud routing for frontier models is table stakes now — expect inference prices to drop and cross-cloud model-parity to start mattering in vendor evaluations.

Then both labs made a stranger move. Anthropic and OpenAI launched enterprise consulting arms on the same day, aimed at different mar…

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Read more

PyTorch has sort of became one of the de facto standards for creating Neural Networks now, and I love its interface. Yet, it is somehow a little difficult for beginners to get a hold of.

I remember picking PyTorch up only after some extensive experimentation a couple of years back. To tell you the truth, it took me a lot of time to pick it up but am I glad that I moved from Keras to PyTorch . With its high customizability and pythonic syntax,PyTorch is just a joy to work with, and I would recommend it to anyone who wants to do some heavy lifting with Deep Learning.

So, in this PyTorch guide, I will try to ease some of the pain with PyTorch for starters and go through some of the most important classes and modules that you will require while creating any Neural Network with Pytorch.

But, that is not to say that this is aimed at beginners only as I will also talk about the high customizability PyTorch provides and will talk about custom Layers, Datasets, Dataloaders, and Loss functions.

So let’s get some coffee ☕ ️and start it up.

Tensors

Tensors are the basic building blocks in PyTorch and put very simply, they are NumPy arrays but on GPU. In this part, I will list down some of the most used operations we can use while working with Tensors. This is by no means an exhaustive list of operations you can do with Tensors, but it is helpful to understand what tensors are before going towards the more exciting parts.

1. Create a Tensor

We can create a PyTorch tensor in multiple ways. This includes converting to tensor from a NumPy array. Below is just a small gist with some examples to start with, but you can do a whole lot of more things with tensors just like you can do with NumPy arrays.

# Using torch.Tensor
t = torch.Tensor([[1,2,3],[3,4,5]])
print(f"Created Tensor Using torch.Tensor:\n{t}")

# Using torch.randn
t = torch.randn(3, 5)
print(f"Created Tensor Using torch.randn:\n{t}")

# using torch.[ones|zeros](*size)
t = torch.ones(3, 5)
print(f"Created Tensor Using torch.ones:\n{t}")
t = torch.zeros(3, 5)
print(f"Created Tensor Using torch.zeros:\n{t}")

# using torch.randint - a tensor of size 4,5 with entries between 0 and 10(excluded)
t = torch.randint(low = 0,high = 10,size = (4,5))
print(f"Created Tensor Using torch.randint:\n{t}")

# Using from_numpy to convert from Numpy Array to Tensor
a = np.array([[1,2,3],[3,4,5]])
t = torch.from_numpy(a)
print(f"Convert to Tensor From Numpy Array:\n{t}")

# Using .numpy() to convert from Tensor to Numpy array
t = t.numpy()
print(f"Convert to Numpy Array From Tensor:\n{t}")

2. Tensor Operations

Again, there are a lot of operations you can do on these tensors. The full list of functions can be found here .

A = torch.randn(3,4)
W = torch.randn(4,2)
# Multiply Matrix A and W
t = A.mm(W)
print(f"Created Tensor t by Multiplying A and W:\n{t}")
# Transpose Tensor t
t = t.t()
print(f"Transpose of Tensor t:\n{t}")
# Square each element of t
t = t**2
print(f"Square each element of Tensor t:\n{t}")
# return the size of a tensor
print(f"Size of Tensor t using .size():\n{t.size()}")

Note: What are PyTorch Variables? In the previous versions of Pytorch, Tensor and Variables used to be different and provided different functionality, but now the Variable API is deprecated , and all methods for variables work with Tensors. So, if you don’t know about them, it’s fine as they re not needed, and if you know them, you can forget about them.

The nn.Module

Here comes the fun part as we are now going to talk about some of the most used constructs in Pytorch while creating deep learning projects. nn.Module lets you create your Deep Learning models as a class. You can inherit from nn.Moduleto define any model as a class. Every model class necessarily contains an __init__ procedure block and a block for the forward pass.

• In the __init__ part, the user can define all the layers the network is going to have but doesn’t yet define how those layers would be connected to each other.

• In the forward pass block, the user defines how data flows from one layer to another inside the network.

So, put simply, any network we define will look like:

class myNeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        # Define all Layers Here
        self.lin1 = nn.Linear(784, 30)
        self.lin2 = nn.Linear(30, 10)
    def forward(self, x):
        # Connect the layer Outputs here to define the forward pass
        x = self.lin1(x)
        x = self.lin2(x)
        return x

Here we have defined a very simple Network that takes an input of size 784 and passes it through two linear layers in a sequential manner. But the thing to note is that we can define any sort of calculation while defining the forward pass, and that makes PyTorch highly customizable for research purposes. For example, in our crazy experimentation mode, we might have used the below network where we arbitrarily attach our layers. Here we send back the output from the second linear layer back again to the first one after adding the input to it(skip connection) back again(I honestly don’t know what that will do).

class myCrazyNeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        # Define all Layers Here
        self.lin1 = nn.Linear(784, 30)
        self.lin2 = nn.Linear(30, 784)
        self.lin3 = nn.Linear(30, 10)

    def forward(self, x):
        # Connect the layer Outputs here to define the forward pass
        x_lin1 = self.lin1(x)
        x_lin2 = x + self.lin2(x_lin1)
        x_lin2 = self.lin1(x_lin2)
        x = self.lin3(x_lin2)
        return x

We can also check if the neural network forward pass works. I usually do that by first creating some random input and just passing that through the network I have created.

x = torch.randn((100,784))
model = myCrazyNeuralNet()
model(x).size()
--------------------------
torch.Size([100, 10])

A word about Layers

Pytorch is pretty powerful, and you can actually create any new experimental layer by yourself using nn.Module. For example, rather than using the predefined Linear Layer nn.Linear from Pytorch above, we could have created our custom linear layer.

class myCustomLinearLayer(nn.Module):
    def __init__(self,in_size,out_size):
        super().__init__()
        self.weights = nn.Parameter(torch.randn(in_size, out_size))
        self.bias = nn.Parameter(torch.zeros(out_size))
    def forward(self, x):
        return x.mm(self.weights) + self.bias

You can see how we wrap our weights tensor in nn.Parameter. This is done to make the tensor to be considered as a model parameter. From PyTorch docs :

Parameters are <code>*Tensor*</code> subclasses, that have a very special property when used with Module - when they’re assigned as Module attributes they are automatically added to the list of its parameters, and will appear in parameters() iterator

As you will later see, the model.parameters() iterator will be an input to the optimizer. But more on that later.

Right now, we can now use this custom layer in any PyTorch network, just like any other layer.

class myCustomNeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        # Define all Layers Here
        self.lin1 = myCustomLinearLayer(784,10)

    def forward(self, x):
        # Connect the layer Outputs here to define the forward pass
        x = self.lin1(x)
        return x
x = torch.randn((100,784))
model = myCustomNeuralNet()
model(x).size()
------------------------------------------
torch.Size([100, 10])

But then again, Pytorch would not be so widely used if it didn’t provide a lot of ready to made layers used very frequently in wide varieties of Neural Network architectures. Some examples are: nn.Linear , nn.Conv2d , nn.MaxPool2d , nn.ReLU , nn.BatchNorm2d , nn.Dropout , nn.Embedding , nn.GRU / nn.LSTM , nn.Softmax , nn.LogSoftmax , nn.MultiheadAttention , nn.TransformerEncoder , nn.TransformerDecoder

I have linked all the layers to their source where you could read all about them, but to show how I usually try to understand a layer and read the docs, I would try to look at a very simple convolutional layer here.

So, a Conv2d Layer needs as input an Image of height H and width W, with Cin channels. Now, for the first layer in a convnet, the number of in_channels would be 3(RGB), and the number of out_channels can be defined by the user. The kernel_size mostly used is 3x3, and the stride normally used is 1.

To check a new layer which I don’t know much about, I usually try to see the input as well as output for the layer like below where I would first initialize the layer:

conv_layer = nn.Conv2d(in_channels = 3, out_channels = 64, kernel_size = (3,3), stride = 1, padding=1)

And then pass some random input through it. Here 100 is the batch size.

x = torch.randn((100,3,24,24))
conv_layer(x).size()
--------------------------------
torch.Size([100, 64, 24, 24])

So, we get the output from the convolution operation as required, and I have sufficient information on how to use this layer in any Neural Network I design.

Datasets and DataLoaders

How would we pass data to our Neural nets while training or while testing? We can definitely pass tensors as we have done above, but Pytorch also provides us with pre-built Datasets to make it easier for us to pass data to our neural nets. You can check out the complete list of datasets provided at torchvision.datasets and torchtext.datasets . But, to give a concrete example for datasets, let’s say we had to pass images to an Image Neural net using a folder which has images in this structure:

data
    train
        sailboat
        kayak
        .
        .

We can use torchvision.datasets.ImageFolder dataset to get an example image like below:

from torchvision import transforms
from torchvision.datasets import ImageFolder
traindir = "data/train/"
t = transforms.Compose([
        transforms.Resize(size=256),
    transforms.CenterCrop(size=224),
        transforms.ToTensor()])
train_dataset = ImageFolder(root=traindir,transform=t)
print("Num Images in Dataset:", len(train_dataset))
print("Example Image and Label:", train_dataset[2])

This dataset has 847 images, and we can get an image and its label using an index. Now we can pass images one by one to any image neural network using a for loop:

for i in range(0,len(train_dataset)):
    image ,label = train_dataset[i]
    pred = model(image)

But that is not optimal. We want to do batching. We can actually write some more code to append images and labels in a batch and then pass it to the Neural network. But Pytorch provides us with a utility iterator torch.utils.data.DataLoader to do precisely that. Now we can simply wrap our train_dataset in the Dataloader, and we will get batches instead of individual examples.

train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=True, num_workers=10)

We can simply iterate with batches using:

for image_batch, label_batch in train_dataloader:
    print(image_batch.size(),label_batch.size())
    break
-------------------------------------------------
torch.Size([64, 3, 224, 224]) torch.Size([64])

So actually, the whole process of using datasets and Dataloaders becomes:

t = transforms.Compose([
        transforms.Resize(size=256),
    transforms.CenterCrop(size=224),
        transforms.ToTensor()])

train_dataset = torchvision.datasets.ImageFolder(root=traindir,transform=t)
train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=True, num_workers=10)

for image_batch, label_batch in train_dataloader:
    pred = myImageNeuralNet(image_batch)

You can look at this particular example in action in my previous blogpost on Image classification using Deep Learning here .

This is great, and Pytorch does provide a lot of functionality out of the box. But the main power of Pytorch comes with its immense customization. We can also create our own custom datasets if the datasets provided by PyTorch don’t fit our use case.

Understanding Custom Datasets

To write our custom datasets, we can make use of the abstract class torch.utils.data.Dataset provided by Pytorch. We need to inherit this Dataset class and need to define two methods to create a custom Dataset.

• __len__ : a function that returns the size of the dataset. This one is pretty simple to write in most cases.

• __getitem__: a function that takes as input an index i and returns the sample at index i.

For example, we can create a simple custom dataset that returns an image and a label from a folder. See that most of the tasks are happening in __init__ part where we use glob.glob to get image names and do some general preprocessing.

from glob import glob
from PIL import Image
from torch.utils.data import Dataset

class customImageFolderDataset(Dataset):
    """Custom Image Loader dataset."""
    def __init__(self, root, transform=None):
        """
        Args:
            root (string): Path to the images organized in a particular folder structure.
            transform: Any Pytorch transform to be applied
        """
        # Get all image paths from a directory
        self.image_paths = glob(f"{root}/*/*")
        # Get the labels from the image paths
        self.labels = [x.split("/")[-2] for x in self.image_paths]
        # Create a dictionary mapping each label to a index from 0 to len(classes).
        self.label_to_idx = {x:i for i,x in enumerate(set(self.labels))}
        self.transform = transform

    def __len__(self):
        # return length of dataset
        return len(self.image_paths)

    def __getitem__(self, idx):
        # open and send one image and label
        img_name = self.image_paths[idx]
        label = self.labels[idx]
        image = Image.open(img_name)
        if self.transform:
            image = self.transform(image)
        return image,self.label_to_idx[label]

Also, note that we open our images one at a time in the __getitem__ method and not while initializing. This is not done in __init__ because we don’t want to load all our images in the memory and just need to load the required ones.

We can now use this dataset with the utility Dataloader just like before. It works just like the previous dataset provided by PyTorch but without some utility functions.

t = transforms.Compose([
        transforms.Resize(size=256),
    transforms.CenterCrop(size=224),
        transforms.ToTensor()])

train_dataset = customImageFolderDataset(root=traindir,transform=t)
train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=True, num_workers=10)

for image_batch, label_batch in train_dataloader:
    pred = myImageNeuralNet(image_batch)

Understanding Custom DataLoaders

This particular section is a little advanced and can be skipped going through this post as it will not be needed in a lot of situations. But I am adding it for completeness here.

So let’s say you are looking to provide batches to a network that processes text input, and the network could take sequences with any sequence size as long as the size remains constant in the batch. For example, we can have a BiLSTM network that can process sequences of any length. It’s alright if you don’t understand the layers used in it right now; just know that it can process sequences with variable sizes.

class BiLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden_size = 64
        drp = 0.1
        max_features, embed_size = 10000,300
        self.embedding = nn.Embedding(max_features, embed_size)
        self.lstm = nn.LSTM(embed_size, self.hidden_size, bidirectional=True, batch_first=True)
        self.linear = nn.Linear(self.hidden_size*4 , 64)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(drp)
        self.out = nn.Linear(64, 1)


    def forward(self, x):
        h_embedding = self.embedding(x)
        h_embedding = torch.squeeze(torch.unsqueeze(h_embedding, 0))

        h_lstm, _ = self.lstm(h_embedding)
        avg_pool = torch.mean(h_lstm, 1)
        max_pool, _ = torch.max(h_lstm, 1)
        conc = torch.cat(( avg_pool, max_pool), 1)
        conc = self.relu(self.linear(conc))
        conc = self.dropout(conc)
        out = self.out(conc)
        return out

This network expects its input to be of shape (batch_size, seq_length) and works with any seq_length. We can check this by passing our model two random batches with different sequence lengths(10 and 25).

model = BiLSTM()
input_batch_1 = torch.randint(low = 0,high = 10000, size = (100,**10**))
input_batch_2 = torch.randint(low = 0,high = 10000, size = (100,**25**))
print(model(input_batch_1).size())
print(model(input_batch_2).size())
------------------------------------------------------------------
torch.Size([100, 1])
torch.Size([100, 1])

Now, we want to provide tight batches to this model, such that each batch has the same sequence length based on the max sequence length in the batch to minimize padding. This has an added benefit of making the neural net run faster. It was, in fact, one of the methods used in the winning submission of the Quora Insincere challenge in Kaggle, where running time was of utmost importance.

So, how do we do this? Let’s write a very simple custom dataset class first.

class CustomTextDataset(Dataset):
    '''
    Simple Dataset initializes with X and y vectors
    We start by sorting our X and y vectors by sequence lengths
    '''
    def __init__(self,X,y=None):
        self.data = list(zip(X,y))
        # Sort by length of first element in tuple
        self.data = sorted(self.data, key=lambda x: len(x[0]))

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]

Also, let’s generate some random data which we will use with this custom Dataset.

import numpy as np
train_data_size = 1024
sizes = np.random.randint(low=50,high=300,size=(train_data_size,))
X = [np.random.randint(0,10000, (sizes[i])) for i in range(train_data_size)]
y = np.random.rand(train_data_size).round()
#checking one example in dataset
print((X[0],y[0]))

Example of one random sequence and label. Each integer in the sequence corresponds to a word in the sentence.

We can use the custom dataset now using:

train_dataset = CustomTextDataset(X,y)

If we now try to use the Dataloader on this dataset with batch_size>1, we will get an error. Why is that?

train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=False, num_workers=10)
for xb,yb in train_dataloader:
    print(xb.size(),yb.size())

This happens because the sequences have different lengths, and our data loader expects our sequences of the same length. Remember that in the previous image example, we resized all images to size 224 using the transforms, so we didn’t face this error.

So, how do we iterate through this dataset so that each batch has sequences with the same length, but different batches may have different sequence lengths?

We can use collate_fn parameter in the DataLoader that lets us define how to stack sequences in a particular batch. To use this, we need to define a function that takes as input a batch and returns (x_batch, y_batch ) with padded sequence lengths based on max_sequence_length in the batch. The functions I have used in the below function are simple NumPy operations. Also, the function is properly commented so you can understand what is happening.

def collate_text(batch):
    # get text sequences in batch
    data = [item[0] for item in batch]
    # get labels in batch
    target = [item[1] for item in batch]
    # get max_seq_length in batch
    max_seq_len = max([len(x) for x in data])
    # pad text sequences based on max_seq_len
    data = [np.pad(p, (0, max_seq_len - len(p)), 'constant') for p in data]
    # convert data and target to tensor
    data = torch.LongTensor(data)
    target = torch.LongTensor(target)
    return [data, target]

We can now use this collate_fn with our Dataloader as:

train_dataloader = DataLoader(train_dataset,batch_size = 64, shuffle=False, num_workers=10,collate_fn = collate_text)

for xb,yb in train_dataloader:
    print(xb.size(),yb.size())

It will work this time as we have provided a custom collate_fn. And see that the batches have different sequence lengths now. Thus we would be able to train our BiLSTM using variable input sizes just like we wanted.

Training a Neural Network

We know how to create a neural network using nn.Module. But how to train it? Any neural network that has to be trained will have a training loop that will look something similar to below:

num_epochs = 5
for epoch in range(num_epochs):
    # Set model to train mode
    model.train()
    for x_batch,y_batch in train_dataloader:
        # Clear gradients
        optimizer.zero_grad()
        # Forward pass - Predicted outputs
        pred = model(x_batch)
        # Find Loss and backpropagation of gradients
        loss = loss_criterion(pred, y_batch)
        loss.backward()
        # Update the parameters
        optimizer.step()
    model.eval()
    for x_batch,y_batch in valid_dataloader:
        pred = model(x_batch)
        val_loss = loss_criterion(pred, y_batch)

In the above code, we are running five epochs and in each epoch:

1. We iterate through the dataset using a data loader.

2. In each iteration, we do a forward pass using model(x_batch)

3. We calculate the Loss using a loss_criterion

4. We back-propagate that loss using loss.backward() call. We don’t have to worry about the calculation of the gradients at all, as this simple call does it all for us.

5. Take an optimizer step to change the weights in the whole network using optimizer.step(). This is where weights of the network get modified using the gradients calculated in loss.backward() call.

6. We go through the validation data loader to check the validation score/metrics. Before doing validation, we set the model to eval mode using model.eval().Please note we don’t back-propagate losses in eval mode.

Till now, we have talked about how to use nn.Module to create networks and how to use Custom Datasets and Dataloaders with Pytorch. So let’s talk about the various options available for Loss Functions and Optimizers.

Loss functions

Pytorch provides us with a variety of loss functions for our most common tasks, like Classification and Regression. Some most used examples are nn.CrossEntropyLoss , nn.NLLLoss , nn.KLDivLoss and nn.MSELoss . You can read the documentation of each loss function, but to explain how to use these loss functions, I will go through the example of nn.NLLLoss

The documentation for NLLLoss is pretty succinct. As in, this loss function is used for Multiclass classification, and based on the documentation:

• the input expected needs to be of size (batch_size x Num_Classes ) — These are the predictions from the Neural Network we have created.

• We need to have the log-probabilities of each class in the input — To get log-probabilities from a Neural Network, we can add a LogSoftmax Layer as the last layer of our network.

• The target needs to be a tensor of classes with class numbers in the range(0, C-1) where C is the number of classes.

So, we can try to use this Loss function for a simple classification network. Please note the LogSoftmax layer after the final linear layer. If you don’t want to use this LogSoftmax layer, you could have just used <code>nn.CrossEntropyLoss</code>

class myClassificationNet(nn.Module):
    def __init__(self):
        super().__init__()
        # Define all Layers Here
        self.lin = nn.Linear(784, 10)
        self.logsoftmax = nn.LogSoftmax(dim=1)
    def forward(self, x):
        # Connect the layer Outputs here to define the forward pass
        x = self.lin(x)
        x = self.logsoftmax(x)
        return x

Let’s define a random input to pass to our network to test it:

# some random input:

X = torch.randn(100,784)
y = torch.randint(low = 0,high = 10,size = (100,))

And pass it through the model to get predictions:

model = myClassificationNet()
preds = model(X)

We can now get the loss as:

criterion = nn.NLLLoss()
loss = criterion(preds,y)
loss
------------------------------------------
tensor(2.4852, grad_fn=<NllLossBackward>)

Custom Loss Function

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Let me admit something upfront → there is always a small, mildly annoying blocker before picking up a new tool.

That little <do I really want to learn yet another thing?> feeling — even when half of LinkedIn is screaming that you must.

I felt it with Claude Code. I had it installed on my machine for two full weeks before I actually sat down to use it. And when I finally did, the first hour was a lot of reading docs, clicking “yes” to prompts I did not understand, and wondering if this was genuinely going to pay off or if I had just given another AI tool access to my filesystem.

Here is the thing about Claude Code, though. The hello-world is easy.

You npm install, you type claude, you ask it to fix a bug, it fixes the bug. You feel clever. Then you look at the docs and you feel lost. Are you making the most out of it?

This is so confusing. Which of this actually matters? And which of it can you safely ignore for now?

I’ve been using Claude Code daily for months since that slow start, and in the first week I made every setup mistake I could. I clicked “yes” to permission prompts a thousand times before I realized acceptEdits and dangerously-skip-permissions existed. Racked up a $200 API bill before I discovered the Max plan can be used to login as well. Ended up starting forty-something terminal claude sessions which got lost as I didn’t know claude -c was a thing.

This post is the shortcut I wish someone had handed me in week one — an opinionated setup that gets you from “I installed it” to “I actually use this daily” in a weekend. The 20% of the surface area that delivers 80% of the value. Plus honest opinions on which features are worth your time and which ones you can skip.

Let’s dive in.

1. What Claude Code Actually Is (And What It Isn’t)

Before we install anything, let’s get the mental model right. This is the single most common reason people bounce off Claude Code in the first hour.

Claude Code is not a VS Code extension that autocompletes your code. It is not a chat sidebar. It is not Cursor or Copilot. Yes, it has a VS Code integration, but the integration is a thin window over a CLI tool that runs in your terminal.

What Claude Code actually is: an agentic CLI. You run claude in your terminal, you tell it what you want in plain English, and it then reads files, writes files, runs bash commands, executes your tests, browses the web, calls APIs, and keeps iterating until the task is done. You watch it work, you can interrupt at any point, you can steer.

Think of it less like autocomplete and more like pair-programming with a junior developer who types fast, never gets tired, sometimes goes off the rails, and occasionally needs a hard “no, do not do that.”

That loop above — shown in the diagram at the start of this section — is the whole product. Read, think, act, observe, repeat. Your job is to give it good context up front and steer when it drifts.

If this sounds familiar, it should — I wrote about my first dance with vibe coding using Claude Pro a while back. Claude Code is what happens when that same loop moves out of a chat window and into your actual terminal, with access to your actual files and your actual tests.

And it is not a small thing. As of February 2026, roughly 4% of all public commits on GitHub were authored by Claude Code — about 135,000 commits per day. Anthropic itself says 90% of their internal code is now AI-written. ServiceNow has 29,000 daily users on it. This is not a Twitter fad. This is real and this is here to stay.

The question is no longer “is this useful.” The question is how you go from “I installed it” to “I actually use it well.” Which is the rest of this post.

2. Install and Get Logged In

Installation is genuinely a one-liner now.

npm install -g @anthropic-ai/claude-code

You need Node 18 or newer. That is the whole prerequisite list.

Open a terminal and Run claude in any directory. The first time, it walks you through login. You get two choices: a claude.ai account (Pro or Max plan) or an API key from the Anthropic Console. Pick the first one if you are a human writing code daily. Pick the second one only if you have a specific reason — Bedrock, Vertex, headless CI, Corporate use and that kind of thing.

Pricing reality check: Max plan is $100 a month for individuals, $200 a month for teams. A typical 30-to-60-minute Claude Code session costs roughly $0.50 to $3.00 on the API. Do that math for yourself. If you are coding even a few hours a day, Max is the cheaper option, and it gives you Opus access without rate-limit anxiety.

If you want my full breakdown of how the AI subscriptions compare across coding, research, and general use, I broke it down here.

Once you are in, run /status to see which settings are active and how many tokens you have used up. We will come back to that command when things get weird.

3. CLAUDE.md — The One File You Have to Write

If you only learn one thing from this entire post, learn this.

CLAUDE.md is a markdown file at the root of your project. Every time you start Claude Code in that directory, the contents of this file get loaded into the conversation automatically. Think of this file as your project’s onboarding document — except the onboardee shows up every single session and reads it cover to cover.

This is where you tell Claude things like → what the project does, which command runs the tests, what the build tool is, conventions the team follows, paths it should never touch, and the gotchas.

Here is a real-world CLAUDE.md for a Python data science project, slightly trimmed:

Read more

🏆 Story of the Week: OpenAI Just Bought 10% of Cerebras

For two years, I’ve been hearing the same question in every infrastructure conversation: how does the Nvidia monopoly end?

This week, we got the answer. And it’s not what anyone predicted.

It wasn’t a DOJ antitrust case. It was a procurement contract — but a procurement contract structured like nothing we’ve ever seen in this industry.

Cerebras filed its S-1 on Friday. But buried inside was a deal between OpenAI and Cerebras that is pretty hard to believe.

Here’s what OpenAI committed to:

• $20+ billion in chip spending through 2028

• 750 MW of capacity, with an option to expand to 2 GW

• A $1 billion loan to OpenAI from Cerebras at 6% interest

In exchange, OpenAI got about 10% of Cerebras post-IPO.

Read that again. The customer got equity in the supplier. The customer also got a billion-dollar loan from the supplier.

OpenAI is now Cerebras’s biggest customer, biggest creditor, and one of its biggest shareholders. People are calling it “circular…

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Every tech revolution follows the same pattern. First, we solve the problem one way. Then, we realize we’ve been solving the wrong problem.

Natural language processing: we spent a decade on classification (sentiment, NER, QA as pick-the-right-answer).

Then GPT said: generation subsumes classification. Just generate the output.

Recommendation systems are having their moment right now. We spent years building the pipeline that would retrieve 10K candidates with Two-Tower, score 1K with a ranker, re-rank the top 100. But what if the recommender could just generate the next item directly? That’s where this series is headed.

But you can’t understand the generative revolution without understanding what came before it.

In this two-part post, we’ll trace the full evolution: how RNNs first cracked sequential recommendation, how Transformers took over, and ultimately how generative models are rewriting the rules entirely.

This is Part 1 — covering

• GRU4Rec (2016),

• SASRec (2018),

• the BERT4Rec controversy, and

• production deployment patterns.

Part 2 will cover Semantic IDs, TIGER, HSTU, and who’s deploying generative recommenders in production today.

Let’s dive in!

1. The Sequential Problem: Why Order Matters

Let’s say that you just finished watching Inception. Netflix recommends Interstellar. You watch it. Next up: Arrival.

As you can see this watching order is not random. This is not just “you like sci-fi” and so are watching sci-fi movies.

There’s a trajectory here. The recommender is following your path through Christopher Nolan’s mind-bending sci-fi catalog — what you watched second might change what you should see third.

In the first post of this series, we covered collaborative filtering, which treats user history as an unordered matrix — a bag of items. So, if you watched [Inception, Interstellar, Arrival], traditional CF treats that the same as [Arrival, Inception, Interstellar]. But the order you watched them in tells you something completely different about what to recommend next.

Sequential models fixed that. They learned to predict not just “what you might like” but “what comes next.”

Formalising the Problem

Given a sequence of items a user has interacted with:

[i₁, i₂, i₃, ..., iₙ]

Predict the next item: iₙ₊₁

This formulation applies across domains:

• E-commerce: product browsing → purchase prediction

• Streaming: watch history → next video

• Music: listening sequence → next song

• News: reading pattern → next article

The Benchmark: Steam Games Dataset

For this post, we’ll use the Steam Games dataset — a rich gaming interaction dataset from UCSD’s repository for building our models:

• 67,287 users (raw) → 56,808 users (after 5-core filtering)

• 32,133 games (raw) → 6,382 games (after 5-core filtering)

• 2,235,453 interactions (playtime > 1 hour)

• Average sequence length: 39.4 games (median: 26)

5-core filtering is a technique that removes all users and items with fewer than 5 interactions, applied iteratively until every remaining user and item has at least 5. It’s a standard preprocessing step in RecSys research to eliminate extreme cold-start cases (users who tried one game, games nobody played) that add noise without enough signal to learn from.

Now let’s see how different architectures tackle sequential prediction.

2. GRU4Rec — When RNNs Met Recommendations (2016)

In 2016, Gravity R&D published “Session-based Recommendations with Recurrent Neural Networks” at ICLR. First major work applying RNNs to sequential recommendation. It dominated the field for nearly two years.

Read more

🏆 Story of the Week: The AI Stack Fragments — From Silicon to Society

This was the week the AI industry’s single-vendor era officially ended. Not in one dramatic announcement, but through a cascade of moves at every layer of the stack that collectively redrew the competitive map.

Start at the bottom: Intel surged 4.2% on Monday after confirming its participation in Elon Musk’s Terafab project — the first serious attempt at a domestic US AI chip fab. The same day, reports confirmed that DeepSeek is building V4 entirely on Huawei Ascend 950PR chips, fully decoupling from Nvidia. And Broadcom jumped 6.1% on expanded TPU deals with both Google and Anthropic. In a single session, the market priced in four distinct AI chip supply chains: Nvidia/TSMC (incumbent), Google/Broadcom TPUs, Intel/Terafab (domestic US), and Huawei/Ascend (China). The telling number: Nvidia fell 1.6% while the rest of the AI chip ecosystem rallied. SEMI data confirmed the investment cycle is real — global chip equipm…

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

Here’s something they don’t teach you in ML courses:

A perfectly relevant recommendation list is usually a terrible one.

You spend months training a ranking model. Features, architectures, multi-task objectives — the works. Then the product team walks in: “Can you make sure we don’t show 5 horror movies in a row? And boost new releases? Oh, and reserve slot 3 for promoted content.”

Each request costs you relevance. The question isn’t whether to spend — it’s how much.

Think of it as a budget. Your ranking model gives you relevance scores for every item. Re-ranking is the art of spending that relevance wisely — trading some accuracy for diversity, freshness, fairness, and business value.

This is Part 8 of the RecSys for MLEs series. We’ve covered the fundamentals, the evolution from CF to deep learning, the 3-stage funnel where I first introduced re-ranking as the “business layer,” two-tower retrieval, vector search, the ranking layer, and the cold start problem.

Today, we’re opening up that final layer. Here’s what we’ll cover:

• The Set Problem → Why sorting by relevance produces bad recommendations

• Diversity → From dedup rules to Determinantal Point Processes (YouTube’s production system)

• Calibration → Matching your recommendations to the user’s taste distribution

• Freshness → Getting new content into the feed without wrecking relevance

• Business Constraints → The product rules that shape the final feed

• Multi-Objective Re-Ranking → Combining everything: scalarization, constraints, and 2D layouts

• The Practitioner’s Playbook → When to use what, and the pitfalls that trip everyone up

Let’s dive in!

1. Why Re-Ranking Exists — The Set Problem

Your ranking model scores items independently. Item A gets 0.92. Item B gets 0.89. Item C gets 0.87. Sort descending. Done.

Except it’s not done. Because when you look at your top-10 list, items A, B, and C are all psychological thrillers from the same director. Items D through G are also thrillers. The model did exactly what you asked — it found the most relevant items. But the resulting set is terrible.

Here’s how to think about it. Ranking answers: “How relevant is this item to this user?” Re-ranking answers a harder question: “What’s the best collection of items to show this user?”

The input to re-ranking is typically 100-500 scored items from your ranker. The output is the final 10-50 items in their display order. And the constraints are everything your ranking model doesn’t know about: diversity requirements, content freshness, promotional obligations, fairness targets, and a dozen product-specific rules.

I remember a team meeting where someone pulled up our top-10 list for a test user: ten nearly identical sci-fi action movies. “The model is working perfectly,” someone said. Technically correct — and completely useless. The top-10 wasn’t a recommendation; it was a redundancy report.

Netflix does this at massive scale — 15,000+ shows, nearly 300 million users, and a homepage that needs to feel both personally relevant and excitingly diverse. Their page construction system doesn’t just rank shows; it considers the composition of each row and the relationships between rows.

Here’s the key mental model I want you to hold for this entire post: re-ranking is spending a relevance budget. Your ranking model gives you a relevance score for each item. That score is currency. Every diversity constraint, every freshness boost, every business rule costs some of that relevance. The art is deciding how much to spend on each.

Let’s look at the algorithms that make this possible.

2. Diversity — From Rules to Determinantal Point Processes

Diversity is the most visible re-ranking objective. When a user sees 10 items from the same genre, something has clearly gone wrong. But “add diversity” is easy to say and surprisingly hard to get right. Three levels of sophistication:

Level 1: Rule-Based Dedup

The simplest approach is just writing rules: - “No more than 2 items from the same category in the top 5” - “No two items from the same creator in a row” - “At least 1 item from ‘trending’ in top 3”

Before YouTube deployed their DPP system(we will talk about this), they used exactly these kinds of heuristics: fuzzy deduplication (removing items too similar to ones already selected) and sliding window constraints (at most n out of every m items from the same type).

Rules are fast, interpretable, and easy to debug. But they’re also brittle. They can’t capture nuanced notions of similarity — “these are both thrillers” is a rule; “these have similar emotional arcs” is not. And they compose badly: stack 5 rules on top of each other and you’ll find they frequently conflict.

Level 2: Maximal Marginal Relevance (MMR)

MMR is the first real algorithmic approach to diversity. It was originally proposed for document retrieval, but it maps perfectly to recommendations.

The idea is beautifully simple. Instead of selecting items by relevance alone, you select greedily: at each step, pick the item that best balances relevance with dissimilarity to items you’ve already selected.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

def mmr_rerank(relevance_scores, item_embeddings, lambda_param=0.5, top_k=10):
    “”“
    Maximal Marginal Relevance re-ranking.

    Args:
        relevance_scores: array of shape (N,) — ranking model scores
        item_embeddings: array of shape (N, d) — item feature vectors
        lambda_param: trade-off between relevance (1.0) and diversity (0.0)
        top_k: number of items to select

    Returns:
        selected: list of indices in selection order
    “”“
    n_items = len(relevance_scores)
    sim_matrix = cosine_similarity(item_embeddings)

    selected = []
    candidates = list(range(n_items))

    for _ in range(top_k):
        best_score = -np.inf
        best_idx = None

        for idx in candidates:
            # Relevance term
            rel = relevance_scores[idx]

            # Max similarity to any already-selected item
            if selected:
                max_sim = max(sim_matrix[idx][s] for s in selected)
            else:
                max_sim = 0

            # MMR score: balance relevance vs. novelty
            score = lambda_param * rel - (1 - lambda_param) * max_sim

            if score > best_score:
                best_score = score
                best_idx = idx

        selected.append(best_idx)
        candidates.remove(best_idx)

    return selected

The lambda_param is your knob. At λ=1.0, MMR is pure relevance (no diversity). At λ=0.0, it’s pure diversity (ignores relevance). In practice, values between 0.5 and 0.7 work well.

MMR’s complexity is O(Nk) per selection, which is fast. But it has a fundamental limitation: it’s myopic. At each step, it only compares the candidate to items already selected. It never evaluates the global quality of the final set.

Level 3: Determinantal Point Processes (DPP)

This is where things get interesting.

A DPP is a probabilistic model that assigns higher probability to subsets of items that are both high-quality AND diverse. Unlike MMR’s pairwise comparisons, a DPP evaluates the entire subset at once.

Here’s the intuition: Imagine each item as an arrow in a high-dimensional space. The arrow’s length represents quality (the ranking model’s score). The arrow’s direction represents the item’s characteristics (its embedding). A DPP selects the set of arrows that spans the maximum volume — you want arrows that are both long (high quality) AND point in different directions (diverse).

Mathematically, we define a kernel matrix L where each entry captures both quality and similarity:

L[i,j] = q_i × q_j × similarity(i,j)

where q_i is item i’s quality score (from your ranker) and similarity(i,j) is the cosine similarity between item embeddings. The probability of selecting a subset S is proportional to det(L_S) — the determinant of the submatrix formed by those items, which is exactly the volume of the parallelogram those item vectors span.

That’s abstract. Let me walk through it with three movies.

Read more

Hey, Rahul here! 👋 Each week, I publish long-form ML+AI posts covering ML, AI, and System design for MLwhiz. Paid subscribers also get how-to guides with full code walkthroughs. I publish occasional extra articles. If you’d like to become a paid subscriber, here’s a button for that:

Subscribe now

A user signs up for your streaming platform. They’ve never watched anything. They’ve never rated anything. They’ve never even scrolled. And your recommendation engine — the same engine that serves 200 million personalized feeds per day — stares at this blank profile and essentially says: “I have no idea who you are.”

This is the Cold Start Problem, and I’ve been fighting it for the better part of four years — at Meta, where new creators needed to find their audience from day one, and in the streaming world, where every new user expects a personalized experience the moment they log in. It’s the problem that’s been discussed on HackerNews since 2010, has a 400-page book written about it (Andrew Chen’s The Cold Start Problem), and STILL doesn’t have a clean answer.

This is the next installment in my RecSys series. We’ve covered the algorithmic evolution of recommendation systems, built two-tower retrieval from scratch, dissected the ranking layer — all assuming we have user data to work with. Today we drop that assumption entirely.

Here’s what we’ll cover:

• The 3 types of cold start — they’re different problems with different solutions

• Classical approaches — the baselines everyone ships first, and where they hit a ceiling

• 3 modern frontiers: contextual bandits, meta-learning (MAML, prototypical networks, CMML), and LLMs (feature extraction, reasoning, data generation)

• How Spotify, TikTok, and YouTube actually solve this in production — with specific engineering details

• A decision framework — so you know which approach fits your system, your data, and your budget

This is meant to be the definitive practitioner’s guide. Let’s dive in!

The Three Faces of Cold Start

Before we jump into solutions, let’s be precise about what we’re solving. “Cold start” isn’t one problem — it’s three distinct problems, and confusing them is one of the most common mistakes I see engineers make.

New User Cold Start

A user signs up. Zero watch history. Zero ratings. Zero clicks. Your collaborative filtering model — the one that works beautifully for your 50 million existing users — is completely blind. It relies on the user-item interaction matrix R where entry R(u, i) represents user u‘s interaction with item i. For a new user u_new, the entire row R(u_new, :) is empty — a zero vector.

This means every technique that depends on finding similar users (nearest-neighbor CF), decomposing the interaction matrix (matrix factorization), or learning user embeddings from behavior (deep learning approaches) — the entire algorithmic evolution we covered in Part 2 of this series — has literally nothing to work with. The new user is a point with no coordinates in preference space.

Mathematically, collaborative filtering predicts a rating as:

r̂(u, i) = μ + q_i^T · p_u

where p_u is the user’s latent factor vector and q_i is the item’s latent factor vector (Koren et al., 2009, “Matrix Factorization Techniques for Recommender Systems”, IEEE Computer). For a new user, p_u is undefined — there are no interactions to learn it from. You can initialize it randomly, but then your predictions are random noise.

New Item Cold Start

A video gets uploaded. A product gets listed. A song gets released. No one has interacted with it yet. Even if your model is phenomenal at scoring items with behavioral data, this item has zero behavioral signal — no clicks, no watches, no purchases. Its column R(:, i_new) in the interaction matrix is all zeros.

This creates a vicious cycle that I’ve seen destroy content platforms: the item is invisible to your retrieval-ranking pipeline because it has no engagement data. Because it’s invisible, it gets no exposure. Because it gets no exposure, it accumulates no engagement data. The item is trapped in a black hole of non-existence.

This isn’t an abstract concern — it directly affects creator retention on any content platform. If a creator uploads a show and it gets zero impressions for a week, that creator doesn’t come back. And losing the creator means losing not just that show, but everything they would have made in the future.

New System Cold Start

You’re launching a recommendation system from scratch. No users with behavioral data, no items with engagement history, no interaction matrix at all. R is entirely empty. This is the rarest variant, but it’s also the one that every startup and every new product line faces.

Here’s the uncomfortable truth that most blog posts skip: in production, the new item problem is often harder and more damaging than the new user problem. New users at least have some context you can exploit (device, location, time). New items have nothing but their own metadata. And the business cost of item-side cold start — creator churn, catalog invisibility, content deserts — compounds far faster than user-side cold start.

There’s also a regime between cold and warm that’s arguably even more important in practice: warm start — when you have 1-5 interactions. Not zero, but not enough for your models to be confident. This is where your system spends most of its time for the long tail of users and items, and it’s where the modern approaches we’ll cover really shine.

Classical Solutions (And Why They’re Not Enough)

Every recommendation system starts here. These are the baseline approaches — they work, they ship fast, and they’re better than showing nothing. But they all hit a ceiling, and understanding exactly where that ceiling is tells you when to invest in the modern approaches.

Popularity-Based Ranking

The simplest possible move: show new users whatever is trending right now. It’s the “most popular dish” approach — safe, zero personalization required, trivial to implement. You’re essentially replacing the personalized score r̂(u, i) with the global popularity score:

score(i) = Σ_u R(u, i)  or  score(i) = count(clicks on i in last 24h)

The obvious problem: you’ll never discover that this specific user hates action movies and loves documentaries. Everyone gets the same feed, and you learn nothing about individual preferences. It also creates a rich-get-richer feedback loop — popular items get shown more, get more clicks, become more popular. This is the Matthew Effect in recommendation systems(rich get richer), and it’s brutal for new content.

That said, popularity-based ranking has one underappreciated strength: it surfaces items that are currently relevant. A user might not know they want to watch the Oscar-nominated film that just released, but a time-decayed popularity score will surface it naturally.

Content-Based Fallback

Instead of using behavioral signals (which don’t exist for cold-start entities), use the item’s features directly. A movie’s genre, director, cast, plot keywords, runtime, year, language — these are all available from day one, before anyone watches it.

The basic approach: represent each item as a feature vector f_i (using TF-IDF, one-hot encoding, or pretrained embeddings), represent the user as a weighted average of the features of items they’ve interacted with:

p_u = (1/|I_u|) Σ_{i ∈ I_u} w_i · f_i

where I_u are the items user interacted with
w_i is weight for item (can be 1 for basic cases, or can be a time decay)

Then score new items by cosine similarity:

score(u, i) = cos(p_u, f_i) = (p_u · f_i) / (||p_u|| · ||f_i||)

This is exactly how content-based and collaborative filtering differ at their core. Content-based doesn’t need the interaction matrix — it needs item features and at least some user signal.

The catch: this works far better for new items than for new users. A new item has features you can compute similarity against. A new user with zero interactions has no profile vector p_u at all — you can’t compute an average of an empty set. You’d need at least one click to get started.

Demographic Heuristics

Use whatever you get at signup: geo-location, device type, language, age bracket, operating system. A user signing up from Tokyo on an iPhone at 11 PM likely has different preferences than someone from Texas on a smart TV at 2 PM on a Saturday.

Formally, you’re replacing the missing behavioral profile with a demographic profile:

p_u = g(demographics_u)

where g is a learned function (often a simple lookup table or a shallow neural network) that maps demographic features to the same embedding space as your warm users. You train g on your existing warm users — learning, for example, that users aged 25-34 in urban Japan tend to prefer anime and J-drama.

The obvious limitation: demographics are a coarse proxy for user preferences. Not every 30-year-old in Tokyo likes the same content. You’re fighting the stereotype problem — making assumptions about individuals based on group statistics. But in the zero-data regime, coarse is better than random.

Onboarding Surveys

“Choose 3 genres you like.” “Rate these 5 movies.” “Pick your favorite artists.” Direct, explicit preference signal that bypasses the cold-start chicken-and-egg entirely.

The catch? Every additional question increases signup friction and hurts conversion. Research from the Baymard Institute shows that each additional step beyond 3-4 in a signup flow increases abandonment significantly — and streaming onboarding is no exception. And users lie — or more precisely, they pick aspirationally rather than truthfully. (A user saying ”Yes, I definitely want to watch cerebral documentaries about climate change” might proceed to binge Love Island for 6 hours.)

There’s a rich literature on optimal onboarding question selection. Golbandi et al. (2011, “Adaptive Bootstrapping of Recommender Systems Using Decision Trees”, WSDM) showed you can use decision trees to pick the maximally informative items to show in an onboarding survey — items where the user’s response tells you the most about their latent preferences.

Hybrid Switching

The textbook answer: start with content-based or popularity-based recommendations, gradually switch to collaborative filtering as behavioral data accumulates. Formally:

r̂(u, i) = α(u) · r̂_PB(u, i) + (1 - α(u)) · r̂_CF(u, i)

where α(u) is a function of how much data you have for user u — close to 1 for cold users (trust popularity-based), close to 0 for warm users (trust collaborative filtering).

Sounds clean in a blog post but it could be incredibly messy in production. Here’s why:

• The blending weight α(u) needs a functional form. Is it a step function (hard cutover at 10 interactions)? Sigmoid? Linear ramp? Each choice creates different user experiences, and there’s no universal right answer — you have to tune it per domain.

• Score calibration is a nightmare. Content-based or popularity scores and CF scores live on completely different scales and distributions. Naively adding them produces garbage — you need score normalization (min-max? z-score? rank-based?) that itself requires careful calibration.

• The transition can jar users. A user at α=0.6 today and α=0.3 tomorrow might see a completely different feed. Without smoothing, users experience sudden recommendation “personality shifts” that erode trust.

• You’re running two serving pipelines. Two models to train, two feature stores to maintain, two sets of latency budgets. The operational complexity of production recommendation systems is one of the most underappreciated challenges — and hybrid switching doubles it.

The Ceiling

All of these classical approaches are basically guessing. Educated guessing, sure — but still guessing. They don’t actively try to learn about the user. They wait passively for data to trickle in and hope the user sticks around long enough. That’s fine for day one, maybe week one. But if your cold-start strategy is still “show popular stuff and pray” after a month, you’re leaving massive value on the table.

Here’s a simple Python sketch of the popularity + content-based fallback:

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

def cold_start_recommend(user, items, item_features, popular_items, n=10):
    “”“Simple cold-start fallback: content-based if we have ANY
    signal, popularity otherwise.”“”

    if user.interactions:  # even 1 click gives us something
        # Build user profile from interacted item features
        profile = np.mean(
            [item_features[i] for i in user.interactions], axis=0
        )
        scores = cosine_similarity([profile], item_features)[0]
        top_items = np.argsort(scores)[::-1][:n]
        return [items[i] for i in top_items]

    # Zero interactions → fall back to popularity
    return popular_items[:n]

This is fine for day one. But the three modern approaches that follow are what separate a good recommendation system from a great one.

The rest of this post covers the three modern frontiers — contextual bandits, meta-learning, and LLMs — plus how Spotify, TikTok, and YouTube solve cold start in production, and a decision framework for choosing the right approach. Subscribe to continue reading.

Read more

🏆 Story of the Week: The Agent Platform War Has Officially Started

For years, the AI industry has been in a model quality race. GPT vs Claude vs Gemini, benchmark after benchmark, parameter count after parameter count. This week, that race ended — and a new one began.

On Sunday, OpenAI’s CEO of Applications Fidji Simo announced what insiders are calling “Code Red”: the consolidation of ChatGPT, the Codex coding platform, and the Atlas browser into a single desktop superapp built around agentic task handling. The catalyst? Internal data showing Anthropic’s enterprise market share climbing to 40% while OpenAI’s fell to roughly 27%. Simo told employees they could no longer afford “side quests” — a direct shot at Sora, which briefly hit #1 in the App Store before usage flatlined.

But this isn’t just an OpenAI crisis story — it’s an industry-wide convergence. Within the same week, Meta shipped “My Computer”, a desktop agent from its $2B Manus acquisition, already integrated into Meta Ads Man…

Read more