The Filter

A Sparse Autoencoder decomposes embeddings into interpretable features. It works on any text, not just this dataset.

By Ian Johnson

From local to global

In the previous articles, we used tools that are local to a dataset. Each time we ran UMAP, we got a new projection. Each time we clustered, we got new groups. These are useful for exploring a specific collection, but they don't generalize. The clusters you find in dad jokes tell you nothing about a collection of news articles.

What if we could build a tool that decomposes any text into interpretable features? Not groups specific to this dataset, but a universal vocabulary of concepts that the embedding model recognizes?

This is what a Sparse Autoencoder (SAE) does. It learns to break an embedding into a small number of active features from a much larger set of possible features. Think of it like a polarizing filter on a microscope: it isolates specific wavelengths of meaning from the full signal.

Decomposing a joke

An embedding is a dense vector: all 384 dimensions have nonzero values, and none of them are individually interpretable. An SAE transforms this into a sparse vector: out of thousands of possible features, only a handful are active for any given text. And each feature has a human-readable label.

Pick a joke below and see its features:

Figure 1. SAE decomposition of a dad joke. Each bar is an active feature. The length shows activation strength: how strongly this feature responds to the text.

Look at how the features describe different aspects of the joke. A joke about atoms might activate "science and physics," "wordplay and puns," and "double meaning." These aren't clusters specific to the dad jokes dataset. They're features the SAE learned from a large, diverse training corpus. They apply to any text.

Shared features across jokes

The same feature can activate across very different texts. "Wordplay and puns" might light up for a joke about atoms, a joke about noodles, and a joke about bicycles. This is what makes SAE features useful as a taxonomy: a shared vocabulary for describing what the model sees in any text.

Compare two jokes below. Notice which features they share and which are unique to each:

Figure 2. Two jokes decomposed side by side. Features they share appear in both columns. Features unique to each joke reveal what makes them different in the model's eyes.

A global vocabulary

The crucial difference between SAE features and clusters is scope. Clusters are computed from a dataset and apply only to that dataset. SAE features are trained to match the embedding model itself. They work on any text the model can embed.

This is what we mean by calling the SAE a global tool. It provides a consistent set of concepts for describing what the model sees, regardless of what data you're looking at. If clustering is like sorting your desk into piles, the SAE is like having a filing system that works for any document.

Clustering asks: "what groups exist in this data?" SAE asks: "what concepts does the model recognize in this text?" The first is local. The second is global.

Together, these features form a taxonomy of what the embedding model understands. In the next article, we'll pair this taxonomy with a global map: a permanent coordinate system where every text has a consistent address.