Interpretability and Feature Manifolds

How can we understand what is going on inside an AI system? Today, frontier labs test large language models for dangerous behaviour by monitoring their chain of thought and also looking into their “internals.”

What does that mean? A cartoon picture to have in mind: in an LLM, the words of the prompt are broken into smaller parts (tokens), and each token is represented by a vector in a large vector space with thousands of dimensions. These vectors, called activations, are transformed layer by layer: each layer adds an update to a token's vector. At the final layer, the vector at the last position is compared, via dot products, with a fixed family of unembedding vectors, one per token, and turned into a probability distribution via the softmax function (exponentiate and renormalize). Finally, sampling from this distribution produces the next token. So we can think about the contents of an LLM's mind as collections of vector spaces separated by layers.

These vector spaces of activations have surprising geometric properties that we can use to understand how an LLM is thinking.

The Linear Representation Hypothesis

A key example is the linear representation hypothesis: the affine structure of these vector spaces encodes important information about their semantic meaning. A classical illustration from the earlier word-embedding literature is king − man + woman ≈ queen. Importantly, in LLMs one can project the activations at any layer back onto tokens. This is built in at the final layer, but can be done approximately at every layer using the logit lens or tuned lens, and one finds analogous identities such as rouge − red + green ≈ vert.

The linear representation hypothesis has become foundational to mechanistic interpretability techniques such as linear probes and sparse autoencoders (SAEs). For example, SAEs are used to identify models that have a hidden objective or produce harmful responses. In the latter case, SAEs can identify a certain “misaligned persona” vector, and it is possible to steer a model toward or away from misalignment along that direction.

Feature Manifolds

But these methods have serious limitations because activation space also contains a great deal of structure that is not linear. One example is feature manifolds: activations lying on a low-dimensional submanifold with a meaningful interpretation. Asked “what day is 5 days after Friday?” or “what month is 6 months after November?”, a model represents days of the week or months of the year on a circle and solves the modular-arithmetic problem by rotating it. Similarly, when deciding where to break a line in fixed-width text, Claude 3.5 Haiku tracks the running character count on a curved one-dimensional manifold, which attention heads then twist to compare against the line width. When models are trained to perform group operations, irreducible representations of the group appear in their activation spaces.

Nonlinear structure can be stranger still: transformers trained to predict hidden Markov processes encode the Bayesian belief state of an optimal predictor as a Sierpiński-triangle-like fractal, linearly embedded in activation space.

A key open problem in the field is to develop systematic methods for discovering these nonlinear geometric structures, understanding how they form and how models use them, and turning these insights into scalable interpretability tools.