The word "curiosity" does a lot of work in AI research. It refers, depending on the paper, to count-based exploration bonuses, prediction-error-based rewards, information-gain objectives, novelty signals, and learning-progress measures. These approaches share a family resemblance — they all attempt to formalise the intuition that an agent should seek out informative experience — but they differ substantially in what they actually measure and what kinds of exploration behaviour they produce.
We have spent a good deal of time working through the taxonomy, and our view is that most formalisations of intrinsic motivation underspecify what "informative" means. They reward novelty, or surprise, or prediction error — all of which are proxies for something, but proxies that diverge from the underlying intuition in important and sometimes damaging ways. The formalisation that matters most, in our view, is grounded in a different concept: prediction complexity.
The early approaches and their limitations
Count-based exploration is the oldest and most theoretically grounded approach. In a fully observable, finite state space, an agent that gives decreasing bonus to revisiting already-seen states will eventually explore every reachable state. The visit-count approach has clean convergence guarantees in small tabular environments. Its limitation is obvious: in high-dimensional environments with continuous state spaces, counts are useless — every state is visited at most once, so the count signal provides no gradient.
The pseudo-count approach, which attempts to estimate visit frequency from the density of states under a learned model, provides a path beyond tabular environments but introduces the usual difficulties of density estimation in high dimensions — and, crucially, couples the exploration signal to the quality of the density model in ways that are hard to control.
Prediction-error-based intrinsic reward defines bonus as the error of a forward model predicting the next state given the current state and action. The agent is rewarded for being in states where its predictions are wrong. The intuition is that high prediction error marks the frontier of the agent's model — states where more learning can happen. This is broadly right, but it fails in a specific way: stochastic environments. In a setting where state transitions have genuine randomness — TV static, noisy physical processes — prediction error is permanently high, not because the agent hasn't learned the dynamics, but because the dynamics are inherently unpredictable. The agent is rewarded for watching TV static forever. This is the "noisy TV problem", and it points at a fundamental issue with raw prediction error as an exploration signal.
Distinguishing learnable and unlearnable complexity
The noisy TV problem reveals that what matters is not how much prediction error remains, but how much prediction error is reducible by further learning. A stochastic outcome has high prediction error that cannot be reduced by any amount of observation. A structured but unfamiliar outcome has high prediction error that will reduce as the agent accumulates experience. These two situations feel very different from the inside of a learning system, and they demand very different responses.
The concept of learning progress, developed in various forms in the intrinsic motivation literature, attempts to capture this distinction. Rather than rewarding current prediction error, it rewards the rate of decrease of prediction error over time — the improvement in model quality as the agent observes a region of state space. A region that is truly random produces no learning progress even though it maintains high prediction error; a region that is structured and learnable produces high learning progress initially, which decreases as the model converges.
This is closer to the right concept, but it requires estimating a derivative — the rate of change of model accuracy — which is harder than estimating a quantity. In practice, learning-progress signals are noisy and require careful implementation to avoid pathological behaviour.
Prediction complexity as a grounding concept
We have been investigating what we call prediction complexity as the more fundamental quantity underlying useful intrinsic motivation. Informally: prediction complexity asks not "how wrong is my model?" or "how fast is my model improving?" but "how complex is the generative process that produced this observation, measured relative to what my current model can represent?"
This framing connects intrinsic motivation to ideas from algorithmic information theory. An observation is interesting — worth exploring further — not when it is unpredictable in absolute terms, but when it requires expanding the agent's representational capacity to be fully described. Randomness does not require representational expansion; it is already maximally described by the distribution from which it is drawn. Structure that the agent's model cannot yet capture does require expansion — it is incompressible not because it is random but because the agent lacks the right abstractions.
In practice, measuring prediction complexity exactly is computationally intractable. But there are useful approximations. One approach: maintain a fixed-capacity prediction model alongside a more expressive one. The gap in prediction error between the two models, at a given state, serves as an index of how much structure exists in that state that the current model architecture cannot represent. States where the gap is large are states where the model's expressivity is the binding constraint — the most informative frontier for learning.
Intrinsic motivation and representation learning
One underappreciated connection is between intrinsic motivation and the quality of learned representations. The exploration behaviour an agent exhibits determines what experience it accumulates; the experience determines what representations are formed by the learning process. An agent with poor intrinsic motivation will fail to explore the parts of the state space that contain the most informative structure, and will form representations that are biased toward the easy, frequently-visited regions.
This suggests that intrinsic motivation is not just about data collection efficiency. It is part of the representation learning process itself. An agent with a richer and more accurate sense of where informative structure lies — a better intrinsic motivation signal — will, over time, form better representations, because it will direct its exploration toward the regions where the most learning is available. The two problems are coupled.
We are not claiming that prediction-complexity-based intrinsic motivation solves the exploration problem in general, or that it is the only valid approach. Simpler methods — random network distillation, novelty-based bonuses — work well in many settings and have substantial practical advantages in terms of implementation simplicity. The claim is more specific: for settings where the environment contains genuinely structured, learnable complexity alongside noise or low-information regions, a signal grounded in prediction complexity makes better distinctions than prediction-error-based alternatives, and the resulting exploration behaviour produces representations better suited to transfer.
The open questions
The hardest remaining problem in intrinsic motivation, in our view, is temporal abstraction. Most formalisations of curiosity operate at the level of single-step state transitions. But interesting structure in complex environments is often present at longer temporal scales — regularities that span many steps, causal relationships with long delays between cause and effect, skills that require extended sequences of actions to execute. A curiosity signal that operates only at the single-step level will systematically undervalue exploration of temporally extended behaviour.
Getting intrinsic motivation right at multiple temporal scales probably requires connecting it to the options framework — learning temporally extended actions — or to goal-conditioned exploration, where the agent can set and pursue subgoals over longer horizons. These are tractable research directions, and we think they are more important for practical progress in exploration than continued refinement of single-step prediction objectives. The infrastructure for understanding intelligence that genuinely explores — rather than exploits familiarity — runs through these longer-horizon versions of the problem.
