Reward-Free Exploration

The problem with extrinsic reward

Reinforcement learning has produced striking results in settings where a well-specified reward function is available. But designing a reward function is itself a form of annotation. A reward function embeds assumptions about what matters and what constitutes good behaviour — assumptions that a human has to specify in advance. In many real-world settings, these assumptions are difficult to articulate, easy to misspecify, and the resulting reward function is a poor guide to genuinely intelligent behaviour.

We are interested in agents that can build a useful understanding of an environment without any external reward directing their exploration. The hypothesis: an agent exploring freely, driven only by its own internal states, can construct a world-model — a compressed, predictive representation of environment dynamics — that transfers efficiently to subsequent tasks, none of which need to be specified during the exploration phase.

Intrinsic motivation as a substitute

The agent must be driven to explore by something. In reward-free exploration research, this is typically a form of intrinsic motivation: a signal derived from the agent's own state and experience rather than from an external evaluator.

Common formalisations include prediction error — the agent is drawn toward situations its current model cannot yet accurately predict — novelty signals derived from count-based or density-based measures of state visitation, and competence-based drives such as learning progress, where the agent seeks situations where its model is improving. Each formalisation has different properties. Prediction error, for example, degrades under high environmental stochasticity, where uncertainty cannot be resolved by additional data. We are studying which formalisations are most robust across environments with different stochastic structure.

Current investigations

• Comparing families of intrinsic motivation objectives — prediction error, count-based novelty, learning progress, and variants — across environments with varying degrees of aleatoric and epistemic uncertainty
• Studying the relationship between exploration coverage (how thoroughly an agent visits the state space during reward-free pre-training) and the quality and transferability of the resulting world-model
• Examining how the duration and structure of the reward-free pre-training phase affects the efficiency of subsequent task adaptation
• Investigating whether intrinsic motivation objectives can be designed to be task-agnostic in their exploration behaviour while producing representations reliably useful across a range of downstream tasks

The downstream challenge

Even if reward-free exploration produces a rich world-model, using that world-model to perform well on a subsequently specified task is non-trivial. The gap between good exploration and good task performance is not well understood. We are investigating how the structure of the pre-training phase — the breadth of states visited, the objectives used to drive exploration, the architecture of the world-model — shapes the agent's capacity to adapt rapidly when a task reward is eventually introduced.