Much of the work in reward-free learning is implicitly about reward design — about what it would mean to not need a reward, and therefore about what a reward is for. You cannot build a principled alternative to reward-based learning unless you understand why reward-based learning works when it works, and fails when it fails. That understanding turns out to be harder to come by than it seems.
Reward signals are often described as if they were simply a measurement of how well an agent is doing. In practice, they are hypotheses about which aspects of agent behaviour are worth improving. A reward function encodes a designer's beliefs about what success looks like, compressed into a scalar. The agent's job is to maximise that scalar. The designer's job is to ensure that maximising the scalar actually produces the behaviour they want. These two jobs are almost never perfectly aligned, and the gap between them is the source of most of the interesting problems in reinforcement learning.
The specification gap
Consider a navigation agent rewarded for reaching a goal location quickly. The reward function seems straightforward: sparse reward on reaching the goal, no reward otherwise. But the agent optimising this reward will, absent other constraints, find the fastest path to the goal as measured by the reward — which means it might happily pass through states that a human designer would consider deeply undesirable, if those states lie on the shortest path. The reward function specified what to do (reach the goal), but not what not to do, because the designer's assumption that the unspecified constraints were obvious turned out to be an assumption the reward function could not enforce.
This is not a failure of the reward function as a mathematical object. It is a failure of the reward function as a specification of intent. The designer had more in mind than the scalar encoded, and the agent — being a scalar-maximiser — had no access to anything not in the scalar.
The specification gap is not easily closed by adding more terms to the reward. Each additional term is another hypothesis about what matters, and each introduces its own misalignment risk. There is a combinatorial problem lurking: the number of failure modes that arise from reward misspecification grows faster than the number of terms that can be added to address them, because each new term interacts with all existing terms and with the full range of situations the agent might encounter.
Reward shaping and its consequences
A common engineering response to sparse or weak rewards is reward shaping: augmenting the original signal with intermediate rewards that guide the agent toward useful behaviour. Reward shaping, when done carefully, can dramatically accelerate learning by providing denser signal in the early stages when sparse rewards are too infrequent to produce useful gradient information.
But shaping introduces a new version of the specification problem. The shaped reward now encodes not just the designer's beliefs about success, but also their beliefs about which intermediate states are worth reaching. If those beliefs are correct, shaping helps. If they are wrong — if the designer's model of useful intermediate states does not match the actual problem structure — the shaped reward can misdirect the agent into deep local optima that look good by the intermediate measure but are far from optimal by the original goal.
In domains where the designer does not fully understand the problem structure — which is most interesting domains — reward shaping is therefore dangerous in a specific way: it encodes the designer's existing understanding so strongly into the training signal that the agent can have difficulty going beyond that understanding. The reward becomes a compressed representation of what the designer knows, and the agent becomes a learner of the designer's knowledge rather than a discoverer of the domain's structure.
What makes a reward signal informative
Setting aside the specification and shaping issues, what properties does a reward signal need to have to be useful as a learning driver at all? This is an underexplored question. The standard answer — reward should reflect task performance — is circular. A more useful decomposition might look like this:
- Density: How frequently does the agent receive a reward signal during training? Very sparse rewards make learning slow or impossible in large state spaces; very dense rewards risk rewarding the proxy measure rather than the real objective.
- Faithfulness: Does reward track the dimension of performance we actually care about, or a correlated proxy? A proxy is always a simplification, and optimising a proxy hard enough will always eventually diverge from optimising the real objective.
- Responsiveness: Does reward change meaningfully when the agent's behaviour changes in the ways we care about? A reward that is flat over a large region of behaviour space cannot discriminate between actions in that region, providing no gradient for the agent to follow.
- Robustness: Does reward remain aligned with intent across the full range of states the agent might reach — including states the designer did not anticipate?
Most real-world reward designs fail at least one of these properties in at least some part of the state space. The combination of failures creates the pathological behaviours that practitioners have documented extensively: reward hacking, distributional exploitation, mode collapse, myopic optimisation.
The case for reward-free approaches
Understanding reward design carefully is, paradoxically, the argument for stepping away from reward altogether in some settings. If a reward signal is a compression of the designer's knowledge of what matters, and if the problem of interest is precisely one where the designer does not yet know what matters — then the reward signal is not a guide. It is a statement of ignorance, dressed up as a training signal.
Reward-free exploration proceeds from the idea that an agent can usefully learn about the structure of an environment before any task is specified. If the agent builds a good world model — an internal model of the environment's transition dynamics and latent structure — it is in a much better position to solve any task that is later specified, because it does not have to simultaneously learn what the world is like and what to do in it.
We are not arguing that reward is always wrong. Reward remains the appropriate training signal when the designer does know what they want, when the specification can be made faithful, and when the exploration dynamics are manageable. The argument is narrower: in domains where reward design is structurally hard because the designer lacks the knowledge to write a faithful specification, the reward-free approach is not just a convenience — it is the epistemically correct response. Deploying an impoverished reward signal in such settings does not merely slow learning; it actively miseducates the agent in ways that can be difficult to correct.
First principles, concretely
Thinking about reward from first principles means asking, before writing any reward function: what is the actual structure of what I want? Can I specify it faithfully in a scalar? What happens at the edges of the state space where my specification is vague? What will the agent do when it finds a way to maximise the scalar that I did not anticipate?
These questions are worth asking even when you are going to use reward. They reveal the assumptions baked into the signal and flag where the agent is most likely to behave unexpectedly. And they motivate the reward-free research programme as something more than a stylistic preference: a recognition that in some domains, the questions the reward function would need to answer cannot yet be answered, and honest learning must proceed from what is actually known rather than from a confident-sounding proxy for what is not.
