Continual Learning Without Forgetting

Catastrophic forgetting as a structural problem

When a neural network is trained on a new task, gradient updates tend to overwrite the weight configurations that encoded prior knowledge. This phenomenon — catastrophic forgetting, or catastrophic interference — is well documented and not a pathology of any particular architecture. It is a consequence of how distributed representations work: the weights that matter for one task overlap with the weights that matter for another, and optimising for the new task degrades performance on the old one.

This is not a minor limitation. A system that forgets is not accumulating intelligence. It is repeatedly starting over with different initial conditions. The aspiration for continual learning is a system where new experience extends, rather than replaces, existing knowledge — the plasticity-stability balance that biological learning systems achieve and artificial ones do not.

Existing approaches and their limits

• Regularisation-based methods — of which Elastic Weight Consolidation (EWC) is the most studied example — protect weights that were important for previous tasks from large updates. These work reasonably well in short, well-defined task sequences but degrade as the number of tasks grows, because the set of protected weights expands until the model lacks the freedom to learn anything new
• Memory replay stores examples from previous tasks and interleaves them with current training data. This reduces forgetting but introduces constraints around data retention, storage cost, and privacy, and does not address the underlying problem of distributional shift across a long task sequence
• Architectural expansion allocates new capacity for each new task, avoiding interference but preventing the shared representations and cross-task generalisation that would constitute genuine accumulation of knowledge

Our direction

We are investigating an angle that receives less attention than the three above: the relationship between the quality of the initial representation and susceptibility to catastrophic forgetting during subsequent learning. The hypothesis is that representations formed through self-supervised pre-training — representations that capture the compositional structure of experience rather than surface statistics — may provide a more stable foundation for continual adaptation, because they support more disentangled, less overlapping encodings of distinct types of knowledge.

This links our continual learning research directly to our self-supervised representation work. We are studying whether representations formed without labels, under appropriate objectives and at sufficient scale, exhibit structural properties that make them more amenable to long task sequences without the progressive rigidification that regularisation-based methods eventually produce.

Open questions

Can we characterise what makes a representation resistant to catastrophic forgetting before any forgetting has occurred — that is, prospectively rather than diagnostically? Are there training regimes — particular curriculum structures, regularisation schedules, or objective functions during pre-training — that predictably produce more continually-stable representations? And what does the organisation of biological memory — particularly the complementary learning systems hypothesis and the distinction between hippocampal and neocortical memory — suggest about the architectural conditions under which accumulation rather than forgetting is the default?