2025
Preprint — March 2025
Towards Stable Contrastive Objectives for Representation Learning Without Labels
Amara Osei, David Silver
We investigate the conditions under which contrastive and non-contrastive self-supervised objectives converge to embedding spaces with useful geometric properties, in the absence of any labelled evaluation signal. We document the collapse and mode-dropping failure modes of several common objectives under variations in batch composition and augmentation strength, and describe candidate modifications that exhibit more consistent behaviour on held-out probing tasks.
Full text (Partner Access)
Technical Report — June 2025
Intrinsic Motivation Under Distribution Shift: A Comparative Study
Tobias Wenzel, Priya Narayanan
Intrinsic motivation objectives behave inconsistently when the environment distribution changes mid-training. This report documents six common formalisations across three distribution-shift regimes, characterising how prediction error, novelty, and competence-based drives respond. We find that competence-based drives are most robust to gradual shift but degrade rapidly under abrupt change.
Full text (Partner Access)
Research Memo — September 2025
On the Relationship Between Representation Quality and Catastrophic Forgetting
Tobias Wenzel, Amara Osei
We present a preliminary investigation of whether representations formed through self-supervised pre-training exhibit structural properties that reduce susceptibility to catastrophic forgetting during subsequent task learning. Early results are mixed: compositional structure in the embedding space correlates with forgetting resistance in low-plasticity regimes but not in high-plasticity ones.
Full text (Partner Access)
2024
Technical Report — February 2024
Reward-Free Pre-Training: Coverage, Efficiency, and Downstream Adaptation
David Silver, Tobias Wenzel, Priya Narayanan
This report examines the conditions under which reward-free exploration phases produce world-models that transfer efficiently to downstream tasks. We identify coverage — the extent to which an exploration policy visits the state space — as a primary determinant of transfer quality, and we study the tradeoff between exploration breadth and the specificity of learned dynamics models.
Full text (Partner Access)
Workshop Contribution — July 2024
Probing Representation Geometry Without Labels: Methods and Limitations
Amara Osei
Evaluating the quality of unsupervised representations without access to labelled downstream tasks requires geometric proxies. This contribution surveys and critically assesses five such proxies — isotropy, neighbourhood consistency, linear separability of known structural invariants, and two novel measures — against a retrospective labelled evaluation to assess their predictive validity.
Full text (Partner Access)
2022 – 2023
Technical Report — November 2023
Masked Prediction as a Self-Supervised Objective: Training Stability and Representation Diversity
Amara Osei, Priya Narayanan
We study masked-prediction objectives as a form of self-generated supervision across a range of input modalities. Our focus is training stability and the diversity of representations formed under different masking strategies. We find that random masking with high masking ratios produces more transfer-useful representations than structured masking in the modalities we tested.
Full text (Partner Access)
Research Memo — March 2022
Founding Questions: A Research Agenda for Learning Without Labels
David Silver
Written at the time of Ineffable Intelligence's founding, this memo sets out the central questions that motivated the laboratory's formation: what are the limits of supervised learning, what does unsupervised pre-training actually learn, and under what conditions does a learning signal derived from data structure alone produce representations that support generalisation? It is published here as context for subsequent work. The questions have not changed substantially; some partial answers have emerged since 2022, most have not.
Full text (Partner Access)