We started Ineffable Intelligence in 2021 with a specific bet: that learning without human-labelled data was not merely a niche technique but a foundational question about what AI systems can become. Five years in, the field around that question has changed substantially. Some of our original assumptions turned out to be correct. Others were wrong in interesting ways. This piece is an attempt at honest accounting — where the progress has been real, where it has been slower than hoped, and where the real difficulties still live.
What has moved faster than expected
The most significant positive surprise has been in self-supervised representation learning. When we started, contrastive and masked-prediction approaches were producing representations that were useful but clearly secondary to supervised pretraining in quality. The gap has closed substantially — in many evaluation settings, representations formed without any label match the quality of supervised representations, and in some transfer settings they generalise better. The theoretical understanding of why this works — why the structure of unlabelled data is rich enough to form semantically meaningful categories — has also developed, with better accounts of the inductive biases required and the conditions under which self-supervised objectives succeed or fail.
The second positive development has been in the breadth of domains where self-supervised methods have proven applicable. Early work was concentrated in vision and language; the methods have since been adapted successfully to audio, protein structure, time series, and several scientific data modalities. Each adaptation has required domain-specific choices about augmentation, architecture, and objective design, but the core framework has transferred more widely than we anticipated.
Third, the question of whether and how to scale self-supervised approaches has become tractable as a research direction, even if the answers are not yet settled. The scaling behaviour of supervised models is well-characterised; understanding whether unsupervised approaches exhibit comparable scaling laws is now an active research programme rather than a speculative idea. The data to answer the question at meaningful scale does not yet exist, but the question is being asked seriously by a substantial part of the community.
What has moved slower than expected
The harder and more honest part is cataloguing what has not moved. Reward-free exploration — building agents that develop rich world models through intrinsically motivated exploration without any externally specified task — remains significantly harder than the self-supervised representation learning problem. The theoretical frameworks are there: information-theoretic objectives, competence-based intrinsic motivation, learning-progress signals. The empirical results in controlled environments are encouraging. But the gap between results in controlled simulated environments and robust exploration behaviour in complex, partially observable, real-world-like settings has not closed at the rate we hoped.
Part of the reason is that exploration interacts with everything else: with the quality of the world model, with the granularity of the action space, with the temporal horizon over which useful exploration takes place. Improving exploration in isolation is tractable; improving it in a way that generalises to the full complexity of the problem is much harder.
Continual learning has also moved more slowly than the research activity level would suggest. There are many papers, many methods, and measurable progress on standard benchmarks. But the benchmark assumptions — discrete task boundaries, finite task sets, labelled training data throughout — do not match the practical settings where continual learning would matter most. Progress on the benchmark problem has not cleanly transferred to progress on the practical problem, and understanding why that gap exists is a research project in itself.
Where the real difficulties still live
Setting aside the pace of progress, there are structural difficulties in the label-free learning agenda that have become clearer over five years of work.
The first is evaluation. Supervised learning has natural evaluation: you hold out labelled test data and measure performance on it. Self-supervised learning produces representations, not answers, and the quality of a representation depends on what downstream task you evaluate it on. This creates a fundamental evaluation problem: without a fixed downstream task, there is no objective way to compare two self-supervised representations. The linear probing paradigm — train a linear head on top of frozen representations using labelled downstream data — solves this practically but reintroduces labels at evaluation time. Evaluating representations in a way that is genuinely label-free remains unsolved.
The second structural difficulty is sample efficiency. At large scale, self-supervised methods can compensate for per-example information content by training on massive datasets. At small scale — the setting relevant for scientific domains with limited data — the efficiency advantage of labels is harder to overcome. A labelled example conveys a specific signal about the annotation's category; an unlabelled example conveys a richer but more diffuse signal about its structure. In low-data regimes, the specific signal of the label can be worth more than the diffuse signal of the unlabelled example, and the advantage of label-free learning only becomes clear when data volume is large enough to compensate. Many of the most interesting scientific domains have precisely the combination of data scarcity and annotation expense that makes this trade-off most unfavourable.
The third difficulty is specification. We argued early on that self-supervised learning could produce representations that capture structure that annotators did not know to look for. This is true as a theoretical possibility. In practice, the structure that emerges from self-supervised pretraining reflects the choices of objective function and augmentation — choices that encode prior beliefs about what kinds of structure should be preserved and what kinds should be ignored. These priors are less explicit than annotations, but they are priors nonetheless. The hope that self-supervised learning is genuinely prior-free, discovering structure from first principles without any human-encoded assumptions, is not quite borne out in practice. The priors are there; they are just embedded in objective design rather than label specification.
An honest position on where the field stands
We are not arguing that label-free learning has failed. The progress in self-supervised representation learning has been substantial and real, and it has changed what is possible in domains where labelled data was previously the binding constraint. The question we started with in 2021 — can AI systems learn meaningful representations and develop genuine competence without human-assigned labels — has been answered in the affirmative for a significant class of problems.
What has not been answered, and what remains the harder and more important question, is whether the same can be true in domains where the structure of the problem is itself unknown — where not just the labels but the right categories, the right features, and the right notion of competence are all yet to be determined. The self-play intuition that motivated this laboratory was never specifically about learning in domains where the categories are known but labels are expensive. It was about learning in domains where the right ontology is itself part of what needs to be discovered.
That harder problem — open-ended knowledge acquisition in genuinely uncharted domains — is still where the real work is. We have better tools for parts of it than we had five years ago. The distance between current capability and what that problem requires is still large, and we would rather say so clearly than represent incremental progress as a greater advance than it is. The interesting work is ahead, and it is harder than the work that has already been done.
