The question of what it means for a model to "understand" something sits uncomfortably between engineering and philosophy. On the engineering side, we have operational definitions: a model understands a task if it performs well on held-out examples from the same distribution. On the philosophical side, this answer looks thin. A system that has memorised statistical patterns in labelled data and a system that has formed some deeper structural understanding of the domain might be indistinguishable on a benchmark while being very different in every other respect.
For practical AI systems operating on familiar tasks within stable distributions, this distinction may not matter much. But for systems that need to generalise beyond the training distribution, to transfer knowledge to new tasks, or to operate in domains where labelled data is sparse — the nature of the representation becomes the binding constraint. This is why representation learning is not just a preprocessing step. It is close to the core of the intelligence problem.
What a representation is doing
A representation, in the machine learning sense, is a mapping from raw input to a structured intermediate space. The purpose of this mapping is to make downstream tasks easier: a well-designed representation separates the features relevant to a task from those that are irrelevant, organises similar inputs near one another, and preserves structure that is useful for generalisation.
The traditional approach is to learn representations implicitly during supervised training. The representation is whatever the internal activations of a network look like when that network has been optimised to produce correct labels. This works well when the labels are dense, consistent, and aligned with the structure of the problem. The representation that forms is shaped by the label distribution — which features of the input correlate with the output categories the model is trying to predict.
The fundamental issue with this approach is that the representation is responsive only to the features the labels encode. If the training labels distinguish cats from dogs, the representation will form features that separate cats from dogs. Whether it forms features useful for distinguishing healthy from diseased tissue, or for detecting scene geometry, or for any other downstream task, depends on how much those distinctions share structure with the original labelling.
Unsupervised and self-supervised alternatives
Representation learning without labels proceeds by defining a different kind of objective — one that does not require per-example labels from a human, but instead derives signal from the structure of the data itself.
Autoencoder-style objectives define a reconstruction task: compress the input into a low-dimensional code, then reconstruct the input from that code, minimising reconstruction error. The representation that forms in the bottleneck must retain the information necessary to reconstruct the input. What gets compressed away is, in principle, noise; what is retained is structure. The limitation is that "what is necessary to reconstruct" does not always align with "what is useful for downstream tasks". Pixel-level textures that are important for reconstruction may matter less than semantic structure for most applications.
Contrastive objectives take a different approach. Given two augmented views of the same data point — say, two crops of the same image, or two short segments from the same audio sequence — the objective is to produce similar representations for the two views, while pushing apart representations of different data points. This is the core intuition of contrastive self-supervised learning: invariances that should be preserved (the object being the same object under different crops) are enforced by the training objective, while differences that should be discriminated (different objects) are pushed apart.
The augmentation choice is crucial. What augmentations are applied determines what invariances are learned. Colour-jitter augmentations produce representations invariant to colour; occlusion augmentations produce representations invariant to partial occlusion. Choosing augmentations is a form of domain knowledge injection — but it is much weaker and less labour-intensive than per-example labelling.
How unsupervised representations compare
Empirically, representations learned through self-supervised objectives have proven capable of matching or approaching representations learned through full supervision on a range of evaluation tasks, particularly when the downstream evaluation involves linear probing — fitting a linear classifier on frozen representations to a new labelled dataset. The implication is that the representations form features that are semantically meaningful, even without ever seeing a label.
This should not be surprising in retrospect. The data itself contains vast statistical regularities — co-occurrence patterns, spatial relations, temporal structure — that are informative about the structure of the world. A model that has learned to capture these regularities has built internal categories, whether or not those categories were handed to it from outside. The categories emerge from the structure of the data; the labels, in supervised learning, were always derived from the same underlying structure. Self-supervised learning is, in some sense, extracting what the labels were always pointing at, more directly.
We are not arguing that self-supervised representations are always better than supervised ones. For narrow tasks where the supervision signal is precisely aligned with the representation needed, supervised learning is more direct and typically more sample-efficient within its training distribution. The argument is about breadth: representations formed from the data's own structure tend to generalise more broadly, because they are responsive to the full structure of the domain rather than to the particular slice of it that the annotation schema captures.
The question of understanding
Does a model that has formed good self-supervised representations "understand" anything? This depends entirely on what you mean by understand. If understanding requires something like conscious access to internal representations, no machine learning model qualifies and the question is probably not useful for AI research. If understanding means having formed internal distinctions that correspond to real structure in the world and that support generalisation, then the answer is at least "more than you might think".
One useful proxy is out-of-distribution generalisation. A model that has built a deep internal model of domain structure — rather than fitting surface statistics — should degrade more gracefully when the distribution shifts. We investigate this as one dimension of representation quality: not just performance on held-out in-distribution examples, but the rate and nature of performance degradation as distribution shift increases.
A second proxy is the structure of the representation space itself. If similar inputs in some meaningful sense cluster together in the representation space — if the geometry of the learned embedding reflects the geometry of the domain — that is evidence that something more than surface pattern-matching is occurring. Probing studies, which train simple predictors on top of frozen representations to assess what information is encoded, are one way to examine this.
Where the open questions are
The most interesting open question in self-supervised representation learning, in our view, is not about contrastive objectives or architecture choices. It is about what inductive biases are needed to learn representations that are not just useful for current downstream tasks but that capture something more like the generative structure of the domain.
Current methods tend to learn representations that are useful when the downstream task has similar structure to the pretraining domain. They do not obviously produce the kind of abstract, compositional representations that would allow a model to handle genuinely novel task types — tasks that require combining domain knowledge in ways not seen during pretraining. Getting there requires not just better objectives but probably better priors about what structure representations should have: hierarchical organisation, compositionality, causal sensitivity. These are active areas of investigation across the field, and they point at questions that are, ultimately, about what representations are for — which is a question about the nature of understanding itself.
