Self-Supervised Representation Learning

The dependency on human labels

Every modern machine learning system that achieves strong task performance has been trained, at some stage, on human-labelled examples. This is an architectural constraint that is rarely examined critically. Labels tell a system what matters and what does not — they encode human judgement about which distinctions are relevant. Without them, how does a system decide where to direct its learning?

We investigate a class of approaches collectively described as self-supervised learning: methods that construct learning signals from the data itself, without any external annotation. The underlying premise is that any coherent signal — an image, a sequence of observations, a trajectory — contains far more internal structure than its surface appearance suggests, and that a suitably designed objective can induce a model to discover and encode that structure.

Our current directions

• Contrastive and non-contrastive objectives that encourage consistent representation across augmented views of the same datum — including investigation of the collapse and mode-dropping failure modes that arise when these objectives are poorly conditioned
• Masked prediction: predicting withheld portions of an input as a form of self-generated supervision, with particular attention to how masking ratio and strategy affect the diversity and transferability of the learned representation
• Probing the geometry of learned embedding spaces — isotropy, neighbourhood consistency, linear separability of known structural invariants — to understand what structure these objectives preserve and what they discard
• Studying the relationship between representation quality and downstream task transfer without reference to a fixed labelled benchmark, which requires developing evaluation proxies that do not themselves require annotation

Why this is hard

The difficulty is not building a system that learns a representation. The difficulty is building one whose representation is genuinely useful — that captures aspects of the world relevant to intelligent action across a range of settings, rather than superficial statistical regularities that happen to appear in the training distribution and do not transfer.

There is a second difficulty: evaluation. Measuring representation quality without access to labelled downstream tasks is an open problem. Many geometric proxies — dimensionality, alignment, uniformity — correlate with downstream performance in some regimes and fail to predict it in others. We are working on approaches that are more structurally motivated, aiming to characterise what a representation has learned rather than only whether it performs.

Open questions we are investigating

Can a model trained entirely without labels produce representations that generalise to genuinely novel domains — ones not structurally similar to its training distribution? What learning objectives give rise to representations that support compositional reasoning? What is the relationship between the breadth of the unlabelled data distribution and the breadth of the resulting representation?

These questions are interrelated and we do not have definitive answers. Our current work is largely investigative: mapping the landscape of methods and failure modes, identifying which questions are tractable, and building the evaluation tools that will eventually let us know whether we are making progress.