The scaling hypothesis — that increasing model size, compute, and data reliably yields better performance — has been one of the most empirically robust findings in machine learning over the past several years. The relationship between compute and capability, when both are measured carefully, has been surprisingly smooth and predictable across multiple orders of magnitude. This empirical stability has given the scaling approach a kind of gravity: if you want to improve performance, scaling is the first thing to try, and often the thing that works.
Almost all of this work has been done in what I would call supervised or weakly supervised settings — next-token prediction on text (which has labelled targets in the form of the next token), image-text contrastive pretraining (which uses paired data as implicit supervision), and fine-tuning stages that use human feedback or labelled examples. The question we have been thinking about seriously since 2024 is: what happens when you apply the scaling lens to the genuinely unsupervised case — where no label, no pairing, no human feedback of any kind is available, and the training signal comes entirely from the structure of the raw data?
Why the question is not obvious
The scaling laws that govern supervised and weakly supervised pretraining were not obvious before they were observed. The finding that test loss scales as a power law in compute across many orders of magnitude — with relatively stable exponents — was a nontrivial empirical discovery, not something that followed straightforwardly from the theory of neural networks.
There is no strong reason to expect that the same scaling behaviour holds in unsupervised settings, for several reasons. First, the loss functions are different. Contrastive losses, predictive coding objectives, and masked autoencoding losses all have different properties, and the relationship between loss value and downstream task quality varies by objective and architecture. Second, the data pipeline has different properties. In supervised settings, the training data contains explicit structure in the labels that can be aligned with the model's outputs; in unsupervised settings, the signal extraction is purely from the data's internal structure, which may scale differently with data volume. Third, the failure modes are different. Supervised models that are undertrained fail gracefully; unsupervised models that are undertrained may form representations that are systematically misleading, because the training objective can be satisfied by spurious features without being penalised by a label.
The question of whether and how scaling works in the unsupervised regime is therefore genuinely open, and the answer has significant implications for whether unsupervised learning can ultimately compete with the supervised paradigm on capability at scale.
What we observe at small scale
Our compute budget does not allow frontier-scale experiments. We work at small-to-medium scale — model sizes in the range of tens to hundreds of millions of parameters, training runs measured in days to weeks on modest GPU clusters rather than months on thousands of accelerators. Within these constraints, we have been investigating how downstream task performance scales with training compute under purely self-supervised objectives, specifically contrastive and predictive ones applied to unlabelled image and sequence data.
The observations are tentative and should be read as such. At the scales we operate, the training dynamics of purely self-supervised models are less predictable than those of supervised models trained on the same architectures and compute budgets. Loss curves are noisier. The relationship between training loss and downstream performance is less stable — the correlation is present but weaker than in the supervised case, and there are more instances of training runs where loss decreases steadily but downstream performance plateaus or regresses. This is consistent with the failure mode where the model learns to satisfy the self-supervised objective through features that are statistically convenient but semantically shallow.
These instabilities suggest that scaling alone is not sufficient in the unsupervised regime — that objective design and architectural choice interact with scale in ways that are not yet well-characterised, and that the "scale first, worry about objectives later" approach that has worked in supervised settings may not transfer cleanly.
The infrastructure differences
There is also a practical infrastructure dimension that is underappreciated in discussions of unsupervised scaling. In supervised pretraining, the data pipeline is relatively straightforward: you have labelled examples, you form batches, you compute loss against the labels. The main engineering challenge is throughput — moving data fast enough to keep the accelerators busy.
Contrastive self-supervised learning introduces additional complexity at the data pipeline level. Contrastive objectives require the construction of positive and negative pairs within each batch, typically through data augmentation. The quality of the contrastive signal depends heavily on the batch composition — specifically on the diversity and difficulty of the negatives. A batch of similar examples provides easy negatives that produce weak gradient signal; a batch of diverse examples produces better gradient but may require careful curation to avoid false negatives.
At scale, managing batch composition becomes a significant engineering problem. Sampling strategies that worked at small scale — random batching from a shuffled dataset — degrade in quality as model size and data volume increase, because the proportion of hard negatives in a random batch decreases as the model improves and the easy cases are already well-separated. Curriculum strategies that maintain a supply of hard negatives as training progresses require maintaining and updating some representation of sample difficulty, which adds memory and compute overhead that does not arise in the supervised case.
These infrastructure issues are solvable, but solving them requires engineering attention that is not needed in supervised pipelines. One underappreciated consequence is that the effective compute efficiency of unsupervised training — FLOP-for-FLOP, how much representation quality you get per unit of compute — is currently lower than the supervised equivalent, partly because of these infrastructure inefficiencies rather than any fundamental property of the objective.
What scaling without supervision might unlock
If the practical barriers can be addressed — better objective design, smarter batch construction, more stable training dynamics at scale — the potential payoff is substantial. The distinguishing property of the unsupervised regime is that the data requirement is not bounded by annotation cost. Labelled datasets are expensive to produce and limited in scale; unlabelled data is not. A domain with rich unlabelled data but scarce labelled examples — medical imaging, scientific instrument data, low-resource languages, material characterisation — is one where an unsupervised scaling approach could access training signal that supervised approaches cannot.
More fundamentally, unsupervised scaling at sufficient scale would, if it works as hoped, produce representations formed from the full distributional structure of the domain rather than from the slice that annotations capture. Whether those representations are qualitatively richer in the ways that matter for generalisation and transfer is an empirical question, and answering it requires running the experiments at a scale where the difference — if it exists — would be visible. We are not at that scale yet. But it is the question we are building toward, and the infrastructure work is part of that path.
We want to be careful not to overclaim the prospects. There is no guarantee that unsupervised scaling exhibits the same predictable power-law improvements that have been observed in supervised settings. The instabilities we observe at small scale are a genuine warning sign. But the absence of a ceiling on unlabelled data — and the kinds of domains where unlabelled data is all that exists — makes this worth investigating seriously, even without the certainty that scaling will work as cleanly as it has elsewhere.
