Small models, big deployments: where inference is heading

For several years the story was straightforward: bigger models were better models, and the interesting work happened at the frontier. That story is still partly true and has stopped being the whole picture.

In production, the binding constraint is rarely peak capability. It is latency, cost per call, and the ability to run somewhere specific — a phone, a laptop, a machine inside a hospital network that will never be allowed to make an outbound request.

The bar is task-specific

A model that scores in the ninetieth percentile on a broad reasoning benchmark may be no better than one twenty times smaller at classifying support tickets into eleven categories. The benchmark measures general capability; the deployment needs one narrow competence delivered reliably at 40 milliseconds.

Once teams start measuring against their actual task rather than a leaderboard, the model selection usually moves down a tier or two.

Choose the smallest model that clears the bar, then spend the savings on evaluation.

What gets you there

Three techniques do most of the work. Distillation, where a large model generates training data for a small one. Quantisation, which trades a small amount of accuracy for a large reduction in memory. And straightforward fine-tuning on a few thousand examples of the exact job.

The combination routinely produces something that runs on commodity hardware and beats a much larger general model on the specific task, because it has seen the shape of the problem.

• Distil from a frontier model, evaluate against held-out real traffic.
• Quantise to 4-bit and measure the delta — it is usually smaller than expected.
• Keep a frontier model in the loop for the hard tail, routed by confidence.

The architecture that is winning

The pattern that keeps appearing is a cascade. A small fast model handles the majority of requests. A confidence signal routes the ambiguous remainder to something larger. Costs drop by an order of magnitude and the quality ceiling stays where it was.

It is not a novel idea — it is how spam filtering worked twenty years ago — but it is being rediscovered because the economics now demand it.