Distillation vs. cascades

Two different patterns solve the "we need this cheaper and faster" problem, and they suit different traffic shapes. Distillation trains a smaller, faster student model to mimic a larger teacher's outputs -- a one-time training cost that then serves every request at the smaller model's latency and price, which pays off when traffic is high-volume and reasonably uniform in difficulty. A cascade (or funnel) instead routes each request dynamically: a cheap, fast model or heuristic handles the easy majority of traffic, and only the harder or more uncertain cases escalate to a larger, more expensive model.

Cascades tend to dominate when traffic is skewed -- when only a small fraction of requests are the genuinely hard ones that need the expensive model, sending 100% of traffic through a single distilled model wastes the accuracy the large model could have provided on exactly the cases that need it most. The right choice depends on measuring that skew, not on a general preference for one pattern over the other.

Worked examples

When would you distill rather than build a funnel?

They solve different shapes of problem. Distillation makes one expensive model cheaper everywhere it runs — the right choice when almost every request genuinely needs the full…