LLM Inference Costs Dropped 90%. Here Is Where the Money Went

The great deflation

Eighteen months ago, serving a frontier-class language model cost roughly $50 per million tokens. Today, comparable quality costs under $5. This is one of the fastest cost collapses in computing history.

The technical levers

Three techniques did most of the work. Quantization compressed models to 4-bit precision with negligible quality loss. Speculative decoding let small draft models predict tokens that big models merely verify. And continuous batching pushed GPU utilization from 30% to over 80%.

The business consequences

Cheap inference changes product economics fundamentally. Features that were unaffordable at scale — summarizing every document, classifying every event — are now table stakes. The winners of this era will be the companies that redesign their products around abundant intelligence rather than rationing it.