AI Inference
Inference is running a trained model on new inputs to produce outputs — the phase where AI meets users. For LLMs it means generating tokens step by step; inference cost, latency, and throughput shape product economics more than training does.
Why inference is the real cost
Training is a one-off capital expense; inference is the recurring operational cost paid on every request. For an LLM, each generated token requires a forward pass over the model — which is why API pricing is per-token and why techniques like quantization, distillation, KV-cache reuse, and smaller specialist models matter commercially.
Inference location is an architecture decision: hosted APIs offer frontier capability with per-token billing; self-hosted open-weight models trade ops overhead for data control and predictable unit cost. Reasoning models add “thinking tokens” — longer inference, better answers, higher cost.
Further Reading
Need help with this?
We consult and build across all of these domains — tell us what you are working on.
