Logo
AI Governance

AI Inference

Inference is running a trained model on new inputs to produce outputs — the phase where AI meets users. For LLMs it means generating tokens step by step; inference cost, latency, and throughput shape product economics more than training does.

Why inference is the real cost

Training is a one-off capital expense; inference is the recurring operational cost paid on every request. For an LLM, each generated token requires a forward pass over the model — which is why API pricing is per-token and why techniques like quantization, distillation, KV-cache reuse, and smaller specialist models matter commercially.

Inference location is an architecture decision: hosted APIs offer frontier capability with per-token billing; self-hosted open-weight models trade ops overhead for data control and predictable unit cost. Reasoning models add “thinking tokens” — longer inference, better answers, higher cost.

Need help with this?

We consult and build across all of these domains — tell us what you are working on.

Privacy & Cookie Preferences

We use cookies to enhance your experience, analyze site performance, and support our marketing efforts. Your privacy matters, and you can withdraw consent at any time.