Traditional cloud cost management operates under the premise that costs correlate directly with allocated capacities, such as instance hours, gigabytes stored, commitment percentage rates, and utilisation of reserved instances. Inferences performed using artificial intelligence algorithms disrupt this logic twofold. When using managed machine learning APIs, the final cost correlates strictly with token usage; it does not relate whatsoever to any allocation conducted by the organisation's FinOps division. On the contrary, using self-hosted models requires paying based on accelerator usage. However, productivity rates vary according to batch sizes, memory, and latency constraints – none of which can be determined through typical infrastructure dashboards.
These considerations have led the FinOps Foundation to introduce an updated guide outlining FinOps strategies for AI environments. The document acknowledges significant changes within the industry and describes them thoroughly. According to the authors, cost-per-token should now be seen alongside traditional FinOps metrics such as GPU scarcity and fluctuations in pricing.
Two cost models, two failure modes
Managed APIs bill each request by input and output tokens, priced separately. On Anthropic's published price list, Claude Sonnet 5.5 costs $2 per million input tokens and $10 per million output tokens, Claude Haiku 5.5 costs $0.10 and $0.50 for prompts up to 100,000 tokens, and the Batch API halves both rates for asynchronous work. There is no idle cost, but there is no natural ceiling either: spend scales with traffic, prompt length and verbosity.
Self-hosted inference inverts the structure. The accelerator is billed whether or not it serves requests, so cost follows uptime while value follows tokens delivered within SLO. The governing metric is cost per million tokens at SLO:
cost per MTok = hourly accelerator cost ÷ (sustained goodput tokens per hour ÷ 10⁶)
Assume, purely for illustration, an all-in cost of $4.00 per GPU-hour and a measured sustainable output of 1,500 tokens per second while meeting latency targets. That yields 5.4 million tokens per hour, or about $0.74 per million output tokens. If average demand uses only 30% of that capacity, effective cost rises to about $2.47 per million tokens. The hardware price did not change; idle capacity did.
Why GPU utilisation misleads
The utilisation figure reported by nvidia-smi is the share of the sampling period during which at least one kernel was executing, as NVIDIA's developer forum explains; a single thread on one streaming multiprocessor can register as fully busy. NVIDIA's DCGM profiling documentation points to finer metrics such as SM activity and notes that a value of 0.8 or higher is necessary, but not sufficient, for effective GPU use.
LLM decoding is frequently constrained by memory bandwidth and KV-cache capacity rather than arithmetic. The vLLM team reported that earlier serving systems wasted 60–80% of KV-cache memory through fragmentation and over-reservation, limiting batch size; PagedAttention reduced that waste to under 4%. Batching is itself a trade-off: NVIDIA's GenAI-Perf guidance notes that serving more concurrent requests can raise output-token throughput while increasing inter-token latency. A GPU pushed to high utilisation with oversized batches may emit more tokens per second while breaching time-to-first-token targets; tokens delivered outside SLO carry no business value. The defensible efficiency measure is throughput per accelerator at SLO, reported with quality pass rates.
Token economics are architectural
Long context and repetition. Every call re-bills its full input. A support assistant sending an 8,000-token system prompt and tool schema plus 2,000 tokens of user and retrieval context, returning 500 tokens, costs $0.025 per request on Sonnet 5.5. Prompt caching changes the arithmetic: per the pricing page, Sonnet 5.5 cache hits bill at 5% of the base input rate, and five-minute cache writes at 1.25x. With the static 8,000-token prefix served from cache, the request costs about $0.0098 before write amortisation, but only if prefixes are byte-identical and traffic arrives within the cache lifetime.
Output generation. Output tokens cost five times input on these models, so verbosity, reasoning traces and unbounded max_tokens settings are direct cost drivers.
Agent loops. Multi-step agents resend accumulated context on every step. An agent whose context starts at 10,000 tokens and grows by 3,000 per step consumes 235,000 input tokens over 10 steps; with 500 output tokens per step, a run costs about $0.52 on Sonnet 5.5, roughly 21 times the single-call example. Input cost grows quadratically with step count, so step and token budgets belong in agent design.
Failure. If that agent completes 80% of tasks, cost per successful task is $0.65, not $0.52. Retries and abandoned runs are cost without value.
Routing, validated
Model routing sends simple requests to small models and reserves capable models for hard ones. At the prices above, a request with 2,000 input and 300 output tokens costs $0.00035 on Haiku 5.5 versus $0.007 on Sonnet 5.5. The RouteLLM study reported cost reductions of more than 2x on public benchmarks with minimal quality loss, but those results are benchmark-specific. In production a router is a classifier that can be wrong, so it must be evaluated against task-level quality, tail latency including escalation paths, and failure handling. If the small model resolves 85% of routed requests and the rest escalate, the expected cost is about $0.0014, provided failures are detected reliably.
Optimisation levers
- Autoscaling on queue depth and token throughput rather than GPU utilisation.
- Right-sizing model size, precision and accelerator memory to observed context lengths.
- Workload scheduling that moves latency-tolerant work to batch tiers or off-peak capacity.
- Prompt optimisation: trimming static context, stabilising cacheable prefixes and capping output.
- Cost attribution by tagging each request with product, team and feature at the gateway.
A continuous FinOps framework for AI
- Instrument. Capture per-request tokens (input, cached, output), model, latency, outcome and owner; for self-hosted fleets, join DCGM metrics with serving throughput.
- Define units. Track cost per successful task, cost per resolved request and throughput per accelerator at SLO, each paired with explicit quality and latency objectives.
- Decide. Require cost-per-outcome estimates in design reviews and gate routing or model changes on offline evaluation.
- Account. Show back or charge back by product and team; set budgets and anomaly alerts in unit terms.
- Review. Compare unit cost with business value per outcome, and repeat as prices, models and traffic change.
Spend reduction is the wrong objective. A falling bill alongside a rising cost per resolved request is a regression; a rising bill with falling unit cost and stable SLOs is growth working as intended.
