Inference Economics and AI COGS: What CFOs Should Measure Beyond Token Price

Training cost is capital; inference cost is operations. When software embeds large language models, every user query, agent step, and retry adds variable expense that scales with usage—not with head count. Inference economics is the discipline of measuring and managing that ongoing cost: defining a unit of AI work, attributing full-stack spend to it, and tracking whether margin improves as volume grows.

This note goes deeper than headline token pricing. For workflow-level FinOps, ROI dashboards, and finance KPIs, see the companion post AI FinOps in corporate finance.

Training vs Inference: Who Pays What

Cost type	Who typically bears it	Finance treatment
Training (foundation or fine-tune)	Model vendor or large enterprise R&D	Capex or R&D expense
Inference (each production call)	Product owner embedding the model	Variable COGS / opex

Vista Equity Partners’ inference economics framework describes the shift for enterprise software: when a platform embeds AI, the software company pays the model provider for every inference its customers generate. Cost grows with every query and every agent action—unlike traditional hosting that scales more modestly with users.

Deloitte’s CFO guide to AI token economics notes that many organizations already exceed billions of tokens per month, with average enterprise AI model usage spend rising sharply year over year. Token dynamics belong in forecasts, margins, and capital risk projections—not only engineering dashboards.

AI COGS as a Margin Line Item

For AI-native and AI-heavy products, inference can exceed classic hosting as a cost of goods sold line. Vista’s analysis cites directional figures where inference averages a material share of revenue at AI-native B2B companies—magnified for agentic products where each user request triggers multiple model calls.

Finance should report:

• Gross margin by product/segment with AI COGS separated from non-AI infrastructure.
• Cost per active customer per month including fully loaded inference.
• Cost per use case (support, search, generation, close automation) to spot margin leaks.

Beyond Cost per Token: Cost per Successful Output

Raw tokens mislead. A failed agent loop that retries five times burns tokens without business value. FinOps Foundation GenAI guidance (summarized in industry writeups such as CloudZero on inference economics) recommends modeling inference and training by use case, with attribution from infrastructure spend to business activity.

Better units:

• Cost per inference — tokens × blended rate + embedding + reranking costs.
• Cost per successful task — above, divided by tasks that pass quality gates and human acceptance.
• Cost per agent run — includes tool calls, context re-injection, and self-correction loops.

Track separately:

• Input vs output tokens (pricing asymmetry)
• Context window size (long-context FP&A Q&A costs more per call)
• Cache hit rate (repeat context should be cheaper)
• Model tier mix (routing small tasks to smaller models)

The Agent Loop Multiplier

Agentic workflows send packages of instructions, history, tools, and task descriptions to the model on each step. Vista’s internal research (as described in their public framework) found that context reuse and prompt design can change inference cost by large multiples—agent prompt optimization often yields additional savings beyond architecture choices.

Finance and product should jointly monitor:

• Average steps per completed workflow
• Token budget per task (hard caps where possible)
• Escalation rate to expensive reasoning models

Pair with multi-agent orchestration logging so loops are visible, not buried in one API bill.

Driver-Based Forecast Template

A practical forecast chain for CFO offices:

Active users
  × requests per user per day
  × tokens per request (input + output)
  × model mix and blended $/M tokens
  + embedding / retrieval costs
  + GPU overhead if self-hosted
  = daily inference $

Refresh monthly. Bridge engineering projections to actuals; agent adoption surprises often appear as token growth per request, not just user growth.

Scenario cases to model:

• Cost spike — longer context, deeper agent loops, model upgrade
• Efficiency win — caching, routing, prompt compression
• Adoption surge — free tier or new feature without pricing guardrails

Pricing and Guardrails for Consumption Products

When finance sells consumption-based AI (API, credits, per-seat with limits):

• Define included units and overage economics before launch.
• Align sales discounts with margin floors on inference-heavy SKUs.
• Instrument cost per customer alerts when subsidy exceeds threshold (directional practice cited in Deloitte’s token economics piece for high-growth startups tracking per-engineer usage).

Model Portability and COGS

Vendor concentration affects COGS risk. A model-agnostic stack lets finance shift volume when token prices fall or a cheaper model passes regression tests—if evaluation harnesses exist before swaps.

What to Do This Quarter

1. Separate AI COGS on the P&L or management pack where material.
2. Pick one product or workflow and instrument cost per successful output.
3. Add inference drivers to the rolling forecast alongside head count and cloud spend.
4. Review agent designs with engineering for loop and context cost—not only accuracy.

The Finance AI strategy hub links inference economics to compute procurement, controls, and capability signals.