LLM Cost Tracking: How to Track LLM API Costs Across OpenAI, Anthropic, and Bedrock

LLM cost tracking means capturing the token spend of every model call and attributing it to a team, feature, or customer before the invoice arrives. There are three practical layers: the provider console gives you a per-key total, an observability tool such as Langfuse attaches cost to each trace so you can see what one conversation cost, and a gateway such as LiteLLM sits inline and can refuse a call that would break a budget. Provider dashboards alone are not enough once you use two model vendors, because nothing normalizes them into one number and none of it is real time.

The pattern is familiar to anyone who has run a cloud bill. Usage-based pricing plus a tool that is trivially easy to call means spend grows quietly for a few months and then jumps in a week. The difference with model APIs is speed: a retry loop in an agent can spend more in six hours than the team spent all quarter, and nobody finds out until the monthly statement. The FinOps Foundation's State of FinOps 2026 survey of 1,192 practitioners, representing more than 83 billion dollars of cloud spend, found 98 percent now manage AI spend, up from 31 percent two years earlier. This went from an edge case to standard scope in about 24 months.

What is LLM cost tracking?

LLM cost tracking is the practice of measuring what each model call costs and grouping those costs by something meaningful: a team, an application, a feature, a customer, or a model. A call's cost is input tokens times the input rate plus output tokens times the output rate, adjusted for cached tokens and any batch or reasoning-token pricing the provider applies. The arithmetic is simple. The hard part is that the number is only useful if it carries a label, and labels have to be attached at the moment of the call.

That last point catches most teams out. If you did not pass a team or feature identifier with the request, no tool can reconstruct it later from a bill. Unlike cloud resources, which at least persist and can be tagged after the fact, an API call is gone the moment it returns.

Where does AI spend actually show up?

AI cost is rarely one line on one bill. It typically arrives in four places at once, which is why finance so often ends up adding it up by hand.

Each layer has a different owner and a different reporting tool, and none of them naturally roll up. An engineering team can be diligent about token spend while the same company adds 400 AI seats through a renewal nobody reviewed. Pulling all four into one view is the reason a dedicated platform starts to pay for itself, and it is the gap our guide to the best AI cost management tools compares vendor by vendor.

How do you track LLM API costs?

Route calls through a gateway or instrument them with a tracing SDK, tag every request with team, feature, and environment, and store the resulting cost so you can group it later. In practice that is four steps.

• Decide the labels before you instrument. Team plus feature covers most questions, environment separates the experiments from production, and a customer or tenant identifier is worth adding early if you might ever need cost per customer. Retrofitting a label means rewriting call sites.
• Pick your capture point. A gateway captures everything that passes through it with no per-call code changes. A tracing SDK gives richer context but has to be wired into each service. Many teams run both: the gateway for control, tracing for detail.
• Store cost with the trace, not just the total. Per-request cost data is what lets you answer why the bill moved. A monthly total tells you it did.
• Alert on the shape, not just the ceiling. A budget alert fires after you cross a number you guessed. Anomaly detection models the normal daily pattern and flags deviations as they start, which is the difference between catching a runaway agent on day one and finding it on the invoice. The same logic applies to cloud spend, which we cover in cost anomaly detection vs budget alerts.

What is the difference between an LLM gateway and LLM observability?

A gateway sits inline in the request path, so it can both measure spend and block a call that exceeds a budget. Observability records what already happened and explains it in detail, but it cannot stop anything. This is the single most important distinction in the category and the one buyers most often get wrong.

If your risk is a looping agent overnight, you need the gateway. If your risk is not understanding why last month grew 40 percent, telemetry is enough. If both worry you, run the gateway for control and tracing for diagnosis, which is the common production setup at teams past their first serious AI bill.

How do you set a spending limit on OpenAI or Anthropic API usage?

Provider consoles offer account-level monthly limits and usage alerts, which protect the company but not an individual team. For per-team or per-key caps you need a gateway in front: LiteLLM enforces hard limits and soft alerts per user, key, and team and rejects calls past the ceiling. Do both. The provider limit is your backstop against catastrophe, and the gateway is your day-to-day control.

Two operational details matter. Set the soft alert well below the hard cap, because a limit that only trips at the ceiling gives nobody time to act. And decide in advance what a blocked call should do: fail loudly to a human, fall back to a smaller model, or queue. A cap that silently breaks a customer-facing feature is worse than the overspend it prevented.

How do you allocate LLM costs to a team or feature?

Pass the ownership dimension as metadata on every call and as a tag on every AI resource, then group by it. For provider APIs that usually means one API key per team or a metadata field the gateway records. For managed cloud AI it means the same cost allocation tags and cost categories you already use for the rest of the bill, so Bedrock and Azure OpenAI spend lands in the same team totals as EC2, which is exactly the exercise described in cloud cost allocation.

GPU compute needs its own answer. The cloud bill charges per node, so a shared cluster running several teams' inference workloads produces one number nobody owns. Cluster-level tooling that allocates by namespace, pod, and label is the only way to break that apart, and idle GPU time should be reported separately rather than smeared across teams, since an accelerator sitting unused is a capacity decision, not a team's consumption.

How do you reduce LLM costs without hurting quality?

The largest reliable wins are context hygiene, caching, and model routing, in that order. Teams tend to reach for a cheaper model first, when the bigger number is usually sitting in the prompt.

• Stop paying for context you do not need. Sending raw HTML, full PDFs, or an entire document when a structured extract would do is the most common and most expensive habit in production LLM code. Pre-processing sources into clean, structured data an LLM can read before they reach the context window cuts input tokens on every single call, which compounds far faster than a one-time model swap.
• Cache the repeated parts. System prompts, retrieved chunks, and few-shot examples are often identical across thousands of calls. Provider prompt caching charges a fraction of the standard input rate for those tokens.
• Route by difficulty. Send classification and extraction to a small model and reserve a frontier model for genuine reasoning. Measure quality on a fixed evaluation set before and after, so the saving is real rather than assumed.
• Cap retries and loop depth. Most catastrophic AI bills are not a pricing problem, they are a control-flow bug. An agent with no maximum step count will find the ceiling for you.

Measure per-trace cost before you optimize anything. Teams routinely misjudge which calls dominate the bill, and the honest ranking of expensive paths is often nothing like the guess.

Do you need a dedicated AI cost management tool?

Not always. If AI spend is one team calling one provider, the provider console plus your cloud billing tags will do the job for free, and you should exhaust that first. The case for a dedicated tool appears when spend crosses two model vendors and two clouds so nothing normalizes into one number, when a team needs a hard budget cap, when finance needs cost per customer to defend gross margin, or when GPU capacity sits idle in a shared cluster and nobody can see whose it is.

The other trigger is organizational rather than technical. Once AI cost is material enough that the CFO asks about it in a monthly review, the answer has to include the seats and subscriptions as well as the tokens, because that is how the invoice actually reads. Seeing model spend next to SaaS spend management in one view is what turns four partial answers into one number, and it is why anomaly alerts on AI spend belong in the same place as the rest of your cost anomaly detection.

Where to start this week

Pick your labels: team, feature, environment. Put a gateway in front of your model calls even if you only use it for measurement at first, because the day you need a hard cap you will not want to be doing an integration. Turn on the provider account limit as a backstop. Then pull one report that puts token spend, managed cloud AI, GPU compute, and AI subscriptions on the same page, and show it to whoever owns the budget. Almost every team that does this finds at least one line they did not know they were paying for.