When you take a generative AI pilot and push it to production, the financial reality hits fast. Pay-as-you-go APIs work well for quick developer experiments, but they scale poorly at enterprise volume.
To keep your margins from collapsing, you must understand the line between token price vs token cost AI.
- Token Price is what you pay to rent. It is the metered rate per million tokens charged by public API vendors, who bundle physical compute with their own margins, research subsidies, and data transfer fees.
- Token Cost is what it costs to own. It is the electricity, silicon depreciation, rack space, and software orchestration needed to produce those same tokens on hardware you control.
The math is straightforward. If your teams process hundreds of millions of tokens every day, renting leaves your balance sheet at the mercy of variable, uncapped fees. This breakdown works through the real drivers of inference cost to answer one question: at what point does it make more sense to own the machine than to rent the tokens?
The Math of the Margin: Retail Markup vs. Raw Compute
The gap between commercial API pricing and raw hardware cost is wider than most teams realize. When you buy from public API vendors, you pay a steep premium for convenience, hosting, and hardware maintenance.
To see through this markup, you have to look at how vendors calculate a retail unit of LLM compute. Commercial APIs charge a flat or tiered rate per token, but physical hardware utilization over time (GPU-seconds) is the actual metric driving their internal infrastructure costs.
Your monthly API invoice pays for three distinct buckets:
- The physical compute slice: The micro-amortization of the GPU during the milliseconds your request was processed.
- The capacity buffer: The cost of idle hardware that the provider must maintain to handle sudden, multi-tenant traffic spikes.
- The commercial markup: High gross margins required to fund ongoing research and development of future frontier models.
Self-hosting changes the game. By running models on dedicated, single-server infrastructure, your cost structure shifts from variable-use to fixed-use. Instead of a meter running on every query, your token cost becomes a predictable function of hardware amortization, facility power, and software optimization. If you run consistent workloads, this approach makes your bills drop fast.
Understanding the AI Token as a Billing Unit
To make sense of LLM economics, you have to start with the unit of consumption: the token.
A token is rarely a whole word. For English text, think of a token as a word fragment averaging about four characters; one hundred tokens yield roughly seventy-five words. Large language models do not read raw strings of text. They parse and output numerical vectors.
Using tokens as a billing yardstick is standard practice across the industry for two core reasons.
- Direct Correlation to Attention Complexity: The self-attention math inside a Transformer model runs on sequence lengths. Because of this, computational complexity scales quadratically with the number of tokens inside your context window.
- Granular Performance Profiling: Standard hardware metrics like FLOPS or raw GPU hours are bad indicators of software performance. They do not show how well a model turns silicon power into readable text. Token metrics, like tokens per second and cost per million tokens, give you an objective baseline to compare different hardware, architectures, and quantization levels.
Why Input and Output Tokens Cost Different Amounts
The real trick to understanding API bills lies in the split between how a model reads your prompt and how it writes its answer.
[ User Prompt: N Tokens ]
│
▼
┌────────────────────────────────────────────────────────┐
│ PRE-FILL PHASE (Input) │
│ - Processes entire input sequence concurrently │
│ - Compute-bound (Matrix multiplication) │
│ - Highly parallelized, high GPU utilization │
└────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ DECODING PHASE (Output) │
│ - Generates tokens sequentially, one-by-one │
│ - Memory-bound (High KV-cache read/write overhead) │
│ - Serial processing, lower parallel GPU utilization │
└────────────────────────────────────────────────────────┘
│
▼
[ Generated Token: +1 ]
Every single inference request goes through two distinct phases. Each phase taxes the underlying server hardware in different ways.
1. Input (Pre-fill) Phase
First, the model ingests your entire prompt all at once. The hardware calculates key-value activations for the whole input batch. Since the system processes these tokens in parallel, this phase is compute-bound.
It fully utilizes the Tensor Cores on your GPU. This high parallel efficiency means reading prompt tokens is actually cheap.
2. Output (Decoding) Phase
Next comes generation. The model produces text autoregressively, meaning it writes one single token at a time. To generate a single new token, the system must fetch the entire model weight set and the history of previous tokens from high-bandwidth memory into the GPU registers.
The system repeats this heavy lift for every single word fragment it outputs. This makes the decoding phase memory-bandwidth bound. Your expensive GPU cores spend most of their time sitting idle, waiting on memory retrieval. This systemic bottleneck is why API vendors like OpenAI and Anthropic charge a massive premium for output, typically pricing output tokens five to six times higher than inputs.
The High Cost of Pay-As-You-Go Public APIs
Metered pricing makes starting cheap, but as soon as you scale up production, variable billing turns into a massive, unpredictable liability for enterprise budgets.
Frontier Models Demand a Premium
OpenAI and Anthropic own the proprietary market, selling hosted access to black-box models like OpenAI’s GPT-5.6 or Anthropic’s Claude Opus 5 on a strictly metered basis.
While it works for a weekend hackathon, this model is a financial trap for enterprise scale.
- Financial Volatility: You pay for every single syllable. If a developer deploys a sloppy prompt template that pulls in massive system histories, your API bill spikes overnight. Budgets become impossible to predict.
- Funding Their Next Training Run: Your API fees do not just cover the electricity running the GPU. You are directly subsidizing their multi-billion-dollar research and development budgets to build the next foundation model.
- Compliance and Sovereignty Barriers: Regulated industries cannot send customer data to third-party endpoints. This external data transit violates strict localization mandates, including the EU AI Act, making public endpoints a non-starter.
The Limits of Metered Open-Weight Hosters
Providers like Together AI and Fireworks AI arrived to challenge the closed-source monopoly. They host open-weight models like Kimi K3, DeepSeek V4, and GLM-5.2 on highly optimized hardware, shaving down the cost per token.
Yet, the fundamental billing model remains identical. You are still renting space on someone else’s infrastructure, meaning your bills scale linearly alongside your user growth. Because these are shared, multi-tenant cloud environments, they cannot satisfy the strict compliance requirements of a true air-gapped, on-premises deployment.
The Anatomy of Self-Hosted Token Costs
To bypass hyperscaler markups, you have to look at the raw physical math. Running open-weight models like Ornith-1.0, Kimi K3, DeepSeek V4, or GLM-5.2 on your own iron means trading variable bills for predictable hardware variables.
Forget paying a tax on every single word. Your actual self-hosted expense comes down to a mix of silicon amortization, power, and how hard your software stack can push the chips.
GPU Amortization and Hardware Lifecycle
Silicon is your main upfront cost. For serious enterprise workloads, the current generation is Blackwell, and it spans a wide range of single-server configurations. The RTX PRO 6000 Blackwell Server Edition is the volume option: 96GB of GDDR7 and 600W per card over PCIe Gen5, with up to eight cards in a standard 2U or 4U chassis. Above that sit HGX B200 nodes at 180GB of HBM3e per GPU, and HGX B300 (Blackwell Ultra) at 288GB — 2.3TB across eight cards, enough to hold a trillion-parameter model resident on a single box. The previous generation, H100, H200, and L40S, is still widely deployed and still pencils out, especially on the secondary market or for smaller models.
Finance teams usually write these assets off over a 3-year to 5-year depreciation lifecycle.
- CapEx Baseline: A single server packed with high-end GPUs, dual CPUs, deep system memory, and fast NVMe storage goes on your books as a capital expense that loses value monthly.
- The Utilization Variable: With cloud APIs, you pay for what you use. On-prem, your server costs the exact same amount whether it sits completely idle or runs at a flat-out 95% capacity all day and night. You must maximize utilization to drive your amortized cost per token straight to the floor.
Power, Cooling, and Facility Overhead
Chips run hot. Calculating the real running cost means adding up the electricity sucked back by the server itself, plus the power needed to cool the room, which you calculate using Power Usage Effectiveness (PUE).
- Active Draw: An 8-GPU node peaks at 6 kW with RTX PRO 6000 Server Edition cards, but can go north of 10 kW if fitted with B300s, once you add host CPUs, memory, and storage on top of the accelerators.
- Idle Draw: Even waiting around, that same machine still sips 1 kW to 2.5 kW, depending on the cards.
- Tokens-Per-Watt Efficiency: The number that actually matters is how many tokens you get out of every single watt. Highly tuned serving stacks squeeze the absolute maximum work out of every active cycle, which keeps you from wasting money on idle power.
Software Stack Efficiency and Model Optimization
Your software layer controls the hardware. If you run stock, unoptimized inference code on high-end GPUs, you are leaving expensive computing power on the table.
We use hardware-native runtimes like vLLM, TensorRT-LLM, or custom bare-metal engines because they squeeze more work out of the exact same silicon. They do this through a few specific techniques:
- PagedAttention: This stops KV-cache memory fragmentation by splitting memory into non-contiguous virtual blocks, letting you handle much larger batches of users concurrently.
- Continuous Batching: Instead of waiting for every user request in a batch to finish before starting the next round, the system injects new requests the millisecond an older one finishes.
- Native 4-bit Formats (NVFP4, MXFP4): Blackwell’s fifth-generation Tensor Cores run FP4 natively, and NVFP4 is roughly 2.35x faster than INT4 on the same silicon at close to baseline accuracy. Published checkpoints still need conversion work, and the gap is not academic. Ornith-1.0 ships as FP8 at roughly 397GB, which does not fit on four RTX PRO 6000 cards. Converted to NVFP4 it lands near 198GB and leaves most of the board free for KV cache. Format choice and KV-cache precision then decide how many concurrent requests the node can hold.
Comparing the Economics: Retail API vs. Hyperscaler Cloud vs. Private Infrastructure
The two figures behave differently, and the difference is structural rather than a matter of degree. The table below breaks down what sits inside each one.
| Dimension | Retail Token Price (rent) | Self-Hosted Token Cost (own) |
|---|---|---|
| Unit of billing | Per million tokens, billed separately for input and output | Per node-hour, converted to a token figure by measured throughput |
| What the figure covers | GPU time slice, idle capacity buffer, vendor margin and R&D recovery | Hardware amortization, power and cooling, platform engineering time |
| Input vs. output | Priced separately, output typically 5x to 6x input | Same physical asymmetry, but no separate line item |
| Curve against volume | Linear. Double the usage, double the invoice | Flat until the node saturates, then steps up with the next server |
| Marginal cost of one more token | The list rate, every time | Close to zero while spare capacity exists |
| Where the waste goes | Idle capacity you pay for inside the rate, invisible to you | Idle capacity you own and can see |
| Failure mode | An invoice nobody forecast | Stranded hardware you already bought |
The Real Cost of Renting Cloud GPUs
Renting virtualized GPUs from AWS or Azure looks like an easy compromise. You avoid public API margins without the upfront cost of buying physical hardware, but this middle path brings a fresh set of expensive headaches.
First, you pay a heavy virtualization tax. Hyperscalers pad their cloud margins by marking up raw compute, meaning a virtualized instance costs significantly more than equivalent bare metal. Next come the network egress fees. Moving your enterprise data into a public cloud is free, but pulling it back out to on-premises systems or external clients triggers bandwidth charges that mount quickly.
Finally, renting a raw GPU gives you a blank slate, not a working AI service. Your team still has to handle the tedious work of configuring operating systems, installing CUDA drivers, setting up Kubernetes, and optimizing inference engines like vLLM. This creates a heavy, continuous engineering drain.
The Utilization Threshold for Dedicated Hardware
Renting tokens and owning the machine that makes them cross at exactly one number, and this is it.
Utilization is the share of your node’s output capacity you actually consume: output tokens produced divided by output tokens the machine could produce running flat out. It measures throughput, not uptime, and the gap between those two is where most self-hosting business cases quietly fail. A server answering queries all day at a fraction of its batch capacity is fully occupied and barely utilized.
The asymmetry is what makes the number decisive. Renting tracks consumption: use half as much, pay half as much. Owning bills you for capacity whether you draw on it or not, depreciating and drawing power the whole time. So the decision reduces to a single question. Do you consume enough to own instead of rent?
What an hour on the machine actually costs
Take a GB200 NVL4: four Blackwell GPUs and two Grace CPUs on a single board, 768GB of HBM3e, 5.5 kW at the wall, roughly $235,000 to buy. Depreciate it over three years, colocate it for $700 a month, and pay $0.15/kWh for industrial power at a PUE of 1.3.
| Line item | Working | $/hour |
|---|---|---|
| Hardware amortization | $235,000 / (3 yr x 8,760 hr) | 8.94 |
| Power | 5.5 kW x 1.3 PUE x $0.15/kWh | 1.07 |
| Facility | $700/mo / 730 hr | 0.96 |
| Total | 10.97 |
Just under eleven dollars an hour, running or idle. Now convert that into tokens. Serving a trillion-parameter open-weight model on this box, call it 1,000 output tokens per second sustained. That works out to 3.6 million output tokens per hour at saturation.
$10.97 / 3.6 = $3.05 per million output tokens.
That is the floor: silicon, electricity, and floor space, with nobody operating it yet.
The curve
Your real cost per token is that floor divided by your utilization. Run the machine at half capacity and every token costs twice as much.
| Utilization | Cost per million output tokens |
|---|---|
| 100% | $3.05 |
| 50% | $6.09 |
| 25% | $12.19 |
| 10% | $30.47 |
API pricing has no such curve. You pay list price on the first token of the month and the same rate on the billionth. So the comparison collapses to one line of arithmetic:
Break-even utilization = your cost per million at saturation / the API price per million.
Above that line the machine is cheaper. Below it, the meter wins.
The variable nobody budgets for
The floor above leaves out the one line that varies by an order of magnitude: the people who keep the stack running.
This is not ordinary server administration. Someone has to convert an MXFP4 checkpoint into the format Blackwell runs natively and prove the conversion did not cost accuracy. Someone has to size KV-cache precision against context length, tune continuous batching to the traffic shape, and repeat the whole exercise every time a serving engine ships a release or a lab publishes new weights. That skill set is scarce, and it overlaps very little with the people already keeping your infrastructure alive.
| Ops model | Annual | $/hr | $/M tokens | Break-even vs Opus 5 | vs Sonnet 5 |
|---|---|---|---|---|---|
| Excluded (floor) | — | 10.97 | 3.05 | 12% | 20% |
| Licensed platform | $80,000 | 20.10 | 5.58 | 22% | 37% |
| In-house team (2 FTE) | $300,000 | 45.22 | 12.56 | 50% | 84% |
Read the Sonnet row. Twenty percent if operations were free. Thirty-seven with a licensed platform. Eighty-four percent if you staff it yourself, which is less a threshold than a ceiling, since no production workload holds that utilization once demand varies across a day. Against Sonnet-tier pricing, a self-built stack on a single node does not pay for itself.
The Opus row runs the other way. Fifty percent is demanding but reachable, so if you are displacing frontier-tier calls, even a bespoke build can clear the bar.
The shape of the answer is the same either way. Two platform engineers cost $34.25 an hour. The GB200 NVL4, its electricity, and the rack it occupies cost $10.97 an hour combined. Even the licensed path, at $9.13 an hour, sits level with the $8.94 of amortized silicon. The expensive part of running your own inference is not the hardware.
One caveat on the in-house column. Platform teams do not scale per server. You hire that capability once, so at a fleet of one the entire salary lands on a single machine, and the per-node figure drops as you add more. A licensed platform stays flat per node. Which one wins depends on how large your fleet is going to get, and most enterprises testing the water start at one.
Match the machine to the model
The GB200 NVL4 above is the expensive end, and most workloads do not need it.
Ornith-1.0 is 397B parameters, MIT licensed, and posts 82.4 on SWE-Bench Verified, ahead of Claude Opus 4.7. It is a coding and agentic model, so the comparison against Sonnet 5 holds for those workloads rather than as a general claim. Converted from its published FP8 checkpoint to NVFP4 it occupies roughly 198GB, which sits comfortably on four RTX PRO 6000 Server Edition cards. That box costs $75,000, draws 3.0 kW, and sustains around 700 output tokens per second.
| 4x RTX PRO 6000 SE | GB200 NVL4 | |
|---|---|---|
| CapEx | $75,000 | $235,000 |
| Power at the wall | 3.0 kW | 5.5 kW |
| Sustained output | 700 tok/s | 1,000 tok/s |
| Bare node | $3.99/hr | $10.97/hr |
| With licensed platform | $7.99/hr | $20.10/hr |
| Cost per million tokens | $3.17 | $5.58 |
| Break-even vs Sonnet 5 | 21% | 37% |
| Break-even vs Opus 5 | 13% | 22% |
The smaller box produces cheaper tokens, and not marginally: $3.17 against $5.58. Spending three times as much on hardware does not buy cheaper inference, it buys a larger model. If a 397B model handles your workload, your economics are roughly twice as good as the team running a trillion-parameter model on frontier silicon to do the same job.
The same arithmetic makes build-versus-buy more lopsided, not less. Two platform engineers cost $34.25 an hour regardless of what they operate. On the GB200 that is 3.1 times the cost of the infrastructure. On the RTX box it is 8.6 times, and break-even against Sonnet 5 climbs past 100%, meaning a self-built stack on that machine never pays for itself at any utilization. The cheaper your hardware, the more the salary line decides the outcome.
Which tier you are replacing
The table compares against Opus 5 and Sonnet 5 rather than the cheaper tiers, and that choice is deliberate. An open-weight model that trades benchmark wins with frontier systems is not a substitute for a small fast model, so pricing it against one would flatter the result. The fair test is the tier your open model actually displaces.
It also explains why a single threshold was never going to work. On the licensed platform line, break-even against Fable 5 at $50 per million output is 11%. Against Haiku 4.5 at $5, it exceeds 100%, meaning it never breaks even at all. Same hardware, same software, same electricity bill. The tier you are substituting for swings the answer by a factor of ten, further than your token volume does.
Running the numbers on your own workload
Four figures give you your own threshold:
- Measured output tokens per second on your hardware, under your traffic, not from a vendor benchmark.
- All-in hourly node cost, including whoever operates it.
- Your current blended API rate per million, weighted by your real input-to-output ratio.
- The model tier your open-weight candidate genuinely replaces on your tasks.
Divide the second by the first for your saturated cost per million. Divide that by the third. The result is the utilization you need to justify the move. Compare it against what you are achieving today.
Owning Without Building
Owning the infrastructure does not mean building the platform. Those are separate decisions, and the arithmetic above only punishes the second one. A machine you own can run a system somebody else has already engineered.
That distinction is what the licensed platform line in the tables above represents: a fixed annual fee per server, scaled to the class of machine, with no metering on top of it.
DiscreteStack ships as one system: model, serving stack, quantization, and control plane, built and versioned together. The entry configuration pairs Ornith-1.0 with four RTX PRO 6000 Server Edition cards. Seven hundred output tokens per second is what that configuration does out of the box.
- The optimization ships with the product. Format conversion, KV-cache precision, batching behaviour, and memory layout are fixed properties of the build, chosen for the model and the silicon it runs on and validated against baseline accuracy before release. Each version carries the tuning of the ones before it.
- One fee per server, no token metering. Your bill is a function of how many machines you run, not how hard your teams use them. That is what makes the curve in the previous section work: every additional token pushed through a node you have already paid for arrives at close to zero marginal cost.
- Governance built for European obligations. Identity and access management, audit trails on every request, and access logs that survive an ISO or EU AI Act review. The platform is built in the EU and runs fully air-gapped where required. For workloads that fail step one of the framework below, this is the entire reason to own rather than rent.
The model question is settled before the appliance arrives. Several of the strongest open-weight models available today come out of Chinese labs, which puts every European organisation running them into a compliance conversation about provenance. Here the model is selected, validated, and shipped as part of the system, under an EU-built control plane, on hardware inside your building.
What this does not fix. If your API spend sits below the monthly cost of a node, a licence does not rescue the case and you should keep renting. If your demand arrives in short spikes separated by long idle stretches, you are buying capacity you will not use, and the utilization arithmetic will say so plainly. And past a certain fleet size the calculation inverts: with enough machines, the salary that looks absurd against one box becomes reasonable spread across ten, and you should hire the team.
Choosing Your Enterprise Inference Strategy
Three questions, in that order. The first can settle it before the other two get a vote.
START
|
v
[1] Compliance or sovereignty?
|
NO <------+------> YES
| |
v |
[2] API spend above |
$5,800/mo? |
| |
NO <---+---> YES |
| | |
| +------+-------+
| |
v v
+--------+ +---------+
| RENT | | OWN |
+--------+ +---------+
|
v
[3] Platform team to spare?
|
NO <------+------> YES
| |
v v
BUY BUILD
(~8x node cost)
1. Map your security and compliance boundaries
Handling personally identifiable information, proprietary source code, patient records, or financial transaction logs on public APIs introduces massive risk. When your internal audit demands zero-trust isolation or fully air-gapped deployments, public clouds are out. You need private on-premises infrastructure to guarantee absolute data sovereignty.
2. Compare your bill to the cost of a node
An entry-class node costs about $5,800 a month, all in, running or idle. That is the reference point for the whole decision. If you spend less than that on metered endpoints today, renting is cheaper and the arithmetic ends there.
Spend more, and the question turns to utilization. A node’s cost is fixed, so every additional token you push through it lowers your cost per million. At 21% of capacity you match Sonnet 5 list pricing, and past that the gap widens in your favour.
3. Decide whether to build or buy the platform
Owning the hardware and building the software that runs on it are separate choices, and only the second one is expensive. Two platform engineers cost the same whether they tune one node or ten, so on a single entry-class box that salary runs roughly eight times the hourly cost of the hardware, and break-even against Sonnet 5 never arrives at any utilization.
The gap between what a token costs and what you are charged for it is real, and at scale it is large. On its own it settles nothing. Ownership pays when you use enough of the capacity you have bought, and when operating that capacity costs less than the capacity itself. Both are arithmetic rather than judgement, and both are specific to your workload. Run them.