Moving off a closed API: when do your own GPUs pay off?
Three ways to run the same traffic, priced with list prices from one day. The answer depends less on the price of a GPU than on how busy you can keep it.

Three ways to run the same traffic#
- A closed API from OpenAI, Anthropic or Google. The strongest models and nothing to operate. You pay per token.
- A per-token API for an open model, from hosts such as DeepInfra, Together AI or Hyperstack. Open weights, nothing to operate, and prices that differ tenfold for the same model.
- Dedicated GPUs, run by your own team or set up through a service such as ours. You pay for every hour, whether requests arrive or not.
List prices per million tokens on 5 October 2026:
| Option | Model | Input | Output |
|---|---|---|---|
| OpenAI | gpt-6.1-sol | $2.00 | $10.00 |
| OpenAI | gpt-6-luna | $0.10 | $0.50 |
| Anthropic | Claude Sonnet 5.5 | $2.00 | $10.00 |
| Anthropic | Claude Haiku 4.5 | $1.00 | $5.00 |
| Gemini 3.8 Flash (promotional until 31 December) | $0.75 | $3.75 | |
| DeepInfra | Llama 3.3 70B Turbo (FP8) | $0.10 | $0.32 |
| Hyperstack AI Studio | Llama 3.3 70B | $0.80 | $0.80 |
| Together AI | Llama 3.3 70B | $1.04 | $1.04 |
Two per-token options for Llama 3.3 70B have gone this year. Groq now lists the model as "Contact Sales", and Fireworks serves it only on dedicated GPUs.
Token counts aren't identical across these models. Anthropic says its newer tokenizer, used by Sonnet 5.5, produces about 30% more tokens for the same text, so the closed-model rows below flatter Sonnet slightly.
What two dedicated H100s cost per request#
NVIDIA publishes throughput for Llama 3.3 70B with its TensorRT-LLM engine. On two H100s at FP8, with 1,000 tokens in and 1,000 out per request, it measured 2,209 output tokens per second per GPU at full load (NVIDIA, updated 26 September 2026). The pair therefore produces about 15.9 million output tokens an hour.
At $4.00 per GPU-hour, roughly the middle of the specialist clouds' range in this month's H100 prices:
cost per million output tokens = GPU price per hour × GPUs ÷ millions of output tokens per hour
= $4.00 × 2 ÷ 15.9
= $0.50
With 1,000 output tokens per request, 1,000 requests are a million output tokens, so the pair costs $0.50 per 1,000 requests at full load. Processing the input tokens is already inside NVIDIA's measurement. At AWS's on-demand list price ($6.88) it is $0.87.
Cost per 1,000 requests (1,000 tokens in, 1,000 out), list prices on 5 October 2026
| Option | Bar | USD per 1,000 requests |
|---|---|---|
| DeepInfra, Llama 3.3 70B per-token API, FP8 | $0.42 | |
| Your own 2× H100 at $4.00 100% busy | $0.50 | |
| OpenAI gpt-6-luna | $0.60 | |
| Your own 2× H100 on AWS $6.88 list, 100% busy | $0.87 | |
| Together AI, Llama 3.3 70B per-token API | $2.08 | |
| Claude Haiku 4.5 | $6.00 | |
| Claude Sonnet 5.5, gpt-6.1-sol | $12.00 |
This is a best case, for three reasons. NVIDIA measured maximum throughput with no latency target, so each user's tokens arrive slowly at that load. Your engine, version and settings will produce a different number. And nobody keeps two GPUs 100% busy. Treat 2,209 as an input you replace with your own measurement.
Utilisation decides it#
Dedicated GPUs cost the same per hour at 3 a.m. as at midday. If they are busy a quarter of the time, every useful request carries four times the full-load cost. So the break-even point is a utilisation, not a token price:
break-even utilisation = self-hosted cost per 1,000 requests at full load ÷ API cost per 1,000 requests
| Compared with | API cost per 1,000 requests | Break-even utilisation | Daily output tokens at break-even |
|---|---|---|---|
| DeepInfra, Llama 3.3 70B | $0.42 | never | n/a |
| OpenAI gpt-6-luna | $0.60 | 84% | 320 million |
| Together AI, Llama 3.3 70B | $2.08 | 24% | 92 million |
| Gemini 3.8 Flash (promotional price) | $4.50 | 11% | 43 million |
| Claude Haiku 4.5 | $6.00 | 8% | 32 million |
| Claude Sonnet 5.5 or gpt-6.1-sol | $12.00 | 4% | 16 million |
The same two H100s ($4.00 per GPU-hour) at different levels of use
| Utilisation | Bar | USD per 1,000 requests |
|---|---|---|
| Busy 100% of the time | $0.50 | |
| Busy 50% | $1.01 | |
| Busy 25% | $2.01 | |
| Busy 10% | $5.03 | |
| Busy 5% CAST AI's 2026 average | $10.06 | |
| Together AI, Llama 3.3 70B for comparison | $2.08 | |
| Claude Sonnet 5.5 for comparison | $12.00 |
How busy do GPUs get in practice? CAST AI measured an average of 5% GPU utilisation across tens of thousands of production Kubernetes clusters, and its best example ran 136 H200s at 49% (CAST AI, April 2026). A June 2026 study found that on identical H100 hardware, effective cost ranged from $0.21 to $15.25 per million output tokens depending on load (Patil, 2026).
A single endpoint with steady traffic can run far above 5%. It won't run at 100%, because traffic follows your users' day. And since loading a 70B model takes 73 GB of weights, most teams keep at least one replica running through the quiet hours.
Long prompts change the answer#
Retrieval-augmented generation sends long prompts. With 8,192 tokens in and 1,024 out, NVIDIA's figure for the same two H100s drops to 398 output tokens per second per GPU, because each request carries eight times as much input to process. The pair's full-load cost rises to $2.79 per million output tokens.
Per 1,000 requests with 8,000 tokens in and 1,000 out, the APIs charge $1.12 (DeepInfra), $9.36 (Together AI) and $26.00 (Claude Sonnet 5.5). Break-even moves to 30% against Together and 11% against Sonnet.
Prompt caching narrows the gap further. If 7,000 of those 8,000 input tokens are a shared prefix such as instructions or reference documents, Sonnet 5.5 charges $0.20 per million for the cached part instead of $2.00, and the cost falls to $13.40. One case study of coding agents found that a 99.3% cache hit rate made a Claude API cheaper than a dedicated GPU slice (Peng, Lin and Lee, 2026). Self-hosted engines cache shared prefixes too, so measure both with your real prompts.
Costs that aren't per token#
The hourly rate leaves out work:
- someone responsible for the endpoint when it fails at night
- upgrading the inference engine and the model
- finding capacity when your provider runs out in your region
- re-running evaluations when you change model or precision
This is the work ClearCompute takes on: we compare the GPU, engine and provider for your workload, deploy the setup you pick behind one OpenAI-compatible endpoint, and keep it running. When a faster or cheaper setup appears, we tell you, and you decide whether to move. If your team already runs GPU infrastructure, the figures above are what you'll pay.
Who should move, and who shouldn't#
Dedicated GPUs pay off when:
- Volume is steady and high. Above roughly 90 million output tokens a day against mid-priced open-model APIs, or 16 million against Sonnet-class pricing, provided an open model passes your evaluations.
- The model is yours. A fine-tune, or an open model nobody serves per token at a fair price.
- Data has to stay put. Traffic that must remain in one region or off shared infrastructure.
- Latency has to be predictable. No shared queue and no rate limits set by someone else.
Paying per token is still the better deal when:
- Traffic is small or spiky. A few million tokens a day, or bursts with long quiet gaps, leave dedicated GPUs idle most of the time.
- Someone already serves your exact model cheaply. DeepInfra charges $0.42 per 1,000 requests for Llama 3.3 70B, less than two fully busy H100s cost to run.
- Only a frontier closed model passes your evaluations. Then price isn't the question.
Those teams aren't who we built ClearCompute for, and a per-token API will serve them well. We work with the first group: teams moving off a closed API, or running their own model, with enough steady traffic to keep GPUs busy.
Buying GPUs outright is a different calculation. One study that assumed purchased hardware found payback within three months for models under 30B parameters, and often beyond two years for large ones (Pan et al., 2025).
How to decide in an afternoon#
- Export a week of production traffic and count input and output tokens per hour.
- Run your evaluation set against candidate open models. If none passes, stop: keep the closed API. Together AI's migration guide is a good checklist for this step, though it doesn't price self-hosting.
- Price per-token hosts for the model that passed.
- Size the deployment for your busiest hour using how much GPU memory your model needs, then price the GPUs across providers.
- Work out your utilisation from the hourly token counts and put it into the formula above.
If you'd rather send us the token counts, we'll do steps 3 to 5 and tell you which option wins, including when it isn't us.
How we worked this out#
Sources#
All accessed 5 October 2026.
- API pricing: OpenAI, Anthropic, Google Gemini, DeepInfra, Together AI, Hyperstack, Groq models, Fireworks AI model page
- Throughput: NVIDIA TensorRT-LLM performance overview
- Utilisation: CAST AI, 2026 State of Kubernetes Resource Optimization, Patil, "Beyond Per-Token Pricing" (2026)
- Break-even studies: Peng, Lin and Lee, "Inference Economics of Enterprise Coding Agents" (2026), Pan et al., "A Cost-Benefit Analysis of On-Premise Large Language Model Deployment" (2025)
- Together AI, "Migrating from closed to open source models" (September 2026)

