Moving off a closed API: when do your own GPUs pay off?

Three ways to run the same traffic, priced with list prices from one day. The answer depends less on the price of a GPU than on how busy you can keep it.

Stacked layers of options in blue, with one square on the top layer chosen in vermilion

Three ways to run the same traffic#

  1. A closed API from OpenAI, Anthropic or Google. The strongest models and nothing to operate. You pay per token.
  2. A per-token API for an open model, from hosts such as DeepInfra, Together AI or Hyperstack. Open weights, nothing to operate, and prices that differ tenfold for the same model.
  3. Dedicated GPUs, run by your own team or set up through a service such as ours. You pay for every hour, whether requests arrive or not.

List prices per million tokens on 5 October 2026:

OptionModelInputOutput
OpenAIgpt-6.1-sol$2.00$10.00
OpenAIgpt-6-luna$0.10$0.50
AnthropicClaude Sonnet 5.5$2.00$10.00
AnthropicClaude Haiku 4.5$1.00$5.00
GoogleGemini 3.8 Flash (promotional until 31 December)$0.75$3.75
DeepInfraLlama 3.3 70B Turbo (FP8)$0.10$0.32
Hyperstack AI StudioLlama 3.3 70B$0.80$0.80
Together AILlama 3.3 70B$1.04$1.04

Two per-token options for Llama 3.3 70B have gone this year. Groq now lists the model as "Contact Sales", and Fireworks serves it only on dedicated GPUs.

Token counts aren't identical across these models. Anthropic says its newer tokenizer, used by Sonnet 5.5, produces about 30% more tokens for the same text, so the closed-model rows below flatter Sonnet slightly.

What two dedicated H100s cost per request#

NVIDIA publishes throughput for Llama 3.3 70B with its TensorRT-LLM engine. On two H100s at FP8, with 1,000 tokens in and 1,000 out per request, it measured 2,209 output tokens per second per GPU at full load (NVIDIA, updated 26 September 2026). The pair therefore produces about 15.9 million output tokens an hour.

At $4.00 per GPU-hour, roughly the middle of the specialist clouds' range in this month's H100 prices:

cost per million output tokens = GPU price per hour × GPUs ÷ millions of output tokens per hour
                               = $4.00 × 2 ÷ 15.9
                               = $0.50

With 1,000 output tokens per request, 1,000 requests are a million output tokens, so the pair costs $0.50 per 1,000 requests at full load. Processing the input tokens is already inside NVIDIA's measurement. At AWS's on-demand list price ($6.88) it is $0.87.

Cost per 1,000 requests (1,000 tokens in, 1,000 out), list prices on 5 October 2026

Cost per 1,000 requests (1,000 tokens in, 1,000 out), list prices on 5 October 2026
OptionBarUSD per 1,000 requests
DeepInfra, Llama 3.3 70B per-token API, FP8$0.42
Your own 2× H100 at $4.00 100% busy$0.50
OpenAI gpt-6-luna$0.60
Your own 2× H100 on AWS $6.88 list, 100% busy$0.87
Together AI, Llama 3.3 70B per-token API$2.08
Claude Haiku 4.5$6.00
Claude Sonnet 5.5, gpt-6.1-sol$12.00
Self-hosted rows: Llama 3.3 70B at FP8 on two H100s, using NVIDIA's published TensorRT-LLM throughput of 2,209 output tokens per second per GPU at full load. That is a best case: a GPU pair is never 100% busy in production. Closed models are not equivalent in quality to Llama 3.3 70B.

This is a best case, for three reasons. NVIDIA measured maximum throughput with no latency target, so each user's tokens arrive slowly at that load. Your engine, version and settings will produce a different number. And nobody keeps two GPUs 100% busy. Treat 2,209 as an input you replace with your own measurement.

Utilisation decides it#

Dedicated GPUs cost the same per hour at 3 a.m. as at midday. If they are busy a quarter of the time, every useful request carries four times the full-load cost. So the break-even point is a utilisation, not a token price:

break-even utilisation = self-hosted cost per 1,000 requests at full load ÷ API cost per 1,000 requests
Compared withAPI cost per 1,000 requestsBreak-even utilisationDaily output tokens at break-even
DeepInfra, Llama 3.3 70B$0.42nevern/a
OpenAI gpt-6-luna$0.6084%320 million
Together AI, Llama 3.3 70B$2.0824%92 million
Gemini 3.8 Flash (promotional price)$4.5011%43 million
Claude Haiku 4.5$6.008%32 million
Claude Sonnet 5.5 or gpt-6.1-sol$12.004%16 million

The same two H100s ($4.00 per GPU-hour) at different levels of use

The same two H100s ($4.00 per GPU-hour) at different levels of use
UtilisationBarUSD per 1,000 requests
Busy 100% of the time$0.50
Busy 50%$1.01
Busy 25%$2.01
Busy 10%$5.03
Busy 5% CAST AI's 2026 average$10.06
Together AI, Llama 3.3 70B for comparison$2.08
Claude Sonnet 5.5 for comparison$12.00
Cost per 1,000 requests (1,000 tokens in, 1,000 out) is the full-load cost divided by the share of time the GPUs are busy. CAST AI measured 5% average GPU utilisation across tens of thousands of production Kubernetes clusters (April 2026).

How busy do GPUs get in practice? CAST AI measured an average of 5% GPU utilisation across tens of thousands of production Kubernetes clusters, and its best example ran 136 H200s at 49% (CAST AI, April 2026). A June 2026 study found that on identical H100 hardware, effective cost ranged from $0.21 to $15.25 per million output tokens depending on load (Patil, 2026).

A single endpoint with steady traffic can run far above 5%. It won't run at 100%, because traffic follows your users' day. And since loading a 70B model takes 73 GB of weights, most teams keep at least one replica running through the quiet hours.

Long prompts change the answer#

Retrieval-augmented generation sends long prompts. With 8,192 tokens in and 1,024 out, NVIDIA's figure for the same two H100s drops to 398 output tokens per second per GPU, because each request carries eight times as much input to process. The pair's full-load cost rises to $2.79 per million output tokens.

Per 1,000 requests with 8,000 tokens in and 1,000 out, the APIs charge $1.12 (DeepInfra), $9.36 (Together AI) and $26.00 (Claude Sonnet 5.5). Break-even moves to 30% against Together and 11% against Sonnet.

Prompt caching narrows the gap further. If 7,000 of those 8,000 input tokens are a shared prefix such as instructions or reference documents, Sonnet 5.5 charges $0.20 per million for the cached part instead of $2.00, and the cost falls to $13.40. One case study of coding agents found that a 99.3% cache hit rate made a Claude API cheaper than a dedicated GPU slice (Peng, Lin and Lee, 2026). Self-hosted engines cache shared prefixes too, so measure both with your real prompts.

Costs that aren't per token#

The hourly rate leaves out work:

  • someone responsible for the endpoint when it fails at night
  • upgrading the inference engine and the model
  • finding capacity when your provider runs out in your region
  • re-running evaluations when you change model or precision

This is the work ClearCompute takes on: we compare the GPU, engine and provider for your workload, deploy the setup you pick behind one OpenAI-compatible endpoint, and keep it running. When a faster or cheaper setup appears, we tell you, and you decide whether to move. If your team already runs GPU infrastructure, the figures above are what you'll pay.

Who should move, and who shouldn't#

Dedicated GPUs pay off when:

  • Volume is steady and high. Above roughly 90 million output tokens a day against mid-priced open-model APIs, or 16 million against Sonnet-class pricing, provided an open model passes your evaluations.
  • The model is yours. A fine-tune, or an open model nobody serves per token at a fair price.
  • Data has to stay put. Traffic that must remain in one region or off shared infrastructure.
  • Latency has to be predictable. No shared queue and no rate limits set by someone else.

Paying per token is still the better deal when:

  • Traffic is small or spiky. A few million tokens a day, or bursts with long quiet gaps, leave dedicated GPUs idle most of the time.
  • Someone already serves your exact model cheaply. DeepInfra charges $0.42 per 1,000 requests for Llama 3.3 70B, less than two fully busy H100s cost to run.
  • Only a frontier closed model passes your evaluations. Then price isn't the question.

Those teams aren't who we built ClearCompute for, and a per-token API will serve them well. We work with the first group: teams moving off a closed API, or running their own model, with enough steady traffic to keep GPUs busy.

Buying GPUs outright is a different calculation. One study that assumed purchased hardware found payback within three months for models under 30B parameters, and often beyond two years for large ones (Pan et al., 2025).

How to decide in an afternoon#

  1. Export a week of production traffic and count input and output tokens per hour.
  2. Run your evaluation set against candidate open models. If none passes, stop: keep the closed API. Together AI's migration guide is a good checklist for this step, though it doesn't price self-hosting.
  3. Price per-token hosts for the model that passed.
  4. Size the deployment for your busiest hour using how much GPU memory your model needs, then price the GPUs across providers.
  5. Work out your utilisation from the hourly token counts and put it into the formula above.

If you'd rather send us the token counts, we'll do steps 3 to 5 and tell you which option wins, including when it isn't us.

How we worked this out#

Sources#

All accessed 5 October 2026.

Frequently asked questions

When is self-hosting an LLM cheaper than using an API?
When your GPUs stay busy enough. For Llama 3.3 70B on two H100s at $4.00 per GPU-hour, the break-even is about 4% utilisation against Claude Sonnet 5.5 or gpt-6.1-sol and 24% against Together AI's Llama 3.3 70B (list prices, 5 October 2026). For small or spiky traffic on a model that DeepInfra already serves at $0.10/$0.32 per million tokens, paying per token stays cheaper.
How do you calculate the break-even point for self-hosting?
Divide the self-hosted cost per 1,000 requests at full load by the API's cost for the same 1,000 requests. The result is the share of the time your GPUs must be busy to break even. Full-load cost is GPU price per hour × number of GPUs ÷ millions of output tokens per hour.
How much does it cost to run Llama 3.3 70B on your own GPUs?
Two H100s at $4.00 per GPU-hour cost $5,840 a month around the clock. At NVIDIA's published full-load throughput they produce about 11.6 billion output tokens a month, which is $0.50 per million output tokens.
What GPU utilisation is realistic?
CAST AI measured 5% average GPU utilisation across tens of thousands of production Kubernetes clusters in April 2026; its best example ran 136 H200s at 49%. A dedicated endpoint with steady traffic can do much better than 5%, but never 100%, because traffic varies through the day.
Should we move off OpenAI or Anthropic?
Only if an open model passes your own evaluations on real traffic. If one does, and your volume is steady, dedicated GPUs cost a fraction of Sonnet-class prices. If no open model is good enough, stay on the closed API.