# Moving off a closed API: when do your own GPUs pay off?

> We priced the same Llama 3.3 70B traffic three ways on 5 October 2026: a closed API, a per-token open-model API and dedicated H100s. Dedicated GPUs beat Sonnet-class prices above about 4% utilisation; for small or spiky traffic on a widely hosted model, the cheapest per-token hosts still cost less.

By Ronit Ahuja, ClearCompute. Published 2026-10-05, updated 2026-10-26. Canonical: https://clearcompute.cc/blog/self-hosting-vs-api-cost

> **The short answer.** At list prices on 5 October 2026, Llama 3.3 70B on two H100s costs about **$0.50 per 1,000 requests** (1,000 tokens in and 1,000 out) when the GPUs are busy all the time. If an open model is good enough for your task, that beats Claude Sonnet 5.5 or gpt-6.1-sol ($12.00) once the GPUs are busy more than **4%** of the time, and Together AI's Llama 3.3 70B ($2.08) above **24%**. Dedicated GPUs pay off for teams with steady traffic, their own fine-tuned model, or data that has to stay in one region. For small or spiky traffic on a model a host such as DeepInfra already serves ($0.42 per 1,000 requests), paying per token stays cheaper, and we would tell you so.

## Three ways to run the same traffic

1. **A closed API** from OpenAI, Anthropic or Google. The strongest models and nothing to operate. You pay per token.
2. **A per-token API for an open model**, from hosts such as DeepInfra, Together AI or Hyperstack. Open weights, nothing to operate, and prices that differ tenfold for the same model.
3. **Dedicated GPUs**, run by your own team or set up through a service such as ours. You pay for every hour, whether requests arrive or not.

List prices per million tokens on 5 October 2026:

| Option | Model | Input | Output |
|---|---|---|---|
| OpenAI | gpt-6.1-sol | $2.00 | $10.00 |
| OpenAI | gpt-6-luna | $0.10 | $0.50 |
| Anthropic | Claude Sonnet 5.5 | $2.00 | $10.00 |
| Anthropic | Claude Haiku 4.5 | $1.00 | $5.00 |
| Google | Gemini 3.8 Flash (promotional until 31 December) | $0.75 | $3.75 |
| DeepInfra | Llama 3.3 70B Turbo (FP8) | $0.10 | $0.32 |
| Hyperstack AI Studio | Llama 3.3 70B | $0.80 | $0.80 |
| Together AI | Llama 3.3 70B | $1.04 | $1.04 |

Two per-token options for Llama 3.3 70B have gone this year. Groq now lists the model as "Contact Sales", and Fireworks serves it only on dedicated GPUs.

Token counts aren't identical across these models. Anthropic says its newer tokenizer, used by Sonnet 5.5, produces about 30% more tokens for the same text, so the closed-model rows below flatter Sonnet slightly.

## What two dedicated H100s cost per request

NVIDIA publishes throughput for Llama 3.3 70B with its TensorRT-LLM engine. On two H100s at FP8, with 1,000 tokens in and 1,000 out per request, it measured **2,209 output tokens per second per GPU** at full load ([NVIDIA, updated 26 September 2026](https://nvidia.github.io/TensorRT-LLM/latest/developer-guide/perf-overview.html)). The pair therefore produces about 15.9 million output tokens an hour.

At $4.00 per GPU-hour, roughly the middle of the specialist clouds' range in [this month's H100 prices](/blog/h100-cloud-prices):

```text
cost per million output tokens = GPU price per hour × GPUs ÷ millions of output tokens per hour
                               = $4.00 × 2 ÷ 15.9
                               = $0.50
```

With 1,000 output tokens per request, 1,000 requests are a million output tokens, so the pair costs **$0.50 per 1,000 requests** at full load. Processing the input tokens is already inside NVIDIA's measurement. At AWS's on-demand list price ($6.88) it is $0.87.

**Cost per 1,000 requests (1,000 tokens in, 1,000 out), list prices on 5 October 2026**

| Option | USD per 1,000 requests |
|---|---|
| DeepInfra, Llama 3.3 70B (per-token API, FP8) | $0.42 |
| Your own 2× H100 at $4.00 (100% busy) | $0.50 |
| OpenAI gpt-6-luna | $0.60 |
| Your own 2× H100 on AWS ($6.88 list, 100% busy) | $0.87 |
| Together AI, Llama 3.3 70B (per-token API) | $2.08 |
| Claude Haiku 4.5 | $6.00 |
| Claude Sonnet 5.5, gpt-6.1-sol | $12.00 |

*Self-hosted rows: Llama 3.3 70B at FP8 on two H100s, using NVIDIA's published TensorRT-LLM throughput of 2,209 output tokens per second per GPU at full load. That is a best case: a GPU pair is never 100% busy in production. Closed models are not equivalent in quality to Llama 3.3 70B.*

This is a best case, for three reasons. NVIDIA measured maximum throughput with no latency target, so each user's tokens arrive slowly at that load. Your engine, version and settings will produce a different number. And nobody keeps two GPUs 100% busy. Treat 2,209 as an input you replace with your own measurement.

## Utilisation decides it

Dedicated GPUs cost the same per hour at 3 a.m. as at midday. If they are busy a quarter of the time, every useful request carries four times the full-load cost. So the break-even point is a utilisation, not a token price:

```text
break-even utilisation = self-hosted cost per 1,000 requests at full load ÷ API cost per 1,000 requests
```

| Compared with | API cost per 1,000 requests | Break-even utilisation | Daily output tokens at break-even |
|---|---|---|---|
| DeepInfra, Llama 3.3 70B | $0.42 | never | n/a |
| OpenAI gpt-6-luna | $0.60 | 84% | 320 million |
| Together AI, Llama 3.3 70B | $2.08 | 24% | 92 million |
| Gemini 3.8 Flash (promotional price) | $4.50 | 11% | 43 million |
| Claude Haiku 4.5 | $6.00 | 8% | 32 million |
| Claude Sonnet 5.5 or gpt-6.1-sol | $12.00 | 4% | 16 million |

**The same two H100s ($4.00 per GPU-hour) at different levels of use**

| Utilisation | USD per 1,000 requests |
|---|---|
| Busy 100% of the time | $0.50 |
| Busy 50% | $1.01 |
| Busy 25% | $2.01 |
| Busy 10% | $5.03 |
| Busy 5% (CAST AI's 2026 average) | $10.06 |
| Together AI, Llama 3.3 70B (for comparison) | $2.08 |
| Claude Sonnet 5.5 (for comparison) | $12.00 |

*Cost per 1,000 requests (1,000 tokens in, 1,000 out) is the full-load cost divided by the share of time the GPUs are busy. CAST AI measured 5% average GPU utilisation across tens of thousands of production Kubernetes clusters (April 2026).*

How busy do GPUs get in practice? CAST AI measured an average of **5% GPU utilisation** across tens of thousands of production Kubernetes clusters, and its best example ran 136 H200s at 49% ([CAST AI, April 2026](https://cast.ai/blog/2026-state-of-kubernetes-resource-optimization-cpu-at-8-memory-at-20-and-getting-worse/)). A June 2026 study found that on identical H100 hardware, effective cost ranged from $0.21 to $15.25 per million output tokens depending on load ([Patil, 2026](https://arxiv.org/abs/2606.11690)).

A single endpoint with steady traffic can run far above 5%. It won't run at 100%, because traffic follows your users' day. And since loading a 70B model takes 73 GB of weights, most teams keep at least one replica running through the quiet hours.

## Long prompts change the answer

Retrieval-augmented generation sends long prompts. With 8,192 tokens in and 1,024 out, NVIDIA's figure for the same two H100s drops to 398 output tokens per second per GPU, because each request carries eight times as much input to process. The pair's full-load cost rises to $2.79 per million output tokens.

Per 1,000 requests with 8,000 tokens in and 1,000 out, the APIs charge $1.12 (DeepInfra), $9.36 (Together AI) and $26.00 (Claude Sonnet 5.5). Break-even moves to 30% against Together and 11% against Sonnet.

Prompt caching narrows the gap further. If 7,000 of those 8,000 input tokens are a shared prefix such as instructions or reference documents, Sonnet 5.5 charges $0.20 per million for the cached part instead of $2.00, and the cost falls to $13.40. One case study of coding agents found that a 99.3% cache hit rate made a Claude API cheaper than a dedicated GPU slice ([Peng, Lin and Lee, 2026](https://arxiv.org/abs/2607.13080)). Self-hosted engines cache shared prefixes too, so measure both with your real prompts.

## Costs that aren't per token

The hourly rate leaves out work:

- someone responsible for the endpoint when it fails at night
- upgrading the inference engine and the model
- finding capacity when your provider runs out in your region
- re-running evaluations when you change model or precision

This is the work ClearCompute takes on: we compare the GPU, engine and provider for your workload, deploy the setup you pick behind one OpenAI-compatible endpoint, and keep it running. When a faster or cheaper setup appears, we tell you, and you decide whether to move. If your team already runs GPU infrastructure, the figures above are what you'll pay.

## Who should move, and who shouldn't

Dedicated GPUs pay off when:

- **Volume is steady and high.** Above roughly 90 million output tokens a day against mid-priced open-model APIs, or 16 million against Sonnet-class pricing, provided an open model passes your evaluations.
- **The model is yours.** A fine-tune, or an open model nobody serves per token at a fair price.
- **Data has to stay put.** Traffic that must remain in one region or off shared infrastructure.
- **Latency has to be predictable.** No shared queue and no rate limits set by someone else.

Paying per token is still the better deal when:

- **Traffic is small or spiky.** A few million tokens a day, or bursts with long quiet gaps, leave dedicated GPUs idle most of the time.
- **Someone already serves your exact model cheaply.** DeepInfra charges $0.42 per 1,000 requests for Llama 3.3 70B, less than two fully busy H100s cost to run.
- **Only a frontier closed model passes your evaluations.** Then price isn't the question.

Those teams aren't who we built ClearCompute for, and a per-token API will serve them well. We work with the first group: teams moving off a closed API, or running their own model, with enough steady traffic to keep GPUs busy.

Buying GPUs outright is a different calculation. One study that assumed purchased hardware found payback within three months for models under 30B parameters, and often beyond two years for large ones ([Pan et al., 2025](https://arxiv.org/abs/2509.18101)).

## How to decide in an afternoon

1. Export a week of production traffic and count input and output tokens per hour.
2. Run your evaluation set against candidate open models. If none passes, stop: keep the closed API. Together AI's [migration guide](https://www.together.ai/blog/migrating-from-closed-to-open-source-models) is a good checklist for this step, though it doesn't price self-hosting.
3. Price per-token hosts for the model that passed.
4. Size the deployment for your busiest hour using [how much GPU memory your model needs](/blog/gpu-memory-70b-model), then price the GPUs across providers.
5. Work out your utilisation from the hourly token counts and put it into the formula above.

If you'd rather send us the token counts, we'll do steps 3 to 5 and tell you which option wins, including when it isn't us.

## How we worked this out

> **How we worked this out.** API prices come from each provider's public pricing page on 5 October 2026, at standard rates without batch or committed-use discounts. Self-hosted throughput is NVIDIA's published TensorRT-LLM figure for Llama 3.3 70B at FP8 with tensor parallelism across two H100 SXM GPUs (2,209 output tokens per second per GPU at 1,000 tokens in and out, and 398 at 8,192 in and 1,024 out), as shown on 26 September 2026. GPU prices are on-demand list prices from [our October survey](/blog/h100-cloud-prices). We count a request as 1,000 input and 1,000 output tokens and ignore storage, egress and engineering time. Closed models are not equivalent in quality to Llama 3.3 70B, and token counts differ between tokenizers, so treat the closed-model comparisons as conditional on an open model passing your evaluations. We did not run these benchmarks ourselves.

## Sources

All accessed 5 October 2026.

- API pricing: [OpenAI](https://developers.openai.com/api/docs/pricing), [Anthropic](https://platform.claude.com/docs/en/about-claude/pricing), [Google Gemini](https://ai.google.dev/gemini-api/docs/pricing), [DeepInfra](https://deepinfra.com/meta-llama/Llama-3.3-70B-Instruct-Turbo), [Together AI](https://www.together.ai/pricing), [Hyperstack](https://www.hyperstack.cloud/gpu-pricing), [Groq models](https://console.groq.com/docs/models), [Fireworks AI model page](https://fireworks.ai/models/fireworks/llama-v3p3-70b-instruct)
- Throughput: [NVIDIA TensorRT-LLM performance overview](https://nvidia.github.io/TensorRT-LLM/latest/developer-guide/perf-overview.html)
- Utilisation: [CAST AI, 2026 State of Kubernetes Resource Optimization](https://cast.ai/blog/2026-state-of-kubernetes-resource-optimization-cpu-at-8-memory-at-20-and-getting-worse/), [Patil, "Beyond Per-Token Pricing" (2026)](https://arxiv.org/abs/2606.11690)
- Break-even studies: [Peng, Lin and Lee, "Inference Economics of Enterprise Coding Agents" (2026)](https://arxiv.org/abs/2607.13080), [Pan et al., "A Cost-Benefit Analysis of On-Premise Large Language Model Deployment" (2025)](https://arxiv.org/abs/2509.18101)
- [Together AI, "Migrating from closed to open source models" (September 2026)](https://www.together.ai/blog/migrating-from-closed-to-open-source-models)

## Frequently asked questions

### When is self-hosting an LLM cheaper than using an API?

When your GPUs stay busy enough. For Llama 3.3 70B on two H100s at $4.00 per GPU-hour, the break-even is about 4% utilisation against Claude Sonnet 5.5 or gpt-6.1-sol and 24% against Together AI's Llama 3.3 70B (list prices, 5 October 2026). For small or spiky traffic on a model that DeepInfra already serves at $0.10/$0.32 per million tokens, paying per token stays cheaper.

### How do you calculate the break-even point for self-hosting?

Divide the self-hosted cost per 1,000 requests at full load by the API's cost for the same 1,000 requests. The result is the share of the time your GPUs must be busy to break even. Full-load cost is GPU price per hour × number of GPUs ÷ millions of output tokens per hour.

### How much does it cost to run Llama 3.3 70B on your own GPUs?

Two H100s at $4.00 per GPU-hour cost $5,840 a month around the clock. At NVIDIA's published full-load throughput they produce about 11.6 billion output tokens a month, which is $0.50 per million output tokens.

### What GPU utilisation is realistic?

CAST AI measured 5% average GPU utilisation across tens of thousands of production Kubernetes clusters in April 2026; its best example ran 136 H200s at 49%. A dedicated endpoint with steady traffic can do much better than 5%, but never 100%, because traffic varies through the day.

### Should we move off OpenAI or Anthropic?

Only if an open model passes your own evaluations on real traffic. If one does, and your volume is steady, dedicated GPUs cost a fraction of Sonnet-class prices. If no open model is good enough, stay on the closed API.
