# How much GPU memory does a 70B model need? Fitting it is the easy part

> Llama 3.3 70B needs 141 GB of weights at BF16 and 73 GB at FP8, and every token of a user's context adds 320 KiB of KV cache. The arithmetic for how many users fit on an H100, two H100s, an H200 or a B200.

By Ronit Ahuja, ClearCompute. Published 2026-10-05, updated 2026-10-26. Canonical: https://clearcompute.cc/blog/gpu-memory-70b-model

> **The short answer.** Llama 3.3 70B needs about **141 GB for its weights at BF16, 73 GB at FP8 and 40 GB at 4-bit**. Every token a user holds in context adds **320 KiB of KV cache** at BF16, or 160 KiB with an FP8 cache, so one 8,192-token conversation takes 2.5 GiB. At FP8 the model loads on one 80 GB H100 but leaves room for only about 10,600 tokens of cache under vLLM's defaults: one long conversation at a time. Two H100s hold about 486,000 tokens and one H200 about 387,000 with an FP8 cache, which makes them the practical minimum for a small production service.

## The short version

Llama 3.3 70B at FP8, served with vLLM's default settings:

| Setup | Room left for the KV cache | Tokens of cache (BF16 cache) | Tokens of cache (FP8 cache) | 8K-token conversations at once (FP8 cache) |
|---|---|---|---|---|
| 1× H100 80GB | 3.5 GB | 10,600 | 21,200 | 2 |
| 2× H100 80GB | 79.6 GB | 243,000 | 486,000 | 59 |
| 1× H200 141GB | 63.4 GB | 194,000 | 387,000 | 47 |
| 1× B200 180GB | 90.9 GB | 277,000 | 555,000 | 67 |

The rest of this post shows where each number comes from, so you can redo it for your own model and traffic.

## Weights: 141 GB, 73 GB or 40 GB

Llama 3.3 70B has 70,553,706,496 parameters. At two bytes each in BF16 that is 141.1 GB, which matches the 141.11 GB checkpoint on Hugging Face. FP8 checkpoints come to 72.7 GB rather than the 70.6 GB you would expect, because about 2.1 billion parameters (the embeddings and the output layer) stay in BF16. A 4-bit W4A16 checkpoint is 39.5 GB. That is 28% of BF16 rather than 25%, for the same reason plus the scales stored alongside the 4-bit weights.

Lower precision costs little on published benchmarks:

| Benchmark | BF16 | FP8 (W8A8) | 4-bit (W4A16) |
|---|---|---|---|
| MMLU, 5-shot | 81.60 | 81.31 | 80.62 |
| MMLU, chain of thought | 86.58 | 86.34 | 85.81 |
| GSM-8K, chain of thought | 94.16 | 94.92 | 94.47 |
| HumanEval, pass@1 | 83.20 | 83.70 | 83.40 |

These are from Red Hat's model cards for its [FP8](https://huggingface.co/RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic) and [4-bit](https://huggingface.co/RedHatAI/Llama-3.3-70B-Instruct-quantized.w4a16) checkpoints. NVIDIA's own [FP8 checkpoint](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP8) moves MMLU from 83.3 to 83.2. A study of more than 500,000 evaluations on Llama 3.1 found FP8 "effectively lossless" at every model size ([Kurtic et al., ACL 2025](https://arxiv.org/abs/2411.02355)).

We start from FP8 on H100, H200 and B200, which compute in FP8 natively. An A100 runs FP8 checkpoints with the weights stored in FP8 and the arithmetic done in 16-bit, which saves memory but not compute. Benchmarks don't cover your prompts, so run your own evaluation before you switch.

## KV cache: 320 KiB per token

While a model generates, it keeps a key and a value vector for every token of every conversation it is serving, in every layer. That store is the KV cache, and it grows with each user and each token of their context.

The size per token is:

```text
bytes per token = 2 (key and value) × layers × KV heads × head dimension × bytes per value
```

For Llama 3.3 70B, from its config.json:

```python
layers, kv_heads, head_dim = 80, 8, 128   # num_hidden_layers, num_key_value_heads, head_dim
bytes_per_value = 2                       # BF16; 1 with an FP8 cache

per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
print(per_token)                          # 327680 bytes, or 320 KiB
print(per_token * 8_192 / 2**30)          # 2.5 GiB for one 8K-token conversation
```

Use the number of KV heads, not attention heads. Llama 3 uses grouped-query attention, where 64 query heads share 8 key-value heads. Plugging in 64 by mistake makes the cache look eight times bigger than it is. And read head_dim from the config instead of dividing the hidden size by the number of heads: Qwen3-32B has 64 heads of 128 dimensions, which is 8,192, while its hidden size is 5,120.

| Context per conversation | KV cache, BF16 | KV cache, FP8 |
|---|---|---|
| 4,096 tokens | 1.25 GiB | 0.63 GiB |
| 8,192 tokens | 2.5 GiB | 1.25 GiB |
| 32,768 tokens | 10 GiB | 5 GiB |
| 131,072 tokens (the model's maximum) | 40 GiB | 20 GiB |

One user at the full 131,072-token context needs 40 GiB of cache at BF16: half an H100 for a single conversation.

An FP8 cache halves all of this. vLLM keeps the cache at the model's 16-bit precision unless you pass `--kv-cache-dtype fp8`. In tests published on the [vLLM blog in April 2026](https://vllm.ai/blog/2026-04-22-fp8-kvcache), Llama 3.3 70B kept about 97 to 98% of its long-context retrieval score with an FP8 cache. Check your own long-context tasks before turning it on.

## What the engine actually gets

Three things stand between the number on the box and the space left for users.

1. **Memory the software sees.** An "80 GB" H100 reports 85.52 GB (79.65 GiB) to NVIDIA's driver tools, as [Jonathon Belotti explains](https://thundergolfer.com/blog/nvidia-gpu-memory-capacity), and PyTorch reports a total capacity of 79.11 GiB (84.9 GB), as you can see in [public out-of-memory reports](https://github.com/modelscope/ms-swift/issues/6509). An H200 shows about 139.8 GiB (150.1 GB) ([example](https://github.com/vllm-project/vllm/issues/59317)).
2. **vLLM's share.** vLLM uses 92% of that memory by default. The `gpu_memory_utilization` default moved from 0.9 to 0.92 in version 0.20.0, released in April 2026 ([engine arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html)).
3. **Working memory.** At start-up, vLLM runs a test pass to measure peak activation memory, subtracts it and the weights from its share, and gives the rest to the KV cache. CUDA graphs need memory too. We assume 2 GB per GPU for both together; the real figure depends on your settings.

**Where the memory goes: Llama 3.3 70B at FP8 under vLLM's defaults**

| Setup | Model weights (FP8) | Engine working memory | KV cache for your users | Held back by vLLM (8%) | Visible memory |
|---|---|---|---|---|---|
| 1× H100 | 72.7 GB | 2.0 GB | 3.5 GB | 6.8 GB | 84.9 GB |
| 2× H100 | 72.7 GB | 4.0 GB | 79.6 GB | 13.6 GB | 169.9 GB |
| 1× H200 | 72.7 GB | 2.0 GB | 63.4 GB | 12.0 GB | 150.1 GB |
| 1× B200 | 72.7 GB | 2.0 GB | 90.9 GB | 14.4 GB | 180.0 GB |

*Memory visible to software: 79.11 GiB (84.9 GB) per H100 and about 139.8 GiB (150.1 GB) per H200, as reported by PyTorch; 180 GB per B200 as marketed. vLLM uses 92% of it by default. Engine working memory (activations, CUDA graphs) is our assumption of 2 GB per GPU; vLLM measures the real figure at start-up.*

On one H100, 72.7 GB of FP8 weights leave 3.5 GB for users. That is about 10,600 tokens with vLLM's default 16-bit cache: one 8K-token conversation, with nothing left for a second. An FP8 cache doubles that to two.

When vLLM starts, it prints how much memory it gave the KV cache and how many tokens that holds. That log line beats any calculator, ours included.

## How many users fit

With FP8 weights:

| Setup | Cache tokens (FP8 cache) | Conversations at 8K tokens | Conversations at 32K tokens |
|---|---|---|---|
| 1× H100 | 21,200 | 2 | 0 |
| 1× H100, 4-bit weights instead | 223,000 | 27 | 6 |
| 2× H100 | 486,000 | 59 | 14 |
| 1× H200 | 387,000 | 47 | 11 |
| 1× B200 | 555,000 | 67 | 16 |

"Conversations at once" assumes every user holds a full context at the same moment, which is the worst case. vLLM hands out cache in small blocks as tokens arrive, so a mix of short and long conversations fits more users. Fitting is also different from being fast: the more conversations a GPU serves at once, the slower each one's tokens arrive.

What we would do with each:

- **One H100 (80 GB):** fine for development and a few users with short prompts. If one card is your budget, use the 4-bit checkpoint (about 99% of BF16 quality on the benchmarks above) or a smaller model.
- **Two H100s or one H200:** the practical minimum for a small production service. At specialist clouds that rent single GPUs, one H200 costs $3.59 to $5.40 an hour, against $5.38 to $9.00 for two H100s ([October 2026 prices](/blog/h100-cloud-prices)), so it holds cache more cheaply. The H100 pair has 40% more total memory bandwidth (6.7 TB/s against 4.8), which matters when many users generate at once.
- **One B200 (180 GB):** the most cache on a single GPU, at $5.98 to $8.50 an hour at specialist clouds that rent single GPUs. Worth pricing if your users bring long documents.
- **BF16 weights:** need four H100s or two H200s. With FP8 scoring within a point on the benchmarks above, there are few reasons to pay for that.

## Long context is a memory problem first

At the full 131,072-token context, one conversation's FP8 cache is 21.5 GB. Two H100s hold three such conversations at once, one H200 holds two and a B200 holds four. Long prompts also take longer to process before the first token appears, but you run out of memory first.

If your users send long documents, size for the longest context you will accept, and cap the rest with vLLM's `--max-model-len` so one request can't take the whole cache.

## A smaller model changes the picture

Qwen3-32B at FP8 is a 34.3 GB checkpoint with 256 KiB of BF16 cache per token (128 KiB in FP8). On one H100 it leaves about 41.8 GB for the cache: roughly 160,000 tokens at vLLM's defaults, or 320,000 with an FP8 cache, enough for about 38 conversations of 8K tokens. Its native context is 32,768 tokens (131,072 with YaRN scaling).

So if one H100 is your budget, a 32B model will serve far more people than a 70B one. Whether it serves them well enough is a question for your evaluation set, not for a memory calculation. Once you know which model passes, see [what that setup costs per million tokens](/blog/self-hosting-vs-api-cost).

## How we worked this out

> **How we worked this out.** Llama 3.3 70B's architecture comes from its config.json, read from three ungated copies on Hugging Face because the official repository requires accepting Meta's licence, and checked against Meta's own model list on GitHub. Parameter counts come from the official repository's metadata; checkpoint sizes from each repository's safetensors index. Memory visible to software is PyTorch's reported total capacity for the H100 (79.11 GiB) and H200 (about 139.8 GiB), as shown in public GitHub issues, and NVIDIA's listed capacity for the B200 (180 GB). We applied vLLM's default 0.92 memory utilisation and assumed 2 GB per GPU of working memory. With tensor parallelism across two or four GPUs, the eight KV heads are split between the GPUs, so their cache space adds up. We did not run these models for this post: the arithmetic is exact for weights and cache, and an estimate for working memory.

## Sources

All accessed 5 October 2026.

- Meta Llama 3.3 70B: [model metadata](https://huggingface.co/api/models/meta-llama/Llama-3.3-70B-Instruct), [ungated config copy](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/resolve/main/config.json), [Meta's sku_list.py](https://github.com/meta-llama/llama-models/blob/main/models/sku_list.py)
- Checkpoints and evaluations: [Red Hat FP8](https://huggingface.co/RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic), [Red Hat W4A16](https://huggingface.co/RedHatAI/Llama-3.3-70B-Instruct-quantized.w4a16), [NVIDIA FP8](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP8), [Kurtic et al., "Give Me BF16 or Give Me Death"](https://arxiv.org/abs/2411.02355)
- [Qwen3-32B config](https://huggingface.co/Qwen/Qwen3-32B/resolve/main/config.json)
- KV cache: [NVIDIA, Mastering LLM techniques: inference optimization](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/), [Ainslie et al., grouped-query attention](https://arxiv.org/abs/2305.13245), [vLLM quantized KV cache](https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html), [vLLM blog on FP8 KV cache](https://vllm.ai/blog/2026-04-22-fp8-kvcache)
- vLLM memory: [engine arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html), [conserving memory](https://docs.vllm.ai/en/latest/configuration/conserving_memory.html), [cache configuration source, v0.31.0](https://github.com/vllm-project/vllm/blob/v0.31.0/vllm/config/cache.py)
- GPU memory: [Jonathon Belotti, "Why does an NVIDIA H100 80GB card offer 85.52 GB?"](https://thundergolfer.com/blog/nvidia-gpu-memory-capacity), [H100 report of 79.11 GiB](https://github.com/modelscope/ms-swift/issues/6509), [H200 report of 139.79 GiB](https://github.com/vllm-project/vllm/issues/59317), NVIDIA [H100](https://www.nvidia.com/en-us/data-center/h100/), [H200](https://www.nvidia.com/en-us/data-center/h200/) and [DGX B200](https://www.nvidia.com/en-us/data-center/dgx-b200/) specifications

## Frequently asked questions

### How much VRAM does Llama 3.3 70B need?

The weights take about 141 GB at BF16, 73 GB at FP8 and 40 GB at 4-bit (W4A16). On top of that, the KV cache needs 320 KiB per token of context at BF16, or 160 KiB with an FP8 cache, so one 8,192-token conversation adds 2.5 GiB.

### Can one H100 run Llama 3.3 70B?

At FP8 it loads, but under vLLM's defaults it leaves room for only about 10,600 tokens of KV cache, or 21,200 with an FP8 cache: one or two long conversations at a time. For real traffic, use a 4-bit checkpoint, two H100s or one H200.

### How do you calculate KV cache size?

Bytes per token = 2 × layers × KV heads × head dimension × bytes per value. For Llama 3.3 70B that is 2 × 80 × 8 × 128 × 2 = 327,680 bytes at BF16. Use the number of KV heads (8), not query heads (64), and read head_dim from the model's config.json.

### Does FP8 reduce the quality of Llama 3.3 70B?

Very little on published benchmarks. Red Hat's FP8 checkpoint scores 99.6% to 100.8% of BF16 (MMLU 81.31 against 81.60), and its 4-bit W4A16 checkpoint about 99% (MMLU 80.62). Test on your own tasks before switching.

### How many users can one H200 serve with a 70B model?

With FP8 weights and an FP8 KV cache, about 387,000 tokens of cache: room for about 47 conversations holding 8,192 tokens each at the same moment, or 11 at 32,768 tokens.

### Is one H200 or two H100s better for a 70B model?

Two H100s hold more cache (about 486,000 tokens against 387,000) and have 40% more total memory bandwidth. One H200 costs less per hour at most specialist clouds and avoids splitting the model across GPUs.
