How much GPU memory does a 70B model need? Fitting it is the easy part
The weights decide whether a model loads. The KV cache decides how many people can use it at once, and that is where most sizing guides stop.

The short version#
Llama 3.3 70B at FP8, served with vLLM's default settings:
| Setup | Room left for the KV cache | Tokens of cache (BF16 cache) | Tokens of cache (FP8 cache) | 8K-token conversations at once (FP8 cache) |
|---|---|---|---|---|
| 1× H100 80GB | 3.5 GB | 10,600 | 21,200 | 2 |
| 2× H100 80GB | 79.6 GB | 243,000 | 486,000 | 59 |
| 1× H200 141GB | 63.4 GB | 194,000 | 387,000 | 47 |
| 1× B200 180GB | 90.9 GB | 277,000 | 555,000 | 67 |
The rest of this post shows where each number comes from, so you can redo it for your own model and traffic.
Weights: 141 GB, 73 GB or 40 GB#
Llama 3.3 70B has 70,553,706,496 parameters. At two bytes each in BF16 that is 141.1 GB, which matches the 141.11 GB checkpoint on Hugging Face. FP8 checkpoints come to 72.7 GB rather than the 70.6 GB you would expect, because about 2.1 billion parameters (the embeddings and the output layer) stay in BF16. A 4-bit W4A16 checkpoint is 39.5 GB. That is 28% of BF16 rather than 25%, for the same reason plus the scales stored alongside the 4-bit weights.
Lower precision costs little on published benchmarks:
| Benchmark | BF16 | FP8 (W8A8) | 4-bit (W4A16) |
|---|---|---|---|
| MMLU, 5-shot | 81.60 | 81.31 | 80.62 |
| MMLU, chain of thought | 86.58 | 86.34 | 85.81 |
| GSM-8K, chain of thought | 94.16 | 94.92 | 94.47 |
| HumanEval, pass@1 | 83.20 | 83.70 | 83.40 |
These are from Red Hat's model cards for its FP8 and 4-bit checkpoints. NVIDIA's own FP8 checkpoint moves MMLU from 83.3 to 83.2. A study of more than 500,000 evaluations on Llama 3.1 found FP8 "effectively lossless" at every model size (Kurtic et al., ACL 2025).
We start from FP8 on H100, H200 and B200, which compute in FP8 natively. An A100 runs FP8 checkpoints with the weights stored in FP8 and the arithmetic done in 16-bit, which saves memory but not compute. Benchmarks don't cover your prompts, so run your own evaluation before you switch.
KV cache: 320 KiB per token#
While a model generates, it keeps a key and a value vector for every token of every conversation it is serving, in every layer. That store is the KV cache, and it grows with each user and each token of their context.
The size per token is:
bytes per token = 2 (key and value) × layers × KV heads × head dimension × bytes per value
For Llama 3.3 70B, from its config.json:
layers, kv_heads, head_dim = 80, 8, 128 # num_hidden_layers, num_key_value_heads, head_dim
bytes_per_value = 2 # BF16; 1 with an FP8 cache
per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
print(per_token) # 327680 bytes, or 320 KiB
print(per_token * 8_192 / 2**30) # 2.5 GiB for one 8K-token conversation
Use the number of KV heads, not attention heads. Llama 3 uses grouped-query attention, where 64 query heads share 8 key-value heads. Plugging in 64 by mistake makes the cache look eight times bigger than it is. And read head_dim from the config instead of dividing the hidden size by the number of heads: Qwen3-32B has 64 heads of 128 dimensions, which is 8,192, while its hidden size is 5,120.
| Context per conversation | KV cache, BF16 | KV cache, FP8 |
|---|---|---|
| 4,096 tokens | 1.25 GiB | 0.63 GiB |
| 8,192 tokens | 2.5 GiB | 1.25 GiB |
| 32,768 tokens | 10 GiB | 5 GiB |
| 131,072 tokens (the model's maximum) | 40 GiB | 20 GiB |
One user at the full 131,072-token context needs 40 GiB of cache at BF16: half an H100 for a single conversation.
An FP8 cache halves all of this. vLLM keeps the cache at the model's 16-bit precision unless you pass --kv-cache-dtype fp8. In tests published on the vLLM blog in April 2026, Llama 3.3 70B kept about 97 to 98% of its long-context retrieval score with an FP8 cache. Check your own long-context tasks before turning it on.
What the engine actually gets#
Three things stand between the number on the box and the space left for users.
- Memory the software sees. An "80 GB" H100 reports 85.52 GB (79.65 GiB) to NVIDIA's driver tools, as Jonathon Belotti explains, and PyTorch reports a total capacity of 79.11 GiB (84.9 GB), as you can see in public out-of-memory reports. An H200 shows about 139.8 GiB (150.1 GB) (example).
- vLLM's share. vLLM uses 92% of that memory by default. The
gpu_memory_utilizationdefault moved from 0.9 to 0.92 in version 0.20.0, released in April 2026 (engine arguments). - Working memory. At start-up, vLLM runs a test pass to measure peak activation memory, subtracts it and the weights from its share, and gives the rest to the KV cache. CUDA graphs need memory too. We assume 2 GB per GPU for both together; the real figure depends on your settings.
Where the memory goes: Llama 3.3 70B at FP8 under vLLM's defaults
| Setup | Breakdown | Visible memory |
|---|---|---|
| 1× H100 | Model weights (FP8) 72.7 GB, Engine working memory 2.0 GB, KV cache for your users 3.5 GB, Held back by vLLM (8%) 6.8 GB | 84.9 GB |
| 2× H100 | Model weights (FP8) 72.7 GB, Engine working memory 4.0 GB, KV cache for your users 79.6 GB, Held back by vLLM (8%) 13.6 GB | 169.9 GB |
| 1× H200 | Model weights (FP8) 72.7 GB, Engine working memory 2.0 GB, KV cache for your users 63.4 GB, Held back by vLLM (8%) 12.0 GB | 150.1 GB |
| 1× B200 | Model weights (FP8) 72.7 GB, Engine working memory 2.0 GB, KV cache for your users 90.9 GB, Held back by vLLM (8%) 14.4 GB | 180.0 GB |
On one H100, 72.7 GB of FP8 weights leave 3.5 GB for users. That is about 10,600 tokens with vLLM's default 16-bit cache: one 8K-token conversation, with nothing left for a second. An FP8 cache doubles that to two.
When vLLM starts, it prints how much memory it gave the KV cache and how many tokens that holds. That log line beats any calculator, ours included.
How many users fit#
With FP8 weights:
| Setup | Cache tokens (FP8 cache) | Conversations at 8K tokens | Conversations at 32K tokens |
|---|---|---|---|
| 1× H100 | 21,200 | 2 | 0 |
| 1× H100, 4-bit weights instead | 223,000 | 27 | 6 |
| 2× H100 | 486,000 | 59 | 14 |
| 1× H200 | 387,000 | 47 | 11 |
| 1× B200 | 555,000 | 67 | 16 |
"Conversations at once" assumes every user holds a full context at the same moment, which is the worst case. vLLM hands out cache in small blocks as tokens arrive, so a mix of short and long conversations fits more users. Fitting is also different from being fast: the more conversations a GPU serves at once, the slower each one's tokens arrive.
What we would do with each:
- One H100 (80 GB): fine for development and a few users with short prompts. If one card is your budget, use the 4-bit checkpoint (about 99% of BF16 quality on the benchmarks above) or a smaller model.
- Two H100s or one H200: the practical minimum for a small production service. At specialist clouds that rent single GPUs, one H200 costs $3.59 to $5.40 an hour, against $5.38 to $9.00 for two H100s (October 2026 prices), so it holds cache more cheaply. The H100 pair has 40% more total memory bandwidth (6.7 TB/s against 4.8), which matters when many users generate at once.
- One B200 (180 GB): the most cache on a single GPU, at $5.98 to $8.50 an hour at specialist clouds that rent single GPUs. Worth pricing if your users bring long documents.
- BF16 weights: need four H100s or two H200s. With FP8 scoring within a point on the benchmarks above, there are few reasons to pay for that.
Long context is a memory problem first#
At the full 131,072-token context, one conversation's FP8 cache is 21.5 GB. Two H100s hold three such conversations at once, one H200 holds two and a B200 holds four. Long prompts also take longer to process before the first token appears, but you run out of memory first.
If your users send long documents, size for the longest context you will accept, and cap the rest with vLLM's --max-model-len so one request can't take the whole cache.
A smaller model changes the picture#
Qwen3-32B at FP8 is a 34.3 GB checkpoint with 256 KiB of BF16 cache per token (128 KiB in FP8). On one H100 it leaves about 41.8 GB for the cache: roughly 160,000 tokens at vLLM's defaults, or 320,000 with an FP8 cache, enough for about 38 conversations of 8K tokens. Its native context is 32,768 tokens (131,072 with YaRN scaling).
So if one H100 is your budget, a 32B model will serve far more people than a 70B one. Whether it serves them well enough is a question for your evaluation set, not for a memory calculation. Once you know which model passes, see what that setup costs per million tokens.
How we worked this out#
Sources#
All accessed 5 October 2026.
- Meta Llama 3.3 70B: model metadata, ungated config copy, Meta's sku_list.py
- Checkpoints and evaluations: Red Hat FP8, Red Hat W4A16, NVIDIA FP8, Kurtic et al., "Give Me BF16 or Give Me Death"
- Qwen3-32B config
- KV cache: NVIDIA, Mastering LLM techniques: inference optimization, Ainslie et al., grouped-query attention, vLLM quantized KV cache, vLLM blog on FP8 KV cache
- vLLM memory: engine arguments, conserving memory, cache configuration source, v0.31.0
- GPU memory: Jonathon Belotti, "Why does an NVIDIA H100 80GB card offer 85.52 GB?", H100 report of 79.11 GiB, H200 report of 139.79 GiB, NVIDIA H100, H200 and DGX B200 specifications

