Give me the back-of-envelope math for how much GPU memo...

The starting arithmetic is simple: parameter count times bytes per parameter. A 70-billion-parameter model in bf16 (2 bytes per parameter) needs roughly 140GB just to hold the weights in memory — before anything else happens.

The real number in production is substantially higher because weights aren’t the only thing occupying GPU memory during inference. The KV cache — the cached keys and values for every token in every active sequence, across every layer and attention head — grows with both context length and batch size, and at long context or high concurrency it can rival or exceed the weights themselves. On top of that there’s activation memory during the forward pass and general framework/CUDA overhead. A realistic planning number is often 1.5 to 3x the raw weight size once you account for a production KV cache and batch size, which is exactly why “will it fit in memory” for a given GPU configuration is a question you answer with the full picture, not just the parameter count.