How do you provision inference hardware for a 70B-total...

Start with total parameters to set the memory floor; at BF16, this requires at least two H100-80GB GPUs with expert-parallel sharding. While latency per token is determined by the 7B activated-parameter FLOPs, batch throughput is limited by the all-to-all communication dispatch required for expert routing. Benchmark throughput degradation against a dense stack before finalizing the deployment.