Walk me through the memory implications of weight-only...

Weight-only INT8 quantization reduces the memory footprint of a 70B model from 140 GB to 70 GB. This is significant because it allows the model to fit on a single H100 80 GB GPU, whereas the BF16 version would require two. Because only the weights are quantized and activations remain in BF16, the memory usage for the KV cache and intermediate tensors remains unchanged. The primary performance benefit is that per-token decode latency is approximately halved, as the runtime is dominated by the time required to load weights from HBM.