How would you fit a 70B model on a single 8×H100 node?

With a 70B parameter model, you can use tensor parallelism (TP=8) to distribute the model across 8 GPUs. Each GPU holds 17.5 GB of parameters (140 GB total / 8). By splitting weight matrices into column-parallel and row-parallel projections, you perform one AllReduce per transformer layer per direction. This fits the model within the node’s memory with only about 10% communication overhead on NVLink.