DeepSeek-V3 has no tensor parallelism. Why not?

DeepSeek-V3 uses expert parallelism rather than tensor parallelism (TP). Because expert MLP layers dominate the parameter count in a 671B MoE model, distributing these experts across 64 GPUs via token routing effectively reduces memory usage. Implementing TP would force per-layer all-reduces on the attention and dense MLP layers, which would decrease net throughput while offering minimal additional memory savings.