Mixture-of-Experts (MoE) models enable substantial growth in parameter count while keeping per-token computation modest through sparse expert activation. Serving frontier MoE models requires distributing experts across many GPUs using Expert Parallelism (EP).
At every MoE layer, each token's activation is sent to the devices hosting its selected experts (dispatch), and the expert outputs are sent back (combine). When the model spans multiple nodes, both transfers carry the full hidden state across the network. For DeepSeek-V3 served with 128 GPUs, these collectives account for 60% of transformer-layer latency, and still 32% after overlapping communication with computation. As accelerator compute grows faster than interconnect bandwidth, EP communication increasingly limits MoE throughput and latency.
Prior work either redesigns the architecture before pretraining, which cannot be applied to existing models, or reduces the cost of each communicated element through faster kernels or lower precision. Regardless of precision, every MoE layer still transfers all activation dimensions on both collectives.