Communicate the Change: Efficient Expert Parallelism via Activation Compression

Introducing EchoMoE

1Mila – Quebec AI Institute 2McGill University 3Periodic Labs 4Canada CIFAR AI Chair

*Equal Contribution. Author ordering determined by coin flip.

Correspondence: {fengyuan.liu, jay.gala}@mila.quebec

TL;DR

Mixture-of-experts (MoE) models rely on expert parallelism, where every MoE layer dispatches hidden states to the activated experts and combines the expert outputs. As compute has outpaced interconnect bandwidth, this communication has become a major bottleneck. Because transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed change. EchoMoE retrofits existing MoE models without retraining from scratch.

58%less inter-node RDMA traffic
+22.6%throughput improvement
2.38×throughput over Ethernet
8,500×fewer training tokens than pretraining

Same route. Smaller payload.

Swipe the diagram to follow the full route →

Standard expert parallelism and EchoMoE Two stacked diagrams show four GPUs on two nodes. In both, attention sends token A to local expert E zero and remote expert E two, then combines their outputs. Standard expert parallelism moves nine-block payloads. EchoMoE compresses before dispatch and combine, sending two-block payloads and reconstructing them at the destination. Purple cached references support dispatch compression. Red arrows cross nodes; dark arrows stay within a node. Working components light up in sequence. AttentionDP Dispatchall-to-all ExpertsEP Combineall-to-all OutputDP Standard EP Node 0 Node 1 GPU 0 GPU 1 GPU 2 GPU 3 AAttn E0 A′ BAttn E1 B′ CAttn E2 C′ DAttn E3 D′ AttentionDP Dispatchall-to-all ExpertsEP Combineall-to-all OutputDP EchoMoE Node 0 Node 1 GPU 0 GPU 1 GPU 2 GPU 3 z z AAttn E0 A′ Compress Decompress Compress Decompress BAttn E1 B′ Compress Decompress Compress Decompress CAttn E2 C′ Compress Decompress Compress Decompress DAttn E3 D′ Compress Decompress Compress Decompress
Standard EPAttention computes the activations
EchoMoEAttention computes the activations
Between nodes Within a node Compress Decompress Cached reference More blocks, more data
Follow token A to one local and one remote expert. Animation timing and chunk sizes are illustrative.

Figure 1. Same route, smaller payload. Standard EP sends full activations across nodes on both dispatch and combine. With a cached reference, EchoMoE only needs to send a compact change.

ProblemCommunication bottleneck of expert parallelism

Mixture-of-Experts (MoE) models enable substantial growth in parameter count while keeping per-token computation modest through sparse expert activation. Serving frontier MoE models requires distributing experts across many GPUs using Expert Parallelism (EP).

At every MoE layer, each token's activation is sent to the devices hosting its selected experts (dispatch), and the expert outputs are sent back (combine). When the model spans multiple nodes, both transfers carry the full hidden state across the network. For DeepSeek-V3 served with 128 GPUs, these collectives account for 60% of transformer-layer latency, and still 32% after overlapping communication with computation. As accelerator compute grows faster than interconnect bandwidth, EP communication increasingly limits MoE throughput and latency.

Prior work either redesigns the architecture before pretraining, which cannot be applied to existing models, or reduces the cost of each communicated element through faster kernels or lower precision. Regardless of precision, every MoE layer still transfers all activation dimensions on both collectives.

ObservationNearby MoE layers receive similar inputs

Each transformer block adds its output to the residual stream rather than overwriting it, so a token's hidden state changes gradually with depth and nearby MoE layers read highly similar inputs.

>90% average similarity between a token's inputs to neighboring MoE layers
>80% of the next MoE layer's input can be predicted from the previous one with a simple linear fit

Yet standard EP still communicates the full activation at every layer, even when much of it is predictable from an earlier state. Given a cached activation from an earlier layer, we only need to communicate the change.

MethodCommunicate the change

EchoMoE partitions the MoE layers into groups of $s$ consecutive layers. Full-width hidden states are transmitted once per layer group rather than at every MoE layer.

Cache a reference

The first layer of each group is a reference layer. It broadcasts the full hidden state to every rank, which caches it as $z$. The broadcast is asynchronous and does not block the forward pass.

Compress dispatch

The following $s-1$ layers reuse $z$. The owner rank compresses and dispatches only the difference between the current state $x_\ell$ and the scaled reference:

$$c_d = C_d\,(x_\ell - a \odot z)$$

Each expert rank reconstructs its input from the cached reference $z$ and the dispatched code $c_d$.

Compress combine

Expert outputs are updates to the residual stream and can be represented in a lower-dimensional space. Each node returns a single compressed state, which the owner rank decompresses once.

All compressors and decompressors are linear, so they fold into the expert weights and add no extra serving-time operations.

Follow one token, from bottom to top.

Reference states are stored once and reused in following layers.A token flows upward through normalization, attention, and a standard MoE layer, with black residual bypass paths. The input to the standard MoE travels asynchronously into a purple reference cache. Later, the cached reference is reused by an EchoMoE layer. × L/s× (s − 1)× 1RMSNormAttentionRMSNormStandard MoERMSNormAttentionRMSNormEchoMoEreference cachez = xrstoreoncereusexrxℓhrhr+s

Swipe to follow the full layer

An EchoMoE layer sends compact changes and returns compact outputs.Execute from bottom to top. Compress the current activation relative to the cached reference. Dispatch small teal payloads; purple cached references reach expert inputs at the same time. Experts execute input projection, activation, and output projection. Their compressed outputs are weighted, combined, and decompressed once.EchoMoE layer3Decompress onceon owner rank (DP)2Experts read the compressed changeon expert ranks (EP)1Compress only the changeon owner rank (DP)z ∈ ℝda ⊙ zcachedscaledCdRouter (top-k)reads full xℓxℓ ∈ ℝdcd ∈ ℝrdS, wAll-to-all dispatchzE1w1zcdW̃in,1σ(·)W̃out,1E2w2zcdW̃in,2σ(·)W̃out,2EkwkzcdW̃in,kσ(·)W̃out,k⋯All-to-all combineDcŷ ∈ ℝdcc ∈ ℝrc
Inside an EchoMoE layer: compact codes travel between ranks.Read from bottom to top. The change from the cached reference is compressed and dispatched as a small teal code. Each of the k selected experts reads the cached reference z and the code together, computes, and sends back a weighted compact output, which is combined and decompressed once. EchoMoE layer 3Decompress onceon owner rank (DP) 2Experts read thecompressed changeon expert ranks (EP) 1Compress onlythe changeon owner rank (DP) ŷ ∈ ℝd Dc cc ∈ ℝrc All-to-all combine × k Ei wi W̃out,i σ(·) W̃in,i zcd z local All-to-all dispatch cd ∈ ℝrd S, w Cd Router(top-k) cached scaled z ∈ ℝd a ⊙ z reads full xℓ xℓ ∈ ℝd
REFERENCE LAYER

The token enters the reference layer.

activation / residual cached reference compressed activation routing metadata

Figure 2. Store once, reuse across layers. A reference is shared once per layer group, and every following layer communicates only the compressed change. Routing is unchanged.

Retrofitting an existing model

EchoMoE preserves the original model's behavior by first initializing the compressors to minimize activation reconstruction error, then training end-to-end to match the uncompressed model. This requires 8,500× fewer training tokens than full pretraining.

ResultsLess RDMA traffic, higher throughput

We evaluate EchoMoE on Qwen3-30B-A3B across 18 benchmarks spanning knowledge & reasoning, mathematics, coding and instruction following, served with vLLM at EP32 on four H100 nodes.

EchoMoE Plain compression

Task performance

Inference throughput

Show data
Method(s, rd, rc)RDMA reductionTask score (%)Throughput
EchoMoE(4, 1280, 1280)1.46×65.5 (−1.4)1.025×
EchoMoE(6, 1280, 1280)1.50×65.1 (−1.8)1.055×
EchoMoE(4, 512, 1280)1.84×64.9 (−2.0)1.115×
EchoMoE(4, 512, 1024)2.08×63.8 (−3.1)1.169×
EchoMoE(4, 512, 768)2.39×63.1 (−3.8)1.226×
EchoMoE(6, 512, 768)2.61×61.5 (−5.4)1.276×
EchoMoE(4, 512, 512)2.81×60.6 (−6.4)1.287×
EchoMoE(12, 768, 512)2.92×59.4 (−7.6)1.348×
EchoMoE(6, 256, 768)3.02×58.3 (−8.6)1.325×
EchoMoE(6, 256, 512)3.72×56.9 (−10.1)1.396×
EchoMoE(8, 256, 512)4.03×55.7 (−11.2)1.429×
Plain compression(–, 1280, 1024)1.78×62.8 (−4.1)–
Plain compression(–, 1024, 1024)2.00×61.6 (−5.3)–
Plain compression(–, 768, 768)2.67×58.0 (−8.9)–
Plain compression(–, 512, 512)4.00×51.4 (−15.5)–

Figure 3. Less communication translates directly into higher throughput. EchoMoE keeps task performance close to the uncompressed model and outperforms the plain compression baseline at every level of RDMA savings, while throughput rises steadily as more traffic is saved. Hover over a point for details.

EchoMoE reduces RDMA traffic by 58% with only a 3.8 pp drop in task performance, and improves throughput by 22.6%. Milder compression drops only 2.0 pp, and more aggressive compression improves throughput by 42.9%.

EchoMoE outperforms plain compression. At similar task performance, EchoMoE sends 26% less RDMA traffic than plain compression, which directly compresses dispatch activations.

AnalysisReference states improve dispatch compression

(a) Compression-path ablation

One path compressed, the other at full width

(b) Reference-stride trade-off

Reference dispatch, combine at full width

(c) Decode gains

Eleven configurations, three highlighted

Show data

(a, b) Score change (pp), combine or dispatch kept at full width

Widths=4s=6s=8s=12Plain dispatchCombine score (%)
12800.0−1.1−1.0−1.6–65.2 (−1.7)
1024−0.6−1.8−2.3−3.0−7.164.1 (−2.8)
768−1.0−2.5−3.9−5.3−12.460.6 (−6.3)
512−2.6−5.5−6.3−10.6−21.154.7 (−12.2)
256−5.6−10.7−15.1−20.8–41.4 (−25.5)

(c) Decode speedup by requests per GPU

(s, rd, rc)RDMA reduction432128512
(4, 1280, 1280)1.46×0.941×0.974×0.998×0.973×
(6, 1280, 1280)1.50×0.948×0.980×1.013×1.001×
(4, 512, 1280)1.84×0.927×0.987×1.037×1.037×
(4, 512, 1024)2.08×0.975×1.002×1.057×1.080×
(4, 512, 768)2.39×0.953×1.008×1.077×1.120×
(6, 512, 768)2.61×0.948×1.017×1.098×1.163×
(4, 512, 512)2.81×0.984×1.016×1.098×1.154×
(12, 768, 512)2.92×0.977×1.027×1.131×1.211×
(6, 256, 768)3.02×0.965×1.023×1.117×1.196×
(6, 256, 512)3.72×0.982×1.030×1.141×1.237×
(8, 256, 512)4.03×0.982×1.034×1.154×1.270×

Figure 4. The reference is what makes compression work. (a) Compressing the change loses far less than compressing the activation directly. (b) Refreshing the reference more often is the better use of bandwidth. (c) Decode gains grow with load.

  • Fig. 4a Reference states substantially improve dispatch compression. At width 512, plain dispatch loses 21.1 pp, whereas reference dispatch loses only 2.6 pp.
  • Fig. 4a Expert outputs are more compressible than dispatch activations. This is consistent with their lower-dimensional structure.
  • Fig. 4b Smaller strides with narrower codes outperform larger strides with wider codes. Communication is better spent on more frequent references.

InferenceWhere the speedup comes from

EchoMoE shortens prefill mainly by reducing communication. Prefill communication drops from 115 to 64 ms, while compression adds only a few milliseconds.

Communication Compression MoE compute Attention compute Other (compute, sync wait, CPU)

Prefill step (4,096 tokens per GPU)

Decode step (128 tokens per GPU)

Show data (ms)
Step(s, rd, rc)Comm.Compr.MoEAttentionOther computeSync waitCPUTotal
PrefillUncompressed114.60.037.127.720.111.79.4220.7
Prefill(4, 512, 1280)79.46.935.627.722.28.16.2186.1
Prefill(4, 512, 768)64.46.431.127.721.613.26.7171.3
Prefill(8, 256, 512)42.66.527.627.719.56.07.7137.6
DecodeUncompressed11.40.02.619.22.11.53.540.4
Decode(4, 512, 1280)9.90.92.219.22.11.53.739.6
Decode(4, 512, 768)9.00.82.119.12.11.03.737.9
Decode(8, 256, 512)7.30.91.919.12.02.03.737.0

Figure 5. The speedup comes from communication. Communication time shrinks the most, while compression adds only a small cost.

Gains grow on slower networks

TCP over Ethernet, FP8 TCP over Ethernet, BF16 InfiniBand, BF16 (reference)
Show data
(s, rd, rc)RDMA reductionTCP, BF16TCP, FP8InfiniBand, BF16
(4, 1280, 1280)1.46×1.49×1.87×0.96×
(6, 1280, 1280)1.50×1.52×1.86×0.97×
(4, 512, 1280)1.84×1.69×2.06×1.00×
(4, 512, 1024)2.08×1.83×2.14×1.01×
(4, 512, 768)2.39×1.93×2.16×1.04×
(6, 512, 768)2.61×1.97×2.26×1.06×
(4, 512, 512)2.81×2.10×2.28×1.07×
(12, 768, 512)2.92×2.11×2.20×1.08×
(6, 256, 768)3.02×2.09×3.00×1.08×
(6, 256, 512)3.72×2.32×3.12×1.09×
(8, 256, 512)4.03×2.38×3.24×1.10×

Figure 6. The slower the network, the larger the gain. Over Ethernet, EchoMoE improves throughput by up to 2.38× with BF16 and 3.24× with FP8 communication, while the same configurations give small gains on InfiniBand. Hover over the chart for each configuration.

Gains grow on slower networks. Over bandwidth-constrained Ethernet (about 23 Gb/s per node), EchoMoE improves inference throughput by up to 2.38×, and up to 3.24× with FP8 communication, paving the way for wide-area distributed inference.

GeneralityOther MoE models

Applied to Gemma 4, Kanana-2, Nemotron-3.5 and Moonlight, EchoMoE roughly halves their RDMA traffic with a 2.4–5.9 pp task performance drop. EchoMoE also composes cleanly with FP8 quantization, with no additional loss in task performance.

Conclusion

EchoMoE exploits the cross-layer predictability of MoE inputs by caching an earlier activation as a reference and communicating only the compressed change. It establishes cross-layer predictability as a practical mechanism for substantially reducing EP communication in existing MoE models, and could also be incorporated as a novel architecture for pretraining.

CitationBibTeX

@misc{liu2026communicatechangeefficientexpert,
  title={Communicate the Change: Efficient Expert Parallelism via Activation Compression},
  author={Fengyuan Liu and Jay Gala and Siva Reddy and Dzmitry Bahdanau},
  year={2026},
  eprint={2611.22215},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2611.22215},
}

Acknowledgements

We acknowledge the support of the 2026 Google TPU Research & Education Awards (TPU Builders Research Program). Siva Reddy is supported by the Canada CIFAR AI Chairs program. We thank the Mila IDT team and the Digital Research Alliance of Canada for providing the computational resources used in this work. We also thank members of McGill University, Mila, and ServiceNow Research, especially Tomás Vergara Browne and Spandana Gella.