2026-d2-1400-paper-session-c-distributed-ai-communication

Sorry for rushing the slides through! Regarding attention mechanisms, I'd like to recommend htor.inf.ethz.ch/blog/…/lets-stop-calling-everything-linear-attention for a detailed explanation. Per-package bandwidth, yes. The quantity on the critical path is the hot expert's inbound byte count, which is fixed by the model and the batch. Higher per-endpoint bandwidth drains it proportionally faster. Our IB counters say we are nowhere near fabric-limited: peak 82-84 Gbps per node against 4x NDR200 = 800 Gbps installed, with port_xmit_wait delta 0 in every sampled iteration. Therefore it's endpoint-bound, not fabric-bound, so per-package bandwidth is the right thing to buy.

Thank you so much for this fantastic question! I just pulled our measured data and they showed some really interesting patterns: predicting experts works better on some models than others. In models like Qwen3-30B-A3B (GQA), the most-used experts stay fairly predictable from one step to the next, with only about 7.6% of the active experts changing per turn. In models like Qwen3.5-35B-A3B (Qwen switched to Gated DeltaNet Hybrid from Qwen3.5), expert usage shifts rapidly, causing a 72.6% change per turn.
Even when predictions miss, it isn't a total failure, because expert usage in fast-changing models is spread out evenly, picking any set of top experts still covers around 40–50% of your total workload. Missing the exact right expert doesn't hurt performance as much as we might think.

We mean receiver-side occupancy. A hot destination rank has in-flight payloads from all P-1 senders resident simultaneously, so per-endpoint buffer demand scales with fan-in x message size. Sparser routing makes that worse in count, not bytes: more experts per token means more, smaller messages in flight at once.

Caveat: Low routing imbalance does not necessarily imply a superior model, and none of the observations below should be taken as a evaluation of overall capability (dense baselines like Qwen3.8-27B remain highly capable).
Interestingly, load imbalance can dynamically shift during training. For example, NVIDIA-Nemotron-3-Nano-30B-A3B initially exhibits high imbalance with concentrated token routing, but naturally rebalances after several hundred steps. Furthermore, for continued pretraining on out-of-domain corpora, hybrid MoE architectures with Gated DeltaNet layers (e.g., Qwen3.5-35B-A3B) appear to be quite robust. However, because they may not yield the optimal aggregate per-rank Gini coefficient, the best architecture ultimately depends on your specific workload requirements.

I think what matters here is the level of measurement: at the collective level and our operating point, it is dense; but it's not the case for individual token and it can be sparse. In one forward pass, each token goes to k experts, so at most k ranks, but a collective carries a whole batch, and the union covers every destination. Therefore, the primitive is a dense AllToAllv with a skewed count matrix, which is why a uniform AllToAll microbenchmark mis-sizes it. There's one caveat: in very high EP Degree cases (e.g. EP=128) where Local Experts per GPU = 1, we observed that 1.5-4.1% of the ranks might receive nothing during a forward pass, but it constitutes a small fraction.

Regarding how to test it, I've recently noticed more efforts targeting the miscalibration issue I mentioned during the presentation. You might be interested in CCL-Bench: cclbench.ai/index.html

There are many applications using NVSHMEM for both AI inference and HPC. AI examples include deepEP v1 and HPC applications include GROMACS and QUDA

they are scalar put and get APIs

e.g. nvshmem_int_p puts exactly one integer

passed as an immediate