MoE transport design report · 17 August 2026

Fixed pooled-wave all-to-all for the 8-of-384 MoE

TL;DR: The merged sender CF 1.10 and receiver CF 1.15 configuration completed a 20-step one-rack gate without an OOM. Median throughput for steps 2 through 19 was 250,691 tokens/s. The transport keeps all collective shapes static, uses three waves, and does not exchange token counts. It reports sender and receiver drops separately. The short hero gate does not establish the final drop rate.

one GB200 rack 64 GPUs BF16 device parameters FP32 pinned-host master parameters no FP8 no micro-batches no metadata collective

384routed experts

top 8expert assignments per token

6 / GPUlocal routed experts

3 wavessame static shapes in each wave

250,691current median tokens/s, steps 2–19

no OOMcurrent 20-step one-rack gate

01 · Model position

Where the transport fits

Each of the 48 transformer blocks has an attention branch and an MoE branch. The fixed pooled-wave transport is only in the routed expert path. The two shared experts stay local.

Block inputwidth 6144

RMSNorm + attention48 heads, SConv, residual

Router384 scores, select top 8

Latent down6144 → 3072

Fixed pooled-wave A2Aroute, expert MLP, return

Latent up3072 → 6144

Two shared expertslocal dense path, width 6144, intermediate 3072

Add routed and shared outputsthen SConv

Residual addblock output, width 6144

Routed path: The router reads full-width tokens. The latent projection reduces each row to 3072 values before the all-to-all. This cuts routed activation traffic in half.

Shared path: The two shared experts do not use expert parallel all-to-all. Their output joins the routed result after the latent up projection.

02 · Rack view

One pool for each destination GPU

The sender does not allocate one transport cell for each of 384 experts. It allocates 64 destination pools for each wave. The receiver reads the local expert ID from the same BF16 payload.

Route local tokensEach GPU has 65,536 tokens. Top-8 routing makes 524,288 assignments.
Map each assignmentThe expert ID gives a destination GPU and one local expert ID from 0 to 5.
Stripe destination rankswave = rank mod 3. pool row = rank ÷ 3.
Build 64 fixed poolsEach destination gets 3,004 BF16 rows in one wave. Extra sender rows are dropped.
Send the payloadThe first full-width row carries packed expert IDs. No count exchange is required.
Compact on the receiverThe receiver groups rows into six static expert arrays. It drops rows above the receiver capacity.
Compute and returnOne batched SwiGLU GEMM serves the six experts. A fixed reverse all-to-all returns the outputs.

Expert-parallel rack

64 GPUs. Each sender has one pool for every GPU.

GPU 00

GPU 42

fixed A2A sends one pool from GPU 00 to GPU 42

GPU 42 decodes six local expert IDs from the payload.

E252

E253

E254

E255

E256

E257

03 · One wave

Static shapes from send to combine

All three waves use the same array shapes. The waves run in order and add their results to one FP32 output accumulator. Each wave has a forward all-to-all and a reverse all-to-all.

1. Sender pools

[64, 3004, 3072]

One fixed pool for each destination. One extra full-width header row carries the local expert IDs.

2. Fixed all-to-all

[64, 3005, 3072]

The shape includes the header. Each GPU sends and receives the same number of BF16 values.

3. Receiver compact

[6, 33497, 3072]

The receiver groups rows by local expert. Rows above 33,497 for one expert in one wave are dropped.

4. Expert MLP + return

6 × SwiGLU

The receiver computes all six local experts in one batched operation. It expands the rows and uses the reverse fixed all-to-all.

Wave 0destination ranks 0, 3, 6, …

Wave 1destination ranks 1, 4, 7, …

Wave 2destination ranks 2, 5, 8, …

04 · Capacity and memory

Two limits control two different risks

Sender capacity controls traffic from one source to one destination. Receiver capacity controls the final load for one local expert. The second limit is necessary because traffic from 64 senders can select the same expert.

Assignments on each GPU

65,536 tokens × top 8 = 524,288

The global batch is 1024. The sequence length is 4096. The expert axis has 64 GPUs.

Sender pool capacity

ceil(1.10 × 524,288 ÷ 64 ÷ 3) = 3,004

This fixed pool is for one destination GPU in one wave.

Receiver expert capacity

ceil(1.15 × 524,288 ÷ 6 ÷ 3) = 33,497

This fixed array is for one local expert in one wave.

BF16 activation-bank size for one array

These values show array payload size. They do not show full device peak memory. The full peak also includes parameters, gradients, copies, compiler temporaries, and allocator reserve.

Historical direct fixed CF 1.33
[6,64,1816,3072]

3.99 GiB

Current sender pool, one wave
[64,3004,3072]

1.10 GiB

Current receiver compact, one wave
[6,33497,3072]

1.15 GiB

The receiver processes all six local experts in each wave. The per-wave receiver bank is one third of the equivalent three-wave capacity. The sender and receiver payload arrays are 1.10 GiB and 1.15 GiB before other live data.

05 · Design basis

What the profiles changed

The design came from measured memory limits and XProf traces. Some choices have a direct A/B result. Other choices use the measured fit boundary and a simple memory model.

OOM evidence

Pool by destination GPU

Direct fixed expert cells failed at the 8-of-384 shape. MHEP-055 had a 192.65 GiB XLA estimate and failed on a 123.49 GiB CUDA allocation. A destination pool removes the global expert axis from the transport buffer.

Memory boundary

Use waves

A three-expert bank fit at 182.47 GiB in MHEP-057. A six-expert bank failed at 193.77 GiB in MHEP-058. Sequential waves keep only part of the receiver bank active.

A/B profile

Use three waves

MHEP-101 used two local expert-capacity units in each of three waves. It reached 270,799 tokens/s. MHEP-102 used three units in each of two waves. It reached 268,462 tokens/s. Three waves were 0.87% faster in this short profile test.

XProf hotspot

Control traffic size

At receiver CF 1.33, the main fixed A2A shape grew to [6,64,1816,3072]. Fixed A2A self-time increased by 34.36%. NCCL send and receive time increased by 35.27%. Communication was 27.0% of exclusive device time.

Protocol constraint

Carry expert IDs in-band

Each sender and receiver knows all static shapes. The expert ID uses one packed full-width header row in the activation payload. This avoids a count exchange and a second metadata collective.

Metric definition

Report both receiver rates

For total assignments T, sender drops S, and receiver drops R, the additive rates are S/T and R/T. They add to total drop rate. The conditional receiver rate is R/(T-S).

Fit test

Use BF16 device parameters

The six-expert bank failed before a timed step with the original device parameter storage. The BF16 parameter test MHEP-059 completed 10 steps at 286,467 tokens/s. The optimizer state and the FP32 master parameters stay in pinned host memory.

Traffic reduction

Route the 3072-wide latent

The router still reads the 6144-wide model state. The routed activation is projected to 3072 before dispatch. This halves the width of the values sent through the expert transport.

Current rack gate

Use sender CF 1.10 and receiver CF 1.15

MHEP-118 completed 20 steps without an OOM or preemption. Median throughput for steps 2 through 19 was 250,691 tokens/s. The lower receiver factor reduces its BF16 compact array from 1.33 GiB to 1.15 GiB. This run is too short to measure the final drop rate.

06 · Results

Evidence from the one-rack runs

Short one-rack runs found fit and profile limits. MHEP-103 is the 200-step result for the prior 1.05 sender and 1.33 receiver factors. MHEP-118 tests the merged 1.10 sender and 1.15 receiver factors for 20 steps. Early drop values change as router balance changes during training.

Run	Change	Result	Measured value
MHEP-055	Direct fixed A2A, 8-of-384	OOM	XLA 192.65 GiB. CUDA request 123.49 GiB.
MHEP-057	Three-expert bank, CF 1.0	Fit	XLA 182.47 GiB. 264,613 tokens/s. Communication 21.2%.
MHEP-058	Six-expert bank, CF 1.0	OOM	XLA 193.77 GiB. CUDA request 120.26 GiB.
MHEP-059	BF16 device parameters	Fit	286,467 tokens/s over the short steady window.
MHEP-061	Receiver CF 1.33	Fit, slower	255,310 tokens/s. Fixed A2A self-time +34.36%.
MHEP-101	Three-wave pooled profile	Selected	270,799 tokens/s over steps 2–9.
MHEP-102	Two-wave pooled profile	Valid	268,462 tokens/s over steps 2–9. It was 0.87% slower.
MHEP-103	Prior CF 1.05 / 1.33, 200 steps	Passed	256,818 tokens/s and 2.41% drop over steps 150–199.
MHEP-118	Merged CF 1.10 / 1.15, 20 steps	Passed	250,691 median tokens/s over steps 2–19. Final drop was 19.33%: 7.14% sender and 12.19% receiver. The conditional receiver rate was 13.12%.
4-rack d2560	CF 1.15 / 1.15, 24 layers	In progress	At step 17,563, the prior 501-step medians were 6.38M tokens/s and 3.08% drop: 2.92% sender and 0.162% receiver.

XProf signal at CF 1.33

27.0%communication

NCCL send and receive was the largest single kernel group. It used 17.9% of exclusive profiled device time in MHEP-061.

● communication ● compute

Prior hero long-run result

MHEP-103 completed all 200 steps on 16 nodes and 64 GPUs with the prior factors. It had no OOM, non-finite loss, failure, or preemption. The final loss was 3.25104.

Longer drop signal at another scale

The four-rack d2560 run had processed 294.7B tokens at step 17,563 of 88,242. Its prior 501-step median receiver drop was 0.162%, while sender drop was 2.92%. This run uses both capacity factors at 1.15 and a smaller 24-layer model, so its throughput is not a direct hero comparison.

07 · Scope

What this data does not prove

The report separates measured results from the current engineering judgment.

Three waves won one short A/B profile. This is not a full search over all wave counts and pool layouts.
The merged 1.10 sender and 1.15 receiver factors have one 20-step hero result. The prior 1.05 sender and 1.33 receiver factors have one full 200-step hero result.
MHEP-118 verified sender and receiver metrics on one rack. Its 19.33% final drop is an early-run value and is not a final drop estimate.
The four-rack d2560 run supports the drop trend, but it uses a smaller model and both capacity factors at 1.15.
The merged receiver keeps rows in fixed source order and does not use router scores for overflow. Open PR #8347 proposes a per-receiver rotation so overflow does not always select the same last source.
The buffer sizes in this report are payload estimates. XLA peak memory includes more live data.
The selected throughput has one rack placement. The memory and drop conclusions have higher confidence than the exact throughput value.

08 · Links

Code and run records

The HTML file has no external script or style dependency. Run links need network access.

Report basis: merged main at a4dcd392f, saved Iris monitoring records, W&B history, and XProf summaries. Generated on 17 August 2026. The four-rack W&B values are a step-17,563 snapshot from an active run.