id
flops/deepep-expert-parallel-dispatch-combine
Arithmetic intensity
Not FLOPs - bytes, and specifically bytes that cross NVLink or RDMA rather than HBM. Dispatch moves each routed token's activation to the rank owning its expert; combine moves expert outputs back. The kernel-level content is the overlap: num_sms bounds how many SMs the communication kernel occupies so expert GEMMs can run concurrently, and defer_epilogue=True with async_with_compute_stream=True returns an EventOverlap so the dispatch epilogue runs on the current stream while compute proceeds. With do_cpu_sync=False (the decode path) the kernel avoids a CPU synchronization entirely, allocating outputs to configured capacity and exposing psum_num_recv_tokens_per_expert for the downstream grouped GEMM to find valid ranges - that is the difference between a decode step that stalls on a host round trip and one that does not. FP8 dispatch halves the bytes on the wire.
Scales with
num_experts and experts_per_token (total bytes moved is tokens x topk)hidden_widthinterconnect bandwidth (NVLink intranode, RDMA internode)num_sms (communication/compute overlap budget)expert redundancy (NVLink weight and gradient exchange for redundant experts)
Hardware parameters
parametermemory_bandwidtheffectSo on a part without NVLink (or on a PCIe-only host) DeepEP is not the path, and expert parallelism becomes bandwidth-bound on the slowest transport rather than hidden behind MMA.basisunverifiedvalues_fornvidia-h100-sxm,nvidia-h200-sxm,nvidia-b200,nvidia-gb200-nvl72
Workarounds
num_sms to cap the comm kernel's SM footprint and leave SMs for the expert GEMMFP8 dispatch to halve wire bytesdo_cpu_sync=False for decode, with GPU-side receive counts consumed by the masked grouped GEMMdefer_epilogue + async_with_compute_stream for an explicit overlap handlelb_prefetch_weights / lb_reduce_grads to move expert weights and quantization scales over NVLink before the expert compute and accumulate redundant experts' FP32 gradients after
notes
Hardware scope is explicit and worth quoting: 'EP dispatch and combine require GPU SMs; zero-SM RDMA EP is not supported', requires NVIDIA Hopper or newer, NVLink intranode plus RDMA internode, and auto-detects bandwidth with nvidia-smi and ibstat. So on a part without NVLink (or on a PCIe-only host) DeepEP is not the path, and expert parallelism becomes bandwidth-bound on the slowest transport rather than hidden behind MMA. Two claims NOT to conflate: 'dispatch/combine overlap' reduces exposed latency, while DeepGEMM's Mega MoE reduces the actual HBM round trips by fusing the stages. Ascend ports exist (DeepEP-Ascend).
HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.