NCCL issue 2428: NCCL_IGNORE_NET_MISMATCH turns a heterogeneous-HCA misconfiguration into a silent hang

Fields

id

sources/mgpu-nccl-2428-net-mismatch-hang

URL

https://github.com/NVIDIA/nccl/issues/2428

Publisher

NVIDIA/nccl

Kind

repo

Published

2026-09-21

Accessed

2026-10-03

notes

2 nodes x 8x H20 over RoCE/eRDMA; rank 0 builds channels on an HCA rank 1 cannot open. Init completes then spins forever: SM 100%, memory throughput 0%, ~118 W of ~400 W, perf shows recvProxyProgress/_raw_spin_lock.

Notes

2 nodes x 8x H20 over RoCE/eRDMA; rank 0 builds channels on an HCA rank 1 cannot open. Init completes then spins forever: SM 100%, memory throughput 0%, ~118 W of ~400 W, perf shows recvProxyProgress/_raw_spin_lock.

Cross-references

This record references no other records.