verifiedhigh · 0.85updated 2026-10-03

NCCL issue 2379: Probabilistic multi-communicator init hang in the cuMem UDS fd exchange on single-node PCIe

repo

Fields

sources/mgpu-nccl-2379-cumem-uds-hang

URL

https://github.com/NVIDIA/nccl/issues/2379

Publisher

NVIDIA/nccl

Kind

repo

Published

2026-08-29

Accessed

2026-10-03

notes

8x RTX 3090, no NVLink, TP2xPP4+EP creating ~50 communicators; ~1 boot in 7 never comes up. NCCL_CUMEM_ENABLE=0 gives 20/20 clean boots vs 5/9, Fisher p=0.00106.

Notes

8x RTX 3090, no NVLink, TP2xPP4+EP creating ~50 communicators; ~1 boot in 7 never comes up. NCCL_CUMEM_ENABLE=0 gives 20/20 clean boots vs 5/9, Fisher p=0.00106.

Cross-references

This record references no other records.