verifiedhigh · 0.85updated 2026-10-03

RCCL issue 2191: Multi-communicator concurrent collectives deadlock on gfx90a (FSDP2+TP)

repo

Fields

sources/mgpu-rccl-2191-multicomm-deadlock

URL

https://github.com/ROCm/rccl/issues/2191

Publisher

ROCm/rccl

Kind

repo

Published

2026-09-10

Accessed

2026-10-03

notes

4x gfx90a GCD on MI250, RCCL 2.30.4; 50-100% hang with 2-3 concurrent communicators. NCCL_LAUNCH_ORDER_IMPLICIT=1 rescues the 2-comm case (10/10) but not 3-comm (0/10).

Notes

4x gfx90a GCD on MI250, RCCL 2.30.4; 50-100% hang with 2-3 concurrent communicators. NCCL_LAUNCH_ORDER_IMPLICIT=1 rescues the 2-comm case (10/10) but not 3-comm (0/10).

Cross-references

This record references no other records.