inference/research
verifiedhigh · 0.85blockerupdated 2026-10-11

ROCr runtime refuses kernel dispatches declaring hostcall buffers when PCIe root port lacks AtomicOps, causing all RCCL collectives to fail

[blocker] driver

Fields

id
gotchas/rocm-rccl-pcie-atomicops-requirement
Class
driver
Severity
blocker
Concepts
rocmrcclpcieatomicopshostcallmulti-gpu
Symptom
When initializing multi-GPU inference with RCCL/vLLM on consumer motherboards or virtualized setups (VFIO/KVM), all RCCL collective operations fail at kernel dispatch. The ROCr runtime refuses any kernel execution that declares a hostcall buffer.
Root cause
The PCIe root port or bridge above the consumer GPU fails to advertise PCIe 3.0/4.0 AtomicOps capability. The ROCr runtime strictly requires AtomicOps to allocate and manage hostcall buffers used by RCCL collectives for multi-GPU synchronization. If the root port does not advertise AtomicOps, dispatch is rejected by the runtime.
Workaround
On bare metal, enable PCIe AtomicOps, ACS, and ReBAR in the motherboard BIOS (and update BIOS firmware). On virtual machines (QEMU/KVM/VFIO), configure the root PCIe port with AtomicOps routing/forwarding enabled (e.g. qemu host bridge flags) so the guest kernel can see root port AtomicOps support.
notes
Documented in dual-radeon-vllm and reported in ROCm issue #6520; reproduced on both bare metal consumer boards and virtualized topologies.

Notes

Documented in dual-radeon-vllm and reported in ROCm issue #6520; reproduced on both bare metal consumer boards and virtualized topologies.

Cross-references