inference/research
verifiedhigh · 0.85updated 2026-10-03

NCCL issue 949: Slow (~56%) inter-node all_reduce_perf performance on A100 Virtual Machine

repo

Fields

id
sources/mgpu-nccl-949-vm-topology-perf
URL
https://github.com/NVIDIA/nccl/issues/949
Publisher
NVIDIA/nccl
Kind
repo
Published
2023-08-07
Accessed
2026-10-03
notes
2x A100 SXM4*8 nodes with SR-IOV VF InfiniBand pass-through into VMs; an NCCL_TOPO_FILE xml was needed to tell NCCL the topology, and inter-node all-reduce still ran at ~56% of expected.

Notes

2x A100 SXM4*8 nodes with SR-IOV VF InfiniBand pass-through into VMs; an NCCL_TOPO_FILE xml was needed to tell NCCL the topology, and inter-node all-reduce still ran at ~56% of expected.

Cross-references

This record references no other records.