Dynamo#12463: UCX_NET_DEVICES unset causes garbage tokens in P/D serving

Fields

id

sources/src5-gotcha-dynamo-12463-ucx-net-devices

URL

https://github.com/ai-dynamo/dynamo/issues/12463

Publisher

NVIDIA Dynamo

Kind

forum

Published

2026-07-30

Accessed

2026-10-04

notes

Critical silent failure: vLLM disaggregated serving over InfiniBand returns garbage tokens when UCX_NET_DEVICES is unset. HTTP 200 with well-formed response but corrupt content. Setting UCX_NET_DEVICES=mlx5_0:1 fixes it. Affects 2-node H200 cluster with NixlConnector. No error surfaced to client. Demonstrates multi-GPU/NVLink silent failure.

Notes

Critical silent failure: vLLM disaggregated serving over InfiniBand returns garbage tokens when UCX_NET_DEVICES is unset. HTTP 200 with well-formed response but corrupt content. Setting UCX_NET_DEVICES=mlx5_0:1 fixes it. Affects 2-node H200 cluster with NixlConnector. No error surfaced to client. Demonstrates multi-GPU/NVLink silent failure.

Cross-references

This record references no other records.