vLLM: Meta-Llama-3.1-405B-Instruct-FP8 on 8x H100 (TP=8), aggregate output throughput

Engine: vLLM (July 2024 launch build), launched as `vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tensor-parallel-size 8`. Hardware: 8x H100 on a single Lambda 1-Click node with InfiniBand. Tensor-parallel size 8, pipeline parallel 1. Synthetic workload with average input length 1024 tokens and average output length 128 tokens. Server sustained 2.82 requests/s, giving 2884.86 input tokens/s and 291.53 output tokens/s. Concurrency: NOT stated and NOT controlled - the source reports a sustained server rate under load and does not give the request arrival pattern, the number of concurrent requests in flight, or the batch size actually achieved. Figure is AGGREGATE output tokens/s across the whole 8-GPU server (about 36 tok/s per GPU), not per-request. Omitted by the source: exact vLLM commit, container image, clock settings.

Published by the vLLM project with Meta co-announcement, on cloud hardware vLLM does not control, and the post itself calls the performance 'preliminary' / 'early reference points'. Reproducible in the sense that the command line and workload shape are published, but the missing arrival pattern means the exact 291.53 is not re-derivable; reproducible is therefore false. The same post reports GSM8K 8-shot CoT exact match 95.38% (+/- 0.56) for the FP8 checkpoint vs 96.8% official BF16 - a useful quality companion to this throughput number.