Silex (third-party genai-perf): Llama 3.3 70B Instruct bf16 on 4x H100 80GB SXM, ~7000 total tokens/s at 200/200

Engine: NVIDIA NIM container nvcr.io/nim/meta/llama-3.3-70b-instruct:1.8.2, which serves through TensorRT-LLM with the default NIM profile: bfloat16 weights, tensor-parallel size 4, pipeline-parallel size 1. Hardware: 4x NVIDIA H100 80GB SXM5 in one host. Prompt length 200 tokens, output length 200 tokens, synthetic prompts generated by genai-perf ('translation / Q&A' pattern). Concurrency: swept 1, 2, 5, 10, 50, 100, 250, 500 users; the ~7,000 TPS figure is the peak of that sweep, reached near 500 concurrent users. Figure is AGGREGATE total tokens/s (prompt + generated) for the 4-GPU host, as reported by genai-perf's 'Total Tokens/Second'. Omitted by the source: the value is read off a published chart and is stated by the author as an approximation; genai-perf request rate / saturation policy and the achieved batch size are not stated; TTFT is reported as a median without the exact percentile at that point. Same post reports ~2,600 TPS at 250 users for the 1000-in / 200-out workload, and A100 80GB PCIe on the same setup saturating near ~570 TPS at ~50 users.

Third-party run with the hardware, container, parallelism and workload shape all pinned, and the benchmark scripts plus raw data published in a public GitHub repo (derekelewis/ml-notes), so it is reproducible in the practical sense - which is why reproducible is true despite the value being read from a chart. Confidence is kept at 0.6 because the headline number itself is the author's '~' approximation from a chart image rather than a raw counter, and because bf16 (not FP8) is used, which is the worst case for Hopper. Note the A100 host in the same post used PCIe rather than SXM form factor, so the stated ~12x H100-vs-A100 gap conflates architecture with form factor and interconnect - do not quote that 12x as a clean Hopper-vs-Ampere number.