Perplexity: Llama 2 70B FP8 on 8x H100 80GB, TP=2 / batch 128, 767 output tokens/s per GPU

Engine: TensorRT-LLM v0.5.0, the open-source library - Perplexity ran it on their own infrastructure (AWS p5.48xlarge, 8x H100-HBM3-80GB). Model: Llama 2 70B, FP8 weights and FP8 activations (H100 native FP8). Prompt length 1024 tokens, output length 512 tokens, synthetic dataset chosen to match Perplexity's observed Llama-2-70B production traffic distribution. Batch size 128, concurrency 128 in-flight requests. Tensor-parallel size 2 (the serving server spans TP=2, so per-GPU figures are normalised across the 8 GPUs of the host). Figure is AGGREGATE output tokens/s per GPU, not per-request. This is the throughput-MAXIMISING point of the sweep and the post is explicit that it is also the slowest: response latency was 'closer to 42000 ms' for the full 512 output tokens. The post gives two better-balanced tradeoffs in absolute terms: TP=4 / batch 128 at 626 tok/s/GPU with 26188 ms response time, and TP=2 / batch 32 at 435 tok/s/GPU with 18821 ms response time. Omitted by the source: exact TensorRT-LLM commit beyond the v0.5.0 tag, clock settings, and whether KV cache was FP8.

Independent third party (Perplexity) running NVIDIA's engine, which is exactly the case the measured_by field exists for - neither vendor benchmark nor our own run. Their stated purpose was to confirm NVIDIA's H100 marketing, which they did (184-202% throughput vs A100 at matched settings, 251-460% under a latency constraint), but the comparison baseline is Perplexity's own A100 fleet, not a vendor-supplied A100 number. Reproducible is false because the exact engine build and clock settings are not pinned and the absolute figures come from their own figure images rather than a released data file. Accuracy companion from the same post: WikiText perplexity change <1% for FP8 and w8a8 SmoothQuant, ~7% for w4a16 - the reason they rejected w4a16. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: MLPerf-style server scenario. ABSENT CONDITIONS (1 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - warmup: unrecoverable from bench-nv-perplexity-h100-llama2-blog - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted.