Turbocharging Llama 2 70B with NVIDIA H100 (Perplexity)

Perplexity third-party study of Llama 2 70B with TensorRT-LLM v0.5.0 on 8x80GB H100 (p5.48xlarge) vs 8x A100. Synthetic dataset of 1024 input / 512 output tokens, TP sweep 2/4/8, batch sweep 1..128. Absolute figures given for the throughput-maximising and two compromise configurations, plus perplexity deltas (<1% for fp8 and w8a8-SmoothQuant, 7% for w4a16 on WikiText).