AWS Neuron Inf2: flan-t5-xl, inf2.24xlarge TP8, batch 1, 128in/84out, throughput-optimized
117.61 tokens/s (input+output counted, TP8)
Fields
id
benchmarks/aws-inf2-flan-t5-xl-tp8-b1-128io-tputMetric
tps_aggregate
Value
117.61
Unit
tokens/s (input+output counted, TP8)
Model
google/flan-t5-xl
Methodology
VENDOR MEASURED (AWS Neuron, Neuron SDK 2.17.0, NeuronX Distributed). inf2.24xlarge (Inferentia2), TP8/DP1, batch 1, seq 128, in 128/out 84, flan-t5-xl FP32 autocast Matmult-BF16. Latency per token P50 8.51 ms / P99 8.53 ms.
Measured by
vendor
Reproducible
yes
notes
Gives aws-inferentia2 a second benchmark record alongside the existing fp16 alt-aws-inf2 row, and enables a same-model Trainium1-vs-Inferentia2 comparison (122.5 vs 117.6 tok/s).
Notes
Gives aws-inferentia2 a second benchmark record alongside the existing fp16 alt-aws-inf2 row, and enables a same-model Trainium1-vs-Inferentia2 comparison (122.5 vs 117.6 tok/s).