AWS Neuron Inf2: bert-large-uncased fp32, inf2.xlarge, batch 4, throughput-optimized
950.05 inferences/s (batch-throughput-optimized, seq len 128)
Fields
id
benchmarks/aws-inf2-bert-large-batch4-tputMetric
tps_aggregate
Value
950.05
Unit
inferences/s (batch-throughput-optimized, seq len 128)
Model
bert-large-uncased
Methodology
VENDOR MEASURED (AWS NeuronPerf, Neuron SDK 2.21.0). inf2.xlarge, bert-large-uncased FP32 autocast Matmult-BF16, seq 128, batch 4, data-parallel, 'Batch' application. Latency P50 8.41 ms / P99 8.85 ms.
Measured by
vendor
Reproducible
yes
notes
Direct gen-over-gen comparison to aws-trainium1's bert-large row: Trn1 hits 1104 inf/s on the same model/batch config where Inf2 manages 950 — consistent with Trainium being the higher-compute sibling.
Notes
Direct gen-over-gen comparison to aws-trainium1's bert-large row: Trn1 hits 1104 inf/s on the same model/batch config where Inf2 manages 950 — consistent with Trainium being the higher-compute sibling.