AWS Neuron Inf1: BERT-base-uncased fp32, inf1.xlarge, batch 6, throughput-optimized
1181.0 inferences/s (batch-throughput-optimized, seq len 128)
Fields
id
benchmarks/aws-inf1-bert-base-batch128-tputMetric
tps_aggregate
Value
1,181
Unit
inferences/s (batch-throughput-optimized, seq len 128)
Model
bert-base-uncased
Methodology
VENDOR MEASURED (AWS NeuronPerf, Neuron SDK 2.20.0). inf1.xlarge = 1 Inferentia chip (4 NeuronCores). bert-base-uncased FP32, sequence length 128, batch size 6, data-parallel run mode, 'Batch' application type (max throughput). Latency P50 41 ms / P99 45 ms. AGGREGATE across the 4 NeuronCores; not a decode tok/s — BERT is an encoder, so the figure is complete inferences/s.
Measured by
vendor
Reproducible
yes
notes
Throughput-optimized row from AWS's Inf1 table. The companion latency-optimized run of the same model gives 285 inf/s at batch 3 (P50 10 ms) — recorded as a separate record. This is the strongest fp32 encoder figure AWS publishes for a single Inferentia gen-1 chip and the only measured throughput row the repo has for this part.
Notes
Throughput-optimized row from AWS's Inf1 table. The companion latency-optimized run of the same model gives 285 inf/s at batch 3 (P50 10 ms) — recorded as a separate record. This is the strongest fp32 encoder figure AWS publishes for a single Inferentia gen-1 chip and the only measured throughput row the repo has for this part.