AWS Neuron Inf1: BERT-base-uncased fp32, inf1.xlarge, batch 3, latency-optimized
285.0 inferences/s (latency-optimized, seq len 128)
Fields
id
benchmarks/aws-inf1-bert-base-lat-opt-tputMetric
tps_aggregate
Value
285
Unit
inferences/s (latency-optimized, seq len 128)
Model
bert-base-uncased
Methodology
VENDOR MEASURED (AWS NeuronPerf, Neuron SDK 2.20.0). inf1.xlarge, bert-base-uncased FP32, seq len 128, batch 3, data-parallel, 'Real Time' application type (min-latency). Latency P50 10 ms / P99 11 ms. Aggregate across 4 NeuronCores.
Measured by
vendor
Reproducible
yes
notes
Latency-optimized row; the batch-optimized counterpart achieves 1181 inf/s at batch 6 — the ~4x throughput-vs-latency tradeoff is characteristic of the part. Records the same model/quant/config axis as the throughput row.
Notes
Latency-optimized row; the batch-optimized counterpart achieves 1181 inf/s at batch 6 — the ~4x throughput-vs-latency tradeoff is characteristic of the part. Records the same model/quant/config axis as the throughput row.