AWS Neuron Trn1: bert-large-uncased fp32, trn1.2xlarge, batch 4, throughput-optimized
1104.44 inferences/s (batch-throughput-optimized, seq len 128)
Fields
id
benchmarks/aws-trn1-bert-large-batch4-tputMetric
tps_aggregate
Value
1,104.44
Unit
inferences/s (batch-throughput-optimized, seq len 128)
Model
bert-large-uncased
Methodology
VENDOR MEASURED (AWS NeuronPerf, Neuron SDK 2.27.0). trn1.2xlarge = 1 Trainium chip. bert-large-uncased FP32 autocast Matmult-BF16, seq 128, batch 4, data-parallel, 'Batch' application. Latency P50 7.24 ms / P99 7.3 ms.
Measured by
vendor
Reproducible
yes
notes
Encoder throughput row — the strongest single-chip number AWS publishes for gen-1 Trainium and the direct comparison point to the Inf1 record for the same model class.
Notes
Encoder throughput row — the strongest single-chip number AWS publishes for gen-1 Trainium and the direct comparison point to the Inf1 record for the same model class.