AWS Neuron docs: bert-base-cased on inf1.xlarge batch, 1095 inf/s (p50 58ms)
1095.0 inferences/s (encoder, batch 8, seq_len 128, fp32)
Fields
id
benchmarks/aws-neuron-inf1-bert-base-batch-inf-sMetric
tps_aggregate
Value
1,095
Unit
inferences/s (encoder, batch 8, seq_len 128, fp32)
Model
bert-base-cased
Methodology
VENDOR-MEASURED (AWS Neuron docs, Inf1 Inference Performance, NeuronPerf, last update 2024-09-16, Neuron 2.20). inf1.xlarge = 1 Inferentia1 chip (4 NeuronCores). Encoder/CNN models only — NO generative LLM rows exist for Inf1. Units are inferences/sec (BERT, seq_len 128) or images/sec (ResNet/YOLO), NOT tokens/sec. PyTorch 1.13.1, batch 8, Data Parallel.
Measured by
vendor
Reproducible
yes
notes
Batch-optimized encoder row. Inf1 is EOL for new work (Neuron docs mark Inf1 pages archived) but remains the cheapest AWS inference silicon; recorded as the vendor reference number.
Notes
Batch-optimized encoder row. Inf1 is EOL for new work (Neuron docs mark Inf1 pages archived) but remains the cheapest AWS inference silicon; recorded as the vendor reference number.