AWS Neuron docs: t5-3b on trn1.32xlarge TP=8, 116.29 tok/s (in+out)
116.29 tok/s (in+out tokens, batch-1, TP=8 over 16-chip trn1.32xlarge)
Fields
id
benchmarks/aws-neuron-trn1-t5-3b-tp8-128io-tpsMetric
tps_aggregate
Value
116.29
Unit
tok/s (in+out tokens, batch-1, TP=8 over 16-chip trn1.32xlarge)
Model
t5-3b
Methodology
VENDOR-MEASURED (AWS Neuron docs, Trn1/Trn1n Inference Performance, NeuronPerf). trn1.32xlarge (16 Trainium1 chips). NeuronX Distributed tensor-parallel TP=8, DP=1, batch 1, seq_len 128, input 128 + output ~84 tokens, FP32 weights with Matmult-BF16 autocast. 'Throughput (tokens/second)' counts BOTH input and output tokens (vendor note). Encoder-decoder text-generation rows.
Measured by
vendor
Reproducible
yes
notes
Latency/token p50 8.58 ms. NeuronX Distributed compile path; Neuron SDK ~2.17. Trainium1 is not a consumer part but IS a budget cloud inference path (trn1 cheaper than GPU equivalents for compiled models); first benchmark coverage for aws-trainium1.
Notes
Latency/token p50 8.58 ms. NeuronX Distributed compile path; Neuron SDK ~2.17. Trainium1 is not a consumer part but IS a budget cloud inference path (trn1 cheaper than GPU equivalents for compiled models); first benchmark coverage for aws-trainium1.