AWS Neuron docs: google/flan-t5-xl on trn1.32xlarge TP=8, 122.52 tok/s (in+out)
122.52 tok/s (in+out tokens, batch-1, TP=8 over 16-chip trn1.32xlarge)
Fields
id
benchmarks/aws-neuron-trn1-google-flan-t5-xl-tp8-128io-tpsMetric
tps_aggregate
Value
122.52
Unit
tok/s (in+out tokens, batch-1, TP=8 over 16-chip trn1.32xlarge)
Model
google/flan-t5-xl
Methodology
VENDOR-MEASURED (AWS Neuron docs, Trn1/Trn1n Inference Performance, NeuronPerf). trn1.32xlarge (16 Trainium1 chips). NeuronX Distributed tensor-parallel TP=8, DP=1, batch 1, seq_len 128, input 128 + output ~84 tokens, FP32 weights with Matmult-BF16 autocast. 'Throughput (tokens/second)' counts BOTH input and output tokens (vendor note). Encoder-decoder text-generation rows.
Measured by
vendor
Reproducible
yes
notes
Latency/token p50 8.16 ms. NeuronX Distributed compile path; Neuron SDK ~2.17. Trainium1 is not a consumer part but IS a budget cloud inference path (trn1 cheaper than GPU equivalents for compiled models); first benchmark coverage for aws-trainium1.
Notes
Latency/token p50 8.16 ms. NeuronX Distributed compile path; Neuron SDK ~2.17. Trainium1 is not a consumer part but IS a budget cloud inference path (trn1 cheaper than GPU equivalents for compiled models); first benchmark coverage for aws-trainium1.