AWS Neuron: trn1.32xlarge (Trainium1), NeuronX Distributed, google/flan-t5-xl, TP=8, batch 1, throughput-opt
122.52 tok/s (counts input+output tokens; TP=8 across 16 Trainium chips)
Fields
id
benchmarks/aws-trn1-neuronxd-flant5xl-fp32-tp8-b1-tpsMetric
tps_aggregate
Value
122.52
Unit
tok/s (counts input+output tokens; TP=8 across 16 Trainium chips)
Model
google/flan-t5-xl
Methodology
VENDOR-MEASURED (AWS Neuron docs, NeuronPerf). NeuronX Distributed on trn1.32xlarge (16x Trainium1), Tensor Parallel degree 8, DP 1, batch 1, seq_len 128, in_len 128, out_len 84, FP32 weights / Matmult-BF16 autocast, Neuron 2.17.0. Throughput tokens/sec counts BOTH input and output tokens — NOT pure decode.
Measured by
vendor
Reproducible
yes
notes
Encoder-decoder text-generation row. Engine = AWS NeuronX Distributed (Neuron SDK PyTorch tensor-parallel runtime); engine_id uses the repo's alt-neuron-compiler engine tag.
Notes
Encoder-decoder text-generation row. Engine = AWS NeuronX Distributed (Neuron SDK PyTorch tensor-parallel runtime); engine_id uses the repo's alt-neuron-compiler engine tag.