AWS Neuron documentation: Inf2 Inference Performance tables

Fields

id

sources/aws-neuron-inf2-inference-performance

URL

https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/benchmarks/inf2/inf2-performance.html

Publisher

AWS

Kind

benchmark

Accessed

2026-10-15

notes

Vendor reference inference numbers for Inf2 (Inferentia2). Encoder models reported as inference/sec; encoder-decoder (flan-t5-xl, t5-3b) as tokens/sec counting BOTH input and output tokens, NeuronX Distributed TP=8 on inf2.24xlarge. Neuron SDK 2.17-2.28, model dtype FP32 autocast Matmult-BF16. Last update 2026-02-26.

Notes

Vendor reference inference numbers for Inf2 (Inferentia2). Encoder models reported as inference/sec; encoder-decoder (flan-t5-xl, t5-3b) as tokens/sec counting BOTH input and output tokens, NeuronX Distributed TP=8 on inf2.24xlarge. Neuron SDK 2.17-2.28, model dtype FP32 autocast Matmult-BF16. Last update 2026-02-26.

Cross-references

This record references no other records.