HF optimum-neuron on AWS inf2.48xlarge: Llama2 13B (latency config, bs1), 145 tok/s at 256 new tokens
145.0 tok/s (single request; Neuron-compiled)
Fields
id
benchmarks/hf-optimum-neuron-inf2-llama2-13b-l-1x-256new-tpsMetric
tps_aggregate
Value
145
Unit
tok/s (single request; Neuron-compiled)
Model
meta-llama/Llama-2-13b-hf
Methodology
VENDOR-ADJACENT MEASURED (Hugging Face optimum-neuron team, huggingface.co/blog/inferentia-llama2). AWS inf2.48xlarge (12 Inferentia2 chips, 24 NeuronCores in use), model Neuron-compiled for max_seq 2048, input 256 tokens + 256 generated tokens, batch 1. Throughput = batch*(input+output)/e2e-latency per the blog's convention — aggregate, not per-request decode. Latency-oriented Neuron compile (bs1).
Measured by
vendor
Reproducible
yes
notes
Numbers from the published throughput table at 256 new tokens (256-input). Compiled-model Neuron SDK figures — no llama.cpp/vLLM path exists on Inferentia. inf2.48xlarge = 12x Inferentia2 chips over NeuronLink v2; the 24-core compile uses 2 chips' worth of cores (each chip has 2 NeuronCores v2) — per the blog all non-budget configs span the full instance.
Notes
Numbers from the published throughput table at 256 new tokens (256-input). Compiled-model Neuron SDK figures — no llama.cpp/vLLM path exists on Inferentia. inf2.48xlarge = 12x Inferentia2 chips over NeuronLink v2; the 24-core compile uses 2 chips' worth of cores (each chip has 2 NeuronCores v2) — per the blog all non-budget configs span the full instance.