Aggregate output throughput, MPT-7B static batch 64, 1x A100

Databricks Table 2, MPT-7B row '1 x A100', batch size 64 column: peak throughput 12.5 requests/s with static batching. Hardware: 1x NVIDIA A100 40GB. IMPORTANT: this repo has no 40GB A100 accelerator record, so accelerator_ids is deliberately empty rather than pointing at the 80GB A100 records, which are a different part. Backend: FasterTransformer-based static batching. Requests: 512 input tokens, 64 output tokens, uniform. Measured by Databricks/MosaicML on their own production serving stack. DERIVED VALUE: the source table reports requests/s, not tokens/s. 12.5 req/s x 64 output tokens = 800 output tokens/s aggregate. Derivation is stated in notes because the benchmark schema has no *_basis field and forbids additional properties.

DERIVATION BASIS: Databricks Table 2 gives 12.5 req/s at batch size 64 on 1x A100 for MPT-7B at 512 input / 64 output tokens; multiplied by the fixed 64-token output length this is 800 aggregate output tokens/s. This number is ONLY meaningful next to mpt7b-a100-bs64-per-user-decode-tps (12.5 tok/s per user) and mpt7b-a100-bs1-per-user-decode-tps (57.6 tok/s per user) - the aggregate figure is 64x the per-user figure here purely because 64 requests share it. Quoting 800 tok/s as 'the speed of this model' is the single most common misread in serving benchmarks: no individual user sees 800 tok/s. Same table, same model, same GPU: batch 1 is 0.9 req/s. Going from batch 1 to batch 64 raises aggregate throughput 13.9x while per-user decode speed FALLS 4.6x, and the blog states the tradeoff explicitly - 'with one NVIDIA A100 GPU, if we maximize throughput with a batch size of 64, latency increases by 4x while throughput increases by 14x'. The blog also warns that after the compute-bound crossover every further doubling of batch size raises latency without raising throughput. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: NOT STATED by the source - absent, not inferred. A throughput figure with no scenario cannot be placed against an MLPerf Offline or Server number (they are different measurements; on identical hardware the Offline-minus-Server difference is negative in 5 of 9 systems measured). ABSENT CONDITIONS (2 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - engine version: unrecoverable from bench-meth-databricks-inference-tuning - engine_id=fastertransformer is named but no build/commit/container digest is published - warmup: unrecoverable from bench-meth-databricks-inference-tuning - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted.