Per-user decode tok/s, MPT-7B static batch 1, 1x A100

Databricks Table 2, MPT-7B row '1 x A100', batch size 1 column: peak throughput 0.9 requests/s with static batching. Hardware: 1x NVIDIA A100 40GB. IMPORTANT: this repo has no 40GB A100 accelerator record, so accelerator_ids is deliberately empty rather than pointing at the 80GB A100 records, which are a different part. Backend: FasterTransformer-based static batching. Requests: 512 input tokens, 64 output tokens, uniform. Measured by Databricks/MosaicML on their own production serving stack. DERIVED VALUE: 0.9 req/s x 64 output tokens = 57.6 output tokens/s; at batch size 1 there is exactly one user, so per-user and aggregate are the same number. Equivalent to a time-per-output-token of about 17.4 ms/token.

DERIVATION BASIS: 0.9 req/s x 64 output tokens = 57.6 tok/s at batch size 1 on 1x A100, per Databricks Table 2. This is the per-user counterpart to mpt7b-a100-bs64-aggregate-output-tps. The pair is the trap: the SAME model, SAME GPU, SAME engine, SAME request shape, and the aggregate throughput number is 13.9x larger (800 vs 57.6 tok/s) while the number the user actually feels is 4.6x SMALLER (12.5 vs 57.6 tok/s). Any throughput number without its batch size is uninterpretable. The blog makes the definitional point explicitly: throughput is 'the number of output tokens per second an inference server can generate across all users and requests', while time-per-output-token is per user, and the two trade directly against each other - at 16 concurrent queries you get higher throughput but longer per-user token latency. Note this is a static-batching measurement, which the blog says underperforms dynamic batching; continuous batching is quoted at 10x-20x better throughput than dynamic batching. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: NOT STATED by the source - absent, not inferred. A throughput figure with no scenario cannot be placed against an MLPerf Offline or Server number (they are different measurements; on identical hardware the Offline-minus-Server difference is negative in 5 of 9 systems measured). ABSENT CONDITIONS (2 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - engine version: unrecoverable from bench-meth-databricks-inference-tuning - engine_id=fastertransformer is named but no build/commit/container digest is published - warmup: unrecoverable from bench-meth-databricks-inference-tuning - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted. ACCELERATOR POINTER AUDIT 2026-10-05 (agent w6-benchmarks): VALUE UNCHANGED, and the pointer this record defends in its methodology is now STALE. The methodology says "this repo has no 40GB A100 accelerator record, so accelerator_ids is deliberately empty rather than pointing at the 80GB A100 records, which are a different part". That was true when written and is false now: data/accelerators/nvidia-a100-40gb-sxm4 exists, verified, 1555 GB/s, 40 GB, form factor sxm, and its own notes state it is DISTINCT from the other two 40 GB A100 forms. So the reasoning still holds - an empty pointer is better than a wrong one, and the 40GB SXM4 is still not the part Databricks measured - but the justification is no longer accurate, and a reader who trusts the sentence will go looking for a missing record and find this one instead. NOT CHANGED because the fix is a wording change in a sibling record that a human should make with the three mpt7b records in view; they are a set and editing one alone would make the trio inconsistent. FLAG: the roofline sweep can now compute this record: 1x A100 40GB SXM4 at 1555 GB/s, MPT-7B FP16 = 14 GB of weights, roof = 1555e9/14e9 = 111.1 tok/s against the recorded 57.6 = 51.9% of roof, which is a plausible and healthy fraction for a 2023 FasterTransformer production stack and is why no roofline finding is raised against it. The instrument working here is the reason the stale pointer is worth correcting: with a pointer the sweep could not have run this check at all.