TTFT, MPT-7B, 512-token prompt, batch size 1, 1x A100 40GB

Databricks Table 1, 'Time to first token (ms)', row MPT-7B, column '1xA100-40GB': 46 ms. Request length 512 input tokens, batch size 1. Backend is Databricks' TensorRT-LLM-based inference server - NOT the FasterTransformer stack used in their Table 2 throughput sweep, so this record is not directly comparable to the batch-size sweep records; it is here to be compared against the per-user decode figure at the same batch size and prompt length. Tensor-parallel size 1, single GPU. No 40GB A100 accelerator record exists in this repo, so accelerator_ids is empty rather than pointing at the 80GB parts, which are a different part.

TTFT IS NOT A SPEED METRIC AND IS NOT COMPARABLE TO INTER-TOKEN LATENCY. At batch size 1 with a 512-token prompt, this MPT-7B model takes 46 ms to emit its first token and about 17.4 ms per token after that (derived from mpt7b-a100-bs1-per-user-decode-tps). TTFT is therefore about 2.6x a single inter-token interval, and for a 64-token response the entire prefill cost is roughly 43 percent of total latency. Three consequences. (1) A model can have a superb tok/s number and an awful TTFT, because TTFT is dominated by prefill, which is compute-bound and scales with PROMPT length, whereas decode is memory-bandwidth-bound and scales with batch. (2) The same source reports that adding 512 INPUT tokens costs more latency than producing 8 additional OUTPUT tokens on MPT models, so a tok/s figure quoted without prompt length omits the dominant term for interactive workloads. (3) Response latency is TTFT + TPOT x output tokens, so a per-user tok/s number and a TTFT number describe different halves of the same request and cannot be substituted for one another - which is exactly the mix-up this record exists to prevent. Table 1 also shows TTFT is the metric tensor parallelism actually helps: MPT-7B goes 46 -> 34 -> 26 ms across 1x/2x/4x A100, while Llama2-70B goes 154 -> 114 ms going 4x -> 8x, only 0.74x, because communication overhead and lower model-bandwidth-utilization eat the gain. Same batch size and prompt length as the per-user decode record, which is the only reason these two are comparable at all. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: MLPerf-style server scenario. ABSENT CONDITIONS (2 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - engine version: unrecoverable from bench-meth-databricks-inference-tuning - engine_id=fastertransformer is named but no build/commit/container digest is published - warmup: unrecoverable from bench-meth-databricks-inference-tuning - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted.