Long-Context Benchmarks: Who's Actually Fast at 128K Tokens? (GeneralCompute)

TTFT and decode speed at 128K input context, batch size 1 (single request), 512 output tokens, 3 runs per model with median reported. Models loaded at native weight precision. Hardware: GeneralCompute inference infrastructure (specific GPUs not stated). Engine: not explicitly stated (likely vLLM or SGLang). Cache state: cold (no prefix caching mentioned). Models tested: Qwen2.5 7B, Llama 4 Scout (~17B active MoE), Gemma 3 27B, DeepSeek V3 (~37B active MoE), Qwen2.5 72B, Mistral Large 123B.