inference/research
verifiedhigh · 0.85updated 2026-10-05

EAGLE-3 throughput improvement in SGLang at batch size 64

1.38 value 1.38 with metric speedup_ratio: 1.38x the BASELINE throughput at batch size 64 in SGLang, NOT 1.38 output tokens/s. BASELINE: the same SGLang build without EAGLE-3, batch 64, draft tree depth 8.

Fields

id
benchmarks/sd-eagle3-throughput-sglang-batch64
Metric
speedup_ratio
Value
1.38
Unit
value 1.38 with metric speedup_ratio: 1.38x the BASELINE throughput at batch size 64 in SGLang, NOT 1.38 output tokens/s. BASELINE: the same SGLang build without EAGLE-3, batch 64, draft tree depth 8.
Model
LLaMA-Instruct 3.1 8B
Methodology
SGLang framework, batch size 64. EAGLE-3 with draft tree depth 8. Throughput improvement 1.38x over baseline.
Measured by
self
Reproducible
yes
notes
This is a THROUGHPUT measurement at moderate batch size, not a latency measurement at batch 1. The 1.38x at batch 64 should be compared to EAGLE-2's 3-6.5x at batch 1 - the dramatic difference illustrates the batch dilution problem. EAGLE-3 achieves up to 6.5x speedup at batch 1. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: NOT STATED by the source - absent, not inferred. A throughput figure with no scenario cannot be placed against an MLPerf Offline or Server number (they are different measurements; on identical hardware the Offline-minus-Server difference is negative in 5 of 9 systems measured). ABSENT CONDITIONS (5 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - hardware: unrecoverable from sd-eagle3-paper - no accelerator model is named anywhere in the source - engine version: unrecoverable from sd-eagle3-paper - engine_id=sglang is named but no build/commit/container digest is published - input length: unrecoverable from sd-eagle3-paper - would need the run's ISL or the prepared dataset's input-length profile - output length: unrecoverable from sd-eagle3-paper - would need the run's OSL or the harness's configured max_tokens - warmup: unrecoverable from sd-eagle3-paper - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted. UNIT/METRIC DIRECTION CORRECTION 2026-10-04 (benchmark throughput + latency audit). The stored value is UNCHANGED; what was wrong is the `metric` enum, which labelled this record as a raw tokens/s measurement when the number is a dimensionless RATIO. Stored value is 1.38 with unit `speedup_ratio`: 1.38x the BASELINE throughput at batch size 64, not 1.38 output tokens/s. The record names its own baseline, and the number is a correct ratio of two throughputs - only its `metric` key misfiles it. This is the error docs/07-benchmarking-methodology.md section 8 warns about from the other side: a reader filtering data/benchmarks/ by `metric: decode_tok_s` expects tokens per second, so a speedup_ratio filed under that key is not a wrong number, it is a number of the wrong KIND in the one field a capacity planner is most likely to filter on. CORRECT ENUM MEMBER: none of decode_tok_s / prefill_tok_s / tps_aggregate / tok_s_per_user describes a ratio, and schemas/benchmark.schema.json is out of scope for this audit so the record is left with an explicit `unit` and this note rather than silently reclassified. Anyone selecting benchmarks by metric MUST additionally check `unit` until a speedup_ratio member exists in the enum. METRIC CORRECTION 2026-10-05 (schema authority pass, agent w5-schema). The stored VALUE is UNCHANGED. What changed is `metric`, which labelled a DIMENSIONLESS RATIO as a dimensioned rate. schemas/benchmark.schema.json now has a speedup_ratio member, so the number no longer has to borrow a rate key it does not belong under, and every filter on data/benchmarks/ that selects tok/s no longer returns a ratio. Value and baseline unchanged; only the kind of quantity is now stated by the field that means it. BREAK-EVEN CHECK 2026-10-05 (agent w6-benchmarks): VALUE UNCHANGED, and the claimed 1.38x is a REAL win rather than a regression dressed as one - which the existing notes assume but did not demonstrate. Checked against A_be = 1 + P_d/(B x P_t) in the compute-bound regime that batch 64 sits in: with a same-size draft on an 8B target, A_be = 1 + 1/64 = 1.0156, so 1.38 clears break-even by 36%. The memory-bound formula A_be = (1 + P_d/P_t)/k does not apply here, and using it would have been the error: it is B-independent by construction and would have reported this batch-64 number against a k-dependent threshold that has nothing to do with batch 64. This is the record that makes the batch-dilution claim in its notes checkable, and it checks out. The instrument was validated first against the repo's own anchor - (1 + 1e9/68.98e9)/4 = 0.2536 at k=4 - before being applied. Full sweep including why the ten acceptance records cannot be checked at all: data/gotchas/sd-acceptance-length-is-not-a-quality-score-and-makes-break-even-uncheckable.json.

Notes

This is a THROUGHPUT measurement at moderate batch size, not a latency measurement at batch 1. The 1.38x at batch 64 should be compared to EAGLE-2's 3-6.5x at batch 1 - the dramatic difference illustrates the batch dilution problem. EAGLE-3 achieves up to 6.5x speedup at batch 1.

LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: NOT STATED by the source - absent, not inferred. A throughput figure with no scenario cannot be placed against an MLPerf Offline or Server number (they are different measurements; on identical hardware the Offline-minus-Server difference is negative in 5 of 9 systems measured). ABSENT CONDITIONS (5 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - hardware: unrecoverable from sd-eagle3-paper - no accelerator model is named anywhere in the source - engine version: unrecoverable from sd-eagle3-paper - engine_id=sglang is named but no build/commit/container digest is published - input length: unrecoverable from sd-eagle3-paper - would need the run's ISL or the prepared dataset's input-length profile - output length: unrecoverable from sd-eagle3-paper - would need the run's OSL or the harness's configured max_tokens - warmup: unrecoverable from sd-eagle3-paper - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted.

UNIT/METRIC DIRECTION CORRECTION 2026-10-04 (benchmark throughput + latency audit). The stored value is UNCHANGED; what was wrong is the `metric` enum, which labelled this record as a raw tokens/s measurement when the number is a dimensionless RATIO. Stored value is 1.38 with unit `speedup_ratio`: 1.38x the BASELINE throughput at batch size 64, not 1.38 output tokens/s. The record names its own baseline, and the number is a correct ratio of two throughputs - only its `metric` key misfiles it. This is the error docs/07-benchmarking-methodology.md section 8 warns about from the other side: a reader filtering data/benchmarks/ by `metric: decode_tok_s` expects tokens per second, so a speedup_ratio filed under that key is not a wrong number, it is a number of the wrong KIND in the one field a capacity planner is most likely to filter on. CORRECT ENUM MEMBER: none of decode_tok_s / prefill_tok_s / tps_aggregate / tok_s_per_user describes a ratio, and schemas/benchmark.schema.json is out of scope for this audit so the record is left with an explicit `unit` and this note rather than silently reclassified. Anyone selecting benchmarks by metric MUST additionally check `unit` until a speedup_ratio member exists in the enum.

METRIC CORRECTION 2026-10-05 (schema authority pass, agent w5-schema). The stored VALUE is UNCHANGED. What changed is `metric`, which labelled a DIMENSIONLESS RATIO as a dimensioned rate. schemas/benchmark.schema.json now has a speedup_ratio member, so the number no longer has to borrow a rate key it does not belong under, and every filter on data/benchmarks/ that selects tok/s no longer returns a ratio. Value and baseline unchanged; only the kind of quantity is now stated by the field that means it.

BREAK-EVEN CHECK 2026-10-05 (agent w6-benchmarks): VALUE UNCHANGED, and the claimed 1.38x is a REAL win rather than a regression dressed as one - which the existing notes assume but did not demonstrate. Checked against A_be = 1 + P_d/(B x P_t) in the compute-bound regime that batch 64 sits in: with a same-size draft on an 8B target, A_be = 1 + 1/64 = 1.0156, so 1.38 clears break-even by 36%. The memory-bound formula A_be = (1 + P_d/P_t)/k does not apply here, and using it would have been the error: it is B-independent by construction and would have reported this batch-64 number against a k-dependent threshold that has nothing to do with batch 64. This is the record that makes the batch-dilution claim in its notes checkable, and it checks out. The instrument was validated first against the repo's own anchor - (1 + 1e9/68.98e9)/4 = 0.2536 at k=4 - before being applied. Full sweep including why the ten acceptance records cannot be checked at all: data/gotchas/sd-acceptance-length-is-not-a-quality-score-and-makes-break-even-uncheckable.json.

Cross-references

Engine (1)