Jetson AGX Thor SGLang: Qwen3.6-35B-A3B FP8 decode (Multimodalflow)
14.6 tok/s (generation, per-request)
Fields
id
benchmarks/thor-sglang-qwen36-35b-fp8-decodeMetric
decode_tok_s
Value
14.6
Unit
tok/s (generation, per-request)
Model
Qwen3.6-35B-A3B-FP8
Methodology
Multimodalflow live-hardware post dated 2026-06-01 on a Jetson AGX Thor Developer Kit. CUDA 13.0, compute capability 11.0, unified memory reported as 125,771 MiB (~123 GB), kernel 6.8.12-tegra. Not the MLPerf agentic-edge submission already in this repo. SGLang server (`python3 -m sglang.launch_server`, version not printed) serving the FP8 checkpoint fully in unified memory (~36 GB footprint, ~101 GB system RAM with model loaded). Short-prompt case: 24 prompt tokens, 200 output tokens, 13.69 s, 3-run average 14.6 t/s. Long-prompt case (268 in / 300 out) measured 14.7 t/s and is not a separate record. Per request, concurrency 1, not an aggregate. FP8 variant (e4m3 vs e5m2) is not stated; fp8-e4m3 is the common inference default and is flagged as an assumption in notes.
Measured by
third_party
Reproducible
yes
notes
Author states numbers come from curl against a live server, three-run average. Blog, not a lab with a published raw log, so confidence stays at 0.55. format_id fp8-e4m3 is an assumption: the post only says FP8. SGLang version and batch size are not printed. Unified memory, so no PCIe interconnect id.
Notes
Author states numbers come from curl against a live server, three-run average. Blog, not a lab with a published raw log, so confidence stays at 0.55. format_id fp8-e4m3 is an assumption: the post only says FP8. SGLang version and batch size are not printed. Unified memory, so no PCIe interconnect id.