Furiosa RNGD single card: GPT-J 6B about 12 queries/s (Hot Chips 2024, pre-optimization)
12 queries/s (MLPerf-style, single card, not tokens/s)
Fields
id
benchmarks/furiosa-rngd-gptj-6b-12qps-hotchips-2024Metric
tps_aggregate
Value
12
Unit
queries/s (MLPerf-style, single card, not tokens/s)
Model
GPT-J 6B
Methodology
VENDOR, approximate. Furiosa Hot Chips 2024 blog: a single RNGD was generating about 12 queries per second on GPT-J 6B at the announcement, with the caveat that the number should rise as the compiler is refined. The comparison table on the same page is an image captioned unverified by MLPerf and was not transcribed. Not a per-request decode tok/s. Batch, prompt length, and output length follow the MLPerf GPT-J scenario only by implication and are not stated in the prose.
Measured by
vendor
Reproducible
no
Mean power (W)
180
Power scope
device
notes
ENGINE_ID NOTE: pinned to vllm, the closest existing engine; the 2024 blog does not name the runtime. 12 is queries per second, not tokens per second, and the unit string says so. Draft because the figure is about 12 and the scenario lengths are not in the prose. A later Furiosa MLPerf doc (developer center 2025.2.0) shows how to run gpt-j-offline on one RNGD but publishes no result number. power_w is the card TDP printed on the Hot Chips blog, not a measured draw during the 12 queries/s run.
Notes
ENGINE_ID NOTE: pinned to vllm, the closest existing engine; the 2024 blog does not name the runtime. 12 is queries per second, not tokens per second, and the unit string says so. Draft because the figure is about 12 and the scenario lengths are not in the prose. A later Furiosa MLPerf doc (developer center 2025.2.0) shows how to run gpt-j-offline on one RNGD but publishes no result number. power_w is the card TDP printed on the Hot Chips blog, not a measured draw during the 12 queries/s run.