r/LocalLLaMA 'llama.cpp RPC Performance' - one user pools 56GB of VRAM across three machines and reports 4-5 tok/s decode
forum
Fields
id
sources/comm-reddit-localllama-1hapq7e-llamacpp-rpc-perfPublisher
r/LocalLLaMA
Kind
forum
Accessed
2026-10-04
notes
OP u/RazzmatazzReal4129. Hardware: one machine with an RTX 4090 plus two other machines each with a 4060 Ti; 56GB VRAM total across the three. Model Llama 3.3 70B Q4_K_M. Reports two requests: prompt eval 145.97 tok/s then 37.47 tok/s, decode 4.38 tok/s then 3.98 tok/s, at n_ctx_slot 8192. Posts full llama-server slot timing logs, which is what makes this reproducible-ish rather than a bare claim. Self-described as a first attempt: 'I haven't found much online as I was trying to set this up.' SINGLE REPORTER, and the network topology between the three machines is not stated - so the 4-5 tok/s must NOT be read as an RPC-over-ethernet ceiling.
Notes
OP u/RazzmatazzReal4129. Hardware: one machine with an RTX 4090 plus two other machines each with a 4060 Ti; 56GB VRAM total across the three. Model Llama 3.3 70B Q4_K_M. Reports two requests: prompt eval 145.97 tok/s then 37.47 tok/s, decode 4.38 tok/s then 3.98 tok/s, at n_ctx_slot 8192. Posts full llama-server slot timing logs, which is what makes this reproducible-ish rather than a bare claim. Self-described as a first attempt: 'I haven't found much online as I was trying to set this up.' SINGLE REPORTER, and the network topology between the three machines is not stated - so the 4-5 tok/s must NOT be read as an RPC-over-ethernet ceiling.
Cross-references
This record references no other records.