XiongjieDai/GPU-Benchmarks-on-LLM-Inference — llama.cpp on NVIDIA A100 PCIe 80GB et al. (RunPod, LLaMA-3 8B/70B Q4_K_M + F16, 1024-token generate + prompt-eval)

Fields

id

sources/xiongjiedai-gpu-benchmarks-llm-inference-a100-pcie

URL

https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

Publisher

Xiongjie Dai (RunPod rental)

Kind

repo

Accessed

2026-10-14

notes

llama.cpp built with cuBLAS (make LLAMA_CUBLAS=1, era ~May 2024 for the runpod table). './main -ngl 10000' full offload, -n 1024 generation and a 1024-token prompt; README tables report average tokens/s. Per-request batch-1, NOT aggregate. Multi-GPU rows marked '* N'; single-GPU A100 PCIe 80GB, A100 SXM 80GB, H100 PCIe, L40S, RTX 6000 Ada, A40 rows all present. F16 uses fp16 weights (larger VRAM).

Notes

llama.cpp built with cuBLAS (make LLAMA_CUBLAS=1, era ~May 2024 for the runpod table). './main -ngl 10000' full offload, -n 1024 generation and a 1024-token prompt; README tables report average tokens/s. Per-request batch-1, NOT aggregate. Multi-GPU rows marked '* N'; single-GPU A100 PCIe 80GB, A100 SXM 80GB, H100 PCIe, L40S, RTX 6000 Ada, A40 rows all present. F16 uses fp16 weights (larger VRAM).

Cross-references

This record references no other records.