Tetrate - Envoy AI Gateway and Tetrate Agent Router performance benchmarks (2026 living reference)

Last updated July 2026. The page is explicitly a curated summary of OTHER people's benchmarks, which is its value and its bias: 'Where we cite a number, we link the methodology so you can reproduce it' and 'Our consistent recommendation: benchmark any gateway under your own traffic and policy load before deciding.' Four data points recorded. (1) MCP tool-call overhead, Dec 2025: Envoy AI Gateway added 'roughly 160-390ms over direct, unproxied calls to the upstream MCP server - comparable to other cloud-native solutions tested', average difference vs a competing implementation ~0.2 ms; method was a Go benchmark against an echo tool with 100 key-derivation iterations, and Tetrate argues it is negligible because 'LLM reasoning typically takes several seconds'. (2) Control-plane scaling, Feb 2026: tested to 2,000 AIGatewayRoutes with 'consistent route-readiness latency, linear and predictable CPU/memory growth, and zero routing failures'; REQUIRES TUNING - 'As route count grew, the xDS configuration payload between the Envoy Gateway control plane and the AI Gateway extension server exceeded the default 4MB gRPC message size. Raising it to 25MB ... resolved it cleanly'; harness used a Mock Cassette Server replaying static responses 'so anyone can reproduce it without incurring provider API bills'; runs were WITHOUT headerMutation, so 'the effective route ceiling depends on per-route configuration complexity and the ~1MB Kubernetes object size limit'. (3) Documented LiteLLM behaviour under sustained load (NOT a Tetrate run, attributed to their own source): 'LiteLLM begins to bottleneck beyond approximately 300 RPS, with latency degrading from around 200ms to over 12 seconds under load - and adding instances did not resolve it. Under sustained traffic, LiteLLM containers have been reported to grow to 12GB and crash.' Architectural contrast claimed: Envoy-based, 'Logging stays off the critical request path (written asynchronously), and policies and pricing are cached in memory so requests do not block on database calls.' (4) Data-plane latency, July 2026, independent Broadcom/VMware Cloud Foundation validation: 'roughly 2 milliseconds of gateway overhead - about 0.01% of end-to-end latency - under sustained enterprise LLM load', flat at peak saturation; three-hour endurance at 190 concurrent users, average TTFT 0.103 s, ~0.035 s per output token, saturated at 224 concurrent users where TTFT climbed sharply - 'a GPU compute ceiling on the four-H100 test cluster, not a gateway limit', KV cache preemption stayed at zero. OPERATOR NOTE that matters for k8s: 'Early runs showed multi-second apparent latency from a Linux CFS throttling trap when Envoy worker threads exceeded the container CPU quota. Pinning Envoy --concurrency to allocated CPU limits eliminated that overhead.' Their framing table: model generation seconds, TTFT ~100 ms, gateway overhead ~2 ms. Competitor benchmarks quoted from vendor sources with caveats: Kong (mock backends, default configs, large claimed advantages over Portkey and LiteLLM), Bifrost/Maxim (~11 microseconds at 5,000 RPS under Maxim's conditions), and the general warning 'A gateway's overhead with auth, attribution, guardrails, and rate limiting active is the number that matters for production - and it is rarely the number in a headline.'