inference/research
verifiedhigh · 0.93updated 2026-10-04

Triton Inference Server: a KV-cache occupancy metric - VERIFIED ABSENT

triton-inference-server · unknown

Fields

id
metric-exposures/w3m-triton-inference-server-no-nv-inference-kv-cache-usage
Metric name
nv_inference_kv_cache_usage
Metric type
unknown
Exposure
unknown
Verified negatively
yes
notes
VERIFIED NEGATIVE. Triton Inference Server does not emit a KV-cache occupancy metric. nv_inference_kv_cache_usage is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. Triton core has no model-internal state metrics at all - it measures the server, not the model. KV occupancy exists only if the backend registers it, and the TRT-LLM backend registers it as nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type="fraction"|"used"|"free"|"max"} - you filter a label rather than name a metric. What was searched: the docs' metric tables contain no cache-occupancy family; backends/v3-style per-model state is explicitly delegated to the backend custom metrics API (TRITONSERVER_MetricFamily*).. Do not read this record's metric_name as something the engine exports; it is the name that is missing.

Notes

VERIFIED NEGATIVE. Triton Inference Server does not emit a KV-cache occupancy metric. nv_inference_kv_cache_usage is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. Triton core has no model-internal state metrics at all - it measures the server, not the model. KV occupancy exists only if the backend registers it, and the TRT-LLM backend registers it as nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type="fraction"|"used"|"free"|"max"} - you filter a label rather than name a metric. What was searched: the docs' metric tables contain no cache-occupancy family; backends/v3-style per-model state is explicitly delegated to the backend custom metrics API (TRITONSERVER_MetricFamily*).. Do not read this record's metric_name as something the engine exports; it is the name that is missing.

Cross-references