inference/research
verifiedhigh · 0.93updated 2026-10-04

TGI: any KV-cache occupancy metric - VERIFIED ABSENT

text-generation-inference · unknown

Fields

id
metric-exposures/w3m-text-generation-inference-no-tgi-kv-cache-usage
Metric name
tgi_kv_cache_usage
Metric type
unknown
Exposure
unknown
Verified negatively
yes
notes
VERIFIED NEGATIVE. TGI does not emit any KV-cache occupancy metric. tgi_kv_cache_usage is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. No cache-usage number is exported, so a TGI deployment cannot be capacity-planned from KV occupancy. tgi_batch_current_max_tokens and tgi_batch_max_total_tokens are the nearest signals and both are batch-shape numbers rather than cache occupancy - they describe what the batch is doing, not how full the cache is. What was searched: case-insensitive grep for 'cache_usage' and 'kv_cache_usage' returns 0 hits in router/src/server.rs, backends/v3/src/backend.rs and backends/v3/src/block_allocator.rs; block_allocator.rs contains no metric calls at all.. Do not read this record's metric_name as something the engine exports; it is the name that is missing.

Notes

VERIFIED NEGATIVE. TGI does not emit any KV-cache occupancy metric. tgi_kv_cache_usage is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. No cache-usage number is exported, so a TGI deployment cannot be capacity-planned from KV occupancy. tgi_batch_current_max_tokens and tgi_batch_max_total_tokens are the nearest signals and both are batch-shape numbers rather than cache occupancy - they describe what the batch is doing, not how full the cache is. What was searched: case-insensitive grep for 'cache_usage' and 'kv_cache_usage' returns 0 hits in router/src/server.rs, backends/v3/src/backend.rs and backends/v3/src/block_allocator.rs; block_allocator.rs contains no metric calls at all.. Do not read this record's metric_name as something the engine exports; it is the name that is missing.

Cross-references