obs-lmdeploy-metrics-doc · inference-research

Fields

id

sources/obs-lmdeploy-metrics-doc

URL

https://github.com/InternLM/lmdeploy/blob/main/docs/en/advance/metrics.md

Publisher

InternLM

Kind

repo

Accessed

2026-10-04

notes

Documents that lmdeploy:gpu_cache_usage_perc means different things per backend: on the PyTorch backend 'logical GPU blocks referenced by live requests' with prefix-cache-only and stopped-session blocks excluded, switchable via LMDEPLOY_ENABLE_REQUEST_CACHE_USAGE_METRIC=0; on Turbomind it is bytes of live prefix-cache and checkpoint objects over the configured cache-region size. Also: 'Turbomind currently reports scheduler metrics for DP 1; per-rank Turbomind metrics with DP greater than 1 are not yet supported.'

Notes

Documents that lmdeploy:gpu_cache_usage_perc means different things per backend: on the PyTorch backend 'logical GPU blocks referenced by live requests' with prefix-cache-only and stopped-session blocks excluded, switchable via LMDEPLOY_ENABLE_REQUEST_CACHE_USAGE_METRIC=0; on Turbomind it is bytes of live prefix-cache and checkpoint objects over the configured cache-region size. Also: 'Turbomind currently reports scheduler metrics for DP 1; per-rank Turbomind metrics with DP greater than 1 are not yet supported.'

Cross-references

This record references no other records.