TEI inference-duration histogram
niche-text-embeddings-inference · histogram
Fields
id
metric-exposures/w5e-tei-te-request-inference-durationMetric name
te_request_inference_duration
Metric type
histogram
Unit
seconds
Exposure
prometheus
Endpoint
:9000/metrics (--prometheus-port, default 9000)
Histogram buckets
0.00002250.000033750.0000506250000000000040.00007593750.000113906250000000010.0001708593750.00025628906250.000384433593750.00057665039062500010.00086497558593750.00129746337890625010.0019461950683593750.00291929260253906250.0043789389038085940.0065684083557128910.0098526125335693360.0147789188003540040.0221683782005310070.033252567300796510.0498788509511947640.074818276426792150.112227414640188220.168341121960282340.25251168294042350.378767524410635270.56815128661595280.85222692992392931.2783403948858941.9175105923288412.87626588849326134.3143988327398926.4715982491098389.70739737366475614.56109606049713521.841644090745703
Enabled by default
yes
Phase scoped
yes
notes
router/src/lib.rs: metrics::histogram!("te_request_inference_duration").record(self.inference_time.as_secs_f64()). Same Suffix("duration") ladder as every other duration histogram. Because it is separated from queue time, TEI can attribute a latency regression to admission pressure versus model work - which neither vLLM nor SGLang can do from a single request-duration histogram alone.
Notes
router/src/lib.rs: metrics::histogram!("te_request_inference_duration").record(self.inference_time.as_secs_f64()). Same Suffix("duration") ladder as every other duration histogram. Because it is separated from queue time, TEI can attribute a latency regression to admission pressure versus model work - which neither vLLM nor SGLang can do from a single request-duration histogram alone.