TEI next-batch-size histogram
niche-text-embeddings-inference · histogram
Fields
id
metric-exposures/w5e-tei-te-batch-next-sizeMetric name
te_batch_next_size
Metric type
histogram
Unit
requests
Exposure
prometheus
Endpoint
:9000/metrics (--prometheus-port, default 9000)
Histogram buckets
1248163264128256512102420484096
Enabled by default
yes
notes
Matcher::Full("te_batch_next_size"), 13 bounds 2^0..2^12 (1 to 4096). A batching-efficiency signal, not a latency signal: sustained mass in the low buckets means the router is admitting small batches and the GPU is underfilled. EASY TO MISREAD as a load metric.
Notes
Matcher::Full("te_batch_next_size"), 13 bounds 2^0..2^12 (1 to 4096). A batching-efficiency signal, not a latency signal: sustained mass in the low buckets means the router is admitting small batches and the GPU is underfilled. EASY TO MISREAD as a load metric.