inference/research
verifiedhigh · 0.90updated 2026-10-05

Envoy AI Gateway server request duration histogram (OTel gen_ai semconv)

envoy-ai-gateway · histogram

Fields

id
metric-exposures/w5e-envoy-ai-gateway-gen-ai-server-request-duration
Metric name
gen_ai.server.request.duration
Metric type
histogram
Unit
seconds
Exposure
prometheus
Endpoint
:8080/metrics (Prometheus reader always attached)
Histogram buckets
0.010.020.040.080.160.320.641.282.565.1210.2420.4840.9681.92
Labels
gen_ai.operation.namegen_ai.provider.namegen_ai.request.modelgen_ai.response.modelgen_ai.original.modelgen_ai.backend
Phase scoped
no
notes
WithDescription('Generative AI server request duration such as time-to-last byte or last output token.') - note the gateway is a PROXY, so this measures the full round trip including the upstream model's own latency, and it is measured at the gateway rather than inside any engine. 14 explicit bounds in a strict doubling series from 10 ms to ~82 s; the ladder stops at 81.92 s, so a percentile above that is not computable from this histogram. Envoy AI Gateway main (envoyproxy/ai-gateway), read 2026-10-05. Metrics go through an OpenTelemetry MeterProvider (internal/metrics/metrics.go NewMeterFromEnv), not a bare prometheus_client registry: a Prometheus reader is ALWAYS attached, and a console or OTLP exporter is added additionally when OTEL_METRICS_EXPORTER is 'console', or when it is neither 'none' nor 'prometheus' and an OTLP endpoint env var is set. OTEL_SDK_DISABLED=true turns off the extra exporters but not the Prometheus reader. That means the metric names are dotted OTel names (gen_ai.server.*), which the Prometheus exporter renders with dots replaced by underscores and a unit suffix appended - so the scraped series is NOT the literal constant in genai.go. Names follow the OpenTelemetry GenAI semantic conventions, not any engine's private scheme.

Notes

WithDescription('Generative AI server request duration such as time-to-last byte or last output token.') - note the gateway is a PROXY, so this measures the full round trip including the upstream model's own latency, and it is measured at the gateway rather than inside any engine. 14 explicit bounds in a strict doubling series from 10 ms to ~82 s; the ladder stops at 81.92 s, so a percentile above that is not computable from this histogram. Envoy AI Gateway main (envoyproxy/ai-gateway), read 2026-10-05. Metrics go through an OpenTelemetry MeterProvider (internal/metrics/metrics.go NewMeterFromEnv), not a bare prometheus_client registry: a Prometheus reader is ALWAYS attached, and a console or OTLP exporter is added additionally when OTEL_METRICS_EXPORTER is 'console', or when it is neither 'none' nor 'prometheus' and an OTLP endpoint env var is set. OTEL_SDK_DISABLED=true turns off the extra exporters but not the Prometheus reader. That means the metric names are dotted OTel names (gen_ai.server.*), which the Prometheus exporter renders with dots replaced by underscores and a unit suffix appended - so the scraped series is NOT the literal constant in genai.go. Names follow the OpenTelemetry GenAI semantic conventions, not any engine's private scheme.

Cross-references