inference/research
verifiedhigh · 0.90updated 2026-10-05

Envoy AI Gateway time-per-output-token histogram (OTel gen_ai semconv)

envoy-ai-gateway · histogram

Fields

id
metric-exposures/w5e-envoy-ai-gateway-gen-ai-server-time-per-output-token
Metric name
gen_ai.server.time_per_output_token
Metric type
histogram
Unit
seconds
Exposure
prometheus
Endpoint
:8080/metrics (Prometheus reader always attached)
Histogram buckets
0.010.0250.050.0750.10.150.20.30.40.50.7512.5
Labels
gen_ai.operation.namegen_ai.provider.namegen_ai.request.modelgen_ai.response.modelgen_ai.original.modelgen_ai.backend
Phase scoped
yes
notes
mustRegisterHistogram with WithDescription('Time per output token generated after the first token for successful responses.') - the 'after the first token' and 'for successful responses' qualifiers are load-bearing. The first token is excluded, so this is not simply request_duration/n_tokens, and FAILED requests contribute nothing, so a regression that manifests as errors rather than slowness does not move this series at all. 13 explicit bounds, tight around interactive decode rates. Envoy AI Gateway main (envoyproxy/ai-gateway), read 2026-10-05. Metrics go through an OpenTelemetry MeterProvider (internal/metrics/metrics.go NewMeterFromEnv), not a bare prometheus_client registry: a Prometheus reader is ALWAYS attached, and a console or OTLP exporter is added additionally when OTEL_METRICS_EXPORTER is 'console', or when it is neither 'none' nor 'prometheus' and an OTLP endpoint env var is set. OTEL_SDK_DISABLED=true turns off the extra exporters but not the Prometheus reader. That means the metric names are dotted OTel names (gen_ai.server.*), which the Prometheus exporter renders with dots replaced by underscores and a unit suffix appended - so the scraped series is NOT the literal constant in genai.go. Names follow the OpenTelemetry GenAI semantic conventions, not any engine's private scheme.

Notes

mustRegisterHistogram with WithDescription('Time per output token generated after the first token for successful responses.') - the 'after the first token' and 'for successful responses' qualifiers are load-bearing. The first token is excluded, so this is not simply request_duration/n_tokens, and FAILED requests contribute nothing, so a regression that manifests as errors rather than slowness does not move this series at all. 13 explicit bounds, tight around interactive decode rates. Envoy AI Gateway main (envoyproxy/ai-gateway), read 2026-10-05. Metrics go through an OpenTelemetry MeterProvider (internal/metrics/metrics.go NewMeterFromEnv), not a bare prometheus_client registry: a Prometheus reader is ALWAYS attached, and a console or OTLP exporter is added additionally when OTEL_METRICS_EXPORTER is 'console', or when it is neither 'none' nor 'prometheus' and an OTLP endpoint env var is set. OTEL_SDK_DISABLED=true turns off the extra exporters but not the Prometheus reader. That means the metric names are dotted OTel names (gen_ai.server.*), which the Prometheus exporter renders with dots replaced by underscores and a unit suffix appended - so the scraped series is NOT the literal constant in genai.go. Names follow the OpenTelemetry GenAI semantic conventions, not any engine's private scheme.

Cross-references