obs-tgi-router-server-rs · inference-research

Fields

id

sources/obs-tgi-router-server-rs

URL

https://github.com/huggingface/text-generation-inference/blob/main/router/src/server.rs

Publisher

huggingface

Kind

repo

Accessed

2026-10-04

notes

Where the TGI metrics are actually emitted (the Rust router). Line ~363-422 is the canonical path: total_time = start.elapsed(); validation_time = response.queued - start; queue_time = response.start - response.queued; inference_time = now - response.start; time_per_token = inference_time / generated_text.generated_tokens. There is no TTFT anywhere in this file - verified by grep returning zero matches for time_to_first.

Notes

Where the TGI metrics are actually emitted (the Rust router). Line ~363-422 is the canonical path: total_time = start.elapsed(); validation_time = response.queued - start; queue_time = response.start - response.queued; inference_time = now - response.start; time_per_token = inference_time / generated_text.generated_tokens. There is no TTFT anywhere in this file - verified by grep returning zero matches for time_to_first.

Cross-references

This record references no other records.