TGI backend queue implementation (unbounded mpsc channel into the batching task)
Fields
sources/ops-tgi-queue-unboundedNotes
Queue::new() creates mpsc::unbounded_channel() and spawns queue_task; append() sends QueueCommand::Append(Box::new(entry), ...) and unwraps the result. The queue state is a VecDeque::with_capacity(128) of (u64, Entry), each Entry carrying an mpsc::UnboundedSender<Result<InferStreamResponse, InferError>> plus queue_time and batch_time instants. next_batch() is asked for a batch under (min_size, max_size, prefill_token_budget, token_budget) - the token budget is enforced at BATCH FORMATION time by the batching task, not at admission. metrics::gauge!("tgi_queue_size").set(state.entries.len()) is emitted on each NextBatch. Operationally: nothing in this file bounds the number of accepted waiting entries, so the only admission bound on the TGI path is the router's concurrency semaphore.
Cross-references
This record references no other records.