Hugging Face text-generation-launcher argument reference (MAX_TOTAL_TOKENS, MAX_CONCURRENT_REQUESTS, MAX_BATCH_TOTAL_TOKENS, MAX_WAITING_TOKENS)

Verbatim: --max-concurrent-requests 'The maximum amount of concurrent requests for this particular deployment. Having a low limit will refuse clients requests instead of having them wait for too long and is usually good to handle backpressure correctly', default 128. --max-total-tokens 'This is the most important value to set as it defines the "memory budget" of running clients requests. Clients will send input sequences and ask to generate max_new_tokens on top. With a value of 1512 users can send either a prompt of 1000 and ask for 512 new tokens, or send a prompt of 1 and ask for 1511 max_new_tokens. The larger this value, the larger amount each request will be in your RAM and the less effective batching can be.', default min(max_position_embeddings, 4096) - note this is a PER-REQUEST budget, not a fleet total. --max-batch-total-tokens is flagged IMPORTANT and 'should be the largest possible amount that fits the remaining memory (after the model is loaded). Since the actual memory overhead depends on other parameters like if you are using quantization, flash attention or the model implementation, text-generation-inference cannot infer this number automatically.' --max-waiting-tokens 'defines how many tokens can be passed before forcing the waiting queries to be put on the batch', with 'With a value too small, queries will always steal the compute to run prefill and running queries will be delayed by a lot. With a value too big, waiting queries could wait for a very long time before be [ing served].' --waiting-served-ratio default 0.3, the ratio of waiting to running queries at which the router pauses running queries to admit waiting ones, 'only applied if there is room in the batch as defined by max_batch_total_tokens'.