inference/research
verifiedhigh · 0.95updated 2026-10-04

obs-sglang-schedule-batch-retract

repo

Fields

id
sources/obs-sglang-schedule-batch-retract
Publisher
sgl-project/sglang
Kind
repo
Accessed
2026-10-04
notes
release_req(): release_kv_cache(req, tree_cache, is_insert=False) - 'don't insert into the tree because we need the space instantly' - then evict_from_tree_cache for the freed budget, then req.reset_for_retract() which increments retraction_count and zeroes num_matched_prefix_tokens. This is the recompute-on-resume path that makes retraction expensive.

Notes

release_req(): release_kv_cache(req, tree_cache, is_insert=False) - 'don't insert into the tree because we need the space instantly' - then evict_from_tree_cache for the freed budget, then req.reset_for_retract() which increments retraction_count and zeroes num_matched_prefix_tokens. This is the recompute-on-resume path that makes retraction expensive.

Cross-references

This record references no other records.