inference/research
verifiedmedium · 0.80majorupdated 2026-10-04

On a streaming endpoint an output filter that inspects completed text cannot un-send the tokens it already streamed

[major] security

Fields

id
gotchas/sec-streaming-output-scan-cannot-unretract
Class
security
Severity
major
Concepts
streamingoutput-filteringmoderationttftdetokenizer
Symptom
You add a moderation check on the assistant turn. The compliance test passes because the test client is non-streaming. In production the first sentence of the offending content is already delivered before the check has completed, and there is no way to take it back over the stream.
Root cause
Output scanning is a post-hoc decision on text that already exists. On a streaming HTTP endpoint the token stream is delivered incrementally, so the check necessarily runs after some prefix has been transmitted. Buffering the whole completion before release makes the scan sound and reinstates the whole time-to-first-token cost of non-streaming generation; incremental sentence-level checks shrink the exposed prefix but cannot eliminate it, and a payload harmful only in aggregate defeats sentence-level granularity.
Workaround
Decide explicitly, per endpoint, which of three contracts you are offering: (a) buffered - scan then release, costs TTFT, the only sound version; (b) streaming with a stated exposure prefix - document how much text can escape before the check fires; (c) streaming unfiltered, with the filter applied only to the next turn or to logging. Silently choosing (c) because the framework streams by default is the common failure.
notes
Residual risk for (b) and (c) is the same and it is not zero: the exposure window is a function of generation speed, so a faster server is a larger breach. This interacts with the speculative-decoding records elsewhere in this repo - higher decode throughput widens the window. SCHEME RECLASSIFIED (2026-10-04): was 'framework'.

Notes

Residual risk for (b) and (c) is the same and it is not zero: the exposure window is a function of generation speed, so a faster server is a larger breach. This interacts with the speculative-decoding records elsewhere in this repo - higher decode throughput widens the window. SCHEME RECLASSIFIED (2026-10-04): was 'framework'.

Cross-references