Vision encoder runs on separate schedule from decoder
[major] config
Fields
id
gotchas/vision-encoder-runs-on-separate-schedule-from-decoderClass
config
Severity
major
Concepts
vision-encoderdecoderschedulingcuda-streamprefetch
Symptom
The vision encoder and LLM decoder run on separate schedules, leading to underutilization. While the encoder is running, the decoder is idle, and vice versa.
Root cause
The vision encoder is a separate forward pass that must complete before the LLM prefill can start. Engines that do not overlap these phases leave the GPU underutilized.
Workaround
Use an engine with explicit encoder-decoder overlap. TRT-LLM has a Multimodal Encoder Side Stream that prefetches encoder work on a separate CUDA stream. This overlaps the vision encoder forward pass with the LLM's text prefill, improving utilization.
notes
ENCODER-DECODER OVERLAP IS CRITICAL FOR UTILIZATION: Without it, the vision encoder and LLM prefill run serially, leaving the GPU idle during each phase. TRT-LLM's Multimodal Encoder Side Stream prefetches encoder work on a separate CUDA stream, overlapping it with the LLM's text prefill. This is especially important for high-resolution images where the encoder cost is significant. The overlap can improve throughput by 20-30% for vision-language workloads. The mechanism is from TRT-LLM docs; the utilization analysis is derived from the vision encoder FLOP counts.
Notes
ENCODER-DECODER OVERLAP IS CRITICAL FOR UTILIZATION: Without it, the vision encoder and LLM prefill run serially, leaving the GPU idle during each phase. TRT-LLM's Multimodal Encoder Side Stream prefetches encoder work on a separate CUDA stream, overlapping it with the LLM's text prefill. This is especially important for high-resolution images where the encoder cost is significant. The overlap can improve throughput by 20-30% for vision-language workloads. The mechanism is from TRT-LLM docs; the utilization analysis is derived from the vision encoder FLOP counts.