vLLM Hardware Plugin for Intel Gaudi - GitHub README

vLLM plugin for Intel Gaudi. Key facts: (1) The HabanaAI/vllm-fork reached EOL at v1.23.0 and is deprecated in v1.24.0; the vllm-gaudi plugin is production-ready in v1.23.0 and default by v1.24.0. (2) Version 0.26.0 built on vLLM 0.26.0, compatible with Gaudi v1.24.1, PyTorch 2.11. (3) Version 0.24.0 adds Qwen3-Next architecture support, FP8/INC quantization, Mamba/GDN rewrite, TPC-native causal_conv1d. (4) Version 0.19.0 adds Qwen 3.5, Mamba prefix caching, MxFP4 weight dequantization, LMCache integration, custom depthwise conv1d TPC kernel for MambaMixer2. (5) On HPU, multi-card serving uses vLLM's mp (multiprocessing) backend when world_size > 1; worker start method auto-overridden to spawn (not fork) because forking after HPU driver init causes hangs. (6) Custom ops registered: hpu_fused_moe, hpu_grouped_topk_router, hpu_layernorm, hpu_lora, hpu_mamba_mixer2, hpu_rotary_embedding, hpu_compressed_tensors, hpu_fp8, hpu_gptq, hpu_awq, hpu_mm_encoder_attention, hpu_rejection_sampler. (7) Supported models include Llama 3.1 8B/70B/405B, Mistral 7B, Mixtral 8x7B/8x22B, Qwen2.5 7B/14B/32B/72B, Qwen3, DeepSeek-R1-Distill-Llama-70B. (8) Bucketing mechanism, FP8 quantization, custom graph caching for fast warm-up.