vLLM Gaudi Quickstart Guide · inference-research

Fields

id

sources/xpu-vllm-gaudi-quickstart

URL

https://vllm-gaudi.readthedocs.io/en/latest/getting_started/quickstart/quickstart.html

Publisher

vllm-project

Kind

spec-sheet

Published

2026-01-01

Accessed

2026-10-04

notes

Requirements: Ubuntu 22.04/24.04, RHEL 9.4/9.6, Python 3.10, Intel Gaudi 2 or 3, Gaudi software v1.24.1+. Validated models and TP sizes: Llama-3.1-8B (TP1), Llama-3.1-70B (TP4), Llama-3.1-405B (TP8), Mistral-7B (TP1), Mixtral-8x7B (TP2), Mixtral-8x22B (TP4), Qwen2.5-7B/14B/32B (TP1), Qwen2.5-72B (TP4), DeepSeek-R1-Distill-Llama-70B (TP8), Qwen3-0.6B (TP1), Qwen3-30B-A3B (TP1). Docker image: vault.habana.ai/gaudi-docker/1.24.1/ubuntu24.04/habanalabs/vllm-0.26.0-ptupstream-2.11.0:latest. Performance optimization guide references docs.habana.ai Optimizing Training Platform Guide.

Notes

Requirements: Ubuntu 22.04/24.04, RHEL 9.4/9.6, Python 3.10, Intel Gaudi 2 or 3, Gaudi software v1.24.1+. Validated models and TP sizes: Llama-3.1-8B (TP1), Llama-3.1-70B (TP4), Llama-3.1-405B (TP8), Mistral-7B (TP1), Mixtral-8x7B (TP2), Mixtral-8x22B (TP4), Qwen2.5-7B/14B/32B (TP1), Qwen2.5-72B (TP4), DeepSeek-R1-Distill-Llama-70B (TP8), Qwen3-0.6B (TP1), Qwen3-30B-A3B (TP1). Docker image: vault.habana.ai/gaudi-docker/1.24.1/ubuntu24.04/habanalabs/vllm-0.26.0-ptupstream-2.11.0:latest. Performance optimization guide references docs.habana.ai Optimizing Training Platform Guide.

Cross-references

This record references no other records.