Raspberry Pi AI HAT+ 2 (Hailo-10H) tops out at <=2B models despite the 'GenAI' badge and 8GB
[major] hardware
Fields
id
gotchas/gotcha-pi-ai-hat2-cannot-run-7bClass
hardware
Severity
major
Concepts
hailo-10hpi-ai-hatedge-inferencecompiled-modelsmall-modelno-7b
Symptom
A Pi AI HAT+ 2 bought to run a 7B-class model cannot - the compiled-model set for the Hailo-10H tops out at <=2B parameters (Llama-3.2-1B at 9.89 TPS is the best-supported row). The 'GenAI' branding and 8GB onboard RAM suggest more headroom than exists.
Root cause
Hailo-10H is a 40-TOPS INT4 edge accelerator. Its GenAI support is a fixed set of Hailo-compiled models served via Hailo-Ollama/HailoRT, and that set tops out at 1.5-2B. There is no path to load an arbitrary 7B GGUF - the model must be a Hailo-compiled artifact, and Hailo has not shipped one at 7B+.
Workaround
Use it for its actual purpose - a private-edge 1-2B small-model box at ~8-10 TPS via a clean Ollama experience for $200. Do not buy it for 7B. NOTE the original AI HAT+ ($70/$110) is Hailo-8/8L - vision only, NO LLM at all; only the AI HAT+ 2 has Hailo-10H.
notes
A real product with real measured tok/s - genuinely impressive for $200 if 1-2B models are the target. The trap is buying it for the 7B goal. Verdict: TRAP for 7B; legit toy for <=2B. Full compiled-model table in record comm-hailo10h-pi-ai-hat2-llama32-1b-tps.
Notes
A real product with real measured tok/s - genuinely impressive for $200 if 1-2B models are the target. The trap is buying it for the 7B goal. Verdict: TRAP for 7B; legit toy for <=2B. Full compiled-model table in record comm-hailo10h-pi-ai-hat2-llama32-1b-tps.