Qualcomm Cloud AI 100 runs LLMs, but only via ahead-of-time compile - not load-a-GGUF

A used Qualcomm Cloud AI 100 PCIe card (~$200-600, 32GB LPDDR4x) looks like cheap 8B-capable silicon, and it technically is - but only through the QPC compile-and-serve pipeline: quic/efficient-transformers converts an HF checkpoint to a Qualcomm Package (QPC) with AWQ-INT4/mxfp6 quantization and mxint8 KV cache, then the ghcr.io/quic/cloud_ai_inference_vllm container serves it. There is no GGUF path, no load-and-go.

The card is a fixed-compile NPU: models must be compiled to QPC with a specific quantization recipe (AWQ-INT4 / mxfp6) before serving. vLLM is available but as a Qualcomm container image, not upstream.

Use the public SDK: qualcomm/cloud-ai-sdk (BSD-3) + quic/efficient-transformers for HF->QPC compile + ghcr.io/quic/cloud_ai_inference_vllm for serving. Meta-Llama-3.1-8B-Instruct-AWQ-INT4 runs out of the box. Expert-only; if you cannot drive a compile pipeline it becomes a paperweight.

Verdict class "viable-ish / expert trap": the SDK is genuinely public and runs open models (Llama-3.1-8B AWQ-INT4, Qwen3-30B on 4x cards, even Llama-3.3-70B demos exist), but it is a compile-and-serve platform, not an inference-device ecosystem. Sweet spot for a single 32GB card is 7-8B INT4.