Repo
kvcache-ai/ktransformers
Design goal
heterogeneous CPU-GPUresearchconsumer-hardware MoE
Scheduling
continuous-batchingprefix-cacheexpert-offload-scheduling
Backends
cudacpu-avx512cpu-avx2npu-ascend
Formats
safetensorsfp8int4rawint4
Notable features
Keeps attention and most of the dense weights on the GPU while streaming MoE experts from CPU RAM, so huge MoE models run on consumer GPUsCPU-GPU expert scheduling, native BF16 and FP8 per-channel precision paths, and native RAWINT4 routed experts for Kimi K2.5/K2.6Day-0 model support cadence (GLM-5, MiniMax-M2/M2.5/M3, Kimi-K2.5, DeepSeek-V4-Flash, GLM-5.2, GLM-5.3-flash)Also an SFT path: MoE LoRA and full-parameter fine-tuning including block-FP8 LoRA and BF16 full fine-tuning, wired through LlamaFactoryAVX512 x86 including AMD servers, and an AVX2-only CPU backend for lower-end hostsAscend NPU support with CPU expert offload (DeepSeek-V4-Flash on a single NPU)1M-token context and multimodal input on consumer GPUs via the GLM-5.3-flash pathThree-tier GPU-CPU-Disk prefix cache reuse, the only engine in this repo whose prefix cache has a disk tier - a consequence of its weights not fitting in VRAM rather than a design flourish
Hardware caveats
It is a research project first: the CPU-GPU split is tuned per model, and an unsupported architecture falls back to something much slowerThe whole premise is that experts live in system RAM - you need enough RAM and memory bandwidth for the offloaded experts, and a discrete GPU is strongly implied for the resident halfCPU capability still gates throughput: AMX was originally required for the fine-tuning path, relaxed in v0.7.0 to compatible AVX512 x86, and an AVX2-only backend exists for inference but is a separate tutorialAscend NPU support is CPU-expert-offload dependent, so the NPU is not doing the expert mathFine-tuning and inference are now split between kt-kernel and a LlamaFactory cookbook, and the original integrated framework was moved to archive/ - so docs and code are in two placesDay-0 support means a working tutorial, not parity with a datacenter engine on the same model
Best for
Running a very large MoE model on a consumer GPU with large system RAM, when datacenter hardware is not availableResearching heterogeneous placement and expert scheduling - that is what the project is forMoE fine-tuning on a single workstation-class machine
Avoid for
Datacenter serving, where a CUDA-first engine with expert parallelism is the right toolDense models that fit comfortably in VRAM, where plain vLLM or llama.cpp is simplerProduction use without reading the per-model tutorial, because behaviour varies sharply per architecture
notes
Scheduling terms are approximate here: KTransformers is not a paged-KV serving engine in the vLLM sense. It does its own cache management plus CPU-GPU expert scheduling, so only 'prefix-cache' and 'expert-offload-scheduling' are recorded as literal mechanisms. Published as SOSP 2025 (doi 10.1145/3731569.3764843). ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-09-30 (0.1 months), latest release v0.7.1 published 2026-09-15, 129 contributors, open_issues_count 513 (issues PLUS pull requests), NOT archived, 19561 stars. Actively developed with a real team behind it. DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - see [[ev-production-evidence-is-scarce-and-nearly-always-single-site]]. FEATURE AXES - PREFIX CACHING is the distinguishing one and it is why this engine exists: a THREE-TIER GPU-CPU-DISK prefix cache, 'Support 3-layer (GPU-CPU-Disk) prefix cache reuse' (2025-06-30). Every other engine in this repo keeps its prefix cache in memory; this one spills to disk, which is the necessary rather than the elegant answer for an engine whose weights do not fit in VRAM at all. MULTIMODAL: 'Added native support for GLM-5.3-flash, bringing 1M-token context and multimodal input to consumer GPUs' (2026-08-26). Speculative decoding, chunked prefill, disaggregated P/D and structured output: UNSTATED, and a keyword scan of the README returns nothing for any of the four, so they are recorded as unstated rather than absent.