llama.cpp SYCL backend (hand-written kernels over oneDNN, oneMKL and joint_matrix)
Fields
compilers/llama-cpp-sycl-backendNotes
Filed as a compiler record rather than an engine record because the repo already tracks llama.cpp as an engine and the thing worth recording here is the kernel technique, not the runtime. The technique that transfers to a hand-written stack: on XMX the bandwidth-bound decode and MoE-prefill GEMMs are limited by the dequantize step, not by the matrix unit. Dequantizing each weight tile directly into the joint_matrix operand (the 'XMX gather' paths) overlaps the dequant with the DPAS issue and removes the full-precision weight buffer. The docs name the joint_matrix shapes the paths may pick, per architecture: on Xe2, Xe3 and Xe-HPC (which is Ponte Vecchio) f16 8x16x16, 16x16x16, 32x64x16 and 32x64x32, tf32 8x16x8 and bf16 8x16x16; on Xe-HPG (Arc A770, Arrow Lake-H) only f16 8x8x16 and bf16 8x8x16. Accumulation is always f32. The operand type is chosen at runtime by GGML_SYCL_DYNAMIC_PRECISION: F16 is fastest but overflows above 65504, BF16 keeps the f32 range with a 7-bit mantissa, TF32 keeps the f16 mantissa and the f32 range but is about 30% slower and needs Xe2, Xe3 or Xe-HPC, and F32 turns the XMX paths off. The best supported combination is picked automatically and logged as fg_pick_combo. Weight formats the paths accept, as a bitmask: IQ4_NL, IQ3_S, IQ4_XS, IQ3_XXS, IQ2_XXS, IQ2_XS, IQ2_S, IQ1_S, IQ1_M, plus Q8_0, Q4_K, Q5_K and Q6_K for the MoE MUL_MAT_ID path only. Two separate, older optimizations are not this one and should not be conflated with it. The reorder optimization (scales split from weights for coalesced loads) landed for Q4_0, Q4_K, Q5_K, Q6_K and Q8_0 in 2026.04-05 and is what [[sycl-quant-reorder-gap-makes-some-quants-4x-slower]] measures. The 2025.2 Q4_0 MUL_MAT optimization raised llama-2-7b.Q4_0.gguf decode by 39 to 73 tok/s (+87%) on a Max 1550, 39 to 50 (+28%) on Flex 170, 42 to 55 (+30%) on Arc A770, 13 to 16 (+23%) on Meteor Lake and 14 to 17 (+21%) on Arrow Lake-H; the docs give no baseline configuration beyond the model file, so those five rows are the backend's own claim. Also in the 2026.04-05 changelog and not separately quantified: fused MoE. The docs name oneDNN as the default GEMM library since 2024.8, which is the thing the XMX paths beat inside their tile constraints and lose to outside them — see [[sycl-xmx-dequant-gemm-falls-back-past-tile-constraints]]. Vendor-claim status: every number above comes from the backend's own documentation, written by the people who ship the kernel, so treat them as the authors' measurements rather than as an independent result. target_backends uses spirv because that is the schema's nearest value to a Level Zero/SPIR-V stack; the backend does not target CUDA, ROCm, HIP, Metal or Vulkan.
Cross-references
This record references no other records — it is a leaf. It is cited by 0.