FlashInfer GitHub repository

README. Describes itself as 'a library and kernel generator for inference' providing unified APIs for attention, GEMM and MoE with 'multiple backend implementations including FlashAttention-2/3, cuDNN, CUTLASS, and TensorRT-LLM', and 'Automatically selects the best backend for your hardware and workload'. GPU support table: Turing SM75; Ampere SM80/SM86; Ada SM89; Hopper SM90; Blackwell SM100/SM103 (B200/B300), SM110 (Jetson Thor), SM120/SM121 (RTX 50 series, DGX Spark) - with the explicit warning 'Not all features are supported across all compute capabilities.' Kernel features: paged and ragged KV cache, separate decode/prefill/append kernels, native MLA support, cascade attention (hierarchical KV reuse for shared prefixes), block-sparse and variable block-sparse, POD-attention (fused prefill+decode for mixed batching); BF16 GEMM for SM10.0+, FP8 GEMM per-tensor and groupwise, FP4 GEMM (NVFP4 and MXFP4) for Blackwell; fused MoE with DeepSeek-V3 / Llama-4 / standard top-k routing and FP8/FP4 block-scaled expert weights; sorting-free top-k/top-p/min-p sampling; AllReduce, MNNVL multi-node NVLink, NVSHMEM integration; RoPE, RMSNorm/LayerNorm/Gemma fused norms, fused-gated SiLU/GELU. Distribution matters for cold start: flashinfer-python compiles/downloads kernels on first use, flashinfer-cubin ships pre-compiled binaries for all supported architectures, flashinfer-jit-cache installs architecture-specific pre-built kernel providers. Blackwell support added in v0.4.0 (2025-10-08). No performance number is taken from this record.