Design goal
ease-of-useprivate-offlineragresearch
Formats
safetensorsggufawqgptqadapter-lora
Notable features
The 'Stateful Open-Source RAG + LLM inference system' - a full private document-QA stack, not an inference kernel. Reads PDFs, Excel, Word, images, video frames, YouTube, audio, code, text and MarkdownPluggable vector stores: Chroma, Weaviate or in-memory FAISS, with selectable embedding models (instructor-large, all-MiniLM-L6-v2)HYDE (Hypothetical Document Embeddings) for retrieval using the LLM's own hypothetical answers as the querySemantic chunking, which the README flags as requiring a GPUAttention sinks for arbitrarily long generation on LLaMA-2, Mistral, MPT, Pythia and FalconTwo inference backends in one project - HuggingFace models and LLaMA.cpp GGML on GPU; HF, LLaMA.cpp and GPT4ALL on CPU. This dual path is the niche interestSupports AutoGPTQ at 4-bit/8-bit and LORA adaptersGradio UI or CLI with streaming, plus a wide set of side models (LLaVA and other vision models, Stable Diffusion, Whisper STT, Microsoft Speech T5 TTS, Flux)Claims 80 tokens/s summarization with a 13B LLaMA2 model (vendor claim, no methodology given)Claims over 1000 unit and integration tests taking over 24 GPU-hours as a quality signal
Hardware caveats
ARCHIVED. The GitHub API reports archived=true on h2oai/h2ogpt as of 2026-10-03. Last push 2025-10-09; last release 0.2.1 from 2024-06-04, i.e. releases stopped over two years ago while 1188 open issues accumulatedThe repo points at a commercial successor: the README's first substantive link is 'Try Enterprise Version for Free' at h2ogpte.genai.h2o.ai. The open-source line was retired into the enterprise product rather than superseded by another OSS runtimeNAME COLLISION - do not confuse this with the 'H2O' kNN-LLM paper by Shi et al. (kNN-LM: Generalization through Memorization, arXiv:1911.00172). Unrelated projects, unrelated authors, wildly different reception. This is H2O.ai, the Wisconsin company behind H2O-3The RAG side is where the GPU requirement hides: the README explicitly marks semantic chunking as needing a GPU, so the 'CPU support' claim covers LLM inference but not the full pipelineIts dual HF-plus-llama.cpp design means it inherits both projects' format and quality problems, including that GGML-era and HF-era checkpoints of the same model give different answers11954 stars and an archived flag is the whole story: this is the fate of OSS stacks that outgrow their serving engine
Best for
Studying what a complete private-RAG product needs beyond an inference engine, and how much of that is vector-store plumbing rather than model servingReading the HYDE and semantic-chunking implementations as concrete codeHistorical: it is the clearest public example of the OSS-RAG-stack-to-enterprise-product transition
Avoid for
Anything new - the repo is archived and releases stopped in June 2024Serving: it is a RAG application, not an engine. If you want the LLM serving part use [[llama-cpp]] or [[vllm]]Retrieval research: use ColBERT / PLAID (see [[niche-colbert]]) or a maintained vector database instead of a wrapper around Chroma and FAISS
notes
Verified via the GitHub API on 2026-10-03: created 2023-03-24, archived=true, last push 2025-10-09, latest release 0.2.1 (2024-06-04), Apache-2.0, 1188 open issues, 11954 stars. Explicitly disambiguated from the kNN-LLM 'H2O' paper of the same name as required. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2025-10-09 (11.8 months before this record), latest release 0.2.1 published 2024-06-04, 78 contributors (distinct committers including anonymous), open_issues_count 328 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), ARCHIVED (read-only), 11954 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. ||| DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - repositories, READMEs, docs and issue trackers were searched. See [[ev-production-evidence-is-scarce-and-nearly-always-single-site]] for the grading scheme; note that for an artifact-evaluation or paper-code release, non-deployment is documented intent rather than a negative finding.