CiteRAG
A fail-closed RAG system for local technical docs
The problem
CiteRAG is a self-hosted question-answering system for local technical documentation. It indexes Markdown and text files, retrieves the passages relevant to a question, and answers with citations attached. Most RAG demos return an answer even when the retrieved context does not actually support it. CiteRAG checks every answer for citations before returning it. When the model can't back a claim with a source, the answer is replaced with a refusal instead of being shown as fact.
Architecture
Ingestion
Search and answer
How it works
Ingestion
Documents are loaded, cleaned, and split into chunks. Each chunk gets a dense embedding (BAAI/bge-m3) and a sparse embedding, and both are indexed into Qdrant. Ingestion is incremental. Files are hashed, so a re-run skips anything unchanged, and chunks from edited or deleted files are cleaned up automatically instead of accumulating stale data.
Search and answer
A query is embedded and searched against Qdrant with hybrid dense and sparse retrieval, fused with Reciprocal Rank Fusion so both semantic and keyword matches surface. The candidate set is reranked with a cross-encoder (BAAI/bge-reranker-v2-m3) before it reaches the language model. The context builder attaches citation labels to the retrieved passages, the model generates an answer grounded in that context, and the answer is checked for citations before it's returned. An uncited answer becomes a refusal.
Optional HyDE retrieval
An optional HyDE mode retrieves using a model-generated hypothetical answer instead of the raw query. It helps on short or keyword-poor questions. It only changes retrieval. The final answer is still grounded in the real question that was asked.
Interfaces and infrastructure
A CLI, a FastAPI backend, and a Streamlit UI all call the same RAGService, so the ingestion and search logic isn't duplicated per interface. Qdrant runs in Docker for vector storage and Ollama runs the language model locally. All inference (embedding, reranking, and generation) happens on the local machine. No document content or query leaves it.
Testing and evaluation
137 unit tests cover the codebase, with the model and embedding calls mocked out. They confirm the code runs correctly, not that the answers are good. CI runs these tests, plus ruff for formatting and linting and mypy for type checking, on every push.
A separate eval harness checks answer quality against a live pipeline: 10 fixed questions (6 with expected keywords and a minimum citation count, 4 that should be refused as off-topic) run against a real Qdrant, Ollama, and FastAPI stack and get scored pass or fail. It needs that live stack, so it doesn't run in CI. It also supports comparing two retrieval configurations against each other, such as the baseline against HyDE, by writing each run to its own report.
Limitations
- ▸The eval harness isn't part of CI. It has to be run manually against a live stack, so answer-quality regressions between commits aren't caught automatically.
- ▸The golden set is 10 questions. That's enough to catch a regression or confirm an intentional change, not a statistically rigorous benchmark of answer quality.
- ▸Everything runs on one machine with a GPU for embedding, reranking, and generation. There's no distributed or multi-tenant deployment story here. It's built for one person's local documentation.