Research assistant over 100k+ papers
Combines keyword and meaning-based search across arXiv papers and keeps only the useful passages in memory. Recall@10 went from 0.62 to 0.89, at 40% lower token cost.
- Role
- Lead AI Systems Engineer
- Built with
- Python, Qdrant, Redis, LangChain, FastAPI, Docker
Results
- Recall@10: 0.62 → 0.89
- 40% lower multi-turn token cost
- 100k+ arXiv corpus
In more detail
Hybrid dense (HNSW) + sparse (BM25) retrieval and Reciprocal Rank Fusion over 100k+ scholarly publications (arXiv), persisting only pruned, citation-grounded passages in Redis active memory to enable long-horizon research sessions at 40% lower token cost (boosting Recall@10 from 0.62 to 0.89).
Key decisions
Hybrid Dense HNSW + Sparse BM25 with Reciprocal Rank Fusion (RRF)
Combined semantic dense embeddings with exact lexical BM25 sparse vectors in Qdrant, fusing candidate rankings via RRF to boost Recall@10 from 0.62 to 0.89 on technical nomenclature and mathematical symbols.
Redis Active Context Memory & Citation-Grounded Pruning
Persisted only verified, citation-grounded passages in Redis session memory while pruning low-utility intermediate chunks—enabling long-horizon literature review sessions at 40% lower token expenditure.