cfe9c949a7
<!-- .github/pull_request_template.md --> ## Description <!-- Provide a clear description of the changes in this PR --> - Comparative Framework: Independent benchmarking system for evaluating different RAG/QA systems - HotpotQA Dataset: 50 instances corpus and corresponding QA pairs for standardized evaluation - Base Class: Abstract QABenchmarkRAG with async pipeline for document ingestion and question answering - Three Benchmarks: Standalone implementations for Mem0, LightRAG, and Graphiti with specific dependencies ## DCO Affirmation I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin. --------- Co-authored-by: hajdul88 <52442977+hajdul88@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| helpers | ||
| hotpot_50_corpus.json | ||
| hotpot_50_qa_pairs.json | ||
| qa_benchmark_base.py | ||
| qa_benchmark_graphiti.py | ||
| qa_benchmark_lightrag.py | ||
| qa_benchmark_mem0.py | ||
| README.md | ||
Comparative QA Benchmarks
Independent benchmarks for different QA/RAG systems using HotpotQA dataset.
Dataset Files
hotpot_50_corpus.json- 50 instances from HotpotQAhotpot_50_qa_pairs.json- Corresponding question-answer pairs
Benchmarks
Each benchmark can be run independently with appropriate dependencies:
Mem0
pip install mem0ai openai
python qa_benchmark_mem0.py
LightRAG
pip install "lightrag-hku[api]"
python qa_benchmark_lightrag.py
Graphiti
pip install graphiti-core
python qa_benchmark_graphiti.py
Environment
Create .env with required API keys:
OPENAI_API_KEY(all benchmarks)NEO4J_URI,NEO4J_USER,NEO4J_PASSWORD(Graphiti only)
Usage
Each benchmark inherits from QABenchmarkRAG base class and can be configured independently.
Results
Updated results will be posted soon.