1ad0bf82f9
This contribution adds a comprehensive evaluation system using the RAGAS framework to assess LightRAG's retrieval and generation quality. Features: - RAGEvaluator class with four key metrics: * Faithfulness: Answer accuracy vs context * Answer Relevance: Query-response alignment * Context Recall: Retrieval completeness * Context Precision: Retrieved context quality - HTTP API integration for live system testing - JSON and CSV report generation - Configurable test datasets - Complete documentation with examples - Sample test dataset included Changes: - Added lightrag/evaluation/eval_rag_quality.py (RAGAS evaluator implementation) - Added lightrag/evaluation/README.md (comprehensive documentation) - Added lightrag/evaluation/__init__.py (package initialization) - Updated pyproject.toml with optional 'evaluation' dependencies - Updated .gitignore to exclude evaluation results directory Installation: pip install lightrag-hku[evaluation] Dependencies: - ragas>=0.3.7 - datasets>=4.3.0 - httpx>=0.28.1 - pytest>=8.4.2 - pytest-asyncio>=1.2.0
16 lines
379 B
Python
16 lines
379 B
Python
"""
|
|
LightRAG Evaluation Module
|
|
|
|
RAGAS-based evaluation framework for assessing RAG system quality.
|
|
|
|
Usage:
|
|
from lightrag.evaluation.eval_rag_quality import RAGEvaluator
|
|
|
|
evaluator = RAGEvaluator()
|
|
results = await evaluator.run()
|
|
|
|
Note: RAGEvaluator is imported dynamically to avoid import errors
|
|
when ragas/datasets are not installed.
|
|
"""
|
|
|
|
__all__ = ["RAGEvaluator"]
|