cognee/evals
alekszievr 6653d73556
Feat/cog 950 improve metric selection (#435)
* QA eval dataset as argument, with hotpot and 2wikimultihop as options. Json schema validation for datasets.

* Load dataset file by filename, outsource utilities

* restructure metric selection

* Add comprehensiveness, diversity and empowerment metrics

* add promptfoo as an option

* refactor RAG solution in eval;2C

* LLM as a judge metrics implemented in a uniform way

* Use requests.get instead of wget

* clean up promptfoo config template

* minor fixes

* get promptfoo path instead of hardcoding

* minor fixes

* Add LLM as a judge prompts

* Minor refactor and logger usage
2025-01-15 10:45:55 +01:00
..
cloud Add -y to setup_ubuntu_instance.sh commands and update EC2_README 2024-11-29 11:30:39 +01:00
test_datasets/initial_test Updated evals, added falkordb 2024-05-20 14:41:08 +02:00
__init__.py add test for linter 2024-05-25 22:18:07 +02:00
deepeval_metrics.py Feat/cog 950 improve metric selection (#435) 2025-01-15 10:45:55 +01:00
EC2_README.md Add code formating to usermod command 2024-11-29 11:30:39 +01:00
eval_on_hotpot.py Feat/cog 950 improve metric selection (#435) 2025-01-15 10:45:55 +01:00
eval_swe_bench.py Fix: deleting incorrect repo path 2025-01-10 15:54:45 +01:00
eval_utils.py ruff format 2025-01-05 19:09:08 +01:00
generate_test_set.py Fix linter issues 2025-01-05 19:48:35 +01:00
official_hotpot_metrics.py ruff format 2025-01-05 19:09:08 +01:00
promptfoo_config_template.yaml Feat/cog 950 improve metric selection (#435) 2025-01-15 10:45:55 +01:00
promptfoo_metrics.py Feat/cog 950 improve metric selection (#435) 2025-01-15 10:45:55 +01:00
promptfoo_wrapper.py Feat/cog 950 improve metric selection (#435) 2025-01-15 10:45:55 +01:00
promptfooprompt.json Feat/cog 950 improve metric selection (#435) 2025-01-15 10:45:55 +01:00
qa_dataset_utils.py Feat/cog 950 improve metric selection (#435) 2025-01-15 10:45:55 +01:00
qa_metrics_utils.py Feat/cog 950 improve metric selection (#435) 2025-01-15 10:45:55 +01:00
simple_rag_vs_cognee_eval.py Fix linter issues 2025-01-05 19:48:35 +01:00