24ca4cc6b7
### What problem does this PR solve? This PR investigates the cause of #7957. TL;DR: Incorrect similarity calculations lead to too many candidates. Since candidate selection involves interaction with the LLM, this causes significant delays in the program. What this PR does: 1. **Fix similarity calculation**: When processing a 64 pages government document, the corrected similarity calculation reduces the number of candidates from over 100,000 to around 16,000. With a default batch size of 100 pairs per LLM call, this fix reduces unnecessary LLM interactions from over 1,000 calls to around 160, a roughly 10x improvement. 2. **Add concurrency and timeout limits**: Up to 5 entity types are processed in "parallel", each with a 180-second timeout. These limits may be configurable in future updates. 3. **Improve logging**: The candidate resolution process now reports progress in real time. 4. **Mitigates potential concurrency risks** ### Type of change - [x] Bug Fix (non-breaking change which fixes an issue) - [x] Refactoring |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| community_report_prompt.py | ||
| community_reports_extractor.py | ||
| entity_embedding.py | ||
| extractor.py | ||
| graph_extractor.py | ||
| graph_prompt.py | ||
| index.py | ||
| leiden.py | ||
| mind_map_extractor.py | ||
| mind_map_prompt.py | ||
| smoke.py | ||