gmakstutis/LightRAG
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
LightRAG/lightrag/kg
History
clssck 48c7732edc feat: add automatic entity resolution with 3-layer matching 
Implement automatic entity resolution to prevent duplicate nodes in the
knowledge graph. The system uses a 3-layer approach:

1. Case-insensitive exact matching (free, instant)
2. Fuzzy string matching >85% threshold (free, instant)
3. Vector similarity + LLM verification (for acronyms/synonyms)

Key features:
- Pre-resolution phase prevents race conditions in parallel processing
- Numeric suffix detection blocks false matches (IL-4 ≠ IL-13)
- PostgreSQL alias cache for fast lookups on subsequent ingestion
- Configurable thresholds via environment variables

Bug fixes included:
- Fix fuzzy matching false positives for numbered entities
- Fix alias cache not being populated (missing db parameter)
- Skip entity_aliases table from generic id index creation

New files:
- lightrag/entity_resolution/ - Core resolution module
- tests/test_entity_resolution/ - Unit tests
- docker/postgres-age-vector/ - Custom PG image with pgvector + AGE
- docker-compose.test.yml - Integration test environment

Configuration (env.example):
- ENTITY_RESOLUTION_ENABLED=true
- ENTITY_RESOLUTION_FUZZY_THRESHOLD=0.85
- ENTITY_RESOLUTION_VECTOR_THRESHOLD=0.5
- ENTITY_RESOLUTION_MAX_CANDIDATES=3
2025-11-27 15:35:02 +01:00
..
deprecated Preserve ordering in get_by_ids methods across all storage implementations 2025-10-11 12:37:59 +08:00
__init__.py Improve storage config validation and add config.ini fallback support 2025-11-08 22:48:49 +08:00
faiss_impl.py Remove final_namespace attribute for in-memory storage and use namespace in clean_llm_query_cache.py 2025-11-17 12:54:33 +08:00
json_doc_status_impl.py Remove final_namespace attribute for in-memory storage and use namespace in clean_llm_query_cache.py 2025-11-17 12:54:33 +08:00
json_kv_impl.py Remove final_namespace attribute for in-memory storage and use namespace in clean_llm_query_cache.py 2025-11-17 12:54:33 +08:00
memgraph_impl.py Refactor workspace handling to use default workspace and namespace locks 2025-11-17 12:54:33 +08:00
milvus_impl.py Standardize empty workspace handling from "_" to "" across storage 2025-11-17 12:54:33 +08:00
mongo_impl.py Standardize empty workspace handling from "_" to "" across storage 2025-11-17 12:54:33 +08:00
nano_vector_db_impl.py Standardize empty workspace handling from "_" to "" across storage 2025-11-17 12:54:33 +08:00
neo4j_impl.py Refactor: Extract retry decorator to reduce code duplication in Neo4J storage 2025-11-25 01:35:21 +08:00
networkx_impl.py Remove final_namespace attribute for in-memory storage and use namespace in clean_llm_query_cache.py 2025-11-17 12:54:33 +08:00
postgres_impl.py feat: add automatic entity resolution with 3-layer matching 2025-11-27 15:35:02 +01:00
qdrant_impl.py Refactor workspace handling to use default workspace and namespace locks 2025-11-17 12:54:33 +08:00
redis_impl.py Standardize empty workspace handling from "_" to "" across storage 2025-11-17 12:54:33 +08:00
shared_storage.py Add pipeline status lock function for legacy compatibility 2025-11-25 18:24:39 +08:00