gmakstutis/LightRAG
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
4665 commits 5 branches 72 tags 705 MiB
Commit graph

805 commits

Author SHA1 Message Date
Matt23-star
9804a1885b feat: refactor parameter handling in database queries to use lists for improved consistency 2025-08-28 16:17:35 -07:00
Matt23-star
015e9ae3dd Merge branch 'main' into feature/optimization 
# Conflicts:
#	lightrag/kg/postgres_impl.py
 2025-08-20 16:05:38 +08:00
Matt23-star
874ddda605 feat: remove unused parameter from query methods across multiple implementations 2025-08-20 15:59:05 +08:00
yangdx
a4c4b1182a Fix logging level usage in Redis retry decorator 
* Replace string with logging.WARNING constant
 2025-08-20 05:21:15 +08:00
yangdx
cdfbd2114f Merge branch 'main' into pg-optimization 2025-08-18 22:24:37 +08:00
yangdx
a9d6807432 Fix query windows size limitation for Milvus data migration 2025-08-18 16:29:03 +08:00
yangdx
47b8caaf64 Stop execution on validation errors in Milvus storage 
• Stop execution on validation errors to prevent potential data loss
 2025-08-18 14:15:07 +08:00
yangdx
dcec511f72 feat: increase file path length limit to 32768 and add schema migration for Milvus DB 
- Bump path limit to 32768 chars
- Add migration detection logic
- Implement dual-client migration
- Auto-migrate old collections
 2025-08-18 04:37:12 +08:00
Matt23-star
60564cf453 fix: correct parameter usage in database query for improved reliability 2025-08-17 13:50:41 +08:00
yangdx
d84715bae7 Improve MongoDB index migration with better conflict detection 
• Enhanced conflict detection logic
• Improved index comparison method
 2025-08-17 12:53:05 +08:00
yangdx
61469c0a56 Add Chinese pinyin sorting support across document operations 
• Replace pyuca with centralized utils function
• Add pinyin sort keys for file paths
• Update MongoDB indexes with zh collation
• Migrate existing indexes for compatibility
• Support Chinese chars in Redis/JSON storage
• Keep PostgreSQL sorting order controled by Database Collate order
 2025-08-17 12:45:48 +08:00
yangdx
185b576101 Fix parameter reference and apply code formatting improvements 2025-08-17 04:02:43 +08:00
Matt23-star
a0593ec1c9 feat: enhance query performance by restructuring relationships, entities, and chunks retrieval in PostgreSQL. 
Fixed: duplicate items query
 2025-08-16 22:49:54 +08:00
Matt23-star
6a7e3092ea feat: optimize node and edge queries in PostgreSQL. query tables Directly 2025-08-16 22:37:48 +08:00
Matt23-star
a7da48e05c feat: add batch size parameter to node and edge retrieval methods 2025-08-16 22:35:22 +08:00
yangdx
2a781dfb91 Update Neo4j database naming in env.example 2025-08-15 19:14:38 +08:00
yangdx
3a227e37b8 Add get_vectors_by_ids method to MongoVectorDBStorage 2025-08-15 16:53:14 +08:00
yangdx
7a7385a200 Add efficient vector retrieval by IDs to PGVectorStorage 2025-08-15 16:51:41 +08:00
yangdx
8f7031b882 Add get_vectors_by_ids method to QdrantVectorDBStorage 2025-08-15 16:46:52 +08:00
yangdx
a71499a180 Add get_vectors_by_ids method to MilvusVectorDBStorage 2025-08-15 16:36:50 +08:00
yangdx
1e2d5252d7 Add get_vectors_by_ids method and filter out vector data from query results 2025-08-15 16:32:26 +08:00
yangdx
f85e2aa4bf Merge branch 'main' into pick-trunk-by-vector 2025-08-14 03:54:26 +08:00
yangdx
0b22ffb252 Refac: uniformly protected with the get_data_init_lock for all storage initializations 2025-08-14 03:46:19 +08:00
yangdx
f1dafa0d01 feat: KG related chunks selection by vector similarity 
- Add env switch to toggle weighted polling vs vector-similarity strategy
- Implement similarity-based sorting with fallback to weighted
- Introduce batch vector read API for vector storage
- Implement vector store and retrive funtion for Nanovector DB
- Preserve default behavior (weighted polling selection method)
 2025-08-13 18:16:42 +08:00
yangdx
578bdaa410 Pin pymilvus version to 2.5.2 to avoid Protobuf version warning 2025-08-12 18:22:00 +08:00
yangdx
5d1bc8b49d Relocate client creation to the initialize method to prevent race conditions in multi-process mode. 2025-08-12 18:20:56 +08:00
yangdx
74783d7781 Remove redundant debug logging for Qdrant operations 2025-08-12 17:29:05 +08:00
yangdx
41f8ef05b9 Restore thread safety to MongoDB client manager 
- Protected client creation with lock
- Protected client release with lock
 2025-08-12 16:42:53 +08:00
yangdx
0b2c3d06c7 - Remove redundant collection listing check 2025-08-12 15:24:06 +08:00
yangdx
fc8ca1a706 Fix: add muti-process lock for initialize and drop method for all storage 2025-08-12 04:25:09 +08:00
yangdx
ca00b9c8ee Fix: Resolve workspace isolation problem for PostgreSQL with multiple LightRAG instances 2025-08-12 01:27:05 +08:00
yangdx
d9c1f935f5 Fix: Resolve workspace isolation issues in in-memory database with multiple LightRAG instances 2025-08-12 01:26:09 +08:00
yangdx
095e0cbfa2 Refac: Add workspace infomation to all logger output for all storage type 2025-08-12 01:19:09 +08:00
yangdx
16c9a81f4c feat: support config.ini for PostgreSQL vector index settings 
- Add support for reading vector_index_type, hnsw_m, hnsw_ef, and ivfflat_lists from config.ini
- Maintain backward compatibility with environment variables
- Update config.ini.example with new PostgreSQL vector index options
- Follow existing configuration priority: env vars > config.ini > defaults
 2025-08-08 02:55:49 +08:00
yangdx
dec4148075 Merge branch 'main' into Matt23-star/main 2025-08-08 02:24:34 +08:00
yangdx
f38e10559e Update PostgreSQL vector index configuration 
- Remove FLAT index support
- Standardize on HNSW as default
- Add dimension validation
- Improve error logging
- Clean up index creation code
 2025-08-08 02:21:06 +08:00
yangdx
f4ef254de2 fix(neo4j): enhance connection lifecycle management to prevent timeout errors 
- Add max_connection_lifetime, liveness_check_timeout, keep_alive parameters
- Extend retry mechanisms for connection reset scenarios
- Update config examples with new Neo4j connection options
- Resolves ClientTimeoutException during data insertion operations
 2025-08-08 01:07:45 +08:00
Matt23-star
727ca43d3c feat: add vector index creation functionality for PostgreSQL 2025-08-07 23:07:18 +08:00
yangdx
a04c11a598 Remove deprecated storage 2025-08-06 00:02:50 +08:00
yangdx
cc1f7118e7 Remove deprecated cache_by_modes functionality from all storage 2025-08-05 23:20:26 +08:00
yangdx
8294d6d1b7 Remove deprecated mode field from LLM cache schema 
- Drop mode column from LLM cache table
- Update primary key to exclude mode
- Remove mode from all SQL queries
- Deprecate mode-related methods
- Update schema migration logic
 2025-08-05 23:18:54 +08:00
yangdx
0463963520 fix: include all query parameters in LLM cache hash key generation 
- Add missing query parameters (top_k, enable_rerank, max_tokens, etc.) to cache key generation in kg_query, naive_query, and extract_keywords_only functions
- Add queryparam field to CacheData structure and PostgreSQL storage for debugging
- Update PostgreSQL schema with automatic migration for queryparam JSONB column
- Prevent incorrect cache hits between queries with different parameters

Fixes issue where different query parameters incorrectly shared the same cached results.
 2025-08-05 18:03:10 +08:00
yangdx
7b3a9c09ca Fix: add missing colume to LLM cache of PostgreSQL implementation 2025-08-04 11:12:59 +08:00
yangdx
5513155808 Fix namespace tablename translate error 
- Reorder namespace table map for PostgreSQL
- Ensure specific namespaces come first
 2025-08-04 00:21:20 +08:00
yangdx
952d1feb07 feat: Add support for KV_STORE_FULL_ENTITIES and KV_STORE_FULL_RELATIONS namespaces in PGKVStorage 
- Add LIGHTRAG_FULL_ENTITIES and LIGHTRAG_FULL_RELATIONS table schemas
- Implement complete CRUD operations for both namespaces
- Add automatic table creation and migration support
- Add SQL templates and namespace mappings
- Ensure workspace isolation and proper indexing
 2025-08-03 22:54:56 +08:00
yangdx
d2dd137f83 feat: implement get_all_nodes and get_all_edges methods for graph storage backends 
Add get_all_nodes() and get_all_edges() methods to Neo4JStorage, PGGraphStorage, MongoGraphStorage, and MemgraphStorage classes. These methods return all nodes and edges in the graph with consistent formatting matching NetworkXStorage for compatibility across different storage backends.
 2025-08-03 11:02:37 +08:00
yangdx
091f2b42c3 feat(performance): Optimize document deletion with entity/relation index 
- Introduces an index mapping documents to their corresponding entities and relations. This significantly speeds up `adelete_by_doc_id` by replacing slow graph traversal with a fast key-value lookup.
- Refactors the ingestion pipeline (`merge_nodes_and_edges`) to populate this new index. Adds a one-time data migration script to backfill the index for existing data.
 2025-08-03 09:19:02 +08:00
yangdx
2f0aa7ed12 Optimize graph query by simplifying MATCH pattern 
- Simplify MATCH clause to ()-[r]-()
- Remove node type constraints
- Improve query performance
 2025-08-02 12:54:22 +08:00
yangdx
e00690b41b Fix: workspace isolation problem for json KV storage 
- Use workspace+namespace as final_namespace identifier
- Update all related storage operations
- Maintain backward compatibility
 2025-08-02 11:30:19 +08:00
yangdx
9a8f58826d fix: Add safe handling for missing file_path and metadata in PostgreSQL doc status functions 
- Add null-safe file_path handling with "no-file-path" fallback in get_docs_by_status and get_docs_by_track_id
- Enhance metadata validation to ensure dict type after JSON parsing
- Align PostgreSQL implementation with JSON implementation safety patterns
- Prevent KeyError exceptions when database records have missing fields
 2025-07-31 18:07:53 +08:00