gmakstutis/LightRAG
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
5655 commits 5 branches 72 tags 705 MiB
Commit graph

590 commits

Author SHA1 Message Date
yangdx
14a6c24ed7 Add configurable embedding token limit with validation 
- Add EMBEDDING_TOKEN_LIMIT env var
- Set max_token_size on embedding func
- Add token limit property to LightRAG
- Validate summary length vs limit
- Log warning when limit exceeded
 2025-11-17 12:54:32 +08:00
yangdx
03cc6262c4 Prohibit direct access to internal functions of EmbeddingFunc. 
• Fix similarity search error in query stage
• Remove redundant null checks
• Improve log readability
 2025-11-08 01:43:36 +08:00
yangdx
ec2ea4fd3f Rename function and variables for clarity in context building 
- Rename _build_llm_context to _build_context_str
- Change text_units_context to chunks_context
- Move string building before early return
- Update log messages and comments
- Consistent variable naming throughout
 2025-11-01 12:15:24 +08:00
yangdx
3fa79026e0 Fix Entity Source IDs Tracking Problem 
- Handle existing node updates properly in edge merging stage
- Fix source_ids merging logic
- Reorder entity deletion and optimize node operations
- Delete relationships before entities
- Add edge existence debugging logs
 2025-10-29 01:19:55 +08:00
yangdx
29c4a91dc3 Move relationship ID sorting to before vector DB operations 
• Remove verbose entity rebuild logging
• Sort IDs before vector DB updates
• Keep graph storage with original order
 2025-10-28 19:13:48 +08:00
yangdx
5ee9a2f8c6 Fix entity consistency in knowledge graph rebuilding and merging 
• Sort src/tgt for consistent ordering
• Create missing nodes before edges
• Update entity chunks storage
• Pass entity_vdb to rebuild function
• Ensure entities exist in all storages
 2025-10-25 21:37:03 +08:00
yangdx
97a2ee4ef1 Rename rebuild function name and improve relationship logging format 2025-10-25 11:17:43 +08:00
yangdx
a9ec15e669 Resolve lock leakage issue during user cancellation handling 
• Change default log level to INFO
• Force enable error logging output
• Add lock cleanup rollback protection
• Handle LLM cache persistence errors
• Fix async task exception handling
 2025-10-25 03:06:45 +08:00
yangdx
77336e50b6 Improve error handling and add cancellation checks in pipeline 2025-10-24 17:54:17 +08:00
yangdx
78ad8873b8 Add cancellation check in delete loop 2025-10-24 14:47:20 +08:00
yangdx
743aefc655 Add pipeline cancellation feature for graceful processing termination 
• Add cancel_pipeline API endpoint
• Implement PipelineCancelledException
• Add cancellation checks in main loop
• Handle task cancellation gracefully
• Mark cancelled docs as FAILED
 2025-10-24 14:08:12 +08:00
yangdx
00aa5e53a7 Improve entity identifier truncation warning message format 2025-10-22 15:56:19 +08:00
yangdx
904b1f46f9 Add entity name length truncation with configurable limit 2025-10-22 14:02:30 +08:00
yangdx
a809245aed Preserve file path order by using lists instead of sets 2025-10-21 18:57:54 +08:00
yangdx
fe890fca15 Improve formatting of limit method info in rebuild functions 2025-10-21 18:34:06 +08:00
yangdx
3ed2abd82c Improve logging to show source ID ratios when skipping entities/edges 2025-10-21 16:20:34 +08:00
yangdx
80668aae22 Improve file path truncation labels and UI consistency 
• Standardize FIFO/KEEP truncation labels
• Update UI truncation text format
 2025-10-21 15:39:31 +08:00
yangdx
be3d274a0b Refactor node and edge merging logic with improved code structure 
• Add numbered steps for clarity
• Improve early return handling
• Enhance file path limiting logic
 2025-10-21 15:16:47 +08:00
yangdx
a5253244f9 Simplify skip logging and reduce pipeline status updates 2025-10-21 06:33:34 +08:00
yangdx
cd1c48beaf Standardize placeholder format to use colon separator consistently 2025-10-21 05:03:57 +08:00
yangdx
1154c5683f Refactor deduplication calculation and remove unused variables 2025-10-21 04:41:15 +08:00
yangdx
665f60b90f Refactor entity/relation merge to consolidate VDB operations within functions 
• Move VDB upserts into merge functions
• Fix early return data structure issues
• Update status messages (IGNORE_NEW → KEEP)
• Consolidate error handling paths
• Improve relationship content format
 2025-10-21 03:19:34 +08:00
yangdx
e01c998ee9 Track placeholders in file paths for accurate source count display 
• Add has_placeholder tracking variable
• Detect placeholder patterns in paths
• Show + sign for truncated counts
 2025-10-20 23:48:04 +08:00
yangdx
e0fd31a60d Fix logging message formatting 2025-10-20 22:09:09 +08:00
yangdx
a9fec26798 Add file path limit configuration for entities and relations 
• Add MAX_FILE_PATHS env variable
• Implement file path count limiting
• Support KEEP/FIFO strategies
• Add truncation placeholder
• Remove old build_file_path function
 2025-10-20 20:12:53 +08:00
yangdx
dc62c78f98 Add entity/relation chunk tracking with configurable source ID limits 
- Add entity_chunks & relation_chunks storage
- Implement KEEP/FIFO limit strategies
- Update env.example with new settings
- Add migration for chunk tracking data
- Support all KV storage
 2025-10-20 15:24:15 +08:00
yangdx
9f49e56a44 Merge branch 'main' into feat-entity-size-caps 2025-10-17 15:59:44 +08:00
yangdx
35cd567c9e Allow related chunks missing in knowledge graph queries 2025-10-17 00:19:30 +08:00
DivinesLight
c06522b927 Get max source Id config from .env and lightRAG init 2025-10-15 18:24:38 +05:00
yangdx
29bac49fb9 Handle empty query results by returning None instead of fail responses 
• Return None when no context found
• Add structured failure metadata
• Use PROMPTS["fail_response"] for content
• Keep API compatible
 2025-10-15 12:04:49 +08:00
haseebuchiha
d52c3377b4 Import from env and use default if none and removed useless import 2025-10-14 16:14:03 +05:00
DivinesLight
54f0a7d1ca Quick fix to limit source_id ballooning while inserting nodes 2025-10-14 14:47:04 +05:00
yangdx
85d1a563b3 Merge branch 'adminunblinded/main' 2025-10-10 12:31:47 +08:00
NeelM0906
f6d1fb98ac Fix Linting errors 2025-10-09 16:52:22 -04:00
yangdx
aac787bafb Clarify chunk tracking log message in _build_llm_context 2025-10-05 13:33:55 +08:00
yangdx
37e8898cf6 Simplify reference formatting in LLM context generation 
- Remove extra newlines in reference lists
- Change code block type from text to generic
 2025-10-01 22:20:58 +08:00
yangdx
dbb0b3afb4 Fix hl_keywords and ll_keywords cache logic 
- Remove hl_keywords and ll_keywords from keywork extracht cache
- Add hl_keywords and ll_keywords to LLM query cache
 2025-09-27 15:26:52 +08:00
yangdx
8cd4139cbf refactor: fix double query problem by add aquery_llm function for consistent response handling 
- Add new aquery_llm/query_llm methods providing structured responses
- Consolidate /query and /query/stream endpoints to use unified aquery_llm
- Optimize cache handling by moving cache checks before LLM calls
 2025-09-26 19:05:03 +08:00
yangdx
cbdc4c4bdf Refactor prompts and context building for better maintainability 
- Extract context templates to PROMPTS
- Unify token calculation logic
- Simplify user_prompt formatting
- Reduce code duplication
- Improve prompt structure consistency
 2025-09-26 12:39:06 +08:00
yangdx
fba2356c81 Move user_prompt to system prompt 
- Refactor query prompt handling to separate user prompts in system context
- Simplify user_query to only contain query
- Apply changes to both kg_query and naive_query
 2025-09-26 10:02:01 +08:00
yangdx
b848ca49e6 Fix linting 2025-09-25 16:22:00 +08:00
yangdx
b08b8a6a6a Add reference list support to query API endpoints with unified result handling 
• Add include_references param to QueryRequest
• Extend QueryResponse with references field
• Create unified QueryResult data structures
• Refactor kg_query and naive_query functions
• Update streaming to send references first
 2025-09-25 16:21:42 +08:00
yangdx
5eb4a4b799 feat: simplify citations, add reference merging, and restructure API response format 2025-09-24 14:30:10 +08:00
yangdx
367f3df038 Fix log message 2025-09-23 11:25:55 +08:00
yangdx
a4442a8613 Optimize log message 2025-09-23 11:22:14 +08:00
yangdx
86186c0c85 Update log message 2025-09-23 11:08:33 +08:00
yangdx
6e2eab5c23 Add ID fields to entities, relations, and chunks in raw data query results 2025-09-21 23:31:35 +08:00
yangdx
18e886d7e9 Improve context item identification with meaningful IDs 
- Add EN prefix to entitie IDs
- Add RE prefix to relation IDs
-Add DC prefix chunk IDs
- Enhance traceability across contexts
 2025-09-21 20:19:14 +08:00
yangdx
8f0fb3c9eb Include user query in prompt returns 2025-09-21 15:24:20 +08:00
yangdx
6eb37e270a Refactor query handling and improve RAG response prompts 
- Move user_prompt to query concatenation
- Remove DEFAULT_USER_PROMPT constant
- Enhance prompt clarity and structure
- Standardize citation formatting
- Improve step-by-step instructions
 2025-09-21 15:16:24 +08:00