yangdx
ced3aef7cb
refactor: simplify text encoding by removing redundant safe_encode_for_llm
2025-08-19 19:37:46 +08:00
yangdx
806081645f
Refactor text cleaning to use sanitize_text_for_encoding consistently
...
• Replace clean_text with sanitize_text
• Remove deprecated clean_text function
• Add whitespace trimming to sanitizer
• Improve UTF-8 encoding safety
• Consolidate text cleaning logic
2025-08-19 19:20:01 +08:00
yangdx
f9cf544805
Add text sanitization to prevent UTF-8 encoding errors in LLM calls
...
• Remove surrogate characters
• Clean control characters
• Sanitize input and history messages
• Add comprehensive error handling
• Log sanitization activities
2025-08-19 18:50:52 +08:00
yangdx
64015548df
Refactor MD5 hash functions and consolidate Unicode error handling
2025-08-19 17:49:23 +08:00
yangdx
64058c771f
Refactor: Harden compute_args_hash against Unicode errors
2025-08-19 17:19:39 +08:00
yangdx
2603e99005
Enhance file deletion to remove files from both input and enqueued dirs
2025-08-19 17:13:58 +08:00
yangdx
1f86543772
Update i18n translation and webui assets
2025-08-19 16:23:05 +08:00
yangdx
c6b30f1a03
Fix file type mappings for proper MIME type handling
2025-08-19 15:26:21 +08:00
yangdx
0aa1bc8bf9
Update webui assets and bump api version to 0205
2025-08-19 15:11:34 +08:00
yangdx
e38df464ea
Ensure front-end file type uploads are synchronized with back-end
2025-08-19 15:10:13 +08:00
yangdx
ac33cf693d
Refactor keyword extraction rules and remove overlap constraint
...
• Require content in both keyword categories
• Remove no-overlap rule between lists
• Simplify edge case handling
• Clarify source of truth requirement
2025-08-19 15:07:40 +08:00
yangdx
9ed5b93467
Add [File Extraction] prefix to error messages and logs
2025-08-19 11:33:28 +08:00
Daniel.y
ce35b1dfd4
Merge pull request #1977 from danielaskdd/keywork-extract
...
Optimize keyword extraction prompt, and remove conversation history from keyword extraction
2025-08-19 00:47:02 +08:00
yangdx
92c0ad0076
Fix linting
2025-08-19 00:45:29 +08:00
yangdx
23334e7e51
Update prompt.py
2025-08-19 00:29:33 +08:00
yangdx
2a7fec2873
Optimize keyword extraction prompt, and remove conversation history from keywork extraction.
...
- Remove history context processing
- Update prompt to focus on single query
- Clarify high/low level keyword types
- Improve JSON output instructions
- Add edge case handling guidance
2025-08-18 23:35:04 +08:00
yangdx
ee15629f26
Merge branch 'pg-optimization'
2025-08-18 22:34:08 +08:00
yangdx
cdfbd2114f
Merge branch 'main' into pg-optimization
2025-08-18 22:24:37 +08:00
yangdx
d54c8f973b
Merge branch 'Matt23-star/main' into pg-optimization
2025-08-18 22:23:47 +08:00
yangdx
1c4d6fde58
Change log level from info to debug for document storage message
2025-08-18 20:04:29 +08:00
Daniel.y
5fc2400a70
Merge pull request #1976 from danielaskdd/kg-context-file-path
...
Refactor: Remove file_path and created_at from entity and relation query context send to LLM
2025-08-18 19:40:54 +08:00
yangdx
368d2b00d6
Update webui assets and bump api version to 0204
2025-08-18 19:33:46 +08:00
yangdx
d5e8f1e860
Update default query parameters for better performance
...
- Increase chunk_top_k from 10 to 20
- Reduce max_entity_tokens to 6000
- Reduce max_relation_tokens to 8000
- Update web UI default values
- Fix max_total_tokens to 30000
2025-08-18 19:32:11 +08:00
yangdx
8d7a7e4ad6
Refactor prompt templates with improved guidelines and citation formats
2025-08-18 19:14:32 +08:00
yangdx
d3fde60938
refactor: remove file_path and created_at from context, improve token truncation
...
- Remove file_path and created_at fields from entity and relationship contexts
- Update token truncation to include full JSON serialization instead of content only
2025-08-18 18:30:09 +08:00
Daniel.y
1484c4adfa
Merge pull request #1975 from danielaskdd/milvus-file-path-len
...
Refac: Increase file_path field length to 32768 and add schema migration for Milvus DB
2025-08-18 17:17:00 +08:00
yangdx
a9d6807432
Fix query windows size limitation for Milvus data migration
2025-08-18 16:29:03 +08:00
yangdx
47b8caaf64
Stop execution on validation errors in Milvus storage
...
• Stop execution on validation errors to prevent potential data loss
2025-08-18 14:15:07 +08:00
yangdx
453efeb924
Fix file path length checking to use UTF-8 byte length instead of char count
2025-08-18 13:59:27 +08:00
Daniel.y
b27664298a
Merge pull request #1971 from danielaskdd/failed-2-pending
...
Change the status from PROCESSING/FAILED to PENDING at the beginning of document processing pipeline
2025-08-18 12:03:49 +08:00
yangdx
dcec511f72
feat: increase file path length limit to 32768 and add schema migration for Milvus DB
...
- Bump path limit to 32768 chars
- Add migration detection logic
- Implement dual-client migration
- Auto-migrate old collections
2025-08-18 04:37:12 +08:00
yangdx
377f1a022e
fix: reset PROCESSING/FAILED docs to PENDING at the beginging of document processing pipeline
...
- Reset documents with PROCESSING/FAILED status to PENDING when they pass consistency checks
- Update doc_status storage and clear error messages/metadata on reset
2025-08-18 00:49:52 +08:00
yangdx
add8b07a21
Improve logging messages for document processing clarity
2025-08-18 00:22:04 +08:00
yangdx
14e083a1a6
fix: replace pyuca with pypinyin for Chinese pinyin sorting and add file_path sort
2025-08-17 15:21:24 +08:00
Matt23-star
60564cf453
fix: correct parameter usage in database query for improved reliability
2025-08-17 13:50:41 +08:00
yangdx
1941df9cf6
Simplify warning message format for document deletion
2025-08-17 13:30:55 +08:00
Daniel.y
9cc9d62c89
Merge pull request #1967 from danielaskdd/pinyin-sort
...
Add Chinese pinyin sorting support across document operations
2025-08-17 13:18:59 +08:00
yangdx
d84715bae7
Improve MongoDB index migration with better conflict detection
...
• Enhanced conflict detection logic
• Improved index comparison method
2025-08-17 12:53:05 +08:00
yangdx
61469c0a56
Add Chinese pinyin sorting support across document operations
...
• Replace pyuca with centralized utils function
• Add pinyin sort keys for file paths
• Update MongoDB indexes with zh collation
• Migrate existing indexes for compatibility
• Support Chinese chars in Redis/JSON storage
• Keep PostgreSQL sorting order controled by Database Collate order
2025-08-17 12:45:48 +08:00
Daniel.y
a635d0625e
Merge pull request #1966 from danielaskdd/fix-select-all
...
Fix Document Selection Issues After Pagination Implementation
2025-08-17 10:54:19 +08:00
yangdx
6196bab00a
Update webui assets and bump api version to 0203
2025-08-17 10:39:16 +08:00
yangdx
1af0803c62
fix(ui): fix selection state management in paginated views
...
- Replace DeselectDocumentsDialog with smart selection button
- Auto-reset selection on page/filter changes
- Remove deletion restrictions and update i18n
2025-08-17 10:38:12 +08:00
yangdx
3e4214cef3
Standardize document deletion warning messages for consistency
2025-08-17 09:35:46 +08:00
yangdx
f76d926512
Merge branch 'main' into pg-optimization
2025-08-17 08:57:24 +08:00
yangdx
185b576101
Fix parameter reference and apply code formatting improvements
2025-08-17 04:02:43 +08:00
yangdx
3a7310873c
Merge branch 'bedrock-support'
2025-08-17 02:23:44 +08:00
yangdx
da7e4b79e5
Update documentation in README files
2025-08-17 02:23:14 +08:00
yangdx
1ed77a2e53
Remove openai-ollama binding from LightRAG level args
2025-08-17 02:13:50 +08:00
Daniel.y
459b0e4c44
Merge pull request #1965 from danielaskdd/rm-enqueued-file
...
Feat: Optimize error handling for document processing pipeline
2025-08-17 01:59:33 +08:00
yangdx
301acfc274
Update webui assets
2025-08-17 01:54:39 +08:00
