Commit graph

4444 commits

Author SHA1 Message Date
yangdx
78b4ff2d0b Bump api version to 0195 2025-07-31 02:00:26 +08:00
yangdx
3e5efd0b27 Add /documents/paginated to filtered logging paths 2025-07-31 02:00:00 +08:00
yangdx
a8f7e125c4 Update i18n translation 2025-07-31 01:59:26 +08:00
yangdx
45f27fccc3 feat(webui): Implement intelligent polling and responsive health checks
- Relocate the health check functionality from aap.tsx to state.ts to enable initialization by other components.
- Replaced the fixed 5-second polling with a dynamic interval. The polling interval is extended to 30 seconds when the no files in pending an processing state.
- Data refresh is triggered instantly when  `pipelineBusy` state changed
- Health check is triggered  after clicking "Scan New Documents" or "Clear Documents"
2025-07-31 01:37:24 +08:00
yangdx
93dede163d feat: move document list reset button to right side with ghost style 2025-07-30 22:37:22 +08:00
yangdx
08da46ac0f Bump api version to 0194 2025-07-30 18:50:53 +08:00
yangdx
83dc672f21 Update webui assets 2025-07-30 18:50:22 +08:00
yangdx
0eac1a883a Feat: add file path sorting for document manager
- Add file_path sorting support to all database backends (JSON, Redis, PostgreSQL, MongoDB)
- Implement smart column header switching between "ID" and "File Name" based on display mode
- Add automatic sort field switching when toggling between ID and file name display
- Create composite indexes for workspace+file_path in PostgreSQL and MongoDB for better query performance
- Update frontend to maintain sort state when switching display modes
- Add internationalization support for "fileName" in English and Chinese locales

This enhancement improves user experience by providing intuitive file-based sorting
while maintaining performance through optimized database indexes.
2025-07-30 18:46:55 +08:00
yangdx
e60c26ea77 feat: set default page size to 10 and persist in localStorage
- Change default documents page size from 20 to 10 rows
- Add documentsPageSize setting to settings store with persistence
- Update DocumentManager to use and save page size preference
- Include migration for existing users to get new default value
2025-07-30 18:07:38 +08:00
yangdx
74eecc46e5 feat(pagination): Implement document list pagination backends and frontend UI
- Add pagination support to BaseDocStatusStorage interface and all implementations (PostgreSQL, MongoDB, Redis, JSON)
- Implement RESTful API endpoints for paginated document queries and status counts
- Create reusable pagination UI components with internationalization support
- Optimize performance with database-level pagination and efficient in-memory processing
- Maintain backward compatibility while adding configurable page sizes (10-200 items)
2025-07-30 17:58:32 +08:00
yangdx
c24c2ff2f6 Remove deprecated temp file saving function
- Delete unused save_temp_file function
2025-07-30 14:23:08 +08:00
yangdx
b456bb0942 Fix linting 2025-07-30 13:59:50 +08:00
yangdx
a788899285 Fix linting 2025-07-30 13:32:15 +08:00
yangdx
797dcc1ff1 Update README 2025-07-30 13:31:47 +08:00
yangdx
50621d5a94 Update docs 2025-07-30 13:16:26 +08:00
yangdx
aba46213a7 Update README 2025-07-30 13:13:59 +08:00
yangdx
c7bc4fc42c Add track_id return to document processing pipeline 2025-07-30 10:27:12 +08:00
Daniel.y
444593bda8
Merge pull request #1878 from Ja1aia/main
fix timeout issue
2025-07-30 09:19:46 +08:00
Daniel.y
6d0a644844
Merge pull request #1883 from danielaskdd/add-metadata
feat: add processing time tracking to document status with metadata field
2025-07-30 09:15:36 +08:00
Daniel.y
1c35322088
Merge pull request #1882 from danielaskdd/add-track-id-support
Feat: add Document Processing Track ID Support for Frontend
2025-07-30 09:15:21 +08:00
Daniel.y
6fa5a6f634
Merge pull request #1881 from danielaskdd/remove-content-in-doc-status
Refact remove content in doc status to improve performance
2025-07-30 09:15:06 +08:00
yangdx
29e829113b Fix status key serialization issue in get_rack_status 2025-07-30 04:45:48 +08:00
yangdx
30f71c8acf Remove _id field and improve index handling in MongoDB
- Remove MongoDB _id field from documents
- Improve index existence check and creation
2025-07-30 04:17:26 +08:00
yangdx
cfb7117dd6 Fix track_id missing for query in PostgreSQL 2025-07-30 03:44:20 +08:00
yangdx
5ec7eedf37 Bump api version to 0193 2025-07-30 03:11:44 +08:00
yangdx
faa59cac72 Update webui assets 2025-07-30 03:11:19 +08:00
yangdx
cbaede8455 Add ScanResponse type for scan endpoint in webui 2025-07-30 03:11:09 +08:00
yangdx
7207598fc4 Fix track_id bugs and add track_id to scanning response 2025-07-30 03:06:20 +08:00
yangdx
75de799353 Remove deprecated content field from doc status storage
- Remove content field from JSON storage
- Remove content field from MongoDB storage
- Remove content field from Redis storage
2025-07-30 01:00:06 +08:00
yangdx
3ef3b8e155 Update webui assets 2025-07-30 00:06:27 +08:00
yangdx
6f958d5aee feat: add metadata timestamps to document processing and update frontend compatibility
- Add metadata field to doc_status storage with Unix timestamps for processing start/end times
- Update frontend API types: error -> error_msg, add track_id and metadata support
- Add getTrackStatus API method for document tracking functionality
- Fix frontend DocumentManager to use error_msg field for proper error display
- Ensure full compatibility between backend metadata changes and frontend UI
2025-07-30 00:04:27 +08:00
yangdx
93afa7d8a7 feat: add processing time tracking to document status with metadata field
- Add metadata field to DocProcessingStatus with start_time and end_time tracking
- Record processing timestamps using Unix time format (seconds precision)
- Update all storage backends (JSON, MongoDB, Redis, PostgreSQL) for new field support
- Maintain backward compatibility with default values for existing data
- Add error_msg field for better error tracking during document processing
2025-07-29 23:42:33 +08:00
yangdx
7206c07468 Remove deprecated content field from doc status
- Drop content column from LIGHTRAG_DOC_STATUS
- Clean up doc status handling code
- Maintain backward compatibility
2025-07-29 23:19:36 +08:00
yangdx
1e1adcb64a Add index on track_id column in doc status table of PostgreSQL 2025-07-29 23:03:09 +08:00
yangdx
6014b9bf73 feat: add track_id support for document processing progress monitoring
- Add get_docs_by_track_id() method to all storage backends (MongoDB, PostgreSQL, Redis, JSON)
- Implement automatic track_id generation with upload_/insert_ prefixes
- Add /track_status/{track_id} API endpoint for frontend progress queries
- Create database indexes for efficient track_id lookups
- Enable real-time document processing status tracking across all storage types
2025-07-29 22:24:21 +08:00
yangdx
dafdf92715 Remove content fallback logic in get_docs_by_status from Redis 2025-07-29 19:13:07 +08:00
yangdx
40a4cacee0 Merge branch 'main' into remove-content-from-doc-status 2025-07-29 16:15:01 +08:00
yangdx
92bbb7a1b3 Remove content fallback and standardize doc status handling
- Remove content_summary fallback logic
- Standardize doc status processing
- Handle missing file_path consistently
2025-07-29 16:13:51 +08:00
yangdx
24c36d876c Remove content field from DocProcessingStatus, update MongoDB and PostgreSQL implementation 2025-07-29 14:52:45 +08:00
administrator
9c3e1505b5 fix timeout issue 2025-07-29 13:38:46 +07:00
yangdx
8274ed52d1 feat: separate document content from doc_status to improve performance
This optimization significantly improves doc_status query/update performance by avoiding large string operations during frequent status checks.
2025-07-29 14:20:07 +08:00
Daniel.y
c4544c23a7
Merge pull request #1876 from Ja1aia/main
Fix: corrected unterminated f-string in config.py
2025-07-29 14:12:57 +08:00
administrator
c26dfa33de Fix: corrected unterminated f-string in config.py 2025-07-29 11:21:23 +07:00
Daniel.y
7a5df185a5
Merge pull request #1875 from danielaskdd/remove-embedding-max-token-size
refactor: Remove deprecated `max_token_size` from embedding configura…
2025-07-29 11:23:54 +08:00
yangdx
9923821d75 refactor: Remove deprecated max_token_size from embedding configuration
This parameter is no longer used. Its removal simplifies the API and clarifies that token length management is handled by upstream text chunking logic rather than the embedding wrapper.
2025-07-29 10:49:35 +08:00
yangdx
d26d413d97 Merge branch 'patch-1' 2025-07-29 09:57:58 +08:00
yangdx
f4c2dc327d Fix linting 2025-07-29 09:57:41 +08:00
yangdx
75d1b1e9f8 Update Ollama context length configuration
- Rename OLLAMA_NUM_CTX to OLLAMA_LLM_NUM_CTX
- Increase default context window size
- Add requirement for minimum context size
- Update documentation examples
2025-07-29 09:53:37 +08:00
yangdx
645f81f7c8 fixes a critical bug where Ollama options were not being applied correctly
`dict.update()` modifies the dictionary in-place and returns `None`.
2025-07-29 09:52:25 +08:00
yangdx
9bdbdae120 Update README 2025-07-29 08:22:48 +08:00