Commit graph

1228 commits

Author SHA1 Message Date
yangdx
831e658ed8 Update readme 2025-11-06 16:26:07 +08:00
yangdx
6e36ff41e1 Fix linting 2025-11-06 16:01:24 +08:00
yangdx
5f49cee20f Merge branch 'main' into VOXWAVE-FOUNDRY/main 2025-11-06 15:37:35 +08:00
anouarbm
c9e1c6c1c2 fix(api): change content field to list in query responses
BREAKING CHANGE: content field is now List[str] instead of str

- Add ReferenceItem Pydantic model for type safety
- Update /query and /query/stream to return content as list
- Update OpenAPI schema and examples
- Add migration guide to API README
- Fix RAGAS evaluation to handle list format

Addresses PR #2297 feedback. Tested with RAGAS: 97.37% score.
2025-11-03 04:57:08 +01:00
anouarbm
9d69e8d776 fix(api): Change content field from string to list in query responses
BREAKING CHANGE: The `content` field in query response references is now
an array of strings instead of a concatenated string. This preserves
individual chunk boundaries when a single file has multiple chunks.

Changes:
- Update QueryResponse Pydantic model to accept List[str] for content
- Modify query_text endpoint to return content as list (query_routes.py:425)
- Modify query_text_stream endpoint to support chunk content enrichment
- Update OpenAPI schema and examples to reflect array structure
- Update API README with breaking change notice and migration guide
- Fix RAGAS evaluation to flatten chunk content lists
2025-11-03 04:37:09 +01:00
anouarbm
0b5e3f9dc4 Use logger in RAG evaluation and optimize reference content joins 2025-11-02 18:43:53 +01:00
anouarbm
963ad4c637 docs: Add documentation and examples for include_chunk_content parameter
Added comprehensive documentation for the new include_chunk_content parameter
that enables retrieval of actual chunk text content in API responses.

Documentation Updates:
- Added "Include Chunk Content in References" section to API README
- Explained use cases: RAG evaluation, debugging, citations, transparency
- Provided JSON request/response examples
- Clarified parameter interaction with include_references

OpenAPI/Swagger Examples:
- Added "Response with chunk content" example to /query endpoint
- Shows complete reference structure with content field
- Demonstrates realistic chunk text content

This makes the feature discoverable through:
1. API documentation (README.md)
2. Interactive Swagger UI (http://localhost:9621/docs)
3. Code examples for developers
2025-11-02 17:53:27 +01:00
anouarbm
0bbef9814e Optimize RAGAS evaluation with parallel execution and chunk content enrichment
Added efficient RAG evaluation system with optimized API calls and comprehensive benchmarking.

Key Features:
- Single API call per evaluation (2x faster than before)
- Parallel evaluation based on MAX_ASYNC environment variable
- Chunk content enrichment in /query endpoint responses
- Comprehensive benchmark statistics (moyennes)
- NaN-safe metric calculations

API Changes:
- Added include_chunk_content parameter to QueryRequest (backward compatible)
- /query endpoint enriches references with actual chunk content when requested
- No breaking changes - default behavior unchanged

Evaluation Improvements:
- Parallel execution using asyncio.Semaphore (respects MAX_ASYNC)
- Shared HTTP client with connection pooling
- Proper timeout handling (3min connect, 5min read)
- Debug output for context retrieval verification
- Benchmark statistics with averages, min/max scores

Results:
- Moyenne RAGAS Score: 0.9772
- Perfect Faithfulness: 1.0000
- Perfect Context Recall: 1.0000
- Perfect Context Precision: 1.0000
- Excellent Answer Relevance: 0.9087
2025-11-02 17:39:43 +01:00
yangdx
61b57cbb5d Add PDF decryption support for password-protected files
• Add PDF_DECRYPT_PASSWORD env variable
• Check encryption status before reading
• Handle decrypt errors gracefully
• Log detailed error messages
• Support both encrypted/plain PDFs
2025-11-01 15:01:17 +08:00
yangdx
728721b14f Remove redundant separator lines in gunicorn shutdown handler 2025-11-01 12:53:54 +08:00
yangdx
6d4a55100e Remove redundant shutdown message from gunicorn 2025-11-01 12:52:22 +08:00
yangdx
7ccc1fdd27 Add frontend rebuild warning indicator to version display
- Return bool from check_frontend_build()
- Add ⚠️ symbol to outdated versions
- Show tooltip with rebuild message
- Add translations for warning text
- Fix tailwind config filename typo
2025-10-31 06:09:46 +08:00
yangdx
e5414c61ef Bump core version to 1.4.9.8 and API version to 0250 2025-10-31 05:23:48 +08:00
yangdx
c46c1b26a9 Add pycryptodome dependency for PDF encryption support 2025-10-31 01:49:42 +08:00
yangdx
c9e73bb450 Bump core version to 1.4.9.7 and API version to 0249 2025-10-30 19:43:35 +08:00
yangdx
f610fdaf9b Merge branch 'main' into Anush008/main 2025-10-30 11:07:39 +08:00
yangdx
3a7f753560 Bump core version to 1.4.9.6 and API version to 0248 2025-10-29 19:08:32 +08:00
yangdx
d5bcd14c6f Refactor service deployment to use direct process execution
- Remove bash wrapper script
- Update systemd service configuration
- Improve process management for gunicorn
- Simplify shared storage cleanup logic
- Update documentation for deployment
2025-10-29 18:55:47 +08:00
yangdx
6489aaa7f0 Remove worker_exit hook and improve cleanup logging
• Remove unreliable worker_exit function
• Add debug logs for cleanup modes
• Move DEBUG_LOCKS to top of file
2025-10-29 15:15:13 +08:00
yangdx
4a46d39c93 Replace GUNICORN_CMD_ARGS with custom LIGHTRAG_GUNICORN_MODE flag
• Use custom env var for mode detection
• Improve Gunicorn mode reliability
2025-10-29 14:06:03 +08:00
yangdx
816feefd84 Fix cleanup coordination between Gunicorn and UvicornWorker lifecycles
• Document UvicornWorker hook limitations
• Add GUNICORN_CMD_ARGS cleanup guard
• Prevent double cleanup in workers
2025-10-29 13:53:46 +08:00
yangdx
72b29659c9 Fix worker process cleanup to prevent shared resource conflicts
• Add worker_exit hook in gunicorn config
• Add shutdown_manager parameter in finalize_share_data of share_storage
• Prevent Manager shutdown in workers
• Remove custom signal handlers
2025-10-29 13:33:21 +08:00
yangdx
da2e9efd11 Bump API version to 0247 2025-10-29 01:39:55 +08:00
yangdx
88d12beae2 Add offline Swagger UI support with custom static file serving
- Disable default docs URL
- Add custom /docs endpoint
- Mount static Swagger UI files
- Include OAuth2 redirect handler
- Support offline documentation access
2025-10-28 02:23:08 +08:00
yangdx
5155edd8d2 feat: Improve entity merge and edit UX
- **API:** The `graph/entity/edit` endpoint now returns a detailed `operation_summary` for better client-side handling of update, rename, and merge outcomes.
- **Web UI:** Added an "auto-merge on rename" option. The UI now gracefully handles merge success, partial failures (update OK, merge fail), and other errors with specific user feedback.
2025-10-27 23:42:08 +08:00
yangdx
97034f06e3 Add allow_merge parameter to entity update API endpoint 2025-10-27 14:30:27 +08:00
yangdx
94f24a66f2 Bump API version to 0246 2025-10-27 12:28:46 +08:00
yangdx
6015e8bc68 Refactor graph utils to use unified persistence callback
- Add _persist_graph_updates function
- Remove duplicate callback functions
2025-10-26 20:20:16 +08:00
yangdx
bf1897a67e Normalize entity order for undirected graph consistency
• Normalize entity pairs for storage
• Update API docs for undirected edges
2025-10-26 15:53:31 +08:00
Anush008
8584980e3a
refactor: Qdrant Multi-tenancy (Include staged)
Signed-off-by: Anush008 <anushshetty90@gmail.com>
2025-10-26 09:58:24 +05:30
Daniel.y
c82485d94d
Merge pull request #2253 from Mobious/main
Allow users to provide keywords with QueryRequest
2025-10-25 11:26:54 +08:00
yangdx
3eb3a07544 Bump core version to 1.4.9.5 and API version to 0245 2025-10-25 04:23:57 +08:00
yangdx
78ad8873b8 Add cancellation check in delete loop 2025-10-24 14:47:20 +08:00
yangdx
743aefc655 Add pipeline cancellation feature for graceful processing termination
• Add cancel_pipeline API endpoint
• Implement PipelineCancelledException
• Add cancellation checks in main loop
• Handle task cancellation gracefully
• Mark cancelled docs as FAILED
2025-10-24 14:08:12 +08:00
Mobious
f24a261613 Allow users to provide keywords with QueryRequest 2025-10-23 12:53:19 -10:00
yangdx
fdf0fe048b Bump API version to 0244 2025-10-22 23:39:02 +08:00
yangdx
8dc23eeff2 Fix RayAnything compatible problem
• Use "preprocessed" to indicate multimodal processing is required
• Update DocProcessingStatus to process status convertion automatically
• Remove multimodal_processed from DocStatus enum value
• Update UI filter logic
2025-10-22 20:15:29 +08:00
Daniel.y
3ba1d75c97
Merge pull request #2243 from xiaojunxiang2023/main
fix(docs): correct typo "acivate" → "activate"
2025-10-22 14:39:00 +08:00
yangdx
1101562eaf Bump API version to 0243 2025-10-22 12:30:22 +08:00
yangdx
162370b6e6 Add optional LLM cache deletion when deleting documents
• Add delete_llm_cache parameter to API
• Collect cache IDs from text chunks
• Delete cache after graph operations
• Update UI with new checkbox option
• Add i18n translations for cache option
2025-10-22 12:19:23 +08:00
xiaojunxiang
04d9fe0293
Merge branch 'HKUDS:main' into main 2025-10-22 11:01:36 +08:00
xiaojunxiang
9e5004e24f fix(docs): correct typo "acivate" → "activate" 2025-10-22 03:00:47 +00:00
yangdx
637b850ec5 Add truncation indicator and update property labels in graph view
• Add truncate tooltip to source_id field
• Add visual truncation indicator (†)
• Bump API version to 0242
2025-10-20 23:03:01 +08:00
Humphry
0b3d31507e extended to use gemini, sswitched to use gemini-flash-latest 2025-10-20 13:17:16 +03:00
yangdx
dc62c78f98 Add entity/relation chunk tracking with configurable source ID limits
- Add entity_chunks & relation_chunks storage
- Implement KEEP/FIFO limit strategies
- Update env.example with new settings
- Add migration for chunk tracking data
- Support all KV storage
2025-10-20 15:24:15 +08:00
yangdx
012aaada22 Update Swagger API key status description text 2025-10-18 09:40:44 +08:00
yangdx
7bf9d1e8dc Bump API version to 0241 2025-10-17 21:19:03 +08:00
yangdx
46ac5dac53 Improve API description formatting and add ReDoc link 2025-10-17 16:24:01 +08:00
yangdx
c18762e34a Simplify Docker deployment documentation and improve clarity 2025-10-17 15:00:53 +08:00
yangdx
f45dce347a Fix cache control error of index.html
• Retrun no-cache for all HTML responses not just .html files
• Prevent force browser refresh action after front-end rebuild
2025-10-17 12:43:04 +08:00