Commit graph

3373 commits

Author SHA1 Message Date
yangdx
7cddd56461 Revert core version to 1.4.9..2 2025-10-11 13:06:58 +08:00
yangdx
9be22dd666 Preserve ordering in get_by_ids methods across all storage implementations
- Fix result ordering in vector stores
- Update KV storage get_by_ids methods
- Maintain order in doc status storage
- Return None for missing IDs
2025-10-11 12:37:59 +08:00
yangdx
a5c05f1b92 Add offline deployment support with cache management and layered deps
• Add tiktoken cache downloader CLI
• Add layered offline dependencies
• Add offline requirements files
• Add offline deployment guide
2025-10-11 10:28:14 +08:00
yangdx
12facac506 Enhance graph API endpoints with detailed docs and field validation
- Remove redundant README section
- Add Pydantic field validation
- Expand endpoint docstrings
- Include request/response examples
- Document merge operation benefits
2025-10-10 12:49:00 +08:00
yangdx
85d1a563b3 Merge branch 'adminunblinded/main' 2025-10-10 12:31:47 +08:00
NeelM0906
b7c77396a0 Fix entity/relation creation endpoints to properly update vector stores
- Changed create_entity to use rag.acreate_entity() instead of direct graph manipulation
  - Changed create_relation to use rag.acreate_relation() instead of direct graph manipulation
  - This ensures vector embeddings are created and entities/relations are searchable
  - Adds proper concurrency locks and metadata population
2025-10-09 17:02:17 -04:00
NeelM0906
f6d1fb98ac Fix Linting errors 2025-10-09 16:52:22 -04:00
yangdx
b3ed264707 Refactor PostgreSQL retry config to use centralized configuration
• Move retry config to ClientManager
• Remove env var parsing from PostgreSQLDB
• Add config params to test setup
2025-10-10 03:44:13 +08:00
yangdx
e758204ab2 Add PostgreSQL connection retry mechanism with comprehensive error handling
• Implement connection retry with backoff
• Add transient error detection
• Pool management with timeout guards
2025-10-10 03:06:01 +08:00
Yasiru Rangana
ae9f4ae73f fix: Remove trailing whitespace for pre-commit linting 2025-10-09 15:01:53 +11:00
NeelM0906
9f44e89de7 Add knowledge graph manipulation endpoints
Added three new REST API endpoints for direct knowledge graph manipulation:

- POST /graph/entity/create: Create new entities in the knowledge graph
- POST /graph/relation/create: Create relationships between entities
- POST /graph/entities/merge: Merge duplicate/misspelled entities while preserving relationships

The merge endpoint is particularly useful for consolidating entities discovered after document processing, fixing spelling errors, and cleaning up the knowledge graph. All relationships from source entities are transferred to the target entity, with intelligent handling of duplicate relationships.

Updated API documentation in lightrag/api/README.md with usage examples for all three endpoints.
2025-10-08 15:59:47 -04:00
Yasiru Rangana
ec40b17eea feat: Add token tracking support to openai_embed function
- Add optional token_tracker parameter to openai_embed()
- Track prompt_tokens and total_tokens for embedding API calls
- Enables monitoring of embedding token usage alongside LLM calls
- Maintains backward compatibility with existing code
2025-10-08 14:36:08 +11:00
yangdx
f1e0110716 Merge branch 'kevinnkansah/main' 2025-10-07 23:04:59 +08:00
yangdx
f2c0b41e78 Make PostgreSQL statement_cache_size configuration optional
• Remove forced int conversion
• Allow None values for cache size
• Add conditional parameter setting
2025-10-07 22:57:21 +08:00
Aleks Vujić
dd8f44e621
Fixed typo in log message when creating new graph file 2025-10-07 14:30:05 +02:00
kevinnkansah
fdcb034da0 chore: distinguish settings 2025-10-06 12:01:40 +02:00
kevinnkansah
22a7b482c5 fix: renamed PostGreSQL options env variable and allowed LRU cache to be an optional env variable 2025-10-06 11:56:09 +02:00
kevinnkansah
d8a9617c0e fix: fix: asyncpg bouncer connection pool error
Prepared statement caching is disabled by setting
`statement_cache_size=0` in the `asyncpg` connection pool parameters.
This is necessary to prevent
`asyncpg.exceptions.InvalidSQLStatementNameError` when using
transaction-level connection poolers like Supabase Supavisor or
pgbouncer, which do not support prepared statements.
2025-10-06 00:36:25 +02:00
kevinnkansah
108cdbe133 feat: add options for PostGres connection 2025-10-05 23:29:04 +02:00
yangdx
d473f635d8 Update webui assets 2025-10-05 14:02:18 +08:00
yangdx
aac787bafb Clarify chunk tracking log message in _build_llm_context 2025-10-05 13:33:55 +08:00
yangdx
457d51952e Add doc_name field to full docs storage
- Store file_path in full_docs storage
- Update PostgreSQL implementation by map file_path to doc_name
- Other storage implementation automatically handles the new field
2025-10-05 11:44:27 +08:00
yangdx
1574fec7f0 Update webui assets 2025-10-05 10:41:46 +08:00
yangdx
b5f8376756 Update webui assets 2025-10-05 09:27:38 +08:00
yangdx
dde728a32f Bump core version to 1.5.0 and API to 0236 2025-10-05 09:25:57 +08:00
Jon
cf2a024e37 feat: Add endpoint and UI to retry failed documents
Add a new `/documents/reprocess_failed` API endpoint and corresponding
UI button to retry processing of failed and pending documents. This
addresses a common recovery scenario when document processing fails due
to server crashes, network errors, or LLM service outages.

Backend changes:
- Add ReprocessResponse model with status, message, and track_id fields
- Add POST /documents/reprocess_failed endpoint that triggers background
  reprocessing of FAILED, PENDING, and interrupted PROCESSING documents
- Reuses existing apipeline_process_enqueue_documents for consistency
- Includes comprehensive docstring and logging for observability

Frontend changes:
- Add TypeScript types and API function for the new endpoint
- Add retry handler with intelligent polling (fast refresh → normal)
- Add "Retry Failed" button in Documents page toolbar
- Button disabled when pipeline is busy to prevent duplicate operations
- Complete i18n support (English and Chinese translations)

This feature provides a convenient way to recover from processing
failures without requiring a full filesystem rescan.
2025-10-04 16:46:29 -04:00
yangdx
b9c37bd937 Fix linting 2025-10-03 02:10:02 +08:00
yangdx
112349ed5b Modernize type hints and remove Python 3.8 compatibility code
• Use collections.abc.AsyncIterator only
• Remove sys.version_info checks
• Use union syntax for None types
• Simplify string emptiness checks
2025-10-02 23:15:42 +08:00
yangdx
cec784f60e Update webui assets 2025-10-02 22:02:42 +08:00
yangdx
a250d881f2 Update webui assets 2025-10-02 21:24:39 +08:00
yangdx
6bf6f43d96 Remove bold formatting from instruction headers in prompts 2025-10-02 00:58:03 +08:00
yangdx
bb6138e748 fix(prompt): Clarify reference section restrictions in prompt templates 2025-10-01 22:35:26 +08:00
yangdx
37e8898cf6 Simplify reference formatting in LLM context generation
- Remove extra newlines in reference lists
- Change code block type from text to generic
2025-10-01 22:20:58 +08:00
yangdx
f83cde14df fix(prompt): Improve markdown formatting requirements and reference style 2025-10-01 21:41:12 +08:00
yangdx
83d99e1424 fix(OllamaAPI): Add validation to ensure last message is from user role
• Validate last message role is "user"
• Raise 400 error for invalid role
• Improve API request validation
• Prevent invalid message sequences
2025-10-01 20:48:37 +08:00
yangdx
df43afc89b Relax conversation history role validation requirements
• Remove strict role value checking
• Allow any non-empty string roles
2025-09-29 13:10:15 +08:00
yangdx
ba216787c1 Update webui assets 2025-09-28 22:51:06 +08:00
yangdx
6e3e67fc24 Update webui assets 2025-09-28 21:34:09 +08:00
yangdx
81bc1ba025 Update webui assets and bump API version to 0235 2025-09-28 21:05:40 +08:00
yangdx
42d1d04147 Fix boolean parser problem for for LLM environment variable
• Add custom boolean parser for argparse in BindingOptions
2025-09-28 19:23:57 +08:00
yangdx
4957f4f017 Bump core version to 1.4.9.1 2025-09-28 15:25:02 +08:00
yangdx
978f154680 Update webui assets and bump API version to 0234 2025-09-28 15:14:05 +08:00
yangdx
0fd0186414 Improve prompt clarity by standardizing terminology and formatting
• Replace "Source Data" with "Context"
• Add bold formatting for key sections
• Clarify reference_id usage
• Improve JSON/text block formatting
• Standardize data source naming
2025-09-28 13:31:55 +08:00
yangdx
7cba458f22 Limit deprecated documents endpoint to 1000 records with fair distribution 2025-09-28 11:18:10 +08:00
yangdx
fcbab77613 docs: add API key usage example and security guidance 2025-09-28 10:56:52 +08:00
yangdx
2ce6a022ac Fix documentation for user_prompt parameter in QueryParam 2025-09-27 23:41:17 +08:00
yangdx
91be53ffd2 Fix linting 2025-09-27 22:36:38 +08:00
yangdx
e0ac05db90 Simplify query route documentation and clarify conversation history 2025-09-27 22:36:16 +08:00
yangdx
f66a0aad8b Update query streaming endpoint docs to clarify behavior 2025-09-27 22:27:49 +08:00
yangdx
3ca2165c69 Bump API version to 0233 2025-09-27 15:31:30 +08:00