gmakstutis/LightRAG
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
5949 commits 5 branches 72 tags 705 MiB
Commit graph

3744 commits

Author SHA1 Message Date
yangdx
f254ccae8e Merge branch 'fix/missing-langue-in-keywords-extraction-prompt' 2025-12-12 10:11:56 +08:00
yangdx
834778eb01 Reorganize entity extraction prompt for better clarity 
- Move instructions before data section
- Update task description wording
 2025-12-12 06:12:47 +08:00
José Carlos
d8cd48f43b fix(cache): include language parameter in keywords extraction cache key 2025-12-11 12:43:42 -03:00
yangdx
4fd3914194 Merge branch 'main' into fix/openai-prompt-caching-optimization-2355 2025-12-11 19:12:42 +08:00
yangdx
294f75438e Restructure entity extraction prompt format for consistency 
• Move entity_types to user prompt
• Add XML-style formatting tags
• Update examples with entity_types
 2025-12-11 19:12:34 +08:00
José Carlos
fd2ff358bf fix(prompt): use language parameter in keywords_extraction prompt 
The language parameter was being passed to the prompt but was not
being used. Added {language} placeholder to ensure keywords are
extracted in the configured language.
 2025-12-06 09:50:12 -03:00
yangdx
9009abed3e Fix top_n behavior with chunking to limit documents not chunks 
- Disable API-level top_n when chunking
- Apply top_n to aggregated documents
- Add comprehensive test coverage
 2025-12-03 13:08:26 +08:00
yangdx
561ba4e4b5 Fix trailing whitespace and update test mocking for rerank module 
• Remove trailing whitespace
• Fix TiktokenTokenizer import patch
• Add async context manager mocks
• Update aiohttp.ClientSession patch
• Improve test reliability
 2025-12-03 12:40:48 +08:00
yangdx
8e50eef58b Merge branch 'main' into cohere-rerank 2025-12-02 22:19:37 +08:00
yangdx
19c16bc464 Add content deduplication check for document insertion endpoints 
• Check content hash before insertion
• Return duplicated status if exists
• Use sanitized text for hash computation
• Apply to both single and batch inserts
• Prevent duplicate content processing
 2025-12-02 17:49:48 +08:00
yangdx
8d28b95966 Fix duplicate document responses to return original track_id 
- Return existing track_id for duplicates
- Remove track_id generation in reprocess
- Update reprocess response documentation
- Clarify track_id behavior in comments
- Update API response examples
 2025-12-02 14:32:28 +08:00
yangdx
381ddfffd4 Bump API version to 0259 2025-12-02 13:27:02 +08:00
yangdx
2ecf77efe2 Update help text to use correct gunicorn command with workers flag 2025-12-02 02:52:31 +08:00
yangdx
d6019c82af Add CASCADE to AGE extension creation in PostgreSQL implementation 
- Add CASCADE option to CREATE EXTENSION
- Ensure dependencies are installed
- Fix potential AGE setup issues
 2025-12-02 00:17:41 +08:00
yangdx
112ed234c4 Bump API version to 0258 2025-12-01 12:20:27 +08:00
yangdx
ea8d55ab42 Add documentation for embedding provider configuration rules 2025-11-28 17:49:30 +08:00
yangdx
4ab4a7ac94 Allow embedding models to use provider defaults when unspecified 
- Set EMBEDDING_MODEL default to None
- Pass model param only when provided
- Let providers use their own defaults
- Fix lollms embed function params
- Add ollama embed_model default param
 2025-11-28 16:57:33 +08:00
yangdx
881b8d3a50 Bump API version to 0257 2025-11-28 15:39:55 +08:00
yangdx
56e0365cf0 Add configurable model parameter to jina_embed function 
- Add model parameter to jina_embed
- Pass model from API server
- Default to jina-embeddings-v4
- Update function documentation
- Make model selection flexible
 2025-11-28 15:38:29 +08:00
yangdx
6e2946e78a Add max_token_size parameter to azure_openai_embed wrapper 2025-11-28 13:41:01 +08:00
yangdx
4f12fe121d Change entity extraction logging from warning to info level 
• Reduce log noise for empty entities
 2025-11-27 11:00:34 +08:00
Ghazi-raad
4e8e08cf4d
Update lightrag/operate.py 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-11-26 23:18:20 +00:00
Ghazi-raad
56677ae466
Update lightrag/prompt.py 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-11-26 23:18:12 +00:00
Ghazi-raad
207af40f54 Optimize for OpenAI Prompt Caching: Restructure entity extraction prompts 
- Remove input_text from entity_extraction_system_prompt to enable caching
- Move input_text to entity_extraction_user_prompt for per-chunk variability
- Update operate.py to format system prompt once without input_text
- Format user prompts with input_text for each chunk

This enables OpenAI's automatic prompt caching (50% discount on cached tokens):
- ~1300 token system message cached and reused for ALL chunks
- Only ~150 token user message varies per chunk
- Expected 45% cost reduction on prompt tokens during indexing
- 2-3x faster response times from cached prompts

Fixes #2355
 2025-11-26 21:56:25 +00:00
palanisd
a898f0548d
Merge branch 'HKUDS:main' into cohere-rerank 2025-11-25 14:21:43 -05:00
yangdx
93d445dfdd Add pipeline status lock function for legacy compatibility 
- Add get_pipeline_status_lock function
- Return NamespaceLock for consistency
- Support workspace parameter
- Enable logging option
- Legacy code compatibility
 2025-11-25 18:24:39 +08:00
EightyOliveira
8994c70f2f fix:exception handling order error 2025-11-25 16:36:41 +08:00
yangdx
48b67d3077 Handle missing WebUI assets gracefully without blocking server startup 
- Change build check from error to warning
- Redirect to /docs when WebUI unavailable
- Add webui_available to health endpoint
- Only mount /webui if assets exist
- Return status tuple from build check
 2025-11-25 02:51:55 +08:00
yangdx
8c4d7a00ad Refactor: Extract retry decorator to reduce code duplication in Neo4J storage 
• Define READ_RETRY_EXCEPTIONS constant
• Create reusable READ_RETRY decorator
• Replace 11 duplicate retry decorators
• Improve code maintainability
• Add missing retry to edge_degrees_batch
 2025-11-25 01:35:21 +08:00
yangdx
7aaa51cda9 Add retry decorators to Neo4j read operations for resilience 2025-11-24 22:28:15 +08:00
copilot-swe-agent[bot]
8835fc244a Improve edge case handling for max_tokens=1 
Co-authored-by: netbrah <162479981+netbrah@users.noreply.github.com>
 2025-11-24 03:43:05 +00:00
copilot-swe-agent[bot]
1d6ea0c5f7 Fix chunking infinite loop when overlap_tokens >= max_tokens 
Co-authored-by: netbrah <162479981+netbrah@users.noreply.github.com>
 2025-11-24 03:40:58 +00:00
netbrah
a05bbf105e Add Cohere reranker config, chunking, and tests 2025-11-22 16:43:13 -05:00
yangdx
7b76211066 Add fallback to AZURE_OPENAI_API_VERSION for embedding API version 2025-11-22 00:14:35 +08:00
yangdx
ffd8da512e Improve Azure OpenAI compatibility and error handling 
• Reduce log noise for Azure content filters
• Add default API version fallback
• Change warning to debug log level
• Handle empty choices in streaming
• Better Azure OpenAI integration
 2025-11-21 23:51:18 +08:00
yangdx
fafa1791f4 Fix Azure OpenAI model parameter to use deployment name consistently 
- Use deployment name for Azure API calls
- Fix model param in embed function
- Consistent api_model logic
- Prevent Azure model name conflicts
 2025-11-21 23:41:52 +08:00
yangdx
ac9f2574a5 Improve Azure OpenAI wrapper functions with full parameter support 
• Add missing parameters to wrappers
• Update docstrings for clarity
• Ensure API consistency
• Fix parameter forwarding
• Maintain backward compatibility
 2025-11-21 19:24:32 +08:00
yangdx
45f4f82392 Refactor Azure OpenAI client creation to support client_configs merging 
- Handle None client_configs case
- Merge configs with explicit params
- Override client_configs with params
- Use dict unpacking for client init
- Maintain parameter precedence
 2025-11-21 19:14:16 +08:00
yangdx
0c4cba3860 Fix double decoration in azure_openai_embed and document decorator usage 
• Remove redundant @retry decorator
• Call openai_embed.func directly
• Add detailed decorator documentation
• Prevent double parameter injection
• Fix EmbeddingFunc wrapping issues
 2025-11-21 18:03:53 +08:00
yangdx
b46c152306 Fix linting 2025-11-21 17:16:44 +08:00
yangdx
b709f8f869 Consolidate Azure OpenAI implementation into main OpenAI module 
• Unified OpenAI/Azure client creation
• Azure module now re-exports functions
• Backward compatibility maintained
• Reduced code duplication
 2025-11-21 17:12:33 +08:00
yangdx
66d6c7dd6f Refactor main function to provide sync CLI entry point 2025-11-21 13:11:55 +08:00
yangdx
02fdceb959 Update OpenAI client to use stable API and bump minimum version to 2.0.0 
- Remove beta prefix from completions.parse
- Update OpenAI dependency to >=2.0.0
- Fix whitespace formatting
- Update all requirement files
- Clean up pyproject.toml dependencies
 2025-11-21 12:55:44 +08:00
yangdx
9f69c5bf85 feat: Support structured output parsed from OpenAI 
Added support for structured output (JSON mode) from the OpenAI API in `openai.py` and `azure_openai.py`.

When `response_format` is used to request structured data, the new logic checks for the `message.parsed` attribute. If it exists, it's serialized into a JSON string as the final content. If not, the code falls back to the existing `message.content` handling, ensuring backward compatibility.
 2025-11-21 12:46:31 +08:00
yangdx
c9e1c86e81 Refactor keyword extraction handling to centralize response format logic 
• Move response format to core function
• Remove duplicate format assignments
• Standardize keyword extraction flow
• Clean up redundant parameter handling
• Improve Azure OpenAI compatibility
 2025-11-21 12:10:04 +08:00
yangdx
46ce6d9a13 Fix Azure OpenAI embedding model parameter fallback 
- Use model param if provided
- Fall back to deployment name
- Fix embedding API call
- Improve parameter handling
 2025-11-20 18:20:22 +08:00
Amritpal Singh
30e86fa331 use deployment variable which extracted value from .env file or have default value 2025-11-20 09:00:27 +00:00
yangdx
b7de694f48 Add comprehensive error logging across API routes 
- Add error logs to Ollama API endpoints
- Replace logging with unified logger
- Log streaming query errors
- Add data query error logging
- Include stack traces for debugging
 2025-11-19 22:50:06 +08:00
yangdx
0fb2925c6a Remove ascii_colors dependency and fix stream handling errors 
• Remove ascii_colors.trace_exception calls
• Add SafeStreamHandler for closed streams
• Patch ascii_colors console handler
• Prevent ValueError on stream close
• Improve logging error handling
 2025-11-19 21:38:17 +08:00
yangdx
6fea68bff9 Fix ChunkTokenLimitExceededError message formatting 
- Prevent passes two separate string objects to __init__
- Maintain same error output
 2025-11-19 18:50:45 +08:00