Commit graph

5257 commits

Author SHA1 Message Date
yangdx
fa5510e6f6 Fix deadlock in JSON cache migration and prevent same storage selection
- Snapshot JSON data before yielding batches
- Release lock during batch processing
- Exclude source type from target selection
- Add detailed docstring for lock behavior
- Filter available storage types properly

(cherry picked from commit 5be04263b2)
2025-12-04 19:09:03 +08:00
yangdx
5a5e583b9c Improve storage config validation and add config.ini fallback support
• Add MongoDB env requirements
• Support config.ini fallback
• Warn on missing env vars
• Check available storage count
• Show config source info

(cherry picked from commit 1a91bcdb5f)
2025-12-04 19:09:03 +08:00
domices
45c10d7f22 Fix spelling errors in the "使用PostgreSQL存储" section of README-zh.md
(cherry picked from commit 5c0ced6e4a)
2025-12-04 19:09:03 +08:00
yangdx
d1ab42bb36 Translate graph storage test from Chinese to English
(cherry picked from commit f3b2ba8152)
2025-12-04 19:09:03 +08:00
yangdx
cea34d6691 Initialize shared storage for all graph storage types in graph unit test
(cherry picked from commit 36501b82f5)
2025-12-04 19:09:03 +08:00
yangdx
7896c42fba Restructure semaphore control to manage entire evaluation pipeline
• Move rag_semaphore to wrap full function
• Increase RAG concurrency to 2x eval limit
• Prevent memory buildup from slow evals
• Keep eval_semaphore for RAGAS control

(cherry picked from commit e5abe9dd3d)
2025-12-04 19:09:02 +08:00
yangdx
c459caed26 Implement two-stage pipeline for RAG evaluation with separate semaphores
• Split RAG gen and eval stages
• Add rag_semaphore for stage 1
• Add eval_semaphore for stage 2
• Improve concurrency control
• Update connection pool limits

(cherry picked from commit 83715a3ac1)
2025-12-04 19:09:02 +08:00
ben moussa anouar
dd425e5513 Update lightrag/evaluation/eval_rag_quality.py for launguage
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
(cherry picked from commit 98f0464a31)
2025-12-04 19:09:02 +08:00
yangdx
407a2c2ecd Remove redundant shutdown message from gunicorn
(cherry picked from commit 6d4a55100e)
2025-12-04 19:09:02 +08:00
yangdx
df2c24264f Improve entity merge logging by removing redundant message and fixing typo
(cherry picked from commit 9a8742da59)
2025-12-04 19:09:02 +08:00
yangdx
cd0cd99062 Include static files in package distribution
- Add static dir to MANIFEST.in
- Update package data config
- Ensure static assets are bundled
- Fix missing static file issue

(cherry picked from commit 16d3d82a0e)
2025-12-04 19:09:02 +08:00
yangdx
2d85e9f2f8 Fix swagger docs page problem in dev mode
- Add /static to VITE_API_ENDPOINTS
- Update proxy rewrite rules
- Include static file serving
- Sync sample env file

(cherry picked from commit ee7c683fa7)
2025-12-04 19:09:02 +08:00
yangdx
8c7b0017df Remove enable_logging parameter from get_data_init_lock call in MilvusVectorDBStorage
(cherry picked from commit 0692175c7b)
2025-12-04 19:09:01 +08:00
Anush008
e86aa091f4 refactor: Qdrant Multi-tenancy (Include staged)
Signed-off-by: Anush008 <anushshetty90@gmail.com>
(cherry picked from commit 8584980e3a)
2025-12-04 19:09:01 +08:00
yangdx
a42222d7f9 Resolve lock leakage issue during user cancellation handling
• Change default log level to INFO
• Force enable error logging output
• Add lock cleanup rollback protection
• Handle LLM cache persistence errors
• Fix async task exception handling

(cherry picked from commit a9ec15e669)
2025-12-04 19:09:01 +08:00
yangdx
8b6fdef965 Optimize PostgreSQL graph queries to avoid Cypher overhead and complexity
• Replace Cypher with native SQL queries
• Fix O(N²) to O(E) performance issue
• Add error handling for parse failures
• Use direct table access pattern
• Eliminate Cartesian product joins

(cherry picked from commit a97e5dad4c)
2025-12-04 19:09:01 +08:00
yangdx
ec9b4862d0 Simplify pipeline status dialog by consolidating message sections
• Remove separate latest message section
• Combine into single pipeline messages area
• Add overflow-x-hidden for better display
• Change break-words to break-all
• Update translations across all locales

(cherry picked from commit 2476d6b7f8)
2025-12-04 19:09:01 +08:00
yangdx
e4be3549c3 Improve entity identifier truncation warning message format
(cherry picked from commit 00aa5e53a7)
2025-12-04 19:09:00 +08:00
Yasiru Rangana
8a72135a32 Optimize PostgreSQL initialization performance
- Batch index existence checks into single query (16+ queries -> 1 query)
- Batch timestamp column checks into single query (8 queries -> 1 query)
- Batch field length checks into single query (5 queries -> 1 query)

Performance improvement: ~70-80% faster initialization (35s -> 5-10s)

Key optimizations:
1. check_tables(): Use ANY($1) to check all indexes at once
2. _migrate_timestamp_columns(): Batch all column type checks
3. _migrate_field_lengths(): Batch all field definition checks

All changes are backward compatible with no schema or API changes.
Reduces database round-trips by batching information_schema queries.

(cherry picked from commit 2f22336ace)
2025-12-04 19:09:00 +08:00
yangdx
c2620efc5e Update truncation message format in properties tooltip
(cherry picked from commit 019dff5248)
2025-12-04 19:09:00 +08:00
yangdx
3780addc4c Fix logging message formatting
(cherry picked from commit e0fd31a60d)
2025-12-04 19:09:00 +08:00
Lucky Verma
12ebc9f2a9 Refactor SQL queries and improve input handling in PGKVStorage and PGDocStatusStorage
(cherry picked from commit 917e41aa78)
2025-12-04 19:09:00 +08:00
Won-Kyu Park
f4d6fcbe91 remove deprecated dotenv package.
(cherry picked from commit 532400412e)
2025-12-04 19:09:00 +08:00
yangdx
c80a9d6ef0 Remove docling dependency and related packages from project
* Remove docling from pyproject.toml
* Update requirements files
* Clean up uv.lock dependencies
* Reduce offline docker image size

(cherry picked from commit f2b6a068e3)
2025-12-04 19:09:00 +08:00
yangdx
a2d67a7c22 Add build script for multi-platform images
- Add build script for multi-platform images
- Update docker deployment document

(cherry picked from commit ef79821f29)
2025-12-04 19:08:59 +08:00
yangdx
58f818c449 Change default docker image to offline version
• Add lite verion docker image with tiktoken cache
• Update docs and build scripts

(cherry picked from commit daeca17f38)
2025-12-04 19:08:59 +08:00
yangdx
f89c7315fd Migrate Dockerfile from pip to uv package manager for faster builds
• Replace pip with uv for dependencies
• Add offline extras to Dockerfile.offline
• Update UV_LOCK_GUIDE.md with new commands
• Improve build caching and performance

(cherry picked from commit 65c2eb9f99)
2025-12-04 19:08:59 +08:00
yangdx
9229c03d40 Migrate from pip to uv package manager for faster builds
• Replace pip with uv in Dockerfile
• Remove constraints-offline.txt
• Add uv.lock for dependency pinning
• Use uv sync --frozen for builds

(cherry picked from commit 466de2070d)
2025-12-04 19:08:59 +08:00
yangdx
7e5c23e15b docs: clarify docling exclusion in offline Docker image
(cherry picked from commit 388dce2e31)
2025-12-04 19:08:59 +08:00
yangdx
69f38041cc Remove explicit protobuf dependency from offline storage requirements
(cherry picked from commit bc1a70bad0)
2025-12-04 19:08:58 +08:00
yangdx
6bd5b2d95b Add offline deployment support with cache management and layered deps
• Add tiktoken cache downloader CLI
• Add layered offline dependencies
• Add offline requirements files
• Add offline deployment guide

(cherry picked from commit a5c05f1b92)
2025-12-04 19:08:58 +08:00
yangdx
e19a4be0af Preserve ordering in get_by_ids methods across all storage implementations
- Fix result ordering in vector stores
- Update KV storage get_by_ids methods
- Maintain order in doc status storage
- Return None for missing IDs

(cherry picked from commit 9be22dd666)
2025-12-04 19:08:58 +08:00
yangdx
17106225dd Add PostgreSQL connection retry mechanism with comprehensive error handling
• Implement connection retry with backoff
• Add transient error detection
• Pool management with timeout guards

(cherry picked from commit e758204ab2)
2025-12-04 19:08:58 +08:00
yangdx
8f924d6f21 Add PostgreSQL connection retry configuration options
- Add retry environment variables
- Fix asyncpg import in retry tests

(cherry picked from commit bd535e3e7a)
2025-12-04 19:08:57 +08:00
yangdx
60a695539a Refactor PostgreSQL retry config to use centralized configuration
• Move retry config to ClientManager
• Remove env var parsing from PostgreSQLDB
• Add config params to test setup

(cherry picked from commit b3ed264707)
2025-12-04 19:08:57 +08:00
yangdx
d5154bca73 Condensed AGENTS.md to focus on essential development guidelines
(cherry picked from commit 8d3b53ce22)
2025-12-04 19:08:57 +08:00
yangdx
390842a6dd Rename Agments.md to AGENTS.md and standardize formatting
(cherry picked from commit 6e39c0c0ff)
2025-12-04 19:08:57 +08:00
yangdx
f56ba3b599 Add project intelligence files for AI agent collaboration
- Add .clinerules with technical patterns
- Create Agments.md for Codex agent guidance
- Ensures consistent behavior across all team members

(cherry picked from commit 577b9e6882)
2025-12-04 19:08:57 +08:00
yangdx
c6433edb23 Make PostgreSQL statement_cache_size configuration optional
• Remove forced int conversion
• Allow None values for cache size
• Add conditional parameter setting

(cherry picked from commit f2c0b41e78)
2025-12-04 19:08:57 +08:00
kevinnkansah
c8c73ab114 fix: renamed PostGreSQL options env variable and allowed LRU cache to be an optional env variable
(cherry picked from commit 22a7b482c5)
2025-12-04 19:08:56 +08:00
kevinnkansah
7ce46bacb6 feat: add options for PostGres connection
(cherry picked from commit 108cdbe133)
2025-12-04 19:08:56 +08:00
yangdx
fe05563ecb Remove future dependency and replace passlib with direct bcrypt
(cherry picked from commit fc44f11368)
2025-12-04 19:08:56 +08:00
yangdx
ad6b36143e Update .env loading and add API authentication to RAG evaluator
• Load .env from current directory
• Support LIGHTRAG_API_KEY auth header
• Override=False for env precedence
• Add Bearer token to API requests
• Enable per-instance .env configs

(cherry picked from commit 72db042667)
2025-12-04 19:08:56 +08:00
yangdx
6de4bb9113 Fix logging message formatting
(cherry picked from commit e0fd31a60d)
2025-12-04 19:08:46 +08:00
Lucky Verma
80dcbc696a Refactor SQL queries and improve input handling in PGKVStorage and PGDocStatusStorage
(cherry picked from commit 917e41aa78)
2025-12-04 19:08:41 +08:00
Won-Kyu Park
dd5b220e58 remove deprecated dotenv package.
(cherry picked from commit 532400412e)
2025-12-04 19:08:40 +08:00
yangdx
f142a8c375 Remove docling dependency and related packages from project
* Remove docling from pyproject.toml
* Update requirements files
* Clean up uv.lock dependencies
* Reduce offline docker image size

(cherry picked from commit f2b6a068e3)
2025-12-04 19:08:28 +08:00
yangdx
9a23234c6c Add build script for multi-platform images
- Add build script for multi-platform images
- Update docker deployment document

(cherry picked from commit ef79821f29)
2025-12-04 19:08:21 +08:00
yangdx
aa61e82820 Migrate from pip to uv package manager for faster builds
• Replace pip with uv in Dockerfile
• Remove constraints-offline.txt
• Add uv.lock for dependency pinning
• Use uv sync --frozen for builds

(cherry picked from commit 466de2070d)
2025-12-04 19:08:10 +08:00
yangdx
8c3a325193 docs: clarify docling exclusion in offline Docker image
(cherry picked from commit 388dce2e31)
2025-12-04 19:07:59 +08:00