The language parameter was being passed to the prompt but was not
being used. Added {language} placeholder to ensure keywords are
extracted in the configured language.
• Check content hash before insertion
• Return duplicated status if exists
• Use sanitized text for hash computation
• Apply to both single and batch inserts
• Prevent duplicate content processing
- Set EMBEDDING_MODEL default to None
- Pass model param only when provided
- Let providers use their own defaults
- Fix lollms embed function params
- Add ollama embed_model default param
- Add model parameter to jina_embed
- Pass model from API server
- Default to jina-embeddings-v4
- Update function documentation
- Make model selection flexible
- Remove input_text from entity_extraction_system_prompt to enable caching
- Move input_text to entity_extraction_user_prompt for per-chunk variability
- Update operate.py to format system prompt once without input_text
- Format user prompts with input_text for each chunk
This enables OpenAI's automatic prompt caching (50% discount on cached tokens):
- ~1300 token system message cached and reused for ALL chunks
- Only ~150 token user message varies per chunk
- Expected 45% cost reduction on prompt tokens during indexing
- 2-3x faster response times from cached prompts
Fixes#2355
Why this change is needed:
1. Added clarifying comments to _pg_migrate_workspace_data() parameter handling
2. Removed dead code from PGDocStatusStorage.initialize() that was never executed
Changes:
1. PostgreSQL Migration Parameter Documentation (lightrag/kg/postgres_impl.py:2240-2241):
- Added comments explaining dict rebuild for correct value ordering
- Clarifies that Python 3.7+ dict insertion order is relied upon
- Documents that execute() converts dict to tuple via .values()
2. Dead Code Removal (lightrag/kg/postgres_impl.py:3061-3062):
- Removed unreachable table creation code from PGDocStatusStorage.initialize()
- Table is already created by PostgreSQLDB.initdb() during initialization
- This code path was never executed as table always exists before initialize() is called
- Added NOTE comment explaining where table creation actually happens
Impact:
- No functional changes - only code clarification and cleanup
- Reduces maintenance burden by removing unreachable code
- Improves code readability with better documentation
Testing:
- All 14 PostgreSQL migration tests pass
- All 5 UnifiedLock safety tests pass
- Pre-commit checks pass (ruff-format, ruff)
Why this change is needed:
In multi-tenant deployments, when workspace A migrates first (creating
the new model-suffixed table), subsequent workspace B initialization
enters Case 1 (both tables exist). The original Case 1 logic only
checked if the legacy table was empty globally, without checking if
the current workspace had unmigrated data. This caused workspace B's
data to remain in the legacy table while the application queried the
new table, resulting in data loss for workspace B.
How it solves the problem:
1. Extracted migration logic into _pg_migrate_workspace_data() helper
function to avoid code duplication
2. Modified Case 1 to check if current workspace has data in legacy
table and migrate it if found
3. Both Case 1 and Case 4 now use the same migration helper, ensuring
consistent behavior
4. After migration, only delete the current workspace's data from
legacy table, preserving other workspaces' data
Impact:
- Prevents data loss in multi-tenant PostgreSQL deployments
- Maintains backward compatibility with single-tenant setups
- Reduces code duplication between Case 1 and Case 4
Testing:
All PostgreSQL migration tests pass (8/8)
- Change build check from error to warning
- Redirect to /docs when WebUI unavailable
- Add webui_available to health endpoint
- Only mount /webui if assets exist
- Return status tuple from build check
Critical Bug Fix:
PostgreSQLDB.execute() expects data as dict, but workspace cleanup
was passing a list [workspace], causing cleanup to fail with
"PostgreSQLDB.execute() expects data as dict, got list" error.
Changes:
1. Fixed postgres_impl.py:2522
- Changed: await db.execute(delete_query, [workspace])
- To: await db.execute(delete_query, {"workspace": workspace})
2. Improved test_postgres_migration.py mock
- Enhanced COUNT(*) mock to properly distinguish between:
* Legacy table with workspace filter (returns 50)
* Legacy table without filter after deletion (returns 0)
* New table verification (returns 50)
- Uses storage.legacy_table_name dynamically instead of hardcoded strings
- Detects table type by checking for model suffix patterns
3. Fixed test_unified_lock_safety.py formatting
- Applied ruff formatting to assert statement
Impact:
- Workspace-aware legacy cleanup now works correctly
- Legacy tables properly deleted when all workspace data migrated
- Legacy tables preserved when other workspace data remains
Tests: All 25 unit tests pass
This commit fixes two critical issues in PostgreSQL storage:
BUG 1: Legacy table cleanup causing data loss across workspaces
---------------------------------------------------------------
PROBLEM:
- After migrating workspace_a data from legacy table, the ENTIRE legacy
table was deleted
- This caused workspace_b's data (still in legacy table) to be lost
- Multi-tenant data isolation was violated
FIX:
- Implement workspace-aware cleanup: only delete migrated workspace's data
- Check if other workspaces still have data before dropping table
- Only drop legacy table when it becomes completely empty
- If other workspace data exists, preserve legacy table with remaining records
Location: postgres_impl.py PGVectorStorage.setup_table() lines 2510-2567
Test verification:
- test_workspace_migration_isolation_e2e_postgres validates this fix
BUG 2: PGDocStatusStorage missing table initialization
-------------------------------------------------------
PROBLEM:
- PGDocStatusStorage.initialize() only set workspace, never created table
- Caused "relation 'lightrag_doc_status' does not exist" errors
- document insertion (ainsert) failed immediately
FIX:
- Add table creation to initialize() method using _pg_create_table()
- Consistent with other storage implementations:
* MongoDocStatusStorage creates collections
* JsonDocStatusStorage creates directories
* PGDocStatusStorage now creates tables ✓
Location: postgres_impl.py PGDocStatusStorage.initialize() lines 2965-2971
Test Results:
- Unit tests: 13/13 passed (test_unified_lock_safety,
test_workspace_migration_isolation, test_dimension_mismatch)
- E2E tests require PostgreSQL server
Related: PR #2391 (Vector Storage Model Isolation)
Problem:
When UnifiedLock.__aexit__ encountered an exception during async_lock.release(),
the error recovery logic would incorrectly attempt to release async_lock again
because it only checked main_lock_released flag. This could cause:
- Double-release attempts on already-failed locks
- Masking of original exceptions
- Undefined behavior in lock state
Root Cause:
The recovery logic used only main_lock_released to determine whether to attempt
async_lock release, without tracking whether async_lock.release() had already
been attempted and failed.
Fix:
- Added async_lock_released flag to track async_lock release attempts
- Updated recovery logic condition to check both main_lock_released AND
async_lock_released before attempting async_lock release
- This ensures async_lock.release() is only called once, even if it fails
Testing:
- Added test_aexit_no_double_release_on_async_lock_failure:
Verifies async_lock.release() is called only once when it fails
- Added test_aexit_recovery_on_main_lock_failure:
Verifies recovery logic still works when main lock fails
- All 5 UnifiedLock safety tests pass
Impact:
- Eliminates double-release bugs in multiprocess lock scenarios
- Preserves correct error propagation
- Maintains recovery logic for legitimate failure cases
Files Modified:
- lightrag/kg/shared_storage.py: Added async_lock_released tracking
- tests/test_unified_lock_safety.py: Added 2 new tests (5 total now pass)
Why this change is needed:
Two critical P0 security vulnerabilities were identified in CursorReview:
1. UnifiedLock silently allows unprotected execution when lock is None, creating
false security and potential race conditions in multi-process scenarios
2. PostgreSQL migration copies ALL workspace data during legacy table migration,
violating multi-tenant isolation and causing data leakage
How it solves it:
- UnifiedLock now raises RuntimeError when lock is None instead of WARNING
- Added workspace parameter to setup_table() for proper data isolation
- Migration queries now filter by workspace in both COUNT and SELECT operations
- Added clear error messages to help developers diagnose initialization issues
Impact:
- lightrag/kg/shared_storage.py: UnifiedLock raises exception on None lock
- lightrag/kg/postgres_impl.py: Added workspace filtering to migration logic
- tests/test_unified_lock_safety.py: 3 tests for lock safety
- tests/test_workspace_migration_isolation.py: 3 tests for workspace isolation
- tests/test_dimension_mismatch.py: Updated table names and mocks
- tests/test_postgres_migration.py: Updated mocks for workspace filtering
Testing:
- All 31 tests pass (16 migration + 4 safety + 3 lock + 3 workspace + 5 dimension)
- Backward compatible: existing code continues working unchanged
- Code style verified with ruff and pre-commit hooks
Why this change is needed:
Two critical issues were identified in Codex review of PR #2391:
1. Migration fails when legacy collections/tables use different embedding dimensions
(e.g., upgrading from 1536d to 3072d models causes initialization failures)
2. When model_suffix is empty (no model_name provided), table_name equals legacy_table_name,
causing Case 1 logic to delete the only table/collection on second startup
How it solves it:
- Added dimension compatibility checks before migration in both Qdrant and PostgreSQL
- PostgreSQL uses two-method detection: pg_attribute metadata query + vector sampling fallback
- When dimensions mismatch, skip migration and create new empty table/collection, preserving legacy data
- Added safety check to detect when new and legacy names are identical, preventing deletion
- Both backends log clear warnings about dimension mismatches and skipped migrations
Impact:
- lightrag/kg/qdrant_impl.py: Added dimension check (lines 254-297) and no-suffix safety (lines 163-169)
- lightrag/kg/postgres_impl.py: Added dimension check with fallback (lines 2347-2410) and no-suffix safety (lines 2281-2287)
- tests/test_no_model_suffix_safety.py: New test file with 4 test cases covering edge scenarios
- Backward compatible: All existing scenarios continue working unchanged
Testing:
- All 20 tests pass (16 existing migration tests + 4 new safety tests)
- E2E tests enhanced with explicit verification points for dimension mismatch scenarios
- Verified graceful degradation when dimension detection fails
- Code style verified with ruff and pre-commit hooks
