Add test_case1_sequential_workspace_migration to verify the fix for
the multi-tenant data loss bug in PostgreSQL Case 1 migration.
Problem:
- When workspace_a migrates first (Case 4: only legacy table exists)
- Then workspace_b initializes later (Case 1: both tables exist)
- Bug: Case 1 only checked if legacy table was globally empty
- Result: workspace_b's data was not migrated, causing data loss
Test Scenario:
1. Legacy table contains data from both workspace_a (3 records) and
workspace_b (3 records)
2. workspace_a initializes first → triggers Case 4 migration
3. workspace_b initializes second → triggers Case 1 migration
4. Verify workspace_b's data is correctly migrated to new table
5. Verify workspace_b's data is deleted from legacy table
6. Verify legacy table is dropped when empty
This test uses mock tracking of inserted records to verify migration
behavior without requiring a real PostgreSQL database.
Related: GitHub PR #2391 comment #2553973066
Critical Bug Fix:
PostgreSQLDB.execute() expects data as dict, but workspace cleanup
was passing a list [workspace], causing cleanup to fail with
"PostgreSQLDB.execute() expects data as dict, got list" error.
Changes:
1. Fixed postgres_impl.py:2522
- Changed: await db.execute(delete_query, [workspace])
- To: await db.execute(delete_query, {"workspace": workspace})
2. Improved test_postgres_migration.py mock
- Enhanced COUNT(*) mock to properly distinguish between:
* Legacy table with workspace filter (returns 50)
* Legacy table without filter after deletion (returns 0)
* New table verification (returns 50)
- Uses storage.legacy_table_name dynamically instead of hardcoded strings
- Detects table type by checking for model suffix patterns
3. Fixed test_unified_lock_safety.py formatting
- Applied ruff formatting to assert statement
Impact:
- Workspace-aware legacy cleanup now works correctly
- Legacy tables properly deleted when all workspace data migrated
- Legacy tables preserved when other workspace data remains
Tests: All 25 unit tests pass
Why this change is needed:
Two critical P0 security vulnerabilities were identified in CursorReview:
1. UnifiedLock silently allows unprotected execution when lock is None, creating
false security and potential race conditions in multi-process scenarios
2. PostgreSQL migration copies ALL workspace data during legacy table migration,
violating multi-tenant isolation and causing data leakage
How it solves it:
- UnifiedLock now raises RuntimeError when lock is None instead of WARNING
- Added workspace parameter to setup_table() for proper data isolation
- Migration queries now filter by workspace in both COUNT and SELECT operations
- Added clear error messages to help developers diagnose initialization issues
Impact:
- lightrag/kg/shared_storage.py: UnifiedLock raises exception on None lock
- lightrag/kg/postgres_impl.py: Added workspace filtering to migration logic
- tests/test_unified_lock_safety.py: 3 tests for lock safety
- tests/test_workspace_migration_isolation.py: 3 tests for workspace isolation
- tests/test_dimension_mismatch.py: Updated table names and mocks
- tests/test_postgres_migration.py: Updated mocks for workspace filtering
Testing:
- All 31 tests pass (16 migration + 4 safety + 3 lock + 3 workspace + 5 dimension)
- Backward compatible: existing code continues working unchanged
- Code style verified with ruff and pre-commit hooks
Why this change is needed:
Previously, PostgreSQL and Qdrant had inconsistent migration behavior:
- PostgreSQL kept legacy tables after migration, requiring manual cleanup
- Qdrant auto-deleted legacy collections after migration
This inconsistency caused confusion for users and required different
documentation for each backend.
How it solves the problem:
Unified both backends to follow the same smart cleanup strategy:
- Case 1 (both exist): Auto-delete if legacy is empty, warn if has data
- Case 4 (migration): Auto-delete legacy after successful verification
This provides a fully automated migration experience without manual intervention.
Impact:
- Eliminates need for users to manually delete legacy tables/collections
- Reduces storage waste from duplicate data
- Provides consistent behavior across PostgreSQL and Qdrant
- Simplifies documentation and user experience
Testing:
- All 16 unit tests pass (8 PostgreSQL + 8 Qdrant)
- Added 4 new tests for Case 1 scenarios (empty vs non-empty legacy)
- Updated E2E tests to verify auto-deletion behavior
- All lint checks pass (ruff-format, ruff, trailing-whitespace)
Why this change is needed:
Unit test mock was rejecting dict parameters, but real PostgreSQLDB.execute()
accepts data as dict[str, Any]. This caused unit tests to fail after fixing
the actual migration code to pass dict instead of unpacked positional args.
How it solves it:
- Changed mock_execute signature from (sql, *args) to (sql, data=None)
- Accept dict parameter like real execute() does
- Mock now matches actual PostgreSQLDB.execute() behavior
Impact:
- Fixes Vector Storage Migration unit tests
- Mock now correctly validates migration code
Testing:
- Unit tests will verify this fix
Why this change is needed:
Offline tests were failing with "ModuleNotFoundError: No module named 'qdrant_client'"
because test_e2e_multi_instance.py was being imported during test collection, even
though it's an E2E test that shouldn't run in offline mode. Pytest imports all test
files during collection phase regardless of marks, causing import errors for missing
E2E dependencies (qdrant_client, asyncpg, etc.).
Additionally, the test mocks for PostgreSQL migration were too permissive - they
accepted any parameter format without validation, which allowed bugs (like passing
dict instead of positional args to AsyncPG execute()) to slip through undetected.
How it solves it:
1. E2E Import Fix:
- Use pytest.importorskip() to conditionally import qdrant_client
- E2E tests are now skipped cleanly when dependencies are missing
- Offline tests can collect and run without E2E dependencies
2. Stricter Test Mocks:
- Enhanced mock_pg_db fixture to validate AsyncPG parameter format
- Mock execute() now raises TypeError if dict/list passed as single argument
- Ensures tests catch parameter passing bugs that would fail in production
3. Parameter Validation Test:
- Added test_postgres_migration_params.py for explicit parameter validation
- Verifies migration passes positional args correctly to AsyncPG
- Provides detailed output for debugging parameter issues
Impact:
- Offline tests no longer fail due to missing E2E dependencies
- Future bugs in AsyncPG parameter passing will be caught by tests
- Better test isolation between offline and E2E test suites
- Improved test coverage for migration parameter handling
Testing:
- Verified with `pytest tests/ -m offline -v` - no import errors
- All PostgreSQL migration tests pass (6/6 unit + 1 strict validation)
- Pre-commit hooks pass (ruff-format, ruff)
Why this change is needed:
PostgreSQLDB class doesn't have a fetch() method. The migration code
was incorrectly using db.fetch() for batch data retrieval, causing
AttributeError during E2E tests.
How it solves it:
1. Changed db.fetch(sql, params) to db.query(sql, params, multirows=True)
2. Updated all test mocks to support the multirows parameter
3. Consolidated mock_query implementation to handle both single and multi-row queries
Impact:
- PostgreSQL legacy data migration now works correctly in E2E tests
- All unit tests pass (6/6)
- Aligns with PostgreSQLDB's actual API
Testing:
- pytest tests/test_postgres_migration.py -v (6/6 passed)
- Updated test_postgres_migration_trigger mock
- Updated test_scenario_2_legacy_upgrade_migration mock
- Updated base mock_pg_db fixture
Why this change is needed:
PostgreSQL vector storage needs model isolation to prevent dimension
conflicts when different workspaces use different embedding models.
Without this, the first workspace locks the vector dimension for all
subsequent workspaces, causing failures.
How it solves it:
- Implements dynamic table naming with model suffix: {table}_{model}_{dim}d
- Adds setup_table() method mirroring Qdrant's approach for consistency
- Implements 4-branch migration logic: both exist -> warn, only new -> use,
neither -> create, only legacy -> migrate
- Batch migration: 500 records/batch (same as Qdrant)
- No automatic rollback to support idempotent re-runs
Impact:
- PostgreSQL tables now isolated by embedding model and dimension
- Automatic data migration from legacy tables on startup
- Backward compatible: model_name=None defaults to "unknown"
- All SQL operations use dynamic table names
Testing:
- 6 new tests for PostgreSQL migration (100% pass)
- Tests cover: naming, migration trigger, scenarios 1-3
- 3 additional scenario tests added for Qdrant completeness
Co-Authored-By: Claude <noreply@anthropic.com>
