Commit graph

5772 commits

Author SHA1 Message Date
yangdx
87de2b3e9e Update XLSX extraction documentation to reflect current implementation 2025-11-19 04:26:41 +08:00
yangdx
0244699d81 Optimize XLSX extraction by using sheet.max_column instead of two-pass scan
• Remove two-pass row scanning approach
• Use built-in sheet.max_column property
• Simplify column width detection logic
• Improve memory efficiency
• Maintain column alignment preservation
2025-11-19 04:02:39 +08:00
yangdx
2b16016312 Optimize XLSX extraction to avoid storing all rows in memory
• Remove intermediate row storage
• Use iterator twice instead of list()
• Preserve column alignment logic
• Reduce memory footprint
• Maintain same output format
2025-11-19 03:48:36 +08:00
yangdx
ef659a1e09 Preserve column alignment in XLSX extraction with two-pass processing
• Two-pass approach for consistent width
• Maintain tabular structure integrity
• Determine max columns first pass
• Extract with alignment second pass
• Prevent column misalignment issues
2025-11-19 03:34:22 +08:00
yangdx
3efb1716b4 Enhance XLSX extraction with structured tab-delimited format and escaping
- Add clear sheet separators
- Escape special characters
- Trim trailing empty columns
- Preserve row structure
- Single-pass optimization
2025-11-19 03:06:29 +08:00
Daniel.y
efbbaaf7f9
Merge pull request #2383 from danielaskdd/doc-table
Feat: Enhanced DOCX Extraction with Table Content Support
2025-11-19 02:26:02 +08:00
yangdx
e7d2803a65 Remove text stripping in DOCX extraction to preserve whitespace
• Keep original paragraph spacing
• Preserve cell whitespace in tables
• Maintain document formatting
• Don't strip leading/trailing spaces
2025-11-19 02:12:27 +08:00
yangdx
186c8f0e16 Preserve blank paragraphs in DOCX extraction to maintain spacing
• Remove text emptiness check
• Always append paragraph text
• Maintain document formatting
• Preserve original spacing
2025-11-19 02:03:10 +08:00
yangdx
fa887d811b Fix table column structure preservation in DOCX extraction
• Always append cell text to maintain columns
• Preserve empty cells in table structure
• Check for any content before adding rows
• Use tab separation for proper alignment
• Improve table formatting consistency
2025-11-19 01:52:02 +08:00
yangdx
4438ba41a3 Enhance DOCX extraction to preserve document order with tables
• Include tables in extracted content
• Maintain original document order
• Add spacing around tables
• Use tabs to separate table cells
• Process all body elements sequentially
2025-11-19 01:31:33 +08:00
yangdx
d16c7840ab Bump API version to 0256 2025-11-18 23:15:31 +08:00
yangdx
e77340d4a1 Adjust chunking parameters to match the default environment variable settings 2025-11-18 23:14:50 +08:00
yangdx
24423c9215 Merge branch 'fix_chunk_comment' 2025-11-18 22:47:23 +08:00
yangdx
1bfa1f81cb Merge branch 'main' into fix_chunk_comment 2025-11-18 22:38:50 +08:00
yangdx
9c10c87554 Fix linting 2025-11-18 22:38:43 +08:00
yangdx
9109509b1a Merge branch 'dev-postgres-vchordrq' 2025-11-18 22:25:35 +08:00
yangdx
dbae327a17 Merge branch 'main' into dev-postgres-vchordrq 2025-11-18 22:13:27 +08:00
yangdx
b583b8a59d Merge branch 'feature/postgres-vchordrq-indexes' into dev-postgres-vchordrq 2025-11-18 22:05:48 +08:00
yangdx
3096f844fb fix(postgres): allow vchordrq.epsilon config when probes is empty
Previously, configure_vchordrq would fail silently when probes was empty
(the default), preventing epsilon from being configured. Now each parameter
is handled independently with conditional execution, and configuration
errors fail-fast instead of being swallowed.

This fixes the documented epsilon setting being impossible to use in the
default configuration.
2025-11-18 21:58:36 +08:00
EightyOliveira
dacca334e0 refactor(chunking): rename params and improve docstring for chunking_by_token_size 2025-11-18 15:46:28 +08:00
wmsnp
f4bf5d279c
fix: add logger to configure_vchordrq() and format code 2025-11-18 15:31:08 +08:00
Daniel.y
dfbc97363c
Merge pull request #2369 from HKUDS/workspace-isolation
Feat: Add Workspace Isolation for Pipeline Status and In-memory Storage
2025-11-18 15:21:10 +08:00
yangdx
702cfd2981 Fix document deletion concurrency control and validation logic
• Clarify job naming for single vs batch deletion
• Update job name validation in busy pipeline check
2025-11-18 13:59:24 +08:00
yangdx
656025b75e Rename GitHub workflow from "Tests" to "Offline Unit Tests" 2025-11-18 13:36:00 +08:00
yangdx
7e9c8ed1e8 Rename test classes to prevent warning from pytest
• TestResult → ExecutionResult
• TestStats → ExecutionStats
• Update class docstrings
• Update type hints
• Update variable references
2025-11-18 13:33:05 +08:00
yangdx
4048fc4b89 Fix: auto-acquire pipeline when idle in document deletion
• Track if we acquired the pipeline lock
• Auto-acquire pipeline when idle
• Only release if we acquired it
• Prevent concurrent deletion conflicts
• Improve deletion job validation
2025-11-18 13:25:13 +08:00
yangdx
1745b30a5f Fix missing workspace parameter in update flags status call 2025-11-18 12:55:48 +08:00
yangdx
f8dd2e0724 Fix namespace parsing when workspace contains colons
• Use rsplit instead of split
• Handle colons in workspace names
2025-11-18 12:23:05 +08:00
yangdx
472b498ade Replace pytest group reference with explicit dependencies in evaluation
• Remove pytest group dependency
• Add explicit pytest>=8.4.2
• Add pytest-asyncio>=1.2.0
• Add pre-commit directly
• Fix potential circular dependency
2025-11-18 12:17:21 +08:00
yangdx
a11912ffa5 Add testing workflow guidelines to basic development rules
* Define pytest marker patterns
* Document CI/CD test execution
* Specify offline vs integration tests
* Add test isolation best practices
* Reference testing guidelines doc
2025-11-18 11:54:19 +08:00
yangdx
41bf6d0283 Fix test to use default workspace parameter behavior 2025-11-18 11:51:17 +08:00
wmsnp
d07023c962
feat(postgres_impl): add vchordrq vector index support and unify vector index creation logic 2025-11-18 11:45:16 +08:00
yangdx
4ea2124001 Add GitHub CI workflow and test markers for offline/integration tests
- Add GitHub Actions workflow for CI
- Mark integration tests requiring services
- Add offline test markers for isolated tests
- Skip integration tests by default
- Configure pytest markers and collection
2025-11-18 11:36:10 +08:00
yangdx
4fef731f37 Standardize test directory creation and remove tempfile dependency
• Remove unused tempfile import
• Use consistent project temp/ structure
• Clean up existing directories first
• Create directories with os.makedirs
• Use descriptive test directory names
2025-11-18 10:39:54 +08:00
yangdx
1fe05df211 Refactor test configuration to use pytest fixtures and CLI options
• Add pytest command-line options
• Create session-scoped fixtures
• Remove hardcoded environment vars
• Update test function signatures
• Improve configuration priority
2025-11-18 10:31:53 +08:00
yangdx
6ae0c14438 test: add concurrent execution to workspace isolation test
• Add async sleep to mock functions
• Test concurrent ainsert operations
• Use asyncio.gather for parallel exec
• Measure concurrent execution time
2025-11-18 10:17:34 +08:00
yangdx
6cef8df159 Reduce log level and improve workspace mismatch message clarity
• Change warning to info level
• Simplify workspace mismatch wording
2025-11-18 08:25:21 +08:00
yangdx
fc9f7c705e Fix linting 2025-11-18 08:07:54 +08:00
yangdx
f83b475ab1 Remove Dependabot configuration file
• Delete .github/dependabot.yml
• Remove weekly pip updates
2025-11-18 01:42:15 +08:00
yangdx
21ad990e36 Improve workspace isolation tests with better parallelism checks and cleanup
• Add finalize_share_data cleanup
• Refactor lock timing measurement
• Add timeline overlap validation
• Include purpose/scope documentation
• Fix tokenizer integration
2025-11-18 01:38:31 +08:00
yangdx
5da82bb096 Add pre-commit to pytest dependencies and format test code
• Add pre-commit to pytest extra deps
• Update lock file dependencies
2025-11-18 00:42:04 +08:00
yangdx
99262adaaa Enhance workspace isolation test with distinct mock data and persistence
• Use different mock LLM per workspace
• Add persistent test directory
• Create workspace-specific responses
• Skip cleanup for inspection
2025-11-18 00:38:31 +08:00
yangdx
b7b8d15632 Refactor pytest dependencies into separate optional group
- Extract pytest deps to own group
- Reference pytest group in evaluation
- Add pytest config to pyproject.toml
- Update uv.lock with new structure
2025-11-17 23:52:13 +08:00
yangdx
1874cfaf73 Fix linting 2025-11-17 23:32:38 +08:00
Daniel.y
3806892a40
Merge pull request #2371 from BukeLy/pytest-style-conversion
test: Convert test_workspace_isolation.py to pytest style
2025-11-17 23:28:56 +08:00
BukeLy
1a1837028a docs: Update test file docstring to reflect all 11 test scenarios
Previous docstring mentioned only 4 scenarios but the file actually contains
11 comprehensive test cases. Updated to list all scenarios:

1. Pipeline Status Isolation
2. Lock Mechanism (Parallel/Serial)
3. Backward Compatibility
4. Multi-Workspace Concurrency
5. NamespaceLock Re-entrance Protection
6. Different Namespace Lock Isolation
7. Error Handling
8. Update Flags Workspace Isolation
9. Empty Workspace Standardization
10. JsonKVStorage Workspace Isolation
11. LightRAG End-to-End Workspace Isolation

This makes the file header accurately describe its contents.
2025-11-17 19:02:46 +08:00
BukeLy
3ec736932e test: Enhance E2E workspace isolation detection with content verification
Add specific content assertions to detect cross-contamination between workspaces.
Previously only checked that workspaces had different data, now verifies:

- Each workspace contains only its own text content
- Each workspace does NOT contain the other workspace's content
- Cross-contamination would be immediately detected

This ensures the test can find problems, not just pass.

Changes:
- Add assertions for "Artificial Intelligence" and "Machine Learning" in project_a
- Add assertions for "Deep Learning" and "Neural Networks" in project_b
- Add negative assertions to verify data leakage doesn't occur
- Add detailed output messages showing what was verified

Testing:
- pytest tests/test_workspace_isolation.py::test_lightrag_end_to_end_workspace_isolation
- Test passes with proper content isolation verified
2025-11-17 18:55:45 +08:00
BukeLy
a990c1d40b fix: Correct Mock LLM output format in E2E test
Why this change is needed:
The mock LLM function was returning JSON format, which is incorrect
for LightRAG's entity extraction. This caused "Complete delimiter
can not be found" warnings and resulted in 0 entities/relations
being extracted during tests.

How it solves it:
- Updated mock_llm_func to return correct tuple-delimited format
- Format: entity<|#|>name<|#|>type<|#|>description
- Format: relation<|#|>source<|#|>target<|#|>keywords<|#|>description
- Added proper completion delimiter: <|COMPLETE|>
- Now correctly extracts 2 entities and 1 relation

Impact:
- E2E test now properly validates entity/relation extraction
- No more "Complete delimiter" warnings
- Tests can now detect extraction-related bugs
- Graph files contain actual data (2 nodes, 1 edge) instead of empty graphs

Testing:
All 11 tests pass in 2.42s with proper entity extraction:
- Chunk 1 of 1 extracted 2 Ent + 1 Rel (previously 0 Ent + 0 Rel)
- Graph files now 2564 bytes (previously 310 bytes)
2025-11-17 18:49:54 +08:00
BukeLy
288498ccdc test: Convert test_workspace_isolation.py to pytest style
Why this change is needed:
The test file was using a custom TestResults class for tracking test
execution and results, which is not standard practice for pytest-based
test suites. This makes the tests harder to integrate with CI/CD pipelines
and reduces compatibility with pytest plugins and tooling.

How it solves it:
- Removed custom TestResults class and manual result tracking
- Added @pytest.mark.asyncio decorator to all async test functions
- Converted all results.add() calls to standard pytest assert statements
- Added pytest fixture (setup_shared_data) for common test setup
- Removed custom main() runner (pytest handles test discovery/execution)
- Kept all test logic, assertions, and debugging print statements intact

Impact:
- All 11 test functions maintain identical behavior and coverage
- Tests now follow pytest conventions and integrate with pytest ecosystem
- Test output is cleaner and more informative with pytest's reporting
- Easier to run selective tests using pytest's filtering options

Testing:
Verified by running: uv run pytest tests/test_workspace_isolation.py -v
Result: All 11 tests passed in 2.41s
2025-11-17 18:24:52 +08:00
yangdx
ddc76f0c80 Merge branch 'main' into workspace-isolation 2025-11-17 17:08:07 +08:00