Commit graph

51 commits

Author SHA1 Message Date
Raphaël MANSUY
50d882c7a7 cherry-pick efd50064 2025-12-04 19:19:22 +08:00
Raphaël MANSUY
b2c2eb267d cherry-pick 4b31942e 2025-12-04 19:19:03 +08:00
Raphaël MANSUY
1afb70f563 cherry-pick e0966b65 2025-12-04 19:19:00 +08:00
Raphaël MANSUY
86df9402c7 cherry-pick c61b7bd4 2025-12-04 19:18:35 +08:00
Raphaël MANSUY
41f163362e cherry-pick a8bbce3a 2025-12-04 19:18:35 +08:00
Raphaël MANSUY
78c1c69b0a cherry-pick be9e6d16 2025-12-04 19:18:16 +08:00
Raphaël MANSUY
a00b4e4bc9 cherry-pick 6c05f0f8 2025-12-04 19:18:16 +08:00
Raphaël MANSUY
bf1a8cf8ba cherry-pick dde728a3 2025-12-04 19:17:42 +08:00
Raphaël MANSUY
f7f9a9e6cf fix: sync all core modules with upstream after Wave 1 2025-12-04 19:13:48 +08:00
yangdx
7248e09fc4 Allow related chunks missing in knowledge graph queries
(cherry picked from commit 35cd567c9e)
2025-12-04 19:11:18 +08:00
yangdx
4e94f3a67c Improve Docker build workflow with automated multi-arch script and docs
(cherry picked from commit 0e0b4a94dc)
2025-12-04 19:09:08 +08:00
yangdx
a2d67a7c22 Add build script for multi-platform images
- Add build script for multi-platform images
- Update docker deployment document

(cherry picked from commit ef79821f29)
2025-12-04 19:08:59 +08:00
yangdx
58f818c449 Change default docker image to offline version
• Add lite verion docker image with tiktoken cache
• Update docs and build scripts

(cherry picked from commit daeca17f38)
2025-12-04 19:08:59 +08:00
yangdx
f89c7315fd Migrate Dockerfile from pip to uv package manager for faster builds
• Replace pip with uv for dependencies
• Add offline extras to Dockerfile.offline
• Update UV_LOCK_GUIDE.md with new commands
• Improve build caching and performance

(cherry picked from commit 65c2eb9f99)
2025-12-04 19:08:59 +08:00
yangdx
7e5c23e15b docs: clarify docling exclusion in offline Docker image
(cherry picked from commit 388dce2e31)
2025-12-04 19:08:59 +08:00
yangdx
9a23234c6c Add build script for multi-platform images
- Add build script for multi-platform images
- Update docker deployment document

(cherry picked from commit ef79821f29)
2025-12-04 19:08:21 +08:00
yangdx
aa61e82820 Migrate from pip to uv package manager for faster builds
• Replace pip with uv in Dockerfile
• Remove constraints-offline.txt
• Add uv.lock for dependency pinning
• Use uv sync --frozen for builds

(cherry picked from commit 466de2070d)
2025-12-04 19:08:10 +08:00
yangdx
8c3a325193 docs: clarify docling exclusion in offline Docker image
(cherry picked from commit 388dce2e31)
2025-12-04 19:07:59 +08:00
yangdx
b0bdbb5839 Add offline deployment support with cache management and layered deps
• Add tiktoken cache downloader CLI
• Add layered offline dependencies
• Add offline requirements files
• Add offline deployment guide

(cherry picked from commit a5c05f1b92)
2025-12-04 19:07:09 +08:00
Raphaël MANSUY
6cca895ba9 Add logs for recent actions and decisions regarding upstream changes
- Documented major changes after pulling from upstream (HKUDS/LightRAG), focusing on multi-tenant support, security hardening, and RLS/RBAC.
- Created concise documentation under docs/diff_hku, including migration guides and security audits.
- Enumerated unmerged upstream commits and summarized substantive features and fixes.
- Outlined next steps for DB migrations, CI tests, and potential cherry-picking of upstream fixes.
2025-12-04 18:28:44 +08:00
Raphael MANSUY
2b292d4924
docs: Enterprise Edition & Multi-tenancy attribution (#5)
* Remove outdated documentation files: Quick Start Guide, Apache AGE Analysis, and Scratchpad.

* Add multi-tenant testing strategy and ADR index documentation

- Introduced ADR 008 detailing the multi-tenant testing strategy for the ./starter environment, covering compatibility and multi-tenant modes, testing scenarios, and implementation details.
- Created a comprehensive ADR index (README.md) summarizing all architecture decision records related to the multi-tenant implementation, including purpose, key sections, and reading paths for different roles.

* feat(docs): Add comprehensive multi-tenancy guide and README for LightRAG Enterprise

- Introduced `0008-multi-tenancy.md` detailing multi-tenancy architecture, key concepts, roles, permissions, configuration, and API endpoints.
- Created `README.md` as the main documentation index, outlining features, quick start, system overview, and deployment options.
- Documented the LightRAG architecture, storage backends, LLM integrations, and query modes.
- Established a task log (`2025-01-21-lightrag-documentation-log.md`) summarizing documentation creation actions, decisions, and insights.
2025-12-04 18:09:15 +08:00
Raphael MANSUY
fe9b8ec02a
tests: stabilize integration tests + skip external services; fix multi-tenant API behavior and idempotency (#4)
* feat: Implement multi-tenant architecture with tenant and knowledge base models

- Added data models for tenants, knowledge bases, and related configurations.
- Introduced role and permission management for users in the multi-tenant system.
- Created a service layer for managing tenants and knowledge bases, including CRUD operations.
- Developed a tenant-aware instance manager for LightRAG with caching and isolation features.
- Added a migration script to transition existing workspace-based deployments to the new multi-tenant architecture.

* chore: ignore lightrag/api/webui/assets/ directory

* chore: stop tracking lightrag/api/webui/assets (ignore in .gitignore)

* feat: Initialize LightRAG Multi-Tenant Stack with PostgreSQL

- Added README.md for project overview, setup instructions, and architecture details.
- Created docker-compose.yml to define services: PostgreSQL, Redis, LightRAG API, and Web UI.
- Introduced env.example for environment variable configuration.
- Implemented init-postgres.sql for PostgreSQL schema initialization with multi-tenant support.
- Added reproduce_issue.py for testing default tenant access via API.

* feat: Enhance TenantSelector and update related components for improved multi-tenant support

* feat: Enhance testing capabilities and update documentation

- Updated Makefile to include new test commands for various modes (compatibility, isolation, multi-tenant, security, coverage, and dry-run).
- Modified API health check endpoint in Makefile to reflect new port configuration.
- Updated QUICK_START.md and README.md to reflect changes in service URLs and ports.
- Added environment variables for testing modes in env.example.
- Introduced run_all_tests.sh script to automate testing across different modes.
- Created conftest.py for pytest configuration, including database fixtures and mock services.
- Implemented database helper functions for streamlined database operations in tests.
- Added test collection hooks to skip tests based on the current MULTITENANT_MODE.

* feat: Implement multi-tenant support with demo mode enabled by default

- Added multi-tenant configuration to the environment and Docker setup.
- Created pre-configured demo tenants (acme-corp and techstart) for testing.
- Updated API endpoints to support tenant-specific data access.
- Enhanced Makefile commands for better service management and database operations.
- Introduced user-tenant membership system with role-based access control.
- Added comprehensive documentation for multi-tenant setup and usage.
- Fixed issues with document visibility in multi-tenant environments.
- Implemented necessary database migrations for user memberships and legacy support.

* feat(audit): Add final audit report for multi-tenant implementation

- Documented overall assessment, architecture overview, test results, security findings, and recommendations.
- Included detailed findings on critical security issues and architectural concerns.

fix(security): Implement security fixes based on audit findings

- Removed global RAG fallback and enforced strict tenant context.
- Configured super-admin access and required user authentication for tenant access.
- Cleared localStorage on logout and improved error handling in WebUI.

chore(logs): Create task logs for audit and security fixes implementation

- Documented actions, decisions, and next steps for both audit and security fixes.
- Summarized test results and remaining recommendations.

chore(scripts): Enhance development stack management scripts

- Added scripts for cleaning, starting, and stopping the development stack.
- Improved output messages and ensured graceful shutdown of services.

feat(starter): Initialize PostgreSQL with AGE extension support

- Created initialization scripts for PostgreSQL extensions including uuid-ossp, vector, and AGE.
- Ensured successful installation and verification of extensions.

* feat: Implement auto-select for first tenant and KB on initial load in WebUI

- Removed WEBUI_INITIAL_STATE_FIX.md as the issue is resolved.
- Added useTenantInitialization hook to automatically select the first available tenant and KB on app load.
- Integrated the new hook into the Root component of the WebUI.
- Updated RetrievalTesting component to ensure a KB is selected before allowing user interaction.
- Created end-to-end tests for multi-tenant isolation and real service interactions.
- Added scripts for starting, stopping, and cleaning the development stack.
- Enhanced API and tenant routes to support tenant-specific pipeline status initialization.
- Updated constants for backend URL to reflect the correct port.
- Improved error handling and logging in various components.

* feat: Add multi-tenant support with enhanced E2E testing scripts and client functionality

* update client

* Add integration and unit tests for multi-tenant API, models, security, and storage

- Implement integration tests for tenant and knowledge base management endpoints in `test_tenant_api_routes.py`.
- Create unit tests for tenant isolation, model validation, and role permissions in `test_tenant_models.py`.
- Add security tests to enforce role-based permissions and context validation in `test_tenant_security.py`.
- Develop tests for tenant-aware storage operations and context isolation in `test_tenant_storage_phase3.py`.

* feat(e2e): Implement OpenAI model support and database reset functionality

* Add comprehensive test suite for gpt-5-nano compatibility

- Introduced tests for parameter normalization, embeddings, and entity extraction.
- Implemented direct API testing for gpt-5-nano.
- Validated .env configuration loading and OpenAI API connectivity.
- Analyzed reasoning token overhead with various token limits.
- Documented test procedures and expected outcomes in README files.
- Ensured all tests pass for production readiness.

* kg(postgres_impl): ensure AGE extension is loaded in session and configure graph initialization

* dev: add hybrid dev helper scripts, Makefile, docker-compose.dev-db and local development docs

* feat(dev): add dev helper scripts and local development documentation for hybrid setup

* feat(multi-tenant): add detailed specifications and logs for multi-tenant improvements, including UX, backend handling, and ingestion pipeline

* feat(migration): add generated tenant/kb columns, indexes, triggers; drop unused tables; update schema and docs

* test(backward-compat): adapt tests to new StorageNameSpace/TenantService APIs (use concrete dummy storages)

* chore: multi-tenant and UX updates — docs, webui, storage, tenant service adjustments

* tests: stabilize integration tests + skip external services; fix multi-tenant API behavior and idempotency

- gpt5_nano_compatibility: add pytest-asyncio markers, skip when OPENAI key missing, prevent module-level asyncio.run collection, add conftest
- Ollama tests: add server availability check and skip markers; avoid pytest collection warnings by renaming helper classes
- Graph storage tests: rename interactive test functions to avoid pytest collection
- Document & Tenant routes: support external_ids for idempotency; ensure HTTPExceptions are re-raised
- LightRAG core: support external_ids in apipeline_enqueue_documents and idempotent logic
- Tests updated to match API changes (tenant routes & document routes)
- Add logs and scripts for inspection and audit
2025-12-04 16:04:21 +08:00
Raphaël MANSUY
a5eb441124 feat: Add multi-tenant architecture ADRs and deployment guide
- Introduced ADR 007: Deployment Guide and Quick Reference, detailing multi-tenant architecture components, setup instructions, and testing procedures.
- Created DELIVERY_MANIFEST.txt summarizing the multi-tenant ADR delivery, including document purposes, lengths, and key insights.
- Added README.md as a comprehensive index for all ADRs, providing navigation paths and role-specific reading recommendations.
2025-11-20 15:27:31 +08:00
yangdx
580cb7906c feat: Add multiple rerank provider support to LightRAG Server by adding new env vars and cli params
- Add --enable-rerank CLI argument and ENABLE_RERANK env var
- Simplify rerank configuration logic to only check enable flag and binding
- Update health endpoint to show enable_rerank and rerank_configured status
- Improve logging messages for rerank enable/disable states
- Maintain backward compatibility with default value True
2025-08-22 19:29:45 +08:00
yangdx
9923821d75 refactor: Remove deprecated max_token_size from embedding configuration
This parameter is no longer used. Its removal simplifies the API and clarifies that token length management is handled by upstream text chunking logic rather than the embedding wrapper.
2025-07-29 10:49:35 +08:00
yangdx
3f5ade47cd Update README 2025-07-27 17:26:49 +08:00
yangdx
88bf695de5 Update doc for rerank 2025-07-20 00:37:36 +08:00
zrguo
9a9f0f2463 Update rerank_example & readme 2025-07-15 12:17:27 +08:00
yangdx
ab805b35c4 Update doc: concurrent explain 2025-07-13 21:50:30 +08:00
zrguo
cf26e52d89 fix init 2025-07-08 15:13:09 +08:00
zrguo
f5c80d7cde Simplify Configuration 2025-07-08 11:16:34 +08:00
zrguo
75dd4f3498 add rerank model 2025-07-07 22:44:59 +08:00
zrguo
03dd99912d RAG-Anything Integration 2025-06-17 01:16:02 +08:00
zrguo
ea2fabe6b0
Merge pull request #1619 from earayu/add_doc_for_parall
Add doc for explaining LightRAG's multi-document concurrent processing mechanism
2025-06-09 09:50:41 +08:00
zrguo
cc9040d70c fix lint 2025-06-05 17:37:11 +08:00
zrguo
962974589a Add example of directly using modal processors 2025-06-05 17:36:05 +08:00
zrguo
8a726f6e08 MinerU integration 2025-06-05 17:02:48 +08:00
earayu
2679f619b6 feat: add doc 2025-05-23 11:57:45 +08:00
earayu
6d6aefa2ff feat: add doc 2025-05-23 11:54:40 +08:00
earayu
2520ad01da feat: add doc 2025-05-23 11:53:06 +08:00
earayu
8b530698cc feat: add doc 2025-05-23 11:52:28 +08:00
earayu
8bafa49d5d feat: add doc 2025-05-23 11:52:06 +08:00
Saifeddine ALOUI
6e04df5fab
Create Algorithm.md 2025-01-24 21:19:04 +01:00
yangdx
57093c3571 Merge commit '548ad1f299c875b59df21147f7edf9eab2d73d2c' into fix-RAG-param-missing 2025-01-23 01:41:52 +08:00
yangdx
54c11d7734 Delete outdated doc (new version is lightrag/api/README.md) 2025-01-23 01:17:21 +08:00
Nick French
78fa56e2a7
Replacing ParisNeo with this repo owner, HKUDS 2025-01-21 10:50:27 -05:00
Saifeddine ALOUI
58f1058198 added some explanation to document 2025-01-17 02:03:02 +01:00
Saifeddine ALOUI
5fe28d31e9 Fixed linting 2025-01-17 01:36:16 +01:00
Saifeddine ALOUI
c5e027aa9a Added documentation about used environment variables 2025-01-17 00:42:22 +01:00
Saifeddine ALOUI
b2e7c75f5a Added Docker container setup 2025-01-16 22:28:28 +01:00