Commit graph

62 commits

Author SHA1 Message Date
Raphael MANSUY
fe9b8ec02a
tests: stabilize integration tests + skip external services; fix multi-tenant API behavior and idempotency (#4)
* feat: Implement multi-tenant architecture with tenant and knowledge base models

- Added data models for tenants, knowledge bases, and related configurations.
- Introduced role and permission management for users in the multi-tenant system.
- Created a service layer for managing tenants and knowledge bases, including CRUD operations.
- Developed a tenant-aware instance manager for LightRAG with caching and isolation features.
- Added a migration script to transition existing workspace-based deployments to the new multi-tenant architecture.

* chore: ignore lightrag/api/webui/assets/ directory

* chore: stop tracking lightrag/api/webui/assets (ignore in .gitignore)

* feat: Initialize LightRAG Multi-Tenant Stack with PostgreSQL

- Added README.md for project overview, setup instructions, and architecture details.
- Created docker-compose.yml to define services: PostgreSQL, Redis, LightRAG API, and Web UI.
- Introduced env.example for environment variable configuration.
- Implemented init-postgres.sql for PostgreSQL schema initialization with multi-tenant support.
- Added reproduce_issue.py for testing default tenant access via API.

* feat: Enhance TenantSelector and update related components for improved multi-tenant support

* feat: Enhance testing capabilities and update documentation

- Updated Makefile to include new test commands for various modes (compatibility, isolation, multi-tenant, security, coverage, and dry-run).
- Modified API health check endpoint in Makefile to reflect new port configuration.
- Updated QUICK_START.md and README.md to reflect changes in service URLs and ports.
- Added environment variables for testing modes in env.example.
- Introduced run_all_tests.sh script to automate testing across different modes.
- Created conftest.py for pytest configuration, including database fixtures and mock services.
- Implemented database helper functions for streamlined database operations in tests.
- Added test collection hooks to skip tests based on the current MULTITENANT_MODE.

* feat: Implement multi-tenant support with demo mode enabled by default

- Added multi-tenant configuration to the environment and Docker setup.
- Created pre-configured demo tenants (acme-corp and techstart) for testing.
- Updated API endpoints to support tenant-specific data access.
- Enhanced Makefile commands for better service management and database operations.
- Introduced user-tenant membership system with role-based access control.
- Added comprehensive documentation for multi-tenant setup and usage.
- Fixed issues with document visibility in multi-tenant environments.
- Implemented necessary database migrations for user memberships and legacy support.

* feat(audit): Add final audit report for multi-tenant implementation

- Documented overall assessment, architecture overview, test results, security findings, and recommendations.
- Included detailed findings on critical security issues and architectural concerns.

fix(security): Implement security fixes based on audit findings

- Removed global RAG fallback and enforced strict tenant context.
- Configured super-admin access and required user authentication for tenant access.
- Cleared localStorage on logout and improved error handling in WebUI.

chore(logs): Create task logs for audit and security fixes implementation

- Documented actions, decisions, and next steps for both audit and security fixes.
- Summarized test results and remaining recommendations.

chore(scripts): Enhance development stack management scripts

- Added scripts for cleaning, starting, and stopping the development stack.
- Improved output messages and ensured graceful shutdown of services.

feat(starter): Initialize PostgreSQL with AGE extension support

- Created initialization scripts for PostgreSQL extensions including uuid-ossp, vector, and AGE.
- Ensured successful installation and verification of extensions.

* feat: Implement auto-select for first tenant and KB on initial load in WebUI

- Removed WEBUI_INITIAL_STATE_FIX.md as the issue is resolved.
- Added useTenantInitialization hook to automatically select the first available tenant and KB on app load.
- Integrated the new hook into the Root component of the WebUI.
- Updated RetrievalTesting component to ensure a KB is selected before allowing user interaction.
- Created end-to-end tests for multi-tenant isolation and real service interactions.
- Added scripts for starting, stopping, and cleaning the development stack.
- Enhanced API and tenant routes to support tenant-specific pipeline status initialization.
- Updated constants for backend URL to reflect the correct port.
- Improved error handling and logging in various components.

* feat: Add multi-tenant support with enhanced E2E testing scripts and client functionality

* update client

* Add integration and unit tests for multi-tenant API, models, security, and storage

- Implement integration tests for tenant and knowledge base management endpoints in `test_tenant_api_routes.py`.
- Create unit tests for tenant isolation, model validation, and role permissions in `test_tenant_models.py`.
- Add security tests to enforce role-based permissions and context validation in `test_tenant_security.py`.
- Develop tests for tenant-aware storage operations and context isolation in `test_tenant_storage_phase3.py`.

* feat(e2e): Implement OpenAI model support and database reset functionality

* Add comprehensive test suite for gpt-5-nano compatibility

- Introduced tests for parameter normalization, embeddings, and entity extraction.
- Implemented direct API testing for gpt-5-nano.
- Validated .env configuration loading and OpenAI API connectivity.
- Analyzed reasoning token overhead with various token limits.
- Documented test procedures and expected outcomes in README files.
- Ensured all tests pass for production readiness.

* kg(postgres_impl): ensure AGE extension is loaded in session and configure graph initialization

* dev: add hybrid dev helper scripts, Makefile, docker-compose.dev-db and local development docs

* feat(dev): add dev helper scripts and local development documentation for hybrid setup

* feat(multi-tenant): add detailed specifications and logs for multi-tenant improvements, including UX, backend handling, and ingestion pipeline

* feat(migration): add generated tenant/kb columns, indexes, triggers; drop unused tables; update schema and docs

* test(backward-compat): adapt tests to new StorageNameSpace/TenantService APIs (use concrete dummy storages)

* chore: multi-tenant and UX updates — docs, webui, storage, tenant service adjustments

* tests: stabilize integration tests + skip external services; fix multi-tenant API behavior and idempotency

- gpt5_nano_compatibility: add pytest-asyncio markers, skip when OPENAI key missing, prevent module-level asyncio.run collection, add conftest
- Ollama tests: add server availability check and skip markers; avoid pytest collection warnings by renaming helper classes
- Graph storage tests: rename interactive test functions to avoid pytest collection
- Document & Tenant routes: support external_ids for idempotency; ensure HTTPExceptions are re-raised
- LightRAG core: support external_ids in apipeline_enqueue_documents and idempotent logic
- Tests updated to match API changes (tenant routes & document routes)
- Add logs and scripts for inspection and audit
2025-12-04 16:04:21 +08:00
yangdx
9db8f2fce5 feat: Add popular labels and search APIs with history management
- Add popular/search label endpoints
- Implement SearchHistoryManager utility
- Replace client-side with server search
- Add graph data version tracking
- Update UI for better label discovery
2025-09-20 02:03:47 +08:00
yangdx
eeff0d5c67 Fix thinking state management and UI improvements for chat messages
- Reset expansion state on new thinking
- Prevent content leakage during thinking
- Update thinkingTime type to allow null
- Add memory leak prevention cleanup
- Add missing i18n translations
2025-09-08 18:47:41 +08:00
yangdx
2ba10dbb5b Add AI thinking process display with expandable UI and timing
- Add thinking fields to Message type
- Create expandable thinking section
- Track thinking time during streaming
2025-09-08 18:09:04 +08:00
yangdx
de2daf6565 refac: Rename summary_max_tokens to summary_context_size, comprehensive parameter validation for summary configuration
- Update algorithm logic in operate.py for better token management
- Fix health endpoint to use correct parameter names
2025-08-26 01:35:50 +08:00
yangdx
47485b130d refac(ui): Show rerank binding info on status card
- Remove separate ENABLE_RERANK flag in favor of rerank_binding="null"
- Change default rerank binding from "cohere" to "null" (disabled)
- Update UI to display both rerank binding and model information
2025-08-23 02:04:14 +08:00
yangdx
c22315ea6d refactor: remove selective LLM cache clearing functionality
- Remove optional 'modes' parameter from aclear_cache() and clear_cache() methods
- Replace deprecated drop_cache_by_modes() with drop() method for complete cache clearing
- Update API endpoint to ignore mode-specific parameters and clear all cache
- Simplify frontend clearCache() function to send empty request body

This change ensures all LLM cache is cleared together.
2025-08-05 23:51:51 +08:00
yangdx
0eac1a883a Feat: add file path sorting for document manager
- Add file_path sorting support to all database backends (JSON, Redis, PostgreSQL, MongoDB)
- Implement smart column header switching between "ID" and "File Name" based on display mode
- Add automatic sort field switching when toggling between ID and file name display
- Create composite indexes for workspace+file_path in PostgreSQL and MongoDB for better query performance
- Update frontend to maintain sort state when switching display modes
- Add internationalization support for "fileName" in English and Chinese locales

This enhancement improves user experience by providing intuitive file-based sorting
while maintaining performance through optimized database indexes.
2025-07-30 18:46:55 +08:00
yangdx
74eecc46e5 feat(pagination): Implement document list pagination backends and frontend UI
- Add pagination support to BaseDocStatusStorage interface and all implementations (PostgreSQL, MongoDB, Redis, JSON)
- Implement RESTful API endpoints for paginated document queries and status counts
- Create reusable pagination UI components with internationalization support
- Optimize performance with database-level pagination and efficient in-memory processing
- Maintain backward compatibility while adding configurable page sizes (10-200 items)
2025-07-30 17:58:32 +08:00
yangdx
cbaede8455 Add ScanResponse type for scan endpoint in webui 2025-07-30 03:11:09 +08:00
yangdx
6f958d5aee feat: add metadata timestamps to document processing and update frontend compatibility
- Add metadata field to doc_status storage with Unix timestamps for processing start/end times
- Update frontend API types: error -> error_msg, add track_id and metadata support
- Add getTrackStatus API method for document tracking functionality
- Fix frontend DocumentManager to use error_msg field for proper error display
- Ensure full compatibility between backend metadata changes and frontend UI
2025-07-30 00:04:27 +08:00
yangdx
84b09aa5da feat: add threshold status line to StatusCard with i18n support
- Add cosine_threshold, min_rerank_score, related_chunk_number to LightragStatus type
2025-07-28 02:51:36 +08:00
yangdx
5aceca0052 feat(webui): enhance status card with new health endpoint data
- Update StatusCard to display consolidated server info with parallel insert limits and summary settings
- Merge LLM and embedding configurations with async parameters for cleaner display
- Add new status fields to TypeScript interface (summary_language, max_parallel_insert, etc.)
2025-07-28 02:19:27 +08:00
zrguo
7c882313bb remove chunk_rerank_top_k 2025-07-15 11:52:34 +08:00
zrguo
4e425b1b59 Revert "update from main"
This reverts commit 1d0376d6a9.
2025-07-14 16:29:00 +08:00
zrguo
1d0376d6a9 update from main 2025-07-14 16:27:49 +08:00
zrguo
c9cbd2d3e0 Merge branch 'main' into rerank 2025-07-14 16:24:29 +08:00
zrguo
ef2115d437 Update token limit 2025-07-14 15:53:48 +08:00
yangdx
ab561196ff Feat: Added reranker config and lock status to status card of WebUI 2025-07-13 00:41:54 +08:00
yangdx
db22cad2c8 feat: add workspace and MAX_GRAPH_NODES to /health endpoint and webui 2025-07-07 01:39:48 +08:00
yangdx
bdcd55a871 Feat: Add delete upload file option to document deletion 2025-06-25 19:02:46 +08:00
yangdx
8ea7d7ad85 Feat: Add document deletion for WebUI 2025-06-25 17:09:10 +08:00
yangdx
d914de441e Add user_prompt to WebUI 2025-05-08 04:15:21 +08:00
yangdx
147f73002d Remove ll-keywords and hl-keywords from WebUI 2025-05-07 19:07:31 +08:00
yangdx
8234886c45 Optimize webui error handling of streaming response 2025-04-23 21:08:13 +08:00
choizhang
333c43fbdf refactor: Fixed redundant blank characters in message content 2025-04-22 01:25:18 +08:00
choizhang
4900cf07d9 Merge branch 'main' into edit-node 2025-04-13 11:51:55 +08:00
choizhang
5e5f3640d7 2025-04-13 10:50:42 +08:00
choizhang
7e3e685763 feat(graph): Add editing function for entity and relationship attributes 2025-04-12 00:48:19 +08:00
choizhang
a12d60e4ea feat: Add query mode 'bypass' to bypass knowledge retrieval and directly use LLM 2025-04-11 02:57:34 +08:00
yangdx
1541b7b486 Fix linting 2025-04-11 00:31:14 +08:00
choizhang
a524520b80 refactor(lightrag): use fetch replace axios 2025-04-11 00:31:14 +08:00
yangdx
299c508f27 Fix webtitle display problem 2025-04-05 02:24:23 +08:00
yangdx
1c0b94c46f Fix web title cleaning problem 2025-04-05 00:41:49 +08:00
yangdx
29d01e1c74 Add title support for webui 2025-04-04 22:51:07 +08:00
yangdx
1a1f934e27 Feat: replace min_degree with max_nodes in webui 2025-04-02 18:35:32 +08:00
yangdx
4c9c600472 Refactoring clear document UI 2025-04-01 13:31:14 +08:00
yangdx
35d4503677 Feat: add duplicate file check 2025-03-28 16:48:46 +08:00
yangdx
814f3b3308 feat: add pipeline busy status indicator with breathing effect
- Add pipeline_busy field to health check response
- Track pipeline busy state in frontend store
- Add breathing animation for pipeline status button
- Enhance dark mode visibility with stronger contrast
2025-03-26 13:11:53 +08:00
yangdx
d7c0b420b9 feat: add pipeline status monitoring dialog
- Add pipeline status API and types
- Create PipelineStatusDialog component with position control
- Unify modal overlay style across components
2025-03-26 12:05:54 +08:00
yangdx
d456269718 feat: Add file name display in WebUI
Backend:
- Add file_path field to DocStatusResponse
- Update document status response creation

Frontend:
- Remove metadata column
- Improve filename display with truncation and tooltips
- Add show/hide filename toggle with proper styling
- Update translations for all supported languages"
2025-03-25 22:42:46 +08:00
yangdx
a444edd9eb Added auth mode status to health check endpoint.
- Introduced auth_mode in health check response
- Updated App.tsx to handle guest mode logic
2025-03-23 02:43:44 +08:00
yangdx
3609dc5b26 Improve authentication flow and navigation handling
- Replace never-resolving promise with rejection
- Remove unnecessary setTimeout in navigation
2025-03-23 01:39:43 +08:00
yangdx
31c7923067 Added core and API version to health check
- Bumped API version to 1.2.2
2025-03-23 00:28:50 +08:00
yangdx
0df0ba269d Implemented version display in SiteHeader of webui
- Updated API version to 1.2.0
- Stored versions in localStorage
2025-03-22 01:51:39 +08:00
yangdx
99814b57d9 Refactor navigation and authentication flow
- Move navigation setup to AppRouter
- Prevent protected route logic to handle login 401
2025-03-19 19:08:09 +08:00
yangdx
50a8b5fb6d Refactor navigation and authentication flow, prevent mounting login page multiple times
- Improved protected route handling
- Enhanced direct login access detection
- Centralized navigation logic
- Optimized state reset process
- Fixed logout navigation behavior
2025-03-19 12:45:08 +08:00
yangdx
c42f08c0e6 Add navigation service for centralized login redirect routing
- Simplify token validation logic in API
- Update axios interceptor to use navigation service
2025-03-18 19:45:43 +08:00
yangdx
43996656d3 Fix linting 2025-03-18 03:30:43 +08:00
yangdx
f8440c8f80 feat(auth): implement auto guest mode and enhance token system
- Add role-based token system with metadata support
- Implement automatic guest mode for unconfigured authentication
- Create new /auth-status endpoint for authentication status checking
- Modify frontend to auto-detect auth status and bypass login when appropriate
- Add guest mode indicator in site header for better UX

This change allows users to automatically access the system without manual
login when authentication is not configured, while maintaining secure
authentication when credentials are properly set up.
2025-03-18 02:56:02 +08:00