fe9b8ec02a
* feat: Implement multi-tenant architecture with tenant and knowledge base models - Added data models for tenants, knowledge bases, and related configurations. - Introduced role and permission management for users in the multi-tenant system. - Created a service layer for managing tenants and knowledge bases, including CRUD operations. - Developed a tenant-aware instance manager for LightRAG with caching and isolation features. - Added a migration script to transition existing workspace-based deployments to the new multi-tenant architecture. * chore: ignore lightrag/api/webui/assets/ directory * chore: stop tracking lightrag/api/webui/assets (ignore in .gitignore) * feat: Initialize LightRAG Multi-Tenant Stack with PostgreSQL - Added README.md for project overview, setup instructions, and architecture details. - Created docker-compose.yml to define services: PostgreSQL, Redis, LightRAG API, and Web UI. - Introduced env.example for environment variable configuration. - Implemented init-postgres.sql for PostgreSQL schema initialization with multi-tenant support. - Added reproduce_issue.py for testing default tenant access via API. * feat: Enhance TenantSelector and update related components for improved multi-tenant support * feat: Enhance testing capabilities and update documentation - Updated Makefile to include new test commands for various modes (compatibility, isolation, multi-tenant, security, coverage, and dry-run). - Modified API health check endpoint in Makefile to reflect new port configuration. - Updated QUICK_START.md and README.md to reflect changes in service URLs and ports. - Added environment variables for testing modes in env.example. - Introduced run_all_tests.sh script to automate testing across different modes. - Created conftest.py for pytest configuration, including database fixtures and mock services. - Implemented database helper functions for streamlined database operations in tests. - Added test collection hooks to skip tests based on the current MULTITENANT_MODE. * feat: Implement multi-tenant support with demo mode enabled by default - Added multi-tenant configuration to the environment and Docker setup. - Created pre-configured demo tenants (acme-corp and techstart) for testing. - Updated API endpoints to support tenant-specific data access. - Enhanced Makefile commands for better service management and database operations. - Introduced user-tenant membership system with role-based access control. - Added comprehensive documentation for multi-tenant setup and usage. - Fixed issues with document visibility in multi-tenant environments. - Implemented necessary database migrations for user memberships and legacy support. * feat(audit): Add final audit report for multi-tenant implementation - Documented overall assessment, architecture overview, test results, security findings, and recommendations. - Included detailed findings on critical security issues and architectural concerns. fix(security): Implement security fixes based on audit findings - Removed global RAG fallback and enforced strict tenant context. - Configured super-admin access and required user authentication for tenant access. - Cleared localStorage on logout and improved error handling in WebUI. chore(logs): Create task logs for audit and security fixes implementation - Documented actions, decisions, and next steps for both audit and security fixes. - Summarized test results and remaining recommendations. chore(scripts): Enhance development stack management scripts - Added scripts for cleaning, starting, and stopping the development stack. - Improved output messages and ensured graceful shutdown of services. feat(starter): Initialize PostgreSQL with AGE extension support - Created initialization scripts for PostgreSQL extensions including uuid-ossp, vector, and AGE. - Ensured successful installation and verification of extensions. * feat: Implement auto-select for first tenant and KB on initial load in WebUI - Removed WEBUI_INITIAL_STATE_FIX.md as the issue is resolved. - Added useTenantInitialization hook to automatically select the first available tenant and KB on app load. - Integrated the new hook into the Root component of the WebUI. - Updated RetrievalTesting component to ensure a KB is selected before allowing user interaction. - Created end-to-end tests for multi-tenant isolation and real service interactions. - Added scripts for starting, stopping, and cleaning the development stack. - Enhanced API and tenant routes to support tenant-specific pipeline status initialization. - Updated constants for backend URL to reflect the correct port. - Improved error handling and logging in various components. * feat: Add multi-tenant support with enhanced E2E testing scripts and client functionality * update client * Add integration and unit tests for multi-tenant API, models, security, and storage - Implement integration tests for tenant and knowledge base management endpoints in `test_tenant_api_routes.py`. - Create unit tests for tenant isolation, model validation, and role permissions in `test_tenant_models.py`. - Add security tests to enforce role-based permissions and context validation in `test_tenant_security.py`. - Develop tests for tenant-aware storage operations and context isolation in `test_tenant_storage_phase3.py`. * feat(e2e): Implement OpenAI model support and database reset functionality * Add comprehensive test suite for gpt-5-nano compatibility - Introduced tests for parameter normalization, embeddings, and entity extraction. - Implemented direct API testing for gpt-5-nano. - Validated .env configuration loading and OpenAI API connectivity. - Analyzed reasoning token overhead with various token limits. - Documented test procedures and expected outcomes in README files. - Ensured all tests pass for production readiness. * kg(postgres_impl): ensure AGE extension is loaded in session and configure graph initialization * dev: add hybrid dev helper scripts, Makefile, docker-compose.dev-db and local development docs * feat(dev): add dev helper scripts and local development documentation for hybrid setup * feat(multi-tenant): add detailed specifications and logs for multi-tenant improvements, including UX, backend handling, and ingestion pipeline * feat(migration): add generated tenant/kb columns, indexes, triggers; drop unused tables; update schema and docs * test(backward-compat): adapt tests to new StorageNameSpace/TenantService APIs (use concrete dummy storages) * chore: multi-tenant and UX updates — docs, webui, storage, tenant service adjustments * tests: stabilize integration tests + skip external services; fix multi-tenant API behavior and idempotency - gpt5_nano_compatibility: add pytest-asyncio markers, skip when OPENAI key missing, prevent module-level asyncio.run collection, add conftest - Ollama tests: add server availability check and skip markers; avoid pytest collection warnings by renaming helper classes - Graph storage tests: rename interactive test functions to avoid pytest collection - Document & Tenant routes: support external_ids for idempotency; ensure HTTPExceptions are re-raised - LightRAG core: support external_ids in apipeline_enqueue_documents and idempotent logic - Tests updated to match API changes (tenant routes & document routes) - Add logs and scripts for inspection and audit
337 lines
12 KiB
Markdown
337 lines
12 KiB
Markdown
# Multi-Tenant Implementation Audit - Final Report
|
|
|
|
**Date:** November 29, 2025
|
|
**Auditor:** Automated Audit Process
|
|
**Version:** 1.0
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
This audit reviewed the multi-tenant implementation of the LightRAG stack from Web UI through REST API to Storage layer. The implementation uses a **workspace-based isolation model** where `workspace = tenant_id/kb_id`.
|
|
|
|
### Overall Assessment: ⚠️ FUNCTIONAL WITH CONCERNS
|
|
|
|
The multi-tenant implementation is **functionally correct** but has several security and architectural concerns that should be addressed before production deployment.
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Web UI (React) │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
|
|
│ │ TenantStore │ │ KBSelector │ │ Axios Interceptor │ │
|
|
│ │ (Zustand) │ │ │ │ X-Tenant-ID Header │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
HTTP Headers (X-Tenant-ID, X-KB-ID)
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ REST API (FastAPI) │
|
|
│ ┌──────────────────┐ ┌────────────────┐ ┌─────────────────┐ │
|
|
│ │ get_tenant_ │ │ TenantContext │ │ TenantRAG │ │
|
|
│ │ context() │──│ (dataclass) │──│ Manager │ │
|
|
│ │ dependency │ │ │ │ (per-tenant RAG)│ │
|
|
│ └──────────────────┘ └────────────────┘ └─────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
workspace = tenant_id/kb_id
|
|
│
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Storage Layer │
|
|
│ ┌──────────────────────────────────────────────────────────┐ │
|
|
│ │ PostgreSQL Tables with workspace column │ │
|
|
│ │ PRIMARY KEY (workspace, id) │ │
|
|
│ │ │ │
|
|
│ │ Tables: LIGHTRAG_DOC_FULL, LIGHTRAG_DOC_CHUNKS, │ │
|
|
│ │ LIGHTRAG_VDB_*, LIGHTRAG_TENANTS, etc. │ │
|
|
│ └──────────────────────────────────────────────────────────┘ │
|
|
│ ┌─────────────────┐ ┌─────────────────┐ ┌────────────────┐ │
|
|
│ │ Redis │ │ Apache AGE │ │ Vector Store │ │
|
|
│ │ (key prefix) │ │ (graph per ws) │ │ (ws filter) │ │
|
|
│ └─────────────────┘ └─────────────────┘ └────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Test Results Summary
|
|
|
|
### Unit Tests: ✅ ALL PASSED
|
|
|
|
| Test Suite | Tests | Result |
|
|
|------------|-------|--------|
|
|
| test_multitenant_e2e.py | 32 | ✅ PASSED |
|
|
| test_tenant_security.py | 11 | ✅ PASSED |
|
|
| test_multi_tenant_backends.py | 37 | ✅ PASSED |
|
|
| test_tenant_storage_phase3.py | 22 | ✅ PASSED |
|
|
| **TOTAL** | **102** | **✅ ALL PASSED** |
|
|
|
|
### Key Verifications
|
|
- ✅ Cross-tenant data access prevention
|
|
- ✅ Knowledge base isolation within tenants
|
|
- ✅ Role-based permission enforcement
|
|
- ✅ Composite key uniqueness
|
|
- ✅ Redis namespace isolation
|
|
- ✅ Vector ID uniqueness per tenant
|
|
|
|
---
|
|
|
|
## Security Findings
|
|
|
|
### Critical Findings
|
|
|
|
| ID | Severity | Component | Finding |
|
|
|----|----------|-----------|---------|
|
|
| SEC-001 | 🔴 HIGH | API | `get_tenant_context_optional()` allows global RAG fallback |
|
|
| SEC-002 | 🔴 HIGH | API | Admin user bypass in `tenant_service.py` |
|
|
| SEC-003 | 🟡 MEDIUM | API | `user_id` parameter optional in RAG manager |
|
|
| SEC-004 | 🟡 MEDIUM | Storage | No PostgreSQL Row-Level Security (RLS) |
|
|
| SEC-005 | 🟢 LOW | Storage | Tenant context set for logging only |
|
|
| SEC-006 | 🟢 LOW | WebUI | localStorage used for token storage |
|
|
|
|
### SEC-001: Global RAG Fallback (HIGH)
|
|
|
|
**Location:** `lightrag/api/routers/document_routes.py`, `query_routes.py`
|
|
|
|
**Issue:** Using `get_tenant_context_optional` allows requests without tenant headers to use global RAG:
|
|
|
|
```python
|
|
async def get_tenant_rag(
|
|
tenant_context: Optional[TenantContext] = Depends(get_tenant_context_optional)
|
|
) -> LightRAG:
|
|
if rag_manager and tenant_context:
|
|
return await rag_manager.get_rag_instance(...)
|
|
return rag # Falls back to global RAG!
|
|
```
|
|
|
|
**Risk:** Data leakage if global RAG contains multi-tenant data.
|
|
|
|
**Recommendation:** Use `get_tenant_context` (required) for all data endpoints in multi-tenant mode.
|
|
|
|
### SEC-002: Admin User Bypass (HIGH)
|
|
|
|
**Location:** `lightrag/services/tenant_service.py`
|
|
|
|
**Issue:**
|
|
```python
|
|
if user_id.lower() == "admin":
|
|
return True # Bypass all access checks
|
|
```
|
|
|
|
**Risk:** Any user with username "admin" can access any tenant's data.
|
|
|
|
**Recommendation:** Implement proper super-admin role with separate authentication.
|
|
|
|
### SEC-003: Optional User ID (MEDIUM)
|
|
|
|
**Location:** `lightrag/tenant_rag_manager.py`
|
|
|
|
**Issue:** `user_id` parameter is optional, allowing access without authentication check.
|
|
|
|
**Recommendation:** Deprecate unauthenticated access path.
|
|
|
|
### SEC-004: No Row-Level Security (MEDIUM)
|
|
|
|
**Location:** PostgreSQL Tables
|
|
|
|
**Issue:** Tenant isolation relies entirely on application-level `workspace` filtering. No database-level enforcement.
|
|
|
|
**Recommendation:** Enable PostgreSQL RLS as defense-in-depth:
|
|
```sql
|
|
ALTER TABLE LIGHTRAG_DOC_FULL ENABLE ROW LEVEL SECURITY;
|
|
CREATE POLICY tenant_isolation ON LIGHTRAG_DOC_FULL
|
|
USING (workspace = current_setting('app.current_workspace'));
|
|
```
|
|
|
|
---
|
|
|
|
## Architectural Findings
|
|
|
|
### ARCH-001: Workspace Encoding
|
|
|
|
**Pattern:** `workspace = "{tenant_id}/{kb_id}"`
|
|
|
|
**Strengths:**
|
|
- Simple, hierarchical namespace
|
|
- Clear data ownership
|
|
- Compatible with file-based storage
|
|
|
|
**Concerns:**
|
|
- No explicit validation of workspace format
|
|
- Potential collision if tenant IDs contain `/`
|
|
|
|
### ARCH-002: Context Propagation
|
|
|
|
**Pattern:** Python `ContextVar` for tenant context
|
|
|
|
```python
|
|
tenant_id_var: ContextVar[Optional[str]] = ContextVar("tenant_id")
|
|
```
|
|
|
|
**Current Use:** Set for PostgreSQL session logging, but NOT used for security.
|
|
|
|
**Recommendation:** Use context variable for RLS enforcement.
|
|
|
|
### ARCH-003: RAG Instance Caching
|
|
|
|
**Pattern:** LRU cache keyed by `(tenant_id, kb_id)`
|
|
|
|
**Strengths:**
|
|
- Memory efficient
|
|
- Fast subsequent access
|
|
|
|
**Concerns:**
|
|
- Cache size limit may cause thrashing with many tenants
|
|
- No cache invalidation on tenant deletion
|
|
|
|
---
|
|
|
|
## Web UI Findings
|
|
|
|
### UI-001: Tenant State Management
|
|
|
|
**Location:** `lightrag_webui/src/stores/tenant.ts`
|
|
|
|
**Status:** ✅ Well implemented
|
|
|
|
- Zustand store with persistence
|
|
- Automatic header injection via Axios interceptor
|
|
- Console logging for debugging
|
|
|
|
### UI-002: Document State Clearing
|
|
|
|
**Location:** `lightrag_webui/src/features/DocumentManager.tsx`
|
|
|
|
**Recent Fix:** State clearing on tenant/KB change
|
|
|
|
```typescript
|
|
useEffect(() => {
|
|
setCurrentPageDocs([]);
|
|
setDocs(null);
|
|
setStatusCounts({ all: 0 });
|
|
// ...
|
|
}, [selectedTenant?.tenant_id, selectedKB?.kb_id]);
|
|
```
|
|
|
|
**Status:** ✅ Fixed (per task log dated 2025-02-25)
|
|
|
|
### UI-003: Token Storage
|
|
|
|
**Finding:** Authentication token stored in localStorage
|
|
|
|
**Risk:** XSS vulnerability could expose tokens
|
|
|
|
**Recommendation:** Use httpOnly cookies for production
|
|
|
|
---
|
|
|
|
## Recommendations
|
|
|
|
### Priority 1: Critical Security Fixes
|
|
|
|
1. **Replace `get_tenant_context_optional` with `get_tenant_context`** for all data endpoints
|
|
2. **Remove admin bypass** in `tenant_service.py`
|
|
3. **Make `user_id` required** in RAG manager
|
|
|
|
### Priority 2: Defense-in-Depth
|
|
|
|
4. **Enable PostgreSQL RLS** for all tenant-scoped tables
|
|
5. **Validate workspace format** to prevent injection
|
|
6. **Add audit logging** for cross-tenant access attempts
|
|
|
|
### Priority 3: Operational Improvements
|
|
|
|
7. **Add cache invalidation** on tenant/KB deletion
|
|
8. **Implement rate limiting** per tenant
|
|
9. **Add metrics/monitoring** for tenant isolation verification
|
|
|
|
---
|
|
|
|
## Test Protocol for Manual Verification
|
|
|
|
### Prerequisites
|
|
```bash
|
|
# Start test databases
|
|
docker compose -f docker-compose.test-db.yml up -d
|
|
|
|
# Verify containers
|
|
docker ps --filter "name=lightrag-audit"
|
|
```
|
|
|
|
### Test Scenarios
|
|
|
|
#### Scenario 1: Cross-Tenant Data Access Prevention
|
|
1. Create Tenant A and KB
|
|
2. Upload document to Tenant A
|
|
3. Attempt to list documents with Tenant B headers
|
|
4. **Expected:** Empty list for Tenant B
|
|
|
|
#### Scenario 2: Missing Tenant Header
|
|
1. Send document list request without X-Tenant-ID header
|
|
2. **Expected:** 400 Bad Request OR empty response (not global data)
|
|
|
|
#### Scenario 3: Query Isolation
|
|
1. Insert data into Tenant A KB
|
|
2. Query with Tenant B context
|
|
3. **Expected:** Query returns no results from Tenant A
|
|
|
|
### Test Script
|
|
```bash
|
|
# Run API isolation tests
|
|
python docs/action_plan/multitenant-audit/scripts/test_api_isolation.py
|
|
```
|
|
|
|
---
|
|
|
|
## Conclusion
|
|
|
|
The LightRAG multi-tenant implementation provides **functional tenant isolation** through:
|
|
- Workspace-based table partitioning
|
|
- Per-tenant RAG instance management
|
|
- Header-based tenant context extraction
|
|
- Role-based permission model
|
|
|
|
However, the implementation has **security gaps** that must be addressed:
|
|
1. Optional tenant context fallback to global RAG
|
|
2. Admin user bypass in access control
|
|
3. No database-level security (RLS)
|
|
|
|
### Recommended Next Steps
|
|
|
|
1. 🔴 **Immediate:** Fix SEC-001 and SEC-002 before production
|
|
2. 🟡 **Short-term:** Enable PostgreSQL RLS
|
|
3. 🟢 **Long-term:** Add comprehensive audit logging and monitoring
|
|
|
|
---
|
|
|
|
## Appendices
|
|
|
|
### A. Files Reviewed
|
|
|
|
| Layer | Files |
|
|
|-------|-------|
|
|
| Web UI | `tenant.ts`, `client.ts`, `DocumentManager.tsx` |
|
|
| API | `dependencies.py`, `document_routes.py`, `query_routes.py`, `tenant_routes.py` |
|
|
| Service | `tenant_service.py`, `tenant_rag_manager.py` |
|
|
| Storage | `postgres_impl.py`, `utils_context.py`, `namespace.py` |
|
|
| Tests | `test_multitenant_e2e.py`, `test_tenant_security.py`, `test_multi_tenant_backends.py` |
|
|
|
|
### B. Test Environment
|
|
|
|
| Component | Version/Config |
|
|
|-----------|----------------|
|
|
| Python | 3.12.12 |
|
|
| PostgreSQL | pgvector/pgvector:pg16 |
|
|
| Redis | 7-alpine |
|
|
| OS | macOS (arm64) |
|
|
|
|
### C. Related Documentation
|
|
|
|
- `docs/0001-multi-tenant-architecture.md`
|
|
- `docs/0002-multi-tenant-visual-reference.md`
|
|
- `docs/0003-multi-tenant-documentation-index.md`
|
|
- `MULTITENANT_IMPLEMENTATION_SUMMARY.md`
|