From c1b935a0b9b48a6afcc4ab725af1c20047bd2f58 Mon Sep 17 00:00:00 2001
From: Copilot <198982749+Copilot@users.noreply.github.com>
Date: Wed, 1 Oct 2025 12:00:26 +0800
Subject: [PATCH] Add comprehensive TypeScript migration documentation for
LightRAG (#1)
* Initial plan
* Initialize reverse documentation directory and create analysis plan
Co-authored-by: raphaelmansuy <1003084+raphaelmansuy@users.noreply.github.com>
* Add executive summary and comprehensive architecture documentation
Co-authored-by: raphaelmansuy <1003084+raphaelmansuy@users.noreply.github.com>
* Add comprehensive data models and dependency migration documentation
Co-authored-by: raphaelmansuy <1003084+raphaelmansuy@users.noreply.github.com>
* Complete comprehensive TypeScript migration documentation suite
Co-authored-by: raphaelmansuy <1003084+raphaelmansuy@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: raphaelmansuy <1003084+raphaelmansuy@users.noreply.github.com>
---
reverse_documentation/00-README.md | 290 ++++
reverse_documentation/01-executive-summary.md | 271 ++++
.../02-architecture-documentation.md | 568 ++++++++
.../03-data-models-and-schemas.md | 890 +++++++++++++
.../04-dependency-migration-guide.md | 894 +++++++++++++
...ypescript-project-structure-and-roadmap.md | 1169 +++++++++++++++++
.../06-implementation-guide.md | 1004 ++++++++++++++
reverse_documentation/scratchpad.md | 84 ++
8 files changed, 5170 insertions(+)
create mode 100644 reverse_documentation/00-README.md
create mode 100644 reverse_documentation/01-executive-summary.md
create mode 100644 reverse_documentation/02-architecture-documentation.md
create mode 100644 reverse_documentation/03-data-models-and-schemas.md
create mode 100644 reverse_documentation/04-dependency-migration-guide.md
create mode 100644 reverse_documentation/05-typescript-project-structure-and-roadmap.md
create mode 100644 reverse_documentation/06-implementation-guide.md
create mode 100644 reverse_documentation/scratchpad.md
diff --git a/reverse_documentation/00-README.md b/reverse_documentation/00-README.md
new file mode 100644
index 00000000..5a6837f7
--- /dev/null
+++ b/reverse_documentation/00-README.md
@@ -0,0 +1,290 @@
+# LightRAG TypeScript Migration Documentation
+
+## Overview
+
+This documentation suite provides comprehensive technical analysis and migration guidance for reimplementing LightRAG from Python to TypeScript/Node.js. The documentation is designed for senior developers and architects who need to build a production-ready TypeScript version of LightRAG while maintaining functional parity with the original implementation.
+
+## Documentation Structure
+
+### 1. [Executive Summary](./01-executive-summary.md) (16KB)
+High-level overview of the LightRAG system, its capabilities, architecture, and key migration challenges.
+
+**Contents:**
+- System overview and core capabilities
+- Architecture at a glance with Mermaid diagram
+- Key technical characteristics
+- Migration challenges and recommended solutions
+- TypeScript technology stack recommendations
+- Success metrics and next steps
+
+**Target Audience:** Decision makers, project managers, senior architects
+
+### 2. [Architecture Documentation](./02-architecture-documentation.md) (33KB)
+Detailed system architecture with comprehensive diagrams showing component interactions, data flows, and design patterns.
+
+**Contents:**
+- 5-layer system architecture
+- Component interaction patterns
+- Document indexing data flow (with sequence diagram)
+- Query processing data flow (with sequence diagram)
+- Storage layer architecture
+- Concurrency and state management patterns
+- TypeScript migration considerations for each pattern
+
+**Key Diagrams:**
+- System architecture (5 layers)
+- Component interactions
+- Indexing sequence diagram
+- Query sequence diagram
+- Storage layer structure
+- Concurrency control patterns
+
+**Target Audience:** System architects, technical leads
+
+### 3. [Data Models and Schemas](./03-data-models-and-schemas.md) (27KB)
+Complete type system documentation with Python and TypeScript definitions side-by-side.
+
+**Contents:**
+- Core data models (TextChunk, Entity, Relationship, DocStatus)
+- Storage schema definitions (KV, Vector, Graph, DocStatus)
+- Query and response models
+- Configuration models
+- Python to TypeScript type mapping guide
+- Validation and serialization strategies
+
+**Key Features:**
+- Every data structure documented with field descriptions
+- TypeScript type definitions provided
+- Validation rules and constraints
+- Storage patterns explained
+- Runtime validation examples with Zod
+
+**Target Audience:** Developers, data engineers
+
+### 4. [Dependency Migration Guide](./04-dependency-migration-guide.md) (27KB)
+Comprehensive mapping of Python packages to Node.js/npm equivalents with complexity assessment.
+
+**Contents:**
+- Core dependencies mapping (40+ packages)
+- Storage driver equivalents (PostgreSQL, MongoDB, Redis, Neo4j, etc.)
+- LLM and embedding provider equivalents (OpenAI, Anthropic, Ollama)
+- API and web framework alternatives (FastAPI → Fastify)
+- Utility library equivalents
+- Async/await pattern differences
+- Migration complexity assessment (Low/Medium/High)
+- Version compatibility matrix
+
+**Key Tables:**
+- Python package → npm package mapping
+- Migration complexity per category
+- Recommended versions with notes
+- Code comparison examples
+
+**Target Audience:** Developers, technical leads
+
+### 5. [TypeScript Project Structure and Migration Roadmap](./05-typescript-project-structure-and-roadmap.md) (29KB)
+Complete project organization, configuration, and phase-by-phase implementation plan.
+
+**Contents:**
+- Recommended directory structure
+- Module organization patterns
+- Complete configuration files (package.json, tsconfig.json, etc.)
+- Build and development workflow
+- Testing strategy (unit, integration, E2E)
+- 14-week phase-by-phase migration roadmap
+- CI/CD pipeline configuration
+- Docker and deployment setup
+
+**Key Sections:**
+- Detailed file structure with example code
+- Configuration for all build tools
+- Testing examples with Vitest
+- Phase-by-phase roadmap with deliverables
+- Docker and Kubernetes configuration
+- GitHub Actions CI/CD
+
+**Target Audience:** Developers, DevOps engineers
+
+## Quick Start Guide
+
+### For Decision Makers
+1. Read [Executive Summary](./01-executive-summary.md) for high-level overview
+2. Review migration challenges and technology stack recommendations
+3. Check estimated timeline in [Migration Roadmap](./05-typescript-project-structure-and-roadmap.md#phase-by-phase-migration-roadmap)
+
+### For Architects
+1. Read [Architecture Documentation](./02-architecture-documentation.md) to understand system design
+2. Study component interaction patterns and data flows
+3. Review [Data Models](./03-data-models-and-schemas.md) for data architecture
+4. Check [TypeScript Project Structure](./05-typescript-project-structure-and-roadmap.md) for implementation approach
+
+### For Developers
+1. Start with [Data Models and Schemas](./03-data-models-and-schemas.md) to understand types
+2. Review [Dependency Migration Guide](./04-dependency-migration-guide.md) for library equivalents
+3. Study [Project Structure](./05-typescript-project-structure-and-roadmap.md) for code organization
+4. Follow phase-by-phase roadmap for implementation sequence
+
+## Key Insights
+
+### System Architecture
+- **5-layer architecture**: Presentation → API Gateway → Business Logic → Integration → Infrastructure
+- **4 storage types**: KV (cache/chunks), Vector (embeddings), Graph (entities/relations), DocStatus (pipeline state)
+- **6 query modes**: local, global, hybrid, mix, naive, bypass
+- **Async-first design**: Semaphore-based rate limiting, keyed locks, task queues
+
+### Migration Feasibility
+- **Overall Complexity**: Medium (12-14 weeks with small team)
+- **High-Risk Areas**: Vector search (FAISS alternatives), NetworkX (use graphology)
+- **Low-Risk Areas**: PostgreSQL, MongoDB, Redis, Neo4j, OpenAI, API layer
+- **Recommended Stack**: Node.js 20 LTS, TypeScript 5.3+, Fastify, Zod, pnpm
+
+### Technology Choices
+
+**Storage**:
+- PostgreSQL: `pg` + optional `drizzle-orm` for type safety
+- MongoDB: Official `mongodb` driver
+- Redis: `ioredis` for best TypeScript support
+- Neo4j: Official `neo4j-driver`
+- Graph: `graphology` (NetworkX equivalent)
+- Vector: Qdrant, Milvus, or PostgreSQL with pgvector
+
+**LLM Integration**:
+- OpenAI: Official `openai` SDK (v4+)
+- Anthropic: `@anthropic-ai/sdk`
+- Ollama: Official `ollama` package
+- Tokenization: `@dqbd/tiktoken` (WASM port)
+
+**Web Framework**:
+- API: `fastify` (FastAPI equivalent)
+- Validation: `zod` (Pydantic equivalent)
+- Authentication: `@fastify/jwt`
+- Documentation: `@fastify/swagger`
+
+**Utilities**:
+- Async control: `p-limit`, `p-queue`, `p-retry`
+- Logging: `pino` (fast, structured)
+- Testing: `vitest` (fast, TypeScript-native)
+- Build: `tsup` (fast bundler)
+
+## Implementation Roadmap Summary
+
+### Phase 1-2: Foundation & Storage (Weeks 1-5)
+- Set up project structure and tooling
+- Implement storage abstractions and PostgreSQL reference implementation
+- Add alternative storage backends (MongoDB, Redis, File-based)
+- **Deliverable**: Working storage layer with tests
+
+### Phase 3-4: LLM & Core Engine (Weeks 6-8)
+- Integrate LLM providers (OpenAI, Anthropic, Ollama)
+- Implement document processing pipeline (chunking, extraction, merging)
+- Add vector embedding and indexing
+- **Deliverable**: Complete document ingestion pipeline
+
+### Phase 5: Query Engine (Weeks 9-10)
+- Implement all 6 query modes
+- Add token budget management
+- Integrate reranking
+- **Deliverable**: Complete query functionality
+
+### Phase 6: API Layer (Week 11)
+- Build REST API with Fastify
+- Add authentication and authorization
+- Implement streaming responses
+- **Deliverable**: Production API
+
+### Phase 7-8: Testing & Production (Weeks 12-14)
+- Comprehensive testing (unit, integration, E2E)
+- Performance optimization
+- Production hardening (monitoring, logging, deployment)
+- **Deliverable**: Production-ready system
+
+## Documentation Statistics
+
+- **Total Documentation**: ~140KB across 5 major documents
+- **Mermaid Diagrams**: 6 comprehensive architecture diagrams
+- **Code Examples**: 100+ Python/TypeScript comparison snippets
+- **Dependency Mapping**: 40+ Python packages → npm equivalents
+- **Type Definitions**: Complete TypeScript types for all data structures
+- **Configuration Files**: 10+ complete config examples
+
+## Success Criteria
+
+A successful TypeScript migration achieves:
+
+1. **Functional Parity**: All query modes, storage backends, and LLM providers working identically
+2. **API Compatibility**: Existing WebUI works without modification
+3. **Performance**: Comparable or better throughput and latency
+4. **Type Safety**: Full TypeScript coverage, minimal use of `any`
+5. **Test Coverage**: >80% code coverage with comprehensive tests
+6. **Production Ready**: Handles errors gracefully, provides observability, scales horizontally
+7. **Documentation**: Complete API docs, deployment guides, migration notes
+
+## Additional Resources
+
+### Python Repository
+- **URL**: https://github.com/HKUDS/LightRAG
+- **Paper**: EMNLP 2025 - "LightRAG: Simple and Fast Retrieval-Augmented Generation"
+- **License**: MIT
+
+### Related Documentation
+- **README.md**: Repository root documentation
+- **README-zh.md**: Chinese version of documentation
+- **API Documentation**: `lightrag/api/README.md`
+- **Examples**: `examples/` directory with usage samples
+
+### Existing TypeScript Reference
+The repository already includes a TypeScript WebUI (`lightrag_webui/`) that provides:
+- TypeScript type definitions for API responses
+- API client implementation patterns
+- Data model usage examples
+- Component patterns that can be referenced
+
+## Getting Help
+
+### Common Questions
+
+**Q: Can I use alternative storage backends not mentioned?**
+A: Yes, as long as they implement the storage interfaces. The documentation provides patterns for implementing custom storage backends.
+
+**Q: Do I need to implement all query modes?**
+A: For a complete migration, yes. However, you can prioritize modes based on your use case (mix and naive are most commonly used).
+
+**Q: Can I use a different web framework than Fastify?**
+A: Yes, Express is a viable alternative. Fastify is recommended for performance and TypeScript support, but the core logic is framework-agnostic.
+
+**Q: How do I handle FAISS migration?**
+A: Use Qdrant or Milvus for production, or hnswlib-node for a local alternative. See the dependency guide for detailed comparison.
+
+**Q: Is the 14-week timeline realistic?**
+A: Yes, with a team of 2-3 experienced TypeScript developers working full-time. Adjust based on your team size and experience.
+
+### Contact
+
+For questions about this documentation or the migration:
+- Open an issue in the LightRAG repository
+- Refer to the original paper for algorithm details
+- Check existing examples in the `examples/` directory
+
+## Maintenance and Updates
+
+This documentation reflects the LightRAG codebase as of the repository snapshot date. Key version references:
+- **Python Version**: 3.10+
+- **Node.js Version**: 20 LTS recommended (18+ minimum)
+- **TypeScript Version**: 5.3+ recommended
+
+When updating this documentation:
+1. Keep Python and TypeScript examples in sync
+2. Update version numbers in dependency tables
+3. Validate code examples against latest library versions
+4. Update Mermaid diagrams if architecture changes
+5. Maintain consistent styling and formatting
+
+## License
+
+This documentation inherits the MIT license from the LightRAG project. Feel free to use, modify, and distribute as needed for your TypeScript implementation.
+
+---
+
+**Last Updated**: 2024
+**Documentation Version**: 1.0
+**Target LightRAG Version**: Latest (main branch)
diff --git a/reverse_documentation/01-executive-summary.md b/reverse_documentation/01-executive-summary.md
new file mode 100644
index 00000000..318c1d37
--- /dev/null
+++ b/reverse_documentation/01-executive-summary.md
@@ -0,0 +1,271 @@
+# Executive Summary: LightRAG TypeScript Migration
+
+## Overview
+
+LightRAG (Light Retrieval-Augmented Generation) is a sophisticated, graph-based RAG system implemented in Python that combines knowledge graph construction with vector retrieval to deliver contextually rich question-answering capabilities. This document provides comprehensive technical analysis to enable a production-ready TypeScript/Node.js reimplementation.
+
+**Repository**: [HKUDS/LightRAG](https://github.com/HKUDS/LightRAG)
+**Paper**: EMNLP 2025 - "LightRAG: Simple and Fast Retrieval-Augmented Generation"
+**Current Implementation**: Python 3.10+, ~58 Python files, ~500KB of core code
+**License**: MIT
+
+## System Capabilities
+
+LightRAG delivers a comprehensive RAG solution with the following core capabilities:
+
+### Document Processing Pipeline
+The system implements a multi-stage document processing pipeline that transforms raw documents into a queryable knowledge graph. Documents are ingested, split into semantic chunks, processed through LLM-based entity and relationship extraction, and merged into a unified knowledge graph with vector embeddings for retrieval. The pipeline supports multiple file formats (PDF, DOCX, PPTX, CSV, TXT) and handles batch processing with status tracking and error recovery.
+
+### Knowledge Graph Construction
+At the heart of LightRAG is an automated knowledge graph construction system that extracts entities and relationships from text using large language models. The system identifies entity types (Person, Organization, Location, Event, Concept, etc.), establishes relationships between entities, and maintains entity descriptions and relationship metadata. Graph merging algorithms consolidate duplicate entities and relationships, while maintaining source attribution and citation tracking.
+
+### Multi-Modal Retrieval Strategies
+LightRAG implements six distinct query modes to balance between specificity and coverage:
+- **Local Mode**: Focuses on entity-centric retrieval using low-level keywords to find specific, context-dependent information
+- **Global Mode**: Emphasizes relationship-centric retrieval using high-level keywords for broader, interconnected insights
+- **Hybrid Mode**: Combines local and global results using round-robin merging for balanced coverage
+- **Mix Mode**: Integrates knowledge graph data with vector-retrieved document chunks for comprehensive context
+- **Naive Mode**: Pure vector retrieval without knowledge graph integration for simple similarity search
+- **Bypass Mode**: Direct LLM query without retrieval for general questions
+
+### Flexible Storage Architecture
+The system supports multiple storage backends through a pluggable architecture:
+- **Key-Value Storage**: JSON files, PostgreSQL, MongoDB, Redis (for LLM cache, chunks, and documents)
+- **Vector Storage**: NanoVectorDB, FAISS, Milvus, Qdrant, PostgreSQL with pgvector (for embeddings)
+- **Graph Storage**: NetworkX, Neo4j, Memgraph, PostgreSQL (for entity-relationship graphs)
+- **Document Status Storage**: JSON files, PostgreSQL, MongoDB (for pipeline tracking)
+
+### Production Features
+The system includes enterprise-ready features for production deployment:
+- RESTful API with FastAPI and OpenAPI documentation
+- WebUI for document management, graph visualization, and querying
+- Authentication and authorization (JWT-based)
+- Streaming responses for real-time user feedback
+- Ollama-compatible API for integration with AI chatbots
+- Workspace isolation for multi-tenant deployments
+- Pipeline status tracking and monitoring
+- Configurable rate limiting and concurrency control
+- Error handling and retry mechanisms
+- Citation and source attribution support
+
+## Architecture at a Glance
+
+```mermaid
+graph TB
+ subgraph "Client Layer"
+ WebUI["WebUI
(TypeScript/React)"]
+ API["REST API Client"]
+ end
+
+ subgraph "API Layer"
+ FastAPI["FastAPI Server
(lightrag_server.py)"]
+ Routes["Route Handlers
Query/Document/Graph"]
+ Auth["Authentication
(JWT)"]
+ end
+
+ subgraph "Core LightRAG Engine"
+ LightRAG["LightRAG Class
(lightrag.py)"]
+ Operations["Operations Layer
(operate.py)"]
+ Utils["Utilities
(utils.py)"]
+ end
+
+ subgraph "Processing Pipeline"
+ Chunking["Text Chunking
(Token-based)"]
+ Extraction["Entity Extraction
(LLM-based)"]
+ Merging["Graph Merging
(Deduplication)"]
+ Indexing["Vector Indexing
(Embeddings)"]
+ end
+
+ subgraph "Query Pipeline"
+ KeywordExtract["Keyword Extraction
(High/Low Level)"]
+ GraphRetrieval["Graph Retrieval
(Entities/Relations)"]
+ VectorRetrieval["Vector Retrieval
(Chunks)"]
+ ContextBuild["Context Building
(Token Budget)"]
+ LLMGen["LLM Generation
(Response)"]
+ end
+
+ subgraph "LLM Integration"
+ LLMProvider["LLM Provider
(OpenAI, Ollama, etc.)"]
+ EmbedProvider["Embedding Provider
(text-embedding-3-small)"]
+ end
+
+ subgraph "Storage Layer"
+ KVStorage["KV Storage
(Cache/Chunks/Docs)"]
+ VectorStorage["Vector Storage
(Embeddings)"]
+ GraphStorage["Graph Storage
(Entities/Relations)"]
+ DocStatus["Doc Status Storage
(Pipeline State)"]
+ end
+
+ WebUI --> FastAPI
+ API --> FastAPI
+ FastAPI --> Routes
+ Routes --> Auth
+ Routes --> LightRAG
+
+ LightRAG --> Operations
+ LightRAG --> Utils
+
+ LightRAG --> Chunking
+ Chunking --> Extraction
+ Extraction --> Merging
+ Merging --> Indexing
+
+ LightRAG --> KeywordExtract
+ KeywordExtract --> GraphRetrieval
+ KeywordExtract --> VectorRetrieval
+ GraphRetrieval --> ContextBuild
+ VectorRetrieval --> ContextBuild
+ ContextBuild --> LLMGen
+
+ Operations --> LLMProvider
+ Operations --> EmbedProvider
+
+ LightRAG --> KVStorage
+ LightRAG --> VectorStorage
+ LightRAG --> GraphStorage
+ LightRAG --> DocStatus
+
+ style WebUI fill:#E6F3FF
+ style FastAPI fill:#FFE6E6
+ style LightRAG fill:#E6FFE6
+ style LLMProvider fill:#FFF5E6
+ style KVStorage fill:#FFE6E6
+ style VectorStorage fill:#FFE6E6
+ style GraphStorage fill:#FFE6E6
+ style DocStatus fill:#FFE6E6
+```
+
+## Key Technical Characteristics
+
+### Async-First Architecture
+The entire system is built on Python's asyncio, with extensive use of async/await patterns, semaphores for rate limiting, and task queues for concurrent processing. This design enables efficient handling of I/O-bound operations and supports high concurrency for embedding generation and LLM calls.
+
+### Storage Abstraction Pattern
+LightRAG implements a clean abstraction layer over storage backends through base classes (BaseKVStorage, BaseVectorStorage, BaseGraphStorage, DocStatusStorage). This pattern enables seamless switching between different storage implementations without modifying core logic, supporting everything from in-memory JSON files to enterprise databases like PostgreSQL and Neo4j.
+
+### Pipeline-Based Processing
+Document ingestion follows a multi-stage pipeline pattern: enqueue → validate → chunk → extract → merge → index. Each stage is idempotent and resumable, with comprehensive status tracking enabling fault tolerance and progress monitoring. Documents flow through the pipeline with track IDs for monitoring and debugging.
+
+### Token Budget Management
+The system implements sophisticated token budget management for query contexts, allocating tokens across entities, relationships, and chunks while respecting LLM context window limits. This unified token control system ensures optimal use of available context space and prevents token overflow errors.
+
+### Modular LLM Integration
+LLM and embedding providers are abstracted behind function interfaces, supporting multiple providers (OpenAI, Ollama, Anthropic, AWS Bedrock, Azure OpenAI, Hugging Face, and more) with consistent error handling, retry logic, and rate limiting across all providers.
+
+## Key Migration Challenges
+
+### Challenge 1: Monolithic File Structure
+**Issue**: Core logic is concentrated in large files (lightrag.py: 141KB, operate.py: 164KB, utils.py: 106KB) with high cyclomatic complexity.
+**Impact**: Direct translation would create unmaintainable TypeScript code.
+**Strategy**: Refactor into smaller, focused modules following single responsibility principle. Break down large classes into composition patterns. Leverage TypeScript's module system for better organization.
+
+### Challenge 2: Python-Specific Language Features
+**Issue**: Heavy use of Python dataclasses, decorators (@final, @dataclass), type hints (TypedDict, Literal, overload), and metaprogramming patterns.
+**Impact**: These features don't have direct TypeScript equivalents.
+**Strategy**: Use TypeScript classes with decorators from libraries like class-validator and class-transformer. Leverage TypeScript's type system for Literal types and union types. Replace overload decorators with function overloading syntax.
+
+### Challenge 3: Async/Await Pattern Differences
+**Issue**: Python's asyncio model differs from Node.js event loop, particularly in semaphore usage, task cancellation, and exception handling in concurrent operations.
+**Impact**: Concurrency patterns require redesign for Node.js runtime.
+**Strategy**: Use p-limit for semaphore-like behavior, AbortController for cancellation, and Promise.allSettled for concurrent operations with individual error handling. Leverage async iterators for streaming.
+
+### Challenge 4: Storage Driver Ecosystem
+**Issue**: Python has mature drivers for PostgreSQL (asyncpg), MongoDB (motor), Neo4j (neo4j-driver), Redis (redis-py), while Node.js alternatives have different APIs and capabilities.
+**Impact**: Storage layer requires careful driver selection and adapter implementation.
+**Strategy**: Use node-postgres for PostgreSQL, mongodb driver for MongoDB, neo4j-driver-lite for Neo4j, ioredis for Redis. Create consistent adapter layer to abstract driver differences.
+
+### Challenge 5: Embedding and Tokenization Libraries
+**Issue**: tiktoken (OpenAI's tokenizer) and sentence-transformers have limited or no Node.js support.
+**Impact**: Need alternative approaches for tokenization and local embeddings.
+**Strategy**: Use @dqbd/tiktoken (WASM port) for tokenization, or js-tiktoken as alternative. For embeddings, use OpenAI API, Hugging Face Inference API, or ONNX Runtime for local model inference.
+
+### Challenge 6: Complex State Management
+**Issue**: Python uses global dictionaries and namespace-based state management (global_config, pipeline_status, keyed locks) with multiprocessing considerations.
+**Impact**: State management in Node.js requires different patterns.
+**Strategy**: Use class-based state management with dependency injection. Implement singleton pattern for shared state. Use Redis or similar for distributed state in multi-process deployments.
+
+## Recommended TypeScript Technology Stack
+
+### Runtime and Core
+- **Runtime**: Node.js 20 LTS (for latest async features and stability)
+- **Language**: TypeScript 5.3+ (for latest type system features)
+- **Build Tool**: esbuild or swc (for fast builds)
+- **Package Manager**: pnpm (for efficient dependency management)
+
+### Web Framework
+- **API Framework**: Fastify or Express with TypeScript
+- **Validation**: Zod or class-validator
+- **OpenAPI**: @fastify/swagger or tsoa
+
+### Storage Drivers
+- **PostgreSQL**: pg with @types/pg, or Drizzle ORM for type-safe queries
+- **MongoDB**: mongodb driver with TypeScript support
+- **Neo4j**: neo4j-driver with TypeScript bindings
+- **Redis**: ioredis (best TypeScript support)
+- **Vector**: @pinecone-database/pinecone, qdrant-client, or pg with pgvector
+
+### LLM and Embeddings
+- **OpenAI**: openai (official SDK)
+- **Anthropic**: @anthropic-ai/sdk
+- **Generic LLM**: langchain or custom adapters
+- **Tokenization**: @dqbd/tiktoken or js-tiktoken
+- **Embeddings**: OpenAI API, or @xenova/transformers for local
+
+### Utilities
+- **Async Control**: p-limit, p-queue, bottleneck
+- **Logging**: pino or winston
+- **Configuration**: dotenv, convict
+- **Testing**: vitest (fast, TypeScript-native)
+- **Hashing**: crypto (built-in), or js-md5
+- **JSON Repair**: json-repair-ts
+
+## Migration Approach Recommendation
+
+### Phase 1: Core Abstractions (Weeks 1-2)
+Establish foundational abstractions: storage interfaces, base classes, type definitions, and configuration management. This creates the contract layer that all other components will depend on. Implement basic in-memory storage to enable early testing.
+
+### Phase 2: Storage Layer (Weeks 3-5)
+Implement storage adapters for primary backends (PostgreSQL, NetworkX-equivalent using graphology, NanoVectorDB-equivalent). Focus on KV and Vector storage first, then Graph storage, finally Doc Status storage. Each storage type should pass identical test suites regardless of backend.
+
+### Phase 3: LLM Integration (Weeks 4-6, parallel)
+Build LLM and embedding provider adapters, starting with OpenAI as reference implementation. Implement retry logic, rate limiting, and error handling. Create abstract interfaces that other providers can implement. Add streaming support for responses.
+
+### Phase 4: Core Engine (Weeks 6-8)
+Implement the LightRAG core engine: chunking, entity extraction, graph merging, and indexing pipeline. This requires integrating storage, LLM, and utility layers. Focus on making the pipeline idempotent and resumable with comprehensive state tracking.
+
+### Phase 5: Query Pipeline (Weeks 8-10)
+Build the query engine with all six retrieval modes. Implement keyword extraction, graph retrieval, vector retrieval, context building with token budgets, and response generation. Add support for conversation history and streaming responses.
+
+### Phase 6: API Layer (Weeks 10-11)
+Develop RESTful API with Fastify or Express, implementing all endpoints from the Python version. Add authentication, authorization, request validation, and OpenAPI documentation. Ensure API compatibility with existing WebUI.
+
+### Phase 7: Testing and Optimization (Weeks 11-13)
+Comprehensive testing including unit tests, integration tests, and end-to-end tests. Performance testing and optimization, particularly for concurrent operations. Load testing for production readiness. Documentation updates.
+
+### Phase 8: Production Hardening (Weeks 13-14)
+Add monitoring, logging, error tracking, health checks, and deployment configurations. Implement graceful shutdown, connection pooling, and resource cleanup. Create Docker images and Kubernetes configurations.
+
+## Success Metrics
+
+A successful TypeScript implementation should achieve:
+
+1. **Functional Parity**: All query modes, storage backends, and LLM providers working identically to Python version
+2. **API Compatibility**: Existing WebUI works without modification against TypeScript API
+3. **Performance**: Comparable or better throughput and latency for document ingestion and query operations
+4. **Type Safety**: Full TypeScript type coverage with no 'any' types in core logic
+5. **Test Coverage**: >80% code coverage with unit and integration tests
+6. **Production Ready**: Handles errors gracefully, provides observability, scales horizontally
+7. **Documentation**: Complete API documentation, deployment guides, and migration notes
+
+## Next Steps
+
+This executive summary provides the foundation for detailed technical documentation. The following documents dive deeper into:
+
+- **Architecture Documentation**: Detailed system design with comprehensive diagrams
+- **Data Models and Schemas**: Complete type definitions for all data structures
+- **Storage Layer Specification**: In-depth analysis of each storage implementation
+- **LLM Integration Guide**: Provider-specific integration patterns
+- **API Reference**: Complete endpoint documentation with TypeScript types
+- **Implementation Roadmap**: Detailed phase-by-phase migration guide
+
+Each subsequent document builds on this foundation, providing the specific technical details needed to implement a production-ready TypeScript version of LightRAG.
diff --git a/reverse_documentation/02-architecture-documentation.md b/reverse_documentation/02-architecture-documentation.md
new file mode 100644
index 00000000..e0da4614
--- /dev/null
+++ b/reverse_documentation/02-architecture-documentation.md
@@ -0,0 +1,568 @@
+# Architecture Documentation: LightRAG System Design
+
+## Table of Contents
+1. [System Architecture Overview](#system-architecture-overview)
+2. [Component Interaction Architecture](#component-interaction-architecture)
+3. [Document Indexing Data Flow](#document-indexing-data-flow)
+4. [Query Processing Data Flow](#query-processing-data-flow)
+5. [Storage Layer Architecture](#storage-layer-architecture)
+6. [Concurrency and State Management](#concurrency-and-state-management)
+
+## System Architecture Overview
+
+LightRAG follows a layered architecture pattern with clear separation of concerns. The system is structured into five primary layers, each with specific responsibilities and well-defined interfaces.
+
+```mermaid
+graph TB
+ subgraph "Presentation Layer"
+ A1["WebUI (TypeScript)
React + Vite"]
+ A2["API Clients
REST/OpenAPI"]
+ end
+
+ subgraph "API Gateway Layer"
+ B1["FastAPI Server
lightrag_server.py"]
+ B2["Authentication
JWT + API Keys"]
+ B3["Request Validation
Pydantic Models"]
+ B4["Route Handlers
Query/Document/Graph"]
+ end
+
+ subgraph "Business Logic Layer"
+ C1["LightRAG Core
lightrag.py"]
+ C2["Operations Module
operate.py"]
+ C3["Utilities
utils.py + utils_graph.py"]
+ C4["Prompt Templates
prompt.py"]
+ end
+
+ subgraph "Integration Layer"
+ D1["LLM Providers
OpenAI, Ollama, etc."]
+ D2["Embedding Providers
text-embedding-*"]
+ D3["Storage Adapters
KV/Vector/Graph/Status"]
+ end
+
+ subgraph "Infrastructure Layer"
+ E1["PostgreSQL
Relational + Vector"]
+ E2["Neo4j/Memgraph
Graph Database"]
+ E3["Redis/MongoDB
Cache + NoSQL"]
+ E4["File System
JSON + FAISS"]
+ end
+
+ A1 --> B1
+ A2 --> B1
+ B1 --> B2
+ B2 --> B3
+ B3 --> B4
+ B4 --> C1
+ C1 --> C2
+ C1 --> C3
+ C2 --> C4
+ C1 --> D1
+ C1 --> D2
+ C1 --> D3
+ D3 --> E1
+ D3 --> E2
+ D3 --> E3
+ D3 --> E4
+
+ style A1 fill:#E6F3FF
+ style B1 fill:#FFE6E6
+ style C1 fill:#E6FFE6
+ style D1 fill:#FFF5E6
+ style E1 fill:#FFE6F5
+```
+
+### Layer Responsibilities
+
+**Presentation Layer**: Handles user interactions through a React-based WebUI and provides REST API client capabilities. Responsible for rendering data, handling user input, and managing client-side state. Written in TypeScript with React, this layer communicates with the API Gateway exclusively through HTTP/HTTPS.
+
+**API Gateway Layer**: Manages all external communication with the system. Implements authentication and authorization using JWT tokens and API keys, validates incoming requests using Pydantic models, handles rate limiting, and routes requests to appropriate handlers. Built with FastAPI, it provides automatic OpenAPI documentation and request/response validation.
+
+**Business Logic Layer**: Contains the core intelligence of LightRAG. The LightRAG class orchestrates all operations, managing document processing pipelines, query execution, and storage coordination. The Operations module handles entity extraction, graph merging, and retrieval algorithms. Utilities provide helper functions for text processing, tokenization, hashing, and caching. Prompt templates define structured prompts for LLM interactions.
+
+**Integration Layer**: Abstracts external dependencies through consistent interfaces. LLM provider adapters normalize different API formats (OpenAI, Anthropic, Ollama, etc.) into a common interface. Embedding provider adapters handle various embedding services. Storage adapters implement the abstract storage interfaces (BaseKVStorage, BaseVectorStorage, BaseGraphStorage, DocStatusStorage) for different backends.
+
+**Infrastructure Layer**: Provides the foundational data persistence and retrieval capabilities. Supports multiple database systems including PostgreSQL (with pgvector for vector storage), Neo4j and Memgraph (for graph storage), Redis and MongoDB (for caching and document storage), and file-based storage (JSON files, FAISS indexes) for development and small deployments.
+
+## Component Interaction Architecture
+
+This diagram illustrates how major components interact during typical operations, showing both document indexing and query execution flows.
+
+```mermaid
+graph LR
+ subgraph "External Systems"
+ LLM["LLM Service
(OpenAI/Ollama)"]
+ Embed["Embedding Service"]
+ end
+
+ subgraph "Core Components"
+ RAG["LightRAG
Core Engine"]
+ OPS["Operations
Module"]
+ PIPE["Pipeline
Manager"]
+ end
+
+ subgraph "Storage System"
+ KV["KV Storage
Chunks/Cache"]
+ VEC["Vector Storage
Embeddings"]
+ GRAPH["Graph Storage
Entities/Relations"]
+ STATUS["Status Storage
Pipeline State"]
+ end
+
+ subgraph "Processing Components"
+ CHUNK["Chunking
Engine"]
+ EXTRACT["Entity
Extraction"]
+ MERGE["Graph
Merging"]
+ QUERY["Query
Engine"]
+ end
+
+ RAG --> PIPE
+ PIPE --> CHUNK
+ CHUNK --> EXTRACT
+ EXTRACT --> OPS
+ OPS --> LLM
+ OPS --> Embed
+ OPS --> MERGE
+ MERGE --> VEC
+ MERGE --> GRAPH
+ CHUNK --> KV
+ PIPE --> STATUS
+
+ RAG --> QUERY
+ QUERY --> OPS
+ QUERY --> VEC
+ QUERY --> GRAPH
+ QUERY --> KV
+ QUERY --> LLM
+
+ style RAG fill:#E6FFE6
+ style LLM fill:#FFF5E6
+ style KV fill:#FFE6E6
+ style VEC fill:#FFE6E6
+ style GRAPH fill:#FFE6E6
+ style STATUS fill:#FFE6E6
+```
+
+### Component Interaction Patterns
+
+**Document Ingestion Pattern**: Client submits documents to the API Gateway, which authenticates and validates the request before passing it to the LightRAG core. The core initializes a pipeline instance with a unique track ID, stores the document in KV storage, and updates its status in Status storage. The Pipeline Manager coordinates the chunking, extraction, merging, and indexing stages, maintaining progress information throughout.
+
+**Entity Extraction Pattern**: The Operations module receives text chunks from the Pipeline Manager and constructs prompts using templates from prompt.py. These prompts are sent to the configured LLM service, which returns structured entity and relationship data. The Operations module parses the response, normalizes entity names, and prepares data for graph merging.
+
+**Graph Merging Pattern**: When new entities and relationships are extracted, the Merge component compares them against existing graph data. For matching entities (based on name similarity), it consolidates descriptions, merges metadata, and updates source references. For relationships, it deduplicates based on source-target pairs and aggregates weights. The merged data is then stored in Graph storage and vector representations are computed and stored in Vector storage.
+
+**Query Execution Pattern**: The Query Engine receives a user query and determines the appropriate retrieval strategy based on the query mode. It extracts high-level and low-level keywords using the LLM, retrieves relevant entities and relationships from Graph storage, fetches related chunks from Vector storage, builds a context respecting token budgets, and finally generates a response using the LLM with the assembled context.
+
+## Document Indexing Data Flow
+
+This sequence diagram details the complete flow of document processing from ingestion to indexing.
+
+```mermaid
+sequenceDiagram
+ participant Client
+ participant API as API Server
+ participant Core as LightRAG Core
+ participant Pipeline
+ participant Chunking
+ participant Extraction
+ participant Merging
+ participant KV as KV Storage
+ participant Vec as Vector Storage
+ participant Graph as Graph Storage
+ participant Status as Status Storage
+ participant LLM as LLM Service
+ participant Embed as Embedding Service
+
+ Client->>API: POST /documents/upload
+ API->>API: Authenticate & Validate
+ API->>Core: ainsert(document, file_path)
+
+ Note over Core: Initialization Phase
+ Core->>Core: Generate track_id
+ Core->>Status: Create doc status (PENDING)
+ Core->>KV: Store document content
+ Core->>Pipeline: apipeline_process_enqueue_documents()
+
+ Note over Pipeline: Chunking Phase
+ Pipeline->>Status: Update status (CHUNKING)
+ Pipeline->>Chunking: chunking_by_token_size()
+ Chunking->>Chunking: Tokenize & split by overlap
+ Chunking-->>Pipeline: chunks[]
+ Pipeline->>KV: Store chunks with metadata
+
+ Note over Pipeline: Extraction Phase
+ Pipeline->>Status: Update status (EXTRACTING)
+ loop For each chunk
+ Pipeline->>Extraction: extract_entities(chunk)
+ Extraction->>LLM: Generate entities/relations
+ LLM-->>Extraction: Structured output
+ Extraction->>Extraction: Parse & normalize
+ Extraction-->>Pipeline: entities[], relations[]
+ end
+
+ Note over Pipeline: Merging Phase
+ Pipeline->>Status: Update status (MERGING)
+ Pipeline->>Merging: merge_nodes_and_edges()
+
+ par Parallel Entity Processing
+ loop For each entity
+ Merging->>Graph: Check if entity exists
+ alt Entity exists
+ Merging->>LLM: Summarize descriptions
+ LLM-->>Merging: Merged description
+ Merging->>Graph: Update entity
+ else New entity
+ Merging->>Graph: Insert entity
+ end
+ Merging->>Embed: Generate embedding
+ Embed-->>Merging: embedding vector
+ Merging->>Vec: Store entity embedding
+ end
+ and Parallel Relationship Processing
+ loop For each relationship
+ Merging->>Graph: Check if relation exists
+ alt Relation exists
+ Merging->>Graph: Update weight & metadata
+ else New relation
+ Merging->>Graph: Insert relation
+ end
+ Merging->>Embed: Generate embedding
+ Embed-->>Merging: embedding vector
+ Merging->>Vec: Store relation embedding
+ end
+ end
+
+ Note over Pipeline: Indexing Phase
+ Pipeline->>Status: Update status (INDEXING)
+ Pipeline->>Vec: Index chunk embeddings
+ Pipeline->>Graph: Build graph indices
+ Pipeline->>KV: Commit cache
+
+ Note over Pipeline: Completion
+ Pipeline->>Status: Update status (COMPLETED)
+ Pipeline-->>Core: Success
+ Core-->>API: track_id, status
+ API-->>Client: 200 OK {track_id}
+
+ Note over Client: Client can poll /status/{track_id}
+```
+
+### Indexing Phase Details
+
+**Document Reception and Validation**: The API server receives the document, validates the file format and size, authenticates the request, and generates a unique track ID for monitoring. The document content is immediately stored in KV storage with metadata including file path, upload timestamp, and original filename.
+
+**Chunking Strategy**: Documents are split into overlapping chunks using a token-based approach. The system tokenizes the entire document using tiktoken, creates chunks of configurable size (default 1200 tokens), adds overlap between consecutive chunks (default 100 tokens) to preserve context, and stores each chunk with position metadata and references to the source document.
+
+**Entity and Relationship Extraction**: For each chunk, the system constructs a specialized prompt that instructs the LLM to identify entities with specific types and relationships between them. The LLM returns structured output in a specific format (entity|name|type|description for entities, relation|source|target|keywords|description for relationships). The system parses this output, normalizes entity names using case-insensitive matching, and validates the structure before proceeding.
+
+**Graph Construction and Merging**: New entities are compared against existing entities in the graph using fuzzy matching. When duplicates are found, descriptions are merged using either LLM-based summarization (for complex cases) or simple concatenation (for simple cases). Relationships are deduplicated based on source-target pairs, with weights aggregated when duplicates are found. All graph modifications are protected by keyed locks to ensure consistency in concurrent operations.
+
+**Vector Embedding Generation**: Entity descriptions, relationship descriptions, and chunk content are sent to the embedding service in batches for efficient processing. Embeddings are generated using the configured model (e.g., text-embedding-3-small for OpenAI), and vectors are stored in Vector storage with metadata linking back to their source entities, relationships, or chunks. The system uses semaphores to limit concurrent embedding requests and prevent rate limit errors.
+
+**Status Tracking Throughout**: Every stage updates the document status in Status storage, recording the current phase, progress percentage, error messages if any, and timing information. This enables clients to poll for progress and provides diagnostic information for debugging failed indexing operations.
+
+## Query Processing Data Flow
+
+This sequence diagram illustrates the retrieval and response generation process for different query modes.
+
+```mermaid
+sequenceDiagram
+ participant Client
+ participant API as API Server
+ participant Core as LightRAG Core
+ participant Query as Query Engine
+ participant KW as Keyword Extractor
+ participant Graph as Graph Storage
+ participant Vec as Vector Storage
+ participant KV as KV Storage
+ participant Context as Context Builder
+ participant LLM as LLM Service
+ participant Rerank as Rerank Service
+
+ Client->>API: POST /query (query, mode, params)
+ API->>API: Authenticate & Validate
+ API->>Core: aquery(query, QueryParam)
+ Core->>Query: Execute query
+
+ Note over Query: Keyword Extraction Phase
+ Query->>KW: Extract keywords
+ KW->>LLM: Generate high/low level keywords
+ LLM-->>KW: {hl_keywords[], ll_keywords[]}
+ KW-->>Query: keywords
+
+ alt Mode: local (Entity-centric)
+ Note over Query: Local Mode - Focus on Entities
+ Query->>Vec: Query entity vectors (ll_keywords)
+ Vec-->>Query: top_k entity_ids[]
+ Query->>Graph: Get entities by IDs
+ Graph-->>Query: entities[]
+ Query->>Graph: Get connected relations
+ Graph-->>Query: relations[]
+
+ else Mode: global (Relationship-centric)
+ Note over Query: Global Mode - Focus on Relations
+ Query->>Vec: Query relation vectors (hl_keywords)
+ Vec-->>Query: top_k relation_ids[]
+ Query->>Graph: Get relations by IDs
+ Graph-->>Query: relations[]
+ Query->>Graph: Get connected entities
+ Graph-->>Query: entities[]
+
+ else Mode: hybrid
+ Note over Query: Hybrid Mode - Combined
+ par Parallel Retrieval
+ Query->>Vec: Query entity vectors
+ Vec-->>Query: entity_ids[]
+ and
+ Query->>Vec: Query relation vectors
+ Vec-->>Query: relation_ids[]
+ end
+ Query->>Graph: Get entities and relations
+ Graph-->>Query: entities[], relations[]
+ Query->>Query: Merge with round-robin
+
+ else Mode: mix
+ Note over Query: Mix Mode - KG + Chunks
+ par Parallel Retrieval
+ Query->>Vec: Query entity vectors
+ Vec-->>Query: entity_ids[]
+ Query->>Graph: Get entities
+ Graph-->>Query: entities[]
+ and
+ Query->>Vec: Query chunk vectors
+ Vec-->>Query: chunk_ids[]
+ end
+
+ else Mode: naive
+ Note over Query: Naive Mode - Pure Vector
+ Query->>Vec: Query chunk vectors only
+ Vec-->>Query: top_k chunk_ids[]
+ end
+
+ Note over Query: Chunk Retrieval Phase
+ alt Mode != bypass
+ Query->>Query: Get related chunks from entities/relations
+ Query->>KV: Get chunks by IDs
+ KV-->>Query: chunks[]
+
+ opt Rerank enabled
+ Query->>Rerank: Rerank chunks
+ Rerank-->>Query: reranked_chunks[]
+ end
+ end
+
+ Note over Query: Context Building Phase
+ Query->>Context: Build context with token budget
+ Context->>Context: Allocate tokens (entities/relations/chunks)
+ Context->>Context: Truncate to fit budget
+ Context->>Context: Format entities/relations/chunks
+ Context-->>Query: context_string, references[]
+
+ Note over Query: Response Generation Phase
+ Query->>Query: Build prompt with context
+ opt Include conversation history
+ Query->>Query: Add history messages
+ end
+
+ alt Stream enabled
+ Query->>LLM: Stream generate (prompt, context)
+ loop Streaming chunks
+ LLM-->>Query: chunk
+ Query-->>API: chunk
+ API-->>Client: SSE chunk
+ end
+ else Stream disabled
+ Query->>LLM: Generate (prompt, context)
+ LLM-->>Query: response
+ Query-->>Core: response, references
+ Core-->>API: {response, references, metadata}
+ API-->>Client: 200 OK {response}
+ end
+```
+
+### Query Processing Phase Details
+
+**Keyword Extraction Phase**: The system sends the user query to the LLM with a specialized prompt that asks for high-level keywords (abstract concepts and themes) and low-level keywords (specific entities and terms). The LLM returns structured JSON with both keyword types, which guide the subsequent retrieval strategy. This two-level keyword approach enables the system to retrieve both broad contextual information and specific detailed facts.
+
+**Mode-Specific Retrieval Strategies**:
+
+*Local Mode* focuses on entity-centric retrieval by querying the vector storage using low-level keywords to find the most relevant entities, retrieving full entity details including descriptions and metadata, and then fetching all relationships connected to those entities. This mode is optimal for questions about specific entities or localized information.
+
+*Global Mode* emphasizes relationship-centric retrieval by querying vector storage using high-level keywords to find relevant relationships, retrieving relationship details including keywords and descriptions, and then fetching the entities connected by those relationships. This mode excels at questions about connections, patterns, and higher-level concepts.
+
+*Hybrid Mode* combines both approaches by running local and global retrieval in parallel and then merging results using a round-robin strategy to balance entity and relationship information. This provides comprehensive coverage for complex queries that require both types of information.
+
+*Mix Mode* integrates knowledge graph retrieval with direct chunk retrieval by querying entity vectors to get graph-based context, simultaneously querying chunk vectors for relevant document sections, and combining both types of results. This mode provides the most complete context by including both structured knowledge and raw document content.
+
+*Naive Mode* performs pure vector similarity search without using the knowledge graph, simply retrieving the most similar chunks based on embedding distance. This mode is fastest and works well for simple similarity-based retrieval without needing entity or relationship context.
+
+*Bypass Mode* skips retrieval entirely and sends the query directly to the LLM, useful for general questions that don't require specific document context or when testing the LLM's base knowledge.
+
+**Context Building with Token Budgets**: The system implements a sophisticated token budget management system that allocates a maximum number of tokens across different context components. It allocates tokens to entity descriptions (default 6000 tokens), relationship descriptions (default 8000 tokens), and chunk content (remaining budget, with a cap defined by chunk_top_k). The system truncates each component to fit within its budget using the tokenizer, prioritizing higher-ranked items when truncation is necessary, and ensures the total context doesn't exceed the max_total_tokens limit (default 30000 tokens).
+
+**Reranking for Improved Relevance**: When enabled, the reranking phase takes retrieved chunks and reranks them using a specialized reranking model (like Cohere rerank or Jina rerank). This cross-encoder approach provides more accurate relevance scoring than pure vector similarity, especially for semantic matching. Chunks below the minimum rerank score threshold are filtered out, and only the top-k chunks after reranking are included in the final context.
+
+**Response Generation with Streaming**: For streaming responses, the system establishes a connection to the LLM with stream=True, receives response tokens incrementally, and immediately forwards them to the client via Server-Sent Events (SSE). This provides real-time feedback to users and reduces perceived latency. For non-streaming responses, the system waits for the complete LLM response before returning it to the client along with metadata about entities, relationships, and chunks used in the context.
+
+## Storage Layer Architecture
+
+The storage layer implements a plugin architecture with abstract base classes defining the contract for each storage type.
+
+```mermaid
+graph TB
+ subgraph "Abstract Interfaces"
+ BASE["StorageNameSpace
(Base Class)"]
+ KV_BASE["BaseKVStorage"]
+ VEC_BASE["BaseVectorStorage"]
+ GRAPH_BASE["BaseGraphStorage"]
+ STATUS_BASE["DocStatusStorage"]
+ end
+
+ subgraph "KV Storage Implementations"
+ JSON_KV["JsonKVStorage
(File-based)"]
+ PG_KV["PGKVStorage
(PostgreSQL)"]
+ MONGO_KV["MongoKVStorage
(MongoDB)"]
+ REDIS_KV["RedisKVStorage
(Redis)"]
+ end
+
+ subgraph "Vector Storage Implementations"
+ NANO["NanoVectorDBStorage
(In-memory)"]
+ FAISS["FaissVectorDBStorage
(FAISS)"]
+ PG_VEC["PGVectorStorage
(pgvector)"]
+ MILVUS["MilvusVectorStorage
(Milvus)"]
+ QDRANT["QdrantVectorStorage
(Qdrant)"]
+ end
+
+ subgraph "Graph Storage Implementations"
+ NX["NetworkXStorage
(NetworkX)"]
+ NEO4J["Neo4jStorage
(Neo4j)"]
+ MEMGRAPH["MemgraphStorage
(Memgraph)"]
+ PG_GRAPH["PGGraphStorage
(PostgreSQL)"]
+ end
+
+ subgraph "Doc Status Implementations"
+ JSON_STATUS["JsonDocStatusStorage
(File-based)"]
+ PG_STATUS["PGDocStatusStorage
(PostgreSQL)"]
+ MONGO_STATUS["MongoDocStatusStorage
(MongoDB)"]
+ end
+
+ BASE --> KV_BASE
+ BASE --> VEC_BASE
+ BASE --> GRAPH_BASE
+ BASE --> STATUS_BASE
+
+ KV_BASE --> JSON_KV
+ KV_BASE --> PG_KV
+ KV_BASE --> MONGO_KV
+ KV_BASE --> REDIS_KV
+
+ VEC_BASE --> NANO
+ VEC_BASE --> FAISS
+ VEC_BASE --> PG_VEC
+ VEC_BASE --> MILVUS
+ VEC_BASE --> QDRANT
+
+ GRAPH_BASE --> NX
+ GRAPH_BASE --> NEO4J
+ GRAPH_BASE --> MEMGRAPH
+ GRAPH_BASE --> PG_GRAPH
+
+ STATUS_BASE --> JSON_STATUS
+ STATUS_BASE --> PG_STATUS
+ STATUS_BASE --> MONGO_STATUS
+
+ style BASE fill:#E6FFE6
+ style JSON_KV fill:#E6F3FF
+ style NANO fill:#FFE6E6
+ style NX fill:#FFF5E6
+ style JSON_STATUS fill:#FFE6F5
+```
+
+### Storage Interface Contracts
+
+**BaseKVStorage Interface**: Key-value storage manages cached data, text chunks, and full documents. Core methods include get_by_id(id) for retrieving a single value, get_by_ids(ids) for batch retrieval, filter_keys(keys) to check which keys don't exist, upsert(data) for inserting or updating entries, delete(ids) for removing entries, and index_done_callback() for persisting changes to disk. This interface supports both in-memory implementations with persistence and direct database implementations.
+
+**BaseVectorStorage Interface**: Vector storage handles embeddings for entities, relationships, and chunks. Core methods include query(query, top_k, query_embedding) for similarity search, upsert(data) for storing vectors with metadata, delete(ids) for removing vectors, delete_entity(entity_name) for removing entity-related vectors, delete_entity_relation(entity_name) for removing relationship vectors, get_by_id(id) and get_by_ids(ids) for retrieving full vector data, and get_vectors_by_ids(ids) for efficient vector-only retrieval. All implementations must support cosine similarity search and metadata filtering.
+
+**BaseGraphStorage Interface**: Graph storage maintains the entity-relationship graph structure. Core methods include has_node(node_id) and has_edge(source, target) for existence checks, node_degree(node_id) and edge_degree(src, tgt) for connectivity metrics, get_node(node_id) and upsert_node(node_id, data) for node operations, upsert_edge(source, target, data) for relationship operations, get_knowledge_graph(node_label, max_depth, max_nodes) for graph traversal and export, and delete operations for nodes and edges. The interface treats all relationships as undirected unless explicitly specified.
+
+**DocStatusStorage Interface**: Document status storage tracks processing pipeline state for each document. Core methods include upsert(status) for updating document status, get_by_id(doc_id) for retrieving status, get_by_ids(doc_ids) for batch retrieval, filter_ids(doc_ids) for checking existence, delete(doc_ids) for cleanup, get_by_status(status) for finding documents in a specific state, and count_by_status() for pipeline metrics. This enables comprehensive monitoring and recovery of document processing operations.
+
+### Storage Implementation Patterns
+
+**File-Based Storage (JSON)**: Simple implementations store data in JSON files with in-memory caching for performance. All modifications are held in memory until index_done_callback() triggers a write to disk. These implementations are suitable for development, small deployments, and single-process scenarios. They provide atomic writes using temporary files and rename operations, handle concurrent access through file locking, and support workspace isolation through directory structure.
+
+**PostgreSQL Storage**: Comprehensive implementations that leverage PostgreSQL's capabilities including JSON columns for flexible metadata, pgvector extension for vector similarity search, advisory locks for distributed coordination, connection pooling for performance, and transaction support for consistency. PostgreSQL implementations can handle all four storage types in a single database, simplifying deployment and backup. They support multi-tenant deployments through schema-based isolation and provide excellent performance for mixed workloads.
+
+**Specialized Vector Databases**: Dedicated vector storage implementations like FAISS, Milvus, and Qdrant provide optimized vector similarity search with features like approximate nearest neighbor (ANN) search, GPU acceleration for large-scale similarity search, advanced indexing strategies (IVF, HNSW), and high-performance batch operations. These are recommended for deployments with large document sets (>1M chunks) or high query throughput requirements.
+
+**Graph Databases (Neo4j/Memgraph)**: Specialized graph implementations optimize graph traversal and pattern matching with native graph storage and indexing, Cypher query language for complex graph queries, visualization capabilities for knowledge graph exploration, and optimized algorithms for shortest path, centrality, and community detection. These are ideal for use cases requiring complex graph analytics and when graph visualization is a primary feature.
+
+## Concurrency and State Management
+
+LightRAG implements sophisticated concurrency control to handle parallel document processing and query execution.
+
+```mermaid
+graph TB
+ subgraph "Concurrency Control"
+ SEM1["Semaphore
LLM Calls
(max_async)"]
+ SEM2["Semaphore
Embeddings
(embedding_func_max_async)"]
+ SEM3["Semaphore
Graph Merging
(graph_max_async)"]
+ LOCK1["Keyed Locks
Entity Processing"]
+ LOCK2["Keyed Locks
Relation Processing"]
+ LOCK3["Pipeline Status Lock"]
+ end
+
+ subgraph "State Management"
+ GLOBAL["Global Config
workspace, paths, settings"]
+ PIPELINE["Pipeline Status
current job, progress"]
+ NAMESPACE["Namespace Data
storage instances, locks"]
+ end
+
+ subgraph "Task Coordination"
+ QUEUE["Task Queue
Document Processing"]
+ PRIORITY["Priority Limiter
Async Function Calls"]
+ TRACK["Track ID System
Monitoring & Logging"]
+ end
+
+ SEM1 --> PRIORITY
+ SEM2 --> PRIORITY
+ SEM3 --> PRIORITY
+
+ LOCK1 --> NAMESPACE
+ LOCK2 --> NAMESPACE
+ LOCK3 --> PIPELINE
+
+ QUEUE --> TRACK
+ PRIORITY --> TRACK
+
+ GLOBAL --> NAMESPACE
+ PIPELINE --> TRACK
+
+ style SEM1 fill:#FFE6E6
+ style GLOBAL fill:#E6FFE6
+ style QUEUE fill:#E6F3FF
+```
+
+### Concurrency Patterns
+
+**Semaphore-Based Rate Limiting**: The system uses asyncio semaphores to limit concurrent operations and prevent overwhelming external services or exhausting resources. Different semaphores control different types of operations: LLM calls are limited by max_async (default 4), embedding function calls by embedding_func_max_async (default 8), and graph merging operations by graph_max_async (calculated as llm_model_max_async * 2). These semaphores ensure respectful API usage and prevent rate limit errors.
+
+**Keyed Locks for Data Consistency**: When processing entities and relationships concurrently, the system uses keyed locks to ensure that multiple processes don't modify the same entity or relationship simultaneously. Each entity or relationship gets a unique lock based on its identifier, preventing race conditions during graph merging while still allowing parallel processing of different entities. This pattern enables high concurrency without sacrificing data consistency.
+
+**Pipeline Status Tracking**: A shared pipeline status object tracks the current state of document processing including the active job name, number of documents being processed, current batch number, latest status message, and history of messages for debugging. This status is protected by an async lock and can be queried by clients to monitor progress. The status persists across the entire pipeline and provides visibility into long-running operations.
+
+**Workspace Isolation**: The workspace concept provides multi-tenant isolation by prefixing all storage namespaces with a workspace identifier. Different workspaces maintain completely separate data including separate storage instances, independent configuration, isolated locks and semaphores, and separate pipeline status. This enables running multiple LightRAG instances in the same infrastructure without interference.
+
+### TypeScript Migration Considerations for Concurrency
+
+**Semaphore Implementation**: Node.js doesn't have built-in semaphores, but the pattern can be implemented using the p-limit library, which provides similar functionality with a cleaner API. Example: `const limiter = pLimit(4); await limiter(() => callLLM())`.
+
+**Keyed Locks**: For single-process deployments, a Map can implement keyed locks. For multi-process deployments, consider using Redis with the Redlock algorithm or a dedicated lock service. The key insight is ensuring that operations on the same entity/relationship are serialized while different entities can be processed in parallel.
+
+**Shared State Management**: Python's global dictionaries need to be replaced with class-based state management in TypeScript. For multi-process deployments, shared state should be externalized to Redis or a similar store. For single-process deployments, singleton classes can manage state with proper TypeScript visibility controls.
+
+**Pipeline Status Updates**: Real-time status updates can be implemented using EventEmitter in Node.js for in-process communication, or Redis Pub/Sub for multi-process scenarios. WebSocket connections can provide real-time updates to clients without polling.
+
+## Summary
+
+This architecture documentation provides a comprehensive view of LightRAG's design, from high-level layer organization to detailed component interactions and concurrency patterns. The system's modular design, with clear interfaces and abstractions, makes it well-suited for migration to TypeScript. The key architectural principles—layered separation of concerns, plugin-based storage abstraction, async-first concurrency, and comprehensive state tracking—translate well to TypeScript and Node.js idioms.
+
+The subsequent documentation sections build on this architectural foundation, providing detailed specifications for data models, storage implementations, LLM integrations, and API contracts. Together, these documents form a complete blueprint for implementing a production-ready TypeScript version of LightRAG.
diff --git a/reverse_documentation/03-data-models-and-schemas.md b/reverse_documentation/03-data-models-and-schemas.md
new file mode 100644
index 00000000..8bcbfff2
--- /dev/null
+++ b/reverse_documentation/03-data-models-and-schemas.md
@@ -0,0 +1,890 @@
+# Data Models and Schemas: LightRAG Type System
+
+## Table of Contents
+1. [Core Data Models](#core-data-models)
+2. [Storage Schema Definitions](#storage-schema-definitions)
+3. [Query and Response Models](#query-and-response-models)
+4. [Configuration Models](#configuration-models)
+5. [TypeScript Type Mapping](#typescript-type-mapping)
+
+## Core Data Models
+
+### Text Chunk Schema
+
+Text chunks are the fundamental unit of document processing in LightRAG. Documents are split into overlapping chunks that preserve context while fitting within token limits.
+
+**Python Definition** (`lightrag/base.py:75-79`):
+```python
+class TextChunkSchema(TypedDict):
+ tokens: int
+ content: str
+ full_doc_id: str
+ chunk_order_index: int
+```
+
+**TypeScript Definition**:
+```typescript
+interface TextChunkSchema {
+ tokens: number;
+ content: string;
+ full_doc_id: string;
+ chunk_order_index: number;
+}
+```
+
+**Field Descriptions**:
+- `tokens`: Number of tokens in the chunk according to the configured tokenizer (e.g., tiktoken for GPT models)
+- `content`: The actual text content of the chunk, UTF-8 encoded
+- `full_doc_id`: MD5 hash of the complete document, used as a foreign key to link chunks to their source document
+- `chunk_order_index`: Zero-based index indicating the chunk's position in the original document sequence
+
+**Storage Pattern**: Chunks are stored in KV storage with keys following the pattern `{full_doc_id}_{chunk_order_index}`. The chunk content is also embedded and stored in Vector storage for similarity search.
+
+**Validation Rules**:
+- `tokens` must be > 0 and typically < 2048
+- `content` must not be empty
+- `full_doc_id` must be a valid MD5 hash (32 hexadecimal characters)
+- `chunk_order_index` must be >= 0
+
+### Entity Schema
+
+Entities represent key concepts, people, organizations, locations, and other named entities extracted from documents.
+
+**Python Definition** (Implicit in `lightrag/operate.py`):
+```python
+entity_data = {
+ "entity_name": str, # Normalized entity name (title case)
+ "entity_type": str, # One of DEFAULT_ENTITY_TYPES
+ "description": str, # Consolidated description
+ "source_id": str, # Chunk IDs joined by GRAPH_FIELD_SEP
+ "file_path": str, # File paths joined by GRAPH_FIELD_SEP
+ "created_at": str, # ISO 8601 timestamp
+ "updated_at": str, # ISO 8601 timestamp
+}
+```
+
+**TypeScript Definition**:
+```typescript
+interface EntityData {
+ entity_name: string;
+ entity_type: EntityType;
+ description: string;
+ source_id: string; // Pipe-separated chunk IDs
+ file_path: string; // Pipe-separated file paths
+ created_at: string; // ISO 8601
+ updated_at: string; // ISO 8601
+}
+
+type EntityType =
+ | "Person"
+ | "Creature"
+ | "Organization"
+ | "Location"
+ | "Event"
+ | "Concept"
+ | "Method"
+ | "Content"
+ | "Data"
+ | "Artifact"
+ | "NaturalObject"
+ | "Other";
+```
+
+**Field Descriptions**:
+- `entity_name`: Normalized name using title case for consistency (e.g., "John Smith", "OpenAI")
+- `entity_type`: Classification of the entity using predefined types from `DEFAULT_ENTITY_TYPES`
+- `description`: Rich text description of the entity's attributes, activities, and context. May be merged from multiple sources using LLM summarization
+- `source_id`: Pipe-separated (``) list of chunk IDs where this entity was mentioned, enabling citation tracking
+- `file_path`: Pipe-separated list of source file paths for traceability
+- `created_at`: ISO 8601 timestamp when the entity was first created
+- `updated_at`: ISO 8601 timestamp when the entity was last modified
+
+**Storage Locations**:
+1. Graph Storage: Entity as a node with `entity_name` as the node ID
+2. Vector Storage: Entity description embedding with metadata
+3. Full Entities KV Storage: Complete entity data for retrieval
+
+**Normalization Rules**:
+- Entity names are case-insensitive for matching but stored in title case
+- Multiple mentions of the same entity (fuzzy matched) are merged
+- Descriptions are consolidated using LLM summarization when they exceed token limits
+- `source_id` and `file_path` are deduplicated when merging
+
+### Relationship Schema
+
+Relationships represent connections between entities, forming the edges of the knowledge graph.
+
+**Python Definition** (Implicit in `lightrag/operate.py`):
+```python
+relationship_data = {
+ "src_id": str, # Source entity name
+ "tgt_id": str, # Target entity name
+ "description": str, # Relationship description
+ "keywords": str, # Comma-separated keywords
+ "weight": float, # Relationship strength (0-1)
+ "source_id": str, # Chunk IDs joined by GRAPH_FIELD_SEP
+ "file_path": str, # File paths joined by GRAPH_FIELD_SEP
+ "created_at": str, # ISO 8601 timestamp
+ "updated_at": str, # ISO 8601 timestamp
+}
+```
+
+**TypeScript Definition**:
+```typescript
+interface RelationshipData {
+ src_id: string;
+ tgt_id: string;
+ description: string;
+ keywords: string; // Comma-separated
+ weight: number; // 0.0 to 1.0
+ source_id: string; // Pipe-separated chunk IDs
+ file_path: string; // Pipe-separated file paths
+ created_at: string; // ISO 8601
+ updated_at: string; // ISO 8601
+}
+```
+
+**Field Descriptions**:
+- `src_id`: Name of the source entity (must match an existing entity)
+- `tgt_id`: Name of the target entity (must match an existing entity)
+- `description`: Explanation of how and why the entities are related
+- `keywords`: High-level keywords summarizing the relationship nature (e.g., "collaboration, project, research")
+- `weight`: Numeric weight indicating relationship strength, aggregated when merging duplicates
+- `source_id`: Chunk IDs where this relationship was mentioned
+- `file_path`: Source file paths for citation
+- `created_at`: Creation timestamp
+- `updated_at`: Last modification timestamp
+
+**Storage Locations**:
+1. Graph Storage: Edge between source and target nodes
+2. Vector Storage: Relationship description embedding with metadata
+3. Full Relations KV Storage: Complete relationship data
+
+**Validation Rules**:
+- Relationships are treated as undirected (bidirectional)
+- `src_id` and `tgt_id` must reference existing entities
+- `weight` must be between 0.0 and 1.0
+- Duplicate relationships (same src_id, tgt_id pair) are merged with weights summed
+
+### Document Processing Status
+
+Tracks the processing state of documents through the ingestion pipeline.
+
+**Python Definition** (`lightrag/base.py:679-724`):
+```python
+@dataclass
+class DocProcessingStatus:
+ content_summary: str
+ content_length: int
+ file_path: str
+ status: DocStatus
+ created_at: str
+ updated_at: str
+ track_id: str | None = None
+ chunks_count: int | None = None
+ chunks_list: list[str] | None = field(default_factory=list)
+ entities_count: int | None = None
+ relations_count: int | None = None
+ batch_number: int | None = None
+ error_message: str | None = None
+```
+
+**TypeScript Definition**:
+```typescript
+enum DocStatus {
+ PENDING = "PENDING",
+ CHUNKING = "CHUNKING",
+ EXTRACTING = "EXTRACTING",
+ MERGING = "MERGING",
+ INDEXING = "INDEXING",
+ COMPLETED = "COMPLETED",
+ FAILED = "FAILED"
+}
+
+interface DocProcessingStatus {
+ content_summary: string;
+ content_length: number;
+ file_path: string;
+ status: DocStatus;
+ created_at: string; // ISO 8601
+ updated_at: string; // ISO 8601
+ track_id?: string;
+ chunks_count?: number;
+ chunks_list?: string[];
+ entities_count?: number;
+ relations_count?: number;
+ batch_number?: number;
+ error_message?: string;
+}
+```
+
+**Field Descriptions**:
+- `content_summary`: First 100 characters of document for preview
+- `content_length`: Total character length of the document
+- `file_path`: Original file path or identifier
+- `status`: Current processing stage (see DocStatus enum)
+- `created_at`: Document submission timestamp
+- `updated_at`: Last status update timestamp
+- `track_id`: Optional tracking ID for batch monitoring (shared across multiple documents)
+- `chunks_count`: Number of chunks created during splitting
+- `chunks_list`: Array of chunk IDs for reference
+- `entities_count`: Number of entities extracted
+- `relations_count`: Number of relationships extracted
+- `batch_number`: Batch identifier for processing order
+- `error_message`: Error details if status is FAILED
+
+**State Transitions**:
+```
+PENDING → CHUNKING → EXTRACTING → MERGING → INDEXING → COMPLETED
+ ↓
+ FAILED
+```
+
+Any stage can transition to FAILED on error, with `error_message` populated with diagnostic information.
+
+## Storage Schema Definitions
+
+### KV Storage Schema
+
+KV storage handles three types of data: LLM response cache, text chunks, and full documents.
+
+#### LLM Cache Entry
+**Key Format**: `cache:{hash(prompt+model+params)}`
+
+**Value Schema**:
+```typescript
+interface LLMCacheEntry {
+ return_message: string;
+ embedding_dim: number;
+ model: string;
+ timestamp: string;
+}
+```
+
+#### Text Chunk Entry
+**Key Format**: `{full_doc_id}_{chunk_order_index}`
+
+**Value Schema**:
+```typescript
+interface ChunkEntry extends TextChunkSchema {
+ // Additional metadata can be stored
+ file_path?: string;
+ created_at?: string;
+}
+```
+
+#### Full Document Entry
+**Key Format**: Document ID (MD5 hash of content)
+
+**Value Schema**:
+```typescript
+interface FullDocEntry {
+ content: string;
+ file_path?: string;
+ created_at?: string;
+ metadata?: Record;
+}
+```
+
+### Vector Storage Schema
+
+Vector storage maintains embeddings for entities, relationships, and chunks.
+
+**Entry Schema**:
+```typescript
+interface VectorEntry {
+ id: string; // Unique identifier
+ vector: number[]; // Embedding vector (e.g., 1536 dimensions for OpenAI)
+ metadata: {
+ content: string; // Original text that was embedded
+ type: "entity" | "relation" | "chunk";
+ entity_name?: string; // For entities
+ source_id?: string; // Chunk IDs where this appears
+ file_path?: string; // Source file paths
+ [key: string]: any; // Additional metadata
+ };
+}
+```
+
+**Index Requirements**:
+- Cosine similarity search support
+- Efficient top-k retrieval (ANN algorithms recommended for large datasets)
+- Metadata filtering capabilities
+- Batch upsert and deletion support
+
+**Storage Size Estimates**:
+- Entity vectors: ~6KB each (1536 floats × 4 bytes)
+- Relationship vectors: ~6KB each
+- Chunk vectors: ~6KB each
+- 10,000 documents ≈ 100,000 chunks ≈ 600MB vector storage
+
+### Graph Storage Schema
+
+Graph storage maintains the entity-relationship graph structure.
+
+#### Node Schema
+**Node ID**: Entity name (normalized, title case)
+
+**Node Properties**:
+```typescript
+interface GraphNode {
+ entity_name: string; // Node ID
+ entity_type: string; // Entity classification
+ description: string; // Entity description
+ source_id: string; // Pipe-separated chunk IDs
+ file_path: string; // Pipe-separated file paths
+ created_at: string; // ISO 8601
+ updated_at: string; // ISO 8601
+}
+```
+
+#### Edge Schema
+**Edge ID**: Combination of source and target node IDs (undirected)
+
+**Edge Properties**:
+```typescript
+interface GraphEdge {
+ src_id: string; // Source entity name
+ tgt_id: string; // Target entity name
+ description: string; // Relationship description
+ keywords: string; // Comma-separated
+ weight: number; // 0.0 to 1.0
+ source_id: string; // Pipe-separated chunk IDs
+ file_path: string; // Pipe-separated file paths
+ created_at: string; // ISO 8601
+ updated_at: string; // ISO 8601
+}
+```
+
+**Graph Constraints**:
+- Undirected edges: (A, B) and (B, A) represent the same relationship
+- No self-loops: src_id ≠ tgt_id
+- Unique edge constraint: Only one edge per (src_id, tgt_id) pair
+- Node must exist before creating edges
+
+#### Query Capabilities Required
+- Node existence check: `has_node(node_id)`
+- Edge existence check: `has_edge(src_id, tgt_id)`
+- Degree calculation: `node_degree(node_id)`, `edge_degree(src_id, tgt_id)`
+- Node retrieval: `get_node(node_id)`, `get_nodes_batch(node_ids[])`
+- Edge retrieval: `get_edge(src_id, tgt_id)`, `get_edges_batch(pairs[])`
+- Neighborhood query: `get_node_edges(node_id)`
+- Graph traversal: `get_knowledge_graph(start_node, max_depth, max_nodes)`
+- Label listing: `get_all_labels()`
+
+### Document Status Storage Schema
+
+Document status storage is a specialized KV storage for tracking pipeline state.
+
+**Key Format**: Document ID (MD5 hash)
+
+**Value Schema**: `DocProcessingStatus` (see above)
+
+**Required Capabilities**:
+- Get by ID: `get_by_id(doc_id)`
+- Get by IDs: `get_by_ids(doc_ids[])`
+- Get by status: `get_by_status(status)` → all documents in that state
+- Get by track ID: `get_by_track_id(track_id)` → all documents in that batch
+- Status counts: `get_status_counts()` → count of documents in each state
+- Upsert: `upsert(doc_id, status_data)`
+- Delete: `delete(doc_ids[])`
+
+## Query and Response Models
+
+### Query Parameter Model
+
+Comprehensive configuration for query execution.
+
+**Python Definition** (`lightrag/base.py:86-171`):
+```python
+@dataclass
+class QueryParam:
+ mode: Literal["local", "global", "hybrid", "naive", "mix", "bypass"] = "mix"
+ only_need_context: bool = False
+ only_need_prompt: bool = False
+ response_type: str = "Multiple Paragraphs"
+ stream: bool = False
+ top_k: int = 40
+ chunk_top_k: int = 20
+ max_entity_tokens: int = 6000
+ max_relation_tokens: int = 8000
+ max_total_tokens: int = 30000
+ hl_keywords: list[str] = field(default_factory=list)
+ ll_keywords: list[str] = field(default_factory=list)
+ conversation_history: list[dict[str, str]] = field(default_factory=list)
+ history_turns: int = 0
+ model_func: Callable[..., object] | None = None
+ user_prompt: str | None = None
+ enable_rerank: bool = True
+ include_references: bool = False
+```
+
+**TypeScript Definition**:
+```typescript
+type QueryMode = "local" | "global" | "hybrid" | "naive" | "mix" | "bypass";
+
+interface ConversationMessage {
+ role: "user" | "assistant" | "system";
+ content: string;
+}
+
+interface QueryParam {
+ mode?: QueryMode;
+ only_need_context?: boolean;
+ only_need_prompt?: boolean;
+ response_type?: string;
+ stream?: boolean;
+ top_k?: number;
+ chunk_top_k?: number;
+ max_entity_tokens?: number;
+ max_relation_tokens?: number;
+ max_total_tokens?: number;
+ hl_keywords?: string[];
+ ll_keywords?: string[];
+ conversation_history?: ConversationMessage[];
+ history_turns?: number;
+ model_func?: (...args: any[]) => Promise;
+ user_prompt?: string;
+ enable_rerank?: boolean;
+ include_references?: boolean;
+}
+```
+
+**Field Descriptions**:
+
+**Retrieval Configuration**:
+- `mode`: Query strategy (see Query Processing documentation)
+- `top_k`: Number of entities (local) or relations (global) to retrieve
+- `chunk_top_k`: Number of text chunks to keep after reranking
+
+**Token Budget**:
+- `max_entity_tokens`: Token budget for entity descriptions in context
+- `max_relation_tokens`: Token budget for relationship descriptions
+- `max_total_tokens`: Total context budget including system prompt
+
+**Keyword Guidance**:
+- `hl_keywords`: High-level keywords for global retrieval (themes, concepts)
+- `ll_keywords`: Low-level keywords for local retrieval (specific terms)
+
+**Conversation Context**:
+- `conversation_history`: Previous messages for multi-turn dialogue
+- `history_turns`: Number of conversation turns to include (deprecated, all history sent)
+
+**Response Configuration**:
+- `response_type`: Desired format ("Multiple Paragraphs", "Single Paragraph", "Bullet Points", etc.)
+- `stream`: Enable streaming responses via SSE
+- `user_prompt`: Additional instructions to inject into the LLM prompt
+- `enable_rerank`: Use reranking model for chunk relevance scoring
+- `include_references`: Include citation information in response
+
+**Debug Options**:
+- `only_need_context`: Return retrieved context without LLM generation
+- `only_need_prompt`: Return the constructed prompt without generation
+
+### Query Result Model
+
+Unified response structure for all query types.
+
+**Python Definition** (`lightrag/base.py:778-820`):
+```python
+@dataclass
+class QueryResult:
+ content: Optional[str] = None
+ response_iterator: Optional[AsyncIterator[str]] = None
+ raw_data: Optional[Dict[str, Any]] = None
+ is_streaming: bool = False
+```
+
+**TypeScript Definition**:
+```typescript
+interface QueryResult {
+ content?: string;
+ response_iterator?: AsyncIterableIterator;
+ raw_data?: QueryRawData;
+ is_streaming: boolean;
+}
+
+interface QueryRawData {
+ response: string;
+ references?: ReferenceEntry[];
+ entities?: EntityData[];
+ relationships?: RelationshipData[];
+ chunks?: ChunkData[];
+ processing_info?: ProcessingInfo;
+}
+
+interface ReferenceEntry {
+ reference_id: string;
+ file_path: string;
+}
+
+interface ChunkData {
+ content: string;
+ tokens: number;
+ source_id: string;
+ file_path: string;
+}
+
+interface ProcessingInfo {
+ mode: QueryMode;
+ keyword_extraction: {
+ high_level_keywords: string[];
+ low_level_keywords: string[];
+ };
+ retrieval_stats: {
+ entities_retrieved: number;
+ relationships_retrieved: number;
+ chunks_retrieved: number;
+ chunks_after_rerank?: number;
+ };
+ context_stats: {
+ entity_tokens: number;
+ relation_tokens: number;
+ chunk_tokens: number;
+ total_tokens: number;
+ };
+ token_budget: {
+ max_entity_tokens: number;
+ max_relation_tokens: number;
+ max_total_tokens: number;
+ final_entity_tokens: number;
+ final_relation_tokens: number;
+ final_chunk_tokens: number;
+ };
+}
+```
+
+**Usage Patterns**:
+
+For non-streaming responses:
+```typescript
+const result = await rag.query("What is AI?", { stream: false });
+console.log(result.content); // Complete response text
+console.log(result.raw_data?.references); // Citation information
+```
+
+For streaming responses:
+```typescript
+const result = await rag.query("What is AI?", { stream: true });
+for await (const chunk of result.response_iterator!) {
+ process.stdout.write(chunk); // Stream to output
+}
+```
+
+For context-only retrieval:
+```typescript
+const result = await rag.query("What is AI?", { only_need_context: true });
+console.log(result.raw_data?.entities); // Retrieved entities
+console.log(result.raw_data?.chunks); // Retrieved chunks
+```
+
+## Configuration Models
+
+### LightRAG Configuration
+
+Complete configuration for a LightRAG instance.
+
+**Python Definition** (`lightrag/lightrag.py:116-384`):
+```python
+@dataclass
+class LightRAG:
+ # Storage
+ working_dir: str = "./rag_storage"
+ kv_storage: str = "JsonKVStorage"
+ vector_storage: str = "NanoVectorDBStorage"
+ graph_storage: str = "NetworkXStorage"
+ doc_status_storage: str = "JsonDocStatusStorage"
+ workspace: str = ""
+
+ # LLM and Embedding
+ llm_model_func: Callable | None = None
+ llm_model_name: str = "gpt-4o-mini"
+ llm_model_max_async: int = 4
+ llm_model_timeout: int = 180
+ embedding_func: EmbeddingFunc | None = None
+ embedding_batch_num: int = 10
+ embedding_func_max_async: int = 8
+ default_embedding_timeout: int = 30
+
+ # Chunking
+ chunk_token_size: int = 1200
+ chunk_overlap_token_size: int = 100
+ tokenizer: Optional[Tokenizer] = None
+ tiktoken_model_name: str = "gpt-4o-mini"
+
+ # Extraction
+ entity_extract_max_gleaning: int = 1
+ entity_types: list[str] = field(default_factory=lambda: DEFAULT_ENTITY_TYPES)
+ force_llm_summary_on_merge: int = 8
+ summary_max_tokens: int = 1200
+ summary_language: str = "English"
+
+ # Query
+ top_k: int = 40
+ chunk_top_k: int = 20
+ max_entity_tokens: int = 6000
+ max_relation_tokens: int = 8000
+ max_total_tokens: int = 30000
+ cosine_threshold: int = 0.2
+ related_chunk_number: int = 5
+ kg_chunk_pick_method: str = "VECTOR"
+
+ # Reranking
+ enable_rerank: bool = True
+ rerank_model_func: Callable | None = None
+ min_rerank_score: float = 0.0
+
+ # Concurrency
+ max_async: int = 4
+ max_parallel_insert: int = 2
+
+ # Optional
+ addon_params: dict[str, Any] = field(default_factory=dict)
+```
+
+**TypeScript Definition**:
+```typescript
+interface LightRAGConfig {
+ // Storage
+ working_dir?: string;
+ kv_storage?: string;
+ vector_storage?: string;
+ graph_storage?: string;
+ doc_status_storage?: string;
+ workspace?: string;
+
+ // LLM and Embedding
+ llm_model_func?: LLMFunction;
+ llm_model_name?: string;
+ llm_model_max_async?: number;
+ llm_model_timeout?: number;
+ embedding_func?: EmbeddingFunction;
+ embedding_batch_num?: number;
+ embedding_func_max_async?: number;
+ default_embedding_timeout?: number;
+
+ // Chunking
+ chunk_token_size?: number;
+ chunk_overlap_token_size?: number;
+ tokenizer?: Tokenizer;
+ tiktoken_model_name?: string;
+
+ // Extraction
+ entity_extract_max_gleaning?: number;
+ entity_types?: string[];
+ force_llm_summary_on_merge?: number;
+ summary_max_tokens?: number;
+ summary_language?: string;
+
+ // Query
+ top_k?: number;
+ chunk_top_k?: number;
+ max_entity_tokens?: number;
+ max_relation_tokens?: number;
+ max_total_tokens?: number;
+ cosine_threshold?: number;
+ related_chunk_number?: number;
+ kg_chunk_pick_method?: "VECTOR" | "WEIGHT";
+
+ // Reranking
+ enable_rerank?: boolean;
+ rerank_model_func?: RerankFunction;
+ min_rerank_score?: number;
+
+ // Concurrency
+ max_async?: number;
+ max_parallel_insert?: number;
+
+ // Optional
+ addon_params?: Record;
+}
+
+type LLMFunction = (
+ prompt: string,
+ system_prompt?: string,
+ history_messages?: ConversationMessage[],
+ stream?: boolean,
+ **kwargs: any
+) => Promise | AsyncIterableIterator;
+
+type EmbeddingFunction = (texts: string[]) => Promise;
+
+type RerankFunction = (
+ query: string,
+ documents: string[]
+) => Promise>;
+
+interface Tokenizer {
+ encode(text: string): number[];
+ decode(tokens: number[]): string;
+}
+```
+
+## TypeScript Type Mapping
+
+### Python to TypeScript Type Conversion
+
+| Python Type | TypeScript Type | Notes |
+|------------|----------------|-------|
+| `str` | `string` | Direct mapping |
+| `int` | `number` | JavaScript/TypeScript uses `number` for all numerics |
+| `float` | `number` | Same as `int` |
+| `bool` | `boolean` | Direct mapping |
+| `list[T]` | `T[]` or `Array` | Both notations are valid in TypeScript |
+| `dict[K, V]` | `Record` or `Map` | `Record` for simple objects, `Map` for dynamic keys |
+| `set[T]` | `Set` | Direct mapping |
+| `tuple[T1, T2]` | `[T1, T2]` | TypeScript tuple syntax |
+| `Literal["a", "b"]` | `"a" \| "b"` | Union of literal types |
+| `Optional[T]` | `T \| undefined` or `T?` | Optional property syntax |
+| `Union[T1, T2]` | `T1 \| T2` | Union type |
+| `Any` | `any` | Avoid if possible, use `unknown` for type-safe any |
+| `TypedDict` | `interface` | TypeScript interface |
+| `@dataclass` | `class` or `interface` | Use `class` for behavior, `interface` for pure data |
+| `Callable[..., T]` | `(...args: any[]) => T` | Function type |
+| `AsyncIterator[T]` | `AsyncIterableIterator` | Async iteration support |
+
+### Python-Specific Features Requiring Special Handling
+
+**Dataclasses with field defaults**:
+```python
+# Python
+@dataclass
+class Example:
+ name: str
+ items: list[str] = field(default_factory=list)
+```
+
+```typescript
+// TypeScript - Option 1: Class with constructor
+class Example {
+ name: string;
+ items: string[];
+
+ constructor(name: string, items: string[] = []) {
+ this.name = name;
+ this.items = items;
+ }
+}
+
+// TypeScript - Option 2: Interface with builder
+interface Example {
+ name: string;
+ items: string[];
+}
+
+function createExample(name: string, items: string[] = []): Example {
+ return { name, items };
+}
+```
+
+**Multiple inheritance from ABC**:
+```python
+# Python
+@dataclass
+class BaseGraphStorage(StorageNameSpace, ABC):
+ pass
+```
+
+```typescript
+// TypeScript - Use composition over inheritance
+abstract class StorageNameSpace {
+ abstract initialize(): Promise;
+}
+
+abstract class BaseGraphStorage extends StorageNameSpace {
+ // Additional abstract methods
+}
+
+// Or use interfaces for pure contracts
+interface IStorageNameSpace {
+ initialize(): Promise;
+}
+
+interface IGraphStorage extends IStorageNameSpace {
+ // Graph-specific methods
+}
+```
+
+**Overloaded functions**:
+```python
+# Python
+@overload
+def get(id: str) -> dict | None: ...
+@overload
+def get(ids: list[str]) -> list[dict]: ...
+```
+
+```typescript
+// TypeScript - Native overload support
+function get(id: string): Promise | null>;
+function get(ids: string[]): Promise[]>;
+function get(idOrIds: string | string[]): Promise {
+ if (typeof idOrIds === 'string') {
+ // Single ID logic
+ } else {
+ // Multiple IDs logic
+ }
+}
+```
+
+### Validation and Serialization
+
+For runtime validation and serialization in TypeScript, consider using:
+
+**Zod for schema validation**:
+```typescript
+import { z } from 'zod';
+
+const TextChunkSchema = z.object({
+ tokens: z.number().positive(),
+ content: z.string().min(1),
+ full_doc_id: z.string().regex(/^[a-f0-9]{32}$/),
+ chunk_order_index: z.number().nonnegative(),
+});
+
+type TextChunk = z.infer;
+
+// Validate at runtime
+const chunk = TextChunkSchema.parse(data);
+```
+
+**class-transformer for class serialization**:
+```typescript
+import { plainToClass, classToPlain } from 'class-transformer';
+import { IsString, IsNumber } from 'class-validator';
+
+class TextChunk {
+ @IsNumber()
+ tokens: number;
+
+ @IsString()
+ content: string;
+
+ @IsString()
+ full_doc_id: string;
+
+ @IsNumber()
+ chunk_order_index: number;
+}
+
+// Convert plain object to class instance
+const chunk = plainToClass(TextChunk, jsonData);
+
+// Convert class instance to plain object
+const json = classToPlain(chunk);
+```
+
+## Summary
+
+This comprehensive data models documentation provides:
+
+1. **Complete type definitions** for all core data structures in both Python and TypeScript
+2. **Storage schemas** detailing how data is persisted in each storage layer
+3. **Query and response models** with full field descriptions and usage patterns
+4. **Configuration models** for system setup and customization
+5. **Type mapping guide** for Python-to-TypeScript conversion
+6. **Validation strategies** using TypeScript libraries
+
+These type definitions form the contract layer between all components of the system, ensuring type safety and consistent data structures throughout the implementation. The TypeScript definitions leverage the language's strong type system to provide compile-time safety while maintaining compatibility with the original Python design.
+
+The next documentation sections will use these type definitions extensively when describing storage implementations, API contracts, and LLM integrations.
diff --git a/reverse_documentation/04-dependency-migration-guide.md b/reverse_documentation/04-dependency-migration-guide.md
new file mode 100644
index 00000000..74fa4791
--- /dev/null
+++ b/reverse_documentation/04-dependency-migration-guide.md
@@ -0,0 +1,894 @@
+# Dependency Migration Guide: Python to TypeScript/Node.js
+
+## Table of Contents
+1. [Core Dependencies Mapping](#core-dependencies-mapping)
+2. [Storage Driver Dependencies](#storage-driver-dependencies)
+3. [LLM and Embedding Dependencies](#llm-and-embedding-dependencies)
+4. [API and Web Framework Dependencies](#api-and-web-framework-dependencies)
+5. [Utility and Helper Dependencies](#utility-and-helper-dependencies)
+6. [Migration Complexity Assessment](#migration-complexity-assessment)
+
+## Core Dependencies Mapping
+
+### Python Standard Library → Node.js/TypeScript
+
+| Python Package | Purpose | TypeScript/Node.js Equivalent | Migration Notes |
+|----------------|---------|-------------------------------|-----------------|
+| `asyncio` | Async/await runtime | Native Node.js (async/await) | Direct support, different patterns (see below) |
+| `dataclasses` | Data class definitions | TypeScript classes/interfaces | Use `class` with constructor or `interface` |
+| `typing` | Type hints | Native TypeScript types | Superior type system in TS |
+| `functools` | Function tools (partial, lru_cache) | `lodash/partial`, custom decorators | `lodash.partial` or arrow functions |
+| `collections` | Counter, defaultdict, deque | Native Map/Set, or `collections-js` | Map/Set cover most cases |
+| `json` | JSON parsing | Native `JSON` | Direct support |
+| `hashlib` | MD5, SHA hashing | `crypto` (built-in) | `crypto.createHash('md5')` |
+| `os` | OS operations | `fs`, `path` (built-in) | Similar APIs |
+| `time` | Time operations | Native `Date`, `performance.now()` | Different epoch (JS uses ms) |
+| `datetime` | Date/time handling | Native `Date` or `date-fns` | `date-fns` recommended for manipulation |
+| `configparser` | INI file parsing | `ini` npm package | Direct equivalent |
+| `warnings` | Warning system | `console.warn()` or custom | Simpler in Node.js |
+| `traceback` | Stack traces | Native Error stack | `error.stack` property |
+
+### Core Python Packages → npm Packages
+
+| Python Package | Purpose | npm Package | Version | Migration Complexity |
+|----------------|---------|-------------|---------|---------------------|
+| `aiohttp` | Async HTTP client | `axios` or `undici` | ^1.6.0 or ^6.0.0 | Low - similar APIs |
+| `json_repair` | Fix malformed JSON | `json-repair` | ^0.2.0 | Low - direct port exists |
+| `numpy` | Numerical arrays | `@tensorflow/tfjs` or `ndarray` | ^4.0.0 or ^1.0.0 | Medium - different paradigm |
+| `pandas` | Data manipulation | `danfojs` | ^1.1.0 | Medium - similar but less mature |
+| `pydantic` | Data validation | `zod` or `class-validator` | ^3.22.0 or ^0.14.0 | Low - excellent TS support |
+| `python-dotenv` | Environment variables | `dotenv` | ^16.0.0 | Low - identical functionality |
+| `tenacity` | Retry logic | `async-retry` or `p-retry` | ^3.0.0 or ^6.0.0 | Low - similar patterns |
+| `tiktoken` | OpenAI tokenizer | `@dqbd/tiktoken` or `js-tiktoken` | ^1.0.0 or ^1.0.0 | Medium - WASM-based port |
+| `pypinyin` | Chinese pinyin | `pinyin` npm package | ^3.0.0 | Low - direct equivalent |
+
+### Async/Await Pattern Differences
+
+**Python asyncio patterns:**
+```python
+import asyncio
+
+# Semaphore
+semaphore = asyncio.Semaphore(4)
+async with semaphore:
+ await do_work()
+
+# Gather with error handling
+results = await asyncio.gather(*tasks, return_exceptions=True)
+
+# Task cancellation
+task.cancel()
+await task # Raises CancelledError
+
+# Wait with timeout
+try:
+ result = await asyncio.wait_for(coro, timeout=30)
+except asyncio.TimeoutError:
+ pass
+```
+
+**TypeScript/Node.js equivalents:**
+```typescript
+import pLimit from 'p-limit';
+import pTimeout from 'p-timeout';
+
+// Semaphore using p-limit
+const limit = pLimit(4);
+await limit(() => doWork());
+
+// Promise.allSettled for error handling
+const results = await Promise.allSettled(promises);
+results.forEach(result => {
+ if (result.status === 'fulfilled') {
+ console.log(result.value);
+ } else {
+ console.error(result.reason);
+ }
+});
+
+// AbortController for cancellation
+const controller = new AbortController();
+fetch(url, { signal: controller.signal });
+controller.abort();
+
+// Timeout using p-timeout
+try {
+ const result = await pTimeout(promise, { milliseconds: 30000 });
+} catch (error) {
+ if (error.name === 'TimeoutError') {
+ // Handle timeout
+ }
+}
+```
+
+**Recommended npm packages for async patterns:**
+- `p-limit` (^5.0.0): Rate limiting / semaphore
+- `p-queue` (^8.0.0): Priority queue for async tasks
+- `p-retry` (^6.0.0): Retry with exponential backoff
+- `p-timeout` (^6.0.0): Timeout for promises
+- `bottleneck` (^2.19.0): Advanced rate limiting
+
+## Storage Driver Dependencies
+
+### PostgreSQL
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `asyncpg` | `pg` | ^8.11.0 | Most popular, excellent TypeScript support |
+| | `drizzle-orm` | ^0.29.0 | Optional: Type-safe query builder |
+| | `@neondatabase/serverless` | ^0.9.0 | For serverless environments |
+
+**Migration complexity**: Low
+**Recommendation**: Use `pg` with connection pooling. Consider `drizzle-orm` for type-safe queries.
+
+```typescript
+// PostgreSQL connection with pg
+import { Pool } from 'pg';
+
+const pool = new Pool({
+ host: process.env.PG_HOST,
+ port: parseInt(process.env.PG_PORT || '5432'),
+ database: process.env.PG_DATABASE,
+ user: process.env.PG_USER,
+ password: process.env.PG_PASSWORD,
+ max: 20, // Connection pool size
+ idleTimeoutMillis: 30000,
+ connectionTimeoutMillis: 2000,
+});
+
+// With Drizzle ORM for type safety
+import { drizzle } from 'drizzle-orm/node-postgres';
+const db = drizzle(pool);
+```
+
+### PostgreSQL pgvector Extension
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `pgvector` | `pgvector` | ^0.1.0 | Official Node.js client |
+
+**Migration complexity**: Low
+**Implementation**: Use `pgvector` npm package with `pg`:
+
+```typescript
+import pgvector from 'pgvector/pg';
+
+// Register pgvector type
+await pgvector.registerType(pool);
+
+// Insert vector
+await pool.query(
+ 'INSERT INTO items (embedding) VALUES ($1)',
+ [pgvector.toSql([1.0, 2.0, 3.0])]
+);
+
+// Query by similarity
+const result = await pool.query(
+ 'SELECT * FROM items ORDER BY embedding <-> $1 LIMIT 10',
+ [pgvector.toSql(queryVector)]
+);
+```
+
+### MongoDB
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `motor` | `mongodb` | ^6.3.0 | Official MongoDB driver |
+| | `mongoose` | ^8.0.0 | Optional: ODM for schemas |
+
+**Migration complexity**: Low
+**Recommendation**: Use official `mongodb` driver. Add `mongoose` if you need schema validation.
+
+```typescript
+import { MongoClient } from 'mongodb';
+
+const client = new MongoClient(process.env.MONGODB_URI!, {
+ maxPoolSize: 50,
+ minPoolSize: 10,
+ serverSelectionTimeoutMS: 5000,
+});
+
+await client.connect();
+const db = client.db('lightrag');
+const collection = db.collection('entities');
+```
+
+### Redis
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `redis-py` | `ioredis` | ^5.3.0 | Better TypeScript support than `redis` |
+| | `redis` | ^4.6.0 | Official client, good but less TS-friendly |
+
+**Migration complexity**: Low
+**Recommendation**: Use `ioredis` for better TypeScript experience and cluster support.
+
+```typescript
+import Redis from 'ioredis';
+
+const redis = new Redis({
+ host: process.env.REDIS_HOST,
+ port: parseInt(process.env.REDIS_PORT || '6379'),
+ password: process.env.REDIS_PASSWORD,
+ db: 0,
+ maxRetriesPerRequest: 3,
+ enableReadyCheck: true,
+ lazyConnect: false,
+});
+
+// Cluster support
+const cluster = new Redis.Cluster([
+ { host: 'node1', port: 6379 },
+ { host: 'node2', port: 6379 },
+]);
+```
+
+### Neo4j
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `neo4j` | `neo4j-driver` | ^5.15.0 | Official driver with TypeScript support |
+
+**Migration complexity**: Low
+**Recommendation**: Use official driver with TypeScript type definitions.
+
+```typescript
+import neo4j from 'neo4j-driver';
+
+const driver = neo4j.driver(
+ process.env.NEO4J_URI!,
+ neo4j.auth.basic(
+ process.env.NEO4J_USER!,
+ process.env.NEO4J_PASSWORD!
+ ),
+ {
+ maxConnectionPoolSize: 50,
+ connectionTimeout: 30000,
+ }
+);
+
+const session = driver.session({ database: 'neo4j' });
+try {
+ const result = await session.run(
+ 'MATCH (n:Entity {name: $name}) RETURN n',
+ { name: 'John' }
+ );
+} finally {
+ await session.close();
+}
+```
+
+### Memgraph
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `neo4j` | `neo4j-driver` | ^5.15.0 | Compatible with Neo4j driver |
+
+**Migration complexity**: Low
+**Note**: Memgraph uses the Neo4j protocol, so the same driver works.
+
+### NetworkX (Graph Library)
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `networkx` | `graphology` | ^0.25.0 | Modern graph library for JS |
+| | `cytoscape` | ^3.28.0 | Alternative with visualization |
+
+**Migration complexity**: Medium
+**Recommendation**: Use `graphology` - most feature-complete and maintained.
+
+```typescript
+import Graph from 'graphology';
+
+const graph = new Graph({ type: 'undirected' });
+
+// Add nodes and edges
+graph.addNode('A', { name: 'Node A', type: 'entity' });
+graph.addNode('B', { name: 'Node B', type: 'entity' });
+graph.addEdge('A', 'B', { weight: 0.5, description: 'related to' });
+
+// Query
+const degree = graph.degree('A');
+const neighbors = graph.neighbors('A');
+const hasEdge = graph.hasEdge('A', 'B');
+
+// Serialization
+import { serialize, deserialize } from 'graphology-utils';
+const json = serialize(graph);
+const newGraph = deserialize(json);
+```
+
+### FAISS (Vector Search)
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `faiss-cpu` | **No direct port** | N/A | Need alternative approach |
+
+**Migration complexity**: High
+**Alternatives**:
+1. **hnswlib-node** (^2.0.0): HNSW algorithm implementation
+2. **vectra** (^0.4.0): Simple vector database for Node.js
+3. Use cloud services: Pinecone, Weaviate, Qdrant
+
+```typescript
+// Option 1: hnswlib-node (closest to FAISS)
+import { HierarchicalNSW } from 'hnswlib-node';
+
+const index = new HierarchicalNSW('cosine', 1536);
+index.initIndex(10000); // Max elements
+index.addItems(vectors, labels);
+
+const results = index.searchKnn(queryVector, 10);
+
+// Option 2: vectra (simpler, file-based)
+import { LocalIndex } from 'vectra';
+
+const index = new LocalIndex('./vectors');
+await index.createIndex();
+await index.insertItem({
+ id: 'item1',
+ vector: [0.1, 0.2, ...],
+ metadata: { text: 'content' }
+});
+
+const results = await index.queryItems(queryVector, 10);
+```
+
+### Milvus
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `pymilvus` | `@zilliz/milvus2-sdk-node` | ^2.3.0 | Official client |
+
+**Migration complexity**: Low
+**Recommendation**: Use official SDK.
+
+```typescript
+import { MilvusClient } from '@zilliz/milvus2-sdk-node';
+
+const client = new MilvusClient({
+ address: process.env.MILVUS_ADDRESS!,
+ username: process.env.MILVUS_USER,
+ password: process.env.MILVUS_PASSWORD,
+});
+
+// Create collection
+await client.createCollection({
+ collection_name: 'entities',
+ fields: [
+ { name: 'id', data_type: DataType.VarChar, is_primary_key: true },
+ { name: 'vector', data_type: DataType.FloatVector, dim: 1536 },
+ ],
+});
+
+// Insert vectors
+await client.insert({
+ collection_name: 'entities',
+ data: [{ id: '1', vector: [...] }],
+});
+
+// Search
+const results = await client.search({
+ collection_name: 'entities',
+ vector: queryVector,
+ limit: 10,
+});
+```
+
+### Qdrant
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `qdrant-client` | `@qdrant/js-client-rest` | ^1.8.0 | Official REST client |
+
+**Migration complexity**: Low
+**Recommendation**: Use official client.
+
+```typescript
+import { QdrantClient } from '@qdrant/js-client-rest';
+
+const client = new QdrantClient({
+ url: process.env.QDRANT_URL!,
+ apiKey: process.env.QDRANT_API_KEY,
+});
+
+// Create collection
+await client.createCollection('entities', {
+ vectors: { size: 1536, distance: 'Cosine' },
+});
+
+// Upsert vectors
+await client.upsert('entities', {
+ points: [
+ {
+ id: 1,
+ vector: [...],
+ payload: { text: 'content' },
+ },
+ ],
+});
+
+// Search
+const results = await client.search('entities', {
+ vector: queryVector,
+ limit: 10,
+});
+```
+
+## LLM and Embedding Dependencies
+
+### OpenAI
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `openai` | `openai` | ^4.28.0 | Official SDK with TypeScript |
+
+**Migration complexity**: Low
+**Recommendation**: Use official SDK - excellent TypeScript support.
+
+```typescript
+import OpenAI from 'openai';
+
+const openai = new OpenAI({
+ apiKey: process.env.OPENAI_API_KEY,
+ maxRetries: 3,
+ timeout: 60000,
+});
+
+// Chat completion
+const completion = await openai.chat.completions.create({
+ model: 'gpt-4o-mini',
+ messages: [{ role: 'user', content: 'Hello!' }],
+ temperature: 0.7,
+});
+
+// Streaming
+const stream = await openai.chat.completions.create({
+ model: 'gpt-4o-mini',
+ messages: [{ role: 'user', content: 'Hello!' }],
+ stream: true,
+});
+
+for await (const chunk of stream) {
+ process.stdout.write(chunk.choices[0]?.delta?.content || '');
+}
+
+// Embeddings
+const embedding = await openai.embeddings.create({
+ model: 'text-embedding-3-small',
+ input: 'Text to embed',
+});
+```
+
+### Anthropic
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `anthropic` | `@anthropic-ai/sdk` | ^0.17.0 | Official SDK |
+
+**Migration complexity**: Low
+
+```typescript
+import Anthropic from '@anthropic-ai/sdk';
+
+const client = new Anthropic({
+ apiKey: process.env.ANTHROPIC_API_KEY,
+});
+
+const message = await client.messages.create({
+ model: 'claude-3-opus-20240229',
+ max_tokens: 1024,
+ messages: [{ role: 'user', content: 'Hello!' }],
+});
+
+// Streaming
+const stream = await client.messages.stream({
+ model: 'claude-3-opus-20240229',
+ max_tokens: 1024,
+ messages: [{ role: 'user', content: 'Hello!' }],
+});
+
+for await (const chunk of stream) {
+ process.stdout.write(chunk.delta.text || '');
+}
+```
+
+### Ollama
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `ollama` | `ollama` | ^0.5.0 | Official Node.js library |
+
+**Migration complexity**: Low
+
+```typescript
+import ollama from 'ollama';
+
+const response = await ollama.chat({
+ model: 'llama2',
+ messages: [{ role: 'user', content: 'Hello!' }],
+});
+
+// Streaming
+const stream = await ollama.chat({
+ model: 'llama2',
+ messages: [{ role: 'user', content: 'Hello!' }],
+ stream: true,
+});
+
+for await (const chunk of stream) {
+ process.stdout.write(chunk.message.content);
+}
+
+// Embeddings
+const embedding = await ollama.embeddings({
+ model: 'llama2',
+ prompt: 'Text to embed',
+});
+```
+
+### Hugging Face
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `transformers` | `@xenova/transformers` | ^2.12.0 | Transformers.js (ONNX-based) |
+| `sentence-transformers` | **Use Inference API** | N/A | No direct equivalent |
+
+**Migration complexity**: Medium to High
+**Recommendation**: Use Hugging Face Inference API for server-side, or Transformers.js for client-side.
+
+```typescript
+// Option 1: Hugging Face Inference API (recommended for server)
+import { HfInference } from '@huggingface/inference';
+
+const hf = new HfInference(process.env.HF_TOKEN);
+
+const result = await hf.textGeneration({
+ model: 'meta-llama/Llama-2-7b-chat-hf',
+ inputs: 'The answer to the universe is',
+});
+
+// Option 2: Transformers.js (ONNX models, client-side or server)
+import { pipeline } from '@xenova/transformers';
+
+const embedder = await pipeline('feature-extraction',
+ 'Xenova/all-MiniLM-L6-v2');
+const embeddings = await embedder('Text to embed', {
+ pooling: 'mean',
+ normalize: true,
+});
+```
+
+### Tokenization (tiktoken)
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `tiktoken` | `@dqbd/tiktoken` | ^1.0.7 | WASM-based port |
+| | `js-tiktoken` | ^1.0.10 | Alternative pure JS |
+
+**Migration complexity**: Medium
+**Recommendation**: Use `@dqbd/tiktoken` for best compatibility.
+
+```typescript
+import { encoding_for_model } from '@dqbd/tiktoken';
+
+// Initialize encoder for specific model
+const encoder = encoding_for_model('gpt-4o-mini');
+
+// Encode text to tokens
+const tokens = encoder.encode('Hello, world!');
+console.log(`Token count: ${tokens.length}`);
+
+// Decode tokens back to text
+const text = encoder.decode(tokens);
+
+// Don't forget to free resources
+encoder.free();
+
+// Alternative: js-tiktoken (pure JS, no WASM)
+import { encodingForModel } from 'js-tiktoken';
+const enc = encodingForModel('gpt-4o-mini');
+const tokenCount = enc.encode('Hello, world!').length;
+```
+
+## API and Web Framework Dependencies
+
+### FastAPI → Node.js Framework
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `fastapi` | `fastify` | ^4.25.0 | Fast, low overhead, TypeScript-friendly |
+| | `@fastify/swagger` | ^8.13.0 | OpenAPI documentation |
+| | `@fastify/swagger-ui` | ^2.1.0 | Swagger UI |
+| | `@fastify/cors` | ^8.5.0 | CORS support |
+| | `@fastify/jwt` | ^7.2.0 | JWT authentication |
+
+**Alternative**: Express.js (^4.18.0) - More familiar but slower
+
+**Migration complexity**: Low
+**Recommendation**: Use Fastify for similar performance to FastAPI.
+
+```typescript
+import Fastify from 'fastify';
+import fastifySwagger from '@fastify/swagger';
+import fastifySwaggerUi from '@fastify/swagger-ui';
+import fastifyJwt from '@fastify/jwt';
+
+const app = Fastify({ logger: true });
+
+// Swagger/OpenAPI
+await app.register(fastifySwagger, {
+ openapi: {
+ info: { title: 'LightRAG API', version: '1.0.0' },
+ },
+});
+await app.register(fastifySwaggerUi, {
+ routePrefix: '/docs',
+});
+
+// JWT
+await app.register(fastifyJwt, {
+ secret: process.env.JWT_SECRET!,
+});
+
+// Route with schema
+app.post('/query', {
+ schema: {
+ body: {
+ type: 'object',
+ required: ['query'],
+ properties: {
+ query: { type: 'string' },
+ mode: { type: 'string', enum: ['local', 'global', 'hybrid'] },
+ },
+ },
+ },
+}, async (request, reply) => {
+ return { response: 'Answer' };
+});
+
+await app.listen({ port: 9621, host: '0.0.0.0' });
+```
+
+### Pydantic → TypeScript Validation
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `pydantic` | `zod` | ^3.22.0 | Runtime validation, best TS integration |
+| | `class-validator` | ^0.14.0 | Decorator-based validation |
+| | `joi` | ^17.12.0 | Traditional validation library |
+
+**Migration complexity**: Low
+**Recommendation**: Use `zod` for runtime validation with excellent TypeScript inference.
+
+```typescript
+import { z } from 'zod';
+
+// Define schema
+const QuerySchema = z.object({
+ query: z.string().min(3),
+ mode: z.enum(['local', 'global', 'hybrid', 'mix', 'naive', 'bypass']).default('mix'),
+ top_k: z.number().positive().default(40),
+ stream: z.boolean().default(false),
+});
+
+// Infer TypeScript type from schema
+type QueryInput = z.infer;
+
+// Validate at runtime
+function handleQuery(data: unknown) {
+ const validated = QuerySchema.parse(data); // Throws on invalid
+ // validated is now typed as QueryInput
+ return validated;
+}
+
+// Safe parse (returns result object)
+const result = QuerySchema.safeParse(data);
+if (result.success) {
+ console.log(result.data);
+} else {
+ console.error(result.error);
+}
+```
+
+### Uvicorn → Node.js Server
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `uvicorn` | Native Node.js | N/A | Node.js has built-in HTTP server |
+
+**Migration complexity**: None
+**Note**: Fastify/Express handle the server internally. For clustering:
+
+```typescript
+import cluster from 'cluster';
+import os from 'os';
+
+if (cluster.isPrimary) {
+ const numCPUs = os.cpus().length;
+ for (let i = 0; i < numCPUs; i++) {
+ cluster.fork();
+ }
+} else {
+ // Start Fastify server
+ await app.listen({ port: 9621 });
+}
+```
+
+## Utility and Helper Dependencies
+
+### Text Processing and Utilities
+
+| Python Package | npm Package | Version | Migration Complexity |
+|----------------|-------------|---------|---------------------|
+| `pypinyin` | `pinyin` | ^3.0.0 | Low - direct equivalent |
+| `xlsxwriter` | `xlsx` or `exceljs` | ^0.18.0 or ^4.3.0 | Low - good alternatives |
+| `pypdf2` | `pdf-parse` | ^1.1.1 | Medium - different API |
+| `python-docx` | `docx` | ^8.5.0 | Low - similar API |
+| `python-pptx` | `pptxgenjs` | ^3.12.0 | Medium - different focus |
+| `openpyxl` | `xlsx` | ^0.18.0 | Low - handles both read/write |
+
+### Logging and Monitoring
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `logging` | `pino` | ^8.17.0 | Fast, structured logging |
+| | `winston` | ^3.11.0 | Feature-rich alternative |
+| `psutil` | `systeminformation` | ^5.21.0 | System monitoring |
+
+```typescript
+import pino from 'pino';
+
+const logger = pino({
+ level: process.env.LOG_LEVEL || 'info',
+ transport: {
+ target: 'pino-pretty',
+ options: { colorize: true },
+ },
+});
+
+logger.info({ user: 'John' }, 'User logged in');
+logger.error({ err: error }, 'Operation failed');
+```
+
+### Authentication
+
+| Python Package | npm Package | Version | Notes |
+|----------------|-------------|---------|-------|
+| `PyJWT` | `jsonwebtoken` | ^9.0.2 | JWT creation and verification |
+| `python-jose` | `jose` | ^5.2.0 | More complete JWT/JWE/JWS library |
+| `passlib` | `bcrypt` | ^5.1.0 | Password hashing |
+
+```typescript
+import jwt from 'jsonwebtoken';
+import bcrypt from 'bcrypt';
+
+// JWT
+const token = jwt.sign(
+ { user_id: 123 },
+ process.env.JWT_SECRET!,
+ { expiresIn: '24h' }
+);
+
+const decoded = jwt.verify(token, process.env.JWT_SECRET!);
+
+// Password hashing
+const hash = await bcrypt.hash('password', 10);
+const isValid = await bcrypt.compare('password', hash);
+```
+
+## Migration Complexity Assessment
+
+### Complexity Levels Defined
+
+**Low Complexity** (1-2 days per component):
+- Direct npm equivalent exists with similar API
+- Well-documented TypeScript support
+- Minimal code changes required
+- Examples: PostgreSQL, MongoDB, Redis, OpenAI, Anthropic
+
+**Medium Complexity** (3-5 days per component):
+- npm equivalent exists but with different API patterns
+- Requires adapter layer or wrapper
+- Good TypeScript support but needs configuration
+- Examples: NetworkX → graphology, tiktoken, Hugging Face
+
+**High Complexity** (1-2 weeks per component):
+- No direct npm equivalent
+- Requires custom implementation or major architectural changes
+- May need cloud service integration
+- Examples: FAISS → alternatives, sentence-transformers
+
+### Overall Migration Complexity by Category
+
+| Category | Complexity | Estimated Effort | Risk Level |
+|----------|-----------|------------------|-----------|
+| Core Storage (PostgreSQL, MongoDB, Redis) | Low | 1 week | Low |
+| Graph Storage (Neo4j, NetworkX) | Medium | 1-2 weeks | Medium |
+| Vector Storage (FAISS alternatives) | Medium-High | 2 weeks | Medium |
+| LLM Integration (OpenAI, Anthropic, Ollama) | Low | 3 days | Low |
+| Tokenization (tiktoken) | Medium | 3 days | Medium |
+| API Framework (FastAPI → Fastify) | Low | 1 week | Low |
+| Validation (Pydantic → Zod) | Low | 3 days | Low |
+| Authentication & Security | Low | 3 days | Low |
+| File Processing | Medium | 1 week | Medium |
+| Utilities & Helpers | Low | 3 days | Low |
+
+### Version Compatibility Matrix
+
+| Dependency | Recommended Version | Minimum Node.js | Notes |
+|-----------|-------------------|----------------|-------|
+| Node.js | 20 LTS | 18.0.0 | Use LTS for production |
+| TypeScript | 5.3.0+ | N/A | For latest type features |
+| pg | ^8.11.0 | 14.0.0 | PostgreSQL client |
+| mongodb | ^6.3.0 | 16.0.0 | MongoDB driver |
+| ioredis | ^5.3.0 | 14.0.0 | Redis client |
+| neo4j-driver | ^5.15.0 | 14.0.0 | Neo4j client |
+| graphology | ^0.25.0 | 14.0.0 | Graph library |
+| openai | ^4.28.0 | 18.0.0 | OpenAI SDK |
+| fastify | ^4.25.0 | 18.0.0 | Web framework |
+| zod | ^3.22.0 | 16.0.0 | Validation |
+| pino | ^8.17.0 | 14.0.0 | Logging |
+| @dqbd/tiktoken | ^1.0.7 | 14.0.0 | Tokenization |
+
+### Migration Strategy Recommendations
+
+**Phase 1 - Core Infrastructure** (Weeks 1-2):
+- Set up TypeScript project structure
+- Migrate storage abstractions and interfaces
+- Implement PostgreSQL storage (reference implementation)
+- Set up testing framework
+
+**Phase 2 - Storage Implementations** (Weeks 3-4):
+- Migrate MongoDB, Redis implementations
+- Implement NetworkX → graphology migration
+- Set up vector storage alternatives (Qdrant or Milvus)
+- Implement document status storage
+
+**Phase 3 - LLM Integration** (Week 5):
+- Migrate OpenAI integration
+- Add Anthropic and Ollama support
+- Implement tiktoken for tokenization
+- Add retry and rate limiting logic
+
+**Phase 4 - Core Logic** (Weeks 6-8):
+- Migrate chunking logic
+- Implement entity extraction
+- Implement graph merging
+- Set up pipeline processing
+
+**Phase 5 - Query Engine** (Weeks 8-9):
+- Implement query modes
+- Add context building
+- Integrate reranking
+- Add streaming support
+
+**Phase 6 - API Layer** (Week 10):
+- Build Fastify server
+- Implement all endpoints
+- Add authentication
+- Set up OpenAPI docs
+
+**Phase 7 - Testing & Polish** (Weeks 11-12):
+- Comprehensive testing
+- Performance optimization
+- Documentation
+- Deployment setup
+
+## Summary
+
+This dependency migration guide provides:
+
+1. **Complete mapping** of Python packages to Node.js/npm equivalents
+2. **Code examples** showing API differences and migration patterns
+3. **Complexity assessment** for each dependency category
+4. **Version recommendations** with compatibility notes
+5. **Migration strategy** with phased approach
+6. **Risk assessment** for high-complexity migrations
+
+The migration is very feasible with most dependencies having good or excellent npm equivalents. The main challenges are:
+- FAISS vector search (use Qdrant, Milvus, or hnswlib-node)
+- NetworkX graph library (use graphology)
+- Sentence transformers (use Hugging Face Inference API)
+
+With proper planning and the recommended npm packages, a production-ready TypeScript implementation can be achieved in 12-14 weeks with a small team.
diff --git a/reverse_documentation/05-typescript-project-structure-and-roadmap.md b/reverse_documentation/05-typescript-project-structure-and-roadmap.md
new file mode 100644
index 00000000..edbf459e
--- /dev/null
+++ b/reverse_documentation/05-typescript-project-structure-and-roadmap.md
@@ -0,0 +1,1169 @@
+# TypeScript Project Structure and Migration Roadmap
+
+## Table of Contents
+1. [Recommended Project Structure](#recommended-project-structure)
+2. [Module Organization](#module-organization)
+3. [Configuration Files](#configuration-files)
+4. [Build and Development Workflow](#build-and-development-workflow)
+5. [Testing Strategy](#testing-strategy)
+6. [Phase-by-Phase Migration Roadmap](#phase-by-phase-migration-roadmap)
+7. [Deployment and CI/CD](#deployment-and-cicd)
+
+## Recommended Project Structure
+
+### Directory Layout
+
+```
+lightrag-ts/
+├── src/
+│ ├── core/
+│ │ ├── LightRAG.ts # Main LightRAG class
+│ │ ├── Pipeline.ts # Document processing pipeline
+│ │ ├── QueryEngine.ts # Query execution engine
+│ │ └── index.ts
+│ ├── storage/
+│ │ ├── base/
+│ │ │ ├── BaseKVStorage.ts # KV storage interface
+│ │ │ ├── BaseVectorStorage.ts # Vector storage interface
+│ │ │ ├── BaseGraphStorage.ts # Graph storage interface
+│ │ │ ├── DocStatusStorage.ts # Status storage interface
+│ │ │ └── index.ts
+│ │ ├── implementations/
+│ │ │ ├── kv/
+│ │ │ │ ├── JsonKVStorage.ts
+│ │ │ │ ├── PostgresKVStorage.ts
+│ │ │ │ ├── MongoKVStorage.ts
+│ │ │ │ └── RedisKVStorage.ts
+│ │ │ ├── vector/
+│ │ │ │ ├── NanoVectorStorage.ts
+│ │ │ │ ├── PostgresVectorStorage.ts
+│ │ │ │ ├── QdrantVectorStorage.ts
+│ │ │ │ └── MilvusVectorStorage.ts
+│ │ │ ├── graph/
+│ │ │ │ ├── GraphologyStorage.ts
+│ │ │ │ ├── Neo4jStorage.ts
+│ │ │ │ ├── PostgresGraphStorage.ts
+│ │ │ │ └── MemgraphStorage.ts
+│ │ │ └── status/
+│ │ │ ├── JsonDocStatusStorage.ts
+│ │ │ ├── PostgresDocStatusStorage.ts
+│ │ │ └── MongoDocStatusStorage.ts
+│ │ ├── StorageFactory.ts # Factory for creating storage instances
+│ │ └── index.ts
+│ ├── llm/
+│ │ ├── base/
+│ │ │ ├── LLMProvider.ts # LLM provider interface
+│ │ │ ├── EmbeddingProvider.ts # Embedding interface
+│ │ │ └── index.ts
+│ │ ├── providers/
+│ │ │ ├── OpenAIProvider.ts
+│ │ │ ├── AnthropicProvider.ts
+│ │ │ ├── OllamaProvider.ts
+│ │ │ ├── BedrockProvider.ts
+│ │ │ └── HuggingFaceProvider.ts
+│ │ ├── LLMFactory.ts
+│ │ └── index.ts
+│ ├── operations/
+│ │ ├── chunking.ts # Text chunking logic
+│ │ ├── extraction.ts # Entity/relation extraction
+│ │ ├── merging.ts # Graph merging algorithms
+│ │ ├── retrieval.ts # Retrieval strategies
+│ │ └── index.ts
+│ ├── api/
+│ │ ├── server.ts # Fastify server setup
+│ │ ├── routes/
+│ │ │ ├── query.ts
+│ │ │ ├── documents.ts
+│ │ │ ├── graph.ts
+│ │ │ ├── status.ts
+│ │ │ └── index.ts
+│ │ ├── middleware/
+│ │ │ ├── auth.ts
+│ │ │ ├── validation.ts
+│ │ │ ├── errorHandler.ts
+│ │ │ └── index.ts
+│ │ ├── schemas/
+│ │ │ ├── query.schema.ts
+│ │ │ ├── document.schema.ts
+│ │ │ └── index.ts
+│ │ └── index.ts
+│ ├── types/
+│ │ ├── models.ts # Core data models
+│ │ ├── config.ts # Configuration types
+│ │ ├── storage.ts # Storage-related types
+│ │ ├── query.ts # Query-related types
+│ │ └── index.ts
+│ ├── utils/
+│ │ ├── tokenizer.ts # Tokenization utilities
+│ │ ├── hashing.ts # MD5, SHA utilities
+│ │ ├── cache.ts # Caching utilities
+│ │ ├── retry.ts # Retry logic
+│ │ ├── concurrency.ts # Rate limiting, semaphores
+│ │ ├── logger.ts # Logging setup
+│ │ └── index.ts
+│ ├── prompts/
+│ │ ├── extraction.ts # Entity extraction prompts
+│ │ ├── keywords.ts # Keyword extraction prompts
+│ │ ├── summary.ts # Summary prompts
+│ │ └── index.ts
+│ └── index.ts # Main entry point
+├── tests/
+│ ├── unit/
+│ │ ├── core/
+│ │ ├── storage/
+│ │ ├── llm/
+│ │ ├── operations/
+│ │ └── utils/
+│ ├── integration/
+│ │ ├── storage/
+│ │ ├── llm/
+│ │ └── api/
+│ ├── e2e/
+│ │ └── scenarios/
+│ ├── fixtures/
+│ │ └── test-data.ts
+│ └── setup.ts
+├── examples/
+│ ├── basic-usage.ts
+│ ├── custom-storage.ts
+│ ├── streaming-query.ts
+│ └── batch-processing.ts
+├── docs/
+│ ├── api/
+│ ├── guides/
+│ └── migration/
+├── scripts/
+│ ├── build.ts
+│ ├── migrate-data.ts
+│ └── generate-types.ts
+├── .github/
+│ └── workflows/
+│ ├── ci.yml
+│ ├── release.yml
+│ └── deploy.yml
+├── package.json
+├── tsconfig.json
+├── tsconfig.build.json
+├── vitest.config.ts
+├── .env.example
+├── .eslintrc.js
+├── .prettierrc
+├── Dockerfile
+├── docker-compose.yml
+└── README.md
+```
+
+## Module Organization
+
+### Core Module (`src/core/`)
+
+The core module contains the main LightRAG orchestration logic.
+
+**LightRAG.ts** - Main class:
+```typescript
+import { LightRAGConfig } from '../types/config';
+import { StorageFactory } from '../storage/StorageFactory';
+import { LLMFactory } from '../llm/LLMFactory';
+import { Pipeline } from './Pipeline';
+import { QueryEngine } from './QueryEngine';
+
+export class LightRAG {
+ private config: LightRAGConfig;
+ private kvStorage: BaseKVStorage;
+ private vectorStorage: BaseVectorStorage;
+ private graphStorage: BaseGraphStorage;
+ private statusStorage: DocStatusStorage;
+ private llmProvider: LLMProvider;
+ private embeddingProvider: EmbeddingProvider;
+ private pipeline: Pipeline;
+ private queryEngine: QueryEngine;
+
+ constructor(config: LightRAGConfig) {
+ this.config = config;
+ this.initializeStorages();
+ this.initializeProviders();
+ this.pipeline = new Pipeline(this);
+ this.queryEngine = new QueryEngine(this);
+ }
+
+ async initialize(): Promise {
+ await Promise.all([
+ this.kvStorage.initialize(),
+ this.vectorStorage.initialize(),
+ this.graphStorage.initialize(),
+ this.statusStorage.initialize(),
+ ]);
+ }
+
+ async insert(documents: string | string[]): Promise {
+ return this.pipeline.processDocuments(documents);
+ }
+
+ async query(query: string, params?: QueryParam): Promise {
+ return this.queryEngine.execute(query, params);
+ }
+
+ async close(): Promise {
+ await Promise.all([
+ this.kvStorage.finalize(),
+ this.vectorStorage.finalize(),
+ this.graphStorage.finalize(),
+ this.statusStorage.finalize(),
+ ]);
+ }
+}
+```
+
+### Storage Module (`src/storage/`)
+
+Storage implementations follow a factory pattern for easy swapping.
+
+**StorageFactory.ts**:
+```typescript
+import { LightRAGConfig } from '../types/config';
+import { BaseKVStorage, BaseVectorStorage, BaseGraphStorage } from './base';
+
+export class StorageFactory {
+ static createKVStorage(config: LightRAGConfig): BaseKVStorage {
+ const { kv_storage } = config;
+
+ switch (kv_storage) {
+ case 'JsonKVStorage':
+ return new JsonKVStorage(config);
+ case 'PostgresKVStorage':
+ return new PostgresKVStorage(config);
+ case 'MongoKVStorage':
+ return new MongoKVStorage(config);
+ case 'RedisKVStorage':
+ return new RedisKVStorage(config);
+ default:
+ throw new Error(`Unknown KV storage: ${kv_storage}`);
+ }
+ }
+
+ static createVectorStorage(config: LightRAGConfig): BaseVectorStorage {
+ // Similar pattern for vector storage
+ }
+
+ static createGraphStorage(config: LightRAGConfig): BaseGraphStorage {
+ // Similar pattern for graph storage
+ }
+}
+```
+
+### LLM Module (`src/llm/`)
+
+LLM providers implement a common interface for consistent usage.
+
+**base/LLMProvider.ts**:
+```typescript
+export interface LLMProvider {
+ name: string;
+
+ chat(
+ messages: ConversationMessage[],
+ options?: LLMOptions
+ ): Promise;
+
+ streamChat(
+ messages: ConversationMessage[],
+ options?: LLMOptions
+ ): AsyncIterableIterator;
+
+ generateEmbeddings(
+ texts: string[],
+ options?: EmbeddingOptions
+ ): Promise;
+}
+
+export interface LLMOptions {
+ temperature?: number;
+ max_tokens?: number;
+ top_p?: number;
+ model?: string;
+ stop?: string[];
+}
+```
+
+**providers/OpenAIProvider.ts**:
+```typescript
+import OpenAI from 'openai';
+import { LLMProvider, LLMOptions } from '../base/LLMProvider';
+
+export class OpenAIProvider implements LLMProvider {
+ name = 'openai';
+ private client: OpenAI;
+
+ constructor(apiKey: string) {
+ this.client = new OpenAI({ apiKey });
+ }
+
+ async chat(
+ messages: ConversationMessage[],
+ options: LLMOptions = {}
+ ): Promise {
+ const completion = await this.client.chat.completions.create({
+ model: options.model || 'gpt-4o-mini',
+ messages,
+ temperature: options.temperature ?? 0.7,
+ max_tokens: options.max_tokens,
+ });
+
+ return completion.choices[0].message.content || '';
+ }
+
+ async *streamChat(
+ messages: ConversationMessage[],
+ options: LLMOptions = {}
+ ): AsyncIterableIterator {
+ const stream = await this.client.chat.completions.create({
+ model: options.model || 'gpt-4o-mini',
+ messages,
+ stream: true,
+ temperature: options.temperature ?? 0.7,
+ });
+
+ for await (const chunk of stream) {
+ const content = chunk.choices[0]?.delta?.content;
+ if (content) yield content;
+ }
+ }
+
+ async generateEmbeddings(texts: string[]): Promise {
+ const response = await this.client.embeddings.create({
+ model: 'text-embedding-3-small',
+ input: texts,
+ });
+
+ return response.data.map(item => item.embedding);
+ }
+}
+```
+
+### API Module (`src/api/`)
+
+RESTful API using Fastify.
+
+**server.ts**:
+```typescript
+import Fastify from 'fastify';
+import cors from '@fastify/cors';
+import helmet from '@fastify/helmet';
+import jwt from '@fastify/jwt';
+import swagger from '@fastify/swagger';
+import swaggerUi from '@fastify/swagger-ui';
+import { queryRoutes } from './routes/query';
+import { documentRoutes } from './routes/documents';
+import { errorHandler } from './middleware/errorHandler';
+
+export async function createServer(rag: LightRAG) {
+ const app = Fastify({
+ logger: {
+ level: process.env.LOG_LEVEL || 'info',
+ },
+ });
+
+ // Security
+ await app.register(helmet);
+ await app.register(cors, {
+ origin: process.env.CORS_ORIGIN || '*',
+ });
+
+ // JWT
+ await app.register(jwt, {
+ secret: process.env.JWT_SECRET!,
+ });
+
+ // OpenAPI docs
+ await app.register(swagger, {
+ openapi: {
+ info: {
+ title: 'LightRAG API',
+ version: '1.0.0',
+ },
+ servers: [
+ { url: 'http://localhost:9621' },
+ ],
+ },
+ });
+
+ await app.register(swaggerUi, {
+ routePrefix: '/docs',
+ });
+
+ // Routes
+ await app.register(queryRoutes, { rag });
+ await app.register(documentRoutes, { rag });
+
+ // Error handling
+ app.setErrorHandler(errorHandler);
+
+ return app;
+}
+```
+
+## Configuration Files
+
+### package.json
+
+```json
+{
+ "name": "lightrag-ts",
+ "version": "1.0.0",
+ "description": "TypeScript implementation of LightRAG",
+ "main": "dist/index.js",
+ "types": "dist/index.d.ts",
+ "engines": {
+ "node": ">=18.0.0"
+ },
+ "scripts": {
+ "dev": "tsx watch src/index.ts",
+ "build": "tsup",
+ "test": "vitest",
+ "test:unit": "vitest run --testPathPattern=tests/unit",
+ "test:integration": "vitest run --testPathPattern=tests/integration",
+ "test:e2e": "vitest run --testPathPattern=tests/e2e",
+ "test:coverage": "vitest run --coverage",
+ "lint": "eslint src --ext .ts",
+ "lint:fix": "eslint src --ext .ts --fix",
+ "format": "prettier --write \"src/**/*.ts\"",
+ "typecheck": "tsc --noEmit",
+ "start": "node dist/index.js",
+ "start:api": "node dist/api/server.js"
+ },
+ "dependencies": {
+ "@anthropic-ai/sdk": "^0.17.0",
+ "@dqbd/tiktoken": "^1.0.7",
+ "@fastify/cors": "^8.5.0",
+ "@fastify/helmet": "^11.1.0",
+ "@fastify/jwt": "^7.2.0",
+ "@fastify/swagger": "^8.13.0",
+ "@fastify/swagger-ui": "^2.1.0",
+ "@qdrant/js-client-rest": "^1.8.0",
+ "@zilliz/milvus2-sdk-node": "^2.3.0",
+ "axios": "^1.6.0",
+ "bcrypt": "^5.1.0",
+ "bottleneck": "^2.19.0",
+ "date-fns": "^3.0.0",
+ "dotenv": "^16.0.0",
+ "fastify": "^4.25.0",
+ "graphology": "^0.25.0",
+ "ioredis": "^5.3.0",
+ "json-repair": "^0.2.0",
+ "jsonwebtoken": "^9.0.2",
+ "mongodb": "^6.3.0",
+ "neo4j-driver": "^5.15.0",
+ "ollama": "^0.5.0",
+ "openai": "^4.28.0",
+ "p-limit": "^5.0.0",
+ "p-queue": "^8.0.0",
+ "p-retry": "^6.0.0",
+ "p-timeout": "^6.0.0",
+ "pg": "^8.11.0",
+ "pgvector": "^0.1.0",
+ "pino": "^8.17.0",
+ "pino-pretty": "^10.3.0",
+ "zod": "^3.22.0"
+ },
+ "devDependencies": {
+ "@types/bcrypt": "^5.0.0",
+ "@types/jsonwebtoken": "^9.0.0",
+ "@types/node": "^20.10.0",
+ "@types/pg": "^8.10.0",
+ "@typescript-eslint/eslint-plugin": "^6.15.0",
+ "@typescript-eslint/parser": "^6.15.0",
+ "@vitest/coverage-v8": "^1.1.0",
+ "eslint": "^8.56.0",
+ "eslint-config-prettier": "^9.1.0",
+ "eslint-plugin-import": "^2.29.0",
+ "prettier": "^3.1.0",
+ "tsup": "^8.0.0",
+ "tsx": "^4.7.0",
+ "typescript": "^5.3.0",
+ "vitest": "^1.1.0"
+ }
+}
+```
+
+### tsconfig.json
+
+```json
+{
+ "compilerOptions": {
+ "target": "ES2022",
+ "module": "ESNext",
+ "lib": ["ES2022"],
+ "moduleResolution": "bundler",
+ "resolveJsonModule": true,
+ "allowJs": false,
+ "declaration": true,
+ "declarationMap": true,
+ "sourceMap": true,
+ "outDir": "./dist",
+ "rootDir": "./src",
+ "removeComments": true,
+ "esModuleInterop": true,
+ "forceConsistentCasingInFileNames": true,
+ "strict": true,
+ "noImplicitAny": true,
+ "strictNullChecks": true,
+ "strictFunctionTypes": true,
+ "strictBindCallApply": true,
+ "strictPropertyInitialization": true,
+ "noImplicitThis": true,
+ "alwaysStrict": true,
+ "noUnusedLocals": true,
+ "noUnusedParameters": true,
+ "noImplicitReturns": true,
+ "noFallthroughCasesInSwitch": true,
+ "skipLibCheck": true,
+ "paths": {
+ "@/*": ["./src/*"],
+ "@core/*": ["./src/core/*"],
+ "@storage/*": ["./src/storage/*"],
+ "@llm/*": ["./src/llm/*"],
+ "@utils/*": ["./src/utils/*"]
+ }
+ },
+ "include": ["src/**/*"],
+ "exclude": ["node_modules", "dist", "tests"]
+}
+```
+
+### vitest.config.ts
+
+```typescript
+import { defineConfig } from 'vitest/config';
+import path from 'path';
+
+export default defineConfig({
+ test: {
+ globals: true,
+ environment: 'node',
+ coverage: {
+ provider: 'v8',
+ reporter: ['text', 'json', 'html'],
+ exclude: [
+ 'node_modules/',
+ 'dist/',
+ 'tests/',
+ '**/*.spec.ts',
+ '**/*.test.ts',
+ ],
+ },
+ setupFiles: ['./tests/setup.ts'],
+ },
+ resolve: {
+ alias: {
+ '@': path.resolve(__dirname, './src'),
+ '@core': path.resolve(__dirname, './src/core'),
+ '@storage': path.resolve(__dirname, './src/storage'),
+ '@llm': path.resolve(__dirname, './src/llm'),
+ '@utils': path.resolve(__dirname, './src/utils'),
+ },
+ },
+});
+```
+
+### .eslintrc.js
+
+```javascript
+module.exports = {
+ parser: '@typescript-eslint/parser',
+ extends: [
+ 'eslint:recommended',
+ 'plugin:@typescript-eslint/recommended',
+ 'plugin:import/recommended',
+ 'plugin:import/typescript',
+ 'prettier',
+ ],
+ plugins: ['@typescript-eslint', 'import'],
+ rules: {
+ '@typescript-eslint/no-explicit-any': 'warn',
+ '@typescript-eslint/no-unused-vars': ['error', {
+ argsIgnorePattern: '^_',
+ varsIgnorePattern: '^_'
+ }],
+ 'import/order': ['error', {
+ 'groups': [
+ 'builtin',
+ 'external',
+ 'internal',
+ 'parent',
+ 'sibling',
+ 'index'
+ ],
+ 'newlines-between': 'always',
+ 'alphabetize': { order: 'asc' }
+ }],
+ },
+};
+```
+
+## Build and Development Workflow
+
+### Development Mode
+
+```bash
+# Install dependencies
+pnpm install
+
+# Start development server with hot reload
+pnpm dev
+
+# Run tests in watch mode
+pnpm test
+
+# Type checking
+pnpm typecheck
+```
+
+### Build for Production
+
+```bash
+# Build optimized bundle
+pnpm build
+
+# Run production build
+pnpm start
+```
+
+### Build Configuration (tsup.config.ts)
+
+```typescript
+import { defineConfig } from 'tsup';
+
+export default defineConfig({
+ entry: ['src/index.ts', 'src/api/server.ts'],
+ format: ['esm'],
+ dts: true,
+ splitting: false,
+ sourcemap: true,
+ clean: true,
+ minify: true,
+ treeshake: true,
+ target: 'es2022',
+ outDir: 'dist',
+});
+```
+
+## Testing Strategy
+
+### Unit Tests
+
+Test individual functions and classes in isolation.
+
+**Example: Storage unit test**
+```typescript
+// tests/unit/storage/JsonKVStorage.test.ts
+import { describe, it, expect, beforeEach } from 'vitest';
+import { JsonKVStorage } from '@storage/implementations/kv/JsonKVStorage';
+
+describe('JsonKVStorage', () => {
+ let storage: JsonKVStorage;
+
+ beforeEach(() => {
+ storage = new JsonKVStorage({
+ working_dir: './test-storage',
+ workspace: 'test',
+ });
+ });
+
+ it('should store and retrieve values', async () => {
+ await storage.upsert({ key1: { value: 'test' } });
+ const result = await storage.get_by_id('key1');
+ expect(result).toEqual({ value: 'test' });
+ });
+
+ it('should return null for non-existent keys', async () => {
+ const result = await storage.get_by_id('nonexistent');
+ expect(result).toBeNull();
+ });
+});
+```
+
+### Integration Tests
+
+Test interactions between multiple components.
+
+**Example: LLM + Storage integration**
+```typescript
+// tests/integration/extraction.test.ts
+import { describe, it, expect } from 'vitest';
+import { LightRAG } from '@core/LightRAG';
+
+describe('Entity Extraction Integration', () => {
+ it('should extract and store entities', async () => {
+ const rag = new LightRAG({
+ llm_provider: 'openai',
+ openai_api_key: process.env.OPENAI_API_KEY,
+ kv_storage: 'JsonKVStorage',
+ });
+
+ await rag.initialize();
+
+ const docId = await rag.insert(
+ 'John Smith works at OpenAI in San Francisco.'
+ );
+
+ const entities = await rag.graphStorage.get_all_labels();
+ expect(entities).toContain('John Smith');
+ expect(entities).toContain('OpenAI');
+
+ await rag.close();
+ });
+});
+```
+
+### End-to-End Tests
+
+Test complete workflows from API to storage.
+
+**Example: Complete query flow**
+```typescript
+// tests/e2e/query-flow.test.ts
+import { describe, it, expect } from 'vitest';
+import { createServer } from '@/api/server';
+import { LightRAG } from '@core/LightRAG';
+
+describe('Complete Query Flow', () => {
+ it('should handle document upload and query', async () => {
+ const rag = new LightRAG(config);
+ await rag.initialize();
+
+ const app = await createServer(rag);
+
+ // Upload document
+ const uploadResponse = await app.inject({
+ method: 'POST',
+ url: '/documents',
+ payload: {
+ content: 'Test document about AI.',
+ },
+ });
+
+ expect(uploadResponse.statusCode).toBe(200);
+
+ // Query
+ const queryResponse = await app.inject({
+ method: 'POST',
+ url: '/query',
+ payload: {
+ query: 'What is AI?',
+ mode: 'mix',
+ },
+ });
+
+ expect(queryResponse.statusCode).toBe(200);
+ const result = JSON.parse(queryResponse.body);
+ expect(result.response).toBeDefined();
+
+ await app.close();
+ await rag.close();
+ });
+});
+```
+
+### Test Coverage Goals
+
+- **Unit Tests**: >80% code coverage
+- **Integration Tests**: Cover all component interactions
+- **E2E Tests**: Cover critical user flows
+- **Performance Tests**: Benchmark indexing and query latency
+
+## Phase-by-Phase Migration Roadmap
+
+### Phase 1: Foundation (Weeks 1-2)
+
+**Goal**: Establish project structure and core abstractions
+
+**Tasks**:
+1. Set up TypeScript project with build configuration
+2. Configure linting, formatting, and testing
+3. Define all TypeScript types and interfaces
+4. Implement base storage classes (abstract interfaces)
+5. Set up logger and basic utilities
+6. Create tokenizer wrapper for tiktoken
+
+**Deliverables**:
+- Complete project structure
+- Type definitions matching Python data models
+- Abstract storage interfaces
+- Basic utility functions
+- Test infrastructure
+
+**Success Criteria**:
+- Project builds without errors
+- All types properly defined
+- Tests can run (even if no implementations yet)
+
+### Phase 2: Storage Layer (Weeks 3-5)
+
+**Goal**: Implement storage backends
+
+**Week 3: PostgreSQL Storage**
+- PostgresKVStorage (with pooling)
+- PostgresVectorStorage (with pgvector)
+- PostgresGraphStorage
+- PostgresDocStatusStorage
+- Connection management and error handling
+
+**Week 4: Alternative Storage**
+- JsonKVStorage (file-based)
+- GraphologyStorage (NetworkX equivalent)
+- NanoVectorStorage (in-memory)
+- JsonDocStatusStorage
+
+**Week 5: Additional Storage & Testing**
+- MongoKVStorage and MongoDocStatusStorage
+- RedisKVStorage
+- QdrantVectorStorage
+- Comprehensive storage tests
+
+**Deliverables**:
+- 4+ complete storage implementations
+- Storage factory pattern
+- 100+ storage tests
+- Performance benchmarks
+
+**Success Criteria**:
+- All storage tests pass
+- Can switch storage backends via config
+- Performance comparable to Python
+
+### Phase 3: LLM Integration (Week 6)
+
+**Goal**: Implement LLM and embedding providers
+
+**Tasks**:
+1. Implement OpenAIProvider (chat + embeddings)
+2. Implement AnthropicProvider
+3. Implement OllamaProvider
+4. Add retry logic and rate limiting
+5. Implement streaming support
+6. Add embedding batch processing
+
+**Deliverables**:
+- 3+ LLM provider implementations
+- LLM factory pattern
+- Streaming support
+- Comprehensive error handling
+
+**Success Criteria**:
+- All providers work with streaming
+- Rate limiting prevents API errors
+- Retry logic handles transient failures
+
+### Phase 4: Core Engine - Part 1 (Weeks 7-8)
+
+**Goal**: Implement document processing pipeline
+
+**Week 7: Chunking & Extraction**
+- Text chunking with overlap
+- Entity extraction with LLM
+- Relationship extraction
+- Prompt template system
+
+**Week 8: Graph Merging & Indexing**
+- Entity deduplication and merging
+- Relationship consolidation
+- Vector embedding generation
+- Pipeline status tracking
+
+**Deliverables**:
+- Complete chunking implementation
+- Entity/relation extraction working
+- Graph merging algorithms
+- Vector indexing
+
+**Success Criteria**:
+- Documents successfully indexed
+- Entities and relations extracted
+- Graph properly merged
+- Status tracking works
+
+### Phase 5: Core Engine - Part 2 (Weeks 9-10)
+
+**Goal**: Implement query engine
+
+**Week 9: Retrieval Strategies**
+- Keyword extraction
+- Local mode (entity-centric)
+- Global mode (relationship-centric)
+- Hybrid mode (combined)
+
+**Week 10: Query Modes & Context**
+- Mix mode (KG + chunks)
+- Naive mode (vector only)
+- Bypass mode (direct LLM)
+- Token budget management
+- Reranking support
+
+**Deliverables**:
+- All 6 query modes working
+- Context building with token budgets
+- Reranking integration
+- Query result formatting
+
+**Success Criteria**:
+- All query modes produce correct results
+- Token budgets respected
+- Comparable results to Python version
+
+### Phase 6: API Layer (Week 11)
+
+**Goal**: Build REST API
+
+**Tasks**:
+1. Set up Fastify server
+2. Implement query endpoints
+3. Implement document management endpoints
+4. Add authentication (JWT)
+5. Add OpenAPI documentation
+6. Implement streaming responses
+
+**Deliverables**:
+- Complete REST API
+- Authentication working
+- OpenAPI docs auto-generated
+- Streaming query support
+
+**Success Criteria**:
+- All endpoints functional
+- API compatible with existing WebUI
+- Authentication secure
+- Documentation complete
+
+### Phase 7: Testing & Optimization (Week 12)
+
+**Goal**: Comprehensive testing and optimization
+
+**Tasks**:
+1. Complete unit test coverage
+2. Integration test suite
+3. E2E test scenarios
+4. Performance benchmarking
+5. Memory profiling
+6. Load testing
+
+**Deliverables**:
+- >80% test coverage
+- Performance benchmarks
+- Load test results
+- Optimization recommendations
+
+**Success Criteria**:
+- All tests pass
+- Performance meets targets
+- No memory leaks
+- Handles expected load
+
+### Phase 8: Production Hardening (Weeks 13-14)
+
+**Goal**: Production readiness
+
+**Tasks**:
+1. Add monitoring and observability
+2. Implement health checks
+3. Add graceful shutdown
+4. Create Docker images
+5. Write deployment docs
+6. Set up CI/CD pipelines
+
+**Deliverables**:
+- Docker images
+- Kubernetes manifests
+- Deployment documentation
+- CI/CD pipelines
+- Monitoring dashboards
+
+**Success Criteria**:
+- Deploys successfully
+- Monitoring works
+- Zero-downtime updates possible
+- Documentation complete
+
+## Deployment and CI/CD
+
+### Dockerfile
+
+```dockerfile
+# Build stage
+FROM node:20-alpine AS builder
+
+WORKDIR /app
+
+# Copy package files
+COPY package.json pnpm-lock.yaml ./
+RUN npm install -g pnpm && pnpm install --frozen-lockfile
+
+# Copy source
+COPY . .
+
+# Build
+RUN pnpm build
+
+# Production stage
+FROM node:20-alpine
+
+WORKDIR /app
+
+# Install production dependencies only
+COPY package.json pnpm-lock.yaml ./
+RUN npm install -g pnpm && pnpm install --prod --frozen-lockfile
+
+# Copy built files
+COPY --from=builder /app/dist ./dist
+
+# Create data directories
+RUN mkdir -p /app/data/rag_storage /app/data/inputs
+
+# Environment
+ENV NODE_ENV=production
+ENV WORKING_DIR=/app/data/rag_storage
+ENV INPUT_DIR=/app/data/inputs
+
+EXPOSE 9621
+
+CMD ["node", "dist/api/server.js"]
+```
+
+### docker-compose.yml
+
+```yaml
+version: '3.8'
+
+services:
+ lightrag:
+ build: .
+ ports:
+ - "9621:9621"
+ environment:
+ - NODE_ENV=production
+ - OPENAI_API_KEY=${OPENAI_API_KEY}
+ - PG_HOST=postgres
+ - PG_DATABASE=lightrag
+ - REDIS_HOST=redis
+ depends_on:
+ - postgres
+ - redis
+ volumes:
+ - ./data:/app/data
+
+ postgres:
+ image: ankane/pgvector:latest
+ environment:
+ - POSTGRES_DB=lightrag
+ - POSTGRES_USER=lightrag
+ - POSTGRES_PASSWORD=password
+ ports:
+ - "5432:5432"
+ volumes:
+ - postgres-data:/var/lib/postgresql/data
+
+ redis:
+ image: redis:7-alpine
+ ports:
+ - "6379:6379"
+ volumes:
+ - redis-data:/data
+
+volumes:
+ postgres-data:
+ redis-data:
+```
+
+### GitHub Actions CI/CD
+
+**.github/workflows/ci.yml**:
+```yaml
+name: CI
+
+on:
+ push:
+ branches: [main, develop]
+ pull_request:
+ branches: [main]
+
+jobs:
+ test:
+ runs-on: ubuntu-latest
+
+ services:
+ postgres:
+ image: ankane/pgvector:latest
+ env:
+ POSTGRES_DB: lightrag_test
+ POSTGRES_USER: test
+ POSTGRES_PASSWORD: test
+ ports:
+ - 5432:5432
+
+ redis:
+ image: redis:7-alpine
+ ports:
+ - 6379:6379
+
+ steps:
+ - uses: actions/checkout@v4
+
+ - uses: pnpm/action-setup@v2
+ with:
+ version: 8
+
+ - uses: actions/setup-node@v4
+ with:
+ node-version: 20
+ cache: 'pnpm'
+
+ - run: pnpm install --frozen-lockfile
+
+ - run: pnpm lint
+
+ - run: pnpm typecheck
+
+ - run: pnpm test:coverage
+
+ - uses: codecov/codecov-action@v3
+ with:
+ files: ./coverage/coverage-final.json
+
+ build:
+ runs-on: ubuntu-latest
+ needs: test
+
+ steps:
+ - uses: actions/checkout@v4
+
+ - uses: pnpm/action-setup@v2
+ with:
+ version: 8
+
+ - uses: actions/setup-node@v4
+ with:
+ node-version: 20
+ cache: 'pnpm'
+
+ - run: pnpm install --frozen-lockfile
+
+ - run: pnpm build
+
+ - uses: actions/upload-artifact@v3
+ with:
+ name: dist
+ path: dist/
+```
+
+## Summary
+
+This comprehensive project structure and roadmap provides:
+
+1. **Complete directory organization** following TypeScript best practices
+2. **Module structure** with clear separation of concerns
+3. **Configuration files** optimized for TypeScript development
+4. **Testing strategy** with unit, integration, and E2E tests
+5. **14-week migration roadmap** with clear phases and deliverables
+6. **CI/CD setup** for automated testing and deployment
+7. **Docker configuration** for containerized deployment
+
+The structure is designed to be maintainable, scalable, and production-ready while following TypeScript and Node.js ecosystem best practices. Each phase builds on previous work, allowing for incremental delivery and testing throughout the migration process.
diff --git a/reverse_documentation/06-implementation-guide.md b/reverse_documentation/06-implementation-guide.md
new file mode 100644
index 00000000..4c4d8aa5
--- /dev/null
+++ b/reverse_documentation/06-implementation-guide.md
@@ -0,0 +1,1004 @@
+# Implementation Guide: Key Components and Patterns
+
+## Table of Contents
+1. [Storage Implementation Patterns](#storage-implementation-patterns)
+2. [LLM Integration Implementation](#llm-integration-implementation)
+3. [Document Processing Pipeline](#document-processing-pipeline)
+4. [Query Engine Implementation](#query-engine-implementation)
+5. [Error Handling and Resilience](#error-handling-and-resilience)
+6. [Performance Optimization](#performance-optimization)
+7. [Testing Patterns](#testing-patterns)
+
+## Storage Implementation Patterns
+
+### PostgreSQL KV Storage Implementation
+
+A complete example showing connection pooling, error handling, and workspace isolation.
+
+```typescript
+import { Pool, PoolClient } from 'pg';
+import { BaseKVStorage } from '../base/BaseKVStorage';
+import { EmbeddingFunc } from '@/types/models';
+
+export class PostgresKVStorage extends BaseKVStorage {
+ private pool: Pool;
+ private namespace: string;
+ private tableName: string;
+
+ constructor(config: StorageConfig) {
+ super(config.namespace, config.workspace, config.global_config);
+
+ // Create connection pool
+ this.pool = new Pool({
+ host: process.env.PG_HOST,
+ port: parseInt(process.env.PG_PORT || '5432'),
+ database: process.env.PG_DATABASE,
+ user: process.env.PG_USER,
+ password: process.env.PG_PASSWORD,
+ max: 20, // Max connections in pool
+ idleTimeoutMillis: 30000,
+ connectionTimeoutMillis: 2000,
+ });
+
+ // Workspace-based namespace
+ this.namespace = config.workspace
+ ? `${config.workspace}_${config.namespace}`
+ : config.namespace;
+
+ this.tableName = `kv_${this.namespace}`;
+ }
+
+ async initialize(): Promise {
+ const client = await this.pool.connect();
+ try {
+ // Create table if not exists
+ await client.query(`
+ CREATE TABLE IF NOT EXISTS ${this.tableName} (
+ id TEXT PRIMARY KEY,
+ data JSONB NOT NULL,
+ created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+ updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+ )
+ `);
+
+ // Create index on JSONB data
+ await client.query(`
+ CREATE INDEX IF NOT EXISTS idx_${this.tableName}_data
+ ON ${this.tableName} USING GIN (data)
+ `);
+
+ logger.info(`PostgresKVStorage initialized: ${this.tableName}`);
+ } finally {
+ client.release();
+ }
+ }
+
+ async get_by_id(id: string): Promise | null> {
+ try {
+ const result = await this.pool.query(
+ `SELECT data FROM ${this.tableName} WHERE id = $1`,
+ [id]
+ );
+
+ return result.rows[0]?.data || null;
+ } catch (error) {
+ logger.error({ error, id }, 'Failed to get by id');
+ throw new StorageError('Failed to retrieve data', { cause: error });
+ }
+ }
+
+ async get_by_ids(ids: string[]): Promise[]> {
+ if (ids.length === 0) return [];
+
+ try {
+ const result = await this.pool.query(
+ `SELECT data FROM ${this.tableName} WHERE id = ANY($1)`,
+ [ids]
+ );
+
+ return result.rows.map(row => row.data);
+ } catch (error) {
+ logger.error({ error, ids }, 'Failed to get by ids');
+ throw new StorageError('Failed to retrieve data', { cause: error });
+ }
+ }
+
+ async filter_keys(keys: Set): Promise> {
+ if (keys.size === 0) return new Set();
+
+ const keyArray = Array.from(keys);
+
+ try {
+ const result = await this.pool.query(
+ `SELECT id FROM ${this.tableName} WHERE id = ANY($1)`,
+ [keyArray]
+ );
+
+ const existingKeys = new Set(result.rows.map(row => row.id));
+ return new Set(keyArray.filter(key => !existingKeys.has(key)));
+ } catch (error) {
+ logger.error({ error }, 'Failed to filter keys');
+ throw new StorageError('Failed to filter keys', { cause: error });
+ }
+ }
+
+ async upsert(data: Record>): Promise {
+ if (Object.keys(data).length === 0) return;
+
+ const client = await this.pool.connect();
+ try {
+ await client.query('BEGIN');
+
+ // Batch upsert using unnest
+ const ids = Object.keys(data);
+ const values = ids.map(id => data[id]);
+
+ await client.query(`
+ INSERT INTO ${this.tableName} (id, data, updated_at)
+ SELECT * FROM unnest($1::text[], $2::jsonb[])
+ ON CONFLICT (id)
+ DO UPDATE SET
+ data = EXCLUDED.data,
+ updated_at = CURRENT_TIMESTAMP
+ `, [ids, values]);
+
+ await client.query('COMMIT');
+
+ logger.debug(`Upserted ${ids.length} entries to ${this.tableName}`);
+ } catch (error) {
+ await client.query('ROLLBACK');
+ logger.error({ error }, 'Failed to upsert data');
+ throw new StorageError('Failed to upsert data', { cause: error });
+ } finally {
+ client.release();
+ }
+ }
+
+ async delete(ids: string[]): Promise {
+ if (ids.length === 0) return;
+
+ try {
+ await this.pool.query(
+ `DELETE FROM ${this.tableName} WHERE id = ANY($1)`,
+ [ids]
+ );
+
+ logger.debug(`Deleted ${ids.length} entries from ${this.tableName}`);
+ } catch (error) {
+ logger.error({ error, ids }, 'Failed to delete');
+ throw new StorageError('Failed to delete data', { cause: error });
+ }
+ }
+
+ async index_done_callback(): Promise {
+ // For PostgreSQL, operations are immediately persisted
+ // This callback is primarily for in-memory storages
+ logger.debug(`Index done callback for ${this.tableName}`);
+ }
+
+ async drop(): Promise<{ status: string; message: string }> {
+ try {
+ await this.pool.query(`DROP TABLE IF EXISTS ${this.tableName}`);
+ return { status: 'success', message: 'data dropped' };
+ } catch (error) {
+ logger.error({ error }, 'Failed to drop table');
+ return { status: 'error', message: String(error) };
+ }
+ }
+
+ async finalize(): Promise {
+ await this.pool.end();
+ logger.info(`PostgresKVStorage finalized: ${this.tableName}`);
+ }
+}
+```
+
+### Vector Storage with pgvector
+
+Example showing vector similarity search with PostgreSQL's pgvector extension.
+
+```typescript
+import { Pool } from 'pg';
+import pgvector from 'pgvector/pg';
+import { BaseVectorStorage } from '../base/BaseVectorStorage';
+
+export class PostgresVectorStorage extends BaseVectorStorage {
+ private pool: Pool;
+ private tableName: string;
+ private dimension: number = 1536; // OpenAI text-embedding-3-small
+
+ async initialize(): Promise {
+ const client = await this.pool.connect();
+ try {
+ // Register pgvector type
+ await pgvector.registerType(client);
+
+ // Create extension
+ await client.query('CREATE EXTENSION IF NOT EXISTS vector');
+
+ // Create table
+ await client.query(`
+ CREATE TABLE IF NOT EXISTS ${this.tableName} (
+ id TEXT PRIMARY KEY,
+ vector vector(${this.dimension}),
+ metadata JSONB,
+ created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+ )
+ `);
+
+ // Create index for vector similarity search
+ await client.query(`
+ CREATE INDEX IF NOT EXISTS idx_${this.tableName}_vector
+ ON ${this.tableName}
+ USING ivfflat (vector vector_cosine_ops)
+ WITH (lists = 100)
+ `);
+ } finally {
+ client.release();
+ }
+ }
+
+ async query(
+ query: string,
+ top_k: number,
+ query_embedding?: number[]
+ ): Promise>> {
+ // Generate embedding if not provided
+ const embedding = query_embedding ||
+ (await this.embedding_func([query]))[0];
+
+ try {
+ // Cosine similarity search
+ const result = await this.pool.query(`
+ SELECT
+ id,
+ metadata,
+ 1 - (vector <=> $1::vector) AS similarity
+ FROM ${this.tableName}
+ WHERE 1 - (vector <=> $1::vector) > $2
+ ORDER BY vector <=> $1::vector
+ LIMIT $3
+ `, [
+ pgvector.toSql(embedding),
+ this.cosine_better_than_threshold,
+ top_k
+ ]);
+
+ return result.rows.map(row => ({
+ id: row.id,
+ ...row.metadata,
+ similarity: row.similarity,
+ }));
+ } catch (error) {
+ logger.error({ error }, 'Vector query failed');
+ throw new StorageError('Vector query failed', { cause: error });
+ }
+ }
+
+ async upsert(data: Record>): Promise {
+ const entries = Object.entries(data);
+ if (entries.length === 0) return;
+
+ // Generate embeddings for new entries
+ const textsToEmbed: string[] = [];
+ const idsToEmbed: string[] = [];
+
+ for (const [id, entry] of entries) {
+ if (!entry.vector && entry.content) {
+ textsToEmbed.push(entry.content);
+ idsToEmbed.push(id);
+ }
+ }
+
+ let embeddings: number[][] = [];
+ if (textsToEmbed.length > 0) {
+ embeddings = await this.embedding_func(textsToEmbed);
+ }
+
+ // Map embeddings back to entries
+ idsToEmbed.forEach((id, idx) => {
+ data[id].vector = embeddings[idx];
+ });
+
+ const client = await this.pool.connect();
+ try {
+ await client.query('BEGIN');
+
+ for (const [id, entry] of entries) {
+ const { vector, ...metadata } = entry;
+
+ await client.query(`
+ INSERT INTO ${this.tableName} (id, vector, metadata)
+ VALUES ($1, $2, $3)
+ ON CONFLICT (id)
+ DO UPDATE SET
+ vector = EXCLUDED.vector,
+ metadata = EXCLUDED.metadata
+ `, [
+ id,
+ pgvector.toSql(vector),
+ JSON.stringify(metadata)
+ ]);
+ }
+
+ await client.query('COMMIT');
+ } catch (error) {
+ await client.query('ROLLBACK');
+ throw new StorageError('Upsert failed', { cause: error });
+ } finally {
+ client.release();
+ }
+ }
+}
+```
+
+### Graph Storage with Graphology
+
+NetworkX alternative using graphology for in-memory graph operations.
+
+```typescript
+import Graph from 'graphology';
+import { Attributes } from 'graphology-types';
+import { BaseGraphStorage } from '../base/BaseGraphStorage';
+import fs from 'fs/promises';
+import path from 'path';
+
+export class GraphologyStorage extends BaseGraphStorage {
+ private graph: Graph;
+ private filePath: string;
+
+ constructor(config: StorageConfig) {
+ super(config.namespace, config.workspace, config.global_config);
+ this.graph = new Graph({ type: 'undirected' });
+ this.filePath = path.join(
+ config.working_dir,
+ config.workspace || '',
+ `${config.namespace}_graph.json`
+ );
+ }
+
+ async initialize(): Promise {
+ // Load existing graph from disk if available
+ try {
+ const data = await fs.readFile(this.filePath, 'utf-8');
+ const serialized = JSON.parse(data);
+ this.graph.import(serialized);
+ logger.info(`Loaded graph from ${this.filePath}`);
+ } catch (error) {
+ // File doesn't exist, start with empty graph
+ logger.info(`Starting with empty graph at ${this.filePath}`);
+ }
+ }
+
+ async has_node(node_id: string): Promise {
+ return this.graph.hasNode(node_id);
+ }
+
+ async has_edge(source_node_id: string, target_node_id: string): Promise {
+ return this.graph.hasEdge(source_node_id, target_node_id);
+ }
+
+ async node_degree(node_id: string): Promise {
+ if (!this.graph.hasNode(node_id)) return 0;
+ return this.graph.degree(node_id);
+ }
+
+ async edge_degree(src_id: string, tgt_id: string): Promise {
+ const srcDegree = await this.node_degree(src_id);
+ const tgtDegree = await this.node_degree(tgt_id);
+ return srcDegree + tgtDegree;
+ }
+
+ async get_node(node_id: string): Promise | null> {
+ if (!this.graph.hasNode(node_id)) return null;
+ return this.graph.getNodeAttributes(node_id);
+ }
+
+ async get_edge(
+ source_node_id: string,
+ target_node_id: string
+ ): Promise | null> {
+ if (!this.graph.hasEdge(source_node_id, target_node_id)) return null;
+ return this.graph.getEdgeAttributes(source_node_id, target_node_id);
+ }
+
+ async get_node_edges(source_node_id: string): Promise | null> {
+ if (!this.graph.hasNode(source_node_id)) return null;
+
+ const edges: Array<[string, string]> = [];
+ this.graph.forEachEdge(source_node_id, (edge, attributes, source, target) => {
+ edges.push([source, target]);
+ });
+
+ return edges;
+ }
+
+ async upsert_node(node_id: string, node_data: Record): Promise {
+ if (this.graph.hasNode(node_id)) {
+ this.graph.mergeNodeAttributes(node_id, node_data);
+ } else {
+ this.graph.addNode(node_id, node_data);
+ }
+ }
+
+ async upsert_edge(
+ source_node_id: string,
+ target_node_id: string,
+ edge_data: Record
+ ): Promise {
+ // Ensure nodes exist
+ if (!this.graph.hasNode(source_node_id)) {
+ this.graph.addNode(source_node_id, {});
+ }
+ if (!this.graph.hasNode(target_node_id)) {
+ this.graph.addNode(target_node_id, {});
+ }
+
+ // Add or update edge (undirected)
+ if (this.graph.hasEdge(source_node_id, target_node_id)) {
+ this.graph.mergeEdgeAttributes(source_node_id, target_node_id, edge_data);
+ } else {
+ this.graph.addEdge(source_node_id, target_node_id, edge_data);
+ }
+ }
+
+ async get_all_labels(): Promise {
+ return this.graph.nodes();
+ }
+
+ async get_knowledge_graph(
+ node_label: string,
+ max_depth: number = 3,
+ max_nodes: number = 1000
+ ): Promise {
+ const nodes: KnowledgeGraphNode[] = [];
+ const edges: KnowledgeGraphEdge[] = [];
+ const visited = new Set();
+ const queue: Array<{ id: string; depth: number }> = [{ id: node_label, depth: 0 }];
+
+ while (queue.length > 0 && visited.size < max_nodes) {
+ const { id, depth } = queue.shift()!;
+
+ if (visited.has(id) || depth > max_depth) continue;
+ visited.add(id);
+
+ // Add node
+ const nodeAttrs = this.graph.getNodeAttributes(id);
+ nodes.push({
+ id,
+ labels: [nodeAttrs.entity_type || 'Entity'],
+ properties: nodeAttrs,
+ });
+
+ // Add edges and queue neighbors
+ if (depth < max_depth) {
+ this.graph.forEachEdge(id, (edge, edgeAttrs, source, target) => {
+ const neighbor = source === id ? target : source;
+
+ if (!visited.has(neighbor)) {
+ queue.push({ id: neighbor, depth: depth + 1 });
+ }
+
+ // Add edge if both nodes will be included
+ if (visited.has(neighbor) || visited.size < max_nodes) {
+ edges.push({
+ id: edge,
+ source,
+ target,
+ type: edgeAttrs.keywords || 'related',
+ properties: edgeAttrs,
+ });
+ }
+ });
+ }
+ }
+
+ return {
+ nodes,
+ edges,
+ is_truncated: visited.size >= max_nodes,
+ };
+ }
+
+ async index_done_callback(): Promise {
+ // Persist graph to disk
+ try {
+ const serialized = this.graph.export();
+ await fs.mkdir(path.dirname(this.filePath), { recursive: true });
+ await fs.writeFile(
+ this.filePath,
+ JSON.stringify(serialized, null, 2),
+ 'utf-8'
+ );
+ logger.info(`Persisted graph to ${this.filePath}`);
+ } catch (error) {
+ logger.error({ error }, 'Failed to persist graph');
+ throw new StorageError('Failed to persist graph', { cause: error });
+ }
+ }
+
+ async drop(): Promise<{ status: string; message: string }> {
+ try {
+ this.graph.clear();
+ await fs.unlink(this.filePath);
+ return { status: 'success', message: 'data dropped' };
+ } catch (error) {
+ return { status: 'error', message: String(error) };
+ }
+ }
+}
+```
+
+## LLM Integration Implementation
+
+### OpenAI Provider with Retry Logic
+
+```typescript
+import OpenAI from 'openai';
+import pRetry from 'p-retry';
+import { LLMProvider } from '../base/LLMProvider';
+
+export class OpenAIProvider implements LLMProvider {
+ name = 'openai';
+ private client: OpenAI;
+ private defaultModel: string;
+
+ constructor(apiKey: string, model: string = 'gpt-4o-mini') {
+ this.client = new OpenAI({
+ apiKey,
+ maxRetries: 0, // We handle retries ourselves
+ timeout: 180000, // 3 minutes
+ });
+ this.defaultModel = model;
+ }
+
+ async chat(
+ messages: ConversationMessage[],
+ options: LLMOptions = {}
+ ): Promise {
+ return pRetry(
+ async () => {
+ const completion = await this.client.chat.completions.create({
+ model: options.model || this.defaultModel,
+ messages,
+ temperature: options.temperature ?? 0.7,
+ max_tokens: options.max_tokens,
+ top_p: options.top_p,
+ stop: options.stop,
+ });
+
+ const content = completion.choices[0].message.content;
+ if (!content) {
+ throw new Error('Empty response from LLM');
+ }
+
+ return content;
+ },
+ {
+ retries: 3,
+ onFailedAttempt: (error) => {
+ logger.warn({
+ attempt: error.attemptNumber,
+ retriesLeft: error.retriesLeft,
+ error: error.message,
+ }, 'LLM call failed, retrying...');
+ },
+ shouldRetry: (error) => {
+ // Retry on rate limits and temporary errors
+ return error instanceof OpenAI.APIError &&
+ (error.status === 429 || error.status >= 500);
+ },
+ }
+ );
+ }
+
+ async *streamChat(
+ messages: ConversationMessage[],
+ options: LLMOptions = {}
+ ): AsyncIterableIterator {
+ const stream = await this.client.chat.completions.create({
+ model: options.model || this.defaultModel,
+ messages,
+ stream: true,
+ temperature: options.temperature ?? 0.7,
+ max_tokens: options.max_tokens,
+ });
+
+ for await (const chunk of stream) {
+ const content = chunk.choices[0]?.delta?.content;
+ if (content) yield content;
+ }
+ }
+
+ async generateEmbeddings(
+ texts: string[],
+ batchSize: number = 100
+ ): Promise {
+ const allEmbeddings: number[][] = [];
+
+ // Process in batches
+ for (let i = 0; i < texts.length; i += batchSize) {
+ const batch = texts.slice(i, i + batchSize);
+
+ const response = await pRetry(
+ () => this.client.embeddings.create({
+ model: 'text-embedding-3-small',
+ input: batch,
+ }),
+ { retries: 3 }
+ );
+
+ allEmbeddings.push(...response.data.map(item => item.embedding));
+ }
+
+ return allEmbeddings;
+ }
+}
+```
+
+## Document Processing Pipeline
+
+### Chunking Implementation
+
+```typescript
+import { Tiktoken, encoding_for_model } from '@dqbd/tiktoken';
+
+export class TextChunker {
+ private encoder: Tiktoken;
+
+ constructor(modelName: string = 'gpt-4o-mini') {
+ this.encoder = encoding_for_model(modelName);
+ }
+
+ chunk(
+ content: string,
+ chunkSize: number = 1200,
+ overlap: number = 100
+ ): TextChunkSchema[] {
+ const tokens = this.encoder.encode(content);
+ const chunks: TextChunkSchema[] = [];
+
+ let chunkIndex = 0;
+ for (let i = 0; i < tokens.length; i += chunkSize - overlap) {
+ const chunkTokens = tokens.slice(i, i + chunkSize);
+ const chunkContent = this.encoder.decode(chunkTokens);
+
+ chunks.push({
+ tokens: chunkTokens.length,
+ content: chunkContent,
+ full_doc_id: '', // Set by caller
+ chunk_order_index: chunkIndex++,
+ });
+ }
+
+ return chunks;
+ }
+
+ dispose() {
+ this.encoder.free();
+ }
+}
+```
+
+### Entity Extraction
+
+```typescript
+import { LLMProvider } from '@/llm/base/LLMProvider';
+import { EXTRACTION_PROMPT } from '@/prompts/extraction';
+
+export async function extractEntities(
+ chunk: string,
+ llmProvider: LLMProvider
+): Promise<{ entities: EntityData[]; relationships: RelationshipData[] }> {
+ const prompt = EXTRACTION_PROMPT
+ .replace('{input_text}', chunk)
+ .replace('{entity_types}', DEFAULT_ENTITY_TYPES.join(', '))
+ .replace('{language}', 'English');
+
+ const response = await llmProvider.chat([
+ { role: 'system', content: prompt },
+ { role: 'user', content: 'Extract entities and relationships.' },
+ ]);
+
+ return parseExtractionResponse(response);
+}
+
+function parseExtractionResponse(response: string): {
+ entities: EntityData[];
+ relationships: RelationshipData[];
+} {
+ const entities: EntityData[] = [];
+ const relationships: RelationshipData[] = [];
+
+ const lines = response.split('\n');
+
+ for (const line of lines) {
+ if (line.startsWith('entity<|#|>')) {
+ const parts = line.split('<|#|>');
+ if (parts.length >= 4) {
+ entities.push({
+ entity_name: parts[1].trim(),
+ entity_type: parts[2].trim(),
+ description: parts[3].trim(),
+ source_id: '',
+ file_path: '',
+ created_at: new Date().toISOString(),
+ updated_at: new Date().toISOString(),
+ });
+ }
+ } else if (line.startsWith('relation<|#|>')) {
+ const parts = line.split('<|#|>');
+ if (parts.length >= 5) {
+ relationships.push({
+ src_id: parts[1].trim(),
+ tgt_id: parts[2].trim(),
+ keywords: parts[3].trim(),
+ description: parts[4].trim(),
+ weight: 1.0,
+ source_id: '',
+ file_path: '',
+ created_at: new Date().toISOString(),
+ updated_at: new Date().toISOString(),
+ });
+ }
+ }
+ }
+
+ return { entities, relationships };
+}
+```
+
+## Error Handling and Resilience
+
+### Custom Error Classes
+
+```typescript
+export class LightRAGError extends Error {
+ constructor(message: string, public cause?: unknown) {
+ super(message);
+ this.name = 'LightRAGError';
+ }
+}
+
+export class StorageError extends LightRAGError {
+ constructor(message: string, cause?: unknown) {
+ super(message, cause);
+ this.name = 'StorageError';
+ }
+}
+
+export class LLMError extends LightRAGError {
+ constructor(message: string, cause?: unknown) {
+ super(message, cause);
+ this.name = 'LLMError';
+ }
+}
+
+export class ValidationError extends LightRAGError {
+ constructor(message: string, cause?: unknown) {
+ super(message, cause);
+ this.name = 'ValidationError';
+ }
+}
+```
+
+### Circuit Breaker Pattern
+
+```typescript
+import { CircuitBreaker } from './circuitBreaker';
+
+export class ResilientLLMProvider {
+ private provider: LLMProvider;
+ private breaker: CircuitBreaker;
+
+ constructor(provider: LLMProvider) {
+ this.provider = provider;
+ this.breaker = new CircuitBreaker({
+ failureThreshold: 5,
+ resetTimeout: 60000, // 1 minute
+ });
+ }
+
+ async chat(
+ messages: ConversationMessage[],
+ options?: LLMOptions
+ ): Promise {
+ return this.breaker.execute(() =>
+ this.provider.chat(messages, options)
+ );
+ }
+}
+
+class CircuitBreaker {
+ private state: 'CLOSED' | 'OPEN' | 'HALF_OPEN' = 'CLOSED';
+ private failureCount = 0;
+ private lastFailureTime?: number;
+
+ constructor(private config: {
+ failureThreshold: number;
+ resetTimeout: number;
+ }) {}
+
+ async execute(fn: () => Promise): Promise {
+ if (this.state === 'OPEN') {
+ const now = Date.now();
+ if (now - this.lastFailureTime! > this.config.resetTimeout) {
+ this.state = 'HALF_OPEN';
+ } else {
+ throw new Error('Circuit breaker is OPEN');
+ }
+ }
+
+ try {
+ const result = await fn();
+ this.onSuccess();
+ return result;
+ } catch (error) {
+ this.onFailure();
+ throw error;
+ }
+ }
+
+ private onSuccess() {
+ this.failureCount = 0;
+ this.state = 'CLOSED';
+ }
+
+ private onFailure() {
+ this.failureCount++;
+ this.lastFailureTime = Date.now();
+
+ if (this.failureCount >= this.config.failureThreshold) {
+ this.state = 'OPEN';
+ }
+ }
+}
+```
+
+## Performance Optimization
+
+### Connection Pooling
+
+```typescript
+export class ConnectionPoolManager {
+ private pools = new Map();
+
+ getPool(config: DatabaseConfig): Pool {
+ const key = `${config.host}:${config.port}/${config.database}`;
+
+ if (!this.pools.has(key)) {
+ this.pools.set(key, new Pool({
+ ...config,
+ max: 20,
+ idleTimeoutMillis: 30000,
+ connectionTimeoutMillis: 2000,
+ }));
+ }
+
+ return this.pools.get(key)!;
+ }
+
+ async closeAll(): Promise {
+ await Promise.all(
+ Array.from(this.pools.values()).map(pool => pool.end())
+ );
+ this.pools.clear();
+ }
+}
+```
+
+### Batch Processing with Concurrency Control
+
+```typescript
+import pLimit from 'p-limit';
+
+export async function processBatch(
+ items: T[],
+ processor: (item: T) => Promise,
+ concurrency: number = 4
+): Promise {
+ const limit = pLimit(concurrency);
+
+ return Promise.all(
+ items.map(item => limit(() => processor(item)))
+ );
+}
+
+// Usage in document processing
+async function processDocuments(documents: string[]): Promise {
+ return processBatch(
+ documents,
+ async (doc) => {
+ const docId = computeMD5(doc);
+ await extractAndIndex(doc, docId);
+ return docId;
+ },
+ 4 // Max 4 concurrent document processing
+ );
+}
+```
+
+## Testing Patterns
+
+### Mock Storage for Testing
+
+```typescript
+export class MockKVStorage extends BaseKVStorage {
+ private data = new Map>();
+
+ async get_by_id(id: string): Promise | null> {
+ return this.data.get(id) || null;
+ }
+
+ async upsert(data: Record>): Promise {
+ for (const [id, value] of Object.entries(data)) {
+ this.data.set(id, value);
+ }
+ }
+
+ async delete(ids: string[]): Promise {
+ ids.forEach(id => this.data.delete(id));
+ }
+
+ clear() {
+ this.data.clear();
+ }
+}
+```
+
+### Integration Test Helper
+
+```typescript
+export class TestHelper {
+ static async setupTestEnvironment(): Promise {
+ const config: LightRAGConfig = {
+ working_dir: './test-storage',
+ workspace: `test-${Date.now()}`,
+ kv_storage: 'MockKVStorage',
+ vector_storage: 'MockVectorStorage',
+ graph_storage: 'MockGraphStorage',
+ llm_provider: 'mock',
+ };
+
+ const rag = new LightRAG(config);
+ await rag.initialize();
+ return rag;
+ }
+
+ static async cleanupTestEnvironment(rag: LightRAG): Promise {
+ await rag.drop();
+ await rag.close();
+ }
+}
+
+// Usage
+describe('Document Processing', () => {
+ let rag: LightRAG;
+
+ beforeEach(async () => {
+ rag = await TestHelper.setupTestEnvironment();
+ });
+
+ afterEach(async () => {
+ await TestHelper.cleanupTestEnvironment(rag);
+ });
+
+ it('should process document', async () => {
+ const docId = await rag.insert('Test document');
+ expect(docId).toBeDefined();
+ });
+});
+```
+
+## Summary
+
+This implementation guide provides:
+
+1. **Complete storage implementations** with PostgreSQL, pgvector, and graphology
+2. **LLM integration patterns** with retry logic and circuit breakers
+3. **Document processing examples** for chunking and extraction
+4. **Error handling patterns** with custom error classes
+5. **Performance optimization** techniques for production
+6. **Testing patterns** for unit and integration tests
+
+These patterns form the foundation for a robust, production-ready TypeScript implementation of LightRAG. Each pattern can be adapted and extended based on specific requirements while maintaining the core architectural principles.
diff --git a/reverse_documentation/scratchpad.md b/reverse_documentation/scratchpad.md
new file mode 100644
index 00000000..574f14a8
--- /dev/null
+++ b/reverse_documentation/scratchpad.md
@@ -0,0 +1,84 @@
+# LightRAG Analysis Scratchpad
+
+## Initial Observations (Phase 1 - Repository Structure)
+
+### Core Files Analysis
+- **lightrag.py** (141KB): Main class implementation - massive file with ~2200 lines
+- **operate.py** (164KB): Core operations for entity extraction, chunking, querying - ~1782 lines
+- **utils.py** (106KB): Utility functions - extensive helper library
+- **base.py** (30KB): Base classes and interfaces for storage abstractions
+- **prompt.py** (28KB): LLM prompt templates
+- **utils_graph.py** (43KB): Graph utility functions
+
+### Architecture Components Identified
+
+#### 1. Storage Layer (lightrag/kg/)
+Storage implementations (15+ files):
+- **JSON-based**: json_kv_impl.py, json_doc_status_impl.py
+- **Vector DBs**: nano_vector_db_impl.py, faiss_impl.py, milvus_impl.py, qdrant_impl.py
+- **Graph DBs**: networkx_impl.py, neo4j_impl.py (79KB), memgraph_impl.py (49KB)
+- **SQL**: postgres_impl.py (200KB - largest storage impl!)
+- **NoSQL**: mongo_impl.py (95KB), redis_impl.py (46KB)
+- **Shared**: shared_storage.py (48KB) - centralized storage management
+
+Key insight: PostgreSQL implementation is massive - suggests it's a comprehensive reference implementation
+
+#### 2. LLM Integration Layer (lightrag/llm/)
+LLM providers (14 files):
+- openai.py (24KB - most comprehensive)
+- binding_options.py (27KB) - configuration management
+- ollama.py, azure_openai.py, anthropic.py, bedrock.py
+- hf.py, llama_index_impl.py, lmdeploy.py
+- jina.py (embedding), zhipu.py, nvidia_openai.py, siliconcloud.py, lollms.py
+
+#### 3. API Layer (lightrag/api/)
+REST API implementation:
+- lightrag_server.py - FastAPI server
+- routers/ - modular route handlers
+ - query_routes.py - query endpoints
+ - document_routes.py - document management
+ - graph_routes.py - graph visualization
+ - ollama_api.py - Ollama compatibility layer
+
+#### 4. WebUI (lightrag_webui/)
+TypeScript/React frontend - already exists! This provides reference for:
+- TypeScript type definitions (lightrag.ts)
+- API client patterns
+- Data models used by frontend
+
+## Priority Order for Documentation
+1. Executive Summary + Architecture Overview
+2. Core Data Models (base.py, types.py)
+3. Storage Layer Architecture
+4. LLM Integration Patterns
+5. Query Pipeline
+6. Indexing Pipeline
+7. Dependency Migration Guide
+8. TypeScript Implementation Roadmap
+
+## Documentation Progress - Update
+
+### Completed Documents (4/8):
+1. ✅ Executive Summary (16KB) - Complete system overview
+2. ✅ Architecture Documentation (33KB) - 6 comprehensive Mermaid diagrams
+3. ✅ Data Models and Schemas (27KB) - Complete type system
+4. ✅ Dependency Migration Guide (27KB) - Full npm mapping with complexity assessment
+
+### Next Priority Documents:
+5. Storage Layer Implementation Guide - Deep dive into each storage backend
+6. TypeScript Project Structure and Migration Roadmap
+7. LLM Integration Patterns
+8. API Reference with TypeScript Types
+
+### Key Insights for Remaining Docs:
+- Focus on practical implementation examples
+- Include performance considerations
+- Document error handling patterns
+- Provide testing strategies
+- Add deployment configurations
+
+### Total Documentation So Far:
+- ~103KB of technical documentation
+- 6 Mermaid architecture diagrams
+- 50+ code comparison examples
+- Complete dependency mapping for 40+ packages