Add project intelligence files for AI agent collaboration
- Add .clinerules with technical patterns
- Create Agments.md for Codex agent guidance
- Ensures consistent behavior across all team members
(cherry picked from commit 577b9e6882)
This commit is contained in:
yangdx
Raphaël MANSUY
parent
e0e228673c
commit
06ceae92ab
3 changed files with 315 additions and 1 deletions
207
.clinerules/01-basic.md
Normal file
207
.clinerules/01-basic.md
Normal file
|
|
@ -0,0 +1,207 @@
|
||||||
|
# LightRAG Project Intelligence (.clinerules)
|
||||||
|
|
||||||
|
## Project Overview
|
||||||
|
LightRAG is a mature, production-ready Retrieval-Augmented Generation (RAG) system with comprehensive knowledge graph capabilities. The system has evolved from experimental to production-ready status with extensive functionality across all major components.
|
||||||
|
|
||||||
|
## Current System State (August 15, 2025)
|
||||||
|
- **Status**: Production Ready - Stable and Mature
|
||||||
|
- **Configuration**: Gemini 2.5 Flash + BAAI/bge-m3 embeddings via custom endpoints
|
||||||
|
- **Storage**: Default in-memory with file persistence (JsonKVStorage, NetworkXStorage, NanoVectorDBStorage)
|
||||||
|
- **Language**: Chinese for summaries
|
||||||
|
- **Workspace**: `space1` for data isolation
|
||||||
|
- **Authentication**: JWT-based with admin/user accounts
|
||||||
|
|
||||||
|
## Critical Implementation Patterns
|
||||||
|
|
||||||
|
### 1. Embedding Format Compatibility (CRITICAL)
|
||||||
|
**Pattern**: Always handle both base64 and raw array embedding formats
|
||||||
|
**Location**: `lightrag/llm/openai.py` - `openai_embed` function
|
||||||
|
**Issue**: Custom OpenAI-compatible endpoints return embeddings as raw arrays, not base64 strings
|
||||||
|
**Solution**:
|
||||||
|
```python
|
||||||
|
np.array(dp.embedding, dtype=np.float32) if isinstance(dp.embedding, list)
|
||||||
|
else np.frombuffer(base64.b64decode(dp.embedding), dtype=np.float32)
|
||||||
|
```
|
||||||
|
**Impact**: Document processing fails completely without this dual format support
|
||||||
|
|
||||||
|
### 2. Async Pattern Consistency (CRITICAL)
|
||||||
|
**Pattern**: Always await coroutines before calling methods on the result
|
||||||
|
**Common Error**: `coroutine.method()` instead of `(await coroutine).method()`
|
||||||
|
**Locations**: MongoDB implementations, Neo4j operations
|
||||||
|
**Example**: `await self._data.list_indexes()` then `await cursor.to_list()`
|
||||||
|
|
||||||
|
### 3. Storage Layer Data Compatibility (CRITICAL)
|
||||||
|
**Pattern**: Always filter deprecated/incompatible fields during deserialization
|
||||||
|
**Common Fields to Remove**: `content`, `_id` (MongoDB), database-specific fields
|
||||||
|
**Implementation**: `data.pop('field_name', None)` before creating dataclass objects
|
||||||
|
**Locations**: All storage implementations (JSON, Redis, MongoDB, PostgreSQL)
|
||||||
|
|
||||||
|
### 4. Lock Key Generation (CRITICAL)
|
||||||
|
**Pattern**: Always sort relationship pairs for consistent lock keys
|
||||||
|
**Implementation**: `sorted_key_parts = sorted([src, tgt])` then `f"{sorted_key_parts[0]}-{sorted_key_parts[1]}"`
|
||||||
|
**Impact**: Prevents deadlocks in concurrent relationship processing
|
||||||
|
|
||||||
|
### 5. Event Loop Management (CRITICAL)
|
||||||
|
**Pattern**: Handle event loop mismatches during shutdown gracefully
|
||||||
|
**Implementation**: Timeout + specific RuntimeError handling for "attached to a different loop"
|
||||||
|
**Location**: Neo4j storage finalization
|
||||||
|
**Impact**: Prevents application shutdown failures
|
||||||
|
|
||||||
|
## Architecture Patterns
|
||||||
|
|
||||||
|
### 1. Dependency Injection
|
||||||
|
**Pattern**: Pass configuration through object constructors, not direct imports
|
||||||
|
**Example**: OllamaAPI receives configuration through LightRAG object
|
||||||
|
**Benefit**: Better testability and modularity
|
||||||
|
|
||||||
|
### 2. Memory Bank Documentation
|
||||||
|
**Pattern**: Maintain comprehensive memory bank for development continuity
|
||||||
|
**Structure**: Core files (projectbrief.md, activeContext.md, progress.md, etc.)
|
||||||
|
**Purpose**: Essential for context preservation across development sessions
|
||||||
|
|
||||||
|
### 3. Configuration Management
|
||||||
|
**Pattern**: Centralize defaults in constants.py, use environment variables for runtime config
|
||||||
|
**Implementation**: Default values in constants, override via .env file
|
||||||
|
**Benefit**: Consistent configuration across components
|
||||||
|
|
||||||
|
## Development Workflow Patterns
|
||||||
|
|
||||||
|
### 1. Frontend Development (CRITICAL)
|
||||||
|
**Package Manager**: **ALWAYS USE BUN** - Never use npm or yarn unless Bun is unavailable
|
||||||
|
**Commands**:
|
||||||
|
- `bun install` - Install dependencies
|
||||||
|
- `bun run dev` - Start development server
|
||||||
|
- `bun run build` - Build for production
|
||||||
|
- `bun run lint` - Run linting
|
||||||
|
- `bun test` - Run tests
|
||||||
|
- `bun run preview` - Preview production build
|
||||||
|
|
||||||
|
**Pattern**: All frontend operations must use Bun commands
|
||||||
|
**Fallback**: Only use npm/yarn if Bun installation fails
|
||||||
|
**Testing**: Use `bun test` for all frontend testing
|
||||||
|
|
||||||
|
### 2. Bug Fix Approach
|
||||||
|
1. **Identify root cause** - Don't just fix symptoms
|
||||||
|
2. **Implement robust solution** - Handle edge cases and format variations
|
||||||
|
3. **Maintain backward compatibility** - Preserve existing functionality
|
||||||
|
4. **Add comprehensive error handling** - Graceful degradation
|
||||||
|
5. **Document the fix** - Update memory bank with technical details
|
||||||
|
|
||||||
|
### 3. Feature Implementation
|
||||||
|
1. **Follow existing patterns** - Maintain architectural consistency
|
||||||
|
2. **Use dependency injection** - Avoid direct imports between modules
|
||||||
|
3. **Implement comprehensive error handling** - Handle all failure modes
|
||||||
|
4. **Add proper logging** - Debug and warning messages
|
||||||
|
5. **Update documentation** - Memory bank and code comments
|
||||||
|
6. **Comment Language** - Use English for comments and documentation
|
||||||
|
|
||||||
|
### 4. Performance Optimization
|
||||||
|
1. **Profile before optimizing** - Identify actual bottlenecks
|
||||||
|
2. **Maintain algorithmic correctness** - Don't sacrifice functionality for speed
|
||||||
|
3. **Use appropriate data structures** - Match structure to access patterns
|
||||||
|
4. **Implement caching strategically** - Cache expensive operations
|
||||||
|
5. **Monitor memory usage** - Prevent memory leaks
|
||||||
|
|
||||||
|
## Technology Stack Intelligence
|
||||||
|
|
||||||
|
### 1. LLM Integration
|
||||||
|
- **Primary**: Gemini 2.5 Flash via custom endpoint
|
||||||
|
- **Embedding**: BAAI/bge-m3 via custom endpoint
|
||||||
|
- **Reranking**: BAAI/bge-reranker-v2-m3
|
||||||
|
- **Pattern**: Always handle multiple provider formats
|
||||||
|
|
||||||
|
### 2. Storage Backends
|
||||||
|
- **Default**: In-memory with file persistence
|
||||||
|
- **Production Options**: PostgreSQL, MongoDB, Redis, Neo4j
|
||||||
|
- **Pattern**: Abstract storage interface with multiple implementations
|
||||||
|
|
||||||
|
### 3. API Architecture
|
||||||
|
- **Framework**: FastAPI with Gunicorn for production
|
||||||
|
- **Authentication**: JWT-based with role support
|
||||||
|
- **Compatibility**: Ollama-compatible endpoints for easy integration
|
||||||
|
|
||||||
|
### 4. Frontend
|
||||||
|
- **Framework**: React with TypeScript
|
||||||
|
- **Package Manager**: **BUN (REQUIRED)** - Always use Bun for all frontend operations
|
||||||
|
- **Build Tool**: Vite with Bun runtime
|
||||||
|
- **Visualization**: Sigma.js for graph rendering
|
||||||
|
- **State Management**: React hooks with context
|
||||||
|
- **Internationalization**: i18next for multi-language support
|
||||||
|
|
||||||
|
## Common Pitfalls and Solutions
|
||||||
|
|
||||||
|
### 1. Embedding Format Issues
|
||||||
|
**Pitfall**: Assuming all endpoints return base64-encoded embeddings
|
||||||
|
**Solution**: Always check format and handle both base64 and raw arrays
|
||||||
|
|
||||||
|
### 2. Async/Await Patterns
|
||||||
|
**Pitfall**: Calling methods on coroutines instead of awaited results
|
||||||
|
**Solution**: Always await coroutines before accessing their methods
|
||||||
|
|
||||||
|
### 3. Data Model Evolution
|
||||||
|
**Pitfall**: Breaking changes when removing fields from dataclasses
|
||||||
|
**Solution**: Filter deprecated fields during deserialization, don't break storage
|
||||||
|
|
||||||
|
### 4. Concurrency Issues
|
||||||
|
**Pitfall**: Inconsistent lock key generation causing deadlocks
|
||||||
|
**Solution**: Always sort keys for deterministic lock ordering
|
||||||
|
|
||||||
|
### 5. Event Loop Management
|
||||||
|
**Pitfall**: Event loop mismatches during shutdown
|
||||||
|
**Solution**: Implement timeout and specific error handling for loop issues
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
### 1. Query Context Building
|
||||||
|
- **Algorithm**: Linear gradient weighted polling for fair resource allocation
|
||||||
|
- **Optimization**: Round-robin merging to eliminate mode bias
|
||||||
|
- **Pattern**: Smart chunk selection based on cross-entity occurrence
|
||||||
|
|
||||||
|
### 2. Graph Operations
|
||||||
|
- **Optimization**: Batch operations where possible
|
||||||
|
- **Pattern**: Use appropriate indexing for large datasets
|
||||||
|
- **Consideration**: Memory usage with large graphs
|
||||||
|
|
||||||
|
### 3. LLM Request Management
|
||||||
|
- **Pattern**: Priority-based queue for request ordering
|
||||||
|
- **Optimization**: Connection pooling and retry mechanisms
|
||||||
|
- **Consideration**: Rate limiting and cost management
|
||||||
|
|
||||||
|
## Security Patterns
|
||||||
|
|
||||||
|
### 1. Authentication
|
||||||
|
- **Implementation**: JWT tokens with role-based access
|
||||||
|
- **Pattern**: Stateless authentication with configurable expiration
|
||||||
|
- **Security**: Proper token validation and refresh mechanisms
|
||||||
|
|
||||||
|
### 2. API Security
|
||||||
|
- **Pattern**: Input validation and sanitization
|
||||||
|
- **Implementation**: FastAPI dependency injection for auth
|
||||||
|
- **Consideration**: Rate limiting and abuse prevention
|
||||||
|
|
||||||
|
## Maintenance Guidelines
|
||||||
|
|
||||||
|
### 1. Memory Bank Updates
|
||||||
|
- **Trigger**: After significant changes or bug fixes
|
||||||
|
- **Pattern**: Update activeContext.md and progress.md
|
||||||
|
- **Purpose**: Maintain development continuity
|
||||||
|
|
||||||
|
### 2. Configuration Management
|
||||||
|
- **Pattern**: Environment-based configuration with sensible defaults
|
||||||
|
- **Implementation**: .env files with example templates
|
||||||
|
- **Consideration**: Security for production deployments
|
||||||
|
|
||||||
|
### 3. Error Handling
|
||||||
|
- **Pattern**: Comprehensive logging with appropriate levels
|
||||||
|
- **Implementation**: Graceful degradation where possible
|
||||||
|
- **Consideration**: User-friendly error messages
|
||||||
|
|
||||||
|
## Project Evolution Notes
|
||||||
|
|
||||||
|
The project has evolved from experimental to production-ready status. Key milestones:
|
||||||
|
- **Early 2025**: Basic RAG implementation
|
||||||
|
- **Mid 2025**: Multiple storage backends and LLM providers
|
||||||
|
- **July 2025**: Major query optimization and algorithm improvements
|
||||||
|
- **August 2025**: Production-ready stable state
|
||||||
|
|
||||||
|
The system now supports enterprise-level deployments with comprehensive functionality across all components.
|
||||||
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -106,4 +106,3 @@ gui/
|
||||||
# Cline files
|
# Cline files
|
||||||
memory-bank
|
memory-bank
|
||||||
memory-bank/
|
memory-bank/
|
||||||
.clinerules
|
|
||||||
|
|
|
||||||
108
Agments.md
Normal file
108
Agments.md
Normal file
|
|
@ -0,0 +1,108 @@
|
||||||
|
# Project Guide for AI Agents
|
||||||
|
|
||||||
|
This Agments.md file provides operational guidance for AI assistants collaborating on the LightRAG codebase. Use it to understand the repository layout, preferred tooling, and expectations for adding or modifying functionality.
|
||||||
|
|
||||||
|
## Core Purpose
|
||||||
|
|
||||||
|
LightRAG is an advanced Retrieval-Augmented Generation (RAG) framework designed to enhance information retrieval and generation through graph-based knowledge representation. The project aims to provide a more intelligent and efficient way to process and retrieve information from documents by leveraging both graph structures and vector embeddings.
|
||||||
|
|
||||||
|
## Project Structure for Navigation
|
||||||
|
|
||||||
|
- `/lightrag`: Core Python package (ingestion, querying, storage abstractions, utilities). Key modules include `lightrag/lightrag.py` orchestration, `operate.py` pipeline helpers, `kg/` storage backends, `llm/` bindings, and `utils*.py`.
|
||||||
|
- `/lightrag/api`: FastAPI with Gunicorn for production. FastAPI service for LightRAG , auth, WebUI assets live in `lightrag_server.py`. Routers live in `routers/`, shared helpers in `utils_api.py`. Gunicorn startup logic lives in `run_with_gunicorn.py`.
|
||||||
|
- `/lightrag_webui`: React 19 + TypeScript + Tailwind front-end built with Vite/Bun. Uses component folders under `src/` and configuration via `env.*.sample`.
|
||||||
|
- `/inputs`, `/rag_storage`, `/dickens`, `/temp`: data directories. Treat contents as mutable working data; avoid committing generated artefacts.
|
||||||
|
- `/tests` and root-level `test_*.py`: Integration and smoke-test scripts (graph storage, API endpoints, behaviour regressions). Many expect specific environment variables or services.
|
||||||
|
- `/docs`, `/k8s-deploy`, `docker-compose.yml`: Deployment notes, Kubernetes manifests, and container orchestration helpers.
|
||||||
|
- Configuration templates: `.env.example`, `config.ini.example`, `lightrag.service.example`. Copy and adapt for local runs without committing secrets.
|
||||||
|
|
||||||
|
## Environment Setup and Tooling
|
||||||
|
|
||||||
|
- Python 3.10 is required. Recommended bootstrap:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Development installation
|
||||||
|
python -m venv .venv
|
||||||
|
source .venv/bin/activate
|
||||||
|
pip install -e .
|
||||||
|
pip install -e .[api]
|
||||||
|
|
||||||
|
# Start API server
|
||||||
|
lightrag-server
|
||||||
|
|
||||||
|
# Production deployment
|
||||||
|
lightrag-gunicorn --workers 3
|
||||||
|
```
|
||||||
|
|
||||||
|
- Duplicate `.env.example` to `.env` and adjust storage, LLM, and reranker bindings. Mirror `config.ini.example` when customising pipeline defaults.
|
||||||
|
- Storage backends (PostgreSQL, Redis, Neo4j, Milvus, etc.) are selected via `LIGHTRAG_*` environment variables. Ensure connection URLs and credentials are in place before running ingestion or tests.
|
||||||
|
- CLI entry points: `python -m lightrag` for package usage, `lightrag-server` (or `uvicorn lightrag.api.lightrag_server:app --reload`) for the API, `lightrag-gunicorn` for production gunicorn runs.
|
||||||
|
- Front-end work: install dependencies with `bun install` (preferred) or `npm install`, then use `bunx --bun vite` commands defined in `package.json`.
|
||||||
|
|
||||||
|
## Frontend Development
|
||||||
|
|
||||||
|
* **Package Manager**: **ALWAYS USE BUN** - Never use npm or yarn unless Bun is unavailable
|
||||||
|
**Commands**:
|
||||||
|
|
||||||
|
- `bun install` - Install dependencies
|
||||||
|
|
||||||
|
- `bun run dev` - Start development server
|
||||||
|
|
||||||
|
- `bun run build` - Build for production
|
||||||
|
|
||||||
|
- `bun run lint` - Run linting
|
||||||
|
|
||||||
|
- `bun test` - Run tests
|
||||||
|
|
||||||
|
- `bun run preview` - Preview production build
|
||||||
|
|
||||||
|
|
||||||
|
* **Pattern**: All frontend operations must use Bun commands
|
||||||
|
* **Testing**: Use `bun test` for all frontend testing
|
||||||
|
|
||||||
|
## Coding Conventions
|
||||||
|
|
||||||
|
- Embrace type hints, dataclasses, and asynchronous patterns already present in `lightrag/lightrag.py` and storage implementations. Keep long-running jobs within `asyncio` flows and reuse helpers from `lightrag.operate`.
|
||||||
|
- Honour abstraction boundaries: new storage providers should inherit from the relevant base classes in `lightrag.base`; reusable logic belongs in `utils.py`/`utils_graph.py`.
|
||||||
|
- Use `lightrag.utils.logger` (not bare `print`) and let environment toggles (`VERBOSE`, `LOG_LEVEL`) control verbosity.
|
||||||
|
- Respect configuration defaults in `lightrag/constants.py`, extending with care and synchronising related documentation when behaviour changes.
|
||||||
|
- API additions should live under `lightrag/api/routers`, leverage dependency injections from `utils_api.py`, and return structured responses consistent with existing handlers.
|
||||||
|
- Front-end code should remain in TypeScript, rely on functional React components with hooks, and follow Tailwind utility style. Co-locate component-specific styles; reserve custom CSS for cases Tailwind cannot cover.
|
||||||
|
- Storage Backends
|
||||||
|
- **Default**: In-memory with file persistence
|
||||||
|
- **Production Options**: PostgreSQL, MongoDB, Redis, Neo4j
|
||||||
|
- **Pattern**: Abstract storage interface with multiple implementations
|
||||||
|
|
||||||
|
* Lock Key Generation Consistency
|
||||||
|
- **Critical Pattern**: Always sort parameters for lock key generation to prevent deadlocks
|
||||||
|
- **Example**: `sorted_key_parts = sorted([src, tgt])` before creating lock key
|
||||||
|
- **Why**: Prevents different lock keys for same relationship pair processed in different orders
|
||||||
|
- **Apply to**: Any function that uses locks with multiple parameters
|
||||||
|
* Priority Queue Implementation
|
||||||
|
* **Pattern**: Use priority-based task queuing for LLM requests
|
||||||
|
* **Benefits**: Critical operations get higher priority
|
||||||
|
* **Implementation**: Lower priority values = higher priority
|
||||||
|
|
||||||
|
## Testing and Quality Gates
|
||||||
|
|
||||||
|
- Run Python tests with `python -m pytest tests` for the FastAPI suite, and execute targeted scripts (for example `python tests/test_graph_storage.py`, `python test_lock_fix.py`) when touching related functionality. Many scripts require running backing services; check `.env` for prerequisites.
|
||||||
|
- Perform linting via `ruff check .` (configured in `pyproject.toml`) and address warnings. For formatting, match the existing style rather than introducing new tools.
|
||||||
|
- Front-end validation: `bun test`, `bunx --bun vite build`, and `bunx --bun vite lint`. The `*-no-bun` scripts exist if Bun is unavailable.
|
||||||
|
- When touching deployment assets, ensure `docker-compose config` or relevant `kubectl` dry-runs succeed before submitting changes.
|
||||||
|
|
||||||
|
## Runtime and Operational Notes
|
||||||
|
|
||||||
|
- Knowledge ingestion expects documents inside `inputs/` and writes intermediate state to `rag_storage/`. Keep these directories gitignored; never check in private data or large artefacts.
|
||||||
|
- Use `operate.py` helpers (e.g., `chunking_by_token_size`, `extract_entities`) to keep ingestion behaviour consistent. If extending the pipeline, document new steps in `docs/` and update any affected CLI usage.
|
||||||
|
- The API and core package rely on `.env`/`config.ini` being co-located with the current working directory. Scripts such as `tests/test_graph_storage.py` dynamically read these files; ensure they are in sync.
|
||||||
|
|
||||||
|
## Contribution Checklist
|
||||||
|
|
||||||
|
1. Run `pre-commit run --all-files` before sumitting PR.
|
||||||
|
2. Describe the change, affected modules, and operational impact in your PR. Mention any new environment knobs or storage dependencies.
|
||||||
|
3. Link related issues or discussions when available.
|
||||||
|
4. Confirm all applicable checks pass (`ruff`, pytest suite, targeted integration scripts, front-end build/tests when touched).
|
||||||
|
5. Capture screenshots or GIFs for front-end or API changes that affect user-visible behaviour.
|
||||||
|
6. Keep each PR focused on a single concern and update documentation (`README.md`, `docs/`, `.env.example`) when behaviour or configuration changes.
|
||||||
|
|
||||||
|
Follow this playbook to keep LightRAG contributions predictable, testable, and production-ready.
|
||||||
Loading…
Add table
Reference in a new issue