Implements full three-tier retrieval system with RAGFlow integration.
Changes:
- Complete Tier 1: KB routing with rule-based, LLM-based, and auto modes
- Complete Tier 2: Document filtering with metadata support
- Complete Tier 3: Chunk refinement with vector search integration
- Integration with RAGFlow's Dealer and search infrastructure
- Add hierarchical_retrieval_config field to Dialog model
- Database migration for configuration storage
- 29 passing unit tests (6 skipped due to NLTK environment dependency)
Implementation Details:
- HierarchicalRetrieval: Main orchestrator with RAGFlow integration
- KBRouter: Standalone router using keyword matching
- DocumentFilter: Metadata-based filtering
- ChunkRefiner: Vector search integration via rag.nlp.search.Dealer
- Rule-based routing uses token overlap scoring
- Auto routing analyzes query characteristics
- Tier 3 integrates with existing DocStoreConnection and embedding models
Test Results:
✅ 29/29 tests passing
- All tier tests working
- Integration scenarios validated
- Config and result dataclasses tested
- Edge cases handled
Addresses owner feedback: Complete implementation rather than skeleton.
Related to #11610
272534df64