066d6d3754
Implements comprehensive database connector with advanced features for production-grade data synchronization and vectorization. Core Features (1378 lines - database_connector.py): - Connection pooling with thread-safe management - Secure credential encryption using Fernet - Query result caching with LRU eviction - Rate limiting with token bucket algorithm - SQL injection prevention and validation - Comprehensive error handling and retry logic - Batch processing with memory management - Incremental sync with timestamp tracking - Real-time metrics and monitoring - Health checks and diagnostics Security: - Encrypted credential storage at rest - SSL/TLS connection support - SQL injection pattern detection - Parameterized query enforcement - Secure password handling Performance: - Connection pool (5-20 connections) - Query result caching (LRU, configurable TTL) - Rate limiting (100 calls/min default) - Batch processing (1000 rows/batch) - Query timeout management - Automatic retry with exponential backoff UI Configuration (693 lines - database_config_ui.py): - Complete UI schema for frontend integration - Field validation and conditional rendering - Example configurations for common use cases - Connection testing utilities - Schema discovery from SQL queries - Sample data preview - Row count estimation Supported Databases: - MySQL 5.7+ - MariaDB 10.2+ - PostgreSQL 10+ Configuration Options: - Batch vs Incremental sync modes - Field mapping (vectorization vs metadata) - Custom field transformations - Validation rules - SSL/TLS settings - Performance tuning (pool size, timeouts, cache) - Rate limiting configuration Use Cases: - Product catalogs - Customer support tickets - Internal documentation - FAQ databases - Real-time data feeds - Scheduled batch imports Dependencies: - mysql-connector-python (MySQL/MariaDB) - psycopg2 (PostgreSQL) - cryptography (encryption) Test Coverage: - Unit tests for all major components - Configuration validation - Document conversion - Field transformation - Error handling Fixes #11560 |
||
|---|---|---|
| .. | ||
| testcases | ||
| unit_test | ||