hsparks.codes
066d6d3754
feat: Enterprise-grade MySQL/PostgreSQL database connector (2071 lines)
Implements comprehensive database connector with advanced features for
production-grade data synchronization and vectorization.
Core Features (1378 lines - database_connector.py):
- Connection pooling with thread-safe management
- Secure credential encryption using Fernet
- Query result caching with LRU eviction
- Rate limiting with token bucket algorithm
- SQL injection prevention and validation
- Comprehensive error handling and retry logic
- Batch processing with memory management
- Incremental sync with timestamp tracking
- Real-time metrics and monitoring
- Health checks and diagnostics
Security:
- Encrypted credential storage at rest
- SSL/TLS connection support
- SQL injection pattern detection
- Parameterized query enforcement
- Secure password handling
Performance:
- Connection pool (5-20 connections)
- Query result caching (LRU, configurable TTL)
- Rate limiting (100 calls/min default)
- Batch processing (1000 rows/batch)
- Query timeout management
- Automatic retry with exponential backoff
UI Configuration (693 lines - database_config_ui.py):
- Complete UI schema for frontend integration
- Field validation and conditional rendering
- Example configurations for common use cases
- Connection testing utilities
- Schema discovery from SQL queries
- Sample data preview
- Row count estimation
Supported Databases:
- MySQL 5.7+
- MariaDB 10.2+
- PostgreSQL 10+
Configuration Options:
- Batch vs Incremental sync modes
- Field mapping (vectorization vs metadata)
- Custom field transformations
- Validation rules
- SSL/TLS settings
- Performance tuning (pool size, timeouts, cache)
- Rate limiting configuration
Use Cases:
- Product catalogs
- Customer support tickets
- Internal documentation
- FAQ databases
- Real-time data feeds
- Scheduled batch imports
Dependencies:
- mysql-connector-python (MySQL/MariaDB)
- psycopg2 (PostgreSQL)
- cryptography (encryption)
Test Coverage:
- Unit tests for all major components
- Configuration validation
- Document conversion
- Field transformation
- Error handling
Fixes #11560