gmakstutis/ragflow
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ragflow/common
History
hsparks.codes 066d6d3754 feat: Enterprise-grade MySQL/PostgreSQL database connector (2071 lines) 
Implements comprehensive database connector with advanced features for
production-grade data synchronization and vectorization.

Core Features (1378 lines - database_connector.py):
- Connection pooling with thread-safe management
- Secure credential encryption using Fernet
- Query result caching with LRU eviction
- Rate limiting with token bucket algorithm
- SQL injection prevention and validation
- Comprehensive error handling and retry logic
- Batch processing with memory management
- Incremental sync with timestamp tracking
- Real-time metrics and monitoring
- Health checks and diagnostics

Security:
- Encrypted credential storage at rest
- SSL/TLS connection support
- SQL injection pattern detection
- Parameterized query enforcement
- Secure password handling

Performance:
- Connection pool (5-20 connections)
- Query result caching (LRU, configurable TTL)
- Rate limiting (100 calls/min default)
- Batch processing (1000 rows/batch)
- Query timeout management
- Automatic retry with exponential backoff

UI Configuration (693 lines - database_config_ui.py):
- Complete UI schema for frontend integration
- Field validation and conditional rendering
- Example configurations for common use cases
- Connection testing utilities
- Schema discovery from SQL queries
- Sample data preview
- Row count estimation

Supported Databases:
- MySQL 5.7+
- MariaDB 10.2+
- PostgreSQL 10+

Configuration Options:
- Batch vs Incremental sync modes
- Field mapping (vectorization vs metadata)
- Custom field transformations
- Validation rules
- SSL/TLS settings
- Performance tuning (pool size, timeouts, cache)
- Rate limiting configuration

Use Cases:
- Product catalogs
- Customer support tickets
- Internal documentation
- FAQ databases
- Real-time data feeds
- Scheduled batch imports

Dependencies:
- mysql-connector-python (MySQL/MariaDB)
- psycopg2 (PostgreSQL)
- cryptography (encryption)

Test Coverage:
- Unit tests for all major components
- Configuration validation
- Document conversion
- Field transformation
- Error handling

Fixes #11560
2025-12-03 12:27:24 +01:00
..
data_source feat: Enterprise-grade MySQL/PostgreSQL database connector (2071 lines) 2025-12-03 12:27:24 +01:00
__init__.py Refactor: rename rmSpace to remove_redundant_spaces (#10796) 2025-10-28 09:46:32 +08:00
config_utils.py Move some enumerate type to constants.py (#10998) 2025-11-04 19:25:25 +08:00
connection_utils.py Fix: refine error msg. (#11380) 2025-11-19 19:10:45 +08:00
constants.py fix(files): replace hard coded status codes with constants (#11544) 2025-11-27 09:41:24 +08:00
decorator.py Move singleton to common directory (#10935) 2025-11-02 12:24:08 +08:00
exceptions.py Feat: GraphRAG handle cancel gracefully (#11061) 2025-11-06 16:12:20 +08:00
file_utils.py Refactor file utils (#10970) 2025-11-03 18:54:55 +08:00
float_utils.py Refactor: rename rmSpace to remove_redundant_spaces (#10796) 2025-10-28 09:46:32 +08:00
http_client.py Refa: make RAGFlow more asynchronous (#11601) 2025-12-01 14:24:06 +08:00
log_utils.py Feat: add Jira connector (#11285) 2025-11-17 09:38:04 +08:00
mcp_tool_call_conn.py Refactor: move mcp connection utilities to common (#11304) 2025-11-17 15:34:17 +08:00
misc_utils.py Refactor code (#11694) 2025-12-03 15:15:00 +08:00
settings.py Use Infinity single-field-multi-index (#11444) 2025-11-26 11:06:37 +08:00
signal_utils.py Fix and refactor imports (#11010) 2025-11-05 11:07:54 +08:00
string_utils.py Refactor: rename rmSpace to remove_redundant_spaces (#10796) 2025-10-28 09:46:32 +08:00
time_utils.py Add time utils (#10849) 2025-10-28 19:09:14 +08:00
token_utils.py fix(llm): handle None response in total_token_count_from_response (#10941) 2025-11-20 10:04:03 +08:00
versions.py Admin: add 'show version' (#11079) 2025-11-06 19:24:46 +08:00