ragflow/rag/utils
Andrea Bugeja 74afb8d710
feat: Add Single Bucket Mode for MinIO/S3 (#11416)
## Overview

This PR adds support for **Single Bucket Mode** in RAGFlow, allowing
users to configure MinIO/S3 to use a single bucket with a directory
structure instead of creating multiple buckets per Knowledge Base and
user folder.

## Problem Statement

The current implementation creates one bucket per Knowledge Base and one
bucket per user folder, which can be problematic when:
- Cloud providers charge per bucket
- IAM policies restrict bucket creation
- Organizations want centralized data management in a single bucket

## Solution

Added a `prefix_path` configuration option to the MinIO connector that
enables:
- Using a single bucket with directory-based organization
- Backward compatibility with existing multi-bucket deployments
- Support for MinIO, AWS S3, and other S3-compatible storage backends

## Changes

- **`rag/utils/minio_conn.py`**: Enhanced MinIO connector to support
single bucket mode with prefix paths
- **`conf/service_conf.yaml`**: Added new configuration options
(`bucket` and `prefix_path`)
- **`docker/service_conf.yaml.template`**: Updated template with single
bucket configuration examples
- **`docker/.env.single-bucket-example`**: Added example environment
variables for single bucket setup
- **`docs/single-bucket-mode.md`**: Comprehensive documentation covering
usage, migration, and troubleshooting

## Configuration Example

```yaml
minio:
  user: "access-key"
  password: "secret-key"
  host: "minio.example.com:443"
  bucket: "ragflow-bucket"      # Single bucket name
  prefix_path: "ragflow"         # Optional prefix path
```

## Backward Compatibility

 Fully backward compatible - existing deployments continue to work
without any changes
- If `bucket` is not configured, uses default multi-bucket behavior
- If `bucket` is configured without `prefix_path`, uses bucket root
- If both are configured, uses `bucket/prefix_path/` structure

## Testing

- Tested with MinIO (local and cloud)
- Verified backward compatibility with existing multi-bucket mode
- Validated IAM policy restrictions work correctly

## Documentation

Included comprehensive documentation in `docs/single-bucket-mode.md`
covering:
- Configuration examples
- Migration guide from multi-bucket to single-bucket mode
- IAM policy examples
- Troubleshooting guide

---

**Related Issue**: Addresses use cases where bucket creation is
restricted or costly
2025-12-11 19:22:47 +08:00
..
__init__.py Move token related functions to common (#10942) 2025-11-03 08:50:05 +08:00
azure_sas_conn.py Move api.settings to common.settings (#11036) 2025-11-06 09:36:38 +08:00
azure_spn_conn.py Refactor function name (#11210) 2025-11-12 19:00:15 +08:00
base64_image.py Refa:replace trio with asyncio (#11831) 2025-12-09 19:23:14 +08:00
doc_store_conn.py Refactor function name (#11210) 2025-11-12 19:00:15 +08:00
es_conn.py Fix: Table parse method issue. (#11627) 2025-12-01 12:42:35 +08:00
file_utils.py Move some funcs from api to rag module (#10972) 2025-11-03 19:26:09 +08:00
gcs_conn.py feat(gcs): Add support for Google Cloud Storage (GCS) integration (#11718) 2025-12-04 10:44:05 +08:00
infinity_conn.py Bump infinity to v0.6.11. Requires python>=3.11 (#11814) 2025-12-09 16:23:37 +08:00
minio_conn.py feat: Add Single Bucket Mode for MinIO/S3 (#11416) 2025-12-11 19:22:47 +08:00
ob_conn.py feat: enhance OBConnection.search (#11876) 2025-12-10 19:13:37 +08:00
opendal_conn.py Fix:async issue and sensitive logging (#11895) 2025-12-11 13:54:47 +08:00
opensearch_conn.py Refactor function name (#11210) 2025-11-12 19:00:15 +08:00
oss_conn.py Refactor function name (#11210) 2025-11-12 19:00:15 +08:00
raptor_utils.py feat: Auto-disable Raptor for structured data (Issue #11653) (#11676) 2025-12-03 17:02:29 +08:00
redis_conn.py Refa:replace trio with asyncio (#11831) 2025-12-09 19:23:14 +08:00
s3_conn.py Refactor function name (#11210) 2025-11-12 19:00:15 +08:00
storage_factory.py Move api.settings to common.settings (#11036) 2025-11-06 09:36:38 +08:00
tavily_conn.py Remove 'get_lan_ip' and add common misc_utils.py (#10880) 2025-10-31 16:42:01 +08:00