gmakstutis/openrag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #34 from langflow-ai/readme-udpate

Browse source 
chore: revamp the readme with detailed setup +tui docs
This commit is contained in:
Sebastián Estévez 2025-09-12 12:28:35 -04:00 • committed by GitHub
commit b492f87e2f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 826 additions and 42 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

289
CONTRIBUTING.md Normal file
View file

@ -0,0 +1,289 @@
# Contributing to OpenRAG
Thank you for your interest in contributing to OpenRAG! This guide will help you set up your development environment and understand the development workflow.
## 🛠️ Development Setup
### Prerequisites
- Docker or Podman with Compose installed
- Make (for development commands)
- Python 3.13+ with uv package manager
- Node.js 18+ and npm
### Environment Setup
```bash
# Clone the repository
git clone <repository-url>
cd openrag
# Setup development environment
make setup # Creates .env and installs dependencies
```
### Configuration
Edit `.env` with your API keys and credentials. See the main README for required environment variables.
## 🔧 Development Commands
All development tasks are managed through the Makefile. Run `make help` to see all available commands.
### Environment Management
```bash
# Setup development environment
make setup # Initial setup: creates .env, installs dependencies
# Start development environments
make dev # Full stack with GPU support
make dev-cpu # Full stack with CPU only
make infra # Infrastructure only (for local development)
# Container management
make stop # Stop all containers
make restart # Restart all containers
make clean # Stop and remove containers/volumes
make status # Show container status
make health # Check service health
```
### Local Development Workflow
For faster development iteration, run infrastructure in Docker and backend/frontend locally:
```bash
# Terminal 1: Start infrastructure
make infra
# Terminal 2: Run backend locally
make backend
# Terminal 3: Run frontend locally
make frontend
```
This setup provides:
- Faster code reloading
- Direct access to logs and debugging
- Easier testing and iteration
### Dependency Management
```bash
make install # Install all dependencies
make install-be # Install backend dependencies (uv)
make install-fe # Install frontend dependencies (npm)
```
### Building and Testing
```bash
# Build Docker images
make build # Build all images
make build-be # Build backend image only
make build-fe # Build frontend image only
# Testing and quality
make test # Run backend tests
make lint # Run linting checks
```
### Debugging
```bash
# View logs
make logs # All container logs
make logs-be # Backend logs only
make logs-fe # Frontend logs only
make logs-lf # Langflow logs only
make logs-os # OpenSearch logs only
# Shell access
make shell-be # Shell into backend container
make shell-lf # Shell into langflow container
make shell-os # Shell into opensearch container
```
### Database Operations
```bash
# Reset OpenSearch indices
make db-reset # Delete and recreate indices
```
### Flow Management
```bash
# Upload flow to Langflow
make flow-upload FLOW_FILE=path/to/flow.json
```
## 🏗️ Architecture Overview
### Backend (Python/Starlette)
- **API Layer**: RESTful endpoints in `src/api/`
- **Services**: Business logic in `src/services/`
- **Models**: Data models and processors in `src/models/`
- **Connectors**: External service integrations in `src/connectors/`
- **Configuration**: Settings management in `src/config/`
### Frontend (Next.js/React)
- **Pages**: Next.js app router in `frontend/src/app/`
- **Components**: Reusable UI components in `frontend/src/components/`
- **Contexts**: State management in `frontend/src/contexts/`
- **Hooks**: Custom React hooks in `frontend/hooks/`
### Infrastructure
- **OpenSearch**: Vector database and search engine
- **Langflow**: Visual flow builder for LLM workflows
- **Docker**: Containerization and orchestration
## 🧪 Testing
### Backend Tests
```bash
make test # Run all backend tests
uv run pytest # Direct pytest execution
uv run pytest -v # Verbose test output
```
### Frontend Tests
```bash
cd frontend && npm test # Run frontend tests
cd frontend && npm run lint # Frontend linting
```
## 📝 Code Style
### Backend
- Follow PEP 8 style guidelines
- Use type hints where appropriate
- Document functions and classes with docstrings
- Use structured logging with `structlog`
### Frontend
- Follow React/Next.js best practices
- Use TypeScript for type safety
- Follow the established component structure
- Use Tailwind CSS for styling
## 🔍 Debugging Tips
### Backend Debugging
```bash
# Enable debug logging
export LOG_LEVEL=DEBUG
# Run backend locally for debugging
make infra && make backend
# Check OpenSearch indices
curl -X GET "http://localhost:9200/_cat/indices?v" \
-u admin:$(grep OPENSEARCH_PASSWORD .env | cut -d= -f2)
```
### Frontend Debugging
```bash
# Run with detailed logs
cd frontend && npm run dev
# Build and analyze bundle
cd frontend && npm run build
```
### Container Debugging
```bash
# Check container status
make status
# View real-time logs
make logs
# Shell into containers
make shell-be # Backend container
make shell-lf # Langflow container
```
## 🚀 Deployment Testing
### Local Testing
```bash
# Test full stack deployment
make clean && make dev
# Test CPU-only deployment
make clean && make dev-cpu
```
### Performance Testing
```bash
# Monitor resource usage
docker stats
# Check service health
make health
```
## 📚 Development Resources
### Key Files
- `src/main.py` - Backend application entry point
- `src/config/settings.py` - Configuration management
- `frontend/src/app/layout.tsx` - Frontend root layout
- `docker-compose.yml` - Container orchestration
- `Makefile` - Development commands
### Documentation
- API documentation: Available at `http://localhost:8000/docs` when backend is running
- Component Storybook: (if implemented) at `http://localhost:6006`
- OpenSearch: `http://localhost:5601` (Dashboards)
- Langflow: `http://localhost:7860`
## 🐛 Common Issues
### Port Conflicts
Ensure these ports are available:
- 3000 (Frontend)
- 7860 (Langflow)
- 8000 (Backend)
- 9200 (OpenSearch)
- 5601 (OpenSearch Dashboards)
### Memory Issues
- Use `make dev-cpu` for CPU-only mode
- Increase Docker memory allocation
- Podman on macOS: increase the VM memory if needed
```bash
podman machine stop
podman machine rm
podman machine init --memory 8192 # 8 GB example
podman machine start
```
### Environment Issues
```bash
# Reset environment
make clean
cp .env.example .env # Reconfigure as needed
make setup
```
## 📋 Pull Request Guidelines
1. **Fork and Branch**: Create a feature branch from `main`
2. **Test**: Ensure all tests pass with `make test` and `make lint`
3. **Documentation**: Update relevant documentation
4. **Commit Messages**: Use clear, descriptive commit messages
5. **PR Description**: Explain changes and include testing instructions
## 🤝 Getting Help
- Check existing issues and discussions
- Use `make status` and `make health` for debugging
- Review logs with `make logs`
- Join our community discussions
Thank you for contributing to OpenRAG! 🚀

184
README.md
View file

@ -1,68 +1,168 @@
## OpenRAG <div align="center">
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/phact/openrag) # OpenRAG
### getting started </div>
<div align="center">
<a href="#quick-start" style="color: #0366d6;">🚀 Quick Start</a> &nbsp;&nbsp;|&nbsp;&nbsp;
<a href="#tui-interface" style="color: #0366d6;">💻 TUI Interface</a> &nbsp;&nbsp;|&nbsp;&nbsp;
<a href="#docker-deployment" style="color: #0366d6;">🐳 Docker Deployment</a> &nbsp;&nbsp;|&nbsp;&nbsp;
<a href="#development" style="color: #0366d6;">⚙️ Development</a> &nbsp;&nbsp;|&nbsp;&nbsp;
<a href="#troubleshooting" style="color: #0366d6;">🔧 Troubleshooting</a>
</div>
Set up your secrets:
cp .env.example .env
Populate the values in .env OpenRAG is a comprehensive Retrieval-Augmented Generation platform that enables intelligent document search and AI-powered conversations. Users can upload, process, and query documents through a chat interface backed by large language models and semantic search capabilities. The system utilizes Langflow for document ingestion, retrieval workflows, and intelligent nudges, providing a seamless RAG experience. Built with Starlette, Next.js, OpenSearch, and Langflow integration. [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/phact/openrag)
Requirements:
Docker or podman with compose installed. <div align="center">
<a href="https://github.com/langflow-ai/langflow"><img src="https://img.shields.io/badge/Langflow-1C1C1E?style=flat&logo=langflow" alt="Langflow"></a>
&nbsp;&nbsp;
<a href="https://github.com/opensearch-project/OpenSearch"><img src="https://img.shields.io/badge/OpenSearch-005EB8?style=flat&logo=opensearch&logoColor=white" alt="OpenSearch"></a>
&nbsp;&nbsp;
<a href="https://github.com/encode/starlette"><img src="https://img.shields.io/badge/Starlette-009639?style=flat&logo=fastapi&logoColor=white" alt="Starlette"></a>
&nbsp;&nbsp;
<a href="https://github.com/vercel/next.js"><img src="https://img.shields.io/badge/Next.js-000000?style=flat&logo=next.js&logoColor=white" alt="Next.js"></a>
Run OpenRAG: </div>
docker compose build
docker compose up
CPU only:
docker compose -f docker-compose-cpu.yml up
If you need to reset state:
docker compose up --build --force-recreate --remove-orphans ## 🚀 Quick Start
### Configuration ### Prerequisites
OpenRAG uses environment variables for configuration. Copy `.env.example` to `.env` and populate with your values: - Docker or Podman with Compose installed
- Make (for development commands)
### 1. Environment Setup
```bash ```bash
cp .env.example .env # Clone and setup environment
git clone https://github.com/langflow-ai/openrag.git
cd openrag
make setup # Creates .env and installs dependencies
``` ```
#### Key Environment Variables ### 2. Configure Environment
**Required:** Edit `.env` with your API keys and credentials:
- `OPENAI_API_KEY`: Your OpenAI API key
- `OPENSEARCH_PASSWORD`: Password for OpenSearch admin user
- `LANGFLOW_SUPERUSER`: Langflow admin username
- `LANGFLOW_SUPERUSER_PASSWORD`: Langflow admin password
- `LANGFLOW_CHAT_FLOW_ID`: ID of your Langflow chat flow
- `LANGFLOW_INGEST_FLOW_ID`: ID of your Langflow ingestion flow
**Ingestion Configuration:** ```bash
- `DISABLE_INGEST_WITH_LANGFLOW`: Disable Langflow ingestion pipeline (default: `false`) # Required
- `false` or unset: Uses Langflow pipeline (upload → ingest → delete) OPENAI_API_KEY=your_openai_api_key
- `true`: Uses traditional OpenRAG processor for document ingestion OPENSEARCH_PASSWORD=your_secure_password
LANGFLOW_SUPERUSER=admin
LANGFLOW_SUPERUSER_PASSWORD=your_secure_password
LANGFLOW_CHAT_FLOW_ID=your_chat_flow_id
LANGFLOW_INGEST_FLOW_ID=your_ingest_flow_id
NUDGES_FLOW_ID=your_nudges_flow_id
```
ee extended configuration, including ingestion and optional variables: [docs/configuration.md](docs/
configuration.md)
### 3. Start OpenRAG
**Optional:** ```bash
- `LANGFLOW_PUBLIC_URL`: Public URL for Langflow (default: `http://localhost:7860`) # Full stack with GPU support
- `GOOGLE_OAUTH_CLIENT_ID` / `GOOGLE_OAUTH_CLIENT_SECRET`: For Google OAuth authentication make dev
- `MICROSOFT_GRAPH_OAUTH_CLIENT_ID` / `MICROSOFT_GRAPH_OAUTH_CLIENT_SECRET`: For Microsoft OAuth
- `WEBHOOK_BASE_URL`: Base URL for webhook endpoints
- `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`: For AWS integrations
See `.env.example` for a complete list with descriptions, or check the docker-compose.yml files. # Or CPU only
make dev-cpu
```
For podman on mac you may have to increase your VM memory (`podman stats` should not show limit at only 2gb): Access the services:
- **Frontend**: http://localhost:3000
- **Backend API**: http://localhost:8000
- **Langflow**: http://localhost:7860
- **OpenSearch**: http://localhost:9200
- **OpenSearch Dashboards**: http://localhost:5601
podman machine stop ## 🖥️ TUI Interface
podman machine rm
podman machine init --memory 8192 # example: 8 GB OpenRAG includes a powerful Terminal User Interface (TUI) for easy setup, configuration, and monitoring. The TUI provides a user-friendly way to manage your OpenRAG installation without complex command-line operations.
podman machine start
![OpenRAG TUI Interface](assets/OpenRAG_TUI_2025-09-10T13_04_11_757637.svg)
### Launching the TUI
```bash
# Install dependencies first
uv sync
# Launch the TUI
uv run openrag
```
### TUI Features
See the full TUI guide for features, navigation, and benefits: [docs/tui.md](docs/tui.md)
## 🐳 Docker Deployment
### Standard Deployment
```bash
# Build and start all services
docker compose build
docker compose up -d
```
### CPU-Only Deployment
For environments without GPU support:
```bash
docker compose -f docker-compose-cpu.yml up -d
```
More deployment commands and tips: [docs/docker.md](docs/docker.md)
## 🔧 Troubleshooting
### Podman on macOS
If using Podman on macOS, you may need to increase VM memory:
```bash
podman machine stop
podman machine rm
podman machine init --memory 8192 # 8 GB example
podman machine start
```
### Common Issues
See common issues and fixes: [docs/troubleshooting.md](docs/troubleshooting.md)
## 🛠️ Development
For developers wanting to contribute to OpenRAG or set up a development environment, please see our comprehensive development guide:
**[📚 See CONTRIBUTING.md for detailed development instructions](CONTRIBUTING.md)**
The contributing guide includes:
- Complete development environment setup
- Local development workflows
- Testing and debugging procedures
- Code style guidelines
- Architecture overview
- Pull request guidelines
### Quick Development Commands
```bash
make help # See all available commands
make setup # Initial development setup
make infra # Start infrastructure services
make backend # Run backend locally
make frontend # Run frontend locally
```

237
assets/OpenRAG_TUI_2025-09-10T13_04_11_757637.svg Normal file
View file

File diff suppressed because one or more lines are too long
Side by side

After

Width:  |  Height:  |  Size: 35 KiB

43
docs/configuration.md Normal file
View file

@ -0,0 +1,43 @@
# Configuration
OpenRAG uses environment variables for configuration. Copy `.env.example` to `.env` and populate with your values:
```bash
cp .env.example .env
```
## Required Variables
| Variable | Description |
|----------|-------------|
| `OPENAI_API_KEY` | Your OpenAI API key |
| `OPENSEARCH_PASSWORD` | Password for OpenSearch admin user |
| `LANGFLOW_SUPERUSER` | Langflow admin username |
| `LANGFLOW_SUPERUSER_PASSWORD` | Langflow admin password |
| `LANGFLOW_CHAT_FLOW_ID` | ID of your Langflow chat flow |
| `LANGFLOW_INGEST_FLOW_ID` | ID of your Langflow ingestion flow |
| `NUDGES_FLOW_ID` | ID of your Langflow nudges/suggestions flow |
## Ingestion Configuration
| Variable | Description |
|----------|-------------|
| `DISABLE_INGEST_WITH_LANGFLOW` | Disable Langflow ingestion pipeline (default: `false`) |
- `false` or unset: Uses Langflow pipeline (upload → ingest → delete)
- `true`: Uses traditional OpenRAG processor for document ingestion
## Optional Variables
| Variable | Description |
|----------|-------------|
| `LANGFLOW_PUBLIC_URL` | Public URL for Langflow (default: `http://localhost:7860`) |
| `GOOGLE_OAUTH_CLIENT_ID` / `GOOGLE_OAUTH_CLIENT_SECRET` | Google OAuth authentication |
| `MICROSOFT_GRAPH_OAUTH_CLIENT_ID` / `MICROSOFT_GRAPH_OAUTH_CLIENT_SECRET` | Microsoft OAuth |
| `WEBHOOK_BASE_URL` | Base URL for webhook endpoints |
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | AWS integrations |
| `SESSION_SECRET` | Session management (default: auto-generated, change in production) |
| `LANGFLOW_KEY` | Explicit Langflow API key (auto-generated if not provided) |
| `LANGFLOW_SECRET_KEY` | Secret key for Langflow internal operations |
See `.env.example` for a complete list with descriptions, and `docker-compose*.yml` for runtime usage.

35
docs/docker.md Normal file
View file

@ -0,0 +1,35 @@
# Docker Deployment
## Standard Deployment
```bash
# Build and start all services
docker compose build
docker compose up -d
```
## CPU-Only Deployment
For environments without GPU support:
```bash
docker compose -f docker-compose-cpu.yml up -d
```
## Force Rebuild
If you need to reset state or rebuild everything:
```bash
docker compose up --build --force-recreate --remove-orphans
```
## Service URLs
After deployment, services are available at:
- Frontend: http://localhost:3000
- Backend API: http://localhost:8000
- Langflow: http://localhost:7860
- OpenSearch: http://localhost:9200
- OpenSearch Dashboards: http://localhost:5601

19
docs/troubleshooting.md Normal file
View file

@ -0,0 +1,19 @@
# Troubleshooting
## Podman on macOS
If using Podman on macOS, you may need to increase VM memory:
```bash
podman machine stop
podman machine rm
podman machine init --memory 8192 # 8 GB example
podman machine start
```
## Common Issues
1. OpenSearch fails to start: Check that `OPENSEARCH_PASSWORD` is set and meets requirements
2. Langflow connection issues: Verify `LANGFLOW_SUPERUSER` credentials are correct
3. Out of memory errors: Increase Docker memory allocation or use CPU-only mode
4. Port conflicts: Ensure ports 3000, 7860, 8000, 9200, 5601 are available

61
docs/tui.md Normal file
View file

@ -0,0 +1,61 @@
# OpenRAG TUI Guide
The OpenRAG Terminal User Interface (TUI) provides a streamlined way to set up, configure, and monitor your OpenRAG deployment directly from the terminal.
![OpenRAG TUI Interface](../assets/OpenRAG_TUI_2025-09-10T13_04_11_757637.svg)
## Launch
```bash
uv sync
uv run openrag
```
## Features
### Welcome Screen
- Quick setup options: basic (no auth) or advanced (OAuth)
- Service monitoring: container status at a glance
- Quick actions: diagnostics, logs, configuration
### Configuration Screen
- Environment variables: guided forms for required settings
- API keys: secure input with validation
- OAuth setup: Google and Microsoft
- Document paths: configure ingestion directories
- Auto-save: generates and updates `.env`
### Service Monitor
- Container status: real-time state of services
- Resource usage: CPU, memory, network
- Service control: start/stop/restart
- Health checks: health indicators for all components
### Log Viewer
- Live logs: stream logs across services
- Filtering: by service (backend, frontend, Langflow, OpenSearch)
- Levels: DEBUG/INFO/WARNING/ERROR
- Export: save logs for later analysis
### Diagnostics
- System checks: Docker/Podman availability and configuration
- Environment validation: verify required variables
- Network tests: connectivity between services
- Performance metrics: system capacity and recommendations
## Navigation
- Arrow keys: move between options
- Tab/Shift+Tab: switch fields and buttons
- Enter: select/confirm
- Escape: back
- Q: quit
- Number keys (1-4): quick access to main screens
## Benefits
1. Simplified setup without manual file edits
2. Clear visual feedback and error messages
3. Integrated monitoring and control
4. Cross-platform: Linux, macOS, Windows
5. Fully terminal-based; no browser required