No description
This commit introduces an asynchronous method to fetch flow definitions from the Langflow API and validates the ingestion flow structure. It includes detailed extraction of components from the flow data, ensuring required components are present and logging appropriate warnings for optional components. The validation results are cached for efficiency, enhancing the overall robustness of the ingestion process. |
||
|---|---|---|
| .github/workflows | ||
| documents | ||
| flows | ||
| frontend | ||
| keys | ||
| securityconfig | ||
| src | ||
| .dockerignore | ||
| .DS_Store | ||
| .env.example | ||
| .gitignore | ||
| .python-version | ||
| docker-compose-cpu.yml | ||
| docker-compose.yml | ||
| Dockerfile | ||
| Dockerfile.backend | ||
| Dockerfile.frontend | ||
| Dockerfile.langflow | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
| warm_up_docling.py | ||
OpenRAG
getting started
Set up your secrets:
cp .env.example .env
Populate the values in .env
Requirements:
Docker or podman with compose installed.
Run OpenRAG:
docker compose build
docker compose up
CPU only:
docker compose -f docker-compose-cpu.yml up
If you need to reset state:
docker compose up --build --force-recreate --remove-orphans
Configuration
OpenRAG uses environment variables for configuration. Copy .env.example to .env and populate with your values:
cp .env.example .env
Key Environment Variables
Required:
OPENAI_API_KEY: Your OpenAI API keyOPENSEARCH_PASSWORD: Password for OpenSearch admin userLANGFLOW_SUPERUSER: Langflow admin usernameLANGFLOW_SUPERUSER_PASSWORD: Langflow admin passwordLANGFLOW_CHAT_FLOW_ID: ID of your Langflow chat flowLANGFLOW_INGEST_FLOW_ID: ID of your Langflow ingestion flow
Ingestion Configuration:
DISABLE_INGEST_WITH_LANGFLOW: Disable Langflow ingestion pipeline (default:false)falseor unset: Uses Langflow pipeline (upload → ingest → delete)true: Uses traditional OpenRAG processor for document ingestion
Optional:
LANGFLOW_PUBLIC_URL: Public URL for Langflow (default:http://localhost:7860)GOOGLE_OAUTH_CLIENT_ID/GOOGLE_OAUTH_CLIENT_SECRET: For Google OAuth authenticationMICROSOFT_GRAPH_OAUTH_CLIENT_ID/MICROSOFT_GRAPH_OAUTH_CLIENT_SECRET: For Microsoft OAuthWEBHOOK_BASE_URL: Base URL for webhook endpointsAWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY: For AWS integrations
See .env.example for a complete list with descriptions, or check the docker-compose.yml files.
For podman on mac you may have to increase your VM memory (podman stats should not show limit at only 2gb):
podman machine stop
podman machine rm
podman machine init --memory 8192 # example: 8 GB
podman machine start