MLO-446: Adds API key authentication support to LightRAG client (#12)

* Adds LightRAG API key support to deployment and secrets Introduces a new environment variable for the LightRAG API key sourced from secrets to enable authenticated access. Updates Helm values and templates to include LightRAG API key management alongside the existing OpenAI key, improving configuration consistency and security. Relates to MLO-339 * Adds optional API key authentication support to LightRAG client Enables passing custom headers, including an API key from environment variables, to all LightRAG HTTP requests for authentication. Improves security by allowing authenticated access without breaking existing unauthenticated usage. Relates to MLO-446 * Adds basic user authentication support to Helm deployment Introduces configurable user accounts and token secret in values and templates to enable authentication. Generates an encoded authentication string from account data stored in secrets and exposes relevant environment variables in the deployment only when authentication is enabled and configured. This enhancement allows secure management of multiple user credentials and token secrets, improving the deployment's security and flexibility. Relates to MLO-446 * Adds support for external secret references in PostgreSQL auth Introduces parameters to allow PostgreSQL credentials to be sourced from existing Kubernetes secrets instead of inline passwords. Improves security and flexibility by enabling integration with external secret management without changing deployment structure. Relates to MLO-446 * Streamline deployment docs and remove preset environment configs Consolidates deployment instructions by removing separate dev and prod values files and related workflows, encouraging users to customize a single values file instead. Simplifies the README to focus on flexible chart deployment without environment-specific templates or variable substitution, improving maintainability and clarity. * Adds Helm packaging and publishing Makefile for LightRAG Introduces a Makefile to automate Helm chart packaging, versioning, and publishing to a container registry. Uses git tags or user-defined versions for chart versioning with sanitization. Ensures streamlined CI/CD by handling dependencies, packaging, registry login, and cleanup, simplifying release workflows. Relates to MLO-446
2025-10-29 14:31:56 +02:00 · 2025-10-29 14:31:56 +02:00 · 748ded40fb
commit 748ded40fb
parent aabe4aa434
9 changed files with 158 additions and 244 deletions
--- a/62
+++ b/62
@ -0,0 +1,62 @@
+# Makefile for LightRAG Helm packaging
+
+# Configuration
+CHART_NAME := lightrag-minimal
+CHART_DIR := k8s-deploy/$(CHART_NAME)
+CHART_PACKAGE_DIR := dist/charts
+HELM_REGISTRY := ghcr.io/neuro-inc/helm-charts
+
+RAW_VERSION := $(if $(VERSION),$(VERSION),$(shell git describe --tags --always --dirty 2>/dev/null))
+SANITIZED_VERSION := $(shell RAW="$(RAW_VERSION)" python - <<'PY'
+import os, re
+raw = os.environ.get("RAW", "").strip()
+if not raw:
+    raw = "0.0.0"
+raw = raw.lstrip("v")
+sanitized = re.sub(r"[^0-9A-Za-z\\.\\-]", "-", raw)
+print(sanitized or "0.0.0")
+PY
+)
+CHART_VERSION := $(SANITIZED_VERSION)
+CHART_PACKAGE := $(CHART_PACKAGE_DIR)/$(CHART_NAME)-$(CHART_VERSION).tgz
+
+GITHUB_USERNAME := $(shell echo "$$APOLO_GITHUB_TOKEN" | base64 -d 2>/dev/null | cut -d: -f1 2>/dev/null || echo "oauth2")
+
+.PHONY: help helm-package helm-push clean
+
+help:
+	@echo "Available targets:"
+	@echo "  helm-package         - Package the LightRAG Helm chart (version: $(CHART_VERSION))"
+	@echo "  helm-push            - Package and push the chart to $(HELM_REGISTRY)"
+	@echo "  clean                - Remove packaged charts from $(CHART_PACKAGE_DIR)"
+	@echo "\nSet VERSION=1.2.3 to override the git-derived chart version."
+
+helm-package:
+	@if [ -z "$(CHART_VERSION)" ]; then \
+		echo "Error: unable to determine chart version."; \
+		exit 1; \
+	fi
+	@echo "Packaging $(CHART_NAME) chart version $(CHART_VERSION)..."
+	@mkdir -p $(CHART_PACKAGE_DIR)
+	helm dependency update $(CHART_DIR) >/dev/null
+	helm package $(CHART_DIR) \
+		--version $(CHART_VERSION) \
+		--app-version $(CHART_VERSION) \
+		-d $(CHART_PACKAGE_DIR)
+	@echo "✅ Chart packaged at $(CHART_PACKAGE)"
+
+helm-push: helm-package
+	@if [ -z "$(APOLO_GITHUB_TOKEN)" ]; then \
+		echo "Error: APOLO_GITHUB_TOKEN not set. Please export a token with write:packages."; \
+		exit 1; \
+	fi
+	@echo "Logging into Helm registry ghcr.io as $(GITHUB_USERNAME)..."
+	echo "$(APOLO_GITHUB_TOKEN)" | helm registry login ghcr.io -u $(GITHUB_USERNAME) --password-stdin >/dev/null
+	@echo "Pushing chart $(CHART_NAME):$(CHART_VERSION) to $(HELM_REGISTRY)..."
+	helm push $(CHART_PACKAGE) oci://$(HELM_REGISTRY)
+	@echo "✅ Chart pushed to $(HELM_REGISTRY)"
+
+clean:
+	@echo "Removing packaged charts..."
+	rm -rf $(CHART_PACKAGE_DIR)
+	@echo "✅ Cleaned"
--- a/k8s-deploy/lightrag-minimal/README.md
+++ b/k8s-deploy/lightrag-minimal/README.md
@ -22,31 +22,23 @@ This chart provides a comprehensive LightRAG deployment with:

 ## Validated Installation Steps

-### Development/Local Setup (Minikube)
+### Deploying the Chart

-1. **Prepare Helm repositories**:
 ```bash
 cd lightrag-minimal
 helm repo add bitnami https://charts.bitnami.com/bitnami
 helm repo update
 helm dependency update
-```

-2. **Set your OpenAI API key**:
-```bash
-export OPENAI_API_KEY="your-openai-api-key-here"
-```
+# (Optional) create a copy of values.yaml and customize it for your environment
+cp values.yaml my-values.yaml
+# edit my-values.yaml as needed (OpenAI keys, storage class, Postgres password, etc.)

-3. **Deploy for development**:
-```bash
-# Substitute environment variables and deploy
-envsubst < values-dev.yaml > values-dev-final.yaml
 helm install lightrag-minimal . \
-  -f values-dev-final.yaml \
+  -f my-values.yaml \
  --namespace lightrag \
  --create-namespace

-# Wait for deployment
 kubectl wait --namespace lightrag \
  --for=condition=ready pod \
  -l app.kubernetes.io/name=postgresql \
@ -57,31 +49,14 @@ kubectl wait --namespace lightrag \
  -l app.kubernetes.io/name=lightrag-minimal \
  --timeout=120s

-# Clean up temporary file
-rm values-dev-final.yaml
-
-# Start port forwarding
+# Optional: expose the service locally
 kubectl port-forward --namespace lightrag svc/lightrag-minimal 9621:9621 &
 ```

-### Production Setup
-
-```bash
-# Customize values-prod.yaml first (domain, storage classes, etc.)
-envsubst < values-prod.yaml > values-prod-final.yaml
-helm install lightrag-minimal . \
-  -f values-prod-final.yaml \
-  --namespace lightrag \
-  --create-namespace
-rm values-prod-final.yaml
-```
-
 ## Configuration Options

 ### Validated Environment Configuration

-Both `values-dev.yaml` and `values-prod.yaml` include these critical settings:
-
 ```yaml
 env:
  # OpenAI API Configuration (REQUIRED)
@ -376,4 +351,4 @@ kubectl delete namespace lightrag
 | Persistence | Local volumes | PersistentVolumeClaims |
 | Monitoring | Manual | Kubernetes native |

-This chart maintains the same conservative, working configuration as the Docker Compose setup while adding Kubernetes-native features for production deployment.
+This chart maintains the same conservative, working configuration as the Docker Compose setup while adding Kubernetes-native features for production deployment.
--- a/k8s-deploy/lightrag-minimal/templates/_helpers.tpl
+++ b/k8s-deploy/lightrag-minimal/templates/_helpers.tpl
@ -77,4 +77,15 @@ PostgreSQL connection string
 {{- else }}
 {{- .Values.env.POSTGRES_HOST }}
 {{- end }}
-{{- end }}
+{{- end }}
+
+{{/*
+Generate AUTH_ACCOUNTS string from configured accounts
+*/}}
+{{- define "lightrag-minimal.authAccounts" -}}
+{{- if .Values.auth.accounts }}
+{{- range $index, $account := .Values.auth.accounts -}}
+{{- if gt $index 0 }},{{ end -}}{{ $account.username }}:{{ $account.password }}
+{{- end -}}
+{{- end -}}
+{{- end }}
--- a/k8s-deploy/lightrag-minimal/templates/deployment.yaml
+++ b/k8s-deploy/lightrag-minimal/templates/deployment.yaml
@ -77,7 +77,26 @@ spec:
                secretKeyRef:
                  name: {{ include "lightrag-minimal.secretName" . }}
                  key: embedding-api-key
-            
+            - name: LIGHTRAG_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "lightrag-minimal.secretName" . }}
+                  key: lightrag-api-key
+            {{- if and .Values.auth.enabled (gt (len .Values.auth.accounts) 0) }}
+            - name: AUTH_ACCOUNTS
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "lightrag-minimal.secretName" . }}
+                  key: auth-accounts
+            {{- end }}
+            {{- if and .Values.auth.enabled .Values.auth.tokenSecret }}
+            - name: TOKEN_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "lightrag-minimal.secretName" . }}
+                  key: token-secret
+            {{- end }}
+
            # Storage configuration
            - name: LIGHTRAG_KV_STORAGE
              value: {{ .Values.env.LIGHTRAG_KV_STORAGE | quote }}
@ -156,4 +175,4 @@ spec:
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
-      {{- end }}
+      {{- end }}
--- a/k8s-deploy/lightrag-minimal/templates/secret.yaml
+++ b/k8s-deploy/lightrag-minimal/templates/secret.yaml
@ -9,4 +9,11 @@ data:
  openai-api-key: {{ .Values.secrets.openaiApiKey | b64enc | quote }}
  llm-api-key: {{ .Values.secrets.llmApiKey | b64enc | quote }}
  embedding-api-key: {{ .Values.secrets.embeddingApiKey | b64enc | quote }}
-  postgres-password: {{ .Values.postgresql.auth.password | b64enc | quote }}
+  lightrag-api-key: {{ .Values.secrets.lightragApiKey | default "" | b64enc | quote }}
+  postgres-password: {{ .Values.postgresql.auth.password | b64enc | quote }}
+  {{- if and .Values.auth.enabled (gt (len .Values.auth.accounts) 0) }}
+  auth-accounts: {{ include "lightrag-minimal.authAccounts" . | b64enc | quote }}
+  {{- end }}
+  {{- if and .Values.auth.enabled .Values.auth.tokenSecret }}
+  token-secret: {{ .Values.auth.tokenSecret | b64enc | quote }}
+  {{- end }}
--- a/k8s-deploy/lightrag-minimal/values-dev.yaml
+++ b/k8s-deploy/lightrag-minimal/values-dev.yaml
@ -1,75 +0,0 @@
-# Development/Minikube Values
-# Optimized for local development with reduced resource requirements
-
-# Environment configuration
-env:
-  LLM_MODEL: "gpt-4o"
-  WEBUI_TITLE: "Apolo Copilot - LightRAG (Development)"
-  WEBUI_DESCRIPTION: "Development LightRAG for Apolo Documentation"
-  
-  # OpenAI API Configuration
-  LLM_BINDING: "openai"
-  LLM_BINDING_HOST: "https://api.openai.com/v1"
-  EMBEDDING_BINDING: "openai"
-  EMBEDDING_BINDING_HOST: "https://api.openai.com/v1"
-  EMBEDDING_MODEL: "text-embedding-ada-002"
-  EMBEDDING_DIM: "1536"
-  
-  # Concurrency settings (conservative for API stability)
-  MAX_ASYNC: "4"
-  MAX_PARALLEL_INSERT: "2"
-  
-  # LLM Configuration
-  ENABLE_LLM_CACHE: "true"
-  ENABLE_LLM_CACHE_FOR_EXTRACT: "true"
-  TIMEOUT: "240"
-  TEMPERATURE: "0"
-  MAX_TOKENS: "32768"
-
-# Reduced resources for local development
-resources:
-  limits:
-    cpu: 1000m
-    memory: 2Gi
-  requests:
-    cpu: 250m
-    memory: 512Mi
-
-# Smaller storage for development
-persistence:
-  ragStorage:
-    size: 5Gi
-  inputs:
-    size: 2Gi
-
-# PostgreSQL with reduced resources
-postgresql:
-  # Use pgvector image for vector support
-  image:
-    registry: docker.io
-    repository: pgvector/pgvector
-    tag: pg16
-  auth:
-    password: "dev-lightrag-pass"
-  primary:
-    persistence:
-      size: 5Gi
-    resources:
-      limits:
-        cpu: 500m
-        memory: 1Gi
-      requests:
-        cpu: 100m
-        memory: 256Mi
-
-# OpenAI API key (set via environment variable)
-secrets:
-  openaiApiKey: "${OPENAI_API_KEY}"
-
-# Disable ingress for local development (use port-forward)
-ingress:
-  enabled: false
-
-# Disable autoscaling for development
-autoscaling:
-  enabled: false
--- a/k8s-deploy/lightrag-minimal/values-prod.yaml
+++ b/k8s-deploy/lightrag-minimal/values-prod.yaml
@ -1,120 +0,0 @@
-# Production Values
-# Optimized for production with HA, scaling, and monitoring
-
-# Environment configuration
-env:
-  LLM_MODEL: "gpt-4o"
-  WEBUI_TITLE: "Apolo Copilot - LightRAG"
-  WEBUI_DESCRIPTION: "Production LightRAG for Apolo Documentation"
-  
-  # OpenAI API Configuration
-  LLM_BINDING: "openai"
-  LLM_BINDING_HOST: "https://api.openai.com/v1"
-  EMBEDDING_BINDING: "openai"
-  EMBEDDING_BINDING_HOST: "https://api.openai.com/v1"
-  EMBEDDING_MODEL: "text-embedding-ada-002"
-  EMBEDDING_DIM: "1536"
-  
-  # Concurrency settings (conservative for API stability)
-  MAX_ASYNC: "4"
-  MAX_PARALLEL_INSERT: "2"
-  
-  # LLM Configuration
-  ENABLE_LLM_CACHE: "true"
-  ENABLE_LLM_CACHE_FOR_EXTRACT: "true"
-  TIMEOUT: "240"
-  TEMPERATURE: "0"
-  MAX_TOKENS: "32768"
-
-# Production resources
-resources:
-  limits:
-    cpu: 4000m
-    memory: 8Gi
-  requests:
-    cpu: 1000m
-    memory: 2Gi
-
-# Production storage with fast storage class
-persistence:
-  ragStorage:
-    size: 100Gi
-    storageClass: "fast-ssd"  # Adjust for your cluster
-  inputs:
-    size: 50Gi
-    storageClass: "fast-ssd"
-
-# PostgreSQL with production resources
-postgresql:
-  # Use pgvector image for vector support
-  image:
-    registry: docker.io
-    repository: pgvector/pgvector
-    tag: pg16
-  auth:
-    password: "secure-production-password"  # Use external secret in real production
-  primary:
-    persistence:
-      size: 200Gi
-      storageClass: "fast-ssd"
-    resources:
-      limits:
-        cpu: 2000m
-        memory: 4Gi
-      requests:
-        cpu: 500m
-        memory: 1Gi
-
-# OpenAI API key (use external secret manager in production)
-secrets:
-  openaiApiKey: "${OPENAI_API_KEY}"
-
-# Enable ingress for production
-ingress:
-  enabled: true
-  className: "nginx"
-  annotations:
-    cert-manager.io/cluster-issuer: "letsencrypt-prod"
-    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
-    nginx.ingress.kubernetes.io/ssl-redirect: "true"
-  hosts:
-    - host: lightrag.yourdomain.com
-      paths:
-        - path: /
-          pathType: Prefix
-  tls:
-    - secretName: lightrag-tls
-      hosts:
-        - lightrag.yourdomain.com
-
-# Enable autoscaling for production
-autoscaling:
-  enabled: true
-  minReplicas: 2
-  maxReplicas: 10
-  targetCPUUtilizationPercentage: 70
-  targetMemoryUtilizationPercentage: 80
-
-# Production security context
-securityContext:
-  runAsNonRoot: true
-  runAsUser: 1000
-  fsGroup: 1000
-
-podSecurityContext:
-  seccompProfile:
-    type: RuntimeDefault
-
-# Node affinity for production workloads
-affinity:
-  podAntiAffinity:
-    preferredDuringSchedulingIgnoredDuringExecution:
-    - weight: 100
-      podAffinityTerm:
-        labelSelector:
-          matchExpressions:
-          - key: app.kubernetes.io/name
-            operator: In
-            values:
-            - lightrag-minimal
-        topologyKey: kubernetes.io/hostname
--- a/k8s-deploy/lightrag-minimal/values.yaml
+++ b/k8s-deploy/lightrag-minimal/values.yaml
@ -60,7 +60,10 @@ postgresql:
  auth:
    database: lightrag
    username: lightrag_user
-    password: lightrag_pass
+    password: ""
+    existingSecret: ""
+    secretKeys:
+      userPasswordKey: postgres-password
  primary:
    persistence:
      enabled: true
@ -132,7 +135,13 @@ secrets:
  openaiApiKey: "" # Legacy field, kept for backward compatibility
  llmApiKey: "" # API key for LLM service (e.g., OpenRouter)
  embeddingApiKey: "" # API key for embedding service (e.g., Google Gemini)
-  
+  lightragApiKey: ""
+
+auth:
+  enabled: false
+  accounts: []
+  tokenSecret: ""
+

 # Node selector and affinity
 nodeSelector: {}
--- a/load_docs.py
+++ b/load_docs.py
@ -8,22 +8,27 @@ import asyncio
 import httpx
 import argparse
 import sys
+import os
 from pathlib import Path
-from typing import List, Optional
+from typing import Dict, List, Optional


 async def load_document_to_lightrag(
    content: str, 
    title: str, 
    doc_url: str,
-    endpoint: str = "http://localhost:9621"
+    endpoint: str = "http://localhost:9621",
+    headers: Optional[Dict[str, str]] = None
 ) -> bool:
    """Load a single document to LightRAG with URL reference"""
    try:
        async with httpx.AsyncClient(timeout=30.0) as client:
+            request_headers = {"Content-Type": "application/json"}
+            if headers:
+                request_headers.update(headers)
            response = await client.post(
                f"{endpoint}/documents/text",
-                headers={"Content-Type": "application/json"},
+                headers=request_headers,
                json={
                    "text": content,
                    "file_source": doc_url
@ -148,11 +153,14 @@ Source: {source_info}
    return documents


-async def test_lightrag_health(endpoint: str = "http://localhost:9621") -> bool:
+async def test_lightrag_health(
+    endpoint: str = "http://localhost:9621",
+    headers: Optional[Dict[str, str]] = None
+) -> bool:
    """Test if LightRAG is accessible"""
    try:
        async with httpx.AsyncClient(timeout=10.0) as client:
-            response = await client.get(f"{endpoint}/health")
+            response = await client.get(f"{endpoint}/health", headers=headers)
            if response.status_code == 200:
                health_data = response.json()
                print(f"✅ LightRAG is healthy: {health_data.get('status')}")
@ -165,14 +173,20 @@ async def test_lightrag_health(endpoint: str = "http://localhost:9621") -> bool:
        return False


-async def test_query(endpoint: str = "http://localhost:9621") -> None:
+async def test_query(
+    endpoint: str = "http://localhost:9621",
+    headers: Optional[Dict[str, str]] = None
+) -> None:
    """Test a sample query"""
    print(f"\n🧪 Testing query...")
    try:
        async with httpx.AsyncClient(timeout=30.0) as client:
+            request_headers = {"Content-Type": "application/json"}
+            if headers:
+                request_headers.update(headers)
            response = await client.post(
                f"{endpoint}/query",
-                headers={"Content-Type": "application/json"},
+                headers=request_headers,
                json={"query": "What is this documentation about?", "mode": "local"}
            )
            
@ -247,6 +261,12 @@ Examples:
    )
    
    args = parser.parse_args()
+    api_key = os.getenv("LIGHTRAG_API_KEY")
+    if api_key:
+        auth_headers = {"X-API-Key": api_key}
+    else:
+        auth_headers = None
+        print("ℹ️ LIGHTRAG_API_KEY not set, continuing without authentication.")
    
    print("🚀 Loading Documentation into LightRAG")
    print("=" * 60)
@ -262,7 +282,7 @@ Examples:
    print()
    
    # Test LightRAG connectivity
-    if not await test_lightrag_health(args.endpoint):
+    if not await test_lightrag_health(args.endpoint, headers=auth_headers):
        print("❌ Cannot connect to LightRAG. Please ensure it's running and accessible.")
        sys.exit(1)
    
@ -292,7 +312,13 @@ Examples:
    print(f"\n🔄 Starting to load documents...")
    
    for i, (content, title, doc_url) in enumerate(documents):
-        success = await load_document_to_lightrag(content, title, doc_url, args.endpoint)
+        success = await load_document_to_lightrag(
+            content,
+            title,
+            doc_url,
+            args.endpoint,
+            headers=auth_headers
+        )
        
        if success:
            successful += 1
@ -312,8 +338,8 @@ Examples:
    
    # Test query unless disabled
    if not args.no_test and successful > 0:
-        await test_query(args.endpoint)
+        await test_query(args.endpoint, headers=auth_headers)


 if __name__ == "__main__":
-    asyncio.run(main())
+    asyncio.run(main())