# Upstream (HKUDS/main @ f0d67f16) — features & fixes missing from this version

Summary: this file lists new features, bug fixes, CI/docs/tooling changes and important refactors that exist in upstream `HKUDS/LightRAG` main (commit f0d67f16 and its ancestors) but are not merged into this local branch. I grouped changes into functional areas and included short remediation notes where appropriate.

NOTE: upstream/main contains many small dependency bumps and documentation commits; this document focuses on substantive features and functional fixes that affect runtime behavior, storage, security, tooling and testing.

1) Core storage, DB & Postgres improvements
- Add PostgreSQL vchordrq vector index support and unify vector index creation logic (dev-postgres-vchordrq) — improves Postgres vector indexing semantics and config handling.
- Add CASCADE to AGE extension creation in Postgres init scripts (avoid failures when recreating extension)
- Add postgres_impl fixes, retry improvements and support for vchordrq epsilon config when probes empty
- Postgres RLS-related and storage refinements (note: local branch already added postgres_rls.sql; upstream brings complementary DB/VECTOR engine improvements and fixes). Remediation: merge upstream postgres/vchordrq changes and ensure migration scripts align.

2) Chunking, indexing and document ingestion fixes
- Fix top_n behavior: limit by documents instead of chunks to avoid over-counting. (important for retrieval ranking)
- Fix infinite loop when overlap_tokens >= max_tokens and edge-case handling for max_tokens == 1.
- Add comprehensive tests for chunking logic (multi-token tokenizer, recursive split) and chunking parameters tuning.
- Add content deduplication check for document insertion endpoints and fix duplicate document response handling to return original track_id. (prevents duplicates and preserves original IDs)

3) Embeddings & LLM / cloud provider support improvements
- Major improvements in OpenAI/OLLAMA/Azure/Bedrock embedding wrappers and clients:
  - Allow embedding provider defaults when unspecified
  - Add configurable embedding token limits and validation
  - Fix Azure OpenAI compatibility and support various deployments, fallback to AZURE_OPENAI_API_VERSION
  - Convert OpenAI client to use a stable API and bump minimum version (>=2.0.0)
  - Add support for structured OpenAI outputs via parsed field
  - Improve Bedrock error handling and add retry logic/custom exceptions
  - Additional refactors for embedding function wrapping rules, model param handling and function attribute inheritance
  - Add helper flags like configurable model parameter to jina_embed
  - Support async chunking functions for large, async chunkers
  - Add new LLM support, additions under lightrag/llm (e.g., gemini file added upstream)

4) Document / file extraction improvements
- DOCX/XLSX handling fixes (preserve table structure, whitespace, column alignment; optimize memory use)
- Replace PyPDF2 with pypdf for PDF processing (faster, more reliable parsing)

5) Workspace isolation, pipeline status, RAG lifecycle fixes
- Fix document deletion concurrency control and auto-acquire pipeline when idle.
- Auto-initialize pipeline status on LightRAG.initialize_storages() (reduces error-prone manual calls)
- Namespace, workspace handling and locking fixes: improvements to NamespaceLock (ContextVar), default workspace handling, filtering logic, consistent empty workspace handling and many concurrency bug fixes.

6) Web UI — upgrades, feature additions, fixes
- Large set of dependency upgrades for `lightrag_webui` (vite, react-i18next, plugin-react-swc, syntax highlighter, etc.). Upstream also cleaned duplicate deps and improved build tooling.
- Add new UI components / improvements (MergeDialog, graph features, translations updates, many components updated).
- Handle missing WebUI assets gracefully so server startup is not blocked.
- Add static swagger UI assets for API docs (swagger-ui files added upstream).

7) CI, testing, and developer tooling
- New/updated GitHub workflows and test runners: tests.yml, improved offline/integration CI markers, Copilot setup steps, docker-build* workflows and improved GitHub Actions versions.
- Drop older Python versions in test matrices (3.10/3.11 removed; 3.13/3.14 added) — keep CI modern.
- Add ruff to pytest extras, add pre-commit hooks and refine pytest fixtures and markers.
- Add many new tests including workspace isolation, chunking tests, overlap validation, postgres retry integration tests, rerank chunking tests, and E2E test improvements.

8) Tools & CLI
- New helper tools: clean_llm_query_cache.py, migrate_llm_cache.py, download_cache.py and related README docs for cleaning/migrating LLM caches.
- Add `lightrag-clean-llmqc` console script entrypoint.

9) Docs & deployment support
- Added docs: FrontendBuildGuide.md, OfflineDeployment.md, UV_LOCK_GUIDE.md and evaluation assets.
- Added Dockerfile.lite and docker-build-push.sh to support smaller builds and multi-format distribution.

10) KaTeX & math / feature parity
- Upstream adds KaTeX copy‑tex extension support and mhchem extension for chemistry formulas (enables better formula copying and chemistry rendering). Also fixed KaTeX loading in startup.

11) JSON, sanitization and performance
- Multiple JSON write/sanitizer enhancements (specialized sanitizers to handle tuples/dict keys/UTF8 errors, optimize sanitization performance) and fixes to avoid memory corruption on migrations.

12) Cloud model & misc improvements
- Improve cloud model detection/safety, macOS fork-safety check for Gunicorn multiworker cases; many small fixes for cloud model defaults and config.

13) Security / dependency hygiene
- Remove future dependency and replace passlib usage with direct bcrypt (adopt modern libs)

Actionable remediation checklist (priority):
- Merge and test upstream changes that affect: chunking, embeddings/LLM wrappers, doc processing, and Postgres vector indexing + RLS compatibility (High).
- Add or adapt DB migrations to incorporate any upstream schema changes required by tenant features and ensure no conflicts (High).
- Update CI matrix and tests to incorporate upstream tests (esp. workspace isolation and chunking tests) to verify no regressions (High).
- Merge Web UI updates separately behind feature flag/workflow (Medium) — major dependency churn.

If you want, I can automatically generate:
- a full commit-by-commit list (746 commits) in docs/diff_hku/unmerged_upstream_commits.txt (raw) — useful for exhaustive audit.
- cherry-pick safe/high-priority upstream commits onto this branch and prepare a candidate PR with resolved conflicts.