Commit graph

78 commits

Author SHA1 Message Date
Liu An
0aa3c4cdae
Docs: Update version references to v0.20.2 in READMEs and docs (#9559)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.20.1 to v0.20.2
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2025-08-19 17:26:49 +08:00
Liu An
c8bbf7452d
Env: Update dependencies for proxy support (#9519)
### What problem does this PR solve?

- Update httpx dependency to include socks support in pyproject.toml
- Update lockfile with new socksio dependency

### Type of change

- [x] Update dependencies for proxy support
2025-08-18 12:04:16 +08:00
Yongteng Lei
99df0766fe
Feat: add SMTP support for user invitation emails (#9479)
### What problem does this PR solve?

Add SMTP support for user invitation emails

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-08-15 18:12:20 +08:00
Kevin Hu
b6e34e3aa7
Fix: PyPDF's Manipulated FlateDecode streams can exhaust RAM (#9469)
### What problem does this PR solve?

#3951
#8463 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-08-14 13:45:19 +08:00
Jay Xu
569ab011c4
Add fallback to use 'calamine' parse engine in excel_parser.py (#9374)
### What problem does this PR solve?

add fallback to `calamine` engine when parse error raised using the
default `openpyxl` / `xlrd` engine.
e.g. the following error can be fixed:
```
Traceback (most recent call last):
  File "/ragflow/deepdoc/parser/excel_parser.py", line 53, in _load_excel_to_workbook
    df = pd.read_excel(file_like_object)
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_base.py", line 495, in read_excel
    io = ExcelFile(
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_base.py", line 1567, in __init__
    self._reader = self._engines[engine](
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_xlrd.py", line 46, in __init__
    super().__init__(
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_base.py", line 573, in __init__
    self.book = self.load_workbook(self.handles.handle, engine_kwargs)
  File "/ragflow/.venv/lib/python3.10/site-packages/pandas/io/excel/_xlrd.py", line 63, in load_workbook
    return open_workbook(file_contents=data, **engine_kwargs)
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/__init__.py", line 172, in open_workbook
    bk = open_workbook_xls(
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/book.py", line 68, in open_workbook_xls
    bk.biff2_8_load(
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/book.py", line 641, in biff2_8_load
    cd.locate_named_stream(UNICODE_LITERAL(qname))
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/compdoc.py", line 398, in locate_named_stream
    result = self._locate_stream(
  File "/ragflow/.venv/lib/python3.10/site-packages/xlrd/compdoc.py", line 429, in _locate_stream
    raise CompDocError("%s corruption: seen[%d] == %d" % (qname, s, self.seen[s]))
xlrd.compdoc.CompDocError: Workbook corruption: seen[2] == 4
```

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-08-12 12:41:33 +08:00
Yongteng Lei
83771e500c
Refa: migrate chat models to LiteLLM (#9394)
### What problem does this PR solve?

All models pass the mock response tests, which means that if a model can
return the correct response, everything should work as expected.
However, not all models have been fully tested in a real environment,
the real API_KEY. I suggest actively monitoring the refactored models
over the coming period to ensure they work correctly and fixing them
step by step, or waiting to merge until most have been tested in
practical environment.

### Type of change

- [x] Refactoring
2025-08-12 10:59:20 +08:00
Liu An
b9eeb8e64f
Docs: Update version references to v0.20.1 in READMEs and docs (#9335)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.20.0 to v0.20.1
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2025-08-08 18:17:25 +08:00
Liu An
95534f5cf2
Docs: Update version references to v0.20.0 in READMEs and docs (#9164)
### What problem does this PR solve?

- Update version tags in README files (including translations) from
v0.19.1 to v0.20.0
- Modify Docker image references and documentation to reflect new
version
- Update version badges and image descriptions
- Maintain consistency across all language variants of README files

### Type of change

- [x] Documentation Update
2025-08-01 20:41:44 +08:00
Kevin Hu
d9fe279dde
Feat: Redesign and refactor agent module (#9113)
### What problem does this PR solve?

#9082 #6365

<u> **WARNING: it's not compatible with the older version of `Agent`
module, which means that `Agent` from older versions can not work
anymore.**</u>

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-07-30 19:41:09 +08:00
Zhichang Yu
ad177951e9
Bump to infinity v0.6.0-dev4 (#9013)
### What problem does this PR solve?

Bump to infinity v0.6.0-dev4.
WARNNING: infinity v0.6.0-dev4 has very different meta data format with
older versions. You have to destroy infinity data volume are restart
infinity container if there's existing data.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-07-23 19:27:57 +08:00
湛露先生
f6ae570417
fix redundancy package import. (#8754)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [x] Performance Improvement
- [ ] Other (please describe):

Signed-off-by: zhanluxianshen <zhanluxianshen@163.com>
2025-07-11 19:11:52 +08:00
Yongteng Lei
03656da4dd
Refa: upgrade MCP SDK to v1.9.4 (#8421)
### What problem does this PR solve?

Upgrade MCP SDK to v1.9.4 (latest).

### Type of change

- [x] Refactoring
2025-06-23 16:53:59 +08:00
Liu An
7e87eb2e23
Docs: Update version references to v0.19.1 in READMEs and docs (#8366)
### What problem does this PR solve?

- Update Docker image version badges and references from v0.19.0 to
v0.19.1
- Modify version mentions in all localized README files (id, ja, ko,
pt_br, tzh, zh)
- Update version in docker/README.md and related documentation files
- Includes updates to Helm values and Python SDK dependencies

### Type of change

- [x] Documentation Update
2025-06-19 14:39:27 +08:00
africa-worker
44287fb05f
Oss support opendal(including mysql) (#8204)
### What problem does this PR solve?

#8074
Oss support opendal(including mysql)

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2025-06-12 11:37:42 +08:00
Liu An
52c814b89d
Refa: Move HTTP API tests to top-level test directory (#8042)
### What problem does this PR solve?

Move test cases only - CI still runs tests under sdk/python

### Type of change

- [x] Refactoring
2025-06-04 13:16:17 +08:00
Liu An
e702431fcb
Feat: sync test group to top pyproject.toml (#8015)
### What problem does this PR solve?

sync test group from sdk/python/pyproject.toml to top pyproject.toml

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-06-03 15:21:06 +08:00
He Wang
aaefc3f44c
update xgboost and dep scripts for local build on MacOS (#7857)
### What problem does this PR solve?

There are two main changes:

1. Update xgboost to 1.6.0 to build the project on MacOS with Apple
chips, this change refers to the issue:
https://github.com/infiniflow/ragflow/issues/5114.

2. When `use_china_mirrors` is set in `download_deps.py`, the names of
chrome files downloaded by the script will be different from the file
names used in Dockerfile, so I added the file name in `get_urls`
function to solve this problem.

I think it's better to add testing for Docker image
`infiniflow/ragflow_deps` to the test workflow, but since the workflow
is currently running on a self-hosted runner, I'm not sure how to modify
it.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2025-05-27 09:28:52 +08:00
liu an
590b9dabab Docs: update for v0.19.0 (#7823)
### What problem does this PR solve?

update for v0.19.0

### Type of change

- [x] Documentation Update
2025-05-23 18:25:47 +08:00
Song Fuchang
a1f06a4fdc
Feat: Support tool calling in Generate component (#7572)
### What problem does this PR solve?

Hello, our use case requires LLM agent to invoke some tools, so I made a
simple implementation here.

This PR does two things:

1. A simple plugin mechanism based on `pluginlib`:

This mechanism lives in the `plugin` directory. It will only load
plugins from `plugin/embedded_plugins` for now.

A sample plugin `bad_calculator.py` is placed in
`plugin/embedded_plugins/llm_tools`, it accepts two numbers `a` and `b`,
then give a wrong result `a + b + 100`.

In the future, it can load plugins from external location with little
code change.

Plugins are divided into different types. The only plugin type supported
in this PR is `llm_tools`, which must implement the `LLMToolPlugin`
class in the `plugin/llm_tool_plugin.py`.
More plugin types can be added in the future.

2. A tool selector in the `Generate` component:

Added a tool selector to select one or more tools for LLM:


![image](https://github.com/user-attachments/assets/74a21fdf-9333-4175-991b-43df6524c5dc)

And with the `bad_calculator` tool, it results this with the `qwen-max`
model:


![image](https://github.com/user-attachments/assets/93aff9c4-8550-414a-90a2-1a15a5249d94)


### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):

Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
2025-05-16 16:32:19 +08:00
Gifford Nowland
0b4d366514
Fix: Setuptools project.license as a TOML table deprecation (#7652)
TOML-table-based project.license is deprecated as per PEP 639, see:
https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license-and-license-files

### What problem does this PR solve?

The following error when building project (e.g. `uv build`)

```
SetuptoolsDeprecationWarning: `project.license` as a TOML table is deprecated
!!

        ********************************************************************************
        Please use a simple string containing a SPDX expression for `project.license`. You can also use `project.license-files`. (Both options available on setuptools>=77.0.0).

        By 2026-Feb-18, you need to update your project and remove deprecated calls
        or your builds will no longer be supported.

        See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
        ********************************************************************************

!!
```

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2025-05-15 12:23:29 +08:00
Gifford Nowland
e7a84bdac2
Fix: setuptools package definitions (#7654)
### What problem does this PR solve?
For `uv package`/`uv pip install ".[full]"`, bug introduced in #6370:

* Removes erroneous (non-package) directories (`helm`, `flask_session`)
* Adds `mcp.server` package
* Resolves "warning: package would be ignored" ambiguity by changing
`sdk` to `sdk.python.ragflow_sdk`
* Resolves "error: package directory 'intergrations' does not exist" by
including `intergrations.chatgpt-on-wechat.plugins` explicitly
* Also rearranges packages in alphabetical order, for DX.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2025-05-15 12:22:56 +08:00
pyyuhao
c8c3b756b0
Feat: Adds OpenSearch2.19.1 as the vector_database support (#7140)
### What problem does this PR solve?

This PR adds the support for latest OpenSearch2.19.1 as the store engine
& search engine option for RAGFlow.

### Main Benefit

1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is
much better than Elasticsearch
2. For search, OpenSearch2.19.1 supports full-text
search、vector_search、hybrid_search those are similar with Elasticsearch
on schema
3. For store, OpenSearch2.19.1 stores text、vector those are quite
simliar with Elasticsearch on schema

### Changes

- Support opensearch_python_connetor. I make a lot of adaptions since
the schema and api/method between ES and Opensearch differs in many
ways(especially the knn_search has a significant gap) :
rag/utils/opensearch_coon.py
- Support static config adaptions by changing:
conf/service_conf.yaml、api/settings.py、rag/settings.py
- Supprt some store&search schema changes between OpenSearch and ES:
conf/os_mapping.json
- Support OpenSearch python sdk : pyproject.toml
- Support docker config for OpenSearch2.19.1 :
docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template

### How to use
- I didn't change the priority that ES as the default doc/search engine.
Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it
will work.


### Others
Our team tested a lot of docs in our environment by using OpenSearch as
the vector database ,it works very well.
All the conifg for OpenSearch is necessary.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Yongteng Lei <yongtengrey@outlook.com>
Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com>
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
2025-04-24 16:03:31 +08:00
liu an
03672df691
Docs: update for v0.18.0 (#7223)
### What problem does this PR solve?

update for v0.18.0

### Type of change

- [x] Documentation Update
2025-04-23 12:02:50 +08:00
Yongteng Lei
68b9dae6c0
Feat: mcp server (#7084)
### What problem does this PR solve?

Add MCP support with a client example.

Issue link: #4344

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-04-21 09:43:20 +08:00
Zhichang Yu
6bf26e2a81
Optimize graphrag again (#6513)
### What problem does this PR solve?

Removed set_entity and set_relation to avoid accessing doc engine during
graph computation.
Introduced GraphChange to avoid writing unchanged chunks.

### Type of change

- [x] Performance Improvement
2025-03-26 15:34:42 +08:00
Yongteng Lei
85eb367ede
Feat: add basic Langfuse support for LLM module (#6443)
### What problem does this PR solve?

#6155

Add basic Langfuse support for LLM module.

A trace example:

<img width="755" alt="image"
src="https://github.com/user-attachments/assets/25c1f852-5116-486c-a47f-6097187142ca"
/>


### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-03-24 13:18:47 +08:00
fansir
4091af4560
Fix: multiple top-level packages error in Python project (#6370)
### What problem does this PR solve?

This PR resolves the issue of multiple top-level packages being detected
in the Python project, which caused errors when using uv pip install.
The problem occurred because the project had multiple directories files
at the root level, leading to a flat-layout error.
To fix this, the pyproject.toml file was updated to explicitly list the
packages using the [tool.setuptools] section. This ensures that the
correct packages are included during installation, avoiding the
flat-layout error.
Type of change

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-03-21 18:44:49 +08:00
Yongteng Lei
674f94228b
Chore: unify Ruff config and enable async checks (ASYNC, TRIO) (#6351)
### What problem does this PR solve?

Unify Ruff config and enable async checks (ASYNC, TRIO)

### Type of change

- [x] CI/CD or tooling improvement
2025-03-20 22:31:18 +08:00
Kevin Hu
4df4bf68a2
DOCS: for release. (#6023)
### What problem does this PR solve?


### Type of change

- [x] Documentation Update
2025-03-13 15:09:29 +08:00
Kevin Hu
d44739283c
Docs: prepare docs for release v0.17.1 (#5900)
### What problem does this PR solve?


### Type of change

- [x] Documentation Update
2025-03-11 14:39:41 +08:00
Zhichang Yu
c813c1ff4c
Made task_executor async to speedup parsing (#5530)
### What problem does this PR solve?

Made task_executor async to speedup parsing

### Type of change

- [x] Performance Improvement
2025-03-03 18:59:49 +08:00
Kevin Hu
d6836444c9
DOC: for release. (#5472)
### What problem does this PR solve?


### Type of change

- [x] Documentation Update

---------

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2025-03-02 18:47:06 +08:00
yihong
37aacb3960
Refa: drop useless fasttext (#5470)
### What problem does this PR solve?

This patch drop useless fastext which is seems useless in the code base 
and its very kind of hard install
should close #4498


### Type of change

- [x] Refactoring

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-02-28 14:30:56 +08:00
Kevin Hu
53b9e7b52f
Add tavily as web searh tool. (#5349)
### What problem does this PR solve?

#5198

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-02-26 10:21:04 +08:00
Kevin Hu
fe9e9a644f
Preparation for release. (#4739)
### What problem does this PR solve?


### Type of change

- [x] Documentation Update
2025-02-06 15:15:13 +08:00
Zhichang Yu
3411d0a2ce
Added cuda_is_available (#4725)
### What problem does this PR solve?

Added cuda_is_available

### Type of change

- [x] Refactoring
2025-02-05 18:01:23 +08:00
Zhichang Yu
e1526846da
Fixed GPU detection on CPU only environment (#4711)
### What problem does this PR solve?

Fixed GPU detection on CPU only environment. Close #4692

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-02-05 12:02:43 +08:00
Zhichang Yu
c4b1c4e6f4
Fix onnxruntime-gpu marks (#4643)
### What problem does this PR solve?

Fix onnxruntime-gpu marks

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-26 09:37:59 +08:00
Zhichang Yu
3c2c8942d5
Removed onnxruntime (#4632)
### What problem does this PR solve?

Removed onnxruntime. It conflicts with the onnxruntime-gpu.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-24 23:41:52 +08:00
Zhichang Yu
8b49734241
Added onnxruntime-gpu (#4631)
### What problem does this PR solve?

Added onnxruntime-gpu

### Type of change

- [x] Refactoring
2025-01-24 14:33:21 +08:00
Kevin Hu
dd0ebbea35
Light GraphRAG (#4585)
### What problem does this PR solve?

#4543

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-01-22 19:43:14 +08:00
Zhichang Yu
2962284c79
Bump akshare (#4536)
### What problem does this PR solve?

Bump akshare. Close #4525 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-20 11:17:59 +08:00
Zhichang Yu
57b4e0c464
Bump infinity to v0.6.0-dev2 (#4497)
### What problem does this PR solve?

Bump infinity to v0.6.0-dev2. Close #4477 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-16 10:07:17 +08:00
Zhichang Yu
d3c07794b5
Replace poetry with uv (#4471)
### What problem does this PR solve?

Replace poetry with uv

### Type of change

- [x] Refactoring
2025-01-14 11:49:43 +08:00
Zhichang Yu
95261f17f6
Bump infinity to v0.6.0-dev1 (#4448)
### What problem does this PR solve?

Bump infinity to v0.6.0-dev1 and poetry to 2.0.1

### Type of change

- [x] Refactoring
2025-01-12 19:54:33 +08:00
Hayden Zhou
6acbd374d8
fix duckduckgo search subsection error (#4430)
### What problem does this PR solve?

duckduckgo search 6.3.0 still has error sometimes, need to update to
7.2.0, after updated, it works ok.
this PR is going to fix this issue
https://github.com/infiniflow/ragflow/issues/4396

### Type of change

 Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: xiaohzho <xiaohzho@cisco.com>
2025-01-10 09:34:20 +08:00
Hayden Zhou
ac89a2dda1
[Fix] fix duckduck go search 202 ratelimit failed (#4398)
this PR is going to fix issue
https://github.com/infiniflow/ragflow/issues/4396

### Type of change

 Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: xiaohzho <xiaohzho@cisco.com>
2025-01-07 18:40:46 +08:00
Jin Hai
439d20e41f
Use LTS polars to resolve some machines don't support AVX CPU flag (#4364)
### What problem does this PR solve?

Some old types of machine or virtual machine doesn't support AVX CPU
flag. This PR is to use lts polars module to avoid this fault.

fix issue: #4349

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring

Signed-off-by: jinhai <haijin.chn@gmail.com>
2025-01-06 18:52:17 +08:00
Jin Hai
1160b58b6e
Update pyproject.toml to 0.15.1 (#4222)
### What problem does this PR solve?

Missing update information

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-12-25 14:06:00 +08:00
Zhichang Yu
b7a7413419
Bump infinity to 0.5.2 (#4207)
### What problem does this PR solve?

Bump infinity to 0.5.2

### Type of change

- [x] Refactoring
2024-12-24 15:17:37 +08:00