Commit graph

4555 commits

Author SHA1 Message Date
Igor Ilic
4afde917a9
Main merge vol4 (#1856)
<!-- .github/pull_request_template.md -->

## Description
Merge main changes into dev branch

## Type of Change
<!-- Please check the relevant option -->
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Other (please specify):

## Screenshots/Videos (if applicable)
<!-- Add screenshots or videos to help explain your changes -->

## Pre-submission Checklist
<!-- Please check all boxes that apply before submitting your PR -->
- [ ] **I have tested my changes thoroughly before submitting this PR**
- [ ] **This PR contains minimal changes necessary to address the
issue/feature**
- [ ] My code follows the project's coding standards and style
guidelines
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added necessary documentation (if applicable)
- [ ] All new and existing tests pass
- [ ] I have searched existing PRs to ensure this change hasn't been
submitted already
- [ ] I have linked any relevant issues in the description
- [ ] My commits have clear and descriptive messages

## DCO Affirmation
I affirm that all code in every commit of this pull request conforms to
the terms of the Topoteretes Developer Certificate of Origin.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Overhauled README: renamed tagline, clarified product positioning,
reorganized Get Started into Open Source and Cloud paths, streamlined
Quickstart, refreshed demos, navigation, and citation/contributing
sections.

* **New Features**
* Added MCP tools for developer rules management, interaction logging,
and enhanced search modes (SUMMARIES, CYPHER, FEELING_LUCKY).

* **Bug Fixes**
  * Improved embedding handling to support alternate response formats.

* **Chores**
  * Updated test/dev dependency versions.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-04 17:48:11 +01:00
Pavel Zorin
386e0c8234
chore: uv lock check in pre-test workflow (#1773)
<!-- .github/pull_request_template.md -->
`.github/actions/cognee_setup/action.yml` - makes `uv.lock` rebuild
disabled by default.
`.github/workflows/pre_test.yml` - lightweight checks that signal that
PR is not ready for testing. For that moment it verifies that `uv.lock`
corresponds to `pyproject.toml`. For the case of failure the run
provides the instructions how to fix the `uv.lock`

Warning: made an exclusion for python 3.13 dependencies. This doesn't
look good to me

## Description
Make
## Type of Change
<!-- Please check the relevant option -->
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Other (please specify): CI improvement

## Screenshots/Videos (if applicable)
<!-- Add screenshots or videos to help explain your changes -->

## Pre-submission Checklist
<!-- Please check all boxes that apply before submitting your PR -->
- [ ] **I have tested my changes thoroughly before submitting this PR**
- [ ] **This PR contains minimal changes necessary to address the
issue/feature**
- [ ] My code follows the project's coding standards and style
guidelines
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added necessary documentation (if applicable)
- [ ] All new and existing tests pass
- [ ] I have searched existing PRs to ensure this change hasn't been
submitted already
- [ ] I have linked any relevant issues in the description
- [ ] My commits have clear and descriptive messages

## DCO Affirmation
I affirm that all code in every commit of this pull request conforms to
the terms of the Topoteretes Developer Certificate of Origin.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
  * Made uv lockfile rebuilding optional in CI (disabled by default).
* Added a pre-validation workflow to check lockfile integrity before
tests.
* Ensured pre-test validation runs before basic and end-to-end test
jobs.
* Updated dependencies to support Python 3.13 with version-specific lxml
constraints and adjusted docs dependency structure.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-04 15:50:39 +01:00
Pavel Zorin
7033b63cd4 update poetry lock 2025-12-04 13:26:53 +01:00
Pavel Zorin
6d1f1d183a Removed python-version from pre-test 2025-12-04 11:51:32 +01:00
Pavel Zorin
ed0c0d1823 Potential fix for code scanning alert no. 407: Workflow does not contain permissions
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-04 11:51:32 +01:00
Pavel Zorin
a50810240b Vesion exclusion for lxml and python 3.13 2025-12-04 11:51:28 +01:00
Pavel Zorin
c7a2138966 CI: uv lock check. Pre-test workflow. 2025-12-04 11:47:35 +01:00
Igor Ilic
7d7f8a249a
Merge branch 'dev' into main-merge-vol4 2025-12-04 10:32:10 +01:00
Igor Ilic
f1c5b9a55f fix: Resolve DB caching issues when deleting databases 2025-12-03 18:05:47 +01:00
Igor Ilic
fd84edeb74 refactor: change getting of tables during deletion 2025-12-03 15:43:41 +01:00
Boris
8cad9ef225
Merge branch 'dev' into feature/cog-3409-add-bedrock-as-supported-llm-provider 2025-12-03 14:58:00 +01:00
Igor Ilic
45f32f8bfd
Merge branch 'dev' into multi-tenant-neo4j 2025-12-03 14:37:13 +01:00
Igor Ilic
1961efcc33 fix: Handle scenario when there is no relational database on prune time 2025-12-03 14:27:06 +01:00
Igor Ilic
f4078d1247 feat: Add ability to delete lance and kuzu datasets, add prune to work with multi user mode 2025-12-03 13:10:18 +01:00
Igor Ilic
5698c609f5 test: Update tests with regards to auto scaling changes 2025-12-03 11:47:10 +01:00
Boris Arzentar
0d2e84f58e
test: test_strip_quotes_from_strings 2025-12-03 10:59:17 +01:00
Boris
3288ef01a4
Merge branch 'dev' into fix/remove-double-quotes-from-llmconfig-str-params 2025-12-03 10:05:49 +01:00
hajdul88
d4d190ac2b
feature: adds triplet embedding via memify (#1832)
<!-- .github/pull_request_template.md -->

## Description
This PR introduces triplet embeddings via a new
create_triplet_embeddings memify pipeline.
The pipeline reads the graph in batches, extracts properties from graph
elements based on their datapoint types, and generates combined triplet
embeddings. These embeddings are stored in the vector database as a new
collection.

Changes in This PR:

-Added a new create_triplet_embeddings memify pipeline.
-Added a new get_triplet_datapoints memify task.
-Introduced a new triplet_completion search type.
-Added full test coverage
--Unit tests: memify task, pipeline, and retriever
--Integration tests: memify task, pipeline, and retriever
--End-to-end tests: updated session history tests and multi-DB search
tests; added tests for triplet_completion and memify pipeline execution

Acceptance Criteria and Testing
Scenario 1:
-Run default add, cognify pipelines
-Run create triplet embeddings memify pipeline
-Verify the vector DB contains a non empty Triplet_text collection.
-Use the new triplet_completion search type and confirm it works
correctly.

Scenario 2:
-Run the default add and cognify pipelines.
-Do not run the triplet embeddings memify pipeline.
-Attempt to use the triplet_completion search type.
-You should receive an error indicating that the triplet embeddings
memify pipeline must be executed first.


## Type of Change
<!-- Please check the relevant option -->
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Other (please specify):

## Screenshots/Videos (if applicable)
<!-- Add screenshots or videos to help explain your changes -->

## Pre-submission Checklist
<!-- Please check all boxes that apply before submitting your PR -->
- [x] **I have tested my changes thoroughly before submitting this PR**
- [x] **This PR contains minimal changes necessary to address the
issue/feature**
- [x] My code follows the project's coding standards and style
guidelines
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added necessary documentation (if applicable)
- [x] All new and existing tests pass
- [x] I have searched existing PRs to ensure this change hasn't been
submitted already
- [x] I have linked any relevant issues in the description
- [x] My commits have clear and descriptive messages

## DCO Affirmation
I affirm that all code in every commit of this pull request conforms to
the terms of the Topoteretes Developer Certificate of Origin.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Triplet-based search with LLM-powered completions (TRIPLET_COMPLETION)
* Batch triplet retrieval and a triplet embeddings pipeline for
extraction, indexing, and optional background processing
* Context retrieval from triplet embeddings with optional caching and
conversation-history support
  * New Triplet data type exposed for indexing and search

* **Examples**
* End-to-end example demonstrating triplet embeddings extraction and
TRIPLET_COMPLETION search

* **Tests**
* Unit and integration tests covering triplet extraction, retrieval,
embedding pipeline, and completion flows

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Pavel Zorin <pazonec@yandex.ru>
2025-12-02 18:27:08 +01:00
Igor Ilic
4e8b2ffc3e chore: Update lock files 2025-12-02 17:51:30 +01:00
Igor Ilic
c7810e9fdb Merge branch 'dev' into main-merge-vol4 2025-12-02 17:40:09 +01:00
Igor Ilic
1282905888 feat: add password encryption for Neo4j 2025-12-02 16:34:16 +01:00
Igor Ilic
2d45db9e0d
Fix distributed issues with latest pydantic version (#1859)
<!-- .github/pull_request_template.md -->

## Description
Resolve distributed issues with poetry lock

## Type of Change
<!-- Please check the relevant option -->
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Other (please specify):

## Screenshots/Videos (if applicable)
<!-- Add screenshots or videos to help explain your changes -->

## Pre-submission Checklist
<!-- Please check all boxes that apply before submitting your PR -->
- [ ] **I have tested my changes thoroughly before submitting this PR**
- [ ] **This PR contains minimal changes necessary to address the
issue/feature**
- [ ] My code follows the project's coding standards and style
guidelines
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added necessary documentation (if applicable)
- [ ] All new and existing tests pass
- [ ] I have searched existing PRs to ensure this change hasn't been
submitted already
- [ ] I have linked any relevant issues in the description
- [ ] My commits have clear and descriptive messages

## DCO Affirmation
I affirm that all code in every commit of this pull request conforms to
the terms of the Topoteretes Developer Certificate of Origin.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated dependency version constraints to improve compatibility and
flexibility with Pydantic and aiofiles packages.
* Removed unused development dependency from the project configuration.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-02 16:08:26 +01:00
Igor Ilic
92448767fe refactor: remove done TODOs 2025-12-02 14:29:51 +01:00
Igor Ilic
702cdb45be
Merge branch 'dev' into multi-tenant-neo4j 2025-12-02 13:11:37 +01:00
Igor Ilic
dbcb35a6da chore: remove unused imports, add optional for delete dataset statement 2025-12-02 13:09:45 +01:00
Boris Arzentar
0ff836b6dd
fix: install latest nvm version 2025-12-02 10:48:28 +01:00
Boris Arzentar
5fe6a17cfd
fix: resolve nvm when not in path 2025-12-02 10:43:57 +01:00
Boris Arzentar
5ee5ae294a
Merge remote-tracking branch 'origin/dev' into feature/cog-3441-cognee-cli-ui-fix 2025-12-01 20:23:01 +01:00
Andrej Milicevic
d473ef12ae fix: small changes based on PR comments 2025-12-01 18:32:55 +01:00
Igor Ilic
8e67471d1e
Merge branch 'dev' into main-merge-vol4 2025-12-01 17:43:46 +01:00
Vasilije
c17f838034
CI: 32 GB machine for Ollama tests (#1857)
<!-- .github/pull_request_template.md -->

## Description
Recently the Llama test became failing with `model requires more system
memory (8.9 GiB) than is available (8.4 GiB)`. Due to `cgroup`
configuration, only 8 GBs are available for containers running on
`buildjet-4vcpu-ubuntu-2204`.
The decision is to change the the machine to
`buildjet-8vcpu-ubuntu-2204`. it costs 0.0016 $ per minute.

Unconfidently changed the model to `phi3:mini`. Any other ideas are
welcome.
<!--
Please provide a clear, human-generated description of the changes in
this PR.
DO NOT use AI-generated descriptions. We want to understand your thought
process and reasoning.
-->

## Type of Change
<!-- Please check the relevant option -->
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Other (please specify):

## Screenshots/Videos (if applicable)
<!-- Add screenshots or videos to help explain your changes -->

## Pre-submission Checklist
<!-- Please check all boxes that apply before submitting your PR -->
- [ ] **I have tested my changes thoroughly before submitting this PR**
- [ ] **This PR contains minimal changes necessary to address the
issue/feature**
- [ ] My code follows the project's coding standards and style
guidelines
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added necessary documentation (if applicable)
- [ ] All new and existing tests pass
- [ ] I have searched existing PRs to ensure this change hasn't been
submitted already
- [ ] I have linked any relevant issues in the description
- [ ] My commits have clear and descriptive messages

## DCO Affirmation
I affirm that all code in every commit of this pull request conforms to
the terms of the Topoteretes Developer Certificate of Origin.
2025-12-01 08:01:24 -08:00
Boris
2df84dba27
Merge branch 'dev' into fix/remove-double-quotes-from-llmconfig-str-params 2025-12-01 16:03:00 +01:00
Igor Ilic
5cfc7b1761 chore: Disable backend access control when not supported 2025-12-01 15:58:19 +01:00
Pavel Zorin
e480acaa7c
COG-3437: Chore: CodeRabbit config (#1833)
## Description
Adds `.coderabbit.yml` that tunes CodeRabbit reviews
## Type of Change
<!-- Please check the relevant option -->
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Other (please specify): chore

## Screenshots/Videos (if applicable)
<!-- Add screenshots or videos to help explain your changes -->

## Pre-submission Checklist
<!-- Please check all boxes that apply before submitting your PR -->
- [ ] **I have tested my changes thoroughly before submitting this PR**
- [ ] **This PR contains minimal changes necessary to address the
issue/feature**
- [ ] My code follows the project's coding standards and style
guidelines
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added necessary documentation (if applicable)
- [ ] All new and existing tests pass
- [ ] I have searched existing PRs to ensure this change hasn't been
submitted already
- [ ] I have linked any relevant issues in the description
- [ ] My commits have clear and descriptive messages

## DCO Affirmation
I affirm that all code in every commit of this pull request conforms to
the terms of the Topoteretes Developer Certificate of Origin.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Added a project-wide review and tooling configuration to standardize
reviews, path-specific guidance, automated incremental reviews, chat
auto-replies, and integrations with linters/validators.
* Configured review behaviors (auto-review, abort-on-close, high-level
summaries, placeholders) and path filtering to focus checks where
needed.

* **Note**
  * No user-visible changes.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-01 15:29:49 +01:00
Pavel Zorin
ba9ca46574 Increase the machine size 2025-12-01 15:23:59 +01:00
Igor Ilic
362aa8df5c
Merge branch 'main' into baml-rate-limit-handling 2025-12-01 15:12:27 +01:00
Igor Ilic
2e493cea4c chore: Disable multi user mode for tests that can't run it 2025-12-01 15:07:01 +01:00
Igor Ilic
524e3e8232 chore: Update lock files 2025-12-01 14:52:01 +01:00
Pavel Zorin
7c9a78abea CI: Smaller embedding model for Ollama test 2025-12-01 14:39:45 +01:00
Igor Ilic
859e98b494 fix: resolve issue with poetry and dev dependency 2025-12-01 13:47:16 +01:00
Boris
76d054b6a5
Merge branch 'dev' into feature/cog-3156-move-codify-pipeline-out-of-main-repo 2025-12-01 11:21:34 +01:00
Igor Ilic
0bb4ece4d8 Merge branch 'main' into main-merge-vol4 2025-12-01 11:16:59 +01:00
Boris
5ce1af8cc0
Merge branch 'dev' into fix/remove-double-quotes-from-llmconfig-str-params 2025-12-01 10:09:53 +01:00
Igor Ilic
d81d63390f test: Add test for dataset database handler creation 2025-11-28 16:33:46 +01:00
Igor Ilic
a0c5867977 chore: disable backend access control 2025-11-28 14:56:33 +01:00
Igor Ilic
7e0be8f167 chore: disable backend access control for tests not supporting mode 2025-11-28 13:48:01 +01:00
Igor Ilic
7844b9a3a5 Merge branch 'multi-tenant-neo4j' of github.com:topoteretes/cognee into multi-tenant-neo4j 2025-11-28 13:12:07 +01:00
Igor Ilic
ed9b774448 chore: disable backend access control for deduplication test 2025-11-28 13:11:45 +01:00
Igor Ilic
0c825b96ff
Merge branch 'dev' into multi-tenant-neo4j 2025-11-28 12:55:48 +01:00
Vasilije
00b60aed6c
backport: Adds lance-namespace version fix to toml (fixes lancedb issue with 0.2.0 lance-namespace version) + crawler ingetration test url fix (#1842)
<!-- .github/pull_request_template.md -->

## Description
Implements a quick fix for the lance-namespace 0.0.21 to 0.2.0 release
issue with lancedb. Later this has to be revisited if they fix it on
their side, for now we fixed the lance-namespace version to the previous
one.


**If Lancedb fixes the issue on their side this can be closed**


Additionally cherry picking crawler integration test fixes from dev

## Type of Change
<!-- Please check the relevant option -->
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Other (please specify):

## Screenshots/Videos (if applicable)
<!-- Add screenshots or videos to help explain your changes -->

## Pre-submission Checklist
<!-- Please check all boxes that apply before submitting your PR -->
- [ ] **I have tested my changes thoroughly before submitting this PR**
- [ ] **This PR contains minimal changes necessary to address the
issue/feature**
- [ ] My code follows the project's coding standards and style
guidelines
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added necessary documentation (if applicable)
- [ ] All new and existing tests pass
- [ ] I have searched existing PRs to ensure this change hasn't been
submitted already
- [ ] I have linked any relevant issues in the description
- [ ] My commits have clear and descriptive messages

## DCO Affirmation
I affirm that all code in every commit of this pull request conforms to
the terms of the Topoteretes Developer Certificate of Origin.
2025-11-27 10:47:00 -08:00