cognee/.github/workflows/disable_independent_workflows.sh
Chaitany 96eb0d448a
feat(#1357): Lexical chunk retriever (#1392)
<!-- .github/pull_request_template.md -->

## Description
<!-- 
Please provide a clear, human-generated description of the changes in
this PR.
DO NOT use AI-generated descriptions. We want to understand your thought
process and reasoning.
-->
I Implemented Lexical Chunk Retriever In the LexicalRetriever class is
Inherite the BaseRetriever and The DocumentChunk are lazy loaded when
first time query is made because it save time during object
initialization
and the function get_context and the get_completion are Implemented same
as the ChunksRetriever the only diffrence is that the DocumentChunk are
converted to match the output type of the ChunksRetriever using function
get_own_properties in the utils.

## Type of Change
<!-- Please check the relevant option -->
- [-] Bug fix (non-breaking change that fixes an issue)
- [-] New feature (non-breaking change that adds functionality)
- [-] Breaking change (fix or feature that would cause existing
functionality to change)
- [-] Documentation update
- [-] Code refactoring
- [-] Performance improvement
- [-] Other (please specify):

## Changes Made
<!-- List the specific changes made in this PR -->
- Added LexicalRetriever base class with customizable tokenizer & scorer
     - Implemented caching of DocumentChunk tokens and payloads 
- Added robust initialization with error handling and logging -
Implemented get_context with top_k ranking and optional scores
- Implemented get_completion consistent with BaseRetriever interface
- Added JaccardChunksRetriever demo using set/multiset Jaccard
similarity
- Support for stopwords and multiset frequency-aware similarity -
Integrated logging for initialization, scoring, and retrieval

## Testing

- Manual tests: initialized retriever, retrieved chunks with toy corpus
    - Edge cases: empty corpus, empty query, scorer/tokenizer errors 
    - Verified Jaccard similarity results for single/multiset cases 
    - Code formatted and linted


## Screenshots/Videos (if applicable)
<!-- Add screenshots or videos to help explain your changes -->

## Pre-submission Checklist
<!-- Please check all boxes that apply before submitting your PR -->
- [-] **I have tested my changes thoroughly before submitting this PR**
- [-] **This PR contains minimal changes necessary to address the
issue/feature**
- [-] My code follows the project's coding standards and style
guidelines
- [-] I have added tests that prove my fix is effective or that my
feature works
- [-] I have added necessary documentation (if applicable)
- [-] All new and existing tests pass
- [-] I have searched existing PRs to ensure this change hasn't been
submitted already
- [-] I have linked any relevant issues in the description
- [-] My commits have clear and descriptive messages

## Related Issues
<!-- Link any related issues using "Fixes #issue_number" or "Relates to
#issue_number" -->
Relates to  #1392
## Additional Notes
<!-- Add any additional notes, concerns, or context for reviewers -->
Int the cognee/modules/chunking/models/DocumentChunk.py
don't remove the optional  from is_part_of attributes.

## DCO Affirmation
I affirm that all code in every commit of this pull request conforms to
the terms of the Topoteretes Developer Certificate of Origin.

---------

Co-authored-by: Andrej Milicevic <milicevicandrej@yahoo.com>
Co-authored-by: Igor Ilic <30923996+dexters1@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Igor Ilic <igorilic03@gmail.com>
Co-authored-by: Vasilije <8619304+Vasilije1990@users.noreply.github.com>
Co-authored-by: Boris <boris@topoteretes.com>
Co-authored-by: lxobr <122801072+lxobr@users.noreply.github.com>
2025-09-19 18:24:33 +02:00

84 lines
No EOL
2.4 KiB
Bash
Executable file

#!/bin/bash
# Navigate to the workflows directory
cd "$(dirname "$0")"
# List of workflows that should only be triggered via test-suites.yml
WORKFLOWS=(
"test_chromadb.yml"
"test_weaviate.yml"
"test_kuzu.yml"
"test_multimetric_qa_eval_run.yaml"
"test_graphrag_vs_rag_notebook.yml"
"test_llms.yml"
"test_multimedia_example.yaml"
"test_deduplication.yml"
"test_eval_framework.yml"
"test_descriptive_graph_metrics.yml"
"test_llama_index_cognee_integration_notebook.yml"
"test_cognee_llama_index_notebook.yml"
"test_cognee_multimedia_notebook.yml"
"test_cognee_server_start.yml"
"test_telemetry.yml"
"test_neo4j.yml"
"test_pgvector.yml"
"test_ollama.yml"
"test_notebook.yml"
"test_simple_example.yml"
"test_code_graph_example.yml"
)
for workflow in "${WORKFLOWS[@]}"; do
if [ -f "$workflow" ]; then
echo "Processing $workflow..."
# Create a backup
cp "$workflow" "${workflow}.bak"
# Check if the file begins with a workflow_call trigger
if grep -q "workflow_call:" "$workflow"; then
echo "$workflow already has workflow_call trigger, skipping..."
continue
fi
# Get the content after the 'on:' section
on_line=$(grep -n "^on:" "$workflow" | cut -d ':' -f1)
if [ -z "$on_line" ]; then
echo "Warning: No 'on:' section found in $workflow, skipping..."
continue
fi
# Create a new file with the modified content
{
# Copy the part before 'on:'
head -n $((on_line-1)) "$workflow"
# Add the new on: section that only includes workflow_call
echo "on:"
echo " workflow_call:"
echo " secrets:"
echo " inherit: true"
# Find where to continue after the original 'on:' section
next_section=$(awk "NR > $on_line && /^[a-z]/ {print NR; exit}" "$workflow")
if [ -z "$next_section" ]; then
next_section=$(wc -l < "$workflow")
next_section=$((next_section+1))
fi
# Copy the rest of the file starting from the next section
tail -n +$next_section "$workflow"
} > "${workflow}.new"
# Replace the original with the new version
mv "${workflow}.new" "$workflow"
echo "Modified $workflow to only run when called from test-suites.yml"
else
echo "Warning: $workflow not found, skipping..."
fi
done
echo "Finished modifying workflows!"