8879f3fbbe
<!-- .github/pull_request_template.md -->
## Description
PR to test Gemini PR from holchan
1. Add Gemini LLM and Gemini Embedding support
2. Fix CodeGraph issue with chunks being bigger than maximum token value
3. Add Tokenizer adapters to CodeGraph
## DCO Affirmation
I affirm that all code in every commit of this pull request conforms to
the terms of the Topoteretes Developer Certificate of Origin
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added support for the Gemini LLM provider.
- Expanded LLM configuration options.
- Introduced a new GitHub Actions workflow for multimetric QA
evaluation.
- Added new environment variables for LLM and embedding configurations
across various workflows.
- **Bug Fixes**
- Improved error handling in various components.
- Updated tokenization and embedding processes.
- Removed warning related to missing `dict` method in data items.
- **Refactor**
- Simplified token extraction and decoding methods.
- Updated tokenizer interfaces.
- Removed deprecated dependencies.
- Enhanced retry logic and error handling in embedding processes.
- **Documentation**
- Updated configuration comments and settings.
- **Chores**
- Updated GitHub Actions workflows to accommodate new secrets and
environment variables.
- Modified evaluation parameters.
- Adjusted dependency management for optional libraries.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: holchan <61059652+holchan@users.noreply.github.com>
Co-authored-by: Boris <boris@topoteretes.com>
83 lines
2.1 KiB
YAML
83 lines
2.1 KiB
YAML
name: test | python 3.11
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
pull_request:
|
|
types: [labeled, synchronize]
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
env:
|
|
RUNTIME__LOG_LEVEL: ERROR
|
|
ENV: 'dev'
|
|
|
|
jobs:
|
|
|
|
run_common:
|
|
name: test
|
|
runs-on: ubuntu-22.04
|
|
strategy:
|
|
fail-fast: false
|
|
defaults:
|
|
run:
|
|
shell: bash
|
|
|
|
steps:
|
|
- name: Check out
|
|
uses: actions/checkout@master
|
|
|
|
- name: Setup Python
|
|
uses: actions/setup-python@v5
|
|
with:
|
|
python-version: '3.11.x'
|
|
|
|
- name: Install Poetry
|
|
# https://github.com/snok/install-poetry#running-on-windows
|
|
uses: snok/install-poetry@v1.4.1
|
|
with:
|
|
virtualenvs-create: true
|
|
virtualenvs-in-project: true
|
|
installer-parallel: true
|
|
|
|
- name: Install dependencies
|
|
run: poetry install --no-interaction -E docs
|
|
|
|
- name: Download NLTK tokenizer data
|
|
run: |
|
|
poetry run python -m nltk.downloader punkt_tab averaged_perceptron_tagger_eng
|
|
|
|
|
|
- name: Run unit tests
|
|
run: poetry run pytest cognee/tests/unit/
|
|
|
|
- name: Run integration tests
|
|
run: poetry run pytest cognee/tests/integration/
|
|
|
|
- name: Run default basic pipeline
|
|
env:
|
|
ENV: 'dev'
|
|
LLM_MODEL: ${{ secrets.LLM_MODEL }}
|
|
LLM_ENDPOINT: ${{ secrets.LLM_ENDPOINT }}
|
|
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
|
|
LLM_API_VERSION: ${{ secrets.LLM_API_VERSION }}
|
|
EMBEDDING_MODEL: ${{ secrets.EMBEDDING_MODEL }}
|
|
EMBEDDING_ENDPOINT: ${{ secrets.EMBEDDING_ENDPOINT }}
|
|
EMBEDDING_API_KEY: ${{ secrets.EMBEDDING_API_KEY }}
|
|
EMBEDDING_API_VERSION: ${{ secrets.EMBEDDING_API_VERSION }}
|
|
run: poetry run python ./cognee/tests/test_library.py
|
|
|
|
- name: Clean up disk space
|
|
run: |
|
|
sudo rm -rf ~/.cache
|
|
sudo rm -rf /tmp/*
|
|
df -h
|
|
|
|
- name: Build with Poetry
|
|
run: poetry build
|
|
|
|
- name: Install Package
|
|
run: |
|
|
cd dist
|
|
pip install *.whl
|