Currently TikToken is used for tokenizing by default which is only supported by OpenAI, this is an initial commit in an attempt to add Cognee tokenizing support for multiple LLMs |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| config.py | ||
Currently TikToken is used for tokenizing by default which is only supported by OpenAI, this is an initial commit in an attempt to add Cognee tokenizing support for multiple LLMs |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| config.py | ||