4 commits

Author	SHA1	Message	Date
Igor Ilic	0a9f1349f2	refactor: Change variable and function names based on PR comments ... Change variable and function names based on PR comments	2025-01-28 10:10:29 +01:00
Igor Ilic	7dea1d54d7	refactor: Add specific max token values to embedding models	2025-01-23 18:18:45 +01:00
Igor Ilic	294ed1d960	feat: Add HuggingFace Tokenizer support	2025-01-23 16:52:35 +01:00
Igor Ilic	93249c72c5	fix: Initial commit to resolve issue with using tokenizer based on LLMs ... Currently TikToken is used for tokenizing by default which is only supported by OpenAI, this is an initial commit in an attempt to add Cognee tokenizing support for multiple LLMs	2025-01-21 19:53:22 +01:00