Increase OpenAIGenericClient max_tokens to 16K and update docs

- Set default max_tokens to 16384 (16K) for OpenAIGenericClient to better support local models
- Add documentation note clarifying OpenAIGenericClient should be used for Ollama and LM Studio
- Previous default was 8192 (8K)
This commit is contained in:
Daniel Chalef 2025-11-10 22:39:45 -08:00
parent 552c9d9634
commit 29b04a08fe
2 changed files with 6 additions and 0 deletions

View file

@ -523,6 +523,8 @@ reranker, leveraging Gemini's log probabilities feature to rank passage relevanc
Graphiti supports Ollama for running local LLMs and embedding models via Ollama's OpenAI-compatible API. This is ideal
for privacy-focused applications or when you want to avoid API costs.
**Note:** Use `OpenAIGenericClient` (not `OpenAIClient`) for Ollama and other OpenAI-compatible providers like LM Studio. The `OpenAIGenericClient` is optimized for local models with a higher default max token limit (16K vs 8K) and full support for structured outputs.
Install the models:
```bash

View file

@ -77,6 +77,10 @@ class OpenAIGenericClient(LLMClient):
if config is None:
config = LLMConfig()
# Override max_tokens default to 16K for better compatibility with local models
if config.max_tokens == DEFAULT_MAX_TOKENS:
config.max_tokens = 16384
super().__init__(config, cache)
if client is None: