graphiti/PROVIDER_CONFIGURATION.md

Provider Configuration System

This document describes the new provider configuration system that replaces hardcoded "ghost variables" with configurable defaults.

Overview

Previously, each LLM provider client had hardcoded model names and configuration values that could not be easily customized without modifying the source code. This created several issues:

1. Maintenance burden: Updating to newer models required code changes
2. Limited flexibility: Users couldn't easily switch to different models
3. Provider constraints: Some models (like Gemini's flash-lite) have specific location constraints that differed from defaults
4. Hidden configurations: Token limits and other operational parameters were buried in the code

New Configuration System

The new system introduces a centralized provider_defaults.py module that:

1. Centralizes all provider defaults in a single location
2. Supports environment variable overrides for easy customization
3. Maintains backward compatibility with existing configurations
4. Provides provider-specific configurations for different LLM providers

Environment Variables

You can now override any provider default using environment variables with the following pattern:

# For OpenAI
export OPENAI_DEFAULT_MODEL="gpt-4"
export OPENAI_DEFAULT_SMALL_MODEL="gpt-4-mini"
export OPENAI_DEFAULT_MAX_TOKENS="8192"
export OPENAI_DEFAULT_TEMPERATURE="0.0"
export OPENAI_EXTRACT_EDGES_MAX_TOKENS="16384"

# For Gemini
export GEMINI_DEFAULT_MODEL="gemini-2.5-flash"
export GEMINI_DEFAULT_SMALL_MODEL="gemini-2.5-flash-lite"
export GEMINI_DEFAULT_MAX_TOKENS="8192"
export GEMINI_DEFAULT_TEMPERATURE="0.0"
export GEMINI_EXTRACT_EDGES_MAX_TOKENS="16384"

# For Anthropic
export ANTHROPIC_DEFAULT_MODEL="claude-3-5-sonnet-latest"
export ANTHROPIC_DEFAULT_SMALL_MODEL="claude-3-5-haiku-latest"
export ANTHROPIC_DEFAULT_MAX_TOKENS="8192"
export ANTHROPIC_DEFAULT_TEMPERATURE="0.0"
export ANTHROPIC_EXTRACT_EDGES_MAX_TOKENS="16384"

# For Groq
export GROQ_DEFAULT_MODEL="llama-3.1-70b-versatile"
export GROQ_DEFAULT_SMALL_MODEL="llama-3.1-8b-instant"
export GROQ_DEFAULT_MAX_TOKENS="8192"
export GROQ_DEFAULT_TEMPERATURE="0.0"
export GROQ_EXTRACT_EDGES_MAX_TOKENS="16384"

# General configuration (for edge operations)
export EXTRACT_EDGES_MAX_TOKENS="16384"

Supported Providers

The system currently supports the following providers:

• openai: OpenAI GPT models
• gemini: Google Gemini models
• anthropic: Anthropic Claude models
• groq: Groq models
• azure_openai: Azure OpenAI models

Usage Examples

Basic Usage

The configuration system works transparently with existing code:

from graphiti_core.llm_client import OpenAIClient
from graphiti_core.llm_client.config import LLMConfig

# Uses default models (configurable via environment variables)
client = OpenAIClient()

# Or with explicit configuration (still uses provider defaults as fallback)
config = LLMConfig(model="gpt-4", small_model="gpt-4-mini")
client = OpenAIClient(config)

Customizing Model Defaults

Instead of hardcoding model names in your application, you can now use environment variables:

# Set up your preferred models
export OPENAI_DEFAULT_MODEL="gpt-4"
export OPENAI_DEFAULT_SMALL_MODEL="gpt-4-mini"

# Your application will automatically use these defaults
python your_app.py

Provider-Specific Configuration

Each provider can have different default models and configurations:

from graphiti_core.llm_client.provider_defaults import get_provider_defaults

# Get defaults for a specific provider
openai_defaults = get_provider_defaults('openai')
print(f"OpenAI default model: {openai_defaults.model}")
print(f"OpenAI small model: {openai_defaults.small_model}")

gemini_defaults = get_provider_defaults('gemini')
print(f"Gemini default model: {gemini_defaults.model}")
print(f"Gemini small model: {gemini_defaults.small_model}")

Migration Guide

Before (with ghost variables)

# In gemini_client.py
DEFAULT_MODEL = 'gemini-2.5-flash'
DEFAULT_SMALL_MODEL = 'models/gemini-2.5-flash-lite-preview-06-17'

def _get_model_for_size(self, model_size: ModelSize) -> str:
    if model_size == ModelSize.small:
        return self.small_model or DEFAULT_SMALL_MODEL
    else:
        return self.model or DEFAULT_MODEL

After (with configurable defaults)

# Configuration is now externalized
from .provider_defaults import get_model_for_size

def _get_model_for_size(self, model_size: ModelSize) -> str:
    return get_model_for_size(
        provider='gemini',
        model_size=model_size.value,
        user_model=self.model,
        user_small_model=self.small_model
    )

Benefits

1. No More Ghost Variables: All defaults are now configurable
2. Easy Model Updates: Update models via environment variables
3. Provider Flexibility: Each provider can have optimized defaults
4. Backward Compatibility: Existing code continues to work unchanged
5. Environment-Specific Configuration: Different environments can use different models
6. Reduced Maintenance: No need to modify source code for model updates

Implementation Details

The new system is implemented in graphiti_core/llm_client/provider_defaults.py and includes:

• ProviderDefaults dataclass for configuration structure
• get_provider_defaults() function with environment variable support
• get_model_for_size() centralized model selection logic
• get_extract_edges_max_tokens_default() for operational parameters

All existing LLM client implementations have been updated to use this new system while maintaining full backward compatibility.