LightRAG/tests/gpt5_nano_compatibility/README.md

# GPT-5-Nano Compatibility Tests

This directory contains comprehensive tests for ensuring LightRAG's compatibility with OpenAI's gpt-5-nano model, including its specific API constraints and parameter requirements.

## Overview

gpt-5-nano is a cost-optimized reasoning model that differs from traditional LLMs in important ways:
- Uses `max_completion_tokens` instead of `max_tokens`
- Does NOT support custom `temperature` parameter
- Has built-in reasoning that consumes tokens from the completion budget
- Requires token budget adjustments to account for reasoning overhead

These tests validate that LightRAG handles these constraints correctly.

## Test Files

### 1. `test_gpt5_nano_compatibility.py` ⭐ Primary Test Suite
**Purpose:** Comprehensive compatibility validation
**Tests:**
- Test 1: Parameter normalization (max_tokens → max_completion_tokens conversion)
- Test 2: Configuration loading from .env
- Test 3: Embeddings generation with gpt-5-nano
- Test 4: Simple LLM completion
- Test 5: Entity extraction tasks

**Run:** `python test_gpt5_nano_compatibility.py`

**Expected Output:**
```
✅ Parameter Normalization: PASSED
✅ Configuration Loading: PASSED
✅ Embeddings: PASSED
✅ Simple Completion: PASSED
✅ Entity Extraction: PASSED
🎉 ALL TESTS PASSED
```

### 2. `test_env_config.py`
**Purpose:** Validate .env configuration is properly respected
**Tests:**
- Part 1: .env file loading
- Part 2: Config parser respects .env variables
- Part 3: OpenAI API connectivity
- Part 4: Embeddings generation with configured model
- Part 5: LLM extraction with configured model
- Part 6: Full RAG pipeline integration

**Run:** `python test_env_config.py`

**Expected Output:**
```
✅ .env Loading: PASSED
✅ Config Parser: PASSED
✅ OpenAI Connectivity: PASSED
✅ Embeddings: PASSED
✅ LLM Extraction: PASSED
✅ Full Integration: PASSED
OVERALL: 6/6 tests passed
```

### 3. `test_direct_gpt5nano.py`
**Purpose:** Direct API testing without LightRAG abstraction
**Validates:** Raw gpt-5-nano API behavior with proper parameters

**Run:** `python test_direct_gpt5nano.py`

**What it does:**
- Sends direct API request to gpt-5-nano
- Uses `max_completion_tokens` parameter
- Prints raw response and token usage

### 4. `test_gpt5_reasoning.py`
**Purpose:** Understand gpt-5-nano's reasoning token overhead
**Tests:** Token allocation with different reasoning effort levels

**Run:** `python test_gpt5_reasoning.py`

**What it does:**
- Test 1: 200 token budget
- Test 2: 50 token budget with `reasoning_effort="low"`
- Outputs actual reasoning tokens consumed

## Prerequisites

### Environment Variables
Create a `.env` file in the repository root with:

```env
# Required for all tests
OPENAI_API_KEY=sk-...

# For LLM tests
LLM_BINDING=openai
LLM_MODEL=gpt-5-nano
LLM_BINDING_API_KEY=sk-...

# For embedding tests
EMBEDDING_BINDING=openai
EMBEDDING_MODEL=text-embedding-3-small
EMBEDDING_BINDING_API_KEY=sk-...
EMBEDDING_DIM=1536
```

Or use existing `.env` configuration if already set up.

### Python Dependencies
```bash
pip install openai
pip install python-dotenv
pip install lightrag  # for integration tests
```

## Running All Tests

### From this directory:
```bash
# Run individual test
python test_gpt5_nano_compatibility.py

# Or run all tests
for test in test_*.py; do 
    echo "Running $test..."
    python "$test"
done
```

### From repository root:
```bash
# Run specific test
python -m pytest tests/gpt5_nano_compatibility/test_gpt5_nano_compatibility.py -v

# Or run all tests in this directory
python -m pytest tests/gpt5_nano_compatibility/ -v
```

## Key Findings & Implementation

### Problem: Parameter Incompatibility
gpt-5-nano requires different parameter names and constraints than other OpenAI models.

**Issue:** 
- Other models use `max_tokens`
- gpt-5-nano requires `max_completion_tokens`

**Solution:**
A normalization function `_normalize_openai_kwargs_for_model()` in `/lightrag/llm/openai.py` that:
1. Detects gpt-5 models
2. Converts `max_tokens` → `max_completion_tokens`
3. Applies 2.5x token multiplier (minimum 300 tokens) to account for reasoning overhead
4. Removes unsupported `temperature` parameter

### Problem: Empty Responses
gpt-5-nano was returning empty responses despite successful API calls.

**Root Cause:**
Internal reasoning consumes tokens from the completion budget. With insufficient token budget, all tokens were consumed by reasoning, leaving nothing for actual output.

**Solution:**
Empirical testing showed that:
- 200 tokens: Often empty responses
- 300+ tokens: Consistent full responses
- 2.5x multiplier: Provides adequate margin for reasoning

### Parameter Handling

**For gpt-5-nano models:**
```python
# Before normalization:
{"max_tokens": 500, "temperature": 0.7}

# After normalization:
{"max_completion_tokens": 1250}  # 500 * 2.5, min 300
```

**For other models:**
```python
# Unchanged
{"max_tokens": 500, "temperature": 0.7}
```

## Test Results Summary

All tests validate:
- ✅ Parameter normalization works correctly
- ✅ gpt-5-nano parameter constraints are handled
- ✅ Backward compatibility maintained (other models unaffected)
- ✅ Configuration from .env is respected
- ✅ OpenAI API integration functions properly
- ✅ Embeddings generation works
- ✅ Entity extraction works with gpt-5-nano
- ✅ Full RAG pipeline integration successful

## Troubleshooting

### "OPENAI_API_KEY not set"
- Ensure `.env` file exists in repository root
- Verify `OPENAI_API_KEY` is set: `echo $OPENAI_API_KEY`

### "max_tokens unsupported with this model"
- This error means parameter normalization isn't being called
- Check that you're using LightRAG functions (not direct OpenAI client)
- Verify the normalization function is in `/lightrag/llm/openai.py`

### "Empty API responses"
- Increase token budget (tests use 100+ tokens)
- If using custom token limits, multiply by 2.5 minimum

### "temperature does not support 0.7"
- gpt-5-nano doesn't accept custom temperature
- The normalization function removes it automatically
- No action needed if using LightRAG functions

## Documentation

For more details, see:
- `/docs/GPT5_NANO_COMPATIBILITY.md` - User guide
- `/docs/GPT5_NANO_COMPATIBILITY_IMPLEMENTATION.md` - Technical implementation details

## Related Files

- `/lightrag/llm/openai.py` - Contains parameter normalization logic
- `/lightrag/llm/azure_openai.py` - Azure OpenAI integration with same normalization
- `/.env` - Configuration file (use `.env.example` as template)

## Maintenance Notes

When updating LightRAG's OpenAI integration:
1. Run all tests to ensure backward compatibility
2. If adding new OpenAI models, test with gpt-5-nano constraints
3. Update parameter normalization logic if OpenAI adds new gpt-5 variants
4. Keep `max_tokens * 2.5` strategy unless OpenAI documents different reasoning overhead

---

**Last Updated:** 2024
**Status:** All tests passing ✅
**Model Tested:** gpt-5-nano
**OpenAI SDK:** Latest (with max_completion_tokens support)