Implement comprehensive trace logging system for agent execution that
returns step-by-step execution traces in API responses.
New modules:
- agent/trace/trace_models.py: Data models for trace events, sessions,
LLM calls, retrievals, and tool calls
- agent/trace/trace_collector.py: Real-time trace event collection with
subscriber pattern for streaming
- agent/trace/trace_formatter.py: Multiple formatters (streaming, compact,
detailed) for different output needs
- api/db/services/trace_service.py: Service layer for trace persistence,
retrieval, and analysis
- api/apps/trace_app.py: REST API endpoints for trace management
Features:
- Real-time trace streaming via SSE
- Multiple trace verbosity levels (minimal, standard, detailed, debug)
- Component execution timing and bottleneck detection
- LLM call tracking with token counts
- Retrieval operation logging with chunk details
- Tool call tracing with arguments and results
- Trace session persistence in Redis
- Analysis and recommendations based on trace data
API Endpoints:
- GET /traces - List trace sessions
- GET /traces/<task_id> - Get trace session
- GET /traces/<task_id>/events - Get filtered events
- GET /traces/<task_id>/summary - Get trace summary
- GET /traces/<task_id>/analysis - Analyze trace
- GET /traces/<task_id>/stream - Stream trace events
- DELETE /traces/<task_id> - Delete trace
- POST /traces/cleanup - Cleanup old traces
- POST /agents/<agent_id>/completions/trace - Completion with trace
Closes#10081
697f8138b6