gmakstutis/LightRAG
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add memory management for pipeline history messages

Browse source 
- Trim history at 10k messages
- Keep latest 5k messages
- Prevent memory growth
- Add logging for trim events
This commit is contained in:
yangdx 2025-09-08 15:56:35 +08:00
parent 298037d8f7
commit 9437df83cc
1 changed files with 9 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
lightrag/lightrag.py
View file

@ -1504,6 +1504,15 @@ class LightRAG:
pipeline_status["latest_message"] = log_message pipeline_status["latest_message"] = log_message
pipeline_status["history_messages"].append(log_message) pipeline_status["history_messages"].append(log_message)
# Prevent memory growth: keep only latest 5000 messages when exceeding 10000
if len(pipeline_status["history_messages"]) > 10000:
pipeline_status["history_messages"] = (
pipeline_status["history_messages"][-5000:]
)
logger.info(
f"Trimming pipeline history from {len(pipeline_status['history_messages'])} to 5000 messages"
)
# Get document content from full_docs # Get document content from full_docs
content_data = await self.full_docs.get_by_id(doc_id) content_data = await self.full_docs.get_by_id(doc_id)
if not content_data: if not content_data: