12ab6ebb42
Implements high-quality entity extraction for three languages using best-in-class tools:
- Chinese: HanLP (F1 95%)
- English: spaCy (F1 90%)
- Swedish: spaCy (F1 80-85%)
**Why not GLiNER?**
Quality gap too large:
- Chinese: 95% vs 24% (-71%)
- English: 90% vs 60% (-30%)
- Swedish: 85% vs 50% (-35%)
**Key Features:**
1. Lazy loading (memory efficient)
- Loads models on-demand
- Only one model in memory at a time (~1.5-1.8 GB)
- Not 4-5 GB simultaneously
2. High quality
- Each language uses optimal tool
- Chinese: HanLP (specialized for Chinese)
- English/Swedish: spaCy (official support)
3. Easy to use
- Simple API: extract(text, language='zh'/'en'/'sv')
- Automatic model management
- Error handling and logging
**Files Added:**
- lightrag/kg/trilingual_entity_extractor.py - Core extractor class
- requirements-trilingual.txt - Dependencies (spacy + hanlp)
- scripts/install_trilingual_models.sh - One-click installation
- scripts/test_trilingual_extractor.py - Comprehensive test suite
- docs/TrilingualNER-Usage-zh.md - Complete usage guide
**Installation:**
```bash
# Method 1: One-click install
./scripts/install_trilingual_models.sh
# Method 2: Manual install
pip install -r requirements-trilingual.txt
python -m spacy download en_core_web_trf
python -m spacy download sv_core_news_lg
# HanLP downloads automatically on first use
```
**Usage:**
```python
from lightrag.kg.trilingual_entity_extractor import TrilingualEntityExtractor
extractor = TrilingualEntityExtractor()
# Chinese
entities = extractor.extract("苹果公司由史蒂夫·乔布斯创立。", language='zh')
# English
entities = extractor.extract("Apple Inc. was founded by Steve Jobs.", language='en')
# Swedish
entities = extractor.extract("Volvo grundades i Göteborg.", language='sv')
```
**Testing:**
```bash
python scripts/test_trilingual_extractor.py
```
**Resource Requirements:**
- Disk: ~1.4 GB (440MB + 545MB + 400MB)
- Memory: ~1.5-1.8 GB per language (lazy loaded)
**Performance (CPU):**
- Chinese: ~12 docs/s
- English: ~29 docs/s
- Swedish: ~26 docs/s
Addresses user's specific needs: pure Chinese, pure English, and pure Swedish documents.
57 lines
1.7 KiB
Bash
Executable file
57 lines
1.7 KiB
Bash
Executable file
#!/bin/bash
|
||
# 三语言实体提取器安装脚本
|
||
# 自动安装 spaCy + HanLP 及相关模型
|
||
|
||
set -e # 遇到错误立即退出
|
||
|
||
echo "=================================================="
|
||
echo " 三语言实体提取器安装"
|
||
echo " 支持: 中文 (HanLP) + 英文 (spaCy) + 瑞典语 (spaCy)"
|
||
echo "=================================================="
|
||
echo ""
|
||
|
||
# 检查 Python 版本
|
||
echo "🔍 检查 Python 版本..."
|
||
python_version=$(python3 --version 2>&1 | awk '{print $2}')
|
||
echo " Python 版本: $python_version"
|
||
|
||
# 安装 Python 依赖
|
||
echo ""
|
||
echo "📦 安装 Python 依赖包..."
|
||
echo " - spaCy (英文 + 瑞典语)"
|
||
echo " - HanLP (中文)"
|
||
pip install -r requirements-trilingual.txt
|
||
|
||
# 下载 spaCy 英文模型
|
||
echo ""
|
||
echo "⬇️ 下载 spaCy 英文模型 (en_core_web_trf, ~440 MB)..."
|
||
python3 -m spacy download en_core_web_trf
|
||
|
||
# 下载 spaCy 瑞典语模型
|
||
echo ""
|
||
echo "⬇️ 下载 spaCy 瑞典语模型 (sv_core_news_lg, ~545 MB)..."
|
||
python3 -m spacy download sv_core_news_lg
|
||
|
||
# HanLP 提示
|
||
echo ""
|
||
echo "ℹ️ HanLP 中文模型会在首次使用时自动下载 (~400 MB)"
|
||
|
||
# 完成
|
||
echo ""
|
||
echo "=================================================="
|
||
echo " ✅ 安装完成！"
|
||
echo "=================================================="
|
||
echo ""
|
||
echo "磁盘空间使用:"
|
||
echo " - spaCy 英文模型: ~440 MB"
|
||
echo " - spaCy 瑞典语模型: ~545 MB"
|
||
echo " - HanLP 中文模型: ~400 MB (首次使用时下载)"
|
||
echo " - 总计: ~1.4 GB"
|
||
echo ""
|
||
echo "内存占用:"
|
||
echo " - 按需加载: 同时只加载一个语言模型 (~1.5-1.8 GB)"
|
||
echo " - 不会同时占用 4-5 GB 内存"
|
||
echo ""
|
||
echo "运行测试:"
|
||
echo " python3 scripts/test_trilingual_extractor.py"
|
||
echo ""
|