Repository: lancopku/pkuseg-python
Branch: master
Commit: 071d57c7df9a
Files: 27
Total size: 131.6 KB
Directory structure:
gitextract_zd_f4sqf/
├── .gitignore
├── LICENSE
├── README.md
├── pkuseg/
│ ├── __init__.py
│ ├── config.py
│ ├── data.py
│ ├── dicts/
│ │ ├── __init__.py
│ │ └── default.pkl
│ ├── download.py
│ ├── feature_extractor.pyx
│ ├── gradient.py
│ ├── inference.pyx
│ ├── model.py
│ ├── optimizer.py
│ ├── postag/
│ │ ├── __init__.py
│ │ ├── feature_extractor.pyx
│ │ └── model.py
│ ├── res_summarize.py
│ ├── scorer.py
│ └── trainer.py
├── readme/
│ ├── comparison.md
│ ├── environment.md
│ ├── history.md
│ ├── interface.md
│ ├── multiprocess.md
│ └── readme_english.md
└── setup.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.vscode/
*.cpp
*.pyd
*.html
**/__pycache__/
*.egg-info
*.txt*
build
tmp/
data/
models/
*stats
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2018-2019 pkuseg authors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# pkuseg：一个多领域中文分词工具包 [**(English Version)**](readme/readme_english.md)
pkuseg 是基于论文[[Luo et. al, 2019](#论文引用)]的工具包。其简单易用，支持细分领域分词，有效提升了分词准确度。
## 目录
* [主要亮点](#主要亮点)
* [编译和安装](#编译和安装)
* [各类分词工具包的性能对比](#各类分词工具包的性能对比)
* [使用方式](#使用方式)
* [论文引用](#论文引用)
* [作者](#作者)
* [常见问题及解答](#常见问题及解答)
## 主要亮点
pkuseg具有如下几个特点：
1. 多领域分词。不同于以往的通用中文分词工具，此工具包同时致力于为不同领域的数据提供个性化的预训练模型。根据待分词文本的领域特点，用户可以自由地选择不同的模型。 我们目前支持了新闻领域，网络领域，医药领域，旅游领域，以及混合领域的分词预训练模型。在使用中，如果用户明确待分词的领域，可加载对应的模型进行分词。如果用户无法确定具体领域，推荐使用在混合领域上训练的通用模型。各领域分词样例可参考 [**example.txt**](https://github.com/lancopku/pkuseg-python/blob/master/example.txt)。
2. 更高的分词准确率。相比于其他的分词工具包，当使用相同的训练数据和测试数据，pkuseg可以取得更高的分词准确率。
3. 支持用户自训练模型。支持用户使用全新的标注数据进行训练。
4. 支持词性标注。
## 编译和安装
- 目前**仅支持python3**
- **为了获得好的效果和速度，强烈建议大家通过pip install更新到目前的最新版本**
1. 通过PyPI安装(自带模型文件)：
```
pip3 install pkuseg
之后通过import pkuseg来引用
```
**建议更新到最新版本**以获得更好的开箱体验：
```
pip3 install -U pkuseg
```
2. 如果PyPI官方源下载速度不理想，建议使用镜像源，比如：
初次安装：
```
pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple pkuseg
```
更新：
```
pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple -U pkuseg
```
3. 如果不使用pip安装方式，选择从GitHub下载，可运行以下命令安装：
```
python setup.py build_ext -i
```
GitHub的代码并不包括预训练模型，因此需要用户自行下载或训练模型，预训练模型可详见[release](https://github.com/lancopku/pkuseg-python/releases)。使用时需设定"model_name"为模型文件。
注意：**安装方式1和2目前仅支持linux(ubuntu)、mac、windows 64 位的python3版本**。如果非以上系统，请使用安装方式3进行本地编译安装。
## 各类分词工具包的性能对比
我们选择jieba、THULAC等国内代表分词工具包与pkuseg做性能比较，详细设置可参考[实验环境](readme/environment.md)。
#### 细领域训练及测试结果
以下是在不同数据集上的对比结果：
| MSRA | Precision | Recall | F-score |
| :----- | --------: | -----: | --------: |
| jieba | 87.01 | 89.88 | 88.42 |
| THULAC | 95.60 | 95.91 | 95.71 |
| pkuseg | 96.94 | 96.81 | **96.88** |
| WEIBO | Precision | Recall | F-score |
| :----- | --------: | -----: | --------: |
| jieba | 87.79 | 87.54 | 87.66 |
| THULAC | 93.40 | 92.40 | 92.87 |
| pkuseg | 93.78 | 94.65 | **94.21** |
#### 默认模型在不同领域的测试效果
考虑到很多用户在尝试分词工具的时候，大多数时候会使用工具包自带模型测试。为了直接对比“初始”性能，我们也比较了各个工具包的默认模型在不同领域的测试效果。请注意，这样的比较只是为了说明默认情况下的效果，并不一定是公平的。
| Default | MSRA | CTB8 | PKU | WEIBO | All Average |
| ------- | :---: | :---: | :---: | :---: | :---------: |
| jieba | 81.45 | 79.58 | 81.83 | 83.56 | 81.61 |
| THULAC | 85.55 | 87.84 | 92.29 | 86.65 | 88.08 |
| pkuseg | 87.29 | 91.77 | 92.68 | 93.43 | **91.29** |
其中，`All Average`显示的是在所有测试集上F-score的平均。
更多详细比较可参见[和现有工具包的比较](readme/comparison.md)。
## 使用方式
#### 代码示例
以下代码示例适用于python交互式环境。
代码示例1：使用默认配置进行分词（**如果用户无法确定分词领域，推荐使用默认模型分词**）
```python3
import pkuseg
seg = pkuseg.pkuseg() # 以默认配置加载模型
text = seg.cut('我爱北京天安门') # 进行分词
print(text)
```
代码示例2：细领域分词（**如果用户明确分词领域，推荐使用细领域模型分词**）
```python3
import pkuseg
seg = pkuseg.pkuseg(model_name='medicine') # 程序会自动下载所对应的细领域模型
text = seg.cut('我爱北京天安门') # 进行分词
print(text)
```
代码示例3：分词同时进行词性标注，各词性标签的详细含义可参考 [tags.txt](https://github.com/lancopku/pkuseg-python/blob/master/tags.txt)
```python3
import pkuseg
seg = pkuseg.pkuseg(postag=True) # 开启词性标注功能
text = seg.cut('我爱北京天安门') # 进行分词和词性标注
print(text)
```
代码示例4：对文件分词
```python3
import pkuseg
# 对input.txt的文件分词输出到output.txt中
# 开20个进程
pkuseg.test('input.txt', 'output.txt', nthread=20)
```
其他使用示例可参见[详细代码示例](readme/interface.md)。
#### 参数说明
模型配置
```
pkuseg.pkuseg(model_name = "default", user_dict = "default", postag = False)
model_name 模型路径。
"default"，默认参数，表示使用我们预训练好的混合领域模型(仅对pip下载的用户)。
"news", 使用新闻领域模型。
"web", 使用网络领域模型。
"medicine", 使用医药领域模型。
"tourism", 使用旅游领域模型。
model_path, 从用户指定路径加载模型。
user_dict 设置用户词典。
"default", 默认参数，使用我们提供的词典。
None, 不使用词典。
dict_path, 在使用默认词典的同时会额外使用用户自定义词典，可以填自己的用户词典的路径，词典格式为一行一个词（如果选择进行词性标注并且已知该词的词性，则在该行写下词和词性，中间用tab字符隔开）。
postag 是否进行词性分析。
False, 默认参数，只进行分词，不进行词性标注。
True, 会在分词的同时进行词性标注。
```
对文件进行分词
```
pkuseg.test(readFile, outputFile, model_name = "default", user_dict = "default", postag = False, nthread = 10)
readFile 输入文件路径。
outputFile 输出文件路径。
model_name 模型路径。同pkuseg.pkuseg
user_dict 设置用户词典。同pkuseg.pkuseg
postag 设置是否开启词性分析功能。同pkuseg.pkuseg
nthread 测试时开的进程数。
```
模型训练
```
pkuseg.train(trainFile, testFile, savedir, train_iter = 20, init_model = None)
trainFile 训练文件路径。
testFile 测试文件路径。
savedir 训练模型的保存路径。
train_iter 训练轮数。
init_model 初始化模型，默认为None表示使用默认初始化，用户可以填自己想要初始化的模型的路径如init_model='./models/'。
```
#### 多进程分词
当将以上代码示例置于文件中运行时，如涉及多进程功能，请务必使用`if __name__ == '__main__'`保护全局语句，详见[多进程分词](readme/multiprocess.md)。
## 预训练模型
从pip安装的用户在使用细领域分词功能时，只需要设置model_name字段为对应的领域即可，会自动下载对应的细领域模型。
从github下载的用户则需要自己下载对应的预训练模型，并设置model_name字段为预训练模型路径。预训练模型可以在[release](https://github.com/lancopku/pkuseg-python/releases)部分下载。以下是对预训练模型的说明：
- **news**: 在MSRA（新闻语料）上训练的模型。
- **web**: 在微博（网络文本语料）上训练的模型。
- **medicine**: 在医药领域上训练的模型。
- **tourism**: 在旅游领域上训练的模型。
- **mixed**: 混合数据集训练的通用模型。随pip包附带的是此模型。
我们还通过领域自适应的方法，利用维基百科的未标注数据实现了几个细领域预训练模型的自动构建以及通用模型的优化，这些模型目前仅可以在release中下载：
- **art**: 在艺术与文化领域上训练的模型。
- **entertainment**: 在娱乐与体育领域上训练的模型。
- **science**: 在科学领域上训练的模型。
- **default_v2**: 使用领域自适应方法得到的优化后的通用模型，相较于默认模型规模更大，但泛化性能更好。
欢迎更多用户可以分享自己训练好的细分领域模型。
## 版本历史
详见[版本历史](readme/history.md)。
## 开源协议
1. 本代码采用MIT许可证。
2. 欢迎对该工具包提出任何宝贵意见和建议，请发邮件至jingjingxu@pku.edu.cn。
## 论文引用
该代码包主要基于以下科研论文，如使用了本工具，请引用以下论文：
* Ruixuan Luo, Jingjing Xu, Yi Zhang, Zhiyuan Zhang, Xuancheng Ren, Xu Sun. [PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation](https://arxiv.org/abs/1906.11455). Arxiv. 2019.
```
@article{pkuseg,
author = {Luo, Ruixuan and Xu, Jingjing and Zhang, Yi and Zhang, Zhiyuan and Ren, Xuancheng and Sun, Xu},
journal = {CoRR},
title = {PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation.},
url = {https://arxiv.org/abs/1906.11455},
volume = {abs/1906.11455},
year = 2019
}
```
## 其他相关论文
* Xu Sun, Houfeng Wang, Wenjie Li. Fast Online Training with Frequency-Adaptive Learning Rates for Chinese Word Segmentation and New Word Detection. ACL. 2012.
* Jingjing Xu and Xu Sun. Dependency-based gated recursive neural network for chinese word segmentation. ACL. 2016.
* Jingjing Xu and Xu Sun. Transfer learning for low-resource chinese word segmentation with a novel neural network. NLPCC. 2017.
## 常见问题及解答
1. [为什么要发布pkuseg？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#1-为什么要发布pkuseg)
2. [pkuseg使用了哪些技术？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#2-pkuseg使用了哪些技术)
3. [无法使用多进程分词和训练功能，提示RuntimeError和BrokenPipeError。](https://github.com/lancopku/pkuseg-python/wiki/FAQ#3-无法使用多进程分词和训练功能提示runtimeerror和brokenpipeerror)
4. [是如何跟其它工具包在细领域数据上进行比较的？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#4-是如何跟其它工具包在细领域数据上进行比较的)
5. [在黑盒测试集上进行比较的话，效果如何？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#5-在黑盒测试集上进行比较的话效果如何)
6. [如果我不了解待分词语料的所属领域呢？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#6-如果我不了解待分词语料的所属领域呢)
7. [如何看待在一些特定样例上的分词结果？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#7-如何看待在一些特定样例上的分词结果)
8. [关于运行速度问题？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#8-关于运行速度问题)
9. [关于多进程速度问题？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#9-关于多进程速度问题)
## 致谢
感谢俞士汶教授（北京大学计算语言所）与邱立坤博士提供的训练数据集！
## 作者
Ruixuan Luo （罗睿轩）, Jingjing Xu（许晶晶）, Xuancheng Ren（任宣丞）, Yi Zhang（张艺）, Zhiyuan Zhang（张之远）, Bingzhen Wei（位冰镇）， Xu Sun （孙栩）
北京大学 [语言计算与机器学习研究组](http://lanco.pku.edu.cn/)
================================================
FILE: pkuseg/__init__.py
================================================
from __future__ import print_function
import sys
if sys.version_info[0] < 3:
print("pkuseg does not support python2", file=sys.stderr)
sys.exit(1)
import os
import time
import pickle as pkl
import multiprocessing
from multiprocessing import Process, Queue
import pkuseg.trainer as trainer
import pkuseg.inference as _inf
from pkuseg.config import config
from pkuseg.feature_extractor import FeatureExtractor
from pkuseg.model import Model
from pkuseg.download import download_model
from pkuseg.postag import Postag
class TrieNode:
"""建立词典的Trie树节点"""
def __init__(self, isword):
self.isword = isword
self.usertag = ''
self.children = {}
class Preprocesser:
"""预处理器，在用户词典中的词强制分割"""
def __init__(self, dict_file):
"""初始化建立Trie树"""
if dict_file is None:
dict_file = []
self.dict_data = dict_file
if isinstance(dict_file, str):
with open(dict_file, encoding="utf-8") as f:
lines = f.readlines()
self.trie = TrieNode(False)
for line in lines:
fields = line.strip().split('\t')
word = fields[0].strip()
usertag = fields[1].strip() if len(fields) > 1 else ''
self.insert(word, usertag)
else:
self.trie = TrieNode(False)
for w_t in dict_file:
if isinstance(w_t, str):
w = w_t.strip()
t = ''
else:
assert isinstance(w_t, tuple)
assert len(w_t)==2
w, t = map(lambda x:x.strip(), w_t)
self.insert(w, t)
def insert(self, word, usertag):
"""Trie树中插入单词"""
l = len(word)
now = self.trie
for i in range(l):
c = word[i]
if not c in now.children:
now.children[c] = TrieNode(False)
now = now.children[c]
now.isword = True
now.usertag = usertag
def solve(self, txt):
"""对文本进行预处理"""
outlst = []
iswlst = []
taglst = []
l = len(txt)
last = 0
i = 0
while i < l:
now = self.trie
j = i
found = False
usertag = ''
last_word_idx = -1 # 表示从当前位置i往后匹配，最长匹配词词尾的idx
while True:
c = txt[j]
if not c in now.children and last_word_idx != -1:
found = True
break
if not c in now.children and last_word_idx == -1:
break
now = now.children[c]
if now.isword:
last_word_idx = j
usertag = now.usertag
j += 1
if j == l and last_word_idx == -1:
break
if j == l and last_word_idx != -1 :
j = last_word_idx + 1
found = True
break
if found:
if last != i:
outlst.append(txt[last:i])
iswlst.append(False)
taglst.append('')
outlst.append(txt[i:j])
iswlst.append(True)
taglst.append(usertag)
last = j
i = j
else:
i += 1
if last < l:
outlst.append(txt[last:l])
iswlst.append(False)
taglst.append('')
return outlst, iswlst, taglst
class Postprocesser:
"""对分词结果后处理"""
def __init__(self, common_name, other_names):
if common_name is None and other_names is None:
self.do_process = False
return
self.do_process = True
if common_name is None:
self.common_words = set()
else:
# with open(common_name, encoding='utf-8') as f:
# lines = f.readlines()
# self.common_words = set(map(lambda x:x.strip(), lines))
with open(common_name, "rb") as f:
all_words = pkl.load(f).strip().split("\n")
self.common_words = set(all_words)
if other_names is None:
self.other_words = set()
else:
self.other_words = set()
for other_name in other_names:
# with open(other_name, encoding='utf-8') as f:
# lines = f.readlines()
# self.other_words.update(set(map(lambda x:x.strip(), lines)))
with open(other_name, "rb") as f:
all_words = pkl.load(f).strip().split("\n")
self.other_words.update(set(all_words))
def post_process(self, sent, check_seperated):
for m in reversed(range(2, 8)):
end = len(sent)-m
if end < 0:
continue
i = 0
while (i < end + 1):
merged_words = ''.join(sent[i:i+m])
if merged_words in self.common_words:
do_seg = True
elif merged_words in self.other_words:
if check_seperated:
seperated = all(((w in self.common_words)
or (w in self.other_words)) for w in sent[i:i+m])
else:
seperated = False
if seperated:
do_seg = False
else:
do_seg = True
else:
do_seg = False
if do_seg:
for k in range(m):
del sent[i]
sent.insert(i, merged_words)
i += 1
end = len(sent) - m
else:
i += 1
return sent
def __call__(self, sent):
if not self.do_process:
return sent
return self.post_process(sent, check_seperated=True)
class pkuseg:
def __init__(self, model_name="default", user_dict="default", postag=False):
"""初始化函数，加载模型及用户词典"""
# print("loading model")
# config = Config()
# self.config = config
self.postag = postag
if model_name in ["default"]:
config.modelDir = os.path.join(
os.path.dirname(os.path.realpath(__file__)),
"models",
model_name,
)
elif model_name in config.available_models:
config.modelDir = os.path.join(
config.pkuseg_home,
model_name,
)
download_model(config.model_urls[model_name], config.pkuseg_home, config.model_hash[model_name])
else:
config.modelDir = model_name
# config.fModel = os.path.join(config.modelDir, "model.txt")
if user_dict is None:
file_name = None
other_names = None
else:
if user_dict not in config.available_models:
file_name = user_dict
else:
file_name = None
if model_name in config.models_with_dict:
other_name = os.path.join(
config.pkuseg_home,
model_name,
model_name+"_dict.pkl",
)
default_name = os.path.join(
os.path.dirname(os.path.realpath(__file__)),
"dicts", "default.pkl",
)
other_names = [other_name, default_name]
else:
default_name = os.path.join(
os.path.dirname(os.path.realpath(__file__)),
"dicts", "default.pkl",
)
other_names = [default_name]
self.preprocesser = Preprocesser(file_name)
# self.preprocesser = Preprocesser([])
self.postprocesser = Postprocesser(None, other_names)
self.feature_extractor = FeatureExtractor.load()
self.model = Model.load()
self.idx_to_tag = {
idx: tag for tag, idx in self.feature_extractor.tag_to_idx.items()
}
self.n_feature = len(self.feature_extractor.feature_to_idx)
self.n_tag = len(self.feature_extractor.tag_to_idx)
if postag:
download_model(config.model_urls["postag"], config.pkuseg_home, config.model_hash["postag"])
postag_dir = os.path.join(
config.pkuseg_home,
"postag",
)
self.tagger = Postag(postag_dir)
# print("finish")
def _cut(self, text):
"""
直接对文本分词
"""
examples = list(self.feature_extractor.normalize_text(text))
length = len(examples)
all_feature = [] # type: List[List[int]]
for idx in range(length):
node_feature_idx = self.feature_extractor.get_node_features_idx(
idx, examples
)
# node_feature = self.feature_extractor.get_node_features(
# idx, examples
# )
# node_feature_idx = []
# for feature in node_feature:
# feature_idx = self.feature_extractor.feature_to_idx.get(feature)
# if feature_idx is not None:
# node_feature_idx.append(feature_idx)
# if not node_feature_idx:
# node_feature_idx.append(0)
all_feature.append(node_feature_idx)
_, tags = _inf.decodeViterbi_fast(all_feature, self.model)
words = []
current_word = None
is_start = True
for tag, char in zip(tags, text):
if is_start:
current_word = char
is_start = False
elif "B" in self.idx_to_tag[tag]:
words.append(current_word)
current_word = char
else:
current_word += char
if current_word:
words.append(current_word)
return words
def cut(self, txt):
"""分词，结果返回一个list"""
txt = txt.strip()
ret = []
usertags = []
if not txt:
return ret
imary = txt.split() # 根据空格分为多个片段
# 对每个片段分词
for w0 in imary:
if not w0:
continue
# 根据用户词典拆成更多片段
lst, isword, taglst = self.preprocesser.solve(w0)
for w, isw, usertag in zip(lst, isword, taglst):
if isw:
ret.append(w)
usertags.append(usertag)
continue
output = self._cut(w)
post_output = self.postprocesser(output)
ret.extend(post_output)
usertags.extend(['']*len(post_output))
if self.postag:
tags = self.tagger.tag(ret.copy())
for i, usertag in enumerate(usertags):
if usertag:
tags[i] = usertag
ret = list(zip(ret, tags))
return ret
def train(trainFile, testFile, savedir, train_iter=20, init_model=None):
"""用于训练模型"""
# config = Config()
starttime = time.time()
if not os.path.exists(trainFile):
raise Exception("trainfile does not exist.")
if not os.path.exists(testFile):
raise Exception("testfile does not exist.")
if not os.path.exists(config.tempFile):
os.makedirs(config.tempFile)
if not os.path.exists(config.tempFile + "/output"):
os.mkdir(config.tempFile + "/output")
# config.runMode = "train"
config.trainFile = trainFile
config.testFile = testFile
config.modelDir = savedir
# config.fModel = os.path.join(config.modelDir, "model.txt")
config.nThread = 1
config.ttlIter = train_iter
config.init_model = init_model
os.makedirs(config.modelDir, exist_ok=True)
trainer.train(config)
# pkuseg.main.run(config)
# clearDir(config.tempFile)
print("Total time: " + str(time.time() - starttime))
def _test_single_proc(
input_file, output_file, model_name="default", user_dict="default", postag=False, verbose=False
):
times = []
times.append(time.time())
seg = pkuseg(model_name, user_dict, postag=postag)
times.append(time.time())
if not os.path.exists(input_file):
raise Exception("input_file {} does not exist.".format(input_file))
with open(input_file, "r", encoding="utf-8") as f:
lines = f.readlines()
times.append(time.time())
results = []
for line in lines:
if not postag:
results.append(" ".join(seg.cut(line)))
else:
results.append(" ".join(map(lambda x:"/".join(x), seg.cut(line))))
times.append(time.time())
with open(output_file, "w", encoding="utf-8") as f:
f.write("\n".join(results))
times.append(time.time())
print("total_time:\t{:.3f}".format(times[-1] - times[0]))
if verbose:
time_strs = ["load_model", "read_file", "word_seg", "write_file"]
for key, value in zip(
time_strs,
[end - start for start, end in zip(times[:-1], times[1:])],
):
print("{}:\t{:.3f}".format(key, value))
def _proc_deprecated(seg, lines, start, end, q):
for i in range(start, end):
l = lines[i].strip()
ret = seg.cut(l)
q.put((i, " ".join(ret)))
def _proc(seg, in_queue, out_queue):
# TODO: load seg (json or pickle serialization) in sub_process
# to avoid pickle seg online when using start method other
# than fork
while True:
item = in_queue.get()
if item is None:
return
idx, line = item
if not seg.postag:
output_str = " ".join(seg.cut(line))
else:
output_str = " ".join(map(lambda x:"/".join(x), seg.cut(line)))
out_queue.put((idx, output_str))
def _proc_alt(model_name, user_dict, postag, in_queue, out_queue):
seg = pkuseg(model_name, user_dict, postag=postag)
while True:
item = in_queue.get()
if item is None:
return
idx, line = item
if not postag:
output_str = " ".join(seg.cut(line))
else:
output_str = " ".join(map(lambda x:"/".join(x), seg.cut(line)))
out_queue.put((idx, output_str))
def _test_multi_proc(
input_file,
output_file,
nthread,
model_name="default",
user_dict="default",
postag=False,
verbose=False,
):
alt = multiprocessing.get_start_method() == "spawn"
times = []
times.append(time.time())
if alt:
seg = None
else:
seg = pkuseg(model_name, user_dict, postag)
times.append(time.time())
if not os.path.exists(input_file):
raise Exception("input_file {} does not exist.".format(input_file))
with open(input_file, "r", encoding="utf-8") as f:
lines = f.readlines()
times.append(time.time())
in_queue = Queue()
out_queue = Queue()
procs = []
for _ in range(nthread):
if alt:
p = Process(
target=_proc_alt,
args=(model_name, user_dict, postag, in_queue, out_queue),
)
else:
p = Process(target=_proc, args=(seg, in_queue, out_queue))
procs.append(p)
for idx, line in enumerate(lines):
in_queue.put((idx, line))
for proc in procs:
in_queue.put(None)
proc.start()
times.append(time.time())
result = [None] * len(lines)
for _ in result:
idx, line = out_queue.get()
result[idx] = line
times.append(time.time())
for p in procs:
p.join()
times.append(time.time())
with open(output_file, "w", encoding="utf-8") as f:
f.write("\n".join(result))
times.append(time.time())
print("total_time:\t{:.3f}".format(times[-1] - times[0]))
if verbose:
time_strs = [
"load_model",
"read_file",
"start_proc",
"word_seg",
"join_proc",
"write_file",
]
if alt:
times = times[1:]
time_strs = time_strs[1:]
time_strs[2] = "load_modal & word_seg"
for key, value in zip(
time_strs,
[end - start for start, end in zip(times[:-1], times[1:])],
):
print("{}:\t{:.3f}".format(key, value))
def test(
input_file,
output_file,
model_name="default",
user_dict="default",
nthread=10,
postag=False,
verbose=False,
):
if nthread > 1:
_test_multi_proc(
input_file, output_file, nthread, model_name, user_dict, postag, verbose
)
else:
_test_single_proc(
input_file, output_file, model_name, user_dict, postag, verbose
)
================================================
FILE: pkuseg/config.py
================================================
import os
import tempfile
class Config:
lineEnd = "\n"
biLineEnd = "\n\n"
triLineEnd = "\n\n\n"
undrln = "_"
blank = " "
tab = "\t"
star = "*"
slash = "/"
comma = ","
delimInFeature = "."
B = "B"
num = "0123456789.几二三四五六七八九十千万亿兆零１２３４５６７８９０％"
letter = "ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺａｂｃｄｅｆｇｈｉｇｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ／・－"
mark = "*"
model_urls = {
"postag": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/postag.zip",
"medicine": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/medicine.zip",
"tourism": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/tourism.zip",
"news": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/news.zip",
"web": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/web.zip",
}
model_hash = {
"postag": "afdf15f4e39bc47a39be4c37e3761b0c8f6ad1783f3cd3aff52984aebc0a1da9",
"medicine": "773d655713acd27dd1ea9f97d91349cc1b6aa2fc5b158cd742dc924e6f239dfc",
"tourism": "1c84a0366fe6fda73eda93e2f31fd399923b2f5df2818603f426a200b05cbce9",
"news": "18188b68e76b06fc437ec91edf8883a537fe25fa606641534f6f004d2f9a2e42",
"web": "4867f5817f187246889f4db259298c3fcee07c0b03a2d09444155b28c366579e",
}
available_models = ["default", "medicine", "tourism", "web", "news"]
models_with_dict = ["medicine", "tourism"]
def __init__(self):
# main setting
self.pkuseg_home = os.path.expanduser(os.getenv('PKUSEG_HOME', '~/.pkuseg'))
self.trainFile = os.path.join("data", "small_training.utf8")
self.testFile = os.path.join("data", "small_test.utf8")
self._tmp_dir = tempfile.TemporaryDirectory()
self.homepath = self._tmp_dir.name
self.tempFile = os.path.join(self.homepath, ".pkuseg", "temp")
self.readFile = os.path.join("data", "small_test.utf8")
self.outputFile = os.path.join("data", "small_test_output.utf8")
self.modelOptimizer = "crf.adf"
self.rate0 = 0.05 # init value of decay rate in SGD and ADF training
# self.reg = 1
# self.regs = [1]
# self.regList = self.regs.copy()
self.random = (
0
) # 0 for 0-initialization of model weights, 1 for random init of model weights
self.evalMetric = (
"f1"
) # tok.acc (token accuracy), str.acc (string accuracy), f1 (F1-score)
self.trainSizeScale = 1 # for scaling the size of training data
self.ttlIter = 20 # of training iterations
self.nUpdate = 10 # for ADF training
self.outFolder = os.path.join(self.tempFile, "output")
self.save = 1 # save model file
self.rawResWrite = True
self.miniBatch = 1 # mini-batch in stochastic training
self.nThread = 10 # number of processes
# ADF training
self.upper = 0.995 # was tuned for nUpdate = 10
self.lower = 0.6 # was tuned for nUpdate = 10
# global variables
self.metric = None
self.reg = 1
self.outDir = self.outFolder
self.testrawDir = "rawinputs/"
self.testinputDir = "inputs/"
self.tempDir = os.path.join(self.homepath, ".pkuseg", "temp")
self.testoutputDir = "entityoutputs/"
# self.GL_init = True
self.weightRegMode = "L2" # choosing weight regularizer: L2, L1)
self.c_train = os.path.join(self.tempFile, "train.conll.txt")
self.f_train = os.path.join(self.tempFile, "train.feat.txt")
self.c_test = os.path.join(self.tempFile, "test.conll.txt")
self.f_test = os.path.join(self.tempFile, "test.feat.txt")
self.fTune = "tune.txt"
self.fLog = "trainLog.txt"
self.fResSum = "summarizeResult.txt"
self.fResRaw = "rawResult.txt"
self.fOutput = "outputTag-{}.txt"
self.fFeatureTrain = os.path.join(self.tempFile, "ftrain.txt")
self.fGoldTrain = os.path.join(self.tempFile, "gtrain.txt")
self.fFeatureTest = os.path.join(self.tempFile, "ftest.txt")
self.fGoldTest = os.path.join(self.tempFile, "gtest.txt")
self.modelDir = os.path.join(
os.path.dirname(os.path.realpath(__file__)), "models", "ctb8"
)
self.fModel = os.path.join(self.modelDir, "model.txt")
# feature
self.numLetterNorm = True
self.featureTrim = 0
self.wordFeature = True
self.wordMax = 6
self.wordMin = 2
self.nLabel = 5
self.order = 1
def globalCheck(self):
if self.evalMetric == "f1":
self.metric = "f-score"
elif self.evalMetric == "tok.acc":
self.metric = "token-accuracy"
elif self.evalMetric == "str.acc":
self.metric = "string-accuracy"
else:
raise Exception("invalid eval metric")
assert self.rate0 > 0
assert self.trainSizeScale > 0
assert self.ttlIter > 0
assert self.nUpdate > 0
assert self.miniBatch > 0
assert self.reg > 0
config = Config()
================================================
FILE: pkuseg/data.py
================================================
# from .config import Config
# from pkuseg.feature_generator import
# import os
import copy
import random
# class dataFormat:
# def __init__(self, config):
# self.featureIndexMap = {}
# self.tagIndexMap = {}
# self.config = config
# def convert(self):
# config = self.config
# if config.runMode.find("train") >= 0:
# self.getMaps(config.fTrain)
# self.saveFeature(config.modelDir + "/featureIndex.txt")
# self.convertFile(config.fTrain)
# else:
# self.readFeature(config.modelDir + "/featureIndex.txt")
# self.readTag(config.modelDir + "/tagIndex.txt")
# self.convertFile(config.fTest)
# if config.dev:
# self.convertFile(config.fDev)
# def saveFeature(self, file):
# featureList = list(self.featureIndexMap.keys())
# num = len(featureList) // 10
# for i in range(10):
# l = i * num
# r = (i + 1) * num if i < 9 else len(featureList)
# with open(file + "_" + str(i), "w", encoding="utf-8") as sw:
# for w in range(l, r):
# word = featureList[w]
# sw.write(word + " " + str(self.featureIndexMap[word]) + "\n")
# def readFeature(self, file):
# featureList = []
# for i in range(10):
# featureList.append([])
# with open(file + "_" + str(i), encoding="utf-8") as f:
# lines = f.readlines()
# for line in lines:
# featureList[i].append(line.strip())
# feature = []
# for i in range(10):
# for line in featureList[i]:
# word, index = line.split(" ")
# self.featureIndexMap[word] = int(index)
# def readFeatureNormal(self, path):
# with open(path, encoding="utf-8") as f:
# lines = f.readlines()
# for line in lines:
# u, v = line.split(" ")
# self.featureIndexMap[u] = int(v)
# def readTag(self, path):
# with open(path, encoding="utf-8") as f:
# lines = f.readlines()
# for line in lines:
# u, v = line.split(" ")
# self.tagIndexMap[u] = int(v)
# def getMaps(self, file):
# config = self.config
# if not os.path.exists(file):
# print("file {} not exist!".format(file))
# print("file {} converting...".format(file))
# featureFreqMap = {}
# tagSet = set()
# with open(file, encoding="utf-8") as f:
# lines = f.readlines()
# for line in lines:
# line = line.replace("\t", " ")
# line = line.replace("\r", "").strip()
# if line == "":
# continue
# ary = line.split(config.blank)
# for i in range(1, len(ary) - 1):
# if ary[i] == "" or ary[i] == "/":
# continue
# if config.weightRegMode == "GL":
# if not config.GL_init and config.groupTrim[i - 1]:
# continue
# ary2 = ary[i].split(config.slash)
# feature = str(i) + "." + ary2[0]
# if not feature in featureFreqMap:
# featureFreqMap[feature] = 0
# featureFreqMap[feature] += 1
# tag = ary[-1]
# tagSet.add(tag)
# sortList = []
# for k in featureFreqMap:
# sortList.append(k + " " + str(featureFreqMap[k]))
# if config.weightRegMode == "GL":
# sortList.sort(key=lambda x: (int(x.split(config.blank)[1].strip()), x))
# with open("featureTemp_sorted.txt", "w", encoding="utf-8") as f:
# for x in sortList:
# f.write(x + "\n")
# config.groupStart = [0]
# config.groupEnd = []
# for k in range(1, len(sortList)):
# thisAry = sortList[k].split(config.dot)
# preAry = sortList[k - 1].split(config.dot)
# s = thisAry[0]
# preAry = preAry[0]
# if s != preAry:
# config.groupStart.append(k)
# config.groupEnd.append(k)
# config.groupEnd.append(len(sortList))
# else:
# sortList.sort(
# key=lambda x: (int(x.split(config.blank)[1].strip()), x), reverse=True
# )
# if config.weightRegMode == "GL" and config.GL_init:
# if nFeatTemp != len(config.groupStart):
# raise Exception(
# "inconsistent # of features per line, check the feature file for consistency!"
# )
# with open(
# os.path.join(config.modelDir, "featureIndex.txt"), "w", encoding="utf-8"
# ) as swFeat:
# for i, l in enumerate(sortList):
# ary = l.split(config.blank)
# self.featureIndexMap[ary[0]] = i
# swFeat.write("{} {}\n".format(ary[0].strip(), i))
# with open(os.path.join(config.modelDir, "tagIndex.txt"), "w", encoding="utf-8") as swTag:
# tagSortList = []
# for tag in tagSet:
# tagSortList.append(tag)
# tagSortList.sort()
# for i, l in enumerate(tagSortList):
# self.tagIndexMap[l] = i
# swTag.write("{} {}\n".format(l, i))
# def convertFile(self, file):
# config = self.config
# if not os.path.exists(file):
# print("file {} not exist!".format(file))
# print("file converting...")
# if file == config.fTrain:
# swFeature = open(config.fFeatureTrain, "w", encoding="utf-8")
# swGold = open(config.fGoldTrain, "w", encoding="utf-8")
# else:
# swFeature = open(config.fFeatureTest, "w", encoding="utf-8")
# swGold = open(config.fGoldTest, "w", encoding="utf-8")
# swFeature.write(str(len(self.featureIndexMap)) + "\n\n")
# swGold.write(str(len(self.tagIndexMap)) + "\n\n")
# with open(file, encoding="utf-8") as sr:
# readLines = sr.readlines()
# featureList = []
# goldList = []
# for k in range(len(readLines)):
# line = readLines[k]
# line = line.replace("\t", "").strip()
# featureLine = ""
# goldLine = ""
# if line == "":
# featureLine = featureLine + "\n"
# goldLine = goldLine + "\n\n"
# featureList.append(featureLine)
# goldList.append(goldLine)
# continue
# flag = 0
# ary = line.split(config.blank)
# tmp = []
# for i in ary:
# if i != "":
# tmp.append(i)
# ary = tmp
# for i in range(1, len(ary) - 1):
# if ary[i] == "/":
# continue
# ary2 = ary[i].split(config.slash)
# tmp = []
# for j in ary2:
# if j != "":
# tmp.append(j)
# ary2 = tmp
# feature = str(i) + "." + ary2[0]
# value = ""
# real = False
# if len(ary2) > 1:
# value = ary2[1]
# real = True
# if not feature in self.featureIndexMap:
# continue
# flag = 1
# fIndex = self.featureIndexMap[feature]
# if not real:
# featureLine = featureLine + str(fIndex) + ","
# else:
# featureLine = featureLine + str(fIndex) + "/" + value + ","
# if flag == 0:
# featureLine = featureLine + "0"
# featureLine = featureLine + "\n"
# tag = ary[-1]
# tIndex = self.tagIndexMap[tag]
# goldLine = goldLine + str(tIndex) + ","
# featureList.append(featureLine)
# goldList.append(goldLine)
# for i in range(len(featureList)):
# swFeature.write(featureList[i])
# swGold.write(goldList[i])
# swFeature.close()
# swGold.close()
class DataSet:
def __init__(self, n_tag=0, n_feature=0):
self.lst = [] # type: List[Example]
self.n_tag = n_tag
self.n_feature = n_feature
# if len(args) == 2:
# if type(args[0]) == int:
# self.nTag, self.nFeature = args
# else:
# self.load(args[0], args[1])
def __len__(self):
return len(self.lst)
def __iter__(self):
return self.iterator()
def __getitem__(self, x):
return self.lst[x]
def iterator(self):
for i in self.lst:
yield i
def append(self, x):
self.lst.append(x)
def clear(self):
self.lst = []
def randomShuffle(self):
cp = copy.deepcopy(self)
random.shuffle(cp.lst)
return cp
# def setDataInfo(self, X):
# self.nTag = X.nTag
# self.nFeature = X.nFeature
def resize(self, scale):
dataset = DataSet(self.n_tag, self.n_feature)
new_size = int(len(self) * scale)
old_size = len(self)
for i in range(new_size):
if i >= old_size:
i %= old_size
dataset.append(self[i])
return dataset
@classmethod
def load(cls, feature_idx_file, tag_idx_file):
dataset = cls.__new__(cls)
# def load(self, fileFeature, fileTag):
with open(feature_idx_file, encoding="utf-8") as f_reader, open(
tag_idx_file, encoding="utf-8"
) as t_reader:
example_strs = f_reader.read().split("\n\n")[:-1]
tags_strs = t_reader.read().split("\n\n")[:-1]
assert len(example_strs) == len(
tags_strs
), "lengths do not match:\t{}\n{}\n".format(example_strs, tags_strs)
n_feature = int(example_strs[0])
n_tag = int(tags_strs[0])
dataset.n_feature = n_feature
dataset.n_tag = n_tag
dataset.lst = []
for example_str, tags_str in zip(example_strs[1:], tags_strs[1:]):
features = [
list(map(int, feature_line.split(",")))
for feature_line in example_str.split("\n")
]
tags = tags_str.split(",")
example = Example(features, tags)
dataset.lst.append(example)
return dataset
# txt = srfileFeature.read()
# txt.replace("\r", "")
# fAry = txt.split(Config.biLineEnd)
# tmp = []
# for i in fAry:
# if i != "":
# tmp.append(i)
# fAry = tmp
# txt = srfileTag.read()
# txt.replace("\r", "")
# tAry = txt.split(Config.biLineEnd)
# tmp = []
# for i in tAry:
# if i != "":
# tmp.append(i)
# tAry = tmp
# assert len(fAry) == len(tAry)
# self.nFeature = int(fAry[0])
# self.nTag = int(tAry[0])
# for i in range(1, len(fAry)):
# features = fAry[i]
# tags = tAry[i]
# seq = dataSeq()
# seq.read(features, tags)
# self.append(seq)
# @property
# def NTag(self):
# return self.nTag
class Example:
def __init__(self, features, tags):
self.features = features # type: List[List[int]]
self.tags = list(map(int, tags)) # type: List[int]
self.predicted_tags = None
def __len__(self):
return len(self.features)
# class dataSeq:
# def __init__(self, *args):
# self.featureTemps = []
# self.yGold = []
# if len(args) == 2:
# self.featureTemps = copy.deepcopy(args[0])
# self.yGold = copy.deepcopy(args[1])
# elif len(args) == 3:
# x, n, length = args
# end = min(n + length, len(x))
# for i in range(n, end):
# self.featureTemps.append(x.featureTemps[i])
# yGold.append(x.yGold[i])
# def __len__(self):
# return len(self.featureTemps)
# def read(self, a, b):
# lineAry = a.split(Config.lineEnd)
# for im in lineAry:
# if im == "":
# continue
# nodeList = []
# imAry = im.split(Config.comma)
# for imm in imAry:
# if imm == "":
# continue
# if imm.find("/") >= 0:
# biAry = imm.split(Config.slash)
# ft = featureTemp(int(biAry[0], float(biAry[1])))
# nodeList.append(ft)
# else:
# ft = featureTemp(int(imm), 1)
# nodeList.append(ft)
# self.featureTemps.append(nodeList)
# lineAry = b.split(Config.comma)
# for im in lineAry:
# if im == "":
# continue
# self.yGold.append(int(im))
# # def load(self, feature):
# # for imAry in feature:
# # nodeList = []
# # for imm in imAry:
# # if imm == "":
# # continue
# # if imm.find("/") >= 0:
# # biAry = imm.split(Config.slash)
# # ft = featureTemp(int(biAry[0], float(biAry[1])))
# # nodeList.append(ft)
# # else:
# # ft = featureTemp(int(imm), 1)
# # nodeList.append(ft)
# # self.featureTemps.append(nodeList)
# # self.yGold.append(0)
# def getFeatureTemp(self, *args):
# return (
# self.featureTemps if len(args) == 0 else self.featureTemps[args[0]]
# )
# def getTags(self, *args):
# return self.yGold if len(args) == 0 else self.yGold[args[0]]
# def setTags(self, lst):
# assert len(lst) == len(self.yGold)
# for i in range(len(lst)):
# self.yGold[i] = lst[i]
# class dataSeqTest:
# def __init__(self, x, yOutput):
# self._x = x
# self._yOutput = yOutput
================================================
FILE: pkuseg/dicts/__init__.py
================================================
================================================
FILE: pkuseg/download.py
================================================
from __future__ import absolute_import, division, print_function, unicode_literals
import hashlib
import os
import re
import shutil
import sys
import tempfile
import zipfile
try:
from requests.utils import urlparse
from requests import get as urlopen
requests_available = True
except ImportError:
requests_available = False
if sys.version_info[0] == 2:
from urlparse import urlparse # noqa f811
from urllib2 import urlopen # noqa f811
else:
from urllib.request import urlopen
from urllib.parse import urlparse
try:
from tqdm import tqdm
except ImportError:
tqdm = None # defined below
HASH_REGEX = re.compile(r'-([a-f0-9]*)\.')
def download_model(url, model_dir, hash_prefix, progress=True):
if not os.path.exists(model_dir):
os.makedirs(model_dir)
parts = urlparse(url)
filename = os.path.basename(parts.path)
cached_file = os.path.join(model_dir, filename)
if not os.path.exists(cached_file):
sys.stderr.write('Downloading: "{}" to {}\n'.format(url, cached_file))
_download_url_to_file(url, cached_file, hash_prefix, progress=progress)
unzip_file(cached_file, os.path.join(model_dir, filename.split('.')[0]))
def _download_url_to_file(url, dst, hash_prefix, progress):
if requests_available:
u = urlopen(url, stream=True, timeout=5)
file_size = int(u.headers["Content-Length"])
u = u.raw
else:
u = urlopen(url, timeout=5)
meta = u.info()
if hasattr(meta, 'getheaders'):
file_size = int(meta.getheaders("Content-Length")[0])
else:
file_size = int(meta.get_all("Content-Length")[0])
f = tempfile.NamedTemporaryFile(delete=False)
try:
if hash_prefix is not None:
sha256 = hashlib.sha256()
with tqdm(total=file_size, disable=not progress) as pbar:
while True:
buffer = u.read(8192)
if len(buffer) == 0:
break
f.write(buffer)
if hash_prefix is not None:
sha256.update(buffer)
pbar.update(len(buffer))
f.close()
if hash_prefix is not None:
digest = sha256.hexdigest()
if digest[:len(hash_prefix)] != hash_prefix:
raise RuntimeError('invalid hash value (expected "{}", got "{}")'
.format(hash_prefix, digest))
shutil.move(f.name, dst)
finally:
f.close()
if os.path.exists(f.name):
os.remove(f.name)
if tqdm is None:
# fake tqdm if it's not installed
class tqdm(object):
def __init__(self, total, disable=False):
self.total = total
self.disable = disable
self.n = 0
def update(self, n):
if self.disable:
return
self.n += n
sys.stderr.write("\r{0:.1f}%".format(100 * self.n / float(self.total)))
sys.stderr.flush()
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
if self.disable:
return
sys.stderr.write('\n')
def unzip_file(zip_name, target_dir):
if not os.path.exists(target_dir):
os.makedirs(target_dir)
file_zip = zipfile.ZipFile(zip_name, 'r')
for file in file_zip.namelist():
file_zip.extract(file, target_dir)
file_zip.close()
if __name__ == '__main__':
url = 'https://github.com/lancopku/pkuseg-python/releases/download/v0.0.14/mixed.zip'
download_model(url, '.')
================================================
FILE: pkuseg/feature_extractor.pyx
================================================
# distutils: language = c++
# cython: infer_types=True
# cython: language_level=3
import json
import os
import sys
import pickle
from collections import Counter
from itertools import product
import cython
from pkuseg.config import config
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef get_slice_str(iterable, int start, int length, int all_len):
if start < 0 or start >= all_len:
return ""
if start + length >= all_len + 1:
return ""
return "".join(iterable[start : start + length])
@cython.boundscheck(False)
@cython.wraparound(False)
@cython.nonecheck(False)
def __get_node_features_idx(object config not None, int idx, list nodes not None, dict feature_to_idx not None, set unigram not None):
cdef:
list flist = []
Py_ssize_t i = idx
int length = len(nodes)
int word_max = config.wordMax
int word_min = config.wordMin
int word_range = word_max - word_min + 1
c = nodes[i]
# $$ starts feature
flist.append(0)
# 8 unigram/bgiram feature
feat = 'c.' + c
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
if i > 0:
prev_c = nodes[i-1]
feat = 'c-1.' + prev_c
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
feat = 'c-1c.' + prev_c + '.' + c
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
if i + 1 < length:
next_c = nodes[i+1]
feat = 'c1.' + next_c
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
feat = 'cc1.' + c + '.' + next_c
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
if i > 1:
prepre_char = nodes[i-2]
feat = 'c-2.' + prepre_char
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
feat = 'c-2c-1.' + prepre_char + '.' + nodes[i-1]
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
if i + 2 < length:
feat = 'c2.' + nodes[i+2]
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
# no num/letter based features
if not config.wordFeature:
return flist
# 2 * (wordMax-wordMin+1) word features (default: 2*(6-2+1)=10 )
# the character starts or ends a word
prelst_in = []
for l in range(word_max, word_min - 1, -1):
# length 6 ... 2 (default)
# "prefix including current c" wordary[n-l+1, n]
# current character ends word
tmp = get_slice_str(nodes, i - l + 1, l, length)
if tmp in unigram:
feat = 'w-1.' + tmp
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
prelst_in.append(tmp)
else:
prelst_in.append("**noWord")
postlst_in = []
for l in range(word_max, word_min - 1, -1):
# "suffix" wordary[n, n+l-1]
# current character starts word
tmp = get_slice_str(nodes, i, l, length)
if tmp in unigram:
feat = 'w1.' + tmp
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
postlst_in.append(tmp)
else:
postlst_in.append("**noWord")
# these are not in feature list
prelst_ex = []
for l in range(word_max, word_min - 1, -1):
# "prefix excluding current c" wordary[n-l, n-1]
tmp = get_slice_str(nodes, i - l, l, length)
if tmp in unigram:
prelst_ex.append(tmp)
else:
prelst_ex.append("**noWord")
postlst_ex = []
for l in range(word_max, word_min - 1, -1):
# "suffix excluding current c" wordary[n+1, n+l]
tmp = get_slice_str(nodes, i + 1, l, length)
if tmp in unigram:
postlst_ex.append(tmp)
else:
postlst_ex.append("**noWord")
# this character is in the middle of a word
# 2*(wordMax-wordMin+1)^2 (default: 2*(6-2+1)^2=50)
for pre in prelst_ex:
for post in postlst_in:
feat = 'ww.l.' + pre + '*' + post
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
for pre in prelst_in:
for post in postlst_ex:
feat = 'ww.r.' + pre + '*' + post
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
return flist
class FeatureExtractor:
keywords = "-._,|/*:"
num = set("0123456789." "几二三四五六七八九十千万亿兆零" "１２３４５６７８９０％")
letter = set(
"ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ" "ａｂｃｄｅｆｇｈｉｇｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ" "／・－"
)
keywords_translate_table = str.maketrans("-._,|/*:", "&&&&&&&&")
@classmethod
def keyword_rename(cls, text):
return text.translate(cls.keywords_translate_table)
@classmethod
def _num_letter_normalize_char(cls, character):
if character in cls.num:
return "**Num"
if character in cls.letter:
return "**Letter"
return character
@classmethod
def normalize_text(cls, text):
text = cls.keyword_rename(text)
for character in text:
if config.numLetterNorm:
yield cls._num_letter_normalize_char(character)
else:
yield character
def __init__(self):
self.unigram = set() # type: Set[str]
self.bigram = set() # type: Set[str]
self.feature_to_idx = {} # type: Dict[str, int]
self.tag_to_idx = {} # type: Dict[str, int]
def build(self, train_file):
with open(train_file, "r", encoding="utf8") as reader:
lines = reader.readlines()
examples = [] # type: List[List[List[str]]]
# first pass to collect unigram and bigram and tag info
word_length_info = Counter()
specials = set()
for line in lines:
line = line.strip("\n\r") # .replace("\t", " ")
if not line:
continue
line = self.keyword_rename(line)
# str.split() without sep sees consecutive whiltespaces as one separator
# e.g., '\ra \t b \r\n'.split() = ['a', 'b']
words = [word for word in line.split()]
word_length_info.update(map(len, words))
specials.update(word for word in words if len(word)>=10)
self.unigram.update(words)
for pre, suf in zip(words[:-1], words[1:]):
self.bigram.add("{}*{}".format(pre, suf))
example = [
self._num_letter_normalize_char(character)
for word in words
for character in word
]
examples.append(example)
max_word_length = max(word_length_info.keys())
for length in range(1, max_word_length + 1):
print("length = {} : {}".format(length, word_length_info[length]))
# print('special words: {}'.format(', '.join(specials)))
# second pass to get features
feature_freq = Counter()
for example in examples:
for i, _ in enumerate(example):
node_features = self.get_node_features(i, example)
feature_freq.update(
feature for feature in node_features if feature != "/"
)
feature_set = (
feature
for feature, freq in feature_freq.most_common()
if freq > config.featureTrim
)
tot = len(self.feature_to_idx)
for feature in feature_set:
if not feature in self.feature_to_idx:
self.feature_to_idx[feature] = tot
tot += 1
# self.feature_to_idx = {
# feature: idx for idx, feature in enumerate(feature_set)
# }
if config.nLabel == 2:
B = B_single = "B"
I_first = I = I_end = "I"
elif config.nLabel == 3:
B = B_single = "B"
I_first = I = "I"
I_end = "I_end"
elif config.nLabel == 4:
B = "B"
B_single = "B_single"
I_first = I = "I"
I_end = "I_end"
elif config.nLabel == 5:
B = "B"
B_single = "B_single"
I_first = "I_first"
I = "I"
I_end = "I_end"
tag_set = {B, B_single, I_first, I, I_end}
self.tag_to_idx = {tag: idx for idx, tag in enumerate(sorted(tag_set))}
def get_node_features_idx(self, idx, nodes):
return __get_node_features_idx(config, idx, nodes, self.feature_to_idx, self.unigram)
def get_node_features(self, idx, wordary):
cdef int length = len(wordary)
w = wordary[idx]
flist = []
# 1 start feature
flist.append("$$")
# 8 unigram/bgiram feature
flist.append("c." + w)
if idx > 0:
flist.append("c-1." + wordary[idx - 1])
else:
flist.append("/")
if idx < len(wordary) - 1:
flist.append("c1." + wordary[idx + 1])
else:
flist.append("/")
if idx > 1:
flist.append("c-2." + wordary[idx - 2])
else:
flist.append("/")
if idx < len(wordary) - 2:
flist.append("c2." + wordary[idx + 2])
else:
flist.append("/")
if idx > 0:
flist.append("c-1c." + wordary[idx - 1] + config.delimInFeature + w)
else:
flist.append("/")
if idx < len(wordary) - 1:
flist.append("cc1." + w + config.delimInFeature + wordary[idx + 1])
else:
flist.append("/")
if idx > 1:
flist.append(
"c-2c-1."
+ wordary[idx - 2]
+ config.delimInFeature
+ wordary[idx - 1]
)
else:
flist.append("/")
# no num/letter based features
if not config.wordFeature:
return flist
# 2 * (wordMax-wordMin+1) word features (default: 2*(6-2+1)=10 )
# the character starts or ends a word
tmplst = []
for l in range(config.wordMax, config.wordMin - 1, -1):
# length 6 ... 2 (default)
# "prefix including current c" wordary[n-l+1, n]
# current character ends word
tmp = get_slice_str(wordary, idx - l + 1, l, length)
if tmp != "":
if tmp in self.unigram:
flist.append("w-1." + tmp)
tmplst.append(tmp)
else:
flist.append("/")
tmplst.append("**noWord")
else:
flist.append("/")
tmplst.append("**noWord")
prelst_in = tmplst
tmplst = []
for l in range(config.wordMax, config.wordMin - 1, -1):
# "suffix" wordary[n, n+l-1]
# current character starts word
tmp = get_slice_str(wordary, idx, l, length)
if tmp != "":
if tmp in self.unigram:
flist.append("w1." + tmp)
tmplst.append(tmp)
else:
flist.append("/")
tmplst.append("**noWord")
else:
flist.append("/")
tmplst.append("**noWord")
postlst_in = tmplst
# these are not in feature list
tmplst = []
for l in range(config.wordMax, config.wordMin - 1, -1):
# "prefix excluding current c" wordary[n-l, n-1]
tmp = get_slice_str(wordary, idx - l, l, length)
if tmp != "":
if tmp in self.unigram:
tmplst.append(tmp)
else:
tmplst.append("**noWord")
else:
tmplst.append("**noWord")
prelst_ex = tmplst
tmplst = []
for l in range(config.wordMax, config.wordMin - 1, -1):
# "suffix excluding current c" wordary[n+1, n+l]
tmp = get_slice_str(wordary, idx + 1, l, length)
if tmp != "":
if tmp in self.unigram:
tmplst.append(tmp)
else:
tmplst.append("**noWord")
else:
tmplst.append("**noWord")
postlst_ex = tmplst
# this character is in the middle of a word
# 2*(wordMax-wordMin+1)^2 (default: 2*(6-2+1)^2=50)
for pre in prelst_ex:
for post in postlst_in:
bigram = pre + "*" + post
if bigram in self.bigram:
flist.append("ww.l." + bigram)
else:
flist.append("/")
for pre in prelst_in:
for post in postlst_ex:
bigram = pre + "*" + post
if bigram in self.bigram:
flist.append("ww.r." + bigram)
else:
flist.append("/")
return flist
def convert_feature_file_to_idx_file(
self, feature_file, feature_idx_file, tag_idx_file
):
with open(feature_file, "r", encoding="utf8") as reader:
lines = reader.readlines()
with open(feature_idx_file, "w", encoding="utf8") as f_writer, open(
tag_idx_file, "w", encoding="utf8"
) as t_writer:
f_writer.write("{}\n\n".format(len(self.feature_to_idx)))
t_writer.write("{}\n\n".format(len(self.tag_to_idx)))
tags_idx = [] # type: List[str]
features_idx = [] # type: List[List[str]]
for line in lines:
line = line.strip()
if not line:
# sentence finish
for feature_idx in features_idx:
if not feature_idx:
f_writer.write("0\n")
else:
f_writer.write(",".join(map(str, feature_idx)))
f_writer.write("\n")
f_writer.write("\n")
t_writer.write(",".join(map(str, tags_idx)))
t_writer.write("\n\n")
tags_idx = []
features_idx = []
continue
splits = line.split(" ")
feature_idx = [
self.feature_to_idx[feat]
for feat in splits[:-1]
if feat in self.feature_to_idx
]
features_idx.append(feature_idx)
tags_idx.append(self.tag_to_idx[splits[-1]])
def convert_text_file_to_feature_file(
self, text_file, conll_file=None, feature_file=None
):
if conll_file is None:
conll_file = "{}.conll{}".format(*os.path.split(text_file))
if feature_file is None:
feature_file = "{}.feat{}".format(*os.path.split(text_file))
if config.nLabel == 2:
B = B_single = "B"
I_first = I = I_end = "I"
elif config.nLabel == 3:
B = B_single = "B"
I_first = I = "I"
I_end = "I_end"
elif config.nLabel == 4:
B = "B"
B_single = "B_single"
I_first = I = "I"
I_end = "I_end"
elif config.nLabel == 5:
B = "B"
B_single = "B_single"
I_first = "I_first"
I = "I"
I_end = "I_end"
conll_line_format = "{} {}\n"
with open(text_file, "r", encoding="utf8") as reader, open(
conll_file, "w", encoding="utf8"
) as c_writer, open(feature_file, "w", encoding="utf8") as f_writer:
for line in reader:
line = line.strip()
if not line:
continue
words = self.keyword_rename(line).split()
example = []
tags = []
for word in words:
word_length = len(word)
for idx, character in enumerate(word):
if word_length == 1:
tag = B_single
elif idx == 0:
tag = B
elif idx == word_length - 1:
tag = I_end
elif idx == 1:
tag = I_first
else:
tag = I
c_writer.write(conll_line_format.format(character, tag))
if config.numLetterNorm:
example.append(
self._num_letter_normalize_char(character)
)
else:
example.append(character)
tags.append(tag)
c_writer.write("\n")
for idx, tag in enumerate(tags):
features = self.get_node_features(idx, example)
features = [
(feature if feature in self.feature_to_idx else "/")
for feature in features
]
features.append(tag)
f_writer.write(" ".join(features))
f_writer.write("\n")
f_writer.write("\n")
def save(self, model_dir=None):
if model_dir is None:
model_dir = config.modelDir
data = {}
data["unigram"] = sorted(list(self.unigram))
data["bigram"] = sorted(list(self.bigram))
data["feature_to_idx"] = self.feature_to_idx
data["tag_to_idx"] = self.tag_to_idx
with open(os.path.join(model_dir, 'features.pkl'), 'wb') as writer:
pickle.dump(data, writer, protocol=pickle.HIGHEST_PROTOCOL)
# with open(
# os.path.join(config.modelDir, "features.json"), "w", encoding="utf8"
# ) as writer:
# json.dump(data, writer, ensure_ascii=False)
@classmethod
def load(cls, model_dir=None):
if model_dir is None:
model_dir = config.modelDir
extractor = cls.__new__(cls)
feature_path = os.path.join(model_dir, "features.pkl")
if os.path.exists(feature_path):
with open(feature_path, "rb") as reader:
data = pickle.load(reader)
extractor.unigram = set(data["unigram"])
extractor.bigram = set(data["bigram"])
extractor.feature_to_idx = data["feature_to_idx"]
extractor.tag_to_idx = data["tag_to_idx"]
return extractor
print(
"WARNING: features.pkl does not exist, try loading features.json",
file=sys.stderr,
)
feature_path = os.path.join(model_dir, "features.json")
if os.path.exists(feature_path):
with open(feature_path, "r", encoding="utf8") as reader:
data = json.load(reader)
extractor.unigram = set(data["unigram"])
extractor.bigram = set(data["bigram"])
extractor.feature_to_idx = data["feature_to_idx"]
extractor.tag_to_idx = data["tag_to_idx"]
extractor.save(model_dir)
return extractor
print(
"WARNING: features.json does not exist, try loading using old format",
file=sys.stderr,
)
with open(
os.path.join(model_dir, "unigram_word.txt"),
"r",
encoding="utf8",
) as reader:
extractor.unigram = set([line.strip() for line in reader])
with open(
os.path.join(model_dir, "bigram_word.txt"),
"r",
encoding="utf8",
) as reader:
extractor.bigram = set(line.strip() for line in reader)
extractor.feature_to_idx = {}
feature_base_name = os.path.join(model_dir, "featureIndex.txt")
for i in range(10):
with open(
"{}_{}".format(feature_base_name, i), "r", encoding="utf8"
) as reader:
for line in reader:
feature, index = line.split(" ")
feature = ".".join(feature.split(".")[1:])
extractor.feature_to_idx[feature] = int(index)
extractor.tag_to_idx = {}
with open(
os.path.join(model_dir, "tagIndex.txt"), "r", encoding="utf8"
) as reader:
for line in reader:
tag, index = line.split(" ")
extractor.tag_to_idx[tag] = int(index)
print(
"INFO: features.json is saved",
file=sys.stderr,
)
extractor.save(model_dir)
return extractor
================================================
FILE: pkuseg/gradient.py
================================================
import pkuseg.model
from typing import List
import pkuseg.inference as _inf
import pkuseg.data
def get_grad_SGD_minibatch(
grad: List[float], model: pkuseg.model.Model, X: List[pkuseg.data.Example]
):
# if idset is not None:
# idset.clear()
all_id_set = set()
errors = 0
for x in X:
error, id_set = get_grad_CRF(grad, model, x)
errors += error
all_id_set.update(id_set)
return errors, all_id_set
def get_grad_CRF(
grad: List[float], model: pkuseg.model.Model, x: pkuseg.data.Example
):
id_set = set()
n_tag = model.n_tag
bel = _inf.belief(len(x), n_tag)
belMasked = _inf.belief(len(x), n_tag)
Ylist, YYlist, maskYlist, maskYYlist = _inf.getYYandY(model, x)
Z, sum_edge = _inf.get_beliefs(bel, model, x, Ylist, YYlist)
ZGold, sum_edge_masked = _inf.get_beliefs(belMasked, model, x, maskYlist, maskYYlist)
for i, node_feature_list in enumerate(x.features):
for feature_id in node_feature_list:
trans_id = model._get_node_tag_feature_id(feature_id, 0)
id_set.update(range(trans_id, trans_id + n_tag))
grad[trans_id:trans_id+n_tag] += bel.belState[i] - belMasked.belState[i]
backoff = model.n_feature * n_tag
grad[backoff:] += sum_edge - sum_edge_masked
id_set.update(range(backoff, backoff + n_tag * n_tag))
return Z - ZGold, id_set
================================================
FILE: pkuseg/inference.pyx
================================================
# distutils: language = c++
# cython: infer_types=True
# cython: language_level=3
cimport cython
import numpy as np
cimport numpy as np
from libcpp.vector cimport vector
from libc.math cimport exp, log
np.import_array()
class belief:
def __init__(self, nNodes, nStates):
self.belState = np.zeros((nNodes, nStates))
self.belEdge = np.zeros((nNodes, nStates * nStates))
self.Z = 0
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef get_beliefs(object bel, object m, object x, np.ndarray[double, ndim=2] Y, np.ndarray[double, ndim=2] YY):
cdef:
np.ndarray[double, ndim=2] belState = bel.belState
np.ndarray[double, ndim=2] belEdge = bel.belEdge
int nNodes = len(x)
int nTag = m.n_tag
double Z = 0
np.ndarray[double, ndim=1] alpha_Y = np.zeros(nTag)
np.ndarray[double, ndim=1] newAlpha_Y = np.zeros(nTag)
np.ndarray[double, ndim=1] tmp_Y = np.zeros(nTag)
np.ndarray[double, ndim=2] YY_trans = YY.transpose()
np.ndarray[double, ndim=1] YY_t_r = YY_trans.reshape(-1)
np.ndarray[double, ndim=1] sum_edge = np.zeros(nTag * nTag)
for i in range(nNodes - 1, 0, -1):
tmp_Y = belState[i] + Y[i]
belState[i-1] = logMultiply(YY, tmp_Y)
for i in range(nNodes):
if i > 0:
tmp_Y = alpha_Y.copy()
newAlpha_Y = logMultiply(YY_trans, tmp_Y) + Y[i]
else:
newAlpha_Y = Y[i].copy()
if i > 0:
tmp_Y = Y[i] + belState[i]
belEdge[i] = YY_t_r
for yPre in range(nTag):
for y in range(nTag):
belEdge[i, y * nTag + yPre] += tmp_Y[y] + alpha_Y[yPre]
belState[i] = belState[i] + newAlpha_Y
alpha_Y = newAlpha_Y
Z = logSum(alpha_Y)
for i in range(nNodes):
belState[i] = np.exp(belState[i] - Z)
for i in range(1, nNodes):
sum_edge += np.exp(belEdge[i] - Z)
return Z, sum_edge
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef run_viterbi(np.ndarray[double, ndim=2] node_score, np.ndarray[double, ndim=2] edge_score):
cdef int i, y, y_pre, i_pre, tag, w=node_score.shape[0], h=node_score.shape[1]
cdef double ma, sc
cdef np.ndarray[double, ndim=2] max_score = np.zeros((w, h), dtype=np.float64)
cdef np.ndarray[int, ndim=2] pre_tag = np.zeros((w, h), dtype=np.int32)
cdef np.ndarray[unsigned char, ndim=2] init_check = np.zeros((w, h), dtype=np.uint8)
cdef np.ndarray[int, ndim=1] states = np.zeros(w, dtype=np.int32)
for y in range(h):
max_score[w-1, y] = node_score[w-1, y]
for i in range(w - 2, -1, -1):
for y in range(h):
for y_pre in range(h):
i_pre = i + 1
sc = max_score[i_pre, y_pre] + node_score[i, y] + edge_score[y, y_pre]
if not init_check[i, y]:
init_check[i, y] = 1
max_score[i, y] = sc
pre_tag[i, y] = y_pre
elif sc >= max_score[i, y]:
max_score[i, y] = sc
pre_tag[i, y] = y_pre
ma = max_score[0, 0]
tag = 0
for y in range(1, h):
sc = max_score[0, y]
if ma < sc:
ma = sc
tag = y
states[0] = tag
for i in range(1, w):
tag = pre_tag[i-1, tag]
states[i] = tag
if ma > 300:
ma = 300
return exp(ma), states
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef getLogYY(vector[vector[int]] feature_temp, int num_tag, int backoff, np.ndarray[double, ndim=1] w, double scalar):
cdef:
int num_node = feature_temp.size()
np.ndarray[double, ndim=2] node_score = np.zeros((num_node, num_tag), dtype=np.float64)
np.ndarray[double, ndim=2] edge_score = np.ones((num_tag, num_tag), dtype=np.float64)
int s, s_pre, i
double maskValue, tmp
vector[int] f_list
int f, ft
for i in range(num_node):
f_list = feature_temp[i]
for ft in f_list:
for s in range(num_tag):
f = ft * num_tag + s
node_score[i, s] += w[f] * scalar
for s in range(num_tag):
for s_pre in range(num_tag):
f = backoff + s * num_tag + s_pre
edge_score[s_pre, s] += w[f] * scalar
return node_score, edge_score
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef maskY(object tags, int nNodes, int nTag, np.ndarray[double, ndim=2] Y):
cdef np.ndarray[double, ndim=2] mask_Yi = Y.copy()
cdef double maskValue = -1e100
cdef list tagList = tags
cdef int i
for i in range(nNodes):
for s in range(nTag):
if tagList[i] != s:
mask_Yi[i, s] = maskValue
return mask_Yi
@cython.boundscheck(False)
@cython.wraparound(False)
cdef logMultiply(np.ndarray[double, ndim=2] A, np.ndarray[double, ndim=1] B):
cdef int r, c
cdef np.ndarray[double, ndim=2] toSumLists = np.zeros_like(A)
cdef np.ndarray[double, ndim=1] ret = np.zeros(A.shape[0])
for r in range(A.shape[0]):
for c in range(A.shape[1]):
toSumLists[r, c] = A[r, c] + B[c]
for r in range(A.shape[0]):
ret[r] = logSum(toSumLists[r])
return ret
@cython.boundscheck(False)
@cython.wraparound(False)
cdef logSum(double[:] a):
cdef int n = a.shape[0]
cdef double s = a[0]
cdef double m1
cdef double m2
for i in range(1, n):
if s >= a[i]:
m1, m2 = s, a[i]
else:
m1, m2 = a[i], s
s = m1 + log(1 + exp(m2 - m1))
return s
def decodeViterbi_fast(feature_temp, model):
Y, YY = getLogYY(feature_temp, model.n_tag, model.n_feature*model.n_tag, model.w, 1.0)
numer, tags = run_viterbi(Y, YY)
tags = list(tags)
return numer, tags
def getYYandY(model, example):
Y, YY = getLogYY(example.features, model.n_tag, model.n_feature*model.n_tag, model.w, 1.0)
mask_Y = maskY(example.tags, len(example), model.n_tag, Y)
mask_YY = YY
return Y, YY, mask_Y, mask_YY
================================================
FILE: pkuseg/model.py
================================================
import os
import sys
import numpy as np
from .config import config
class Model:
def __init__(self, n_feature, n_tag):
self.n_tag = n_tag
self.n_feature = n_feature
self.n_transition_feature = n_tag * (n_feature + n_tag)
if config.random:
self.w = np.random.random(size=(self.n_transition_feature,)) * 2 - 1
else:
self.w = np.zeros(self.n_transition_feature)
def expand(self, n_feature, n_tag):
new_transition_feature = n_tag * (n_feature + n_tag)
if config.random:
new_w = np.random.random(size=(new_transition_feature,)) * 2 - 1
else:
new_w = np.zeros(new_transition_feature)
n_node = self.n_tag * self.n_feature
n_edge = self.n_tag * self.n_tag
new_w[:n_node] = self.w[:n_node]
new_w[-n_edge:] = self.w[-n_edge:]
self.n_tag = n_tag
self.n_feature = n_feature
self.n_transition_feature = new_transition_feature
self.w = new_w
def _get_node_tag_feature_id(self, feature_id, tag_id):
return feature_id * self.n_tag + tag_id
def _get_tag_tag_feature_id(self, pre_tag_id, tag_id):
return self.n_feature * self.n_tag + tag_id * self.n_tag + pre_tag_id
@classmethod
def load(cls, model_dir=None):
if model_dir is None:
model_dir = config.modelDir
model_path = os.path.join(model_dir, "weights.npz")
if os.path.exists(model_path):
npz = np.load(model_path)
sizes = npz["sizes"]
w = npz["w"]
model = cls.__new__(cls)
model.n_tag = int(sizes[0])
model.n_feature = int(sizes[1])
model.n_transition_feature = model.n_tag * (
model.n_feature + model.n_tag
)
model.w = w
assert model.w.shape[0] == model.n_transition_feature
return model
print(
"WARNING: weights.npz does not exist, try loading using old format",
file=sys.stderr,
)
model_path = os.path.join(model_dir, "model.txt")
with open(model_path, encoding="utf-8") as f:
ary = f.readlines()
model = cls.__new__(cls)
model.n_tag = int(ary[0].strip())
wsize = int(ary[1].strip())
w = np.zeros(wsize)
for i in range(2, wsize):
w[i - 2] = float(ary[i].strip())
model.w = w
model.n_feature = wsize // model.n_tag - model.n_tag
model.n_transition_feature = wsize
model.save(model_dir)
return model
@classmethod
def new(cls, model, copy_weight=True):
new_model = cls.__new__(cls)
new_model.n_tag = model.n_tag
if copy_weight:
new_model.w = model.w.copy()
else:
new_model.w = np.zeros_like(model.w)
new_model.n_feature = (
new_model.w.shape[0] // new_model.n_tag - new_model.n_tag
)
new_model.n_transition_feature = new_model.w.shape[0]
return new_model
def save(self, model_dir=None):
if model_dir is None:
model_dir = config.modelDir
sizes = np.array([self.n_tag, self.n_feature])
np.savez(
os.path.join(model_dir, "weights.npz"), sizes=sizes, w=self.w
)
# np.save
# with open(file, "w", encoding="utf-8") as f:
# f.write("{}\n{}\n".format(self.n_tag, self.w.shape[0]))
# for value in self.w:
# f.write("{:.4f}\n".format(value))
================================================
FILE: pkuseg/optimizer.py
================================================
import random
import numpy as np
import pkuseg.gradient as _grad
# from pkuseg.config import config
class Optimizer:
def __init__(self):
self._preVals = []
def converge_test(self, err):
val = 1e100
if len(self._preVals) > 1:
prevVal = self._preVals[0]
if len(self._preVals) == 10:
self._preVals.pop(0)
avgImprovement = (prevVal - err) / len(self._preVals)
relAvg = avgImprovement / abs(err)
val = relAvg
self._preVals.append(err)
return val
def optimize(self):
raise NotImplementedError()
class ADF(Optimizer):
def __init__(self, config, dataset, model):
super().__init__()
self.config = config
self._model = model
self._X = dataset
self.decayList = np.ones_like(self._model.w) * config.rate0
def optimize(self):
config = self.config
sample_size = 0
w = self._model.w
fsize = w.shape[0]
xsize = len(self._X)
grad = np.zeros(fsize)
error = 0
feature_count_list = np.zeros(fsize)
# feature_count_list = [0] * fsize
ri = list(range(xsize))
random.shuffle(ri)
update_interval = xsize // config.nUpdate
# config.interval = xsize // config.nUpdate
n_sample = 0
for t in range(0, xsize, config.miniBatch):
XX = []
end = False
for k in range(t, t + config.miniBatch):
i = ri[k]
x = self._X[i]
XX.append(x)
if k == xsize - 1:
end = True
break
mb_size = len(XX)
n_sample += mb_size
# fSet = set()
err, feature_set = _grad.get_grad_SGD_minibatch(
grad, self._model, XX
)
error += err
feature_set = list(feature_set)
feature_count_list[feature_set] += 1
# for i in feature_set:
# feature_count_list[i] += 1
check = False
for k in range(t, t + config.miniBatch):
if t != 0 and k % update_interval == 0:
check = True
# update decay rates
if check or end:
self.decayList *= (
config.upper
- (config.upper - config.lower)
* feature_count_list
/ n_sample
)
feature_count_list.fill(0)
# for i in range(fsize):
# v = feature_count_list[i]
# u = v / n_sample
# eta = config.upper - (config.upper - config.lower) * u
# self.decayList[i] *= eta
# feature_count_list
# for i in range(len(feature_count_list)):
# feature_count_list[i] = 0
# update weights
w[feature_set] -= self.decayList[feature_set] * grad[feature_set]
grad[feature_set] = 0
# for i in feature_set:
# w[i] -= self.decayList[i] * grad[i]
# grad[i] = 0
# reg
if check or end:
if config.reg != 0:
w -= self.decayList * (
w / (config.reg * config.reg) * n_sample / xsize
)
# for i in range(fsize):
# grad_i = (
# w[i] / (config.reg * config.reg) * (n_sample / xsize)
# )
# w[i] -= self.decayList[i] * grad_i
n_sample = 0
sample_size += mb_size
if config.reg != 0:
s = (w * w).sum()
error += s / (2.0 * config.reg * config.reg)
diff = self.converge_test(error)
return error, sample_size, diff
================================================
FILE: pkuseg/postag/__init__.py
================================================
from __future__ import print_function
import sys
import os
import time
from ..inference import decodeViterbi_fast
from .feature_extractor import FeatureExtractor
from .model import Model
class Postag:
def __init__(self, model_name):
modelDir = model_name
self.feature_extractor = FeatureExtractor.load(modelDir)
self.model = Model.load(modelDir)
self.idx_to_tag = {
idx: tag for tag, idx in self.feature_extractor.tag_to_idx.items()
}
self.n_feature = len(self.feature_extractor.feature_to_idx)
self.n_tag = len(self.feature_extractor.tag_to_idx)
# print("finish")
def _cut(self, text):
examples = list(self.feature_extractor.normalize_text(text))
length = len(examples)
all_feature = [] # type: List[List[int]]
for idx in range(length):
node_feature_idx = self.feature_extractor.get_node_features_idx(
idx, examples
)
# node_feature = self.feature_extractor.get_node_features(
# idx, examples
# )
# node_feature_idx = []
# for feature in node_feature:
# feature_idx = self.feature_extractor.feature_to_idx.get(feature)
# if feature_idx is not None:
# node_feature_idx.append(feature_idx)
# if not node_feature_idx:
# node_feature_idx.append(0)
all_feature.append(node_feature_idx)
_, tags = decodeViterbi_fast(all_feature, self.model)
tags = list(map(lambda x:self.idx_to_tag[x], tags))
return tags
def tag(self, sen):
"""txt: list[str], tags: list[str]"""
tags = self._cut(sen)
return tags
================================================
FILE: pkuseg/postag/feature_extractor.pyx
================================================
# distutils: language = c++
# cython: infer_types=True
# cython: language_level=3
import json
import os
import sys
import pickle
from collections import Counter
from itertools import product
import cython
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef get_slice_str(iterable, int start, int length, int all_len):
if start < 0 or start >= all_len:
return ""
if start + length >= all_len + 1:
return ""
return "".join(iterable[start : start + length])
@cython.boundscheck(False)
@cython.wraparound(False)
@cython.nonecheck(False)
def __get_node_features_idx(int idx, list nodes not None, dict feature_to_idx not None):
cdef:
list flist = []
Py_ssize_t i = idx
int length = len(nodes)
int j
w = nodes[i]
# $$ starts feature
flist.append(0)
# unigram/bgiram feature
feat = "w." + w
if feat in feature_to_idx:
feature = feature_to_idx[feat]
flist.append(feature)
for j in range(1, 4):
if len(w)>=j:
feat = "tr1.pre.%d.%s"%(j, w[:j])
if feat in feature_to_idx:
flist.append(feature_to_idx[feat])
feat = "tr1.post.%d.%s"%(j, w[-j:])
if feat in feature_to_idx:
flist.append(feature_to_idx[feat])
if i > 0:
feat = "tr1.w-1." + nodes[i - 1]
else:
feat = "tr1.w-1.BOS"
if feat in feature_to_idx:
flist.append(feature_to_idx[feat])
if i < length - 1:
feat = "tr1.w1." + nodes[i + 1]
else:
feat = "tr1.w1.EOS"
if feat in feature_to_idx:
flist.append(feature_to_idx[feat])
if i > 1:
feat = "tr1.w-2." + nodes[i - 2]
else:
feat = "tr1.w-2.BOS"
if feat in feature_to_idx:
flist.append(feature_to_idx[feat])
if i < length - 2:
feat = "tr1.w2." + nodes[i + 2]
else:
feat = "tr1.w2.EOS"
if feat in feature_to_idx:
flist.append(feature_to_idx[feat])
if i > 0:
feat = "tr1.w_-1_0." + nodes[i - 1] + "." + w
else:
feat = "tr1.w_-1_0.BOS"
if feat in feature_to_idx:
flist.append(feature_to_idx[feat])
if i < length - 1:
feat = "tr1.w_0_1." + w + "." + nodes[i + 1]
else:
feat = "tr1.w_0_1.EOS"
if feat in feature_to_idx:
flist.append(feature_to_idx[feat])
return flist
class FeatureExtractor:
keywords = "-._,|/*:"
num = set("0123456789." "几二三四五六七八九十千万亿兆零" "１２３４５６７８９０％")
letter = set(
"ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ" "ａｂｃｄｅｆｇｈｉｇｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ" "／・－"
)
keywords_translate_table = str.maketrans("-._,|/*:", "&&&&&&&&")
@classmethod
def keyword_rename(cls, text):
return text.translate(cls.keywords_translate_table)
@classmethod
def _num_letter_normalize(cls, word):
if not list(filter(lambda x:x not in cls.num, word)):
return "**Num"
return word
@classmethod
def normalize_text(cls, text):
for i in range(len(text)):
text[i] = cls.keyword_rename(text[i])
for character in text:
yield cls._num_letter_normalize(character)
def __init__(self):
# self.unigram = set() # type: Set[str]
# self.bigram = set() # type: Set[str]
self.feature_to_idx = {} # type: Dict[str, int]
self.tag_to_idx = {} # type: Dict[str, int]
def get_node_features_idx(self, idx, nodes):
return __get_node_features_idx(idx, nodes, self.feature_to_idx)
def get_node_features(self, idx, wordary):
cdef int length = len(wordary)
w = wordary[idx]
flist = []
# 1 start feature
flist.append("$$")
# 8 unigram/bgiram feature
flist.append("w." + w)
# prefix/suffix
for i in range(1, 4):
if len(w)>=i:
flist.append("tr1.pre.%d.%s"%(i, w[:i]))
flist.append("tr1.post.%d.%s"%(i, w[-i:]))
else:
flist.append("/")
flist.append("/")
if idx > 0:
flist.append("tr1.w-1." + wordary[idx - 1])
else:
flist.append("tr1.w-1.BOS")
if idx < len(wordary) - 1:
flist.append("tr1.w1." + wordary[idx + 1])
else:
flist.append("tr1.w1.EOS")
if idx > 1:
flist.append("tr1.w-2." + wordary[idx - 2])
else:
flist.append("tr1.w-2.BOS")
if idx < len(wordary) - 2:
flist.append("tr1.w2." + wordary[idx + 2])
else:
flist.append("tr1.w2.EOS")
if idx > 0:
flist.append("tr1.w_-1_0." + wordary[idx - 1] + "." + w)
else:
flist.append("tr1.w_-1_0.BOS")
if idx < len(wordary) - 1:
flist.append("tr1.w_0_1." + w + "." + wordary[idx + 1])
else:
flist.append("tr1.w_0_1.EOS")
return flist
def convert_feature_file_to_idx_file(
self, feature_file, feature_idx_file, tag_idx_file
):
with open(feature_file, "r", encoding="utf8") as reader:
lines = reader.readlines()
with open(feature_idx_file, "w", encoding="utf8") as f_writer, open(
tag_idx_file, "w", encoding="utf8"
) as t_writer:
f_writer.write("{}\n\n".format(len(self.feature_to_idx)))
t_writer.write("{}\n\n".format(len(self.tag_to_idx)))
tags_idx = [] # type: List[str]
features_idx = [] # type: List[List[str]]
for line in lines:
line = line.strip()
if not line:
# sentence finish
for feature_idx in features_idx:
if not feature_idx:
f_writer.write("0\n")
else:
f_writer.write(",".join(map(str, feature_idx)))
f_writer.write("\n")
f_writer.write("\n")
t_writer.write(",".join(map(str, tags_idx)))
t_writer.write("\n\n")
tags_idx = []
features_idx = []
continue
splits = line.split(" ")
feature_idx = [
self.feature_to_idx[feat]
for feat in splits[:-1]
if feat in self.feature_to_idx
]
features_idx.append(feature_idx)
if not splits[-1] in self.tag_to_idx:
tags_idx.append(-1)
else:
tags_idx.append(self.tag_to_idx[splits[-1]])
def convert_text_file_to_feature_file(
self, text_file, conll_file=None, feature_file=None
):
if conll_file is None:
conll_file = "{}.conll{}".format(*os.path.split(text_file))
if feature_file is None:
feature_file = "{}.feat{}".format(*os.path.split(text_file))
conll_line_format = "{} {}\n"
with open(text_file, "r", encoding="utf8") as reader, open(
conll_file, "w", encoding="utf8"
) as c_writer, open(feature_file, "w", encoding="utf8") as f_writer:
for line in reader.read().strip().replace("\r", "").split("\n\n"):
line = line.strip()
if not line:
continue
line = self.keyword_rename(line).split("\n")
words = []
tags = []
for word_tag in line:
word, tag = word_tag.split()
words.append(word)
tags.append(tag)
example = [
self._num_letter_normalize(word)
for word in words
]
for word, tag in zip(example, tags):
c_writer.write(conll_line_format.format(word, tag))
c_writer.write("\n")
for idx, tag in enumerate(tags):
features = self.get_node_features(idx, example)
features = [
(feature if feature in self.feature_to_idx else "/")
for feature in features
]
features.append(tag)
f_writer.write(" ".join(features))
f_writer.write("\n")
f_writer.write("\n")
def save(self, model_dir):
data = {}
data["feature_to_idx"] = self.feature_to_idx
data["tag_to_idx"] = self.tag_to_idx
with open(os.path.join(model_dir, 'features.pkl'), 'wb') as writer:
pickle.dump(data, writer, protocol=pickle.HIGHEST_PROTOCOL)
@classmethod
def load(cls, model_dir):
extractor = cls.__new__(cls)
feature_path = os.path.join(model_dir, "features.pkl")
if os.path.exists(feature_path):
with open(feature_path, "rb") as reader:
data = pickle.load(reader)
extractor.feature_to_idx = data["feature_to_idx"]
extractor.tag_to_idx = data["tag_to_idx"]
return extractor
print(
"WARNING: features.pkl does not exist, try loading features.json",
file=sys.stderr,
)
feature_path = os.path.join(model_dir, "features.json")
if os.path.exists(feature_path):
with open(feature_path, "r", encoding="utf8") as reader:
data = json.load(reader)
extractor.feature_to_idx = data["feature_to_idx"]
extractor.tag_to_idx = data["tag_to_idx"]
extractor.save(model_dir)
return extractor
print(
"WARNING: features.json does not exist, try loading using old format",
file=sys.stderr,
)
extractor.feature_to_idx = {}
feature_base_name = os.path.join(model_dir, "featureIndex.txt")
for i in range(10):
with open(
"{}_{}".format(feature_base_name, i), "r", encoding="utf8"
) as reader:
for line in reader:
feature, index = line.split(" ")
feature = ".".join(feature.split(".")[1:])
extractor.feature_to_idx[feature] = int(index)
extractor.tag_to_idx = {}
with open(
os.path.join(model_dir, "tagIndex.txt"), "r", encoding="utf8"
) as reader:
for line in reader:
tag, index = line.split(" ")
extractor.tag_to_idx[tag] = int(index)
print(
"INFO: features.json is saved",
file=sys.stderr,
)
extractor.save(model_dir)
return extractor
================================================
FILE: pkuseg/postag/model.py
================================================
import os
import sys
import numpy as np
class Model:
def __init__(self, n_feature, n_tag):
self.n_tag = n_tag
self.n_feature = n_feature
self.n_transition_feature = n_tag * (n_feature + n_tag)
self.w = np.zeros(self.n_transition_feature)
def _get_node_tag_feature_id(self, feature_id, tag_id):
return feature_id * self.n_tag + tag_id
def _get_tag_tag_feature_id(self, pre_tag_id, tag_id):
return self.n_feature * self.n_tag + tag_id * self.n_tag + pre_tag_id
@classmethod
def load(cls, model_dir):
model_path = os.path.join(model_dir, "weights.npz")
if os.path.exists(model_path):
npz = np.load(model_path)
sizes = npz["sizes"]
w = npz["w"]
model = cls.__new__(cls)
model.n_tag = int(sizes[0])
model.n_feature = int(sizes[1])
model.n_transition_feature = model.n_tag * (
model.n_feature + model.n_tag
)
model.w = w
assert model.w.shape[0] == model.n_transition_feature
return model
print(
"WARNING: weights.npz does not exist, try loading using old format",
file=sys.stderr,
)
model_path = os.path.join(model_dir, "model.txt")
with open(model_path, encoding="utf-8") as f:
ary = f.readlines()
model = cls.__new__(cls)
model.n_tag = int(ary[0].strip())
wsize = int(ary[1].strip())
w = np.zeros(wsize)
for i in range(2, wsize):
w[i - 2] = float(ary[i].strip())
model.w = w
model.n_feature = wsize // model.n_tag - model.n_tag
model.n_transition_feature = wsize
model.save(model_dir)
return model
@classmethod
def new(cls, model, copy_weight=True):
new_model = cls.__new__(cls)
new_model.n_tag = model.n_tag
if copy_weight:
new_model.w = model.w.copy()
else:
new_model.w = np.zeros_like(model.w)
new_model.n_feature = (
new_model.w.shape[0] // new_model.n_tag - new_model.n_tag
)
new_model.n_transition_feature = new_model.w.shape[0]
return new_model
def save(self, model_dir):
sizes = np.array([self.n_tag, self.n_feature])
np.savez(
os.path.join(model_dir, "weights.npz"), sizes=sizes, w=self.w
)
# np.save
# with open(file, "w", encoding="utf-8") as f:
# f.write("{}\n{}\n".format(self.n_tag, self.w.shape[0]))
# for value in self.w:
# f.write("{:.4f}\n".format(value))
================================================
FILE: pkuseg/res_summarize.py
================================================
import numpy as np
from .config import Config
import os
def tomatrix(s):
lines = s.split(Config.lineEnd)
lst = []
for line in lines:
if line == "":
continue
if not line.startswith("%"):
tmp = []
for i in line.split(Config.comma):
tmp.append(float(i))
lst.append(tmp)
return np.array(lst)
def summarize(config):
with open(
os.path.join(config.outDir, config.fResRaw), encoding="utf-8"
) as sr:
txt = sr.read()
txt = txt.replace("\r", "")
regions = txt.split(config.triLineEnd)
with open(
os.path.join(config.outDir, config.fResSum), "w", encoding="utf-8"
) as sw:
for region in regions:
if region == "":
continue
blocks = region.split(config.biLineEnd)
mList = []
for im in blocks:
mList.append(tomatrix(im))
avgM = np.zeros_like(mList[0])
for m in mList:
avgM = avgM + m
avgM = avgM / len(mList)
sqravgM = np.zeros_like(mList[0])
for m in mList:
sqravgM += m * m
sqravgM = sqravgM / len(mList)
deviM = (sqravgM - avgM * avgM) ** 0.5
sw.write("%averaged values:\n")
for i in range(avgM.shape[0]):
for j in range(avgM.shape[1]):
sw.write("{:.2f},".format(avgM[i, j]))
sw.write("\n")
sw.write("\n%deviations:\n")
for i in range(deviM.shape[0]):
for j in range(deviM.shape[1]):
sw.write("{:.2f},".format(deviM[i, j]))
# sw.write(("%.2f" % deviM[i, j]) + ",")
sw.write("\n")
sw.write("\n%avg & devi:\n")
for i in range(avgM.shape[0]):
for j in range(avgM.shape[1]):
sw.write("{:.2f}+-{:,2f},".format(avgM[i, j], deviM[i, j]))
sw.write("\n")
sw.write("%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\n\n")
def write(config, timeList, errList, diffList, scoreListList):
def log(message):
if config.rawResWrite:
config.swResRaw.write(message)
log("% training results:" + config.metric + "\n")
for i in range(config.ttlIter):
it = i
log("% iter#={} ".format(it))
lst = scoreListList[i]
if config.evalMetric == "f1":
log(
"% f-score={:.2f}% precision={:.2f}% recall={:.2f}% ".format(
lst[0], lst[1], lst[2]
)
)
else:
log("% {}={:.2f}% ".format(config.metric, lst[0]))
time = 0
for k in range(i + 1):
time += timeList[k]
log(
"cumulative-time(sec)={:.2f} objective={:.2f} diff={:.2f}\n".format(
time, errList[i], diffList[i]
)
)
# #ttlScore = 0
# for i in range(config.ttlIter):
# it = i + 1
# log("% iter#={} ".format(it))
# lst = scoreListList[i]
# # ttlScore += lst[0]
# if config.evalMetric == "f1":
# log(
# "% f-score={:.2f}% precision={:.2f}% recall={:.2f}% ".format(
# lst[0], lst[1], lst[2]
# )
# )
# else:
# log("% {}={:.2f}% ".format(config.metric, lst[0]))
# time = 0
# for k in range(i + 1):
# time += timeList[k]
# log(
# "cumulative-time(sec)={:.2f} objective={:.2f} diff={:.2f}\n".format(
# time, errList[i], diffList[i]
# )
# )
================================================
FILE: pkuseg/scorer.py
================================================
from pkuseg.config import Config
def getFscore(goldTagList, resTagList, idx_to_chunk_tag):
scoreList = []
assert len(resTagList) == len(goldTagList)
getNewTagList(idx_to_chunk_tag, goldTagList)
getNewTagList(idx_to_chunk_tag, resTagList)
goldChunkList = getChunks(goldTagList)
resChunkList = getChunks(resTagList)
gold_chunk = 0
res_chunk = 0
correct_chunk = 0
for i in range(len(goldChunkList)):
res = resChunkList[i]
gold = goldChunkList[i]
resChunkAry = res.split(Config.comma)
tmp = []
for t in resChunkAry:
if len(t) > 0:
tmp.append(t)
resChunkAry = tmp
goldChunkAry = gold.split(Config.comma)
tmp = []
for t in goldChunkAry:
if len(t) > 0:
tmp.append(t)
goldChunkAry = tmp
gold_chunk += len(goldChunkAry)
res_chunk += len(resChunkAry)
goldChunkSet = set()
for im in goldChunkAry:
goldChunkSet.add(im)
for im in resChunkAry:
if im in goldChunkSet:
correct_chunk += 1
pre = correct_chunk / res_chunk * 100
rec = correct_chunk / gold_chunk * 100
f1 = 0 if correct_chunk == 0 else 2 * pre * rec / (pre + rec)
scoreList.append(f1)
scoreList.append(pre)
scoreList.append(rec)
infoList = []
infoList.append(gold_chunk)
infoList.append(res_chunk)
infoList.append(correct_chunk)
return scoreList, infoList
def getNewTagList(tagMap, tagList):
tmpList = []
for im in tagList:
tagAry = im.split(Config.comma)
for i in range(len(tagAry)):
if tagAry[i] == "":
continue
index = int(tagAry[i])
if not index in tagMap:
raise Exception("Error")
tagAry[i] = tagMap[index]
newTags = ",".join(tagAry)
tmpList.append(newTags)
tagList.clear()
for im in tmpList:
tagList.append(im)
def getChunks(tagList):
tmpList = []
for im in tagList:
tagAry = im.split(Config.comma)
tmp = []
for t in tagAry:
if t != "":
tmp.append(t)
tagAry = tmp
chunks = ""
for i in range(len(tagAry)):
if tagAry[i].startswith("B"):
pos = i
length = 1
ty = tagAry[i]
for j in range(i + 1, len(tagAry)):
if tagAry[j] == "I":
length += 1
else:
break
chunk = ty + "*" + str(length) + "*" + str(pos)
chunks = chunks + chunk + ","
tmpList.append(chunks)
return tmpList
================================================
FILE: pkuseg/trainer.py
================================================
# from .config import config
# from .feature import *
# from .data_format import *
# from .toolbox import *
import os
import time
from multiprocessing import Process, Queue
from pkuseg import res_summarize
# from .inference import *
# from .config import Config
from pkuseg.config import Config, config
from pkuseg.data import DataSet
from pkuseg.feature_extractor import FeatureExtractor
# from .feature_generator import *
from pkuseg.model import Model
import pkuseg.inference as _inf
# from .inference import *
# from .gradient import *
from pkuseg.optimizer import ADF
from pkuseg.scorer import getFscore
# from typing import TextIO
# from .res_summarize import summarize
# from .res_summarize import write as reswrite
# from pkuseg.trainer import Trainer
def train(config=None):
if config is None:
config = Config()
if config.init_model is None:
feature_extractor = FeatureExtractor()
else:
feature_extractor = FeatureExtractor.load(config.init_model)
feature_extractor.build(config.trainFile)
feature_extractor.save()
feature_extractor.convert_text_file_to_feature_file(
config.trainFile, config.c_train, config.f_train
)
feature_extractor.convert_text_file_to_feature_file(
config.testFile, config.c_test, config.f_test
)
feature_extractor.convert_feature_file_to_idx_file(
config.f_train, config.fFeatureTrain, config.fGoldTrain
)
feature_extractor.convert_feature_file_to_idx_file(
config.f_test, config.fFeatureTest, config.fGoldTest
)
config.globalCheck()
config.swLog = open(os.path.join(config.outDir, config.fLog), "w")
config.swResRaw = open(os.path.join(config.outDir, config.fResRaw), "w")
config.swTune = open(os.path.join(config.outDir, config.fTune), "w")
print("\nstart training...")
config.swLog.write("\nstart training...\n")
print("\nreading training & test data...")
config.swLog.write("\nreading training & test data...\n")
trainset = DataSet.load(config.fFeatureTrain, config.fGoldTrain)
testset = DataSet.load(config.fFeatureTest, config.fGoldTest)
trainset = trainset.resize(config.trainSizeScale)
print(
"done! train/test data sizes: {}/{}".format(len(trainset), len(testset))
)
config.swLog.write(
"done! train/test data sizes: {}/{}\n".format(
len(trainset), len(testset)
)
)
config.swLog.write("\nr: {}\n".format(config.reg))
print("\nr: {}".format(config.reg))
if config.rawResWrite:
config.swResRaw.write("\n%r: {}\n".format(config.reg))
trainer = Trainer(config, trainset, feature_extractor)
time_list = []
err_list = []
diff_list = []
score_list_list = []
for i in range(config.ttlIter):
# config.glbIter += 1
time_s = time.time()
err, sample_size, diff = trainer.train_epoch()
time_t = time.time() - time_s
time_list.append(time_t)
err_list.append(err)
diff_list.append(diff)
score_list = trainer.test(testset, i)
score_list_list.append(score_list)
score = score_list[0]
logstr = "iter{} diff={:.2e} train-time(sec)={:.2f} {}={:.2f}%".format(
i, diff, time_t, config.metric, score
)
config.swLog.write(logstr + "\n")
config.swLog.write("------------------------------------------------\n")
config.swLog.flush()
print(logstr)
res_summarize.write(config, time_list, err_list, diff_list, score_list_list)
if config.save == 1:
trainer.model.save()
config.swLog.close()
config.swResRaw.close()
config.swTune.close()
res_summarize.summarize(config)
print("finished.")
class Trainer:
def __init__(self, config, dataset, feature_extractor):
self.config = config
self.X = dataset
self.n_feature = dataset.n_feature
self.n_tag = dataset.n_tag
if config.init_model is None:
self.model = Model(self.n_feature, self.n_tag)
else:
self.model = Model.load(config.init_model)
self.model.expand(self.n_feature, self.n_tag)
self.optim = self._get_optimizer(dataset, self.model)
self.feature_extractor = feature_extractor
self.idx_to_chunk_tag = {}
for tag, idx in feature_extractor.tag_to_idx.items():
if tag.startswith("I"):
tag = "I"
if tag.startswith("O"):
tag = "O"
self.idx_to_chunk_tag[idx] = tag
def _get_optimizer(self, dataset, model):
config = self.config
if "adf" in config.modelOptimizer:
return ADF(config, dataset, model)
raise ValueError("Invalid Optimizer")
def train_epoch(self):
return self.optim.optimize()
def test(self, testset, iteration):
outfile = os.path.join(config.outDir, config.fOutput.format(iteration))
func_mapping = {
"tok.acc": self._decode_tokAcc,
"str.acc": self._decode_strAcc,
"f1": self._decode_fscore,
}
with open(outfile, "w", encoding="utf8") as writer:
score_list = func_mapping[config.evalMetric](
testset, self.model, writer
)
for example in testset:
example.predicted_tags = None
return score_list
def _decode(self, testset: DataSet, model: Model):
if config.nThread == 1:
self._decode_single(testset, model)
else:
self._decode_multi_proc(testset, model)
def _decode_single(self, testset: DataSet, model: Model):
# n_tag = model.n_tag
for example in testset:
_, tags = _inf.decodeViterbi_fast(example.features, model)
example.predicted_tags = tags
@staticmethod
def _decode_proc(model, in_queue, out_queue):
while True:
item = in_queue.get()
if item is None:
return
idx, features = item
_, tags = _inf.decodeViterbi_fast(features, model)
out_queue.put((idx, tags))
def _decode_multi_proc(self, testset: DataSet, model: Model):
in_queue = Queue()
out_queue = Queue()
procs = []
nthread = self.config.nThread
for i in range(nthread):
p = Process(
target=self._decode_proc, args=(model, in_queue, out_queue)
)
procs.append(p)
for idx, example in enumerate(testset):
in_queue.put((idx, example.features))
for proc in procs:
in_queue.put(None)
proc.start()
for _ in range(len(testset)):
idx, tags = out_queue.get()
testset[idx].predicted_tags = tags
for p in procs:
p.join()
# token accuracy
def _decode_tokAcc(self, dataset, model, writer):
config = self.config
self._decode(dataset, model)
n_tag = model.n_tag
all_correct = [0] * n_tag
all_pred = [0] * n_tag
all_gold = [0] * n_tag
for example in dataset:
pred = example.predicted_tags
gold = example.tags
if writer is not None:
writer.write(",".join(map(str, pred)))
writer.write("\n")
for pred_tag, gold_tag in zip(pred, gold):
all_pred[pred_tag] += 1
all_gold[gold_tag] += 1
if pred_tag == gold_tag:
all_correct[gold_tag] += 1
config.swLog.write(
"% tag-type #gold #output #correct-output token-precision token-recall token-f-score\n"
)
sumGold = 0
sumOutput = 0
sumCorrOutput = 0
for i, (correct, gold, pred) in enumerate(
zip(all_correct, all_gold, all_pred)
):
sumGold += gold
sumOutput += pred
sumCorrOutput += correct
if gold == 0:
rec = 0
else:
rec = correct * 100.0 / gold
if pred == 0:
prec = 0
else:
prec = correct * 100.0 / pred
config.swLog.write(
"% {}: {} {} {} {:.2f} {:.2f} {:.2f}\n".format(
i,
gold,
pred,
correct,
prec,
rec,
(2 * prec * rec / (prec + rec)),
)
)
if sumGold == 0:
rec = 0
else:
rec = sumCorrOutput * 100.0 / sumGold
if sumOutput == 0:
prec = 0
else:
prec = sumCorrOutput * 100.0 / sumOutput
if prec == 0 and rec == 0:
fscore = 0
else:
fscore = 2 * prec * rec / (prec + rec)
config.swLog.write(
"% overall-tags: {} {} {} {:.2f} {:.2f} {:.2f}\n".format(
sumGold, sumOutput, sumCorrOutput, prec, rec, fscore
)
)
config.swLog.flush()
return [fscore]
def _decode_strAcc(self, dataset, model, writer):
config = self.config
self._decode(dataset, model)
correct = 0
total = len(dataset)
for example in dataset:
pred = example.predicted_tags
gold = example.tags
if writer is not None:
writer.write(",".join(map(str, pred)))
writer.write("\n")
for pred_tag, gold_tag in zip(pred, gold):
if pred_tag != gold_tag:
break
else:
correct += 1
acc = correct / total * 100.0
config.swLog.write(
"total-tag-strings={} correct-tag-strings={} string-accuracy={}%".format(
total, correct, acc
)
)
return [acc]
def _decode_fscore(self, dataset, model, writer):
config = self.config
self._decode(dataset, model)
gold_tags = []
pred_tags = []
for example in dataset:
pred = example.predicted_tags
gold = example.tags
pred_str = ",".join(map(str, pred))
pred_tags.append(pred_str)
if writer is not None:
writer.write(pred_str)
writer.write("\n")
gold_tags.append(",".join(map(str, gold)))
scoreList, infoList = getFscore(
gold_tags, pred_tags, self.idx_to_chunk_tag
)
config.swLog.write(
"#gold-chunk={} #output-chunk={} #correct-output-chunk={} precision={:.2f} recall={:.2f} f-score={:.2f}\n".format(
infoList[0],
infoList[1],
infoList[2],
scoreList[1],
scoreList[2],
scoreList[0],
)
)
return scoreList
# acc = correct / total * 100.0
# config.swLog.write(
# "total-tag-strings={} correct-tag-strings={} string-accuracy={}%".format(
# total, correct, acc
# )
# )
# goldTagList = []
# resTagList = []
# for x in X2:
# res = ""
# for im in x._yOutput:
# res += str(im) + ","
# resTagList.append(res)
# # if not dynamic:
# if writer is not None:
# for i in range(len(x._yOutput)):
# writer.write(str(x._yOutput[i]) + ",")
# writer.write("\n")
# goldTags = x._x.getTags()
# gold = ""
# for im in goldTags:
# gold += str(im) + ","
# goldTagList.append(gold)
# # if dynamic:
# # return resTagList
# scoreList = []
# if config.runMode == "train":
# infoList = []
# scoreList = getFscore(
# goldTagList, resTagList, infoList, self.idx_to_chunk_tag
# )
# config.swLog.write(
# "#gold-chunk={} #output-chunk={} #correct-output-chunk={} precision={:.2f} recall={:.2f} f-score={:.2f}\n".format(
# infoList[0],
# infoList[1],
# infoList[2],
# "%.2f" % scoreList[1],
# "%.2f" % scoreList[2],
# "%.2f" % scoreList[0],
# )
# )
# return scoreList
# # def multiThreading(self, X, X2):
# config = self.config
# # if dynamic:
# # for i in range(len(X)):
# # X2.append(dataSeqTest(X[i], []))
# # for k, x in enumerate(X2):
# # tags = []
# # prob = self.Inf.decodeViterbi_fast(self.Model, x._x, tags)
# # X2[k]._yOutput.clear()
# # X2[k]._yOutput.extend(tags)
# # return
# for i in range(len(X)):
# X2.append(dataSeqTest(X[i], []))
# if len(X) < config.nThread:
# config.nThread = len(X)
# interval = (len(X2) + config.nThread - 1) // config.nThread
# procs = []
# Q = Queue(5000)
# for i in range(config.nThread):
# start = i * interval
# end = min(start + interval, len(X2))
# proc = Process(
# target=Trainer.taskRunner_test,
# args=(self.Inf, self.Model, X2, start, end, Q),
# )
# proc.start()
# procs.append(proc)
# for i in range(len(X2)):
# t = Q.get()
# k, tags = t
# X2[k]._yOutput.clear()
# X2[k]._yOutput.extend(tags)
# for proc in procs:
# proc.join()
# @staticmethod
# def taskRunner_test(Inf, Model, X2, start, end, Q):
# for k in range(start, end):
# x = X2[k]
# tags = []
# prob = Inf.decodeViterbi_fast(Model, x._x, tags)
# Q.put((k, tags))
================================================
FILE: readme/comparison.md
================================================
# ϸѵԽ
ڲͬݼϵĶԱȽ
| MSRA | Precision | Recall | F-score |
| :----- | --------: | -----: | --------: |
| jieba | 87.01 | 89.88 | 88.42 |
| THULAC | 95.60 | 95.91 | 95.71 |
| pkuseg | 96.94 | 96.81 | **96.88** |
| CTB8 | Precision | Recall | F-score |
| :----- | --------: | -----: | --------: |
| jieba | 88.63 | 85.71 | 87.14 |
| THULAC | 93.90 | 95.30 | 94.56 |
| pkuseg | 95.99 | 95.39 | **95.69** |
| WEIBO | Precision | Recall | F-score |
| :----- | --------: | -----: | --------: |
| jieba | 87.79 | 87.54 | 87.66 |
| THULAC | 93.40 | 92.40 | 92.87 |
| pkuseg | 93.78 | 94.65 | **94.21** |
#### Խ
ѡ˻CTB8ϵѵѵͬʱвԣģģڡںݡϵķִЧѡCTB8ϵԭǣCTB8ڻϣµЧãڲǷCTB8ѵģͣй߰ԶԻøߵƽЧǿԵĽ
| CTB8 Training | MSRA | CTB8 | PKU | WEIBO | All Average | OOD Average |
| ------------- | ----- | ----- | ----- | ----- | ----------- | ----------- |
| jieba | 82.75 | 87.14 | 87.12 | 85.68 | 85.67 | 85.18 |
| THULAC | 83.50 | 94.56 | 89.13 | 91.00 | 89.55 | 87.88 |
| pkuseg | 83.67 | 95.69 | 89.67 | 91.19 | 90.06 | **88.18** |
У`All Average`ʾвԼ(CTB8Լ)F-scoreƽ`OOD Average` (Out-of-domain Average)ʾڳCTB8Լƽ
#### ĬģڲͬIJЧ
ǵܶûڳԷִʹߵʱʱʹù߰ԴģͲԡΪֱӶԱȡʼܣҲȽ˸߰ĬģڲͬIJЧע⣬ıȽֻΪ˵ĬµЧһǹƽġ
| Default | MSRA | CTB8 | PKU | WEIBO | All Average |
| ------- | :---: | :---: | :---: | :---: | :---------: |
| jieba | 81.45 | 79.58 | 81.83 | 83.56 | 81.61 |
| THULAC | 85.55 | 87.84 | 92.29 | 86.65 | 88.08 |
| pkuseg | 87.29 | 91.77 | 92.68 | 93.43 | **91.29** |
У`All Average`ʾвԼF-scoreƽ
================================================
FILE: readme/environment.md
================================================
# 实验环境
考虑到jieba分词和THULAC工具包等并没有提供细领域的预训练模型，为了便于比较，我们重新使用它们提供的训练接口在细领域的数据集上进行训练，用训练得到的模型进行中文分词。
我们选择Linux作为测试环境，在新闻数据(MSRA)、混合型文本(CTB8)、网络文本(WEIBO)数据上对不同工具包进行了准确率测试。我们使用了第二届国际汉语分词评测比赛提供的分词评价脚本。其中MSRA与WEIBO使用标准训练集测试集划分，CTB8采用随机划分。对于不同的分词工具包，训练测试数据的划分都是一致的；**即所有的分词工具包都在相同的训练集上训练，在相同的测试集上测试**。对于所有数据集，pkuseg使用了不使用词典的训练和测试接口。以下是pkuseg训练和测试代码示例:
```
pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models')
pkuseg.test('msr_test.raw', 'output.txt', user_dict=None)
```
================================================
FILE: readme/history.md
================================================
# 汾ʷ
- v0.0.11(2019-01-09)
- ĬãCTB8ΪĬģͣʹôʵ
- v0.0.14(2019-01-23)
- ˴ʵ䴦˴ʵ䣬ִЧ
- **ЧʽŻٶȽ֮ǰ汾9**
- ڴģݼѵͨģͣΪĬʹģ
- v0.0.15(2019-01-30)
- ֧fine-tuneѵԤصģͼѵ֧趨ѵ
- v0.0.18(2019-02-20)
- **ִ֧Աעҽơϸģ**
================================================
FILE: readme/interface.md
================================================
# 代码示例
以下代码示例适用于python交互式环境。
代码示例1：使用默认配置进行分词（**如果用户无法确定分词领域，推荐使用默认模型分词**）
```python3
import pkuseg
seg = pkuseg.pkuseg() # 以默认配置加载模型
text = seg.cut('我爱北京天安门') # 进行分词
print(text)
```
代码示例2：细领域分词（**如果用户明确分词领域，推荐使用细领域模型分词**）
```python3
import pkuseg
seg = pkuseg.pkuseg(model_name='medicine') # 程序会自动下载所对应的细领域模型
text = seg.cut('我爱北京天安门') # 进行分词
print(text)
```
代码示例3：分词同时进行词性标注，各词性标签的详细含义可参考 [tags.txt](https://github.com/lancopku/pkuseg-python/blob/master/tags.txt)
```python3
import pkuseg
seg = pkuseg.pkuseg(postag=True) # 开启词性标注功能
text = seg.cut('我爱北京天安门') # 进行分词和词性标注
print(text)
```
代码示例4：对文件分词
```python3
import pkuseg
# 对input.txt的文件分词输出到output.txt中
# 开20个进程
pkuseg.test('input.txt', 'output.txt', nthread=20)
```
代码示例5：额外使用用户自定义词典
```python3
import pkuseg
seg = pkuseg.pkuseg(user_dict='my_dict.txt') # 给定用户词典为当前目录下的"my_dict.txt"
text = seg.cut('我爱北京天安门') # 进行分词
print(text)
```
代码示例6：使用自训练模型分词（以CTB8模型为例）
```python3
import pkuseg
seg = pkuseg.pkuseg(model_name='./ctb8') # 假设用户已经下载好了ctb8的模型并放在了'./ctb8'目录下，通过设置model_name加载该模型
text = seg.cut('我爱北京天安门') # 进行分词
print(text)
```
代码示例7：训练新模型 （模型随机初始化）
```python3
import pkuseg
# 训练文件为'msr_training.utf8'
# 测试文件为'msr_test_gold.utf8'
# 训练好的模型存到'./models'目录下
# 训练模式下会保存最后一轮模型作为最终模型
# 目前仅支持utf-8编码，训练集和测试集要求所有单词以单个或多个空格分开
pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models')
```
代码示例8：fine-tune训练（从预加载的模型继续训练）
```python3
import pkuseg
# 训练文件为'train.txt'
# 测试文件为'test.txt'
# 加载'./pretrained'目录下的模型，训练好的模型保存在'./models'，训练10轮
pkuseg.train('train.txt', 'test.txt', './models', train_iter=10, init_model='./pretrained')
================================================
FILE: readme/multiprocess.md
================================================
# 多进程分词
当将以上代码示例置于文件中运行时，如涉及多进程功能，请务必使用`if __name__ == '__main__'`保护全局语句，如：
mp.py文件
```python3
import pkuseg
if __name__ == '__main__':
pkuseg.test('input.txt', 'output.txt', nthread=20)
pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models', nthread=20)
```
运行
```
python3 mp.py
```
详见[无法使用多进程分词和训练功能，提示RuntimeError和BrokenPipeError](https://github.com/lancopku/pkuseg-python/wiki#3-无法使用多进程分词和训练功能提示runtimeerror和brokenpipeerror)。
**在Windows平台上，请当文件足够大时再使用多进程分词功能**，详见[关于多进程速度问题](https://github.com/lancopku/pkuseg-python/wiki#9-关于多进程速度问题)。
================================================
FILE: readme/readme_english.md
================================================
# Pkuseg
A multi-domain Chinese word segmentation toolkit.
## Highlights
The pkuseg-python toolkit has the following features:
1. Supporting multi-domain Chinese word segmentation. Pkuseg-python supports multi-domain segmentation, including domains like news, web, medicine, and tourism. Users are free to choose different pre-trained models according to the domain features of the text to be segmented. If not sure the domain of the text, users are recommended to use the default model trained on mixed-domain data.
2. Higher word segmentation results. Compared with existing word segmentation toolkits, pkuseg-python can achieve higher F1 scores on the same dataset.
3. Supporting model training. Pkuseg-python also supports users to train a new segmentation model with their own data.
4. Supporting POS tagging. We also provide users POS tagging interfaces for further lexical analysis.
## Installation
- Requirements: python3
1. Install pkuseg-python by using PyPI: (with the default model trained on mixed-doimain data)
```
pip3 install pkuseg
```
or update to the latest version (**suggested**):
```
pip3 install -U pkuseg
```
2. Install pkuseg-python by using image source for fast speed:
```
pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple pkuseg
```
or update to the latest version (**suggested**):
```
pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple -U pkuseg
```
Note: The previous two installing commands only support python3.5, python3.6, python3.7 on linux, mac, and **windows 64 bit**.
3. If the code is downloaded from GitHub, please run the following command to install pkuseg-python:
```
python setup.py build_ext -i
```
Note: the github code does not contain the pre-trained models, users need to download the pre-trained models from [release](https://github.com/lancopku/pkuseg-python/releases), and set parameter 'model_name' as the model path.
## Usage
#### Examples
Example 1: Segmentation under the default configuration. **If users are not sure the domain of the text to be segmented, the default configuration is recommended.**
```python3
import pkuseg
seg = pkuseg.pkuseg() #load the default model
text = seg.cut('我爱北京天安门')
print(text)
```
Example 2: Domain-specific segmentation. **If users know the text domain, they can select a pre-trained domain model according to the domain features.**
```python3
import pkuseg
seg = pkuseg.pkuseg(model_name='medicine')
#Automatically download the domain-specific model.
text = seg.cut('我爱北京天安门')
print(text)
```
Example 3：Segmentation and POS tagging. For the detailed meaning of each POS tag, please refer to [tags.txt](https://github.com/lancopku/pkuseg-python/blob/master/tags.txt).
```python3
import pkuseg
seg = pkuseg.pkuseg(postag=True)
text = seg.cut('我爱北京天安门')
print(text)
```
Example 4：Segmentation with a text file as input.
```python3
import pkuseg
#Take file 'input.txt' as input.
#The segmented result is stored in file 'output.txt'.
pkuseg.test('input.txt', 'output.txt', nthread=20)
```
Example 5: Segmentation with a user-defined dictionary.
```python3
import pkuseg
seg = pkuseg.pkuseg(user_dict='my_dict.txt')
text = seg.cut('我爱北京天安门')
print(text)
```
Example 6: Segmentation with a user-trained model. Take CTB8 as an example.
```python3
import pkuseg
seg = pkuseg.pkuseg(model_name='./ctb8')
text = seg.cut('我爱北京天安门')
print(text)
```
Example 7: Training a new model (randomly initialized).
```python3
import pkuseg
# Training file: 'msr_training.utf8'.
# Test file: 'msr_test_gold.utf8'.
# Save the trained model to './models'.
# The training and test files are in utf-8 encoding.
pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models')
```
Example 8: Fine-tuning. Take a pre-trained model as input.
```python3
import pkuseg
# Training file: 'train.txt'.
# Testing file'test.txt'.
# The path of the pre-trained model: './pretrained'.
# Save the trained model to './models'.
# The training and test files are in utf-8 encoding.
pkuseg.train('train.txt', 'test.txt', './models', train_iter=10, init_model='./pretrained')
```
#### Parameter Settings
Segmentation for sentences.
```
pkuseg.pkuseg(model_name = "default", user_dict = "default", postag = False)
model_name The path of the used model.
"default". The default mixed-domain model.
"news". The model trained on news domain data.
"web". The model trained on web domain data.
"medicine". The model trained on medicine domain data.
"tourism". The model trained on tourism domain data.
model_path. Load a model from the user-specified path.
user_dict Set up the user dictionary.
"default". Use the default dictionary.
None. No dictionary is used.
dict_path. The path of the user-defined dictionary. Each line only contains one word.
postag POS tagging or not.
False. The default setting. Segmentation without POS tagging.
True. Segmentation with POS tagging.
```
Segmentation for documents.
```
pkuseg.test(readFile, outputFile, model_name = "default", user_dict = "default", postag = False, nthread = 10)
readFile The path of the input file.
outputFile The path of the output file.
model_name The path of the used model. Refer to pkuseg.pkuseg.
user_dict The path of the user dictionary. Refer to pkuseg.pkuseg.
postag POS tagging or not. Refer to pkuseg.pkuseg.
nthread The number of threads.
```
Model training.
```
pkuseg.train(trainFile, testFile, savedir, train_iter = 20, init_model = None)
trainFile The path of the training file.
testFile The path of the test file.
savedir The saved path of the trained model.
train_iter The maximum number of training epochs.
init_model By default, None means random initialization. Users can also load a pre-trained model as initialization, like init_model='./models/'.
```
## Publication
The toolkit is mainly based on the following publication. If you use the toolkit, please cite the paper:
* Ruixuan Luo, Jingjing Xu, Yi Zhang, Zhiyuan Zhang, Xuancheng Ren, Xu Sun. [PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation](https://arxiv.org/abs/1906.11455). Arxiv. 2019.
```
@article{pkuseg,
author = {Luo, Ruixuan and Xu, Jingjing and Zhang, Yi and Zhang, Zhiyuan and Ren, Xuancheng and Sun, Xu},
journal = {CoRR},
title = {PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation.},
url = {https://arxiv.org/abs/1906.11455},
volume = {abs/1906.11455},
year = 2019
}
```
## Related Work
* Xu Sun, Houfeng Wang, Wenjie Li. Fast Online Training with Frequency-Adaptive Learning Rates for Chinese Word Segmentation and New Word Detection. ACL. 2012.
* Jingjing Xu and Xu Sun. Dependency-based gated recursive neural network for chinese word segmentation. ACL. 2016.
* Jingjing Xu and Xu Sun. Transfer learning for low-resource chinese word segmentation with a novel neural network. NLPCC. 2017.
## Authors
Ruixuan Luo, Jingjing Xu, Xuancheng Ren, Yi Zhang, Zhiyuan Zhang, Bingzhen Wei, Xu Sun
[Language Computing and Machine Learning Group](http://lanco.pku.edu.cn/), Peking University
================================================
FILE: setup.py
================================================
import setuptools
import os
from distutils.extension import Extension
import numpy as np
def is_source_release(path):
return os.path.exists(os.path.join(path, "PKG-INFO"))
def setup_package():
root = os.path.abspath(os.path.dirname(__file__))
long_description = "pkuseg-python"
extensions = [
Extension(
"pkuseg.inference",
["pkuseg/inference.pyx"],
include_dirs=[np.get_include()],
language="c++"
),
Extension(
"pkuseg.feature_extractor",
["pkuseg/feature_extractor.pyx"],
include_dirs=[np.get_include()],
),
Extension(
"pkuseg.postag.feature_extractor",
["pkuseg/postag/feature_extractor.pyx"],
include_dirs=[np.get_include()],
),
]
if not is_source_release(root):
from Cython.Build import cythonize
extensions = cythonize(extensions, annotate=True)
setuptools.setup(
name="pkuseg",
version="0.0.25",
author="Lanco",
author_email="luoruixuan97@pku.edu.cn",
description="A small package for Chinese word segmentation",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/lancopku/pkuseg-python",
packages=setuptools.find_packages(),
package_data={"": ["*.txt*", "*.pkl", "*.npz", "*.pyx", "*.pxd"]},
classifiers=[
"Programming Language :: Python :: 3",
"License :: Other/Proprietary License",
"Operating System :: OS Independent",
],
install_requires=["cython", "numpy>=1.16.0"],
setup_requires=["cython", "numpy>=1.16.0"],
ext_modules=extensions,
zip_safe=False,
)
if __name__ == "__main__":
setup_package()
gitextract_zd_f4sqf/ ├── .gitignore ├── LICENSE ├── README.md ├── pkuseg/ │ ├── __init__.py │ ├── config.py │ ├── data.py │ ├── dicts/ │ │ ├── __init__.py │ │ └── default.pkl │ ├── download.py │ ├── feature_extractor.pyx │ ├── gradient.py │ ├── inference.pyx │ ├── model.py │ ├── optimizer.py │ ├── postag/ │ │ ├── __init__.py │ │ ├── feature_extractor.pyx │ │ └── model.py │ ├── res_summarize.py │ ├── scorer.py │ └── trainer.py ├── readme/ │ ├── comparison.md │ ├── environment.md │ ├── history.md │ ├── interface.md │ ├── multiprocess.md │ └── readme_english.md └── setup.py
SYMBOL INDEX (95 symbols across 13 files)
FILE: pkuseg/__init__.py
class TrieNode (line 24) | class TrieNode:
method __init__ (line 27) | def __init__(self, isword):
class Preprocesser (line 33) | class Preprocesser:
method __init__ (line 36) | def __init__(self, dict_file):
method insert (line 62) | def insert(self, word, usertag):
method solve (line 74) | def solve(self, txt):
class Postprocesser (line 124) | class Postprocesser:
method __init__ (line 126) | def __init__(self, common_name, other_names):
method post_process (line 152) | def post_process(self, sent, check_seperated):
method __call__ (line 184) | def __call__(self, sent):
class pkuseg (line 189) | class pkuseg:
method __init__ (line 190) | def __init__(self, model_name="default", user_dict="default", postag=F...
method _cut (line 261) | def _cut(self, text):
method cut (line 307) | def cut(self, txt):
function train (line 348) | def train(trainFile, testFile, savedir, train_iter=20, init_model=None):
function _test_single_proc (line 378) | def _test_single_proc(
function _proc_deprecated (line 416) | def _proc_deprecated(seg, lines, start, end, q):
function _proc (line 423) | def _proc(seg, in_queue, out_queue):
function _proc_alt (line 439) | def _proc_alt(model_name, user_dict, postag, in_queue, out_queue):
function _test_multi_proc (line 453) | def _test_multi_proc(
function test (line 539) | def test(
FILE: pkuseg/config.py
class Config (line 5) | class Config:
method __init__ (line 38) | def __init__(self):
method globalCheck (line 115) | def globalCheck(self):
FILE: pkuseg/data.py
class DataSet (line 213) | class DataSet:
method __init__ (line 214) | def __init__(self, n_tag=0, n_feature=0):
method __len__ (line 224) | def __len__(self):
method __iter__ (line 227) | def __iter__(self):
method __getitem__ (line 230) | def __getitem__(self, x):
method iterator (line 233) | def iterator(self):
method append (line 237) | def append(self, x):
method clear (line 240) | def clear(self):
method randomShuffle (line 243) | def randomShuffle(self):
method resize (line 252) | def resize(self, scale):
method load (line 263) | def load(cls, feature_idx_file, tag_idx_file):
class Example (line 327) | class Example:
method __init__ (line 328) | def __init__(self, features, tags):
method __len__ (line 333) | def __len__(self):
FILE: pkuseg/download.py
function download_model (line 30) | def download_model(url, model_dir, hash_prefix, progress=True):
function _download_url_to_file (line 42) | def _download_url_to_file(url, dst, hash_prefix, progress):
class tqdm (line 84) | class tqdm(object):
method __init__ (line 86) | def __init__(self, total, disable=False):
method update (line 91) | def update(self, n):
method __enter__ (line 99) | def __enter__(self):
method __exit__ (line 102) | def __exit__(self, exc_type, exc_val, exc_tb):
function unzip_file (line 108) | def unzip_file(zip_name, target_dir):
FILE: pkuseg/gradient.py
function get_grad_SGD_minibatch (line 8) | def get_grad_SGD_minibatch(
function get_grad_CRF (line 23) | def get_grad_CRF(
FILE: pkuseg/model.py
class Model (line 8) | class Model:
method __init__ (line 9) | def __init__(self, n_feature, n_tag):
method expand (line 19) | def expand(self, n_feature, n_tag):
method _get_node_tag_feature_id (line 34) | def _get_node_tag_feature_id(self, feature_id, tag_id):
method _get_tag_tag_feature_id (line 37) | def _get_tag_tag_feature_id(self, pre_tag_id, tag_id):
method load (line 41) | def load(cls, model_dir=None):
method new (line 82) | def new(cls, model, copy_weight=True):
method save (line 96) | def save(self, model_dir=None):
FILE: pkuseg/optimizer.py
class Optimizer (line 9) | class Optimizer:
method __init__ (line 10) | def __init__(self):
method converge_test (line 13) | def converge_test(self, err):
method optimize (line 25) | def optimize(self):
class ADF (line 29) | class ADF(Optimizer):
method __init__ (line 30) | def __init__(self, config, dataset, model):
method optimize (line 40) | def optimize(self):
FILE: pkuseg/postag/__init__.py
class Postag (line 13) | class Postag:
method __init__ (line 14) | def __init__(self, model_name):
method _cut (line 28) | def _cut(self, text):
method tag (line 55) | def tag(self, sen):
FILE: pkuseg/postag/model.py
class Model (line 5) | class Model:
method __init__ (line 6) | def __init__(self, n_feature, n_tag):
method _get_node_tag_feature_id (line 13) | def _get_node_tag_feature_id(self, feature_id, tag_id):
method _get_tag_tag_feature_id (line 16) | def _get_tag_tag_feature_id(self, pre_tag_id, tag_id):
method load (line 20) | def load(cls, model_dir):
method new (line 59) | def new(cls, model, copy_weight=True):
method save (line 73) | def save(self, model_dir):
FILE: pkuseg/res_summarize.py
function tomatrix (line 6) | def tomatrix(s):
function summarize (line 20) | def summarize(config):
function write (line 74) | def write(config, timeList, errList, diffList, scoreListList):
FILE: pkuseg/scorer.py
function getFscore (line 4) | def getFscore(goldTagList, resTagList, idx_to_chunk_tag):
function getNewTagList (line 50) | def getNewTagList(tagMap, tagList):
function getChunks (line 68) | def getChunks(tagList):
FILE: pkuseg/trainer.py
function train (line 34) | def train(config=None):
class Trainer (line 131) | class Trainer:
method __init__ (line 132) | def __init__(self, config, dataset, feature_extractor):
method _get_optimizer (line 155) | def _get_optimizer(self, dataset, model):
method train_epoch (line 162) | def train_epoch(self):
method test (line 165) | def test(self, testset, iteration):
method _decode (line 185) | def _decode(self, testset: DataSet, model: Model):
method _decode_single (line 191) | def _decode_single(self, testset: DataSet, model: Model):
method _decode_proc (line 198) | def _decode_proc(model, in_queue, out_queue):
method _decode_multi_proc (line 207) | def _decode_multi_proc(self, testset: DataSet, model: Model):
method _decode_tokAcc (line 233) | def _decode_tokAcc(self, dataset, model, writer):
method _decode_strAcc (line 314) | def _decode_strAcc(self, dataset, model, writer):
method _decode_fscore (line 345) | def _decode_fscore(self, dataset, model, writer):
FILE: setup.py
function is_source_release (line 7) | def is_source_release(path):
function setup_package (line 10) | def setup_package():
Condensed preview — 27 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (142K chars).
[
{
"path": ".gitignore",
"chars": 93,
"preview": ".vscode/\n*.cpp\n*.pyd\n*.html\n**/__pycache__/\n*.egg-info\n*.txt*\nbuild\ntmp/\ndata/\nmodels/\n*stats"
},
{
"path": "LICENSE",
"chars": 1076,
"preview": "MIT License\n\nCopyright (c) 2018-2019 pkuseg authors\n\nPermission is hereby granted, free of charge, to any person obtaini"
},
{
"path": "README.md",
"chars": 7698,
"preview": "# pkuseg：一个多领域中文分词工具包 [**(English Version)**](readme/readme_english.md)\r\n\r\npkuseg 是基于论文[[Luo et. al, 2019](#论文引用)]的工具包。其"
},
{
"path": "pkuseg/__init__.py",
"chars": 16945,
"preview": "from __future__ import print_function\nimport sys\n\nif sys.version_info[0] < 3:\n print(\"pkuseg does not support python2"
},
{
"path": "pkuseg/config.py",
"chars": 5167,
"preview": "import os\nimport tempfile\n\n\nclass Config:\n lineEnd = \"\\n\"\n biLineEnd = \"\\n\\n\"\n triLineEnd = \"\\n\\n\\n\"\n undrln"
},
{
"path": "pkuseg/data.py",
"chars": 14421,
"preview": "# from .config import Config\n# from pkuseg.feature_generator import\n# import os\nimport copy\nimport random\n\n\n# class data"
},
{
"path": "pkuseg/dicts/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "pkuseg/download.py",
"chars": 3658,
"preview": "from __future__ import absolute_import, division, print_function, unicode_literals\n\nimport hashlib\nimport os\nimport re\ni"
},
{
"path": "pkuseg/feature_extractor.pyx",
"chars": 21245,
"preview": "# distutils: language = c++\n# cython: infer_types=True\n# cython: language_level=3\nimport json\nimport os\nimport sys\nimpor"
},
{
"path": "pkuseg/gradient.py",
"chars": 1394,
"preview": "import pkuseg.model\nfrom typing import List\n\nimport pkuseg.inference as _inf\nimport pkuseg.data\n\n\ndef get_grad_SGD_minib"
},
{
"path": "pkuseg/inference.pyx",
"chars": 6080,
"preview": "# distutils: language = c++\n# cython: infer_types=True\n# cython: language_level=3\ncimport cython\nimport numpy as np\ncimp"
},
{
"path": "pkuseg/model.py",
"chars": 3568,
"preview": "import os\nimport sys\nimport numpy as np\n\nfrom .config import config\n\n\nclass Model:\n def __init__(self, n_feature, n_t"
},
{
"path": "pkuseg/optimizer.py",
"chars": 3983,
"preview": "import random\n\nimport numpy as np\nimport pkuseg.gradient as _grad\n\n# from pkuseg.config import config\n\n\nclass Optimizer:"
},
{
"path": "pkuseg/postag/__init__.py",
"chars": 1772,
"preview": "from __future__ import print_function\nimport sys\n\nimport os\nimport time\n\nfrom ..inference import decodeViterbi_fast\n\n\nfr"
},
{
"path": "pkuseg/postag/feature_extractor.pyx",
"chars": 10790,
"preview": "# distutils: language = c++\n# cython: infer_types=True\n# cython: language_level=3\nimport json\nimport os\nimport sys\nimpor"
},
{
"path": "pkuseg/postag/model.py",
"chars": 2678,
"preview": "import os\nimport sys\nimport numpy as np\n\nclass Model:\n def __init__(self, n_feature, n_tag):\n\n self.n_tag = n_"
},
{
"path": "pkuseg/res_summarize.py",
"chars": 3732,
"preview": "import numpy as np\nfrom .config import Config\nimport os\n\n\ndef tomatrix(s):\n lines = s.split(Config.lineEnd)\n lst ="
},
{
"path": "pkuseg/scorer.py",
"chars": 2753,
"preview": "from pkuseg.config import Config\n\n\ndef getFscore(goldTagList, resTagList, idx_to_chunk_tag):\n scoreList = []\n asse"
},
{
"path": "pkuseg/trainer.py",
"chars": 14174,
"preview": "# from .config import config\n# from .feature import *\n# from .data_format import *\n# from .toolbox import *\nimport os\nim"
},
{
"path": "readme/comparison.md",
"chars": 1597,
"preview": "\n\n# ϸѵԽ\n\nڲͬݼϵĶԱȽ\n\n| MSRA | Precision | Recall | F-score |\n| :----- | --------: | -----: | --------: |\n| jieba | "
},
{
"path": "readme/environment.md",
"chars": 480,
"preview": "# 实验环境\n\n考虑到jieba分词和THULAC工具包等并没有提供细领域的预训练模型，为了便于比较，我们重新使用它们提供的训练接口在细领域的数据集上进行训练，用训练得到的模型进行中文分词。\n\n我们选择Linux作为测试环境，在新闻数据(M"
},
{
"path": "readme/history.md",
"chars": 202,
"preview": "# 汾ʷ\n\n\n- v0.0.11(2019-01-09)\n - ĬãCTB8ΪĬģͣʹôʵ\n- v0.0.14(2019-01-23)\n - ˴ʵ䴦˴ʵ䣬ִЧ\n - **ЧʽŻٶȽ֮ǰ汾9**\n - ڴģݼѵͨģͣΪĬʹģ\n- "
},
{
"path": "readme/interface.md",
"chars": 1666,
"preview": "# 代码示例\n\n以下代码示例适用于python交互式环境。\n\n代码示例1：使用默认配置进行分词（**如果用户无法确定分词领域，推荐使用默认模型分词**）\n```python3\nimport pkuseg\n\nseg = pkuseg.pkus"
},
{
"path": "readme/multiprocess.md",
"chars": 567,
"preview": "\n# 多进程分词\n\n当将以上代码示例置于文件中运行时，如涉及多进程功能，请务必使用`if __name__ == '__main__'`保护全局语句，如： \nmp.py文件\n```python3\nimport pkuseg\n\nif __n"
},
{
"path": "readme/readme_english.md",
"chars": 7166,
"preview": "\n# Pkuseg \n\nA multi-domain Chinese word segmentation toolkit.\n\n## Highlights\n\nThe pkuseg-python toolkit has the followin"
},
{
"path": "setup.py",
"chars": 1862,
"preview": "import setuptools\nimport os\nfrom distutils.extension import Extension\n\nimport numpy as np\n\ndef is_source_release(path):\n"
}
]
// ... and 1 more files (download for full content)
About this extraction
This page contains the full source code of the lancopku/pkuseg-python GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 27 files (131.6 KB), approximately 36.8k tokens, and a symbol index with 95 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.