Repository: lancopku/pkuseg-python Branch: master Commit: 071d57c7df9a Files: 27 Total size: 131.6 KB Directory structure: gitextract_zd_f4sqf/ ├── .gitignore ├── LICENSE ├── README.md ├── pkuseg/ │ ├── __init__.py │ ├── config.py │ ├── data.py │ ├── dicts/ │ │ ├── __init__.py │ │ └── default.pkl │ ├── download.py │ ├── feature_extractor.pyx │ ├── gradient.py │ ├── inference.pyx │ ├── model.py │ ├── optimizer.py │ ├── postag/ │ │ ├── __init__.py │ │ ├── feature_extractor.pyx │ │ └── model.py │ ├── res_summarize.py │ ├── scorer.py │ └── trainer.py ├── readme/ │ ├── comparison.md │ ├── environment.md │ ├── history.md │ ├── interface.md │ ├── multiprocess.md │ └── readme_english.md └── setup.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ .vscode/ *.cpp *.pyd *.html **/__pycache__/ *.egg-info *.txt* build tmp/ data/ models/ *stats ================================================ FILE: LICENSE ================================================ MIT License Copyright (c) 2018-2019 pkuseg authors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ # pkuseg：一个多领域中文分词工具包 [**(English Version)**](readme/readme_english.md) pkuseg 是基于论文[[Luo et. al, 2019](#论文引用)]的工具包。其简单易用，支持细分领域分词，有效提升了分词准确度。 ## 目录 * [主要亮点](#主要亮点) * [编译和安装](#编译和安装) * [各类分词工具包的性能对比](#各类分词工具包的性能对比) * [使用方式](#使用方式) * [论文引用](#论文引用) * [作者](#作者) * [常见问题及解答](#常见问题及解答) ## 主要亮点 pkuseg具有如下几个特点： 1. 多领域分词。不同于以往的通用中文分词工具，此工具包同时致力于为不同领域的数据提供个性化的预训练模型。根据待分词文本的领域特点，用户可以自由地选择不同的模型。 我们目前支持了新闻领域，网络领域，医药领域，旅游领域，以及混合领域的分词预训练模型。在使用中，如果用户明确待分词的领域，可加载对应的模型进行分词。如果用户无法确定具体领域，推荐使用在混合领域上训练的通用模型。各领域分词样例可参考 [**example.txt**](https://github.com/lancopku/pkuseg-python/blob/master/example.txt)。 2. 更高的分词准确率。相比于其他的分词工具包，当使用相同的训练数据和测试数据，pkuseg可以取得更高的分词准确率。 3. 支持用户自训练模型。支持用户使用全新的标注数据进行训练。 4. 支持词性标注。 ## 编译和安装 - 目前**仅支持python3** - **为了获得好的效果和速度，强烈建议大家通过pip install更新到目前的最新版本** 1. 通过PyPI安装(自带模型文件)： ``` pip3 install pkuseg 之后通过import pkuseg来引用 ``` **建议更新到最新版本**以获得更好的开箱体验： ``` pip3 install -U pkuseg ``` 2. 如果PyPI官方源下载速度不理想，建议使用镜像源，比如： 初次安装： ``` pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple pkuseg ``` 更新： ``` pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple -U pkuseg ``` 3. 如果不使用pip安装方式，选择从GitHub下载，可运行以下命令安装： ``` python setup.py build_ext -i ``` GitHub的代码并不包括预训练模型，因此需要用户自行下载或训练模型，预训练模型可详见[release](https://github.com/lancopku/pkuseg-python/releases)。使用时需设定"model_name"为模型文件。 注意：**安装方式1和2目前仅支持linux(ubuntu)、mac、windows 64 位的python3版本**。如果非以上系统，请使用安装方式3进行本地编译安装。 ## 各类分词工具包的性能对比 我们选择jieba、THULAC等国内代表分词工具包与pkuseg做性能比较，详细设置可参考[实验环境](readme/environment.md)。 #### 细领域训练及测试结果 以下是在不同数据集上的对比结果： | MSRA | Precision | Recall | F-score | | :----- | --------: | -----: | --------: | | jieba | 87.01 | 89.88 | 88.42 | | THULAC | 95.60 | 95.91 | 95.71 | | pkuseg | 96.94 | 96.81 | **96.88** | | WEIBO | Precision | Recall | F-score | | :----- | --------: | -----: | --------: | | jieba | 87.79 | 87.54 | 87.66 | | THULAC | 93.40 | 92.40 | 92.87 | | pkuseg | 93.78 | 94.65 | **94.21** | #### 默认模型在不同领域的测试效果 考虑到很多用户在尝试分词工具的时候，大多数时候会使用工具包自带模型测试。为了直接对比“初始”性能，我们也比较了各个工具包的默认模型在不同领域的测试效果。请注意，这样的比较只是为了说明默认情况下的效果，并不一定是公平的。 | Default | MSRA | CTB8 | PKU | WEIBO | All Average | | ------- | :---: | :---: | :---: | :---: | :---------: | | jieba | 81.45 | 79.58 | 81.83 | 83.56 | 81.61 | | THULAC | 85.55 | 87.84 | 92.29 | 86.65 | 88.08 | | pkuseg | 87.29 | 91.77 | 92.68 | 93.43 | **91.29** | 其中，`All Average`显示的是在所有测试集上F-score的平均。 更多详细比较可参见[和现有工具包的比较](readme/comparison.md)。 ## 使用方式 #### 代码示例 以下代码示例适用于python交互式环境。 代码示例1：使用默认配置进行分词（**如果用户无法确定分词领域，推荐使用默认模型分词**） ```python3 import pkuseg seg = pkuseg.pkuseg() # 以默认配置加载模型 text = seg.cut('我爱北京天安门') # 进行分词 print(text) ``` 代码示例2：细领域分词（**如果用户明确分词领域，推荐使用细领域模型分词**） ```python3 import pkuseg seg = pkuseg.pkuseg(model_name='medicine') # 程序会自动下载所对应的细领域模型 text = seg.cut('我爱北京天安门') # 进行分词 print(text) ``` 代码示例3：分词同时进行词性标注，各词性标签的详细含义可参考 [tags.txt](https://github.com/lancopku/pkuseg-python/blob/master/tags.txt) ```python3 import pkuseg seg = pkuseg.pkuseg(postag=True) # 开启词性标注功能 text = seg.cut('我爱北京天安门') # 进行分词和词性标注 print(text) ``` 代码示例4：对文件分词 ```python3 import pkuseg # 对input.txt的文件分词输出到output.txt中 # 开20个进程 pkuseg.test('input.txt', 'output.txt', nthread=20) ``` 其他使用示例可参见[详细代码示例](readme/interface.md)。 #### 参数说明 模型配置 ``` pkuseg.pkuseg(model_name = "default", user_dict = "default", postag = False) model_name 模型路径。 "default"，默认参数，表示使用我们预训练好的混合领域模型(仅对pip下载的用户)。 "news", 使用新闻领域模型。 "web", 使用网络领域模型。 "medicine", 使用医药领域模型。 "tourism", 使用旅游领域模型。 model_path, 从用户指定路径加载模型。 user_dict 设置用户词典。 "default", 默认参数，使用我们提供的词典。 None, 不使用词典。 dict_path, 在使用默认词典的同时会额外使用用户自定义词典，可以填自己的用户词典的路径，词典格式为一行一个词（如果选择进行词性标注并且已知该词的词性，则在该行写下词和词性，中间用tab字符隔开）。 postag 是否进行词性分析。 False, 默认参数，只进行分词，不进行词性标注。 True, 会在分词的同时进行词性标注。 ``` 对文件进行分词 ``` pkuseg.test(readFile, outputFile, model_name = "default", user_dict = "default", postag = False, nthread = 10) readFile 输入文件路径。 outputFile 输出文件路径。 model_name 模型路径。同pkuseg.pkuseg user_dict 设置用户词典。同pkuseg.pkuseg postag 设置是否开启词性分析功能。同pkuseg.pkuseg nthread 测试时开的进程数。 ``` 模型训练 ``` pkuseg.train(trainFile, testFile, savedir, train_iter = 20, init_model = None) trainFile 训练文件路径。 testFile 测试文件路径。 savedir 训练模型的保存路径。 train_iter 训练轮数。 init_model 初始化模型，默认为None表示使用默认初始化，用户可以填自己想要初始化的模型的路径如init_model='./models/'。 ``` #### 多进程分词 当将以上代码示例置于文件中运行时，如涉及多进程功能，请务必使用`if __name__ == '__main__'`保护全局语句，详见[多进程分词](readme/multiprocess.md)。 ## 预训练模型 从pip安装的用户在使用细领域分词功能时，只需要设置model_name字段为对应的领域即可，会自动下载对应的细领域模型。 从github下载的用户则需要自己下载对应的预训练模型，并设置model_name字段为预训练模型路径。预训练模型可以在[release](https://github.com/lancopku/pkuseg-python/releases)部分下载。以下是对预训练模型的说明： - **news**: 在MSRA（新闻语料）上训练的模型。 - **web**: 在微博（网络文本语料）上训练的模型。 - **medicine**: 在医药领域上训练的模型。 - **tourism**: 在旅游领域上训练的模型。 - **mixed**: 混合数据集训练的通用模型。随pip包附带的是此模型。 我们还通过领域自适应的方法，利用维基百科的未标注数据实现了几个细领域预训练模型的自动构建以及通用模型的优化，这些模型目前仅可以在release中下载： - **art**: 在艺术与文化领域上训练的模型。 - **entertainment**: 在娱乐与体育领域上训练的模型。 - **science**: 在科学领域上训练的模型。 - **default_v2**: 使用领域自适应方法得到的优化后的通用模型，相较于默认模型规模更大，但泛化性能更好。 欢迎更多用户可以分享自己训练好的细分领域模型。 ## 版本历史 详见[版本历史](readme/history.md)。 ## 开源协议 1. 本代码采用MIT许可证。 2. 欢迎对该工具包提出任何宝贵意见和建议，请发邮件至jingjingxu@pku.edu.cn。 ## 论文引用 该代码包主要基于以下科研论文，如使用了本工具，请引用以下论文： * Ruixuan Luo, Jingjing Xu, Yi Zhang, Zhiyuan Zhang, Xuancheng Ren, Xu Sun. [PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation](https://arxiv.org/abs/1906.11455). Arxiv. 2019. ``` @article{pkuseg, author = {Luo, Ruixuan and Xu, Jingjing and Zhang, Yi and Zhang, Zhiyuan and Ren, Xuancheng and Sun, Xu}, journal = {CoRR}, title = {PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation.}, url = {https://arxiv.org/abs/1906.11455}, volume = {abs/1906.11455}, year = 2019 } ``` ## 其他相关论文 * Xu Sun, Houfeng Wang, Wenjie Li. Fast Online Training with Frequency-Adaptive Learning Rates for Chinese Word Segmentation and New Word Detection. ACL. 2012. * Jingjing Xu and Xu Sun. Dependency-based gated recursive neural network for chinese word segmentation. ACL. 2016. * Jingjing Xu and Xu Sun. Transfer learning for low-resource chinese word segmentation with a novel neural network. NLPCC. 2017. ## 常见问题及解答 1. [为什么要发布pkuseg？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#1-为什么要发布pkuseg) 2. [pkuseg使用了哪些技术？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#2-pkuseg使用了哪些技术) 3. [无法使用多进程分词和训练功能，提示RuntimeError和BrokenPipeError。](https://github.com/lancopku/pkuseg-python/wiki/FAQ#3-无法使用多进程分词和训练功能提示runtimeerror和brokenpipeerror) 4. [是如何跟其它工具包在细领域数据上进行比较的？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#4-是如何跟其它工具包在细领域数据上进行比较的) 5. [在黑盒测试集上进行比较的话，效果如何？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#5-在黑盒测试集上进行比较的话效果如何) 6. [如果我不了解待分词语料的所属领域呢？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#6-如果我不了解待分词语料的所属领域呢) 7. [如何看待在一些特定样例上的分词结果？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#7-如何看待在一些特定样例上的分词结果) 8. [关于运行速度问题？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#8-关于运行速度问题) 9. [关于多进程速度问题？](https://github.com/lancopku/pkuseg-python/wiki/FAQ#9-关于多进程速度问题) ## 致谢 感谢俞士汶教授（北京大学计算语言所）与邱立坤博士提供的训练数据集！ ## 作者 Ruixuan Luo （罗睿轩）, Jingjing Xu（许晶晶）, Xuancheng Ren（任宣丞）, Yi Zhang（张艺）, Zhiyuan Zhang（张之远）, Bingzhen Wei（位冰镇）， Xu Sun （孙栩） 北京大学 [语言计算与机器学习研究组](http://lanco.pku.edu.cn/) ================================================ FILE: pkuseg/__init__.py ================================================ from __future__ import print_function import sys if sys.version_info[0] < 3: print("pkuseg does not support python2", file=sys.stderr) sys.exit(1) import os import time import pickle as pkl import multiprocessing from multiprocessing import Process, Queue import pkuseg.trainer as trainer import pkuseg.inference as _inf from pkuseg.config import config from pkuseg.feature_extractor import FeatureExtractor from pkuseg.model import Model from pkuseg.download import download_model from pkuseg.postag import Postag class TrieNode: """建立词典的Trie树节点""" def __init__(self, isword): self.isword = isword self.usertag = '' self.children = {} class Preprocesser: """预处理器，在用户词典中的词强制分割""" def __init__(self, dict_file): """初始化建立Trie树""" if dict_file is None: dict_file = [] self.dict_data = dict_file if isinstance(dict_file, str): with open(dict_file, encoding="utf-8") as f: lines = f.readlines() self.trie = TrieNode(False) for line in lines: fields = line.strip().split('\t') word = fields[0].strip() usertag = fields[1].strip() if len(fields) > 1 else '' self.insert(word, usertag) else: self.trie = TrieNode(False) for w_t in dict_file: if isinstance(w_t, str): w = w_t.strip() t = '' else: assert isinstance(w_t, tuple) assert len(w_t)==2 w, t = map(lambda x:x.strip(), w_t) self.insert(w, t) def insert(self, word, usertag): """Trie树中插入单词""" l = len(word) now = self.trie for i in range(l): c = word[i] if not c in now.children: now.children[c] = TrieNode(False) now = now.children[c] now.isword = True now.usertag = usertag def solve(self, txt): """对文本进行预处理""" outlst = [] iswlst = [] taglst = [] l = len(txt) last = 0 i = 0 while i < l: now = self.trie j = i found = False usertag = '' last_word_idx = -1 # 表示从当前位置i往后匹配，最长匹配词词尾的idx while True: c = txt[j] if not c in now.children and last_word_idx != -1: found = True break if not c in now.children and last_word_idx == -1: break now = now.children[c] if now.isword: last_word_idx = j usertag = now.usertag j += 1 if j == l and last_word_idx == -1: break if j == l and last_word_idx != -1 : j = last_word_idx + 1 found = True break if found: if last != i: outlst.append(txt[last:i]) iswlst.append(False) taglst.append('') outlst.append(txt[i:j]) iswlst.append(True) taglst.append(usertag) last = j i = j else: i += 1 if last < l: outlst.append(txt[last:l]) iswlst.append(False) taglst.append('') return outlst, iswlst, taglst class Postprocesser: """对分词结果后处理""" def __init__(self, common_name, other_names): if common_name is None and other_names is None: self.do_process = False return self.do_process = True if common_name is None: self.common_words = set() else: # with open(common_name, encoding='utf-8') as f: # lines = f.readlines() # self.common_words = set(map(lambda x:x.strip(), lines)) with open(common_name, "rb") as f: all_words = pkl.load(f).strip().split("\n") self.common_words = set(all_words) if other_names is None: self.other_words = set() else: self.other_words = set() for other_name in other_names: # with open(other_name, encoding='utf-8') as f: # lines = f.readlines() # self.other_words.update(set(map(lambda x:x.strip(), lines))) with open(other_name, "rb") as f: all_words = pkl.load(f).strip().split("\n") self.other_words.update(set(all_words)) def post_process(self, sent, check_seperated): for m in reversed(range(2, 8)): end = len(sent)-m if end < 0: continue i = 0 while (i < end + 1): merged_words = ''.join(sent[i:i+m]) if merged_words in self.common_words: do_seg = True elif merged_words in self.other_words: if check_seperated: seperated = all(((w in self.common_words) or (w in self.other_words)) for w in sent[i:i+m]) else: seperated = False if seperated: do_seg = False else: do_seg = True else: do_seg = False if do_seg: for k in range(m): del sent[i] sent.insert(i, merged_words) i += 1 end = len(sent) - m else: i += 1 return sent def __call__(self, sent): if not self.do_process: return sent return self.post_process(sent, check_seperated=True) class pkuseg: def __init__(self, model_name="default", user_dict="default", postag=False): """初始化函数，加载模型及用户词典""" # print("loading model") # config = Config() # self.config = config self.postag = postag if model_name in ["default"]: config.modelDir = os.path.join( os.path.dirname(os.path.realpath(__file__)), "models", model_name, ) elif model_name in config.available_models: config.modelDir = os.path.join( config.pkuseg_home, model_name, ) download_model(config.model_urls[model_name], config.pkuseg_home, config.model_hash[model_name]) else: config.modelDir = model_name # config.fModel = os.path.join(config.modelDir, "model.txt") if user_dict is None: file_name = None other_names = None else: if user_dict not in config.available_models: file_name = user_dict else: file_name = None if model_name in config.models_with_dict: other_name = os.path.join( config.pkuseg_home, model_name, model_name+"_dict.pkl", ) default_name = os.path.join( os.path.dirname(os.path.realpath(__file__)), "dicts", "default.pkl", ) other_names = [other_name, default_name] else: default_name = os.path.join( os.path.dirname(os.path.realpath(__file__)), "dicts", "default.pkl", ) other_names = [default_name] self.preprocesser = Preprocesser(file_name) # self.preprocesser = Preprocesser([]) self.postprocesser = Postprocesser(None, other_names) self.feature_extractor = FeatureExtractor.load() self.model = Model.load() self.idx_to_tag = { idx: tag for tag, idx in self.feature_extractor.tag_to_idx.items() } self.n_feature = len(self.feature_extractor.feature_to_idx) self.n_tag = len(self.feature_extractor.tag_to_idx) if postag: download_model(config.model_urls["postag"], config.pkuseg_home, config.model_hash["postag"]) postag_dir = os.path.join( config.pkuseg_home, "postag", ) self.tagger = Postag(postag_dir) # print("finish") def _cut(self, text): """ 直接对文本分词 """ examples = list(self.feature_extractor.normalize_text(text)) length = len(examples) all_feature = [] # type: List[List[int]] for idx in range(length): node_feature_idx = self.feature_extractor.get_node_features_idx( idx, examples ) # node_feature = self.feature_extractor.get_node_features( # idx, examples # ) # node_feature_idx = [] # for feature in node_feature: # feature_idx = self.feature_extractor.feature_to_idx.get(feature) # if feature_idx is not None: # node_feature_idx.append(feature_idx) # if not node_feature_idx: # node_feature_idx.append(0) all_feature.append(node_feature_idx) _, tags = _inf.decodeViterbi_fast(all_feature, self.model) words = [] current_word = None is_start = True for tag, char in zip(tags, text): if is_start: current_word = char is_start = False elif "B" in self.idx_to_tag[tag]: words.append(current_word) current_word = char else: current_word += char if current_word: words.append(current_word) return words def cut(self, txt): """分词，结果返回一个list""" txt = txt.strip() ret = [] usertags = [] if not txt: return ret imary = txt.split() # 根据空格分为多个片段 # 对每个片段分词 for w0 in imary: if not w0: continue # 根据用户词典拆成更多片段 lst, isword, taglst = self.preprocesser.solve(w0) for w, isw, usertag in zip(lst, isword, taglst): if isw: ret.append(w) usertags.append(usertag) continue output = self._cut(w) post_output = self.postprocesser(output) ret.extend(post_output) usertags.extend(['']*len(post_output)) if self.postag: tags = self.tagger.tag(ret.copy()) for i, usertag in enumerate(usertags): if usertag: tags[i] = usertag ret = list(zip(ret, tags)) return ret def train(trainFile, testFile, savedir, train_iter=20, init_model=None): """用于训练模型""" # config = Config() starttime = time.time() if not os.path.exists(trainFile): raise Exception("trainfile does not exist.") if not os.path.exists(testFile): raise Exception("testfile does not exist.") if not os.path.exists(config.tempFile): os.makedirs(config.tempFile) if not os.path.exists(config.tempFile + "/output"): os.mkdir(config.tempFile + "/output") # config.runMode = "train" config.trainFile = trainFile config.testFile = testFile config.modelDir = savedir # config.fModel = os.path.join(config.modelDir, "model.txt") config.nThread = 1 config.ttlIter = train_iter config.init_model = init_model os.makedirs(config.modelDir, exist_ok=True) trainer.train(config) # pkuseg.main.run(config) # clearDir(config.tempFile) print("Total time: " + str(time.time() - starttime)) def _test_single_proc( input_file, output_file, model_name="default", user_dict="default", postag=False, verbose=False ): times = [] times.append(time.time()) seg = pkuseg(model_name, user_dict, postag=postag) times.append(time.time()) if not os.path.exists(input_file): raise Exception("input_file {} does not exist.".format(input_file)) with open(input_file, "r", encoding="utf-8") as f: lines = f.readlines() times.append(time.time()) results = [] for line in lines: if not postag: results.append(" ".join(seg.cut(line))) else: results.append(" ".join(map(lambda x:"/".join(x), seg.cut(line)))) times.append(time.time()) with open(output_file, "w", encoding="utf-8") as f: f.write("\n".join(results)) times.append(time.time()) print("total_time:\t{:.3f}".format(times[-1] - times[0])) if verbose: time_strs = ["load_model", "read_file", "word_seg", "write_file"] for key, value in zip( time_strs, [end - start for start, end in zip(times[:-1], times[1:])], ): print("{}:\t{:.3f}".format(key, value)) def _proc_deprecated(seg, lines, start, end, q): for i in range(start, end): l = lines[i].strip() ret = seg.cut(l) q.put((i, " ".join(ret))) def _proc(seg, in_queue, out_queue): # TODO: load seg (json or pickle serialization) in sub_process # to avoid pickle seg online when using start method other # than fork while True: item = in_queue.get() if item is None: return idx, line = item if not seg.postag: output_str = " ".join(seg.cut(line)) else: output_str = " ".join(map(lambda x:"/".join(x), seg.cut(line))) out_queue.put((idx, output_str)) def _proc_alt(model_name, user_dict, postag, in_queue, out_queue): seg = pkuseg(model_name, user_dict, postag=postag) while True: item = in_queue.get() if item is None: return idx, line = item if not postag: output_str = " ".join(seg.cut(line)) else: output_str = " ".join(map(lambda x:"/".join(x), seg.cut(line))) out_queue.put((idx, output_str)) def _test_multi_proc( input_file, output_file, nthread, model_name="default", user_dict="default", postag=False, verbose=False, ): alt = multiprocessing.get_start_method() == "spawn" times = [] times.append(time.time()) if alt: seg = None else: seg = pkuseg(model_name, user_dict, postag) times.append(time.time()) if not os.path.exists(input_file): raise Exception("input_file {} does not exist.".format(input_file)) with open(input_file, "r", encoding="utf-8") as f: lines = f.readlines() times.append(time.time()) in_queue = Queue() out_queue = Queue() procs = [] for _ in range(nthread): if alt: p = Process( target=_proc_alt, args=(model_name, user_dict, postag, in_queue, out_queue), ) else: p = Process(target=_proc, args=(seg, in_queue, out_queue)) procs.append(p) for idx, line in enumerate(lines): in_queue.put((idx, line)) for proc in procs: in_queue.put(None) proc.start() times.append(time.time()) result = [None] * len(lines) for _ in result: idx, line = out_queue.get() result[idx] = line times.append(time.time()) for p in procs: p.join() times.append(time.time()) with open(output_file, "w", encoding="utf-8") as f: f.write("\n".join(result)) times.append(time.time()) print("total_time:\t{:.3f}".format(times[-1] - times[0])) if verbose: time_strs = [ "load_model", "read_file", "start_proc", "word_seg", "join_proc", "write_file", ] if alt: times = times[1:] time_strs = time_strs[1:] time_strs[2] = "load_modal & word_seg" for key, value in zip( time_strs, [end - start for start, end in zip(times[:-1], times[1:])], ): print("{}:\t{:.3f}".format(key, value)) def test( input_file, output_file, model_name="default", user_dict="default", nthread=10, postag=False, verbose=False, ): if nthread > 1: _test_multi_proc( input_file, output_file, nthread, model_name, user_dict, postag, verbose ) else: _test_single_proc( input_file, output_file, model_name, user_dict, postag, verbose ) ================================================ FILE: pkuseg/config.py ================================================ import os import tempfile class Config: lineEnd = "\n" biLineEnd = "\n\n" triLineEnd = "\n\n\n" undrln = "_" blank = " " tab = "\t" star = "*" slash = "/" comma = "," delimInFeature = "." B = "B" num = "0123456789.几二三四五六七八九十千万亿兆零１２３４５６７８９０％" letter = "ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺａｂｃｄｅｆｇｈｉｇｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ／・－" mark = "*" model_urls = { "postag": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/postag.zip", "medicine": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/medicine.zip", "tourism": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/tourism.zip", "news": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/news.zip", "web": "https://github.com/lancopku/pkuseg-python/releases/download/v0.0.16/web.zip", } model_hash = { "postag": "afdf15f4e39bc47a39be4c37e3761b0c8f6ad1783f3cd3aff52984aebc0a1da9", "medicine": "773d655713acd27dd1ea9f97d91349cc1b6aa2fc5b158cd742dc924e6f239dfc", "tourism": "1c84a0366fe6fda73eda93e2f31fd399923b2f5df2818603f426a200b05cbce9", "news": "18188b68e76b06fc437ec91edf8883a537fe25fa606641534f6f004d2f9a2e42", "web": "4867f5817f187246889f4db259298c3fcee07c0b03a2d09444155b28c366579e", } available_models = ["default", "medicine", "tourism", "web", "news"] models_with_dict = ["medicine", "tourism"] def __init__(self): # main setting self.pkuseg_home = os.path.expanduser(os.getenv('PKUSEG_HOME', '~/.pkuseg')) self.trainFile = os.path.join("data", "small_training.utf8") self.testFile = os.path.join("data", "small_test.utf8") self._tmp_dir = tempfile.TemporaryDirectory() self.homepath = self._tmp_dir.name self.tempFile = os.path.join(self.homepath, ".pkuseg", "temp") self.readFile = os.path.join("data", "small_test.utf8") self.outputFile = os.path.join("data", "small_test_output.utf8") self.modelOptimizer = "crf.adf" self.rate0 = 0.05 # init value of decay rate in SGD and ADF training # self.reg = 1 # self.regs = [1] # self.regList = self.regs.copy() self.random = ( 0 ) # 0 for 0-initialization of model weights, 1 for random init of model weights self.evalMetric = ( "f1" ) # tok.acc (token accuracy), str.acc (string accuracy), f1 (F1-score) self.trainSizeScale = 1 # for scaling the size of training data self.ttlIter = 20 # of training iterations self.nUpdate = 10 # for ADF training self.outFolder = os.path.join(self.tempFile, "output") self.save = 1 # save model file self.rawResWrite = True self.miniBatch = 1 # mini-batch in stochastic training self.nThread = 10 # number of processes # ADF training self.upper = 0.995 # was tuned for nUpdate = 10 self.lower = 0.6 # was tuned for nUpdate = 10 # global variables self.metric = None self.reg = 1 self.outDir = self.outFolder self.testrawDir = "rawinputs/" self.testinputDir = "inputs/" self.tempDir = os.path.join(self.homepath, ".pkuseg", "temp") self.testoutputDir = "entityoutputs/" # self.GL_init = True self.weightRegMode = "L2" # choosing weight regularizer: L2, L1) self.c_train = os.path.join(self.tempFile, "train.conll.txt") self.f_train = os.path.join(self.tempFile, "train.feat.txt") self.c_test = os.path.join(self.tempFile, "test.conll.txt") self.f_test = os.path.join(self.tempFile, "test.feat.txt") self.fTune = "tune.txt" self.fLog = "trainLog.txt" self.fResSum = "summarizeResult.txt" self.fResRaw = "rawResult.txt" self.fOutput = "outputTag-{}.txt" self.fFeatureTrain = os.path.join(self.tempFile, "ftrain.txt") self.fGoldTrain = os.path.join(self.tempFile, "gtrain.txt") self.fFeatureTest = os.path.join(self.tempFile, "ftest.txt") self.fGoldTest = os.path.join(self.tempFile, "gtest.txt") self.modelDir = os.path.join( os.path.dirname(os.path.realpath(__file__)), "models", "ctb8" ) self.fModel = os.path.join(self.modelDir, "model.txt") # feature self.numLetterNorm = True self.featureTrim = 0 self.wordFeature = True self.wordMax = 6 self.wordMin = 2 self.nLabel = 5 self.order = 1 def globalCheck(self): if self.evalMetric == "f1": self.metric = "f-score" elif self.evalMetric == "tok.acc": self.metric = "token-accuracy" elif self.evalMetric == "str.acc": self.metric = "string-accuracy" else: raise Exception("invalid eval metric") assert self.rate0 > 0 assert self.trainSizeScale > 0 assert self.ttlIter > 0 assert self.nUpdate > 0 assert self.miniBatch > 0 assert self.reg > 0 config = Config() ================================================ FILE: pkuseg/data.py ================================================ # from .config import Config # from pkuseg.feature_generator import # import os import copy import random # class dataFormat: # def __init__(self, config): # self.featureIndexMap = {} # self.tagIndexMap = {} # self.config = config # def convert(self): # config = self.config # if config.runMode.find("train") >= 0: # self.getMaps(config.fTrain) # self.saveFeature(config.modelDir + "/featureIndex.txt") # self.convertFile(config.fTrain) # else: # self.readFeature(config.modelDir + "/featureIndex.txt") # self.readTag(config.modelDir + "/tagIndex.txt") # self.convertFile(config.fTest) # if config.dev: # self.convertFile(config.fDev) # def saveFeature(self, file): # featureList = list(self.featureIndexMap.keys()) # num = len(featureList) // 10 # for i in range(10): # l = i * num # r = (i + 1) * num if i < 9 else len(featureList) # with open(file + "_" + str(i), "w", encoding="utf-8") as sw: # for w in range(l, r): # word = featureList[w] # sw.write(word + " " + str(self.featureIndexMap[word]) + "\n") # def readFeature(self, file): # featureList = [] # for i in range(10): # featureList.append([]) # with open(file + "_" + str(i), encoding="utf-8") as f: # lines = f.readlines() # for line in lines: # featureList[i].append(line.strip()) # feature = [] # for i in range(10): # for line in featureList[i]: # word, index = line.split(" ") # self.featureIndexMap[word] = int(index) # def readFeatureNormal(self, path): # with open(path, encoding="utf-8") as f: # lines = f.readlines() # for line in lines: # u, v = line.split(" ") # self.featureIndexMap[u] = int(v) # def readTag(self, path): # with open(path, encoding="utf-8") as f: # lines = f.readlines() # for line in lines: # u, v = line.split(" ") # self.tagIndexMap[u] = int(v) # def getMaps(self, file): # config = self.config # if not os.path.exists(file): # print("file {} not exist!".format(file)) # print("file {} converting...".format(file)) # featureFreqMap = {} # tagSet = set() # with open(file, encoding="utf-8") as f: # lines = f.readlines() # for line in lines: # line = line.replace("\t", " ") # line = line.replace("\r", "").strip() # if line == "": # continue # ary = line.split(config.blank) # for i in range(1, len(ary) - 1): # if ary[i] == "" or ary[i] == "/": # continue # if config.weightRegMode == "GL": # if not config.GL_init and config.groupTrim[i - 1]: # continue # ary2 = ary[i].split(config.slash) # feature = str(i) + "." + ary2[0] # if not feature in featureFreqMap: # featureFreqMap[feature] = 0 # featureFreqMap[feature] += 1 # tag = ary[-1] # tagSet.add(tag) # sortList = [] # for k in featureFreqMap: # sortList.append(k + " " + str(featureFreqMap[k])) # if config.weightRegMode == "GL": # sortList.sort(key=lambda x: (int(x.split(config.blank)[1].strip()), x)) # with open("featureTemp_sorted.txt", "w", encoding="utf-8") as f: # for x in sortList: # f.write(x + "\n") # config.groupStart = [0] # config.groupEnd = [] # for k in range(1, len(sortList)): # thisAry = sortList[k].split(config.dot) # preAry = sortList[k - 1].split(config.dot) # s = thisAry[0] # preAry = preAry[0] # if s != preAry: # config.groupStart.append(k) # config.groupEnd.append(k) # config.groupEnd.append(len(sortList)) # else: # sortList.sort( # key=lambda x: (int(x.split(config.blank)[1].strip()), x), reverse=True # ) # if config.weightRegMode == "GL" and config.GL_init: # if nFeatTemp != len(config.groupStart): # raise Exception( # "inconsistent # of features per line, check the feature file for consistency!" # ) # with open( # os.path.join(config.modelDir, "featureIndex.txt"), "w", encoding="utf-8" # ) as swFeat: # for i, l in enumerate(sortList): # ary = l.split(config.blank) # self.featureIndexMap[ary[0]] = i # swFeat.write("{} {}\n".format(ary[0].strip(), i)) # with open(os.path.join(config.modelDir, "tagIndex.txt"), "w", encoding="utf-8") as swTag: # tagSortList = [] # for tag in tagSet: # tagSortList.append(tag) # tagSortList.sort() # for i, l in enumerate(tagSortList): # self.tagIndexMap[l] = i # swTag.write("{} {}\n".format(l, i)) # def convertFile(self, file): # config = self.config # if not os.path.exists(file): # print("file {} not exist!".format(file)) # print("file converting...") # if file == config.fTrain: # swFeature = open(config.fFeatureTrain, "w", encoding="utf-8") # swGold = open(config.fGoldTrain, "w", encoding="utf-8") # else: # swFeature = open(config.fFeatureTest, "w", encoding="utf-8") # swGold = open(config.fGoldTest, "w", encoding="utf-8") # swFeature.write(str(len(self.featureIndexMap)) + "\n\n") # swGold.write(str(len(self.tagIndexMap)) + "\n\n") # with open(file, encoding="utf-8") as sr: # readLines = sr.readlines() # featureList = [] # goldList = [] # for k in range(len(readLines)): # line = readLines[k] # line = line.replace("\t", "").strip() # featureLine = "" # goldLine = "" # if line == "": # featureLine = featureLine + "\n" # goldLine = goldLine + "\n\n" # featureList.append(featureLine) # goldList.append(goldLine) # continue # flag = 0 # ary = line.split(config.blank) # tmp = [] # for i in ary: # if i != "": # tmp.append(i) # ary = tmp # for i in range(1, len(ary) - 1): # if ary[i] == "/": # continue # ary2 = ary[i].split(config.slash) # tmp = [] # for j in ary2: # if j != "": # tmp.append(j) # ary2 = tmp # feature = str(i) + "." + ary2[0] # value = "" # real = False # if len(ary2) > 1: # value = ary2[1] # real = True # if not feature in self.featureIndexMap: # continue # flag = 1 # fIndex = self.featureIndexMap[feature] # if not real: # featureLine = featureLine + str(fIndex) + "," # else: # featureLine = featureLine + str(fIndex) + "/" + value + "," # if flag == 0: # featureLine = featureLine + "0" # featureLine = featureLine + "\n" # tag = ary[-1] # tIndex = self.tagIndexMap[tag] # goldLine = goldLine + str(tIndex) + "," # featureList.append(featureLine) # goldList.append(goldLine) # for i in range(len(featureList)): # swFeature.write(featureList[i]) # swGold.write(goldList[i]) # swFeature.close() # swGold.close() class DataSet: def __init__(self, n_tag=0, n_feature=0): self.lst = [] # type: List[Example] self.n_tag = n_tag self.n_feature = n_feature # if len(args) == 2: # if type(args[0]) == int: # self.nTag, self.nFeature = args # else: # self.load(args[0], args[1]) def __len__(self): return len(self.lst) def __iter__(self): return self.iterator() def __getitem__(self, x): return self.lst[x] def iterator(self): for i in self.lst: yield i def append(self, x): self.lst.append(x) def clear(self): self.lst = [] def randomShuffle(self): cp = copy.deepcopy(self) random.shuffle(cp.lst) return cp # def setDataInfo(self, X): # self.nTag = X.nTag # self.nFeature = X.nFeature def resize(self, scale): dataset = DataSet(self.n_tag, self.n_feature) new_size = int(len(self) * scale) old_size = len(self) for i in range(new_size): if i >= old_size: i %= old_size dataset.append(self[i]) return dataset @classmethod def load(cls, feature_idx_file, tag_idx_file): dataset = cls.__new__(cls) # def load(self, fileFeature, fileTag): with open(feature_idx_file, encoding="utf-8") as f_reader, open( tag_idx_file, encoding="utf-8" ) as t_reader: example_strs = f_reader.read().split("\n\n")[:-1] tags_strs = t_reader.read().split("\n\n")[:-1] assert len(example_strs) == len( tags_strs ), "lengths do not match:\t{}\n{}\n".format(example_strs, tags_strs) n_feature = int(example_strs[0]) n_tag = int(tags_strs[0]) dataset.n_feature = n_feature dataset.n_tag = n_tag dataset.lst = [] for example_str, tags_str in zip(example_strs[1:], tags_strs[1:]): features = [ list(map(int, feature_line.split(","))) for feature_line in example_str.split("\n") ] tags = tags_str.split(",") example = Example(features, tags) dataset.lst.append(example) return dataset # txt = srfileFeature.read() # txt.replace("\r", "") # fAry = txt.split(Config.biLineEnd) # tmp = [] # for i in fAry: # if i != "": # tmp.append(i) # fAry = tmp # txt = srfileTag.read() # txt.replace("\r", "") # tAry = txt.split(Config.biLineEnd) # tmp = [] # for i in tAry: # if i != "": # tmp.append(i) # tAry = tmp # assert len(fAry) == len(tAry) # self.nFeature = int(fAry[0]) # self.nTag = int(tAry[0]) # for i in range(1, len(fAry)): # features = fAry[i] # tags = tAry[i] # seq = dataSeq() # seq.read(features, tags) # self.append(seq) # @property # def NTag(self): # return self.nTag class Example: def __init__(self, features, tags): self.features = features # type: List[List[int]] self.tags = list(map(int, tags)) # type: List[int] self.predicted_tags = None def __len__(self): return len(self.features) # class dataSeq: # def __init__(self, *args): # self.featureTemps = [] # self.yGold = [] # if len(args) == 2: # self.featureTemps = copy.deepcopy(args[0]) # self.yGold = copy.deepcopy(args[1]) # elif len(args) == 3: # x, n, length = args # end = min(n + length, len(x)) # for i in range(n, end): # self.featureTemps.append(x.featureTemps[i]) # yGold.append(x.yGold[i]) # def __len__(self): # return len(self.featureTemps) # def read(self, a, b): # lineAry = a.split(Config.lineEnd) # for im in lineAry: # if im == "": # continue # nodeList = [] # imAry = im.split(Config.comma) # for imm in imAry: # if imm == "": # continue # if imm.find("/") >= 0: # biAry = imm.split(Config.slash) # ft = featureTemp(int(biAry[0], float(biAry[1]))) # nodeList.append(ft) # else: # ft = featureTemp(int(imm), 1) # nodeList.append(ft) # self.featureTemps.append(nodeList) # lineAry = b.split(Config.comma) # for im in lineAry: # if im == "": # continue # self.yGold.append(int(im)) # # def load(self, feature): # # for imAry in feature: # # nodeList = [] # # for imm in imAry: # # if imm == "": # # continue # # if imm.find("/") >= 0: # # biAry = imm.split(Config.slash) # # ft = featureTemp(int(biAry[0], float(biAry[1]))) # # nodeList.append(ft) # # else: # # ft = featureTemp(int(imm), 1) # # nodeList.append(ft) # # self.featureTemps.append(nodeList) # # self.yGold.append(0) # def getFeatureTemp(self, *args): # return ( # self.featureTemps if len(args) == 0 else self.featureTemps[args[0]] # ) # def getTags(self, *args): # return self.yGold if len(args) == 0 else self.yGold[args[0]] # def setTags(self, lst): # assert len(lst) == len(self.yGold) # for i in range(len(lst)): # self.yGold[i] = lst[i] # class dataSeqTest: # def __init__(self, x, yOutput): # self._x = x # self._yOutput = yOutput ================================================ FILE: pkuseg/dicts/__init__.py ================================================ ================================================ FILE: pkuseg/download.py ================================================ from __future__ import absolute_import, division, print_function, unicode_literals import hashlib import os import re import shutil import sys import tempfile import zipfile try: from requests.utils import urlparse from requests import get as urlopen requests_available = True except ImportError: requests_available = False if sys.version_info[0] == 2: from urlparse import urlparse # noqa f811 from urllib2 import urlopen # noqa f811 else: from urllib.request import urlopen from urllib.parse import urlparse try: from tqdm import tqdm except ImportError: tqdm = None # defined below HASH_REGEX = re.compile(r'-([a-f0-9]*)\.') def download_model(url, model_dir, hash_prefix, progress=True): if not os.path.exists(model_dir): os.makedirs(model_dir) parts = urlparse(url) filename = os.path.basename(parts.path) cached_file = os.path.join(model_dir, filename) if not os.path.exists(cached_file): sys.stderr.write('Downloading: "{}" to {}\n'.format(url, cached_file)) _download_url_to_file(url, cached_file, hash_prefix, progress=progress) unzip_file(cached_file, os.path.join(model_dir, filename.split('.')[0])) def _download_url_to_file(url, dst, hash_prefix, progress): if requests_available: u = urlopen(url, stream=True, timeout=5) file_size = int(u.headers["Content-Length"]) u = u.raw else: u = urlopen(url, timeout=5) meta = u.info() if hasattr(meta, 'getheaders'): file_size = int(meta.getheaders("Content-Length")[0]) else: file_size = int(meta.get_all("Content-Length")[0]) f = tempfile.NamedTemporaryFile(delete=False) try: if hash_prefix is not None: sha256 = hashlib.sha256() with tqdm(total=file_size, disable=not progress) as pbar: while True: buffer = u.read(8192) if len(buffer) == 0: break f.write(buffer) if hash_prefix is not None: sha256.update(buffer) pbar.update(len(buffer)) f.close() if hash_prefix is not None: digest = sha256.hexdigest() if digest[:len(hash_prefix)] != hash_prefix: raise RuntimeError('invalid hash value (expected "{}", got "{}")' .format(hash_prefix, digest)) shutil.move(f.name, dst) finally: f.close() if os.path.exists(f.name): os.remove(f.name) if tqdm is None: # fake tqdm if it's not installed class tqdm(object): def __init__(self, total, disable=False): self.total = total self.disable = disable self.n = 0 def update(self, n): if self.disable: return self.n += n sys.stderr.write("\r{0:.1f}%".format(100 * self.n / float(self.total))) sys.stderr.flush() def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): if self.disable: return sys.stderr.write('\n') def unzip_file(zip_name, target_dir): if not os.path.exists(target_dir): os.makedirs(target_dir) file_zip = zipfile.ZipFile(zip_name, 'r') for file in file_zip.namelist(): file_zip.extract(file, target_dir) file_zip.close() if __name__ == '__main__': url = 'https://github.com/lancopku/pkuseg-python/releases/download/v0.0.14/mixed.zip' download_model(url, '.') ================================================ FILE: pkuseg/feature_extractor.pyx ================================================ # distutils: language = c++ # cython: infer_types=True # cython: language_level=3 import json import os import sys import pickle from collections import Counter from itertools import product import cython from pkuseg.config import config @cython.boundscheck(False) @cython.wraparound(False) cpdef get_slice_str(iterable, int start, int length, int all_len): if start < 0 or start >= all_len: return "" if start + length >= all_len + 1: return "" return "".join(iterable[start : start + length]) @cython.boundscheck(False) @cython.wraparound(False) @cython.nonecheck(False) def __get_node_features_idx(object config not None, int idx, list nodes not None, dict feature_to_idx not None, set unigram not None): cdef: list flist = [] Py_ssize_t i = idx int length = len(nodes) int word_max = config.wordMax int word_min = config.wordMin int word_range = word_max - word_min + 1 c = nodes[i] # $$ starts feature flist.append(0) # 8 unigram/bgiram feature feat = 'c.' + c if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) if i > 0: prev_c = nodes[i-1] feat = 'c-1.' + prev_c if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) feat = 'c-1c.' + prev_c + '.' + c if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) if i + 1 < length: next_c = nodes[i+1] feat = 'c1.' + next_c if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) feat = 'cc1.' + c + '.' + next_c if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) if i > 1: prepre_char = nodes[i-2] feat = 'c-2.' + prepre_char if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) feat = 'c-2c-1.' + prepre_char + '.' + nodes[i-1] if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) if i + 2 < length: feat = 'c2.' + nodes[i+2] if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) # no num/letter based features if not config.wordFeature: return flist # 2 * (wordMax-wordMin+1) word features (default: 2*(6-2+1)=10 ) # the character starts or ends a word prelst_in = [] for l in range(word_max, word_min - 1, -1): # length 6 ... 2 (default) # "prefix including current c" wordary[n-l+1, n] # current character ends word tmp = get_slice_str(nodes, i - l + 1, l, length) if tmp in unigram: feat = 'w-1.' + tmp if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) prelst_in.append(tmp) else: prelst_in.append("**noWord") postlst_in = [] for l in range(word_max, word_min - 1, -1): # "suffix" wordary[n, n+l-1] # current character starts word tmp = get_slice_str(nodes, i, l, length) if tmp in unigram: feat = 'w1.' + tmp if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) postlst_in.append(tmp) else: postlst_in.append("**noWord") # these are not in feature list prelst_ex = [] for l in range(word_max, word_min - 1, -1): # "prefix excluding current c" wordary[n-l, n-1] tmp = get_slice_str(nodes, i - l, l, length) if tmp in unigram: prelst_ex.append(tmp) else: prelst_ex.append("**noWord") postlst_ex = [] for l in range(word_max, word_min - 1, -1): # "suffix excluding current c" wordary[n+1, n+l] tmp = get_slice_str(nodes, i + 1, l, length) if tmp in unigram: postlst_ex.append(tmp) else: postlst_ex.append("**noWord") # this character is in the middle of a word # 2*(wordMax-wordMin+1)^2 (default: 2*(6-2+1)^2=50) for pre in prelst_ex: for post in postlst_in: feat = 'ww.l.' + pre + '*' + post if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) for pre in prelst_in: for post in postlst_ex: feat = 'ww.r.' + pre + '*' + post if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) return flist class FeatureExtractor: keywords = "-._,|/*:" num = set("0123456789." "几二三四五六七八九十千万亿兆零" "１２３４５６７８９０％") letter = set( "ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ" "ａｂｃｄｅｆｇｈｉｇｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ" "／・－" ) keywords_translate_table = str.maketrans("-._,|/*:", "&&&&&&&&") @classmethod def keyword_rename(cls, text): return text.translate(cls.keywords_translate_table) @classmethod def _num_letter_normalize_char(cls, character): if character in cls.num: return "**Num" if character in cls.letter: return "**Letter" return character @classmethod def normalize_text(cls, text): text = cls.keyword_rename(text) for character in text: if config.numLetterNorm: yield cls._num_letter_normalize_char(character) else: yield character def __init__(self): self.unigram = set() # type: Set[str] self.bigram = set() # type: Set[str] self.feature_to_idx = {} # type: Dict[str, int] self.tag_to_idx = {} # type: Dict[str, int] def build(self, train_file): with open(train_file, "r", encoding="utf8") as reader: lines = reader.readlines() examples = [] # type: List[List[List[str]]] # first pass to collect unigram and bigram and tag info word_length_info = Counter() specials = set() for line in lines: line = line.strip("\n\r") # .replace("\t", " ") if not line: continue line = self.keyword_rename(line) # str.split() without sep sees consecutive whiltespaces as one separator # e.g., '\ra \t　b \r\n'.split() = ['a', 'b'] words = [word for word in line.split()] word_length_info.update(map(len, words)) specials.update(word for word in words if len(word)>=10) self.unigram.update(words) for pre, suf in zip(words[:-1], words[1:]): self.bigram.add("{}*{}".format(pre, suf)) example = [ self._num_letter_normalize_char(character) for word in words for character in word ] examples.append(example) max_word_length = max(word_length_info.keys()) for length in range(1, max_word_length + 1): print("length = {} : {}".format(length, word_length_info[length])) # print('special words: {}'.format(', '.join(specials))) # second pass to get features feature_freq = Counter() for example in examples: for i, _ in enumerate(example): node_features = self.get_node_features(i, example) feature_freq.update( feature for feature in node_features if feature != "/" ) feature_set = ( feature for feature, freq in feature_freq.most_common() if freq > config.featureTrim ) tot = len(self.feature_to_idx) for feature in feature_set: if not feature in self.feature_to_idx: self.feature_to_idx[feature] = tot tot += 1 # self.feature_to_idx = { # feature: idx for idx, feature in enumerate(feature_set) # } if config.nLabel == 2: B = B_single = "B" I_first = I = I_end = "I" elif config.nLabel == 3: B = B_single = "B" I_first = I = "I" I_end = "I_end" elif config.nLabel == 4: B = "B" B_single = "B_single" I_first = I = "I" I_end = "I_end" elif config.nLabel == 5: B = "B" B_single = "B_single" I_first = "I_first" I = "I" I_end = "I_end" tag_set = {B, B_single, I_first, I, I_end} self.tag_to_idx = {tag: idx for idx, tag in enumerate(sorted(tag_set))} def get_node_features_idx(self, idx, nodes): return __get_node_features_idx(config, idx, nodes, self.feature_to_idx, self.unigram) def get_node_features(self, idx, wordary): cdef int length = len(wordary) w = wordary[idx] flist = [] # 1 start feature flist.append("$$") # 8 unigram/bgiram feature flist.append("c." + w) if idx > 0: flist.append("c-1." + wordary[idx - 1]) else: flist.append("/") if idx < len(wordary) - 1: flist.append("c1." + wordary[idx + 1]) else: flist.append("/") if idx > 1: flist.append("c-2." + wordary[idx - 2]) else: flist.append("/") if idx < len(wordary) - 2: flist.append("c2." + wordary[idx + 2]) else: flist.append("/") if idx > 0: flist.append("c-1c." + wordary[idx - 1] + config.delimInFeature + w) else: flist.append("/") if idx < len(wordary) - 1: flist.append("cc1." + w + config.delimInFeature + wordary[idx + 1]) else: flist.append("/") if idx > 1: flist.append( "c-2c-1." + wordary[idx - 2] + config.delimInFeature + wordary[idx - 1] ) else: flist.append("/") # no num/letter based features if not config.wordFeature: return flist # 2 * (wordMax-wordMin+1) word features (default: 2*(6-2+1)=10 ) # the character starts or ends a word tmplst = [] for l in range(config.wordMax, config.wordMin - 1, -1): # length 6 ... 2 (default) # "prefix including current c" wordary[n-l+1, n] # current character ends word tmp = get_slice_str(wordary, idx - l + 1, l, length) if tmp != "": if tmp in self.unigram: flist.append("w-1." + tmp) tmplst.append(tmp) else: flist.append("/") tmplst.append("**noWord") else: flist.append("/") tmplst.append("**noWord") prelst_in = tmplst tmplst = [] for l in range(config.wordMax, config.wordMin - 1, -1): # "suffix" wordary[n, n+l-1] # current character starts word tmp = get_slice_str(wordary, idx, l, length) if tmp != "": if tmp in self.unigram: flist.append("w1." + tmp) tmplst.append(tmp) else: flist.append("/") tmplst.append("**noWord") else: flist.append("/") tmplst.append("**noWord") postlst_in = tmplst # these are not in feature list tmplst = [] for l in range(config.wordMax, config.wordMin - 1, -1): # "prefix excluding current c" wordary[n-l, n-1] tmp = get_slice_str(wordary, idx - l, l, length) if tmp != "": if tmp in self.unigram: tmplst.append(tmp) else: tmplst.append("**noWord") else: tmplst.append("**noWord") prelst_ex = tmplst tmplst = [] for l in range(config.wordMax, config.wordMin - 1, -1): # "suffix excluding current c" wordary[n+1, n+l] tmp = get_slice_str(wordary, idx + 1, l, length) if tmp != "": if tmp in self.unigram: tmplst.append(tmp) else: tmplst.append("**noWord") else: tmplst.append("**noWord") postlst_ex = tmplst # this character is in the middle of a word # 2*(wordMax-wordMin+1)^2 (default: 2*(6-2+1)^2=50) for pre in prelst_ex: for post in postlst_in: bigram = pre + "*" + post if bigram in self.bigram: flist.append("ww.l." + bigram) else: flist.append("/") for pre in prelst_in: for post in postlst_ex: bigram = pre + "*" + post if bigram in self.bigram: flist.append("ww.r." + bigram) else: flist.append("/") return flist def convert_feature_file_to_idx_file( self, feature_file, feature_idx_file, tag_idx_file ): with open(feature_file, "r", encoding="utf8") as reader: lines = reader.readlines() with open(feature_idx_file, "w", encoding="utf8") as f_writer, open( tag_idx_file, "w", encoding="utf8" ) as t_writer: f_writer.write("{}\n\n".format(len(self.feature_to_idx))) t_writer.write("{}\n\n".format(len(self.tag_to_idx))) tags_idx = [] # type: List[str] features_idx = [] # type: List[List[str]] for line in lines: line = line.strip() if not line: # sentence finish for feature_idx in features_idx: if not feature_idx: f_writer.write("0\n") else: f_writer.write(",".join(map(str, feature_idx))) f_writer.write("\n") f_writer.write("\n") t_writer.write(",".join(map(str, tags_idx))) t_writer.write("\n\n") tags_idx = [] features_idx = [] continue splits = line.split(" ") feature_idx = [ self.feature_to_idx[feat] for feat in splits[:-1] if feat in self.feature_to_idx ] features_idx.append(feature_idx) tags_idx.append(self.tag_to_idx[splits[-1]]) def convert_text_file_to_feature_file( self, text_file, conll_file=None, feature_file=None ): if conll_file is None: conll_file = "{}.conll{}".format(*os.path.split(text_file)) if feature_file is None: feature_file = "{}.feat{}".format(*os.path.split(text_file)) if config.nLabel == 2: B = B_single = "B" I_first = I = I_end = "I" elif config.nLabel == 3: B = B_single = "B" I_first = I = "I" I_end = "I_end" elif config.nLabel == 4: B = "B" B_single = "B_single" I_first = I = "I" I_end = "I_end" elif config.nLabel == 5: B = "B" B_single = "B_single" I_first = "I_first" I = "I" I_end = "I_end" conll_line_format = "{} {}\n" with open(text_file, "r", encoding="utf8") as reader, open( conll_file, "w", encoding="utf8" ) as c_writer, open(feature_file, "w", encoding="utf8") as f_writer: for line in reader: line = line.strip() if not line: continue words = self.keyword_rename(line).split() example = [] tags = [] for word in words: word_length = len(word) for idx, character in enumerate(word): if word_length == 1: tag = B_single elif idx == 0: tag = B elif idx == word_length - 1: tag = I_end elif idx == 1: tag = I_first else: tag = I c_writer.write(conll_line_format.format(character, tag)) if config.numLetterNorm: example.append( self._num_letter_normalize_char(character) ) else: example.append(character) tags.append(tag) c_writer.write("\n") for idx, tag in enumerate(tags): features = self.get_node_features(idx, example) features = [ (feature if feature in self.feature_to_idx else "/") for feature in features ] features.append(tag) f_writer.write(" ".join(features)) f_writer.write("\n") f_writer.write("\n") def save(self, model_dir=None): if model_dir is None: model_dir = config.modelDir data = {} data["unigram"] = sorted(list(self.unigram)) data["bigram"] = sorted(list(self.bigram)) data["feature_to_idx"] = self.feature_to_idx data["tag_to_idx"] = self.tag_to_idx with open(os.path.join(model_dir, 'features.pkl'), 'wb') as writer: pickle.dump(data, writer, protocol=pickle.HIGHEST_PROTOCOL) # with open( # os.path.join(config.modelDir, "features.json"), "w", encoding="utf8" # ) as writer: # json.dump(data, writer, ensure_ascii=False) @classmethod def load(cls, model_dir=None): if model_dir is None: model_dir = config.modelDir extractor = cls.__new__(cls) feature_path = os.path.join(model_dir, "features.pkl") if os.path.exists(feature_path): with open(feature_path, "rb") as reader: data = pickle.load(reader) extractor.unigram = set(data["unigram"]) extractor.bigram = set(data["bigram"]) extractor.feature_to_idx = data["feature_to_idx"] extractor.tag_to_idx = data["tag_to_idx"] return extractor print( "WARNING: features.pkl does not exist, try loading features.json", file=sys.stderr, ) feature_path = os.path.join(model_dir, "features.json") if os.path.exists(feature_path): with open(feature_path, "r", encoding="utf8") as reader: data = json.load(reader) extractor.unigram = set(data["unigram"]) extractor.bigram = set(data["bigram"]) extractor.feature_to_idx = data["feature_to_idx"] extractor.tag_to_idx = data["tag_to_idx"] extractor.save(model_dir) return extractor print( "WARNING: features.json does not exist, try loading using old format", file=sys.stderr, ) with open( os.path.join(model_dir, "unigram_word.txt"), "r", encoding="utf8", ) as reader: extractor.unigram = set([line.strip() for line in reader]) with open( os.path.join(model_dir, "bigram_word.txt"), "r", encoding="utf8", ) as reader: extractor.bigram = set(line.strip() for line in reader) extractor.feature_to_idx = {} feature_base_name = os.path.join(model_dir, "featureIndex.txt") for i in range(10): with open( "{}_{}".format(feature_base_name, i), "r", encoding="utf8" ) as reader: for line in reader: feature, index = line.split(" ") feature = ".".join(feature.split(".")[1:]) extractor.feature_to_idx[feature] = int(index) extractor.tag_to_idx = {} with open( os.path.join(model_dir, "tagIndex.txt"), "r", encoding="utf8" ) as reader: for line in reader: tag, index = line.split(" ") extractor.tag_to_idx[tag] = int(index) print( "INFO: features.json is saved", file=sys.stderr, ) extractor.save(model_dir) return extractor ================================================ FILE: pkuseg/gradient.py ================================================ import pkuseg.model from typing import List import pkuseg.inference as _inf import pkuseg.data def get_grad_SGD_minibatch( grad: List[float], model: pkuseg.model.Model, X: List[pkuseg.data.Example] ): # if idset is not None: # idset.clear() all_id_set = set() errors = 0 for x in X: error, id_set = get_grad_CRF(grad, model, x) errors += error all_id_set.update(id_set) return errors, all_id_set def get_grad_CRF( grad: List[float], model: pkuseg.model.Model, x: pkuseg.data.Example ): id_set = set() n_tag = model.n_tag bel = _inf.belief(len(x), n_tag) belMasked = _inf.belief(len(x), n_tag) Ylist, YYlist, maskYlist, maskYYlist = _inf.getYYandY(model, x) Z, sum_edge = _inf.get_beliefs(bel, model, x, Ylist, YYlist) ZGold, sum_edge_masked = _inf.get_beliefs(belMasked, model, x, maskYlist, maskYYlist) for i, node_feature_list in enumerate(x.features): for feature_id in node_feature_list: trans_id = model._get_node_tag_feature_id(feature_id, 0) id_set.update(range(trans_id, trans_id + n_tag)) grad[trans_id:trans_id+n_tag] += bel.belState[i] - belMasked.belState[i] backoff = model.n_feature * n_tag grad[backoff:] += sum_edge - sum_edge_masked id_set.update(range(backoff, backoff + n_tag * n_tag)) return Z - ZGold, id_set ================================================ FILE: pkuseg/inference.pyx ================================================ # distutils: language = c++ # cython: infer_types=True # cython: language_level=3 cimport cython import numpy as np cimport numpy as np from libcpp.vector cimport vector from libc.math cimport exp, log np.import_array() class belief: def __init__(self, nNodes, nStates): self.belState = np.zeros((nNodes, nStates)) self.belEdge = np.zeros((nNodes, nStates * nStates)) self.Z = 0 @cython.boundscheck(False) @cython.wraparound(False) cpdef get_beliefs(object bel, object m, object x, np.ndarray[double, ndim=2] Y, np.ndarray[double, ndim=2] YY): cdef: np.ndarray[double, ndim=2] belState = bel.belState np.ndarray[double, ndim=2] belEdge = bel.belEdge int nNodes = len(x) int nTag = m.n_tag double Z = 0 np.ndarray[double, ndim=1] alpha_Y = np.zeros(nTag) np.ndarray[double, ndim=1] newAlpha_Y = np.zeros(nTag) np.ndarray[double, ndim=1] tmp_Y = np.zeros(nTag) np.ndarray[double, ndim=2] YY_trans = YY.transpose() np.ndarray[double, ndim=1] YY_t_r = YY_trans.reshape(-1) np.ndarray[double, ndim=1] sum_edge = np.zeros(nTag * nTag) for i in range(nNodes - 1, 0, -1): tmp_Y = belState[i] + Y[i] belState[i-1] = logMultiply(YY, tmp_Y) for i in range(nNodes): if i > 0: tmp_Y = alpha_Y.copy() newAlpha_Y = logMultiply(YY_trans, tmp_Y) + Y[i] else: newAlpha_Y = Y[i].copy() if i > 0: tmp_Y = Y[i] + belState[i] belEdge[i] = YY_t_r for yPre in range(nTag): for y in range(nTag): belEdge[i, y * nTag + yPre] += tmp_Y[y] + alpha_Y[yPre] belState[i] = belState[i] + newAlpha_Y alpha_Y = newAlpha_Y Z = logSum(alpha_Y) for i in range(nNodes): belState[i] = np.exp(belState[i] - Z) for i in range(1, nNodes): sum_edge += np.exp(belEdge[i] - Z) return Z, sum_edge @cython.boundscheck(False) @cython.wraparound(False) cpdef run_viterbi(np.ndarray[double, ndim=2] node_score, np.ndarray[double, ndim=2] edge_score): cdef int i, y, y_pre, i_pre, tag, w=node_score.shape[0], h=node_score.shape[1] cdef double ma, sc cdef np.ndarray[double, ndim=2] max_score = np.zeros((w, h), dtype=np.float64) cdef np.ndarray[int, ndim=2] pre_tag = np.zeros((w, h), dtype=np.int32) cdef np.ndarray[unsigned char, ndim=2] init_check = np.zeros((w, h), dtype=np.uint8) cdef np.ndarray[int, ndim=1] states = np.zeros(w, dtype=np.int32) for y in range(h): max_score[w-1, y] = node_score[w-1, y] for i in range(w - 2, -1, -1): for y in range(h): for y_pre in range(h): i_pre = i + 1 sc = max_score[i_pre, y_pre] + node_score[i, y] + edge_score[y, y_pre] if not init_check[i, y]: init_check[i, y] = 1 max_score[i, y] = sc pre_tag[i, y] = y_pre elif sc >= max_score[i, y]: max_score[i, y] = sc pre_tag[i, y] = y_pre ma = max_score[0, 0] tag = 0 for y in range(1, h): sc = max_score[0, y] if ma < sc: ma = sc tag = y states[0] = tag for i in range(1, w): tag = pre_tag[i-1, tag] states[i] = tag if ma > 300: ma = 300 return exp(ma), states @cython.boundscheck(False) @cython.wraparound(False) cpdef getLogYY(vector[vector[int]] feature_temp, int num_tag, int backoff, np.ndarray[double, ndim=1] w, double scalar): cdef: int num_node = feature_temp.size() np.ndarray[double, ndim=2] node_score = np.zeros((num_node, num_tag), dtype=np.float64) np.ndarray[double, ndim=2] edge_score = np.ones((num_tag, num_tag), dtype=np.float64) int s, s_pre, i double maskValue, tmp vector[int] f_list int f, ft for i in range(num_node): f_list = feature_temp[i] for ft in f_list: for s in range(num_tag): f = ft * num_tag + s node_score[i, s] += w[f] * scalar for s in range(num_tag): for s_pre in range(num_tag): f = backoff + s * num_tag + s_pre edge_score[s_pre, s] += w[f] * scalar return node_score, edge_score @cython.boundscheck(False) @cython.wraparound(False) cpdef maskY(object tags, int nNodes, int nTag, np.ndarray[double, ndim=2] Y): cdef np.ndarray[double, ndim=2] mask_Yi = Y.copy() cdef double maskValue = -1e100 cdef list tagList = tags cdef int i for i in range(nNodes): for s in range(nTag): if tagList[i] != s: mask_Yi[i, s] = maskValue return mask_Yi @cython.boundscheck(False) @cython.wraparound(False) cdef logMultiply(np.ndarray[double, ndim=2] A, np.ndarray[double, ndim=1] B): cdef int r, c cdef np.ndarray[double, ndim=2] toSumLists = np.zeros_like(A) cdef np.ndarray[double, ndim=1] ret = np.zeros(A.shape[0]) for r in range(A.shape[0]): for c in range(A.shape[1]): toSumLists[r, c] = A[r, c] + B[c] for r in range(A.shape[0]): ret[r] = logSum(toSumLists[r]) return ret @cython.boundscheck(False) @cython.wraparound(False) cdef logSum(double[:] a): cdef int n = a.shape[0] cdef double s = a[0] cdef double m1 cdef double m2 for i in range(1, n): if s >= a[i]: m1, m2 = s, a[i] else: m1, m2 = a[i], s s = m1 + log(1 + exp(m2 - m1)) return s def decodeViterbi_fast(feature_temp, model): Y, YY = getLogYY(feature_temp, model.n_tag, model.n_feature*model.n_tag, model.w, 1.0) numer, tags = run_viterbi(Y, YY) tags = list(tags) return numer, tags def getYYandY(model, example): Y, YY = getLogYY(example.features, model.n_tag, model.n_feature*model.n_tag, model.w, 1.0) mask_Y = maskY(example.tags, len(example), model.n_tag, Y) mask_YY = YY return Y, YY, mask_Y, mask_YY ================================================ FILE: pkuseg/model.py ================================================ import os import sys import numpy as np from .config import config class Model: def __init__(self, n_feature, n_tag): self.n_tag = n_tag self.n_feature = n_feature self.n_transition_feature = n_tag * (n_feature + n_tag) if config.random: self.w = np.random.random(size=(self.n_transition_feature,)) * 2 - 1 else: self.w = np.zeros(self.n_transition_feature) def expand(self, n_feature, n_tag): new_transition_feature = n_tag * (n_feature + n_tag) if config.random: new_w = np.random.random(size=(new_transition_feature,)) * 2 - 1 else: new_w = np.zeros(new_transition_feature) n_node = self.n_tag * self.n_feature n_edge = self.n_tag * self.n_tag new_w[:n_node] = self.w[:n_node] new_w[-n_edge:] = self.w[-n_edge:] self.n_tag = n_tag self.n_feature = n_feature self.n_transition_feature = new_transition_feature self.w = new_w def _get_node_tag_feature_id(self, feature_id, tag_id): return feature_id * self.n_tag + tag_id def _get_tag_tag_feature_id(self, pre_tag_id, tag_id): return self.n_feature * self.n_tag + tag_id * self.n_tag + pre_tag_id @classmethod def load(cls, model_dir=None): if model_dir is None: model_dir = config.modelDir model_path = os.path.join(model_dir, "weights.npz") if os.path.exists(model_path): npz = np.load(model_path) sizes = npz["sizes"] w = npz["w"] model = cls.__new__(cls) model.n_tag = int(sizes[0]) model.n_feature = int(sizes[1]) model.n_transition_feature = model.n_tag * ( model.n_feature + model.n_tag ) model.w = w assert model.w.shape[0] == model.n_transition_feature return model print( "WARNING: weights.npz does not exist, try loading using old format", file=sys.stderr, ) model_path = os.path.join(model_dir, "model.txt") with open(model_path, encoding="utf-8") as f: ary = f.readlines() model = cls.__new__(cls) model.n_tag = int(ary[0].strip()) wsize = int(ary[1].strip()) w = np.zeros(wsize) for i in range(2, wsize): w[i - 2] = float(ary[i].strip()) model.w = w model.n_feature = wsize // model.n_tag - model.n_tag model.n_transition_feature = wsize model.save(model_dir) return model @classmethod def new(cls, model, copy_weight=True): new_model = cls.__new__(cls) new_model.n_tag = model.n_tag if copy_weight: new_model.w = model.w.copy() else: new_model.w = np.zeros_like(model.w) new_model.n_feature = ( new_model.w.shape[0] // new_model.n_tag - new_model.n_tag ) new_model.n_transition_feature = new_model.w.shape[0] return new_model def save(self, model_dir=None): if model_dir is None: model_dir = config.modelDir sizes = np.array([self.n_tag, self.n_feature]) np.savez( os.path.join(model_dir, "weights.npz"), sizes=sizes, w=self.w ) # np.save # with open(file, "w", encoding="utf-8") as f: # f.write("{}\n{}\n".format(self.n_tag, self.w.shape[0])) # for value in self.w: # f.write("{:.4f}\n".format(value)) ================================================ FILE: pkuseg/optimizer.py ================================================ import random import numpy as np import pkuseg.gradient as _grad # from pkuseg.config import config class Optimizer: def __init__(self): self._preVals = [] def converge_test(self, err): val = 1e100 if len(self._preVals) > 1: prevVal = self._preVals[0] if len(self._preVals) == 10: self._preVals.pop(0) avgImprovement = (prevVal - err) / len(self._preVals) relAvg = avgImprovement / abs(err) val = relAvg self._preVals.append(err) return val def optimize(self): raise NotImplementedError() class ADF(Optimizer): def __init__(self, config, dataset, model): super().__init__() self.config = config self._model = model self._X = dataset self.decayList = np.ones_like(self._model.w) * config.rate0 def optimize(self): config = self.config sample_size = 0 w = self._model.w fsize = w.shape[0] xsize = len(self._X) grad = np.zeros(fsize) error = 0 feature_count_list = np.zeros(fsize) # feature_count_list = [0] * fsize ri = list(range(xsize)) random.shuffle(ri) update_interval = xsize // config.nUpdate # config.interval = xsize // config.nUpdate n_sample = 0 for t in range(0, xsize, config.miniBatch): XX = [] end = False for k in range(t, t + config.miniBatch): i = ri[k] x = self._X[i] XX.append(x) if k == xsize - 1: end = True break mb_size = len(XX) n_sample += mb_size # fSet = set() err, feature_set = _grad.get_grad_SGD_minibatch( grad, self._model, XX ) error += err feature_set = list(feature_set) feature_count_list[feature_set] += 1 # for i in feature_set: # feature_count_list[i] += 1 check = False for k in range(t, t + config.miniBatch): if t != 0 and k % update_interval == 0: check = True # update decay rates if check or end: self.decayList *= ( config.upper - (config.upper - config.lower) * feature_count_list / n_sample ) feature_count_list.fill(0) # for i in range(fsize): # v = feature_count_list[i] # u = v / n_sample # eta = config.upper - (config.upper - config.lower) * u # self.decayList[i] *= eta # feature_count_list # for i in range(len(feature_count_list)): # feature_count_list[i] = 0 # update weights w[feature_set] -= self.decayList[feature_set] * grad[feature_set] grad[feature_set] = 0 # for i in feature_set: # w[i] -= self.decayList[i] * grad[i] # grad[i] = 0 # reg if check or end: if config.reg != 0: w -= self.decayList * ( w / (config.reg * config.reg) * n_sample / xsize ) # for i in range(fsize): # grad_i = ( # w[i] / (config.reg * config.reg) * (n_sample / xsize) # ) # w[i] -= self.decayList[i] * grad_i n_sample = 0 sample_size += mb_size if config.reg != 0: s = (w * w).sum() error += s / (2.0 * config.reg * config.reg) diff = self.converge_test(error) return error, sample_size, diff ================================================ FILE: pkuseg/postag/__init__.py ================================================ from __future__ import print_function import sys import os import time from ..inference import decodeViterbi_fast from .feature_extractor import FeatureExtractor from .model import Model class Postag: def __init__(self, model_name): modelDir = model_name self.feature_extractor = FeatureExtractor.load(modelDir) self.model = Model.load(modelDir) self.idx_to_tag = { idx: tag for tag, idx in self.feature_extractor.tag_to_idx.items() } self.n_feature = len(self.feature_extractor.feature_to_idx) self.n_tag = len(self.feature_extractor.tag_to_idx) # print("finish") def _cut(self, text): examples = list(self.feature_extractor.normalize_text(text)) length = len(examples) all_feature = [] # type: List[List[int]] for idx in range(length): node_feature_idx = self.feature_extractor.get_node_features_idx( idx, examples ) # node_feature = self.feature_extractor.get_node_features( # idx, examples # ) # node_feature_idx = [] # for feature in node_feature: # feature_idx = self.feature_extractor.feature_to_idx.get(feature) # if feature_idx is not None: # node_feature_idx.append(feature_idx) # if not node_feature_idx: # node_feature_idx.append(0) all_feature.append(node_feature_idx) _, tags = decodeViterbi_fast(all_feature, self.model) tags = list(map(lambda x:self.idx_to_tag[x], tags)) return tags def tag(self, sen): """txt: list[str], tags: list[str]""" tags = self._cut(sen) return tags ================================================ FILE: pkuseg/postag/feature_extractor.pyx ================================================ # distutils: language = c++ # cython: infer_types=True # cython: language_level=3 import json import os import sys import pickle from collections import Counter from itertools import product import cython @cython.boundscheck(False) @cython.wraparound(False) cpdef get_slice_str(iterable, int start, int length, int all_len): if start < 0 or start >= all_len: return "" if start + length >= all_len + 1: return "" return "".join(iterable[start : start + length]) @cython.boundscheck(False) @cython.wraparound(False) @cython.nonecheck(False) def __get_node_features_idx(int idx, list nodes not None, dict feature_to_idx not None): cdef: list flist = [] Py_ssize_t i = idx int length = len(nodes) int j w = nodes[i] # $$ starts feature flist.append(0) # unigram/bgiram feature feat = "w." + w if feat in feature_to_idx: feature = feature_to_idx[feat] flist.append(feature) for j in range(1, 4): if len(w)>=j: feat = "tr1.pre.%d.%s"%(j, w[:j]) if feat in feature_to_idx: flist.append(feature_to_idx[feat]) feat = "tr1.post.%d.%s"%(j, w[-j:]) if feat in feature_to_idx: flist.append(feature_to_idx[feat]) if i > 0: feat = "tr1.w-1." + nodes[i - 1] else: feat = "tr1.w-1.BOS" if feat in feature_to_idx: flist.append(feature_to_idx[feat]) if i < length - 1: feat = "tr1.w1." + nodes[i + 1] else: feat = "tr1.w1.EOS" if feat in feature_to_idx: flist.append(feature_to_idx[feat]) if i > 1: feat = "tr1.w-2." + nodes[i - 2] else: feat = "tr1.w-2.BOS" if feat in feature_to_idx: flist.append(feature_to_idx[feat]) if i < length - 2: feat = "tr1.w2." + nodes[i + 2] else: feat = "tr1.w2.EOS" if feat in feature_to_idx: flist.append(feature_to_idx[feat]) if i > 0: feat = "tr1.w_-1_0." + nodes[i - 1] + "." + w else: feat = "tr1.w_-1_0.BOS" if feat in feature_to_idx: flist.append(feature_to_idx[feat]) if i < length - 1: feat = "tr1.w_0_1." + w + "." + nodes[i + 1] else: feat = "tr1.w_0_1.EOS" if feat in feature_to_idx: flist.append(feature_to_idx[feat]) return flist class FeatureExtractor: keywords = "-._,|/*:" num = set("0123456789." "几二三四五六七八九十千万亿兆零" "１２３４５６７８９０％") letter = set( "ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ" "ａｂｃｄｅｆｇｈｉｇｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ" "／・－" ) keywords_translate_table = str.maketrans("-._,|/*:", "&&&&&&&&") @classmethod def keyword_rename(cls, text): return text.translate(cls.keywords_translate_table) @classmethod def _num_letter_normalize(cls, word): if not list(filter(lambda x:x not in cls.num, word)): return "**Num" return word @classmethod def normalize_text(cls, text): for i in range(len(text)): text[i] = cls.keyword_rename(text[i]) for character in text: yield cls._num_letter_normalize(character) def __init__(self): # self.unigram = set() # type: Set[str] # self.bigram = set() # type: Set[str] self.feature_to_idx = {} # type: Dict[str, int] self.tag_to_idx = {} # type: Dict[str, int] def get_node_features_idx(self, idx, nodes): return __get_node_features_idx(idx, nodes, self.feature_to_idx) def get_node_features(self, idx, wordary): cdef int length = len(wordary) w = wordary[idx] flist = [] # 1 start feature flist.append("$$") # 8 unigram/bgiram feature flist.append("w." + w) # prefix/suffix for i in range(1, 4): if len(w)>=i: flist.append("tr1.pre.%d.%s"%(i, w[:i])) flist.append("tr1.post.%d.%s"%(i, w[-i:])) else: flist.append("/") flist.append("/") if idx > 0: flist.append("tr1.w-1." + wordary[idx - 1]) else: flist.append("tr1.w-1.BOS") if idx < len(wordary) - 1: flist.append("tr1.w1." + wordary[idx + 1]) else: flist.append("tr1.w1.EOS") if idx > 1: flist.append("tr1.w-2." + wordary[idx - 2]) else: flist.append("tr1.w-2.BOS") if idx < len(wordary) - 2: flist.append("tr1.w2." + wordary[idx + 2]) else: flist.append("tr1.w2.EOS") if idx > 0: flist.append("tr1.w_-1_0." + wordary[idx - 1] + "." + w) else: flist.append("tr1.w_-1_0.BOS") if idx < len(wordary) - 1: flist.append("tr1.w_0_1." + w + "." + wordary[idx + 1]) else: flist.append("tr1.w_0_1.EOS") return flist def convert_feature_file_to_idx_file( self, feature_file, feature_idx_file, tag_idx_file ): with open(feature_file, "r", encoding="utf8") as reader: lines = reader.readlines() with open(feature_idx_file, "w", encoding="utf8") as f_writer, open( tag_idx_file, "w", encoding="utf8" ) as t_writer: f_writer.write("{}\n\n".format(len(self.feature_to_idx))) t_writer.write("{}\n\n".format(len(self.tag_to_idx))) tags_idx = [] # type: List[str] features_idx = [] # type: List[List[str]] for line in lines: line = line.strip() if not line: # sentence finish for feature_idx in features_idx: if not feature_idx: f_writer.write("0\n") else: f_writer.write(",".join(map(str, feature_idx))) f_writer.write("\n") f_writer.write("\n") t_writer.write(",".join(map(str, tags_idx))) t_writer.write("\n\n") tags_idx = [] features_idx = [] continue splits = line.split(" ") feature_idx = [ self.feature_to_idx[feat] for feat in splits[:-1] if feat in self.feature_to_idx ] features_idx.append(feature_idx) if not splits[-1] in self.tag_to_idx: tags_idx.append(-1) else: tags_idx.append(self.tag_to_idx[splits[-1]]) def convert_text_file_to_feature_file( self, text_file, conll_file=None, feature_file=None ): if conll_file is None: conll_file = "{}.conll{}".format(*os.path.split(text_file)) if feature_file is None: feature_file = "{}.feat{}".format(*os.path.split(text_file)) conll_line_format = "{} {}\n" with open(text_file, "r", encoding="utf8") as reader, open( conll_file, "w", encoding="utf8" ) as c_writer, open(feature_file, "w", encoding="utf8") as f_writer: for line in reader.read().strip().replace("\r", "").split("\n\n"): line = line.strip() if not line: continue line = self.keyword_rename(line).split("\n") words = [] tags = [] for word_tag in line: word, tag = word_tag.split() words.append(word) tags.append(tag) example = [ self._num_letter_normalize(word) for word in words ] for word, tag in zip(example, tags): c_writer.write(conll_line_format.format(word, tag)) c_writer.write("\n") for idx, tag in enumerate(tags): features = self.get_node_features(idx, example) features = [ (feature if feature in self.feature_to_idx else "/") for feature in features ] features.append(tag) f_writer.write(" ".join(features)) f_writer.write("\n") f_writer.write("\n") def save(self, model_dir): data = {} data["feature_to_idx"] = self.feature_to_idx data["tag_to_idx"] = self.tag_to_idx with open(os.path.join(model_dir, 'features.pkl'), 'wb') as writer: pickle.dump(data, writer, protocol=pickle.HIGHEST_PROTOCOL) @classmethod def load(cls, model_dir): extractor = cls.__new__(cls) feature_path = os.path.join(model_dir, "features.pkl") if os.path.exists(feature_path): with open(feature_path, "rb") as reader: data = pickle.load(reader) extractor.feature_to_idx = data["feature_to_idx"] extractor.tag_to_idx = data["tag_to_idx"] return extractor print( "WARNING: features.pkl does not exist, try loading features.json", file=sys.stderr, ) feature_path = os.path.join(model_dir, "features.json") if os.path.exists(feature_path): with open(feature_path, "r", encoding="utf8") as reader: data = json.load(reader) extractor.feature_to_idx = data["feature_to_idx"] extractor.tag_to_idx = data["tag_to_idx"] extractor.save(model_dir) return extractor print( "WARNING: features.json does not exist, try loading using old format", file=sys.stderr, ) extractor.feature_to_idx = {} feature_base_name = os.path.join(model_dir, "featureIndex.txt") for i in range(10): with open( "{}_{}".format(feature_base_name, i), "r", encoding="utf8" ) as reader: for line in reader: feature, index = line.split(" ") feature = ".".join(feature.split(".")[1:]) extractor.feature_to_idx[feature] = int(index) extractor.tag_to_idx = {} with open( os.path.join(model_dir, "tagIndex.txt"), "r", encoding="utf8" ) as reader: for line in reader: tag, index = line.split(" ") extractor.tag_to_idx[tag] = int(index) print( "INFO: features.json is saved", file=sys.stderr, ) extractor.save(model_dir) return extractor ================================================ FILE: pkuseg/postag/model.py ================================================ import os import sys import numpy as np class Model: def __init__(self, n_feature, n_tag): self.n_tag = n_tag self.n_feature = n_feature self.n_transition_feature = n_tag * (n_feature + n_tag) self.w = np.zeros(self.n_transition_feature) def _get_node_tag_feature_id(self, feature_id, tag_id): return feature_id * self.n_tag + tag_id def _get_tag_tag_feature_id(self, pre_tag_id, tag_id): return self.n_feature * self.n_tag + tag_id * self.n_tag + pre_tag_id @classmethod def load(cls, model_dir): model_path = os.path.join(model_dir, "weights.npz") if os.path.exists(model_path): npz = np.load(model_path) sizes = npz["sizes"] w = npz["w"] model = cls.__new__(cls) model.n_tag = int(sizes[0]) model.n_feature = int(sizes[1]) model.n_transition_feature = model.n_tag * ( model.n_feature + model.n_tag ) model.w = w assert model.w.shape[0] == model.n_transition_feature return model print( "WARNING: weights.npz does not exist, try loading using old format", file=sys.stderr, ) model_path = os.path.join(model_dir, "model.txt") with open(model_path, encoding="utf-8") as f: ary = f.readlines() model = cls.__new__(cls) model.n_tag = int(ary[0].strip()) wsize = int(ary[1].strip()) w = np.zeros(wsize) for i in range(2, wsize): w[i - 2] = float(ary[i].strip()) model.w = w model.n_feature = wsize // model.n_tag - model.n_tag model.n_transition_feature = wsize model.save(model_dir) return model @classmethod def new(cls, model, copy_weight=True): new_model = cls.__new__(cls) new_model.n_tag = model.n_tag if copy_weight: new_model.w = model.w.copy() else: new_model.w = np.zeros_like(model.w) new_model.n_feature = ( new_model.w.shape[0] // new_model.n_tag - new_model.n_tag ) new_model.n_transition_feature = new_model.w.shape[0] return new_model def save(self, model_dir): sizes = np.array([self.n_tag, self.n_feature]) np.savez( os.path.join(model_dir, "weights.npz"), sizes=sizes, w=self.w ) # np.save # with open(file, "w", encoding="utf-8") as f: # f.write("{}\n{}\n".format(self.n_tag, self.w.shape[0])) # for value in self.w: # f.write("{:.4f}\n".format(value)) ================================================ FILE: pkuseg/res_summarize.py ================================================ import numpy as np from .config import Config import os def tomatrix(s): lines = s.split(Config.lineEnd) lst = [] for line in lines: if line == "": continue if not line.startswith("%"): tmp = [] for i in line.split(Config.comma): tmp.append(float(i)) lst.append(tmp) return np.array(lst) def summarize(config): with open( os.path.join(config.outDir, config.fResRaw), encoding="utf-8" ) as sr: txt = sr.read() txt = txt.replace("\r", "") regions = txt.split(config.triLineEnd) with open( os.path.join(config.outDir, config.fResSum), "w", encoding="utf-8" ) as sw: for region in regions: if region == "": continue blocks = region.split(config.biLineEnd) mList = [] for im in blocks: mList.append(tomatrix(im)) avgM = np.zeros_like(mList[0]) for m in mList: avgM = avgM + m avgM = avgM / len(mList) sqravgM = np.zeros_like(mList[0]) for m in mList: sqravgM += m * m sqravgM = sqravgM / len(mList) deviM = (sqravgM - avgM * avgM) ** 0.5 sw.write("%averaged values:\n") for i in range(avgM.shape[0]): for j in range(avgM.shape[1]): sw.write("{:.2f},".format(avgM[i, j])) sw.write("\n") sw.write("\n%deviations:\n") for i in range(deviM.shape[0]): for j in range(deviM.shape[1]): sw.write("{:.2f},".format(deviM[i, j])) # sw.write(("%.2f" % deviM[i, j]) + ",") sw.write("\n") sw.write("\n%avg & devi:\n") for i in range(avgM.shape[0]): for j in range(avgM.shape[1]): sw.write("{:.2f}+-{:,2f},".format(avgM[i, j], deviM[i, j])) sw.write("\n") sw.write("%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\n\n") def write(config, timeList, errList, diffList, scoreListList): def log(message): if config.rawResWrite: config.swResRaw.write(message) log("% training results:" + config.metric + "\n") for i in range(config.ttlIter): it = i log("% iter#={} ".format(it)) lst = scoreListList[i] if config.evalMetric == "f1": log( "% f-score={:.2f}% precision={:.2f}% recall={:.2f}% ".format( lst[0], lst[1], lst[2] ) ) else: log("% {}={:.2f}% ".format(config.metric, lst[0])) time = 0 for k in range(i + 1): time += timeList[k] log( "cumulative-time(sec)={:.2f} objective={:.2f} diff={:.2f}\n".format( time, errList[i], diffList[i] ) ) # #ttlScore = 0 # for i in range(config.ttlIter): # it = i + 1 # log("% iter#={} ".format(it)) # lst = scoreListList[i] # # ttlScore += lst[0] # if config.evalMetric == "f1": # log( # "% f-score={:.2f}% precision={:.2f}% recall={:.2f}% ".format( # lst[0], lst[1], lst[2] # ) # ) # else: # log("% {}={:.2f}% ".format(config.metric, lst[0])) # time = 0 # for k in range(i + 1): # time += timeList[k] # log( # "cumulative-time(sec)={:.2f} objective={:.2f} diff={:.2f}\n".format( # time, errList[i], diffList[i] # ) # ) ================================================ FILE: pkuseg/scorer.py ================================================ from pkuseg.config import Config def getFscore(goldTagList, resTagList, idx_to_chunk_tag): scoreList = [] assert len(resTagList) == len(goldTagList) getNewTagList(idx_to_chunk_tag, goldTagList) getNewTagList(idx_to_chunk_tag, resTagList) goldChunkList = getChunks(goldTagList) resChunkList = getChunks(resTagList) gold_chunk = 0 res_chunk = 0 correct_chunk = 0 for i in range(len(goldChunkList)): res = resChunkList[i] gold = goldChunkList[i] resChunkAry = res.split(Config.comma) tmp = [] for t in resChunkAry: if len(t) > 0: tmp.append(t) resChunkAry = tmp goldChunkAry = gold.split(Config.comma) tmp = [] for t in goldChunkAry: if len(t) > 0: tmp.append(t) goldChunkAry = tmp gold_chunk += len(goldChunkAry) res_chunk += len(resChunkAry) goldChunkSet = set() for im in goldChunkAry: goldChunkSet.add(im) for im in resChunkAry: if im in goldChunkSet: correct_chunk += 1 pre = correct_chunk / res_chunk * 100 rec = correct_chunk / gold_chunk * 100 f1 = 0 if correct_chunk == 0 else 2 * pre * rec / (pre + rec) scoreList.append(f1) scoreList.append(pre) scoreList.append(rec) infoList = [] infoList.append(gold_chunk) infoList.append(res_chunk) infoList.append(correct_chunk) return scoreList, infoList def getNewTagList(tagMap, tagList): tmpList = [] for im in tagList: tagAry = im.split(Config.comma) for i in range(len(tagAry)): if tagAry[i] == "": continue index = int(tagAry[i]) if not index in tagMap: raise Exception("Error") tagAry[i] = tagMap[index] newTags = ",".join(tagAry) tmpList.append(newTags) tagList.clear() for im in tmpList: tagList.append(im) def getChunks(tagList): tmpList = [] for im in tagList: tagAry = im.split(Config.comma) tmp = [] for t in tagAry: if t != "": tmp.append(t) tagAry = tmp chunks = "" for i in range(len(tagAry)): if tagAry[i].startswith("B"): pos = i length = 1 ty = tagAry[i] for j in range(i + 1, len(tagAry)): if tagAry[j] == "I": length += 1 else: break chunk = ty + "*" + str(length) + "*" + str(pos) chunks = chunks + chunk + "," tmpList.append(chunks) return tmpList ================================================ FILE: pkuseg/trainer.py ================================================ # from .config import config # from .feature import * # from .data_format import * # from .toolbox import * import os import time from multiprocessing import Process, Queue from pkuseg import res_summarize # from .inference import * # from .config import Config from pkuseg.config import Config, config from pkuseg.data import DataSet from pkuseg.feature_extractor import FeatureExtractor # from .feature_generator import * from pkuseg.model import Model import pkuseg.inference as _inf # from .inference import * # from .gradient import * from pkuseg.optimizer import ADF from pkuseg.scorer import getFscore # from typing import TextIO # from .res_summarize import summarize # from .res_summarize import write as reswrite # from pkuseg.trainer import Trainer def train(config=None): if config is None: config = Config() if config.init_model is None: feature_extractor = FeatureExtractor() else: feature_extractor = FeatureExtractor.load(config.init_model) feature_extractor.build(config.trainFile) feature_extractor.save() feature_extractor.convert_text_file_to_feature_file( config.trainFile, config.c_train, config.f_train ) feature_extractor.convert_text_file_to_feature_file( config.testFile, config.c_test, config.f_test ) feature_extractor.convert_feature_file_to_idx_file( config.f_train, config.fFeatureTrain, config.fGoldTrain ) feature_extractor.convert_feature_file_to_idx_file( config.f_test, config.fFeatureTest, config.fGoldTest ) config.globalCheck() config.swLog = open(os.path.join(config.outDir, config.fLog), "w") config.swResRaw = open(os.path.join(config.outDir, config.fResRaw), "w") config.swTune = open(os.path.join(config.outDir, config.fTune), "w") print("\nstart training...") config.swLog.write("\nstart training...\n") print("\nreading training & test data...") config.swLog.write("\nreading training & test data...\n") trainset = DataSet.load(config.fFeatureTrain, config.fGoldTrain) testset = DataSet.load(config.fFeatureTest, config.fGoldTest) trainset = trainset.resize(config.trainSizeScale) print( "done! train/test data sizes: {}/{}".format(len(trainset), len(testset)) ) config.swLog.write( "done! train/test data sizes: {}/{}\n".format( len(trainset), len(testset) ) ) config.swLog.write("\nr: {}\n".format(config.reg)) print("\nr: {}".format(config.reg)) if config.rawResWrite: config.swResRaw.write("\n%r: {}\n".format(config.reg)) trainer = Trainer(config, trainset, feature_extractor) time_list = [] err_list = [] diff_list = [] score_list_list = [] for i in range(config.ttlIter): # config.glbIter += 1 time_s = time.time() err, sample_size, diff = trainer.train_epoch() time_t = time.time() - time_s time_list.append(time_t) err_list.append(err) diff_list.append(diff) score_list = trainer.test(testset, i) score_list_list.append(score_list) score = score_list[0] logstr = "iter{} diff={:.2e} train-time(sec)={:.2f} {}={:.2f}%".format( i, diff, time_t, config.metric, score ) config.swLog.write(logstr + "\n") config.swLog.write("------------------------------------------------\n") config.swLog.flush() print(logstr) res_summarize.write(config, time_list, err_list, diff_list, score_list_list) if config.save == 1: trainer.model.save() config.swLog.close() config.swResRaw.close() config.swTune.close() res_summarize.summarize(config) print("finished.") class Trainer: def __init__(self, config, dataset, feature_extractor): self.config = config self.X = dataset self.n_feature = dataset.n_feature self.n_tag = dataset.n_tag if config.init_model is None: self.model = Model(self.n_feature, self.n_tag) else: self.model = Model.load(config.init_model) self.model.expand(self.n_feature, self.n_tag) self.optim = self._get_optimizer(dataset, self.model) self.feature_extractor = feature_extractor self.idx_to_chunk_tag = {} for tag, idx in feature_extractor.tag_to_idx.items(): if tag.startswith("I"): tag = "I" if tag.startswith("O"): tag = "O" self.idx_to_chunk_tag[idx] = tag def _get_optimizer(self, dataset, model): config = self.config if "adf" in config.modelOptimizer: return ADF(config, dataset, model) raise ValueError("Invalid Optimizer") def train_epoch(self): return self.optim.optimize() def test(self, testset, iteration): outfile = os.path.join(config.outDir, config.fOutput.format(iteration)) func_mapping = { "tok.acc": self._decode_tokAcc, "str.acc": self._decode_strAcc, "f1": self._decode_fscore, } with open(outfile, "w", encoding="utf8") as writer: score_list = func_mapping[config.evalMetric]( testset, self.model, writer ) for example in testset: example.predicted_tags = None return score_list def _decode(self, testset: DataSet, model: Model): if config.nThread == 1: self._decode_single(testset, model) else: self._decode_multi_proc(testset, model) def _decode_single(self, testset: DataSet, model: Model): # n_tag = model.n_tag for example in testset: _, tags = _inf.decodeViterbi_fast(example.features, model) example.predicted_tags = tags @staticmethod def _decode_proc(model, in_queue, out_queue): while True: item = in_queue.get() if item is None: return idx, features = item _, tags = _inf.decodeViterbi_fast(features, model) out_queue.put((idx, tags)) def _decode_multi_proc(self, testset: DataSet, model: Model): in_queue = Queue() out_queue = Queue() procs = [] nthread = self.config.nThread for i in range(nthread): p = Process( target=self._decode_proc, args=(model, in_queue, out_queue) ) procs.append(p) for idx, example in enumerate(testset): in_queue.put((idx, example.features)) for proc in procs: in_queue.put(None) proc.start() for _ in range(len(testset)): idx, tags = out_queue.get() testset[idx].predicted_tags = tags for p in procs: p.join() # token accuracy def _decode_tokAcc(self, dataset, model, writer): config = self.config self._decode(dataset, model) n_tag = model.n_tag all_correct = [0] * n_tag all_pred = [0] * n_tag all_gold = [0] * n_tag for example in dataset: pred = example.predicted_tags gold = example.tags if writer is not None: writer.write(",".join(map(str, pred))) writer.write("\n") for pred_tag, gold_tag in zip(pred, gold): all_pred[pred_tag] += 1 all_gold[gold_tag] += 1 if pred_tag == gold_tag: all_correct[gold_tag] += 1 config.swLog.write( "% tag-type #gold #output #correct-output token-precision token-recall token-f-score\n" ) sumGold = 0 sumOutput = 0 sumCorrOutput = 0 for i, (correct, gold, pred) in enumerate( zip(all_correct, all_gold, all_pred) ): sumGold += gold sumOutput += pred sumCorrOutput += correct if gold == 0: rec = 0 else: rec = correct * 100.0 / gold if pred == 0: prec = 0 else: prec = correct * 100.0 / pred config.swLog.write( "% {}: {} {} {} {:.2f} {:.2f} {:.2f}\n".format( i, gold, pred, correct, prec, rec, (2 * prec * rec / (prec + rec)), ) ) if sumGold == 0: rec = 0 else: rec = sumCorrOutput * 100.0 / sumGold if sumOutput == 0: prec = 0 else: prec = sumCorrOutput * 100.0 / sumOutput if prec == 0 and rec == 0: fscore = 0 else: fscore = 2 * prec * rec / (prec + rec) config.swLog.write( "% overall-tags: {} {} {} {:.2f} {:.2f} {:.2f}\n".format( sumGold, sumOutput, sumCorrOutput, prec, rec, fscore ) ) config.swLog.flush() return [fscore] def _decode_strAcc(self, dataset, model, writer): config = self.config self._decode(dataset, model) correct = 0 total = len(dataset) for example in dataset: pred = example.predicted_tags gold = example.tags if writer is not None: writer.write(",".join(map(str, pred))) writer.write("\n") for pred_tag, gold_tag in zip(pred, gold): if pred_tag != gold_tag: break else: correct += 1 acc = correct / total * 100.0 config.swLog.write( "total-tag-strings={} correct-tag-strings={} string-accuracy={}%".format( total, correct, acc ) ) return [acc] def _decode_fscore(self, dataset, model, writer): config = self.config self._decode(dataset, model) gold_tags = [] pred_tags = [] for example in dataset: pred = example.predicted_tags gold = example.tags pred_str = ",".join(map(str, pred)) pred_tags.append(pred_str) if writer is not None: writer.write(pred_str) writer.write("\n") gold_tags.append(",".join(map(str, gold))) scoreList, infoList = getFscore( gold_tags, pred_tags, self.idx_to_chunk_tag ) config.swLog.write( "#gold-chunk={} #output-chunk={} #correct-output-chunk={} precision={:.2f} recall={:.2f} f-score={:.2f}\n".format( infoList[0], infoList[1], infoList[2], scoreList[1], scoreList[2], scoreList[0], ) ) return scoreList # acc = correct / total * 100.0 # config.swLog.write( # "total-tag-strings={} correct-tag-strings={} string-accuracy={}%".format( # total, correct, acc # ) # ) # goldTagList = [] # resTagList = [] # for x in X2: # res = "" # for im in x._yOutput: # res += str(im) + "," # resTagList.append(res) # # if not dynamic: # if writer is not None: # for i in range(len(x._yOutput)): # writer.write(str(x._yOutput[i]) + ",") # writer.write("\n") # goldTags = x._x.getTags() # gold = "" # for im in goldTags: # gold += str(im) + "," # goldTagList.append(gold) # # if dynamic: # # return resTagList # scoreList = [] # if config.runMode == "train": # infoList = [] # scoreList = getFscore( # goldTagList, resTagList, infoList, self.idx_to_chunk_tag # ) # config.swLog.write( # "#gold-chunk={} #output-chunk={} #correct-output-chunk={} precision={:.2f} recall={:.2f} f-score={:.2f}\n".format( # infoList[0], # infoList[1], # infoList[2], # "%.2f" % scoreList[1], # "%.2f" % scoreList[2], # "%.2f" % scoreList[0], # ) # ) # return scoreList # # def multiThreading(self, X, X2): # config = self.config # # if dynamic: # # for i in range(len(X)): # # X2.append(dataSeqTest(X[i], [])) # # for k, x in enumerate(X2): # # tags = [] # # prob = self.Inf.decodeViterbi_fast(self.Model, x._x, tags) # # X2[k]._yOutput.clear() # # X2[k]._yOutput.extend(tags) # # return # for i in range(len(X)): # X2.append(dataSeqTest(X[i], [])) # if len(X) < config.nThread: # config.nThread = len(X) # interval = (len(X2) + config.nThread - 1) // config.nThread # procs = [] # Q = Queue(5000) # for i in range(config.nThread): # start = i * interval # end = min(start + interval, len(X2)) # proc = Process( # target=Trainer.taskRunner_test, # args=(self.Inf, self.Model, X2, start, end, Q), # ) # proc.start() # procs.append(proc) # for i in range(len(X2)): # t = Q.get() # k, tags = t # X2[k]._yOutput.clear() # X2[k]._yOutput.extend(tags) # for proc in procs: # proc.join() # @staticmethod # def taskRunner_test(Inf, Model, X2, start, end, Q): # for k in range(start, end): # x = X2[k] # tags = [] # prob = Inf.decodeViterbi_fast(Model, x._x, tags) # Q.put((k, tags)) ================================================ FILE: readme/comparison.md ================================================ # ϸѵԽ ڲͬݼϵĶԱȽ | MSRA | Precision | Recall | F-score | | :----- | --------: | -----: | --------: | | jieba | 87.01 | 89.88 | 88.42 | | THULAC | 95.60 | 95.91 | 95.71 | | pkuseg | 96.94 | 96.81 | **96.88** | | CTB8 | Precision | Recall | F-score | | :----- | --------: | -----: | --------: | | jieba | 88.63 | 85.71 | 87.14 | | THULAC | 93.90 | 95.30 | 94.56 | | pkuseg | 95.99 | 95.39 | **95.69** | | WEIBO | Precision | Recall | F-score | | :----- | --------: | -----: | --------: | | jieba | 87.79 | 87.54 | 87.66 | | THULAC | 93.40 | 92.40 | 92.87 | | pkuseg | 93.78 | 94.65 | **94.21** | #### Խ ѡ˻CTB8ϵѵѵͬʱвԣģģڡںݡϵķִЧѡCTB8ϵԭǣCTB8ڻϣµЧãڲǷCTB8ѵģͣй߰ԶԻøߵƽЧǿԵĽ | CTB8 Training | MSRA | CTB8 | PKU | WEIBO | All Average | OOD Average | | ------------- | ----- | ----- | ----- | ----- | ----------- | ----------- | | jieba | 82.75 | 87.14 | 87.12 | 85.68 | 85.67 | 85.18 | | THULAC | 83.50 | 94.56 | 89.13 | 91.00 | 89.55 | 87.88 | | pkuseg | 83.67 | 95.69 | 89.67 | 91.19 | 90.06 | **88.18** | У`All Average`ʾвԼ(CTB8Լ)F-scoreƽ`OOD Average` (Out-of-domain Average)ʾڳCTB8Լƽ #### ĬģڲͬIJЧ ǵܶûڳԷִʹߵʱ򣬴ʱʹù߰ԴģͲԡΪֱӶԱȡʼܣҲȽ˸߰ĬģڲͬIJЧע⣬ıȽֻΪ˵ĬµЧһǹƽġ | Default | MSRA | CTB8 | PKU | WEIBO | All Average | | ------- | :---: | :---: | :---: | :---: | :---------: | | jieba | 81.45 | 79.58 | 81.83 | 83.56 | 81.61 | | THULAC | 85.55 | 87.84 | 92.29 | 86.65 | 88.08 | | pkuseg | 87.29 | 91.77 | 92.68 | 93.43 | **91.29** | У`All Average`ʾвԼF-scoreƽ ================================================ FILE: readme/environment.md ================================================ # 实验环境 考虑到jieba分词和THULAC工具包等并没有提供细领域的预训练模型，为了便于比较，我们重新使用它们提供的训练接口在细领域的数据集上进行训练，用训练得到的模型进行中文分词。 我们选择Linux作为测试环境，在新闻数据(MSRA)、混合型文本(CTB8)、网络文本(WEIBO)数据上对不同工具包进行了准确率测试。我们使用了第二届国际汉语分词评测比赛提供的分词评价脚本。其中MSRA与WEIBO使用标准训练集测试集划分，CTB8采用随机划分。对于不同的分词工具包，训练测试数据的划分都是一致的；**即所有的分词工具包都在相同的训练集上训练，在相同的测试集上测试**。对于所有数据集，pkuseg使用了不使用词典的训练和测试接口。以下是pkuseg训练和测试代码示例: ``` pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models') pkuseg.test('msr_test.raw', 'output.txt', user_dict=None) ``` ================================================ FILE: readme/history.md ================================================ # 汾ʷ - v0.0.11(2019-01-09) - ޶ĬãCTB8ΪĬģͣʹôʵ - v0.0.14(2019-01-23) - ޸˴ʵ䴦˴ʵ䣬ִЧ - **ЧʽŻٶȽ֮ǰ汾9** - ڴģݼѵͨģͣΪĬʹģ - v0.0.15(2019-01-30) - ֧fine-tuneѵԤصģͼѵ֧趨ѵ - v0.0.18(2019-02-20) - **ִ֧Աעҽơϸģ** ================================================ FILE: readme/interface.md ================================================ # 代码示例 以下代码示例适用于python交互式环境。 代码示例1：使用默认配置进行分词（**如果用户无法确定分词领域，推荐使用默认模型分词**） ```python3 import pkuseg seg = pkuseg.pkuseg() # 以默认配置加载模型 text = seg.cut('我爱北京天安门') # 进行分词 print(text) ``` 代码示例2：细领域分词（**如果用户明确分词领域，推荐使用细领域模型分词**） ```python3 import pkuseg seg = pkuseg.pkuseg(model_name='medicine') # 程序会自动下载所对应的细领域模型 text = seg.cut('我爱北京天安门') # 进行分词 print(text) ``` 代码示例3：分词同时进行词性标注，各词性标签的详细含义可参考 [tags.txt](https://github.com/lancopku/pkuseg-python/blob/master/tags.txt) ```python3 import pkuseg seg = pkuseg.pkuseg(postag=True) # 开启词性标注功能 text = seg.cut('我爱北京天安门') # 进行分词和词性标注 print(text) ``` 代码示例4：对文件分词 ```python3 import pkuseg # 对input.txt的文件分词输出到output.txt中 # 开20个进程 pkuseg.test('input.txt', 'output.txt', nthread=20) ``` 代码示例5：额外使用用户自定义词典 ```python3 import pkuseg seg = pkuseg.pkuseg(user_dict='my_dict.txt') # 给定用户词典为当前目录下的"my_dict.txt" text = seg.cut('我爱北京天安门') # 进行分词 print(text) ``` 代码示例6：使用自训练模型分词（以CTB8模型为例） ```python3 import pkuseg seg = pkuseg.pkuseg(model_name='./ctb8') # 假设用户已经下载好了ctb8的模型并放在了'./ctb8'目录下，通过设置model_name加载该模型 text = seg.cut('我爱北京天安门') # 进行分词 print(text) ``` 代码示例7：训练新模型 （模型随机初始化） ```python3 import pkuseg # 训练文件为'msr_training.utf8' # 测试文件为'msr_test_gold.utf8' # 训练好的模型存到'./models'目录下 # 训练模式下会保存最后一轮模型作为最终模型 # 目前仅支持utf-8编码，训练集和测试集要求所有单词以单个或多个空格分开 pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models') ``` 代码示例8：fine-tune训练（从预加载的模型继续训练） ```python3 import pkuseg # 训练文件为'train.txt' # 测试文件为'test.txt' # 加载'./pretrained'目录下的模型，训练好的模型保存在'./models'，训练10轮 pkuseg.train('train.txt', 'test.txt', './models', train_iter=10, init_model='./pretrained') ================================================ FILE: readme/multiprocess.md ================================================ # 多进程分词 当将以上代码示例置于文件中运行时，如涉及多进程功能，请务必使用`if __name__ == '__main__'`保护全局语句，如： mp.py文件 ```python3 import pkuseg if __name__ == '__main__': pkuseg.test('input.txt', 'output.txt', nthread=20) pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models', nthread=20) ``` 运行 ``` python3 mp.py ``` 详见[无法使用多进程分词和训练功能，提示RuntimeError和BrokenPipeError](https://github.com/lancopku/pkuseg-python/wiki#3-无法使用多进程分词和训练功能提示runtimeerror和brokenpipeerror)。 **在Windows平台上，请当文件足够大时再使用多进程分词功能**，详见[关于多进程速度问题](https://github.com/lancopku/pkuseg-python/wiki#9-关于多进程速度问题)。 ================================================ FILE: readme/readme_english.md ================================================ # Pkuseg A multi-domain Chinese word segmentation toolkit. ## Highlights The pkuseg-python toolkit has the following features: 1. Supporting multi-domain Chinese word segmentation. Pkuseg-python supports multi-domain segmentation, including domains like news, web, medicine, and tourism. Users are free to choose different pre-trained models according to the domain features of the text to be segmented. If not sure the domain of the text, users are recommended to use the default model trained on mixed-domain data. 2. Higher word segmentation results. Compared with existing word segmentation toolkits, pkuseg-python can achieve higher F1 scores on the same dataset. 3. Supporting model training. Pkuseg-python also supports users to train a new segmentation model with their own data. 4. Supporting POS tagging. We also provide users POS tagging interfaces for further lexical analysis. ## Installation - Requirements: python3 1. Install pkuseg-python by using PyPI: (with the default model trained on mixed-doimain data) ``` pip3 install pkuseg ``` or update to the latest version (**suggested**): ``` pip3 install -U pkuseg ``` 2. Install pkuseg-python by using image source for fast speed: ``` pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple pkuseg ``` or update to the latest version (**suggested**): ``` pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple -U pkuseg ``` Note: The previous two installing commands only support python3.5, python3.6, python3.7 on linux, mac, and **windows 64 bit**. 3. If the code is downloaded from GitHub, please run the following command to install pkuseg-python: ``` python setup.py build_ext -i ``` Note: the github code does not contain the pre-trained models, users need to download the pre-trained models from [release](https://github.com/lancopku/pkuseg-python/releases), and set parameter 'model_name' as the model path. ## Usage #### Examples Example 1: Segmentation under the default configuration. **If users are not sure the domain of the text to be segmented, the default configuration is recommended.** ```python3 import pkuseg seg = pkuseg.pkuseg() #load the default model text = seg.cut('我爱北京天安门') print(text) ``` Example 2: Domain-specific segmentation. **If users know the text domain, they can select a pre-trained domain model according to the domain features.** ```python3 import pkuseg seg = pkuseg.pkuseg(model_name='medicine') #Automatically download the domain-specific model. text = seg.cut('我爱北京天安门') print(text) ``` Example 3：Segmentation and POS tagging. For the detailed meaning of each POS tag, please refer to [tags.txt](https://github.com/lancopku/pkuseg-python/blob/master/tags.txt). ```python3 import pkuseg seg = pkuseg.pkuseg(postag=True) text = seg.cut('我爱北京天安门') print(text) ``` Example 4：Segmentation with a text file as input. ```python3 import pkuseg #Take file 'input.txt' as input. #The segmented result is stored in file 'output.txt'. pkuseg.test('input.txt', 'output.txt', nthread=20) ``` Example 5: Segmentation with a user-defined dictionary. ```python3 import pkuseg seg = pkuseg.pkuseg(user_dict='my_dict.txt') text = seg.cut('我爱北京天安门') print(text) ``` Example 6: Segmentation with a user-trained model. Take CTB8 as an example. ```python3 import pkuseg seg = pkuseg.pkuseg(model_name='./ctb8') text = seg.cut('我爱北京天安门') print(text) ``` Example 7: Training a new model (randomly initialized). ```python3 import pkuseg # Training file: 'msr_training.utf8'. # Test file: 'msr_test_gold.utf8'. # Save the trained model to './models'. # The training and test files are in utf-8 encoding. pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models') ``` Example 8: Fine-tuning. Take a pre-trained model as input. ```python3 import pkuseg # Training file: 'train.txt'. # Testing file'test.txt'. # The path of the pre-trained model: './pretrained'. # Save the trained model to './models'. # The training and test files are in utf-8 encoding. pkuseg.train('train.txt', 'test.txt', './models', train_iter=10, init_model='./pretrained') ``` #### Parameter Settings Segmentation for sentences. ``` pkuseg.pkuseg(model_name = "default", user_dict = "default", postag = False) model_name The path of the used model. "default". The default mixed-domain model. "news". The model trained on news domain data. "web". The model trained on web domain data. "medicine". The model trained on medicine domain data. "tourism". The model trained on tourism domain data. model_path. Load a model from the user-specified path. user_dict Set up the user dictionary. "default". Use the default dictionary. None. No dictionary is used. dict_path. The path of the user-defined dictionary. Each line only contains one word. postag POS tagging or not. False. The default setting. Segmentation without POS tagging. True. Segmentation with POS tagging. ``` Segmentation for documents. ``` pkuseg.test(readFile, outputFile, model_name = "default", user_dict = "default", postag = False, nthread = 10) readFile The path of the input file. outputFile The path of the output file. model_name The path of the used model. Refer to pkuseg.pkuseg. user_dict The path of the user dictionary. Refer to pkuseg.pkuseg. postag POS tagging or not. Refer to pkuseg.pkuseg. nthread The number of threads. ``` Model training. ``` pkuseg.train(trainFile, testFile, savedir, train_iter = 20, init_model = None) trainFile The path of the training file. testFile The path of the test file. savedir The saved path of the trained model. train_iter The maximum number of training epochs. init_model By default, None means random initialization. Users can also load a pre-trained model as initialization, like init_model='./models/'. ``` ## Publication The toolkit is mainly based on the following publication. If you use the toolkit, please cite the paper: * Ruixuan Luo, Jingjing Xu, Yi Zhang, Zhiyuan Zhang, Xuancheng Ren, Xu Sun. [PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation](https://arxiv.org/abs/1906.11455). Arxiv. 2019. ``` @article{pkuseg, author = {Luo, Ruixuan and Xu, Jingjing and Zhang, Yi and Zhang, Zhiyuan and Ren, Xuancheng and Sun, Xu}, journal = {CoRR}, title = {PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation.}, url = {https://arxiv.org/abs/1906.11455}, volume = {abs/1906.11455}, year = 2019 } ``` ## Related Work * Xu Sun, Houfeng Wang, Wenjie Li. Fast Online Training with Frequency-Adaptive Learning Rates for Chinese Word Segmentation and New Word Detection. ACL. 2012. * Jingjing Xu and Xu Sun. Dependency-based gated recursive neural network for chinese word segmentation. ACL. 2016. * Jingjing Xu and Xu Sun. Transfer learning for low-resource chinese word segmentation with a novel neural network. NLPCC. 2017. ## Authors Ruixuan Luo, Jingjing Xu, Xuancheng Ren, Yi Zhang, Zhiyuan Zhang, Bingzhen Wei, Xu Sun [Language Computing and Machine Learning Group](http://lanco.pku.edu.cn/), Peking University ================================================ FILE: setup.py ================================================ import setuptools import os from distutils.extension import Extension import numpy as np def is_source_release(path): return os.path.exists(os.path.join(path, "PKG-INFO")) def setup_package(): root = os.path.abspath(os.path.dirname(__file__)) long_description = "pkuseg-python" extensions = [ Extension( "pkuseg.inference", ["pkuseg/inference.pyx"], include_dirs=[np.get_include()], language="c++" ), Extension( "pkuseg.feature_extractor", ["pkuseg/feature_extractor.pyx"], include_dirs=[np.get_include()], ), Extension( "pkuseg.postag.feature_extractor", ["pkuseg/postag/feature_extractor.pyx"], include_dirs=[np.get_include()], ), ] if not is_source_release(root): from Cython.Build import cythonize extensions = cythonize(extensions, annotate=True) setuptools.setup( name="pkuseg", version="0.0.25", author="Lanco", author_email="luoruixuan97@pku.edu.cn", description="A small package for Chinese word segmentation", long_description=long_description, long_description_content_type="text/markdown", url="https://github.com/lancopku/pkuseg-python", packages=setuptools.find_packages(), package_data={"": ["*.txt*", "*.pkl", "*.npz", "*.pyx", "*.pxd"]}, classifiers=[ "Programming Language :: Python :: 3", "License :: Other/Proprietary License", "Operating System :: OS Independent", ], install_requires=["cython", "numpy>=1.16.0"], setup_requires=["cython", "numpy>=1.16.0"], ext_modules=extensions, zip_safe=False, ) if __name__ == "__main__": setup_package()