[
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) [2017] [Wang Chunqi]\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE."
  },
  {
    "path": "README.md",
    "content": "# CONV-SEG\nConvolutional neural network for Chinese word segmentation (CWS).\nThe corresponding paper: [**Convolutional Neural Network with Word Embeddings for Chinese Word Segmentation**](https://arxiv.org/pdf/1711.04411.pdf)\n\n## Author\nChunqi Wang\n\n## Dependencies\n* [Python 2.7](https://www.python.org/)\n* [Tensorflow 1.0](https://www.tensorflow.org/)\n\nIt is better to use a nvidia GPU to accelerate the training procedure.\n\n## Data\nDownlaod `data.zip` from [here](https://drive.google.com/file/d/0B-f0oKMQIe6sQVNxeE9JeUJfQ0k/view?usp=sharing&resourcekey=0-pqrbXYjENu6AJxR6mq8zJQ) (Note that the SIGHAN datasets should only be used for research purposes). Extract `data.zip` to this directory. So the file tree would be:\n\n\tconvseg\n\t|\tdata\n\t|\t|\tdatasets\n\t|\t|\t|\tsighan2005-pku\n\t|\t|\t|\t|\ttrain.txt\n\t|\t|\t|\t|\tdev.txt\n\t|\t|\t|\t|\ttest.txt\n\t|\t|\t|\tsighan2005-msr\n\t|\t|\t|\t|\ttrain.txt\n\t|\t|\t|\t|\tdev.txt\n\t|\t|\t|\t|\ttest.txt\n\t|\t|\tembeddings\n\t|\t|\t|\tnews_tensite.w2v200\n\t|\t|\t|\tnews_tensite.pku.words.w2v50\n\t|\t|\t|\tnews_tensite.msr.words.w2v50\n\t|\ttagger.py\n\t|\ttrain_cws.py\n\t|\ttrain_cws.sh\n\t|\ttrain_cws_wemb.sh\n\t|\tscore.perl\n\t|\tREADME.md\n\n## How to use\nFirst, give execute permission to scripts:\n\n\tchmod +x train_cws.sh train_cws_wemb.sh\n\nTrain a preliminary model (CONV-SEG):\n\n\t./train_cws.sh WHICH_DATASET WHICH_GPU\n\t\nTrain a model with word embeddings (WE-CONV-SEG):\n\n\t./train_cws_wemb.sh WHICH_DATASET WHICH_GPU\n\t\nWe have two optional datasets: `pku` and `msr`. If you run the program in CPU environment, just leave the second argument empty.\n\nFor example, if you want to train the model CONV-SEG on the pku dataset and on gpu0, you should run:\n\n\t./train_cws.sh pku 0\n\t\nMore arguments can be set in `train.py`.\n\n## Test Score\n| Model | PKU(dev) | PKU(test) | MSR(dev) | MSR(test) |\n|:------|:---------|:----------|:---------|:----------|\n| CONV-SEG | 96.8 | 95.7 | 97.2 | 97.3\t|\n| WE-CONV-SEG | 97.5 |\t96.5\t| 98.1 |\t98.0 |\n"
  },
  {
    "path": "cws.py",
    "content": "from __future__ import print_function\nimport os, codecs\nfrom itertools import izip\nfrom tagger import data_iterator\n\n\nscope = 'CWS'\n\n\ndef process_train_sentence(sentence, bigram, word_window):\n    sentence = sentence.strip()\n    words = sentence.split()\n    chars = []\n    tags = []\n    ret = []\n    for w in words:\n        chars.extend(list(w))\n        if len(w) == 1:\n            tags.append('S')\n        else:\n            tags.extend(['B'] + ['M'] * (len(w) - 2) + ['E'])\n    ret.append(chars)\n    if bigram:\n        chars = ['', ''] + chars + ['', '']\n        ret.append([a + b if a and b else '' for a, b in zip(chars[:-4], chars[1:])])\n        ret.append([a + b if a and b else '' for a, b in zip(chars[1:-3], chars[2:])])\n        ret.append([a + b if a and b else '' for a, b in zip(chars[2:-2], chars[3:])])\n        ret.append([a + b if a and b else '' for a, b in zip(chars[3:-1], chars[4:])])\n    elif word_window > 0:\n        chars = ['', '', ''] + chars + ['', '', '']\n        # single char\n        if word_window >= 1:\n            ret.append(chars[3:-3])\n        if word_window >= 2:\n            # bi chars\n            ret.append([a + b if a and b else '' for a, b in zip(chars[2:], chars[3:-3])])\n            ret.append([a + b if a and b else '' for a, b in zip(chars[3:-3], chars[4:])])\n        if word_window >= 3:\n            # tri chars\n            ret.append(\n                [a + b + c if a and b and c else '' for a, b, c in zip(chars[1:], chars[2:], chars[3:-3])])\n            ret.append(\n                [a + b + c if a and b and c else '' for a, b, c in zip(chars[2:], chars[3:-3], chars[4:])])\n            ret.append(\n                [a + b + c if a and b and c else '' for a, b, c in zip(chars[3:-3], chars[4:], chars[5:])])\n        if word_window >= 4:\n            # four chars\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                            zip(chars[0:], chars[1:], chars[2:], chars[3:-3])])\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                            zip(chars[1:], chars[2:], chars[3:-3], chars[4:])])\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                            zip(chars[2:], chars[3:-3], chars[4:], chars[5:])])\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                             zip(chars[3:-3], chars[4:], chars[5:], chars[6:])])\n    ret.append(tags)\n    return ret\n\n\ndef process_raw_sentence(sentence, bigram, word_window):\n    sentence = sentence.strip()\n    chars = list(sentence)\n    ret = [chars]\n    if bigram:\n        chars = ['', ''] + chars + ['', '']\n        ret.append([a + b if a and b else '' for a, b in zip(chars[:-4], chars[1:])])\n        ret.append([a + b if a and b else '' for a, b in zip(chars[1:-3], chars[2:])])\n        ret.append([a + b if a and b else '' for a, b in zip(chars[2:-2], chars[3:])])\n        ret.append([a + b if a and b else '' for a, b in zip(chars[3:-1], chars[4:])])\n    elif word_window > 0:\n        chars = ['', '', ''] + chars + ['', '', '']\n        # single char\n        if word_window >= 1:\n            ret.append(chars[3:-3])\n        if word_window >= 2:\n            # bi chars\n            ret.append([a + b if a and b else '' for a, b in zip(chars[2:], chars[3:-3])])\n            ret.append([a + b if a and b else '' for a, b in zip(chars[3:-3], chars[4:])])\n        if word_window >= 3:\n            # tri chars\n            ret.append(\n                [a + b + c if a and b and c else '' for a, b, c in zip(chars[1:], chars[2:], chars[3:-3])])\n            ret.append(\n                [a + b + c if a and b and c else '' for a, b, c in zip(chars[2:], chars[3:-3], chars[4:])])\n            ret.append(\n                [a + b + c if a and b and c else '' for a, b, c in zip(chars[3:-3], chars[4:], chars[5:])])\n        if word_window >= 4:\n            # four chars\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                            zip(chars[0:], chars[1:], chars[2:], chars[3:-3])])\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                            zip(chars[1:], chars[2:], chars[3:-3], chars[4:])])\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                            zip(chars[2:], chars[3:-3], chars[4:], chars[5:])])\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                             zip(chars[3:-3], chars[4:], chars[5:], chars[6:])])\n    return ret\n\n\ndef read_train_file(fin, bigram=False, word_window=4):\n    \"\"\"\n    Read training data.\n    \"\"\"\n    data = []\n    for l in fin:\n        data.append(process_train_sentence(l, bigram, word_window))\n    return zip(*data)\n\n\ndef read_raw_file(fin, batch_size, bigram=False, word_window=4):\n    \"\"\"\n    Read raw data.\n    \"\"\"\n    buffer = []\n    max_buffer = 100000\n    for i, l in enumerate(fin):\n        buffer.append(process_raw_sentence(l, bigram, word_window))\n        if i % max_buffer == 0 and i > 0:\n            for b in data_iterator(zip(*buffer), batch_size, shuffle=False):\n                yield b\n            buffer = []\n    if buffer:\n        for b in data_iterator(zip(*buffer), batch_size, shuffle=False):\n            yield b\n\n\ndef read_raw_file_all(fin, bigram=False, word_window=4):\n    \"\"\"\n    Read raw data.\n    \"\"\"\n    data = []\n    for b in read_raw_file(fin, 1000, bigram, word_window):\n        data.extend(zip(*b))\n    return zip(*data)\n\n\ndef create_output(seqs, stags):\n    \"\"\"\n    Create final output from characters and BMES tags.\n    \"\"\"\n    output = []\n    for seq, stag in izip(seqs, stags):\n        new_sen = []\n        for c, tag in izip(seq, stag):\n            new_sen.append(c)\n            if tag == 'S' or tag == 'E':\n                new_sen.append('  ')\n        output.append(''.join(new_sen))\n    return output\n\n\ndef evaluator(data, output_dir, output_flag):\n    \"\"\"\n    Evaluate presion, recall and F1.\n    \"\"\"\n    seqs, gold_stags, pred_stags = data\n    assert len(seqs) == len(gold_stags) == len(pred_stags)\n    # Create and open temp files.\n    if not os.path.exists(output_dir):\n        os.makedirs(output_dir)\n\n    ref_path = os.path.join(output_dir, '%s.ref' % output_flag)\n    pred_path = os.path.join(output_dir, '%s.pred' % output_flag)\n    score_path = os.path.join(output_dir, '%s.score' % output_flag)\n    # Empty words file.\n    temp_path = os.path.join(output_dir, '%s.temp' % output_flag)\n\n    ref_file = codecs.open(ref_path, 'w', 'utf8')\n    pred_file = codecs.open(pred_path, 'w', 'utf8')\n    for l in create_output(seqs, gold_stags):\n        print(l, file=ref_file)\n    for i, l in enumerate(create_output(seqs, pred_stags)):\n        print(l, file=pred_file)\n    ref_file.close()\n    pred_file.close()\n\n    os.system('echo > %s' % temp_path)\n    os.system('%s  %s %s %s > %s' % ('./score.perl', temp_path, ref_path, pred_path, score_path))\n    # Sighan evaluation results\n    os.system('tail -n 7 %s > %s' % (score_path, temp_path))\n    eval_lines = [l.rstrip() for l in codecs.open(temp_path, 'r', 'utf8')]\n    # Remove temp files.\n    os.remove(ref_path)\n    os.remove(pred_path)\n    os.remove(score_path)\n    os.remove(temp_path)\n    # Precision, Recall and F1 score\n    return (float(eval_lines[1].split(':')[1]),\n            float(eval_lines[0].split(':')[1]),\n            float(eval_lines[2].split(':')[1]))"
  },
  {
    "path": "score.perl",
    "content": "#!/usr/bin/perl -w\n\n###########################################################################\n#                                                                         #\n#                               SIGHAN                                    #\n#                      Copyright (c) 2003,2005                            #\n#                        All Rights Reserved.                             #\n#                                                                         #\n#  Permission is hereby granted, free of charge, to use and distribute    #\n#  this software and its documentation without restriction, including     #\n#  without limitation the rights to use, copy, modify, merge, publish,    #\n#  distribute, sublicense, and/or sell copies of this work, and to        #\n#  permit persons to whom this work is furnished to do so, subject to     #\n#  the following conditions:                                              #\n#   1. The code must retain the above copyright notice, this list of      #\n#      conditions and the following disclaimer.                           #\n#   2. Any modifications must be clearly marked as such.                  #\n#   3. Original authors' names are not deleted.                           #\n#   4. The authors' names are not used to endorse or promote products     #\n#      derived from this software without specific prior written          #\n#      permission.                                                        #\n#                                                                         #\n#  SIGHAN AND THE CONTRIBUTORS TO THIS WORK DISCLAIM ALL WARRANTIES       #\n#  WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF      #\n#  MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL SIGHAN NOR THE          #\n#  CONTRIBUTORS BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL      #\n#  DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA     #\n#  OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER      #\n#  TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR       #\n#  PERFORMANCE OF THIS SOFTWARE.                                          #\n#                                                                         #\n###########################################################################\n#                                                                         #\n# Author: Richard Sproat (rws@uiuc.edu)                                   #\n#         Tom Emerson (tree@basistech.com)                                #\n#                                                                         #\n###########################################################################\n\n## This code depends upon a version of diff (e.g. GNU diffutils 2.7.2)\n## that supports the -y flag:\n##\n## -y     Use the side by side output format.\n##\n## change the following per your installation:\n\n$diff = \"/usr/bin/diff\";\n\n$USAGE = \"Usage:\\t$0 dictionary truth test\\n\\t\";\n\nif (@ARGV != 3) {print \"$USAGE\\n\"; exit;}\n\n$tmp1 = \"/tmp/comp01$$\";\n$tmp2 = \"/tmp/comp02$$\";\n\n%dict = ();\n\nopen (S, $ARGV[0]) or  die \"$ARGV[0]: $!\\n\";\n\nwhile (<S>) {\n    chop;\n    s/^\\s*//;\n    s/\\s*$//;\n    $dict{$_} = 1;\n}\n\nclose(S);\n\nopen (TRUTH, $ARGV[1]) or die \"$ARGV[1]: $!\\n\";\nopen (TEST, $ARGV[2]) or die \"$ARGV[2]: $!\\n\";\n\n$Tot = $Del = $Ins = $Subst = $Truecount = $Testcount = 0;\n$RawRecall = $RawPrecision = 0;\n\n$linenum = 0;\n\n\n$IVMISSED = $OOVMISSED = $OOV = $IV = 0;\n\n$file1 = $ARGV[1];\n$file2 = $ARGV[2];\n$file1 =~ s=^/.*/==;\n$file2 =~ s=^/.*/==;\n\nwhile (defined($truth = <TRUTH>) && defined($test = <TEST>)) {\n    $truth =~ s/^\\s*//;\n    $test =~ s/^\\s*//;\n    $truth =~ s/\\s*$//;\n    $test =~ s/\\s*$//;\n    $truth =~ s/(\\xe3\\x80\\x80)|(\\xa1\\x40)/ /g;\n    $test =~ s/(\\xe3\\x80\\x80)|(\\xa1\\x40)/ /g;\n    $truth =~ s/\n//g;\n    $test =~ s/\n//g;\n    @truthwords = split /\\s+/, $truth;\n    @testwords = split /\\s+/, $test;\n    $truecount = scalar(@truthwords);\n    $testcount = scalar(@testwords);\n    ++$linenum;\n    if ($truecount == 0) { \n\tif ($testcount > 0) {\n\t    print STDERR \"Warning: training is 0 but test is nonzero, possible misalignment at line $linenum.\\n\";\n\t}\n\tnext; \n    }\n    if ($testcount == 0) { \n\tprint STDERR \"Warning: No output in test data where there is in training data, line $linenum\\n\";\n    }\n    open (T1, \">$tmp1\") or die \"Can't open $tmp1\";\n    open (T2, \">$tmp2\") or die \"Can't open $tmp2\";\n    foreach my $w (@truthwords) { print T1 \"$w\\n\"; }\n    foreach my $w (@testwords) {print T2 \"$w\\n\";}\n    close (T1);\n    close (T2);\n    open (P, \"$diff -y $tmp1 $tmp2 |\") \n\tor die \"Can't open pipe.\\n\";\n    print \"--$file1-------$file2----$linenum\\n\";\n    my $del = 0;\n    my $ins = 0;\n    my $subst = 0;\n    my $rawrecall = 0;\n    my $rawprecision = 0;\n    while (<P>) {\n\tmy $err = 0;\n\tif (/\\s\\|\\s/) {$subst++ ; $err++; }\n\telsif (/\\s\\>\\s/) {$ins++ ; $err++; }\n\telsif (/\\s\\<\\s/) {$del++ ; $err++; }\n\tif (/^([^\\s]+)\\s/) { \n\t    my $w = $1;\n\t    if (!$dict{$w}) {++$OOV;}\t    \n\t    else {++$IV;}\n\t    if (/^[^\\s]+\\s.*\\s[\\|\\>\\<]\\s/) {\n\t\tif (!$dict{$w}) {++$OOVMISSED;}\n\t\telse {++$IVMISSED;}\n\t\t++$rawrecall; \n\t    }\n\t}\n\tif (/\\s[\\|\\>\\<]\\s.*[^\\s]$/) { ++$rawprecision; }\n\tprint \"$_\";\n    }\n    close (P);\n    my $tot = $del + $ins + $subst;\n    $Tot += $tot;\n    $Del += $del;\n    $Ins += $ins;\n    $Subst += $subst;\n    $Truecount += $truecount;\n    $Testcount += $testcount;\n    $rawrecall = $truecount - $rawrecall;\n    $rawprecision = $testcount - $rawprecision;\n    $RawRecall += $rawrecall;\n    $RawPrecision += $rawprecision;\n    $rawrecall = sprintf(\"%2.3f\", $rawrecall/$truecount);\n    $rawprecision = sprintf(\"%2.3f\", $rawprecision/$testcount);\n    print \"INSERTIONS:\\t$ins\\n\";\n    print \"DELETIONS:\\t$del\\n\";\n    print \"SUBSTITUTIONS:\\t$subst\\n\";\n    print \"NCHANGE:\\t$tot\\n\";\n    print \"NTRUTH:\\t$truecount\\n\";\n    print \"NTEST:\\t$testcount\\n\";\n    print \"TRUE WORDS RECALL:\\t$rawrecall\\n\";\n    print \"TEST WORDS PRECISION:\\t$rawprecision\\n\";\n}\n\nclose(TRUTH);\nclose(TEST);\nunlink($tmp1);\nunlink($tmp2);\n\nprint \"=== SUMMARY:\\n\";\nprint \"=== TOTAL INSERTIONS:\\t$Ins\\n\";\nprint \"=== TOTAL DELETIONS:\\t$Del\\n\";\nprint \"=== TOTAL SUBSTITUTIONS:\\t$Subst\\n\";\nprint \"=== TOTAL NCHANGE:\\t$Tot\\n\";\nprint \"=== TOTAL TRUE WORD COUNT:\\t$Truecount\\n\";\nprint \"=== TOTAL TEST WORD COUNT:\\t$Testcount\\n\";\n$RawRecall =  $RawRecall/$Truecount;\n$RawPrecision = $RawPrecision/$Testcount;\n$beta = 1;\n$R = $RawRecall;\n$P = $RawPrecision;\n\nif ($R != 0. || $P != 0.) {\n    $F = (1 + $beta)*$P*$R/($beta * $P + $R);\n}\nelse{\n    $F=0.;\n}\n\n$F = sprintf(\"%2.3f\", $F);\n$RawRecall = sprintf(\"%2.3f\", $RawRecall);\n$RawPrecision = sprintf(\"%2.3f\", $RawPrecision);\nprint \"=== TOTAL TRUE WORDS RECALL:\\t$RawRecall\\n\";\nprint \"=== TOTAL TEST WORDS PRECISION:\\t$RawPrecision\\n\";\nprint \"=== F MEASURE:\\t$F\\n\";\nif ($OOV > 0) {\n    $OOVMISSED = sprintf(\"%2.3f\", 1 - $OOVMISSED / $OOV);\n}\nelse {\n    $OOVMISSED = \"--\";\n}\n$OOV = sprintf(\"%2.3f\", $OOV / $Truecount);\nif ($IV > 0) {\n    $IVMISSED = sprintf(\"%2.3f\", 1 - $IVMISSED / $IV);\n}\nelse {\n    $IVMISSED = \"--\";\n}\nprint \"=== OOV Rate:\\t$OOV\\n\";\nprint \"=== OOV Recall Rate:\\t$OOVMISSED\\n\";\nprint \"=== IV Recall Rate:\\t$IVMISSED\\n\";\n\nprint \"###\\t$file2\\t$Ins\\t$Del\\t$Subst\\t$Tot\\t$Truecount\\t$Testcount\\t$RawRecall\\t$RawPrecision\\t$F\\t$OOV\\t$OOVMISSED\\t$IVMISSED\\n\";\nexit(0);\n"
  },
  {
    "path": "server.py",
    "content": "# -*- coding:utf-8 -* \nfrom __future__ import print_function\nimport StringIO\nfrom argparse import ArgumentParser\nimport tornado.ioloop\nimport tornado.web\nimport tensorflow as tf\n\nfrom tagger import Model\n\n\ndef load_template():\n    page = '''\n    <!DOCTYPE html>\n    <html>\n    <head>\n    <meta charset=\"utf-8\">\n    <script>\n    function loadXMLDoc()\n    {\n        var xmlhttp;\n        if (window.XMLHttpRequest)\n        {\n            // IE7+, Firefox, Chrome, Opera, Safari\n            xmlhttp=new XMLHttpRequest();\n        }\n        else\n        {\n            // IE6, IE5\n            xmlhttp=new ActiveXObject(\"Microsoft.XMLHTTP\");\n        }\n        xmlhttp.onreadystatechange=function()\n        {\n            if (xmlhttp.readyState==4 && xmlhttp.status==200)\n            {\n                document.getElementById(\"out\").innerHTML=decodeURI(xmlhttp.responseText);\n            }\n        }\n\n        var sentences=document.getElementById(\"in\").value\n        var req=\"sentences=\"+sentences\n        req=encodeURI(req)\n        xmlhttp.open(\"POST\",\"/%s\",true);\n        xmlhttp.setRequestHeader(\"Content-type\", \"application/x-www-form-urlencoded\");\n        xmlhttp.send(req);\n    }\n    </script>\n    </head>\n    <body>\n    <div>\n    <textarea id=\"in\" style=\"height:400px;width:500px;font-size:18px\"></textarea>\n    <button onclick=\"loadXMLDoc()\" style=\"height:25px;width:80px;font-size:18px\">%s</button>\n    <textarea id=\"out\" style=\"height:400px;width:500px;font-size:18px\"></textarea>\n    </div>\n    <br>\n    <br>\n    This is a %s demo provided by Chunqi Wang (chqiwang@126.com).\n    </body>\n    </html>\n    ''' % (TASK.scope, TASK.scope, TASK.scope)\n    return page\n\n\nclass Tagger(object):\n    def __init__(self, sess, model_dir, scope, batch_size):\n        self.model = Model(scope, sess)\n        self.batch_size = batch_size\n        self.model.load_model(model_dir)\n\n    def tag(self, sentences):\n        sentences = self.preprocess(sentences)\n        sf = StringIO.StringIO()\n        sf.write(sentences)\n        sf.seek(0)\n        data = TASK.read_raw_file_all(sf)\n        output = self.model.tag_all(data, self.batch_size)\n        return TASK.create_output(*output)\n\n    def preprocess(self, sentences):\n        return sentences\n\n\nclass MainHandler(tornado.web.RequestHandler):\n    def get(self):\n        self.write(load_template())\n\n\nclass TaskHandler(tornado.web.RequestHandler):\n    def initialize(self, tagger):\n        self.tagger = tagger\n\n    def post(self):\n        sentences = self.get_argument('sentences')\n        segs = self.tagger.tag(sentences)\n        self.write('\\n'.join(segs))\n\n\ndef make_app(model_dir):\n    config = tf.ConfigProto()\n    config.gpu_options.allow_growth = True\n    config.gpu_options.per_process_gpu_memory_fraction = 1.0\n    config.allow_soft_placement = True\n    config.log_device_placement = True\n    sess = tf.Session(config=config)\n    tagger = Tagger(sess=sess, model_dir=model_dir, scope=TASK.scope, batch_size=200)\n    return tornado.web.Application([\n        (r\"/\", MainHandler),\n        (r\"/%s\" % TASK.scope, TaskHandler, {'tagger': tagger})\n    ])\n\n\nif __name__ == \"__main__\":\n    parser = ArgumentParser()\n    parser.add_argument('--task', dest='task')\n    parser.add_argument('--model_dir', dest='model_dir')\n    parser.add_argument('--port', dest='port', type=int, default=8888)\n    args = parser.parse_args()\n\n    TASK = __import__(args.task)\n\n    app = make_app(args.model_dir)\n    app.listen(args.port)\n    tornado.ioloop.IOLoop.current().start()\n"
  },
  {
    "path": "tagger.py",
    "content": "from __future__ import print_function\nimport tensorflow as tf\nimport tensorflow.contrib.layers as layers\nimport tensorflow.contrib.crf as crf\nimport time\nimport codecs\nimport os\nimport cPickle as pickle\nimport numpy as np\nfrom itertools import izip\n\nINT_TYPE = np.int32\nFLOAT_TYPE = np.float32\n\n\n################################################################################\n#                                 Model                                        #\n################################################################################\nclass Model(object):\n    def __init__(self, scope, sess):\n        self.scope = scope\n        self.sess = sess\n\n    def build_input_graph(self, vocab_size, emb_size, word_vocab_size, word_emb_size, word_window_size):\n        \"\"\"\n        Gather embeddings from lookup tables.\n        \"\"\"\n        seq_ids = tf.placeholder(dtype=INT_TYPE, shape=[None, None], name='seq_ids')\n        seq_word_ids = [tf.placeholder(dtype=INT_TYPE, shape=[None, None], name='seq_feature_%d_ids' % i)\n                        for i in range(word_window_size)]\n        embeddings = tf.get_variable('embeddings', [vocab_size, emb_size])\n        embedding_output = tf.nn.embedding_lookup([embeddings], seq_ids)\n        word_outputs = []\n        word_embeddings = tf.get_variable('word_embeddings', [word_vocab_size, word_emb_size])\n        for i in range(word_window_size):\n            word_outputs.append(tf.nn.embedding_lookup([word_embeddings], seq_word_ids[i]))\n\n        return seq_ids, seq_word_ids, tf.concat([embedding_output] + word_outputs, 2, 'inputs')\n\n    def build_tagging_graph(self, inputs, hidden_layers, channels, num_tags, use_crf, lamd, dropout_emb,\n                            dropout_hidden, kernel_size, use_bn, use_wn, active_type):\n        \"\"\"\n        Build a deep neural model for sequence tagging.\n        \"\"\"\n        stag_ids = tf.placeholder(dtype=INT_TYPE, shape=[None, None], name='stag_ids')\n        seq_lengths = tf.placeholder(dtype=INT_TYPE, shape=[None], name='seq_lengths')\n\n        # Default is not train.\n        is_train = tf.placeholder(dtype=tf.bool, shape=[], name='is_train')\n\n        masks = tf.cast(tf.sequence_mask(seq_lengths), FLOAT_TYPE)\n\n        # Dropout on embedding output.\n        if dropout_emb:\n            inputs = tf.cond(is_train,\n                             lambda: tf.nn.dropout(inputs, 1 - dropout_emb),\n                             lambda: inputs)\n\n        hidden_output = inputs\n        pre_channels = inputs.get_shape()[-1].value\n        for i in xrange(hidden_layers):\n\n            k = kernel_size\n            cur_channels = channels[i]\n            filter_w = tf.get_variable('filter_w_%d' % i, shape=[k, pre_channels, cur_channels], dtype=FLOAT_TYPE)\n            filter_v = tf.get_variable('filter_v_%d' % i, shape=[k, pre_channels, cur_channels], dtype=FLOAT_TYPE)\n            bias_b = tf.get_variable('bias_b_%d' % i, shape=[cur_channels],\n                                     initializer=tf.zeros_initializer(dtype=FLOAT_TYPE))\n            bias_c = tf.get_variable('bias_c_%d' % i, shape=[cur_channels],\n                                     initializer=tf.zeros_initializer(dtype=FLOAT_TYPE))\n\n            # Weight normalization.\n            if use_wn:\n                epsilon = 1e-12\n                g_w = tf.get_variable('g_w_%d' % i, shape=[k, 1, cur_channels], dtype=FLOAT_TYPE)\n                g_v = tf.get_variable('g_v_%d' % i, shape=[k, 1, cur_channels], dtype=FLOAT_TYPE)\n                # Perform wn\n                filter_w = g_w * filter_w / (tf.sqrt(tf.reduce_sum(filter_w ** 2, 1, keep_dims=True)) + epsilon)\n                filter_v = g_v * filter_v / (tf.sqrt(tf.reduce_sum(filter_v ** 2, 1, keep_dims=True)) + epsilon)\n\n            w = tf.nn.conv1d(hidden_output, filter_w, 1, 'SAME') + bias_b\n            v = tf.nn.conv1d(hidden_output, filter_v, 1, 'SAME') + bias_c\n\n            if use_bn:\n                w = layers.batch_norm(inputs=v, decay=0.9, is_training=is_train, center=True, scale=True,\n                                      scope='BatchNorm_w_%d' % i)\n                v = layers.batch_norm(inputs=w, decay=0.9, is_training=is_train, center=True, scale=True,\n                                      scope='BatchNorm_v_%d' % i)\n\n            if active_type == 'glu':\n                hidden_output = w * tf.nn.sigmoid(v)\n            elif active_type == 'relu':\n                hidden_output = tf.nn.relu(w)\n            elif active_type == 'gtu':\n                hidden_output = tf.tanh(w) * tf.nn.sigmoid(v)\n            elif active_type == 'tanh':\n                hidden_output = tf.tanh(w)\n            elif active_type == 'linear':\n                hidden_output = w\n            elif active_type == 'bilinear':\n                hidden_output = w * v\n            \n            # Mask paddings.\n            hidden_output = hidden_output * tf.expand_dims(masks, -1)\n            # Dropout on hidden output.\n            if dropout_hidden:\n                hidden_output = tf.cond(is_train,\n                                        lambda: tf.nn.dropout(hidden_output, 1 - dropout_hidden),\n                                        lambda: hidden_output\n                                        )\n\n            pre_channels = cur_channels\n\n        # Un-scaled log probabilities.\n        scores = layers.fully_connected(hidden_output, num_tags, tf.identity)\n\n        if use_crf:\n            cost, transitions = crf.crf_log_likelihood(inputs=scores, tag_indices=stag_ids,\n                                                       sequence_lengths=seq_lengths)\n            cost = - tf.reduce_mean(cost)\n        else:\n            reshaped_scores = tf.reshape(scores, [-1, num_tags])\n            reshaped_stag_ids = tf.reshape(stag_ids, [-1])\n            real_distribution = layers.one_hot_encoding(reshaped_stag_ids, num_tags)\n            cost = tf.nn.softmax_cross_entropy_with_logits(reshaped_scores, real_distribution)\n            cost = tf.reduce_sum(tf.reshape(cost, tf.shape(stag_ids)) * masks) / tf.cast(tf.shape(inputs)[0],\n                                                                                         FLOAT_TYPE)\n\n        # Calculate L2 penalty.\n        l2_penalty = 0\n        if lamd > 0:\n            for v in tf.trainable_variables():\n                if '/B:' not in v.name and '/biases:' not in v.name:\n                    l2_penalty += lamd * tf.nn.l2_loss(v)\n        train_cost = cost + l2_penalty\n\n        # Summary cost.\n        tf.summary.scalar('cost', cost)\n\n        summaries = tf.summary.merge_all()\n\n        update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n        if update_ops:\n            updates = tf.group(*update_ops)\n            with tf.control_dependencies([updates]):\n                cost = tf.identity(cost)\n\n        return stag_ids, seq_lengths, is_train, cost, train_cost, scores, summaries\n\n    def build_graph(self):\n        parameters = self.parameters\n        with tf.variable_scope(name_or_scope=self.scope, initializer=tf.uniform_unit_scaling_initializer()):\n            seq_ids_pl, seq_other_ids_pls, inputs = self.build_input_graph(vocab_size=parameters['vocab_size'],\n                                                                           emb_size=parameters['emb_size'],\n                                                                           word_window_size=parameters['word_window_size'],\n                                                                           word_vocab_size=parameters['word_vocab_size'],\n                                                                           word_emb_size=parameters['word_emb_size'])\n            stag_ids_pl, seq_lengths_pl, is_train_pl, cost_op, train_cost_op, scores_op, summary_op = \\\n                self.build_tagging_graph(inputs=inputs,\n                                         num_tags=parameters['num_tags'],\n                                         use_crf=parameters['use_crf'],\n                                         lamd=parameters['lamd'],\n                                         dropout_emb=parameters['dropout_emb'],\n                                         dropout_hidden=parameters['dropout_hidden'],\n                                         hidden_layers=parameters['hidden_layers'],\n                                         channels=parameters['channels'],\n                                         kernel_size=parameters['kernel_size'],\n                                         use_bn=parameters['use_bn'],\n                                         use_wn=parameters['use_wn'],\n                                         active_type=parameters['active_type'])\n        self.seq_ids_pl = seq_ids_pl\n        self.seq_other_ids_pls = seq_other_ids_pls\n        self.stag_ids_pl = stag_ids_pl\n        self.seq_lengths_pl = seq_lengths_pl\n        self.is_train_pl = is_train_pl\n        self.cost_op = cost_op\n        self.train_cost_op = train_cost_op\n        self.scores_op = scores_op\n        self.summary_op = summary_op\n\n    def inference(self, scores, sequence_lengths=None):\n        \"\"\"\n        Inference label sequence given scores.\n        If transitions is given, then perform veterbi search, else perform greedy search.\n\n        Args:\n            scores: A numpy array with shape (batch, max_length, num_tags).\n            sequence_lengths: A numpy array with shape (batch,).\n\n        Returns:\n            A numpy array with shape (batch, max_length).\n        \"\"\"\n\n        if not self.parameters['use_crf']:\n            return np.argmax(scores, 2)\n        else:\n            with tf.variable_scope(self.scope, reuse=True):\n                transitions = tf.get_variable('transitions').eval(session=self.sess)\n            paths = np.zeros(scores.shape[:2], dtype=INT_TYPE)\n            for i in xrange(scores.shape[0]):\n                tag_score, length = scores[i], sequence_lengths[i]\n                if length == 0:\n                    continue\n                path, _ = crf.viterbi_decode(tag_score[:length], transitions)\n                paths[i, :length] = path\n            return paths\n\n    def train(self, train_data, dev_data, test_data, model_dir, log_dir, emb_size, word_emb_size, optimizer,\n              hidden_layers, channels, kernel_size, active_type, use_bn, use_wn, use_crf, lamd, dropout_emb,\n              dropout_hidden, evaluator, batch_size, eval_batch_size, pre_trained_emb_path, fix_word_emb,\n              reserve_all_word_emb, pre_trained_word_emb_path, max_epoches, print_freq):\n        \"\"\"\n        This function is the main function for preparing data and training the model.\n        \"\"\"\n        assert len(channels) == hidden_layers\n\n        # Parse optimization method and parameters.\n        optimizer = optimizer.split('_')\n        optimizer_name = optimizer[0]\n        optimizer_options = [eval(i) for i in optimizer[1:]]\n        optimizer = {\n            'sgd': tf.train.GradientDescentOptimizer,\n            'adadelta': tf.train.AdadeltaOptimizer,\n            'adam': tf.train.AdamOptimizer,\n            'mom': tf.train.MomentumOptimizer\n        }[optimizer_name](*optimizer_options)\n\n        print('Preparing data...', end='')\n        if not os.path.exists(model_dir):\n            os.makedirs(model_dir)\n\n        mappings_path = os.path.join(model_dir, 'mappings.pkl')\n        parameters_path = os.path.join(model_dir, 'parameters.pkl')\n\n        # Load character embeddings.\n        pre_trained = {}\n        if pre_trained_emb_path and os.path.isfile(pre_trained_emb_path):\n            for l in codecs.open(pre_trained_emb_path, 'r', 'utf8'):\n                we = l.split()\n                if len(we) == emb_size + 1:\n                    w, e = we[0], np.array(map(float, we[1:]))\n                    pre_trained[w] = e\n\n        # Load word embeddings.\n        pre_trained_word = {}\n        if pre_trained_word_emb_path and os.path.isfile(pre_trained_word_emb_path):\n            for l in codecs.open(pre_trained_word_emb_path, 'r', 'utf8', 'ignore'):\n                we = l.split()\n                if len(we) == word_emb_size + 1:\n                    w, e = we[0], np.array(map(float, we[1:]))\n                    pre_trained_word[w] = e\n\n        # Load or create mappings.\n        if os.path.isfile(mappings_path):\n            item2id, id2item, tag2id, id2tag, word2id, id2word = pickle.load(open(mappings_path, 'r'))\n        else:\n            item2id, id2item = create_mapping(create_dic(train_data[0], add_unk=True, add_pad=True))\n            tag2id, id2tag = create_mapping(create_dic(train_data[-1]))\n\n            words = []\n            for t in train_data[1:-1]:\n                words.extend(t)\n            for t in dev_data[1:-1]:\n                words.extend(t)\n            for t in test_data[1:-1]:\n                words.extend(t)\n            word_dic = create_dic(words, add_unk=True, add_pad=True)\n            for k in word_dic.keys():\n                if k not in pre_trained_word and k != '<UNK>' and k != '<PAD>':\n                    word_dic.pop(k)\n            if reserve_all_word_emb:\n                for w in pre_trained_word:\n                    if w not in word_dic:\n                        word_dic[w] = 0\n            word2id, id2word = create_mapping(word_dic)\n            # Save the mappings to disk.\n            pickle.dump((item2id, id2item, tag2id, id2tag, word2id, id2word), open(mappings_path, 'w'))\n\n        # Hyper parameters.\n        word_window_size = len(train_data) - 2\n        parameters = {\n            'vocab_size': len(item2id),\n            'emb_size': emb_size,\n            'word_window_size': word_window_size,\n            'word_vocab_size': len(word2id),\n            'word_emb_size': word_emb_size,\n            'hidden_layers': hidden_layers,\n            'channels': channels,\n            'kernel_size': kernel_size,\n            'use_bn': use_bn,\n            'use_wn': use_wn,\n            'num_tags': len(tag2id),\n            'use_crf': use_crf,\n            'lamd': lamd,\n            'dropout_emb': dropout_emb,\n            'dropout_hidden': dropout_hidden,\n            'active_type': active_type\n        }\n\n        if os.path.isfile(parameters_path):\n            parameters_old = pickle.load(open(parameters_path, 'r'))\n            if parameters != parameters_old:\n                raise Exception('Network parameters are not consistent!')\n        else:\n            pickle.dump(parameters, open(parameters_path, 'w'))\n\n        self.item2id = item2id\n        self.id2item = id2item\n        self.tag2id = tag2id\n        self.id2tag = id2tag\n        self.word2id = word2id\n        self.id2word = id2word\n        self.parameters = parameters\n\n        # Convert data to corresponding ids.\n        train_data_ids = data_to_ids(\n            train_data, [item2id] + [word2id] * word_window_size + [tag2id]\n        )\n        print('Finished.')\n\n        print(\"Start building the network...\", end='')\n        self.build_graph()\n        print('Finished.')\n\n        def summary(name, dtype=FLOAT_TYPE):\n            value = tf.placeholder(dtype, shape=[])\n            return value, tf.summary.scalar(name, value)\n\n        dev_f1_pl, dev_summary_op = summary('dev f1')\n        test_f1_pl, test_summary_op = summary('test f1')\n\n        # Clip gradients and apply.\n        grads_and_vars = optimizer.compute_gradients(loss=self.train_cost_op, var_list=tf.trainable_variables())\n        grads_and_vars = [(g, v) for g, v in grads_and_vars if g is not None]\n\n        # If use fixed word embeddings, remove the grad\n        if fix_word_emb:\n            grads_and_vars = [(g, v) for g, v in grads_and_vars if '/word_embeddings' not in v.name]\n\n        grads_summary_op = tf.summary.histogram('grads', tf.concat([tf.reshape(g, [-1]) for g, _ in grads_and_vars], 0))\n        grads_norm = tf.sqrt(sum([tf.reduce_sum(tf.pow(g, 2)) for g, _ in grads_and_vars]))\n        grads_and_vars = [(g / (tf.reduce_max([grads_norm, 5]) / 5), v) for g, v in grads_and_vars]\n\n        train_op = optimizer.apply_gradients(grads_and_vars)\n\n        # Variables for recording training procedure.\n        best_epoch = tf.get_variable('best_epoch', shape=[], initializer=tf.zeros_initializer(), trainable=False,\n                                     dtype=INT_TYPE)\n        best_step = tf.get_variable('best_step', shape=[], initializer=tf.zeros_initializer(), trainable=False,\n                                    dtype=INT_TYPE)\n        best_dev_score = tf.get_variable('best_dev_score', shape=[], initializer=tf.zeros_initializer(),\n                                         trainable=False, dtype=FLOAT_TYPE)\n        best_test_score = tf.get_variable('best_test_score', shape=[], initializer=tf.zeros_initializer(),\n                                          trainable=False, dtype=FLOAT_TYPE)\n\n        init_op = tf.global_variables_initializer()\n        saver = tf.train.Saver(tf.global_variables())\n        summary_writer = tf.summary.FileWriter(log_dir + '/summaries')\n\n        print('Finished.')\n        print('Start training the network...')\n        self.sess.run(init_op)\n\n        start_time_begin = time.time()\n\n        try:\n            checkpoint = tf.train.latest_checkpoint(model_dir)\n            saver.restore(self.sess, checkpoint)\n            print('Restore model from %s.' % checkpoint)\n        except (tf.errors.DataLossError, TypeError, Exception):\n            # Failed to restore model from disk. Load pre-trained embeddings.\n            # Load character embeddings.\n            with tf.variable_scope(self.scope, reuse=True):\n                embeddings = tf.get_variable('embeddings')\n            value = self.sess.run(embeddings)\n            count = 0\n            for item in item2id:\n                item_id = item2id[item]\n                if item in pre_trained:\n                    value[item_id] = pre_trained[item]\n                    count += 1\n            # Run assign op.\n            self.sess.run(embeddings.assign(value))\n            del (pre_trained)\n            print('%d of %d character embeddings were loaded from pre-trained.' % (count, len(item2id)))\n\n            # Load word embeddings.\n            with tf.variable_scope(self.scope, reuse=True):\n                word_embeddings = tf.get_variable('word_embeddings')\n            value = self.sess.run(word_embeddings)\n            count = 0\n            for item in word2id:\n                item_id = word2id[item]\n                if item in pre_trained_word:\n                    value[item_id] = pre_trained_word[item]\n                    count += 1\n            # Run assign op.\n            self.sess.run(word_embeddings.assign(value))\n            del (pre_trained_word)\n            print('%d of %d word embeddings were loaded from pre-trained.' % (count, len(word2id)))\n\n        start_epoch, global_step, best_dev_f1 = self.sess.run((best_epoch, best_step, best_dev_score))\n\n        for epoch in range(start_epoch + 1, max_epoches + 1):\n            print('Starting epoch %d...' % epoch)\n            start_time = time.time()\n            loss_ep = 0\n            n_step = 0\n            iterator = data_iterator(train_data_ids, batch_size, shuffle=True)\n            for batch in iterator:\n                batch = create_input(batch)\n                seq_ids, seq_other_ids_list, stag_ids, seq_lengths = batch[0], batch[1: -2], batch[-2], batch[-1]\n                feed_dict = {self.seq_ids_pl: seq_ids.astype(INT_TYPE),\n                             self.stag_ids_pl: stag_ids.astype(INT_TYPE),\n                             self.seq_lengths_pl: seq_lengths.astype(INT_TYPE),\n                             self.is_train_pl: True}\n                assert len(self.seq_other_ids_pls) == len(seq_other_ids_list)\n                for pl, v in zip(self.seq_other_ids_pls, seq_other_ids_list):\n                    feed_dict[pl] = v\n                # feed_dict.update(drop_feed_dict)  # enable noise input\n                loss, summaries, grads_summaries, _ = self.sess.run(\n                    [self.cost_op, self.summary_op, grads_summary_op, train_op],\n                    feed_dict=feed_dict)\n                loss_ep += loss\n                n_step += 1\n                global_step += 1\n                summary_writer.add_summary(summaries, global_step)\n                summary_writer.add_summary(grads_summaries, global_step)\n\n                # Show training information.\n                if global_step % print_freq == 0:\n                    print('  Step %d, current cost %.6f, average cost %.6f' % (global_step, loss, loss_ep / n_step))\n            loss_ep = loss_ep / n_step\n            print('Epoch %d finished. Time: %ds Cost: %.6f' % (epoch, time.time() - start_time, loss_ep))\n\n            # Evaluate precision, recall and f1 with an external script.\n            dev_pre, dev_rec, dev_f1 = \\\n                evaluator((dev_data[0], dev_data[-1], self.tag_all(dev_data[:-1], eval_batch_size)[1]),\n                          log_dir + '/dev', epoch)\n            test_pre, test_rec, test_f1 = \\\n                evaluator((test_data[0], test_data[-1], self.tag_all(test_data[:-1], eval_batch_size)[1]),\n                          log_dir + '/test', epoch)\n\n            # Summary dev and test F1 score.\n            summary_writer.add_summary(self.sess.run(dev_summary_op, {dev_f1_pl: dev_f1}), epoch)\n            summary_writer.add_summary(self.sess.run(test_summary_op, {test_f1_pl: test_f1}), epoch)\n\n            print(\"Dev   precision / recall / f1 score: %.2f / %.2f / %.2f\" %\n                  (dev_pre * 100, dev_rec * 100, dev_f1 * 100))\n            print(\"Test  precision / recall / f1 score: %.2f / %.2f / %.2f\" %\n                  (test_pre * 100, test_rec * 100, test_f1 * 100))\n\n            if dev_f1 > best_dev_f1:\n                best_dev_f1 = dev_f1\n                self.sess.run((tf.assign(best_epoch, epoch),\n                               tf.assign(best_dev_score, dev_f1),\n                               tf.assign(best_test_score, test_f1),\n                               tf.assign(best_step, global_step)))\n\n                path = saver.save(self.sess, model_dir + '/model', epoch)\n                print('New best score on dev.')\n                print('Save model at %s.' % path)\n\n        print('Finished.')\n        print('Total training time: %fs.' % (time.time() - start_time_begin))\n\n    def load_model(self, model_dir):\n        mappings_path = os.path.join(model_dir, 'mappings.pkl')\n        parameters_path = os.path.join(model_dir, 'parameters.pkl')\n        item2id, id2item, tag2id, id2tag, word2id, id2word = \\\n            pickle.load(open(mappings_path, 'r'))\n        parameters = pickle.load(open(parameters_path))\n\n        self.item2id = item2id\n        self.id2item = id2item\n        self.tag2id = tag2id\n        self.id2tag = id2tag\n        self.word2id = word2id\n        self.id2word = id2word\n        self.parameters = parameters\n\n        print(parameters)\n        print('Building input graph...', end='')\n        self.build_graph()\n        print('Finished.')\n        print('Initializing variables...', end='')\n        init_op = tf.initialize_all_variables()\n        self.sess.run(init_op)\n        print('Finished.')\n        print('Reloading parameters...', end='')\n        saver = tf.train.Saver(tf.global_variables())\n        checkpoint = tf.train.latest_checkpoint(model_dir)\n        saver.restore(self.sess, checkpoint)\n        print('Finished.')\n\n    def tag(self, data_iter):\n        \"\"\"A tagging function.\n\n        Args:\n            data_iter: A iterator for generate batches.\n\n        Returns:\n            A generator for tagging result.\n        \"\"\"\n        output = []\n        for data in data_iter:\n            batch = data_to_ids(data, [self.item2id] + [self.word2id] * self.parameters['word_window_size'])\n            batch = create_input(batch)\n            seq_ids, seq_other_ids_list, seq_lengths = batch[0], batch[1: -1], batch[-1]\n            feed_dict = {self.seq_ids_pl: seq_ids.astype(INT_TYPE),\n                         self.seq_lengths_pl: seq_lengths.astype(INT_TYPE),\n                         self.is_train_pl: False}\n            for pl, v in zip(self.seq_other_ids_pls, seq_other_ids_list):\n                feed_dict[pl] = v.astype(INT_TYPE)\n            scores = self.sess.run(self.scores_op, feed_dict)\n            stag_ids = self.inference(scores, seq_lengths)\n            for seq, stag_id, length in izip(data[0], stag_ids, seq_lengths):\n                output.append((seq, [self.id2tag[t] for t in stag_id[:length]]))\n            yield zip(*output)\n            output = []\n\n    def tag_all(self, data, batch_size):\n        data_iter = data_iterator(data, batch_size=batch_size, shuffle=False)\n        output = []\n        for b in self.tag(data_iter):\n            output.extend(zip(*b))\n        return zip(*output)\n\n\n################################################################################\n#                                 DATA UTILS                                   #\n################################################################################\ndef create_dic(item_list, add_unk=False, add_pad=False):\n    \"\"\"\n    Create a dictionary of items from a list of list of items.\n    \"\"\"\n    assert type(item_list) in (list, tuple)\n    dic = {}\n    for items in item_list:\n        for item in items:\n            if item not in dic:\n                dic[item] = 1\n            else:\n                dic[item] += 1\n    # Make sure that <PAD> have a id 0.\n    if add_pad:\n        dic['<PAD>'] = 1e20\n    # If specified, add a special item <UNK>.\n    if add_unk:\n        dic['<UNK>'] = 1e10\n    return dic\n\n\ndef create_mapping(items):\n    \"\"\"\n    Create a mapping (item to ID / ID to item) from a dictionary.\n    Items are ordered by decreasing frequency.\n    \"\"\"\n    if type(items) is dict:\n        sorted_items = sorted(items.items(), key=lambda x: (-x[1], x[0]))\n        id2item = {i: v[0] for i, v in enumerate(sorted_items)}\n        item2id = {v: k for k, v in id2item.items()}\n        return item2id, id2item\n    elif type(items) is list:\n        id2item = {i: v for i, v in enumerate(items)}\n        item2id = {v: k for k, v in id2item.items()}\n        return item2id, id2item\n\n\ndef create_input(batch):\n    \"\"\"\n    Take each sentence data in batch and return an input for\n    the training or the evaluation function.\n    \"\"\"\n    assert len(batch) > 0\n    lengths = [len(seq) for seq in batch[0]]\n    max_len = max(2, max(lengths))\n    ret = []\n    for d in batch:\n        dd = []\n        for seq_id, pos in izip(d, lengths):\n            assert len(seq_id) == pos\n            pad = [0] * (max_len - pos)\n            dd.append(seq_id + pad)\n        ret.append(np.array(dd))\n    ret.append(np.array(lengths))\n    return ret\n\n\ndef data_to_ids(data, mappings):\n    \"\"\"\n    Map text data to ids.\n    \"\"\"\n\n    def strQ2B(ustring):\n        rstring = \"\"\n        for uchar in ustring:\n            inside_code = ord(uchar)\n            if inside_code == 12288:\n                inside_code = 32\n            elif 65281 <= inside_code <= 65374:\n                inside_code -= 65248\n            rstring += unichr(inside_code)\n        return rstring\n\n    def strB2Q(ustring):\n        rstring = \"\"\n        for uchar in ustring:\n            inside_code = ord(uchar)\n            if inside_code == 32:\n                inside_code = 12288\n            elif 32 <= inside_code <= 126:\n                inside_code += 65248\n            rstring += unichr(inside_code)\n        return rstring\n\n    def map(item, mapping):\n        if item in mapping:\n            return mapping[item]\n        item = strB2Q(item)\n        if item in mapping:\n            return mapping[item]\n        item = strQ2B(item)\n        if item in mapping:\n            return mapping[item]\n        return mapping['<UNK>']\n\n    def map_seq(seqs, mapping):\n        return [[map(item, mapping) for item in seq] for seq in seqs]\n\n    ret = []\n    for d, m in izip(data, mappings):\n        ret.append(map_seq(d, m))\n    return tuple(ret)\n\n\ndef data_iterator(inputs, batch_size, shuffle=True, max_length=200):\n    \"\"\"\n    A simple iterator for generating dynamic mini batches.\n    \"\"\"\n    assert len(inputs) > 0\n    assert all([len(item) == len(inputs[0]) for item in inputs])\n    inputs = zip(*inputs)\n    if shuffle:\n        np.random.shuffle(inputs)\n\n    batch = []\n    bs = batch_size\n    for d in inputs:\n        if len(d[0]) > max_length:\n            bs = max(1, min(batch_size * max_length / len(d[0]), bs))\n        if len(batch) < bs:\n            batch.append(d)\n        else:\n            yield zip(*batch)\n            batch = [d]\n            if len(d[0]) < max_length:\n                bs = batch_size\n            else:\n                bs = max(1, batch_size * max_length / len(d[0]))\n    if batch:\n        yield zip(*batch)\n"
  },
  {
    "path": "test.py",
    "content": "from __future__ import print_function\nimport codecs\nimport time\nfrom argparse import ArgumentParser\n\nimport tensorflow as tf\nfrom tagger import Model\n\n\nif __name__ == '__main__':\n\n    parser = ArgumentParser()\n    parser.add_argument('--task', dest='task')\n    parser.add_argument('--input', dest='input')\n    parser.add_argument('--output', dest='output')\n    parser.add_argument('--model_dir', dest='model_dir')\n    parser.add_argument('--batch_size', dest='batch_size', type=int)\n\n    args = parser.parse_args()\n    TASK = __import__(args.task)\n\n    data_iter = TASK.read_raw_file(codecs.open(args.input, 'r', 'utf8'), args.batch_size)\n\n    fout = codecs.open(args.output, 'w', 'utf8')\n\n    config = tf.ConfigProto()\n    config.gpu_options.allow_growth = True\n    config.gpu_options.per_process_gpu_memory_fraction = 1.0\n    config.allow_soft_placement = True\n    config.log_device_placement = True\n\n    with tf.Session(config=config) as sess:\n        model = Model(TASK.scope, sess)\n        model.load_model(args.model_dir)\n        start = time.time()\n        count = 0\n        for seqs, stags in model.tag(data_iter):\n            for l in TASK.create_output(seqs, stags):\n                count += 1\n                print(l, file=fout)\n            print('Tagged %d lines in %d seconds.' % (count, time.time() - start))\n        end = time.time()\n\n    fout.close()\n    print('Finished.')\n"
  },
  {
    "path": "train.py",
    "content": "#!coding=utf8\nfrom __future__ import print_function\nimport sys, codecs\nimport tensorflow as tf\nfrom argparse import ArgumentParser\n\nfrom tagger import Model\n\n\nclass FlushFile:\n    \"\"\"\n    A wrapper for File, allowing users see result immediately.\n    \"\"\"\n    def __init__(self, f):\n        self.f = f\n\n    def write(self, x):\n        self.f.write(x)\n        self.f.flush()\n\n\nif __name__ == '__main__':\n    sys.stdout = FlushFile(sys.stdout)\n\n    parser = ArgumentParser()\n    parser.add_argument('--task', dest='task')\n    parser.add_argument('--training_path', dest='training_path', default='data/datasets/sighan2005-pku/train.txt')\n    parser.add_argument('--dev_path', dest='dev_path', default='data/datasets/sighan2005-pku/dev.txt')\n    parser.add_argument('--test_path', dest='test_path', default='data/datasets/sighan2005-pku/test.txt')\n    parser.add_argument('--pre_trained_emb_path', dest='pre_trained_emb_path', default=None)\n    parser.add_argument('--pre_trained_word_emb_path', dest='pre_trained_word_emb_path', default=None)\n    parser.add_argument('--model_root', dest='model_root', default='model-pku')\n    parser.add_argument('--emb_size', dest='emb_size', type=int, default=200)\n    parser.add_argument('--word_window', dest='word_window', type=int, default=0)\n    parser.add_argument('--hidden_layers', dest='hidden_layers', type=int, default=5)\n    parser.add_argument('--channels', dest='channels', type=int, default=200)\n    parser.add_argument('--kernel_size', dest='kernel_size', type=int, default=3)\n    parser.add_argument('--word_emb_size', dest='word_emb_size', type=int, default=50)\n    parser.add_argument('--use_bn', dest='use_bn', type=int, default=0)\n    parser.add_argument('--use_wn', dest='use_wn', type=int, default=1)\n    parser.add_argument('--dropout_emb', dest='dropout_emb', type=float, default=0.2)\n    parser.add_argument('--dropout_hidden', dest='dropout_hidden', type=float, default=0.2)\n    parser.add_argument('--active_type', dest='active_type', default='glu')\n    parser.add_argument('--lamd', dest='lamd', type=float, default=0)\n    parser.add_argument('--fix_word_emb', dest='fix_word_emb', type=int, default=0)\n    parser.add_argument('--reserve_all_word_emb', dest='reserve_all_word_emb', type=int, default=0)\n    parser.add_argument('--use_crf', dest='use_crf', type=int, default=1)\n    parser.add_argument('--optimizer', dest='optimizer', default='adam_0.001')\n    parser.add_argument('--batch_size', dest='batch_size', type=int, default=100)\n    parser.add_argument('--eval_batch_size', dest='eval_batch_size', type=int, default=1000)\n    parser.add_argument('--max_epoches', dest='max_epoches', type=int, default=100)\n\n    args = parser.parse_args()\n    print(args)\n\n    TASK = __import__(args.task)\n\n    train_data, dev_data, test_data = (TASK.read_train_file(codecs.open(args.training_path, 'r', 'utf8'), word_window=args.word_window),\n                                       TASK.read_train_file(codecs.open(args.dev_path, 'r', 'utf8'), word_window=args.word_window),\n                                       TASK.read_train_file(codecs.open(args.test_path, 'r', 'utf8'), word_window=args.word_window))\n\n    sess = tf.Session()\n    model = Model(TASK.scope, sess)\n\n    model.train(train_data=train_data,\n                dev_data=dev_data,\n                test_data=test_data,\n                model_dir=args.model_root + '/models',\n                log_dir=args.model_root + '/logs',\n                emb_size=args.emb_size,\n                word_emb_size=args.word_emb_size,\n                hidden_layers=args.hidden_layers,\n                channels=[args.channels] * args.hidden_layers,\n                kernel_size=args.kernel_size,\n                use_bn=args.use_bn,\n                use_wn=args.use_wn,\n                active_type=args.active_type,\n                batch_size=args.batch_size,\n                use_crf=args.use_crf,\n                lamd=args.lamd,\n                dropout_emb=args.dropout_emb,\n                dropout_hidden=args.dropout_hidden,\n                optimizer=args.optimizer,\n                evaluator=TASK.evaluator,\n                eval_batch_size=args.eval_batch_size,\n                print_freq=50,\n                pre_trained_emb_path=args.pre_trained_emb_path,\n                pre_trained_word_emb_path=args.pre_trained_word_emb_path,\n                fix_word_emb=args.fix_word_emb,\n                reserve_all_word_emb=args.reserve_all_word_emb,\n                max_epoches=args.max_epoches)\n    sess.close()\n"
  },
  {
    "path": "train_cws.sh",
    "content": "#!/usr/bin/env bash\n\ncorpus=$1\n\nexport CUDA_VISIBLE_DEVICES=$2\n\nif [ \"${corpus}\" != \"pku\" ] && [ \"${corpus}\" != \"msr\" ];then\n    echo \"The first input must be pku or msr!\"\n    exit\nfi\n\nchmod +x score.perl\n\nmodel_root=model-${corpus}\n\nif [ ! -d ${model_root} ];then\n    mkdir ${model_root}\nfi\n\nnohup python train.py   --task cws \\\n                        --training_path data/datasets/sighan2005-${corpus}/train.txt \\\n                        --dev_path data/datasets/sighan2005-${corpus}/dev.txt \\\n                        --test_path data/datasets/sighan2005-${corpus}/test.txt \\\n                        --pre_trained_emb_path data/embeddings/news_tensite.w2v200 \\\n                        --model_root ${model_root} \\\n                        --word_window 0 \\\n                        >>${model_root}/stdout.txt 2>>${model_root}/stderr.txt &\n\necho \"Model and log are saved in ${model_root}.\""
  },
  {
    "path": "train_cws_wemb.sh",
    "content": "#!/usr/bin/env bash\n\ncorpus=$1\n\nexport CUDA_VISIBLE_DEVICES=$2\n\nif [ \"${corpus}\" != \"pku\" ] && [ \"${corpus}\" != \"msr\" ];then\n    echo \"The first input must be pku or msr!\"\n    exit\nfi\n\nchmod +x score.perl\n\nmodel_root=model-wemb-${corpus}\n\nif [ ! -d ${model_root} ];then\n    mkdir ${model_root}\nfi\n\nnohup python train.py   --task cws \\\n                        --training_path data/datasets/sighan2005-${corpus}/train.txt \\\n                        --dev_path data/datasets/sighan2005-${corpus}/dev.txt \\\n                        --test_path data/datasets/sighan2005-${corpus}/test.txt \\\n                        --pre_trained_emb_path data/embeddings/news_tensite.w2v200 \\\n                        --pre_trained_word_emb_path data/embeddings/news_tensite.${corpus}.words.w2v50 \\\n                        --model_root ${model_root} \\\n                        --word_window 4 \\\n                        >>${model_root}/stdout.txt 2>>${model_root}/stderr.txt &\n\necho \"Model and log are saved in ${model_root}.\""
  }
]