[
  {
    "path": "README.md",
    "content": "Insurance-QA deeplearning model\n======\nThis is a repo for Q&A Mathing, includes some deep learning models, such as CNN、RNN.<br>\n1. CNN. Basic CNN model from 《Applying Deep Learning To Answer Selection: A Study And An Open Task》<br>\n2. RNN. RNN seems the best model on Insurance-QA dataset.<br>\n3. SWEM. SWEM is the fastest, and has good effect on other datasets, such as WikiQA ..., but is seems not so good on Insurance-QA dataset. I think that, SWEM is more suitable for Q&Q matching, not Q&A matching.<br>\n\n\nIt's hard to say which model is the best in other datasets, you have to choose the most suitable model for you.<br><br>\nMore models are on the way, pay attention to the updates.<br>\n\n## Requirements\n1. tensorflow 1.4.0<br>\n2. python3.5<br>\n\n## Performance\nmargin loss version<br>\n\nModel/Score | Ins_qa_top1_precision | quora_best_prec\n------------ | ------------- | -------------\nCNN | 62% | None\nLSTM+CNN | 68% | None\nSWEM | <55% | None\n\nlogloss version<br>\n\nModel/Score | Insqa_top1_precision | quora_best_prec\n------------ | ------------- | -------------\nCNN | None | 79.60%\nLSTM+CNN | None | None\nSWEM | <40% | 82.69%\n\n## Running\nChange configuration to your own environment, just like data pathes<br>\n    \n    vim config.py\n\nData processing<br>\n   \n    python3 gen.py\n    \nRun CNN model<br>\n\n    cd ./cnn/tensorflow && python3 insqa_train.py\n    \nIt will take few hours(thousands of epoches) to train this model on a single GPU.<br>\n    \n## Downloads\n1. You can get Insurance-QA data from here https://github.com/shuzi/insuranceQA<br>\n2. You can get Quora data from here http://qim.ec.quoracdn.net/quora_duplicate_questions.tsv<br>\n\n## Links\n1. CNN and RNN textual classification repo  https://github.com/white127/TextClassification_CNN_RNN<br>\n2. 《Applying Deep Learning To Answer Selection: A Study And An Open Task》<br>\n\n"
  },
  {
    "path": "cnn/tensorflow/README.md",
    "content": "\n================result==================\n\n结果和theano版本的差不多，具体数值忘了\n\n虽然代码里写了dropout，但是实际并没有使用，dropout对结果影响不是特别大，不用dropout的话训练速度要快一些。\n\n================dataset================\n\n数据格式和theano版本的是一样的\n\ngithub上给出的是样本数据，如果需要全量的，也可直接联系我\ndataset is large, only test1 sample is given (see ./insuranceQA/test1.sample)\n\nI converted original idx_xx format to real-word format (see ./insuranceQA/train ./insuranceQA/test1.sample)\n\nyou can get the original dataset from https://github.com/shuzi/insuranceQA\n\nword embedding is trained by word2vec toolkit\n\n=================run=====================\n\n./insqa_train.py\n\n我使用的是python3.4，部分代码可能会和python2不兼容，如使用python2需要自己做一些小修改，核心的CNN代码应该\n不用改动的\n代码里的数据路径(类似'/export/...')是需要根据自己的环境修改的，指向自己的数据路径即可。核心的CNN代码无需改动\n"
  },
  {
    "path": "cnn/tensorflow/insqa_cnn.py",
    "content": "import tensorflow as tf\nimport numpy as np\n\n##########################################################################\n#  embedding_lookup + cnn + cosine margine ,  batch\n##########################################################################\nclass InsQACNN(object):\n    def __init__(self, _margin, sequence_length, batch_size,\n            vocab_size, embedding_size,\n            filter_sizes, num_filters, l2_reg_lambda=0.0):\n        self.L, self.B, self.V, self.E, self.FS, self.NF = sequence_length, batch_size, \\\n                vocab_size, embedding_size, filter_sizes, num_filters \n\n        #用户问题,字向量使用embedding_lookup\n        self.q = tf.placeholder(tf.int32, [self.B, self.L], name=\"q\")\n        #待匹配正向问题\n        self.qp = tf.placeholder(tf.int32, [self.B, self.L], name=\"qp\")\n        #负向问题\n        self.qn = tf.placeholder(tf.int32, [self.B, self.L], name=\"qn\")\n        self.dropout_keep_prob = tf.placeholder(tf.float32, name=\"dropout_keep_prob\")\n        l2_loss = tf.constant(0.0)\n\n        # Embedding layer\n        with tf.device('/cpu:0'), tf.name_scope(\"embedding\"):\n            W = tf.get_variable(\n                    initializer=tf.random_uniform([self.V, self.E], -1.0, 1.0), \n                    name='We')\n            self.qe = tf.nn.embedding_lookup(W, self.q)\n            self.qpe = tf.nn.embedding_lookup(W, self.qp)\n            self.qne = tf.nn.embedding_lookup(W, self.qn)\n        self.qe = tf.expand_dims(self.qe, -1)\n        self.qpe = tf.expand_dims(self.qpe, -1)\n        self.qne = tf.expand_dims(self.qne, -1)\n        \n        with tf.variable_scope('shared-conv') as scope:\n            self.qe = self.conv(self.qe)\n            scope.reuse_variables()\n            #tf.get_variable_scope().reuse_variables()\n            self.qpe = self.conv(self.qpe)\n            scope.reuse_variables()\n            #tf.get_variable_scope().reuse_variables()\n            self.qne = self.conv(self.qne)\n        self.cos_q_qp = self.cosine(self.qe, self.qpe)\n        self.cos_q_qn = self.cosine(self.qe, self.qne)\n        zero = tf.constant(0, shape=[self.B], dtype=tf.float32)\n        margin = tf.constant(_margin, shape=[self.B], dtype=tf.float32)\n        with tf.name_scope(\"loss\"):\n            self.losses = tf.maximum(zero, tf.subtract(margin, tf.subtract(self.cos_q_qp, self.cos_q_qn)))\n            self.loss = tf.reduce_sum(self.losses) + l2_reg_lambda * l2_loss\n            print('loss ', self.loss)\n\n        # Accuracy\n        with tf.name_scope(\"accuracy\"):\n            self.correct = tf.equal(zero, self.losses)\n            self.accuracy = tf.reduce_mean(tf.cast(self.correct, \"float\"), name=\"accuracy\")\n\n        for v in tf.trainable_variables():\n            print(v)\n\n    def conv(self, tensor):\n      pooled = []   \n      #with tf.variable_scope(name_or_scope='my-conv', reuse=tf.AUTO_REUSE):\n      with tf.variable_scope(\"my-conv-shared\"):\n          for i, fs in enumerate(self.FS):\n              filter_shape = [fs, self.E, 1, self.NF]\n              W = tf.get_variable(initializer=tf.truncated_normal(filter_shape, stddev=0.1), \n                      name=\"W-%s\" % str(fs))\n              b = tf.get_variable(initializer=tf.constant(0.1, shape=[self.NF]), \n                      name=\"b-%s\" % str(fs))\n              conv = tf.nn.conv2d(\n                      tensor, W, strides=[1, 1, 1, 1], padding='VALID',\n                      name=\"conv\")\n              h = tf.nn.relu(tf.nn.bias_add(conv, b), name=\"relu\")\n              output = tf.nn.max_pool(\n                      h, ksize=[1, self.L - fs + 1, 1, 1], strides=[1, 1, 1, 1], padding='VALID',\n                      name=\"pool\")\n              pooled.append(output)\n          num_filters_total = self.NF * len(self.FS)\n          pooled = tf.reshape(tf.concat(pooled, 3), [-1, num_filters_total])\n          pooled = tf.nn.dropout(pooled, self.dropout_keep_prob)\n          return pooled\n\n    def cosine(self, v1, v2):\n        l1 = tf.sqrt(tf.reduce_sum(tf.multiply(v1, v1), 1))\n        l2 = tf.sqrt(tf.reduce_sum(tf.multiply(v2, v2), 1))\n        a = tf.reduce_sum(tf.multiply(v1, v2), 1)\n        cos = tf.div(a, tf.multiply(l1, l2), name='score')\n        return tf.clip_by_value(cos, 1e-5, 0.99999)\n\n"
  },
  {
    "path": "cnn/tensorflow/insqa_cnn.py.old",
    "content": "import tensorflow as tf\nimport numpy as np\n\n##########################################################################\n#  embedding_lookup + cnn + cosine margine ,  batch\n##########################################################################\nclass InsQACNN1(object):\n    def __init__(\n      self, sequence_length, batch_size,\n      vocab_size, embedding_size,\n      filter_sizes, num_filters, l2_reg_lambda=0.0):\n\n        #用户问题,字向量使用embedding_lookup\n        self.input_x_1 = tf.placeholder(tf.int32, [batch_size, sequence_length], name=\"input_x_1\")\n        #待匹配正向问题\n        self.input_x_2 = tf.placeholder(tf.int32, [batch_size, sequence_length], name=\"input_x_2\")\n        #负向问题\n        self.input_x_3 = tf.placeholder(tf.int32, [batch_size, sequence_length], name=\"input_x_3\")\n        self.dropout_keep_prob = tf.placeholder(tf.float32, name=\"dropout_keep_prob\")\n        l2_loss = tf.constant(0.0)\n        print(\"input_x_1 \", self.input_x_1)\n\n        # Embedding layer\n        with tf.device('/cpu:0'), tf.name_scope(\"embedding\"):\n            W = tf.Variable(\n                tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),\n                name=\"W\")\n            chars_1 = tf.nn.embedding_lookup(W, self.input_x_1)\n            chars_2 = tf.nn.embedding_lookup(W, self.input_x_2)\n            chars_3 = tf.nn.embedding_lookup(W, self.input_x_3)\n            #self.embedded_chars_1 = tf.nn.dropout(chars_1, self.dropout_keep_prob)\n            #self.embedded_chars_2 = tf.nn.dropout(chars_2, self.dropout_keep_prob)\n            #self.embedded_chars_3 = tf.nn.dropout(chars_3, self.dropout_keep_prob)\n            self.embedded_chars_1 = chars_1\n            self.embedded_chars_2 = chars_2\n            self.embedded_chars_3 = chars_3\n        self.embedded_chars_expanded_1 = tf.expand_dims(self.embedded_chars_1, -1)\n        self.embedded_chars_expanded_2 = tf.expand_dims(self.embedded_chars_2, -1)\n        self.embedded_chars_expanded_3 = tf.expand_dims(self.embedded_chars_3, -1)\n\n        pooled_outputs_1 = []\n        pooled_outputs_2 = []\n        pooled_outputs_3 = []\n        for i, filter_size in enumerate(filter_sizes):\n            with tf.name_scope(\"conv-maxpool-%s\" % filter_size):\n                filter_shape = [filter_size, embedding_size, 1, num_filters]\n                W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name=\"W\")\n                b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name=\"b\")\n                conv = tf.nn.conv2d(\n                    self.embedded_chars_expanded_1,\n                    W,\n                    strides=[1, 1, 1, 1],\n                    padding='VALID',\n                    name=\"conv-1\"\n                )\n                h = tf.nn.relu(tf.nn.bias_add(conv, b), name=\"relu-1\")\n                pooled = tf.nn.max_pool(\n                    h,\n                    ksize=[1, sequence_length - filter_size + 1, 1, 1],\n                    strides=[1, 1, 1, 1],\n                    padding='VALID',\n                    name=\"poll-1\"\n                )\n                pooled_outputs_1.append(pooled)\n\n                conv = tf.nn.conv2d(\n                    self.embedded_chars_expanded_2,\n                    W,\n                    strides=[1, 1, 1, 1],\n                    padding='VALID',\n                    name=\"conv-2\"\n                )\n                h = tf.nn.relu(tf.nn.bias_add(conv, b), name=\"relu-2\")\n                pooled = tf.nn.max_pool(\n                    h,\n                    ksize=[1, sequence_length - filter_size + 1, 1, 1],\n                    strides=[1, 1, 1, 1],\n                    padding='VALID',\n                    name=\"poll-2\"\n                )\n                pooled_outputs_2.append(pooled)\n\n                conv = tf.nn.conv2d(\n                    self.embedded_chars_expanded_3,\n                    W,\n                    strides=[1, 1, 1, 1],\n                    padding='VALID',\n                    name=\"conv-3\"\n                )\n                h = tf.nn.relu(tf.nn.bias_add(conv, b), name=\"relu-3\")\n                pooled = tf.nn.max_pool(\n                    h,\n                    ksize=[1, sequence_length - filter_size + 1, 1, 1],\n                    strides=[1, 1, 1, 1],\n                    padding='VALID',\n                    name=\"poll-3\"\n                )\n                pooled_outputs_3.append(pooled)\n        num_filters_total = num_filters * len(filter_sizes)\n        pooled_reshape_1 = tf.reshape(tf.concat(pooled_outputs_1, 3), [-1, num_filters_total]) \n        pooled_reshape_2 = tf.reshape(tf.concat(pooled_outputs_2, 3), [-1, num_filters_total]) \n        pooled_reshape_3 = tf.reshape(tf.concat(pooled_outputs_3, 3), [-1, num_filters_total]) \n        #dropout\n        pooled_flat_1 = tf.nn.dropout(pooled_reshape_1, self.dropout_keep_prob)\n        pooled_flat_2 = tf.nn.dropout(pooled_reshape_2, self.dropout_keep_prob)\n        pooled_flat_3 = tf.nn.dropout(pooled_reshape_3, self.dropout_keep_prob)\n\n        pooled_len_1 = tf.sqrt(tf.reduce_sum(tf.multiply(pooled_flat_1, pooled_flat_1), 1)) #计算向量长度Batch模式\n        pooled_len_2 = tf.sqrt(tf.reduce_sum(tf.multiply(pooled_flat_2, pooled_flat_2), 1))\n        pooled_len_3 = tf.sqrt(tf.reduce_sum(tf.multiply(pooled_flat_3, pooled_flat_3), 1))\n        pooled_mul_12 = tf.reduce_sum(tf.multiply(pooled_flat_1, pooled_flat_2), 1) #计算向量的点乘Batch模式\n        pooled_mul_13 = tf.reduce_sum(tf.multiply(pooled_flat_1, pooled_flat_3), 1)\n\n        with tf.name_scope(\"output\"):\n            self.cos_12 = tf.div(pooled_mul_12, tf.multiply(pooled_len_1, pooled_len_2), name=\"scores\") #计算向量夹角Batch模式\n            self.cos_13 = tf.div(pooled_mul_13, tf.multiply(pooled_len_1, pooled_len_3))\n\n        zero = tf.constant(0, shape=[batch_size], dtype=tf.float32)\n        margin = tf.constant(0.05, shape=[batch_size], dtype=tf.float32)\n        with tf.name_scope(\"loss\"):\n            self.losses = tf.maximum(zero, tf.subtract(margin, tf.subtract(self.cos_12, self.cos_13)))\n            self.loss = tf.reduce_sum(self.losses) + l2_reg_lambda * l2_loss\n            print('loss ', self.loss)\n\n        # Accuracy\n        with tf.name_scope(\"accuracy\"):\n            self.correct = tf.equal(zero, self.losses)\n            self.accuracy = tf.reduce_mean(tf.cast(self.correct, \"float\"), name=\"accuracy\")\n        for v in tf.trainable_variables():\n            print(v)\n        exit(1)\n"
  },
  {
    "path": "cnn/tensorflow/insqa_train.py",
    "content": "#! /usr/bin/env python3.4\n\nimport tensorflow as tf\nimport numpy as np\nimport os, time, datetime, operator, sys\nfrom insqa_cnn import InsQACNN\nsys.path.append('../../')\nimport config, utils\n\nprint(tf.__version__)\n\n# Parameters\n# ==================================================\n\n# Model Hyperparameters\ntf.flags.DEFINE_float(\"margin\", 0.05, \"CNN model margin\")\ntf.flags.DEFINE_integer(\"sequence_length\", 200, \"Max sequence lehgth(default: 200)\")\ntf.flags.DEFINE_integer(\"embedding_dim\", 100, \"Dimensionality of character embedding (default: 128)\")\ntf.flags.DEFINE_string(\"filter_sizes\", \"1,2,3,5\", \"Comma-separated filter sizes (default: '3,4,5')\")\ntf.flags.DEFINE_integer(\"num_filters\", 256, \"Number of filters per filter size (default: 128)\")\ntf.flags.DEFINE_float(\"dropout_keep_prob\", 1.0, \"Dropout keep probability (default: 0.5)\")\ntf.flags.DEFINE_float(\"l2_reg_lambda\", 0, \"L2 regularizaion lambda (default: 0.0)\")\n\n# Training parameters\ntf.flags.DEFINE_integer(\"batch_size\", 256, \"Batch Size (default: 64)\")\ntf.flags.DEFINE_integer(\"num_epochs\", 5000000, \"Number of training epochs (default: 200)\")\ntf.flags.DEFINE_integer(\"evaluate_every\", 3000, \"Evaluate model on dev set after this many steps (default: 100)\")\ntf.flags.DEFINE_integer(\"checkpoint_every\", 3000, \"Save model after this many steps (default: 100)\")\n# Misc Parameters\ntf.flags.DEFINE_boolean(\"allow_soft_placement\", True, \"Allow device soft device placement\")\ntf.flags.DEFINE_boolean(\"log_device_placement\", False, \"Log placement of ops on devices\")\nFLAGS = tf.flags.FLAGS\nFLAGS._parse_flags()\nprint(\"\\nParameters:\")\nfor attr, value in sorted(FLAGS.__flags.items()):\n    print(\"{}={}\".format(attr.upper(), value))\nprint(\"\")\n\n# Data Preparatopn\n# ==================================================\n\n# Load data\nprint(\"Loading data...\")\nvocab, embeddings = utils.load_embeddings()\ntrain_data = utils.load_train_data(vocab, FLAGS.sequence_length)\ntest_data = utils.load_test_data(vocab, FLAGS.sequence_length)\nprint(\"Load done...\")\n\n# Training\n# ==================================================\n\nprev_auc = 0\nwith tf.Graph().as_default():\n  with tf.device(\"/gpu:1\"):\n    session_conf = tf.ConfigProto(\n      allow_soft_placement=FLAGS.allow_soft_placement,\n      log_device_placement=FLAGS.log_device_placement)\n    sess = tf.Session(config=session_conf)\n    with sess.as_default():\n        cnn = InsQACNN(\n            _margin=FLAGS.margin,\n            sequence_length=FLAGS.sequence_length,\n            batch_size=FLAGS.batch_size,\n            vocab_size=len(vocab),\n            embedding_size=FLAGS.embedding_dim,\n            filter_sizes=list(map(int, FLAGS.filter_sizes.split(\",\"))),\n            num_filters=FLAGS.num_filters,\n            l2_reg_lambda=FLAGS.l2_reg_lambda)\n\n        # Define Training procedure\n        global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n        optimizer = tf.train.AdamOptimizer(1e-1)\n        #optimizer = tf.train.GradientDescentOptimizer(1e-2)\n        grads_and_vars = optimizer.compute_gradients(cnn.loss)\n        train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)\n\n        # Keep track of gradient values and sparsity (optional)\n        grad_summaries = []\n        for g, v in grads_and_vars:\n            if g is not None:\n                grad_hist_summary = tf.summary.histogram(\"{}/grad/hist\".format(v.name), g)\n                sparsity_summary = tf.summary.scalar(\"{}/grad/sparsity\".format(v.name), tf.nn.zero_fraction(g))\n                grad_summaries.append(grad_hist_summary)\n                grad_summaries.append(sparsity_summary)\n        grad_summaries_merged = tf.summary.merge(grad_summaries)\n\n        # Output directory for models and summaries\n        timestamp = str(int(time.time()))\n        out_dir = os.path.abspath(os.path.join(os.path.curdir, \"runs\", timestamp))\n        print(\"Writing to {}\\n\".format(out_dir))\n\n        # Summaries for loss and accuracy\n        loss_summary = tf.summary.scalar(\"loss\", cnn.loss)\n        acc_summary = tf.summary.scalar(\"accuracy\", cnn.accuracy)\n\n        # Train Summaries\n        train_summary_op = tf.summary.merge([loss_summary, acc_summary, grad_summaries_merged])\n        train_summary_dir = os.path.join(out_dir, \"summaries\", \"train\")\n        train_summary_writer = tf.summary.FileWriter(train_summary_dir, sess.graph_def)\n\n        # Dev summaries\n        dev_summary_op = tf.summary.merge([loss_summary, acc_summary])\n        dev_summary_dir = os.path.join(out_dir, \"summaries\", \"dev\")\n        dev_summary_writer = tf.summary.FileWriter(dev_summary_dir, sess.graph_def)\n\n        # Checkpoint directory. Tensorflow assumes this directory already exists so we need to create it\n        checkpoint_dir = os.path.abspath(os.path.join(out_dir, \"checkpoints\"))\n        checkpoint_prefix = os.path.join(checkpoint_dir, \"model\")\n        if not os.path.exists(checkpoint_dir):\n            os.makedirs(checkpoint_dir)\n        saver = tf.train.Saver(tf.all_variables())\n\n        # Initialize all variables\n        sess.run(tf.initialize_all_variables())\n\n        def train_step(q, qp, qn):\n            feed_dict = {\n              cnn.q: q, cnn.qp: qp, cnn.qn: qn,\n              #cnn.input_x_1: q, cnn.input_x_2: qp, cnn.input_x_3: qn,\n              cnn.dropout_keep_prob: FLAGS.dropout_keep_prob\n            }\n            _, step, summaries, loss, accuracy, cos1, cos2 = sess.run(\n                [train_op, global_step, train_summary_op, cnn.loss, cnn.accuracy, cnn.cos_q_qp, cnn.cos_q_qn],\n                feed_dict)\n            #print(cos1)\n            #print(cos2)\n            time_str = datetime.datetime.now().isoformat()\n            print(\"{}: step {}, loss {:g}, acc {:g}\".format(time_str, step, loss, accuracy))\n            train_summary_writer.add_summary(summaries, step)\n\n        def test_step():\n          yp, y, group, of = [], [], [], open(config.predict1_file, 'w')\n          for i in range(0, len(test_data), FLAGS.batch_size):\n              f, g, q1, q2 = utils.gen_test_batch_qpn(test_data, i, i+FLAGS.batch_size)\n              feed_dict = {\n                cnn.q: q1, cnn.qp: q2, cnn.qn: q2,\n                #cnn.input_x_1: q1, cnn.input_x_2: q2, cnn.input_x_3: q2,\n                cnn.dropout_keep_prob: 1.0\n              }\n              cos = sess.run([cnn.cos_q_qp], feed_dict)\n              yp.extend(cos[0])\n              y.extend(f)\n              group.extend(g)\n          y, g, yp = y[:len(test_data)], group[:len(test_data)], yp[:len(test_data)]\n          auc = utils.eval_auc(y[:len(test_data)], g, yp[:len(test_data)])\n          top1_prec = utils._eval_top1_prec(y, g, yp)\n          for p in yp[:len(test_data)]: of.write(str(p) + '\\n')\n          of.write(str(top1_prec) + '\\n')\n          of.close()\n          return auc\n\n        # Generate batches\n        # Training loop. For each batch...\n        for i in range(FLAGS.num_epochs):\n            try:\n                q, qp, qn = utils.gen_train_batch_qpn(train_data, FLAGS.batch_size)\n                train_step(q, qp, qn)\n                current_step = tf.train.global_step(sess, global_step)\n                if current_step % FLAGS.evaluate_every == 0:\n                    auc = test_step()\n                    #if auc < prev_auc: break\n                    prev_auc = auc\n                if current_step % FLAGS.checkpoint_every == 0:\n                    path = saver.save(sess, checkpoint_prefix, global_step=current_step)\n                    print(\"Saved model checkpoint to {}\\n\".format(path))\n            except Exception as e:\n                print(e)\n"
  },
  {
    "path": "cnn/tensorflow/insqa_train.py.old",
    "content": "#! /usr/bin/env python3.4\n\nimport tensorflow as tf\nimport numpy as np\nimport os\nimport time\nimport datetime\nimport insurance_qa_data_helpers\nfrom insqa_cnn import InsQACNN1\nimport operator\n\n#print tf.__version__\n\n# Parameters\n# ==================================================\n\n# Model Hyperparameters\ntf.flags.DEFINE_integer(\"embedding_dim\", 100, \"Dimensionality of character embedding (default: 128)\")\ntf.flags.DEFINE_string(\"filter_sizes\", \"1,2,3,5\", \"Comma-separated filter sizes (default: '3,4,5')\")\ntf.flags.DEFINE_integer(\"num_filters\", 256, \"Number of filters per filter size (default: 128)\")\ntf.flags.DEFINE_float(\"dropout_keep_prob\", 1.0, \"Dropout keep probability (default: 0.5)\")\ntf.flags.DEFINE_float(\"l2_reg_lambda\", 0, \"L2 regularizaion lambda (default: 0.0)\")\n\n# Training parameters\ntf.flags.DEFINE_integer(\"batch_size\", 100, \"Batch Size (default: 64)\")\ntf.flags.DEFINE_integer(\"num_epochs\", 5000000, \"Number of training epochs (default: 200)\")\ntf.flags.DEFINE_integer(\"evaluate_every\", 5000, \"Evaluate model on dev set after this many steps (default: 100)\")\ntf.flags.DEFINE_integer(\"checkpoint_every\", 5000, \"Save model after this many steps (default: 100)\")\n# Misc Parameters\ntf.flags.DEFINE_boolean(\"allow_soft_placement\", True, \"Allow device soft device placement\")\ntf.flags.DEFINE_boolean(\"log_device_placement\", False, \"Log placement of ops on devices\")\n\nFLAGS = tf.flags.FLAGS\nFLAGS._parse_flags()\nprint(\"\\nParameters:\")\nfor attr, value in sorted(FLAGS.__flags.items()):\n    print(\"{}={}\".format(attr.upper(), value))\nprint(\"\")\n\n# Data Preparatopn\n# ==================================================\n\n# Load data\nprint(\"Loading data...\")\n\nvocab = insurance_qa_data_helpers.build_vocab()\nalist = insurance_qa_data_helpers.read_alist()\nraw = insurance_qa_data_helpers.read_raw()\nx_train_1, x_train_2, x_train_3 = insurance_qa_data_helpers.load_data_6(vocab, alist, raw, FLAGS.batch_size)\ntestList, vectors = insurance_qa_data_helpers.load_test_and_vectors()\nvectors = ''\nprint('x_train_1', np.shape(x_train_1))\nprint(\"Load done...\")\n\nval_file = '/export/jw/cnn/insuranceQA/test1'\nprecision = '/export/jw/cnn/insuranceQA/test1.acc'\n#x_val, y_val = data_deepqa.load_data_val()\n\n# Training\n# ==================================================\n\nwith tf.Graph().as_default():\n  with tf.device(\"/gpu:1\"):\n    session_conf = tf.ConfigProto(\n      allow_soft_placement=FLAGS.allow_soft_placement,\n      log_device_placement=FLAGS.log_device_placement)\n    sess = tf.Session(config=session_conf)\n    with sess.as_default():\n        cnn = InsQACNN1(\n            sequence_length=x_train_1.shape[1],\n            batch_size=FLAGS.batch_size,\n            vocab_size=len(vocab),\n            embedding_size=FLAGS.embedding_dim,\n            filter_sizes=list(map(int, FLAGS.filter_sizes.split(\",\"))),\n            num_filters=FLAGS.num_filters,\n            l2_reg_lambda=FLAGS.l2_reg_lambda)\n\n        # Define Training procedure\n        global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n        optimizer = tf.train.AdamOptimizer(1e-1)\n        #optimizer = tf.train.GradientDescentOptimizer(1e-2)\n        grads_and_vars = optimizer.compute_gradients(cnn.loss)\n        train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)\n\n        # Keep track of gradient values and sparsity (optional)\n        grad_summaries = []\n        for g, v in grads_and_vars:\n            if g is not None:\n                grad_hist_summary = tf.summary.histogram(\"{}/grad/hist\".format(v.name), g)\n                sparsity_summary = tf.summary.scalar(\"{}/grad/sparsity\".format(v.name), tf.nn.zero_fraction(g))\n                grad_summaries.append(grad_hist_summary)\n                grad_summaries.append(sparsity_summary)\n        grad_summaries_merged = tf.summary.merge(grad_summaries)\n\n        # Output directory for models and summaries\n        timestamp = str(int(time.time()))\n        out_dir = os.path.abspath(os.path.join(os.path.curdir, \"runs\", timestamp))\n        print(\"Writing to {}\\n\".format(out_dir))\n\n        # Summaries for loss and accuracy\n        loss_summary = tf.summary.scalar(\"loss\", cnn.loss)\n        acc_summary = tf.summary.scalar(\"accuracy\", cnn.accuracy)\n\n        # Train Summaries\n        train_summary_op = tf.summary.merge([loss_summary, acc_summary, grad_summaries_merged])\n        train_summary_dir = os.path.join(out_dir, \"summaries\", \"train\")\n        train_summary_writer = tf.summary.FileWriter(train_summary_dir, sess.graph_def)\n\n        # Dev summaries\n        dev_summary_op = tf.summary.merge([loss_summary, acc_summary])\n        dev_summary_dir = os.path.join(out_dir, \"summaries\", \"dev\")\n        dev_summary_writer = tf.summary.FileWriter(dev_summary_dir, sess.graph_def)\n\n        # Checkpoint directory. Tensorflow assumes this directory already exists so we need to create it\n        checkpoint_dir = os.path.abspath(os.path.join(out_dir, \"checkpoints\"))\n        checkpoint_prefix = os.path.join(checkpoint_dir, \"model\")\n        if not os.path.exists(checkpoint_dir):\n            os.makedirs(checkpoint_dir)\n        saver = tf.train.Saver(tf.all_variables())\n\n        # Initialize all variables\n        sess.run(tf.initialize_all_variables())\n\n        def train_step(x_batch_1, x_batch_2, x_batch_3):\n            \"\"\"\n            A single training step\n            \"\"\"\n            feed_dict = {\n              cnn.input_x_1: x_batch_1,\n              cnn.input_x_2: x_batch_2,\n              cnn.input_x_3: x_batch_3,\n              cnn.dropout_keep_prob: FLAGS.dropout_keep_prob\n            }\n            _, step, summaries, loss, accuracy = sess.run(\n                [train_op, global_step, train_summary_op, cnn.loss, cnn.accuracy],\n                feed_dict)\n            time_str = datetime.datetime.now().isoformat()\n            print(\"{}: step {}, loss {:g}, acc {:g}\".format(time_str, step, loss, accuracy))\n            train_summary_writer.add_summary(summaries, step)\n\n        def dev_step():\n          scoreList = []\n          i = int(0)\n          while True:\n              x_test_1, x_test_2, x_test_3 = insurance_qa_data_helpers.load_data_val_6(testList, vocab, i, FLAGS.batch_size)\n              feed_dict = {\n                cnn.input_x_1: x_test_1,\n                cnn.input_x_2: x_test_2,\n                cnn.input_x_3: x_test_3,\n                cnn.dropout_keep_prob: 1.0\n              }\n              batch_scores = sess.run([cnn.cos_12], feed_dict)\n              for score in batch_scores[0]:\n                  scoreList.append(score)\n              i += FLAGS.batch_size\n              if i >= len(testList):\n                  break\n          sessdict = {}\n          index = int(0)\n          for line in open(val_file):\n              items = line.strip().split(' ')\n              qid = items[1].split(':')[1]\n              if not qid in sessdict:\n                  sessdict[qid] = []\n              sessdict[qid].append((scoreList[index], items[0]))\n              index += 1\n              if index >= len(testList):\n                  break\n          lev1 = float(0)\n          lev0 = float(0)\n          of = open(precision, 'a')\n          for k, v in sessdict.items():\n              v.sort(key=operator.itemgetter(0), reverse=True)\n              score, flag = v[0]\n              if flag == '1':\n                  lev1 += 1\n              if flag == '0':\n                  lev0 += 1\n          of.write('lev1:' + str(lev1) + '\\n')\n          of.write('lev0:' + str(lev0) + '\\n')\n          print('lev1 ' + str(lev1))\n          print('lev0 ' + str(lev0))\n          of.close()\n\n        # Generate batches\n        # Training loop. For each batch...\n        for i in range(FLAGS.num_epochs):\n            try:\n                x_batch_1, x_batch_2, x_batch_3 = insurance_qa_data_helpers.load_data_6(vocab, alist, raw, FLAGS.batch_size)\n                train_step(x_batch_1, x_batch_2, x_batch_3)\n                current_step = tf.train.global_step(sess, global_step)\n                if current_step % FLAGS.evaluate_every == 0:\n                    print(\"\\nEvaluation:\")\n                    dev_step()\n                    print(\"\")\n                if current_step % FLAGS.checkpoint_every == 0:\n                    path = saver.save(sess, checkpoint_prefix, global_step=current_step)\n                    print(\"Saved model checkpoint to {}\\n\".format(path))\n            except Exception as e:\n                print(e)\n"
  },
  {
    "path": "cnn/tensorflow/insurance_qa_data_helpers.py",
    "content": "import numpy as np\nimport random\n\nempty_vector = []\nfor i in range(0, 100):\n    empty_vector.append(float(0.0))\nonevector = []\nfor i in range(0, 10):\n    onevector.append(float(1))\nzerovector = []\nfor i in range(0, 10):\n    zerovector.append(float(0))\n\ndef build_vocab():\n    code = int(0)\n    vocab = {}\n    vocab['UNKNOWN'] = code\n    code += 1\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        items = line.strip().split(' ')\n        for i in range(2, 4):\n            words = items[i].split('_')\n            for word in words:\n                if not word in vocab:\n                    vocab[word] = code\n                    code += 1\n    for line in open('/export/jw/cnn/insuranceQA/test1'):\n        items = line.strip().split(' ')\n        for i in range(2, 4):\n            words = items[i].split('_')\n            for word in words:\n                if not word in vocab:\n                    vocab[word] = code\n                    code += 1\n    return vocab\n\ndef rand_qa(qalist):\n    index = random.randint(0, len(qalist) - 1)\n    return qalist[index]\n\ndef read_alist():\n    alist = []\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        items = line.strip().split(' ')\n        alist.append(items[3])\n    print('read_alist done ......')\n    return alist\n\ndef vocab_plus_overlap(vectors, sent, over, size):\n    global onevector\n    global zerovector\n    oldict = {}\n    words = over.split('_')\n    if len(words) < size:\n        size = len(words)\n    for i in range(0, size):\n        if words[i] == '<a>':\n            continue\n        oldict[words[i]] = '#'\n    matrix = []\n    words = sent.split('_')\n    if len(words) < size:\n        size = len(words)\n    for i in range(0, size):\n        vec = read_vector(vectors, words[i])\n        newvec = vec.copy()\n        #if words[i] in oldict:\n        #    newvec += onevector\n        #else:\n        #    newvec += zerovector\n        matrix.append(newvec)\n    return matrix\n\ndef load_vectors():\n    vectors = {}\n    for line in open('/export/jw/cnn/insuranceQA/vectors.nobin'):\n        items = line.strip().split(' ')\n        if (len(items) < 101):\n            continue\n        vec = []\n        for i in range(1, 101):\n            vec.append(float(items[i]))\n        vectors[items[0]] = vec\n    return vectors\n\ndef read_vector(vectors, word):\n    global empty_vector\n    if word in vectors:\n        return vectors[word]\n    else:\n        return empty_vector\n        #return vectors['</s>']\n\ndef load_test_and_vectors():\n    testList = []\n    for line in open('/export/jw/cnn/insuranceQA/test1'):\n        testList.append(line.strip())\n    vectors = load_vectors()\n    return testList, vectors\n\ndef load_train_and_vectors():\n    trainList = []\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        trainList.append(line.strip())\n    vectors = load_vectors()\n    return trainList, vectors\n\ndef load_data_val_10(testList, vectors, index):\n    x_train_1 = []\n    x_train_2 = []\n    x_train_3 = []\n    items = testList[index].split(' ')\n    x_train_1.append(vocab_plus_overlap(vectors, items[2], items[3], 200))\n    x_train_2.append(vocab_plus_overlap(vectors, items[3], items[2], 200))\n    x_train_3.append(vocab_plus_overlap(vectors, items[3], items[2], 200))\n    return np.array(x_train_1), np.array(x_train_2), np.array(x_train_3)\n\ndef read_raw():\n    raw = []\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        items = line.strip().split(' ')\n        if items[0] == '1':\n            raw.append(items)\n    return raw\n\ndef encode_sent(vocab, string, size):\n    x = []\n    words = string.split('_')\n    for i in range(0, 200):\n        if words[i] in vocab:\n            x.append(vocab[words[i]])\n        else:\n            x.append(vocab['UNKNOWN'])\n    return x\n\ndef load_data_6(vocab, alist, raw, size):\n    x_train_1 = []\n    x_train_2 = []\n    x_train_3 = []\n    for i in range(0, size):\n        items = raw[random.randint(0, len(raw) - 1)]\n        nega = rand_qa(alist)\n        x_train_1.append(encode_sent(vocab, items[2], 100))\n        x_train_2.append(encode_sent(vocab, items[3], 100))\n        x_train_3.append(encode_sent(vocab, nega, 100))\n    return np.array(x_train_1), np.array(x_train_2), np.array(x_train_3)\n\ndef load_data_val_6(testList, vocab, index, batch):\n    x_train_1 = []\n    x_train_2 = []\n    x_train_3 = []\n    for i in range(0, batch):\n        true_index = index + i\n        if (true_index >= len(testList)):\n            true_index = len(testList) - 1\n        items = testList[true_index].split(' ')\n        x_train_1.append(encode_sent(vocab, items[2], 100))\n        x_train_2.append(encode_sent(vocab, items[3], 100))\n        x_train_3.append(encode_sent(vocab, items[3], 100))\n    return np.array(x_train_1), np.array(x_train_2), np.array(x_train_3)\n\ndef load_data_9(trainList, vectors, size):\n    x_train_1 = []\n    x_train_2 = []\n    y_train = []\n    for i in range(0, size):\n        pos = trainList[random.randint(0, len(trainList) - 1)]\n        posItems = pos.strip().split(' ')\n        x_train_1.append(vocab_plus_overlap(vectors, posItems[2], posItems[3], 200))\n        x_train_2.append(vocab_plus_overlap(vectors, posItems[3], posItems[2], 200))\n        y_train.append([1, 0])\n        neg = trainList[random.randint(0, len(trainList) - 1)]\n        negItems = neg.strip().split(' ')\n        x_train_1.append(vocab_plus_overlap(vectors, posItems[2], negItems[3], 200))\n        x_train_2.append(vocab_plus_overlap(vectors, negItems[3], posItems[2], 200))\n        y_train.append([0, 1])\n    return np.array(x_train_1), np.array(x_train_2), np.array(y_train)\n\ndef load_data_val_9(testList, vectors, index):\n    x_train_1 = []\n    x_train_2 = []\n    items = testList[index].split(' ')\n    x_train_1.append(vocab_plus_overlap(vectors, items[2], items[3], 200))\n    x_train_2.append(vocab_plus_overlap(vectors, items[3], items[2], 200))\n    return np.array(x_train_1), np.array(x_train_2)\n\ndef load_data_10(vectors, qalist, raw, size):\n    x_train_1 = []\n    x_train_2 = []\n    x_train_3 = []\n    items = raw[random.randint(0, len(raw) - 1)]\n    nega = rand_qa(qalist)\n    x_train_1.append(vocab_plus_overlap(vectors, items[2], items[3], 200))\n    x_train_2.append(vocab_plus_overlap(vectors, items[3], items[2], 200))\n    x_train_3.append(vocab_plus_overlap(vectors, nega, items[2], 200))\n    return np.array(x_train_1), np.array(x_train_2), np.array(x_train_3)\n\ndef load_data_11(vectors, qalist, raw, size):\n    x_train_1 = []\n    x_train_2 = []\n    x_train_3 = []\n    items = raw[random.randint(0, len(raw) - 1)]\n    nega = rand_qa(qalist)\n    x_train_1.append(vocab_plus_overlap(vectors, items[2], items[3], 200))\n    x_train_2.append(vocab_plus_overlap(vectors, items[3], items[2], 200))\n    x_train_3.append(vocab_plus_overlap(vectors, nega, items[2], 200))\n    return np.array(x_train_1), np.array(x_train_2), np.array(x_train_3)\n\ndef batch_iter(data, batch_size, num_epochs, shuffle=True):\n    data = np.array(data)\n    data_size = len(data)\n    num_batches_per_epoch = int(len(data)/batch_size) + 1\n    for epoch in range(num_epochs):\n        # Shuffle the data at each epoch\n        if shuffle:\n            shuffle_indices = np.random.permutation(np.arange(data_size))\n            shuffled_data = data[shuffle_indices]\n        else:\n            shuffled_data = data\n        for batch_num in range(num_batches_per_epoch):\n            start_index = batch_num * batch_size\n            end_index = min((batch_num + 1) * batch_size, data_size)\n            yield shuffled_data[start_index:end_index]\n    \n\n"
  },
  {
    "path": "cnn/tensorflow/test.py",
    "content": "import random\n\n_list = [i for i in range(0, 10)]\n_l1 = random.sample(_list, 2)\n_l2 = random.sample(_list, 2)\nprint(_l1)\nprint(_l2)\nfor i in range(2, 2):\n  print(i)\n\n"
  },
  {
    "path": "cnn/theano/README.md",
    "content": "\n================result==================\ntheano and tensorflow cnn code for insuranceQA\n\ntheano code, test1 top-1 precision : 61.5% (see ./insuranceQA/acc)\ntensorflow code, test1 top-1 precision : 62.6%\n\nthe best precision in the paper is 62.8% (see Applying Deep Leaarning To Answer Selection: A study and an open task)\n\n================dataset================\ndataset is large, only test1 sample is given (see ./insuranceQA/test1.sample)\n\nI converted original idx_xx format to real-word format (see ./insuranceQA/train ./insuranceQA/test1.sample)\n\nyou can get the original dataset from https://github.com/shuzi/insuranceQA\n\nword embedding is trained by word2vec toolkit\n\n=================run=====================\nreformat the original dataset(see my train and test1.sample)\nchange filepath to your dataset(see TODO in insqa_cnn.py)\npython insqa_cnn.py\n"
  },
  {
    "path": "cnn/theano/insqa_cnn.py",
    "content": "\n###########################################################\n# test1 top-1 precision: 62%\n###########################################################\n\nimport os, sys, timeit, random, operator\n\nimport numpy as np\n\nimport theano\nimport theano.tensor as T\nfrom theano.tensor.signal import pool\nfrom theano.tensor.nnet import conv2d\n\n#TODO change path to your dataset\ntrainfile = '/export/jw/cnn/insuranceQA/train'\ntest1file = '/export/jw/cnn/insuranceQA/test1'\nvectorsfile = '/export/jw/cnn/insuranceQA/vectors.nobin'\n\n###########################################################\n# read qa data\n###########################################################\ndef build_vocab():\n    global trainfile\n    code, vocab = int(0), {}\n    vocab['UNKNOWN'] = code\n    code += 1\n    for line in open(trainfile):\n        items = line.strip().split(' ')\n        for i in range(2, 3):\n            for word in items[i].split('_'):\n                if len(word) <= 0:\n                    continue\n                if not word in vocab:\n                    vocab[word] = code\n                    code += 1\n    return vocab\n\ndef load_vectors():\n    global vectorsfile\n    vectors = {}\n    for line in open(vectorsfile):\n        items = line.strip().split(' ')\n        if len(items[0]) <= 0:\n            continue\n        vec = []\n        for i in range(1, 101):\n            vec.append(float(items[i]))\n        vectors[items[0]] = vec\n    return vectors\n\ndef load_word_embeddings(vocab, dim):\n    vectors = load_vectors()\n    embeddings = [] #brute initialization\n    for i in range(0, len(vocab)):\n        vec = []\n        for j in range(0, dim):\n            vec.append(0.01)\n        embeddings.append(vec)\n    for word, code in vocab.items():\n        if word in vectors:\n            embeddings[code] = vectors[word]\n    return np.array(embeddings, dtype='float32')\n\n#be attention initialization of UNKNNOW\ndef encode_sent(vocab, string, size):\n    x = []\n    words = string.split('_')\n    for i in range(0, size):\n        if words[i] in vocab:\n            x.append(vocab[words[i]])\n        else:\n            x.append(vocab['UNKNOWN'])\n    return x\n\ndef load_train_list():\n    global trainfile\n    trainList = []\n    for line in open(trainfile):\n        trainList.append(line.strip().split(' '))\n    return trainList\n\ndef load_test_list():\n    global test1file\n    testList = []\n    for line in open(test1file):\n        testList.append(line.strip().split(' '))\n    return testList\n\ndef load_data(trainList, vocab, batch_size):\n    train_1, train_2, train_3 = [], [], []\n    for i in range(0, batch_size):\n        pos = trainList[random.randint(0, len(trainList)-1)]\n        neg = trainList[random.randint(0, len(trainList)-1)]\n        train_1.append(encode_sent(vocab, pos[2], 100))\n        train_2.append(encode_sent(vocab, pos[3], 100))\n        train_3.append(encode_sent(vocab, neg[3], 100))\n    return np.array(train_1, dtype='float32'), np.array(train_2, dtype='float32'), np.array(train_3, dtype='float32')\n\ndef load_data_val(testList, vocab, index, batch_size):\n    x1, x2, x3 = [], [], []\n    for i in range(0, batch_size):\n        true_index = index + i\n        if true_index >= len(testList):\n            true_index = len(testList) - 1\n        items = testList[true_index]\n        x1.append(encode_sent(vocab, items[2], 100))\n        x2.append(encode_sent(vocab, items[3], 100))\n        x3.append(encode_sent(vocab, items[3], 100))\n    return np.array(x1, dtype='float32'), np.array(x2, dtype='float32'), np.array(x3, dtype='float32')\n\ndef validation(validate_model, testList, vocab, batch_size):\n    index, score_list = int(0), []\n    while True:\n        x1, x2, x3 = load_data_val(testList, vocab, index, batch_size)\n        batch_scores, nouse = validate_model(x1, x2, x3, 1.0)\n        for score in batch_scores:\n            score_list.append(score)\n        index += batch_size\n        if index >= len(testList):\n            break\n        print 'Evaluation ' + str(index)\n    sdict, index = {}, int(0)\n    for items in testList:\n        qid = items[1].split(':')[1]\n        if not qid in sdict:\n            sdict[qid] = []\n        sdict[qid].append((score_list[index], items[0]))\n        index += 1\n    lev0, lev1 = float(0), float(0)\n    for qid, cases in sdict.items():\n        cases.sort(key=operator.itemgetter(0), reverse=True)\n        score, flag = cases[0]\n        if flag == '1':\n            lev1 += 1\n        if flag == '0':\n            lev0 += 1\n    print 'top-1 precition: ' + str(lev1 / (lev0 + lev1))\n\nclass QACnn(object):\n  def __init__(self, input1, input2, input3, word_embeddings, batch_size, sequence_len, embedding_size, filter_sizes, num_filters, keep_prob):\n    rng = np.random.RandomState(23455)\n    self.params = []\n\n    lookup_table = theano.shared(word_embeddings)\n    self.params += [lookup_table]\n    #input1-问题, input2-正向答案, input3-负向答案\n    #将每个字替换成字向量\n    input_matrix1 = lookup_table[T.cast(input1.flatten(), dtype=\"int32\")]\n    input_matrix2 = lookup_table[T.cast(input2.flatten(), dtype=\"int32\")]\n    input_matrix3 = lookup_table[T.cast(input3.flatten(), dtype=\"int32\")]\n\n    #CNN的输入是4维矩阵，这里只是增加了一个维度而已\n    input_x1 = input_matrix1.reshape((batch_size, 1, sequence_len, embedding_size))\n    input_x2 = input_matrix2.reshape((batch_size, 1, sequence_len, embedding_size))\n    input_x3 = input_matrix3.reshape((batch_size, 1, sequence_len, embedding_size))\n    #print(input_x1.shape.eval())\n    self.dbg_x1 = input_x1\n\n    outputs_1, outputs_2, outputs_3 = [], [], []\n    #设置多种大小的filter\n    for filter_size in filter_sizes:\n        #每种大小的filter的数量是num_filters\n        filter_shape = (num_filters, 1, filter_size, embedding_size)\n        image_shape = (batch_size, 1, sequence_len, embedding_size)\n        fan_in = np.prod(filter_shape[1:])\n        fan_out = filter_shape[0] * np.prod(filter_shape[2:])\n        W_bound = np.sqrt(6. / (fan_in + fan_out))\n        W = theano.shared(\n            np.asarray(\n                rng.uniform(low=-W_bound, high=W_bound, size=filter_shape),\n                dtype=theano.config.floatX\n            ),\n            borrow=True\n        )\n        b_values = np.zeros((filter_shape[0],), dtype=theano.config.floatX)\n        b = theano.shared(value=b_values, borrow=True)\n\n        #卷积+max_pooling\n        conv_out = conv2d(input=input_x1, filters=W, filter_shape=filter_shape, input_shape=image_shape)\n        #卷积后的向量的长度为ds\n        pooled_out = pool.pool_2d(input=conv_out, ds=(sequence_len - filter_size + 1, 1), ignore_border=True, mode='max')\n        pooled_active = T.tanh(pooled_out + b.dimshuffle('x', 0, 'x', 'x'))\n        outputs_1.append(pooled_active)\n\n        conv_out = conv2d(input=input_x2, filters=W, filter_shape=filter_shape, input_shape=image_shape)\n        pooled_out = pool.pool_2d(input=conv_out, ds=(sequence_len - filter_size + 1, 1), ignore_border=True, mode='max')\n        pooled_active = T.tanh(pooled_out + b.dimshuffle('x', 0, 'x', 'x'))\n        outputs_2.append(pooled_active)\n\n        conv_out = conv2d(input=input_x3, filters=W, filter_shape=filter_shape, input_shape=image_shape)\n        pooled_out = pool.pool_2d(input=conv_out, ds=(sequence_len - filter_size + 1, 1), ignore_border=True, mode='max')\n        pooled_active = T.tanh(pooled_out + b.dimshuffle('x', 0, 'x', 'x'))\n        outputs_3.append(pooled_active)\n\n        self.params += [W, b]\n        self.dbg_conv_out = conv_out.shape\n\n    num_filters_total = num_filters * len(filter_sizes)\n    self.dbg_outputs_1 = outputs_1[0].shape\n    #每一个句子的语义表示向量的长度为num_filters_total\n    output_flat1 = T.reshape(T.concatenate(outputs_1, axis=1), [batch_size, num_filters_total])\n    output_flat2 = T.reshape(T.concatenate(outputs_2, axis=1), [batch_size, num_filters_total])\n    output_flat3 = T.reshape(T.concatenate(outputs_3, axis=1), [batch_size, num_filters_total])\n    #dropout, keep_prob为1表示不进行dropout\n    output_drop1 = self._dropout(rng, output_flat1, keep_prob)\n    output_drop2 = self._dropout(rng, output_flat2, keep_prob)\n    output_drop3 = self._dropout(rng, output_flat3, keep_prob)\n\n    #计算问题和答案之前的向量夹角\n    #计算向量的长度\n    len1 = T.sqrt(T.sum(output_drop1 * output_drop1, axis=1))\n    len2 = T.sqrt(T.sum(output_drop2 * output_drop2, axis=1))\n    len3 = T.sqrt(T.sum(output_drop3 * output_drop3, axis=1))\n    #计算向量之间的夹角\n    cos12 = T.sum(output_drop1 * output_drop2, axis=1) / (len1 * len2)\n    self.cos12 = cos12\n    cos13 = T.sum(output_drop1 * output_drop3, axis=1) / (len1 * len3)\n    self.cos13 = cos13\n\n    zero = theano.shared(np.zeros(batch_size, dtype=theano.config.floatX), borrow=True)\n    margin = theano.shared(np.full(batch_size, 0.05, dtype=theano.config.floatX), borrow=True)\n    #Loss损失函数\n    diff = T.cast(T.maximum(zero, margin - cos12 + cos13), dtype=theano.config.floatX)\n    self.cost = T.sum(diff, acc_dtype=theano.config.floatX)\n    #mini-batch数据的准确率(如果正向答案和问题之间的cosine大于负向答案和问题的cosine，则认为正确，\n    #否则是错误的)\n    #Loss和Accuracy是用来评估训练中模型时候收敛的两个很重要的指标\n    self.accuracy = T.sum(T.cast(T.eq(zero, diff), dtype='int32')) / float(batch_size)\n\n  def _dropout(self, rng, layer, keep_prob):\n    srng = T.shared_randomstreams.RandomStreams(rng.randint(123456))\n    mask = srng.binomial(n=1, p=keep_prob, size=layer.shape)\n    output = layer * T.cast(mask, theano.config.floatX)\n    output = output / keep_prob\n    return output\n\ndef train():\n    batch_size = int(256)\n    filter_sizes = [2,3,5]\n    num_filters = 500\n    embedding_size = 100\n    learning_rate = 0.001\n    n_epochs = 2000000\n    validation_freq = 1000\n    keep_prob_value = 0.25\n\n    vocab = build_vocab()\n    word_embeddings = load_word_embeddings(vocab, embedding_size)\n    trainList = load_train_list()\n    testList = load_test_list()\n    train_x1, train_x2, train_x3 = load_data(trainList, vocab, batch_size)\n\n    x1, x2, x3 = T.matrix('x1'), T.matrix('x2'), T.matrix('x3')\n    keep_prob = T.fscalar('keep_prob')\n    model = QACnn(\n        input1=x1, input2=x2, input3=x3, keep_prob=keep_prob,\n        word_embeddings=word_embeddings, \n        batch_size=batch_size,\n        sequence_len=train_x1.shape[1],\n        embedding_size=embedding_size,\n        filter_sizes=filter_sizes,\n        num_filters=num_filters)\n    dbg_x1 = model.dbg_x1\n    dbg_outputs_1 = model.dbg_outputs_1\n\n    cost, cos12, cos13 = model.cost, model.cos12, model.cos13\n    print 'cost'\n    print cost\n    params, accuracy = model.params, model.accuracy\n    grads = T.grad(cost, params)\n\n    updates = [\n        (param_i, param_i - learning_rate * grad_i)\n        for param_i, grad_i in zip(params, grads)\n    ]\n\n    p1, p2, p3 = T.matrix('p1'), T.matrix('p2'), T.matrix('p3')\n    prob = T.fscalar('prob')\n    train_model = theano.function(\n        [p1, p2, p3, prob], \n        [cost, accuracy, dbg_x1, dbg_outputs_1], \n        updates=updates,\n        givens={\n            x1: p1, x2: p2, x3: p3, keep_prob: prob\n        }\n    )\n\n    v1, v2, v3 = T.matrix('v1'), T.matrix('v2'), T.matrix('v3')\n    validate_model = theano.function(\n        inputs=[v1, v2, v3, prob],\n        outputs=[cos12, cos13],\n        #updates=updates,\n        givens={\n            x1: v1, x2: v2, x3: v3, keep_prob: prob\n        }\n    )\n\n    epoch = 0\n    done_looping = False\n    while (epoch < n_epochs) and (not done_looping):\n        epoch = epoch + 1\n        train_x1, train_x2, train_x3 = load_data(trainList, vocab, batch_size)\n        #print train_x3.shape\n        cost_ij, acc, dbg_x1, dbg_outputs_1 = train_model(train_x1, train_x2, train_x3, keep_prob_value)\n        print 'load data done ...... epoch:' + str(epoch) + ' cost:' + str(cost_ij) + ', acc:' + str(acc)\n        if epoch % validation_freq == 0:\n            print 'Evaluation ......'\n            validation(validate_model, testList, vocab, batch_size)\n        #print dbg_outputs_1\n\nif __name__ == '__main__':\n    train()\n"
  },
  {
    "path": "config.py",
    "content": "import os\n\ndataset_ins = 'insurance-qa'\ndataset_qur = 'quora-qa'\n\n##################################################################\n# ajust to your runnning environment\n# which data do you want\ndataset = dataset_qur\n# word2vec command path\nw2v_command = '/export/jw/word2vec/word2vec'\n##################################################################\n\nhome = ''\nif dataset == dataset_ins:\n  home = os.path.expanduser('/export/jw/insuranceQA')\nelif dataset == dataset_qur:\n  home = os.path.expanduser('/export/jw/quora')\n\n#Insurance-QA original data directory\nqa_version = 'V1'\nvocab_file = os.path.join(home, qa_version, 'vocabulary')\nanswers_file = os.path.join(home, qa_version, 'answers.label.token_idx')\nquestion_train_file = os.path.join(home, qa_version, 'question.train.token_idx.label')\nquestion_test1_file = os.path.join(home, qa_version, 'question.test1.label.token_idx.pool')\nquestion_test2_file = os.path.join(home, qa_version, 'question.test2.label.token_idx.pool')\nquestion_dev_file = os.path.join(home, qa_version, 'question.dev.label.token_idx.pool')\n#quora original data directory\nqr_file = os.path.join(home, 'quora_duplicate_questions.tsv')\nqr_train_ratio = 0.8\n#processed files\ntrain_file = os.path.join(home, 'data', 'train.prepro')\ntest1_file = os.path.join(home, 'data', 'test1.prepro')\ntest2_file = os.path.join(home, 'data', 'test2.prepro')\nw2v_train_file = os.path.join(home, 'data', 'w2v.train')\nw2v_bin_file = os.path.join(home, 'data', 'w2v.bin')\npredict1_file = os.path.join(home, 'data', 'predict1')\n  \n"
  },
  {
    "path": "gen.py",
    "content": "import config, os, random\n\n#####################################################################\n# function: load vocab\n# return: dict[word] = [word_id]\n#####################################################################\ndef load_vocab():\n  voc = {}\n  for line in open(config.vocab_file):\n    word, _id = line.strip().split('\\t')\n    voc[word] = _id\n  return voc\n\n#####################################################################\n# function: load answers, restore idx to real word\n# return : [answer_1, answer_2, ..., answer_n]\n#####################################################################\ndef ins_load_answers():\n  _list, voc = ['<None>'], load_vocab()\n  for line in open(config.answers_file):\n    _, sent = line.strip().split('\\t')\n    _list.append('_'.join([voc[wid] for wid in sent.split(' ')]))\n  return _list\n\n#####################################################################\n# function: preprea word2vec binary file\n# return : \n#####################################################################\ndef ins_w2v():\n  print('preparing word2vec ......')\n  _data, voc = [], load_vocab()\n  for line in open(config.question_train_file):\n    items = line.strip().split('\\t')\n    _data.append(' '.join([voc[_id] for _id in items[0].split(' ')]))\n  for _file in [config.answers_file, config.question_dev_file, \\\n          config.question_test1_file, config.question_test2_file]:\n    for line in open(_file):\n      items = line.strip().split('\\t')\n      _data.append(' '.join([voc[_id] for _id in items[1].split(' ')]))\n  of = open(config.w2v_train_file, 'w')\n  for s in _data: of.write(s + '\\n')\n  of.close()\n  os.system('time ' + config.w2c_command + ' -train ' + config.w2v_train_file + ' -output ' + config.w2v_bin_file + ' -cbow 0 -size 100 -window 5 -negative 20 -sample 1e-3 -threads 12 -binary 0 -min-count 1')\n\n#####################################################################\n# function: preprea train file\n# file format: flag question answer\n#####################################################################\ndef ins_train():\n  print('preparing train ......')\n  answers, voc, _data = ins_load_answers(), load_vocab(), []\n  for line in open(config.question_train_file):\n    qsent, ids = line.strip().split('\\t')\n    qsent = '_'.join([voc[wid] for wid in qsent.split(' ')])\n    for _id in ids.split(' '):\n      _data.append(' '.join(['1', qsent, answers[int(_id)]]))\n  of = open(config.train_file, 'w')\n  for _s in _data: of.write(_s + '\\n')\n  of.close()\n\n#####################################################################\n# function: preprea test file\n# file format: flag group_id question answer\n#####################################################################\ndef ins_test():\n  print('preparing test ......')\n  answers, voc = ins_load_answers(), load_vocab()\n  for _in, _out in ([(config.question_test2_file, config.test2_file), \\\n          (config.question_test1_file, config.test1_file)]):\n    _data, group = [], int(0)\n    for line in open(_in):\n      pids, qsent, pnids = line.strip().split('\\t')\n      positive = {_id:'#' for _id in pids.split(' ')}\n      qsent = '_'.join([voc[wid] for wid in qsent.split(' ')])\n      for _id in pnids.split(' '):\n        flag = '1' if _id in positive else '0'\n        _data.append(' '.join([flag, str(group), qsent, answers[int(_id)]]))\n      group += 1\n    of = open(_out, 'w')\n    for s in _data: of.write(s + '\\n')\n    of.close()\n\ndef ins_qa():\n  ins_w2v()\n  ins_train()\n  ins_test()\n\ndef qur_prepare():\n  #pretrain word2vec\n  _list = []\n  for line in open(config.qr_file):\n    items = line.strip().split('\\t')\n    if len(items) != 6:\n      continue\n    _list.append(items)\n  _list = _list[1:]\n  random.shuffle(_list)\n  _list = [(f, q1, q2) for _,_,_,q1,q2,f in _list]\n  of = open(config.w2v_train_file, 'w')\n  for f, q1, q2 in _list:\n    of.write(q1 + '\\n')\n    of.write(q2 + '\\n')\n  of.close()\n  os.system('time ' + config.w2v_command + ' -train ' + config.w2v_train_file + ' -output ' + config.w2v_bin_file + ' -cbow 0 -size 100 -window 5 -negative 20 -sample 1e-3 -threads 12 -binary 0 -min-count 1')\n  #train file\n  _newlist = []\n  for f, q1, q2 in _list:\n    if len(q1) <= 1 or len(q2) <= 1: continue\n    q1 = '_'.join(q1.split(' '))\n    q2 = '_'.join(q2.split(' '))\n    _newlist.append((f, q1, q2))\n  _list = _newlist\n  of = open(config.train_file, 'w')\n  for f, q1, q2 in _list[:int(len(_list) * 0.8)]:\n    of.write(' '.join([f, q1, q2]) + '\\n')\n  of.close()\n\n  #test file\n  of = open(config.test1_file, 'w')\n  for f, q1, q2 in _list[int(len(_list) * 0.8):]:\n    of.write(' '.join([f, q1, q2]) + '\\n')\n  of.close()\n\ndef qur_qa():\n  qur_prepare()\n\nif __name__ == '__main__':\n  if config.dataset == config.dataset_ins:\n    ins_qa()\n  elif config.dataset == config.dataset_qur:\n    qur_qa()\n"
  },
  {
    "path": "lstm_cnn/theano/README.md",
    "content": "\ntheano lstm+cnn code for insuranceQA\n\n================result==================\n\ntheano code, test1 top-1 precision : 68.3% \n\nlstm+cnn is better than cnn(61.5%). \n\n================dataset================\n\ndataset is large, only test1 sample is given (see ./insuranceQA/test1.sample)\n\nI converted original idx_xx format to real-word format (see ./insuranceQA/train ./insuranceQA/test1.sample)\n\nyou can get the original dataset from https://github.com/shuzi/insuranceQA\n\nword embedding is trained by word2vec toolkit\n\n=================run=====================\n\nreformat the original dataset(see my train and test1.sample)\n\nchange filepath to your dataset(see TODO in insqa_cnn.py)\n\npython insqa_lstm.py\n"
  },
  {
    "path": "lstm_cnn/theano/insqa_lstm.py",
    "content": "\n############################################################\n# if batch_size is 1, there must be a dtype error when doing \n#   T.grad, this is something about scan func\n#   see https://github.com/Theano/Theano/issues/1772\n#\n# LSTM + cnn\n# test1 top-1 precision: 68.3%\n############################################################\n\nfrom collections import OrderedDict\nimport sys, time, random, operator\n\nimport numpy as np\nimport theano\nfrom theano import config\nimport theano.tensor as T\nfrom theano.tensor.signal import pool\nfrom theano.tensor.nnet import conv2d\nfrom theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams\n\n#TODO change filepath to your local environment\n#include train test1 vectors.nobin\n\ndef build_vocab():\n    code, vocab = int(0), {}\n    vocab['UNKNOWN'] = code\n    code += 1\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        items = line.strip().split(' ')\n        for i in range(2, 3):\n            for word in items[i].split('_'):\n                if len(word) <= 0:\n                    continue\n                if not word in vocab:\n                    vocab[word] = code\n                    code += 1\n    return vocab\n\ndef load_vectors():\n    vectors = {}\n    for line in open('/export/jw/cnn/insuranceQA/vectors.nobin'):\n        items = line.strip().split(' ')\n        if len(items[0]) <= 0:\n            continue\n        vec = []\n        for i in range(1, 101):\n            vec.append(float(items[i]))\n        vectors[items[0]] = vec\n    return vectors\n\ndef load_word_embeddings(vocab, dim):\n    vectors = load_vectors()\n    embeddings = [] #brute initialization\n    for i in range(0, len(vocab)):\n        vec = []\n        for j in range(0, dim):\n            vec.append(0.01)\n        embeddings.append(vec)\n    for word, code in vocab.items():\n        if word in vectors:\n            embeddings[code] = vectors[word]\n    return np.array(embeddings, dtype='float32')\n\n#be attention initialization of UNKNNOW\ndef encode_sent(vocab, string, size):\n    x, m = [], []\n    words = string.split('_')\n    for i in range(0, size):\n        if words[i] in vocab:\n            x.append(vocab[words[i]])\n        else:\n            x.append(vocab['UNKNOWN'])\n        if words[i] == '<a>': #TODO\n            m.append(1) #fixed sequence length, else use 0\n        else:\n            m.append(1)\n    return x, m\n\ndef load_train_list():\n    trainList = []\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        items = line.strip().split(' ')\n        if items[0] == '1':\n            trainList.append(line.strip().split(' '))\n    return trainList\n\ndef load_test_list():\n    testList = []\n    for line in open('/export/jw/cnn/insuranceQA/test1'):\n        testList.append(line.strip().split(' '))\n    return testList\n\ndef load_data(trainList, vocab, batch_size):\n    train_1, train_2, train_3 = [], [], []\n    mask_1, mask_2, mask_3 = [], [], []\n    counter = 0\n    while True:\n        pos = trainList[random.randint(0, len(trainList)-1)]\n        neg = trainList[random.randint(0, len(trainList)-1)]\n        if pos[2].startswith('<a>') or pos[3].startswith('<a>') or neg[3].startswith('<a>'):\n            #print 'empty string ......'\n            continue\n        x, m = encode_sent(vocab, pos[2], 100)\n        train_1.append(x)\n        mask_1.append(m)\n        x, m = encode_sent(vocab, pos[3], 100)\n        train_2.append(x)\n        mask_2.append(m)\n        x, m = encode_sent(vocab, neg[3], 100)\n        train_3.append(x)\n        mask_3.append(m)\n        counter += 1\n        if counter >= batch_size:\n            break\n    return np.transpose(np.array(train_1, dtype=config.floatX)), np.transpose(np.array(train_2, dtype=config.floatX)), np.transpose(np.array(train_3, dtype=config.floatX)), np.transpose(np.array(mask_1, dtype=config.floatX)) , np.transpose(np.array(mask_2, dtype=config.floatX)), np.transpose(np.array(mask_3, dtype=config.floatX))\n\ndef load_data_val(testList, vocab, index, batch_size):\n    x1, x2, x3, m1, m2, m3 = [], [], [], [], [], []\n    for i in range(0, batch_size):\n        true_index = index + i\n        if true_index >= len(testList):\n            true_index = len(testList) - 1\n        items = testList[true_index]\n        x, m = encode_sent(vocab, items[2], 100)\n        x1.append(x)\n        m1.append(m)\n        x, m = encode_sent(vocab, items[3], 100)\n        x2.append(x)\n        m2.append(m)\n        x, m = encode_sent(vocab, items[3], 100)\n        x3.append(x)\n        m3.append(m)\n    return np.transpose(np.array(x1, dtype=config.floatX)), np.transpose(np.array(x2, dtype=config.floatX)), np.transpose(np.array(x3, dtype=config.floatX)), np.transpose(np.array(m1, dtype=config.floatX)) , np.transpose(np.array(m2, dtype=config.floatX)), np.transpose(np.array(m3, dtype=config.floatX))\n\ndef validation(validate_model, testList, vocab, batch_size):\n    index, score_list = int(0), []\n    while True:\n        x1, x2, x3, m1, m2, m3 = load_data_val(testList, vocab, index, batch_size)\n        batch_scores, nouse = validate_model(x1, x2, x3, m1, m2, m3)\n        for score in batch_scores:\n            score_list.append(score)\n        index += batch_size\n        if index >= len(testList):\n            break\n        print 'Evaluation ' + str(index)\n    sdict, index = {}, int(0)\n    for items in testList:\n        qid = items[1].split(':')[1]\n        if not qid in sdict:\n            sdict[qid] = []\n        sdict[qid].append((score_list[index], items[0]))\n        index += 1\n    lev0, lev1 = float(0), float(0)\n    of = open('/export/jw/cnn/insuranceQA/acc.lstm', 'a')\n    for qid, cases in sdict.items():\n        cases.sort(key=operator.itemgetter(0), reverse=True)\n        score, flag = cases[0]\n        if flag == '1':\n            lev1 += 1\n        if flag == '0':\n            lev0 += 1\n    for s in score_list:\n        of.write(str(s) + '\\n')\n    of.write('lev1:' + str(lev1) + '\\n')\n    of.write('lev0:' + str(lev0) + '\\n')\n    print 'lev1:' + str(lev1)\n    print 'lev0:' + str(lev0)\n    of.close()\n\ndef ortho_weight(ndim):\n    W = np.random.randn(ndim, ndim)\n    u, s, v = np.linalg.svd(W)\n    return u.astype(config.floatX)\n\ndef numpy_floatX(data):\n    return np.asarray(data, dtype=config.floatX)\n\ndef param_init_cnn(filter_sizes, num_filters, proj_size, tparams, grad_params):\n    rng = np.random.RandomState(23455)\n    for filter_size in filter_sizes:\n        filter_shape = (num_filters, 1, filter_size, proj_size)\n        fan_in = np.prod(filter_shape[1:])\n        fan_out = filter_shape[0] * np.prod(filter_shape[2:])\n        W_bound = np.sqrt(6. / (fan_in + fan_out))\n        W = theano.shared(\n            np.asarray(\n                rng.uniform(low=-W_bound, high=W_bound, size=filter_shape),\n                dtype=theano.config.floatX\n            ),\n            borrow=True\n        )\n        tparams['cnn_W_' + str(filter_size)] = W\n        b_values = np.zeros((filter_shape[0],), dtype=theano.config.floatX)\n        b = theano.shared(value=b_values, borrow=True)\n        tparams['cnn_b_' + str(filter_size)] = b\n        grad_params += [W, b]\n    return tparams, grad_params\n\ndef param_init_lstm(proj_size, tparams, grad_params):\n    W = np.concatenate([ortho_weight(proj_size),\n                           ortho_weight(proj_size),\n                           ortho_weight(proj_size),\n                           ortho_weight(proj_size)], axis=1)\n    W_t = theano.shared(W, borrow=True)\n    tparams[_p('lstm', 'W')] = W_t\n    U = np.concatenate([ortho_weight(proj_size),\n                           ortho_weight(proj_size),\n                           ortho_weight(proj_size),\n                           ortho_weight(proj_size)], axis=1)\n    U_t = theano.shared(U, borrow=True)\n    tparams[_p('lstm', 'U')] = U_t\n    b = np.zeros((4 * proj_size,))\n    b_t = theano.shared(b.astype(config.floatX), borrow=True)\n    tparams[_p('lstm', 'b')] = b_t\n    grad_params += [W_t, U_t, b_t]\n\n    return tparams, grad_params\n\ndef dropout_layer(state_before, use_noise, trng):\n    proj = T.switch(use_noise,\n                         (state_before *\n                          trng.binomial(state_before.shape,\n                                        p=0.5, n=1,\n                                        dtype=state_before.dtype)),\n                         state_before * 0.5)\n    return proj\n\nclass LSTM(object):\n  def __init__(self, input1, input2, input3, mask1, mask2, mask3, word_embeddings, batch_size, sequence_len, embedding_size, filter_sizes, num_filters):\n    #proj_size means embedding_size\n    #'lstm_W' = [embedding_size, embedding_size]\n    #'lstm_U' = [embedding_size, embedding_size]\n    #'lstm_b' = [embedding_size]\n    proj_size = 100 #TODO, what does proj mean\n    self.params, tparams = [], {}\n    tparams, self.params = param_init_lstm(proj_size, tparams, self.params)\n    tparams, self.params = param_init_cnn(filter_sizes, num_filters, proj_size, tparams, self.params)\n    lookup_table = theano.shared(word_embeddings, borrow=True)\n    tparams['lookup_table'] = lookup_table\n    self.params += [lookup_table]\n\n    n_timesteps = input1.shape[0]\n    n_samples = input1.shape[1]\n\n    lstm1, lstm_whole1 = self._lstm_net(tparams, input1, sequence_len, batch_size, embedding_size, mask1, proj_size)\n    lstm2, lstm_whole2 = self._lstm_net(tparams, input2, sequence_len, batch_size, embedding_size, mask2, proj_size)\n    lstm3, lstm_whole3 = self._lstm_net(tparams, input3, sequence_len, batch_size, embedding_size, mask3, proj_size)\n\n    #dimshuffle [sequence_len, batch_size, proj_size] to [batch_size, sequence_len, proj_size]\n    cnn_input1 = T.reshape(lstm1.dimshuffle(1, 0, 2), [batch_size, 1, sequence_len, proj_size])\n    cnn_input2 = T.reshape(lstm2.dimshuffle(1, 0, 2), [batch_size, 1, sequence_len, proj_size])\n    cnn_input3 = T.reshape(lstm3.dimshuffle(1, 0, 2), [batch_size, 1, sequence_len, proj_size])\n    cnn1 = self._cnn_net(tparams, cnn_input1, batch_size, sequence_len, num_filters, filter_sizes, proj_size)\n    cnn2 = self._cnn_net(tparams, cnn_input2, batch_size, sequence_len, num_filters, filter_sizes, proj_size)\n    cnn3 = self._cnn_net(tparams, cnn_input3, batch_size, sequence_len, num_filters, filter_sizes, proj_size)\n\n    len1 = T.sqrt(T.sum(cnn1 * cnn1, axis=1))\n    len2 = T.sqrt(T.sum(cnn2 * cnn2, axis=1))\n    len3 = T.sqrt(T.sum(cnn3 * cnn3, axis=1))\n\n    self.cos12 = T.sum(cnn1 * cnn2, axis=1) / (len1 * len2)\n    self.cos13 = T.sum(cnn1 * cnn3, axis=1) / (len1 * len3)\n\n    zero = theano.shared(np.zeros(batch_size, dtype=config.floatX), borrow=True)\n    margin = theano.shared(np.full(batch_size, 0.05, dtype=config.floatX), borrow=True)\n    diff = T.cast(T.maximum(zero, margin - self.cos12 + self.cos13), dtype=config.floatX)\n    self.cost = T.sum(diff, acc_dtype=config.floatX)\n    self.accuracy = T.sum(T.cast(T.eq(zero, diff), dtype='int32')) / float(batch_size)\n\n  def _cnn_net(self, tparams, cnn_input, batch_size, sequence_len, num_filters, filter_sizes, proj_size):\n    outputs = []\n    for filter_size in filter_sizes:\n        filter_shape = (num_filters, 1, filter_size, proj_size)\n        image_shape = (batch_size, 1, sequence_len, proj_size)\n        W = tparams['cnn_W_' + str(filter_size)]\n        b = tparams['cnn_b_' + str(filter_size)]\n        conv_out = conv2d(input=cnn_input, filters=W, filter_shape=filter_shape, input_shape=image_shape)\n        pooled_out = pool.pool_2d(input=conv_out, ds=(sequence_len - filter_size + 1, 1), ignore_border=True, mode='max')\n        pooled_active = T.tanh(pooled_out + b.dimshuffle('x', 0, 'x', 'x'))\n        outputs.append(pooled_active)\n    num_filters_total = num_filters * len(filter_sizes)\n    output_tensor = T.reshape(T.concatenate(outputs, axis=1), [batch_size, num_filters_total])\n    return output_tensor\n\n  def _lstm_net(self, tparams, _input, sequence_len, batch_size, embedding_size, mask, proj_size):\n    input_matrix = tparams['lookup_table'][T.cast(_input.flatten(), dtype=\"int32\")]\n    input_x = input_matrix.reshape((sequence_len, batch_size, embedding_size))\n    proj, proj_whole = lstm_layer(tparams, input_x, proj_size, prefix='lstm', mask=mask)\n    #if useMask == True:\n    #proj = (proj * mask[:, :, None]).sum(axis=0)\n    #proj = proj / mask.sum(axis=0)[:, None]\n    #if options['use_dropout']:\n    #proj = dropout_layer(proj, use_noise, trng)\n    return proj, proj_whole\n\n#state_below is word_embbeding tensor(3dim)\ndef lstm_layer(tparams, state_below, proj_size, prefix='lstm', mask=None):\n    #dim-0 steps, dim-1 samples(batch_size), dim-3 word_embedding\n    nsteps = state_below.shape[0]\n    if state_below.ndim == 3:\n        n_samples = state_below.shape[1]\n    else:\n        n_samples = 1\n\n    assert mask is not None\n\n    def _slice(_x, n, dim):\n        if _x.ndim == 3:\n            return _x[:, :, n * dim:(n + 1) * dim]\n        return _x[:, n * dim:(n + 1) * dim]\n\n    #h means hidden output? c means context? so we'll use h?\n    #rval[0] = [sequence_len, batch_size, proj_size], rval[1] the same\n\n    #so preact size must equl to x_(lstm input slice)\n    #if you want change lstm h(t) size, 'lstm_U' and 'lstm_b'\n    #and precat must be changed to another function, like h*U+b\n    #see http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n    #f(t) = sigmoid(Wf * [h(t-1),x(t)] + bf)\n    def _step(m_, x_, h_, c_):\n        preact = T.dot(h_, tparams[_p(prefix, 'U')])\n        preact += x_\n\n        i = T.nnet.sigmoid(_slice(preact, 0, proj_size))\n        f = T.nnet.sigmoid(_slice(preact, 1, proj_size))\n        o = T.nnet.sigmoid(_slice(preact, 2, proj_size))\n        c = T.tanh(_slice(preact, 3, proj_size))\n\n        c = f * c_ + i * c\n        c = m_[:, None] * c + (1. - m_)[:, None] * c_\n\n        h = o * T.tanh(c)\n        #if mask(t-1)==0, than make h(t) = h(t-1)\n        h = m_[:, None] * h + (1. - m_)[:, None] * h_\n\n        return h, c\n\n    state_below = (T.dot(state_below, tparams[_p(prefix, 'W')]) +\n                   tparams[_p(prefix, 'b')])\n\n    dim_proj = proj_size\n    rval, updates = theano.scan(_step,\n                                sequences=[mask, state_below],\n                                outputs_info=[T.alloc(numpy_floatX(0.),\n                                                           n_samples,\n                                                           dim_proj),\n                                              T.alloc(numpy_floatX(0.),\n                                                           n_samples,\n                                                           dim_proj)],\n                                name=_p(prefix, '_layers'),\n                                n_steps=nsteps)\n    return rval[0], rval[1]\n\ndef _p(pp, name):\n    return '%s_%s' % (pp, name)\n\ndef train():\n    batch_size = int(256)\n    embedding_size = 100\n    learning_rate = 0.05\n    n_epochs = 20000000\n    validation_freq = 1000\n    filter_sizes = [1, 2, 3, 5]\n    num_filters = 500\n\n    vocab = build_vocab()\n    word_embeddings = load_word_embeddings(vocab, embedding_size)\n    trainList = load_train_list()\n    testList = load_test_list()\n    train_x1, train_x2, train_x3, mask1, mask2, mask3 = load_data(trainList, vocab, batch_size)\n    x1, x2, x3 = T.fmatrix('x1'), T.fmatrix('x2'), T.fmatrix('x3')\n    m1, m2, m3 = T.fmatrix('m1'), T.fmatrix('m2'), T.fmatrix('m3')\n    model = LSTM(\n        input1=x1, input2=x2, input3=x3,\n        mask1=m1, mask2=m2, mask3=m3,\n        word_embeddings=word_embeddings,\n        batch_size=batch_size,\n        sequence_len=train_x1.shape[0], #row is sequence_len\n        embedding_size=embedding_size,\n        filter_sizes=filter_sizes,\n        num_filters=num_filters)\n\n    cost, cos12, cos13 = model.cost, model.cos12, model.cos13\n    params, accuracy = model.params, model.accuracy\n    grads = T.grad(cost, params)\n    updates = [\n        (param_i, param_i - learning_rate * grad_i)\n        for param_i, grad_i in zip(params, grads)\n    ]\n\n    p1, p2, p3 = T.fmatrix('p1'), T.fmatrix('p2'), T.fmatrix('p3')\n    q1, q2, q3 = T.fmatrix('q1'), T.fmatrix('q2'), T.fmatrix('q3')\n    train_model = theano.function(\n        [p1, p2, p3, q1, q2, q3], \n        [cost, accuracy], \n        updates=updates,\n        givens={\n            x1: p1, x2: p2, x3: p3, m1: q1, m2: q2, m3: q3\n        }\n    )\n\n    v1, v2, v3 = T.matrix('v1'), T.matrix('v2'), T.matrix('v3')\n    u1, u2, u3 = T.matrix('u1'), T.matrix('u2'), T.matrix('u3')\n    validate_model = theano.function(\n        inputs=[v1, v2, v3, u1, u2, u3],\n        outputs=[cos12, cos13],\n        #updates=updates,\n        givens={\n            x1: v1, x2: v2, x3: v3, m1: u1, m2: u2, m3: u3\n        }\n    )\n\n    epoch = 0\n    done_looping = False\n    while (epoch < n_epochs) and (not done_looping):\n        epoch += 1\n        train_x1, train_x2, train_x3, mask1, mask2, mask3 = load_data(trainList, vocab, batch_size)\n        #print('train_x1, train_x2, train_x3')\n        #print(train_x1.shape, train_x2.shape, train_x3.shape)\n        cost_ij, acc = train_model(train_x1, train_x2, train_x3, mask1, mask2, mask3)\n        print 'load data done ...... epoch:' + str(epoch) + ' cost:' + str(cost_ij) + ', acc:' + str(acc)\n        if epoch % validation_freq == 0:\n            print 'Evaluation ......'\n            validation(validate_model, testList, vocab, batch_size)\n\nif __name__ == '__main__':\n    train()\n"
  },
  {
    "path": "rnn_attention/tensorflow/insurance_qa_data_helpers.py",
    "content": "import numpy as np\nimport random\nfrom operator import itemgetter\n\nprecision = '/export/jw/cnn/insuranceQA/acc.lstm'\n\nempty_vector = []\nfor i in range(0, 100):\n    empty_vector.append(float(0.0))\nonevector = []\nfor i in range(0, 10):\n    onevector.append(float(1))\nzerovector = []\nfor i in range(0, 10):\n    zerovector.append(float(0))\n\ndef build_vocab():\n    code, vocab = int(0), {}\n    vocab['UNKNOWN'] = code\n    code += 1\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        items = line.strip().split(' ')\n        for i in range(2, 3):\n            words = items[i].split('_')\n            for word in words:\n                if not word in vocab:\n                    vocab[word] = code\n                    code += 1\n    for line in open('/export/jw/cnn/insuranceQA/test1'):\n        items = line.strip().split(' ')\n        for i in range(2, 3):\n            words = items[i].split('_')\n            for word in words:\n                if not word in vocab:\n                    vocab[word] = code\n                    code += 1\n    return vocab\n\ndef read_alist():\n    alist = []\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        items = line.strip().split(' ')\n        alist.append(items[3])\n    print('read_alist done ......')\n    return alist\n\ndef load_vectors():\n    vectors = {}\n    for line in open('/export/jw/cnn/insuranceQA/vectors.nobin'):\n        items = line.strip().split(' ')\n        if (len(items) < 101):\n            continue\n        vec = []\n        for i in range(1, 101):\n            vec.append(float(items[i]))\n        vectors[items[0]] = vec\n    return vectors\n\ndef read_vector(vectors, word):\n    global empty_vector\n    if word in vectors:\n        return vectors[word]\n    else:\n        return empty_vector\n        #return vectors['</s>']\n\ndef load_train_list():\n    train_list = []\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        items = line.strip().split(' ')\n        if items[0] == '1':\n            train_list.append(line.strip().split(' '))\n    return train_list\n\ndef load_test_list():\n    test_list = []\n    for line in open('/export/jw/cnn/insuranceQA/test1'):\n       test_list.append(line.strip().split(' '))\n    return test_list\n\ndef load_train_and_vectors():\n    trainList = []\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        trainList.append(line.strip())\n    vectors = load_vectors()\n    return trainList, vectors\n\ndef read_raw():\n    raw = []\n    for line in open('/export/jw/cnn/insuranceQA/train'):\n        items = line.strip().split(' ')\n        if items[0] == '1':\n            raw.append(items)\n    return raw\n\ndef encode_sent(vocab, string, size):\n    x, m = [], []\n    words = string.split('_')\n    for i in range(0, size):\n        if words[i] in vocab: x.append(vocab[words[i]])\n        else: x.append(vocab['UNKNOWN'])\n        if words[i] == '<a>': m.append(1)\n        else: m.append(1)\n    return x, m\n\ndef load_val_data(test_list, vocab, index, batch_size, max_len):\n    x1, x2, x3, m1, m2, m3 = [], [], [], [], [], []\n    for i in range(0, batch_size):\n        t_i = index + i\n        if t_i >= len(test_list):\n            t_i = len(test_list) - 1\n        items = test_list[t_i]\n        x, m = encode_sent(vocab, items[2], max_len)\n        x1.append(x)\n        m1.append(m)\n        x, m = encode_sent(vocab, items[3], max_len)\n        x2.append(x)\n        m2.append(m)\n        x, m = encode_sent(vocab, items[3], max_len)\n        x3.append(x)\n        m3.append(m)\n    return np.array(x1, dtype='float32'), np.array(x2, dtype='float32'), np.array(x3, dtype='float32'), np.transpose(np.array(m1, dtype='float32')) , np.transpose(np.array(m2, dtype='float32')), np.transpose(np.array(m3, dtype='float32'))\n\ndef load_train_data(trainList, vocab, batch_size, max_len):\n    train_1, train_2, train_3 = [], [], []\n    mask_1, mask_2, mask_3 = [], [], []\n    counter = 0\n    while True:\n        pos = trainList[random.randint(0, len(trainList)-1)]\n        neg = trainList[random.randint(0, len(trainList)-1)]\n        if pos[2].startswith('<a>') or pos[3].startswith('<a>') or neg[3].startswith('<a>'):\n            #print 'empty string ......'\n            continue\n        x, m = encode_sent(vocab, pos[2], max_len)\n        train_1.append(x)\n        mask_1.append(m)\n        x, m = encode_sent(vocab, pos[3], max_len)\n        train_2.append(x)\n        mask_2.append(m)\n        x, m = encode_sent(vocab, neg[3], max_len)\n        train_3.append(x)\n        mask_3.append(m)\n        counter += 1\n        if counter >= batch_size:\n            break\n    return np.array(train_1, dtype='float32'), np.array(train_2, dtype='float32'), np.array(train_3, dtype='float32'), np.transpose(np.array(mask_1, dtype='float32')) , np.transpose(np.array(mask_2, dtype='float32')), np.transpose(np.array(mask_3, dtype='float32'))\n\ndef evaluation(score_list, test_list):\n    global precision\n    sessdict, index = {}, int(0)\n    for items in test_list:\n        qid = items[1].split(':')[1]\n        if not qid in sessdict:\n            sessdict[qid] = []\n        sessdict[qid].append((score_list[index], items[0]))\n        index += 1\n        if index >= len(test_list):\n            break\n    lev1, lev0 = float(0), float(0)\n    of = open(precision, 'a')\n    for k, v in sessdict.items():\n        v.sort(key=itemgetter(0), reverse=True)\n        score, flag = v[0]\n        if flag == '1': lev1 += 1\n        if flag == '0': lev0 += 1\n    of.write('lev1:' + str(lev1) + '\\n')\n    of.write('lev0:' + str(lev0) + '\\n')\n    print('lev1 ' + str(lev1))\n    print('lev0 ' + str(lev0))\n    print('precision:' + str(lev1 / (lev0 + lev1)))\n    of.close()\n"
  },
  {
    "path": "rnn_attention/tensorflow/tf_rnn_char.py",
    "content": "# -*- coding: utf-8 -*-\n\n####################################################################################\n#test1 top1准确率59%\n####################################################################################\nimport tensorflow as tf\nimport numpy as np\nfrom operator import itemgetter\nimport random, datetime, json, insurance_qa_data_helpers\n\nclass RNN_Model(object):\n    def _rnn_net(self, inputs, mask, embedding, keep_prob, batch_size, embed_dim, num_step, fw_cell, bw_cell):\n        _initial_state = fw_cell.zero_state(batch_size,dtype=tf.float32)\n        inputs=tf.nn.embedding_lookup(embedding, inputs)\n        inputs = tf.nn.dropout(inputs, self.keep_prob)\n        #[batch_size, sequence_length, embedding_size]转换为[sequence_length, batch_size, embedding_size]\n        inputs = tf.transpose(inputs, [1, 0, 2])\n        #[sequence_length, batch_size, embedding_size]转换为list, sequence_length个[batch_size, embedding_size]\n        inputs = tf.unstack(inputs)\n        #inputs = tf.reshape(inputs, [-1, embed_dim])\n        #inputs = tf.split(inputs, num_step, 0)\n        #输出为list, sequence_length个[batch_size, embedding_size * 2]\n        outputs, _, _ = tf.contrib.rnn.static_bidirectional_rnn(fw_cell, bw_cell, inputs, initial_state_fw=_initial_state, initial_state_bw=_initial_state)\n        outputs = tf.transpose(tf.stack(outputs), [1, 0, 2])\n        self.outputs = outputs\n        #对rnn的输出[batch_size, sequence_length, embedding_size],目前采用maxpooling是最好的效果\n        #mean_pooling以及取最后一个step的向量,效果都不好\n        outputs = self._max_pooling(outputs)\n        print outputs\n        \n        #outputs = outputs[-1]\n        #outputs = outputs * mask[:, :, None]\n        #mean pooling\n        #outputs = tf.reduce_sum(outputs, 0) / (tf.reduce_sum(mask, 0)[:,None])\n        return outputs\n\n    def _max_pooling(self, lstm):\n        sequence_length, embedding_size = int(lstm.get_shape()[1]), int(lstm.get_shape()[2])\n        lstm = tf.expand_dims(lstm, -1)\n        output = tf.nn.max_pool(lstm, ksize=[1, sequence_length, 1, 1], strides=[1, 1, 1, 1], padding='VALID')\n        output = tf.reshape(output, [-1, embedding_size])\n        return output\n        \n    def __init__(self, config, is_training=True):\n        self.keep_prob=tf.placeholder(tf.float32, name='dropout_keep_prob')\n        self.batch_size=config.batch_size\n        self.num_step=config.num_step\n\n        self.qlist = tf.placeholder(tf.int32, [self.batch_size, self.num_step])\n        #这个版本没有使用mask\n        self.mask_q = tf.placeholder(tf.float32, [self.num_step, self.batch_size])\n        self.plist = tf.placeholder(tf.int32, [self.batch_size, self.num_step])\n        self.mask_p = tf.placeholder(tf.float32, [self.num_step, self.batch_size])\n        self.nlist = tf.placeholder(tf.int32, [self.batch_size, self.num_step])\n        self.mask_n = tf.placeholder(tf.float32, [self.num_step, self.batch_size])\n\n        hidden_neural_size=config.hidden_neural_size\n        vocabulary_size=config.vocabulary_size\n        self.embed_dim=config.embed_dim\n        hidden_layer_num=config.hidden_layer_num\n\n        #fw_cell = tf.contrib.rnn.BasicLSTMCell(hidden_neural_size,forget_bias=1.0,state_is_tuple=True)\n        fw_cell = tf.contrib.rnn.GRUCell(num_units=hidden_neural_size, activation=tf.nn.relu)\n        fw_cell =  tf.contrib.rnn.DropoutWrapper(\n            fw_cell,output_keep_prob=self.keep_prob\n        )\n        #bw_cell = tf.contrib.rnn.BasicLSTMCell(hidden_neural_size,forget_bias=1.0,state_is_tuple=True)\n        bw_cell = tf.contrib.rnn.GRUCell(num_units=hidden_neural_size, activation=tf.nn.relu)\n        bw_cell =  tf.contrib.rnn.DropoutWrapper(\n            bw_cell,output_keep_prob=self.keep_prob\n        )\n\n        #embedding layer\n        with tf.device(\"/cpu:1\"),tf.name_scope(\"embedding_layer\"):\n            self.embedding = tf.Variable(tf.truncated_normal([vocabulary_size, self.embed_dim], stddev=0.1), name='W')\n            #self.a_embedding = tf.Variable(tf.truncated_normal([vocabulary_size, self.embed_dim], stddev=0.1), name='W')\n\n        q = self._rnn_net(self.qlist, mask_q, self.embedding, self.keep_prob, self.batch_size, self.embed_dim, self.num_step, fw_cell, bw_cell)\n        tf.get_variable_scope().reuse_variables()\n        p = self._rnn_net(self.plist, mask_p, self.embedding, self.keep_prob, self.batch_size, self.embed_dim, self.num_step, fw_cell, bw_cell)\n        tf.get_variable_scope().reuse_variables()\n        n = self._rnn_net(self.nlist, mask_n, self.embedding, self.keep_prob, self.batch_size, self.embed_dim, self.num_step, fw_cell, bw_cell)\n        #len_1 = tf.clip_by_value(tf.sqrt(tf.reduce_sum(tf.multiply(q, q), 1)), 0.01, 100000)\n        #len_2 = tf.clip_by_value(tf.sqrt(tf.reduce_sum(tf.multiply(p, p), 1)), 0.01, 100000)\n        #len_3 = tf.clip_by_value(tf.sqrt(tf.reduce_sum(tf.multiply(n, n), 1)), 0.01, 100000)\n        len_1 = tf.sqrt(tf.reduce_sum(tf.multiply(q, q), 1))\n        len_2 = tf.sqrt(tf.reduce_sum(tf.multiply(p, p), 1))\n        len_3 = tf.sqrt(tf.reduce_sum(tf.multiply(n, n), 1))\n\n        self.cos12 = tf.reduce_sum(tf.multiply(q, p), axis=1) / (len_1 * len_2)\n        self.cos13 = tf.reduce_sum(tf.multiply(q, n), axis=1) / (len_1 * len_3)\n        self.q = q\n        self.p = p\n\n        zero = tf.constant(np.zeros(self.batch_size, dtype='float32'))\n        margin = tf.constant(np.full(self.batch_size, 0.1, dtype='float32'))\n        diff = tf.cast(tf.maximum(zero, margin - self.cos12 + self.cos13), dtype='float32')\n        self.cost = tf.reduce_sum(diff)\n        self.accuracy = tf.reduce_sum(tf.cast(tf.equal(zero, diff), dtype='float32')) / float(self.batch_size)\n\ndef train_step(model, qlist, plist, nlist, mask_q, mask_p, mask_n):\n    fetches = [model.cost, model.accuracy, global_step, train_op, model.cos12, model.q, model.p, model.outputs]\n    feed_dict = {\n        model.qlist: qlist,\n        model.plist: plist,\n        model.nlist: nlist,\n        model.mask_q : mask_q,\n        model.mask_p : mask_p,\n        model.mask_n : mask_n,\n        model.keep_prob: config.keep_prob\n    }\n    cost, accuracy, step, _, cos12, q, p, outputs = sess.run(fetches, feed_dict)\n    time_str = datetime.datetime.now().isoformat()\n    print(\"{}: step {}, loss {:g}, acc {:g}\".format(time_str, step, cost, accuracy))\n\n\ndef dev_step(model, vocab, batch_size, max_len):\n    score_list, i = [], int(0)\n    while True:\n        qlist, plist, nlist, mask_q, mask_p, mask_n = insurance_qa_data_helpers.load_val_data(test_list, vocab, i, FLAGS.batch_size, max_len)\n        feed_dict = {\n            model.qlist: qlist,\n            model.plist: plist,\n            model.nlist: nlist,\n            model.mask_q : mask_q,\n            model.mask_p : mask_p,\n            model.mask_n : mask_n,\n            model.keep_prob: float(1.0)\n        }\n        batch_scores = sess.run([model.cos12], feed_dict)\n        for score in batch_scores[0]:\n            score_list.append(score)\n        i += FLAGS.batch_size\n        if i >= len(test_list):\n            break\n    insurance_qa_data_helpers.evaluation(score_list, test_list)\n\ntf.flags.DEFINE_integer('evaluate_every',10000,'evaluate every')\ntf.flags.DEFINE_integer('batch_size',64,'the batch_size of the training procedure')\ntf.flags.DEFINE_integer('emdedding_dim',100,'embedding dim')\ntf.flags.DEFINE_integer('hidden_neural_size',200,'LSTM hidden neural size')\ntf.flags.DEFINE_integer('hidden_layer_num',1,'LSTM hidden layer num')\ntf.flags.DEFINE_integer('max_len',100,'max_len of training sentence')\ntf.flags.DEFINE_float('init_scale',0.1,'init scale')\ntf.flags.DEFINE_float('keep_prob',0.5,'dropout rate')\ntf.flags.DEFINE_integer('num_epoch',1000000,'num epoch')\ntf.flags.DEFINE_integer('max_grad_norm',5,'max_grad_norm')\n# Misc Parameters\ntf.flags.DEFINE_boolean(\"allow_soft_placement\", True, \"Allow device soft device placement\")\ntf.flags.DEFINE_boolean(\"log_device_placement\", False, \"Log placement of ops on devices\")\nFLAGS = tf.flags.FLAGS\nFLAGS._parse_flags()\n\nvocab = insurance_qa_data_helpers.build_vocab()\ntrain_list = insurance_qa_data_helpers.load_train_list()\nqlist, plist, nlist, mask_q, mask_p, mask_n = insurance_qa_data_helpers.load_train_data(train_list, vocab, FLAGS.batch_size, FLAGS.max_len)\ntest_list = insurance_qa_data_helpers.load_test_list()\n\nclass Config(object):\n    hidden_neural_size=FLAGS.hidden_neural_size\n    vocabulary_size=len(vocab)\n    embed_dim=FLAGS.emdedding_dim\n    hidden_layer_num=FLAGS.hidden_layer_num\n    keep_prob=FLAGS.keep_prob\n    batch_size = FLAGS.batch_size\n    num_step = FLAGS.max_len\n    max_grad_norm=FLAGS.max_grad_norm\n    num_epoch = FLAGS.num_epoch\n\nconfig = Config()\neval_config=Config()\neval_config.keep_prob=1.0\n\nwith tf.Graph().as_default():\n    with tf.device('/gpu:1'):\n      session_conf = tf.ConfigProto(\n              allow_soft_placement=FLAGS.allow_soft_placement,\n              log_device_placement=FLAGS.log_device_placement)\n      sess = tf.Session(config=session_conf)\n      with sess.as_default():\n        initializer = tf.random_uniform_initializer(-1*FLAGS.init_scale,1*FLAGS.init_scale)\n        with tf.variable_scope(\"model\",reuse=None,initializer=initializer):\n            model = RNN_Model(config=config, is_training=True)\n\n        # Define Training procedure\n        global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n        #optimizer = tf.train.RMSPropOptimizer(0.01)\n        #optimizer = tf.train.AdamOptimizer(0.1)\n        optimizer = tf.train.GradientDescentOptimizer(0.2)\n        grads_and_vars = optimizer.compute_gradients(model.cost)\n        train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)\n\n        # Initialize all variables\n        sess.run(tf.global_variables_initializer())\n        for i in range(config.num_epoch):\n            qlist, plist, nlist, mask_q, mask_p, mask_n = insurance_qa_data_helpers.load_train_data(train_list, vocab, FLAGS.batch_size, FLAGS.max_len)\n            train_step(model, qlist, plist, nlist, mask_q, mask_p, mask_n)\n            current_step = tf.train.global_step(sess, global_step)\n            if current_step % FLAGS.evaluate_every == 0:\n                dev_step(model, vocab, FLAGS.batch_size, FLAGS.max_len)\n"
  },
  {
    "path": "swem/swem_hier.py",
    "content": "import numpy as np\nimport tensorflow as tf\nimport time, os, random, datetime, sys\nfrom sklearn import metrics\nsys.path.append('../')\nimport config, utils\n\n################################################################################\n# Insurance-QA\n# AUC 0.96, top 1 precision:31%\n#\n# quora-data\n# best precision: 0.8369, best threshold:0.62\n################################################################################\nclass SWEM_HIER(object):\n  def __init__(self, \n      sequence_length,\n      vocab_size,\n      embedding_size,\n      embeddings):\n    self.x1 = tf.placeholder(tf.int32, [None, sequence_length])\n    self.x2 = tf.placeholder(tf.int32, [None, sequence_length])\n    self.y = tf.placeholder(tf.float32, [None])\n    self.one = tf.placeholder(tf.float32, [None])\n    #self.dropout_keep_prob = tf.placeholder(tf.float32)\n\n    with tf.device('/cpu:0'), tf.name_scope('embedding'):\n      self.word_mat = tf.Variable(embeddings, trainable=True, dtype=tf.float32)\n      x1_mat = tf.nn.embedding_lookup(self.word_mat, self.x1)\n      x2_mat = tf.nn.embedding_lookup(self.word_mat, self.x2)\n      self.x1_mat_exp = tf.expand_dims(x1_mat, -1)\n      self.x2_mat_exp = tf.expand_dims(x2_mat, -1)\n    p1 = tf.nn.avg_pool(self.x1_mat_exp, ksize=[1, 2, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    p2 = tf.nn.avg_pool(self.x2_mat_exp, ksize=[1, 2, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    p1 = tf.reshape(tf.reduce_max(p1, 1), [-1, embedding_size])\n    p2 = tf.reshape(tf.reduce_max(p2, 1), [-1, embedding_size])\n    \"\"\"\n    p11 = tf.nn.avg_pool(self.x1_mat_exp, ksize=[1, 3, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    p21 = tf.nn.avg_pool(self.x2_mat_exp, ksize=[1, 3, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    p11 = tf.reshape(tf.reduce_max(p11, 1), [-1, embedding_size])\n    p21 = tf.reshape(tf.reduce_max(p21, 1), [-1, embedding_size])\n    p1 = tf.concat([p1, p11], 1)\n    p2 = tf.concat([p2, p21], 1)\n    \"\"\"\n\n    self.cos = self.cosine(p1, p2)\n    self.losses = self.logloss(self.y, self.one, self.cos)\n\n  def logloss(self, y, v_one, sim):\n    a = tf.multiply(y, tf.log(sim)) #y*log(p)\n    b = tf.subtract(v_one, y)#1-y\n    c = tf.log(tf.subtract(v_one, sim))#log(1-p)\n    losses = -tf.add(a, tf.multiply(b, c))#y*log(p)+(1-y)*log(1-p)\n    losses = tf.reduce_sum(losses, -1)\n    return losses\n\n  def cosine(self, t1, t2):\n    len1 = tf.sqrt(tf.reduce_sum(tf.multiply(t1, t1), 1))\n    len2 = tf.sqrt(tf.reduce_sum(tf.multiply(t2, t2), 1))\n    multiply = tf.reduce_sum(tf.multiply(t1, t2), 1)\n    cos = tf.div(multiply, tf.multiply(len1, len2))\n    return tf.clip_by_value(cos, 1e-5, 0.99999)\n\ndef get_constant(batch_size):\n  one, zero = [1.0] * batch_size, [0.0] * batch_size\n  return np.array(one), np.array(zero)\n\nmax_len = 100\nnum_epoch = 200000\nbatch_size = 256\ncheckpoint_every = 10000\nvocab, embeddings = utils.load_embeddings()\nembedding_size = len(embeddings[0])\ntrain_data, test_data = utils.load_train_data(vocab, max_len), utils.load_test_data(vocab, max_len)\nprint('load data done ......')\nprint(embeddings.shape)\n\nprev_auc = 0.0\nwith tf.Graph().as_default():\n    session_conf = tf.ConfigProto(\n      allow_soft_placement=True, log_device_placement=False)\n    sess = tf.Session(config=session_conf)\n    with sess.as_default():\n      swem = SWEM_HIER(max_len, len(vocab), embedding_size, embeddings)\n      global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n      optimizer = tf.train.AdamOptimizer(1e-1)\n      #optimizer = tf.train.GradientDescentOptimizer(1e-1)\n      grads_and_vars = optimizer.compute_gradients(swem.losses)\n      train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)\n\n      timestamp = str(int(time.time()))\n      out_dir = os.path.abspath(os.path.join(os.path.curdir, \"runs\", timestamp))\n      checkpoint_dir = os.path.abspath(os.path.join(out_dir, \"checkpoints\"))\n      checkpoint_prefix = os.path.join(checkpoint_dir, \"model\")\n      if not os.path.exists(checkpoint_dir):\n        os.makedirs(checkpoint_dir)\n      saver = tf.train.Saver(tf.all_variables())\n      sess.run(tf.initialize_all_variables())\n\n      def train_step():\n        y, x1, x2 = utils.gen_train_batch_yxx(train_data, batch_size)\n        one, zero = get_constant(batch_size)\n        feed_dict = {swem.x1:x1, swem.x2:x2, swem.y:y, swem.one:one}\n        _, step, loss, cos = sess.run(\n            [train_op, global_step, swem.losses, swem.cos], feed_dict)\n        time_str = datetime.datetime.now().isoformat()\n        print(\"{}: step {}, loss {:g}\".format(time_str, step, loss))\n\n      def test_step():\n        yp, y, group = [], [], []\n        for i in range(0, len(test_data), batch_size):\n          f, g, x1, x2 = utils.gen_test_batch_yxx(test_data, i, i + batch_size)\n          one, zero = get_constant(len(f))\n          feed_dict = {swem.x1:x1, swem.x2:x2, swem.y:f, swem.one:one}\n          loss, cos = sess.run([swem.losses, swem.cos], feed_dict)\n          yp.extend(cos)\n          y.extend(f)\n          group.extend(g)\n        ppp = [(_y, _g, _yp) for _y, _g, _yp in zip(y, group, yp)]\n        #for _y, _g, _yp in ppp:\n        #  print(str(_y) + ' ' + str(_g) + ' ' + str(_yp))\n        return y[:len(test_data)], group[:len(test_data)], yp[:len(test_data)]\n\n      for i in range(num_epoch):\n        train_step()\n        current_step = tf.train.global_step(sess, global_step)\n        if current_step % checkpoint_every == 0:\n          y, g, yp = test_step()\n          utils._eval(y, g, yp)\n\n#utils.save_features(features[0] + features[1] + features[2], './data/gen_sweg_hier_train.f')\n#utils.save_features(features[3], './data/gen_sweg_hier_test.f')\n"
  },
  {
    "path": "swem/swem_hier_margin.py",
    "content": "import numpy as np\nimport tensorflow as tf\nimport time, os, random, datetime, sys\nfrom sklearn import metrics\nsys.path.append('../')\nimport config, utils\n\n#top 1 precision:54%\nclass SWEM_HIER(object):\n  def __init__(self, \n      margin,\n      sequence_length,\n      vocab_size,\n      embedding_size,\n      embeddings):\n    self.zero = tf.placeholder(tf.float32, [None])\n    self.q = tf.placeholder(tf.int32, [None, sequence_length])\n    self.qp = tf.placeholder(tf.int32, [None, sequence_length])\n    self.qn = tf.placeholder(tf.int32, [None, sequence_length])\n\n    with tf.device('/cpu:0'), tf.name_scope('embedding'):\n      self.word_mat = tf.Variable(embeddings, trainable=True, dtype=tf.float32)\n      q_mat = tf.nn.embedding_lookup(self.word_mat, self.q)\n      qp_mat = tf.nn.embedding_lookup(self.word_mat, self.qp)\n      qn_mat = tf.nn.embedding_lookup(self.word_mat, self.qn)\n      self.q_mat_exp = tf.expand_dims(q_mat, -1)\n      self.qp_mat_exp = tf.expand_dims(qp_mat, -1)\n      self.qn_mat_exp = tf.expand_dims(qn_mat, -1)\n\n      self.word_mat1 = tf.Variable(embeddings, trainable=True, dtype=tf.float32)\n      q_mat1 = tf.nn.embedding_lookup(self.word_mat1, self.q)\n      qp_mat1 = tf.nn.embedding_lookup(self.word_mat1, self.qp)\n      qn_mat1 = tf.nn.embedding_lookup(self.word_mat1, self.qn)\n      self.q_mat_exp1 = tf.expand_dims(q_mat1, -1)\n      self.qp_mat_exp1 = tf.expand_dims(qp_mat1, -1)\n      self.qn_mat_exp1 = tf.expand_dims(qn_mat1, -1)\n\n    q = tf.nn.avg_pool(self.q_mat_exp, ksize=[1, 2, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    qp = tf.nn.avg_pool(self.qp_mat_exp, ksize=[1, 2, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    qn = tf.nn.avg_pool(self.qn_mat_exp, ksize=[1, 2, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    q = tf.reshape(tf.reduce_max(q, 1), [-1, embedding_size])\n    qp = tf.reshape(tf.reduce_max(qp, 1), [-1, embedding_size])\n    qn = tf.reshape(tf.reduce_max(qn, 1), [-1, embedding_size])\n\n    q1 = tf.nn.avg_pool(self.q_mat_exp1, ksize=[1, 1, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    qp1 = tf.nn.avg_pool(self.qp_mat_exp1, ksize=[1, 1, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    qn1 = tf.nn.avg_pool(self.qn_mat_exp1, ksize=[1, 1, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    q1 = tf.reshape(tf.reduce_max(q1, 1), [-1, embedding_size])\n    qp1 = tf.reshape(tf.reduce_max(qp1, 1), [-1, embedding_size])\n    qn1 = tf.reshape(tf.reduce_max(qn1, 1), [-1, embedding_size])\n\n    q = tf.concat([q, q1], 1)\n    qp = tf.concat([qp, qp1], 1)\n    qn = tf.concat([qn, qn1], 1)\n\n    self.cos_q_qp = self.cosine(q, qp)\n    self.cos_q_qn = self.cosine(q, qn)\n\n    self.losses, loss_batch = self.margin_loss(self.zero, margin, self.cos_q_qp, self.cos_q_qn)\n\n    correct = tf.equal(self.zero, loss_batch)\n    self.accuracy = tf.reduce_mean(tf.cast(correct, \"float\"))\n\n  def margin_loss(self, zero, margin, cos_q_qp, cos_q_qn):\n    loss_batch = tf.maximum(zero, tf.subtract(margin, tf.subtract(cos_q_qp, cos_q_qn)))\n    losses = tf.reduce_sum(loss_batch)\n    return losses, loss_batch\n\n  def logloss(self, y, v_one, sim):\n    a = tf.multiply(y, tf.log(sim)) #y*log(p)\n    b = tf.subtract(v_one, y)#1-y\n    c = tf.log(tf.subtract(v_one, sim))#log(1-p)\n    losses = -tf.add(a, tf.multiply(b, c))#y*log(p)+(1-y)*log(1-p)\n    losses = tf.reduce_sum(losses, -1)\n    return losses\n\n  def cosine(self, t1, t2):\n    len1 = tf.sqrt(tf.reduce_sum(tf.multiply(t1, t1), 1))\n    len2 = tf.sqrt(tf.reduce_sum(tf.multiply(t2, t2), 1))\n    multiply = tf.reduce_sum(tf.multiply(t1, t2), 1)\n    cos = tf.div(multiply, tf.multiply(len1, len2))\n    return tf.clip_by_value(cos, 1e-5, 0.99999)\n\ndef get_constant(batch_size):\n  one, zero = [1.0] * batch_size, [0.0] * batch_size\n  return np.array(one), np.array(zero)\n\nmargin = 0.05\nmax_len = 200\nnum_epoch = 200000\nbatch_size = 256\ncheckpoint_every = 50000\nvocab, embeddings = utils.load_embeddings()\nembedding_size = len(embeddings[0])\ntrain_data, test_data = utils.load_train_data(vocab, max_len), utils.load_test_data(vocab, max_len)\nprint('load data done ......')\nprint(embeddings.shape)\n\nprev_auc = 0.0\nwith tf.Graph().as_default():\n    session_conf = tf.ConfigProto(\n      allow_soft_placement=True, log_device_placement=False)\n    sess = tf.Session(config=session_conf)\n    with sess.as_default():\n      swem = SWEM_HIER(margin, max_len, len(vocab), embedding_size, embeddings)\n      global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n      optimizer = tf.train.AdamOptimizer(1e-1)\n      #optimizer = tf.train.GradientDescentOptimizer(1e-1)\n      grads_and_vars = optimizer.compute_gradients(swem.losses)\n      train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)\n\n      timestamp = str(int(time.time()))\n      out_dir = os.path.abspath(os.path.join(os.path.curdir, \"runs\", timestamp))\n      checkpoint_dir = os.path.abspath(os.path.join(out_dir, \"checkpoints\"))\n      checkpoint_prefix = os.path.join(checkpoint_dir, \"model\")\n      if not os.path.exists(checkpoint_dir):\n        os.makedirs(checkpoint_dir)\n      saver = tf.train.Saver(tf.all_variables())\n      sess.run(tf.initialize_all_variables())\n\n      def train_step():\n        q, qp, qn = utils.gen_train_batch_qpn(train_data, batch_size)\n        one, zero = get_constant(batch_size)\n        feed_dict = {swem.q:q, swem.qp:qp, swem.qn:qn, swem.zero:zero}\n        _, step, loss, cos, acc = sess.run(\n            [train_op, global_step, swem.losses, swem.cos_q_qp, swem.accuracy], feed_dict)\n        time_str = datetime.datetime.now().isoformat()\n        print(\"{}: step {}, loss {:g}, acc{:g}\".format(time_str, step, loss, acc))\n\n      def test_step():\n        yp, y, group = [], [], []\n        for i in range(0, len(test_data), batch_size):\n          f, g, q1, q2 = utils.gen_test_batch_qpn(test_data, i, i+batch_size)\n          one, zero = get_constant(len(f))\n          feed_dict = {swem.q:q1, swem.qp:q2, swem.qn:q2, swem.zero:zero}\n          loss, cos = sess.run([swem.losses, swem.cos_q_qp], feed_dict)\n          yp.extend(cos)\n          y.extend(f)\n          group.extend(g)\n        ppp = [(_y, _g, _yp) for _y, _g, _yp in zip(y, group, yp)]\n        #for _y, _g, _yp in ppp:\n        #  print(str(_y) + ' ' + str(_g) + ' ' + str(_yp))\n        return y[:len(test_data)], group[:len(test_data)], yp[:len(test_data)]\n\n      for i in range(num_epoch):\n        train_step()\n        current_step = tf.train.global_step(sess, global_step)\n        if current_step % checkpoint_every == 0:\n          y, g, yp = test_step()\n          auc = utils.eval_auc(y, g, yp)\n          top1_prec = utils._eval_top1_prec(y, g, yp)\n          #if auc < prev_auc:\n          #  _flist = [(_f, [s]) for s, _f in zip(score[:len(test_data)], flags)]\n          #  features.append(_flist)\n          #  break\n          #prev_auc = auc\n\n#utils.save_features(features[0] + features[1] + features[2], './data/gen_sweg_hier_train.f')\n#utils.save_features(features[3], './data/gen_sweg_hier_test.f')\n"
  },
  {
    "path": "swem/swem_max_margin.py",
    "content": "import numpy as np\nimport tensorflow as tf\nimport time, os, random, datetime, sys\nfrom sklearn import metrics\nsys.path.append('../')\nimport config, utils\n\nclass SWEM_HIER(object):\n  def __init__(self, \n      margin,\n      sequence_length,\n      vocab_size,\n      embedding_size,\n      embeddings):\n    self.zero = tf.placeholder(tf.float32, [None])\n    self.q = tf.placeholder(tf.int32, [None, sequence_length])\n    self.qp = tf.placeholder(tf.int32, [None, sequence_length])\n    self.qn = tf.placeholder(tf.int32, [None, sequence_length])\n\n    with tf.device('/cpu:0'), tf.name_scope('embedding'):\n      self.word_mat = tf.Variable(embeddings, trainable=True, dtype=tf.float32)\n      q_mat = tf.nn.embedding_lookup(self.word_mat, self.q)\n      qp_mat = tf.nn.embedding_lookup(self.word_mat, self.qp)\n      qn_mat = tf.nn.embedding_lookup(self.word_mat, self.qn)\n      self.q_mat_exp = tf.expand_dims(q_mat, -1)\n      self.qp_mat_exp = tf.expand_dims(qp_mat, -1)\n      self.qn_mat_exp = tf.expand_dims(qn_mat, -1)\n    \"\"\"\n    q = tf.nn.avg_pool(self.q_mat_exp, ksize=[1, 2, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    qp = tf.nn.avg_pool(self.qp_mat_exp, ksize=[1, 2, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    qn = tf.nn.avg_pool(self.qn_mat_exp, ksize=[1, 2, 1, 1], \n        strides=[1, 1, 1, 1], padding='VALID')\n    \"\"\"\n    q = tf.reshape(tf.reduce_max(self.q_mat_exp, 1), [-1, embedding_size])\n    qp = tf.reshape(tf.reduce_max(self.qp_mat_exp, 1), [-1, embedding_size])\n    qn = tf.reshape(tf.reduce_max(self.qn_mat_exp, 1), [-1, embedding_size])\n\n    self.cos_q_qp = self.cosine(q, qp)\n    self.cos_q_qn = self.cosine(q, qn)\n    self.losses, loss_batch = self.margin_loss(self.zero, margin, self.cos_q_qp, self.cos_q_qn)\n\n    correct = tf.equal(self.zero, loss_batch)\n    self.accuracy = tf.reduce_mean(tf.cast(correct, \"float\"))\n\n  def margin_loss(self, zero, margin, cos_q_qp, cos_q_qn):\n    loss_batch = tf.maximum(zero, tf.subtract(margin, tf.subtract(cos_q_qp, cos_q_qn)))\n    losses = tf.reduce_sum(loss_batch)\n    return losses, loss_batch\n\n  def logloss(self, y, v_one, sim):\n    a = tf.multiply(y, tf.log(sim)) #y*log(p)\n    b = tf.subtract(v_one, y)#1-y\n    c = tf.log(tf.subtract(v_one, sim))#log(1-p)\n    losses = -tf.add(a, tf.multiply(b, c))#y*log(p)+(1-y)*log(1-p)\n    losses = tf.reduce_sum(losses, -1)\n    return losses\n\n  def cosine(self, t1, t2):\n    len1 = tf.sqrt(tf.reduce_sum(tf.multiply(t1, t1), 1))\n    len2 = tf.sqrt(tf.reduce_sum(tf.multiply(t2, t2), 1))\n    multiply = tf.reduce_sum(tf.multiply(t1, t2), 1)\n    cos = tf.div(multiply, tf.multiply(len1, len2))\n    return tf.clip_by_value(cos, 1e-5, 0.99999)\n\ndef get_constant(batch_size):\n  one, zero = [1.0] * batch_size, [0.0] * batch_size\n  return np.array(one), np.array(zero)\n\nmargin = 0.05\nmax_len = 200\nnum_epoch = 200000\nbatch_size = 256\ncheckpoint_every = 50000\nvocab, embeddings = utils.load_embeddings()\nembedding_size = len(embeddings[0])\ntrain_data, test_data = utils.load_train_data(vocab, max_len), utils.load_test_data(vocab, max_len)\nprint('load data done ......')\nprint(embeddings.shape)\n\nprev_auc = 0.0\nwith tf.Graph().as_default():\n    session_conf = tf.ConfigProto(\n      allow_soft_placement=True, log_device_placement=False)\n    sess = tf.Session(config=session_conf)\n    with sess.as_default():\n      swem = SWEM_HIER(margin, max_len, len(vocab), embedding_size, embeddings)\n      global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n      optimizer = tf.train.AdamOptimizer(1e-1)\n      #optimizer = tf.train.GradientDescentOptimizer(1e-1)\n      grads_and_vars = optimizer.compute_gradients(swem.losses)\n      train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)\n\n      timestamp = str(int(time.time()))\n      out_dir = os.path.abspath(os.path.join(os.path.curdir, \"runs\", timestamp))\n      checkpoint_dir = os.path.abspath(os.path.join(out_dir, \"checkpoints\"))\n      checkpoint_prefix = os.path.join(checkpoint_dir, \"model\")\n      if not os.path.exists(checkpoint_dir):\n        os.makedirs(checkpoint_dir)\n      saver = tf.train.Saver(tf.all_variables())\n      sess.run(tf.initialize_all_variables())\n\n      def train_step():\n        q, qp, qn = utils.gen_train_batch_qpn(train_data, batch_size)\n        one, zero = get_constant(batch_size)\n        feed_dict = {swem.q:q, swem.qp:qp, swem.qn:qn, swem.zero:zero}\n        _, step, loss, cos, acc = sess.run(\n            [train_op, global_step, swem.losses, swem.cos_q_qp, swem.accuracy], feed_dict)\n        time_str = datetime.datetime.now().isoformat()\n        print(\"{}: step {}, loss {:g}, acc {:g}\".format(time_str, step, loss, acc))\n\n      def test_step():\n        yp, y, group = [], [], []\n        for i in range(0, len(test_data), batch_size):\n          f, g, q1, q2 = utils.gen_test_batch_qpn(test_data, i, i+batch_size)\n          one, zero = get_constant(len(f))\n          feed_dict = {swem.q:q1, swem.qp:q2, swem.qn:q2, swem.zero:zero}\n          loss, cos = sess.run([swem.losses, swem.cos_q_qp], feed_dict)\n          yp.extend(cos)\n          y.extend(f)\n          group.extend(g)\n        ppp = [(_y, _g, _yp) for _y, _g, _yp in zip(y, group, yp)]\n        #for _y, _g, _yp in ppp:\n        #  print(str(_y) + ' ' + str(_g) + ' ' + str(_yp))\n        return y[:len(test_data)], group[:len(test_data)], yp[:len(test_data)]\n\n      for i in range(num_epoch):\n        train_step()\n        current_step = tf.train.global_step(sess, global_step)\n        if current_step % checkpoint_every == 0:\n          y, g, yp = test_step()\n          auc = utils.eval_auc(y, g, yp)\n          top1_prec = utils._eval_top1_prec(y, g, yp)\n          #if auc < prev_auc:\n          #  _flist = [(_f, [s]) for s, _f in zip(score[:len(test_data)], flags)]\n          #  features.append(_flist)\n          #  break\n          #prev_auc = auc\n\n#utils.save_features(features[0] + features[1] + features[2], './data/gen_sweg_hier_train.f')\n#utils.save_features(features[3], './data/gen_sweg_hier_test.f')\n"
  },
  {
    "path": "utils.py",
    "content": "import numpy as np\nimport random, sys, config\nfrom sklearn import metrics\nfrom operator import itemgetter\nfrom itertools import groupby\n\ndef load_embeddings():\n  _data, embeddings, vocab, _id = [], [], {}, int(0)\n  for line in open(config.w2v_bin_file):\n    _data.append(line.strip().split(' '))\n  size, dim = int(_data[0][0]), int(_data[0][1])\n  for i in range(1, len(_data)):\n    w, vec = _data[i][0], [float(_data[i][k]) for k in range(1, dim+1)]\n    embeddings.append(vec)\n    vocab[w] = _id\n    _id += 1\n  embeddings.append([0.01] * dim)\n  vocab['UNKNOWN'] = _id\n  _id += 1\n  embeddings.append([0.01] * dim)\n  vocab['<a>'] = _id\n  return vocab, np.array(embeddings)\n\ndef encode_sent(s, vocab, max_len):\n  ws = [w for w in s.split('_')]\n  ws = ws[:max_len] if len(ws) >= max_len else ws + ['<a>'] * (max_len - len(ws)) \n  nws = []\n  for w in ws:\n    nw = w if w in vocab else 'UNKNOWN'\n    nws.append(vocab[nw])\n  return nws\n\ndef load_train_data(vocab, max_len):\n  if config.dataset == config.dataset_ins:\n    return ins_load_train_data(vocab, max_len)\n  if config.dataset == config.dataset_qur:\n    return qur_load_train_test_data(config.train_file, vocab, max_len)\n  print('bad load_train_data')\n  exit(1)\n\ndef qur_load_train_test_data(_file, vocab, max_len):\n  _data = []\n  for line in open(_file):\n    f, q1, q2 = line.strip().split(' ')\n    q1, q2 = encode_sent(q1, vocab, max_len), encode_sent(q2, vocab, max_len)\n    _data.append((int(f), q1, q2))\n  return _data\n\ndef ins_load_train_data(vocab, max_len):\n  _data = []\n  for line in open(config.train_file):\n    f, q1, q2 = line.strip().split(' ')\n    q1, q2 = encode_sent(q1, vocab, max_len), encode_sent(q2, vocab, max_len)\n    _data.append((q1, q2))\n  return _data\n\ndef load_test_data(vocab, max_len):\n  if config.dataset == config.dataset_ins:\n    return ins_load_test_data(vocab, max_len)\n  if config.dataset == config.dataset_qur:\n    return qur_load_train_test_data(config.test1_file, vocab, max_len)\n  print('bad load_test_data')\n  exit(1)\n\ndef ins_load_test_data(vocab, max_len):\n  _data = []\n  for line in open(config.test1_file):\n    f, g, q1, q2 = line.strip().split(' ')\n    q1, q2 = encode_sent(q1, vocab, max_len), encode_sent(q2, vocab, max_len)\n    _data.append((f, g, q1, q2))\n  return _data\n\ndef gen_train_batch_qpn(_data, batch_size):\n  psample = random.sample(_data, batch_size)\n  nsample = random.sample(_data, batch_size)\n  q = [s1 for s1, s2 in psample]\n  qp = [s2 for s1, s2 in psample]\n  qn = [s2 for s1, s2 in nsample]\n  return np.array(q), np.array(qp), np.array(qn)\n\ndef gen_train_batch_yxx(_data, batch_size):\n  if config.dataset == config.dataset_ins:\n    return ins_gen_train_batch_yxx(_data, batch_size)\n  if config.dataset == config.dataset_qur:\n    return qur_gen_train_batch_yxx(_data, batch_size)\n  print('bad gen_train_batch_yxx')\n  exit(1)\n\ndef qur_gen_train_batch_yxx(_data, batch_size):\n  sample = random.sample(_data, batch_size)\n  y = [i for i,_,_ in sample]\n  x1 = [i for _,i,_ in sample]\n  x2 = [i for _,_,i in sample]\n  return np.array(y), np.array(x1), np.array(x2)\n\ndef ins_gen_train_batch_yxx(_data, batch_size):\n  part_one, part_two = int(batch_size / 4 * 3), int(batch_size / 4)\n  psample = random.sample(_data, part_one)\n  nsample = random.sample(_data, part_two)\n  y = [1.0] * part_one + [0.0] * part_two\n  x1 = [s1 for s1, s2 in psample] + [s1 for s1, s2 in psample[:part_two]]\n  x2 = [s2 for s1, s2 in psample] + [s2 for s1, s2 in nsample]\n  return np.array(y), np.array(x1), np.array(x2)\n\ndef gen_test_batch_qpn(_data, start, end):\n  sample = _data[start:end]\n  for i in range(len(sample), end - start):\n    sample.append(sample[-1])\n  f = [int(i) for i,_,_,_ in sample]\n  g = [int(i) for _,i,_,_ in sample]\n  q1 = [i for _,_,i,_ in sample]\n  q2 = [i for _,_,_,i in sample]\n  return f, g, np.array(q1), np.array(q2)\n\ndef gen_test_batch_yxx(_data, start, end):\n  if config.dataset == config.dataset_ins:\n    return ins_gen_test_batch_yxx(_data, start, end)\n  if config.dataset == config.dataset_qur:\n    return qur_gen_test_batch_yxx(_data, start, end)\n  print('bad gen_test_batch_yxx')\n  exit(1)\n\ndef qur_gen_test_batch_yxx(_data, start, end):\n  sample = _data[start:end]\n  y = [i for i,_,_ in sample]\n  x1 = [i for _,i,_ in sample]\n  x2 = [i for _,_,i in sample]\n  return y, y, np.array(x1), np.array(x2)\n\ndef ins_gen_test_batch_yxx(_data, start, end):\n  sample = _data[start:end]\n  for i in range(len(sample), end - start):\n    sample.append(sample[-1])\n  f = [int(i) for i,_,_,_ in sample]\n  g = [int(i) for _,i,_,_ in sample]\n  q1 = [i for _,_,i,_ in sample]\n  q2 = [i for _,_,_,i in sample]\n  return f, g, np.array(q1), np.array(q2)\n\ndef _eval(y, g, yp):\n  if config.dataset == config.dataset_ins:\n    eval_auc(y, g, yp)\n    eval_top1_prec(y, g, yp)\n  if config.dataset == config.dataset_qur:\n    eval_auc(y, g, yp)\n    eval_best_prec(y, g, yp)\n\ndef eval_best_prec(y, g, yp):\n  best_p, best_s = 0.0, 0.0\n  for i in range(50, 100, 1):\n    i = float(i) / 100\n    positive = 0\n    for _y, _yp in zip(y, yp):\n      p = 1 if _yp >= i else 0\n      if p == _y: positive += 1\n    prec = positive / len(yp)\n    if prec > best_p:\n      best_p = prec\n      best_s = i\n  print('best_prec: ' + str(best_p) + ' best_threshold:' + str(best_s))\n  return best_p, best_s\n\ndef eval_auc(y, g, yp):\n  auc = metrics.roc_auc_score(y, yp)\n  print('auc: ' + str(auc))\n  return auc\n\ndef eval_top1_prec(y, g, yp):\n  _list = [(_y, _g, _yp) for _y, _g, _yp in zip(y, g, yp)]\n  _dict = {}\n  for _y, _g, _yp in _list:\n    if not _g in _dict: _dict[_g] = []\n    _dict[_g].append((_y, _g, _yp))\n  positive, gc = 0 , 0\n  for _, group in _dict.items():\n    group = sorted(group, key=itemgetter(2), reverse=True)\n    gc += 1\n    if group[0][0] == 1: \n      positive += 1\n  prec = positive / gc\n  print('top1 precision ' + str(positive) + '/' + str(gc) + ': '+ str(positive / gc))\n  return prec\n\n"
  }
]