[
  {
    "path": ".gitignore",
    "content": ".idea/\n*/__pycache__/\n__pycache__/\n.cache\n.pytest_cache\n*.pyc\n.DS_Store\n*.log\ndata/Dataset Description.md\n.idea/vcs.xml"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2021 zhhlee\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "README.md",
    "content": "# InterFusion\n\n**KDD 2021: Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding**\n\nInterFusion is an unsupervised MTS anomaly detection and interpretation method. It's core idea is to model the normal patterns of MTS using HVAE with jointly trained hierarchical stochastic latent variables, each of which explicitly learns low-dimensional inter-metric or temporal embeddings. You may refer to our [paper](https://dl.acm.org/doi/abs/10.1145/3447548.3467075) for more details.\n\n## Getting Started\n\n**Clone the repo**\n\n```bash\ngit clone https://github.com/zhhlee/InterFusion.git && cd InterFusion\n```\n\n**Get data**\n\nThe datasets used in this paper are in folder ``data``. You may refer to ``data/Dataset Description`` for more details.\n\n**Install dependencies (with python 3.6+)**\n\n(virtualenv is recommended)\n\n```bash\npip install -r requirements.txt\n```\n\nThe code is tested under the following basic environments:\n\n```\nOS: Ubuntu 18.04\nGPU: GTX 1080 Ti\nCuda: 9.0.176\nPython: 3.6.6\n```\n\n**Run the code**\n\nPlease set the root directory of the project as your Python path.\n\nFor dataset ASD and SMD:\n\n```bash\npython algorithm/stack_train.py --dataset=omi-1\t\t\t# training\npython algorithm/stack_predict.py --load_model_dir=./results/stack_train/\t# evaluation\n```\n\nFor dataset SWaT and WADI (Note: you need to acquire these datasets first following ``data/Dataset Description`` and ``explib/raw_data_converter``):\n\nSWaT:\n\n```bash\npython algorithm/stack_train.py --dataset=SWaT --train.train_start=21600 --train.valid_portion=0.1 --model.window_length=30 '--model.output_shape=[15, 15, 30]' --model.z2_dim=8\t# training\npython algorithm/stack_predict.py --load_model_dir=./results/stack_train/ --mcmc_track=False\t# evaluation\n```\n\nWADI:\n\n```bash\npython algorithm/stack_train.py --dataset=WADI --train.train_start=259200 --train.max_train_size=789371 --train.valid_portion=0.1 --model.window_length=30 '--model.output_shape=[15, 15, 30]' --model.z2_dim=8 # training\npython algorithm/stack_predict.py --load_model_dir=./results/stack_train/ --mcmc_track=False\t# evaluation\n```\n\nThe default model configurations are in ``algorithm/InterFusion.py``, train configs in ``algorithm/stack_train.py``, and evaluation configs in ``algorithm/stack_predict.py``. You may overwrite the configs using command line args. For example:\n\n```bash\npython algorithm/stack_train.py --dataset=omi-1 --model.z_dim=5 --train.batch_size=128\npython algorithm/stack_predict.py --load_model_dir=./results/stack_train/ --test_batch_size=100\n```\n\n**Run on your own dataset**\n\n1. Put your train/test/label files under ``data/processed`` folder. e.g., ``ds_train.pkl``, ``ds_test.pkl``, ``ds_test_label.pkl`` with shape ``(train_length, feature_dim)``, ``(test_length, feature_dim)``, ``(test_length,)``, respectively. \n2. Put the interpretation files (optional) under ``data/interpretation_label`` folder.\n3. Edit ``get_data_dim`` in ``algorithm/utils.py`` to add your dataset info.\n4. Run the code following the instructions above.\n\n**Results**\n\nAfter running the algorithm, the results are shown in the ``results`` folder. The main results are:\n\n```bash\nModel: results/stack_train/result_params/\nTraining config: results/stack_train/config.json\nTesting config: results/stack_predict/config.json\nTesting statistics: results/stack_predict/result.json\n```\n\n\n\nIf you find this code useful for your research, please cite our paper:\n\n```bibTex\n@inproceedings{li2021multivariate,\n  title={Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding},\n  author={Li, Zhihan and Zhao, Youjian and Han, Jiaqi and Su, Ya and Jiao, Rui and Wen, Xidao and Pei, Dan},\n  booktitle={Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery \\& Data Mining},\n  pages={3220--3230},\n  year={2021}\n}\n```\n\n"
  },
  {
    "path": "algorithm/InterFusion.py",
    "content": "from enum import Enum\nfrom typing import Optional, List\n\nimport logging\nimport tensorflow as tf\nfrom tensorflow.contrib.rnn import static_rnn, static_bidirectional_rnn\nfrom tensorflow.contrib.framework import arg_scope\nimport tfsnippet as spt\nfrom tfsnippet.bayes import BayesianNet\nfrom tfsnippet.utils import (instance_reuse,\n                             VarScopeObject,\n                             reopen_variable_scope)\nfrom tfsnippet.distributions import FlowDistribution, Normal\nfrom tfsnippet.layers import l2_regularizer\n\nimport mltk\nfrom algorithm.recurrent_distribution import RecurrentDistribution\nfrom algorithm.real_nvp import dense_real_nvp\nfrom algorithm.conv1d_ import conv1d, deconv1d\n\n\nclass RNNCellType(str, Enum):\n    GRU = 'GRU'\n    LSTM = 'LSTM'\n    Basic = 'Basic'\n\n\nclass ModelConfig(mltk.Config):\n    x_dim: int = -1\n    z_dim: int = 3\n    u_dim: int = 1\n    window_length = 100\n    output_shape: List[int] = [25, 25, 50, 50, 100]\n    z2_dim: int = 13\n    l2_reg = 0.0001\n    posterior_flow_type: Optional[str] = mltk.config_field(choices=['rnvp', 'nf'], default='rnvp')\n    # can be 'rnvp' for RealNVP, 'nf' for planarNF, None for not using posterior flow.\n    posterior_flow_layers = 20\n    rnn_cell: RNNCellType = RNNCellType.GRU  # can be 'GRU', 'LSTM' or 'Basic'\n    rnn_hidden_units = 500\n    use_leaky_relu = False\n    use_bidirectional_rnn = False       # whether to use bidirectional rnn or not\n    use_self_attention = False          # whether to use self-attention on hidden states before infer qz or not.\n    unified_px_logstd = False\n    dropout_feature = False             # dropout on the features in arnn\n    logstd_min = -5.\n    logstd_max = 2.\n    use_prior_flow = False              # If True, use RealNVP prior flow to enhance the representation of p(z).\n    prior_flow_layers = 20\n\n    connect_qz = True\n    connect_pz = True\n\n\n# The final InterFusion model.\nclass MTSAD(VarScopeObject):\n\n    def __init__(self, config: ModelConfig, name=None, scope=None):\n        self.config = config\n        super(MTSAD, self).__init__(name=name, scope=scope)\n\n        with reopen_variable_scope(self.variable_scope):\n            if self.config.rnn_cell == RNNCellType.Basic:\n                self.d_fw_cell = tf.nn.rnn_cell.BasicRNNCell(self.config.rnn_hidden_units, name='d_fw_cell')\n                self.a_fw_cell = tf.nn.rnn_cell.BasicRNNCell(self.config.rnn_hidden_units, name='a_fw_cell')\n                if self.config.use_bidirectional_rnn:\n                    self.d_bw_cell = tf.nn.rnn_cell.BasicRNNCell(self.config.rnn_hidden_units, name='d_bw_cell')\n                    self.a_bw_cell = tf.nn.rnn_cell.BasicRNNCell(self.config.rnn_hidden_units, name='a_bw_cell')\n            elif self.config.rnn_cell == RNNCellType.LSTM:\n                self.d_fw_cell = tf.nn.rnn_cell.LSTMCell(self.config.rnn_hidden_units, name='d_fw_cell')\n                self.a_fw_cell = tf.nn.rnn_cell.LSTMCell(self.config.rnn_hidden_units, name='a_fw_cell')\n                if self.config.use_bidirectional_rnn:\n                    self.d_bw_cell = tf.nn.rnn_cell.LSTMCell(self.config.rnn_hidden_units, name='d_bw_cell')\n                    self.a_bw_cell = tf.nn.rnn_cell.LSTMCell(self.config.rnn_hidden_units, name='a_bw_cell')\n            elif self.config.rnn_cell == RNNCellType.GRU:\n                self.d_fw_cell = tf.nn.rnn_cell.GRUCell(self.config.rnn_hidden_units, name='d_fw_cell')\n                self.a_fw_cell = tf.nn.rnn_cell.GRUCell(self.config.rnn_hidden_units, name='a_fw_cell')\n                if self.config.use_bidirectional_rnn:\n                    self.d_bw_cell = tf.nn.rnn_cell.GRUCell(self.config.rnn_hidden_units, name='d_bw_cell')\n                    self.a_bw_cell = tf.nn.rnn_cell.GRUCell(self.config.rnn_hidden_units, name='a_bw_cell')\n            else:\n                raise ValueError('rnn cell must be one of GRU, LSTM or Basic.')\n\n            if self.config.posterior_flow_type == 'nf':\n                self.posterior_flow = spt.layers.planar_normalizing_flows(n_layers=self.config.posterior_flow_layers,\n                                                                          scope='posterior_flow')\n            elif self.config.posterior_flow_type == 'rnvp':\n                self.posterior_flow = dense_real_nvp(flow_depth=self.config.posterior_flow_layers,\n                                                     activation=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                                                     kernel_regularizer=l2_regularizer(self.config.l2_reg),\n                                                     scope='posterior_flow')\n            else:\n                self.posterior_flow = None\n\n            if self.config.use_prior_flow:\n                self.prior_flow = dense_real_nvp(flow_depth=self.config.prior_flow_layers,\n                                                 activation=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                                                 kernel_regularizer=l2_regularizer(self.config.l2_reg),\n                                                 is_prior_flow=True,\n                                                 scope='prior_flow')\n            else:\n                self.prior_flow = None\n\n    def _my_rnn_net(self, x, window_length, fw_cell, bw_cell=None,\n                    time_axis=1, use_bidirectional_rnn=False):\n        \"\"\"\n        Get the base rnn model.\n        :param x: The rnn input.\n        :param window_length: The window length of input along time axis.\n        :param fw_cell: Forward rnn cell.\n        :param bw_cell: Optional. Backward rnn cell, only use when config.use_bidirectional_rnn=True.\n        :param time_axis: Which is the time axis in input x, default 1.\n        :param use_bidirectional_rnn: Whether or not use bidirectional rnn. Default false.\n        :return: Tensor (batch_size, window_length, rnn_hidden_units). The output of rnn.\n        \"\"\"\n\n        x = tf.unstack(value=x, num=window_length, axis=time_axis)\n\n        if use_bidirectional_rnn:\n            outputs, _, _ = static_bidirectional_rnn(fw_cell, bw_cell, x, dtype=tf.float32)\n        else:\n            outputs, _ = static_rnn(fw_cell, x, dtype=tf.float32)\n\n        outputs = tf.stack(outputs, axis=time_axis)     # (batch_size, window_length, rnn_hidden_units)\n        return outputs\n\n    @instance_reuse\n    def a_rnn_net(self, x, window_length, time_axis=1,\n                  use_bidirectional_rnn=False, use_self_attention=False, is_training=False):\n        \"\"\"\n        Reverse rnn network a, capture the future information in qnet.\n        \"\"\"\n        def dropout_fn(input):\n            return tf.layers.dropout(input, rate=.5, training=is_training)\n\n        flag = False\n        if len(x.shape) == 4:               # (n_samples, batch_size, window_length, x_dim)\n            x, s1, s2 = spt.ops.flatten_to_ndims(x, 3)\n            flag = True\n        elif len(x.shape) != 3:\n            logging.error('rnn input shape error.')\n\n        # reverse the input sequence\n        reversed_x = tf.reverse(x, axis=[time_axis])\n\n        if use_bidirectional_rnn:\n            reversed_outputs = self._my_rnn_net(x=reversed_x, window_length=window_length, fw_cell=self.a_fw_cell,\n                                       bw_cell=self.a_bw_cell, time_axis=time_axis,\n                                       use_bidirectional_rnn=use_bidirectional_rnn)\n        else:\n            reversed_outputs = self._my_rnn_net(x=reversed_x, window_length=window_length, fw_cell=self.a_fw_cell,\n                                       time_axis=time_axis, use_bidirectional_rnn=use_bidirectional_rnn)\n\n        outputs = tf.reverse(reversed_outputs, axis=[time_axis])\n\n        # self attention\n        if use_self_attention:\n            outputs1 = spt.layers.dense(outputs, 500, activation_fn=tf.nn.tanh, use_bias=True, scope='arnn_attention_dense1')\n            outputs1 = tf.nn.softmax(spt.layers.dense(outputs1, window_length,\n                                                      use_bias=False, scope='arnn_attention_dense2'), axis=1)\n            M_t = tf.matmul(tf.transpose(outputs, perm=[0, 2, 1]), outputs1)\n            outputs = tf.transpose(M_t, perm=[0, 2, 1])\n\n        # feature extraction layers\n        outputs = spt.layers.dense(outputs, units=500, activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                                   kernel_regularizer=l2_regularizer(self.config.l2_reg), scope='arnn_feature_dense1')\n        if self.config.dropout_feature:\n            outputs = dropout_fn(outputs)\n        outputs = spt.layers.dense(outputs, units=500, activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                                   kernel_regularizer=l2_regularizer(self.config.l2_reg), scope='arnn_feature_dense2')\n        if self.config.dropout_feature:\n            outputs = dropout_fn(outputs)\n\n        if flag:\n            outputs = spt.ops.unflatten_from_ndims(outputs, s1, s2)\n\n        return outputs\n\n    @instance_reuse\n    def qz_mean_layer(self, x):\n        return spt.layers.dense(x, units=self.config.z_dim, scope='qz_mean')\n\n    @instance_reuse\n    def qz_logstd_layer(self, x):\n        return tf.clip_by_value(spt.layers.dense(x, units=self.config.z_dim, scope='qz_logstd'),\n                                clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n\n    @instance_reuse\n    def pz_mean_layer(self, x):\n        return spt.layers.dense(x, units=self.config.z_dim, scope='pz_mean')\n\n    @instance_reuse\n    def pz_logstd_layer(self, x):\n        return tf.clip_by_value(spt.layers.dense(x, units=self.config.z_dim, scope='pz_logstd'),\n                                clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n\n    @instance_reuse\n    def hz2_deconv(self, z2):\n        with arg_scope([deconv1d],\n                       kernel_size=5,\n                       activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                       kernel_regularizer=l2_regularizer(self.config.l2_reg)):\n            h_z = deconv1d(z2, out_channels=self.config.x_dim, output_shape=self.config.output_shape[0], strides=2)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[1], strides=1)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[2], strides=2)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[3], strides=1)\n            h_z2 = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[4], strides=2)\n        return h_z2\n\n    @instance_reuse\n    def q_net(self, x, observed=None, u=None, n_z=None, is_training=False):\n        # vs.name = self.variable_scope.name + \"/q_net\"\n        logging.info('q_net builder: %r', locals())\n\n        net = BayesianNet(observed=observed)\n\n        def dropout_fn(input):\n            return tf.layers.dropout(input, rate=.5, training=is_training)\n\n        # use the pretrained z2 which compress along the time dimension\n        qz2_mean, qz2_logstd = self.h_for_qz(x, is_training=is_training)\n\n        qz2_distribution = Normal(mean=qz2_mean, logstd=qz2_logstd)\n\n        qz2_distribution = qz2_distribution.batch_ndims_to_value(2)\n\n        z2 = net.add('z2', qz2_distribution, n_samples=n_z, is_reparameterized=True)\n\n        # d_{1:t} from deconv\n        h_z = self.h_for_px(z2)\n\n        # a_{1:t}, (batch_size, window_length, dense_hidden_units)\n        arnn_out = self.a_rnn_net(h_z, window_length=self.config.window_length,\n                                  use_bidirectional_rnn=self.config.use_bidirectional_rnn,\n                                  use_self_attention=self.config.use_self_attention,\n                                  is_training=is_training)\n\n        if self.config.connect_qz:\n            qz_distribution = RecurrentDistribution(arnn_out,\n                                                    mean_layer=self.qz_mean_layer, logstd_layer=self.qz_logstd_layer,\n                                                    z_dim=self.config.z_dim, window_length=self.config.window_length)\n        else:\n            qz_mean = spt.layers.dense(arnn_out, units=self.config.z_dim, scope='qz1_mean')\n            qz_logstd = tf.clip_by_value(spt.layers.dense(arnn_out, units=self.config.z_dim, scope='qz1_logstd'),\n                                         clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n            qz_distribution = Normal(mean=qz_mean, logstd=qz_logstd)\n\n        if self.posterior_flow is not None:\n            qz_distribution = FlowDistribution(distribution=qz_distribution, flow=self.posterior_flow).batch_ndims_to_value(1)\n        else:\n            qz_distribution = qz_distribution.batch_ndims_to_value(2)\n\n        z1 = net.add('z1', qz_distribution, is_reparameterized=True)\n\n        return net\n\n    @instance_reuse\n    def p_net(self, observed=None, u=None, n_z=None, is_training=False):\n        logging.info('p_net builder: %r', locals())\n\n        net = BayesianNet(observed=observed)\n\n        pz2_distribution = Normal(mean=tf.zeros([self.config.z2_dim, self.config.x_dim]),\n                                 logstd=tf.zeros([self.config.z2_dim, self.config.x_dim])).batch_ndims_to_value(2)\n\n        z2 = net.add('z2', pz2_distribution, n_samples=n_z, is_reparameterized=True)\n\n        # e_{1:t} from deconv, shared params\n        h_z2 = self.h_for_px(z2)\n\n        if self.config.connect_pz:\n            pz_distribution = RecurrentDistribution(h_z2,\n                                                    mean_layer=self.pz_mean_layer, logstd_layer=self.pz_logstd_layer,\n                                                    z_dim=self.config.z_dim, window_length=self.config.window_length)\n        else:\n            # non-recurrent pz\n            pz_mean = spt.layers.dense(h_z2, units=self.config.z_dim, scope='pz_mean')\n            pz_logstd = tf.clip_by_value(spt.layers.dense(h_z2,\n                                                          units=self.config.z_dim, scope='pz_logstd'),\n                                                          clip_value_min=self.config.logstd_min,\n                                                          clip_value_max=self.config.logstd_max)\n            pz_distribution = Normal(mean=pz_mean, logstd=pz_logstd)\n\n        if self.prior_flow is not None:\n            pz_distribution = FlowDistribution(distribution=pz_distribution, flow=self.prior_flow).batch_ndims_to_value(1)\n        else:\n            pz_distribution = pz_distribution.batch_ndims_to_value(2)\n\n        z1 = net.add('z1', pz_distribution, is_reparameterized=True)\n\n        h_z1 = spt.layers.dense(z1, units=self.config.x_dim)\n\n        h_z = spt.ops.broadcast_concat(h_z1, h_z2, axis=-1)\n\n        h_z = spt.layers.dense(h_z, units=500, activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                               kernel_regularizer=l2_regularizer(self.config.l2_reg), scope='feature_dense1')\n\n        h_z = spt.layers.dense(h_z, units=500, activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                               kernel_regularizer=l2_regularizer(self.config.l2_reg), scope='feature_dense2')\n\n        x_mean = spt.layers.dense(h_z, units=self.config.x_dim, scope='x_mean')\n        if self.config.unified_px_logstd:\n            x_logstd = tf.clip_by_value(\n                tf.get_variable(name='x_logstd', shape=(), trainable=True, dtype=tf.float32,\n                                initializer=tf.constant_initializer(-1., dtype=tf.float32)),\n                                clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n        else:\n            x_logstd = tf.clip_by_value(spt.layers.dense(h_z, units=self.config.x_dim, scope='x_logstd'),\n                                        clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n\n        x = net.add('x',\n                    Normal(mean=x_mean, logstd=x_logstd).batch_ndims_to_value(2),\n                    is_reparameterized=True)\n\n        return net\n\n    def reconstruct(self, x, u, mask, n_z=None):\n        with tf.name_scope('model.reconstruct'):\n            qnet = self.q_net(x=x, u=u, n_z=n_z)\n            pnet = self.p_net(observed={'z1': qnet['z1'], 'z2': qnet['z2']}, u=u)\n        return pnet['x']\n\n    def get_score(self, x_embed, x_eval, u, n_z=None):\n        with tf.name_scope('model.get_score'):\n            qnet = self.q_net(x=x_embed, u=u, n_z=n_z)\n            pnet = self.p_net(observed={'z1': qnet['z1'], 'z2': qnet['z2']}, u=u)\n            score = pnet['x'].distribution.base_distribution.log_prob(x_eval)\n            recons_mean = pnet['x'].distribution.base_distribution.mean\n            recons_std = pnet['x'].distribution.base_distribution.std\n            if n_z is not None:\n                score = tf.reduce_mean(score, axis=0)\n                recons_mean = tf.reduce_mean(recons_mean, axis=0)\n                recons_std = tf.reduce_mean(recons_std, axis=0)\n        return score, recons_mean, recons_std\n\n    @instance_reuse\n    def h_for_qz(self, x, is_training=False):\n        with arg_scope([conv1d],\n                       kernel_size=5,\n                       activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                       kernel_regularizer=l2_regularizer(self.config.l2_reg)):\n            h_x = conv1d(x, out_channels=self.config.x_dim, strides=2)   # 50\n            h_x = conv1d(h_x, out_channels=self.config.x_dim)\n            h_x = conv1d(h_x, out_channels=self.config.x_dim, strides=2)        # 25\n            h_x = conv1d(h_x, out_channels=self.config.x_dim)\n            h_x = conv1d(h_x, out_channels=self.config.x_dim, strides=2)        # 13\n\n        qz_mean = conv1d(h_x, kernel_size=1, out_channels=self.config.x_dim)\n        qz_logstd = conv1d(h_x, kernel_size=1, out_channels=self.config.x_dim)\n        qz_logstd = tf.clip_by_value(qz_logstd, clip_value_min=self.config.logstd_min,\n                                     clip_value_max=self.config.logstd_max)\n        return qz_mean, qz_logstd\n\n    @instance_reuse\n    def h_for_px(self, z):\n        with arg_scope([deconv1d],\n                       kernel_size=5,\n                       activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                       kernel_regularizer=l2_regularizer(self.config.l2_reg)):\n            h_z = deconv1d(z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[0], strides=2)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[1], strides=1)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[2], strides=2)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[3], strides=1)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[4], strides=2)\n        return h_z\n\n    @instance_reuse\n    def pretrain_q_net(self, x, observed=None, n_z=None, is_training=False):\n        # vs.name = self.variable_scope.name + \"/q_net\"\n        logging.info('pretrain_q_net builder: %r', locals())\n\n        net = BayesianNet(observed=observed)\n\n        def dropout_fn(input):\n            return tf.layers.dropout(input, rate=.5, training=is_training)\n\n        qz_mean, qz_logstd = self.h_for_qz(x, is_training=is_training)\n\n        qz_distribution = Normal(mean=qz_mean, logstd=qz_logstd)\n\n        qz_distribution = qz_distribution.batch_ndims_to_value(2)\n\n        z = net.add('z', qz_distribution, n_samples=n_z, is_reparameterized=True)\n\n        return net\n\n    @instance_reuse\n    def pretrain_p_net(self, observed=None, n_z=None, is_training=False):\n        logging.info('p_net builder: %r', locals())\n\n        net = BayesianNet(observed=observed)\n\n        pz_distribution = Normal(mean=tf.zeros([self.config.z2_dim, self.config.x_dim]),\n                                 logstd=tf.zeros([self.config.z2_dim, self.config.x_dim]))\n\n        pz_distribution = pz_distribution.batch_ndims_to_value(2)\n\n        z = net.add('z',\n                    pz_distribution,\n                    n_samples=n_z, is_reparameterized=True)\n\n        h_z = self.h_for_px(z)\n\n        px_mean = conv1d(h_z, kernel_size=1, out_channels=self.config.x_dim, scope='pre_px_mean')\n        px_logstd = conv1d(h_z, kernel_size=1, out_channels=self.config.x_dim, scope='pre_px_logstd')\n        px_logstd = tf.clip_by_value(px_logstd, clip_value_min=self.config.logstd_min,\n                                     clip_value_max=self.config.logstd_max)\n\n        x = net.add('x',\n                    Normal(mean=px_mean, logstd=px_logstd).batch_ndims_to_value(2),\n                    is_reparameterized=True)\n\n        return net\n"
  },
  {
    "path": "algorithm/InterFusion_swat.py",
    "content": "from enum import Enum\nfrom typing import Optional, List\n\nimport logging\nimport tensorflow as tf\nfrom tensorflow.contrib.rnn import static_rnn, static_bidirectional_rnn\nfrom tensorflow.contrib.framework import arg_scope\nimport tfsnippet as spt\nfrom tfsnippet.bayes import BayesianNet\nfrom tfsnippet.utils import (instance_reuse,\n                             VarScopeObject,\n                             reopen_variable_scope)\nfrom tfsnippet.distributions import FlowDistribution, Normal\nfrom tfsnippet.layers import l2_regularizer\n\nimport mltk\nfrom algorithm.recurrent_distribution import RecurrentDistribution\nfrom algorithm.real_nvp import dense_real_nvp\nfrom algorithm.conv1d_ import conv1d, deconv1d\n\n\nclass RNNCellType(str, Enum):\n    GRU = 'GRU'\n    LSTM = 'LSTM'\n    Basic = 'Basic'\n\n\nclass ModelConfig(mltk.Config):\n    x_dim: int = -1\n    z_dim: int = 3\n    u_dim: int = 1\n    window_length = 30\n    output_shape: List[int] = [15, 15, 30]\n    z2_dim: int = 8\n    l2_reg = 0.0001\n    posterior_flow_type: Optional[str] = mltk.config_field(choices=['rnvp', 'nf'], default='rnvp')\n    # can be 'rnvp' for RealNVP, 'nf' for planarNF, None for not using posterior flow.\n    posterior_flow_layers = 20\n    rnn_cell: RNNCellType = RNNCellType.GRU  # can be 'GRU', 'LSTM' or 'Basic'\n    rnn_hidden_units = 500\n    use_leaky_relu = False\n    use_bidirectional_rnn = False       # whether to use bidirectional rnn or not\n    use_self_attention = False          # whether to use self-attention on hidden states before infer qz or not.\n    unified_px_logstd = False\n    dropout_feature = False             # dropout on the features in arnn\n    logstd_min = -5.\n    logstd_max = 2.\n    use_prior_flow = False              # If True, use RealNVP prior flow to enhance the representation of p(z).\n    prior_flow_layers = 20\n\n    connect_qz = True\n    connect_pz = True\n\n\n# InterFusion model for SWaT & WADI (differ in num of layers)\nclass MTSAD_SWAT(VarScopeObject):\n\n    def __init__(self, config: ModelConfig, name=None, scope=None):\n        self.config = config\n        super(MTSAD_SWAT, self).__init__(name=name, scope=scope)\n\n        with reopen_variable_scope(self.variable_scope):\n            if self.config.rnn_cell == RNNCellType.Basic:\n                self.d_fw_cell = tf.nn.rnn_cell.BasicRNNCell(self.config.rnn_hidden_units, name='d_fw_cell')\n                self.a_fw_cell = tf.nn.rnn_cell.BasicRNNCell(self.config.rnn_hidden_units, name='a_fw_cell')\n                if self.config.use_bidirectional_rnn:\n                    self.d_bw_cell = tf.nn.rnn_cell.BasicRNNCell(self.config.rnn_hidden_units, name='d_bw_cell')\n                    self.a_bw_cell = tf.nn.rnn_cell.BasicRNNCell(self.config.rnn_hidden_units, name='a_bw_cell')\n            elif self.config.rnn_cell == RNNCellType.LSTM:\n                self.d_fw_cell = tf.nn.rnn_cell.LSTMCell(self.config.rnn_hidden_units, name='d_fw_cell')\n                self.a_fw_cell = tf.nn.rnn_cell.LSTMCell(self.config.rnn_hidden_units, name='a_fw_cell')\n                if self.config.use_bidirectional_rnn:\n                    self.d_bw_cell = tf.nn.rnn_cell.LSTMCell(self.config.rnn_hidden_units, name='d_bw_cell')\n                    self.a_bw_cell = tf.nn.rnn_cell.LSTMCell(self.config.rnn_hidden_units, name='a_bw_cell')\n            elif self.config.rnn_cell == RNNCellType.GRU:\n                self.d_fw_cell = tf.nn.rnn_cell.GRUCell(self.config.rnn_hidden_units, name='d_fw_cell')\n                self.a_fw_cell = tf.nn.rnn_cell.GRUCell(self.config.rnn_hidden_units, name='a_fw_cell')\n                if self.config.use_bidirectional_rnn:\n                    self.d_bw_cell = tf.nn.rnn_cell.GRUCell(self.config.rnn_hidden_units, name='d_bw_cell')\n                    self.a_bw_cell = tf.nn.rnn_cell.GRUCell(self.config.rnn_hidden_units, name='a_bw_cell')\n            else:\n                raise ValueError('rnn cell must be one of GRU, LSTM or Basic.')\n\n            if self.config.posterior_flow_type == 'nf':\n                self.posterior_flow = spt.layers.planar_normalizing_flows(n_layers=self.config.posterior_flow_layers,\n                                                                          scope='posterior_flow')\n            elif self.config.posterior_flow_type == 'rnvp':\n                self.posterior_flow = dense_real_nvp(flow_depth=self.config.posterior_flow_layers,\n                                                     activation=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                                                     kernel_regularizer=l2_regularizer(self.config.l2_reg),\n                                                     scope='posterior_flow')\n            else:\n                self.posterior_flow = None\n\n            if self.config.use_prior_flow:\n                self.prior_flow = dense_real_nvp(flow_depth=self.config.prior_flow_layers,\n                                                 activation=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                                                 kernel_regularizer=l2_regularizer(self.config.l2_reg),\n                                                 is_prior_flow=True,\n                                                 scope='prior_flow')\n            else:\n                self.prior_flow = None\n\n    def _my_rnn_net(self, x, window_length, fw_cell, bw_cell=None,\n                    time_axis=1, use_bidirectional_rnn=False):\n        \"\"\"\n        Get the base rnn model for d-net and a-net.\n        :param x: The rnn input.\n        :param window_length: The window length of input along time axis.\n        :param fw_cell: Forward rnn cell.\n        :param bw_cell: Optional. Backward rnn cell, only use when config.use_bidirectional_rnn=True.\n        :param time_axis: Which is the time axis in input x, default 1.\n        :param use_bidirectional_rnn: Whether or not use bidirectional rnn. Default false.\n        :return: Tensor (batch_size, window_length, rnn_hidden_units). The output of rnn.\n        \"\"\"\n\n        x = tf.unstack(value=x, num=window_length, axis=time_axis)\n\n        if use_bidirectional_rnn:\n            outputs, _, _ = static_bidirectional_rnn(fw_cell, bw_cell, x, dtype=tf.float32)\n        else:\n            outputs, _ = static_rnn(fw_cell, x, dtype=tf.float32)\n\n        outputs = tf.stack(outputs, axis=time_axis)     # (batch_size, window_length, rnn_hidden_units)\n        return outputs\n\n    @instance_reuse\n    def a_rnn_net(self, x, window_length, time_axis=1,\n                  use_bidirectional_rnn=False, use_self_attention=False, is_training=False):\n        \"\"\"\n        Reverse rnn network a, capture the future information in qnet.\n        \"\"\"\n        def dropout_fn(input):\n            return tf.layers.dropout(input, rate=.5, training=is_training)\n\n        flag = False\n        if len(x.shape) == 4:               # (n_samples, batch_size, window_length, x_dim)\n            x, s1, s2 = spt.ops.flatten_to_ndims(x, 3)\n            flag = True\n        elif len(x.shape) != 3:\n            logging.error('rnn input shape error.')\n\n        # reverse the input sequence\n        reversed_x = tf.reverse(x, axis=[time_axis])\n\n        if use_bidirectional_rnn:\n            reversed_outputs = self._my_rnn_net(x=reversed_x, window_length=window_length, fw_cell=self.a_fw_cell,\n                                       bw_cell=self.a_bw_cell, time_axis=time_axis,\n                                       use_bidirectional_rnn=use_bidirectional_rnn)\n        else:\n            reversed_outputs = self._my_rnn_net(x=reversed_x, window_length=window_length, fw_cell=self.a_fw_cell,\n                                       time_axis=time_axis, use_bidirectional_rnn=use_bidirectional_rnn)\n\n        outputs = tf.reverse(reversed_outputs, axis=[time_axis])\n\n        # self attention\n        if use_self_attention:\n            outputs1 = spt.layers.dense(outputs, 500, activation_fn=tf.nn.tanh, use_bias=True, scope='arnn_attention_dense1')\n            outputs1 = tf.nn.softmax(spt.layers.dense(outputs1, window_length,\n                                                      use_bias=False, scope='arnn_attention_dense2'), axis=1)\n            M_t = tf.matmul(tf.transpose(outputs, perm=[0, 2, 1]), outputs1)\n            outputs = tf.transpose(M_t, perm=[0, 2, 1])\n\n        # feature extraction layers\n        outputs = spt.layers.dense(outputs, units=500, activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                                   kernel_regularizer=l2_regularizer(self.config.l2_reg), scope='arnn_feature_dense1')\n        if self.config.dropout_feature:\n            outputs = dropout_fn(outputs)\n        outputs = spt.layers.dense(outputs, units=500, activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                                   kernel_regularizer=l2_regularizer(self.config.l2_reg), scope='arnn_feature_dense2')\n        if self.config.dropout_feature:\n            outputs = dropout_fn(outputs)\n\n        if flag:\n            outputs = spt.ops.unflatten_from_ndims(outputs, s1, s2)\n\n        return outputs\n\n    @instance_reuse\n    def qz_mean_layer(self, x):\n        return spt.layers.dense(x, units=self.config.z_dim, scope='qz_mean')\n\n    @instance_reuse\n    def qz_logstd_layer(self, x):\n        return tf.clip_by_value(spt.layers.dense(x, units=self.config.z_dim, scope='qz_logstd'),\n                                clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n\n    @instance_reuse\n    def pz_mean_layer(self, x):\n        return spt.layers.dense(x, units=self.config.z_dim, scope='pz_mean')\n\n    @instance_reuse\n    def pz_logstd_layer(self, x):\n        return tf.clip_by_value(spt.layers.dense(x, units=self.config.z_dim, scope='pz_logstd'),\n                                clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n\n    @instance_reuse\n    def hz2_deconv(self, z2):\n        with arg_scope([deconv1d],\n                       kernel_size=5,\n                       activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                       kernel_regularizer=l2_regularizer(self.config.l2_reg)):\n            h_z = deconv1d(z2, out_channels=self.config.x_dim, output_shape=self.config.output_shape[0], strides=2)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[1], strides=1)\n            h_z2 = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[2], strides=2)\n        return h_z2\n\n    @instance_reuse\n    def q_net(self, x, observed=None, u=None, n_z=None, is_training=False):\n        # vs.name = self.variable_scope.name + \"/q_net\"\n        logging.info('q_net builder: %r', locals())\n\n        net = BayesianNet(observed=observed)\n\n        def dropout_fn(input):\n            return tf.layers.dropout(input, rate=.5, training=is_training)\n\n        # use the pretrained z2 which compress along the time dimension\n        qz2_mean, qz2_logstd = self.h_for_qz(x, is_training=is_training)\n\n        qz2_distribution = Normal(mean=qz2_mean, logstd=qz2_logstd)\n\n        qz2_distribution = qz2_distribution.batch_ndims_to_value(2)\n\n        z2 = net.add('z2', qz2_distribution, n_samples=n_z, is_reparameterized=True)\n\n        # d_{1:t} from deconv\n        h_z = self.h_for_px(z2)\n\n        # a_{1:t}, (batch_size, window_length, dense_hidden_units)\n        arnn_out = self.a_rnn_net(h_z, window_length=self.config.window_length,\n                                  use_bidirectional_rnn=self.config.use_bidirectional_rnn,\n                                  use_self_attention=self.config.use_self_attention,\n                                  is_training=is_training)\n\n        if self.config.connect_qz:\n            qz_distribution = RecurrentDistribution(arnn_out,\n                                                    mean_layer=self.qz_mean_layer, logstd_layer=self.qz_logstd_layer,\n                                                    z_dim=self.config.z_dim, window_length=self.config.window_length)\n        else:\n            qz_mean = spt.layers.dense(arnn_out, units=self.config.z_dim, scope='qz1_mean')\n            qz_logstd = tf.clip_by_value(spt.layers.dense(arnn_out, units=self.config.z_dim, scope='qz1_logstd'),\n                                         clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n            qz_distribution = Normal(mean=qz_mean, logstd=qz_logstd)\n\n        if self.posterior_flow is not None:\n            qz_distribution = FlowDistribution(distribution=qz_distribution, flow=self.posterior_flow).batch_ndims_to_value(1)\n        else:\n            qz_distribution = qz_distribution.batch_ndims_to_value(2)\n\n        z1 = net.add('z1', qz_distribution, is_reparameterized=True)\n\n        return net\n\n    @instance_reuse\n    def p_net(self, observed=None, u=None, n_z=None, is_training=False):\n        logging.info('p_net builder: %r', locals())\n\n        net = BayesianNet(observed=observed)\n\n        pz2_distribution = Normal(mean=tf.zeros([self.config.z2_dim, self.config.x_dim]),\n                                 logstd=tf.zeros([self.config.z2_dim, self.config.x_dim])).batch_ndims_to_value(2)\n\n        z2 = net.add('z2', pz2_distribution, n_samples=n_z, is_reparameterized=True)\n\n        # e_{1:t} from deconv, shared params\n        h_z2 = self.h_for_px(z2)\n\n        if self.config.connect_pz:\n            pz_distribution = RecurrentDistribution(h_z2,\n                                                    mean_layer=self.pz_mean_layer, logstd_layer=self.pz_logstd_layer,\n                                                    z_dim=self.config.z_dim, window_length=self.config.window_length)\n        else:\n            # non-recurrent pz\n            pz_mean = spt.layers.dense(h_z2, units=self.config.z_dim, scope='pz_mean')\n            pz_logstd = tf.clip_by_value(spt.layers.dense(h_z2,\n                                                          units=self.config.z_dim, scope='pz_logstd'),\n                                                          clip_value_min=self.config.logstd_min,\n                                                          clip_value_max=self.config.logstd_max)\n            pz_distribution = Normal(mean=pz_mean, logstd=pz_logstd)\n\n        if self.prior_flow is not None:\n            pz_distribution = FlowDistribution(distribution=pz_distribution, flow=self.prior_flow).batch_ndims_to_value(1)\n        else:\n            pz_distribution = pz_distribution.batch_ndims_to_value(2)\n\n        z1 = net.add('z1', pz_distribution, is_reparameterized=True)\n\n        h_z1 = spt.layers.dense(z1, units=self.config.x_dim)\n\n        h_z = spt.ops.broadcast_concat(h_z1, h_z2, axis=-1)\n\n        h_z = spt.layers.dense(h_z, units=500, activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                               kernel_regularizer=l2_regularizer(self.config.l2_reg), scope='feature_dense1')\n\n        h_z = spt.layers.dense(h_z, units=500, activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                               kernel_regularizer=l2_regularizer(self.config.l2_reg), scope='feature_dense2')\n\n        x_mean = spt.layers.dense(h_z, units=self.config.x_dim, scope='x_mean')\n        if self.config.unified_px_logstd:\n            x_logstd = tf.clip_by_value(\n                tf.get_variable(name='x_logstd', shape=(), trainable=True, dtype=tf.float32,\n                                initializer=tf.constant_initializer(-1., dtype=tf.float32)),\n                                clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n        else:\n            x_logstd = tf.clip_by_value(spt.layers.dense(h_z, units=self.config.x_dim, scope='x_logstd'),\n                                        clip_value_min=self.config.logstd_min, clip_value_max=self.config.logstd_max)\n\n        x = net.add('x',\n                    Normal(mean=x_mean, logstd=x_logstd).batch_ndims_to_value(2),\n                    is_reparameterized=True)\n\n        return net\n\n    def reconstruct(self, x, u, mask, n_z=None):\n        with tf.name_scope('model.reconstruct'):\n            qnet = self.q_net(x=x, u=u, n_z=n_z)\n            pnet = self.p_net(observed={'z1': qnet['z1'], 'z2': qnet['z2']}, u=u)\n        return pnet['x']\n\n    def get_score(self, x_embed, x_eval, u, n_z=None):\n        with tf.name_scope('model.get_score'):\n            qnet = self.q_net(x=x_embed, u=u, n_z=n_z)\n            pnet = self.p_net(observed={'z1': qnet['z1'], 'z2': qnet['z2']}, u=u)\n            score = pnet['x'].distribution.base_distribution.log_prob(x_eval)\n            recons_mean = pnet['x'].distribution.base_distribution.mean\n            recons_std = pnet['x'].distribution.base_distribution.std\n            if n_z is not None:\n                score = tf.reduce_mean(score, axis=0)\n                recons_mean = tf.reduce_mean(recons_mean, axis=0)\n                recons_std = tf.reduce_mean(recons_std, axis=0)\n        return score, recons_mean, recons_std\n\n    @instance_reuse\n    def h_for_qz(self, x, is_training=False):\n        with arg_scope([conv1d],\n                       kernel_size=5,\n                       activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                       kernel_regularizer=l2_regularizer(self.config.l2_reg)):\n            h_x = conv1d(x, out_channels=self.config.x_dim, strides=2)   # 15\n            h_x = conv1d(h_x, out_channels=self.config.x_dim)\n            h_x = conv1d(h_x, out_channels=self.config.x_dim, strides=2)        # 8\n\n        qz_mean = conv1d(h_x, kernel_size=1, out_channels=self.config.x_dim)\n        qz_logstd = conv1d(h_x, kernel_size=1, out_channels=self.config.x_dim)\n        qz_logstd = tf.clip_by_value(qz_logstd, clip_value_min=self.config.logstd_min,\n                                     clip_value_max=self.config.logstd_max)\n        return qz_mean, qz_logstd\n\n    @instance_reuse\n    def h_for_px(self, z):\n        with arg_scope([deconv1d],\n                       kernel_size=5,\n                       activation_fn=tf.nn.leaky_relu if self.config.use_leaky_relu else tf.nn.relu,\n                       kernel_regularizer=l2_regularizer(self.config.l2_reg)):\n            h_z = deconv1d(z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[0], strides=2)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[1], strides=1)\n            h_z = deconv1d(h_z, out_channels=self.config.x_dim, output_shape=self.config.output_shape[2], strides=2)\n        return h_z\n\n    @instance_reuse\n    def pretrain_q_net(self, x, observed=None, n_z=None, is_training=False):\n        # vs.name = self.variable_scope.name + \"/q_net\"\n        logging.info('pretrain_q_net builder: %r', locals())\n\n        net = BayesianNet(observed=observed)\n\n        def dropout_fn(input):\n            return tf.layers.dropout(input, rate=.5, training=is_training)\n\n        qz_mean, qz_logstd = self.h_for_qz(x, is_training=is_training)\n\n        qz_distribution = Normal(mean=qz_mean, logstd=qz_logstd)\n\n        qz_distribution = qz_distribution.batch_ndims_to_value(2)\n\n        z = net.add('z', qz_distribution, n_samples=n_z, is_reparameterized=True)\n\n        return net\n\n    @instance_reuse\n    def pretrain_p_net(self, observed=None, n_z=None, is_training=False):\n        logging.info('p_net builder: %r', locals())\n\n        net = BayesianNet(observed=observed)\n\n        pz_distribution = Normal(mean=tf.zeros([self.config.z2_dim, self.config.x_dim]),\n                                 logstd=tf.zeros([self.config.z2_dim, self.config.x_dim]))\n\n        pz_distribution = pz_distribution.batch_ndims_to_value(2)\n\n        z = net.add('z',\n                    pz_distribution,\n                    n_samples=n_z, is_reparameterized=True)\n\n        h_z = self.h_for_px(z)\n\n        px_mean = conv1d(h_z, kernel_size=1, out_channels=self.config.x_dim, scope='pre_px_mean')\n        px_logstd = conv1d(h_z, kernel_size=1, out_channels=self.config.x_dim, scope='pre_px_logstd')\n        px_logstd = tf.clip_by_value(px_logstd, clip_value_min=self.config.logstd_min,\n                                     clip_value_max=self.config.logstd_max)\n\n        x = net.add('x',\n                    Normal(mean=px_mean, logstd=px_logstd).batch_ndims_to_value(2),\n                    is_reparameterized=True)\n\n        return net\n"
  },
  {
    "path": "algorithm/__init__.py",
    "content": "from .recurrent_distribution import RecurrentDistribution\nfrom .real_nvp import dense_real_nvp\nfrom .utils import *\nfrom .mcmc_recons import *"
  },
  {
    "path": "algorithm/cal_IPS.py",
    "content": "import os\nimport pickle\nimport numpy as np\n\n\ndef cal_IPS(path, dataset, mcmc, is_pretrain):\n    path += '/'\n    # load labels\n    f = open(\"./data/processed/\" + dataset + \"_test_label.pkl\", \"rb\")\n    test_label = pickle.load(f).reshape((-1))\n    f.close()\n    label_dir = './data/interpretation_label/' + dataset + '.txt'\n    try:\n        labels = str(open(label_dir, 'rb').read(), encoding='utf8')\n    except:\n        raise FileNotFoundError('cannot find label via path: ' + label_dir)\n    # parse labels\n    labels = labels.split('\\n')\n    if len(labels[-1]) == 0:\n        labels = labels[:-1]\n    intervals = []\n    dims = []\n    for i in labels:\n        t = i.split(':')\n        assert len(t) == 2\n        intervals.append([int(_) for _ in t[0].split('-')])\n        assert len(intervals[-1]) == 2\n        dims.append([int(_) for _ in t[1].split(',')])\n\n    assert len(intervals) == len(dims)\n    interpret_dict = {}\n    tp_res = {}\n\n    # form the interpret dict: {(idx_st, idx_ed): interpret_dims}\n    for _ in range(len(intervals)):\n        interpret_dict[tuple(intervals[_])] = dims[_]\n\n    if mcmc:\n        assert is_pretrain is False\n        try:\n            tracker = pickle.load(open(path + 'mcmc_tracker.pkl', 'rb'))\n        except:\n            raise FileNotFoundError('cannot find mcmc_tracker.pkl')\n        # preprocess tracker to get:\n        # {idx_ed: best_score_mcmc (np.array(shape=[window_size, x_dim]))}\n        for ed_idx in tracker:\n            shape = tracker[ed_idx]['best_score'].shape\n            assert len(shape) == 3\n            tp_res[ed_idx] = tracker[ed_idx]['best_score'].reshape([shape[1], shape[2]])\n    else:\n        try:\n            if is_pretrain:\n                full_recons = np.load(path + 'pretrain_full_recons_window_probs.npz')\n            else:\n                full_recons = np.load(path + 'full_recons_window_probs.npz')\n        except:\n            raise FileNotFoundError('cannot find full_recons_window_probs.npz!')\n        try:\n            if is_pretrain:\n                test_score = pickle.load(open(path + 'pretrain_test_score.pkl', 'rb'))\n            else:\n                test_score = pickle.load(open(path + 'test_score.pkl', 'rb'))\n        except:\n            raise FileNotFoundError('cannot find test_score.pkl!')\n        # get all tp points idx\n        full_recons = full_recons['full_full_test_recons_window_probs']\n        assert full_recons.shape[0] == len(test_label) - 100 + 1\n\n        test_label = test_label[-len(test_score):]\n        assert len(test_score) == len(test_label)\n\n        t, th = get_best_f1(test_score, test_label)\n        tp_idx = np.logical_and(test_label > 0.5, test_score <= th)\n        tp_idx = np.where(tp_idx)[0]\n        tp_idx += (100 - 1)\n        # get the windows which end by tp_idx\n        for ed_idx in tp_idx:\n            tp_res[ed_idx] = full_recons[ed_idx - 100 + 1]\n            assert tp_res[ed_idx].shape[0] == 100\n\n    results = {}\n\n    for p in [100, 150]:\n        prefix = 'p=' + str(p) + ': '\n        # segment-wise aggregation\n        within_window_funcs = [lambda x: x[-1]]\n        within_window_func_names = ['wd_last']\n        assert len(within_window_func_names) == len(within_window_funcs)\n\n        def min_aggr_and_keep(x):\n            temp = np.min(x, axis=0)\n            return [temp for _ in range(len(x))]\n\n        def ave_aggr_and_keep(x):\n            temp = np.mean(x, axis=0)\n            return [temp for _ in range(len(x))]\n\n        def max_aggr_and_keep(x):\n            temp = np.max(x, axis=0)\n            return [temp for _ in range(len(x))]\n\n        def min_aggr(x):\n            return [np.min(x, axis=0)]\n\n        def ave_aggr(x):\n            return [np.mean(x, axis=0)]\n\n        def max_aggr(x):\n            return [np.max(x, axis=0)]\n\n        within_interval_sc_funcs = [min_aggr_and_keep]\n        within_interval_sc_func_names = ['itv_min_weight']\n\n        for wd_idx, window_func in enumerate(within_window_funcs):\n            for iv_idx, itv_func in enumerate(within_interval_sc_funcs):\n                combine_name = prefix + within_window_func_names[wd_idx] + '_' + within_interval_sc_func_names[iv_idx]\n                # compute aggr score\n                itv = {}\n                for ed_idx in tp_res:\n                    for interval in interpret_dict:\n                        if interval[0] <= ed_idx <= interval[1]:\n                            # this TP in this interval\n                            dim_scores = window_func(tp_res[ed_idx]).reshape((-1))\n                            if interval in itv:\n                                itv[interval].append(dim_scores)\n                            else:\n                                itv[interval] = [dim_scores]\n                            break\n                scores = []\n                labels = []\n                for interval in itv:\n                    temp = itv_func(itv[interval])\n                    scores += temp\n                    labels += [interpret_dict[interval] for _ in range(len(temp))]\n                assert len(scores) == len(labels)\n                # compute Interpretation score\n                hit_rate_collector = []\n                for idx, dim_scores in enumerate(scores):\n                    dim_order = np.argsort(dim_scores) + 1\n                    hit_rate = get_hit_rate(pred=dim_order, label=labels[idx], p=p)\n                    hit_rate_collector.append(hit_rate)\n                hit_rate = np.mean(hit_rate_collector)\n                results[combine_name] = hit_rate\n\n    if is_pretrain:\n        res = {}\n        for _ in results:\n            res['pretrain_' + _] = results[_]\n        results = res\n    if mcmc:\n        res = {}\n        for _ in results:\n            res['mcmc_' + _] = results[_]\n        results = res\n    return results\n\n\ndef get_hit_rate(pred, label, p):\n    chance_num = min(int(p / 100 * len(label)), len(pred))\n    cnt = 0\n    for _ in range(chance_num):\n        if pred[_] in label:\n            cnt += 1\n    hit_rate = cnt / len(label)\n    return hit_rate\n\n\n# here for our refined best-f1 search method\ndef get_best_f1(score, label):\n    '''\n    :param score: 1-D array, input score, tot_length\n    :param label: 1-D array, standard label for anomaly\n    :return: list for results, threshold\n    '''\n\n    assert score.shape == label.shape\n    print('***computing best f1***')\n    search_set = []\n    tot_anomaly = 0\n    for i in range(label.shape[0]):\n        tot_anomaly += (label[i] > 0.5)\n    flag = 0\n    cur_anomaly_len = 0\n    cur_min_anomaly_score = 1e5\n    for i in range(label.shape[0]):\n        if label[i] > 0.5:\n            # here for an anomaly\n            if flag == 1:\n                cur_anomaly_len += 1\n                cur_min_anomaly_score = score[i] if score[i] < cur_min_anomaly_score else cur_min_anomaly_score\n            else:\n                flag = 1\n                cur_anomaly_len = 1\n                cur_min_anomaly_score = score[i]\n        else:\n            # here for normal points\n            if flag == 1:\n                flag = 0\n                search_set.append((cur_min_anomaly_score, cur_anomaly_len, True))\n                search_set.append((score[i], 1, False))\n            else:\n                search_set.append((score[i], 1, False))\n    if flag == 1:\n        search_set.append((cur_min_anomaly_score, cur_anomaly_len, True))\n    search_set.sort(key=lambda x: x[0])\n    best_f1_res = - 1\n    threshold = 1\n    P = 0\n    TP = 0\n    best_P = 0\n    best_TP = 0\n    for i in range(len(search_set)):\n        P += search_set[i][1]\n        if search_set[i][2]:  # for an anomaly point\n            TP += search_set[i][1]\n        precision = TP / (P + 1e-5)\n        recall = TP / (tot_anomaly + 1e-5)\n        f1 = 2 * precision * recall / (precision + recall + 1e-5)\n        if f1 > best_f1_res:\n            best_f1_res = f1\n            threshold = search_set[i][0]\n            best_P = P\n            best_TP = TP\n\n    print('***  best_f1  ***: ', best_f1_res)\n    print('*** threshold ***: ', threshold)\n    return (best_f1_res,\n            best_TP / (best_P + 1e-5),\n            best_TP / (tot_anomaly + 1e-5),\n            best_TP,\n            score.shape[0] - best_P - tot_anomaly + best_TP,\n            best_P - best_TP,\n            tot_anomaly - best_TP), threshold\n"
  },
  {
    "path": "algorithm/conv1d_.py",
    "content": "import numpy as np\nimport tensorflow as tf\nfrom tensorflow.contrib.framework import add_arg_scope\n\nfrom tfsnippet.ops import (assert_rank, assert_scalar_equal, flatten_to_ndims,\n                           unflatten_from_ndims)\nfrom tfsnippet.utils import (validate_positive_int_arg, ParamSpec,\n                             is_tensor_object, assert_deps,\n                             get_shape,\n                             add_name_and_scope_arg_doc, model_variable,\n                             maybe_check_numerics, maybe_add_histogram, InputSpec,\n                             get_static_shape, validate_enum_arg, validate_int_tuple_arg)\n\n__all__ = [\n    'conv1d', 'deconv1d', 'validate_conv1d_input', 'get_deconv_output_length', 'batch_norm_1d'\n]\n\n\n@add_arg_scope\ndef batch_norm_1d(input, channels_last=True, training=False, name=None,\n                  scope=None):\n    \"\"\"\n    Apply batch normalization on 1D convolutional layer.\n\n    Args:\n        input (tf.Tensor): The input tensor.\n        channels_last (bool): Whether or not the channel dimension is at last?\n        training (bool or tf.Tensor): Whether or not the model is under\n            training stage?\n\n    Returns:\n        tf.Tensor: The normalized tensor.\n    \"\"\"\n    with tf.variable_scope(scope, default_name=name or 'batch_norm_1d'):\n        input, s1, s2 = flatten_to_ndims(input, ndims=3)\n        output = tf.layers.batch_normalization(\n            input,\n            axis=-1 if channels_last else -2,\n            training=training,\n            name='norm'\n        )\n        output = unflatten_from_ndims(output, s1, s2)\n        return output\n\n\ndef validate_conv1d_input(input, channels_last, arg_name='input'):\n    \"\"\"\n    Validate the input for 1-d convolution.\n    Args:\n        input: The input tensor, must be at least 3-d.\n        channels_last (bool): Whether or not the last dimension is the\n            channels dimension? (i.e., the data format is (batch, length, channels))\n        arg_name (str): Name of the input argument.\n    Returns:\n        (tf.Tensor, int, str): The validated input tensor, the number of input\n            channels, and the data format.\n    \"\"\"\n    if channels_last:\n        channel_axis = -1\n        input_spec = InputSpec(shape=('...', '?', '?', '*'))\n        data_format = \"NWC\"\n    else:\n        channel_axis = -2\n        input_spec = InputSpec(shape=('...', '?', '*', '?'))\n        data_format = \"NCW\"\n    input = input_spec.validate(arg_name, input)\n    input_shape = get_static_shape(input)\n    in_channels = input_shape[channel_axis]\n\n    return input, in_channels, data_format\n\n\ndef get_deconv_output_length(input_length, kernel_size, strides, padding):\n    \"\"\"\n    Get the output length of deconvolution at a specific dimension.\n    Args:\n        input_length: Input tensor length.\n        kernel_size: The size of the kernel.\n        strides: The stride of convolution.\n        padding: One of {\"same\", \"valid\"}, case in-sensitive\n    Returns:\n        int: The output length of deconvolution.\n    \"\"\"\n    padding = validate_enum_arg(\n        'padding', str(padding).upper(), ['SAME', 'VALID'])\n    output_length = input_length * strides\n    if padding == 'VALID':\n        output_length += max(kernel_size - strides, 0)\n    return output_length\n\n\n@add_arg_scope\n@add_name_and_scope_arg_doc\ndef conv1d(input,\n           out_channels,\n           kernel_size,\n           strides=1,\n           dilations=1,\n           padding='same',\n           channels_last=True,\n           activation_fn=None,\n           normalizer_fn=None,\n           gated=False,\n           gate_sigmoid_bias=2.,\n           kernel=None,\n           kernel_mask=None,\n           kernel_initializer=None,\n           kernel_regularizer=None,\n           kernel_constraint=None,\n           use_bias=None,\n           bias=None,\n           bias_initializer=tf.zeros_initializer(),\n           bias_regularizer=None,\n           bias_constraint=None,\n           trainable=True,\n           name=None,\n           scope=None):\n    \"\"\"\n    1D convolutional layer.\n    Args:\n        input (Tensor): The input tensor, at least 3-d.\n        out_channels (int): The channel numbers of the output.\n        kernel_size (int or tuple(int,)): Kernel size over spatial dimensions.\n        strides (int): Strides over spatial dimensions.\n        dilations (int): The dilation factor over spatial dimensions.\n        padding: One of {\"valid\", \"same\"}, case in-sensitive.\n        channels_last (bool): Whether or not the channel axis is the last\n            axis in `input`? (i.e., the data format is \"NWC\")\n        activation_fn: The activation function.\n        normalizer_fn: The normalizer function.\n        gated (bool): Whether or not to use gate on output?\n            `output = activation_fn(output) * sigmoid(gate)`.\n        gate_sigmoid_bias (Tensor): The bias added to `gate` before applying\n            the `sigmoid` activation.\n        kernel (Tensor): Instead of creating a new variable, use this tensor.\n        kernel_mask (Tensor): If specified, multiply this mask onto `kernel`,\n            i.e., the actual kernel to use will be `kernel * kernel_mask`.\n        kernel_initializer: The initializer for `kernel`.\n            Would be ``default_kernel_initializer(...)`` if not specified.\n        kernel_regularizer: The regularizer for `kernel`.\n        kernel_constraint: The constraint for `kernel`.\n        use_bias (bool or None): Whether or not to use `bias`?\n            If :obj:`True`, will always use bias.\n            If :obj:`None`, will use bias only if `normalizer_fn` is not given.\n            If :obj:`False`, will never use bias.\n            Default is :obj:`None`.\n        bias (Tensor): Instead of creating a new variable, use this tensor.\n        bias_initializer: The initializer for `bias`.\n        bias_regularizer: The regularizer for `bias`.\n        bias_constraint: The constraint for `bias`.\n        trainable (bool): Whether or not the parameters are trainable?\n    Returns:\n        tf.Tensor: The output tensor.\n    \"\"\"\n    if not channels_last:\n        raise ValueError('Currently only channels_last=True is supported.')\n    input, in_channels, data_format = \\\n        validate_conv1d_input(input, channels_last)\n    out_channels = validate_positive_int_arg('out_channels', out_channels)\n    dtype = input.dtype.base_dtype\n    if gated:\n        out_channels *= 2\n\n    # check functional arguments\n    padding = validate_enum_arg(\n        'padding', str(padding).upper(), ['VALID', 'SAME'])\n    dilations = validate_positive_int_arg('dilations', dilations)\n    strides = validate_positive_int_arg('strides', strides)\n\n    if dilations > 1 and not channels_last:\n        raise ValueError('`channels_last` == False is incompatible with '\n                         '`dilations` > 1.')\n\n    if strides > 1 and dilations > 1:\n        raise ValueError('`strides` > 1 is incompatible with `dilations` > 1.')\n\n    if use_bias is None:\n        use_bias = normalizer_fn is None\n\n    # get the specification of outputs and parameters\n    kernel_size = validate_int_tuple_arg('kernel_size', kernel_size)\n    kernel_shape = kernel_size + (in_channels, out_channels)\n    bias_shape = (out_channels,)\n\n    # validate the parameters\n    if kernel is not None:\n        kernel_spec = ParamSpec(shape=kernel_shape, dtype=dtype)\n        kernel = kernel_spec.validate('kernel', kernel)\n    if kernel_mask is not None:\n        kernel_mask_spec = InputSpec(dtype=dtype)\n        kernel_mask = kernel_mask_spec.validate('kernel_mask', kernel_mask)\n    if kernel_initializer is None:\n        kernel_initializer = tf.glorot_normal_initializer()\n    if bias is not None:\n        bias_spec = ParamSpec(shape=bias_shape, dtype=dtype)\n        bias = bias_spec.validate('bias', bias)\n\n    # the main part of the conv1d layer\n    with tf.variable_scope(scope, default_name=name or 'conv1d'):\n        c_axis = -1 if channels_last else -2\n\n        # create the variables\n        if kernel is None:\n            kernel = model_variable(\n                'kernel',\n                shape=kernel_shape,\n                dtype=dtype,\n                initializer=kernel_initializer,\n                regularizer=kernel_regularizer,\n                constraint=kernel_constraint,\n                trainable=trainable\n            )\n\n        if kernel_mask is not None:\n            kernel = kernel * kernel_mask\n\n        maybe_add_histogram(kernel, 'kernel')\n        kernel = maybe_check_numerics(kernel, 'kernel')\n\n        if use_bias and bias is None:\n            bias = model_variable(\n                'bias',\n                shape=bias_shape,\n                initializer=bias_initializer,\n                regularizer=bias_regularizer,\n                constraint=bias_constraint,\n                trainable=trainable\n            )\n            maybe_add_histogram(bias, 'bias')\n            bias = maybe_check_numerics(bias, 'bias')\n\n        # flatten to 3d\n        output, s1, s2 = flatten_to_ndims(input, 3)\n\n        # do convolution\n        if dilations > 1:\n            output = tf.nn.convolution(\n                input=output,\n                filter=kernel,\n                dilation_rate=(dilations,),\n                padding=padding,\n                data_format=data_format\n            )\n        else:\n            output = tf.nn.conv1d(\n                value=output,\n                filters=kernel,\n                stride=strides,\n                padding=padding,\n                data_format=data_format\n            )\n\n        # add bias\n        if use_bias:\n            output = tf.add(output, bias)\n\n        # apply the normalization function if specified\n        if normalizer_fn is not None:\n            output = normalizer_fn(output)\n\n        # split into halves if gated\n        if gated:\n            output, gate = tf.split(output, 2, axis=c_axis)\n\n        # apply the activation function if specified\n        if activation_fn is not None:\n            output = activation_fn(output)\n\n        # apply the gate if required\n        if gated:\n            if gate_sigmoid_bias is None:\n                gate_sigmoid_bias = model_variable(\n                    'gate_sigmoid_bias',\n                    shape=bias_shape,\n                    initializer=bias_initializer,\n                    regularizer=bias_regularizer,\n                    constraint=bias_constraint,\n                    trainable=trainable\n                )\n                maybe_add_histogram(gate_sigmoid_bias, 'gate_sigmoid_bias')\n                gate_sigmoid_bias = maybe_check_numerics(gate_sigmoid_bias, 'gate_sigmoid_bias')\n            output = output * tf.sigmoid(gate + gate_sigmoid_bias, name='gate')\n\n        # unflatten back to original shape\n        output = unflatten_from_ndims(output, s1, s2)\n\n        maybe_add_histogram(output, 'output')\n        output = maybe_check_numerics(output, 'output')\n    return output\n\n\n@add_arg_scope\n@add_name_and_scope_arg_doc\ndef deconv1d(input,\n             out_channels,\n             kernel_size,\n             strides=1,\n             padding='same',\n             channels_last=True,\n             output_shape=None,\n             activation_fn=None,\n             normalizer_fn=None,\n             gated=False,\n             gate_sigmoid_bias=2.,\n             kernel=None,\n             kernel_initializer=None,\n             kernel_regularizer=None,\n             kernel_constraint=None,\n             use_bias=None,\n             bias=None,\n             bias_initializer=tf.zeros_initializer(),\n             bias_regularizer=None,\n             bias_constraint=None,\n             trainable=True,\n             name=None,\n             scope=None):\n    \"\"\"\n    1D deconvolutional layer.\n    Args:\n        input (Tensor): The input tensor, at least 3-d.\n        out_channels (int): The channel numbers of the deconvolution output.\n        kernel_size (int or tuple(int,)): Kernel size over spatial dimensions.\n        strides (int): Strides over spatial dimensions.\n        padding: One of {\"valid\", \"same\"}, case in-sensitive.\n        channels_last (bool): Whether or not the channel axis is the last\n            axis in `input`? (i.e., the data format is \"NWC\")\n        output_shape: If specified, use this as the shape of the\n            deconvolution output; otherwise compute the size of each dimension\n            by::\n                output_size = input_size * strides\n                if padding == 'valid':\n                    output_size += max(kernel_size - strides, 0)\n        activation_fn: The activation function.\n        normalizer_fn: The normalizer function.\n        gated (bool): Whether or not to use gate on output?\n            `output = activation_fn(output) * sigmoid(gate)`.\n        gate_sigmoid_bias (Tensor): The bias added to `gate` before applying\n            the `sigmoid` activation.\n        kernel (Tensor): Instead of creating a new variable, use this tensor.\n        kernel_initializer: The initializer for `kernel`.\n            Would be ``default_kernel_initializer(...)`` if not specified.\n        kernel_regularizer: The regularizer for `kernel`.\n        kernel_constraint: The constraint for `kernel`.\n        use_bias (bool or None): Whether or not to use `bias`?\n            If :obj:`True`, will always use bias.\n            If :obj:`None`, will use bias only if `normalizer_fn` is not given.\n            If :obj:`False`, will never use bias.\n            Default is :obj:`None`.\n        bias (Tensor): Instead of creating a new variable, use this tensor.\n        bias_initializer: The initializer for `bias`.\n        bias_regularizer: The regularizer for `bias`.\n        bias_constraint: The constraint for `bias`.\n        trainable (bool): Whether or not the parameters are trainable?\n    Returns:\n        tf.Tensor: The output tensor.\n    \"\"\"\n    if not channels_last:\n        raise ValueError('Currently only channels_last=True is supported.')\n    input, in_channels, data_format = \\\n        validate_conv1d_input(input, channels_last)\n    out_channels = validate_positive_int_arg('out_channels', out_channels)\n    dtype = input.dtype.base_dtype\n    if gated:\n        out_channels *= 2\n\n    # check functional arguments\n    padding = validate_enum_arg(\n        'padding', str(padding).upper(), ['VALID', 'SAME'])\n    strides = validate_positive_int_arg('strides', strides)\n\n    if use_bias is None:\n        use_bias = normalizer_fn is None\n\n    # get the specification of outputs and parameters\n    kernel_size = validate_int_tuple_arg('kernel_size', kernel_size)\n    kernel_shape = kernel_size + (out_channels, in_channels)\n    bias_shape = (out_channels,)\n\n    given_w = None\n    given_output_shape = output_shape\n\n    if is_tensor_object(given_output_shape):\n        given_output_shape = tf.convert_to_tensor(given_output_shape)\n    elif given_output_shape is not None:\n        given_w = given_output_shape\n\n    # validate the parameters\n    if kernel is not None:\n        kernel_spec = ParamSpec(shape=kernel_shape, dtype=dtype)\n        kernel = kernel_spec.validate('kernel', kernel)\n    if kernel_initializer is None:\n        kernel_initializer = tf.glorot_normal_initializer()\n    if bias is not None:\n        bias_spec = ParamSpec(shape=bias_shape, dtype=dtype)\n        bias = bias_spec.validate('bias', bias)\n\n    # the main part of the conv2d layer\n    with tf.variable_scope(scope, default_name=name or 'deconv1d'):\n        with tf.name_scope('output_shape'):\n            # detect the input shape and axis arrangements\n            input_shape = get_static_shape(input)\n            if channels_last:\n                c_axis, w_axis = -1, -2\n            else:\n                c_axis, w_axis = -2, -1\n\n            output_shape = [None, None, None]\n            output_shape[c_axis] = out_channels\n            if given_output_shape is None:\n                if input_shape[w_axis] is not None:\n                    output_shape[w_axis] = get_deconv_output_length(\n                        input_shape[w_axis], kernel_shape[0], strides[0],\n                        padding\n                    )\n            else:\n                if not is_tensor_object(given_output_shape):\n                    output_shape[w_axis] = given_w\n\n            # infer the batch shape in 3-d\n            batch_shape = input_shape[:-2]\n            if None not in batch_shape:\n                output_shape[0] = int(np.prod(batch_shape))\n\n            # now the static output shape is ready\n            output_static_shape = tf.TensorShape(output_shape)\n\n            # prepare for the dynamic batch shape\n            if output_shape[0] is None:\n                output_shape[0] = tf.reduce_prod(get_shape(input)[:-2])\n\n            # prepare for the dynamic spatial dimensions\n            if output_shape[w_axis] is None:\n                if given_output_shape is None:\n                    input_shape = get_shape(input)\n                    if output_shape[w_axis] is None:\n                        output_shape[w_axis] = get_deconv_output_length(\n                            input_shape[w_axis], kernel_shape[0],\n                            strides[0], padding\n                        )\n                else:\n                    assert(is_tensor_object(given_output_shape))\n                    with assert_deps([\n                        assert_rank(given_output_shape, 1),\n                        assert_scalar_equal(\n                            tf.size(given_output_shape), 1)\n                    ]):\n                        output_shape[w_axis] = given_output_shape[0]\n\n            # compose the final dynamic shape\n            if any(is_tensor_object(s) for s in output_shape):\n                output_shape = tf.stack(output_shape)\n            else:\n                output_shape = tuple(output_shape)\n\n        # create the variables\n        if kernel is None:\n            kernel = model_variable(\n                'kernel',\n                shape=kernel_shape,\n                dtype=dtype,\n                initializer=kernel_initializer,\n                regularizer=kernel_regularizer,\n                constraint=kernel_constraint,\n                trainable=trainable\n            )\n\n        maybe_add_histogram(kernel, 'kernel')\n        kernel = maybe_check_numerics(kernel, 'kernel')\n\n        if use_bias and bias is None:\n            bias = model_variable(\n                'bias',\n                shape=bias_shape,\n                initializer=bias_initializer,\n                regularizer=bias_regularizer,\n                constraint=bias_constraint,\n                trainable=trainable\n            )\n            maybe_add_histogram(bias, 'bias')\n            bias = maybe_check_numerics(bias, 'bias')\n\n        # flatten to 3d\n        output, s1, s2 = flatten_to_ndims(input, 3)\n\n        # do convolution or deconvolution\n        output = tf.contrib.nn.conv1d_transpose(\n            value=output,\n            filter=kernel,\n            output_shape=output_shape,\n            stride=strides,\n            padding=padding,\n            data_format=data_format\n        )\n        if output_static_shape is not None:\n            output.set_shape(output_static_shape)\n\n        # add bias\n        if use_bias:\n            output = tf.add(output, bias)\n\n        # apply the normalization function if specified\n        if normalizer_fn is not None:\n            output = normalizer_fn(output)\n\n        # split into halves if gated\n        if gated:\n            output, gate = tf.split(output, 2, axis=c_axis)\n\n        # apply the activation function if specified\n        if activation_fn is not None:\n            output = activation_fn(output)\n\n        # apply the gate if required\n        if gated:\n            if gate_sigmoid_bias is None:\n                gate_sigmoid_bias = model_variable(\n                    'gate_sigmoid_bias',\n                    shape=bias_shape,\n                    initializer=bias_initializer,\n                    regularizer=bias_regularizer,\n                    constraint=bias_constraint,\n                    trainable=trainable\n                )\n                maybe_add_histogram(gate_sigmoid_bias, 'gate_sigmoid_bias')\n                gate_sigmoid_bias = maybe_check_numerics(gate_sigmoid_bias, 'gate_sigmoid_bias')\n            output = output * tf.sigmoid(gate + gate_sigmoid_bias, name='gate')\n\n        # unflatten back to original shape\n        output = unflatten_from_ndims(output, s1, s2)\n\n        maybe_add_histogram(output, 'output')\n        output = maybe_check_numerics(output, 'output')\n\n    return output\n"
  },
  {
    "path": "algorithm/mcmc_recons.py",
    "content": "import tensorflow as tf\n\n\n__all__ = ['masked_reconstruct', 'mcmc_reconstruct']\n\n\ndef masked_reconstruct(reconstruct, x, u, mask, name=None):\n    \"\"\"\n    Replace masked elements of `x` with reconstructed outputs.\n    The potential anomaly points on x can be masked, and replaced by the reconstructed values.\n    This can make the reconstruction more likely to be the normal pattern x should follow.\n    Args:\n        reconstruct ((tf.Tensor, tf.Tensor, tf.Tensor) -> tf.Tensor): Function for reconstructing `x`.\n        x: The tensor to be reconstructed by `func`.\n        u: Additional input for reconstructing `x`.\n        mask: (tf.Tensor) mask, must be broadcastable into the shape of `x`.\n            Indicating whether or not to mask each element of `x`.\n        name (str): Name of this operation in TensorFlow graph.\n            (default \"masked_reconstruct\")\n    Returns:\n        tf.Tensor: `x` with masked elements replaced by reconstructed outputs.\n    \"\"\"\n    with tf.name_scope(name, default_name='masked_reconstruct'):\n        x = tf.convert_to_tensor(x)  # type: tf.Tensor\n        mask = tf.convert_to_tensor(mask, dtype=tf.int32)  # type: tf.Tensor\n\n        mask = tf.broadcast_to(mask, tf.shape(x))\n\n        # get reconstructed x. Currently only support mask the last point if pixelcnn decoder is used.\n        x_recons = reconstruct(x, u, mask)\n\n        # get masked outputs\n        return tf.where(tf.cast(mask, dtype=tf.bool), x_recons, x)\n\n\ndef mcmc_reconstruct(reconstruct, x, u, mask, iter_count,\n                                 back_prop=True, name=None):\n    \"\"\"\n    Iteratively reconstruct `x` with `mask` for `iter_count` times.\n    This method will call :func:`masked_reconstruct` for `iter_count` times,\n    with the output from previous iteration as the input `x` for the next\n    iteration.  The output of the final iteration would be returned.\n    Args:\n        reconstruct: Function for reconstructing `x`.\n        x: The tensor to be reconstructed by `func`.\n        u: Additional input for reconstructing `x`.\n        mask: (tf.Tensor) mask, must be broadcastable into the shape of `x`.\n            Indicating whether or not to mask each element of `x`.\n        iter_count (int or tf.Tensor):\n            Number of mcmc iterations(must be greater than 1).\n        back_prop (bool): Whether or not to support back-propagation through\n            all the iterations? (default :obj:`True`)\n        name (str): Name of this operation in TensorFlow graph.\n            (default \"iterative_masked_reconstruct\")\n    Returns:\n        tf.Tensor: The iteratively reconstructed `x`.\n    \"\"\"\n    with tf.name_scope(name, default_name='mcmc_reconstruct'):\n\n        # do the masked reconstructions\n        x_recons, _ = tf.while_loop(\n            lambda x_i, i: i < iter_count,\n            lambda x_i, i: (masked_reconstruct(reconstruct, x_i, u, mask), i + 1),\n            [x, tf.constant(0, dtype=tf.int32)],\n            back_prop=back_prop\n        )\n\n        return x_recons\n"
  },
  {
    "path": "algorithm/real_nvp.py",
    "content": "import tensorflow as tf\nimport tfsnippet as spt\nfrom tensorflow.contrib.framework import arg_scope\nimport numpy as np\nfrom tfsnippet.layers.flows.utils import ZeroLogDet\n\n\nclass FeatureReversingFlow(spt.layers.FeatureMappingFlow):\n\n    def __init__(self, axis=-1, value_ndims=1, name=None, scope=None):\n        super(FeatureReversingFlow, self).__init__(\n            axis=int(axis), value_ndims=value_ndims, name=name, scope=scope)\n\n    @property\n    def explicitly_invertible(self):\n        return True\n\n    def _build(self, input=None):\n        pass\n\n    def _reverse_feature(self, x, compute_y, compute_log_det):\n        n_features = spt.utils.get_static_shape(x)[self.axis]\n        if n_features is None:\n            raise ValueError('The feature dimension must be fixed.')\n        assert (0 > self.axis >= -self.value_ndims >=\n                -len(spt.utils.get_static_shape(x)))\n        permutation = np.asarray(list(reversed(range(n_features))),\n                                 dtype=np.int32)\n\n        # compute y\n        y = None\n        if compute_y:\n            y = tf.gather(x, permutation, axis=self.axis)\n\n        # compute log_det\n        log_det = None\n        if compute_log_det:\n            log_det = ZeroLogDet(spt.utils.get_shape(x)[:-self.value_ndims],\n                                 x.dtype.base_dtype)\n\n        return y, log_det\n\n    def _transform(self, x, compute_y, compute_log_det):\n        return self._reverse_feature(x, compute_y, compute_log_det)\n\n    def _inverse_transform(self, y, compute_x, compute_log_det):\n        return self._reverse_feature(y, compute_x, compute_log_det)\n\n\ndef dense_real_nvp(flow_depth: int,\n                   activation,\n                   kernel_regularizer,\n                   scope: str,\n                   use_invertible_flow=True,\n                   strict_invertible=False,\n                   use_actnorm_flow=False,\n                   dense_coupling_n_hidden_layers=1,\n                   dense_coupling_n_hidden_units=100,\n                   coupling_scale_shift_initializer='zero',     # 'zero' or 'normal'\n                   coupling_scale_shift_normal_initializer_stddev=0.001,\n                   coupling_scale_type='sigmoid',               # 'sigmoid' or 'exp'\n                   coupling_sigmoid_scale_bias=2.,\n                   is_prior_flow=False) -> spt.layers.BaseFlow:\n    def shift_and_scale(x1, n2):\n\n        with arg_scope([spt.layers.dense],\n                       activation_fn=activation,\n                       kernel_regularizer=kernel_regularizer):\n            h = x1\n            for j in range(dense_coupling_n_hidden_layers):\n                h = spt.layers.dense(h,\n                                     units=dense_coupling_n_hidden_units,\n                                     scope='hidden_{}'.format(j))\n\n        # compute shift and scale\n        if coupling_scale_shift_initializer == 'zero':\n            pre_params_initializer = tf.zeros_initializer()\n        else:\n            pre_params_initializer = tf.random_normal_initializer(\n                stddev=coupling_scale_shift_normal_initializer_stddev)\n        pre_params = spt.layers.dense(h,\n                                      units=n2 * 2,\n                                      kernel_initializer=pre_params_initializer,\n                                      scope='shift_and_scale',)\n\n        shift = pre_params[..., :n2]\n        scale = pre_params[..., n2:]\n\n        return shift, scale\n\n    with tf.variable_scope(scope):\n        flows = []\n        for i in range(flow_depth):\n            level = []\n            if use_invertible_flow:\n                level.append(\n                    spt.layers.InvertibleDense(\n                        strict_invertible=strict_invertible)\n                )\n            else:\n                level.append(FeatureReversingFlow())\n            level.append(\n                spt.layers.CouplingLayer(\n                    tf.make_template(\n                        'coupling', shift_and_scale, create_scope_now_=True),\n                    scale_type=coupling_scale_type,\n                    sigmoid_scale_bias=coupling_sigmoid_scale_bias,\n                )\n            )\n            if use_actnorm_flow:\n                level.append(spt.layers.ActNorm())\n            flows.extend(level)\n        flow = spt.layers.SequentialFlow(flows)\n\n    if is_prior_flow:\n        flow = flow.invert()\n\n    return flow\n"
  },
  {
    "path": "algorithm/recurrent_distribution.py",
    "content": "import tensorflow as tf\nimport tfsnippet as spt\nfrom tfsnippet.distributions import Distribution, Normal\nfrom tfsnippet.stochastic import StochasticTensor\nimport numpy as np\n\n\nclass RecurrentDistribution(Distribution):\n    def __init__(self, input, mean_layer, logstd_layer, z_dim, window_length,\n                 is_reparameterized=True, check_numerics=False):\n        batch_shape = spt.utils.concat_shapes([spt.utils.get_shape(input)[:-1], [z_dim]])\n        batch_static_shape = tf.TensorShape(spt.utils.get_static_shape(input)[:-1] + (z_dim,))\n\n        super(RecurrentDistribution, self).__init__(dtype=input.dtype,\n                                                    is_continuous=True,\n                                                    is_reparameterized=is_reparameterized,\n                                                    value_ndims=0,\n                                                    batch_shape=batch_shape,\n                                                    batch_static_shape=batch_static_shape)\n\n        self.mean_layer = mean_layer\n        self.logstd_layer = logstd_layer\n        self.z_dim = z_dim\n        self._check_numerics = check_numerics\n        self.window_length = window_length\n        self.origin_input = input\n        if len(input.shape) > 3:\n            input, s1, s2 = spt.ops.flatten_to_ndims(input, 3)\n            self.time_first_input = tf.transpose(input, [1, 0, 2])\n            self.s1 = s1\n            self.s2 = s2\n            self.need_unflatten = True\n        elif len(input.shape) == 3:\n            self.time_first_input = tf.transpose(input, [1, 0, 2])  # (window_length, batch_size, feature_dim)\n            self.s1 = None\n            self.s2 = None\n            self.need_unflatten = False\n        else:\n            raise ValueError('Invalid input shape in recurrent distribution.')\n        self._mu = None\n        self._logstd = None\n\n    def mean(self):\n        return self._mu\n\n    def logstd(self):\n        return self._logstd\n\n    def _normal_pdf(self, x, mu, logstd):\n        c = -0.5 * np.log(2 * np.pi)\n        precision = tf.exp(-2 * logstd)\n        if self._check_numerics:\n            precision = tf.check_numerics(precision, \"precision\")\n        log_prob = c - logstd - 0.5 * precision * tf.square(x - mu)\n        if self._check_numerics:\n            log_prob = tf.check_numerics(log_prob, 'log_prob')\n        return log_prob\n\n    def sample_step(self, a, t):\n        z_previous, mu_z_previous, logstd_z_previous, _ = a\n        noise, input = t\n\n        # use the sampled z to derive the (mu. sigma) on next timestamp. may introduce small noise for each sample step.\n        concat_input = spt.ops.broadcast_concat(input, z_previous, axis=-1)\n\n        mu = self.mean_layer(concat_input)  # n_sample * batch_size * z_dim\n\n        logstd = self.logstd_layer(concat_input)  # n_sample * batch_size * z_dim\n\n        std = spt.utils.maybe_check_numerics(tf.exp(logstd), name='recurrent_distribution_z_std',\n                                             message='z_std in recurrent distribution exceeds.')\n\n        z_n = mu + std * noise\n\n        log_prob = self._normal_pdf(z_n, mu, logstd)\n\n        return z_n, mu, logstd, log_prob\n\n    def log_prob_step(self, a, t):\n        z_previous, _, _, log_prob_previous = a\n        given_n, input_n = t\n\n        concat_input = spt.ops.broadcast_concat(z_previous, input_n, axis=-1)\n\n        mu = self.mean_layer(concat_input)\n\n        logstd = self.logstd_layer(concat_input)\n\n        log_prob_n = self._normal_pdf(given_n, mu, logstd)\n\n        return given_n, mu, logstd, log_prob_n\n\n    def sample(self, n_samples=None, is_reparameterized=None, group_ndims=0, compute_density=False,\n               name=None):\n\n        if n_samples is None:\n            n_samples = 1\n            n_samples_is_none = True\n        else:\n            n_samples_is_none = False\n\n        with tf.name_scope(name=name, default_name='sample'):\n            noise = tf.random_normal(shape=[n_samples, tf.shape(self.time_first_input)[0],\n                                            tf.shape(self.time_first_input)[1], self.z_dim])  # (n_samples, window_length, batch_size, z_dim)\n            noise = tf.transpose(noise, [1, 0, 2, 3])   # (window_length, n_samples, batch_size, z_dim)\n\n            time_indices_shape = tf.convert_to_tensor([n_samples, tf.shape(self.time_first_input)[1], self.z_dim])  # (n_samples, batch_size, z_dim)\n\n            results = tf.scan(fn=self.sample_step,\n                              elems=(noise, self.time_first_input),\n                              initializer=(tf.zeros(time_indices_shape),\n                                           tf.zeros(time_indices_shape),\n                                           tf.zeros(time_indices_shape),\n                                           tf.zeros(time_indices_shape)),\n                              back_prop=True\n                              )  # 4 * window_length * n_samples * batch_size * z_dim\n\n            samples = tf.transpose(results[0], [1, 2, 0, 3])  # n_samples * batch_size * window_length * z_dim\n\n            log_prob = tf.transpose(results[-1], [1, 2, 0, 3])  # (n_samples, batch_size, window_length, z_dim)\n\n            if self.need_unflatten:\n                # unflatten to (n_samples, n_samples_of_input_tensor, batch_size, window_length, z_dim)\n                samples = tf.stack([spt.ops.unflatten_from_ndims(samples[i], self.s1, self.s2) for i in range(n_samples)], axis=0)\n                log_prob = tf.stack([spt.ops.unflatten_from_ndims(log_prob[i], self.s1, self.s2) for i in range(n_samples)], axis=0)\n\n            log_prob = spt.reduce_group_ndims(tf.reduce_sum, log_prob, group_ndims)\n\n            if n_samples_is_none:\n                t = StochasticTensor(\n                    distribution=self,\n                    tensor=tf.reduce_mean(samples, axis=0),\n                    group_ndims=group_ndims,\n                    is_reparameterized=self.is_reparameterized,\n                    log_prob=tf.reduce_mean(log_prob, axis=0)\n                )\n                self._mu = tf.reduce_mean(tf.transpose(results[1], [1, 2, 0, 3]), axis=0)\n                self._logstd = tf.reduce_mean(tf.transpose(results[2], [1, 2, 0, 3]), axis=0)\n                if self.need_unflatten:\n                    self._mu = spt.ops.unflatten_from_ndims(self._mu, self.s1, self.s2)\n                    self._logstd = spt.ops.unflatten_from_ndims(self._logstd, self.s1, self.s2)\n            else:\n                t = StochasticTensor(\n                    distribution=self,\n                    tensor=samples,\n                    n_samples=n_samples,\n                    group_ndims=group_ndims,\n                    is_reparameterized=self.is_reparameterized,\n                    log_prob=log_prob\n                )\n                self._mu = tf.transpose(results[1], [1, 2, 0, 3])\n                self._logstd = tf.transpose(results[2], [1, 2, 0, 3])\n                if self.need_unflatten:\n                    self._mu = tf.stack([spt.ops.unflatten_from_ndims(self._mu[i], self.s1, self.s2) for i in range(n_samples)], axis=0)\n                    self._logstd = tf.stack([spt.ops.unflatten_from_ndims(self._logstd[i], self.s1, self.s2) for i in range(n_samples)], axis=0)\n\n            return t\n\n    def log_prob(self, given, group_ndims=0, name=None):\n        with tf.name_scope(name=name, default_name='log_prob'):\n            if self.need_unflatten:\n                assert len(given.shape) == len(self.origin_input.shape)\n                assert given.shape[0] == self.origin_input.shape[0]\n                time_first_input = tf.transpose(self.origin_input, [2, 0, 1, 3])    # (window, sample, batch, feature)\n                # time_indices_shape: (n_sample, batch_size, z_dim)\n                time_indices_shape = tf.convert_to_tensor([tf.shape(given)[0], tf.shape(time_first_input)[2], self.z_dim])\n                given = tf.transpose(given, [2, 0, 1, 3])\n            else:\n                if len(given.shape) > 3:    # (n_sample, batch_size, window_length, z_dim)\n                    time_indices_shape = tf.convert_to_tensor([tf.shape(given)[0], tf.shape(self.time_first_input)[1], self.z_dim])\n                    given = tf.transpose(given, [2, 0, 1, 3])\n                    time_first_input = self.time_first_input\n                else:                       # (batch_size, window_length, z_dim)\n                    time_indices_shape = tf.convert_to_tensor([tf.shape(self.time_first_input)[1], self.z_dim])\n                    given = tf.transpose(given, [1, 0, 2])\n                    time_first_input = self.time_first_input\n            results = tf.scan(fn=self.log_prob_step,\n                               elems=(given, time_first_input),\n                               initializer=(tf.zeros(time_indices_shape),\n                                            tf.zeros(time_indices_shape),\n                                            tf.zeros(time_indices_shape),\n                                            tf.zeros(time_indices_shape)),\n                               back_prop=True\n                               )        # (window_length, ?, batch_size, z_dim)\n            if len(given.shape) > 3:\n                log_prob = tf.transpose(results[-1], [1, 2, 0, 3])\n            else:\n                log_prob = tf.transpose(results[-1], [1, 0, 2])\n\n            log_prob = spt.reduce_group_ndims(tf.reduce_sum, log_prob, group_ndims)\n            return log_prob\n\n    def prob(self, given, group_ndims=0, name=None):\n        with tf.name_scope(name=name, default_name='prob'):\n            log_prob = self.log_prob(given, group_ndims, name)\n            return tf.exp(log_prob)\n"
  },
  {
    "path": "algorithm/stack_predict.py",
    "content": "import mltk\nimport os\n\nfrom explib.eval_methods import get_best_f1, get_adjusted_composite_metrics\nfrom algorithm.utils import GraphNodes, get_data, time_generator, get_sliding_window_data_flow, get_score, \\\n    get_avg_recons\nimport tfsnippet as spt\nimport tensorflow as tf\nfrom tqdm import tqdm\nfrom algorithm.InterFusion import MTSAD\nfrom algorithm.InterFusion_swat import MTSAD_SWAT\nimport numpy as np\nfrom typing import Optional\nimport pickle\nfrom algorithm.mcmc_recons import mcmc_reconstruct, masked_reconstruct\nfrom algorithm.cal_IPS import cal_IPS\n\n__all__ = ['PredictConfig', 'final_testing', 'build_test_graph']\n\n\nclass PredictConfig(mltk.Config):\n    load_model_dir: Optional[str]\n\n    # evaluation params\n    test_n_z = 100\n    test_batch_size = 50\n    test_start = 0\n    max_test_size = None  # `None` means full test set\n\n    save_results = True\n\n    output_dirs = 'analysis_results'\n    train_score_filename = 'train_score.pkl'\n    test_score_filename = 'test_score.pkl'\n    preserve_feature_dim = False  # whether to preserve the feature dim in score. If `True`, the score will be a 2-dim ndarray\n    anomaly_score_calculate_latency = 1   # How many scores are averaged for the final score at a timestamp. `1` means use last point in each sliding window only.\n    plot_recons_results = True\n\n    use_mcmc = True               # use mcmc on the last point for anomaly detection\n    mcmc_iter = 10\n    mcmc_rand_mask = False\n    n_mc_chain: int = 10\n    pos_mask = True\n    mcmc_track = True             # use mcmc tracker for anomaly interpretation and calculate IPS.\n\n\ndef build_test_graph(chain: spt.VariationalChain, input_x, origin_chain: spt.VariationalChain=None) -> GraphNodes:\n    test_recons = tf.reduce_mean(chain.model['x'].log_prob(), axis=0)\n\n    logpx = chain.model['x'].log_prob()\n    logpz = chain.model['z2'].log_prob() + chain.model['z1'].log_prob()\n    logqz_x = chain.variational['z1'].log_prob() + chain.variational['z2'].log_prob()\n    test_lb = tf.reduce_mean(logpx + logpz - logqz_x, axis=0)\n\n    log_joint = logpx + logpz\n    latent_log_prob = logqz_x\n    test_ll = spt.importance_sampling_log_likelihood(log_joint=log_joint, latent_log_prob=latent_log_prob, axis=0)\n    test_nll = -test_ll\n\n    # average over sample dim\n    if origin_chain is not None:\n        full_recons_prob = tf.reduce_mean(\n            (chain.model['x'].distribution.base_distribution.log_prob(input_x) -\n             origin_chain.model['x'].distribution.base_distribution.log_prob(input_x)),\n            axis=0\n        )\n    else:\n        full_recons_prob = tf.reduce_mean(chain.model['x'].distribution.base_distribution.log_prob(input_x), axis=0)\n\n    if origin_chain is not None:\n        origin_log_joint = origin_chain.model['x'].log_prob() + origin_chain.model['z1'].log_prob() + origin_chain.model['z2'].log_prob()\n        origin_latent_log_prob = origin_chain.variational['z1'].log_prob() + origin_chain.variational['z2'].log_prob()\n        origin_ll = spt.importance_sampling_log_likelihood(log_joint=origin_log_joint, latent_log_prob=origin_latent_log_prob, axis=0)\n        test_ll_score = test_ll - origin_ll\n    else:\n        test_ll_score = test_ll\n\n    outputs = {\n        'test_nll': test_nll,\n        'test_lb': test_lb,\n        'test_recons': test_recons,\n        'test_kl': test_recons - test_lb,\n        'full_recons_prob': full_recons_prob,\n        'test_ll': test_ll_score\n    }\n\n    return GraphNodes(outputs)\n\n\ndef build_recons_graph(chain: spt.VariationalChain, window_length, feature_dim, unified_x_std=False) -> GraphNodes:\n    # average over sample dim\n    recons_x = tf.reduce_mean(chain.model['x'].distribution.base_distribution.mean, axis=0)\n    recons_x = spt.utils.InputSpec(shape=['?', window_length, feature_dim]).validate('recons', recons_x)\n    if unified_x_std:\n        recons_x_std = chain.model['x'].distribution.base_distribution.std\n        recons_x_std = spt.ops.broadcast_to_shape(recons_x_std, tf.shape(recons_x))\n    else:\n        recons_x_std = tf.reduce_mean(chain.model['x'].distribution.base_distribution.std, axis=0)\n    recons_x_std = spt.utils.InputSpec(shape=['?', window_length, feature_dim]).validate('recons_std', recons_x_std)\n    return GraphNodes({'recons_x': recons_x, 'recons_x_std': recons_x_std})\n\n\ndef get_recons_results(recons_nodes: GraphNodes, input_x, input_u, data_flow: spt.DataFlow, total_batch_count, dataset,\n                       mask=None, rand_x=None):\n    data_flow = data_flow.threaded(5)\n    recons_collector = []\n    recons_std_collector = []\n    session = spt.utils.get_default_session_or_error()\n    with data_flow:\n        for batch_x, batch_u in tqdm(data_flow, unit='step', total=total_batch_count, ascii=True):\n            if mask is not None:\n                batch_mask = np.zeros(shape=batch_x.shape)\n                batch_mask[:, -1, :] = 1    # mask all dims of the last point in x\n                if rand_x is not None:\n                    batch_output = recons_nodes.eval(session,\n                                                     feed_dict={input_x: batch_x, input_u: batch_u, mask: batch_mask,\n                                                                rand_x: np.random.random(batch_x.shape)})\n                else:\n                    batch_output = recons_nodes.eval(session, feed_dict={input_x: batch_x, input_u: batch_u, mask: batch_mask})\n            else:\n                batch_output = recons_nodes.eval(session, feed_dict={input_x: batch_x, input_u: batch_u})\n            for k, v in batch_output.items():\n                if k == 'recons_x':\n                    if dataset == 'SWaT' or dataset == 'WADI':\n                        # idx = min(10, v.shape[1])\n                        recons_collector.append(v[:, -10:, :])\n                    else:\n                        recons_collector.append(v)\n                elif k == 'recons_x_std':\n                    if dataset == 'SWaT' or dataset == 'WADI':\n                        # idx = min(10, v.shape[1])\n                        recons_std_collector.append(v[:, -10:, :])\n                    else:\n                        recons_std_collector.append(v)\n\n    all_recons = np.concatenate(recons_collector, axis=0)   # (data_length - window_length + 1, window_length, x_dim)\n    print(all_recons.shape)\n    all_recons_std = np.concatenate(recons_std_collector, axis=0)\n    return all_recons, all_recons_std\n\n\ndef final_testing(test_metrics: GraphNodes, input_x, input_u,\n                  data_flow: spt.DataFlow, total_batch_count, y_test=None, mask=None, rand_x=None):\n    data_flow = data_flow.threaded(5)\n    full_recons_collector = []\n    ll_collector = []\n    epoch_out = {}\n    stats = {}\n    session = spt.utils.get_default_session_or_error()\n    with data_flow:\n        for batch_x, batch_u in tqdm(data_flow, unit='step', total=total_batch_count, ascii=True):\n            if mask is not None:\n                batch_mask = np.zeros(shape=batch_x.shape)\n                batch_mask[:, -1, :] = 1  # mask all dims of the last point in x\n                if rand_x is not None:\n                    batch_output = test_metrics.eval(session, feed_dict={input_x: batch_x, input_u: batch_u, mask: batch_mask,\n                                                                         rand_x: np.random.random(batch_x.shape)})\n                else:\n                    batch_output = test_metrics.eval(session,\n                                                     feed_dict={input_x: batch_x, input_u: batch_u, mask: batch_mask})\n            else:\n                batch_output = test_metrics.eval(session, feed_dict={input_x: batch_x, input_u: batch_u})\n            for k, v in batch_output.items():\n\n                if k == 'full_recons_prob':\n                    full_recons_collector.append(v)\n                elif k == 'test_ll':\n                    ll_collector.append(v)\n                    if k not in epoch_out:\n                        epoch_out[k] = []\n                    epoch_out[k].append(v)\n                else:\n                    if k not in epoch_out:\n                        epoch_out[k] = []\n                    epoch_out[k].append(v)\n\n    # save the results of this epoch, and compute epoch stats. Take average over both batch and window_length dim.\n    for k, v in epoch_out.items():\n        epoch_out[k] = np.concatenate(epoch_out[k], axis=0)\n        if k not in stats:\n            stats[k] = []\n        stats[k].append(float(np.mean(epoch_out[k])))\n\n    # collect full recons prob for calculate anomaly score\n    full_recons_probs = np.concatenate(full_recons_collector, axis=0)   # (data_length-window_length+1, window_length, x_dim)\n    ll = np.concatenate(ll_collector, axis=0)\n\n    if y_test is not None:\n        assert full_recons_probs.shape[0] + full_recons_probs.shape[1] - 1 == len(y_test)\n        tmp1 = []\n        for i in range(full_recons_probs.shape[0]):\n            if y_test[i + full_recons_probs.shape[1] - 1] < 0.5:\n                tmp1.append(np.sum(full_recons_probs[i, -1], axis=-1))  # normal point recons score\n        stats['normal_point_test_recons'] = [float(np.mean(tmp1))]\n\n    # calculate average statistics\n    for k, v in stats.items():\n        stats[k] = float(np.mean(v))\n\n    return stats, full_recons_probs, ll\n\n\ndef mcmc_tracker(flow: spt.DataFlow, baseline, model, input_x, input_u, mask, max_iter, total_window_num,\n                 window_length, x_dim, mask_last=False, pos_mask=False, use_rand_mask=False, n_mc_chain=1):\n    # the baseline is the avg total score in a window on training set.\n    session = spt.utils.get_default_session_or_error()\n    last_x = tf.placeholder(dtype=tf.float32, shape=[None, window_length, x_dim], name='last_x')\n\n    x_r = masked_reconstruct(model.reconstruct, last_x, input_u, mask)\n    score, recons_mean, recons_std = model.get_score(x_embed=x_r, x_eval=input_x, u=input_u)\n    tot_score = tf.reduce_sum(tf.multiply(score, tf.cast((1-mask), score.dtype)))\n\n    def avg_multi_chain(x, n_chain):\n        shape = (-1,) + (n_chain,) + x.shape[1:]\n        return np.mean(x.reshape(shape), axis=1)\n\n    res = {}\n    with flow.threaded(5) as flow:\n        for batch_x, batch_u, batch_score, batch_ori_recons, batch_ori_std, batch_idx \\\n                in tqdm(flow, unit='step', total=total_window_num, ascii=True):\n            batch_idx = batch_idx[0]\n            res[batch_idx] = {'x': [batch_x], 'recons': [batch_ori_recons], 'std': [batch_ori_std], 'score': [batch_score],\n                              'K': [0], 'iter': [-1], 'mask': [np.zeros(shape=batch_x.shape)],\n                              'total_score': [np.mean(np.sum(batch_score, axis=-1))]}\n            best_score = batch_score\n            best_total_score = np.mean(np.sum(batch_score, axis=-1))\n            best_K = 0\n            if pos_mask:\n                pos_scores = np.mean(batch_score, axis=0)   # (window, x_dim)\n                sorted_pos_idx = np.argsort(pos_scores, axis=None)\n                potential_dim_num = np.sum((pos_scores < (baseline/(x_dim*window_length))).astype(np.int32))\n            else:\n                dim_scores = np.mean(batch_score, axis=(-2,-3))     # (x_dim, )\n                sorted_dim_idx = np.argsort(dim_scores)\n                potential_dim_num = np.sum((dim_scores < (baseline/(x_dim*window_length))).astype(np.int32))    # num of dims whose avg score < baseline\n\n            if potential_dim_num > 0:\n                K_init = max(potential_dim_num//5, 1)\n                K_inc = max(potential_dim_num//10, 1)\n            else:\n                res[batch_idx]['best_score'] = best_score\n                res[batch_idx]['best_total_score'] = best_total_score\n                res[batch_idx]['best_K'] = best_K\n                continue\n            if use_rand_mask:\n                rand_x = np.random.random(size=batch_x.shape)\n            if pos_mask:\n                max_K = x_dim * window_length\n            else:\n                max_K = x_dim\n            for K in range(K_init, min(potential_dim_num+1, max_K), K_inc):\n                if pos_mask:\n                    mask_idx = sorted_pos_idx[:K]\n                    batch_mask = np.zeros(shape=batch_x.shape)\n                    batch_mask = batch_mask.reshape([batch_x.shape[0], -1])\n                    batch_mask[:, mask_idx] = 1\n                    batch_mask = batch_mask.reshape(batch_x.shape)\n                else:\n                    mask_idx = sorted_dim_idx[:K]\n                    batch_mask = np.zeros(shape=batch_x.shape)\n                    batch_mask[:, :, mask_idx] = 1\n                if mask_last:\n                    batch_mask[:, -1, :] = 1\n\n                batch_last_x = batch_x\n                if use_rand_mask:\n                    batch_last_x = np.where(batch_mask.astype(np.bool), rand_x, batch_last_x)\n                if n_mc_chain > 1:\n                    init_x = np.repeat(batch_x, n_mc_chain, axis=0)\n                    init_u = np.repeat(batch_u, n_mc_chain, axis=0)\n                    init_mask = np.repeat(batch_mask, n_mc_chain, axis=0)\n                    init_last_x = np.repeat(batch_last_x, n_mc_chain, axis=0)\n                for i in range(max_iter):\n                    if n_mc_chain > 1:\n                        x_mc, x_recons, x_std, x_score, x_tot_score = \\\n                            session.run([x_r, recons_mean, recons_std, score, tot_score],\n                                        feed_dict={input_x: init_x, input_u: init_u, mask: init_mask,\n                                                   last_x: init_last_x})\n                        init_last_x = x_mc\n                        x_mc = avg_multi_chain(x_mc, n_mc_chain)\n                        x_recons = avg_multi_chain(x_recons, n_mc_chain)\n                        x_std = avg_multi_chain(x_std, n_mc_chain)\n                        x_score = avg_multi_chain(x_score, n_mc_chain)\n                        x_tot_score = float(x_tot_score) / float(n_mc_chain)\n                    else:\n                        x_mc, x_recons, x_std, x_score, x_tot_score = \\\n                            session.run([x_r, recons_mean, recons_std, score, tot_score],\n                                        feed_dict={input_x: batch_x, input_u: batch_u, mask: batch_mask, last_x: batch_last_x})\n                        batch_last_x = x_mc\n                    total_score = float(x_tot_score) / (window_length * x_dim - np.sum(batch_mask)) / batch_x.shape[0] * x_dim\n                    res[batch_idx]['x'].append(x_mc)\n                    res[batch_idx]['recons'].append(x_recons)\n                    res[batch_idx]['std'].append(x_std)\n                    res[batch_idx]['score'].append(x_score)\n                    res[batch_idx]['K'].append(K)\n                    res[batch_idx]['iter'].append(i)\n                    res[batch_idx]['mask'].append(batch_mask)\n                    res[batch_idx]['total_score'].append(total_score)\n\n                last_score = res[batch_idx]['total_score'][-1]\n                if last_score >= best_total_score:\n                    best_total_score = last_score\n                    best_score = res[batch_idx]['score'][-1]\n                    best_K = res[batch_idx]['K'][-1]\n\n                if best_total_score >= (baseline/window_length):\n                    break\n            res[batch_idx]['best_score'] = best_score\n            res[batch_idx]['best_total_score'] = best_total_score\n            res[batch_idx]['best_K'] = best_K\n    return res\n\n\ndef log_mean_exp(x, axis, keepdims=False):\n    x_max = np.max(x, axis=axis, keepdims=True)\n    ret = x_max + np.log(np.mean(np.exp(x - x_max), axis=axis, keepdims=True))\n    if not keepdims:\n        ret = np.squeeze(ret, axis=axis)\n    return ret\n\n\ndef log_sum_exp(x, axis, keepdims=False):\n    x_max = np.max(x, axis=axis, keepdims=True)\n    ret = x_max + np.log(np.sum(np.exp(x - x_max), axis=axis, keepdims=True))\n    if not keepdims:\n        ret = np.squeeze(ret, axis=axis)\n    return ret\n\n\ndef main(exp: mltk.Experiment[PredictConfig], test_config: PredictConfig):\n    if test_config.load_model_dir is None:\n        raise ValueError('`--load_model_dir` is required.')\n\n    exp_config_path = os.path.join(test_config.load_model_dir, 'config.json')\n    from algorithm.stack_train import ExpConfig\n    loader = mltk.ConfigLoader(ExpConfig())\n    loader.load_file(exp_config_path)\n    train_config = loader.get()\n\n    print(mltk.format_key_values(train_config, title='Train configurations'))\n    print('')\n    print(mltk.format_key_values(test_config, title='Test configurations'))\n    print('')\n\n    # set TFSnippet settings\n    spt.settings.enable_assertions = False\n    spt.settings.check_numerics = train_config.check_numerics\n\n    exp.make_dirs(test_config.output_dirs)\n\n    # prepare the data\n    # simple data\n    (x_train, _), (x_test, y_test) = \\\n        get_data(train_config.dataset, train_config.train.max_train_size, train_config.test.max_test_size,\n                 train_start=train_config.train.train_start, test_start=train_config.test.test_start,\n                 valid_portion=train_config.train.valid_portion)\n\n    if train_config.use_time_info:\n        u_train = np.asarray([time_generator(_i) for _i in range(len(x_train))])  # (train_size, u_dim)\n        u_test = np.asarray([time_generator(len(x_train) + _i) for _i in range(len(x_test))])  # (test_size, u_dim)\n    else:\n        u_train = np.zeros([len(x_train), train_config.model.u_dim])  # (train_size, u_dim)\n        u_test = np.zeros([len(x_test), train_config.model.u_dim])\n\n    # prepare data_flow\n    test_flow = get_sliding_window_data_flow(window_size=train_config.model.window_length, batch_size=test_config.test_batch_size,\n                                             x=x_test, u=u_test, shuffle=False, skip_incomplete=False)\n    evaluate_score_train_flow = get_sliding_window_data_flow(window_size=train_config.model.window_length,\n                                                             batch_size=test_config.test_batch_size,\n                                                             x=x_train, u=u_train, shuffle=False,\n                                                             skip_incomplete=False)\n\n    # build computation graph\n    if train_config.dataset == 'SWaT' or train_config.dataset == 'WADI':\n        model = MTSAD_SWAT(train_config.model, scope='model')\n    else:\n        model = MTSAD(train_config.model, scope='model')\n\n    # input placeholders\n    input_x = tf.placeholder(dtype=tf.float32, shape=[None, train_config.model.window_length, train_config.model.x_dim], name='input_x')\n    input_u = tf.placeholder(dtype=tf.float32, shape=[None, train_config.model.window_length, train_config.model.u_dim], name='input_u')\n    mask = tf.placeholder(dtype=tf.int32, shape=[None, train_config.model.window_length, train_config.model.x_dim], name='mask')\n    rand_x = tf.placeholder(dtype=tf.float32, shape=[None, train_config.model.window_length, train_config.model.x_dim], name='rand_x')\n\n    tmp_out = None\n    if test_config.use_mcmc:\n        with tf.name_scope('mcmc_init'):\n            tmp_qnet = model.q_net(input_x, u=input_u, n_z=test_config.test_n_z)\n            tmp_chain = tmp_qnet.chain(model.p_net, observed={'x': input_x}, latent_axis=0, u=input_u)\n            tmp_out = tf.reduce_mean(tmp_chain.vi.lower_bound.elbo())\n\n    # derive testing nodes\n    with tf.name_scope('testing'):\n\n        if test_config.use_mcmc:\n            if test_config.mcmc_rand_mask:      # use random value to mask the initial input for mcmc (otherwise use the original one)\n                if test_config.n_mc_chain > 1:  # average the results of multi-mcmc chain for each input x.\n                    init_x = tf.where(tf.cast(mask, dtype=tf.bool), rand_x, input_x)\n                    init_x, s1, s2 = spt.ops.flatten_to_ndims(tf.tile(tf.expand_dims(init_x, 1), [1, test_config.n_mc_chain, 1, 1]), 3)\n                    init_u, _, _ = spt.ops.flatten_to_ndims(tf.tile(tf.expand_dims(input_u, 1), [1, test_config.n_mc_chain, 1, 1]), 3)\n                    init_mask, _, _ = spt.ops.flatten_to_ndims(tf.tile(tf.expand_dims(mask, 1), [1, test_config.n_mc_chain, 1, 1]), 3)\n                    x_mcmc = mcmc_reconstruct(model.reconstruct, init_x, init_u, init_mask, test_config.mcmc_iter, back_prop=False)\n                    x_mcmc = spt.ops.unflatten_from_ndims(x_mcmc, s1, s2)\n                    x_mcmc = tf.reduce_mean(x_mcmc, axis=1)\n                else:\n                    init_x = tf.where(tf.cast(mask, dtype=tf.bool), rand_x, input_x)\n                    x_mcmc = mcmc_reconstruct(model.reconstruct, init_x, input_u, mask, test_config.mcmc_iter, back_prop=False)\n            else:\n                if test_config.n_mc_chain > 1:\n                    init_x, s1, s2 = spt.ops.flatten_to_ndims(tf.tile(tf.expand_dims(input_x, 1), [1, test_config.n_mc_chain, 1, 1]), 3)\n                    init_u, _, _ = spt.ops.flatten_to_ndims(tf.tile(tf.expand_dims(input_u, 1), [1, test_config.n_mc_chain, 1, 1]), 3)\n                    init_mask, _, _ = spt.ops.flatten_to_ndims(tf.tile(tf.expand_dims(mask, 1), [1, test_config.n_mc_chain, 1, 1]), 3)\n                    x_mcmc = mcmc_reconstruct(model.reconstruct, init_x, init_u, init_mask, test_config.mcmc_iter, back_prop=False)\n                    x_mcmc = spt.ops.unflatten_from_ndims(x_mcmc, s1, s2)\n                    x_mcmc = tf.reduce_mean(x_mcmc, axis=1)\n                else:\n                    x_mcmc = mcmc_reconstruct(model.reconstruct, input_x, input_u, mask, test_config.mcmc_iter, back_prop=False)\n        else:\n            x_mcmc = input_x\n\n        test_q_net = model.q_net(x_mcmc, u=input_u, n_z=test_config.test_n_z)\n        test_chain = test_q_net.chain(model.p_net, observed={'x': input_x}, latent_axis=0, u=input_u)\n\n        test_metrics = build_test_graph(test_chain, input_x)\n\n        if test_config.plot_recons_results:\n            recons_nodes = build_recons_graph(test_chain, train_config.model.window_length, train_config.model.x_dim, train_config.model.unified_px_logstd)\n\n    # obtain params to restore\n    variables_to_restore = tf.global_variables()\n\n    restore_path = os.path.join(test_config.load_model_dir, 'result_params/restored_params.dat')\n\n    # obtain the variables initializer\n    var_initializer = tf.variables_initializer(tf.global_variables())\n\n    test_flow = test_flow.threaded(5)\n    evaluate_score_train_flow = evaluate_score_train_flow.threaded(5)\n\n    with spt.utils.create_session().as_default() as session:\n\n        session.run(var_initializer)\n\n        saver = tf.train.Saver(var_list=variables_to_restore)\n        saver.restore(session, restore_path)\n\n        print('Model params restored.')\n\n        # Evaluate the whole network\n        if test_config.use_mcmc:\n            for batch_x, batch_u in test_flow:\n                _ = session.run(tmp_out, feed_dict={input_x: batch_x, input_u: batch_u})\n                break\n\n        # do evaluation\n        print('')\n        print('*************Evaluate score on testing set************')\n\n        test_batch_count = (len(x_test) - train_config.model.window_length + test_config.test_batch_size) // test_config.test_batch_size\n\n        test_stats, test_full_recons_probs, test_ll = final_testing(test_metrics,\n                                                                    input_x,\n                                                                    input_u,\n                                                                    test_flow,\n                                                                    test_batch_count,\n                                                                    y_test,\n                                                                    mask=mask if test_config.use_mcmc else None,\n                                                                    rand_x=rand_x if test_config.mcmc_rand_mask else None)\n\n        print('')\n        print(mltk.format_key_values(test_stats, 'Final testing statistics'))\n        exp.update_results(test_stats)\n\n        test_score = get_score(test_full_recons_probs, preserve_feature_dim=test_config.preserve_feature_dim,\n                               score_avg_window_size=test_config.anomaly_score_calculate_latency)\n\n        # evaluate score on train set\n        print('')\n        print('*************Evaluate score on training set************')\n\n        if train_config.dataset != 'WADI':\n            train_set_batch_count = (len(x_train) - train_config.model.window_length + test_config.test_batch_size) // test_config.test_batch_size\n\n            train_stats, train_full_recons_probs, _ = final_testing(test_metrics,\n                                                                    input_x,\n                                                                    input_u,\n                                                                    evaluate_score_train_flow,\n                                                                    train_set_batch_count,\n                                                                    mask=mask if test_config.use_mcmc else None,\n                                                                    rand_x=rand_x if test_config.mcmc_rand_mask else None)\n\n            print('')\n            print(mltk.format_key_values(train_stats, 'Training set evaluation statistics'))\n\n            train_score = get_score(train_full_recons_probs, preserve_feature_dim=test_config.preserve_feature_dim,\n                                    score_avg_window_size=test_config.anomaly_score_calculate_latency)\n\n            print('train_score shape: ', train_score.shape)\n\n            if test_config.save_results:\n                np.savez(os.path.join((exp.abspath(test_config.output_dirs)), 'full_recons_window_probs.npz'),\n                         full_full_test_recons_window_probs=test_full_recons_probs,\n                         full_train_recons_window_probs=get_score(train_full_recons_probs, preserve_feature_dim=True,\n                                       score_avg_window_size=test_config.anomaly_score_calculate_latency),\n                         full_test_recons_window_probs=get_score(test_full_recons_probs, preserve_feature_dim=True,\n                                       score_avg_window_size=test_config.anomaly_score_calculate_latency))\n\n            if not test_config.mcmc_track:\n                del test_full_recons_probs\n\n            del train_full_recons_probs\n\n        # for reconstruct plotting\n        import matplotlib.pyplot as plt\n        exp.make_dirs('figures')\n        if test_config.plot_recons_results:\n            print('')\n            print('*************Calculating and plotting reconstruction data************')\n            all_test_recons, all_test_recons_std = get_recons_results(recons_nodes,\n                                                                      input_x,\n                                                                      input_u,\n                                                                      test_flow,\n                                                                      test_batch_count,\n                                                                      train_config.dataset,\n                                                                      mask=mask if test_config.use_mcmc else None,\n                                                                      rand_x=rand_x if test_config.mcmc_rand_mask else None)\n            all_train_recons, all_train_recons_std = get_recons_results(recons_nodes,\n                                                                        input_x,\n                                                                        input_u,\n                                                                        evaluate_score_train_flow,\n                                                                        train_set_batch_count,\n                                                                        train_config.dataset,\n                                                                        mask=mask if test_config.use_mcmc else None,\n                                                                        rand_x=rand_x if test_config.mcmc_rand_mask else None)\n            final_test_recons = get_avg_recons(all_test_recons, window_length=train_config.model.window_length, recons_avg_window_size=test_config.anomaly_score_calculate_latency)\n            final_train_recons = get_avg_recons(all_train_recons, window_length=train_config.model.window_length, recons_avg_window_size=test_config.anomaly_score_calculate_latency)\n            if not test_config.mcmc_track:\n                del all_test_recons\n            del all_train_recons\n            if test_config.anomaly_score_calculate_latency == 1:\n                final_test_recons_std = get_avg_recons(all_test_recons_std, window_length=train_config.model.window_length)\n                final_train_recons_std = get_avg_recons(all_train_recons_std, window_length=train_config.model.window_length)\n                np.savez(os.path.join(exp.abspath(test_config.output_dirs), 'recons_plotting_data.npz'),\n                         x_train=x_train, x_test=x_test, x_train_recons=final_train_recons,\n                         x_train_recons_std=final_train_recons_std, x_test_recons=final_test_recons,\n                         x_test_recons_std=final_test_recons_std, y_test=y_test)\n            else:       # average x_recons_std is meaningless if calculate latency > 1\n                np.savez(os.path.join(exp.abspath(test_config.output_dirs), 'recons_plotting_data.npz'),\n                     x_train=x_train, x_test=x_test, x_train_recons=final_train_recons, x_test_recons=final_test_recons, y_test=y_test)\n            if not test_config.mcmc_track:\n                del all_test_recons_std\n            del all_train_recons_std\n\n        if test_config.train_score_filename is not None and train_config.dataset != 'WADI':\n            with open(os.path.join(exp.abspath(test_config.output_dirs), test_config.train_score_filename), 'wb') as file:\n                pickle.dump(train_score, file)\n\n        if test_config.test_score_filename is not None:\n            with open(os.path.join(exp.abspath(test_config.output_dirs), test_config.test_score_filename), 'wb') as file:\n                pickle.dump(test_score, file)\n\n        print('')\n        print('*************Calculating best F1-score*****************')\n\n        y_test = y_test[-len(test_score):]\n\n        # get best f1\n        t, th = get_best_f1(test_score, y_test)\n\n        best_thresh = th\n\n        # output the results\n        exp.update_results({\n            'best-f1': t[0],\n            'precision': t[1],\n            'recall': t[2],\n            'TP': t[3],\n            'TN': t[4],\n            'FP': t[5],\n            'FN': t[6],\n            'threshold': th\n        })\n\n        auroc, ap, _, _, _, _, _ = get_adjusted_composite_metrics(test_score, y_test)\n        exp.update_results({\n            'auroc': auroc,\n            'ap': ap\n        })\n\n        if test_config.mcmc_track:\n            # find the TP points that need to be interpreted according to best_thresh\n            best_idx = np.logical_and(test_score <= best_thresh, y_test > 0.5)\n            best_idx = np.where(best_idx)[0]\n            total_x, total_u, total_score, total_recons, total_std, total_idx = [], [], [], [], [], []\n            for i in best_idx:\n                total_x.append(x_test[i:i+train_config.model.window_length, :])\n                total_u.append(u_test[i:i+train_config.model.window_length, :])\n                total_score.append(test_full_recons_probs[i])\n                total_recons.append(all_test_recons[i])\n                total_std.append(all_test_recons_std[i])\n                total_idx.append(i + train_config.model.window_length - 1)\n            total_x = np.stack(total_x, axis=0)\n            total_u = np.stack(total_u, axis=0)\n            total_score = np.stack(total_score, axis=0)\n            total_recons = np.stack(total_recons, axis=0)\n            total_std = np.stack(total_std, axis=0)\n            total_idx = np.stack(total_idx, axis=0)\n            best_flow = spt.DataFlow.arrays([total_x, total_u, total_score, total_recons, total_std, total_idx],\n                                            batch_size=1, shuffle=False, skip_incomplete=False)\n\n            best_baseline = train_stats['test_recons']\n            best_window_num = total_x.shape[0]\n\n            res = mcmc_tracker(best_flow, best_baseline, model, input_x, input_u, mask, test_config.mcmc_iter,\n                               best_window_num, train_config.model.window_length, train_config.model.x_dim,\n                               test_config.use_mcmc, test_config.pos_mask,\n                               test_config.mcmc_rand_mask, test_config.n_mc_chain)\n\n            with open(os.path.join(exp.abspath(test_config.output_dirs), 'mcmc_tracker.pkl'), 'wb') as file:\n                pickle.dump(res, file)\n\n            del res\n\n            res = cal_IPS(path=exp.abspath(test_config.output_dirs), dataset=train_config.dataset,\n                                    mcmc=True,\n                                    is_pretrain=False)\n            exp.update_results({'IPS': res['mcmc_p=100: wd_last_itv_min_weight'],\n                                'IPS@150%': res['mcmc_p=150: wd_last_itv_min_weight']})\n\n        print('')\n        print(mltk.format_key_values(exp.results), 'Results')\n\n\nif __name__ == '__main__':\n    with mltk.Experiment(PredictConfig()) as exp:\n        main(exp, exp.config)\n"
  },
  {
    "path": "algorithm/stack_train.py",
    "content": "# -*- coding: utf-8 -*-\nimport os\n\nimport logging\nimport time\nimport numpy as np\nimport tensorflow as tf\nimport tfsnippet as spt\nfrom tfsnippet.scaffold import TrainLoop\nfrom tfsnippet.trainer import Trainer, Evaluator\nimport mltk\nfrom algorithm.utils import get_data_dim, get_data, get_sliding_window_data_flow, time_generator, GraphNodes\nimport random\n\nfrom algorithm.InterFusion import ModelConfig, MTSAD\nfrom algorithm.InterFusion_swat import MTSAD_SWAT\nfrom algorithm.stack_predict import PredictConfig\n\n\nclass TrainConfig(mltk.Config):\n    # training params\n    batch_size = 100\n    pretrain_max_epoch = 20\n    max_epoch = 20\n    train_start = 0\n    max_train_size = None  # `None` means full train set\n    initial_lr = 0.001\n    lr_anneal_factor = 0.5\n    lr_anneal_epoch_freq = 10\n    lr_anneal_step_freq = None\n    pretrain_lr_anneal_epoch_freq = 10\n\n    early_stopping = True\n    valid_portion = 0.3\n\n    save_test_stats = True\n\n\nclass ExpConfig(mltk.Config):\n    seed = int(time.time())\n\n    dataset = 'omi-1'\n\n    # model params\n    model = ModelConfig()\n\n    @mltk.root_checker()\n    def _model_post_checker(self, v: 'ExpConfig'):\n        if v.model.x_dim == -1:\n            v.model.x_dim = get_data_dim(v.dataset)\n        if v.dataset == 'SWaT':\n            v.model.z_dim = 2\n        if v.dataset == 'WADI':\n            v.model.z_dim = 4\n\n    use_time_info = False  # whether to use time information (minute, hour, day) as input u. discarded.\n\n    model_type = 'mtsad'\n\n    # train params\n    train = TrainConfig()\n\n    @mltk.root_checker()\n    def _train_post_checker(self, v: 'ExpConfig'):\n        if v.dataset == 'SWaT' or v.dataset == 'WADI':\n            v.train.max_epoch = 15\n            v.train.save_test_stats = False\n            v.train.pretrain_max_epoch = 10\n            v.train.pretrain_lr_anneal_epoch_freq = 5\n            v.train.lr_anneal_epoch_freq = 5\n        if v.dataset == 'SWaT':\n            v.train.initial_lr = 0.0005\n        if v.dataset == 'WADI':\n            v.train.initial_lr = 0.0002\n\n    test = PredictConfig()\n\n    # debugging params\n    write_summary = False\n    write_histogram_summary = False\n    check_numerics = False\n    save_results = True\n    save_ckpt = True\n    ckpt_epoch_freq = 10\n    ckpt_max_keep = 10\n    pretrain_ckpt_epoch_freq = 20\n    pretrain_ckpt_max_keep = 10\n\n    exp_dir_save_path = None    # The file path to save the exp dirs for batch run training on different datasets.\n\n\ndef get_lr_value(init_lr,\n                 anneal_factor,\n                 anneal_freq,\n                 loop: spt.TrainLoop,\n                 ) -> spt.DynamicValue:\n    \"\"\"\n    Get the learning rate scheduler for specified experiment.\n\n    Args:\n        exp: The experiment object.\n        loop: The train loop object.\n\n    Returns:\n        A dynamic value, which returns the learning rate each time\n        its `.get()` is called.\n    \"\"\"\n    return spt.AnnealingScalar(\n        loop=loop,\n        initial_value=init_lr,\n        ratio=anneal_factor,\n        epochs=anneal_freq,\n    )\n\n\ndef sgvb_loss(qnet, pnet, metrics_dict: GraphNodes, prefix='train_', name=None):\n    with tf.name_scope(name, default_name='sgvb_loss'):\n        logpx_z = pnet['x'].log_prob(name='logpx_z')\n        logpz1_z2 = pnet['z1'].log_prob(name='logpz1_z2')\n        logpz2 = pnet['z2'].log_prob(name='logpz2')\n        logpz = logpz1_z2 + logpz2\n        logqz1_x = qnet['z1'].log_prob(name='logqz1_x')\n        logqz2_x = qnet['z2'].log_prob(name='logqz2_x')\n        logqz_x = logqz1_x + logqz2_x\n\n        recons_term = tf.reduce_mean(logpx_z)\n        kl_term = tf.reduce_mean(logqz_x - logpz)\n        metrics_dict[prefix + 'recons'] = recons_term\n        metrics_dict[prefix + 'kl'] = kl_term\n\n        return -tf.reduce_mean(logpx_z + logpz - logqz_x)\n\n\ndef main(exp: mltk.Experiment[ExpConfig], config: ExpConfig):\n    logging.basicConfig(\n        level='INFO',\n        format='%(asctime)s [%(levelname)s] %(name)s: %(message)s'\n    )\n\n    # print the current seed and generate three seeds\n    logging.info('Current random seed: %s', config.seed)\n    np.random.seed(config.seed)\n    random.seed(np.random.randint(0xffffffff))\n    tf.set_random_seed(np.random.randint(0xffffffff))\n    np.random.seed(np.random.randint(0xffffffff))\n\n    spt.settings.check_numerics = config.check_numerics\n    spt.settings.enable_assertions = False\n\n    # print the config\n    print(mltk.format_key_values(config, title='Configurations'))\n    print('')\n\n    # open the result object and prepare for result directories\n    exp.make_dirs('train_summary')\n    exp.make_dirs('result_params')\n    exp.make_dirs('ckpt_params')\n    exp.make_dirs(config.test.output_dirs)\n\n    # prepare the data\n    # simple data\n    (x_train, _), (x_test, y_test) = \\\n        get_data(config.dataset, config.train.max_train_size, config.test.max_test_size,\n                 train_start=config.train.train_start, test_start=config.test.test_start,\n                 valid_portion=config.train.valid_portion)\n\n    if config.use_time_info:\n        u_train = np.asarray([time_generator(_i) for _i in range(len(x_train))])  # (train_size, u_dim)\n        u_test = np.asarray([time_generator(len(x_train) + _i) for _i in range(len(x_test))])  # (test_size, u_dim)\n    else:\n        u_train = np.zeros([len(x_train), config.model.u_dim])  # (train_size, u_dim)\n        u_test = np.zeros([len(x_test), config.model.u_dim])\n\n    split_idx = int(len(x_train) * config.train.valid_portion)\n    x_train, x_valid = x_train[:-split_idx], x_train[-split_idx:]\n    u_train, u_valid = u_train[:-split_idx], u_train[-split_idx:]\n\n    # prepare data_flow\n    train_flow = get_sliding_window_data_flow(window_size=config.model.window_length,\n                                              batch_size=config.train.batch_size,\n                                              x=x_train, u=u_train, shuffle=True, skip_incomplete=True)\n    valid_flow = get_sliding_window_data_flow(window_size=config.model.window_length,\n                                              batch_size=config.train.batch_size,\n                                              x=x_valid, u=u_valid, shuffle=False, skip_incomplete=False)\n\n    # build computation graph\n    if config.dataset == 'SWaT' or config.dataset == 'WADI':\n        model = MTSAD_SWAT(config.model, scope='model')\n    else:\n        model = MTSAD(config.model, scope='model')\n\n    # input placeholders\n    input_x = tf.placeholder(dtype=tf.float32, shape=[None, config.model.window_length, config.model.x_dim],\n                             name='input_x')\n    input_u = tf.placeholder(dtype=tf.float32, shape=[None, config.model.window_length, config.model.u_dim],\n                             name='input_u')\n    learning_rate = tf.placeholder(dtype=tf.float32, shape=(), name='learning_rate')\n    is_training = tf.placeholder(dtype=tf.bool, shape=(), name='is_training')\n\n    # derive training nodes\n    with tf.name_scope('training'):\n        # pretrain time-vae to get z2\n        pretrain_q_net = model.pretrain_q_net(input_x, is_training=is_training)\n        pretrain_chain = pretrain_q_net.chain(model.pretrain_p_net, observed={'x': input_x}, is_training=is_training)\n        pretrain_loss = tf.reduce_mean(pretrain_chain.vi.training.sgvb()) + tf.losses.get_regularization_loss()\n        pretrain_train_recons = tf.reduce_mean(pretrain_chain.model['x'].log_prob())\n\n        # train the whole network with z1 and z2\n        train_q_net = model.q_net(input_x, u=input_u, is_training=is_training)\n        train_chain = train_q_net.chain(model.p_net, observed={'x': input_x}, u=input_u, is_training=is_training)\n        train_metrics = GraphNodes()\n        vae_loss = sgvb_loss(train_chain.variational, train_chain.model, train_metrics, name='train_sgvb_loss')\n        reg_loss = tf.losses.get_regularization_loss()\n        loss = vae_loss + reg_loss\n        train_metrics['loss'] = loss\n\n    with tf.name_scope('validation'):\n        # pretrain validation\n        pretrain_valid_q_net = model.pretrain_q_net(input_x, n_z=config.test.test_n_z)\n        pretrain_valid_chain = pretrain_valid_q_net.chain(model.pretrain_p_net, observed={'x': input_x}, latent_axis=0)\n        pretrain_valid_loss = tf.reduce_mean(pretrain_valid_chain.vi.training.sgvb()) + tf.losses.get_regularization_loss()\n        pretrain_valid_recons = tf.reduce_mean(pretrain_valid_chain.model['x'].log_prob())\n\n        # validation of the whole network\n        valid_q_net = model.q_net(input_x, u=input_u, n_z=config.test.test_n_z)\n        valid_chain = valid_q_net.chain(model.p_net, observed={'x': input_x}, latent_axis=0, u=input_u)\n        valid_metrics = GraphNodes()\n        valid_loss = sgvb_loss(valid_chain.variational, valid_chain.model, valid_metrics, prefix='valid_',\n                               name='valid_sgvb_loss') + tf.losses.get_regularization_loss()\n        valid_metrics['valid_loss'] = valid_loss\n\n    # pretrain\n    pre_variables_to_save = sum(\n            [tf.global_variables('model/pretrain_q_net'), tf.global_variables('model/pretrain_p_net'),\n             tf.global_variables('model/h_for_qz'), tf.global_variables('model/h_for_px')],\n            []\n    )\n    pre_train_params = sum(\n            [tf.trainable_variables('model/pretrain_q_net'), tf.trainable_variables('model/pretrain_p_net'),\n             tf.trainable_variables('model/h_for_qz'), tf.trainable_variables('model/h_for_px')],\n            []\n    )\n    pre_optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n    pre_gradients = pre_optimizer.compute_gradients(pretrain_loss, var_list=pre_train_params)\n    with tf.name_scope('PreClipGradients'):\n        for i, (g, v) in enumerate(pre_gradients):\n            if g is not None:\n                pre_gradients[i] = (tf.clip_by_norm(\n                    spt.utils.maybe_check_numerics(g, message='gradient on %s exceed' % str(v.name)), 10), v)\n    with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):\n        pre_train_op = pre_optimizer.apply_gradients(pre_gradients)\n\n    # obtain params and gradients (whole model)\n    variables_to_save = tf.global_variables()\n    train_params = tf.trainable_variables()\n\n    # optimizer\n    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n    gradients = optimizer.compute_gradients(loss, var_list=train_params)\n    # clip gradient by norm\n    with tf.name_scope('ClipGradients'):\n        for i, (g, v) in enumerate(gradients):\n            if g is not None:\n                gradients[i] = (tf.clip_by_norm(\n                    spt.utils.maybe_check_numerics(g, message=\"gradient on %s exceed\" % str(v.name)), 10), v)\n    with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):\n        train_op = optimizer.apply_gradients(gradients)\n\n    pre_var_groups = [\n        model.variable_scope.name + '/pretrain_q_net',\n\n        model.variable_scope.name + '/pretrain_p_net',\n\n        model.variable_scope.name + '/h_for_qz',\n\n        model.variable_scope.name + '/h_for_px'\n    ]\n\n    var_groups = [\n        # for q_net\n        model.variable_scope.name + '/q_net',\n\n        # for p_net\n        model.variable_scope.name + '/p_net',\n\n        # for flow\n        model.variable_scope.name + '/posterior_flow'\n    ]\n\n    var_initializer = tf.variables_initializer(tf.global_variables())\n\n    train_flow = train_flow.threaded(5)\n    valid_flow = valid_flow.threaded(5)\n\n    pre_loop = TrainLoop(param_vars=pre_variables_to_save,\n                         var_groups=pre_var_groups,\n                         max_epoch=config.train.pretrain_max_epoch,\n                         summary_dir=(exp.abspath('pre_train_summary') if config.write_summary else None),\n                         summary_graph=tf.get_default_graph(),\n                         summary_commit_freqs={'pretrain_loss': 10},\n                         early_stopping=config.train.early_stopping,\n                         valid_metric_name='pretrain_valid_loss',\n                         valid_metric_smaller_is_better=True,\n                         checkpoint_dir=(exp.abspath('pre_ckpt_params') if config.save_ckpt else None),\n                         checkpoint_epoch_freq=config.pretrain_ckpt_epoch_freq,\n                         checkpoint_max_to_keep=config.pretrain_ckpt_max_keep)\n\n    loop = TrainLoop(param_vars=variables_to_save,\n                     var_groups=var_groups,\n                     max_epoch=config.train.max_epoch,\n                     summary_dir=(exp.abspath('train_summary')\n                                  if config.write_summary else None),\n                     summary_graph=tf.get_default_graph(),\n                     summary_commit_freqs={'loss': 10},\n                     early_stopping=config.train.early_stopping,\n                     valid_metric_name='valid_loss',\n                     valid_metric_smaller_is_better=True,\n                     checkpoint_dir=(exp.abspath('ckpt_params')\n                                     if config.save_ckpt else None),\n                     checkpoint_epoch_freq=config.ckpt_epoch_freq,\n                     checkpoint_max_to_keep=config.ckpt_max_keep\n                     )\n\n    if config.write_histogram_summary:\n        summary_op = tf.summary.merge_all()\n    else:\n        summary_op = None\n\n    pre_lr_value = get_lr_value(config.train.initial_lr, config.train.lr_anneal_factor,\n                                config.train.pretrain_lr_anneal_epoch_freq, pre_loop)\n    lr_value = get_lr_value(config.train.initial_lr, config.train.lr_anneal_factor,\n                            config.train.lr_anneal_epoch_freq, loop)\n\n    pre_trainer = Trainer(loop=pre_loop,\n                          train_op=pre_train_op,\n                          inputs=[input_x, input_u],\n                          data_flow=train_flow,\n                          feed_dict={learning_rate: pre_lr_value, is_training: True},\n                          metrics={'pretrain_loss': pretrain_loss, 'pretrain_train_recons': pretrain_train_recons},\n                          summaries=summary_op)\n\n    trainer = Trainer(loop=loop,\n                      train_op=train_op,\n                      inputs=[input_x, input_u],\n                      data_flow=train_flow,\n                      feed_dict={learning_rate: lr_value, is_training: True},\n                      metrics=train_metrics,\n                      summaries=summary_op)\n\n    pre_validator = Evaluator(loop=pre_loop,\n                              metrics={'pretrain_valid_loss': pretrain_valid_loss, 'pretrain_valid_recons': pretrain_valid_recons},\n                              inputs=[input_x, input_u],\n                              data_flow=valid_flow,\n                              time_metric_name='pre_valid_time')\n\n    pre_validator.events.on(\n        spt.EventKeys.AFTER_EXECUTION,\n        lambda e: exp.update_results(pre_validator.last_metrics_dict)\n    )\n\n    validator = Evaluator(loop=loop,\n                          metrics=valid_metrics,\n                          inputs=[input_x, input_u],\n                          data_flow=valid_flow,\n                          time_metric_name='valid_time')\n\n    validator.events.on(\n        spt.EventKeys.AFTER_EXECUTION,\n        lambda e: exp.update_results(validator.last_metrics_dict)\n    )\n\n    train_losses = []\n    tmp_collector = []\n    valid_losses = []\n\n    def on_metrics_collected(loop: TrainLoop, metrics):\n        if 'loss' in metrics:\n            tmp_collector.append(metrics['loss'])\n        if loop.epoch % 1 == 0:\n            if 'valid_loss' in metrics:\n                valid_losses.append(metrics['valid_loss'])\n                train_losses.append(np.mean(tmp_collector))\n                tmp_collector.clear()\n\n    loop.events.on(spt.EventKeys.METRICS_COLLECTED, on_metrics_collected)\n\n    pre_trainer.evaluate_after_epochs(pre_validator, freq=1)\n    pre_trainer.log_after_epochs(freq=1)\n\n    trainer.evaluate_after_epochs(validator, freq=1)\n    trainer.log_after_epochs(freq=1)\n\n    with spt.utils.create_session().as_default() as session:\n\n        session.run(var_initializer)\n\n        with pre_loop:\n            pre_trainer.run()\n\n        print('')\n        print('PreTraining Finished.')\n\n        if config.save_results:\n            saver = tf.train.Saver(var_list=pre_variables_to_save)\n            saver.save(session, os.path.join(exp.abspath('result_params'), 'restored_pretrain_params.dat'))\n\n        print('')\n        print('Pretrain Model saved.')\n\n        print('************Start train the whole network***********')\n\n        with loop:\n            trainer.run()\n\n        print('')\n        print('Training Finished.')\n\n        if config.save_results:\n            saver = tf.train.Saver(var_list=variables_to_save)\n            saver.save(session, os.path.join(exp.abspath('result_params'), \"restored_params.dat\"))\n\n        print('')\n        print('Model saved.')\n\n\nif __name__ == '__main__':\n    with mltk.Experiment(ExpConfig()) as exp:\n        exp.save_config()\n        main(exp, exp.config)\n        if exp.config.exp_dir_save_path is not None:\n            with open(exp.config.exp_dir_save_path, 'a') as f:\n                f.write(\"'\" + exp.config.dataset + ' ' + exp.output_dir + \"'\" + '\\n')\n"
  },
  {
    "path": "algorithm/utils.py",
    "content": "import tfsnippet as spt\nimport numpy as np\nimport os\nimport pickle\nfrom sklearn.preprocessing import MinMaxScaler\nfrom typing import *\nimport tensorflow as tf\nfrom functools import partial\n\n# here, use 'min_max' or 'mean_std' for different method\n# method = 'min_max' or 'mean_std'\nmethod = 'min_max'\nalpha = 4.0  # mean +/- alpha * std\n\n\ndef get_sliding_window_data_flow(window_size, batch_size, x, u=None, y=None, shuffle=False, skip_incomplete=False) -> spt.DataFlow:\n    n = len(x)\n    seq = np.arange(window_size - 1, n, dtype=np.int32).reshape([-1, 1])\n    seq_df: spt.DataFlow = spt.DataFlow.arrays(\n        [seq], shuffle=shuffle, skip_incomplete=skip_incomplete, batch_size=batch_size)\n    offset = np.arange(-window_size + 1, 1, dtype=np.int32)\n\n    if y is not None:\n        if u is not None:\n            df = seq_df.map(lambda idx: (x[idx + offset], u[idx + offset], y[idx + offset]))\n        else:\n            df = seq_df.map(lambda idx: (x[idx + offset], y[idx + offset]))\n    else:\n        if u is not None:\n            df = seq_df.map(lambda idx: (x[idx + offset], u[idx + offset]))\n        else:\n            df = seq_df.map(lambda idx: (x[idx + offset],))\n\n    return df\n\n\ndef time_generator(timestamp):\n    mins = 60\n    hours = 24\n    days = 7\n    timestamp %= (mins * hours * days)\n    res = np.zeros([mins + hours + days])\n    res[int(timestamp / hours / mins)] = 1  # day\n    res[days + int((timestamp % (mins * hours)) / mins)] = 1  # hours\n    res[days + hours + int(timestamp % mins)] = 1  # min\n    return res\n\n\ndef get_data_dim(dataset):\n    if dataset == 'SWaT':\n        return 51\n    elif dataset == 'WADI':\n        return 118\n    elif str(dataset).startswith('machine'):\n        return 38\n    elif str(dataset).startswith('omi'):\n        return 19\n    else:\n        raise ValueError('unknown dataset '+str(dataset))\n\n\ndef get_data(dataset, max_train_size=None, max_test_size=None, print_log=True, do_preprocess=True, train_start=0,\n             test_start=0, valid_portion=0.3, prefix=\"./data/processed\"):\n    \"\"\"\n    get data from pkl files\n    return shape: (([train_size, x_dim], [train_size] or None), ([test_size, x_dim], [test_size]))\n    \"\"\"\n    if max_train_size is None:\n        train_end = None\n    else:\n        train_end = train_start + max_train_size\n    if max_test_size is None:\n        test_end = None\n    else:\n        test_end = test_start + max_test_size\n    print('load data of:', dataset)\n    print(\"train: \", train_start, train_end)\n    print(\"test: \", test_start, test_end)\n    x_dim = get_data_dim(dataset)\n    f = open(os.path.join(prefix, dataset + '_train.pkl'), \"rb\")\n    train_data = pickle.load(f).reshape((-1, x_dim))[train_start:train_end, :]\n    f.close()\n    try:\n        f = open(os.path.join(prefix, dataset + '_test.pkl'), \"rb\")\n        test_data = pickle.load(f).reshape((-1, x_dim))[test_start:test_end, :]\n        f.close()\n    except (KeyError, FileNotFoundError):\n        test_data = None\n    try:\n        f = open(os.path.join(prefix, dataset + \"_test_label.pkl\"), \"rb\")\n        test_label = pickle.load(f).reshape((-1))[test_start:test_end]\n        f.close()\n    except (KeyError, FileNotFoundError):\n        test_label = None\n    if do_preprocess:\n        # train_data = preprocess(train_data)\n        # test_data = preprocess(test_data)\n        train_data, test_data = preprocess(train_data, test_data, valid_portion=valid_portion)\n    print(\"train set shape: \", train_data.shape)\n    print(\"test set shape: \", test_data.shape)\n    print(\"test set label shape: \", test_label.shape)\n    return (train_data, None), (test_data, test_label)\n\n\ndef preprocess(train, test, valid_portion=0):\n    train = np.asarray(train, dtype=np.float32)\n    test = np.asarray(test, dtype=np.float32)\n\n    if len(train.shape) == 1 or len(test.shape) == 1:\n        raise ValueError('Data must be a 2-D array')\n\n    if np.any(sum(np.isnan(train)) != 0):\n        print('Train data contains null values. Will be replaced with 0')\n        train = np.nan_to_num(train)\n\n    if np.any(sum(np.isnan(test)) != 0):\n        print('Test data contains null values. Will be replaced with 0')\n        test = np.nan_to_num(test)\n\n    # revise here for other preprocess methods\n    if method == 'min_max':\n        if valid_portion > 0:\n            split_idx = int(len(train) * valid_portion)\n            train, valid = train[:-split_idx], train[-split_idx:]\n            scaler = MinMaxScaler().fit(train)\n            train = scaler.transform(train)\n            valid = scaler.transform(valid)\n            valid = np.clip(valid, a_min=-3.0, a_max=3.0)\n            test = scaler.transform(test)\n            test = np.clip(test, a_min=-3.0, a_max=3.0)\n            train = np.concatenate([train, valid], axis=0)\n            print('Data normalized with min-max scaler')\n        else:\n            scaler = MinMaxScaler().fit(train)\n            train = scaler.transform(train)\n            test = scaler.transform(test)\n            test = np.clip(test, a_min=-3.0, a_max=3.0)\n            print('Data normalized with min-max scaler')\n\n    elif method == 'mean_std':\n\n        def my_transform(value, ret_all=True, mean=None, std=None):\n            if mean is None:\n                mean = np.mean(value, axis=0)\n            if std is None:\n                std = np.std(value, axis=0)\n            for i in range(value.shape[0]):\n                clip_value = mean + alpha * std  # compute clip value: (mean - a * std, mean + a * std)\n                temp = value[i] < clip_value\n                value[i] = temp * value[i] + (1 - temp) * clip_value\n                clip_value = mean - alpha * std\n                temp = value[i] > clip_value\n                value[i] = temp * value[i] + (1 - temp) * clip_value\n                std = np.maximum(std, 1e-5)  # to avoid std -> 0\n                value[i] = (value[i] - mean) / std  # normalization\n            return value, mean, std if ret_all else value\n\n        train, _mean, _std = my_transform(train)\n        test = my_transform(test, False, _mean, _std)[0]\n        print('Data normalized with standard scaler method')\n\n    elif method == 'none':\n        print('No pre-processing')\n\n    else:\n        raise RuntimeError('unknown preprocess method')\n\n    return train, test\n\n\nTensorLike = Union[tf.Tensor, spt.StochasticTensor]\n\n\nclass GraphNodes(Dict[str, TensorLike]):\n    \"\"\"A dict that maps name to TensorFlow graph nodes.\"\"\"\n\n    def __init__(self, *args, **kwargs):\n        super().__init__(*args, **kwargs)\n\n        for k, v in self.items():\n            if not spt.utils.is_tensor_object(v):\n                raise TypeError(f'The value of `{k}` is not a tensor: {v!r}.')\n\n    def eval(self,\n             session: tf.Session = None,\n             feed_dict: Dict[tf.Tensor, Any] = None) -> Dict[str, Any]:\n        \"\"\"\n        Evaluate all the nodes with the specified `session`.\n        Args:\n            session: The TensorFlow session.\n            feed_dict: The feed dict.\n        Returns:\n            The node evaluation outputs.\n        \"\"\"\n        if session is None:\n            session = spt.utils.get_default_session_or_error()\n\n        keys = list(self)\n        tensors = [self[key] for key in keys]\n        outputs = session.run(tensors, feed_dict=feed_dict)\n\n        return dict(zip(keys, outputs))\n\n    def add_prefix(self, prefix: str) -> 'GraphNodes':\n        \"\"\"\n        Add a common prefix to all metrics in this collection.\n        Args:\n             prefix: The common prefix.\n        \"\"\"\n        return GraphNodes({f'{prefix}{k}': v for k, v in self.items()})\n\n\ndef get_score(recons_probs, preserve_feature_dim=False, score_avg_window_size=1):\n    \"\"\"\n    Evaluate the anomaly score at each timestamp according to the reconstruction probability obtained by model.\n    :param recons_probs: (data_length-window_length+1, window_length, x_dim). The reconstruction probabilities correspond\n    to each timestamp and each dimension of x, evaluated in sliding windows with length 'window_length'. The larger the\n    reconstruction probability, the less likely a point is an anomaly.\n    :param preserve_feature_dim: bool. Whether sum over the feature dimension. If True, preserve the anomaly score on\n    each feature dimension. If False, sum over the anomaly scores along feature dimension and return a single score on\n    each timestamp.\n    :param score_avg_window_size: int. How many scores in different sliding windows are used to evaluate the anomaly score\n    at a given timestamp. By default score_avg_window_size=1, only the score of last point are used in each sliding window,\n    and this score is directly used as the final anomaly score at this timestamp. When score_avg_window_size > 1, then\n    the last 'score_avg_window_size' scores are used in each sliding window. Then for timestamp t, if t is the last point\n    of sliding window k, then the anomaly score of t is now evaluated as the average score_{t} in sliding windows\n    [k, k+1, ..., k+score_avg_window_size-1].\n    :return: Anomaly scores (reconstruction probability) at each timestamps.\n    With shape ``(data_length - window_size + score_avg_window_size,)`` if `preserve_feature_dim` is `False`,\n    or ``(data_length - window_size + score_avg_window_size, x_dim)`` if `preserve_feature_dim` is `True`.\n    The first `window_size - score_avg_window_size` points are discarded since there aren't enough previous values to evaluate the score.\n    \"\"\"\n    data_length = recons_probs.shape[0] + recons_probs.shape[1] - 1\n    window_length = recons_probs.shape[1]\n    score_collector = [[] for i in range(data_length)]\n    for i in range(recons_probs.shape[0]):\n        for j in range(score_avg_window_size):\n            score_collector[i + window_length - j - 1].append(recons_probs[i, -j-1])\n\n    score_collector = score_collector[window_length-score_avg_window_size:]\n    scores = []\n    for i in range(len(score_collector)):\n        scores.append(np.mean(score_collector[i], axis=0))\n    scores = np.array(scores)                 # average over the score_avg_window. (data_length-window_length+score_avg_window_size, x_dim)\n    if not preserve_feature_dim:\n        scores = np.sum(scores, axis=-1)\n    return scores\n\n\ndef get_avg_recons(recons_vals, window_length, recons_avg_window_size=1):\n    \"\"\"\n    Get the averaged reconstruction values for plotting. The last `recons_avg_window_size` points in each reconstruct\n    sliding windows are used, the final reconstruction values at each timestamp is the mean of each value at this timestamp.\n    :param recons_vals: original reconstruction values. shape: (data_length - window_length + 1, window_length, x_dim)\n    :param recons_avg_window_size:  int. How many points are used in each reconstruct sliding window.\n    :return: final reconstruction curve. shape: (data_length, x_dim)\n    The first `window_size - recons_avg_window_size` points use the reconstruction value of the first reconstruction window,\n    others use the averaged values according to `recons_vals` and `recons_avg_window_size`.\n    \"\"\"\n    data_length = recons_vals.shape[0] + window_length - 1\n    recons_collector = [[] for i in range(data_length)]\n    for i in range(recons_vals.shape[0]):\n        for j in range(recons_avg_window_size):\n            recons_collector[i + window_length - j - 1].append(recons_vals[i, -j-1, :])\n\n    if recons_vals.shape[1] < window_length:\n        for i in range(window_length - recons_avg_window_size):\n            recons_collector[i] = [recons_vals[0, -1, :]]\n    else:\n        for i in range(window_length - recons_avg_window_size):\n            recons_collector[i] = [recons_vals[0, i, :]]\n\n    final_recons = []\n    for i in range(len(recons_collector)):\n        final_recons.append(np.mean(recons_collector[i], axis=0))\n    final_recons = np.array(final_recons)    # average over the recons_avg_window. (data_length, x_dim)\n    return final_recons\n"
  },
  {
    "path": "data/interpretation_label/anomaly_type.txt",
    "content": "dataset,anomaly_segment,anomaly_type\nomi-1,760-765,metric\nomi-1,1064-1298,metric-temporal\nomi-1,2758-2772,metric\nomi-1,2874-2885,temporal\nomi-1,3012-3025,metric-temporal\nomi-1,3160-3305,metric-temporal\nomi-1,3626-3638,metric-temporal\nomi-2,486-498,metric-temporal\nomi-2,740-753,temporal\nomi-2,766-783,temporal\nomi-2,1436-1445,temporal\nomi-3,760-820,metric-temporal\nomi-3,842-860,metric-temporal\nomi-3,1080-1120,temporal\nomi-3,2188-2201,temporal\nomi-3,2497-2500,temporal\nomi-3,3428-3434,temporal\nomi-4,727-782,metric-temporal\nomi-4,843-857,metric-temporal\nomi-4,2178-2238,metric-temporal\nomi-4,2494-2501,temporal\nomi-4,3425-3435,temporal\nomi-5,765-781,temporal\nomi-5,1589-1617,metric-temporal\nomi-5,2440-2446,metric\nomi-5,3356-3363,metric-temporal\nomi-5,3416-3423,temporal\nomi-5,3590-3594,metric-temporal\nomi-6,357-392,metric\nomi-6,750-788,metric-temporal\nomi-6,1278-1293,metric\nomi-6,1590-1615,metric-temporal\nomi-6,2592-2604,metric\nomi-6,2734-2765,metric-temporal\nomi-6,3066-3072,metric-temporal\nomi-6,3596-3605,metric-temporal\nomi-6,3610-3622,temporal\nomi-6,4159-4164,temporal\nomi-7,738-778,temporal\nomi-7,1625-1660,temporal\nomi-7,3311-3320,temporal\nomi-8,740-753,temporal\nomi-8,764-780,metric-temporal\nomi-8,2319-2325,temporal\nomi-8,2586-2675,metric-temporal\nomi-8,2897-2956,metric-temporal\nomi-9,738-756,metric-temporal\nomi-9,763-780,metric-temporal\nomi-9,2514-2524,temporal\nomi-9,2603-2668,metric-temporal\nomi-9,2918-2958,metric-temporal\nomi-9,3170-3245,metric-temporal\nomi-9,3508-3528,metric\nomi-9,4064-4108,metric-temporal\nomi-10,740-780,metric-temporal\nomi-10,1345-1406,temporal\nomi-10,2224-2272,metric-temporal\nomi-10,2514-2592,metric\nomi-10,2733-2778,metric\nomi-10,3401-3405,temporal\nomi-10,3414-3459,temporal\nomi-10,3700-3744,metric-temporal\nomi-11,738-755,temporal\nomi-11,757-781,metric-temporal\nomi-11,782-815,temporal\nomi-11,1366-1375,temporal\nomi-11,2301-2314,metric\nomi-11,2365-2381,metric-temporal\nomi-11,2464-2485,metric-temporal\nomi-11,2588-2610,metric-temporal\nomi-11,2650-2675,metric\nomi-11,3012-3072,temporal\nomi-12,740-813,temporal\nomi-12,2486-2493,temporal\nomi-12,2515-2525,metric-temporal\nomi-12,2721-2723,temporal\nomi-12,3500-3535,metric"
  },
  {
    "path": "data/interpretation_label/machine-1-1.txt",
    "content": "15849-16395:1,9,10,12,13,14,15\n16963-17517:1,2,3,4,6,7,9,10,11,12,13,14,15,16,19,20,21,22,24,25,26,27,28,29,30,31,32,33,34,35,36\n18071-18528:1,2,9,10,12,13,14,15\n19367-20088:1,2,3,4,9,10,11,12,13,14,15,16,25,28\n20786-21195:1,9,10,12,13,14,15\n24679-24682:9,13,14,15\n26114-26116:9,13,14,15\n27554-27556:9,13,14,15\n"
  },
  {
    "path": "data/interpretation_label/machine-1-6.txt",
    "content": "246-252:1,2,3,4,6,9,10,11,12,13,15\n653-658:1,2,3,4,6,9,10,11,12,13,15,19,20,21,22,26,28,30,31,32\n2092-2100:1,2,3,4,6,9,10,11,12,13,15,19,20,21,22,26,28,30,31,32\n2884-2888:6\n3534-3539:1,2,3,4,6,9,10,11,12,13,15,19,20,21,22\n4647-5045:5,6,9,10,11,13,17,33,34\n5167-5172:9,10,11,13,18\n5708-5713:9,10,11,13\n5873-5885:2,3,6,9,10,11,13,19,20,21,22,28,31,32\n6022-6027:33,34\n6412-6419:1,2,3,6,9,10,11,13,15,19,20,21,22,28,30,31,32,35,36\n7851-7856:1,2,9,10,11,13,19,20,21,22,23,28,30,31,32\n9291-9298:1,2,3,9,10,11,13,19,20,21,22,23,28,30,31,32\n10731-10736:1,2,3,9,10,11,13,19,20,21,22,23,28,30,31,32\n11467-11471:1,2,3,6,9,10,11,13,19,20,21,22,23,28,30,31,32\n12171-12176:1,2,3,6,9,10,11,13,19,20,21,22,23,28,30,31,32\n13069-13073:33,34\n13277-13280:17\n13613-13619:1,2,9,10,11,12,13,30\n14603-14607:9,10,11,13,19,20,21,22,32,37\n15052-15055:9,10,11,13\n15397-15401:9,12,13\n15802-15805:9,10,11,13\n16491-16499:1,2,3,9,10,11,13,15\n16718-16721:9,10,11,13\n16972-16976:9,12,13,15\n17931-17939:1,2,3,6,9,10,11,12,13,15,19,20,21,22,23,28,30,31,32\n18600-21761:1,2,3,4,6,9,10,11,12,13,14,15,16,23,25,28,30,31,32,35,36\n22252-22260:9,13,15\n22417-22420:11,13\n"
  },
  {
    "path": "data/interpretation_label/machine-1-7.txt",
    "content": "837-858:1,6,9,12,13,14,15,16\n2959-4174:6,9,12,13,14,15,16,33,34\n5849-5940:1,2,3,4,6,7,9,10,11,13,15,19,20,21,22,23,24,25,26,28,30,31,32,35,36\n6031-6034:33,34\n7099-8000:1,9,12,13,14,15,17,18\n8564-8580:1,9,10,11,13,19,20,21,22,23,24,25,26,28,31,32,35,36\n12359-12374:1,9,12,13,14,15,16\n13799-13825:9,12,13,14,15,16\n15239-15270:1,6,9,12,13,14,15,16\n15960-15963:17\n18109-18150:1,2,3,4,6,7,9,12,13,14,15,16,17,19,20,21,22,23,24,25,26,28,29,30,31,32,5,36\n19442-19446:33,34\n19559-19590:1,6,9,12,13,14,15,16\n"
  },
  {
    "path": "data/interpretation_label/machine-2-1.txt",
    "content": "6506-6528:23,25,32,35,36\n7907-7961:9,10,11,13,32\n9340-9386:9,10,11,13,32,33,34\n9779-9805:5,33,34\n12173-12181:19,20,21,22,28\n16136-16588:9,10,24,26\n17365-17382:1,2,3,4,9,14,15,16,18,24,26\n17995-18028:9,10,11,13,15\n18575-18650:1,19,20,21,22\n20020-20036:1,6,7\n20891-20912:10,13\n20340-20352:9,10,11,13,32\n22907-23295:1,2,3,4,5,6,7,9,10,11,12,13,14,16,17,19,20,21,22,23,24,25,26,28,31,32,35,36\n"
  },
  {
    "path": "data/interpretation_label/machine-2-2.txt",
    "content": "12760-12771:1,2,3,4,6,7,23,24,25,26,31,32,33,34,35,36\n14511-14521:1,2,3,4,6,7,23,24,25,26,31,32,35,36\n15630-16502:10,11,12,13\n17090-17942:10,11,12,13\n18541-19382:10,11,12,13\n20820-20823:10,11,12,13\n21024-21027:11\n21280-21295:1,2,3,4,6,7,32\n21405-21428:19,20,21,22,28,31\n22260-22263:6,10,11,12,13\n23099-23299:9,15\n"
  },
  {
    "path": "data/interpretation_label/machine-2-7.txt",
    "content": "664-667:9,13,14\n2104-2107:13,14\n3544-3547:9,13,14,15\n4984-4987:9,13,14,15\n6030-6034:33,34\n7665-7669:30\n7863-7867:9,13,14,15\n9304-9307:9,13,14,15\n10744-10747:9,13,14,15\n12099-12127:1,2,3,4,24,26\n13623-13625:9,13,14,15\n14934-14938:33,34\n15063-15066:9,13,14,15\n16534-16557:9,13,14,15\n17978-17981:9,13,14,15\n18639-18944:1,2,3,4,6,7,11,16,19,20,21,22,23,25,28,29,31,32,34,35,36\n19411-19418:9,13,14,15\n20853-20857:9,13,14,15\n22291-22296:9,13,14,15\n22924-22927:10\n"
  },
  {
    "path": "data/interpretation_label/machine-2-8.txt",
    "content": "17580-17741:1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,28,29,30,31,32,33,34,35,36\n"
  },
  {
    "path": "data/interpretation_label/machine-3-11.txt",
    "content": "21230-21296:1,2,3,4,5,6,7,10,11,16,19,20,21,22,23,24,26,28,30,31,32,35,36\n27825-27951:1,2,3,4,7,19,20,21,22,23,24,26,28,30,31,32,34,35,36,37\n27995-28001:10,11\n"
  },
  {
    "path": "data/interpretation_label/machine-3-3.txt",
    "content": "739-742:10,11,12,13\n1808-1814:18\n2179-2182:10,11,12,13\n3619-3622:10,11,12,13\n4561-4578:1,6,7,9,10,11,12,13,14,15,19,20,21,22,23,24,25,26,28,31,35,36,37,38\n4564-4567:1,6,7,9,10,11,12,13,14,15,19,20,21,22,23,24,25,26,28,31,35,36,37,38\n5059-5062:10,11,12,13\n5449-5453:12,13,15,16,17\n6499-6502:10,11,12,13\n7939-7942:10,11,12,13\n8407-8414:12,13,15,16,17\n9737-9740:15,16,17\n10819-10822:10,11,12,13\n12259-12262:10,11,12,13\n13699-13702:10,11,12,13\n13827-13830:17\n14476-14486:14\n15139-15142:10,11,12,13\n16199-16212:10,11,12,13\n16579-16582:1,2,3,4,19,21,22,25,28,31\n18019-18022:10,11,12,13\n19081-19088:10,11,12,13\n19349-19830:1,2,3\n19459-19462:1,2,3,4,10,11,12,19,20,21,22,23,25,28,31,35,36,37,38\n20897-20903:10,11,12,13\n21878-21893:18\n22339-22342:10,11,12,13\n23159-23180:1,2,3,4,19,21,22,28,31\n"
  },
  {
    "path": "data/interpretation_label/machine-3-4.txt",
    "content": "2734-3520:1,2,3,4,6,7,11,16\n4474-4550:2,11\n6013-6016:10,12,13,14\n10963-10969:30,33,34\n11565-11569:24,33,34\n13699-13709:24,33,34\n18589-18640:1,2,3,4\n18784-18825:1,2,3\n"
  },
  {
    "path": "data/interpretation_label/machine-3-6.txt",
    "content": "1187-1221:1,2,3,4,19,20,21,22,23,24,25,28,29,31,36,35,34,32\n8177-8211:1,2,3,4,34,33\n10637-10743:1,33,34\n15633-15863:1,9,10,12,13,14,15,16\n16788-16995:1,10,12,13,14,15\n18054-18134:1,10,12,13,14,15\n19276-19500:1,10,12,13,14,15\n20681-20865:1,9,10,12,13,14,15\n24503-24506:1,9,10,12,13,14,15\n26138-26149:1,12\n27877-27958:1,2,3,4,6,7,11,16,19,20,21,22,23,24,25,26,28,30,31,32,33,35,36\n"
  },
  {
    "path": "data/interpretation_label/machine-3-8.txt",
    "content": "15144-15266:1,2,3,4,19,20,21,22,28,31,35,36\n15941-16514:1,2,3,4,19,20,21,22,28,30,31\n17063-17276:1,2,3,4,19,20,21,22,23,24,25,26,28,29,30,31,32,33,34,35,36\n21405-21548:9,10,12,13\n23478-23781:9,10,13,14,30\n28253-28270:10,12,13\n"
  },
  {
    "path": "data/interpretation_label/omi-1.txt",
    "content": "760-765:4,5,7,8,9,10,11,12,13,14,15,16,17,18,19\n1064-1298:6,7,9,10,11,12,13,14,15,16,17,18,19\n2758-2772:5,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n2874-2885:7,9,10,11,12,13,14,15,16,17,18\n3012-3025:7,10,11,12,13,14,15,16,17,18,19\n3160-3305:8,9,10,11,12,13,14,15,16,17,18,19\n3626-3638:7,9,10,11,12,13,14,15,16,17,18,19"
  },
  {
    "path": "data/interpretation_label/omi-10.txt",
    "content": "740-780:2,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n1345-1406:7,9,10,11,12,13,14,15,16,17,18\n2224-2272:4,5,6,7,9,10,11,12,13,14,15,16,17,18,19\n2514-2592:4,16\n2733-2778:11,14,15\n3401-3405:1\n3414-3459:8\n3700-3744:7,8,9,10,11,12,13,14,15,16,17,18"
  },
  {
    "path": "data/interpretation_label/omi-11.txt",
    "content": "738-755:2\n757-781:1,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n782-815:1,4,6,8,9,10,11,12,13,14,15,16,17,18,19\n1366-1375:4,5,8,17,19\n2301-2314:4,5,17,19\n2365-2381:19\n2464-2485:4,5,17,19\n2588-2610:4,5,17,19\n2650-2675:19\n3012-3072:1,4,5,8,17,19"
  },
  {
    "path": "data/interpretation_label/omi-12.txt",
    "content": "740-813:1,2,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n2486-2493:4,5\n2515-2525:1,4,5,7,8,9,17,18\n2721-2723:4\n3500-3535:5,6,7,8,11,12,13,14,15,16,17,18,19"
  },
  {
    "path": "data/interpretation_label/omi-2.txt",
    "content": "486-498:1,4,5\n740-753:2\n766-783:1,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n1436-1445:7,9,10,11,12,13,14,15,16,17,18"
  },
  {
    "path": "data/interpretation_label/omi-3.txt",
    "content": "760-820:1,4,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n842-860:7,8,9,10,11,12,13,14,15,16,17,18,19\n1080-1120:7,9,10,11,12,13,14,15,16,17,18\n2188-2201:7,9,10,11,12,13,14,15,16,17,18,19\n2497-2500:6,7,9,10,11,12,13,14,15,16,17,18,19\n3428-3434:7,8,9,10,11,12,13,14,15,16,17,18,19"
  },
  {
    "path": "data/interpretation_label/omi-4.txt",
    "content": "727-782:1,2,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n843-857:6,7,8,9,10,11,12,13,14,15,16,17,18,19\n2178-2238:8,9,10,11,12,13,14,15,16,17,18,19\n2494-2501:7,9,10,11,12,13,14,15,16,17,18,19\n3425-3435:7,8,9,10,11,12,13,14,15,16,17,18,19"
  },
  {
    "path": "data/interpretation_label/omi-5.txt",
    "content": "765-781:6,7,8,9,10,11,12,13,14,15,16,17,18,19\n1589-1617:2,4,17\n2440-2446:8,11,12,13,14,15,16,17,18,19\n3356-3363:5,6,17\n3416-3423:4,5,7,9,10,11,12,13,14,15,16,17,18,19\n3590-3594:1,8,17,18,19"
  },
  {
    "path": "data/interpretation_label/omi-6.txt",
    "content": "357-392:4,19\n750-788:2,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n1278-1293:4,5,6,8,17\n1590-1615:1,9,18\n2592-2604:4,6,9\n2734-2765:4,6,8,19\n3066-3072:4,6,19\n3596-3605:4,5,6,19\n3610-3622:1,4,5\n4159-4164:4,5"
  },
  {
    "path": "data/interpretation_label/omi-7.txt",
    "content": "738-778:1,2,4,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n1625-1660:7,10,11,12,13,14,15,16,17,18,19\n3311-3320:7,10,11,12,13,14,15,16,17,18,19"
  },
  {
    "path": "data/interpretation_label/omi-8.txt",
    "content": "740-753:2,4\n764-780:6,7,8,9,10,11,12,13,14,15,16,17,18,19\n2319-2325:4,5,6\n2586-2675:5,6,8,17,18,19\n2897-2956:2,6,8,17,18,19"
  },
  {
    "path": "data/interpretation_label/omi-9.txt",
    "content": "738-756:2,9,10,11,12,13,14,15,16,17,18,19\n763-780:1,6,7,8,9,10,11,12,13,14,15,16,17,18,19\n2514-2524:1,4,5,7,8,9,17,18\n2603-2668:1,2,5,6,8,17,18,19\n2918-2958:1,2,5,6,8,17,18,19\n3170-3245:1,2,6,8,17,18,19\n3508-3528:2,6,8,19\n4064-4108:1,2,6,7,8,9,17,18,19"
  },
  {
    "path": "explib/__init__.py",
    "content": "from .utils import *\nfrom .eval_methods import *"
  },
  {
    "path": "explib/eval_methods.py",
    "content": "# -*- coding: utf-8 -*-\nimport numpy as np\nimport sklearn.metrics\n\n\ndef calc_point2point(predict, actual):\n    \"\"\"\n    calculate f1 score by predict and actual.\n\n    Args:\n        predict (np.ndarray): the predict label\n        actual (np.ndarray): np.ndarray\n    \"\"\"\n    TP = np.sum(predict * actual)\n    TN = np.sum((1 - predict) * (1 - actual))\n    FP = np.sum(predict * (1 - actual))\n    FN = np.sum((1 - predict) * actual)\n    precision = TP / (TP + FP + 0.00001)\n    recall = TP / (TP + FN + 0.00001)\n    f1 = 2 * precision * recall / (precision + recall + 0.00001)\n    return f1, precision, recall, TP, TN, FP, FN\n\n\ndef adjust_predicts(score, label,\n                    threshold=None,\n                    pred=None,\n                    calc_latency=False):\n    \"\"\"\n    Calculate adjusted predict labels using given `score`, `threshold` (or given `pred`) and `label`.\n\n    Args:\n        score (np.ndarray): The anomaly score\n        label (np.ndarray): The ground-truth label\n        threshold (float): The threshold of anomaly score.\n            A point is labeled as \"anomaly\" if its score is lower than the threshold.\n        pred (np.ndarray or None): if not None, adjust `pred` and ignore `score` and `threshold`,\n        calc_latency (bool):\n\n    Returns:\n        np.ndarray: predict labels\n    \"\"\"\n    if len(score) != len(label):\n        raise ValueError(\"score and label must have the same length\")\n    score = np.asarray(score)\n    label = np.asarray(label)\n    latency = 0\n    if pred is None:\n        predict = score < threshold\n    else:\n        predict = pred\n    actual = label > 0.1\n    anomaly_state = False\n    anomaly_count = 0\n    for i in range(len(score)):\n        if actual[i] and predict[i] and not anomaly_state:\n                anomaly_state = True\n                anomaly_count += 1\n                for j in range(i, 0, -1):\n                    if not actual[j]:\n                        break\n                    else:\n                        if not predict[j]:\n                            predict[j] = True\n                            latency += 1\n        elif not actual[i]:\n            anomaly_state = False\n        if anomaly_state:\n            predict[i] = True\n    if calc_latency:\n        return predict, latency / (anomaly_count + 1e-4)\n    else:\n        return predict\n\n\ndef calc_seq(score, label, threshold, calc_latency=False):\n    \"\"\"\n    Calculate f1 score for a score sequence\n    \"\"\"\n    if calc_latency:\n        predict, latency = adjust_predicts(score, label, threshold, calc_latency=calc_latency)\n        t = list(calc_point2point(predict, label))\n        t.append(latency)\n        return t\n    else:\n        predict = adjust_predicts(score, label, threshold, calc_latency=calc_latency)\n        return calc_point2point(predict, label)\n\n\n# here for our refined best-f1 search method\ndef get_best_f1(score, label):\n    '''\n    :param score: 1-D array, input score, tot_length\n    :param label: 1-D array, standard label for anomaly\n    :return: list for results, threshold\n    '''\n\n    assert score.shape == label.shape\n    print('***computing best f1***')\n    search_set = []\n    tot_anomaly = 0\n    for i in range(label.shape[0]):\n        tot_anomaly += (label[i] > 0.5)\n    flag = 0\n    cur_anomaly_len = 0\n    cur_min_anomaly_score = 1e5\n    for i in range(label.shape[0]):\n        if label[i] > 0.5:\n            # here for an anomaly\n            if flag == 1:\n                cur_anomaly_len += 1\n                cur_min_anomaly_score = score[i] if score[i] < cur_min_anomaly_score else cur_min_anomaly_score\n            else:\n                flag = 1\n                cur_anomaly_len = 1\n                cur_min_anomaly_score = score[i]\n        else:\n            # here for normal points\n            if flag == 1:\n                flag = 0\n                search_set.append((cur_min_anomaly_score, cur_anomaly_len, True))\n                search_set.append((score[i], 1, False))\n            else:\n                search_set.append((score[i], 1, False))\n    if flag == 1:\n        search_set.append((cur_min_anomaly_score, cur_anomaly_len, True))\n    search_set.sort(key=lambda x: x[0])\n    best_f1_res = - 1\n    threshold = 1\n    P = 0\n    TP = 0\n    best_P = 0\n    best_TP = 0\n    for i in range(len(search_set)):\n        P += search_set[i][1]\n        if search_set[i][2]:  # for an anomaly point\n            TP += search_set[i][1]\n        precision = TP / (P + 1e-5)\n        recall = TP / (tot_anomaly + 1e-5)\n        f1 = 2 * precision * recall / (precision + recall + 1e-5)\n        if f1 > best_f1_res:\n            best_f1_res = f1\n            threshold = search_set[i][0]\n            best_P = P\n            best_TP = TP\n\n    print('***  best_f1  ***: ', best_f1_res)\n    print('*** threshold ***: ', threshold)\n    return (best_f1_res,\n            best_TP / (best_P + 1e-5),\n            best_TP / (tot_anomaly + 1e-5),\n            best_TP,\n            score.shape[0] - best_P - tot_anomaly + best_TP,\n            best_P - best_TP,\n            tot_anomaly - best_TP), threshold\n\n\n# calculate evaluation metrics (best-F1, AUROC, AP) under point-adjust approach.\ndef get_adjusted_composite_metrics(score, label):\n    score = -score  # change the recons prob to anomaly score, higher anomaly score means more anomalous\n    # adjust the score for segment detection. i.e., for each ground-truth anomaly segment, use the maximum score\n    # as the score of all points in that segment. This corresponds to point-adjust f1-score.\n    assert len(score) == len(label)\n    splits = np.where(label[1:] != label[:-1])[0] + 1\n    is_anomaly = label[0] == 1\n    pos = 0\n    for sp in splits:\n        if is_anomaly:\n            score[pos:sp] = np.max(score[pos:sp])\n        is_anomaly = not is_anomaly\n        pos = sp\n    sp = len(label)\n    if is_anomaly:\n        score[pos:sp] = np.max(score[pos:sp])\n\n    # now get the adjust score for segment evaluation.\n    fpr, tpr, _ = sklearn.metrics.roc_curve(y_true=label, y_score=score, drop_intermediate=False)\n    auroc = sklearn.metrics.auc(fpr, tpr)\n    precision, recall, _ = sklearn.metrics.precision_recall_curve(y_true=label, probas_pred=score)\n    # validate best f1\n    f1 = np.max(2 * precision * recall / (precision + recall + 1e-5))\n    ap = sklearn.metrics.average_precision_score(y_true=label, y_score=score, average=None)\n    return auroc, ap, f1, precision, recall, fpr, tpr\n"
  },
  {
    "path": "explib/raw_data_converter.py",
    "content": "import numpy as np\nimport pandas as pd\nimport pickle as pkl\n\n\n# preprocess for SWaT. SWaT.A2_Dec2015, version 0\ndf = pd.read_csv('SWaT_Dataset_Attack_v0.csv')\ny = df['Normal/Attack'].to_numpy()\nlabels = []\nfor i in y:\n    if i == 'Attack':\n        labels.append(1)\n    else:\n        labels.append(0)\nlabels = np.array(labels)\nassert len(labels) == 449919\n# pkl.dump(labels, open('SWaT_test_label.pkl', 'wb'))\nprint('SWaT_test_label saved')\n\ndf = df.drop(columns=[' Timestamp', 'Normal/Attack'])\ntest = df.to_numpy()\nassert test.shape == (449919, 51)\n# pkl.dump(test, open('SWaT_test.pkl', 'wb'))\nprint('SWaT_test saved')\n\ndf = pd.read_csv('SWaT_Dataset_Normal_v0.csv')\ndf = df.drop(columns=['Unnamed: 0','Unnamed: 52'])\ntrain = df[1:].to_numpy()\nassert train.shape == (496800, 51)\n# pkl.dump(train, open('SWaT_train.pkl', 'wb'))\nprint('SWaT_train saved')\n\n# preprocess for WADI. WADI.A1\na = str(open('WADI_14days.csv', 'rb').read(), encoding='utf8').split('\\n')[5: -1]\na = '\\n'.join(a)\nwith open('train1.csv', 'wb') as f:\n    f.write(a.encode('utf8'))\na = pd.read_csv('train1.csv', header=None)\n\n\na = a.to_numpy()[:, 3:]\nnan_cols = []\nfor j in range(a.shape[1]):\n    for i in range(a.shape[0]):\n        if a[i][j] != a[i][j]:\n            nan_cols.append(j)\n            break\n# len(nan_cols) == 9\ntrain = np.delete(a, nan_cols, axis=1)\nassert train.shape == (1209601, 118)\n# pkl.dump(train, open('WADI_train.pkl', 'wb'))\nprint('WADI_train saved')\n\ndf = pd.read_csv('WADI_attackdata.csv')\ntest = df.to_numpy()[:, 3:]\ntest = np.delete(test, nan_cols, axis=1)\nassert test.shape == (172801, 118)\n# pkl.dump(test, open('WADI_test.pkl', 'wb'))\nprint('WADI_test saved')\n\nprint('WADI_test_label saved')\n\n# WADI labels.pkl are created manually via the description file of the dataset\n"
  },
  {
    "path": "explib/utils.py",
    "content": "import codecs\nimport json\nimport os\nimport yaml\n\n\ndef parse_file(path):\n    \"\"\"\n    Parse configuration values form the given file.\n\n    Args:\n        path (str): Path of the file.  It should be a JSON file or\n            a YAML file, with corresponding file extension.\n    \"\"\"\n    _, ext = os.path.splitext(path)\n    config_dict = None\n    if ext == '.json':\n        with codecs.open(path, 'rb', 'utf-8') as f:\n            config_dict = dict(json.load(f))\n    elif ext in ('.yml', '.yaml'):\n        with codecs.open(path, 'rb', 'utf-8') as f:\n            config_dict = dict(yaml.load(f))\n    else:\n        raise ValueError('Config file of this type is not supported: {}'.\n                         format(path))\n    return config_dict\n\n\nclass Singleton(object):\n    \"\"\"\n    Base class for singleton classes.\n\n    >>> class Parent(Singleton):\n    ...     pass\n\n    >>> class Child(Parent):\n    ...     pass\n\n    >>> Parent() is Parent()\n    True\n    >>> Child() is Child()\n    True\n    >>> Parent() is not Child()\n    True\n    \"\"\"\n\n    __instances_dict = {}\n\n    def __new__(cls, *args, **kwargs):\n        if cls not in Singleton.__instances_dict:\n            Singleton.__instances_dict[cls] = \\\n                object.__new__(cls, *args, **kwargs)\n        return Singleton.__instances_dict[cls]\n\n"
  },
  {
    "path": "requirements.txt",
    "content": "numpy==1.17.0\ntensorflow-gpu==1.12.0\ntyping-extensions==3.7.4.1\ntyping-inspect==0.5.0\ntqdm==4.31.1\npickleshare==0.7.5\nscikit-learn==0.20.3\nscipy==1.2.1\npandas==0.24.2\nmatplotlib==2.0.2\nseaborn==0.9.0\ndataclasses==0.7\ndataclasses-json==0.3.5\nClick==7.0\nfs==2.4.4\nsix==1.11.0\ngit+https://github.com/thu-ml/zhusuan.git@48c0f4e\ngit+https://github.com/haowen-xu/tfsnippet.git@v0.2.0-alpha4\ngit+https://github.com/haowen-xu/ml-essentials.git\n"
  }
]