[
  {
    "path": ".gitignore",
    "content": "*.pyc\n*cache\n*.idea\n*__pycache__\n*venv\n*nn_builder.egg-info\n*build\n*dist\n.DS_Store\n*to_do_list\n*.ipynb_checkpoints"
  },
  {
    "path": ".travis.yml",
    "content": "language:\n  python\n\npython: 3.7\ndist: xenial\nsudo: true\n\ninstall:\n  - pip install -r requirements.txt -q\n  - pip install sklearn\n  - pip install torchvision\n\nscript:\n  - export PYTHONPATH=\"$PYTHONPATH:$PWD\"\n  - export PYTHONPATH=\"${PYTHONPATH}:/home/travis/build/p-christ/nn_builder/tests\"\n  - export PYTHONPATH=\"${PYTHONPATH}:/home/travis/build/p-christ/nn_builder\"\n  - python -m pytest /home/travis/build/p-christ/nn_builder/tests/\n\n\n\n\n\n\n\n\n\n\n\n"
  },
  {
    "path": "LICENSE",
    "content": "Copyright (c) 2018 The Python Packaging Authority\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE."
  },
  {
    "path": "README.md",
    "content": "[![Downloads](https://pepy.tech/badge/nn-builder)](https://pepy.tech/project/nn-builder) ![Image](https://travis-ci.org/p-christ/nn_builder.svg?branch=master) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/dwyl/esta/issues)  \n\n\n![nn_builder](miscellaneous/material_for_readme/nn_builder_new.png)\n\n\n\n**nn_builder lets you build neural networks with less boilerplate code**. You specify the type of network you want and it builds it.\n\n### Install\n\n`pip install nn_builder`\n\n### Support\n\n| Network Type       | **NN**  | **CNN** | **RNN** |\n| ------- | ------- | ------- | ------- |\n| PyTorch     | :heavy_check_mark: | :heavy_check_mark:    | :heavy_check_mark:    |\n| TensorFlow 2.0  |        :heavy_check_mark:  |  :heavy_check_mark: | :heavy_check_mark: |                             |\n\n\n### Examples\n\nOn the left is how you can create the PyTorch neural network on the right in only 1 line of code using nn_builder:\n\n![Screenshot](miscellaneous/material_for_readme/nn_builder_use_case.png)\n\nSimilarly for TensorFlow on the left is how you can create the CNN on the right in only 1 line of code using nn_builder: \n\n![Screenshot](miscellaneous/material_for_readme/tf_nn_builder_example.png)\n\n### Usage\n\nSee this [colab notebook](https://colab.research.google.com/drive/1UdMT3aVSV0L5Rq11nyLHxMSVtTVZryhW) for lots of examples of how to use the module. \n3 types of PyTorch and TensorFlow network are currently supported: NN, CNN and RNN. Each network takes the following arguments:\n\n| Field | Description | Default |\n| :---: | :----------: | :---: |\n| *input_dim*| Dimension of the input into the network. See below for more detail. Not needed for Tensorflow.  | N/A |\n| *layers_info* | List to indicate the layers of the network you want. Exact requirements depend on network type, see below for more detail  | N/A |\n| *output_activation* | String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads | No activation |                              \n| *hidden_activations* | String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer), default is ReLU and for example \"tanh\" would have tanh applied on all hidden layer activations | ReLU after every hidden layer |\n| *dropout* | Float to indicate what dropout probability you want applied after each hidden layer | 0 |\n| *initialiser* | String to indicate which initialiser you want used to initialise all the parameters | PyTorch & TF Default |\n| *batch_norm* | Boolean to indicate whether you want batch norm applied to the output of every hidden layer | False |\n| *columns of_data_to_be_embedded* | List to indicate the column numbers of the data that you want to be put through an embedding layer before being fed through the hidden layers of the network | No embeddings |\n| *embedding_dimensions* | If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list of the form: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...] | No embeddings |\n| *y_range* | Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks | No range |\n| *random_seed* | Integer to indicate the random seed you want to use | 0 |\n| *return_final_seq_only* | Only needed for RNN. Boolean to indicate whether you only want to return the output for the final timestep (True) or if you want to return the output for all timesteps (False) | True |\n\nEach network type has slightly different requirements for **input_dim** and **layers_info** as explained below:\n\n--- \n\n### 1. NN\n\n* **input_dim**: # Features in PyTorch, not needed for TensorFlow\n* **layers_info**: List of integers to indicate number of hidden units you want per linear layer. \n* For example:\n\n```\nfrom nn_builder.pytorch.NN import NN   \nmodel = NN(input_dim=5, layers_info=[10, 10, 1], output_activation=None, hidden_activations=\"relu\", \n           dropout=0.0, initialiser=\"xavier\", batch_norm=False)            \n```\n--- \n### 2. CNN\n\n* **input_dim**: (# Channels, Height, Width) in PyTorch, not needed for TensorFlow\n* **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a  CNN can be one of these 4 forms: \n    * [\"conv\", channels, kernel size, stride, padding] \n    * [\"maxpool\", kernel size, stride, padding]\n    * [\"avgpool\", kernel size, stride, padding]\n    * [\"linear\", units]\n* For a PyTorch network kernel size, stride, padding and units must be integers. For TensorFlow they must all be integers except for padding which must be one of {“valid”, “same”} \n* For example:\n```\nfrom nn_builder.pytorch.CNN import CNN   \nmodel = CNN(input_dim=(3, 64, 64), \n            layers_info=[[\"conv\", 32, 3, 1, 0], [\"maxpool\", 2, 2, 0], \n                         [\"conv\", 64, 3, 1, 2], [\"avgpool\", 2, 2, 0], \n                         [\"linear\", 10]],\n            hidden_activations=\"relu\", output_activation=\"softmax\", dropout=0.0,\n            initialiser=\"xavier\", batch_norm=True)\n```\n--- \n### 3. RNN\n\n* **input_dim**: # Features in PyTorch, not needed for TensorFlow\n* **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a  CNN can be one of these 4 forms: \n    * [\"lstm\", units] \n    * [\"gru\", units]\n    * [\"linear\", units]\n* For example:\n\n```\nfrom nn_builder.pytorch.CNN import CNN   \nmodel = RNN(input_dim=5, layers_info=[[\"gru\", 50], [\"lstm\", 10], [\"linear\", 2]],\n            hidden_activations=\"relu\", output_activation=\"softmax\", \n            batch_norm=False, dropout=0.0, initialiser=\"xavier\")\n```\n--- \n## Contributing\n\nAnyone is very welcome to contribute via a pull request. Please see the [issues](https://github.com/p-christ/nn_builder/issues) \npage for ideas on the best areas to contribute to and try to:\n1. Add tests to the tests folder that cover any code you write\n1. Write comments for every function\n1. Create a colab notebook demonstrating how any extra functionality you created works\n\nTo help you remember things you learn about machine learning in general checkout [Gizmo](https://gizmo.ai/landing/github_links)\n \n\n \n\n\n\n"
  },
  {
    "path": "miscellaneous/material_for_readme/README.md",
    "content": "[![Downloads](https://pepy.tech/badge/nn-builder)](https://pepy.tech/project/nn-builder) ![Image](https://travis-ci.org/p-christ/nn_builder.svg?branch=master) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/dwyl/esta/issues)  \n\n\n![nn_builder](miscellaneous/material_for_readme/nn_builder_new.png)\n\n\n\n**nn_builder lets you build neural networks without boilerplate code**. You specify the type of network you want and it builds it.\n\n### Install\n\n`pip install nn_builder`\n\n### Support\n\n| Network Type       | **NN**  | **CNN** | **RNN** |\n| ------- | ------- | ------- | ------- |\n| PyTorch     | :heavy_check_mark: | :heavy_check_mark:    | :heavy_check_mark:    |\n| TensorFlow 2.0  |        :heavy_check_mark:  |  :heavy_check_mark: | :heavy_check_mark: |                             |\n\n\n### Examples\n\nOn the left is how you can create the PyTorch neural network on the right in only 1 line of code using nn_builder:\n\n+-----------------------------------------------------------+----------------------------------------------------------+\n| **Building a simple NN with nn_builder**                  | **Building a simple NN without nn_builder**              | \n+-----------------------------------------------------------+----------------------------------------------------------+\n| .. code:: python                                          | .. code:: python                                         |\n|  # With nn_builder                                        |  # Without nn_builder                                    |\n|  from nn_builder.pytorch.NN import NN                     |  import torch.nn as nn                                   |\n|  model = NN(input_dim=25,                                 |  class NN(nn.Module):                                    |\n|             layers=[150, 100, 50, 50, 50, 50, 5],         |    def __init__(self):                                   |\n|             output_activation=\"softmax\", dropout=0.5,     |      nn.Module.__init__(self)                            |\n|             initialiser=\"xavier\", batch_norm=True)        |      self.fc1 = nn.Linear(25, 150)                       |\n|                                                           |      self.fc2 = nn.Linear(150, 100)                      |\n|                                                           |      self.fc3 = nn.Linear(100, 50)                       |\n|                                                           |      self.fc4 = nn.Linear(50, 50)                        |\n|                                                           |      self.fc5 = nn.Linear(50, 50)                        |\n|                                                           |      self.fc6 = nn.Linear(50, 50)                        |\n|                                                           |      self.fc7 = nn.Linear(50, 5)                         |\n|                                                           |                                                          |\n|                                                           |      self.bn1 = nn.BatchNorm1d(150)                      |\n|                                                           |      self.bn2 = nn.BatchNorm1d(100)                      |\n|                                                           |      self.bn3 = nn.BatchNorm1d(50)                       |\n|                                                           |      self.bn4 = nn.BatchNorm1d(50)                       |\n|                                                           |      self.bn5 = nn.BatchNorm1d(50)                       |\n|                                                           |      self.bn6 = nn.BatchNorm1d(50)                       |\n|                                                           |                                                          |\n|                                                           |      self.dropout = nn.Dropout(p=0.5)                    |\n|                                                           |      for params in [self.fc1, self.fc2, self.fc3,        |\n|                                                           |                     self.fc4, self.fc5, self.fc6,        |\n|                                                           |                     self.fc7]:                           |\n|                                                           |        nn.init.xavier_uniform_(params.weight)            |\n|                                                           |    def forward(self, x):                                 |\n|                                                           |      x = self.dropout(self.bn1(nn.ReLU()(self.fc1(x))))  |\n|                                                           |      x = self.dropout(self.bn2(nn.ReLU()(self.fc2(x))))  |\n|                                                           |      x = self.dropout(self.bn3(nn.ReLU()(self.fc3(x))))  |\n|                                                           |      x = self.dropout(self.bn4(nn.ReLU()(self.fc4(x))))  |\n|                                                           |      x = self.dropout(self.bn5(nn.ReLU()(self.fc5(x))))  |\n|                                                           |      x = self.dropout(self.bn6(nn.ReLU()(self.fc6(x))))  |\n|                                                           |      x = self.fc7(x)                                     |\n|                                                           |      x = nn.Softmax(dim=1)(x)                            |\n|                                                           |      return x                                            |\n|                                                           |                                                          |\n|                                                           |  model = NN()                                            |\n+-----------------------------------------------------------+----------------------------------------------------------+      \n                                                           \n\n\n\n\n\n\n![Screenshot](miscellaneous/material_for_readme/nn_builder_use_case.png)\n\nSimilarly for TensorFlow on the left is how you can create the CNN on the right in only 1 line of code using nn_builder: \n\n![Screenshot](miscellaneous/material_for_readme/tf_nn_builder_example.png)\n\n### Usage\n\nSee this [colab notebook](https://colab.research.google.com/drive/1UdMT3aVSV0L5Rq11nyLHxMSVtTVZryhW) for lots of examples of how to use the module. \n3 types of PyTorch and TensorFlow network are currently supported: NN, CNN and RNN. Each network takes the following arguments:\n\n| Field | Description | Default |\n| :---: | :----------: | :---: |\n| *input_dim*| Dimension of the input into the network. See below for more detail. Not needed for Tensorflow.  | N/A |\n| *layers_info* | List to indicate the layers of the network you want. Exact requirements depend on network type, see below for more detail  | N/A |\n| *output_activation* | String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads | No activation |                              \n| *hidden_activations* | String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer), default is ReLU and for example \"tanh\" would have tanh applied on all hidden layer activations | ReLU after every hidden layer |\n| *dropout* | Float to indicate what dropout probability you want applied after each hidden layer | 0 |\n| *initialiser* | String to indicate which initialiser you want used to initialise all the parameters | PyTorch & TF Default |\n| *batch_norm* | Boolean to indicate whether you want batch norm applied to the output of every hidden layer | False |\n| *columns of_data_to_be_embedded* | List to indicate the column numbers of the data that you want to be put through an embedding layer before being fed through the hidden layers of the network | No embeddings |\n| *embedding_dimensions* | If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list of the form: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...] | No embeddings |\n| *y_range* | Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks | No range |\n| *random_seed* | Integer to indicate the random seed you want to use | 0 |\n| *return_final_seq_only* | Only needed for RNN. Boolean to indicate whether you only want to return the output for the final timestep (True) or if you want to return the output for all timesteps (False) | True |\n\nEach network type has slightly different requirements for **input_dim** and **layers_info** as explained below:\n\n--- \n\n##### 1. NN\n\n* **input_dim**: # Features in PyTorch, not needed for TensorFlow\n* **layers_info**: List of integers to indicate number of hidden units you want per linear layer. \n* For example:\n\n```\nfrom nn_builder.pytorch.NN import NN   \nmodel = NN(input_dim=5, layers_info=[10, 10, 1], output_activation=None, hidden_activations=\"relu\", \n           dropout=0.0, initialiser=\"xavier\", batch_norm=False)            \n```\n--- \n##### 2. CNN\n\n* **input_dim**: (# Channels, Height, Width) in PyTorch, not needed for TensorFlow\n* **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a  CNN can be one of these 4 forms: \n    * [\"conv\", channels, kernel size, stride, padding] \n    * [\"maxpool\", kernel size, stride, padding]\n    * [\"avgpool\", kernel size, stride, padding]\n    * [\"linear\", units]\n* For a PyTorch network kernel size, stride, padding and units must be integers. For TensorFlow they must all be integers except for padding which must be one of {“valid”, “same”} \n* For example:\n```\nfrom nn_builder.pytorch.CNN import CNN   \nmodel = CNN(input_dim=(3, 64, 64), \n            layers_info=[[\"conv\", 32, 3, 1, 0], [\"maxpool\", 2, 2, 0], \n                         [\"conv\", 64, 3, 1, 2], [\"avgpool\", 2, 2, 0], \n                         [\"linear\", 10]],\n            hidden_activations=\"relu\", output_activation=\"softmax\", dropout=0.0,\n            initialiser=\"xavier\", batch_norm=True)\n```\n--- \n##### 3. RNN\n\n* **input_dim**: # Features in PyTorch, not needed for TensorFlow\n* **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a  CNN can be one of these 4 forms: \n    * [\"lstm\", units] \n    * [\"gru\", units]\n    * [\"linear\", units]\n* For example:\n\n```\nfrom nn_builder.pytorch.CNN import CNN   \nmodel = RNN(input_dim=5, layers_info=[[\"gru\", 50], [\"lstm\", 10], [\"linear\", 2]],\n            hidden_activations=\"relu\", output_activation=\"softmax\", \n            batch_norm=False, dropout=0.0, initialiser=\"xavier\")\n```\n--- \n## Contributing\n\nTogether we can make something that is useful for thousands of people. Anyone is very welcome to contribute via a pull request. Please see the [issues](https://github.com/p-christ/nn_builder/issues) \npage for ideas on the best areas to contribute to and try to:\n1. Add tests to the tests folder that cover any code you write\n1. Write comments for every function\n1. Create a colab notebook demonstrating how any extra functionality you created works\n\n \n\n \n\n\n\n"
  },
  {
    "path": "miscellaneous/material_for_readme/nn_builder_code",
    "content": "# With nn_builder\nfrom nn_builder.pytorch.NN import NN\n\nmodel = NN(input_dim=25, layers=[150, 100, 50, 50, 50, 50, 5],\n           output_activation=\"softmax\", dropout=0.5, initialiser=\"xavier\",\n           batch_norm=True)\n"
  },
  {
    "path": "miscellaneous/material_for_readme/nn_builder_tf_code",
    "content": "#With nn_builder\nfrom nn_builder.tensorflow.CNN import CNN\nmodel=CNN(layers_info=[[\"conv\", 32, 3, 1, \"valid\"],[\"maxpool\", 3, 3, \"valid\"],\n                       [\"conv\", 64, 3, 1, \"valid\"],[\"maxpool\", 3, 2, \"valid\"],\n                       [\"conv\", 128, 3, 1, \"valid\"],[\"maxpool\", 3, 1, \"valid\"],\n                       [\"conv\", 256, 3, 1, \"valid\"],[\"avgpool\", 3, 1, \"valid\"],\n                       [\"linear\", 64], [\"linear\", 10]],\n            hidden_activations=\"relu\", output_activation=\"softmax\",\n            dropout=0.2, initialiser=\"glorot_normal\", batch_norm=False)\n"
  },
  {
    "path": "miscellaneous/material_for_readme/non_nn_builder_code",
    "content": "# Without nn_builder\nimport torch.nn as nn\n\nclass NN(nn.Module):\n    def __init__(self):\n        nn.Module.__init__(self)\n\n        self.fc1 = nn.Linear(25, 150)\n        self.fc2 = nn.Linear(150, 100)\n        self.fc3 = nn.Linear(100, 50)\n        self.fc4 = nn.Linear(50, 50)\n        self.fc5 = nn.Linear(50, 50)\n        self.fc6 = nn.Linear(50, 50)\n        self.fc7 = nn.Linear(50, 5)\n\n        self.bn1 = nn.BatchNorm1d(150)\n        self.bn2 = nn.BatchNorm1d(100)\n        self.bn3 = nn.BatchNorm1d(50)\n        self.bn4 = nn.BatchNorm1d(50)\n        self.bn5 = nn.BatchNorm1d(50)\n        self.bn6 = nn.BatchNorm1d(50)\n\n        self.dropout = nn.Dropout(p=0.5)\n\n        for params in [self.fc1, self.fc2, self.fc3, self.fc4, self.fc5,\n                       self.fc6, self.fc7]:\n                       nn.init.xavier_uniform_(params.weight)\n\n    def forward(self, x):\n        x = self.dropout(self.bn1(nn.ReLU()(self.fc1(x))))\n        x = self.dropout(self.bn2(nn.ReLU()(self.fc2(x))))\n        x = self.dropout(self.bn3(nn.ReLU()(self.fc3(x))))\n        x = self.dropout(self.bn4(nn.ReLU()(self.fc4(x))))\n        x = self.dropout(self.bn5(nn.ReLU()(self.fc5(x))))\n        x = self.dropout(self.bn6(nn.ReLU()(self.fc6(x))))\n        x = self.fc7(x)\n        x = nn.Softmax(dim=1)(x)\n        return x\n\nmodel = NN()\n"
  },
  {
    "path": "miscellaneous/material_for_readme/non_nn_builder_tf_code",
    "content": "#Without nn_builder\nimport tensorflow as tf\nfrom tensorflow.keras import Model, activations\nfrom tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, \\\n                                    MaxPool2D, AveragePooling2D\nclass CNN(Model):\n    def __init__(self):\n        Model.__init__(self)\n\n        self.conv1 = Conv2D(filters=32, kernel_size=3, strides=1,\n                            padding=\"valid\", activation=\"relu\",\n                            kernel_initializer=\"glorot_normal\")\n        self.conv2 = Conv2D(filters=64, kernel_size=3, strides=1,\n                            padding=\"valid\", activation=\"relu\",\n                            kernel_initializer=\"glorot_normal\")\n        self.conv3 = Conv2D(filters=128, kernel_size=3, strides=1,\n                            padding=\"valid\", activation=\"relu\",\n                            kernel_initializer=\"glorot_normal\")\n        self.conv4 = Conv2D(filters=256, kernel_size=3, strides=1,\n                            padding=\"valid\", activation=\"relu\",\n                            kernel_initializer=\"glorot_normal\")\n\n        self.maxpool1 = MaxPool2D(pool_size=3, strides=3, padding=\"valid\")\n        self.maxpool2 = MaxPool2D(pool_size=3, strides=2, padding=\"valid\")\n        self.maxpool3 = MaxPool2D(pool_size=3, strides=1, padding=\"valid\")\n        self.avgpool = AveragePooling2D(pool_size=3, strides=1, padding=\"valid\")\n\n        self.linear1 = Dense(64, activation=\"relu\")\n        self.linear2 = Dense(10, activation=\"softmax\")\n\n        self.dropout = tf.keras.layers.Dropout(rate=0.2)\n\n    def call(self, x):\n        x = self.dropout(self.maxpool1(self.conv1(x)))\n        x = self.dropout(self.maxpool2(self.conv2(x)))\n        x = self.dropout(self.maxpool3(self.conv3(x)))\n        x = self.dropout(self.avgpool(self.conv4(x)))\n        x = Flatten()(x)\n        x = self.dropout(self.linear1(x))\n        x = self.linear2(x)\n        return x\n\nmodel = CNN()\n"
  },
  {
    "path": "miscellaneous/testreadme.rst",
    "content": "\n+-----------------------------------------------------------+----------------------------------------------------------+\n| **Building a simple NN with nn_builder**                  | **Building a simple NN without nn_builder**              |\n+-----------------------------------------------------------+----------------------------------------------------------+\n| .. code:: python                                          | .. code:: python                                         |\n|  # With nn_builder                                        |  # Without nn_builder                                    |\n|  from nn_builder.pytorch.NN import NN                     |  import torch.nn as nn                                   |\n|  model = NN(input_dim=25,                                 |  class NN(nn.Module):                                    |\n|             layers=[150, 100, 50, 50, 50, 50, 5],         |    def __init__(self):                                   |\n|             output_activation=\"softmax\", dropout=0.5,     |      nn.Module.__init__(self)                            |\n|             initialiser=\"xavier\", batch_norm=True)        |      self.fc1 = nn.Linear(25, 150)                       |\n|                                                           |      self.fc2 = nn.Linear(150, 100)                      |\n|                                                           |      self.fc3 = nn.Linear(100, 50)                       |\n|                                                           |      self.fc4 = nn.Linear(50, 50)                        |\n|                                                           |      self.fc5 = nn.Linear(50, 50)                        |\n|                                                           |      self.fc6 = nn.Linear(50, 50)                        |\n|                                                           |      self.fc7 = nn.Linear(50, 5)                         |\n|                                                           |                                                          |\n|                                                           |      self.bn1 = nn.BatchNorm1d(150)                      |\n|                                                           |      self.bn2 = nn.BatchNorm1d(100)                      |\n|                                                           |      self.bn3 = nn.BatchNorm1d(50)                       |\n|                                                           |      self.bn4 = nn.BatchNorm1d(50)                       |\n|                                                           |      self.bn5 = nn.BatchNorm1d(50)                       |\n|                                                           |      self.bn6 = nn.BatchNorm1d(50)                       |\n|                                                           |                                                          |\n|                                                           |      self.dropout = nn.Dropout(p=0.5)                    |\n|                                                           |      for params in [self.fc1, self.fc2, self.fc3,        |\n|                                                           |                     self.fc4, self.fc5, self.fc6,        |\n|                                                           |                     self.fc7]:                           |\n|                                                           |        nn.init.xavier_uniform_(params.weight)            |\n|                                                           |    def forward(self, x):                                 |\n|                                                           |      x = self.dropout(self.bn1(nn.ReLU()(self.fc1(x))))  |\n|                                                           |      x = self.dropout(self.bn2(nn.ReLU()(self.fc2(x))))  |\n|                                                           |      x = self.dropout(self.bn3(nn.ReLU()(self.fc3(x))))  |\n|                                                           |      x = self.dropout(self.bn4(nn.ReLU()(self.fc4(x))))  |\n|                                                           |      x = self.dropout(self.bn5(nn.ReLU()(self.fc5(x))))  |\n|                                                           |      x = self.dropout(self.bn6(nn.ReLU()(self.fc6(x))))  |\n|                                                           |      x = self.fc7(x)                                     |\n|                                                           |      x = nn.Softmax(dim=1)(x)                            |\n|                                                           |      return x                                            |\n|                                                           |                                                          |\n|                                                           |  model = NN()                                            |\n+-----------------------------------------------------------+----------------------------------------------------------+\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n+------------------------------------------------+--------------------------------------------+\n| **Script to train an SVM on the iris dataset** | **The same script as a Sacred experiment** |\n+------------------------------------------------+--------------------------------------------+\n| .. code:: python                               | .. code:: python                           |\n|  # With nn_builder                                              |                                            |\n|  from nn_builder.pytorch.NN import NN          |   from numpy.random import permutation     |\n|                                                |   from sklearn import svm, datasets        |\n|                                                |   from sacred import Experiment            |\n|                                                |   ex = Experiment('iris_rbf_svm')          |\n|                                                |                                            |\n|                                                |   @ex.config                               |\n|                                                |   def cfg():                               |\n|  C = 1.0                                       |     C = 1.0                                |\n|  gamma = 0.7                                   |     gamma = 0.7                            |\n|                                                |                                            |\n|                                                |   @ex.automain                             |\n|                                                |   def run(C, gamma):                       |\n|  iris = datasets.load_iris()                   |     iris = datasets.load_iris()            |\n|  perm = permutation(iris.target.size)          |     per = permutation(iris.target.size)    |\n|  iris.data = iris.data[perm]                   |     iris.data = iris.data[per]             |\n|  iris.target = iris.target[perm]               |     iris.target = iris.target[per]         |\n|  clf = svm.SVC(C, 'rbf', gamma=gamma)          |     clf = svm.SVC(C, 'rbf', gamma=gamma)   |\n|  clf.fit(iris.data[:90],                       |     clf.fit(iris.data[:90],                |\n|          iris.target[:90])                     |             iris.target[:90])              |\n|  print(clf.score(iris.data[90:],               |     return clf.score(iris.data[90:],       |\n|                  iris.target[90:]))            |                      iris.target[90:])     |\n+------------------------------------------------+--------------------------------------------+\n\n\n# With nn_builder\n\n\nmodel = NN(input_dim=25, layers=[150, 100, 50, 50, 50, 50, 5],\n           output_activation=\"softmax\", dropout=0.5, initialiser=\"xavier\",\n           batch_norm=True)\n\n"
  },
  {
    "path": "nn_builder/Overall_Base_Network.py",
    "content": "from abc import ABC, abstractmethod\n\nclass Overall_Base_Network(ABC):\n\n    def __init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm,\n                 y_range, random_seed):\n\n        self.set_all_random_seeds(random_seed)\n        self.input_dim = input_dim\n        self.layers_info = layers_info\n\n        self.hidden_activations = hidden_activations\n        self.output_activation = output_activation\n        self.dropout = dropout\n        self.initialiser = initialiser\n        self.batch_norm = batch_norm\n        self.y_range = y_range\n\n        self.str_to_activations_converter = self.create_str_to_activations_converter()\n        self.str_to_initialiser_converter = self.create_str_to_initialiser_converter()\n\n        self.check_all_user_inputs_valid()\n\n        self.initialiser_function = self.str_to_initialiser_converter[initialiser.lower()]\n\n        self.hidden_layers = self.create_hidden_layers()\n        self.output_layers = self.create_output_layers()\n        self.dropout_layer = self.create_dropout_layer()\n        if self.batch_norm: self.batch_norm_layers = self.create_batch_norm_layers()\n\n    @abstractmethod\n    def check_all_user_inputs_valid(self):\n        \"\"\"Checks that all the user inputs were valid\"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def create_hidden_layers(self):\n        \"\"\"Creates the hidden layers in the network\"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def create_output_layers(self):\n        \"\"\"Creates the output layers in the network\"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def create_batch_norm_layers(self):\n        \"\"\"Creates the batch norm layers in the network\"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def create_dropout_layer(self):\n        \"\"\"Creates the dropout layers in the network\"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def print_model_summary(self):\n        \"\"\"Prints a summary of the model\"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def set_all_random_seeds(self, random_seed):\n        \"\"\"Sets all random seeds\"\"\"\n        raise NotImplementedError\n\n    def check_NN_layers_valid(self):\n        \"\"\"Checks that user input for hidden_units is valid\"\"\"\n        assert isinstance(self.layers_info, list), \"hidden_units must be a list\"\n        list_error_msg = \"neurons must be a list of integers\"\n        integer_error_msg = \"Every element of hidden_units must be 1 or higher\"\n        activation_error_msg = \"The number of output activations provided should match the number of output layers\"\n        for neurons in self.layers_info[:-1]:\n            assert isinstance(neurons, int), list_error_msg\n            assert neurons > 0, integer_error_msg\n        output_layer = self.layers_info[-1]\n        if isinstance(output_layer, list):\n            assert len(output_layer) == len(self.output_activation), activation_error_msg\n            for output_dim in output_layer:\n                assert isinstance(output_dim, int), list_error_msg\n                assert output_dim > 0, integer_error_msg\n        else:\n            assert isinstance(self.output_activation, str) or self.output_activation is None, activation_error_msg\n            assert isinstance(output_layer, int), list_error_msg\n            assert output_layer > 0, integer_error_msg\n\n    def check_NN_input_dim_valid(self):\n        \"\"\"Checks that user input for input_dim is valid\"\"\"\n        assert isinstance(self.input_dim, int), \"input_dim must be an integer\"\n        assert self.input_dim > 0, \"input_dim must be 1 or higher\"\n\n    def check_activations_valid(self):\n        \"\"\"Checks that user input for hidden_activations and output_activation is valid\"\"\"\n        valid_activations_strings = self.str_to_activations_converter.keys()\n        if self.output_activation is None: self.output_activation = \"None\"\n        if isinstance(self.output_activation, list):\n            for activation in self.output_activation:\n                if activation is not None:\n                    assert activation.lower() in set(valid_activations_strings), \"Output activations must be string from list {}\".format(valid_activations_strings)\n        else:\n            assert self.output_activation.lower() in set(valid_activations_strings), \"Output activation must be string from list {}\".format(valid_activations_strings)\n        assert isinstance(self.hidden_activations, str) or isinstance(self.hidden_activations, list), \"hidden_activations must be a string or a list of strings\"\n        if isinstance(self.hidden_activations, str):\n            assert self.hidden_activations.lower() in set(valid_activations_strings), \"hidden_activations must be from list {}\".format(valid_activations_strings)\n        elif isinstance(self.hidden_activations, list):\n            assert len(self.hidden_activations) == len(self.layers_info), \"if hidden_activations is a list then you must provide 1 activation per hidden layer\"\n            for activation in self.hidden_activations:\n                assert isinstance(activation, str), \"hidden_activations must be a string or list of strings\"\n                assert activation.lower() in set(valid_activations_strings), \"each element in hidden_activations must be from list {}\".format(valid_activations_strings)\n\n    def check_embedding_dimensions_valid(self):\n        \"\"\"Checks that user input for embedding_dimensions is valid\"\"\"\n        assert isinstance(self.embedding_dimensions, list), \"embedding_dimensions must be a list\"\n        for embedding_dim in self.embedding_dimensions:\n            assert len(embedding_dim) == 2 and isinstance(embedding_dim, list), \\\n                \"Each element of embedding_dimensions must be of form (input_dim, output_dim)\"\n\n    def check_y_range_values_valid(self):\n        \"\"\"Checks that user input for y_range is valid\"\"\"\n        if self.y_range:\n            assert isinstance(self.y_range, tuple) and len(self.y_range) == 2, \"y_range must be a tuple of 2 floats or integers\"\n            for elem in range(2):\n                assert isinstance(self.y_range[elem], float) or isinstance(self.y_range[elem], int), \"y_range must be a tuple of 2 floats or integers\"\n            assert self.y_range[0] <= self.y_range[1], \"y_range's first element must be smaller than the second element\"\n\n    def check_timesteps_to_output_valid(self):\n        \"\"\"Checks that user input for timesteps_to_output is valid\"\"\"\n        assert self.timesteps_to_output in [\"all\", \"last\"]\n\n    def check_initialiser_valid(self):\n        \"\"\"Checks that user input for initialiser is valid\"\"\"\n        valid_initialisers = set(self.str_to_initialiser_converter.keys())\n        assert isinstance(self.initialiser, str), \"initialiser must be a string from list {}\".format(valid_initialisers)\n        assert self.initialiser.lower() in valid_initialisers, \"initialiser must be from list {}\".format(valid_initialisers)\n\n    def check_return_final_seq_only_valid(self):\n        \"\"\"Checks whether user input for return_final_seq_only is a boolean and therefore valid. Only relevant for RNNs\"\"\"\n        assert isinstance(self.return_final_seq_only, bool)\n\n    def get_activation(self, activations, ix=None):\n        \"\"\"Gets the activation function\"\"\"\n        if isinstance(activations, list):\n            return self.str_to_activations_converter[str(activations[ix]).lower()]\n        return self.str_to_activations_converter[str(activations).lower()]\n"
  },
  {
    "path": "nn_builder/__init__.py",
    "content": ""
  },
  {
    "path": "nn_builder/pytorch/Base_Network.py",
    "content": "import random\nimport numpy as np\nimport torch\nimport torch.nn as nn\nfrom nn_builder.Overall_Base_Network import Overall_Base_Network\nfrom abc import ABC, abstractmethod\n\nclass Base_Network(Overall_Base_Network, ABC):\n    \"\"\"Base class for PyTorch neural network classes\"\"\"\n    def __init__(self, input_dim, layers_info, output_activation,\n                 hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed):\n        self.str_to_activations_converter = self.create_str_to_activations_converter()\n        self.str_to_initialiser_converter = self.create_str_to_initialiser_converter()\n        super().__init__(input_dim, layers_info, output_activation,\n                 hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed)\n        self.initialise_all_parameters()\n        # Flag we use to run checks on the input data into forward the first time it is entered\n        self.checked_forward_input_data_once = False\n\n    @abstractmethod\n    def initialise_all_parameters(self):\n        \"\"\"Initialises all the parameters of the network\"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def forward(self, input_data):\n        \"\"\"Runs a forward pass of the network\"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def check_input_data_into_forward_once(self, input_data):\n        \"\"\"Checks the input data into the network is of the right form. Only runs the first time data is provided\n        otherwise would slow down training too much\"\"\"\n        raise NotImplementedError\n\n    def set_all_random_seeds(self, random_seed):\n        \"\"\"Sets all possible random seeds so results can be reproduced\"\"\"\n        torch.backends.cudnn.deterministic = True\n        torch.manual_seed(random_seed)\n        random.seed(random_seed)\n        np.random.seed(random_seed)\n        if torch.cuda.is_available(): torch.cuda.manual_seed_all(random_seed)\n\n    def create_str_to_activations_converter(self):\n        \"\"\"Creates a dictionary which converts strings to activations\"\"\"\n        str_to_activations_converter = {\"elu\": nn.ELU(), \"hardshrink\": nn.Hardshrink(), \"hardtanh\": nn.Hardtanh(),\n                                        \"leakyrelu\": nn.LeakyReLU(), \"logsigmoid\": nn.LogSigmoid(), \"prelu\": nn.PReLU(),\n                                        \"relu\": nn.ReLU(), \"relu6\": nn.ReLU6(), \"rrelu\": nn.RReLU(), \"selu\": nn.SELU(),\n                                        \"sigmoid\": nn.Sigmoid(), \"softplus\": nn.Softplus(), \"logsoftmax\": nn.LogSoftmax(),\n                                        \"softshrink\": nn.Softshrink(), \"softsign\": nn.Softsign(), \"tanh\": nn.Tanh(),\n                                        \"tanhshrink\": nn.Tanhshrink(), \"softmin\": nn.Softmin(), \"softmax\": nn.Softmax(dim=1),\n                                         \"none\": None}\n        return str_to_activations_converter\n\n    def create_str_to_initialiser_converter(self):\n        \"\"\"Creates a dictionary which converts strings to initialiser\"\"\"\n        str_to_initialiser_converter = {\"uniform\": nn.init.uniform_, \"normal\": nn.init.normal_,\n                                        \"eye\": nn.init.eye_,\n                                        \"xavier_uniform\": nn.init.xavier_uniform_, \"xavier\": nn.init.xavier_uniform_,\n                                        \"xavier_normal\": nn.init.xavier_normal_,\n                                        \"kaiming_uniform\": nn.init.kaiming_uniform_, \"kaiming\": nn.init.kaiming_uniform_,\n                                        \"kaiming_normal\": nn.init.kaiming_normal_, \"he\": nn.init.kaiming_normal_,\n                                        \"orthogonal\": nn.init.orthogonal_,  \"default\": \"use_default\"}\n        return str_to_initialiser_converter\n\n    def create_dropout_layer(self):\n        \"\"\"Creates a dropout layer\"\"\"\n        return nn.Dropout(p=self.dropout)\n\n    def create_embedding_layers(self):\n        \"\"\"Creates the embedding layers in the network\"\"\"\n        embedding_layers = nn.ModuleList([])\n        for embedding_dimension in self.embedding_dimensions:\n            input_dim, output_dim = embedding_dimension\n            embedding_layers.extend([nn.Embedding(input_dim, output_dim)])\n        return embedding_layers\n\n    def initialise_parameters(self, parameters_list):\n        \"\"\"Initialises the list of parameters given\"\"\"\n        initialiser = self.str_to_initialiser_converter[self.initialiser.lower()]\n        if initialiser != \"use_default\":\n            for parameters in parameters_list:\n                if type(parameters) == nn.Linear or type(parameters) == nn.Conv2d:\n                    initialiser(parameters.weight)\n                elif type(parameters) in [nn.LSTM, nn.RNN, nn.GRU]:\n                    initialiser(parameters.weight_hh_l0)\n                    initialiser(parameters.weight_ih_l0)\n\n    def flatten_tensor(self, tensor):\n        \"\"\"Flattens a tensor of shape (a, b, c, d, ...) into shape (a, b * c * d * .. )\"\"\"\n        return tensor.reshape(tensor.shape[0], -1)\n\n    def print_model_summary(self):\n        print(self)\n\n"
  },
  {
    "path": "nn_builder/pytorch/CNN.py",
    "content": "import torch\nimport torch.nn as nn\nimport numpy as np\nfrom nn_builder.pytorch.Base_Network import Base_Network\n\nclass CNN(nn.Module, Base_Network):\n    \"\"\"Creates a PyTorch convolutional neural network\n    Args:\n        - input_dim: Tuple of integers to indicate the (channels, height, width) dimension of the input\n        - layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be\n                         one of these 6 forms:\n                         - [\"conv\", channels, kernel_size, stride, padding]\n                         - [\"maxpool\", kernel_size, stride, padding]\n                         - [\"avgpool\", kernel_size, stride, padding]\n                         - [\"adaptivemaxpool\", output height, output width]\n                         - [\"adaptiveavgpool\", output height, output width]\n                         - [\"linear\", out]\n        - output_activation: String to indicate the activation function you want the output to go through. Provide a list of\n                             strings if you want multiple output heads\n        - hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers\n                              (not including the output layer). Default is ReLU.\n        - dropout: Float to indicate what dropout probability you want applied after each hidden layer\n        - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch\n                       initialisers are supported. PyTorch's default initialisation is the default.\n        - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False\n        - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the\n                   output values to in regression tasks. Default is no range restriction\n        - random_seed: Integer to indicate the random seed you want to use\n\n    NOTE that this class' forward method expects input data in the form: (batch, channels, height, width)\n    \"\"\"\n    def __init__(self, input_dim, layers_info, output_activation=None,\n                 hidden_activations=\"relu\", dropout=0.0, initialiser=\"default\", batch_norm=False,\n                 y_range= (), random_seed=0, converted_from_tf_model=False):\n        nn.Module.__init__(self)\n        self.valid_cnn_hidden_layer_types = {'conv', 'maxpool', 'avgpool', 'adaptivemaxpool', 'adaptiveavgpool', 'linear'}\n        self.valid_layer_types_with_no_parameters = [nn.MaxPool2d, nn.AvgPool2d, nn.AdaptiveAvgPool2d, nn.AdaptiveMaxPool2d]\n        Base_Network.__init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser,\n                              batch_norm, y_range, random_seed)\n        self.converted_from_tf_model = converted_from_tf_model\n\n    def flatten_tensor(self, tensor):\n        \"\"\"Flattens a tensor of shape (a, b, c, d, ...) into shape (a, b * c * d * .. )\"\"\"\n        if self.converted_from_tf_model:\n            tensor = tensor.permute(0, 2, 3, 1).contiguous()\n            tensor = tensor.view(tensor.size(0), -1)\n        else:\n            tensor = tensor.reshape(tensor.shape[0], -1)\n        return tensor\n\n    def check_all_user_inputs_valid(self):\n        \"\"\"Checks that all the user inputs were valid\"\"\"\n        self.check_CNN_input_dim_valid()\n        self.check_CNN_layers_valid()\n        self.check_activations_valid()\n        self.check_initialiser_valid()\n        self.check_y_range_values_valid()\n\n    def check_CNN_input_dim_valid(self):\n        \"\"\"Checks that the CNN input dim valid\"\"\"\n        error_msg = \"input_dim must be a tuple of 3 integers indicating (channels, height, width)\"\n        assert isinstance(self.input_dim, tuple), error_msg\n        for element in self.input_dim: assert isinstance(element, int) and element > 0\n\n    def check_CNN_layers_valid(self):\n        \"\"\"Checks that the user inputs for cnn_hidden_layers were valid. cnn_hidden_layers must be a list of layers where\n        each layer must be of one of these forms:\n        - [\"conv\", channels, kernel_size, stride, padding]\n        - [\"maxpool\", kernel_size, stride, padding]\n        - [\"avgpool\", kernel_size, stride, padding]\n        - [\"adaptivemaxpool\", output height, output width]\n        - [\"adaptiveavgpool\", output height, output width]\n        - [\"linear\", out]\n        \"\"\"\n        error_msg_layer_type = \"First element in a layer specification must be one of {}\".format(self.valid_cnn_hidden_layer_types)\n        error_msg_conv_layer = \"\"\"Conv layer must be of form ['conv', channels, kernel_size, stride, padding] where the \n                               final 4 elements are non-negative integers\"\"\"\n        error_msg_maxpool_layer = \"\"\"Maxpool layer must be of form ['maxpool', kernel_size, stride, padding] where the \n                                       final 2 elements are non-negative integers\"\"\"\n        error_msg_avgpool_layer = \"\"\"Avgpool layer must be of form ['avgpool', kernel_size, stride, padding] where the \n                                               final 2 elements are non-negative integers\"\"\"\n        error_msg_adaptivemaxpool_layer = \"\"\"Adaptivemaxpool layer must be of form ['adaptivemaxpool', output height, output width]\"\"\"\n        error_msg_adaptiveavgpool_layer = \"\"\"Adaptiveavgpool layer must be of form ['adaptiveavgpool', output height, output width]\"\"\"\n        error_msg_linear_layer = \"\"\"Linear layer must be of form ['linear', out] where out is a non-negative integers\"\"\"\n        assert isinstance(self.layers_info, list), \"layers must be a list\"\n\n        all_layers = self.layers_info[:-1]\n        output_layer = self.layers_info[-1]\n        assert isinstance(output_layer, list), \"layers must be a list\"\n        if isinstance(output_layer[0], list):\n            assert len(output_layer) == len(\n                self.output_activation), \"Number of output activations must equal number of output heads\"\n            for layer in output_layer:\n                all_layers.append(layer)\n                assert layer[0].lower() == \"linear\", \"Final layer must be linear\"\n        else:\n            all_layers.append(output_layer)\n            assert isinstance(output_layer[0], str), error_msg_layer_type\n            assert output_layer[0].lower() == \"linear\", \"Final layer must be linear\"\n\n        for layer in all_layers:\n            assert isinstance(layer, list), \"Each layer must be a list\"\n            assert isinstance(layer[0], str), error_msg_layer_type\n            layer_type_name = layer[0].lower()\n            assert layer_type_name in self.valid_cnn_hidden_layer_types, \"Layer name {} not valid, use one of {}\".format(layer_type_name, self.valid_cnn_hidden_layer_types)\n            if layer_type_name == \"conv\":\n                assert len(layer) == 5, error_msg_conv_layer\n                for ix in range(3): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0, error_msg_conv_layer\n                assert isinstance(layer[4], int) and layer[4] >= 0, error_msg_conv_layer\n            elif layer_type_name == \"maxpool\":\n                assert len(layer) == 4, error_msg_maxpool_layer\n                for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_maxpool_layer\n                if layer[1] != layer[2]: print(\"NOTE that your maxpool kernel size {} isn't the same as your stride {}\".format(layer[1], layer[2]))\n                assert isinstance(layer[3], int) and layer[3] >= 0, error_msg_conv_layer\n            elif layer_type_name == \"avgpool\":\n                assert len(layer) == 4, error_msg_avgpool_layer\n                for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_avgpool_layer\n                assert isinstance(layer[3], int) and layer[3] >= 0, error_msg_conv_layer\n                if layer[1] != layer[2]:print(\"NOTE that your avgpool kernel size {} isn't the same as your stride {}\".format(layer[1], layer[2]))\n            elif layer_type_name == \"adaptivemaxpool\":\n                assert len(layer) == 3, error_msg_adaptivemaxpool_layer\n                for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_adaptivemaxpool_layer\n            elif layer_type_name == \"adaptiveavgpool\":\n                assert len(layer) == 3, error_msg_adaptiveavgpool_layer\n                for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[\n                    ix + 1] > 0, error_msg_adaptiveavgpool_layer\n            elif layer_type_name == \"linear\":\n                assert len(layer) == 2, error_msg_linear_layer\n                for ix in range(1): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0\n            else:\n                raise ValueError(\"Invalid layer name\")\n\n        rest_must_be_linear = False\n        for ix, layer in enumerate(all_layers):\n            if rest_must_be_linear: assert layer[0].lower() == \"linear\", \"If have linear layers then they must come at end\"\n            if layer[0].lower() == \"linear\":\n                rest_must_be_linear = True\n\n    def create_hidden_layers(self):\n        \"\"\"Creates the hidden layers in the network\"\"\"\n        cnn_hidden_layers = nn.ModuleList([])\n        input_dim = self.input_dim\n        for layer in self.layers_info[:-1]:\n            input_dim = self.create_and_append_layer(input_dim, layer, cnn_hidden_layers)\n        self.input_dim_into_final_layer = input_dim\n        return cnn_hidden_layers\n\n    def create_and_append_layer(self, input_dim, layer, list_to_append_layer_to):\n        \"\"\"Creates and appends a layer to the list provided\"\"\"\n        layer_name = layer[0].lower()\n        assert layer_name in self.valid_cnn_hidden_layer_types, \"Layer name {} not valid, use one of {}\".format(\n            layer_name, self.valid_cnn_hidden_layer_types)\n        if layer_name == \"conv\":\n            list_to_append_layer_to.extend([nn.Conv2d(in_channels=input_dim[0], out_channels=layer[1], kernel_size=layer[2],\n                                                stride=layer[3], padding=layer[4])])\n        elif layer_name == \"maxpool\":\n            list_to_append_layer_to.extend([nn.MaxPool2d(kernel_size=layer[1],\n                                                   stride=layer[2], padding=layer[3])])\n        elif layer_name == \"avgpool\":\n            list_to_append_layer_to.extend([nn.AvgPool2d(kernel_size=layer[1],\n                                                   stride=layer[2], padding=layer[3])])\n        elif layer_name == \"adaptivemaxpool\":\n            list_to_append_layer_to.extend([nn.AdaptiveMaxPool2d(output_size=(layer[1], layer[2]))])\n        elif layer_name == \"adaptiveavgpool\":\n            list_to_append_layer_to.extend([nn.AdaptiveAvgPool2d(output_size=(layer[1], layer[2]))])\n        elif layer_name == \"linear\":\n            if isinstance(input_dim, tuple): input_dim = np.prod(np.array(input_dim))\n            list_to_append_layer_to.extend([nn.Linear(in_features=input_dim, out_features=layer[1])])\n        else:\n            raise ValueError(\"Wrong layer name\")\n        input_dim = self.calculate_new_dimensions(input_dim, layer)\n        return input_dim\n\n    def calculate_new_dimensions(self, input_dim, layer):\n        \"\"\"Calculates the new dimensions of the data after passing through a type of layer\"\"\"\n        layer_name = layer[0].lower()\n        if layer_name == \"conv\":\n            new_channels = layer[1]\n            kernel, stride, padding = layer[2], layer[3], layer[4]\n            new_height = int((input_dim[1] - kernel + 2*padding)/stride) + 1\n            new_width = int((input_dim[2] - kernel + 2 * padding) / stride) + 1\n            output_dim = (new_channels, new_height, new_width)\n        elif layer_name in [\"maxpool\", \"avgpool\"]:\n            new_channels = input_dim[0]\n            kernel, stride, padding = layer[1], layer[2], layer[3]\n            new_height = int((input_dim[1] - kernel + 2*padding)/stride) + 1\n            new_width = int((input_dim[2] - kernel + 2 * padding) / stride) + 1\n            output_dim = (new_channels, new_height, new_width)\n        elif layer_name in [\"adaptivemaxpool\", \"adaptiveavgpool\"]:\n            new_channels = input_dim[0]\n            output_dim = (new_channels, layer[1], layer[2])\n        elif layer_name == \"linear\":\n            output_dim = layer[1]\n        return output_dim\n\n    def create_output_layers(self):\n        \"\"\"Creates the output layers in the network\"\"\"\n        output_layers = nn.ModuleList([])\n        input_dim = self.input_dim_into_final_layer\n        if not isinstance(self.layers_info[-1][0], list)  : self.layers_info[-1] = [self.layers_info[-1]]\n        for output_layer in self.layers_info[-1]:\n            self.create_and_append_layer(input_dim, output_layer, output_layers)\n        return output_layers\n\n    def initialise_all_parameters(self):\n        \"\"\"Initialises the parameters in the linear and embedding layers\"\"\"\n        initialisable_layers = [layer for layer in self.hidden_layers if not type(layer) in self.valid_layer_types_with_no_parameters]\n        self.initialise_parameters(nn.ModuleList(initialisable_layers))\n        output_initialisable_layers = [layer for layer in self.output_layers if\n                                not type(layer) in self.valid_layer_types_with_no_parameters]\n        self.initialise_parameters(output_initialisable_layers)\n\n    def create_batch_norm_layers(self):\n        \"\"\"Creates the batch norm layers in the network\"\"\"\n        batch_norm_layers = nn.ModuleList([])\n        for layer in self.layers_info[:-1]:\n            layer_type = layer[0].lower()\n            if layer_type == \"conv\":\n                batch_norm_layers.extend([nn.BatchNorm2d(num_features=layer[1])])\n            elif layer_type == \"linear\":\n                batch_norm_layers.extend([nn.BatchNorm1d(num_features=layer[1])])\n        return batch_norm_layers\n\n    def forward(self, x):\n        \"\"\"Forward pass for the network. Note that it expects input data in the form (Batch, Channels, Height, Width)\"\"\"\n        if not self.checked_forward_input_data_once: self.check_input_data_into_forward_once(x)\n        x = self.process_hidden_layers(x)\n        out = self.process_output_layers(x)\n        if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*nn.Sigmoid()(out)\n        return out\n\n    def check_input_data_into_forward_once(self, x):\n        \"\"\"Checks the input data into forward is of the right format. Then sets a flag indicating that this has happened once\n        so that we don't keep checking as this would slow down the model too much\"\"\"\n        assert len(x.shape) == 4, \"x should have the shape (batch_size, channel, height, width)\"\n        assert x.shape[1:] == self.input_dim, \"Input data must be of shape (channels, height, width) that you provided, not of shape {}\".format(x.shape[1:])\n        self.checked_forward_input_data_once = True #So that it doesn't check again\n\n    def process_hidden_layers(self, x):\n        \"\"\"Puts the data x through all the hidden layers\"\"\"\n        flattened=False\n        valid_batch_norm_layer_ix = 0\n        for layer_ix, layer in enumerate(self.hidden_layers):\n            if type(layer) in self.valid_layer_types_with_no_parameters:\n                x = layer(x)\n            else:\n                if type(layer) == nn.Linear and not flattened:\n                    x = self.flatten_tensor(x)\n                    flattened = True\n                x = self.get_activation(self.hidden_activations, layer_ix)(layer(x))\n                if self.batch_norm:\n                    x = self.batch_norm_layers[valid_batch_norm_layer_ix](x)\n                    valid_batch_norm_layer_ix += 1\n                if self.dropout != 0.0: x = self.dropout_layer(x)\n        if not flattened: x = self.flatten_tensor(x)\n        return x\n\n    def process_output_layers(self, x):\n        \"\"\"Puts the data x through all the output layers\"\"\"\n        out = None\n        for output_layer_ix, output_layer in enumerate(self.output_layers):\n            activation = self.get_activation(self.output_activation, output_layer_ix)\n            temp_output = output_layer(x)\n            if activation is not None: temp_output = activation(temp_output)\n            if out is None: out = temp_output\n            else: out = torch.cat((out, temp_output), dim=1)\n        return out"
  },
  {
    "path": "nn_builder/pytorch/NN.py",
    "content": "import torch\nimport numpy as np\nimport torch.nn as nn\nfrom nn_builder.pytorch.Base_Network import Base_Network\n\nclass NN(nn.Module, Base_Network):\n    \"\"\"Creates a PyTorch neural network\n    Args:\n        - input_dim: Integer to indicate the dimension of the input into the network\n        - layers_info: List of integers to indicate the width and number of linear layers you want in your network,\n                      e.g. [5, 8, 1] would produce a network with 3 linear layers of width 5, 8 and then 1\n        - hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers\n                              (not including the output layer). Default is ReLU.\n        - output_activation: String to indicate the activation function you want the output to go through. Provide a list of\n                             strings if you want multiple output heads\n        - dropout: Float to indicate what dropout probability you want applied after each hidden layer\n        - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch\n                       initialisers are supported. PyTorch's default initialisation is the default.\n        - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False\n        - columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer\n                                          before being fed through the other layers of the network. Default option is no embeddings\n        - embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then\n                                you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1],\n                                [embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings\n        - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the\n                   output values to in regression tasks. Default is no range restriction\n        - random_seed: Integer to indicate the random seed you want to use\n    \"\"\"\n    def __init__(self, input_dim, layers_info, output_activation=None,\n                 hidden_activations=\"relu\", dropout=0.0, initialiser=\"default\", batch_norm=False,\n                 columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (), random_seed=0):\n        nn.Module.__init__(self)\n        self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0\n        self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded\n        self.embedding_dimensions = embedding_dimensions\n        self.embedding_layers = self.create_embedding_layers()\n        Base_Network.__init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser,\n                              batch_norm, y_range, random_seed)\n\n    def check_all_user_inputs_valid(self):\n        \"\"\"Checks that all the user inputs were valid\"\"\"\n        self.check_NN_input_dim_valid()\n        self.check_NN_layers_valid()\n        self.check_activations_valid()\n        self.check_embedding_dimensions_valid()\n        self.check_initialiser_valid()\n        self.check_y_range_values_valid()\n\n    def create_hidden_layers(self):\n        \"\"\"Creates the linear layers in the network\"\"\"\n        linear_layers = nn.ModuleList([])\n        input_dim = int(self.input_dim - len(self.embedding_dimensions) + np.sum([output_dims[1] for output_dims in self.embedding_dimensions]))\n        for hidden_unit in self.layers_info[:-1]:\n            linear_layers.extend([nn.Linear(input_dim, hidden_unit)])\n            input_dim = hidden_unit\n        return linear_layers\n\n    def create_output_layers(self):\n        \"\"\"Creates the output layers in the network\"\"\"\n        output_layers = nn.ModuleList([])\n        if len(self.layers_info) >= 2: input_dim = self.layers_info[-2]\n        else: input_dim = self.input_dim\n        if not isinstance(self.layers_info[-1], list): output_layer = [self.layers_info[-1]]\n        else: output_layer = self.layers_info[-1]\n        for output_dim in output_layer:\n            output_layers.extend([nn.Linear(input_dim, output_dim)])\n        return output_layers\n\n    def create_batch_norm_layers(self):\n        \"\"\"Creates the batch norm layers in the network\"\"\"\n        batch_norm_layers = nn.ModuleList([nn.BatchNorm1d(num_features=hidden_unit) for hidden_unit in self.layers_info[:-1]])\n        return batch_norm_layers\n\n    def initialise_all_parameters(self):\n        \"\"\"Initialises the parameters in the linear and embedding layers\"\"\"\n        self.initialise_parameters(self.hidden_layers)\n        self.initialise_parameters(self.output_layers)\n        self.initialise_parameters(self.embedding_layers)\n\n    def forward(self, x):\n        \"\"\"Forward pass for the network\"\"\"\n        if not self.checked_forward_input_data_once: self.check_input_data_into_forward_once(x)\n        if self.embedding_to_occur: x = self.incorporate_embeddings(x)\n        x = self.process_hidden_layers(x)\n        out = self.process_output_layers(x)\n        if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*nn.Sigmoid()(out)\n        return out\n\n    def check_input_data_into_forward_once(self, x):\n        \"\"\"Checks the input data into forward is of the right format. Then sets a flag indicating that this has happened once\n        so that we don't keep checking as this would slow down the model too much\"\"\"\n        for embedding_dim in self.columns_of_data_to_be_embedded:\n            data = x[:, embedding_dim]\n            data_long = data.long()\n            assert all(data_long >= 0), \"All data to be embedded must be integers 0 and above -- {}\".format(data_long)\n            assert torch.sum(abs(data.float() - data_long.float())) < 0.0001, \"\"\"Data columns to be embedded should be integer \n                                                                                values 0 and above to represent the different \n                                                                                classes\"\"\"\n        if self.input_dim > len(self.columns_of_data_to_be_embedded):\n          assert isinstance(x, torch.FloatTensor) or isinstance(x, torch.cuda.FloatTensor), \"Input data must be a float tensor\"\n        assert len(x.shape) == 2, \"X should be a 2-dimensional tensor: {}\".format(x.shape)\n        self.checked_forward_input_data_once = True #So that it doesn't check again\n\n    def incorporate_embeddings(self, x):\n        \"\"\"Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready\n        to then be put through the linear layers\"\"\"\n        all_embedded_data = []\n        for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded):\n            data = x[:, embedding_var].long()\n            embedded_data = self.embedding_layers[embedding_layer_ix](data)\n            all_embedded_data.append(embedded_data)\n        all_embedded_data = torch.cat(tuple(all_embedded_data), dim=1)\n        x = torch.cat((x[:, [col for col in range(x.shape[1]) if col not in self.columns_of_data_to_be_embedded]].float(), all_embedded_data), dim=1)\n        return x\n\n    def process_hidden_layers(self, x):\n        \"\"\"Puts the data x through all the hidden layers\"\"\"\n        for layer_ix, linear_layer in enumerate(self.hidden_layers):\n            x = self.get_activation(self.hidden_activations, layer_ix)(linear_layer(x))\n            if self.batch_norm: x = self.batch_norm_layers[layer_ix](x)\n            if self.dropout != 0.0: x = self.dropout_layer(x)\n        return x\n\n    def process_output_layers(self, x):\n        \"\"\"Puts the data x through all the output layers\"\"\"\n        out = None\n        for output_layer_ix, output_layer in enumerate(self.output_layers):\n            activation = self.get_activation(self.output_activation, output_layer_ix)\n            temp_output = output_layer(x)\n            if activation is not None: temp_output = activation(temp_output)\n            if out is None: out = temp_output\n            else: out = torch.cat((out, temp_output), dim=1)\n        return out"
  },
  {
    "path": "nn_builder/pytorch/RNN.py",
    "content": "import torch\nimport torch.nn as nn\nimport numpy as np\nfrom nn_builder.pytorch.Base_Network import Base_Network\n\nclass RNN(nn.Module, Base_Network):\n    \"\"\"Creates a PyTorch recurrent neural network\n    Args:\n        - input_dim: Integer to indicate the dimension of the input into the network\n        - layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be\n                         one of these 3 forms:\n                         - [\"lstm\", hidden_units]\n                         - [\"gru\", hidden_units]\n                         - [\"linear\", hidden_units]\n        - hidden_activations: String or list of string to indicate the activations you want used on the output of linear hidden layers\n                              (not including the output layer). Default is ReLU.\n        - output_activation: String to indicate the activation function you want the output to go through. Provide a list of\n                             strings if you want multiple output heads\n        - dropout: Float to indicate what dropout probability you want applied after each hidden layer\n        - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch\n                       initialisers are supported. PyTorch's default initialisation is the default.\n        - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False\n        - columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer\n                                          before being fed through the other layers of the network. Default option is no embeddings\n        - embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then\n                                you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1],\n                                [embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings\n        - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the\n                   output values to in regression tasks. Default is no range restriction\n        - return_final_seq_only: Boolean to indicate whether you only want to return the output for the final timestep (True)\n                                 or if you want to return the output for all timesteps (False)\n        - random_seed: Integer to indicate the random seed you want to use\n\n    NOTE that this class' forward method expects input data in the form: (batch, sequence length, features)\n    \"\"\"\n\n    def __init__(self, input_dim, layers_info, output_activation=None,\n                 hidden_activations=\"relu\", dropout=0.0, initialiser=\"default\", batch_norm=False,\n                 columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (),\n                 return_final_seq_only=True, random_seed=0):\n        nn.Module.__init__(self)\n        self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0\n        self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded\n        self.embedding_dimensions = embedding_dimensions\n        self.embedding_layers = self.create_embedding_layers()\n        self.return_final_seq_only = return_final_seq_only\n        self.valid_RNN_hidden_layer_types = {\"linear\", \"gru\", \"lstm\"}\n        Base_Network.__init__(self, input_dim, layers_info, output_activation,\n                              hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed)\n\n    def check_all_user_inputs_valid(self):\n        \"\"\"Checks that all the user inputs were valid\"\"\"\n        self.check_NN_input_dim_valid()\n        self.check_RNN_layers_valid()\n        self.check_activations_valid()\n        self.check_embedding_dimensions_valid()\n        self.check_initialiser_valid()\n        self.check_y_range_values_valid()\n        self.check_return_final_seq_only_valid()\n\n    def check_RNN_layers_valid(self):\n        \"\"\"Checks that layers provided by user are valid\"\"\"\n        error_msg_layer_type = \"First element in a layer specification must be one of {}\".format(self.valid_RNN_hidden_layer_types)\n        error_msg_layer_form = \"Layer must be of form [layer_name, hidden_units]\"\n        error_msg_layer_list = \"Layers must be provided as a list\"\n        error_msg_output_heads = \"Number of output activations must equal number of output heads\"\n\n        assert isinstance(self.layers_info, list), error_msg_layer_list\n\n        all_layers = self.layers_info[:-1]\n        output_layer = self.layers_info[-1]\n        assert isinstance(output_layer, list), error_msg_layer_list\n        if isinstance(output_layer[0], list):\n            assert len(output_layer) == len(\n                self.output_activation), error_msg_output_heads\n            for layer in output_layer:\n                all_layers.append(layer)\n        else:\n            assert not isinstance(self.output_activation, list) or len(\n                self.output_activation) == 1, error_msg_output_heads\n            all_layers.append(output_layer)\n\n        rest_must_be_linear = False\n        for layer in all_layers:\n            assert isinstance(layer, list), \"Each layer must be a list\"\n            assert isinstance(layer[0], str), error_msg_layer_type\n            layer_type_name = layer[0].lower()\n            assert layer_type_name in self.valid_RNN_hidden_layer_types, \"Layer name {} not valid, use one of {}\".format(\n                layer_type_name, self.valid_RNN_hidden_layer_types)\n\n            assert isinstance(layer[1], int), error_msg_layer_form\n            assert layer[1] > 0, \"Must have hidden_units >= 1\"\n            assert len(layer) == 2, error_msg_layer_form\n\n            if rest_must_be_linear: assert layer[0].lower() == \"linear\", \"If have linear layers then they must come at end\"\n            if layer_type_name == \"linear\": rest_must_be_linear = True\n\n    def create_hidden_layers(self):\n        \"\"\"Creates the hidden layers in the network\"\"\"\n        RNN_hidden_layers = nn.ModuleList([])\n        input_dim = int(self.input_dim - len(self.embedding_dimensions) + np.sum(\n            [output_dims[1] for output_dims in self.embedding_dimensions]))\n        for layer in self.layers_info[:-1]:\n            input_dim = self.create_and_append_layer(input_dim, layer, RNN_hidden_layers)\n        self.input_dim_into_final_layer = input_dim\n        return RNN_hidden_layers\n\n    def create_and_append_layer(self, input_dim, layer, RNN_hidden_layers):\n        layer_type_name = layer[0].lower()\n        hidden_size = layer[1]\n        if layer_type_name == \"lstm\":\n            RNN_hidden_layers.extend([nn.LSTM(input_size=input_dim, hidden_size=hidden_size, batch_first=True)])\n        elif layer_type_name == \"gru\":\n            RNN_hidden_layers.extend(\n                [nn.GRU(input_size=input_dim, hidden_size=hidden_size, batch_first=True)])\n        elif layer_type_name == \"linear\":\n            RNN_hidden_layers.extend([nn.Linear(input_dim, hidden_size)])\n        else:\n            raise ValueError(\"Wrong layer names\")\n        input_dim = hidden_size\n        return input_dim\n\n    def create_output_layers(self):\n        \"\"\"Creates the output layers in the network\"\"\"\n        output_layers = nn.ModuleList([])\n        input_dim = self.input_dim_into_final_layer\n        if not isinstance(self.layers_info[-1][0], list): self.layers_info[-1] = [self.layers_info[-1]]\n        for output_layer in self.layers_info[-1]:\n            self.create_and_append_layer(input_dim, output_layer, output_layers)\n        return output_layers\n\n    def initialise_all_parameters(self):\n        \"\"\"Initialises the parameters in the linear and embedding layers\"\"\"\n        self.initialise_parameters(self.hidden_layers)\n        self.initialise_parameters(self.output_layers)\n        self.initialise_parameters(self.embedding_layers)\n\n    def create_batch_norm_layers(self):\n        \"\"\"Creates the batch norm layers in the network\"\"\"\n        batch_norm_layers = nn.ModuleList([nn.BatchNorm1d(num_features=layer[1]) for layer in self.layers_info[:-1]])\n        return batch_norm_layers\n\n    def get_activation(self, activations, ix=None):\n        \"\"\"Gets the activation function\"\"\"\n        if isinstance(activations, list):\n            activation = self.str_to_activations_converter[str(activations[ix]).lower()]\n        else:\n            activation = self.str_to_activations_converter[str(activations).lower()]\n        return activation\n\n    def forward(self, x):\n        \"\"\"Forward pass for the network. Note that it expects input data in the form (batch, seq length, features)\"\"\"\n        if not self.checked_forward_input_data_once: self.check_input_data_into_forward_once(x)\n        batch_size, seq_length, data_dimension = x.shape\n        if self.embedding_to_occur: x = self.incorporate_embeddings(x, batch_size, seq_length)\n        x = self.process_hidden_layers(x, batch_size, seq_length)\n        out = self.process_output_layers(x, batch_size, seq_length)\n        if self.return_final_seq_only: out = out[:, -1, :]\n        if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*nn.Sigmoid()(out)\n        return out\n\n    def check_input_data_into_forward_once(self, x):\n        \"\"\"Checks the input data into forward is of the right format. Then sets a flag indicating that this has happened once\n        so that we don't keep checking as this would slow down the model too much\"\"\"\n        assert len(x.shape) == 3, \"x should have the shape (batch_size, sequence_length, dimension)\"\n        assert x.shape[2] == self.input_dim, \"x must have the same dimension as the input_dim you provided\"\n        for embedding_dim in self.columns_of_data_to_be_embedded:\n            data = x[:, :, embedding_dim]\n            data = data.contiguous().view(-1, 1)\n            data_long = data.long()\n            assert all(data_long >= 0), \"All data to be embedded must be integers 0 and above -- {}\".format(data_long)\n            assert torch.sum(abs(data.float() - data_long.float())) < 0.0001, \"\"\"Data columns to be embedded should be integer \n                                                                                values 0 and above to represent the different \n                                                                                classes\"\"\"\n        if self.input_dim > len(self.columns_of_data_to_be_embedded):\n          assert isinstance(x, torch.FloatTensor) or isinstance(x, torch.cuda.FloatTensor), \"Input data must be a float tensor\"\n        self.checked_forward_input_data_once = True #So that it doesn't check again\n\n    def incorporate_embeddings(self, x, batch_size, seq_length):\n        \"\"\"Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready\n        to then be put through the hidden layers\"\"\"\n        all_embedded_data = []\n        for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded):\n            data = x[:, :, embedding_var].long()\n            data = data.contiguous().view(batch_size * seq_length, -1)\n            embedded_data = self.embedding_layers[embedding_layer_ix](data)\n            embedded_data = embedded_data.view(batch_size, seq_length, -1)\n            all_embedded_data.append(embedded_data)\n        all_embedded_data = torch.cat(tuple(all_embedded_data), dim=2)\n        non_embedded_data = x[:, :, [col for col in range(x.shape[2]) if col not in self.columns_of_data_to_be_embedded]].float()\n        x = torch.cat((non_embedded_data, all_embedded_data), dim=2)\n        return x\n\n    def process_hidden_layers(self, x, batch_size, seq_length):\n        \"\"\"Puts the data x through all the hidden layers\"\"\"\n        for layer_ix, layer in enumerate(self.hidden_layers):\n            if type(layer) == nn.Linear:\n                x = x.contiguous().view(batch_size * seq_length, -1)\n                activation = self.get_activation(self.hidden_activations, layer_ix)\n                x = activation(layer(x))\n                x = x.view(batch_size, seq_length, layer.out_features)\n            else:\n                x = layer(x)\n                x = x[0] #because we only want to keep the output and not the hidden states\n            if self.batch_norm:\n                x.transpose_(1, 2)\n                x = self.batch_norm_layers[layer_ix](x)\n                x.transpose_(1, 2)\n            if self.dropout != 0.0: x = self.dropout_layer(x)\n        return x\n\n    def process_output_layers(self, x, batch_size, seq_length):\n        \"\"\"Puts the data x through all the output layers\"\"\"\n        out = None\n        for output_layer_ix, output_layer in enumerate(self.output_layers):\n            activation = self.get_activation(self.output_activation, output_layer_ix)\n\n            if type(output_layer) == nn.Linear:\n                x = x.contiguous().view(batch_size * seq_length, -1)\n                temp_output = output_layer(x)\n                if activation is not None:\n                    temp_output = activation(temp_output)\n                temp_output = temp_output.view(batch_size, seq_length, -1)\n                x = x.view(batch_size, seq_length, -1)\n            else:\n                temp_output = output_layer(x)\n                temp_output = temp_output[0]\n                if activation is not None:\n                    if type(activation) == nn.Softmax:\n                        temp_output = temp_output.contiguous().view(batch_size * seq_length, -1)\n                        temp_output = activation(temp_output)\n                        temp_output = temp_output.view(batch_size, seq_length, -1)\n                    else:\n                        temp_output = activation(temp_output)\n            if out is None: out = temp_output\n            else: out = torch.cat((out, temp_output), dim=2)\n        return out"
  },
  {
    "path": "nn_builder/pytorch/__init__.py",
    "content": ""
  },
  {
    "path": "nn_builder/tensorflow/Base_Network.py",
    "content": "from tensorflow.keras.layers import BatchNormalization\nfrom nn_builder.Overall_Base_Network import Overall_Base_Network\nimport tensorflow.keras.activations as activations\nimport tensorflow.keras.initializers as initializers\nimport numpy as np\nimport random\nimport tensorflow as tf\nfrom abc import ABC, abstractmethod\n\nclass Base_Network(Overall_Base_Network, ABC):\n    \"\"\"Base class for TensorFlow neural network classes\"\"\"\n    def __init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range,\n                 random_seed, input_dim):\n        if input_dim is not None: print(\"You don't need to provide input_dim for a tensorflow network\")\n        super().__init__(None, layers_info, output_activation,\n                 hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed)\n\n    @abstractmethod\n    def call(self, x, training=True):\n        \"\"\"Runs a forward pass of the tensorflow model\"\"\"\n        raise NotImplementedError\n\n    @abstractmethod\n    def create_and_append_layer(self, layer, list_to_append_layer_to, activation, output_layer=False):\n        \"\"\"Creates a layer and appends it to the provided list\"\"\"\n        raise NotImplementedError\n\n    def set_all_random_seeds(self, random_seed):\n        \"\"\"Sets all possible random seeds so results can be reproduced\"\"\"\n        np.random.seed(random_seed)\n        tf.random.set_seed(random_seed)\n        random.seed(random_seed)\n\n    def create_str_to_activations_converter(self):\n        \"\"\"Creates a dictionary which converts strings to activations\"\"\"\n        str_to_activations_converter = {\"elu\": activations.elu, \"exponential\": activations.exponential,\n                                        \"hard_sigmoid\": activations.hard_sigmoid, \"linear\": activations.linear,\n                                        \"relu\": activations.relu, \"selu\": activations.selu, \"sigmoid\": activations.sigmoid,\n                                        \"softmax\": activations.softmax, \"softplus\": activations.softplus,\n                                        \"softsign\": activations.softsign, \"tanh\": activations.tanh, \"none\": activations.linear}\n        return str_to_activations_converter\n\n    def create_str_to_initialiser_converter(self):\n        \"\"\"Creates a dictionary which converts strings to initialiser\"\"\"\n        str_to_initialiser_converter = {\"glorot_normal\": initializers.glorot_normal, \"glorot_uniform\": initializers.glorot_uniform,\n                                        \"xavier_normal\": initializers.glorot_normal, \"xavier_uniform\": initializers.glorot_uniform,\n                                        \"xavier\": initializers.glorot_uniform, \"he_normal\": initializers.he_normal(),\n                                        \"he_uniform\": initializers.he_uniform(), \"lecun_normal\": initializers.lecun_normal(),\n                                        \"lecun_uniform\": initializers.lecun_uniform(), \"truncated_normal\": initializers.TruncatedNormal,\n                                        \"variance_scaling\": initializers.VarianceScaling, \"default\": initializers.glorot_uniform}\n        return str_to_initialiser_converter\n\n    def create_dropout_layer(self):\n        \"\"\"Creates a dropout layer\"\"\"\n        return tf.keras.layers.Dropout(rate=self.dropout)\n\n    def create_hidden_layers(self):\n        \"\"\"Creates the hidden layers in the network\"\"\"\n        hidden_layers = []\n        for layer_ix, layer in enumerate(self.layers_info[:-1]):\n            activation = self.get_activation(self.hidden_activations, layer_ix)\n            self.create_and_append_layer(layer, hidden_layers, activation, output_layer=False)\n        return hidden_layers\n\n    def create_output_layers(self):\n        \"\"\"Creates the output layers in the network\"\"\"\n        output_layers = []\n        network_type = type(self).__name__\n        if network_type in [\"CNN\", \"RNN\"]:\n            if not isinstance(self.layers_info[-1][0], list): self.layers_info[-1] = [self.layers_info[-1]]\n        elif network_type == \"NN\":\n            if isinstance(self.layers_info[-1], int): self.layers_info[-1] = [self.layers_info[-1]]\n        else:\n            raise ValueError(\"Network type not recognised\")\n        for output_layer_ix, output_layer in enumerate(self.layers_info[-1]):\n            activation = self.get_activation(self.output_activation, output_layer_ix)\n            self.create_and_append_layer(output_layer, output_layers, activation, output_layer=True)\n        return output_layers\n\n    def create_embedding_layers(self):\n        \"\"\"Creates the embedding layers in the network\"\"\"\n        embedding_layers = []\n        for embedding_dimension in self.embedding_dimensions:\n            input_dim, output_dim = embedding_dimension\n            embedding_layers.extend([tf.keras.layers.Embedding(input_dim, output_dim)])\n        return embedding_layers\n\n    def create_batch_norm_layers(self):\n        \"\"\"Creates the batch norm layers in the network\"\"\"\n        batch_norm_layers = []\n        for layer in self.layers_info[:-1]:\n            batch_norm_layers.extend([BatchNormalization()])\n        return batch_norm_layers\n\n    def print_model_summary(self, input_shape=None):\n        assert input_shape is not None, \"Must provide the input_shape parameter as a tuple\"\n        self.build(input_shape=input_shape)\n        self.dropout_layer.build(input_shape)\n        self.summary()"
  },
  {
    "path": "nn_builder/tensorflow/CNN.py",
    "content": "import numpy as np\nfrom tensorflow.keras import Model, activations\nfrom tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, BatchNormalization, MaxPool2D, AveragePooling2D\nfrom nn_builder.tensorflow.Base_Network import Base_Network\nimport tensorflow as tf\n\nclass CNN(Model, Base_Network):\n    \"\"\"Creates a PyTorch convolutional neural network\n    Args:\n        - layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be\n                         one of these 4 forms:\n                         - [\"conv\", channels, kernel_size, stride, padding]\n                         - [\"maxpool\", kernel_size, stride, padding]\n                         - [\"avgpool\", kernel_size, stride, padding]\n                         - [\"linear\", out]\n                        where all variables are integers except for padding which must be either \"same\" or \"valid\"\n        - output_activation: String to indicate the activation function you want the output to go through. Provide a list of\n                             strings if you want multiple output heads\n        - hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers\n                              (not including the output layer). Default is ReLU.\n        - dropout: Float to indicate what dropout probability you want applied after each hidden layer\n        - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch\n                       initialisers are supported. PyTorch's default initialisation is the default.\n        - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False\n        - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the\n                   output values to in regression tasks. Default is no range restriction\n        - random_seed: Integer to indicate the random seed you want to use\n\n    NOTE that this class' call method expects input data in the form: (batch, channels, height, width)\n    \"\"\"\n    def __init__(self, layers_info, output_activation=None, hidden_activations=\"relu\", dropout= 0.0, initialiser=\"default\",\n                 batch_norm=False, y_range=(), random_seed=0, input_dim=None):\n        Model.__init__(self)\n        self.valid_cnn_hidden_layer_types = {'conv', 'maxpool', 'avgpool', 'linear'}\n        self.valid_layer_types_with_no_parameters = (MaxPool2D, AveragePooling2D)\n        Base_Network.__init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser,\n                              batch_norm, y_range, random_seed, input_dim)\n\n    def check_all_user_inputs_valid(self):\n        \"\"\"Checks that all the user inputs were valid\"\"\"\n        self.check_CNN_layers_valid()\n        self.check_activations_valid()\n        self.check_initialiser_valid()\n        self.check_y_range_values_valid()\n\n    def check_CNN_layers_valid(self):\n        \"\"\"Checks that the user inputs for cnn_hidden_layers were valid. cnn_hidden_layers must be a list of layers where\n        each layer must be of one of these forms:\n        - [\"conv\", channels, kernel_size, stride, padding]\n        - [\"maxpool\", kernel_size, stride, padding]\n        - [\"avgpool\", kernel_size, stride, padding]\n        - [\"linear\", out]\n\n        where all variables must be integers except padding which must be \"valid\" or \"same\"\n        \"\"\"\n        error_msg_layer_type = \"First element in a layer specification must be one of {}\".format(self.valid_cnn_hidden_layer_types)\n        error_msg_conv_layer = \"\"\"Conv layer must be of form ['conv', channels, kernel_size, stride, padding] where the \n                               variables are all non-negative integers except padding which must be either \"valid\" or \"same\"\"\"\n        error_msg_maxpool_layer = \"\"\"Maxpool layer must be of form ['maxpool', kernel_size, stride, padding] where the \n                               variables are all non-negative integers except padding which must be either \"valid\" or \"same\"\"\"\n        error_msg_avgpool_layer = \"\"\"Avgpool layer must be of form ['avgpool', kernel_size, stride, padding] where the \n                               variables are all non-negative integers except padding which must be either \"valid\" or \"same\"\"\"\n        error_msg_linear_layer = \"\"\"Linear layer must be of form ['linear', out] where out is a non-negative integers\"\"\"\n        assert isinstance(self.layers_info, list), \"layers must be a list\"\n\n        all_layers = self.layers_info[:-1]\n        output_layer = self.layers_info[-1]\n        assert isinstance(output_layer, list), \"layers must be a list\"\n        if isinstance(output_layer[0], list):\n            assert len(output_layer) == len(self.output_activation), \"Number of output activations must equal number of output heads\"\n            for layer in output_layer:\n                all_layers.append(layer)\n                assert isinstance(layer[0], str), error_msg_layer_type\n                assert layer[0].lower() == \"linear\", \"Final layer must be linear\"\n        else:\n            all_layers.append(output_layer)\n            assert isinstance(output_layer[0], str), error_msg_layer_type\n            assert output_layer[0].lower() == \"linear\", \"Final layer must be linear\"\n\n        for layer in all_layers:\n            assert isinstance(layer, list), \"Each layer must be a list\"\n            assert isinstance(layer[0], str), error_msg_layer_type\n            layer_type_name = layer[0].lower()\n            assert layer_type_name in self.valid_cnn_hidden_layer_types, \"Layer name {} not valid, use one of {}\".format(layer_type_name, self.valid_cnn_hidden_layer_types)\n            if layer_type_name == \"conv\":\n                assert len(layer) == 5, error_msg_conv_layer\n                for ix in range(3): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0, error_msg_conv_layer\n                assert isinstance(layer[4], str) and layer[4].lower() in [\"valid\", \"same\"], error_msg_conv_layer\n            elif layer_type_name == \"maxpool\":\n                assert len(layer) == 4, error_msg_maxpool_layer\n                for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_maxpool_layer\n                if layer[1] != layer[2]: print(\"NOTE that your maxpool kernel size {} isn't the same as your stride {}\".format(layer[1], layer[2]))\n                assert isinstance(layer[3], str) and layer[3].lower() in [\"valid\", \"same\"], error_msg_maxpool_layer\n            elif layer_type_name == \"avgpool\":\n                assert len(layer) == 4, error_msg_avgpool_layer\n                for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_avgpool_layer\n                assert isinstance(layer[3], str) and layer[3].lower() in [\"valid\", \"same\"], error_msg_avgpool_layer\n                if layer[1] != layer[2]:print(\"NOTE that your avgpool kernel size {} isn't the same as your stride {}\".format(layer[1], layer[2]))\n            elif layer_type_name == \"linear\":\n                assert len(layer) == 2, error_msg_linear_layer\n                for ix in range(1): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0\n            else:\n                raise ValueError(\"Invalid layer name\")\n\n        rest_must_be_linear = False\n        for ix, layer in enumerate(all_layers):\n            if rest_must_be_linear: assert layer[0].lower() == \"linear\", \"If have linear layers then they must come at end\"\n            if layer[0].lower() == \"linear\":\n                rest_must_be_linear = True\n\n    def create_and_append_layer(self, layer, list_to_append_layer_to, activation=None, output_layer=False):\n        \"\"\"Creates and appends a layer to the list provided\"\"\"\n        layer_name = layer[0].lower()\n        assert layer_name in self.valid_cnn_hidden_layer_types, \"Layer name {} not valid, use one of {}\".format(\n            layer_name, self.valid_cnn_hidden_layer_types)\n        if layer_name == \"conv\":\n            list_to_append_layer_to.extend([Conv2D(filters=layer[1], kernel_size=layer[2],\n                                                strides=layer[3], padding=layer[4], activation=activation,\n                                                   kernel_initializer=self.initialiser_function)])\n        elif layer_name == \"maxpool\":\n            list_to_append_layer_to.extend([MaxPool2D(pool_size=(layer[1], layer[1]),\n                                                   strides=(layer[2], layer[2]), padding=layer[3])])\n        elif layer_name == \"avgpool\":\n            list_to_append_layer_to.extend([AveragePooling2D(pool_size=(layer[1], layer[1]),\n                                                   strides=(layer[2], layer[2]), padding=layer[3])])\n        elif layer_name == \"linear\":\n            list_to_append_layer_to.extend([Dense(layer[1], activation=activation, kernel_initializer=self.initialiser_function)])\n        else:\n            raise ValueError(\"Wrong layer name\")\n\n    def create_batch_norm_layers(self):\n        \"\"\"Creates the batch norm layers in the network\"\"\"\n        batch_norm_layers = []\n        for layer in self.layers_info[:-1]:\n            layer_type = layer[0].lower()\n            if layer_type in [\"conv\", \"linear\"]:\n                batch_norm_layers.extend([BatchNormalization()])\n        return batch_norm_layers\n\n    def call(self, x, training=True):\n        \"\"\"Forward pass for the network. Note that it expects input data in the form (Batch, Height, Width, Channels)\"\"\"\n        x = self.process_hidden_layers(x, training)\n        out = self.process_output_layers(x)\n        if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0]) * activations.sigmoid(out)\n        return out\n\n    def process_hidden_layers(self, x, training):\n        \"\"\"Puts the data x through all the hidden layers\"\"\"\n        flattened=False\n        training = training or training is None\n        valid_batch_norm_layer_ix = 0\n        for layer_ix, layer in enumerate(self.hidden_layers):\n            if type(layer) in self.valid_layer_types_with_no_parameters:\n                x = layer(x)\n            else:\n                if type(layer) == Dense and not flattened:\n                    x = Flatten()(x)\n                    flattened = True\n                x = layer(x)\n                if self.batch_norm:\n                    x = self.batch_norm_layers[valid_batch_norm_layer_ix](x, training=False)\n                    valid_batch_norm_layer_ix += 1\n                if self.dropout != 0.0 and training: x = self.dropout_layer(x)\n        if not flattened: x = Flatten()(x)\n        return x\n\n    def process_output_layers(self, x):\n        \"\"\"Puts the data x through all the output layers\"\"\"\n        out = None\n        for output_layer_ix, output_layer in enumerate(self.output_layers):\n            temp_output = output_layer(x)\n            if out is None: out = temp_output\n            else: out = Concatenate(axis=1)([out, temp_output])\n        return out\n"
  },
  {
    "path": "nn_builder/tensorflow/NN.py",
    "content": "import tensorflow as tf\nfrom tensorflow.keras import Model, activations\nimport numpy as np\nfrom tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, BatchNormalization\nfrom nn_builder.tensorflow.Base_Network import Base_Network\n\n\nclass NN(Model, Base_Network):\n    \"\"\"Creates a PyTorch neural network\n    Args:\n        - layers_info: List of integers to indicate the width and number of linear layers you want in your network\n        - hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers\n                              (not including the output layer). Default is ReLU.\n        - output_activation: String to indicate the activation function you want the output to go through. Provide a list of\n                             strings if you want multiple output heads\n        - dropout: Float to indicate what dropout probability you want applied after each hidden layer\n        - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch\n                       initialisers are supported. PyTorch's default initialisation is the default.\n        - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False\n        - columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer\n                                          before being fed through the other layers of the network. Default option is no embeddings\n        - embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then\n                                you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1],\n                                [embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings\n        - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the\n                   output values to in regression tasks. Default is no range restriction\n        - random_seed: Integer to indicate the random seed you want to use\n    \"\"\"\n    def __init__(self, layers_info, output_activation=None, hidden_activations=\"relu\", dropout=0.0, initialiser=\"default\",\n                 batch_norm=False, columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (), random_seed=0,\n                 input_dim=None):\n        Model.__init__(self)\n        self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0\n        self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded\n        self.embedding_dimensions = embedding_dimensions\n        self.embedding_layers = self.create_embedding_layers()\n        Base_Network.__init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser,\n                              batch_norm, y_range, random_seed, input_dim)\n\n    def check_all_user_inputs_valid(self):\n        \"\"\"Checks that all the user inputs were valid\"\"\"\n        self.check_NN_layers_valid()\n        self.check_activations_valid()\n        self.check_embedding_dimensions_valid()\n        self.check_initialiser_valid()\n        self.check_y_range_values_valid()\n\n    def create_and_append_layer(self, layer, list_to_append_layer_to, activation=None, output_layer=False):\n        \"\"\"Creates and appends a layer to the list provided\"\"\"\n        list_to_append_layer_to.extend([Dense(layer, activation=activation, kernel_initializer=self.initialiser_function)])\n\n    def call(self, x, training=True):\n        if self.embedding_to_occur: x = self.incorporate_embeddings(x)\n        x = self.process_hidden_layers(x, training)\n        out = self.process_output_layers(x)\n        if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*activations.sigmoid(out)\n        return out\n\n    def incorporate_embeddings(self, x):\n        \"\"\"Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready\n        to then be put through the hidden layers\"\"\"\n        all_embedded_data = []\n        for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded):\n            data = x[:, embedding_var]\n            embedded_data = self.embedding_layers[embedding_layer_ix](data)\n            all_embedded_data.append(embedded_data)\n        if len(all_embedded_data) > 1: all_embedded_data = Concatenate(axis=1)(all_embedded_data)\n        else: all_embedded_data = all_embedded_data[0]\n        non_embedded_columns = [col for col in range(x.shape[1]) if col not in self.columns_of_data_to_be_embedded]\n        if len(non_embedded_columns) > 0:\n            x = tf.gather(x, non_embedded_columns, axis=1)\n            x = Concatenate(axis=1)([tf.dtypes.cast(x, float), all_embedded_data])\n        else: x = all_embedded_data\n        return x\n\n    def process_hidden_layers(self, x, training):\n        \"\"\"Puts the data x through all the hidden layers\"\"\"\n        for layer_ix, linear_layer in enumerate(self.hidden_layers):\n            x = linear_layer(x)\n            if self.batch_norm: x = self.batch_norm_layers[layer_ix](x, training=False)\n            if self.dropout != 0.0 and (training or training is None):\n                x = self.dropout_layer(x)\n        return x\n\n    def process_output_layers(self, x):\n        \"\"\"Puts the data x through all the output layers\"\"\"\n        out = None\n        for output_layer_ix, output_layer in enumerate(self.output_layers):\n            temp_output = output_layer(x)\n            if out is None: out = temp_output\n            else:\n                out = Concatenate(axis=1)(inputs=[out, temp_output])\n        return out\n\n"
  },
  {
    "path": "nn_builder/tensorflow/RNN.py",
    "content": "import numpy as np\nimport tensorflow as tf\nfrom tensorflow.keras import Model, activations\nfrom tensorflow.keras.layers import Dense, Concatenate, GRU, LSTM\nfrom nn_builder.tensorflow.Base_Network import Base_Network\n\nclass RNN(Model, Base_Network):\n    \"\"\"Creates a TensorFlow recurrent neural network\n    Args:\n        - layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be\n                         one of these 3 forms:\n                         - [\"lstm\", hidden_units]\n                         - [\"gru\", hidden_units]\n                         - [\"linear\", hidden_units]\n        - hidden_activations: String or list of string to indicate the activations you want used on the output of linear hidden layers\n                              (not including the output layer). Default is ReLU.\n        - output_activation: String to indicate the activation function you want the output to go through. Provide a list of\n                             strings if you want multiple output heads\n        - dropout: Float to indicate what dropout probability you want applied after each hidden layer\n        - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch\n                       initialisers are supported. PyTorch's default initialisation is the default.\n        - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False\n        - columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer\n                                          before being fed through the other layers of the network. Default option is no embeddings\n        - embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then\n                                you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1],\n                                [embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings\n        - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the\n                   output values to in regression tasks. Default is no range restriction\n        - return_final_seq_only: Boolean to indicate whether you only want to return the output for the final timestep (True)\n                                 or if you want to return the output for all timesteps (False)\n        - random_seed: Integer to indicate the random seed you want to use\n\n    NOTE that this class' call method expects input data in the form: (batch, sequence length, features)\n    \"\"\"\n    def __init__(self, layers_info, output_activation=None, hidden_activations=\"relu\", dropout=0.0, initialiser=\"default\",\n                 batch_norm=False, columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (),\n                 return_final_seq_only=True, random_seed=0, input_dim=None):\n        Model.__init__(self)\n        self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0\n        self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded\n        self.embedding_dimensions = embedding_dimensions\n        self.embedding_layers = self.create_embedding_layers()\n        self.return_final_seq_only = return_final_seq_only\n        self.valid_RNN_hidden_layer_types = {\"linear\", \"gru\", \"lstm\"}\n        Base_Network.__init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser,\n                              batch_norm, y_range, random_seed, input_dim)\n\n    def check_all_user_inputs_valid(self):\n        \"\"\"Checks that all the user inputs were valid\"\"\"\n        self.check_RNN_layers_valid()\n        self.check_activations_valid()\n        self.check_embedding_dimensions_valid()\n        self.check_initialiser_valid()\n        self.check_y_range_values_valid()\n        self.check_return_final_seq_only_valid()\n\n    def check_RNN_layers_valid(self):\n        \"\"\"Checks that layers provided by user are valid\"\"\"\n        error_msg_layer_type = \"First element in a layer specification must be one of {}\".format(self.valid_RNN_hidden_layer_types)\n        error_msg_layer_form = \"Layer must be of form [layer_name, hidden_units]\"\n        error_msg_layer_list = \"Layers must be provided as a list\"\n        error_msg_output_heads = \"Number of output activations must equal number of output heads\"\n\n        assert isinstance(self.layers_info, list), error_msg_layer_list\n\n        all_layers = self.layers_info[:-1]\n        output_layer = self.layers_info[-1]\n        assert isinstance(output_layer, list), error_msg_layer_list\n        if isinstance(output_layer[0], list):\n            assert len(output_layer) == len(\n                self.output_activation), error_msg_output_heads\n            for layer in output_layer:\n                all_layers.append(layer)\n        else:\n            assert not isinstance(self.output_activation, list) or len(self.output_activation) == 1, error_msg_output_heads\n            all_layers.append(output_layer)\n\n        rest_must_be_linear = False\n        for layer in all_layers:\n            assert isinstance(layer, list), \"Each layer must be a list\"\n            assert isinstance(layer[0], str), error_msg_layer_type\n            layer_type_name = layer[0].lower()\n            assert layer_type_name in self.valid_RNN_hidden_layer_types, \"Layer name {} not valid, use one of {}\".format(\n                layer_type_name, self.valid_RNN_hidden_layer_types)\n\n            assert isinstance(layer[1], int), error_msg_layer_form\n            assert layer[1] > 0, \"Must have hidden_units >= 1\"\n            assert len(layer) == 2, error_msg_layer_form\n\n            if rest_must_be_linear: assert layer[0].lower() == \"linear\", \"If have linear layers then they must come at end\"\n            if layer_type_name == \"linear\": rest_must_be_linear = True\n\n    def create_and_append_layer(self, layer, rnn_hidden_layers, activation, output_layer=False):\n        layer_type_name = layer[0].lower()\n        hidden_size = layer[1]\n        if output_layer and self.return_final_seq_only: return_sequences = False\n        else: return_sequences = True\n        if layer_type_name == \"lstm\":\n            rnn_hidden_layers.extend([LSTM(units=hidden_size, kernel_initializer=self.initialiser_function,\n                                           return_sequences=return_sequences)])\n        elif layer_type_name == \"gru\":\n            rnn_hidden_layers.extend([GRU(units=hidden_size, kernel_initializer=self.initialiser_function,\n                                          return_sequences=return_sequences)])\n        elif layer_type_name == \"linear\":\n            rnn_hidden_layers.extend(\n                [Dense(units=hidden_size, activation=activation, kernel_initializer=self.initialiser_function)])\n        else:\n            raise ValueError(\"Wrong layer names\")\n        input_dim = hidden_size\n        return input_dim\n\n    def call(self, x, training=True):\n        \"\"\"Forward pass for the network. Note that it expects input data in the form (batch, seq length, features)\"\"\"\n        if self.embedding_to_occur: x = self.incorporate_embeddings(x)\n        training = training or training is None\n        x, restricted_to_final_seq = self.process_hidden_layers(x, training)\n        out = self.process_output_layers(x, restricted_to_final_seq)\n        if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0]) * activations.sigmoid(out)\n        return out\n\n    def incorporate_embeddings(self, x):\n        \"\"\"Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready\n        to then be put through the hidden layers\"\"\"\n        all_embedded_data = []\n        for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded):\n            data = x[:, :, embedding_var]\n            embedded_data = self.embedding_layers[embedding_layer_ix](data)\n            all_embedded_data.append(embedded_data)\n        if len(all_embedded_data) > 1: all_embedded_data = Concatenate(axis=2)(all_embedded_data)\n        else: all_embedded_data = all_embedded_data[0]\n        non_embedded_columns = [col for col in range(x.shape[2]) if col not in self.columns_of_data_to_be_embedded]\n        if len(non_embedded_columns) > 0:\n            x = tf.gather(x, non_embedded_columns, axis=2)\n            x = Concatenate(axis=2)([tf.dtypes.cast(x, float), all_embedded_data])\n        else: x = all_embedded_data\n        return x\n\n    def process_hidden_layers(self, x, training):\n        \"\"\"Puts the data x through all the hidden layers\"\"\"\n        restricted_to_final_seq = False\n        for layer_ix, layer in enumerate(self.hidden_layers):\n            if type(layer) == Dense:\n                if self.return_final_seq_only and not restricted_to_final_seq:\n                    x = x[:, -1, :]\n                    restricted_to_final_seq = True\n                x = layer(x)\n            else:\n                x = layer(x)\n            if self.batch_norm:\n                x = self.batch_norm_layers[layer_ix](x, training=False)\n            if self.dropout != 0.0 and training: x = self.dropout_layer(x)\n        return x, restricted_to_final_seq\n\n    def process_output_layers(self, x, restricted_to_final_seq):\n        \"\"\"Puts the data x through all the output layers\"\"\"\n        out = None\n        for output_layer_ix, output_layer in enumerate(self.output_layers):\n            if type(output_layer) == Dense:\n                if self.return_final_seq_only and not restricted_to_final_seq:\n                    x = x[:, -1, :]\n                    restricted_to_final_seq = True\n                temp_output = output_layer(x)\n            else:\n                temp_output = output_layer(x)\n                activation = self.get_activation(self.output_activation, output_layer_ix)\n                temp_output = activation(temp_output)\n            if out is None: out = temp_output\n            else:\n                if restricted_to_final_seq: dim = 1\n                else: dim = 2\n                out = Concatenate(axis=dim)([out, temp_output])\n        return out\n"
  },
  {
    "path": "nn_builder/tensorflow/__init__.py",
    "content": ""
  },
  {
    "path": "requirements.txt",
    "content": "tensorflow==2.0.0a0\ntorch==1.0.1.post2\ntorchvision==0.2.2.post3\nnumpy==1.16.2\nsetuptools==40.8.0\npytest==4.4.0"
  },
  {
    "path": "setup.py",
    "content": "import setuptools\nimport sys\n\nwith open(\"README.md\", \"r\") as fh:\n    long_description = fh.read()\n\nsetuptools.setup(\n    name=\"nn_builder\",\n    version=\"1.0.5\",\n    author=\"Petros Christodoulou\",\n    author_email=\"p.christodoulou2@gmail.com\",\n    description=\"Build neural networks in 1 line\",\n    long_description=long_description,\n    long_description_content_type=\"text/markdown\",\n    url=\"https://github.com/p-christ/nn_builder\",\n    packages=setuptools.find_packages(),\n    classifiers=[\n        \"Programming Language :: Python :: 3\",\n        \"License :: OSI Approved :: MIT License\",\n        \"Operating System :: OS Independent\",\n    ],\n    install_requires=[\"tensorflow==2.0.0a0\" if ( sys.platform.startswith(\"mac\") or sys.platform.startswith(\"darwin\")) else \"tensorflow-gpu==2.0.0a0\"]\n)"
  },
  {
    "path": "tests/pytorch_tests/test_pytorch_CNN.py",
    "content": "# Run from home directory with python -m pytest tests\nimport shutil\nimport pytest\nimport torch\nimport random\nimport numpy as np\nimport torch.nn as nn\nfrom nn_builder.pytorch.CNN import CNN\nimport torch.optim as optim\nfrom torchvision import datasets, transforms\n\nN = 250\nX = torch.randn((N, 1, 5, 5))\nX[0:125, 0, 3, 3] += 20.0\ny = X[:, 0, 3, 3] > 5.0\ny = y.float()\n\ndef test_user_hidden_layers_input_rejections():\n    \"\"\"Tests whether network rejects invalid hidden_layers inputted from user\"\"\"\n    inputs_that_should_fail = [['maxpool', 33, 22, 33], [['a']], [[222, 222, 222, 222]], [[\"conv\", 2, 2, -1]], [[\"conv\", 2, 2]], [[\"conv\", 2, 2, 55, 999, 33]],\n                                [[\"maxpool\", 33, 33]], [[\"maxpool\", -1, 33]], [[\"maxpool\", 33]], [[\"maxpoolX\", 1, 33]],\n                                [[\"cosnv\", 2, 2]], [[\"avgpool\", 33, 33, 333, 99]], [[\"avgpool\", -1, 33]], [[\"avgpool\", 33]], [[\"avgpoolX\", 1, 33]],\n                                [[\"adaptivemaxpool\", 33, 33, 333, 33]], [[\"adaptivemaxpool\", 2]], [[\"adaptivemaxpool\", 33]], [[\"adaptivemaxpoolX\"]],\n                                [[\"adaptiveavgpool\", 33, 33, 333, 11]], [[\"adaptiveavgpool\", 2]], [[\"adaptiveavgpool\", 33]],\n                                [[\"adaptiveavgpoolX\"]], [[\"linear\", 40, -2]], [[\"lineafr\", 40, 2]]]\n    for input in inputs_that_should_fail:\n        print(input)\n        with pytest.raises(AssertionError):\n            CNN(input_dim=1, layers_info=input, hidden_activations=\"relu\",\n                output_activation=\"relu\")\n\ndef test_user_hidden_layers_input_acceptances():\n    \"\"\"Tests whether network rejects invalid hidden_layers inputted from user\"\"\"\n    inputs_that_should_work = [[[\"conv\", 2, 2, 3331, 2]], [[\"CONV\", 2, 2, 3331, 22]], [[\"ConV\", 2, 2, 3331, 1]],\n                               [[\"maxpool\", 2, 2, 3331]], [[\"MAXPOOL\", 2, 2, 3331]], [[\"MaXpOOL\", 2, 2, 3331]],\n                               [[\"avgpool\", 2, 2, 3331]], [[\"AVGPOOL\", 2, 2, 3331]], [[\"avGpOOL\", 2, 2, 3331]],\n                               [[\"adaptiveavgpool\", 3, 22]], [[\"ADAPTIVEAVGpOOL\", 1, 33]], [[\"ADAPTIVEaVGPOOL\", 3, 6]],\n                               [[\"adaptivemaxpool\", 4, 66]], [[\"ADAPTIVEMAXpOOL\", 2, 2]], [[\"ADAPTIVEmaXPOOL\", 3, 3]],\n                               [[\"adaptivemaxpool\", 3, 3]], [[\"ADAPTIVEMAXpOOL\", 3, 1]],  [[\"linear\", 40]], [[\"lineaR\", 2]],\n                               [[\"LINEAR\", 2]]]\n    for ix, input in enumerate(inputs_that_should_work):\n        input.append([\"linear\", 5])\n        CNN(input_dim=(1, 1, 1), layers_info=input, hidden_activations=\"relu\",\n            output_activation=\"relu\")\n\ndef test_hidden_layers_created_correctly():\n    \"\"\"Tests that create_hidden_layers works correctly\"\"\"\n    layers = [[\"conv\", 2, 4, 3, 2], [\"maxpool\", 3, 4, 2], [\"avgpool\", 32, 42, 22], [\"adaptivemaxpool\", 3, 34],\n              [\"adaptiveavgpool\", 23, 44], [\"linear\", 22], [\"linear\", 2222], [\"linear\", 5]]\n\n    cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n\n    assert type(cnn.hidden_layers[0]) == nn.Conv2d\n    assert cnn.hidden_layers[0].in_channels == 3\n    assert cnn.hidden_layers[0].out_channels == 2\n    assert cnn.hidden_layers[0].kernel_size == (4, 4)\n    assert cnn.hidden_layers[0].stride == (3, 3)\n    assert cnn.hidden_layers[0].padding == (2, 2)\n\n    assert type(cnn.hidden_layers[1]) == nn.MaxPool2d\n    assert cnn.hidden_layers[1].kernel_size == 3\n    assert cnn.hidden_layers[1].stride == 4\n    assert cnn.hidden_layers[1].padding == 2\n\n    assert type(cnn.hidden_layers[2]) == nn.AvgPool2d\n    assert cnn.hidden_layers[2].kernel_size == 32\n    assert cnn.hidden_layers[2].stride == 42\n    assert cnn.hidden_layers[2].padding == 22\n\n    assert type(cnn.hidden_layers[3]) == nn.AdaptiveMaxPool2d\n    assert cnn.hidden_layers[3].output_size == (3, 34)\n\n    assert type(cnn.hidden_layers[4]) == nn.AdaptiveAvgPool2d\n    assert cnn.hidden_layers[4].output_size == (23, 44)\n\n    assert type(cnn.hidden_layers[5]) == nn.Linear\n    assert cnn.hidden_layers[5].in_features == 2024\n    assert cnn.hidden_layers[5].out_features == 22\n\n    assert type(cnn.hidden_layers[6]) == nn.Linear\n    assert cnn.hidden_layers[6].in_features == 22\n    assert cnn.hidden_layers[6].out_features == 2222\n\n    assert type(cnn.output_layers[0]) == nn.Linear\n    assert cnn.output_layers[0].in_features == 2222\n    assert cnn.output_layers[0].out_features == 5\n\n\ndef test_output_layers_created_correctly():\n    \"\"\"Tests that create_output_layers works correctly\"\"\"\n    layers = [[\"conv\", 2, 4, 3, 2], [\"maxpool\", 3, 4, 2], [\"avgpool\", 32, 42, 22], [\"adaptivemaxpool\", 3, 34],\n              [\"adaptiveavgpool\", 23, 44], [\"linear\", 22], [\"linear\", 2222], [\"linear\", 2]]\n\n    cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations=\"relu\", output_activation=\"relu\")\n\n\n    assert cnn.output_layers[0].in_features == 2222\n    assert cnn.output_layers[0].out_features == 2\n\n    layers = [[\"conv\", 2, 4, 3, 2], [\"maxpool\", 3, 4, 2], [\"avgpool\", 32, 42, 22], [\"adaptivemaxpool\", 3, 34],\n              [\"adaptiveavgpool\", 23, 44], [\"linear\", 7]]\n\n    cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n\n    assert cnn.output_layers[0].in_features == 23 * 44 * 2\n    assert cnn.output_layers[0].out_features == 7\n\n    layers = [[\"conv\", 5, 4, 3, 2], [\"maxpool\", 3, 4, 2], [\"avgpool\", 32, 42, 22], [\"adaptivemaxpool\", 3, 34], [\"linear\", 6]]\n\n\n    cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n\n    assert cnn.output_layers[0].in_features == 3 * 34 * 5\n    assert cnn.output_layers[0].out_features == 6\n\n    layers = [[\"conv\", 5, 4, 3, 2], [\"maxpool\", 3, 4, 2], [\"avgpool\", 32, 42, 22], [\"adaptivemaxpool\", 3, 34],\n              [[\"linear\", 6], [\"linear\", 22]]]\n\n    cnn = CNN(input_dim=(3, 1000, 1000), layers_info=layers, hidden_activations=\"relu\",\n              output_activation=[\"softmax\", None])\n\n    assert cnn.output_layers[0].in_features == 3 * 34 * 5\n    assert cnn.output_layers[0].out_features == 6\n    assert cnn.output_layers[1].in_features == 3 * 34 * 5\n    assert cnn.output_layers[1].out_features == 22\n\ndef test_output_dim_user_input():\n    \"\"\"Tests whether network rejects an invalid output_dim input from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            CNN(input_dim=(2, 10, 10), layers_info=[2, input_value], hidden_activations=\"relu\",  output_activation=\"relu\")\n        with pytest.raises(AssertionError):\n            CNN(input_dim=(2, 10, 10), layers_info=input_value, hidden_activations=\"relu\", output_activation=\"relu\")\n\ndef test_activations_user_input():\n    \"\"\"Tests whether network rejects an invalid hidden_activations or output_activation from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            CNN(input_dim=(2, 10, 10), layers_info=[[\"conv\", 2, 2, 3331, 2]], hidden_activations=input_value,\n                output_activation=\"relu\")\n            CNN(input_dim=(2, 10, 10), layers_info=[[\"conv\", 2, 2, 3331, 2]], hidden_activations=\"relu\",\n                output_activation=input_value)\n\ndef test_initialiser_user_input():\n    \"\"\"Tests whether network rejects an invalid initialiser from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            CNN(input_dim=(2, 10, 10), layers_info=[[\"conv\", 2, 2, 3331, 2]], hidden_activations=\"relu\",\n                output_activation=\"relu\", initialiser=input_value)\n\n        CNN(layers_info=[[\"conv\", 2, 2, 3331, 2], [\"linear\", 3]], hidden_activations=\"relu\",\n            output_activation=\"relu\", initialiser=\"xavier\", input_dim=(2, 10, 10))\n\ndef test_batch_norm_layers():\n    \"\"\"Tests whether batch_norm_layers method works correctly\"\"\"\n    layers = [[\"conv\", 2, 4, 3, 2], [\"maxpool\", 3, 4, 2], [\"adaptivemaxpool\", 3, 34], [\"linear\", 5]]\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\", input_dim=(2, 10, 10),\n            output_activation=\"relu\", initialiser=\"xavier\", batch_norm=False)\n\n    layers = [[\"conv\", 2, 4, 3, 2], [\"maxpool\", 3, 4, 2], [\"adaptivemaxpool\", 3, 34], [\"linear\", 5]]\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\", input_dim=(2, 10, 10),\n              output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n    assert len(cnn.batch_norm_layers) == 1\n    assert cnn.batch_norm_layers[0].num_features == 2\n\n\n    layers = [[\"conv\", 2, 4, 3, 2], [\"maxpool\", 3, 4, 2], [\"conv\", 12, 4, 3, 2], [\"adaptivemaxpool\", 3, 34], [\"linear\", 22], [\"linear\", 55]]\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\", input_dim=(2, 10, 10),\n              output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n    assert len(cnn.batch_norm_layers) == 3\n    assert cnn.batch_norm_layers[0].num_features == 2\n    assert cnn.batch_norm_layers[1].num_features == 12\n    assert cnn.batch_norm_layers[2].num_features == 22\n\ndef test_linear_layers_acceptance():\n    \"\"\"Tests that only accepts linear layers of correct shape\"\"\"\n    layers_that_shouldnt_work = [[[\"linear\", 2, 5]], [[\"linear\", 2, 5, 5]], [[\"linear\"]], [[\"linear\", 2], [\"linear\", 5, 4]],\n                                 [\"linear\", 0], [\"linear\", -5]]\n    for layers in layers_that_shouldnt_work:\n        with pytest.raises(AssertionError):\n            cnn = CNN(layers_info=layers, hidden_activations=\"relu\", input_dim=(2, 10, 10),\n                      output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n    layers_that_should_work = [[[\"linear\", 44], [\"linear\", 2]], [[\"linear\", 22]]]\n    for layer in layers_that_should_work:\n        assert CNN(layers_info=layer, hidden_activations=\"relu\", input_dim=(2, 10, 10),\n                      output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\ndef test_linear_layers_only_come_at_end():\n    \"\"\"Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they\n    don't only come at the end\"\"\"\n    layers = [[\"conv\", 2, 4, 3, 2], [\"linear\", 55], [\"maxpool\", 3, 4, 2], [\"adaptivemaxpool\", 3, 34]]\n    with pytest.raises(AssertionError):\n        cnn = CNN(layers_info=layers, hidden_activations=\"relu\", input_dim=(2, 10, 10),\n                  output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\n    layers = [[\"conv\", 2, 4, 3, 2], [\"linear\", 55]]\n    assert CNN(layers_info=layers, hidden_activations=\"relu\", input_dim=(2, 10, 10),\n                      output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\n    layers = [[\"conv\", 2, 4, 3, 2], [\"linear\", 55], [\"linear\", 55], [\"linear\", 55]]\n    assert CNN(layers_info=layers, hidden_activations=\"relu\",  input_dim=(2, 10, 10),\n               output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\ndef test_output_activation():\n    \"\"\"Tests whether network outputs data that has gone through correct activation function\"\"\"\n    RANDOM_ITERATIONS = 20\n    input_dim = (5, 100, 100)\n    for _ in range(RANDOM_ITERATIONS):\n        data = torch.randn((1, *input_dim))\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, 2], [\"adaptivemaxpool\", 2, 2], [\"linear\", 50]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = CNN_instance.forward(data)\n        assert all(out.squeeze() >= 0)\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 20, 1, 0], [\"linear\", 5]],\n                           hidden_activations=\"relu\",  input_dim=input_dim,\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = CNN_instance.forward(data)\n        assert all(out.squeeze() >= 0)\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 5, 20, 1, 0], [\"linear\", 5]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = CNN_instance.forward(data)\n        assert all(out.squeeze() >= 0)\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 5, 20, 1, 0], [\"linear\",  22]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           output_activation=\"sigmoid\", initialiser=\"xavier\")\n        out = CNN_instance.forward(data)\n        assert all(out.squeeze() >= 0)\n        assert all(out.squeeze() <= 1)\n        assert round(torch.sum(out.squeeze()).item(), 3) != 1.0\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, 2], [\"adaptivemaxpool\", 2, 2], [\"linear\", 5]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = CNN_instance.forward(data)\n        assert all(out.squeeze() >= 0)\n        assert all(out.squeeze() <= 1)\n        assert round(torch.sum(out.squeeze()).item(), 3) == 1.0\n\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, 2], [\"adaptivemaxpool\", 2, 2], [\"linear\", 5]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           initialiser=\"xavier\")\n        out = CNN_instance.forward(data)\n        assert not all(out.squeeze() >= 0)\n        assert not round(torch.sum(out.squeeze()).item(), 3) == 1.0\n\ndef test_y_range():\n    \"\"\"Tests whether setting a y range works correctly\"\"\"\n    for _ in range(100):\n        val1 = random.random() - 3.0*random.random()\n        val2 = random.random() + 2.0*random.random()\n        lower_bound = min(val1, val2)\n        upper_bound = max(val1, val2)\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, 2], [\"adaptivemaxpool\", 2, 2], [\"linear\", 5]],\n                           hidden_activations=\"relu\", y_range=(lower_bound, upper_bound),\n                           initialiser=\"xavier\", input_dim=(1, 20, 20))\n        random_data = torch.randn((10, 1, 20, 20))\n        out = CNN_instance.forward(random_data)\n        assert torch.sum(out > lower_bound).item() == 10*5, \"lower {} vs. {} \".format(lower_bound, out)\n        assert torch.sum(out < upper_bound).item() == 10*5, \"upper {} vs. {} \".format(upper_bound, out)\n\ndef test_deals_with_None_activation():\n    \"\"\"Tests whether is able to handle user inputting None as output activation\"\"\"\n    assert CNN(layers_info=[[\"conv\", 2, 2, 1, 2], [\"adaptivemaxpool\", 2, 2], [\"linear\", 5]],\n                           hidden_activations=\"relu\", output_activation=None,\n                           initialiser=\"xavier\", input_dim=(5, 5, 5))\n\ndef test_check_input_data_into_forward_once():\n    \"\"\"Tests that check_input_data_into_forward_once method only runs once\"\"\"\n    CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, 2], [\"adaptivemaxpool\", 2, 2], [\"linear\", 6]],\n                       hidden_activations=\"relu\", input_dim=(4, 2, 5),\n                       output_activation=\"relu\", initialiser=\"xavier\")\n\n    data_not_to_throw_error = torch.randn((1, 4, 2, 5))\n    data_to_throw_error = torch.randn((1, 2, 20, 20))\n\n    with pytest.raises(AssertionError):\n        CNN_instance.forward(data_to_throw_error)\n    with pytest.raises(RuntimeError):\n        CNN_instance.forward(data_not_to_throw_error)\n        CNN_instance.forward(data_to_throw_error)\n\ndef test_y_range_user_input():\n    \"\"\"Tests whether network rejects invalid y_range inputs\"\"\"\n    invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)]\n    for y_range_value in invalid_y_range_inputs:\n        with pytest.raises(AssertionError):\n            print(y_range_value)\n            CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, 2], [\"adaptivemaxpool\", 2, 2]],\n                           hidden_activations=\"relu\", y_range=y_range_value, input_dim=(2, 2, 2),\n                           initialiser=\"xavier\")\n\ndef test_model_trains():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    for output_activation in [\"sigmoid\", \"None\"]:\n        CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"adaptivemaxpool\", 1, 1], [\"linear\", 1]], input_dim=(1, 5, 5),\n                           hidden_activations=\"relu\", output_activation=output_activation,\n                           initialiser=\"xavier\")\n        assert solves_simple_problem(X, y, CNN_instance)\n\n    z = X[:, 0:1, 3:4, 3:4] > 5.0\n    z =  torch.cat([z ==1, z==0], dim=1).float()\n    z = z.squeeze(-1).squeeze(-1)\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"linear\", 2]], input_dim=(1, 5, 5),\n                           hidden_activations=\"relu\", output_activation=\"softmax\", dropout=0.01,\n                           initialiser=\"xavier\")\n    assert solves_simple_problem(X, z, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"linear\", 1]], input_dim=(1, 5, 5),\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"linear\", 1]], input_dim=(1, 5, 5),\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\", batch_norm=True)\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"maxpool\", 1, 1, 0], [\"linear\", 1]], input_dim=(1, 5, 5),\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"avgpool\", 1, 1, 0], [\"linear\", 1]], input_dim=(1, 5, 5),\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 5, 3, 1, 0], [\"adaptivemaxpool\", 2, 2], [\"linear\", 1]], input_dim=(1, 5, 5),\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 5, 3, 1, 0], [\"adaptiveavgpool\", 2, 2], [\"linear\", 1]], input_dim=(1, 5, 5),\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\ndef test_model_trains_linear_layer():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    CNN_instance = CNN(layers_info=[[\"conv\", 5, 3, 1, 0], [\"linear\", 5], [\"linear\", 5], [\"linear\", 1]],\n                       input_dim=(1, 5, 5),\n                       hidden_activations=\"relu\", output_activation=\"sigmoid\",\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"linear\", 5], [\"linear\", 5], [\"linear\", 1]], input_dim=(1, 5, 5),\n                       hidden_activations=\"relu\", output_activation=\"sigmoid\",\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\ndef test_max_pool_working():\n    \"\"\"Tests whether max pool layers work properly\"\"\"\n    N = 250\n    X = torch.randn((N, 1, 8, 8))\n    X[0:125, 0, 3, 3] = 999.99\n    CNN_instance = CNN(layers_info=[[\"maxpool\", 2, 2, 0], [\"maxpool\", 2, 2, 0], [\"maxpool\", 2, 2, 0], [\"linear\", 1]],\n                       hidden_activations=\"relu\", input_dim=(1, 8, 8),\n                       initialiser=\"xavier\")\n    assert CNN_instance(X).shape == (N, 1)\n\ndef test_dropout():\n    \"\"\"Tests whether dropout layer reads in probability correctly\"\"\"\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"adaptivemaxpool\", 1, 1], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=\"sigmoid\", dropout=0.9999,\n                           initialiser=\"xavier\", input_dim=(1, 5, 5))\n    assert CNN_instance.dropout_layer.p == 0.9999\n    assert not solves_simple_problem(X, y, CNN_instance)\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"adaptivemaxpool\", 1, 1], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=None, dropout=0.0000001,\n                           initialiser=\"xavier\", input_dim=(1, 5, 5))\n    assert CNN_instance.dropout_layer.p == 0.0000001\n    assert solves_simple_problem(X, y, CNN_instance)\n\n\ndef solves_simple_problem(X, y, nn_instance):\n    \"\"\"Checks if a given network is able to solve a simple problem\"\"\"\n    optimizer = optim.Adam(nn_instance.parameters(), lr=0.15)\n    for ix in range(800):\n        out = nn_instance.forward(X)\n        loss = torch.sum((out.squeeze() - y) ** 2) / N\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n    print(\"LOSS \", loss)\n    return loss < 0.1\n\ndef test_MNIST_progress():\n    \"\"\"Tests that network made using CNN module can make progress on MNIST\"\"\"\n    batch_size = 128\n    train_loader = torch.utils.data.DataLoader(\n        datasets.MNIST(root=\"input/\", train=True, download=True,\n                       transform=transforms.Compose([\n                           transforms.ToTensor(),\n                           transforms.Normalize((0.1307,), (0.3081,))\n                       ])),\n        batch_size=batch_size, shuffle=True)\n\n    for batch_norm in [True, False]:\n\n        cnn = CNN(layers_info=[[\"conv\", 20, 5, 1, 0],\n                                      [\"maxpool\", 2, 2, 0],\n                                      [\"conv\", 50, 5, 1, 0],\n                                      [\"maxpool\", 2, 2, 0],\n                                      [\"linear\", 500], [\"linear\", 10]], hidden_activations=\"relu\",\n                  output_activation=\"softmax\", initialiser=\"xavier\", input_dim=(1, 28, 28), batch_norm=batch_norm)\n\n        loss_fn = nn.CrossEntropyLoss()\n        optimizer = optim.Adam(cnn.parameters(), lr=0.001)\n\n        ix = 0\n        accuracies = []\n        for data, target in train_loader:\n            ix += 1\n\n            output = cnn(data)\n            loss = loss_fn(output, target)\n            optimizer.zero_grad()\n            loss.backward()\n            optimizer.step()\n\n            pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability\n            correct = pred.eq(target.view_as(pred)).sum().item()\n            print(\"Accuracy {}\".format(correct / batch_size))\n            accuracies.append(correct / batch_size)\n\n            if ix > 200:\n                break\n\n\n        assert accuracies[-1] > 0.7, \"Accuracy not good enough {}\".format(accuracies[-1])\n    shutil.rmtree(\"input/\", ignore_errors=False, onerror=None)\n\n\ndef test_all_activations_work():\n    \"\"\"Tests that all activations get accepted\"\"\"\n    nn_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"adaptivemaxpool\", 1, 1], [\"linear\", 1]],\n                                   dropout=0.0000001,\n                                   initialiser=\"xavier\", input_dim=(1, 5, 5))\n    for key in nn_instance.str_to_activations_converter.keys():\n        if key == \"none\": hidden_key = \"relu\"\n        else: hidden_key = key\n        model = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"adaptivemaxpool\", 1, 1], [\"linear\", 1]],\n                                   hidden_activations=hidden_key, output_activation=key, dropout=0.0000001,\n                                   initialiser=\"xavier\", input_dim=(1, 5, 5))\n        model(X)\n\ndef test_all_initialisers_work():\n    \"\"\"Tests that all initialisers get accepted\"\"\"\n    nn_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"adaptivemaxpool\", 1, 1], [\"linear\", 1]],\n                                   dropout=0.0000001,\n                                   initialiser=\"xavier\", input_dim=(1, 5, 5))\n    for key in nn_instance.str_to_initialiser_converter.keys():\n        if key != \"eye\":\n            model = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"adaptivemaxpool\", 1, 1], [\"linear\", 1]],\n                                        dropout=0.0000001,\n                                       initialiser=key, input_dim=(1, 5, 5))\n            model(X)\n\ndef test_print_model_summary():\n    nn_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, 0], [\"adaptivemaxpool\", 1, 1], [\"linear\", 1]],\n                                   dropout=0.0000001,\n                                   initialiser=\"xavier\", input_dim=(1, 5, 5))\n    nn_instance.print_model_summary()\n\ndef test_output_heads_error_catching():\n    \"\"\"Tests that having multiple output heads catches errors from user inputs\"\"\"\n    output_dims_that_should_break = [[\"linear\", 2, 2, \"SAME\", \"conv\", 3, 4, \"SAME\"], [[[\"conv\", 3, 2, \"same\"], [\"linear\", 4]]],\n                                     [[2, 8]], [-33, 33, 33, 33, 33]]\n    for output_dim in output_dims_that_should_break:\n        with pytest.raises(AssertionError):\n            CNN(input_dim=(12, 12, 3), layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1], output_dim],\n                hidden_activations=\"relu\", output_activation=\"relu\")\n    output_activations_that_should_break = [\"relu\", [\"relu\"], [\"relu\", \"softmax\"]]\n    for output_activation in output_activations_that_should_break:\n        with pytest.raises(AssertionError):\n            CNN(input_dim=(12, 12, 3), layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1],\n                                                    [[\"linear\", 4], [\"linear\", 10], [\"linear\", 4]]],\n               hidden_activations=\"relu\", output_activation=output_activation)\n\n\ndef test_output_head_layers():\n    \"\"\"Tests whether the output head layers get created properly\"\"\"\n    for output_dim in [[[\"linear\", 3],[\"linear\", 9]], [[\"linear\", 4], [\"linear\", 20]], [[\"linear\", 1], [\"linear\", 1]]]:\n        nn_instance = CNN(input_dim=(12, 12, 3), layers_info=[[\"conv\", 25, 5, 1, 2], [\"conv\", 25, 5, 1, 3], [\"linear\", 5], output_dim],\n                          hidden_activations=\"relu\", output_activation=[\"softmax\", None])\n        assert nn_instance.output_layers[0].out_features == output_dim[0][1]\n        assert nn_instance.output_layers[0].in_features == 5\n        assert nn_instance.output_layers[1].out_features == output_dim[1][1]\n        assert nn_instance.output_layers[1].in_features == 5\n\ndef test_output_head_activations_work():\n    \"\"\"Tests that output head activations work properly\"\"\"\n\n    output_dim = [[\"linear\", 5], [\"linear\", 10], [\"linear\", 3]]\n    nn_instance = CNN(input_dim=(12, 12, 3), layers_info=[[\"conv\", 3, 2, 1, 1], [\"conv\", 3, 1, 1, 1], [\"linear\", 1], output_dim],\n                          hidden_activations=\"relu\", output_activation=[\"softmax\", None, \"relu\"])\n    x = torch.randn((20, 12, 12, 3)) * -20.0\n    out = nn_instance(x)\n\n    assert out.shape == (20, 18)\n\n    sums = torch.sum(out[:, :5], dim=1).detach().numpy()\n    sums_others = torch.sum(out[:, 5:], dim=1).detach().numpy()\n    sums_others_2 = torch.sum(out[:, 5:15], dim=1).detach().numpy()\n    sums_others_3 = torch.sum(out[:, 15:18], dim=1).detach().numpy()\n\n\n    for row in range(out.shape[0]):\n        assert np.round(sums[row], 4) == 1.0, sums[row]\n        assert not np.round(sums_others[row], 4) == 1.0, sums_others[row]\n        assert not np.round(sums_others_2[row], 4) == 1.0, sums_others_2[row]\n        assert not np.round(sums_others_3[row], 4) == 1.0, sums_others_3[row]\n        for col in range(3):\n            assert out[row, 15 + col] >= 0.0, out[row, 15 + col]\n\ndef test_output_head_shapes_correct():\n    \"\"\"Tests that the output shape of network is correct when using multiple outpout heads\"\"\"\n    N = 20\n    X = torch.randn((N, 25, 25, 2))\n    for _ in range(25):\n        nn_instance = CNN(input_dim=(25, 25, 2),\n            layers_info=[[\"conv\", 25, 2, 1, 1], [\"conv\", 2, 5, 1, 1], [\"linear\", 1], [\"linear\", 12]],\n            hidden_activations=\"relu\")\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 12\n\n    for output_dim in [[ [\"linear\", 10], [\"linear\", 4], [\"linear\", 6]], [[\"linear\", 3], [\"linear\", 8], [\"linear\", 9]]]:\n        nn_instance = CNN(input_dim=(25, 25, 2),\n            layers_info=[[\"conv\", 25, 1, 1, 2], [\"conv\", 25, 5, 1, 1], [\"linear\", 1], output_dim],\n            hidden_activations=\"relu\", output_activation=[\"softmax\", None, \"relu\"])\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 20\n"
  },
  {
    "path": "tests/pytorch_tests/test_pytorch_NN.py",
    "content": "# Run from home directory with python -m pytest tests\nimport copy\nimport shutil\nfrom sklearn.datasets import load_boston\nfrom sklearn.model_selection import train_test_split\nimport pytest\nimport torch\nimport random\nimport numpy as np\nimport torch.nn as nn\nimport torch.optim as optim\nfrom nn_builder.pytorch.NN import NN\nfrom sklearn.utils import shuffle\n\nN = 250\nX = torch.randn((N, 5))\nX[:, [2, 4]] += 10.0\ny = X[:, 0] > 0\ny = y.float()\n\ndef test_linear_hidden_units_user_input():\n    \"\"\"Tests whether network rejects an invalid linear_hidden_units input from user\"\"\"\n    inputs_that_should_fail = [\"a\", [\"a\", \"b\"], [2, 4, \"ss\"], [-2], 2]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            NN(input_dim=2, layers_info=input_value, hidden_activations=\"relu\", output_activation=\"relu\")\n\ndef test_input_dim_output_dim_user_input():\n    \"\"\"Tests whether network rejects an invalid input_dim from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            NN(input_dim=input_value, layers_info=[2], hidden_activations=\"relu\", output_activation=\"relu\")\n\ndef test_activations_user_input():\n    \"\"\"Tests whether network rejects an invalid hidden_activations or output_activation from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            NN(input_dim=2, layers_info=[2], hidden_activations=input_value,\n               output_activation=\"relu\")\n            NN(input_dim=2, layers_info=[2], hidden_activations=\"relu\",\n               output_activation=input_value)\n\ndef test_initialiser_user_input():\n    \"\"\"Tests whether network rejects an invalid initialiser from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            NN(input_dim=2, layers_info=[2], hidden_activations=\"relu\",\n               output_activation=\"relu\", initialiser=input_value)\n        NN(input_dim=2, layers_info=[2], hidden_activations=\"relu\",\n           output_activation=\"relu\", initialiser=\"xavier\")\n\ndef test_output_shape_correct():\n    \"\"\"Tests whether network returns output of the right shape\"\"\"\n    input_dims = [x for x in range(1, 3)]\n    output_dims = [x for x in range(4, 6)]\n    linear_hidden_units_options = [ [2, 3, 4], [2, 9, 1], [55, 55, 55, 234, 15]]\n    for input_dim, output_dim, linear_hidden_units in zip(input_dims, output_dims, linear_hidden_units_options):\n        linear_hidden_units.append(output_dim)\n        nn_instance = NN(input_dim=input_dim, layers_info=linear_hidden_units, hidden_activations=\"relu\",\n                         output_activation=\"relu\", initialiser=\"xavier\")\n        data = torch.randn((25, input_dim))\n        output = nn_instance.forward(data)\n        assert output.shape == (25, output_dim)\n\ndef test_output_activation():\n    \"\"\"Tests whether network outputs data that has gone through correct activation function\"\"\"\n    RANDOM_ITERATIONS = 20\n    for _ in range(RANDOM_ITERATIONS):\n        data = torch.randn((1, 100))\n        nn_instance = NN(input_dim=100, layers_info=[5, 5, 5],\n                         hidden_activations=\"relu\",\n                         output_activation=\"relu\", initialiser=\"xavier\")\n        out = nn_instance.forward(data)\n        assert all(out.squeeze() >= 0)\n\n        nn_instance = NN(input_dim=100, layers_info=[5, 5, 5],\n                         hidden_activations=\"relu\",\n                         output_activation=\"sigmoid\", initialiser=\"xavier\")\n        out = nn_instance.forward(data)\n        assert all(out.squeeze() >= 0)\n        assert all(out.squeeze() <= 1)\n\n        nn_instance = NN(input_dim=100, layers_info=[5, 5, 5],\n                         hidden_activations=\"relu\",\n                         output_activation=\"softmax\", initialiser=\"xavier\")\n        out = nn_instance.forward(data)\n        assert all(out.squeeze() >= 0)\n        assert all(out.squeeze() <= 1)\n        assert round(torch.sum(out.squeeze()).item(), 3) == 1.0\n\n        nn_instance = NN(input_dim=100, layers_info=[5, 5, 5],\n                         hidden_activations=\"relu\",\n                         )\n        out = nn_instance.forward(data)\n        assert not all(out.squeeze() >= 0)\n        assert not round(torch.sum(out.squeeze()).item(), 3) == 1.0\n\ndef test_linear_layers():\n    \"\"\"Tests whether create_hidden_layers method works correctly\"\"\"\n    for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]):\n        hidden_units.append(output_dim)\n        nn_instance = NN(input_dim=input_dim, layers_info=hidden_units,\n                         hidden_activations=\"relu\",\n                         output_activation=\"relu\", initialiser=\"xavier\")\n        for layer in nn_instance.hidden_layers:\n            assert isinstance(layer, nn.Linear)\n        assert nn_instance.hidden_layers[0].in_features == input_dim\n        assert nn_instance.hidden_layers[0].out_features == hidden_units[0]\n        assert nn_instance.hidden_layers[1].in_features == hidden_units[0]\n        assert nn_instance.hidden_layers[1].out_features == hidden_units[1]\n        assert nn_instance.hidden_layers[2].in_features == hidden_units[1]\n        assert nn_instance.hidden_layers[2].out_features == hidden_units[2]\n        assert len(nn_instance.hidden_layers) == 3\n\ndef test_embedding_layers():\n    \"\"\"Tests whether create_embedding_layers method works correctly\"\"\"\n    for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)):\n        nn_instance = NN(input_dim=5, layers_info=[5],\n                         embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]])\n        for layer in nn_instance.embedding_layers:\n            assert isinstance(layer, nn.Embedding)\n        assert len(nn_instance.embedding_layers) == 2\n        assert nn_instance.embedding_layers[0].num_embeddings == embedding_in_dim_1\n        assert nn_instance.embedding_layers[0].embedding_dim == embedding_out_dim_1\n        assert nn_instance.embedding_layers[1].num_embeddings == embedding_in_dim_2\n        assert nn_instance.embedding_layers[1].embedding_dim == embedding_out_dim_2\n\ndef test_non_integer_embeddings_rejected():\n    \"\"\"Tests whether an error is raised if user tries to provide non-integer data to be embedded\"\"\"\n    with pytest.raises(AssertionError):\n        nn_instance = NN(input_dim=5, layers_info=[5],\n                         columns_of_data_to_be_embedded=[2, 4],\n                         embedding_dimensions=[[50, 3],\n                                               [55, 4]])\n        out = nn_instance.forward(X)\n\ndef test_incorporate_embeddings():\n    \"\"\"Tests the method incorporate_embeddings\"\"\"\n    X_new = X\n    X_new[:, [2, 4]] = torch.round(X_new[:, [2, 4]])\n    nn_instance = NN(input_dim=5, layers_info=[5],\n                     columns_of_data_to_be_embedded=[2, 4],\n                     embedding_dimensions=[[50, 3],\n                                                       [55, 4]])\n    out = nn_instance.incorporate_embeddings(X)\n    assert out.shape == (N, X.shape[1]+3+4-2)\n\ndef test_embedding_network_can_solve_simple_problem():\n    \"\"\"Tests whether network can solve simple problem using embeddings\"\"\"\n    X = torch.randn(N, 2) * 5.0 + 20.0\n    y = (X[:, 0] >= 20) * (X[:, 1] <= 20)\n    X = X.long()\n    nn_instance = NN(input_dim=2, layers_info=[5, 1],\n                     columns_of_data_to_be_embedded=[0, 1],\n                     embedding_dimensions=[[50, 3],\n                                           [55, 3]])\n    assert solves_simple_problem(X, y.float(), nn_instance)\n\ndef test_batch_norm_layers():\n    \"\"\"Tests whether batch_norm_layers method works correctly\"\"\"\n    for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]):\n        hidden_units.append(output_dim)\n        nn_instance = NN(input_dim=input_dim, layers_info=hidden_units,\n                         hidden_activations=\"relu\", batch_norm=True,\n                         output_activation=\"relu\", initialiser=\"xavier\")\n        for layer in nn_instance.batch_norm_layers:\n            assert isinstance(layer, nn.BatchNorm1d)\n        assert len(nn_instance.batch_norm_layers) == len(hidden_units) - 1\n        assert nn_instance.batch_norm_layers[0].num_features == hidden_units[0]\n        assert nn_instance.batch_norm_layers[1].num_features == hidden_units[1]\n        assert nn_instance.batch_norm_layers[2].num_features == hidden_units[2]\n\ndef test_model_trains():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    for output_activation in [\"sigmoid\", \"None\"]:\n        nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 10, 1],\n                         output_activation=output_activation, dropout=0.01, batch_norm=True)\n        assert solves_simple_problem(X, y, nn_instance)\n    z = X[:, 0:1] > 0\n    z =  torch.cat([z ==1, z==0], dim=1).float()\n    nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 10, 2],\n                     output_activation=\"softmax\", dropout=0.01, batch_norm=True)\n    assert solves_simple_problem(X, z, nn_instance)\n\ndef solves_simple_problem(X, y, nn_instance):\n    \"\"\"Checks if a given network is able to solve a simple problem\"\"\"\n    optimizer = optim.Adam(nn_instance.parameters(), lr=0.15)\n    for ix in range(800):\n        out = nn_instance.forward(X)\n        loss = torch.sum((out.squeeze() - y) ** 2) / N\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n    return loss < 0.1\n\ndef test_dropout():\n    \"\"\"Tests whether dropout layer reads in probability correctly\"\"\"\n    nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999) \n    assert nn_instance.dropout_layer.p == 0.9999\n    assert not solves_simple_problem(X, y,  nn_instance)\n    nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.00001)\n    assert solves_simple_problem(X, y, nn_instance)\n\ndef test_y_range_user_input():\n    \"\"\"Tests whether network rejects invalid y_range inputs\"\"\"\n    invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)]\n    for y_range_value in invalid_y_range_inputs:\n        with pytest.raises(AssertionError):\n            print(y_range_value)\n            nn_instance = NN(input_dim=5, layers_info=[10, 10, 3],\n                             y_range=y_range_value)\n\ndef test_y_range():\n    \"\"\"Tests whether setting a y range works correctly\"\"\"\n    for _ in range(100):\n        val1 = random.random() - 3.0*random.random()\n        val2 = random.random() + 2.0*random.random()\n        lower_bound = min(val1, val2)\n        upper_bound = max(val1, val2)\n        nn_instance = NN(input_dim=5, layers_info=[10, 10, 3],  y_range=(lower_bound, upper_bound))\n        random_data = torch.randn(15, 5)\n        out = nn_instance.forward(random_data)\n        assert torch.sum(out > lower_bound).item() == 3*15, \"lower {} vs. {} \".format(lower_bound, out)\n        assert torch.sum(out < upper_bound).item() == 3*15, \"upper {} vs. {} \".format(upper_bound, out)\n\ndef test_deals_with_None_activation():\n    \"\"\"Tests whether is able to handle user inputting None as output activation\"\"\"\n    assert NN(input_dim=5, layers_info=[10, 10, 3], output_activation=None)\n\ndef test_check_input_data_into_forward_once():\n    \"\"\"Tests that check_input_data_into_forward_once method only runs once\"\"\"\n    data_to_throw_error = torch.randn(N, 2)\n    X = torch.randn(N, 2) * 5.0 + 20.0\n    y = (X[:, 0] >= 20) * (X[:, 1] <= 20)\n    X = X.long()\n    nn_instance = NN(input_dim=2, layers_info=[5, 1],\n                     columns_of_data_to_be_embedded=[0, 1],\n                     embedding_dimensions=[[50, 3],\n                                           [55, 3]])\n    with pytest.raises(AssertionError):\n        nn_instance.forward(data_to_throw_error)\n    with pytest.raises(RuntimeError):\n        nn_instance.forward(X)\n        nn_instance.forward(data_to_throw_error)\n\ndef test_all_activations_work():\n    \"\"\"Tests that all activations get accepted\"\"\"\n    nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999)\n    for key in nn_instance.str_to_activations_converter.keys():\n        if key == \"none\": hidden_key = \"relu\"\n        else: hidden_key = key\n        model = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999, hidden_activations=hidden_key, output_activation=key)\n        model(X)\n\ndef test_all_initialisers_work():\n    \"\"\"Tests that all initialisers get accepted\"\"\"\n    nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999)\n    for key in nn_instance.str_to_initialiser_converter.keys():\n        model = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999, initialiser=key)\n        model(X)\n\ndef test_print_model_summary():\n    nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999)\n    nn_instance.print_model_summary()\n\ndef test_output_heads_error_catching():\n    \"\"\"Tests that having multiple output heads catches errors from user inputs\"\"\"\n    output_dims_that_should_break = [[[2, 8]], [-33, 33, 33, 33, 33]]\n    for output_dim in output_dims_that_should_break:\n        with pytest.raises(AssertionError):\n            NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations=\"relu\",\n               output_activation=\"relu\")\n\n    output_activations_that_should_break = [\"relu\", [\"relu\"], [\"relu\", \"softmax\"]]\n    for output_activation in output_activations_that_should_break:\n        with pytest.raises(AssertionError):\n            NN(input_dim=2, layers_info=[4, 7, 9, [4, 6, 1]], hidden_activations=\"relu\",\n               output_activation=output_activation)\n\ndef test_output_head_layers():\n    \"\"\"Tests whether the output head layers get created properly\"\"\"\n    for output_dim in [[3, 9], [4, 20], [1, 1]]:\n        nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations=\"relu\",\n                         output_activation=[\"softmax\", None])\n        assert nn_instance.output_layers[0].in_features == 9\n        assert nn_instance.output_layers[1].in_features == 9\n        assert nn_instance.output_layers[0].out_features == output_dim[0]\n        assert nn_instance.output_layers[1].out_features == output_dim[1]\n\ndef test_output_head_activations_work():\n    \"\"\"Tests that output head activations work properly\"\"\"\n    nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, [5, 10, 3]], hidden_activations=\"relu\",\n                     output_activation=[\"softmax\", None, \"relu\"])\n\n    x = torch.randn((20, 2)) * -20.0\n    out = nn_instance.forward(x)\n    assert torch.allclose(torch.sum(out[:, :5], dim=1), torch.Tensor([1.0]))\n    assert not torch.allclose(torch.sum(out[:, 5:], dim=1), torch.Tensor([1.0]))\n    for row in range(out.shape[0]):\n        assert all(out[row, -3:] >= 0)\n\ndef test_output_head_shapes_correct():\n    \"\"\"Tests that the output shape of network is correct when using multiple outpout heads\"\"\"\n    N = 20\n    X = torch.randn((N, 2))\n    for _ in range(25):\n        output_dim = random.randint(1, 100)\n        nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations=\"relu\")\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == output_dim\n\n    for output_dim in [[3, 9, 5, 3], [5, 5, 5, 5], [2, 1, 1, 16]]:\n        nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations=\"relu\",\n                         output_activation=[\"softmax\", None, None, \"relu\"])\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 20\n\ndef train_on_boston_housing(model, X, y):\n    # Compile the model\n    loss_fn = nn.MSELoss()\n    optimizer = optim.Adam(model.parameters(), lr=0.08)\n    X = torch.Tensor(X)\n    y = torch.Tensor(y)\n\n    for _ in range(1000):\n        output = model(X)\n        loss = loss_fn(output.squeeze(-1), y)\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n        print(loss)\n    return loss\n\n\ndef test_boston_housing_progress():\n    \"\"\"Tests that network made using NN module can make progress on boston housing dataset\"\"\"\n    boston = load_boston()\n    X, y = (boston.data, boston.target)\n\n    # Normalise the data\n    mean = np.mean(X, axis=0)\n    std = np.std(X, axis=0)\n    X = (X - mean) / std\n\n    model = NN(layers_info=[30, 10, 1], input_dim=13,\n               hidden_activations=\"relu\", output_activation=None, dropout=0.0,\n               initialiser=\"xavier\", batch_norm=True, y_range=(4.5, 55.0))\n    result = train_on_boston_housing(model, X, y)\n    assert result < 15\n\n    model = NN(layers_info=[15, 15, 15, 15, 1], input_dim=13, random_seed=52,\n                hidden_activations=\"relu\", output_activation=None, dropout=0.0,\n                initialiser=\"xavier\", batch_norm=False)\n    result = train_on_boston_housing(model, X, y)\n    assert result < 15\n\n    model = NN(layers_info=[30, 10, 1], input_dim=13,\n                hidden_activations=\"relu\", output_activation=None, dropout=0.0,\n                initialiser=\"xavier\", batch_norm=True)\n    result = train_on_boston_housing(model, X, y)\n    assert result < 15\n\n    model = NN(layers_info=[15, 15, 15, 1], input_dim=13,\n                hidden_activations=\"relu\", output_activation=None, dropout=0.05,\n                initialiser=\"xavier\", batch_norm=False)\n    result = train_on_boston_housing(model, X, y)\n    assert result < 15\n\n\n\n"
  },
  {
    "path": "tests/pytorch_tests/test_pytorch_RNN.py",
    "content": "# Run from home directory with python -m pytest tests\nimport copy\nimport pytest\nimport random\nimport numpy as np\nimport torch.nn as nn\nimport torch\nfrom nn_builder.pytorch.RNN import RNN\nimport torch.optim as optim\n\nN = 250\nX = torch.randn((N, 5, 15))\nX[0:125, 0, 3] += 20.0\ny = X[:, 0, 3] > 5.0\ny = y.float()\n\ndef test_user_hidden_layers_input_rejections():\n    \"\"\"Tests whether network rejects invalid hidden_layers inputted from user\"\"\"\n    inputs_that_should_fail = [[[\"linearr\", 33]], [[\"linear\", 12, 33]], [[\"gru\", 2, 33]], [[\"lstm\", 2, 33]], [[\"lstmr\", 33]],\n                               [[\"gruu\", 33]], [[\"gru\", 33], [\"xxx\", 33]], [[\"linear\", 33], [\"gru\", 12], [\"gru\", 33]] ]\n    for input in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            RNN(input_dim=1, layers_info=input, hidden_activations=\"relu\",\n                output_activation=\"relu\")\n\ndef test_user_hidden_layers_input_acceptances():\n    \"\"\"Tests whether network rejects invalid hidden_layers inputted from user\"\"\"\n    inputs_that_should_work = [[[\"linear\", 33]], [[\"linear\", 12]], [[\"gru\", 2]], [[\"lstm\", 2]], [[\"lstm\", 1]],\n                               [[\"gru\", 330]], [[\"gru\", 33], [\"linear\", 2]] ]\n    for input in inputs_that_should_work:\n        assert  RNN(input_dim=1, layers_info=input, hidden_activations=\"relu\",\n                output_activation=\"relu\")\n\n\ndef test_hidden_layers_created_correctly():\n    \"\"\"Tests that create_hidden_layers works correctly\"\"\"\n    layers = [[\"gru\", 25], [\"lstm\", 23], [\"linear\", 5], [\"linear\", 10]]\n\n    rnn = RNN(input_dim=5, layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n\n    assert type(rnn.hidden_layers[0]) == nn.GRU\n    assert rnn.hidden_layers[0].input_size == 5\n    assert rnn.hidden_layers[0].hidden_size == 25\n\n    assert type(rnn.hidden_layers[1]) == nn.LSTM\n    assert rnn.hidden_layers[1].input_size == 25\n    assert rnn.hidden_layers[1].hidden_size == 23\n\n    assert type(rnn.hidden_layers[2]) == nn.Linear\n    assert rnn.hidden_layers[2].in_features == 23\n    assert rnn.hidden_layers[2].out_features == 5\n\n    assert type(rnn.output_layers[0]) == nn.Linear\n    assert rnn.output_layers[0].in_features == 5\n    assert rnn.output_layers[0].out_features == 10\n\n\ndef test_output_layers_created_correctly():\n    \"\"\"Tests that create_output_layers works correctly\"\"\"\n    layers = [[\"gru\", 25], [\"lstm\", 23], [\"linear\", 5], [\"linear\", 10]]\n\n    rnn = RNN(input_dim=5, layers_info=layers, hidden_activations=\"relu\", output_activation=\"relu\")\n\n    assert rnn.output_layers[0].in_features == 5\n    assert rnn.output_layers[0].out_features == 10\n\n    layers = [[\"gru\", 25], [\"lstm\", 23], [\"lstm\", 10]]\n\n    rnn = RNN(input_dim=5, layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n\n    assert rnn.output_layers[0].input_size == 23\n    assert rnn.output_layers[0].hidden_size == 10\n\n    layers = [[\"gru\", 25], [\"lstm\", 23], [[\"lstm\", 10], [\"linear\", 15]]]\n    rnn = RNN(input_dim=5, layers_info=layers, hidden_activations=\"relu\",\n              output_activation=[\"relu\", \"softmax\"])\n\n    assert rnn.output_layers[0].input_size == 23\n    assert rnn.output_layers[0].hidden_size == 10\n\n    assert rnn.output_layers[1].in_features == 23\n    assert rnn.output_layers[1].out_features == 15\n\ndef test_output_dim_user_input():\n    \"\"\"Tests whether network rejects an invalid output_dim input from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            RNN(input_dim=3, layers_info=[2, input_value], hidden_activations=\"relu\",  output_activation=\"relu\")\n        with pytest.raises(AssertionError):\n            RNN(input_dim=6, layers_info=input_value, hidden_activations=\"relu\", output_activation=\"relu\")\n\ndef test_activations_user_input():\n    \"\"\"Tests whether network rejects an invalid hidden_activations or output_activation from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            RNN(input_dim=4, layers_info=[[\"linear\", 2]], hidden_activations=input_value,\n                output_activation=\"relu\")\n            RNN(input_dim=4, layers_info=[[\"linear\", 2]], hidden_activations=\"relu\",\n                output_activation=input_value)\n\ndef test_initialiser_user_input():\n    \"\"\"Tests whether network rejects an invalid initialiser from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            RNN(input_dim=4, layers_info=[[\"linear\", 2]], hidden_activations=\"relu\",\n                output_activation=\"relu\", initialiser=input_value)\n\n            RNN(layers_info=[[\"linear\", 2], [\"linear\", 2]], hidden_activations=\"relu\",\n            output_activation=\"relu\", initialiser=\"xavier\", input_dim=4)\n\ndef test_batch_norm_layers():\n    \"\"\"Tests whether batch_norm_layers method works correctly\"\"\"\n    layers = [[\"gru\", 20], [\"lstm\", 3], [\"linear\", 4], [\"linear\", 10]]\n    rnn = RNN(layers_info=layers, hidden_activations=\"relu\", input_dim=5,\n              output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n    assert len(rnn.batch_norm_layers) == 3\n    assert rnn.batch_norm_layers[0].num_features == 20\n    assert rnn.batch_norm_layers[1].num_features == 3\n    assert rnn.batch_norm_layers[2].num_features == 4\n\ndef test_linear_layers_only_come_at_end():\n    \"\"\"Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they\n    don't only come at the end\"\"\"\n    layers = [[\"gru\", 20],  [\"linear\", 4], [\"lstm\", 3], [\"linear\", 10]]\n    with pytest.raises(AssertionError):\n        rnn = RNN(layers_info=layers, hidden_activations=\"relu\", input_dim=4,\n                  output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\n    layers = [[\"gru\", 20], [\"lstm\", 3],  [\"linear\", 4], [\"linear\", 10]]\n    assert RNN(layers_info=layers, hidden_activations=\"relu\", input_dim=4,\n                      output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\ndef test_output_activation():\n    \"\"\"Tests whether network outputs data that has gone through correct activation function\"\"\"\n    RANDOM_ITERATIONS = 20\n    input_dim = 100\n    for _ in range(RANDOM_ITERATIONS):\n        data = torch.randn((25, 10, 100))\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5]],\n                           hidden_activations=\"relu\",  input_dim=input_dim,\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           output_activation=\"sigmoid\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n        assert all(out.reshape(1, -1).squeeze() <= 1)\n        summed_result = torch.sum(out, dim=1)\n        assert all(summed_result.reshape(1, -1).squeeze() != 1.0)\n\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n        assert all(out.reshape(1, -1).squeeze() <= 1)\n        summed_result = torch.sum(out, dim=1)\n        summed_result = summed_result.reshape(1, -1).squeeze()\n        summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))\n        assert all( summed_result == 1.0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n        assert all(out.reshape(1, -1).squeeze() <= 1)\n        summed_result = torch.sum(out, dim=1)\n        summed_result = summed_result.reshape(1, -1).squeeze()\n        summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))\n        assert all( summed_result == 1.0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert not all(out.reshape(1, -1).squeeze() >= 0)\n\n        assert not all(out.reshape(1, -1).squeeze() <= 0)\n        summed_result = torch.sum(out, dim=1)\n        summed_result = summed_result.reshape(1, -1).squeeze()\n        summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))\n        assert not all(summed_result == 1.0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25], [\"linear\", 8]],\n                           hidden_activations=\"relu\", input_dim=input_dim,\n                           initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert not all(out.reshape(1, -1).squeeze() >= 0)\n        assert not all(out.reshape(1, -1).squeeze() <= 0)\n        summed_result = torch.sum(out, dim=1)\n        summed_result = summed_result.reshape(1, -1).squeeze()\n        summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))\n        assert not all( summed_result == 1.0)\n\n\ndef test_output_activation_return_return_final_seq_only_off():\n    \"\"\"Tests whether network outputs data that has gone through correct activation function\"\"\"\n    RANDOM_ITERATIONS = 20\n    input_dim = 100\n    for _ in range(RANDOM_ITERATIONS):\n        data = torch.randn((25, 10, 100))\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\", input_dim=input_dim, return_final_seq_only=False,\n                           output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5]],\n                           hidden_activations=\"relu\",  input_dim=input_dim, return_final_seq_only=False,\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\", input_dim=input_dim, return_final_seq_only=False,\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\", input_dim=input_dim, return_final_seq_only=False,\n                           output_activation=\"sigmoid\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n        assert all(out.reshape(1, -1).squeeze() <= 1)\n        summed_result = torch.sum(out, dim=2)\n        assert all(summed_result.reshape(1, -1).squeeze() != 1.0)\n\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\", input_dim=input_dim, return_final_seq_only=False,\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n        assert all(out.reshape(1, -1).squeeze() <= 1)\n        summed_result = torch.sum(out, dim=2)\n        summed_result = summed_result.reshape(1, -1).squeeze()\n        summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))\n        assert all( summed_result == 1.0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\", input_dim=input_dim, return_final_seq_only=False,\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert all(out.reshape(1, -1).squeeze() >= 0)\n        assert all(out.reshape(1, -1).squeeze() <= 1)\n        summed_result = torch.sum(out, dim=2)\n        summed_result = summed_result.reshape(1, -1).squeeze()\n        summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))\n\n\n\n        assert all( summed_result == 1.0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\", input_dim=input_dim, return_final_seq_only=False,\n                           initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert not all(out.reshape(1, -1).squeeze() >= 0)\n\n        assert not all(out.reshape(1, -1).squeeze() <= 0)\n        summed_result = torch.sum(out, dim=2)\n        summed_result = summed_result.reshape(1, -1).squeeze()\n        summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))\n        assert not all( summed_result == 1.0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25], [\"linear\", 8]],\n                           hidden_activations=\"relu\", input_dim=input_dim, return_final_seq_only=False,\n                           initialiser=\"xavier\")\n        out = RNN_instance.forward(data)\n        assert not all(out.reshape(1, -1).squeeze() >= 0)\n        assert not all(out.reshape(1, -1).squeeze() <= 0)\n        summed_result = torch.sum(out, dim=2)\n        summed_result = summed_result.reshape(1, -1).squeeze()\n        summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))\n        assert not all( summed_result == 1.0)\n\ndef test_y_range():\n    \"\"\"Tests whether setting a y range works correctly\"\"\"\n    for _ in range(100):\n        val1 = random.random() - 3.0*random.random()\n        val2 = random.random() + 2.0*random.random()\n        lower_bound = min(val1, val2)\n        upper_bound = max(val1, val2)\n        rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\", y_range=(lower_bound, upper_bound),\n                           initialiser=\"xavier\", input_dim=22)\n        random_data = torch.randn((10, 11, 22))\n        out = rnn.forward(random_data)\n        out = out.reshape(1, -1).squeeze()\n        assert torch.sum(out > lower_bound).item() == 25*10, \"lower {} vs. {} \".format(lower_bound, out)\n        assert torch.sum(out < upper_bound).item() == 25*10, \"upper {} vs. {} \".format(upper_bound, out)\n\ndef test_deals_with_None_activation():\n    \"\"\"Tests whether is able to handle user inputting None as output activation\"\"\"\n    assert RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\", output_activation=None,\n                           initialiser=\"xavier\", input_dim=5)\n\ndef test_check_input_data_into_forward_once():\n    \"\"\"Tests that check_input_data_into_forward_once method only runs once\"\"\"\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                       hidden_activations=\"relu\", input_dim=5,\n                       output_activation=\"relu\", initialiser=\"xavier\")\n\n    data_not_to_throw_error = torch.randn((1, 4, 5))\n    data_to_throw_error = torch.randn((1, 2, 20))\n\n    with pytest.raises(AssertionError):\n        rnn.forward(data_to_throw_error)\n    with pytest.raises(RuntimeError):\n        rnn.forward(data_not_to_throw_error)\n        rnn.forward(data_to_throw_error)\n\ndef test_y_range_user_input():\n    \"\"\"Tests whether network rejects invalid y_range inputs\"\"\"\n    invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)]\n    for y_range_value in invalid_y_range_inputs:\n        with pytest.raises(AssertionError):\n            print(y_range_value)\n            rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\", y_range=y_range_value, input_dim=5,\n                           initialiser=\"xavier\")\n\ndef solves_simple_problem(X, y, nn_instance):\n    \"\"\"Checks if a given network is able to solve a simple problem\"\"\"\n    optimizer = optim.Adam(nn_instance.parameters(), lr=0.15)\n    for ix in range(800):\n        out = nn_instance.forward(X)\n        loss = torch.sum((out.squeeze() - y) ** 2) / N\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n    print(\"LOSS \", loss)\n    return loss < 0.1\n\ndef test_model_trains():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    for output_activation in [\"sigmoid\", \"None\"]:\n        rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]], input_dim=15,\n                           hidden_activations=\"relu\", output_activation=output_activation,\n                           initialiser=\"xavier\")\n        assert solves_simple_problem(X, y, rnn)\n\n    z = X[:, 0:1, 3:4] > 5.0\n    z =  torch.cat([z ==1, z==0], dim=1).float()\n    z = z.squeeze(-1).squeeze(-1)\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 2]], input_dim=15,\n                           hidden_activations=\"relu\", output_activation=\"softmax\", dropout=0.01,\n                           initialiser=\"xavier\")\n    assert solves_simple_problem(X, z, rnn)\n\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"linear\", 1]], input_dim=15,\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, rnn)\n\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"linear\", 20], [\"linear\", 1]], input_dim=15,\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\", batch_norm=True)\n    assert solves_simple_problem(X, y, rnn)\n\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]], input_dim=15,\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, rnn)\n\ndef test_error_when_provide_negative_data_for_embedding():\n    \"\"\"Tests that it raises an error if we try to do an embedding on negative data\"\"\"\n    N = 250\n    X = torch.randn((N, 5, 15))\n    X[0:125, 0, 3] += 20.0\n    y = X[:, 0, 3] > 5.0\n    y = y.float()\n    with pytest.raises(AssertionError):\n        rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]], input_dim=15,\n                  hidden_activations=\"relu\", output_activation=\"sigmoid\", columns_of_data_to_be_embedded=[0],\n                  embedding_dimensions=[[200, 5]], initialiser=\"xavier\")\n        assert solves_simple_problem(X, y, rnn)\n\n    X[:, :, 0] = abs(X[:, :, 0]).long()\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]], input_dim=15,\n              hidden_activations=\"relu\", output_activation=\"sigmoid\", columns_of_data_to_be_embedded=[0],\n              embedding_dimensions=[[200, 5]], initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, rnn)\n\ndef test_embedding_layers():\n    \"\"\"Tests whether create_embedding_layers method works correctly\"\"\"\n    for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)):\n        nn_instance = RNN(input_dim=15, layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]],\n                         embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]])\n        for layer in nn_instance.embedding_layers:\n            assert isinstance(layer, nn.Embedding)\n        assert len(nn_instance.embedding_layers) == 2\n        assert nn_instance.embedding_layers[0].num_embeddings == embedding_in_dim_1\n        assert nn_instance.embedding_layers[0].embedding_dim == embedding_out_dim_1\n        assert nn_instance.embedding_layers[1].num_embeddings == embedding_in_dim_2\n        assert nn_instance.embedding_layers[1].embedding_dim == embedding_out_dim_2\n\ndef test_model_trains_with_embeddings():\n    \"\"\"Tests that model trains when using embeddings\"\"\"\n    N = 250\n    X = torch.randn((N, 5, 15))\n    X[0:125, 0, 3] += 20.0\n    y = X[:, 0, 3] > 5.0\n    y = y.float()\n\n    Z = copy.deepcopy(X)\n    Z[:, :, 0] = abs(Z[:, :, 0]).long()\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]], input_dim=15,\n              hidden_activations=\"relu\", output_activation=\"sigmoid\", columns_of_data_to_be_embedded=[0],\n              embedding_dimensions=[[200, 5]], initialiser=\"xavier\")\n    assert solves_simple_problem(Z, y, rnn)\n\n    Z = copy.deepcopy(X)\n    Z[:, :, 0:2] = abs(Z[:, :, 0:2]).long()\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]], input_dim=15,\n              hidden_activations=\"relu\", output_activation=\"sigmoid\", columns_of_data_to_be_embedded=[0, 1],\n              embedding_dimensions=[[200, 5], [50, 3]], initialiser=\"xavier\")\n    assert solves_simple_problem(Z, y, rnn)\n\n    Z = copy.deepcopy(X)\n    Z[:, :, 3] = abs(Z[:, :, 3]).long()\n    Z[:, :, 6] = abs(Z[:, :, 6]).long()\n    Z[:, :, 4] = abs(Z[:, :, 4]).long()\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]], input_dim=15,\n              hidden_activations=\"relu\", output_activation=\"sigmoid\", columns_of_data_to_be_embedded=[3, 6, 4],\n              embedding_dimensions=[[200, 5], [50, 3], [50, 12]], initialiser=\"xavier\")\n    assert solves_simple_problem(Z, y, rnn)\n\ndef test_dropout():\n    \"\"\"Tests whether dropout layer reads in probability correctly\"\"\"\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=\"sigmoid\", dropout=0.9999,\n                           initialiser=\"xavier\", input_dim=15)\n    assert rnn.dropout_layer.p == 0.9999\n    assert not solves_simple_problem(X, y, rnn)\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=None, dropout=0.0000001,\n                           initialiser=\"xavier\", input_dim=15)\n    assert rnn.dropout_layer.p == 0.0000001\n    assert solves_simple_problem(X, y, rnn)\n\n\ndef test_all_activations_work():\n    \"\"\"Tests that all activations get accepted\"\"\"\n    nn_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=None, dropout=0.0000001,\n                           initialiser=\"xavier\", input_dim=15)\n    for key in nn_instance.str_to_activations_converter.keys():\n        if key == \"none\": hidden_key = \"relu\"\n        else: hidden_key = key\n        model = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=hidden_key, output_activation=key, dropout=0.0000001,\n                           initialiser=\"xavier\", input_dim=15)\n        model(X)\n\ndef test_all_initialisers_work():\n    \"\"\"Tests that all initialisers get accepted\"\"\"\n    nn_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=None, dropout=0.0000001,\n                           initialiser=\"xavier\", input_dim=15)\n    for key in nn_instance.str_to_initialiser_converter.keys():\n        model = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           dropout=0.0000001,\n                           initialiser=key, input_dim=15)\n        model(X)\n\ndef test_output_shapes():\n    \"\"\"Tests whether network outputs of correct shape\"\"\"\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 3]],\n              hidden_activations=\"relu\", initialiser=\"xavier\", input_dim=15)\n    output = rnn(X)\n    assert output.shape == (N, 3)\n\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n              hidden_activations=\"relu\", initialiser=\"xavier\", return_final_seq_only=False, input_dim=15)\n    output = rnn(X)\n    assert output.shape == (N, 5, 7)\n\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"lstm\", 3]],\n              hidden_activations=\"relu\", initialiser=\"xavier\", input_dim=15)\n    output = rnn(X)\n    assert output.shape == (N, 3)\n\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"lstm\", 7]],\n              hidden_activations=\"relu\", initialiser=\"xavier\", return_final_seq_only=False, input_dim=15)\n    output = rnn(X)\n    assert output.shape == (N, 5, 7)\n\ndef test_return_final_seq_user_input_valid():\n    \"\"\"Checks whether network only accepts a valid boolean value for return_final_seq_only\"\"\"\n    for valid_case in [True, False]:\n        assert RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n                  hidden_activations=\"relu\", initialiser=\"xavier\", return_final_seq_only=valid_case, input_dim=15)\n\n    for invalid_case in [[True], 22, [1, 3], (True, False), (5, False)]:\n        with pytest.raises(AssertionError):\n            print(invalid_case)\n            RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n                hidden_activations=\"relu\", initialiser=\"xavier\", return_final_seq_only=invalid_case, input_dim=15)\n\ndef test_print_model_summary():\n    nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"lstm\", 7]],\n              hidden_activations=\"relu\", initialiser=\"xavier\", return_final_seq_only=False, input_dim=15)\n    nn_instance.print_model_summary()\n\n\ndef test_output_heads_error_catching():\n    \"\"\"Tests that having multiple output heads catches errors from user inputs\"\"\"\n    output_dims_that_should_break = [[\"linear\", 2, 2, \"SAME\", \"conv\", 3, 4, \"SAME\"], [[[\"lstm\", 3], [\"gru\", 4]]],\n                                     [[2, 8]], [-33, 33, 33, 33, 33]]\n    for output_dim in output_dims_that_should_break:\n        with pytest.raises(AssertionError):\n            RNN(input_dim=5, layers_info=[[\"gru\", 20], [\"lstm\", 8], output_dim],\n                hidden_activations=\"relu\", output_activation=\"relu\")\n    output_activations_that_should_break = [\"relu\", [\"relu\"], [\"relu\", \"softmax\"]]\n    for output_activation in output_activations_that_should_break:\n        with pytest.raises(AssertionError):\n            RNN(input_dim=5, layers_info=[[\"gru\", 20], [\"lstm\", 8], [[\"linear\", 5], [\"linear\", 2], [\"linear\", 5]]],\n               hidden_activations=\"relu\", output_activation=output_activation)\n\ndef test_output_head_layers():\n    \"\"\"Tests whether the output head layers get created properly\"\"\"\n    for output_dim in [[[\"linear\", 3],[\"linear\", 9]], [[\"linear\", 4], [\"linear\", 20]], [[\"linear\", 1], [\"linear\", 1]]]:\n        nn_instance = RNN(input_dim=5, layers_info=[[\"gru\", 20], [\"lstm\", 8], output_dim],\n                          hidden_activations=\"relu\", output_activation=[\"softmax\", None])\n        assert nn_instance.output_layers[0].out_features == output_dim[0][1]\n        assert nn_instance.output_layers[0].in_features == 8\n        assert nn_instance.output_layers[1].out_features == output_dim[1][1]\n        assert nn_instance.output_layers[1].in_features == 8\n\ndef test_output_head_activations_work():\n    \"\"\"Tests that output head activations work properly\"\"\"\n\n    output_dim = [[\"linear\", 5], [\"linear\", 10], [\"linear\", 3]]\n    nn_instance = RNN(input_dim=5, layers_info=[[\"gru\", 20], [\"lstm\", 8], output_dim],\n                          hidden_activations=\"relu\", output_activation=[\"softmax\", None, \"relu\"])\n\n    x = torch.randn((20, 12, 5)) * -20.0\n    out = nn_instance(x)\n    assert out.shape == (20, 18)\n    sums = torch.sum(out[:, :5], dim=1).detach().numpy()\n    sums_others = torch.sum(out[:, 5:], dim=1).detach().numpy()\n    sums_others_2 = torch.sum(out[:, 5:15], dim=1).detach().numpy()\n    sums_others_3 = torch.sum(out[:, 15:18], dim=1).detach().numpy()\n\n\n    for row in range(out.shape[0]):\n        assert np.round(sums[row], 4) == 1.0, sums[row]\n        assert not np.round(sums_others[row], 4) == 1.0, sums_others[row]\n        assert not np.round(sums_others_2[row], 4) == 1.0, sums_others_2[row]\n        assert not np.round(sums_others_3[row], 4) == 1.0, sums_others_3[row]\n        for col in range(3):\n            assert out[row, 15 + col] >= 0.0, out[row, 15 + col]\n\ndef test_output_head_shapes_correct():\n    \"\"\"Tests that the output shape of network is correct when using multiple outpout heads\"\"\"\n    N = 20\n    X = torch.randn((N, 10, 4)) * -20.0\n    for _ in range(25):\n        nn_instance = RNN(input_dim=4,\n            layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1], [\"linear\", 12]],\n            hidden_activations=\"relu\")\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 12\n\n    for output_dim in [[ [\"linear\", 10], [\"linear\", 4], [\"linear\", 6]], [[\"linear\", 3], [\"linear\", 8], [\"linear\", 9]]]:\n        nn_instance = RNN(input_dim=4,\n            layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1], [\"linear\", 12], output_dim],\n            hidden_activations=\"relu\", output_activation=[\"softmax\", None, \"relu\"])\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 20\n"
  },
  {
    "path": "tests/tensorflow_tests/test_tf_CNN.py",
    "content": "# Run from home directory with python -m pytest tests\nimport shutil\nimport pytest\nimport random\nimport numpy as np\nimport tensorflow as tf\nimport torch.nn as nn\nfrom nn_builder.tensorflow.CNN import CNN\nfrom tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, BatchNormalization, MaxPool2D, AveragePooling2D\n\n\nN = 250\nX = np.random.random((N, 5, 5, 1))\nX[0:125, 3, 3, 0] += 20.0\ny = X[:, 3, 3, 0] > 5.0\n\ndef test_user_hidden_layers_input_rejections():\n    \"\"\"Tests whether network rejects invalid hidden_layers inputted from user\"\"\"\n    inputs_that_should_fail = [ ('maxpool', 3, 3, 3) , ['maxpool', 33, 22, 33], [['a']], [[222, 222, 222, 222]], [[\"conv\", 2, 2, -1]], [[\"conv\", 2, 2]], [[\"conv\", 2, 2, 55, 999, 33]],\n                                [[\"maxpool\", 33, 33]], [[\"maxpool\", -1, 33]], [[\"maxpool\", 33]], [[\"maxpoolX\", 1, 33]],\n                                [[\"cosnv\", 2, 2]], [[\"avgpool\", 33, 33, 333, 99]], [[\"avgpool\", -1, 33]], [[\"avgpool\", 33]], [[\"avgpoolX\", 1, 33]],\n                                [[\"adaptivemaxpool\", 33, 33, 333, 33]], [[\"adaptivemaxpool\", 2]], [[\"adaptivemaxpool\", 33]], [[\"adaptivemaxpoolX\"]],\n                                [[\"adaptiveavgpool\", 33, 33, 333, 11]], [[\"adaptiveavgpool\", 2]], [[\"adaptiveavgpool\", 33]],\n                                [[\"adaptiveavgpoolX\"]], [[\"linear\", 40, -2]], [[\"lineafr\", 40, 2]]]\n    for input in inputs_that_should_fail:\n        print(input)\n        with pytest.raises(AssertionError):\n            CNN(layers_info=input, hidden_activations=\"relu\",\n                output_activation=\"relu\")\n\ndef test_user_hidden_layers_input_acceptances():\n    \"\"\"Tests whether network rejects invalid hidden_layers inputted from user\"\"\"\n    inputs_that_should_work = [[[\"conv\", 2, 2, 3331, \"VALID\"]], [[\"CONV\", 2, 2, 3331, \"SAME\"]], [[\"ConV\", 2, 2, 3331, \"valid\"]],\n                               [[\"maxpool\", 2, 2, \"same\"]], [[\"MAXPOOL\", 2, 2, \"Valid\"]], [[\"MaXpOOL\", 2, 2, \"SAme\"]],\n                               [[\"avgpool\", 2, 2, \"saME\"]], [[\"AVGPOOL\", 2, 2, \"vaLID\"]], [[\"avGpOOL\", 2, 2, \"same\"]],\n                               [[\"linear\", 40]], [[\"lineaR\", 2]], [[\"LINEAR\", 2]]]\n\n    for ix, input in enumerate(inputs_that_should_work):\n        input.append([\"linear\", 5])\n        CNN(layers_info=input, hidden_activations=\"relu\",\n            output_activation=\"relu\")\n\ndef test_hidden_layers_created_correctly():\n    \"\"\"Tests that create_hidden_layers works correctly\"\"\"\n    layers = [[\"conv\", 2, 4, 3, \"same\"], [\"maxpool\", 3, 4, \"vaLID\"], [\"avgpool\", 32, 42, \"vaLID\"],\n               [\"linear\", 22], [\"linear\", 2222], [\"linear\", 5]]\n\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n\n    assert type(cnn.hidden_layers[0]) == Conv2D\n    assert cnn.hidden_layers[0].filters == 2\n    assert cnn.hidden_layers[0].kernel_size == (4, 4)\n    assert cnn.hidden_layers[0].strides == (3, 3)\n    assert cnn.hidden_layers[0].padding == \"same\"\n\n    assert type(cnn.hidden_layers[1]) == MaxPool2D\n    assert cnn.hidden_layers[1].pool_size == (3, 3)\n    assert cnn.hidden_layers[1].strides == (4, 4)\n    assert cnn.hidden_layers[1].padding == \"valid\"\n\n    assert type(cnn.hidden_layers[2]) == AveragePooling2D\n    assert cnn.hidden_layers[2].pool_size == (32, 32)\n    assert cnn.hidden_layers[2].strides == (42, 42)\n    assert cnn.hidden_layers[2].padding == \"valid\"\n\n    assert type(cnn.hidden_layers[3]) == Dense\n    assert cnn.hidden_layers[3].units == 22\n\n    assert type(cnn.hidden_layers[4]) == Dense\n    assert cnn.hidden_layers[4].units == 2222\n\n    assert type(cnn.output_layers[0]) == Dense\n    assert cnn.output_layers[0].units == 5\n\n\ndef test_output_layers_created_correctly():\n    \"\"\"Tests that create_output_layers works correctly\"\"\"\n    layers = [[\"conv\", 2, 4, 3, \"valid\"], [\"maxpool\", 3, 4, \"same\"], [\"avgpool\", 32, 42, \"valid\"],\n              [\"linear\", 22], [\"linear\", 2222], [\"linear\", 2]]\n\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\", output_activation=\"relu\")\n    assert cnn.output_layers[0].units == 2\n\n    layers = [[\"conv\", 2, 4, 3,\"valid\"], [\"maxpool\", 3, 4, \"same\"], [\"avgpool\", 32, 42, \"same\"], [\"linear\", 7]]\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n    assert cnn.output_layers[0].units == 7\n\n    layers = [[\"conv\", 5, 4, 3, \"valid\"], [\"maxpool\", 3, 4, \"valid\"], [\"avgpool\", 32, 42, \"valid\"], [\"linear\", 6]]\n    cnn = CNN( layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n    assert cnn.output_layers[0].units == 6\n\n    layers = [[\"conv\", 5, 4, 3, \"valid\"], [\"maxpool\", 3, 4, \"valid\"], [\"avgpool\", 32, 42, \"valid\"],\n              [[\"linear\", 6], [\"linear\", 22]]]\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\",\n              output_activation=[\"softmax\", None])\n    assert cnn.output_layers[0].units == 6\n    assert cnn.output_layers[1].units == 22\n\ndef test_output_dim_user_input():\n    \"\"\"Tests whether network rejects an invalid output_dim input from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            CNN(layers_info=[2, input_value], hidden_activations=\"relu\",  output_activation=\"relu\")\n        with pytest.raises(AssertionError):\n            CNN(layers_info=input_value, hidden_activations=\"relu\", output_activation=\"relu\")\n\ndef test_activations_user_input():\n    \"\"\"Tests whether network rejects an invalid hidden_activations or output_activation from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            CNN(layers_info=[[\"conv\", 2, 2, 3331, \"valid\"], [\"linear\", 5] ], hidden_activations=input_value,\n                output_activation=\"relu\")\n            CNN(layers_info=[[\"conv\", 2, 2, 3331, \"valid\"], [\"linear\", 3]], hidden_activations=\"relu\",\n                output_activation=input_value)\n\ndef test_initialiser_user_input():\n    \"\"\"Tests whether network rejects an invalid initialiser from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            CNN(layers_info=[[\"conv\", 2, 2, 3331, \"valid\"], [\"linear\", 3]], hidden_activations=\"relu\",\n                output_activation=\"relu\", initialiser=input_value)\n\n        CNN(layers_info=[[\"conv\", 2, 2, 3331, \"valid\"], [\"linear\", 3]], hidden_activations=\"relu\",\n            output_activation=\"relu\", initialiser=\"xavier\")\n\ndef test_batch_norm_layers():\n    \"\"\"Tests whether batch_norm_layers method works correctly\"\"\"\n    layers =[[\"conv\", 2, 4, 3, \"valid\"], [\"maxpool\", 3, 4, \"valid\"], [\"linear\", 5]]\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\",\n            output_activation=\"relu\", initialiser=\"xavier\", batch_norm=False)\n\n    layers = [[\"conv\", 2, 4, 3, \"valid\"], [\"maxpool\", 3, 4, \"valid\"], [\"linear\", 5]]\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n    assert len(cnn.batch_norm_layers) == 1\n    assert isinstance(cnn.batch_norm_layers[0], tf.keras.layers.BatchNormalization)\n\n    layers = [[\"conv\", 2, 4, 3, \"valid\"], [\"maxpool\", 3, 4, \"valid\"], [\"conv\", 12, 4, 3, \"valid\"], [\"linear\", 22], [\"linear\", 55]]\n    cnn = CNN(layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n    assert len(cnn.batch_norm_layers) == 3\n    for layer in cnn.batch_norm_layers:\n        assert isinstance(layer, tf.keras.layers.BatchNormalization)\n\ndef test_linear_layers_acceptance():\n    \"\"\"Tests that only accepts linear layers of correct shape\"\"\"\n    layers_that_shouldnt_work = [[[\"linear\", 2, 5]], [[\"linear\", 2, 5, 5]], [[\"linear\"]], [[\"linear\", 2], [\"linear\", 5, 4]],\n                                 [\"linear\", 0], [\"linear\", -5]]\n    for layers in layers_that_shouldnt_work:\n        with pytest.raises(AssertionError):\n            cnn = CNN(layers_info=layers, hidden_activations=\"relu\",\n                      output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n    layers_that_should_work = [[[\"linear\", 44], [\"linear\", 2]], [[\"linear\", 22]]]\n    for layer in layers_that_should_work:\n        assert CNN(layers_info=layer, hidden_activations=\"relu\",\n                      output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\ndef test_linear_layers_only_come_at_end():\n    \"\"\"Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they\n    don't only come at the end\"\"\"\n    layers = [[\"conv\", 2, 4, 3, \"valid\"], [\"linear\", 55], [\"maxpool\", 3, 4, \"valid\"]]\n    with pytest.raises(AssertionError):\n        cnn = CNN(layers_info=layers, hidden_activations=\"relu\",\n                  output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\n    layers = [[\"conv\", 2, 4, 3, \"valid\"], [\"linear\", 55]]\n    assert CNN(layers_info=layers, hidden_activations=\"relu\",\n                      output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\n    layers = [[\"conv\", 2, 4, 3, \"valid\"], [\"linear\", 55], [\"linear\", 55], [\"linear\", 55]]\n    assert CNN(layers_info=layers, hidden_activations=\"relu\",\n               output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\ndef test_output_activation():\n    \"\"\"Tests whether network outputs data that has gone through correct activation function\"\"\"\n    RANDOM_ITERATIONS = 20\n    input_dim = (100, 100, 5)\n    for _ in range(RANDOM_ITERATIONS):\n        data = np.random.random((1, *input_dim))\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, \"valid\"],  [\"linear\", 50]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = CNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 20, 1, \"same\"], [\"linear\", 5]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = CNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 5, 20, 1, \"same\"], [\"linear\", 5]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = CNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 5, 2, 1, \"valid\"], [\"linear\",  22]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"sigmoid\", initialiser=\"xavier\")\n        out = CNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n        assert all(tf.reshape(out, [-1]) <= 1)\n        assert not np.round(tf.reduce_sum(out, axis=1), 3) == 1.0\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, \"same\"], [\"linear\", 5]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = CNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n        assert all(tf.reshape(out, [-1]) <= 1)\n        assert np.round(tf.reduce_sum(out, axis=1), 3) == 1.0\n\n\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, \"valid\"], [\"linear\", 5]],\n                           hidden_activations=\"relu\",\n                           initialiser=\"xavier\")\n        out = CNN_instance(data)\n        assert not all(tf.reshape(out, [-1]) >= 0)\n        assert not np.round(tf.reduce_sum(out, axis=1), 3) == 1.0\n\n\ndef test_y_range():\n    \"\"\"Tests whether setting a y range works correctly\"\"\"\n    for _ in range(100):\n        val1 = random.random() - 3.0*random.random()\n        val2 = random.random() + 2.0*random.random()\n        lower_bound = min(val1, val2)\n        upper_bound = max(val1, val2)\n        CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, \"valid\"], [\"linear\", 5]],\n                           hidden_activations=\"relu\", y_range=(lower_bound, upper_bound),\n                           initialiser=\"xavier\")\n        random_data = np.random.random((10, 1, 20, 20))\n        out = CNN_instance(random_data)\n        assert all(tf.reshape(out, [-1]) > lower_bound)\n        assert all(tf.reshape(out, [-1]) < upper_bound)\n\ndef test_deals_with_None_activation():\n    \"\"\"Tests whether is able to handle user inputting None as output activation\"\"\"\n    assert CNN(layers_info=[[\"conv\", 2, 2, 1, \"valid\"], [\"linear\", 5]],\n                           hidden_activations=\"relu\", output_activation=None,\n                           initialiser=\"xavier\")\n\ndef test_y_range_user_input():\n    \"\"\"Tests whether network rejects invalid y_range inputs\"\"\"\n    invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)]\n    for y_range_value in invalid_y_range_inputs:\n        with pytest.raises(AssertionError):\n            print(y_range_value)\n            CNN_instance = CNN(layers_info=[[\"conv\", 2, 2, 1, \"valid\"], [\"linear\", 5]],\n                           hidden_activations=\"relu\", y_range=y_range_value,\n                           initialiser=\"xavier\")\n\ndef test_model_trains():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    for output_activation in [\"sigmoid\", \"None\"]:\n        CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"],  [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=output_activation,\n                           initialiser=\"xavier\")\n        print(CNN_instance.hidden_layers[0].kernel_size)\n        assert solves_simple_problem(X, y, CNN_instance)\n\n\ndef test_model_trains_part_2():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    z = X[:, 3:4, 3:4, 0:1] > 5.0\n    z = np.concatenate([z == 1, z == 0], axis=1)\n    z = z.reshape((-1, 2))\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 2]],\n                           hidden_activations=\"relu\", output_activation=\"softmax\", dropout=0.01,\n                           initialiser=\"xavier\")\n    assert solves_simple_problem(X, z, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"same\"], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\", batch_norm=True)\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"maxpool\", 1, 1, \"same\"], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"same\"], [\"avgpool\", 1, 1, \"same\"], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 5, 3, 1, \"same\"], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"conv\", 5, 3, 1, \"valid\"], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n\ndef solves_simple_problem(X, y, nn_instance):\n    \"\"\"Checks if a given network is able to solve a simple problem\"\"\"\n    nn_instance.compile(optimizer='adam',\n                  loss='mse')\n    nn_instance.fit(X, y, epochs=800)\n    results = nn_instance.evaluate(X, y)\n    print(\"FINAL RESULT \", results)\n    return results < 0.1\n\n\ndef test_model_trains_linear_layer():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    CNN_instance = CNN(layers_info=[[\"conv\", 5, 3, 1, \"valid\"], [\"linear\", 5], [\"linear\", 5], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=\"sigmoid\",\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\n    CNN_instance = CNN(layers_info=[[\"linear\", 5], [\"linear\", 5], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=\"sigmoid\",\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, CNN_instance)\n\ndef test_max_pool_working():\n    \"\"\"Tests whether max pool layers work properly\"\"\"\n    N = 250\n    X = np.random.random((N, 8, 8, 1))\n    X[0:125, 3, 3, 0] = 999.99\n    CNN_instance = CNN(layers_info=[[\"maxpool\", 2, 2, \"valid\"], [\"maxpool\", 2, 2, \"valid\"], [\"maxpool\", 2, 2, \"valid\"], [\"linear\", 1]],\n                       hidden_activations=\"relu\",\n                       initialiser=\"xavier\")\n    assert CNN_instance(X).shape == (N, 1)\n\ndef test_dropout():\n    \"\"\"Tests whether dropout layer reads in probability correctly\"\"\"\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=\"sigmoid\", dropout=0.9999,\n                           initialiser=\"xavier\")\n    assert CNN_instance.dropout_layer.rate == 0.9999\n    assert not solves_simple_problem(X, y, CNN_instance)\n    CNN_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=None, dropout=0.0000001,\n                           initialiser=\"xavier\")\n    assert CNN_instance.dropout_layer.rate == 0.0000001\n    assert solves_simple_problem(X, y, CNN_instance)\n\ndef test_MNIST_progress():\n    \"\"\"Tests that network made using CNN module can make progress on MNIST\"\"\"\n    mnist = tf.keras.datasets.mnist\n\n    (x_train, y_train), (x_test, y_test) = mnist.load_data()\n\n    x_train, x_test = x_train / 255.0, x_test / 255.0\n\n    # Add a channels dimension\n    x_train = x_train[..., tf.newaxis]\n    x_test = x_test[..., tf.newaxis]\n\n    # Create model using nn_builder\n    model = CNN(layers_info=[[\"conv\", 32, 3, 1, \"valid\"], [\"maxpool\", 2, 2, \"valid\"], [\"conv\", 64, 3, 1, \"valid\"],\n                             [\"linear\", 10]],\n                hidden_activations=\"relu\", output_activation=\"softmax\", dropout=0.0,\n                initialiser=\"xavier\", batch_norm=True)\n\n    model.compile(optimizer='adam',\n                  loss='sparse_categorical_crossentropy',\n                  metrics=['accuracy'])\n    model.fit(x_train, y_train, epochs=2, batch_size=64)\n\n\n    model.evaluate(x_test, y_test)\n\n    results = model.evaluate(x_test, y_test)\n    assert results[1] > 0.9\n\n    model = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 10]],\n                           hidden_activations=\"relu\", output_activation=\"softmax\", dropout=0.9999,\n                           initialiser=\"xavier\")\n    model.compile(optimizer='adam',\n                  loss='sparse_categorical_crossentropy',\n                  metrics=['accuracy'])\n\n    model.fit(x_train, y_train, epochs=2, batch_size=64)\n\n    model.evaluate(x_test, y_test)\n\n    results = model.evaluate(x_test, y_test)\n    assert not results[1] > 0.9\n\n    model = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 10]],\n                           hidden_activations=\"relu\", output_activation=\"softmax\", dropout=0.0,\n                           initialiser=\"xavier\")\n    model.compile(optimizer='adam',\n                  loss='sparse_categorical_crossentropy',\n                  metrics=['accuracy'])\n\n    model.fit(x_train, y_train, epochs=2, batch_size=64)\n\n    model.evaluate(x_test, y_test)\n\n    results = model.evaluate(x_test, y_test)\n    assert results[1] > 0.9\n\ndef test_all_activations_work():\n    \"\"\"Tests that all activations get accepted\"\"\"\n    nn_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1]],\n                                   dropout=0.0000001,\n                                   initialiser=\"xavier\")\n    for key in nn_instance.str_to_activations_converter.keys():\n        model = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1]],\n                                   hidden_activations=key, output_activation=key, dropout=0.0000001,\n                                   initialiser=\"xavier\")\n        model(X)\n\ndef test_all_initialisers_work():\n    \"\"\"Tests that all initialisers get accepted\"\"\"\n    nn_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1]],\n                                   dropout=0.0000001,\n                                   initialiser=\"xavier\")\n    for key in nn_instance.str_to_initialiser_converter.keys():\n        print(key)\n        model = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1]],\n                                    dropout=0.0000001,\n                                   initialiser=key)\n        model(X)\n\ndef test_print_model_summary():\n    nn_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1]],\n                      dropout=0.0000001, batch_norm=True,\n                      initialiser=\"xavier\")\n    nn_instance.print_model_summary((64, 11, 11, 3))\n\ndef test_output_heads_error_catching():\n    \"\"\"Tests that having multiple output heads catches errors from user inputs\"\"\"\n    output_dims_that_should_break = [[\"linear\", 2, 2, \"SAME\", \"conv\", 3, 4, \"SAME\"], [[[\"conv\", 3, 2, \"same\"], [\"linear\", 4]]],\n                                     [[2, 8]], [-33, 33, 33, 33, 33]]\n    for output_dim in output_dims_that_should_break:\n        with pytest.raises(AssertionError):\n            CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1], output_dim],\n                hidden_activations=\"relu\", output_activation=\"relu\")\n    output_activations_that_should_break = [\"relu\", [\"relu\"], [\"relu\", \"softmax\"]]\n    for output_activation in output_activations_that_should_break:\n        with pytest.raises(AssertionError):\n            CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1],  [[\"linear\", 4], [\"linear\", 10], [\"linear\", 4]]],\n               hidden_activations=\"relu\", output_activation=output_activation)\n\n\ndef test_output_head_layers():\n    \"\"\"Tests whether the output head layers get created properly\"\"\"\n    for output_dim in [[[\"linear\", 3],[\"linear\", 9]], [[\"linear\", 4], [\"linear\", 20]], [[\"linear\", 1], [\"linear\", 1]]]:\n        nn_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1], output_dim],\n                          hidden_activations=\"relu\", output_activation=[\"softmax\", None])\n        assert nn_instance.output_layers[0].units == output_dim[0][1]\n        assert nn_instance.output_layers[1].units == output_dim[1][1]\n\ndef test_output_head_activations_work():\n    \"\"\"Tests that output head activations work properly\"\"\"\n\n    output_dim = [[\"linear\", 5], [\"linear\", 10], [\"linear\", 3]]\n    nn_instance = CNN(layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1], output_dim],\n                          hidden_activations=\"relu\", output_activation=[\"softmax\", None, \"relu\"])\n    x = np.random.random((20, 10, 10, 4)) * -20.0\n    out = nn_instance(x)\n\n    assert out.shape == (20, 18)\n\n    sums = tf.reduce_sum(out[:, :5], axis=1)\n    sums_others = tf.reduce_sum(out[:, 5:], axis=1)\n    sums_others_2 = tf.reduce_sum(out[:, 5:15], axis=1)\n    sums_others_3 = tf.reduce_sum(out[:, 15:18], axis=1)\n\n\n    for row in range(out.shape[0]):\n        assert tf.math.equal(np.round(sums[row], 4), 1.0), sums[row]\n        assert not tf.math.equal(np.round(sums_others[row], 4), 1.0), np.round(sums_others[row], 4)\n        assert not tf.math.equal(np.round(sums_others_2[row], 4), 1.0), np.round(sums_others_2[row], 4)\n        assert not tf.math.equal(np.round(sums_others_3[row], 4), 1.0), np.round(sums_others_3[row], 4)\n        for col in range(3):\n            assert out[row, 15 + col] >= 0.0, out[row, 15 + col]\n\ndef test_output_head_shapes_correct():\n    \"\"\"Tests that the output shape of network is correct when using multiple outpout heads\"\"\"\n    N = 20\n    X = np.random.random((N, 25, 25, 2))\n    for _ in range(25):\n        nn_instance = CNN(\n            layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1], [\"linear\", 12]],\n            hidden_activations=\"relu\")\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 12\n\n    for output_dim in [[ [\"linear\", 10], [\"linear\", 4], [\"linear\", 6]], [[\"linear\", 3], [\"linear\", 8], [\"linear\", 9]]]:\n        nn_instance = CNN(\n            layers_info=[[\"conv\", 25, 5, 1, \"valid\"], [\"conv\", 25, 5, 1, \"valid\"], [\"linear\", 1], output_dim],\n            hidden_activations=\"relu\", output_activation=[\"softmax\", None, \"relu\"])\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 20\n\n"
  },
  {
    "path": "tests/tensorflow_tests/test_tf_NN.py",
    "content": "# Run from home directory with python -m pytest tests\nimport pytest\nimport copy\nimport random\nimport numpy as np\nimport tensorflow as tf\nfrom nn_builder.tensorflow.NN import NN\nimport tensorflow.keras.initializers as initializers\nimport tensorflow.keras.activations as activations\n\nN = 250\nX = (np.random.random((N, 5)) - 0.5) * 2.0\nX[:, [2, 4]] += 10.0\ny = X[:, 0] > 0 * 1.0\n\ndef test_linear_hidden_units_user_input():\n    \"\"\"Tests whether network rejects an invalid linear_hidden_units input from user\"\"\"\n    inputs_that_should_fail = [\"a\", [\"a\", \"b\"], [2, 4, \"ss\"], [-2], 2]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            NN( layers_info=input_value, hidden_activations=\"relu\", output_activation=\"relu\")\n\ndef test_activations_user_input():\n    \"\"\"Tests whether network rejects an invalid hidden_activations or output_activation from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            NN( layers_info=[2], hidden_activations=input_value,\n               output_activation=\"relu\")\n            NN( layers_info=[2], hidden_activations=\"relu\",\n               output_activation=input_value)\n\ndef test_initialiser_user_input():\n    \"\"\"Tests whether network rejects an invalid initialiser from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            NN( layers_info=[2], hidden_activations=\"relu\",\n               output_activation=\"relu\", initialiser=input_value)\n        NN( layers_info=[2], hidden_activations=\"relu\",\n           output_activation=\"relu\", initialiser=\"xavier\")\n\ndef test_output_shape_correct():\n    \"\"\"Tests whether network returns output of the right shape\"\"\"\n    input_dims = [x for x in range(1, 3)]\n    output_dims = [x for x in range(4, 6)]\n    linear_hidden_units_options = [ [2, 3, 4], [2, 9, 1], [55, 55, 55, 234, 15]]\n    for input_dim, output_dim, linear_hidden_units in zip(input_dims, output_dims, linear_hidden_units_options):\n        linear_hidden_units.append(output_dim)\n        nn_instance = NN(layers_info=linear_hidden_units, hidden_activations=\"relu\",\n                         output_activation=\"relu\", initialiser=\"xavier\")\n        data = 2.0 * (np.random.random((25, input_dim)) - 0.5)\n        output = nn_instance(data)\n        assert output.shape == (25, output_dim)\n\ndef test_output_activation():\n    \"\"\"Tests whether network outputs data that has gone through correct activation function\"\"\"\n    RANDOM_ITERATIONS = 20\n    for _ in range(RANDOM_ITERATIONS):\n        data = 2.0 * (np.random.random((1, 100)) - 0.5)\n        nn_instance = NN(layers_info=[5, 5, 5],\n                         hidden_activations=\"relu\",\n                         output_activation=\"relu\", initialiser=\"xavier\")\n        out = nn_instance(data)\n        assert all(tf.squeeze(out) >= 0)\n\n        nn_instance = NN(layers_info=[5, 5, 5],\n                         hidden_activations=\"relu\",\n                         output_activation=\"sigmoid\", initialiser=\"xavier\")\n        out = nn_instance(data)\n        assert all(tf.squeeze(out) >= 0)\n        assert all(tf.squeeze(out) <= 1)\n\n        nn_instance = NN(layers_info=[5, 5, 5],\n                         hidden_activations=\"relu\",\n                         output_activation=\"softmax\", initialiser=\"xavier\")\n        out = nn_instance(data)\n        assert all(tf.squeeze(out) >= 0)\n        assert all(tf.squeeze(out) <= 1)\n        assert np.round(tf.reduce_sum(tf.squeeze(out)), 3) == 1.0\n\n        nn_instance = NN(layers_info=[5, 5, 5],\n                         hidden_activations=\"relu\")\n\n        out = nn_instance(data)\n        assert not all(tf.squeeze(out) >= 0)\n        assert not np.round(tf.reduce_sum(tf.squeeze(out)), 3) == 1.0\n\ndef test_linear_layers_info():\n    \"\"\"Tests whether create_hidden_layers_info method works correctly\"\"\"\n    for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]):\n        hidden_units.append(output_dim)\n        print(hidden_units)\n        nn_instance = NN(layers_info=copy.copy(hidden_units),\n                         hidden_activations=\"relu\",\n                         output_activation=\"softmax\", initialiser=\"xavier\")\n        print(hidden_units)\n        assert len(nn_instance.hidden_layers) == len(hidden_units) - 1\n        for layer_ix in range(len(hidden_units) - 1):\n            layer = nn_instance.hidden_layers[layer_ix]\n            print(nn_instance.hidden_layers[layer_ix])\n            assert type(layer) == tf.keras.layers.Dense\n            assert layer.units == hidden_units[layer_ix]\n            assert layer.kernel_initializer == initializers.glorot_uniform, layer.kernel_initializer\n            assert layer.activation == activations.relu\n\n        output_layer = nn_instance.output_layers[0]\n        assert type(output_layer) == tf.keras.layers.Dense\n        assert output_layer.units == hidden_units[-1]\n        assert output_layer.kernel_initializer == initializers.glorot_uniform\n        assert output_layer.activation == activations.softmax\n\ndef test_embedding_layers():\n    \"\"\"Tests whether create_embedding_layers_info method works correctly\"\"\"\n    for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)):\n        nn_instance = NN( layers_info=[5], columns_of_data_to_be_embedded=[0, 1],\n                         embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]])\n        for layer in nn_instance.embedding_layers:\n            assert isinstance(layer, tf.keras.layers.Embedding)\n        assert len(nn_instance.embedding_layers) == 2\n        assert nn_instance.embedding_layers[0].input_dim == embedding_in_dim_1\n        assert nn_instance.embedding_layers[0].output_dim == embedding_out_dim_1\n        assert nn_instance.embedding_layers[1].input_dim == embedding_in_dim_2\n        assert nn_instance.embedding_layers[1].output_dim == embedding_out_dim_2\n\ndef test_incorporate_embeddings():\n    \"\"\"Tests the method incorporate_embeddings\"\"\"\n    X_new = X\n    X_new[:, [2, 4]] = tf.round(X_new[:, [2, 4]])\n    nn_instance = NN( layers_info=[10],\n                     columns_of_data_to_be_embedded=[2, 4],\n                     embedding_dimensions=[[50, 3],\n                                                       [55, 4]])\n    out = nn_instance.incorporate_embeddings(X)\n    assert out.shape == (N, X.shape[1]+3+4-2)\n\ndef test_embedding_network_can_solve_simple_problem():\n    \"\"\"Tests whether network can solve simple problem using embeddings\"\"\"\n    X = (np.random.random((N, 5)) - 0.5) * 5.0 + 20.0\n    y = (X[:, 0] >= 20) * (X[:, 1] <= 20) * 1.0\n    nn_instance = NN( layers_info=[5, 1],\n                     columns_of_data_to_be_embedded=[0, 1],\n                     embedding_dimensions=[[50, 3],\n                                           [55, 3]])\n    assert solves_simple_problem(X, y, nn_instance)\n\ndef test_batch_norm_layers_info():\n    \"\"\"Tests whether batch_norm_layers_info method works correctly\"\"\"\n    for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]):\n        hidden_units.append(output_dim)\n        nn_instance = NN( layers_info=hidden_units,\n                         hidden_activations=\"relu\", batch_norm=True,\n                         output_activation=\"relu\", initialiser=\"xavier\")\n        for layer in nn_instance.batch_norm_layers:\n            assert isinstance(layer, tf.keras.layers.BatchNormalization)\n        assert len(nn_instance.batch_norm_layers) == len(hidden_units) - 1\n\ndef test_model_trains():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    for output_activation in [\"sigmoid\", \"None\"]:\n        nn_instance = NN( layers_info=[10, 10, 10, 1],\n                         output_activation=output_activation, dropout=0.01, batch_norm=True)\n        assert solves_simple_problem(X, y, nn_instance)\n    z = X[:, 0:1] > 0\n    z =  np.concatenate([z ==1, z==0], axis=1)\n    nn_instance = NN(layers_info=[10, 10, 10, 2],\n                     output_activation=\"softmax\", dropout=0.01, batch_norm=True)\n    assert solves_simple_problem(X, z, nn_instance)\n\ndef solves_simple_problem(X, y, nn_instance):\n    \"\"\"Checks if a given network is able to solve a simple problem\"\"\"\n    nn_instance.compile(optimizer='adam',\n                  loss='mse')\n    nn_instance.fit(X, y, epochs=800)\n    results = nn_instance.evaluate(X, y)\n    print(\"FINAL RESULT \", results)\n    return results < 0.1\n\ndef test_dropout():\n    \"\"\"Tests whether dropout layer reads in probability correctly\"\"\"\n    nn_instance = NN(layers_info=[10, 10, 1], dropout=0.9999)\n    assert nn_instance.dropout_layer.rate == 0.9999\n    assert not solves_simple_problem(X, y, nn_instance)\n    nn_instance = NN( layers_info=[10, 10, 1], dropout=0.00001)\n    assert nn_instance.dropout_layer.rate == 0.00001\n    assert solves_simple_problem(X, y, nn_instance)\n\ndef test_y_range_user_input():\n    \"\"\"Tests whether network rejects invalid y_range inputs\"\"\"\n    invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)]\n    for y_range_value in invalid_y_range_inputs:\n        with pytest.raises(AssertionError):\n            print(y_range_value)\n            nn_instance = NN( layers_info=[10, 10, 3],\n                             y_range=y_range_value)\n\ndef test_y_range():\n    \"\"\"Tests whether setting a y range works correctly\"\"\"\n    for _ in range(100):\n        val1 = random.random() - 3.0*random.random()\n        val2 = random.random() + 2.0*random.random()\n        lower_bound = min(val1, val2)\n        upper_bound = max(val1, val2)\n        nn_instance = NN( layers_info=[10, 10, 3],  y_range=(lower_bound, upper_bound))\n        random_data = 2.0 * (np.random.random((15, 5)) - 0.5)\n        out = nn_instance(random_data)\n        assert np.sum(out > lower_bound) == 3*15, \"lower {} vs. {} \".format(lower_bound, out)\n        assert np.sum(out < upper_bound) == 3*15, \"upper {} vs. {} \".format(upper_bound, out)\n\ndef test_deals_with_None_activation():\n    \"\"\"Tests whether is able to handle user inputting None as output activation\"\"\"\n    assert NN( layers_info=[10, 10, 3], output_activation=None)\n\ndef test_all_activations_work():\n    \"\"\"Tests that all activations get accepted\"\"\"\n    nn_instance = NN( layers_info=[10, 10, 1], dropout=0.9999)\n    for key in nn_instance.str_to_activations_converter.keys():\n        model = NN(layers_info=[10, 10, 1], dropout=0.9999, hidden_activations=key, output_activation=key)\n        model(X)\n\ndef test_all_initialisers_work():\n    \"\"\"Tests that all initialisers get accepted\"\"\"\n    nn_instance = NN(layers_info=[10, 10, 1], dropout=0.9999)\n    for key in nn_instance.str_to_initialiser_converter.keys():\n        model = NN(layers_info=[10, 10, 1], dropout=0.9999, initialiser=key)\n        model(X)\n\ndef test_print_model_summary():\n    nn_instance = NN(layers_info=[10, 10, 1])\n    nn_instance.print_model_summary((64, 11))\n\ndef test_output_heads_error_catching():\n    \"\"\"Tests that having multiple output heads catches errors from user inputs\"\"\"\n    output_dims_that_should_break = [[[2, 8]], [-33, 33, 33, 33, 33]]\n    for output_dim in output_dims_that_should_break:\n        with pytest.raises(AssertionError):\n            NN(layers_info=[4, 7, 9, output_dim], hidden_activations=\"relu\",\n               output_activation=\"relu\")\n\n    output_activations_that_should_break = [\"relu\", [\"relu\"], [\"relu\", \"softmax\"]]\n    for output_activation in output_activations_that_should_break:\n        with pytest.raises(AssertionError):\n            NN(layers_info=[4, 7, 9, [4, 6, 1]], hidden_activations=\"relu\",\n               output_activation=output_activation)\n\ndef test_output_head_layers():\n    \"\"\"Tests whether the output head layers get created properly\"\"\"\n    for output_dim in [[3, 9], [4, 20], [1, 1]]:\n        nn_instance = NN(layers_info=[4, 7, 9, output_dim], hidden_activations=\"relu\",\n                         output_activation=[\"softmax\", None])\n        assert nn_instance.output_layers[0].units == output_dim[0]\n        assert nn_instance.output_layers[1].units == output_dim[1]\n\ndef test_output_head_activations_work():\n    \"\"\"Tests that output head activations work properly\"\"\"\n    nn_instance = NN(layers_info=[4, 7, 9, [5, 10, 3]], hidden_activations=\"relu\",\n                     output_activation=[\"softmax\", None, \"relu\"])\n\n    x = np.random.random((20, 2)) * -20.0\n    out = nn_instance(x)\n\n    assert out.shape == (20, 18)\n\n    sums = tf.reduce_sum(out[:, :5], axis=1)\n    sums_others = tf.reduce_sum(out[:, 5:], axis=1)\n    sums_others_2 = tf.reduce_sum(out[:, 5:15], axis=1)\n    sums_others_3 = tf.reduce_sum(out[:, 15:18], axis=1)\n\n\n    for row in range(out.shape[0]):\n        assert tf.math.equal(np.round(sums[row], 4), 1.0), sums[row]\n        assert not tf.math.equal(np.round(sums_others[row], 4), 1.0), np.round(sums_others[row], 4)\n        assert not tf.math.equal(np.round(sums_others_2[row], 4), 1.0), np.round(sums_others_2[row], 4)\n        assert not tf.math.equal(np.round(sums_others_3[row], 4), 1.0), np.round(sums_others_3[row], 4)\n        for col in range(3):\n            assert out[row, 15 + col] >= 0.0, out[row, 15 + col]\n\ndef test_output_head_shapes_correct():\n    \"\"\"Tests that the output shape of network is correct when using multiple outpout heads\"\"\"\n    N = 20\n    X = np.random.random((N, 2))\n    for _ in range(25):\n        output_dim = random.randint(1, 100)\n        nn_instance = NN(layers_info=[4, 7, 9, output_dim], hidden_activations=\"relu\")\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == output_dim\n\n    for output_dim in [[3, 9, 5, 3], [5, 5, 5, 5], [2, 1, 1, 16]]:\n        nn_instance = NN(layers_info=[4, 7, 9, output_dim], hidden_activations=\"relu\",\n                         output_activation=[\"softmax\", None, None, \"relu\"])\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 20\n\ndef test_boston_housing_progress():\n    \"\"\"Tests that network made using CNN module can make progress on MNIST\"\"\"\n    boston_housing = tf.keras.datasets.boston_housing\n    (x_train, y_train), (x_test, y_test) = boston_housing.load_data()\n\n    model = NN(layers_info=[30, 10, 1],\n                hidden_activations=\"relu\", output_activation=None, dropout=0.0,\n                initialiser=\"xavier\", batch_norm=True, y_range=(4.5, 55.0))\n\n    model.compile(optimizer='adam',loss='mse')\n    model.fit(x_train, y_train, epochs=200, batch_size=64)\n    results = model.evaluate(x_test, y_test)\n    assert results < 35\n\n    model = NN(layers_info=[30, 10, 1],\n                hidden_activations=\"relu\", output_activation=None, dropout=0.0,\n                initialiser=\"xavier\", batch_norm=False)\n\n    model.compile(optimizer='adam',loss='mse')\n    model.fit(x_train, y_train, epochs=200, batch_size=64)\n    results = model.evaluate(x_test, y_test)\n    assert results < 30\n\n    model = NN(layers_info=[30, 10, 1],\n                hidden_activations=\"relu\", output_activation=None, dropout=0.0,\n                initialiser=\"xavier\", batch_norm=True)\n\n    model.compile(optimizer='adam', loss='mse')\n    model.fit(x_train, y_train, epochs=200, batch_size=64)\n    results = model.evaluate(x_test, y_test)\n    assert results < 30\n\n    model = NN(layers_info=[150, 50, 1],\n                hidden_activations=\"relu\", output_activation=None, dropout=0.05,\n                initialiser=\"xavier\", batch_norm=False)\n\n    model.compile(optimizer='adam', loss='mse')\n    model.fit(x_train, y_train, epochs=200, batch_size=64)\n    results = model.evaluate(x_test, y_test)\n    assert results < 30"
  },
  {
    "path": "tests/tensorflow_tests/test_tf_RNN.py",
    "content": "# Run from home directory with python -m pytest tests\nimport pytest\nimport random\nimport tensorflow as tf\nimport numpy as np\nfrom tensorflow.keras.layers import Dense, Concatenate, BatchNormalization, GRU, LSTM\nfrom nn_builder.tensorflow.RNN import RNN\n\nN = 250\nX = np.random.random((N, 3, 5))\nX = X.astype('float32')\n\nX[0:125, :, 3] += 10.0\ny = X[:, 2, 3] > 5.0\n\ndef test_user_hidden_layers_input_rejections():\n    \"\"\"Tests whether network rejects invalid hidden_layers inputted from user\"\"\"\n    inputs_that_should_fail = [[[\"linearr\", 33]], [[\"linear\", 12, 33]], [[\"gru\", 2, 33]], [[\"lstm\", 2, 33]], [[\"lstmr\", 33]],\n                               [[\"gruu\", 33]], [[\"gru\", 33], [\"xxx\", 33]], [[\"linear\", 33], [\"gru\", 12], [\"gru\", 33]] ]\n    for input in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            RNN(layers_info=input, hidden_activations=\"relu\",\n                output_activation=\"relu\")\n\ndef test_user_hidden_layers_input_acceptances():\n    \"\"\"Tests whether network rejects invalid hidden_layers inputted from user\"\"\"\n    inputs_that_should_work = [[[\"linear\", 33]], [[\"linear\", 12]], [[\"gru\", 2]], [[\"lstm\", 2]], [[\"lstm\", 1]],\n                               [[\"gru\", 330]], [[\"gru\", 33], [\"linear\", 2]] ]\n    for input in inputs_that_should_work:\n        assert  RNN(layers_info=input, hidden_activations=\"relu\",\n                output_activation=\"relu\")\n\n\ndef test_hidden_layers_created_correctly():\n    \"\"\"Tests that create_hidden_layers works correctly\"\"\"\n    layers = [[\"gru\", 25], [\"lstm\", 23], [\"linear\", 5], [\"linear\", 10]]\n\n    rnn = RNN(layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n\n    assert type(rnn.hidden_layers[0]) == GRU\n    assert rnn.hidden_layers[0].units == 25\n\n    assert type(rnn.hidden_layers[1]) == LSTM\n    assert rnn.hidden_layers[1].units == 23\n\n    assert type(rnn.hidden_layers[2]) == Dense\n    assert rnn.hidden_layers[2].units == 5\n\n    assert type(rnn.output_layers[0]) == Dense\n    assert rnn.output_layers[0].units == 10\n\n\ndef test_output_layers_created_correctly():\n    \"\"\"Tests that create_output_layers works correctly\"\"\"\n    layers = [[\"gru\", 25], [\"lstm\", 23], [\"linear\", 5], [\"linear\", 10]]\n\n    rnn = RNN(layers_info=layers, hidden_activations=\"relu\", output_activation=\"relu\")\n    assert rnn.output_layers[0].units == 10\n\n    layers = [[\"gru\", 25], [\"lstm\", 23], [\"lstm\", 10]]\n    rnn = RNN(layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\")\n    assert rnn.output_layers[0].units == 10\n\n    layers = [[\"gru\", 25], [\"lstm\", 23], [[\"lstm\", 10], [\"linear\", 15]]]\n    rnn = RNN(layers_info=layers, hidden_activations=\"relu\",\n              output_activation=[\"relu\", \"softmax\"])\n    assert rnn.output_layers[0].units == 10\n    assert rnn.output_layers[1].units == 15\n\ndef test_output_dim_user_input():\n    \"\"\"Tests whether network rejects an invalid output_dim input from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            RNN(layers_info=[2, input_value], hidden_activations=\"relu\",  output_activation=\"relu\")\n        with pytest.raises(AssertionError):\n            RNN(layers_info=input_value, hidden_activations=\"relu\", output_activation=\"relu\")\n\ndef test_activations_user_input():\n    \"\"\"Tests whether network rejects an invalid hidden_activations or output_activation from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            RNN(layers_info=[[\"linear\", 2]], hidden_activations=input_value,\n                output_activation=\"relu\")\n            RNN(layers_info=[[\"linear\", 2]], hidden_activations=\"relu\",\n                output_activation=input_value)\n\ndef test_initialiser_user_input():\n    \"\"\"Tests whether network rejects an invalid initialiser from user\"\"\"\n    inputs_that_should_fail = [-1, \"aa\", [\"dd\"], [2], 0, 2.5, {2}, \"Xavier_\"]\n    for input_value in inputs_that_should_fail:\n        with pytest.raises(AssertionError):\n            RNN(layers_info=[[\"linear\", 2]], hidden_activations=\"relu\",\n                output_activation=\"relu\", initialiser=input_value)\n\n            RNN(layers_info=[[\"linear\", 2], [\"linear\", 2]], hidden_activations=\"relu\",\n            output_activation=\"relu\", initialiser=\"xavier\")\n\ndef test_batch_norm_layers():\n    \"\"\"Tests whether batch_norm_layers method works correctly\"\"\"\n    layers = [[\"gru\", 20], [\"lstm\", 3], [\"linear\", 4], [\"linear\", 10]]\n    rnn = RNN(layers_info=layers, hidden_activations=\"relu\",\n              output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n    assert len(rnn.batch_norm_layers) == 3\n    for layer in rnn.batch_norm_layers:\n        assert isinstance(layer, BatchNormalization)\n\ndef test_linear_layers_only_come_at_end():\n    \"\"\"Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they\n    don't only come at the end\"\"\"\n    layers = [[\"gru\", 20],  [\"linear\", 4], [\"lstm\", 3], [\"linear\", 10]]\n    with pytest.raises(AssertionError):\n        rnn = RNN(layers_info=layers, hidden_activations=\"relu\",\n                  output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\n    layers = [[\"gru\", 20], [\"lstm\", 3],  [\"linear\", 4], [\"linear\", 10]]\n    assert RNN(layers_info=layers, hidden_activations=\"relu\",\n                      output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n\ndef test_output_activation():\n    \"\"\"Tests whether network outputs data that has gone through correct activation function\"\"\"\n    RANDOM_ITERATIONS = 20\n    for _ in range(RANDOM_ITERATIONS):\n        data = np.random.random((25, 10, 30))\n        data = data.astype('float32')\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"sigmoid\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n        assert all(tf.reshape(out, [-1]) <= 1)\n\n        summed_result = tf.reduce_sum(out, axis=1)\n        summed_result = tf.reshape(summed_result, [-1, 1])\n        assert summed_result != 1.0\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n        assert all(tf.reshape(out, [-1]) <= 1)\n        summed_result = tf.reduce_sum(out, axis=1)\n        assert (np.round(summed_result, 3) == 1.0).all()\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n        assert all(tf.reshape(out, [-1]) <= 1)\n        summed_result = tf.reduce_sum(out, axis=1)\n        assert (np.round(summed_result, 3) == 1.0).all()\n\n        RNN_instance = RNN(layers_info=[[\"linear\", 20], [\"linear\", 50]],\n                           hidden_activations=\"relu\")\n\n        out = RNN_instance(data)\n        assert not all(tf.reshape(out, [-1]) >= 0)\n        assert not all(tf.reshape(out, [-1]) <= 1)\n        summed_result = tf.reduce_sum(out, axis=1)\n        assert not (np.round(summed_result, 3) == 1.0).all()\n\n        RNN_instance = RNN(layers_info=[ [\"lstm\", 25], [\"linear\", 10]],\n                           hidden_activations=\"relu\")\n\n        out = RNN_instance(data)\n        assert not all(tf.reshape(out, [-1]) >= 0)\n        assert not all(tf.reshape(out, [-1]) <= 0)\n        summed_result = tf.reduce_sum(out, axis=1)\n        assert not (np.round(summed_result, 3) == 1.0).all()\n\n\ndef test_output_activation():\n    \"\"\"Tests whether network outputs data that has gone through correct activation function\"\"\"\n    RANDOM_ITERATIONS = 20\n    for _ in range(RANDOM_ITERATIONS):\n        data = np.random.random((25, 10, 30))\n        data = data.astype('float32')\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"relu\", initialiser=\"xavier\", batch_norm=True)\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"relu\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"sigmoid\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n        assert all(tf.reshape(out, [-1]) <= 1)\n\n        summed_result = tf.reduce_sum(out, axis=1)\n        summed_result = tf.reshape(summed_result, [-1, 1])\n        assert summed_result != 1.0\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"linear\", 10], [\"linear\", 3]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n        assert all(tf.reshape(out, [-1]) <= 1)\n        summed_result = tf.reduce_sum(out, axis=1)\n        assert (np.round(summed_result, 3) == 1.0).all()\n\n        RNN_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\",\n                           output_activation=\"softmax\", initialiser=\"xavier\")\n        out = RNN_instance(data)\n        assert all(tf.reshape(out, [-1]) >= 0)\n        assert all(tf.reshape(out, [-1]) <= 1)\n        summed_result = tf.reduce_sum(out, axis=1)\n        assert (np.round(summed_result, 3) == 1.0).all()\n\n        RNN_instance = RNN(layers_info=[[\"linear\", 20], [\"linear\", 50]],\n                           hidden_activations=\"relu\")\n\n        out = RNN_instance(data)\n        assert not all(tf.reshape(out, [-1]) >= 0)\n        assert not all(tf.reshape(out, [-1]) <= 1)\n        summed_result = tf.reduce_sum(out, axis=1)\n        assert not (np.round(summed_result, 3) == 1.0).all()\n\n        RNN_instance = RNN(layers_info=[ [\"lstm\", 25], [\"linear\", 10]],\n                           hidden_activations=\"relu\")\n\n        out = RNN_instance(data)\n        assert not all(tf.reshape(out, [-1]) >= 0)\n        assert not all(tf.reshape(out, [-1]) <= 0)\n        summed_result = tf.reduce_sum(out, axis=1)\n        assert not (np.round(summed_result, 3) == 1.0).all()\n\ndef test_y_range():\n    \"\"\"Tests whether setting a y range works correctly\"\"\"\n    for _ in range(20):\n        val1 = random.random() - 3.0*random.random()\n        val2 = random.random() + 2.0*random.random()\n        lower_bound = min(val1, val2)\n        upper_bound = max(val1, val2)\n        rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                  hidden_activations=\"relu\", y_range=(lower_bound, upper_bound), initialiser=\"xavier\")\n        random_data = np.random.random((10, 11, 22))\n        random_data = random_data.astype('float32')\n        out = rnn(random_data)\n        assert all(tf.reshape(out, [-1]) > lower_bound)\n        assert all(tf.reshape(out, [-1]) < upper_bound)\n\ndef test_deals_with_None_activation():\n    \"\"\"Tests whether is able to handle user inputting None as output activation\"\"\"\n    assert RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\", output_activation=None,\n                           initialiser=\"xavier\")\n\ndef test_y_range_user_input():\n    \"\"\"Tests whether network rejects invalid y_range inputs\"\"\"\n    invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)]\n    for y_range_value in invalid_y_range_inputs:\n        with pytest.raises(AssertionError):\n            print(y_range_value)\n            rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 5], [\"lstm\", 25]],\n                           hidden_activations=\"relu\", y_range=y_range_value,\n                           initialiser=\"xavier\")\n\n\ndef solves_simple_problem(X, y, nn_instance):\n    \"\"\"Checks if a given network is able to solve a simple problem\"\"\"\n    print(\"X shape \", X.shape)\n    print(\"y shape \", y.shape)\n    nn_instance.compile(optimizer='adam',\n                  loss='mse')\n    nn_instance.fit(X, y, epochs=25)\n    results = nn_instance.evaluate(X, y)\n    print(\"FINAL RESULT \", results)\n    return results < 0.1\n\ndef test_model_trains():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    for output_activation in [\"sigmoid\", \"None\"]:\n        rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=output_activation,\n                           initialiser=\"xavier\")\n\n        assert solves_simple_problem(X, y, rnn)\n\ndef test_model_trains_part_2():\n    \"\"\"Tests whether a small range of networks can solve a simple task\"\"\"\n    z = X[:, 2:3, 3:4] > 5.0\n    z = np.concatenate([z == 1, z == 0], axis=1)\n    z = z.reshape((-1, 2))\n\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 2]],\n                           hidden_activations=\"relu\", output_activation=\"softmax\", dropout=0.01,\n                           initialiser=\"xavier\")\n    assert solves_simple_problem(X, z, rnn)\n\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, rnn)\n\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\")\n    assert solves_simple_problem(X, y, rnn)\n\ndef test_model_trains_with_batch_norm():\n    \"\"\"Tests whether a model with batch norm on can solve a simple task\"\"\"\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"linear\", 20], [\"linear\", 1]],\n                       hidden_activations=\"relu\", output_activation=None,\n                       initialiser=\"xavier\", batch_norm=True)\n    assert solves_simple_problem(X, y, rnn)\n\ndef test_dropout():\n    \"\"\"Tests whether dropout layer reads in probability correctly\"\"\"\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=\"sigmoid\", dropout=0.9999,\n                           initialiser=\"xavier\")\n    assert rnn.dropout_layer.rate == 0.9999\n    assert not solves_simple_problem(X, y, rnn)\n    rnn = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=None, dropout=0.0000001,\n                           initialiser=\"xavier\")\n    assert rnn.dropout_layer.rate == 0.0000001\n    assert solves_simple_problem(X, y, rnn)\n\n\ndef test_all_activations_work():\n    \"\"\"Tests that all activations get accepted\"\"\"\n    nn_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=None, dropout=0.0000001,\n                           initialiser=\"xavier\")\n    for key in nn_instance.str_to_activations_converter.keys():\n        assert RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=key, output_activation=key, dropout=0.0000001,\n                           initialiser=\"xavier\")\n\ndef test_all_initialisers_work():\n    \"\"\"Tests that all initialisers get accepted\"\"\"\n    nn_instance = RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           hidden_activations=\"relu\", output_activation=None, dropout=0.0000001,\n                           initialiser=\"xavier\")\n    for key in nn_instance.str_to_initialiser_converter.keys():\n        assert RNN(layers_info=[[\"lstm\", 20], [\"gru\", 10], [\"linear\", 20], [\"linear\", 1]],\n                           dropout=0.0000001,\n                           initialiser=key)\n\ndef test_output_shapes():\n    \"\"\"Tests whether network outputs of correct shape\"\"\"\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 3]],\n              hidden_activations=\"relu\", initialiser=\"xavier\")\n    output = rnn(X)\n    assert output.shape == (N, 3)\n\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n              hidden_activations=\"relu\", initialiser=\"xavier\", return_final_seq_only=False)\n    output = rnn(X)\n    assert output.shape == (N, 3, 7)\n\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"lstm\", 3]],\n              hidden_activations=\"relu\", initialiser=\"xavier\")\n    output = rnn(X)\n    assert output.shape == (N, 3)\n\n    rnn = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"lstm\", 7]],\n              hidden_activations=\"relu\", initialiser=\"xavier\", return_final_seq_only=False)\n    output = rnn(X)\n    assert output.shape == (N, 3, 7)\n\ndef test_return_final_seq_user_input_valid():\n    \"\"\"Checks whether network only accepts a valid boolean value for return_final_seq_only\"\"\"\n    for valid_case in [True, False]:\n        assert RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n                  hidden_activations=\"relu\", initialiser=\"xavier\", return_final_seq_only=valid_case)\n\n    for invalid_case in [[True], 22, [1, 3], (True, False), (5, False)]:\n        with pytest.raises(AssertionError):\n            print(invalid_case)\n            RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n                hidden_activations=\"relu\", initialiser=\"xavier\", return_final_seq_only=invalid_case)\n\ndef test_embedding_layers():\n    \"\"\"Tests whether create_embedding_layers_info method works correctly\"\"\"\n    for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)):\n        nn_instance = RNN( layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]], columns_of_data_to_be_embedded=[0, 1],\n                         embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]])\n        for layer in nn_instance.embedding_layers:\n            assert isinstance(layer, tf.keras.layers.Embedding)\n        assert len(nn_instance.embedding_layers) == 2\n        assert nn_instance.embedding_layers[0].input_dim == embedding_in_dim_1\n        assert nn_instance.embedding_layers[0].output_dim == embedding_out_dim_1\n        assert nn_instance.embedding_layers[1].input_dim == embedding_in_dim_2\n        assert nn_instance.embedding_layers[1].output_dim == embedding_out_dim_2\n\ndef test_incorporate_embeddings():\n    \"\"\"Tests the method incorporate_embeddings\"\"\"\n    X_new = X\n    X_new[:, [0, 2], :] = tf.round(X_new[:, [0, 2], :])\n    nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n                     columns_of_data_to_be_embedded=[0, 2],\n                     embedding_dimensions=[[50, 3], [55, 4]])\n    out = nn_instance.incorporate_embeddings(X)\n    assert out.shape == (N, 3, 10)\n\n    nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n                     columns_of_data_to_be_embedded=[0, 1, 2],\n                     embedding_dimensions=[[50, 3], [55, 4], [55, 4]])\n    out = nn_instance.incorporate_embeddings(X)\n    assert out.shape == (N, 3, 13)\n\n    nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n                     columns_of_data_to_be_embedded=[2],\n                     embedding_dimensions=[[150, 30]])\n    out = nn_instance.incorporate_embeddings(X)\n    assert out.shape == (N, 3, 34)\n\ndef test_embedding_network_can_solve_simple_problem():\n    \"\"\"Tests whether network can solve simple problem using embeddings\"\"\"\n    X = (np.random.random((N, 4, 5)) - 0.5) * 5.0 + 20.0\n    y = (X[:, :, 0] >= 25) * (X[:, :, 1] <= 25) * 1.0\n    nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]],\n                     columns_of_data_to_be_embedded=[0, 1],\n                     embedding_dimensions=[[50, 3],\n                                           [55, 3]])\n    assert solves_simple_problem(X, y, nn_instance)\n\n    nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]],\n                      columns_of_data_to_be_embedded=[1],\n                      embedding_dimensions=[[55, 3]])\n    assert solves_simple_problem(X, y, nn_instance)\n\n    nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1]],\n                      columns_of_data_to_be_embedded=[1, 3, 0],\n                      embedding_dimensions=[[55, 3], [55, 5], [55, 2]])\n    assert solves_simple_problem(X, y, nn_instance)\n\n\ndef test_print_model_summary():\n    nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 7]],\n                     columns_of_data_to_be_embedded=[2],\n                     embedding_dimensions=[[150, 30]])\n    nn_instance.print_model_summary((64, 11, 11))\n\ndef test_output_heads_error_catching():\n    \"\"\"Tests that having multiple output heads catches errors from user inputs\"\"\"\n    output_dims_that_should_break = [[\"linear\", 2, 2, \"SAME\", \"conv\", 3, 4, \"SAME\"], [[[\"lstm\", 3], [\"gru\", 4]]],\n                                     [[2, 8]], [-33, 33, 33, 33, 33]]\n    for output_dim in output_dims_that_should_break:\n        with pytest.raises(AssertionError):\n            RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], output_dim],\n                hidden_activations=\"relu\", output_activation=\"relu\")\n    output_activations_that_should_break = [\"relu\", [\"relu\"], [\"relu\", \"softmax\"]]\n    for output_activation in output_activations_that_should_break:\n        with pytest.raises(AssertionError):\n            RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], [[\"linear\", 5], [\"linear\", 2], [\"linear\", 5]]],\n               hidden_activations=\"relu\", output_activation=output_activation)\n\n\ndef test_output_head_layers():\n    \"\"\"Tests whether the output head layers get created properly\"\"\"\n    for output_dim in [[[\"linear\", 3],[\"linear\", 9]], [[\"linear\", 4], [\"linear\", 20]], [[\"linear\", 1], [\"linear\", 1]]]:\n        nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], output_dim],\n                          hidden_activations=\"relu\", output_activation=[\"softmax\", None])\n        assert nn_instance.output_layers[0].units == output_dim[0][1]\n        assert nn_instance.output_layers[1].units == output_dim[1][1]\n\ndef test_output_head_activations_work():\n    \"\"\"Tests that output head activations work properly\"\"\"\n\n    output_dim = [[\"linear\", 5], [\"linear\", 10], [\"linear\", 3]]\n    nn_instance = RNN(layers_info=[[\"gru\", 20], [\"lstm\", 8], output_dim],\n                          hidden_activations=\"relu\", output_activation=[\"softmax\", None, \"relu\"])\n    x = np.random.random((20, 10, 4)) * -20.0\n    x = x.astype('float32')\n    out = nn_instance(x)\n\n    assert out.shape == (20, 18)\n\n    sums = tf.reduce_sum(out[:, :5], axis=1)\n    sums_others = tf.reduce_sum(out[:, 5:], axis=1)\n    sums_others_2 = tf.reduce_sum(out[:, 5:15], axis=1)\n    sums_others_3 = tf.reduce_sum(out[:, 15:18], axis=1)\n\n\n    for row in range(out.shape[0]):\n        assert tf.math.equal(np.round(sums[row], 4), 1.0), sums[row]\n        assert not tf.math.equal(np.round(sums_others[row], 4), 1.0), np.round(sums_others[row], 4)\n        assert not tf.math.equal(np.round(sums_others_2[row], 4), 1.0), np.round(sums_others_2[row], 4)\n        assert not tf.math.equal(np.round(sums_others_3[row], 4), 1.0), np.round(sums_others_3[row], 4)\n        for col in range(3):\n            assert out[row, 15 + col] >= 0.0, out[row, 15 + col]\n\ndef test_output_head_shapes_correct():\n    \"\"\"Tests that the output shape of network is correct when using multiple outpout heads\"\"\"\n    N = 20\n    X = np.random.random((N, 10, 4)) * -20.0\n    X = X.astype('float32')\n    for _ in range(25):\n        nn_instance = RNN(\n            layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1], [\"linear\", 12]],\n            hidden_activations=\"relu\")\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 12\n\n    for output_dim in [[ [\"linear\", 10], [\"linear\", 4], [\"linear\", 6]], [[\"linear\", 3], [\"linear\", 8], [\"linear\", 9]]]:\n        nn_instance = RNN(\n            layers_info=[[\"gru\", 20], [\"lstm\", 8], [\"linear\", 1], [\"linear\", 12], output_dim],\n            hidden_activations=\"relu\", output_activation=[\"softmax\", None, \"relu\"])\n        out = nn_instance(X)\n        assert out.shape[0] == N\n        assert out.shape[1] == 20\n"
  }
]