Repository: p-christ/nn_builder Branch: master Commit: d58182e579c9 Files: 30 Total size: 254.4 KB Directory structure: gitextract_38pgry39/ ├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── miscellaneous/ │ ├── material_for_readme/ │ │ ├── README.md │ │ ├── nn_builder_code │ │ ├── nn_builder_tf_code │ │ ├── non_nn_builder_code │ │ └── non_nn_builder_tf_code │ └── testreadme.rst ├── nn_builder/ │ ├── Overall_Base_Network.py │ ├── __init__.py │ ├── pytorch/ │ │ ├── Base_Network.py │ │ ├── CNN.py │ │ ├── NN.py │ │ ├── RNN.py │ │ └── __init__.py │ └── tensorflow/ │ ├── Base_Network.py │ ├── CNN.py │ ├── NN.py │ ├── RNN.py │ └── __init__.py ├── requirements.txt ├── setup.py └── tests/ ├── pytorch_tests/ │ ├── test_pytorch_CNN.py │ ├── test_pytorch_NN.py │ └── test_pytorch_RNN.py └── tensorflow_tests/ ├── test_tf_CNN.py ├── test_tf_NN.py └── test_tf_RNN.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ *.pyc *cache *.idea *__pycache__ *venv *nn_builder.egg-info *build *dist .DS_Store *to_do_list *.ipynb_checkpoints ================================================ FILE: .travis.yml ================================================ language: python python: 3.7 dist: xenial sudo: true install: - pip install -r requirements.txt -q - pip install sklearn - pip install torchvision script: - export PYTHONPATH="$PYTHONPATH:$PWD" - export PYTHONPATH="${PYTHONPATH}:/home/travis/build/p-christ/nn_builder/tests" - export PYTHONPATH="${PYTHONPATH}:/home/travis/build/p-christ/nn_builder" - python -m pytest /home/travis/build/p-christ/nn_builder/tests/ ================================================ FILE: LICENSE ================================================ Copyright (c) 2018 The Python Packaging Authority Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ [![Downloads](https://pepy.tech/badge/nn-builder)](https://pepy.tech/project/nn-builder) ![Image](https://travis-ci.org/p-christ/nn_builder.svg?branch=master) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/dwyl/esta/issues) ![nn_builder](miscellaneous/material_for_readme/nn_builder_new.png) **nn_builder lets you build neural networks with less boilerplate code**. You specify the type of network you want and it builds it. ### Install `pip install nn_builder` ### Support | Network Type | **NN** | **CNN** | **RNN** | | ------- | ------- | ------- | ------- | | PyTorch | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | TensorFlow 2.0 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | ### Examples On the left is how you can create the PyTorch neural network on the right in only 1 line of code using nn_builder: ![Screenshot](miscellaneous/material_for_readme/nn_builder_use_case.png) Similarly for TensorFlow on the left is how you can create the CNN on the right in only 1 line of code using nn_builder: ![Screenshot](miscellaneous/material_for_readme/tf_nn_builder_example.png) ### Usage See this [colab notebook](https://colab.research.google.com/drive/1UdMT3aVSV0L5Rq11nyLHxMSVtTVZryhW) for lots of examples of how to use the module. 3 types of PyTorch and TensorFlow network are currently supported: NN, CNN and RNN. Each network takes the following arguments: | Field | Description | Default | | :---: | :----------: | :---: | | *input_dim*| Dimension of the input into the network. See below for more detail. Not needed for Tensorflow. | N/A | | *layers_info* | List to indicate the layers of the network you want. Exact requirements depend on network type, see below for more detail | N/A | | *output_activation* | String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads | No activation | | *hidden_activations* | String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer), default is ReLU and for example "tanh" would have tanh applied on all hidden layer activations | ReLU after every hidden layer | | *dropout* | Float to indicate what dropout probability you want applied after each hidden layer | 0 | | *initialiser* | String to indicate which initialiser you want used to initialise all the parameters | PyTorch & TF Default | | *batch_norm* | Boolean to indicate whether you want batch norm applied to the output of every hidden layer | False | | *columns of_data_to_be_embedded* | List to indicate the column numbers of the data that you want to be put through an embedding layer before being fed through the hidden layers of the network | No embeddings | | *embedding_dimensions* | If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list of the form: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...] | No embeddings | | *y_range* | Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks | No range | | *random_seed* | Integer to indicate the random seed you want to use | 0 | | *return_final_seq_only* | Only needed for RNN. Boolean to indicate whether you only want to return the output for the final timestep (True) or if you want to return the output for all timesteps (False) | True | Each network type has slightly different requirements for **input_dim** and **layers_info** as explained below: --- ### 1. NN * **input_dim**: # Features in PyTorch, not needed for TensorFlow * **layers_info**: List of integers to indicate number of hidden units you want per linear layer. * For example: ``` from nn_builder.pytorch.NN import NN model = NN(input_dim=5, layers_info=[10, 10, 1], output_activation=None, hidden_activations="relu", dropout=0.0, initialiser="xavier", batch_norm=False) ``` --- ### 2. CNN * **input_dim**: (# Channels, Height, Width) in PyTorch, not needed for TensorFlow * **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a CNN can be one of these 4 forms: * ["conv", channels, kernel size, stride, padding] * ["maxpool", kernel size, stride, padding] * ["avgpool", kernel size, stride, padding] * ["linear", units] * For a PyTorch network kernel size, stride, padding and units must be integers. For TensorFlow they must all be integers except for padding which must be one of {“valid”, “same”} * For example: ``` from nn_builder.pytorch.CNN import CNN model = CNN(input_dim=(3, 64, 64), layers_info=[["conv", 32, 3, 1, 0], ["maxpool", 2, 2, 0], ["conv", 64, 3, 1, 2], ["avgpool", 2, 2, 0], ["linear", 10]], hidden_activations="relu", output_activation="softmax", dropout=0.0, initialiser="xavier", batch_norm=True) ``` --- ### 3. RNN * **input_dim**: # Features in PyTorch, not needed for TensorFlow * **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a CNN can be one of these 4 forms: * ["lstm", units] * ["gru", units] * ["linear", units] * For example: ``` from nn_builder.pytorch.CNN import CNN model = RNN(input_dim=5, layers_info=[["gru", 50], ["lstm", 10], ["linear", 2]], hidden_activations="relu", output_activation="softmax", batch_norm=False, dropout=0.0, initialiser="xavier") ``` --- ## Contributing Anyone is very welcome to contribute via a pull request. Please see the [issues](https://github.com/p-christ/nn_builder/issues) page for ideas on the best areas to contribute to and try to: 1. Add tests to the tests folder that cover any code you write 1. Write comments for every function 1. Create a colab notebook demonstrating how any extra functionality you created works To help you remember things you learn about machine learning in general checkout [Gizmo](https://gizmo.ai/landing/github_links) ================================================ FILE: miscellaneous/material_for_readme/README.md ================================================ [![Downloads](https://pepy.tech/badge/nn-builder)](https://pepy.tech/project/nn-builder) ![Image](https://travis-ci.org/p-christ/nn_builder.svg?branch=master) [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/dwyl/esta/issues) ![nn_builder](miscellaneous/material_for_readme/nn_builder_new.png) **nn_builder lets you build neural networks without boilerplate code**. You specify the type of network you want and it builds it. ### Install `pip install nn_builder` ### Support | Network Type | **NN** | **CNN** | **RNN** | | ------- | ------- | ------- | ------- | | PyTorch | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | TensorFlow 2.0 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | ### Examples On the left is how you can create the PyTorch neural network on the right in only 1 line of code using nn_builder: +-----------------------------------------------------------+----------------------------------------------------------+ | **Building a simple NN with nn_builder** | **Building a simple NN without nn_builder** | +-----------------------------------------------------------+----------------------------------------------------------+ | .. code:: python | .. code:: python | | # With nn_builder | # Without nn_builder | | from nn_builder.pytorch.NN import NN | import torch.nn as nn | | model = NN(input_dim=25, | class NN(nn.Module): | | layers=[150, 100, 50, 50, 50, 50, 5], | def __init__(self): | | output_activation="softmax", dropout=0.5, | nn.Module.__init__(self) | | initialiser="xavier", batch_norm=True) | self.fc1 = nn.Linear(25, 150) | | | self.fc2 = nn.Linear(150, 100) | | | self.fc3 = nn.Linear(100, 50) | | | self.fc4 = nn.Linear(50, 50) | | | self.fc5 = nn.Linear(50, 50) | | | self.fc6 = nn.Linear(50, 50) | | | self.fc7 = nn.Linear(50, 5) | | | | | | self.bn1 = nn.BatchNorm1d(150) | | | self.bn2 = nn.BatchNorm1d(100) | | | self.bn3 = nn.BatchNorm1d(50) | | | self.bn4 = nn.BatchNorm1d(50) | | | self.bn5 = nn.BatchNorm1d(50) | | | self.bn6 = nn.BatchNorm1d(50) | | | | | | self.dropout = nn.Dropout(p=0.5) | | | for params in [self.fc1, self.fc2, self.fc3, | | | self.fc4, self.fc5, self.fc6, | | | self.fc7]: | | | nn.init.xavier_uniform_(params.weight) | | | def forward(self, x): | | | x = self.dropout(self.bn1(nn.ReLU()(self.fc1(x)))) | | | x = self.dropout(self.bn2(nn.ReLU()(self.fc2(x)))) | | | x = self.dropout(self.bn3(nn.ReLU()(self.fc3(x)))) | | | x = self.dropout(self.bn4(nn.ReLU()(self.fc4(x)))) | | | x = self.dropout(self.bn5(nn.ReLU()(self.fc5(x)))) | | | x = self.dropout(self.bn6(nn.ReLU()(self.fc6(x)))) | | | x = self.fc7(x) | | | x = nn.Softmax(dim=1)(x) | | | return x | | | | | | model = NN() | +-----------------------------------------------------------+----------------------------------------------------------+ ![Screenshot](miscellaneous/material_for_readme/nn_builder_use_case.png) Similarly for TensorFlow on the left is how you can create the CNN on the right in only 1 line of code using nn_builder: ![Screenshot](miscellaneous/material_for_readme/tf_nn_builder_example.png) ### Usage See this [colab notebook](https://colab.research.google.com/drive/1UdMT3aVSV0L5Rq11nyLHxMSVtTVZryhW) for lots of examples of how to use the module. 3 types of PyTorch and TensorFlow network are currently supported: NN, CNN and RNN. Each network takes the following arguments: | Field | Description | Default | | :---: | :----------: | :---: | | *input_dim*| Dimension of the input into the network. See below for more detail. Not needed for Tensorflow. | N/A | | *layers_info* | List to indicate the layers of the network you want. Exact requirements depend on network type, see below for more detail | N/A | | *output_activation* | String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads | No activation | | *hidden_activations* | String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer), default is ReLU and for example "tanh" would have tanh applied on all hidden layer activations | ReLU after every hidden layer | | *dropout* | Float to indicate what dropout probability you want applied after each hidden layer | 0 | | *initialiser* | String to indicate which initialiser you want used to initialise all the parameters | PyTorch & TF Default | | *batch_norm* | Boolean to indicate whether you want batch norm applied to the output of every hidden layer | False | | *columns of_data_to_be_embedded* | List to indicate the column numbers of the data that you want to be put through an embedding layer before being fed through the hidden layers of the network | No embeddings | | *embedding_dimensions* | If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list of the form: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...] | No embeddings | | *y_range* | Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks | No range | | *random_seed* | Integer to indicate the random seed you want to use | 0 | | *return_final_seq_only* | Only needed for RNN. Boolean to indicate whether you only want to return the output for the final timestep (True) or if you want to return the output for all timesteps (False) | True | Each network type has slightly different requirements for **input_dim** and **layers_info** as explained below: --- ##### 1. NN * **input_dim**: # Features in PyTorch, not needed for TensorFlow * **layers_info**: List of integers to indicate number of hidden units you want per linear layer. * For example: ``` from nn_builder.pytorch.NN import NN model = NN(input_dim=5, layers_info=[10, 10, 1], output_activation=None, hidden_activations="relu", dropout=0.0, initialiser="xavier", batch_norm=False) ``` --- ##### 2. CNN * **input_dim**: (# Channels, Height, Width) in PyTorch, not needed for TensorFlow * **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a CNN can be one of these 4 forms: * ["conv", channels, kernel size, stride, padding] * ["maxpool", kernel size, stride, padding] * ["avgpool", kernel size, stride, padding] * ["linear", units] * For a PyTorch network kernel size, stride, padding and units must be integers. For TensorFlow they must all be integers except for padding which must be one of {“valid”, “same”} * For example: ``` from nn_builder.pytorch.CNN import CNN model = CNN(input_dim=(3, 64, 64), layers_info=[["conv", 32, 3, 1, 0], ["maxpool", 2, 2, 0], ["conv", 64, 3, 1, 2], ["avgpool", 2, 2, 0], ["linear", 10]], hidden_activations="relu", output_activation="softmax", dropout=0.0, initialiser="xavier", batch_norm=True) ``` --- ##### 3. RNN * **input_dim**: # Features in PyTorch, not needed for TensorFlow * **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a CNN can be one of these 4 forms: * ["lstm", units] * ["gru", units] * ["linear", units] * For example: ``` from nn_builder.pytorch.CNN import CNN model = RNN(input_dim=5, layers_info=[["gru", 50], ["lstm", 10], ["linear", 2]], hidden_activations="relu", output_activation="softmax", batch_norm=False, dropout=0.0, initialiser="xavier") ``` --- ## Contributing Together we can make something that is useful for thousands of people. Anyone is very welcome to contribute via a pull request. Please see the [issues](https://github.com/p-christ/nn_builder/issues) page for ideas on the best areas to contribute to and try to: 1. Add tests to the tests folder that cover any code you write 1. Write comments for every function 1. Create a colab notebook demonstrating how any extra functionality you created works ================================================ FILE: miscellaneous/material_for_readme/nn_builder_code ================================================ # With nn_builder from nn_builder.pytorch.NN import NN model = NN(input_dim=25, layers=[150, 100, 50, 50, 50, 50, 5], output_activation="softmax", dropout=0.5, initialiser="xavier", batch_norm=True) ================================================ FILE: miscellaneous/material_for_readme/nn_builder_tf_code ================================================ #With nn_builder from nn_builder.tensorflow.CNN import CNN model=CNN(layers_info=[["conv", 32, 3, 1, "valid"],["maxpool", 3, 3, "valid"], ["conv", 64, 3, 1, "valid"],["maxpool", 3, 2, "valid"], ["conv", 128, 3, 1, "valid"],["maxpool", 3, 1, "valid"], ["conv", 256, 3, 1, "valid"],["avgpool", 3, 1, "valid"], ["linear", 64], ["linear", 10]], hidden_activations="relu", output_activation="softmax", dropout=0.2, initialiser="glorot_normal", batch_norm=False) ================================================ FILE: miscellaneous/material_for_readme/non_nn_builder_code ================================================ # Without nn_builder import torch.nn as nn class NN(nn.Module): def __init__(self): nn.Module.__init__(self) self.fc1 = nn.Linear(25, 150) self.fc2 = nn.Linear(150, 100) self.fc3 = nn.Linear(100, 50) self.fc4 = nn.Linear(50, 50) self.fc5 = nn.Linear(50, 50) self.fc6 = nn.Linear(50, 50) self.fc7 = nn.Linear(50, 5) self.bn1 = nn.BatchNorm1d(150) self.bn2 = nn.BatchNorm1d(100) self.bn3 = nn.BatchNorm1d(50) self.bn4 = nn.BatchNorm1d(50) self.bn5 = nn.BatchNorm1d(50) self.bn6 = nn.BatchNorm1d(50) self.dropout = nn.Dropout(p=0.5) for params in [self.fc1, self.fc2, self.fc3, self.fc4, self.fc5, self.fc6, self.fc7]: nn.init.xavier_uniform_(params.weight) def forward(self, x): x = self.dropout(self.bn1(nn.ReLU()(self.fc1(x)))) x = self.dropout(self.bn2(nn.ReLU()(self.fc2(x)))) x = self.dropout(self.bn3(nn.ReLU()(self.fc3(x)))) x = self.dropout(self.bn4(nn.ReLU()(self.fc4(x)))) x = self.dropout(self.bn5(nn.ReLU()(self.fc5(x)))) x = self.dropout(self.bn6(nn.ReLU()(self.fc6(x)))) x = self.fc7(x) x = nn.Softmax(dim=1)(x) return x model = NN() ================================================ FILE: miscellaneous/material_for_readme/non_nn_builder_tf_code ================================================ #Without nn_builder import tensorflow as tf from tensorflow.keras import Model, activations from tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, \ MaxPool2D, AveragePooling2D class CNN(Model): def __init__(self): Model.__init__(self) self.conv1 = Conv2D(filters=32, kernel_size=3, strides=1, padding="valid", activation="relu", kernel_initializer="glorot_normal") self.conv2 = Conv2D(filters=64, kernel_size=3, strides=1, padding="valid", activation="relu", kernel_initializer="glorot_normal") self.conv3 = Conv2D(filters=128, kernel_size=3, strides=1, padding="valid", activation="relu", kernel_initializer="glorot_normal") self.conv4 = Conv2D(filters=256, kernel_size=3, strides=1, padding="valid", activation="relu", kernel_initializer="glorot_normal") self.maxpool1 = MaxPool2D(pool_size=3, strides=3, padding="valid") self.maxpool2 = MaxPool2D(pool_size=3, strides=2, padding="valid") self.maxpool3 = MaxPool2D(pool_size=3, strides=1, padding="valid") self.avgpool = AveragePooling2D(pool_size=3, strides=1, padding="valid") self.linear1 = Dense(64, activation="relu") self.linear2 = Dense(10, activation="softmax") self.dropout = tf.keras.layers.Dropout(rate=0.2) def call(self, x): x = self.dropout(self.maxpool1(self.conv1(x))) x = self.dropout(self.maxpool2(self.conv2(x))) x = self.dropout(self.maxpool3(self.conv3(x))) x = self.dropout(self.avgpool(self.conv4(x))) x = Flatten()(x) x = self.dropout(self.linear1(x)) x = self.linear2(x) return x model = CNN() ================================================ FILE: miscellaneous/testreadme.rst ================================================ +-----------------------------------------------------------+----------------------------------------------------------+ | **Building a simple NN with nn_builder** | **Building a simple NN without nn_builder** | +-----------------------------------------------------------+----------------------------------------------------------+ | .. code:: python | .. code:: python | | # With nn_builder | # Without nn_builder | | from nn_builder.pytorch.NN import NN | import torch.nn as nn | | model = NN(input_dim=25, | class NN(nn.Module): | | layers=[150, 100, 50, 50, 50, 50, 5], | def __init__(self): | | output_activation="softmax", dropout=0.5, | nn.Module.__init__(self) | | initialiser="xavier", batch_norm=True) | self.fc1 = nn.Linear(25, 150) | | | self.fc2 = nn.Linear(150, 100) | | | self.fc3 = nn.Linear(100, 50) | | | self.fc4 = nn.Linear(50, 50) | | | self.fc5 = nn.Linear(50, 50) | | | self.fc6 = nn.Linear(50, 50) | | | self.fc7 = nn.Linear(50, 5) | | | | | | self.bn1 = nn.BatchNorm1d(150) | | | self.bn2 = nn.BatchNorm1d(100) | | | self.bn3 = nn.BatchNorm1d(50) | | | self.bn4 = nn.BatchNorm1d(50) | | | self.bn5 = nn.BatchNorm1d(50) | | | self.bn6 = nn.BatchNorm1d(50) | | | | | | self.dropout = nn.Dropout(p=0.5) | | | for params in [self.fc1, self.fc2, self.fc3, | | | self.fc4, self.fc5, self.fc6, | | | self.fc7]: | | | nn.init.xavier_uniform_(params.weight) | | | def forward(self, x): | | | x = self.dropout(self.bn1(nn.ReLU()(self.fc1(x)))) | | | x = self.dropout(self.bn2(nn.ReLU()(self.fc2(x)))) | | | x = self.dropout(self.bn3(nn.ReLU()(self.fc3(x)))) | | | x = self.dropout(self.bn4(nn.ReLU()(self.fc4(x)))) | | | x = self.dropout(self.bn5(nn.ReLU()(self.fc5(x)))) | | | x = self.dropout(self.bn6(nn.ReLU()(self.fc6(x)))) | | | x = self.fc7(x) | | | x = nn.Softmax(dim=1)(x) | | | return x | | | | | | model = NN() | +-----------------------------------------------------------+----------------------------------------------------------+ +------------------------------------------------+--------------------------------------------+ | **Script to train an SVM on the iris dataset** | **The same script as a Sacred experiment** | +------------------------------------------------+--------------------------------------------+ | .. code:: python | .. code:: python | | # With nn_builder | | | from nn_builder.pytorch.NN import NN | from numpy.random import permutation | | | from sklearn import svm, datasets | | | from sacred import Experiment | | | ex = Experiment('iris_rbf_svm') | | | | | | @ex.config | | | def cfg(): | | C = 1.0 | C = 1.0 | | gamma = 0.7 | gamma = 0.7 | | | | | | @ex.automain | | | def run(C, gamma): | | iris = datasets.load_iris() | iris = datasets.load_iris() | | perm = permutation(iris.target.size) | per = permutation(iris.target.size) | | iris.data = iris.data[perm] | iris.data = iris.data[per] | | iris.target = iris.target[perm] | iris.target = iris.target[per] | | clf = svm.SVC(C, 'rbf', gamma=gamma) | clf = svm.SVC(C, 'rbf', gamma=gamma) | | clf.fit(iris.data[:90], | clf.fit(iris.data[:90], | | iris.target[:90]) | iris.target[:90]) | | print(clf.score(iris.data[90:], | return clf.score(iris.data[90:], | | iris.target[90:])) | iris.target[90:]) | +------------------------------------------------+--------------------------------------------+ # With nn_builder model = NN(input_dim=25, layers=[150, 100, 50, 50, 50, 50, 5], output_activation="softmax", dropout=0.5, initialiser="xavier", batch_norm=True) ================================================ FILE: nn_builder/Overall_Base_Network.py ================================================ from abc import ABC, abstractmethod class Overall_Base_Network(ABC): def __init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed): self.set_all_random_seeds(random_seed) self.input_dim = input_dim self.layers_info = layers_info self.hidden_activations = hidden_activations self.output_activation = output_activation self.dropout = dropout self.initialiser = initialiser self.batch_norm = batch_norm self.y_range = y_range self.str_to_activations_converter = self.create_str_to_activations_converter() self.str_to_initialiser_converter = self.create_str_to_initialiser_converter() self.check_all_user_inputs_valid() self.initialiser_function = self.str_to_initialiser_converter[initialiser.lower()] self.hidden_layers = self.create_hidden_layers() self.output_layers = self.create_output_layers() self.dropout_layer = self.create_dropout_layer() if self.batch_norm: self.batch_norm_layers = self.create_batch_norm_layers() @abstractmethod def check_all_user_inputs_valid(self): """Checks that all the user inputs were valid""" raise NotImplementedError @abstractmethod def create_hidden_layers(self): """Creates the hidden layers in the network""" raise NotImplementedError @abstractmethod def create_output_layers(self): """Creates the output layers in the network""" raise NotImplementedError @abstractmethod def create_batch_norm_layers(self): """Creates the batch norm layers in the network""" raise NotImplementedError @abstractmethod def create_dropout_layer(self): """Creates the dropout layers in the network""" raise NotImplementedError @abstractmethod def print_model_summary(self): """Prints a summary of the model""" raise NotImplementedError @abstractmethod def set_all_random_seeds(self, random_seed): """Sets all random seeds""" raise NotImplementedError def check_NN_layers_valid(self): """Checks that user input for hidden_units is valid""" assert isinstance(self.layers_info, list), "hidden_units must be a list" list_error_msg = "neurons must be a list of integers" integer_error_msg = "Every element of hidden_units must be 1 or higher" activation_error_msg = "The number of output activations provided should match the number of output layers" for neurons in self.layers_info[:-1]: assert isinstance(neurons, int), list_error_msg assert neurons > 0, integer_error_msg output_layer = self.layers_info[-1] if isinstance(output_layer, list): assert len(output_layer) == len(self.output_activation), activation_error_msg for output_dim in output_layer: assert isinstance(output_dim, int), list_error_msg assert output_dim > 0, integer_error_msg else: assert isinstance(self.output_activation, str) or self.output_activation is None, activation_error_msg assert isinstance(output_layer, int), list_error_msg assert output_layer > 0, integer_error_msg def check_NN_input_dim_valid(self): """Checks that user input for input_dim is valid""" assert isinstance(self.input_dim, int), "input_dim must be an integer" assert self.input_dim > 0, "input_dim must be 1 or higher" def check_activations_valid(self): """Checks that user input for hidden_activations and output_activation is valid""" valid_activations_strings = self.str_to_activations_converter.keys() if self.output_activation is None: self.output_activation = "None" if isinstance(self.output_activation, list): for activation in self.output_activation: if activation is not None: assert activation.lower() in set(valid_activations_strings), "Output activations must be string from list {}".format(valid_activations_strings) else: assert self.output_activation.lower() in set(valid_activations_strings), "Output activation must be string from list {}".format(valid_activations_strings) assert isinstance(self.hidden_activations, str) or isinstance(self.hidden_activations, list), "hidden_activations must be a string or a list of strings" if isinstance(self.hidden_activations, str): assert self.hidden_activations.lower() in set(valid_activations_strings), "hidden_activations must be from list {}".format(valid_activations_strings) elif isinstance(self.hidden_activations, list): assert len(self.hidden_activations) == len(self.layers_info), "if hidden_activations is a list then you must provide 1 activation per hidden layer" for activation in self.hidden_activations: assert isinstance(activation, str), "hidden_activations must be a string or list of strings" assert activation.lower() in set(valid_activations_strings), "each element in hidden_activations must be from list {}".format(valid_activations_strings) def check_embedding_dimensions_valid(self): """Checks that user input for embedding_dimensions is valid""" assert isinstance(self.embedding_dimensions, list), "embedding_dimensions must be a list" for embedding_dim in self.embedding_dimensions: assert len(embedding_dim) == 2 and isinstance(embedding_dim, list), \ "Each element of embedding_dimensions must be of form (input_dim, output_dim)" def check_y_range_values_valid(self): """Checks that user input for y_range is valid""" if self.y_range: assert isinstance(self.y_range, tuple) and len(self.y_range) == 2, "y_range must be a tuple of 2 floats or integers" for elem in range(2): assert isinstance(self.y_range[elem], float) or isinstance(self.y_range[elem], int), "y_range must be a tuple of 2 floats or integers" assert self.y_range[0] <= self.y_range[1], "y_range's first element must be smaller than the second element" def check_timesteps_to_output_valid(self): """Checks that user input for timesteps_to_output is valid""" assert self.timesteps_to_output in ["all", "last"] def check_initialiser_valid(self): """Checks that user input for initialiser is valid""" valid_initialisers = set(self.str_to_initialiser_converter.keys()) assert isinstance(self.initialiser, str), "initialiser must be a string from list {}".format(valid_initialisers) assert self.initialiser.lower() in valid_initialisers, "initialiser must be from list {}".format(valid_initialisers) def check_return_final_seq_only_valid(self): """Checks whether user input for return_final_seq_only is a boolean and therefore valid. Only relevant for RNNs""" assert isinstance(self.return_final_seq_only, bool) def get_activation(self, activations, ix=None): """Gets the activation function""" if isinstance(activations, list): return self.str_to_activations_converter[str(activations[ix]).lower()] return self.str_to_activations_converter[str(activations).lower()] ================================================ FILE: nn_builder/__init__.py ================================================ ================================================ FILE: nn_builder/pytorch/Base_Network.py ================================================ import random import numpy as np import torch import torch.nn as nn from nn_builder.Overall_Base_Network import Overall_Base_Network from abc import ABC, abstractmethod class Base_Network(Overall_Base_Network, ABC): """Base class for PyTorch neural network classes""" def __init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed): self.str_to_activations_converter = self.create_str_to_activations_converter() self.str_to_initialiser_converter = self.create_str_to_initialiser_converter() super().__init__(input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed) self.initialise_all_parameters() # Flag we use to run checks on the input data into forward the first time it is entered self.checked_forward_input_data_once = False @abstractmethod def initialise_all_parameters(self): """Initialises all the parameters of the network""" raise NotImplementedError @abstractmethod def forward(self, input_data): """Runs a forward pass of the network""" raise NotImplementedError @abstractmethod def check_input_data_into_forward_once(self, input_data): """Checks the input data into the network is of the right form. Only runs the first time data is provided otherwise would slow down training too much""" raise NotImplementedError def set_all_random_seeds(self, random_seed): """Sets all possible random seeds so results can be reproduced""" torch.backends.cudnn.deterministic = True torch.manual_seed(random_seed) random.seed(random_seed) np.random.seed(random_seed) if torch.cuda.is_available(): torch.cuda.manual_seed_all(random_seed) def create_str_to_activations_converter(self): """Creates a dictionary which converts strings to activations""" str_to_activations_converter = {"elu": nn.ELU(), "hardshrink": nn.Hardshrink(), "hardtanh": nn.Hardtanh(), "leakyrelu": nn.LeakyReLU(), "logsigmoid": nn.LogSigmoid(), "prelu": nn.PReLU(), "relu": nn.ReLU(), "relu6": nn.ReLU6(), "rrelu": nn.RReLU(), "selu": nn.SELU(), "sigmoid": nn.Sigmoid(), "softplus": nn.Softplus(), "logsoftmax": nn.LogSoftmax(), "softshrink": nn.Softshrink(), "softsign": nn.Softsign(), "tanh": nn.Tanh(), "tanhshrink": nn.Tanhshrink(), "softmin": nn.Softmin(), "softmax": nn.Softmax(dim=1), "none": None} return str_to_activations_converter def create_str_to_initialiser_converter(self): """Creates a dictionary which converts strings to initialiser""" str_to_initialiser_converter = {"uniform": nn.init.uniform_, "normal": nn.init.normal_, "eye": nn.init.eye_, "xavier_uniform": nn.init.xavier_uniform_, "xavier": nn.init.xavier_uniform_, "xavier_normal": nn.init.xavier_normal_, "kaiming_uniform": nn.init.kaiming_uniform_, "kaiming": nn.init.kaiming_uniform_, "kaiming_normal": nn.init.kaiming_normal_, "he": nn.init.kaiming_normal_, "orthogonal": nn.init.orthogonal_, "default": "use_default"} return str_to_initialiser_converter def create_dropout_layer(self): """Creates a dropout layer""" return nn.Dropout(p=self.dropout) def create_embedding_layers(self): """Creates the embedding layers in the network""" embedding_layers = nn.ModuleList([]) for embedding_dimension in self.embedding_dimensions: input_dim, output_dim = embedding_dimension embedding_layers.extend([nn.Embedding(input_dim, output_dim)]) return embedding_layers def initialise_parameters(self, parameters_list): """Initialises the list of parameters given""" initialiser = self.str_to_initialiser_converter[self.initialiser.lower()] if initialiser != "use_default": for parameters in parameters_list: if type(parameters) == nn.Linear or type(parameters) == nn.Conv2d: initialiser(parameters.weight) elif type(parameters) in [nn.LSTM, nn.RNN, nn.GRU]: initialiser(parameters.weight_hh_l0) initialiser(parameters.weight_ih_l0) def flatten_tensor(self, tensor): """Flattens a tensor of shape (a, b, c, d, ...) into shape (a, b * c * d * .. )""" return tensor.reshape(tensor.shape[0], -1) def print_model_summary(self): print(self) ================================================ FILE: nn_builder/pytorch/CNN.py ================================================ import torch import torch.nn as nn import numpy as np from nn_builder.pytorch.Base_Network import Base_Network class CNN(nn.Module, Base_Network): """Creates a PyTorch convolutional neural network Args: - input_dim: Tuple of integers to indicate the (channels, height, width) dimension of the input - layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be one of these 6 forms: - ["conv", channels, kernel_size, stride, padding] - ["maxpool", kernel_size, stride, padding] - ["avgpool", kernel_size, stride, padding] - ["adaptivemaxpool", output height, output width] - ["adaptiveavgpool", output height, output width] - ["linear", out] - output_activation: String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads - hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer). Default is ReLU. - dropout: Float to indicate what dropout probability you want applied after each hidden layer - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch initialisers are supported. PyTorch's default initialisation is the default. - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks. Default is no range restriction - random_seed: Integer to indicate the random seed you want to use NOTE that this class' forward method expects input data in the form: (batch, channels, height, width) """ def __init__(self, input_dim, layers_info, output_activation=None, hidden_activations="relu", dropout=0.0, initialiser="default", batch_norm=False, y_range= (), random_seed=0, converted_from_tf_model=False): nn.Module.__init__(self) self.valid_cnn_hidden_layer_types = {'conv', 'maxpool', 'avgpool', 'adaptivemaxpool', 'adaptiveavgpool', 'linear'} self.valid_layer_types_with_no_parameters = [nn.MaxPool2d, nn.AvgPool2d, nn.AdaptiveAvgPool2d, nn.AdaptiveMaxPool2d] Base_Network.__init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed) self.converted_from_tf_model = converted_from_tf_model def flatten_tensor(self, tensor): """Flattens a tensor of shape (a, b, c, d, ...) into shape (a, b * c * d * .. )""" if self.converted_from_tf_model: tensor = tensor.permute(0, 2, 3, 1).contiguous() tensor = tensor.view(tensor.size(0), -1) else: tensor = tensor.reshape(tensor.shape[0], -1) return tensor def check_all_user_inputs_valid(self): """Checks that all the user inputs were valid""" self.check_CNN_input_dim_valid() self.check_CNN_layers_valid() self.check_activations_valid() self.check_initialiser_valid() self.check_y_range_values_valid() def check_CNN_input_dim_valid(self): """Checks that the CNN input dim valid""" error_msg = "input_dim must be a tuple of 3 integers indicating (channels, height, width)" assert isinstance(self.input_dim, tuple), error_msg for element in self.input_dim: assert isinstance(element, int) and element > 0 def check_CNN_layers_valid(self): """Checks that the user inputs for cnn_hidden_layers were valid. cnn_hidden_layers must be a list of layers where each layer must be of one of these forms: - ["conv", channels, kernel_size, stride, padding] - ["maxpool", kernel_size, stride, padding] - ["avgpool", kernel_size, stride, padding] - ["adaptivemaxpool", output height, output width] - ["adaptiveavgpool", output height, output width] - ["linear", out] """ error_msg_layer_type = "First element in a layer specification must be one of {}".format(self.valid_cnn_hidden_layer_types) error_msg_conv_layer = """Conv layer must be of form ['conv', channels, kernel_size, stride, padding] where the final 4 elements are non-negative integers""" error_msg_maxpool_layer = """Maxpool layer must be of form ['maxpool', kernel_size, stride, padding] where the final 2 elements are non-negative integers""" error_msg_avgpool_layer = """Avgpool layer must be of form ['avgpool', kernel_size, stride, padding] where the final 2 elements are non-negative integers""" error_msg_adaptivemaxpool_layer = """Adaptivemaxpool layer must be of form ['adaptivemaxpool', output height, output width]""" error_msg_adaptiveavgpool_layer = """Adaptiveavgpool layer must be of form ['adaptiveavgpool', output height, output width]""" error_msg_linear_layer = """Linear layer must be of form ['linear', out] where out is a non-negative integers""" assert isinstance(self.layers_info, list), "layers must be a list" all_layers = self.layers_info[:-1] output_layer = self.layers_info[-1] assert isinstance(output_layer, list), "layers must be a list" if isinstance(output_layer[0], list): assert len(output_layer) == len( self.output_activation), "Number of output activations must equal number of output heads" for layer in output_layer: all_layers.append(layer) assert layer[0].lower() == "linear", "Final layer must be linear" else: all_layers.append(output_layer) assert isinstance(output_layer[0], str), error_msg_layer_type assert output_layer[0].lower() == "linear", "Final layer must be linear" for layer in all_layers: assert isinstance(layer, list), "Each layer must be a list" assert isinstance(layer[0], str), error_msg_layer_type layer_type_name = layer[0].lower() assert layer_type_name in self.valid_cnn_hidden_layer_types, "Layer name {} not valid, use one of {}".format(layer_type_name, self.valid_cnn_hidden_layer_types) if layer_type_name == "conv": assert len(layer) == 5, error_msg_conv_layer for ix in range(3): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0, error_msg_conv_layer assert isinstance(layer[4], int) and layer[4] >= 0, error_msg_conv_layer elif layer_type_name == "maxpool": assert len(layer) == 4, error_msg_maxpool_layer for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_maxpool_layer if layer[1] != layer[2]: print("NOTE that your maxpool kernel size {} isn't the same as your stride {}".format(layer[1], layer[2])) assert isinstance(layer[3], int) and layer[3] >= 0, error_msg_conv_layer elif layer_type_name == "avgpool": assert len(layer) == 4, error_msg_avgpool_layer for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_avgpool_layer assert isinstance(layer[3], int) and layer[3] >= 0, error_msg_conv_layer if layer[1] != layer[2]:print("NOTE that your avgpool kernel size {} isn't the same as your stride {}".format(layer[1], layer[2])) elif layer_type_name == "adaptivemaxpool": assert len(layer) == 3, error_msg_adaptivemaxpool_layer for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_adaptivemaxpool_layer elif layer_type_name == "adaptiveavgpool": assert len(layer) == 3, error_msg_adaptiveavgpool_layer for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ ix + 1] > 0, error_msg_adaptiveavgpool_layer elif layer_type_name == "linear": assert len(layer) == 2, error_msg_linear_layer for ix in range(1): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0 else: raise ValueError("Invalid layer name") rest_must_be_linear = False for ix, layer in enumerate(all_layers): if rest_must_be_linear: assert layer[0].lower() == "linear", "If have linear layers then they must come at end" if layer[0].lower() == "linear": rest_must_be_linear = True def create_hidden_layers(self): """Creates the hidden layers in the network""" cnn_hidden_layers = nn.ModuleList([]) input_dim = self.input_dim for layer in self.layers_info[:-1]: input_dim = self.create_and_append_layer(input_dim, layer, cnn_hidden_layers) self.input_dim_into_final_layer = input_dim return cnn_hidden_layers def create_and_append_layer(self, input_dim, layer, list_to_append_layer_to): """Creates and appends a layer to the list provided""" layer_name = layer[0].lower() assert layer_name in self.valid_cnn_hidden_layer_types, "Layer name {} not valid, use one of {}".format( layer_name, self.valid_cnn_hidden_layer_types) if layer_name == "conv": list_to_append_layer_to.extend([nn.Conv2d(in_channels=input_dim[0], out_channels=layer[1], kernel_size=layer[2], stride=layer[3], padding=layer[4])]) elif layer_name == "maxpool": list_to_append_layer_to.extend([nn.MaxPool2d(kernel_size=layer[1], stride=layer[2], padding=layer[3])]) elif layer_name == "avgpool": list_to_append_layer_to.extend([nn.AvgPool2d(kernel_size=layer[1], stride=layer[2], padding=layer[3])]) elif layer_name == "adaptivemaxpool": list_to_append_layer_to.extend([nn.AdaptiveMaxPool2d(output_size=(layer[1], layer[2]))]) elif layer_name == "adaptiveavgpool": list_to_append_layer_to.extend([nn.AdaptiveAvgPool2d(output_size=(layer[1], layer[2]))]) elif layer_name == "linear": if isinstance(input_dim, tuple): input_dim = np.prod(np.array(input_dim)) list_to_append_layer_to.extend([nn.Linear(in_features=input_dim, out_features=layer[1])]) else: raise ValueError("Wrong layer name") input_dim = self.calculate_new_dimensions(input_dim, layer) return input_dim def calculate_new_dimensions(self, input_dim, layer): """Calculates the new dimensions of the data after passing through a type of layer""" layer_name = layer[0].lower() if layer_name == "conv": new_channels = layer[1] kernel, stride, padding = layer[2], layer[3], layer[4] new_height = int((input_dim[1] - kernel + 2*padding)/stride) + 1 new_width = int((input_dim[2] - kernel + 2 * padding) / stride) + 1 output_dim = (new_channels, new_height, new_width) elif layer_name in ["maxpool", "avgpool"]: new_channels = input_dim[0] kernel, stride, padding = layer[1], layer[2], layer[3] new_height = int((input_dim[1] - kernel + 2*padding)/stride) + 1 new_width = int((input_dim[2] - kernel + 2 * padding) / stride) + 1 output_dim = (new_channels, new_height, new_width) elif layer_name in ["adaptivemaxpool", "adaptiveavgpool"]: new_channels = input_dim[0] output_dim = (new_channels, layer[1], layer[2]) elif layer_name == "linear": output_dim = layer[1] return output_dim def create_output_layers(self): """Creates the output layers in the network""" output_layers = nn.ModuleList([]) input_dim = self.input_dim_into_final_layer if not isinstance(self.layers_info[-1][0], list) : self.layers_info[-1] = [self.layers_info[-1]] for output_layer in self.layers_info[-1]: self.create_and_append_layer(input_dim, output_layer, output_layers) return output_layers def initialise_all_parameters(self): """Initialises the parameters in the linear and embedding layers""" initialisable_layers = [layer for layer in self.hidden_layers if not type(layer) in self.valid_layer_types_with_no_parameters] self.initialise_parameters(nn.ModuleList(initialisable_layers)) output_initialisable_layers = [layer for layer in self.output_layers if not type(layer) in self.valid_layer_types_with_no_parameters] self.initialise_parameters(output_initialisable_layers) def create_batch_norm_layers(self): """Creates the batch norm layers in the network""" batch_norm_layers = nn.ModuleList([]) for layer in self.layers_info[:-1]: layer_type = layer[0].lower() if layer_type == "conv": batch_norm_layers.extend([nn.BatchNorm2d(num_features=layer[1])]) elif layer_type == "linear": batch_norm_layers.extend([nn.BatchNorm1d(num_features=layer[1])]) return batch_norm_layers def forward(self, x): """Forward pass for the network. Note that it expects input data in the form (Batch, Channels, Height, Width)""" if not self.checked_forward_input_data_once: self.check_input_data_into_forward_once(x) x = self.process_hidden_layers(x) out = self.process_output_layers(x) if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*nn.Sigmoid()(out) return out def check_input_data_into_forward_once(self, x): """Checks the input data into forward is of the right format. Then sets a flag indicating that this has happened once so that we don't keep checking as this would slow down the model too much""" assert len(x.shape) == 4, "x should have the shape (batch_size, channel, height, width)" assert x.shape[1:] == self.input_dim, "Input data must be of shape (channels, height, width) that you provided, not of shape {}".format(x.shape[1:]) self.checked_forward_input_data_once = True #So that it doesn't check again def process_hidden_layers(self, x): """Puts the data x through all the hidden layers""" flattened=False valid_batch_norm_layer_ix = 0 for layer_ix, layer in enumerate(self.hidden_layers): if type(layer) in self.valid_layer_types_with_no_parameters: x = layer(x) else: if type(layer) == nn.Linear and not flattened: x = self.flatten_tensor(x) flattened = True x = self.get_activation(self.hidden_activations, layer_ix)(layer(x)) if self.batch_norm: x = self.batch_norm_layers[valid_batch_norm_layer_ix](x) valid_batch_norm_layer_ix += 1 if self.dropout != 0.0: x = self.dropout_layer(x) if not flattened: x = self.flatten_tensor(x) return x def process_output_layers(self, x): """Puts the data x through all the output layers""" out = None for output_layer_ix, output_layer in enumerate(self.output_layers): activation = self.get_activation(self.output_activation, output_layer_ix) temp_output = output_layer(x) if activation is not None: temp_output = activation(temp_output) if out is None: out = temp_output else: out = torch.cat((out, temp_output), dim=1) return out ================================================ FILE: nn_builder/pytorch/NN.py ================================================ import torch import numpy as np import torch.nn as nn from nn_builder.pytorch.Base_Network import Base_Network class NN(nn.Module, Base_Network): """Creates a PyTorch neural network Args: - input_dim: Integer to indicate the dimension of the input into the network - layers_info: List of integers to indicate the width and number of linear layers you want in your network, e.g. [5, 8, 1] would produce a network with 3 linear layers of width 5, 8 and then 1 - hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer). Default is ReLU. - output_activation: String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads - dropout: Float to indicate what dropout probability you want applied after each hidden layer - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch initialisers are supported. PyTorch's default initialisation is the default. - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False - columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer before being fed through the other layers of the network. Default option is no embeddings - embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks. Default is no range restriction - random_seed: Integer to indicate the random seed you want to use """ def __init__(self, input_dim, layers_info, output_activation=None, hidden_activations="relu", dropout=0.0, initialiser="default", batch_norm=False, columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (), random_seed=0): nn.Module.__init__(self) self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0 self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded self.embedding_dimensions = embedding_dimensions self.embedding_layers = self.create_embedding_layers() Base_Network.__init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed) def check_all_user_inputs_valid(self): """Checks that all the user inputs were valid""" self.check_NN_input_dim_valid() self.check_NN_layers_valid() self.check_activations_valid() self.check_embedding_dimensions_valid() self.check_initialiser_valid() self.check_y_range_values_valid() def create_hidden_layers(self): """Creates the linear layers in the network""" linear_layers = nn.ModuleList([]) input_dim = int(self.input_dim - len(self.embedding_dimensions) + np.sum([output_dims[1] for output_dims in self.embedding_dimensions])) for hidden_unit in self.layers_info[:-1]: linear_layers.extend([nn.Linear(input_dim, hidden_unit)]) input_dim = hidden_unit return linear_layers def create_output_layers(self): """Creates the output layers in the network""" output_layers = nn.ModuleList([]) if len(self.layers_info) >= 2: input_dim = self.layers_info[-2] else: input_dim = self.input_dim if not isinstance(self.layers_info[-1], list): output_layer = [self.layers_info[-1]] else: output_layer = self.layers_info[-1] for output_dim in output_layer: output_layers.extend([nn.Linear(input_dim, output_dim)]) return output_layers def create_batch_norm_layers(self): """Creates the batch norm layers in the network""" batch_norm_layers = nn.ModuleList([nn.BatchNorm1d(num_features=hidden_unit) for hidden_unit in self.layers_info[:-1]]) return batch_norm_layers def initialise_all_parameters(self): """Initialises the parameters in the linear and embedding layers""" self.initialise_parameters(self.hidden_layers) self.initialise_parameters(self.output_layers) self.initialise_parameters(self.embedding_layers) def forward(self, x): """Forward pass for the network""" if not self.checked_forward_input_data_once: self.check_input_data_into_forward_once(x) if self.embedding_to_occur: x = self.incorporate_embeddings(x) x = self.process_hidden_layers(x) out = self.process_output_layers(x) if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*nn.Sigmoid()(out) return out def check_input_data_into_forward_once(self, x): """Checks the input data into forward is of the right format. Then sets a flag indicating that this has happened once so that we don't keep checking as this would slow down the model too much""" for embedding_dim in self.columns_of_data_to_be_embedded: data = x[:, embedding_dim] data_long = data.long() assert all(data_long >= 0), "All data to be embedded must be integers 0 and above -- {}".format(data_long) assert torch.sum(abs(data.float() - data_long.float())) < 0.0001, """Data columns to be embedded should be integer values 0 and above to represent the different classes""" if self.input_dim > len(self.columns_of_data_to_be_embedded): assert isinstance(x, torch.FloatTensor) or isinstance(x, torch.cuda.FloatTensor), "Input data must be a float tensor" assert len(x.shape) == 2, "X should be a 2-dimensional tensor: {}".format(x.shape) self.checked_forward_input_data_once = True #So that it doesn't check again def incorporate_embeddings(self, x): """Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready to then be put through the linear layers""" all_embedded_data = [] for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded): data = x[:, embedding_var].long() embedded_data = self.embedding_layers[embedding_layer_ix](data) all_embedded_data.append(embedded_data) all_embedded_data = torch.cat(tuple(all_embedded_data), dim=1) x = torch.cat((x[:, [col for col in range(x.shape[1]) if col not in self.columns_of_data_to_be_embedded]].float(), all_embedded_data), dim=1) return x def process_hidden_layers(self, x): """Puts the data x through all the hidden layers""" for layer_ix, linear_layer in enumerate(self.hidden_layers): x = self.get_activation(self.hidden_activations, layer_ix)(linear_layer(x)) if self.batch_norm: x = self.batch_norm_layers[layer_ix](x) if self.dropout != 0.0: x = self.dropout_layer(x) return x def process_output_layers(self, x): """Puts the data x through all the output layers""" out = None for output_layer_ix, output_layer in enumerate(self.output_layers): activation = self.get_activation(self.output_activation, output_layer_ix) temp_output = output_layer(x) if activation is not None: temp_output = activation(temp_output) if out is None: out = temp_output else: out = torch.cat((out, temp_output), dim=1) return out ================================================ FILE: nn_builder/pytorch/RNN.py ================================================ import torch import torch.nn as nn import numpy as np from nn_builder.pytorch.Base_Network import Base_Network class RNN(nn.Module, Base_Network): """Creates a PyTorch recurrent neural network Args: - input_dim: Integer to indicate the dimension of the input into the network - layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be one of these 3 forms: - ["lstm", hidden_units] - ["gru", hidden_units] - ["linear", hidden_units] - hidden_activations: String or list of string to indicate the activations you want used on the output of linear hidden layers (not including the output layer). Default is ReLU. - output_activation: String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads - dropout: Float to indicate what dropout probability you want applied after each hidden layer - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch initialisers are supported. PyTorch's default initialisation is the default. - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False - columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer before being fed through the other layers of the network. Default option is no embeddings - embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks. Default is no range restriction - return_final_seq_only: Boolean to indicate whether you only want to return the output for the final timestep (True) or if you want to return the output for all timesteps (False) - random_seed: Integer to indicate the random seed you want to use NOTE that this class' forward method expects input data in the form: (batch, sequence length, features) """ def __init__(self, input_dim, layers_info, output_activation=None, hidden_activations="relu", dropout=0.0, initialiser="default", batch_norm=False, columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (), return_final_seq_only=True, random_seed=0): nn.Module.__init__(self) self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0 self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded self.embedding_dimensions = embedding_dimensions self.embedding_layers = self.create_embedding_layers() self.return_final_seq_only = return_final_seq_only self.valid_RNN_hidden_layer_types = {"linear", "gru", "lstm"} Base_Network.__init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed) def check_all_user_inputs_valid(self): """Checks that all the user inputs were valid""" self.check_NN_input_dim_valid() self.check_RNN_layers_valid() self.check_activations_valid() self.check_embedding_dimensions_valid() self.check_initialiser_valid() self.check_y_range_values_valid() self.check_return_final_seq_only_valid() def check_RNN_layers_valid(self): """Checks that layers provided by user are valid""" error_msg_layer_type = "First element in a layer specification must be one of {}".format(self.valid_RNN_hidden_layer_types) error_msg_layer_form = "Layer must be of form [layer_name, hidden_units]" error_msg_layer_list = "Layers must be provided as a list" error_msg_output_heads = "Number of output activations must equal number of output heads" assert isinstance(self.layers_info, list), error_msg_layer_list all_layers = self.layers_info[:-1] output_layer = self.layers_info[-1] assert isinstance(output_layer, list), error_msg_layer_list if isinstance(output_layer[0], list): assert len(output_layer) == len( self.output_activation), error_msg_output_heads for layer in output_layer: all_layers.append(layer) else: assert not isinstance(self.output_activation, list) or len( self.output_activation) == 1, error_msg_output_heads all_layers.append(output_layer) rest_must_be_linear = False for layer in all_layers: assert isinstance(layer, list), "Each layer must be a list" assert isinstance(layer[0], str), error_msg_layer_type layer_type_name = layer[0].lower() assert layer_type_name in self.valid_RNN_hidden_layer_types, "Layer name {} not valid, use one of {}".format( layer_type_name, self.valid_RNN_hidden_layer_types) assert isinstance(layer[1], int), error_msg_layer_form assert layer[1] > 0, "Must have hidden_units >= 1" assert len(layer) == 2, error_msg_layer_form if rest_must_be_linear: assert layer[0].lower() == "linear", "If have linear layers then they must come at end" if layer_type_name == "linear": rest_must_be_linear = True def create_hidden_layers(self): """Creates the hidden layers in the network""" RNN_hidden_layers = nn.ModuleList([]) input_dim = int(self.input_dim - len(self.embedding_dimensions) + np.sum( [output_dims[1] for output_dims in self.embedding_dimensions])) for layer in self.layers_info[:-1]: input_dim = self.create_and_append_layer(input_dim, layer, RNN_hidden_layers) self.input_dim_into_final_layer = input_dim return RNN_hidden_layers def create_and_append_layer(self, input_dim, layer, RNN_hidden_layers): layer_type_name = layer[0].lower() hidden_size = layer[1] if layer_type_name == "lstm": RNN_hidden_layers.extend([nn.LSTM(input_size=input_dim, hidden_size=hidden_size, batch_first=True)]) elif layer_type_name == "gru": RNN_hidden_layers.extend( [nn.GRU(input_size=input_dim, hidden_size=hidden_size, batch_first=True)]) elif layer_type_name == "linear": RNN_hidden_layers.extend([nn.Linear(input_dim, hidden_size)]) else: raise ValueError("Wrong layer names") input_dim = hidden_size return input_dim def create_output_layers(self): """Creates the output layers in the network""" output_layers = nn.ModuleList([]) input_dim = self.input_dim_into_final_layer if not isinstance(self.layers_info[-1][0], list): self.layers_info[-1] = [self.layers_info[-1]] for output_layer in self.layers_info[-1]: self.create_and_append_layer(input_dim, output_layer, output_layers) return output_layers def initialise_all_parameters(self): """Initialises the parameters in the linear and embedding layers""" self.initialise_parameters(self.hidden_layers) self.initialise_parameters(self.output_layers) self.initialise_parameters(self.embedding_layers) def create_batch_norm_layers(self): """Creates the batch norm layers in the network""" batch_norm_layers = nn.ModuleList([nn.BatchNorm1d(num_features=layer[1]) for layer in self.layers_info[:-1]]) return batch_norm_layers def get_activation(self, activations, ix=None): """Gets the activation function""" if isinstance(activations, list): activation = self.str_to_activations_converter[str(activations[ix]).lower()] else: activation = self.str_to_activations_converter[str(activations).lower()] return activation def forward(self, x): """Forward pass for the network. Note that it expects input data in the form (batch, seq length, features)""" if not self.checked_forward_input_data_once: self.check_input_data_into_forward_once(x) batch_size, seq_length, data_dimension = x.shape if self.embedding_to_occur: x = self.incorporate_embeddings(x, batch_size, seq_length) x = self.process_hidden_layers(x, batch_size, seq_length) out = self.process_output_layers(x, batch_size, seq_length) if self.return_final_seq_only: out = out[:, -1, :] if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*nn.Sigmoid()(out) return out def check_input_data_into_forward_once(self, x): """Checks the input data into forward is of the right format. Then sets a flag indicating that this has happened once so that we don't keep checking as this would slow down the model too much""" assert len(x.shape) == 3, "x should have the shape (batch_size, sequence_length, dimension)" assert x.shape[2] == self.input_dim, "x must have the same dimension as the input_dim you provided" for embedding_dim in self.columns_of_data_to_be_embedded: data = x[:, :, embedding_dim] data = data.contiguous().view(-1, 1) data_long = data.long() assert all(data_long >= 0), "All data to be embedded must be integers 0 and above -- {}".format(data_long) assert torch.sum(abs(data.float() - data_long.float())) < 0.0001, """Data columns to be embedded should be integer values 0 and above to represent the different classes""" if self.input_dim > len(self.columns_of_data_to_be_embedded): assert isinstance(x, torch.FloatTensor) or isinstance(x, torch.cuda.FloatTensor), "Input data must be a float tensor" self.checked_forward_input_data_once = True #So that it doesn't check again def incorporate_embeddings(self, x, batch_size, seq_length): """Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready to then be put through the hidden layers""" all_embedded_data = [] for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded): data = x[:, :, embedding_var].long() data = data.contiguous().view(batch_size * seq_length, -1) embedded_data = self.embedding_layers[embedding_layer_ix](data) embedded_data = embedded_data.view(batch_size, seq_length, -1) all_embedded_data.append(embedded_data) all_embedded_data = torch.cat(tuple(all_embedded_data), dim=2) non_embedded_data = x[:, :, [col for col in range(x.shape[2]) if col not in self.columns_of_data_to_be_embedded]].float() x = torch.cat((non_embedded_data, all_embedded_data), dim=2) return x def process_hidden_layers(self, x, batch_size, seq_length): """Puts the data x through all the hidden layers""" for layer_ix, layer in enumerate(self.hidden_layers): if type(layer) == nn.Linear: x = x.contiguous().view(batch_size * seq_length, -1) activation = self.get_activation(self.hidden_activations, layer_ix) x = activation(layer(x)) x = x.view(batch_size, seq_length, layer.out_features) else: x = layer(x) x = x[0] #because we only want to keep the output and not the hidden states if self.batch_norm: x.transpose_(1, 2) x = self.batch_norm_layers[layer_ix](x) x.transpose_(1, 2) if self.dropout != 0.0: x = self.dropout_layer(x) return x def process_output_layers(self, x, batch_size, seq_length): """Puts the data x through all the output layers""" out = None for output_layer_ix, output_layer in enumerate(self.output_layers): activation = self.get_activation(self.output_activation, output_layer_ix) if type(output_layer) == nn.Linear: x = x.contiguous().view(batch_size * seq_length, -1) temp_output = output_layer(x) if activation is not None: temp_output = activation(temp_output) temp_output = temp_output.view(batch_size, seq_length, -1) x = x.view(batch_size, seq_length, -1) else: temp_output = output_layer(x) temp_output = temp_output[0] if activation is not None: if type(activation) == nn.Softmax: temp_output = temp_output.contiguous().view(batch_size * seq_length, -1) temp_output = activation(temp_output) temp_output = temp_output.view(batch_size, seq_length, -1) else: temp_output = activation(temp_output) if out is None: out = temp_output else: out = torch.cat((out, temp_output), dim=2) return out ================================================ FILE: nn_builder/pytorch/__init__.py ================================================ ================================================ FILE: nn_builder/tensorflow/Base_Network.py ================================================ from tensorflow.keras.layers import BatchNormalization from nn_builder.Overall_Base_Network import Overall_Base_Network import tensorflow.keras.activations as activations import tensorflow.keras.initializers as initializers import numpy as np import random import tensorflow as tf from abc import ABC, abstractmethod class Base_Network(Overall_Base_Network, ABC): """Base class for TensorFlow neural network classes""" def __init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed, input_dim): if input_dim is not None: print("You don't need to provide input_dim for a tensorflow network") super().__init__(None, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed) @abstractmethod def call(self, x, training=True): """Runs a forward pass of the tensorflow model""" raise NotImplementedError @abstractmethod def create_and_append_layer(self, layer, list_to_append_layer_to, activation, output_layer=False): """Creates a layer and appends it to the provided list""" raise NotImplementedError def set_all_random_seeds(self, random_seed): """Sets all possible random seeds so results can be reproduced""" np.random.seed(random_seed) tf.random.set_seed(random_seed) random.seed(random_seed) def create_str_to_activations_converter(self): """Creates a dictionary which converts strings to activations""" str_to_activations_converter = {"elu": activations.elu, "exponential": activations.exponential, "hard_sigmoid": activations.hard_sigmoid, "linear": activations.linear, "relu": activations.relu, "selu": activations.selu, "sigmoid": activations.sigmoid, "softmax": activations.softmax, "softplus": activations.softplus, "softsign": activations.softsign, "tanh": activations.tanh, "none": activations.linear} return str_to_activations_converter def create_str_to_initialiser_converter(self): """Creates a dictionary which converts strings to initialiser""" str_to_initialiser_converter = {"glorot_normal": initializers.glorot_normal, "glorot_uniform": initializers.glorot_uniform, "xavier_normal": initializers.glorot_normal, "xavier_uniform": initializers.glorot_uniform, "xavier": initializers.glorot_uniform, "he_normal": initializers.he_normal(), "he_uniform": initializers.he_uniform(), "lecun_normal": initializers.lecun_normal(), "lecun_uniform": initializers.lecun_uniform(), "truncated_normal": initializers.TruncatedNormal, "variance_scaling": initializers.VarianceScaling, "default": initializers.glorot_uniform} return str_to_initialiser_converter def create_dropout_layer(self): """Creates a dropout layer""" return tf.keras.layers.Dropout(rate=self.dropout) def create_hidden_layers(self): """Creates the hidden layers in the network""" hidden_layers = [] for layer_ix, layer in enumerate(self.layers_info[:-1]): activation = self.get_activation(self.hidden_activations, layer_ix) self.create_and_append_layer(layer, hidden_layers, activation, output_layer=False) return hidden_layers def create_output_layers(self): """Creates the output layers in the network""" output_layers = [] network_type = type(self).__name__ if network_type in ["CNN", "RNN"]: if not isinstance(self.layers_info[-1][0], list): self.layers_info[-1] = [self.layers_info[-1]] elif network_type == "NN": if isinstance(self.layers_info[-1], int): self.layers_info[-1] = [self.layers_info[-1]] else: raise ValueError("Network type not recognised") for output_layer_ix, output_layer in enumerate(self.layers_info[-1]): activation = self.get_activation(self.output_activation, output_layer_ix) self.create_and_append_layer(output_layer, output_layers, activation, output_layer=True) return output_layers def create_embedding_layers(self): """Creates the embedding layers in the network""" embedding_layers = [] for embedding_dimension in self.embedding_dimensions: input_dim, output_dim = embedding_dimension embedding_layers.extend([tf.keras.layers.Embedding(input_dim, output_dim)]) return embedding_layers def create_batch_norm_layers(self): """Creates the batch norm layers in the network""" batch_norm_layers = [] for layer in self.layers_info[:-1]: batch_norm_layers.extend([BatchNormalization()]) return batch_norm_layers def print_model_summary(self, input_shape=None): assert input_shape is not None, "Must provide the input_shape parameter as a tuple" self.build(input_shape=input_shape) self.dropout_layer.build(input_shape) self.summary() ================================================ FILE: nn_builder/tensorflow/CNN.py ================================================ import numpy as np from tensorflow.keras import Model, activations from tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, BatchNormalization, MaxPool2D, AveragePooling2D from nn_builder.tensorflow.Base_Network import Base_Network import tensorflow as tf class CNN(Model, Base_Network): """Creates a PyTorch convolutional neural network Args: - layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be one of these 4 forms: - ["conv", channels, kernel_size, stride, padding] - ["maxpool", kernel_size, stride, padding] - ["avgpool", kernel_size, stride, padding] - ["linear", out] where all variables are integers except for padding which must be either "same" or "valid" - output_activation: String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads - hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer). Default is ReLU. - dropout: Float to indicate what dropout probability you want applied after each hidden layer - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch initialisers are supported. PyTorch's default initialisation is the default. - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks. Default is no range restriction - random_seed: Integer to indicate the random seed you want to use NOTE that this class' call method expects input data in the form: (batch, channels, height, width) """ def __init__(self, layers_info, output_activation=None, hidden_activations="relu", dropout= 0.0, initialiser="default", batch_norm=False, y_range=(), random_seed=0, input_dim=None): Model.__init__(self) self.valid_cnn_hidden_layer_types = {'conv', 'maxpool', 'avgpool', 'linear'} self.valid_layer_types_with_no_parameters = (MaxPool2D, AveragePooling2D) Base_Network.__init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed, input_dim) def check_all_user_inputs_valid(self): """Checks that all the user inputs were valid""" self.check_CNN_layers_valid() self.check_activations_valid() self.check_initialiser_valid() self.check_y_range_values_valid() def check_CNN_layers_valid(self): """Checks that the user inputs for cnn_hidden_layers were valid. cnn_hidden_layers must be a list of layers where each layer must be of one of these forms: - ["conv", channels, kernel_size, stride, padding] - ["maxpool", kernel_size, stride, padding] - ["avgpool", kernel_size, stride, padding] - ["linear", out] where all variables must be integers except padding which must be "valid" or "same" """ error_msg_layer_type = "First element in a layer specification must be one of {}".format(self.valid_cnn_hidden_layer_types) error_msg_conv_layer = """Conv layer must be of form ['conv', channels, kernel_size, stride, padding] where the variables are all non-negative integers except padding which must be either "valid" or "same""" error_msg_maxpool_layer = """Maxpool layer must be of form ['maxpool', kernel_size, stride, padding] where the variables are all non-negative integers except padding which must be either "valid" or "same""" error_msg_avgpool_layer = """Avgpool layer must be of form ['avgpool', kernel_size, stride, padding] where the variables are all non-negative integers except padding which must be either "valid" or "same""" error_msg_linear_layer = """Linear layer must be of form ['linear', out] where out is a non-negative integers""" assert isinstance(self.layers_info, list), "layers must be a list" all_layers = self.layers_info[:-1] output_layer = self.layers_info[-1] assert isinstance(output_layer, list), "layers must be a list" if isinstance(output_layer[0], list): assert len(output_layer) == len(self.output_activation), "Number of output activations must equal number of output heads" for layer in output_layer: all_layers.append(layer) assert isinstance(layer[0], str), error_msg_layer_type assert layer[0].lower() == "linear", "Final layer must be linear" else: all_layers.append(output_layer) assert isinstance(output_layer[0], str), error_msg_layer_type assert output_layer[0].lower() == "linear", "Final layer must be linear" for layer in all_layers: assert isinstance(layer, list), "Each layer must be a list" assert isinstance(layer[0], str), error_msg_layer_type layer_type_name = layer[0].lower() assert layer_type_name in self.valid_cnn_hidden_layer_types, "Layer name {} not valid, use one of {}".format(layer_type_name, self.valid_cnn_hidden_layer_types) if layer_type_name == "conv": assert len(layer) == 5, error_msg_conv_layer for ix in range(3): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0, error_msg_conv_layer assert isinstance(layer[4], str) and layer[4].lower() in ["valid", "same"], error_msg_conv_layer elif layer_type_name == "maxpool": assert len(layer) == 4, error_msg_maxpool_layer for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_maxpool_layer if layer[1] != layer[2]: print("NOTE that your maxpool kernel size {} isn't the same as your stride {}".format(layer[1], layer[2])) assert isinstance(layer[3], str) and layer[3].lower() in ["valid", "same"], error_msg_maxpool_layer elif layer_type_name == "avgpool": assert len(layer) == 4, error_msg_avgpool_layer for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_avgpool_layer assert isinstance(layer[3], str) and layer[3].lower() in ["valid", "same"], error_msg_avgpool_layer if layer[1] != layer[2]:print("NOTE that your avgpool kernel size {} isn't the same as your stride {}".format(layer[1], layer[2])) elif layer_type_name == "linear": assert len(layer) == 2, error_msg_linear_layer for ix in range(1): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0 else: raise ValueError("Invalid layer name") rest_must_be_linear = False for ix, layer in enumerate(all_layers): if rest_must_be_linear: assert layer[0].lower() == "linear", "If have linear layers then they must come at end" if layer[0].lower() == "linear": rest_must_be_linear = True def create_and_append_layer(self, layer, list_to_append_layer_to, activation=None, output_layer=False): """Creates and appends a layer to the list provided""" layer_name = layer[0].lower() assert layer_name in self.valid_cnn_hidden_layer_types, "Layer name {} not valid, use one of {}".format( layer_name, self.valid_cnn_hidden_layer_types) if layer_name == "conv": list_to_append_layer_to.extend([Conv2D(filters=layer[1], kernel_size=layer[2], strides=layer[3], padding=layer[4], activation=activation, kernel_initializer=self.initialiser_function)]) elif layer_name == "maxpool": list_to_append_layer_to.extend([MaxPool2D(pool_size=(layer[1], layer[1]), strides=(layer[2], layer[2]), padding=layer[3])]) elif layer_name == "avgpool": list_to_append_layer_to.extend([AveragePooling2D(pool_size=(layer[1], layer[1]), strides=(layer[2], layer[2]), padding=layer[3])]) elif layer_name == "linear": list_to_append_layer_to.extend([Dense(layer[1], activation=activation, kernel_initializer=self.initialiser_function)]) else: raise ValueError("Wrong layer name") def create_batch_norm_layers(self): """Creates the batch norm layers in the network""" batch_norm_layers = [] for layer in self.layers_info[:-1]: layer_type = layer[0].lower() if layer_type in ["conv", "linear"]: batch_norm_layers.extend([BatchNormalization()]) return batch_norm_layers def call(self, x, training=True): """Forward pass for the network. Note that it expects input data in the form (Batch, Height, Width, Channels)""" x = self.process_hidden_layers(x, training) out = self.process_output_layers(x) if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0]) * activations.sigmoid(out) return out def process_hidden_layers(self, x, training): """Puts the data x through all the hidden layers""" flattened=False training = training or training is None valid_batch_norm_layer_ix = 0 for layer_ix, layer in enumerate(self.hidden_layers): if type(layer) in self.valid_layer_types_with_no_parameters: x = layer(x) else: if type(layer) == Dense and not flattened: x = Flatten()(x) flattened = True x = layer(x) if self.batch_norm: x = self.batch_norm_layers[valid_batch_norm_layer_ix](x, training=False) valid_batch_norm_layer_ix += 1 if self.dropout != 0.0 and training: x = self.dropout_layer(x) if not flattened: x = Flatten()(x) return x def process_output_layers(self, x): """Puts the data x through all the output layers""" out = None for output_layer_ix, output_layer in enumerate(self.output_layers): temp_output = output_layer(x) if out is None: out = temp_output else: out = Concatenate(axis=1)([out, temp_output]) return out ================================================ FILE: nn_builder/tensorflow/NN.py ================================================ import tensorflow as tf from tensorflow.keras import Model, activations import numpy as np from tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, BatchNormalization from nn_builder.tensorflow.Base_Network import Base_Network class NN(Model, Base_Network): """Creates a PyTorch neural network Args: - layers_info: List of integers to indicate the width and number of linear layers you want in your network - hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer). Default is ReLU. - output_activation: String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads - dropout: Float to indicate what dropout probability you want applied after each hidden layer - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch initialisers are supported. PyTorch's default initialisation is the default. - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False - columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer before being fed through the other layers of the network. Default option is no embeddings - embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks. Default is no range restriction - random_seed: Integer to indicate the random seed you want to use """ def __init__(self, layers_info, output_activation=None, hidden_activations="relu", dropout=0.0, initialiser="default", batch_norm=False, columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (), random_seed=0, input_dim=None): Model.__init__(self) self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0 self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded self.embedding_dimensions = embedding_dimensions self.embedding_layers = self.create_embedding_layers() Base_Network.__init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed, input_dim) def check_all_user_inputs_valid(self): """Checks that all the user inputs were valid""" self.check_NN_layers_valid() self.check_activations_valid() self.check_embedding_dimensions_valid() self.check_initialiser_valid() self.check_y_range_values_valid() def create_and_append_layer(self, layer, list_to_append_layer_to, activation=None, output_layer=False): """Creates and appends a layer to the list provided""" list_to_append_layer_to.extend([Dense(layer, activation=activation, kernel_initializer=self.initialiser_function)]) def call(self, x, training=True): if self.embedding_to_occur: x = self.incorporate_embeddings(x) x = self.process_hidden_layers(x, training) out = self.process_output_layers(x) if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*activations.sigmoid(out) return out def incorporate_embeddings(self, x): """Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready to then be put through the hidden layers""" all_embedded_data = [] for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded): data = x[:, embedding_var] embedded_data = self.embedding_layers[embedding_layer_ix](data) all_embedded_data.append(embedded_data) if len(all_embedded_data) > 1: all_embedded_data = Concatenate(axis=1)(all_embedded_data) else: all_embedded_data = all_embedded_data[0] non_embedded_columns = [col for col in range(x.shape[1]) if col not in self.columns_of_data_to_be_embedded] if len(non_embedded_columns) > 0: x = tf.gather(x, non_embedded_columns, axis=1) x = Concatenate(axis=1)([tf.dtypes.cast(x, float), all_embedded_data]) else: x = all_embedded_data return x def process_hidden_layers(self, x, training): """Puts the data x through all the hidden layers""" for layer_ix, linear_layer in enumerate(self.hidden_layers): x = linear_layer(x) if self.batch_norm: x = self.batch_norm_layers[layer_ix](x, training=False) if self.dropout != 0.0 and (training or training is None): x = self.dropout_layer(x) return x def process_output_layers(self, x): """Puts the data x through all the output layers""" out = None for output_layer_ix, output_layer in enumerate(self.output_layers): temp_output = output_layer(x) if out is None: out = temp_output else: out = Concatenate(axis=1)(inputs=[out, temp_output]) return out ================================================ FILE: nn_builder/tensorflow/RNN.py ================================================ import numpy as np import tensorflow as tf from tensorflow.keras import Model, activations from tensorflow.keras.layers import Dense, Concatenate, GRU, LSTM from nn_builder.tensorflow.Base_Network import Base_Network class RNN(Model, Base_Network): """Creates a TensorFlow recurrent neural network Args: - layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be one of these 3 forms: - ["lstm", hidden_units] - ["gru", hidden_units] - ["linear", hidden_units] - hidden_activations: String or list of string to indicate the activations you want used on the output of linear hidden layers (not including the output layer). Default is ReLU. - output_activation: String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads - dropout: Float to indicate what dropout probability you want applied after each hidden layer - initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch initialisers are supported. PyTorch's default initialisation is the default. - batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False - columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer before being fed through the other layers of the network. Default option is no embeddings - embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings - y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks. Default is no range restriction - return_final_seq_only: Boolean to indicate whether you only want to return the output for the final timestep (True) or if you want to return the output for all timesteps (False) - random_seed: Integer to indicate the random seed you want to use NOTE that this class' call method expects input data in the form: (batch, sequence length, features) """ def __init__(self, layers_info, output_activation=None, hidden_activations="relu", dropout=0.0, initialiser="default", batch_norm=False, columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (), return_final_seq_only=True, random_seed=0, input_dim=None): Model.__init__(self) self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0 self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded self.embedding_dimensions = embedding_dimensions self.embedding_layers = self.create_embedding_layers() self.return_final_seq_only = return_final_seq_only self.valid_RNN_hidden_layer_types = {"linear", "gru", "lstm"} Base_Network.__init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed, input_dim) def check_all_user_inputs_valid(self): """Checks that all the user inputs were valid""" self.check_RNN_layers_valid() self.check_activations_valid() self.check_embedding_dimensions_valid() self.check_initialiser_valid() self.check_y_range_values_valid() self.check_return_final_seq_only_valid() def check_RNN_layers_valid(self): """Checks that layers provided by user are valid""" error_msg_layer_type = "First element in a layer specification must be one of {}".format(self.valid_RNN_hidden_layer_types) error_msg_layer_form = "Layer must be of form [layer_name, hidden_units]" error_msg_layer_list = "Layers must be provided as a list" error_msg_output_heads = "Number of output activations must equal number of output heads" assert isinstance(self.layers_info, list), error_msg_layer_list all_layers = self.layers_info[:-1] output_layer = self.layers_info[-1] assert isinstance(output_layer, list), error_msg_layer_list if isinstance(output_layer[0], list): assert len(output_layer) == len( self.output_activation), error_msg_output_heads for layer in output_layer: all_layers.append(layer) else: assert not isinstance(self.output_activation, list) or len(self.output_activation) == 1, error_msg_output_heads all_layers.append(output_layer) rest_must_be_linear = False for layer in all_layers: assert isinstance(layer, list), "Each layer must be a list" assert isinstance(layer[0], str), error_msg_layer_type layer_type_name = layer[0].lower() assert layer_type_name in self.valid_RNN_hidden_layer_types, "Layer name {} not valid, use one of {}".format( layer_type_name, self.valid_RNN_hidden_layer_types) assert isinstance(layer[1], int), error_msg_layer_form assert layer[1] > 0, "Must have hidden_units >= 1" assert len(layer) == 2, error_msg_layer_form if rest_must_be_linear: assert layer[0].lower() == "linear", "If have linear layers then they must come at end" if layer_type_name == "linear": rest_must_be_linear = True def create_and_append_layer(self, layer, rnn_hidden_layers, activation, output_layer=False): layer_type_name = layer[0].lower() hidden_size = layer[1] if output_layer and self.return_final_seq_only: return_sequences = False else: return_sequences = True if layer_type_name == "lstm": rnn_hidden_layers.extend([LSTM(units=hidden_size, kernel_initializer=self.initialiser_function, return_sequences=return_sequences)]) elif layer_type_name == "gru": rnn_hidden_layers.extend([GRU(units=hidden_size, kernel_initializer=self.initialiser_function, return_sequences=return_sequences)]) elif layer_type_name == "linear": rnn_hidden_layers.extend( [Dense(units=hidden_size, activation=activation, kernel_initializer=self.initialiser_function)]) else: raise ValueError("Wrong layer names") input_dim = hidden_size return input_dim def call(self, x, training=True): """Forward pass for the network. Note that it expects input data in the form (batch, seq length, features)""" if self.embedding_to_occur: x = self.incorporate_embeddings(x) training = training or training is None x, restricted_to_final_seq = self.process_hidden_layers(x, training) out = self.process_output_layers(x, restricted_to_final_seq) if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0]) * activations.sigmoid(out) return out def incorporate_embeddings(self, x): """Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready to then be put through the hidden layers""" all_embedded_data = [] for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded): data = x[:, :, embedding_var] embedded_data = self.embedding_layers[embedding_layer_ix](data) all_embedded_data.append(embedded_data) if len(all_embedded_data) > 1: all_embedded_data = Concatenate(axis=2)(all_embedded_data) else: all_embedded_data = all_embedded_data[0] non_embedded_columns = [col for col in range(x.shape[2]) if col not in self.columns_of_data_to_be_embedded] if len(non_embedded_columns) > 0: x = tf.gather(x, non_embedded_columns, axis=2) x = Concatenate(axis=2)([tf.dtypes.cast(x, float), all_embedded_data]) else: x = all_embedded_data return x def process_hidden_layers(self, x, training): """Puts the data x through all the hidden layers""" restricted_to_final_seq = False for layer_ix, layer in enumerate(self.hidden_layers): if type(layer) == Dense: if self.return_final_seq_only and not restricted_to_final_seq: x = x[:, -1, :] restricted_to_final_seq = True x = layer(x) else: x = layer(x) if self.batch_norm: x = self.batch_norm_layers[layer_ix](x, training=False) if self.dropout != 0.0 and training: x = self.dropout_layer(x) return x, restricted_to_final_seq def process_output_layers(self, x, restricted_to_final_seq): """Puts the data x through all the output layers""" out = None for output_layer_ix, output_layer in enumerate(self.output_layers): if type(output_layer) == Dense: if self.return_final_seq_only and not restricted_to_final_seq: x = x[:, -1, :] restricted_to_final_seq = True temp_output = output_layer(x) else: temp_output = output_layer(x) activation = self.get_activation(self.output_activation, output_layer_ix) temp_output = activation(temp_output) if out is None: out = temp_output else: if restricted_to_final_seq: dim = 1 else: dim = 2 out = Concatenate(axis=dim)([out, temp_output]) return out ================================================ FILE: nn_builder/tensorflow/__init__.py ================================================ ================================================ FILE: requirements.txt ================================================ tensorflow==2.0.0a0 torch==1.0.1.post2 torchvision==0.2.2.post3 numpy==1.16.2 setuptools==40.8.0 pytest==4.4.0 ================================================ FILE: setup.py ================================================ import setuptools import sys with open("README.md", "r") as fh: long_description = fh.read() setuptools.setup( name="nn_builder", version="1.0.5", author="Petros Christodoulou", author_email="p.christodoulou2@gmail.com", description="Build neural networks in 1 line", long_description=long_description, long_description_content_type="text/markdown", url="https://github.com/p-christ/nn_builder", packages=setuptools.find_packages(), classifiers=[ "Programming Language :: Python :: 3", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", ], install_requires=["tensorflow==2.0.0a0" if ( sys.platform.startswith("mac") or sys.platform.startswith("darwin")) else "tensorflow-gpu==2.0.0a0"] ) ================================================ FILE: tests/pytorch_tests/test_pytorch_CNN.py ================================================ # Run from home directory with python -m pytest tests import shutil import pytest import torch import random import numpy as np import torch.nn as nn from nn_builder.pytorch.CNN import CNN import torch.optim as optim from torchvision import datasets, transforms N = 250 X = torch.randn((N, 1, 5, 5)) X[0:125, 0, 3, 3] += 20.0 y = X[:, 0, 3, 3] > 5.0 y = y.float() def test_user_hidden_layers_input_rejections(): """Tests whether network rejects invalid hidden_layers inputted from user""" inputs_that_should_fail = [['maxpool', 33, 22, 33], [['a']], [[222, 222, 222, 222]], [["conv", 2, 2, -1]], [["conv", 2, 2]], [["conv", 2, 2, 55, 999, 33]], [["maxpool", 33, 33]], [["maxpool", -1, 33]], [["maxpool", 33]], [["maxpoolX", 1, 33]], [["cosnv", 2, 2]], [["avgpool", 33, 33, 333, 99]], [["avgpool", -1, 33]], [["avgpool", 33]], [["avgpoolX", 1, 33]], [["adaptivemaxpool", 33, 33, 333, 33]], [["adaptivemaxpool", 2]], [["adaptivemaxpool", 33]], [["adaptivemaxpoolX"]], [["adaptiveavgpool", 33, 33, 333, 11]], [["adaptiveavgpool", 2]], [["adaptiveavgpool", 33]], [["adaptiveavgpoolX"]], [["linear", 40, -2]], [["lineafr", 40, 2]]] for input in inputs_that_should_fail: print(input) with pytest.raises(AssertionError): CNN(input_dim=1, layers_info=input, hidden_activations="relu", output_activation="relu") def test_user_hidden_layers_input_acceptances(): """Tests whether network rejects invalid hidden_layers inputted from user""" inputs_that_should_work = [[["conv", 2, 2, 3331, 2]], [["CONV", 2, 2, 3331, 22]], [["ConV", 2, 2, 3331, 1]], [["maxpool", 2, 2, 3331]], [["MAXPOOL", 2, 2, 3331]], [["MaXpOOL", 2, 2, 3331]], [["avgpool", 2, 2, 3331]], [["AVGPOOL", 2, 2, 3331]], [["avGpOOL", 2, 2, 3331]], [["adaptiveavgpool", 3, 22]], [["ADAPTIVEAVGpOOL", 1, 33]], [["ADAPTIVEaVGPOOL", 3, 6]], [["adaptivemaxpool", 4, 66]], [["ADAPTIVEMAXpOOL", 2, 2]], [["ADAPTIVEmaXPOOL", 3, 3]], [["adaptivemaxpool", 3, 3]], [["ADAPTIVEMAXpOOL", 3, 1]], [["linear", 40]], [["lineaR", 2]], [["LINEAR", 2]]] for ix, input in enumerate(inputs_that_should_work): input.append(["linear", 5]) CNN(input_dim=(1, 1, 1), layers_info=input, hidden_activations="relu", output_activation="relu") def test_hidden_layers_created_correctly(): """Tests that create_hidden_layers works correctly""" layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34], ["adaptiveavgpool", 23, 44], ["linear", 22], ["linear", 2222], ["linear", 5]] cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations="relu", output_activation="relu") assert type(cnn.hidden_layers[0]) == nn.Conv2d assert cnn.hidden_layers[0].in_channels == 3 assert cnn.hidden_layers[0].out_channels == 2 assert cnn.hidden_layers[0].kernel_size == (4, 4) assert cnn.hidden_layers[0].stride == (3, 3) assert cnn.hidden_layers[0].padding == (2, 2) assert type(cnn.hidden_layers[1]) == nn.MaxPool2d assert cnn.hidden_layers[1].kernel_size == 3 assert cnn.hidden_layers[1].stride == 4 assert cnn.hidden_layers[1].padding == 2 assert type(cnn.hidden_layers[2]) == nn.AvgPool2d assert cnn.hidden_layers[2].kernel_size == 32 assert cnn.hidden_layers[2].stride == 42 assert cnn.hidden_layers[2].padding == 22 assert type(cnn.hidden_layers[3]) == nn.AdaptiveMaxPool2d assert cnn.hidden_layers[3].output_size == (3, 34) assert type(cnn.hidden_layers[4]) == nn.AdaptiveAvgPool2d assert cnn.hidden_layers[4].output_size == (23, 44) assert type(cnn.hidden_layers[5]) == nn.Linear assert cnn.hidden_layers[5].in_features == 2024 assert cnn.hidden_layers[5].out_features == 22 assert type(cnn.hidden_layers[6]) == nn.Linear assert cnn.hidden_layers[6].in_features == 22 assert cnn.hidden_layers[6].out_features == 2222 assert type(cnn.output_layers[0]) == nn.Linear assert cnn.output_layers[0].in_features == 2222 assert cnn.output_layers[0].out_features == 5 def test_output_layers_created_correctly(): """Tests that create_output_layers works correctly""" layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34], ["adaptiveavgpool", 23, 44], ["linear", 22], ["linear", 2222], ["linear", 2]] cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations="relu", output_activation="relu") assert cnn.output_layers[0].in_features == 2222 assert cnn.output_layers[0].out_features == 2 layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34], ["adaptiveavgpool", 23, 44], ["linear", 7]] cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations="relu", output_activation="relu") assert cnn.output_layers[0].in_features == 23 * 44 * 2 assert cnn.output_layers[0].out_features == 7 layers = [["conv", 5, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34], ["linear", 6]] cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations="relu", output_activation="relu") assert cnn.output_layers[0].in_features == 3 * 34 * 5 assert cnn.output_layers[0].out_features == 6 layers = [["conv", 5, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34], [["linear", 6], ["linear", 22]]] cnn = CNN(input_dim=(3, 1000, 1000), layers_info=layers, hidden_activations="relu", output_activation=["softmax", None]) assert cnn.output_layers[0].in_features == 3 * 34 * 5 assert cnn.output_layers[0].out_features == 6 assert cnn.output_layers[1].in_features == 3 * 34 * 5 assert cnn.output_layers[1].out_features == 22 def test_output_dim_user_input(): """Tests whether network rejects an invalid output_dim input from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): CNN(input_dim=(2, 10, 10), layers_info=[2, input_value], hidden_activations="relu", output_activation="relu") with pytest.raises(AssertionError): CNN(input_dim=(2, 10, 10), layers_info=input_value, hidden_activations="relu", output_activation="relu") def test_activations_user_input(): """Tests whether network rejects an invalid hidden_activations or output_activation from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): CNN(input_dim=(2, 10, 10), layers_info=[["conv", 2, 2, 3331, 2]], hidden_activations=input_value, output_activation="relu") CNN(input_dim=(2, 10, 10), layers_info=[["conv", 2, 2, 3331, 2]], hidden_activations="relu", output_activation=input_value) def test_initialiser_user_input(): """Tests whether network rejects an invalid initialiser from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): CNN(input_dim=(2, 10, 10), layers_info=[["conv", 2, 2, 3331, 2]], hidden_activations="relu", output_activation="relu", initialiser=input_value) CNN(layers_info=[["conv", 2, 2, 3331, 2], ["linear", 3]], hidden_activations="relu", output_activation="relu", initialiser="xavier", input_dim=(2, 10, 10)) def test_batch_norm_layers(): """Tests whether batch_norm_layers method works correctly""" layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["adaptivemaxpool", 3, 34], ["linear", 5]] cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10), output_activation="relu", initialiser="xavier", batch_norm=False) layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["adaptivemaxpool", 3, 34], ["linear", 5]] cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10), output_activation="relu", initialiser="xavier", batch_norm=True) assert len(cnn.batch_norm_layers) == 1 assert cnn.batch_norm_layers[0].num_features == 2 layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["conv", 12, 4, 3, 2], ["adaptivemaxpool", 3, 34], ["linear", 22], ["linear", 55]] cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10), output_activation="relu", initialiser="xavier", batch_norm=True) assert len(cnn.batch_norm_layers) == 3 assert cnn.batch_norm_layers[0].num_features == 2 assert cnn.batch_norm_layers[1].num_features == 12 assert cnn.batch_norm_layers[2].num_features == 22 def test_linear_layers_acceptance(): """Tests that only accepts linear layers of correct shape""" layers_that_shouldnt_work = [[["linear", 2, 5]], [["linear", 2, 5, 5]], [["linear"]], [["linear", 2], ["linear", 5, 4]], ["linear", 0], ["linear", -5]] for layers in layers_that_shouldnt_work: with pytest.raises(AssertionError): cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10), output_activation="relu", initialiser="xavier", batch_norm=True) layers_that_should_work = [[["linear", 44], ["linear", 2]], [["linear", 22]]] for layer in layers_that_should_work: assert CNN(layers_info=layer, hidden_activations="relu", input_dim=(2, 10, 10), output_activation="relu", initialiser="xavier", batch_norm=True) def test_linear_layers_only_come_at_end(): """Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they don't only come at the end""" layers = [["conv", 2, 4, 3, 2], ["linear", 55], ["maxpool", 3, 4, 2], ["adaptivemaxpool", 3, 34]] with pytest.raises(AssertionError): cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10), output_activation="relu", initialiser="xavier", batch_norm=True) layers = [["conv", 2, 4, 3, 2], ["linear", 55]] assert CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10), output_activation="relu", initialiser="xavier", batch_norm=True) layers = [["conv", 2, 4, 3, 2], ["linear", 55], ["linear", 55], ["linear", 55]] assert CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10), output_activation="relu", initialiser="xavier", batch_norm=True) def test_output_activation(): """Tests whether network outputs data that has gone through correct activation function""" RANDOM_ITERATIONS = 20 input_dim = (5, 100, 100) for _ in range(RANDOM_ITERATIONS): data = torch.randn((1, *input_dim)) CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 50]], hidden_activations="relu", input_dim=input_dim, output_activation="relu", initialiser="xavier") out = CNN_instance.forward(data) assert all(out.squeeze() >= 0) CNN_instance = CNN(layers_info=[["conv", 2, 20, 1, 0], ["linear", 5]], hidden_activations="relu", input_dim=input_dim, output_activation="relu", initialiser="xavier") out = CNN_instance.forward(data) assert all(out.squeeze() >= 0) CNN_instance = CNN(layers_info=[["conv", 5, 20, 1, 0], ["linear", 5]], hidden_activations="relu", input_dim=input_dim, output_activation="relu", initialiser="xavier") out = CNN_instance.forward(data) assert all(out.squeeze() >= 0) CNN_instance = CNN(layers_info=[["conv", 5, 20, 1, 0], ["linear", 22]], hidden_activations="relu", input_dim=input_dim, output_activation="sigmoid", initialiser="xavier") out = CNN_instance.forward(data) assert all(out.squeeze() >= 0) assert all(out.squeeze() <= 1) assert round(torch.sum(out.squeeze()).item(), 3) != 1.0 CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 5]], hidden_activations="relu", input_dim=input_dim, output_activation="softmax", initialiser="xavier") out = CNN_instance.forward(data) assert all(out.squeeze() >= 0) assert all(out.squeeze() <= 1) assert round(torch.sum(out.squeeze()).item(), 3) == 1.0 CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 5]], hidden_activations="relu", input_dim=input_dim, initialiser="xavier") out = CNN_instance.forward(data) assert not all(out.squeeze() >= 0) assert not round(torch.sum(out.squeeze()).item(), 3) == 1.0 def test_y_range(): """Tests whether setting a y range works correctly""" for _ in range(100): val1 = random.random() - 3.0*random.random() val2 = random.random() + 2.0*random.random() lower_bound = min(val1, val2) upper_bound = max(val1, val2) CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 5]], hidden_activations="relu", y_range=(lower_bound, upper_bound), initialiser="xavier", input_dim=(1, 20, 20)) random_data = torch.randn((10, 1, 20, 20)) out = CNN_instance.forward(random_data) assert torch.sum(out > lower_bound).item() == 10*5, "lower {} vs. {} ".format(lower_bound, out) assert torch.sum(out < upper_bound).item() == 10*5, "upper {} vs. {} ".format(upper_bound, out) def test_deals_with_None_activation(): """Tests whether is able to handle user inputting None as output activation""" assert CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 5]], hidden_activations="relu", output_activation=None, initialiser="xavier", input_dim=(5, 5, 5)) def test_check_input_data_into_forward_once(): """Tests that check_input_data_into_forward_once method only runs once""" CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 6]], hidden_activations="relu", input_dim=(4, 2, 5), output_activation="relu", initialiser="xavier") data_not_to_throw_error = torch.randn((1, 4, 2, 5)) data_to_throw_error = torch.randn((1, 2, 20, 20)) with pytest.raises(AssertionError): CNN_instance.forward(data_to_throw_error) with pytest.raises(RuntimeError): CNN_instance.forward(data_not_to_throw_error) CNN_instance.forward(data_to_throw_error) def test_y_range_user_input(): """Tests whether network rejects invalid y_range inputs""" invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)] for y_range_value in invalid_y_range_inputs: with pytest.raises(AssertionError): print(y_range_value) CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2]], hidden_activations="relu", y_range=y_range_value, input_dim=(2, 2, 2), initialiser="xavier") def test_model_trains(): """Tests whether a small range of networks can solve a simple task""" for output_activation in ["sigmoid", "None"]: CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation=output_activation, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) z = X[:, 0:1, 3:4, 3:4] > 5.0 z = torch.cat([z ==1, z==0], dim=1).float() z = z.squeeze(-1).squeeze(-1) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["linear", 2]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation="softmax", dropout=0.01, initialiser="xavier") assert solves_simple_problem(X, z, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["linear", 1]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["linear", 1]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation=None, initialiser="xavier", batch_norm=True) assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["maxpool", 1, 1, 0], ["linear", 1]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["avgpool", 1, 1, 0], ["linear", 1]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 5, 3, 1, 0], ["adaptivemaxpool", 2, 2], ["linear", 1]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 5, 3, 1, 0], ["adaptiveavgpool", 2, 2], ["linear", 1]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) def test_model_trains_linear_layer(): """Tests whether a small range of networks can solve a simple task""" CNN_instance = CNN(layers_info=[["conv", 5, 3, 1, 0], ["linear", 5], ["linear", 5], ["linear", 1]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation="sigmoid", initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["linear", 5], ["linear", 5], ["linear", 1]], input_dim=(1, 5, 5), hidden_activations="relu", output_activation="sigmoid", initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) def test_max_pool_working(): """Tests whether max pool layers work properly""" N = 250 X = torch.randn((N, 1, 8, 8)) X[0:125, 0, 3, 3] = 999.99 CNN_instance = CNN(layers_info=[["maxpool", 2, 2, 0], ["maxpool", 2, 2, 0], ["maxpool", 2, 2, 0], ["linear", 1]], hidden_activations="relu", input_dim=(1, 8, 8), initialiser="xavier") assert CNN_instance(X).shape == (N, 1) def test_dropout(): """Tests whether dropout layer reads in probability correctly""" CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]], hidden_activations="relu", output_activation="sigmoid", dropout=0.9999, initialiser="xavier", input_dim=(1, 5, 5)) assert CNN_instance.dropout_layer.p == 0.9999 assert not solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]], hidden_activations="relu", output_activation=None, dropout=0.0000001, initialiser="xavier", input_dim=(1, 5, 5)) assert CNN_instance.dropout_layer.p == 0.0000001 assert solves_simple_problem(X, y, CNN_instance) def solves_simple_problem(X, y, nn_instance): """Checks if a given network is able to solve a simple problem""" optimizer = optim.Adam(nn_instance.parameters(), lr=0.15) for ix in range(800): out = nn_instance.forward(X) loss = torch.sum((out.squeeze() - y) ** 2) / N optimizer.zero_grad() loss.backward() optimizer.step() print("LOSS ", loss) return loss < 0.1 def test_MNIST_progress(): """Tests that network made using CNN module can make progress on MNIST""" batch_size = 128 train_loader = torch.utils.data.DataLoader( datasets.MNIST(root="input/", train=True, download=True, transform=transforms.Compose([ transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,)) ])), batch_size=batch_size, shuffle=True) for batch_norm in [True, False]: cnn = CNN(layers_info=[["conv", 20, 5, 1, 0], ["maxpool", 2, 2, 0], ["conv", 50, 5, 1, 0], ["maxpool", 2, 2, 0], ["linear", 500], ["linear", 10]], hidden_activations="relu", output_activation="softmax", initialiser="xavier", input_dim=(1, 28, 28), batch_norm=batch_norm) loss_fn = nn.CrossEntropyLoss() optimizer = optim.Adam(cnn.parameters(), lr=0.001) ix = 0 accuracies = [] for data, target in train_loader: ix += 1 output = cnn(data) loss = loss_fn(output, target) optimizer.zero_grad() loss.backward() optimizer.step() pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability correct = pred.eq(target.view_as(pred)).sum().item() print("Accuracy {}".format(correct / batch_size)) accuracies.append(correct / batch_size) if ix > 200: break assert accuracies[-1] > 0.7, "Accuracy not good enough {}".format(accuracies[-1]) shutil.rmtree("input/", ignore_errors=False, onerror=None) def test_all_activations_work(): """Tests that all activations get accepted""" nn_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]], dropout=0.0000001, initialiser="xavier", input_dim=(1, 5, 5)) for key in nn_instance.str_to_activations_converter.keys(): if key == "none": hidden_key = "relu" else: hidden_key = key model = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]], hidden_activations=hidden_key, output_activation=key, dropout=0.0000001, initialiser="xavier", input_dim=(1, 5, 5)) model(X) def test_all_initialisers_work(): """Tests that all initialisers get accepted""" nn_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]], dropout=0.0000001, initialiser="xavier", input_dim=(1, 5, 5)) for key in nn_instance.str_to_initialiser_converter.keys(): if key != "eye": model = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]], dropout=0.0000001, initialiser=key, input_dim=(1, 5, 5)) model(X) def test_print_model_summary(): nn_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]], dropout=0.0000001, initialiser="xavier", input_dim=(1, 5, 5)) nn_instance.print_model_summary() def test_output_heads_error_catching(): """Tests that having multiple output heads catches errors from user inputs""" output_dims_that_should_break = [["linear", 2, 2, "SAME", "conv", 3, 4, "SAME"], [[["conv", 3, 2, "same"], ["linear", 4]]], [[2, 8]], [-33, 33, 33, 33, 33]] for output_dim in output_dims_that_should_break: with pytest.raises(AssertionError): CNN(input_dim=(12, 12, 3), layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1], output_dim], hidden_activations="relu", output_activation="relu") output_activations_that_should_break = ["relu", ["relu"], ["relu", "softmax"]] for output_activation in output_activations_that_should_break: with pytest.raises(AssertionError): CNN(input_dim=(12, 12, 3), layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1], [["linear", 4], ["linear", 10], ["linear", 4]]], hidden_activations="relu", output_activation=output_activation) def test_output_head_layers(): """Tests whether the output head layers get created properly""" for output_dim in [[["linear", 3],["linear", 9]], [["linear", 4], ["linear", 20]], [["linear", 1], ["linear", 1]]]: nn_instance = CNN(input_dim=(12, 12, 3), layers_info=[["conv", 25, 5, 1, 2], ["conv", 25, 5, 1, 3], ["linear", 5], output_dim], hidden_activations="relu", output_activation=["softmax", None]) assert nn_instance.output_layers[0].out_features == output_dim[0][1] assert nn_instance.output_layers[0].in_features == 5 assert nn_instance.output_layers[1].out_features == output_dim[1][1] assert nn_instance.output_layers[1].in_features == 5 def test_output_head_activations_work(): """Tests that output head activations work properly""" output_dim = [["linear", 5], ["linear", 10], ["linear", 3]] nn_instance = CNN(input_dim=(12, 12, 3), layers_info=[["conv", 3, 2, 1, 1], ["conv", 3, 1, 1, 1], ["linear", 1], output_dim], hidden_activations="relu", output_activation=["softmax", None, "relu"]) x = torch.randn((20, 12, 12, 3)) * -20.0 out = nn_instance(x) assert out.shape == (20, 18) sums = torch.sum(out[:, :5], dim=1).detach().numpy() sums_others = torch.sum(out[:, 5:], dim=1).detach().numpy() sums_others_2 = torch.sum(out[:, 5:15], dim=1).detach().numpy() sums_others_3 = torch.sum(out[:, 15:18], dim=1).detach().numpy() for row in range(out.shape[0]): assert np.round(sums[row], 4) == 1.0, sums[row] assert not np.round(sums_others[row], 4) == 1.0, sums_others[row] assert not np.round(sums_others_2[row], 4) == 1.0, sums_others_2[row] assert not np.round(sums_others_3[row], 4) == 1.0, sums_others_3[row] for col in range(3): assert out[row, 15 + col] >= 0.0, out[row, 15 + col] def test_output_head_shapes_correct(): """Tests that the output shape of network is correct when using multiple outpout heads""" N = 20 X = torch.randn((N, 25, 25, 2)) for _ in range(25): nn_instance = CNN(input_dim=(25, 25, 2), layers_info=[["conv", 25, 2, 1, 1], ["conv", 2, 5, 1, 1], ["linear", 1], ["linear", 12]], hidden_activations="relu") out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 12 for output_dim in [[ ["linear", 10], ["linear", 4], ["linear", 6]], [["linear", 3], ["linear", 8], ["linear", 9]]]: nn_instance = CNN(input_dim=(25, 25, 2), layers_info=[["conv", 25, 1, 1, 2], ["conv", 25, 5, 1, 1], ["linear", 1], output_dim], hidden_activations="relu", output_activation=["softmax", None, "relu"]) out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 20 ================================================ FILE: tests/pytorch_tests/test_pytorch_NN.py ================================================ # Run from home directory with python -m pytest tests import copy import shutil from sklearn.datasets import load_boston from sklearn.model_selection import train_test_split import pytest import torch import random import numpy as np import torch.nn as nn import torch.optim as optim from nn_builder.pytorch.NN import NN from sklearn.utils import shuffle N = 250 X = torch.randn((N, 5)) X[:, [2, 4]] += 10.0 y = X[:, 0] > 0 y = y.float() def test_linear_hidden_units_user_input(): """Tests whether network rejects an invalid linear_hidden_units input from user""" inputs_that_should_fail = ["a", ["a", "b"], [2, 4, "ss"], [-2], 2] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): NN(input_dim=2, layers_info=input_value, hidden_activations="relu", output_activation="relu") def test_input_dim_output_dim_user_input(): """Tests whether network rejects an invalid input_dim from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): NN(input_dim=input_value, layers_info=[2], hidden_activations="relu", output_activation="relu") def test_activations_user_input(): """Tests whether network rejects an invalid hidden_activations or output_activation from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): NN(input_dim=2, layers_info=[2], hidden_activations=input_value, output_activation="relu") NN(input_dim=2, layers_info=[2], hidden_activations="relu", output_activation=input_value) def test_initialiser_user_input(): """Tests whether network rejects an invalid initialiser from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): NN(input_dim=2, layers_info=[2], hidden_activations="relu", output_activation="relu", initialiser=input_value) NN(input_dim=2, layers_info=[2], hidden_activations="relu", output_activation="relu", initialiser="xavier") def test_output_shape_correct(): """Tests whether network returns output of the right shape""" input_dims = [x for x in range(1, 3)] output_dims = [x for x in range(4, 6)] linear_hidden_units_options = [ [2, 3, 4], [2, 9, 1], [55, 55, 55, 234, 15]] for input_dim, output_dim, linear_hidden_units in zip(input_dims, output_dims, linear_hidden_units_options): linear_hidden_units.append(output_dim) nn_instance = NN(input_dim=input_dim, layers_info=linear_hidden_units, hidden_activations="relu", output_activation="relu", initialiser="xavier") data = torch.randn((25, input_dim)) output = nn_instance.forward(data) assert output.shape == (25, output_dim) def test_output_activation(): """Tests whether network outputs data that has gone through correct activation function""" RANDOM_ITERATIONS = 20 for _ in range(RANDOM_ITERATIONS): data = torch.randn((1, 100)) nn_instance = NN(input_dim=100, layers_info=[5, 5, 5], hidden_activations="relu", output_activation="relu", initialiser="xavier") out = nn_instance.forward(data) assert all(out.squeeze() >= 0) nn_instance = NN(input_dim=100, layers_info=[5, 5, 5], hidden_activations="relu", output_activation="sigmoid", initialiser="xavier") out = nn_instance.forward(data) assert all(out.squeeze() >= 0) assert all(out.squeeze() <= 1) nn_instance = NN(input_dim=100, layers_info=[5, 5, 5], hidden_activations="relu", output_activation="softmax", initialiser="xavier") out = nn_instance.forward(data) assert all(out.squeeze() >= 0) assert all(out.squeeze() <= 1) assert round(torch.sum(out.squeeze()).item(), 3) == 1.0 nn_instance = NN(input_dim=100, layers_info=[5, 5, 5], hidden_activations="relu", ) out = nn_instance.forward(data) assert not all(out.squeeze() >= 0) assert not round(torch.sum(out.squeeze()).item(), 3) == 1.0 def test_linear_layers(): """Tests whether create_hidden_layers method works correctly""" for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]): hidden_units.append(output_dim) nn_instance = NN(input_dim=input_dim, layers_info=hidden_units, hidden_activations="relu", output_activation="relu", initialiser="xavier") for layer in nn_instance.hidden_layers: assert isinstance(layer, nn.Linear) assert nn_instance.hidden_layers[0].in_features == input_dim assert nn_instance.hidden_layers[0].out_features == hidden_units[0] assert nn_instance.hidden_layers[1].in_features == hidden_units[0] assert nn_instance.hidden_layers[1].out_features == hidden_units[1] assert nn_instance.hidden_layers[2].in_features == hidden_units[1] assert nn_instance.hidden_layers[2].out_features == hidden_units[2] assert len(nn_instance.hidden_layers) == 3 def test_embedding_layers(): """Tests whether create_embedding_layers method works correctly""" for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)): nn_instance = NN(input_dim=5, layers_info=[5], embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]]) for layer in nn_instance.embedding_layers: assert isinstance(layer, nn.Embedding) assert len(nn_instance.embedding_layers) == 2 assert nn_instance.embedding_layers[0].num_embeddings == embedding_in_dim_1 assert nn_instance.embedding_layers[0].embedding_dim == embedding_out_dim_1 assert nn_instance.embedding_layers[1].num_embeddings == embedding_in_dim_2 assert nn_instance.embedding_layers[1].embedding_dim == embedding_out_dim_2 def test_non_integer_embeddings_rejected(): """Tests whether an error is raised if user tries to provide non-integer data to be embedded""" with pytest.raises(AssertionError): nn_instance = NN(input_dim=5, layers_info=[5], columns_of_data_to_be_embedded=[2, 4], embedding_dimensions=[[50, 3], [55, 4]]) out = nn_instance.forward(X) def test_incorporate_embeddings(): """Tests the method incorporate_embeddings""" X_new = X X_new[:, [2, 4]] = torch.round(X_new[:, [2, 4]]) nn_instance = NN(input_dim=5, layers_info=[5], columns_of_data_to_be_embedded=[2, 4], embedding_dimensions=[[50, 3], [55, 4]]) out = nn_instance.incorporate_embeddings(X) assert out.shape == (N, X.shape[1]+3+4-2) def test_embedding_network_can_solve_simple_problem(): """Tests whether network can solve simple problem using embeddings""" X = torch.randn(N, 2) * 5.0 + 20.0 y = (X[:, 0] >= 20) * (X[:, 1] <= 20) X = X.long() nn_instance = NN(input_dim=2, layers_info=[5, 1], columns_of_data_to_be_embedded=[0, 1], embedding_dimensions=[[50, 3], [55, 3]]) assert solves_simple_problem(X, y.float(), nn_instance) def test_batch_norm_layers(): """Tests whether batch_norm_layers method works correctly""" for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]): hidden_units.append(output_dim) nn_instance = NN(input_dim=input_dim, layers_info=hidden_units, hidden_activations="relu", batch_norm=True, output_activation="relu", initialiser="xavier") for layer in nn_instance.batch_norm_layers: assert isinstance(layer, nn.BatchNorm1d) assert len(nn_instance.batch_norm_layers) == len(hidden_units) - 1 assert nn_instance.batch_norm_layers[0].num_features == hidden_units[0] assert nn_instance.batch_norm_layers[1].num_features == hidden_units[1] assert nn_instance.batch_norm_layers[2].num_features == hidden_units[2] def test_model_trains(): """Tests whether a small range of networks can solve a simple task""" for output_activation in ["sigmoid", "None"]: nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 10, 1], output_activation=output_activation, dropout=0.01, batch_norm=True) assert solves_simple_problem(X, y, nn_instance) z = X[:, 0:1] > 0 z = torch.cat([z ==1, z==0], dim=1).float() nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 10, 2], output_activation="softmax", dropout=0.01, batch_norm=True) assert solves_simple_problem(X, z, nn_instance) def solves_simple_problem(X, y, nn_instance): """Checks if a given network is able to solve a simple problem""" optimizer = optim.Adam(nn_instance.parameters(), lr=0.15) for ix in range(800): out = nn_instance.forward(X) loss = torch.sum((out.squeeze() - y) ** 2) / N optimizer.zero_grad() loss.backward() optimizer.step() return loss < 0.1 def test_dropout(): """Tests whether dropout layer reads in probability correctly""" nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999) assert nn_instance.dropout_layer.p == 0.9999 assert not solves_simple_problem(X, y, nn_instance) nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.00001) assert solves_simple_problem(X, y, nn_instance) def test_y_range_user_input(): """Tests whether network rejects invalid y_range inputs""" invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)] for y_range_value in invalid_y_range_inputs: with pytest.raises(AssertionError): print(y_range_value) nn_instance = NN(input_dim=5, layers_info=[10, 10, 3], y_range=y_range_value) def test_y_range(): """Tests whether setting a y range works correctly""" for _ in range(100): val1 = random.random() - 3.0*random.random() val2 = random.random() + 2.0*random.random() lower_bound = min(val1, val2) upper_bound = max(val1, val2) nn_instance = NN(input_dim=5, layers_info=[10, 10, 3], y_range=(lower_bound, upper_bound)) random_data = torch.randn(15, 5) out = nn_instance.forward(random_data) assert torch.sum(out > lower_bound).item() == 3*15, "lower {} vs. {} ".format(lower_bound, out) assert torch.sum(out < upper_bound).item() == 3*15, "upper {} vs. {} ".format(upper_bound, out) def test_deals_with_None_activation(): """Tests whether is able to handle user inputting None as output activation""" assert NN(input_dim=5, layers_info=[10, 10, 3], output_activation=None) def test_check_input_data_into_forward_once(): """Tests that check_input_data_into_forward_once method only runs once""" data_to_throw_error = torch.randn(N, 2) X = torch.randn(N, 2) * 5.0 + 20.0 y = (X[:, 0] >= 20) * (X[:, 1] <= 20) X = X.long() nn_instance = NN(input_dim=2, layers_info=[5, 1], columns_of_data_to_be_embedded=[0, 1], embedding_dimensions=[[50, 3], [55, 3]]) with pytest.raises(AssertionError): nn_instance.forward(data_to_throw_error) with pytest.raises(RuntimeError): nn_instance.forward(X) nn_instance.forward(data_to_throw_error) def test_all_activations_work(): """Tests that all activations get accepted""" nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999) for key in nn_instance.str_to_activations_converter.keys(): if key == "none": hidden_key = "relu" else: hidden_key = key model = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999, hidden_activations=hidden_key, output_activation=key) model(X) def test_all_initialisers_work(): """Tests that all initialisers get accepted""" nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999) for key in nn_instance.str_to_initialiser_converter.keys(): model = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999, initialiser=key) model(X) def test_print_model_summary(): nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999) nn_instance.print_model_summary() def test_output_heads_error_catching(): """Tests that having multiple output heads catches errors from user inputs""" output_dims_that_should_break = [[[2, 8]], [-33, 33, 33, 33, 33]] for output_dim in output_dims_that_should_break: with pytest.raises(AssertionError): NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations="relu", output_activation="relu") output_activations_that_should_break = ["relu", ["relu"], ["relu", "softmax"]] for output_activation in output_activations_that_should_break: with pytest.raises(AssertionError): NN(input_dim=2, layers_info=[4, 7, 9, [4, 6, 1]], hidden_activations="relu", output_activation=output_activation) def test_output_head_layers(): """Tests whether the output head layers get created properly""" for output_dim in [[3, 9], [4, 20], [1, 1]]: nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations="relu", output_activation=["softmax", None]) assert nn_instance.output_layers[0].in_features == 9 assert nn_instance.output_layers[1].in_features == 9 assert nn_instance.output_layers[0].out_features == output_dim[0] assert nn_instance.output_layers[1].out_features == output_dim[1] def test_output_head_activations_work(): """Tests that output head activations work properly""" nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, [5, 10, 3]], hidden_activations="relu", output_activation=["softmax", None, "relu"]) x = torch.randn((20, 2)) * -20.0 out = nn_instance.forward(x) assert torch.allclose(torch.sum(out[:, :5], dim=1), torch.Tensor([1.0])) assert not torch.allclose(torch.sum(out[:, 5:], dim=1), torch.Tensor([1.0])) for row in range(out.shape[0]): assert all(out[row, -3:] >= 0) def test_output_head_shapes_correct(): """Tests that the output shape of network is correct when using multiple outpout heads""" N = 20 X = torch.randn((N, 2)) for _ in range(25): output_dim = random.randint(1, 100) nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations="relu") out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == output_dim for output_dim in [[3, 9, 5, 3], [5, 5, 5, 5], [2, 1, 1, 16]]: nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations="relu", output_activation=["softmax", None, None, "relu"]) out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 20 def train_on_boston_housing(model, X, y): # Compile the model loss_fn = nn.MSELoss() optimizer = optim.Adam(model.parameters(), lr=0.08) X = torch.Tensor(X) y = torch.Tensor(y) for _ in range(1000): output = model(X) loss = loss_fn(output.squeeze(-1), y) optimizer.zero_grad() loss.backward() optimizer.step() print(loss) return loss def test_boston_housing_progress(): """Tests that network made using NN module can make progress on boston housing dataset""" boston = load_boston() X, y = (boston.data, boston.target) # Normalise the data mean = np.mean(X, axis=0) std = np.std(X, axis=0) X = (X - mean) / std model = NN(layers_info=[30, 10, 1], input_dim=13, hidden_activations="relu", output_activation=None, dropout=0.0, initialiser="xavier", batch_norm=True, y_range=(4.5, 55.0)) result = train_on_boston_housing(model, X, y) assert result < 15 model = NN(layers_info=[15, 15, 15, 15, 1], input_dim=13, random_seed=52, hidden_activations="relu", output_activation=None, dropout=0.0, initialiser="xavier", batch_norm=False) result = train_on_boston_housing(model, X, y) assert result < 15 model = NN(layers_info=[30, 10, 1], input_dim=13, hidden_activations="relu", output_activation=None, dropout=0.0, initialiser="xavier", batch_norm=True) result = train_on_boston_housing(model, X, y) assert result < 15 model = NN(layers_info=[15, 15, 15, 1], input_dim=13, hidden_activations="relu", output_activation=None, dropout=0.05, initialiser="xavier", batch_norm=False) result = train_on_boston_housing(model, X, y) assert result < 15 ================================================ FILE: tests/pytorch_tests/test_pytorch_RNN.py ================================================ # Run from home directory with python -m pytest tests import copy import pytest import random import numpy as np import torch.nn as nn import torch from nn_builder.pytorch.RNN import RNN import torch.optim as optim N = 250 X = torch.randn((N, 5, 15)) X[0:125, 0, 3] += 20.0 y = X[:, 0, 3] > 5.0 y = y.float() def test_user_hidden_layers_input_rejections(): """Tests whether network rejects invalid hidden_layers inputted from user""" inputs_that_should_fail = [[["linearr", 33]], [["linear", 12, 33]], [["gru", 2, 33]], [["lstm", 2, 33]], [["lstmr", 33]], [["gruu", 33]], [["gru", 33], ["xxx", 33]], [["linear", 33], ["gru", 12], ["gru", 33]] ] for input in inputs_that_should_fail: with pytest.raises(AssertionError): RNN(input_dim=1, layers_info=input, hidden_activations="relu", output_activation="relu") def test_user_hidden_layers_input_acceptances(): """Tests whether network rejects invalid hidden_layers inputted from user""" inputs_that_should_work = [[["linear", 33]], [["linear", 12]], [["gru", 2]], [["lstm", 2]], [["lstm", 1]], [["gru", 330]], [["gru", 33], ["linear", 2]] ] for input in inputs_that_should_work: assert RNN(input_dim=1, layers_info=input, hidden_activations="relu", output_activation="relu") def test_hidden_layers_created_correctly(): """Tests that create_hidden_layers works correctly""" layers = [["gru", 25], ["lstm", 23], ["linear", 5], ["linear", 10]] rnn = RNN(input_dim=5, layers_info=layers, hidden_activations="relu", output_activation="relu") assert type(rnn.hidden_layers[0]) == nn.GRU assert rnn.hidden_layers[0].input_size == 5 assert rnn.hidden_layers[0].hidden_size == 25 assert type(rnn.hidden_layers[1]) == nn.LSTM assert rnn.hidden_layers[1].input_size == 25 assert rnn.hidden_layers[1].hidden_size == 23 assert type(rnn.hidden_layers[2]) == nn.Linear assert rnn.hidden_layers[2].in_features == 23 assert rnn.hidden_layers[2].out_features == 5 assert type(rnn.output_layers[0]) == nn.Linear assert rnn.output_layers[0].in_features == 5 assert rnn.output_layers[0].out_features == 10 def test_output_layers_created_correctly(): """Tests that create_output_layers works correctly""" layers = [["gru", 25], ["lstm", 23], ["linear", 5], ["linear", 10]] rnn = RNN(input_dim=5, layers_info=layers, hidden_activations="relu", output_activation="relu") assert rnn.output_layers[0].in_features == 5 assert rnn.output_layers[0].out_features == 10 layers = [["gru", 25], ["lstm", 23], ["lstm", 10]] rnn = RNN(input_dim=5, layers_info=layers, hidden_activations="relu", output_activation="relu") assert rnn.output_layers[0].input_size == 23 assert rnn.output_layers[0].hidden_size == 10 layers = [["gru", 25], ["lstm", 23], [["lstm", 10], ["linear", 15]]] rnn = RNN(input_dim=5, layers_info=layers, hidden_activations="relu", output_activation=["relu", "softmax"]) assert rnn.output_layers[0].input_size == 23 assert rnn.output_layers[0].hidden_size == 10 assert rnn.output_layers[1].in_features == 23 assert rnn.output_layers[1].out_features == 15 def test_output_dim_user_input(): """Tests whether network rejects an invalid output_dim input from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): RNN(input_dim=3, layers_info=[2, input_value], hidden_activations="relu", output_activation="relu") with pytest.raises(AssertionError): RNN(input_dim=6, layers_info=input_value, hidden_activations="relu", output_activation="relu") def test_activations_user_input(): """Tests whether network rejects an invalid hidden_activations or output_activation from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): RNN(input_dim=4, layers_info=[["linear", 2]], hidden_activations=input_value, output_activation="relu") RNN(input_dim=4, layers_info=[["linear", 2]], hidden_activations="relu", output_activation=input_value) def test_initialiser_user_input(): """Tests whether network rejects an invalid initialiser from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): RNN(input_dim=4, layers_info=[["linear", 2]], hidden_activations="relu", output_activation="relu", initialiser=input_value) RNN(layers_info=[["linear", 2], ["linear", 2]], hidden_activations="relu", output_activation="relu", initialiser="xavier", input_dim=4) def test_batch_norm_layers(): """Tests whether batch_norm_layers method works correctly""" layers = [["gru", 20], ["lstm", 3], ["linear", 4], ["linear", 10]] rnn = RNN(layers_info=layers, hidden_activations="relu", input_dim=5, output_activation="relu", initialiser="xavier", batch_norm=True) assert len(rnn.batch_norm_layers) == 3 assert rnn.batch_norm_layers[0].num_features == 20 assert rnn.batch_norm_layers[1].num_features == 3 assert rnn.batch_norm_layers[2].num_features == 4 def test_linear_layers_only_come_at_end(): """Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they don't only come at the end""" layers = [["gru", 20], ["linear", 4], ["lstm", 3], ["linear", 10]] with pytest.raises(AssertionError): rnn = RNN(layers_info=layers, hidden_activations="relu", input_dim=4, output_activation="relu", initialiser="xavier", batch_norm=True) layers = [["gru", 20], ["lstm", 3], ["linear", 4], ["linear", 10]] assert RNN(layers_info=layers, hidden_activations="relu", input_dim=4, output_activation="relu", initialiser="xavier", batch_norm=True) def test_output_activation(): """Tests whether network outputs data that has gone through correct activation function""" RANDOM_ITERATIONS = 20 input_dim = 100 for _ in range(RANDOM_ITERATIONS): data = torch.randn((25, 10, 100)) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", input_dim=input_dim, output_activation="relu", initialiser="xavier", batch_norm=True) out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5]], hidden_activations="relu", input_dim=input_dim, output_activation="relu", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", input_dim=input_dim, output_activation="relu", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", input_dim=input_dim, output_activation="sigmoid", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) assert all(out.reshape(1, -1).squeeze() <= 1) summed_result = torch.sum(out, dim=1) assert all(summed_result.reshape(1, -1).squeeze() != 1.0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", input_dim=input_dim, output_activation="softmax", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) assert all(out.reshape(1, -1).squeeze() <= 1) summed_result = torch.sum(out, dim=1) summed_result = summed_result.reshape(1, -1).squeeze() summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5)) assert all( summed_result == 1.0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", input_dim=input_dim, output_activation="softmax", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) assert all(out.reshape(1, -1).squeeze() <= 1) summed_result = torch.sum(out, dim=1) summed_result = summed_result.reshape(1, -1).squeeze() summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5)) assert all( summed_result == 1.0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", input_dim=input_dim, initialiser="xavier") out = RNN_instance.forward(data) assert not all(out.reshape(1, -1).squeeze() >= 0) assert not all(out.reshape(1, -1).squeeze() <= 0) summed_result = torch.sum(out, dim=1) summed_result = summed_result.reshape(1, -1).squeeze() summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5)) assert not all(summed_result == 1.0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25], ["linear", 8]], hidden_activations="relu", input_dim=input_dim, initialiser="xavier") out = RNN_instance.forward(data) assert not all(out.reshape(1, -1).squeeze() >= 0) assert not all(out.reshape(1, -1).squeeze() <= 0) summed_result = torch.sum(out, dim=1) summed_result = summed_result.reshape(1, -1).squeeze() summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5)) assert not all( summed_result == 1.0) def test_output_activation_return_return_final_seq_only_off(): """Tests whether network outputs data that has gone through correct activation function""" RANDOM_ITERATIONS = 20 input_dim = 100 for _ in range(RANDOM_ITERATIONS): data = torch.randn((25, 10, 100)) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False, output_activation="relu", initialiser="xavier", batch_norm=True) out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5]], hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False, output_activation="relu", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False, output_activation="relu", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False, output_activation="sigmoid", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) assert all(out.reshape(1, -1).squeeze() <= 1) summed_result = torch.sum(out, dim=2) assert all(summed_result.reshape(1, -1).squeeze() != 1.0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False, output_activation="softmax", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) assert all(out.reshape(1, -1).squeeze() <= 1) summed_result = torch.sum(out, dim=2) summed_result = summed_result.reshape(1, -1).squeeze() summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5)) assert all( summed_result == 1.0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False, output_activation="softmax", initialiser="xavier") out = RNN_instance.forward(data) assert all(out.reshape(1, -1).squeeze() >= 0) assert all(out.reshape(1, -1).squeeze() <= 1) summed_result = torch.sum(out, dim=2) summed_result = summed_result.reshape(1, -1).squeeze() summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5)) assert all( summed_result == 1.0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False, initialiser="xavier") out = RNN_instance.forward(data) assert not all(out.reshape(1, -1).squeeze() >= 0) assert not all(out.reshape(1, -1).squeeze() <= 0) summed_result = torch.sum(out, dim=2) summed_result = summed_result.reshape(1, -1).squeeze() summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5)) assert not all( summed_result == 1.0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25], ["linear", 8]], hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False, initialiser="xavier") out = RNN_instance.forward(data) assert not all(out.reshape(1, -1).squeeze() >= 0) assert not all(out.reshape(1, -1).squeeze() <= 0) summed_result = torch.sum(out, dim=2) summed_result = summed_result.reshape(1, -1).squeeze() summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5)) assert not all( summed_result == 1.0) def test_y_range(): """Tests whether setting a y range works correctly""" for _ in range(100): val1 = random.random() - 3.0*random.random() val2 = random.random() + 2.0*random.random() lower_bound = min(val1, val2) upper_bound = max(val1, val2) rnn = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", y_range=(lower_bound, upper_bound), initialiser="xavier", input_dim=22) random_data = torch.randn((10, 11, 22)) out = rnn.forward(random_data) out = out.reshape(1, -1).squeeze() assert torch.sum(out > lower_bound).item() == 25*10, "lower {} vs. {} ".format(lower_bound, out) assert torch.sum(out < upper_bound).item() == 25*10, "upper {} vs. {} ".format(upper_bound, out) def test_deals_with_None_activation(): """Tests whether is able to handle user inputting None as output activation""" assert RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", output_activation=None, initialiser="xavier", input_dim=5) def test_check_input_data_into_forward_once(): """Tests that check_input_data_into_forward_once method only runs once""" rnn = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", input_dim=5, output_activation="relu", initialiser="xavier") data_not_to_throw_error = torch.randn((1, 4, 5)) data_to_throw_error = torch.randn((1, 2, 20)) with pytest.raises(AssertionError): rnn.forward(data_to_throw_error) with pytest.raises(RuntimeError): rnn.forward(data_not_to_throw_error) rnn.forward(data_to_throw_error) def test_y_range_user_input(): """Tests whether network rejects invalid y_range inputs""" invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)] for y_range_value in invalid_y_range_inputs: with pytest.raises(AssertionError): print(y_range_value) rnn = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", y_range=y_range_value, input_dim=5, initialiser="xavier") def solves_simple_problem(X, y, nn_instance): """Checks if a given network is able to solve a simple problem""" optimizer = optim.Adam(nn_instance.parameters(), lr=0.15) for ix in range(800): out = nn_instance.forward(X) loss = torch.sum((out.squeeze() - y) ** 2) / N optimizer.zero_grad() loss.backward() optimizer.step() print("LOSS ", loss) return loss < 0.1 def test_model_trains(): """Tests whether a small range of networks can solve a simple task""" for output_activation in ["sigmoid", "None"]: rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15, hidden_activations="relu", output_activation=output_activation, initialiser="xavier") assert solves_simple_problem(X, y, rnn) z = X[:, 0:1, 3:4] > 5.0 z = torch.cat([z ==1, z==0], dim=1).float() z = z.squeeze(-1).squeeze(-1) rnn = RNN(layers_info=[["gru", 20], ["lstm", 2]], input_dim=15, hidden_activations="relu", output_activation="softmax", dropout=0.01, initialiser="xavier") assert solves_simple_problem(X, z, rnn) rnn = RNN(layers_info=[["lstm", 20], ["linear", 1]], input_dim=15, hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, rnn) rnn = RNN(layers_info=[["lstm", 20], ["linear", 20], ["linear", 1]], input_dim=15, hidden_activations="relu", output_activation=None, initialiser="xavier", batch_norm=True) assert solves_simple_problem(X, y, rnn) rnn = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], input_dim=15, hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, rnn) def test_error_when_provide_negative_data_for_embedding(): """Tests that it raises an error if we try to do an embedding on negative data""" N = 250 X = torch.randn((N, 5, 15)) X[0:125, 0, 3] += 20.0 y = X[:, 0, 3] > 5.0 y = y.float() with pytest.raises(AssertionError): rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15, hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[0], embedding_dimensions=[[200, 5]], initialiser="xavier") assert solves_simple_problem(X, y, rnn) X[:, :, 0] = abs(X[:, :, 0]).long() rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15, hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[0], embedding_dimensions=[[200, 5]], initialiser="xavier") assert solves_simple_problem(X, y, rnn) def test_embedding_layers(): """Tests whether create_embedding_layers method works correctly""" for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)): nn_instance = RNN(input_dim=15, layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]]) for layer in nn_instance.embedding_layers: assert isinstance(layer, nn.Embedding) assert len(nn_instance.embedding_layers) == 2 assert nn_instance.embedding_layers[0].num_embeddings == embedding_in_dim_1 assert nn_instance.embedding_layers[0].embedding_dim == embedding_out_dim_1 assert nn_instance.embedding_layers[1].num_embeddings == embedding_in_dim_2 assert nn_instance.embedding_layers[1].embedding_dim == embedding_out_dim_2 def test_model_trains_with_embeddings(): """Tests that model trains when using embeddings""" N = 250 X = torch.randn((N, 5, 15)) X[0:125, 0, 3] += 20.0 y = X[:, 0, 3] > 5.0 y = y.float() Z = copy.deepcopy(X) Z[:, :, 0] = abs(Z[:, :, 0]).long() rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15, hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[0], embedding_dimensions=[[200, 5]], initialiser="xavier") assert solves_simple_problem(Z, y, rnn) Z = copy.deepcopy(X) Z[:, :, 0:2] = abs(Z[:, :, 0:2]).long() rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15, hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[0, 1], embedding_dimensions=[[200, 5], [50, 3]], initialiser="xavier") assert solves_simple_problem(Z, y, rnn) Z = copy.deepcopy(X) Z[:, :, 3] = abs(Z[:, :, 3]).long() Z[:, :, 6] = abs(Z[:, :, 6]).long() Z[:, :, 4] = abs(Z[:, :, 4]).long() rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15, hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[3, 6, 4], embedding_dimensions=[[200, 5], [50, 3], [50, 12]], initialiser="xavier") assert solves_simple_problem(Z, y, rnn) def test_dropout(): """Tests whether dropout layer reads in probability correctly""" rnn = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation="sigmoid", dropout=0.9999, initialiser="xavier", input_dim=15) assert rnn.dropout_layer.p == 0.9999 assert not solves_simple_problem(X, y, rnn) rnn = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation=None, dropout=0.0000001, initialiser="xavier", input_dim=15) assert rnn.dropout_layer.p == 0.0000001 assert solves_simple_problem(X, y, rnn) def test_all_activations_work(): """Tests that all activations get accepted""" nn_instance = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation=None, dropout=0.0000001, initialiser="xavier", input_dim=15) for key in nn_instance.str_to_activations_converter.keys(): if key == "none": hidden_key = "relu" else: hidden_key = key model = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations=hidden_key, output_activation=key, dropout=0.0000001, initialiser="xavier", input_dim=15) model(X) def test_all_initialisers_work(): """Tests that all initialisers get accepted""" nn_instance = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation=None, dropout=0.0000001, initialiser="xavier", input_dim=15) for key in nn_instance.str_to_initialiser_converter.keys(): model = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], dropout=0.0000001, initialiser=key, input_dim=15) model(X) def test_output_shapes(): """Tests whether network outputs of correct shape""" rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 3]], hidden_activations="relu", initialiser="xavier", input_dim=15) output = rnn(X) assert output.shape == (N, 3) rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], hidden_activations="relu", initialiser="xavier", return_final_seq_only=False, input_dim=15) output = rnn(X) assert output.shape == (N, 5, 7) rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["lstm", 3]], hidden_activations="relu", initialiser="xavier", input_dim=15) output = rnn(X) assert output.shape == (N, 3) rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["lstm", 7]], hidden_activations="relu", initialiser="xavier", return_final_seq_only=False, input_dim=15) output = rnn(X) assert output.shape == (N, 5, 7) def test_return_final_seq_user_input_valid(): """Checks whether network only accepts a valid boolean value for return_final_seq_only""" for valid_case in [True, False]: assert RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], hidden_activations="relu", initialiser="xavier", return_final_seq_only=valid_case, input_dim=15) for invalid_case in [[True], 22, [1, 3], (True, False), (5, False)]: with pytest.raises(AssertionError): print(invalid_case) RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], hidden_activations="relu", initialiser="xavier", return_final_seq_only=invalid_case, input_dim=15) def test_print_model_summary(): nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], ["lstm", 7]], hidden_activations="relu", initialiser="xavier", return_final_seq_only=False, input_dim=15) nn_instance.print_model_summary() def test_output_heads_error_catching(): """Tests that having multiple output heads catches errors from user inputs""" output_dims_that_should_break = [["linear", 2, 2, "SAME", "conv", 3, 4, "SAME"], [[["lstm", 3], ["gru", 4]]], [[2, 8]], [-33, 33, 33, 33, 33]] for output_dim in output_dims_that_should_break: with pytest.raises(AssertionError): RNN(input_dim=5, layers_info=[["gru", 20], ["lstm", 8], output_dim], hidden_activations="relu", output_activation="relu") output_activations_that_should_break = ["relu", ["relu"], ["relu", "softmax"]] for output_activation in output_activations_that_should_break: with pytest.raises(AssertionError): RNN(input_dim=5, layers_info=[["gru", 20], ["lstm", 8], [["linear", 5], ["linear", 2], ["linear", 5]]], hidden_activations="relu", output_activation=output_activation) def test_output_head_layers(): """Tests whether the output head layers get created properly""" for output_dim in [[["linear", 3],["linear", 9]], [["linear", 4], ["linear", 20]], [["linear", 1], ["linear", 1]]]: nn_instance = RNN(input_dim=5, layers_info=[["gru", 20], ["lstm", 8], output_dim], hidden_activations="relu", output_activation=["softmax", None]) assert nn_instance.output_layers[0].out_features == output_dim[0][1] assert nn_instance.output_layers[0].in_features == 8 assert nn_instance.output_layers[1].out_features == output_dim[1][1] assert nn_instance.output_layers[1].in_features == 8 def test_output_head_activations_work(): """Tests that output head activations work properly""" output_dim = [["linear", 5], ["linear", 10], ["linear", 3]] nn_instance = RNN(input_dim=5, layers_info=[["gru", 20], ["lstm", 8], output_dim], hidden_activations="relu", output_activation=["softmax", None, "relu"]) x = torch.randn((20, 12, 5)) * -20.0 out = nn_instance(x) assert out.shape == (20, 18) sums = torch.sum(out[:, :5], dim=1).detach().numpy() sums_others = torch.sum(out[:, 5:], dim=1).detach().numpy() sums_others_2 = torch.sum(out[:, 5:15], dim=1).detach().numpy() sums_others_3 = torch.sum(out[:, 15:18], dim=1).detach().numpy() for row in range(out.shape[0]): assert np.round(sums[row], 4) == 1.0, sums[row] assert not np.round(sums_others[row], 4) == 1.0, sums_others[row] assert not np.round(sums_others_2[row], 4) == 1.0, sums_others_2[row] assert not np.round(sums_others_3[row], 4) == 1.0, sums_others_3[row] for col in range(3): assert out[row, 15 + col] >= 0.0, out[row, 15 + col] def test_output_head_shapes_correct(): """Tests that the output shape of network is correct when using multiple outpout heads""" N = 20 X = torch.randn((N, 10, 4)) * -20.0 for _ in range(25): nn_instance = RNN(input_dim=4, layers_info=[["gru", 20], ["lstm", 8], ["linear", 1], ["linear", 12]], hidden_activations="relu") out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 12 for output_dim in [[ ["linear", 10], ["linear", 4], ["linear", 6]], [["linear", 3], ["linear", 8], ["linear", 9]]]: nn_instance = RNN(input_dim=4, layers_info=[["gru", 20], ["lstm", 8], ["linear", 1], ["linear", 12], output_dim], hidden_activations="relu", output_activation=["softmax", None, "relu"]) out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 20 ================================================ FILE: tests/tensorflow_tests/test_tf_CNN.py ================================================ # Run from home directory with python -m pytest tests import shutil import pytest import random import numpy as np import tensorflow as tf import torch.nn as nn from nn_builder.tensorflow.CNN import CNN from tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, BatchNormalization, MaxPool2D, AveragePooling2D N = 250 X = np.random.random((N, 5, 5, 1)) X[0:125, 3, 3, 0] += 20.0 y = X[:, 3, 3, 0] > 5.0 def test_user_hidden_layers_input_rejections(): """Tests whether network rejects invalid hidden_layers inputted from user""" inputs_that_should_fail = [ ('maxpool', 3, 3, 3) , ['maxpool', 33, 22, 33], [['a']], [[222, 222, 222, 222]], [["conv", 2, 2, -1]], [["conv", 2, 2]], [["conv", 2, 2, 55, 999, 33]], [["maxpool", 33, 33]], [["maxpool", -1, 33]], [["maxpool", 33]], [["maxpoolX", 1, 33]], [["cosnv", 2, 2]], [["avgpool", 33, 33, 333, 99]], [["avgpool", -1, 33]], [["avgpool", 33]], [["avgpoolX", 1, 33]], [["adaptivemaxpool", 33, 33, 333, 33]], [["adaptivemaxpool", 2]], [["adaptivemaxpool", 33]], [["adaptivemaxpoolX"]], [["adaptiveavgpool", 33, 33, 333, 11]], [["adaptiveavgpool", 2]], [["adaptiveavgpool", 33]], [["adaptiveavgpoolX"]], [["linear", 40, -2]], [["lineafr", 40, 2]]] for input in inputs_that_should_fail: print(input) with pytest.raises(AssertionError): CNN(layers_info=input, hidden_activations="relu", output_activation="relu") def test_user_hidden_layers_input_acceptances(): """Tests whether network rejects invalid hidden_layers inputted from user""" inputs_that_should_work = [[["conv", 2, 2, 3331, "VALID"]], [["CONV", 2, 2, 3331, "SAME"]], [["ConV", 2, 2, 3331, "valid"]], [["maxpool", 2, 2, "same"]], [["MAXPOOL", 2, 2, "Valid"]], [["MaXpOOL", 2, 2, "SAme"]], [["avgpool", 2, 2, "saME"]], [["AVGPOOL", 2, 2, "vaLID"]], [["avGpOOL", 2, 2, "same"]], [["linear", 40]], [["lineaR", 2]], [["LINEAR", 2]]] for ix, input in enumerate(inputs_that_should_work): input.append(["linear", 5]) CNN(layers_info=input, hidden_activations="relu", output_activation="relu") def test_hidden_layers_created_correctly(): """Tests that create_hidden_layers works correctly""" layers = [["conv", 2, 4, 3, "same"], ["maxpool", 3, 4, "vaLID"], ["avgpool", 32, 42, "vaLID"], ["linear", 22], ["linear", 2222], ["linear", 5]] cnn = CNN(layers_info=layers, hidden_activations="relu", output_activation="relu") assert type(cnn.hidden_layers[0]) == Conv2D assert cnn.hidden_layers[0].filters == 2 assert cnn.hidden_layers[0].kernel_size == (4, 4) assert cnn.hidden_layers[0].strides == (3, 3) assert cnn.hidden_layers[0].padding == "same" assert type(cnn.hidden_layers[1]) == MaxPool2D assert cnn.hidden_layers[1].pool_size == (3, 3) assert cnn.hidden_layers[1].strides == (4, 4) assert cnn.hidden_layers[1].padding == "valid" assert type(cnn.hidden_layers[2]) == AveragePooling2D assert cnn.hidden_layers[2].pool_size == (32, 32) assert cnn.hidden_layers[2].strides == (42, 42) assert cnn.hidden_layers[2].padding == "valid" assert type(cnn.hidden_layers[3]) == Dense assert cnn.hidden_layers[3].units == 22 assert type(cnn.hidden_layers[4]) == Dense assert cnn.hidden_layers[4].units == 2222 assert type(cnn.output_layers[0]) == Dense assert cnn.output_layers[0].units == 5 def test_output_layers_created_correctly(): """Tests that create_output_layers works correctly""" layers = [["conv", 2, 4, 3, "valid"], ["maxpool", 3, 4, "same"], ["avgpool", 32, 42, "valid"], ["linear", 22], ["linear", 2222], ["linear", 2]] cnn = CNN(layers_info=layers, hidden_activations="relu", output_activation="relu") assert cnn.output_layers[0].units == 2 layers = [["conv", 2, 4, 3,"valid"], ["maxpool", 3, 4, "same"], ["avgpool", 32, 42, "same"], ["linear", 7]] cnn = CNN(layers_info=layers, hidden_activations="relu", output_activation="relu") assert cnn.output_layers[0].units == 7 layers = [["conv", 5, 4, 3, "valid"], ["maxpool", 3, 4, "valid"], ["avgpool", 32, 42, "valid"], ["linear", 6]] cnn = CNN( layers_info=layers, hidden_activations="relu", output_activation="relu") assert cnn.output_layers[0].units == 6 layers = [["conv", 5, 4, 3, "valid"], ["maxpool", 3, 4, "valid"], ["avgpool", 32, 42, "valid"], [["linear", 6], ["linear", 22]]] cnn = CNN(layers_info=layers, hidden_activations="relu", output_activation=["softmax", None]) assert cnn.output_layers[0].units == 6 assert cnn.output_layers[1].units == 22 def test_output_dim_user_input(): """Tests whether network rejects an invalid output_dim input from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): CNN(layers_info=[2, input_value], hidden_activations="relu", output_activation="relu") with pytest.raises(AssertionError): CNN(layers_info=input_value, hidden_activations="relu", output_activation="relu") def test_activations_user_input(): """Tests whether network rejects an invalid hidden_activations or output_activation from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): CNN(layers_info=[["conv", 2, 2, 3331, "valid"], ["linear", 5] ], hidden_activations=input_value, output_activation="relu") CNN(layers_info=[["conv", 2, 2, 3331, "valid"], ["linear", 3]], hidden_activations="relu", output_activation=input_value) def test_initialiser_user_input(): """Tests whether network rejects an invalid initialiser from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): CNN(layers_info=[["conv", 2, 2, 3331, "valid"], ["linear", 3]], hidden_activations="relu", output_activation="relu", initialiser=input_value) CNN(layers_info=[["conv", 2, 2, 3331, "valid"], ["linear", 3]], hidden_activations="relu", output_activation="relu", initialiser="xavier") def test_batch_norm_layers(): """Tests whether batch_norm_layers method works correctly""" layers =[["conv", 2, 4, 3, "valid"], ["maxpool", 3, 4, "valid"], ["linear", 5]] cnn = CNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=False) layers = [["conv", 2, 4, 3, "valid"], ["maxpool", 3, 4, "valid"], ["linear", 5]] cnn = CNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) assert len(cnn.batch_norm_layers) == 1 assert isinstance(cnn.batch_norm_layers[0], tf.keras.layers.BatchNormalization) layers = [["conv", 2, 4, 3, "valid"], ["maxpool", 3, 4, "valid"], ["conv", 12, 4, 3, "valid"], ["linear", 22], ["linear", 55]] cnn = CNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) assert len(cnn.batch_norm_layers) == 3 for layer in cnn.batch_norm_layers: assert isinstance(layer, tf.keras.layers.BatchNormalization) def test_linear_layers_acceptance(): """Tests that only accepts linear layers of correct shape""" layers_that_shouldnt_work = [[["linear", 2, 5]], [["linear", 2, 5, 5]], [["linear"]], [["linear", 2], ["linear", 5, 4]], ["linear", 0], ["linear", -5]] for layers in layers_that_shouldnt_work: with pytest.raises(AssertionError): cnn = CNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) layers_that_should_work = [[["linear", 44], ["linear", 2]], [["linear", 22]]] for layer in layers_that_should_work: assert CNN(layers_info=layer, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) def test_linear_layers_only_come_at_end(): """Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they don't only come at the end""" layers = [["conv", 2, 4, 3, "valid"], ["linear", 55], ["maxpool", 3, 4, "valid"]] with pytest.raises(AssertionError): cnn = CNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) layers = [["conv", 2, 4, 3, "valid"], ["linear", 55]] assert CNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) layers = [["conv", 2, 4, 3, "valid"], ["linear", 55], ["linear", 55], ["linear", 55]] assert CNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) def test_output_activation(): """Tests whether network outputs data that has gone through correct activation function""" RANDOM_ITERATIONS = 20 input_dim = (100, 100, 5) for _ in range(RANDOM_ITERATIONS): data = np.random.random((1, *input_dim)) CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, "valid"], ["linear", 50]], hidden_activations="relu", output_activation="relu", initialiser="xavier") out = CNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) CNN_instance = CNN(layers_info=[["conv", 2, 20, 1, "same"], ["linear", 5]], hidden_activations="relu", output_activation="relu", initialiser="xavier") out = CNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) CNN_instance = CNN(layers_info=[["conv", 5, 20, 1, "same"], ["linear", 5]], hidden_activations="relu", output_activation="relu", initialiser="xavier") out = CNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) CNN_instance = CNN(layers_info=[["conv", 5, 2, 1, "valid"], ["linear", 22]], hidden_activations="relu", output_activation="sigmoid", initialiser="xavier") out = CNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) assert all(tf.reshape(out, [-1]) <= 1) assert not np.round(tf.reduce_sum(out, axis=1), 3) == 1.0 CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, "same"], ["linear", 5]], hidden_activations="relu", output_activation="softmax", initialiser="xavier") out = CNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) assert all(tf.reshape(out, [-1]) <= 1) assert np.round(tf.reduce_sum(out, axis=1), 3) == 1.0 CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, "valid"], ["linear", 5]], hidden_activations="relu", initialiser="xavier") out = CNN_instance(data) assert not all(tf.reshape(out, [-1]) >= 0) assert not np.round(tf.reduce_sum(out, axis=1), 3) == 1.0 def test_y_range(): """Tests whether setting a y range works correctly""" for _ in range(100): val1 = random.random() - 3.0*random.random() val2 = random.random() + 2.0*random.random() lower_bound = min(val1, val2) upper_bound = max(val1, val2) CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, "valid"], ["linear", 5]], hidden_activations="relu", y_range=(lower_bound, upper_bound), initialiser="xavier") random_data = np.random.random((10, 1, 20, 20)) out = CNN_instance(random_data) assert all(tf.reshape(out, [-1]) > lower_bound) assert all(tf.reshape(out, [-1]) < upper_bound) def test_deals_with_None_activation(): """Tests whether is able to handle user inputting None as output activation""" assert CNN(layers_info=[["conv", 2, 2, 1, "valid"], ["linear", 5]], hidden_activations="relu", output_activation=None, initialiser="xavier") def test_y_range_user_input(): """Tests whether network rejects invalid y_range inputs""" invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)] for y_range_value in invalid_y_range_inputs: with pytest.raises(AssertionError): print(y_range_value) CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, "valid"], ["linear", 5]], hidden_activations="relu", y_range=y_range_value, initialiser="xavier") def test_model_trains(): """Tests whether a small range of networks can solve a simple task""" for output_activation in ["sigmoid", "None"]: CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 1]], hidden_activations="relu", output_activation=output_activation, initialiser="xavier") print(CNN_instance.hidden_layers[0].kernel_size) assert solves_simple_problem(X, y, CNN_instance) def test_model_trains_part_2(): """Tests whether a small range of networks can solve a simple task""" z = X[:, 3:4, 3:4, 0:1] > 5.0 z = np.concatenate([z == 1, z == 0], axis=1) z = z.reshape((-1, 2)) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 2]], hidden_activations="relu", output_activation="softmax", dropout=0.01, initialiser="xavier") assert solves_simple_problem(X, z, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 1]], hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, "same"], ["linear", 1]], hidden_activations="relu", output_activation=None, initialiser="xavier", batch_norm=True) assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["maxpool", 1, 1, "same"], ["linear", 1]], hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, "same"], ["avgpool", 1, 1, "same"], ["linear", 1]], hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 5, 3, 1, "same"], ["linear", 1]], hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 5, 3, 1, "valid"], ["linear", 1]], hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) def solves_simple_problem(X, y, nn_instance): """Checks if a given network is able to solve a simple problem""" nn_instance.compile(optimizer='adam', loss='mse') nn_instance.fit(X, y, epochs=800) results = nn_instance.evaluate(X, y) print("FINAL RESULT ", results) return results < 0.1 def test_model_trains_linear_layer(): """Tests whether a small range of networks can solve a simple task""" CNN_instance = CNN(layers_info=[["conv", 5, 3, 1, "valid"], ["linear", 5], ["linear", 5], ["linear", 1]], hidden_activations="relu", output_activation="sigmoid", initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["linear", 5], ["linear", 5], ["linear", 1]], hidden_activations="relu", output_activation="sigmoid", initialiser="xavier") assert solves_simple_problem(X, y, CNN_instance) def test_max_pool_working(): """Tests whether max pool layers work properly""" N = 250 X = np.random.random((N, 8, 8, 1)) X[0:125, 3, 3, 0] = 999.99 CNN_instance = CNN(layers_info=[["maxpool", 2, 2, "valid"], ["maxpool", 2, 2, "valid"], ["maxpool", 2, 2, "valid"], ["linear", 1]], hidden_activations="relu", initialiser="xavier") assert CNN_instance(X).shape == (N, 1) def test_dropout(): """Tests whether dropout layer reads in probability correctly""" CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 1]], hidden_activations="relu", output_activation="sigmoid", dropout=0.9999, initialiser="xavier") assert CNN_instance.dropout_layer.rate == 0.9999 assert not solves_simple_problem(X, y, CNN_instance) CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 1]], hidden_activations="relu", output_activation=None, dropout=0.0000001, initialiser="xavier") assert CNN_instance.dropout_layer.rate == 0.0000001 assert solves_simple_problem(X, y, CNN_instance) def test_MNIST_progress(): """Tests that network made using CNN module can make progress on MNIST""" mnist = tf.keras.datasets.mnist (x_train, y_train), (x_test, y_test) = mnist.load_data() x_train, x_test = x_train / 255.0, x_test / 255.0 # Add a channels dimension x_train = x_train[..., tf.newaxis] x_test = x_test[..., tf.newaxis] # Create model using nn_builder model = CNN(layers_info=[["conv", 32, 3, 1, "valid"], ["maxpool", 2, 2, "valid"], ["conv", 64, 3, 1, "valid"], ["linear", 10]], hidden_activations="relu", output_activation="softmax", dropout=0.0, initialiser="xavier", batch_norm=True) model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) model.fit(x_train, y_train, epochs=2, batch_size=64) model.evaluate(x_test, y_test) results = model.evaluate(x_test, y_test) assert results[1] > 0.9 model = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 10]], hidden_activations="relu", output_activation="softmax", dropout=0.9999, initialiser="xavier") model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) model.fit(x_train, y_train, epochs=2, batch_size=64) model.evaluate(x_test, y_test) results = model.evaluate(x_test, y_test) assert not results[1] > 0.9 model = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 10]], hidden_activations="relu", output_activation="softmax", dropout=0.0, initialiser="xavier") model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) model.fit(x_train, y_train, epochs=2, batch_size=64) model.evaluate(x_test, y_test) results = model.evaluate(x_test, y_test) assert results[1] > 0.9 def test_all_activations_work(): """Tests that all activations get accepted""" nn_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 1]], dropout=0.0000001, initialiser="xavier") for key in nn_instance.str_to_activations_converter.keys(): model = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 1]], hidden_activations=key, output_activation=key, dropout=0.0000001, initialiser="xavier") model(X) def test_all_initialisers_work(): """Tests that all initialisers get accepted""" nn_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 1]], dropout=0.0000001, initialiser="xavier") for key in nn_instance.str_to_initialiser_converter.keys(): print(key) model = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["linear", 1]], dropout=0.0000001, initialiser=key) model(X) def test_print_model_summary(): nn_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1]], dropout=0.0000001, batch_norm=True, initialiser="xavier") nn_instance.print_model_summary((64, 11, 11, 3)) def test_output_heads_error_catching(): """Tests that having multiple output heads catches errors from user inputs""" output_dims_that_should_break = [["linear", 2, 2, "SAME", "conv", 3, 4, "SAME"], [[["conv", 3, 2, "same"], ["linear", 4]]], [[2, 8]], [-33, 33, 33, 33, 33]] for output_dim in output_dims_that_should_break: with pytest.raises(AssertionError): CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1], output_dim], hidden_activations="relu", output_activation="relu") output_activations_that_should_break = ["relu", ["relu"], ["relu", "softmax"]] for output_activation in output_activations_that_should_break: with pytest.raises(AssertionError): CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1], [["linear", 4], ["linear", 10], ["linear", 4]]], hidden_activations="relu", output_activation=output_activation) def test_output_head_layers(): """Tests whether the output head layers get created properly""" for output_dim in [[["linear", 3],["linear", 9]], [["linear", 4], ["linear", 20]], [["linear", 1], ["linear", 1]]]: nn_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1], output_dim], hidden_activations="relu", output_activation=["softmax", None]) assert nn_instance.output_layers[0].units == output_dim[0][1] assert nn_instance.output_layers[1].units == output_dim[1][1] def test_output_head_activations_work(): """Tests that output head activations work properly""" output_dim = [["linear", 5], ["linear", 10], ["linear", 3]] nn_instance = CNN(layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1], output_dim], hidden_activations="relu", output_activation=["softmax", None, "relu"]) x = np.random.random((20, 10, 10, 4)) * -20.0 out = nn_instance(x) assert out.shape == (20, 18) sums = tf.reduce_sum(out[:, :5], axis=1) sums_others = tf.reduce_sum(out[:, 5:], axis=1) sums_others_2 = tf.reduce_sum(out[:, 5:15], axis=1) sums_others_3 = tf.reduce_sum(out[:, 15:18], axis=1) for row in range(out.shape[0]): assert tf.math.equal(np.round(sums[row], 4), 1.0), sums[row] assert not tf.math.equal(np.round(sums_others[row], 4), 1.0), np.round(sums_others[row], 4) assert not tf.math.equal(np.round(sums_others_2[row], 4), 1.0), np.round(sums_others_2[row], 4) assert not tf.math.equal(np.round(sums_others_3[row], 4), 1.0), np.round(sums_others_3[row], 4) for col in range(3): assert out[row, 15 + col] >= 0.0, out[row, 15 + col] def test_output_head_shapes_correct(): """Tests that the output shape of network is correct when using multiple outpout heads""" N = 20 X = np.random.random((N, 25, 25, 2)) for _ in range(25): nn_instance = CNN( layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1], ["linear", 12]], hidden_activations="relu") out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 12 for output_dim in [[ ["linear", 10], ["linear", 4], ["linear", 6]], [["linear", 3], ["linear", 8], ["linear", 9]]]: nn_instance = CNN( layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1], output_dim], hidden_activations="relu", output_activation=["softmax", None, "relu"]) out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 20 ================================================ FILE: tests/tensorflow_tests/test_tf_NN.py ================================================ # Run from home directory with python -m pytest tests import pytest import copy import random import numpy as np import tensorflow as tf from nn_builder.tensorflow.NN import NN import tensorflow.keras.initializers as initializers import tensorflow.keras.activations as activations N = 250 X = (np.random.random((N, 5)) - 0.5) * 2.0 X[:, [2, 4]] += 10.0 y = X[:, 0] > 0 * 1.0 def test_linear_hidden_units_user_input(): """Tests whether network rejects an invalid linear_hidden_units input from user""" inputs_that_should_fail = ["a", ["a", "b"], [2, 4, "ss"], [-2], 2] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): NN( layers_info=input_value, hidden_activations="relu", output_activation="relu") def test_activations_user_input(): """Tests whether network rejects an invalid hidden_activations or output_activation from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): NN( layers_info=[2], hidden_activations=input_value, output_activation="relu") NN( layers_info=[2], hidden_activations="relu", output_activation=input_value) def test_initialiser_user_input(): """Tests whether network rejects an invalid initialiser from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): NN( layers_info=[2], hidden_activations="relu", output_activation="relu", initialiser=input_value) NN( layers_info=[2], hidden_activations="relu", output_activation="relu", initialiser="xavier") def test_output_shape_correct(): """Tests whether network returns output of the right shape""" input_dims = [x for x in range(1, 3)] output_dims = [x for x in range(4, 6)] linear_hidden_units_options = [ [2, 3, 4], [2, 9, 1], [55, 55, 55, 234, 15]] for input_dim, output_dim, linear_hidden_units in zip(input_dims, output_dims, linear_hidden_units_options): linear_hidden_units.append(output_dim) nn_instance = NN(layers_info=linear_hidden_units, hidden_activations="relu", output_activation="relu", initialiser="xavier") data = 2.0 * (np.random.random((25, input_dim)) - 0.5) output = nn_instance(data) assert output.shape == (25, output_dim) def test_output_activation(): """Tests whether network outputs data that has gone through correct activation function""" RANDOM_ITERATIONS = 20 for _ in range(RANDOM_ITERATIONS): data = 2.0 * (np.random.random((1, 100)) - 0.5) nn_instance = NN(layers_info=[5, 5, 5], hidden_activations="relu", output_activation="relu", initialiser="xavier") out = nn_instance(data) assert all(tf.squeeze(out) >= 0) nn_instance = NN(layers_info=[5, 5, 5], hidden_activations="relu", output_activation="sigmoid", initialiser="xavier") out = nn_instance(data) assert all(tf.squeeze(out) >= 0) assert all(tf.squeeze(out) <= 1) nn_instance = NN(layers_info=[5, 5, 5], hidden_activations="relu", output_activation="softmax", initialiser="xavier") out = nn_instance(data) assert all(tf.squeeze(out) >= 0) assert all(tf.squeeze(out) <= 1) assert np.round(tf.reduce_sum(tf.squeeze(out)), 3) == 1.0 nn_instance = NN(layers_info=[5, 5, 5], hidden_activations="relu") out = nn_instance(data) assert not all(tf.squeeze(out) >= 0) assert not np.round(tf.reduce_sum(tf.squeeze(out)), 3) == 1.0 def test_linear_layers_info(): """Tests whether create_hidden_layers_info method works correctly""" for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]): hidden_units.append(output_dim) print(hidden_units) nn_instance = NN(layers_info=copy.copy(hidden_units), hidden_activations="relu", output_activation="softmax", initialiser="xavier") print(hidden_units) assert len(nn_instance.hidden_layers) == len(hidden_units) - 1 for layer_ix in range(len(hidden_units) - 1): layer = nn_instance.hidden_layers[layer_ix] print(nn_instance.hidden_layers[layer_ix]) assert type(layer) == tf.keras.layers.Dense assert layer.units == hidden_units[layer_ix] assert layer.kernel_initializer == initializers.glorot_uniform, layer.kernel_initializer assert layer.activation == activations.relu output_layer = nn_instance.output_layers[0] assert type(output_layer) == tf.keras.layers.Dense assert output_layer.units == hidden_units[-1] assert output_layer.kernel_initializer == initializers.glorot_uniform assert output_layer.activation == activations.softmax def test_embedding_layers(): """Tests whether create_embedding_layers_info method works correctly""" for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)): nn_instance = NN( layers_info=[5], columns_of_data_to_be_embedded=[0, 1], embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]]) for layer in nn_instance.embedding_layers: assert isinstance(layer, tf.keras.layers.Embedding) assert len(nn_instance.embedding_layers) == 2 assert nn_instance.embedding_layers[0].input_dim == embedding_in_dim_1 assert nn_instance.embedding_layers[0].output_dim == embedding_out_dim_1 assert nn_instance.embedding_layers[1].input_dim == embedding_in_dim_2 assert nn_instance.embedding_layers[1].output_dim == embedding_out_dim_2 def test_incorporate_embeddings(): """Tests the method incorporate_embeddings""" X_new = X X_new[:, [2, 4]] = tf.round(X_new[:, [2, 4]]) nn_instance = NN( layers_info=[10], columns_of_data_to_be_embedded=[2, 4], embedding_dimensions=[[50, 3], [55, 4]]) out = nn_instance.incorporate_embeddings(X) assert out.shape == (N, X.shape[1]+3+4-2) def test_embedding_network_can_solve_simple_problem(): """Tests whether network can solve simple problem using embeddings""" X = (np.random.random((N, 5)) - 0.5) * 5.0 + 20.0 y = (X[:, 0] >= 20) * (X[:, 1] <= 20) * 1.0 nn_instance = NN( layers_info=[5, 1], columns_of_data_to_be_embedded=[0, 1], embedding_dimensions=[[50, 3], [55, 3]]) assert solves_simple_problem(X, y, nn_instance) def test_batch_norm_layers_info(): """Tests whether batch_norm_layers_info method works correctly""" for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]): hidden_units.append(output_dim) nn_instance = NN( layers_info=hidden_units, hidden_activations="relu", batch_norm=True, output_activation="relu", initialiser="xavier") for layer in nn_instance.batch_norm_layers: assert isinstance(layer, tf.keras.layers.BatchNormalization) assert len(nn_instance.batch_norm_layers) == len(hidden_units) - 1 def test_model_trains(): """Tests whether a small range of networks can solve a simple task""" for output_activation in ["sigmoid", "None"]: nn_instance = NN( layers_info=[10, 10, 10, 1], output_activation=output_activation, dropout=0.01, batch_norm=True) assert solves_simple_problem(X, y, nn_instance) z = X[:, 0:1] > 0 z = np.concatenate([z ==1, z==0], axis=1) nn_instance = NN(layers_info=[10, 10, 10, 2], output_activation="softmax", dropout=0.01, batch_norm=True) assert solves_simple_problem(X, z, nn_instance) def solves_simple_problem(X, y, nn_instance): """Checks if a given network is able to solve a simple problem""" nn_instance.compile(optimizer='adam', loss='mse') nn_instance.fit(X, y, epochs=800) results = nn_instance.evaluate(X, y) print("FINAL RESULT ", results) return results < 0.1 def test_dropout(): """Tests whether dropout layer reads in probability correctly""" nn_instance = NN(layers_info=[10, 10, 1], dropout=0.9999) assert nn_instance.dropout_layer.rate == 0.9999 assert not solves_simple_problem(X, y, nn_instance) nn_instance = NN( layers_info=[10, 10, 1], dropout=0.00001) assert nn_instance.dropout_layer.rate == 0.00001 assert solves_simple_problem(X, y, nn_instance) def test_y_range_user_input(): """Tests whether network rejects invalid y_range inputs""" invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)] for y_range_value in invalid_y_range_inputs: with pytest.raises(AssertionError): print(y_range_value) nn_instance = NN( layers_info=[10, 10, 3], y_range=y_range_value) def test_y_range(): """Tests whether setting a y range works correctly""" for _ in range(100): val1 = random.random() - 3.0*random.random() val2 = random.random() + 2.0*random.random() lower_bound = min(val1, val2) upper_bound = max(val1, val2) nn_instance = NN( layers_info=[10, 10, 3], y_range=(lower_bound, upper_bound)) random_data = 2.0 * (np.random.random((15, 5)) - 0.5) out = nn_instance(random_data) assert np.sum(out > lower_bound) == 3*15, "lower {} vs. {} ".format(lower_bound, out) assert np.sum(out < upper_bound) == 3*15, "upper {} vs. {} ".format(upper_bound, out) def test_deals_with_None_activation(): """Tests whether is able to handle user inputting None as output activation""" assert NN( layers_info=[10, 10, 3], output_activation=None) def test_all_activations_work(): """Tests that all activations get accepted""" nn_instance = NN( layers_info=[10, 10, 1], dropout=0.9999) for key in nn_instance.str_to_activations_converter.keys(): model = NN(layers_info=[10, 10, 1], dropout=0.9999, hidden_activations=key, output_activation=key) model(X) def test_all_initialisers_work(): """Tests that all initialisers get accepted""" nn_instance = NN(layers_info=[10, 10, 1], dropout=0.9999) for key in nn_instance.str_to_initialiser_converter.keys(): model = NN(layers_info=[10, 10, 1], dropout=0.9999, initialiser=key) model(X) def test_print_model_summary(): nn_instance = NN(layers_info=[10, 10, 1]) nn_instance.print_model_summary((64, 11)) def test_output_heads_error_catching(): """Tests that having multiple output heads catches errors from user inputs""" output_dims_that_should_break = [[[2, 8]], [-33, 33, 33, 33, 33]] for output_dim in output_dims_that_should_break: with pytest.raises(AssertionError): NN(layers_info=[4, 7, 9, output_dim], hidden_activations="relu", output_activation="relu") output_activations_that_should_break = ["relu", ["relu"], ["relu", "softmax"]] for output_activation in output_activations_that_should_break: with pytest.raises(AssertionError): NN(layers_info=[4, 7, 9, [4, 6, 1]], hidden_activations="relu", output_activation=output_activation) def test_output_head_layers(): """Tests whether the output head layers get created properly""" for output_dim in [[3, 9], [4, 20], [1, 1]]: nn_instance = NN(layers_info=[4, 7, 9, output_dim], hidden_activations="relu", output_activation=["softmax", None]) assert nn_instance.output_layers[0].units == output_dim[0] assert nn_instance.output_layers[1].units == output_dim[1] def test_output_head_activations_work(): """Tests that output head activations work properly""" nn_instance = NN(layers_info=[4, 7, 9, [5, 10, 3]], hidden_activations="relu", output_activation=["softmax", None, "relu"]) x = np.random.random((20, 2)) * -20.0 out = nn_instance(x) assert out.shape == (20, 18) sums = tf.reduce_sum(out[:, :5], axis=1) sums_others = tf.reduce_sum(out[:, 5:], axis=1) sums_others_2 = tf.reduce_sum(out[:, 5:15], axis=1) sums_others_3 = tf.reduce_sum(out[:, 15:18], axis=1) for row in range(out.shape[0]): assert tf.math.equal(np.round(sums[row], 4), 1.0), sums[row] assert not tf.math.equal(np.round(sums_others[row], 4), 1.0), np.round(sums_others[row], 4) assert not tf.math.equal(np.round(sums_others_2[row], 4), 1.0), np.round(sums_others_2[row], 4) assert not tf.math.equal(np.round(sums_others_3[row], 4), 1.0), np.round(sums_others_3[row], 4) for col in range(3): assert out[row, 15 + col] >= 0.0, out[row, 15 + col] def test_output_head_shapes_correct(): """Tests that the output shape of network is correct when using multiple outpout heads""" N = 20 X = np.random.random((N, 2)) for _ in range(25): output_dim = random.randint(1, 100) nn_instance = NN(layers_info=[4, 7, 9, output_dim], hidden_activations="relu") out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == output_dim for output_dim in [[3, 9, 5, 3], [5, 5, 5, 5], [2, 1, 1, 16]]: nn_instance = NN(layers_info=[4, 7, 9, output_dim], hidden_activations="relu", output_activation=["softmax", None, None, "relu"]) out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 20 def test_boston_housing_progress(): """Tests that network made using CNN module can make progress on MNIST""" boston_housing = tf.keras.datasets.boston_housing (x_train, y_train), (x_test, y_test) = boston_housing.load_data() model = NN(layers_info=[30, 10, 1], hidden_activations="relu", output_activation=None, dropout=0.0, initialiser="xavier", batch_norm=True, y_range=(4.5, 55.0)) model.compile(optimizer='adam',loss='mse') model.fit(x_train, y_train, epochs=200, batch_size=64) results = model.evaluate(x_test, y_test) assert results < 35 model = NN(layers_info=[30, 10, 1], hidden_activations="relu", output_activation=None, dropout=0.0, initialiser="xavier", batch_norm=False) model.compile(optimizer='adam',loss='mse') model.fit(x_train, y_train, epochs=200, batch_size=64) results = model.evaluate(x_test, y_test) assert results < 30 model = NN(layers_info=[30, 10, 1], hidden_activations="relu", output_activation=None, dropout=0.0, initialiser="xavier", batch_norm=True) model.compile(optimizer='adam', loss='mse') model.fit(x_train, y_train, epochs=200, batch_size=64) results = model.evaluate(x_test, y_test) assert results < 30 model = NN(layers_info=[150, 50, 1], hidden_activations="relu", output_activation=None, dropout=0.05, initialiser="xavier", batch_norm=False) model.compile(optimizer='adam', loss='mse') model.fit(x_train, y_train, epochs=200, batch_size=64) results = model.evaluate(x_test, y_test) assert results < 30 ================================================ FILE: tests/tensorflow_tests/test_tf_RNN.py ================================================ # Run from home directory with python -m pytest tests import pytest import random import tensorflow as tf import numpy as np from tensorflow.keras.layers import Dense, Concatenate, BatchNormalization, GRU, LSTM from nn_builder.tensorflow.RNN import RNN N = 250 X = np.random.random((N, 3, 5)) X = X.astype('float32') X[0:125, :, 3] += 10.0 y = X[:, 2, 3] > 5.0 def test_user_hidden_layers_input_rejections(): """Tests whether network rejects invalid hidden_layers inputted from user""" inputs_that_should_fail = [[["linearr", 33]], [["linear", 12, 33]], [["gru", 2, 33]], [["lstm", 2, 33]], [["lstmr", 33]], [["gruu", 33]], [["gru", 33], ["xxx", 33]], [["linear", 33], ["gru", 12], ["gru", 33]] ] for input in inputs_that_should_fail: with pytest.raises(AssertionError): RNN(layers_info=input, hidden_activations="relu", output_activation="relu") def test_user_hidden_layers_input_acceptances(): """Tests whether network rejects invalid hidden_layers inputted from user""" inputs_that_should_work = [[["linear", 33]], [["linear", 12]], [["gru", 2]], [["lstm", 2]], [["lstm", 1]], [["gru", 330]], [["gru", 33], ["linear", 2]] ] for input in inputs_that_should_work: assert RNN(layers_info=input, hidden_activations="relu", output_activation="relu") def test_hidden_layers_created_correctly(): """Tests that create_hidden_layers works correctly""" layers = [["gru", 25], ["lstm", 23], ["linear", 5], ["linear", 10]] rnn = RNN(layers_info=layers, hidden_activations="relu", output_activation="relu") assert type(rnn.hidden_layers[0]) == GRU assert rnn.hidden_layers[0].units == 25 assert type(rnn.hidden_layers[1]) == LSTM assert rnn.hidden_layers[1].units == 23 assert type(rnn.hidden_layers[2]) == Dense assert rnn.hidden_layers[2].units == 5 assert type(rnn.output_layers[0]) == Dense assert rnn.output_layers[0].units == 10 def test_output_layers_created_correctly(): """Tests that create_output_layers works correctly""" layers = [["gru", 25], ["lstm", 23], ["linear", 5], ["linear", 10]] rnn = RNN(layers_info=layers, hidden_activations="relu", output_activation="relu") assert rnn.output_layers[0].units == 10 layers = [["gru", 25], ["lstm", 23], ["lstm", 10]] rnn = RNN(layers_info=layers, hidden_activations="relu", output_activation="relu") assert rnn.output_layers[0].units == 10 layers = [["gru", 25], ["lstm", 23], [["lstm", 10], ["linear", 15]]] rnn = RNN(layers_info=layers, hidden_activations="relu", output_activation=["relu", "softmax"]) assert rnn.output_layers[0].units == 10 assert rnn.output_layers[1].units == 15 def test_output_dim_user_input(): """Tests whether network rejects an invalid output_dim input from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): RNN(layers_info=[2, input_value], hidden_activations="relu", output_activation="relu") with pytest.raises(AssertionError): RNN(layers_info=input_value, hidden_activations="relu", output_activation="relu") def test_activations_user_input(): """Tests whether network rejects an invalid hidden_activations or output_activation from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): RNN(layers_info=[["linear", 2]], hidden_activations=input_value, output_activation="relu") RNN(layers_info=[["linear", 2]], hidden_activations="relu", output_activation=input_value) def test_initialiser_user_input(): """Tests whether network rejects an invalid initialiser from user""" inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"] for input_value in inputs_that_should_fail: with pytest.raises(AssertionError): RNN(layers_info=[["linear", 2]], hidden_activations="relu", output_activation="relu", initialiser=input_value) RNN(layers_info=[["linear", 2], ["linear", 2]], hidden_activations="relu", output_activation="relu", initialiser="xavier") def test_batch_norm_layers(): """Tests whether batch_norm_layers method works correctly""" layers = [["gru", 20], ["lstm", 3], ["linear", 4], ["linear", 10]] rnn = RNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) assert len(rnn.batch_norm_layers) == 3 for layer in rnn.batch_norm_layers: assert isinstance(layer, BatchNormalization) def test_linear_layers_only_come_at_end(): """Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they don't only come at the end""" layers = [["gru", 20], ["linear", 4], ["lstm", 3], ["linear", 10]] with pytest.raises(AssertionError): rnn = RNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) layers = [["gru", 20], ["lstm", 3], ["linear", 4], ["linear", 10]] assert RNN(layers_info=layers, hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) def test_output_activation(): """Tests whether network outputs data that has gone through correct activation function""" RANDOM_ITERATIONS = 20 for _ in range(RANDOM_ITERATIONS): data = np.random.random((25, 10, 30)) data = data.astype('float32') RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5]], hidden_activations="relu", output_activation="relu", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", output_activation="relu", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", output_activation="sigmoid", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) assert all(tf.reshape(out, [-1]) <= 1) summed_result = tf.reduce_sum(out, axis=1) summed_result = tf.reshape(summed_result, [-1, 1]) assert summed_result != 1.0 RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", output_activation="softmax", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) assert all(tf.reshape(out, [-1]) <= 1) summed_result = tf.reduce_sum(out, axis=1) assert (np.round(summed_result, 3) == 1.0).all() RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", output_activation="softmax", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) assert all(tf.reshape(out, [-1]) <= 1) summed_result = tf.reduce_sum(out, axis=1) assert (np.round(summed_result, 3) == 1.0).all() RNN_instance = RNN(layers_info=[["linear", 20], ["linear", 50]], hidden_activations="relu") out = RNN_instance(data) assert not all(tf.reshape(out, [-1]) >= 0) assert not all(tf.reshape(out, [-1]) <= 1) summed_result = tf.reduce_sum(out, axis=1) assert not (np.round(summed_result, 3) == 1.0).all() RNN_instance = RNN(layers_info=[ ["lstm", 25], ["linear", 10]], hidden_activations="relu") out = RNN_instance(data) assert not all(tf.reshape(out, [-1]) >= 0) assert not all(tf.reshape(out, [-1]) <= 0) summed_result = tf.reduce_sum(out, axis=1) assert not (np.round(summed_result, 3) == 1.0).all() def test_output_activation(): """Tests whether network outputs data that has gone through correct activation function""" RANDOM_ITERATIONS = 20 for _ in range(RANDOM_ITERATIONS): data = np.random.random((25, 10, 30)) data = data.astype('float32') RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", output_activation="relu", initialiser="xavier", batch_norm=True) out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5]], hidden_activations="relu", output_activation="relu", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", output_activation="relu", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", output_activation="sigmoid", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) assert all(tf.reshape(out, [-1]) <= 1) summed_result = tf.reduce_sum(out, axis=1) summed_result = tf.reshape(summed_result, [-1, 1]) assert summed_result != 1.0 RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]], hidden_activations="relu", output_activation="softmax", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) assert all(tf.reshape(out, [-1]) <= 1) summed_result = tf.reduce_sum(out, axis=1) assert (np.round(summed_result, 3) == 1.0).all() RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", output_activation="softmax", initialiser="xavier") out = RNN_instance(data) assert all(tf.reshape(out, [-1]) >= 0) assert all(tf.reshape(out, [-1]) <= 1) summed_result = tf.reduce_sum(out, axis=1) assert (np.round(summed_result, 3) == 1.0).all() RNN_instance = RNN(layers_info=[["linear", 20], ["linear", 50]], hidden_activations="relu") out = RNN_instance(data) assert not all(tf.reshape(out, [-1]) >= 0) assert not all(tf.reshape(out, [-1]) <= 1) summed_result = tf.reduce_sum(out, axis=1) assert not (np.round(summed_result, 3) == 1.0).all() RNN_instance = RNN(layers_info=[ ["lstm", 25], ["linear", 10]], hidden_activations="relu") out = RNN_instance(data) assert not all(tf.reshape(out, [-1]) >= 0) assert not all(tf.reshape(out, [-1]) <= 0) summed_result = tf.reduce_sum(out, axis=1) assert not (np.round(summed_result, 3) == 1.0).all() def test_y_range(): """Tests whether setting a y range works correctly""" for _ in range(20): val1 = random.random() - 3.0*random.random() val2 = random.random() + 2.0*random.random() lower_bound = min(val1, val2) upper_bound = max(val1, val2) rnn = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", y_range=(lower_bound, upper_bound), initialiser="xavier") random_data = np.random.random((10, 11, 22)) random_data = random_data.astype('float32') out = rnn(random_data) assert all(tf.reshape(out, [-1]) > lower_bound) assert all(tf.reshape(out, [-1]) < upper_bound) def test_deals_with_None_activation(): """Tests whether is able to handle user inputting None as output activation""" assert RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", output_activation=None, initialiser="xavier") def test_y_range_user_input(): """Tests whether network rejects invalid y_range inputs""" invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)] for y_range_value in invalid_y_range_inputs: with pytest.raises(AssertionError): print(y_range_value) rnn = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]], hidden_activations="relu", y_range=y_range_value, initialiser="xavier") def solves_simple_problem(X, y, nn_instance): """Checks if a given network is able to solve a simple problem""" print("X shape ", X.shape) print("y shape ", y.shape) nn_instance.compile(optimizer='adam', loss='mse') nn_instance.fit(X, y, epochs=25) results = nn_instance.evaluate(X, y) print("FINAL RESULT ", results) return results < 0.1 def test_model_trains(): """Tests whether a small range of networks can solve a simple task""" for output_activation in ["sigmoid", "None"]: rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], hidden_activations="relu", output_activation=output_activation, initialiser="xavier") assert solves_simple_problem(X, y, rnn) def test_model_trains_part_2(): """Tests whether a small range of networks can solve a simple task""" z = X[:, 2:3, 3:4] > 5.0 z = np.concatenate([z == 1, z == 0], axis=1) z = z.reshape((-1, 2)) rnn = RNN(layers_info=[["gru", 20], ["lstm", 2]], hidden_activations="relu", output_activation="softmax", dropout=0.01, initialiser="xavier") assert solves_simple_problem(X, z, rnn) rnn = RNN(layers_info=[["lstm", 20], ["linear", 1]], hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, rnn) rnn = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation=None, initialiser="xavier") assert solves_simple_problem(X, y, rnn) def test_model_trains_with_batch_norm(): """Tests whether a model with batch norm on can solve a simple task""" rnn = RNN(layers_info=[["lstm", 20], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation=None, initialiser="xavier", batch_norm=True) assert solves_simple_problem(X, y, rnn) def test_dropout(): """Tests whether dropout layer reads in probability correctly""" rnn = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation="sigmoid", dropout=0.9999, initialiser="xavier") assert rnn.dropout_layer.rate == 0.9999 assert not solves_simple_problem(X, y, rnn) rnn = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation=None, dropout=0.0000001, initialiser="xavier") assert rnn.dropout_layer.rate == 0.0000001 assert solves_simple_problem(X, y, rnn) def test_all_activations_work(): """Tests that all activations get accepted""" nn_instance = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation=None, dropout=0.0000001, initialiser="xavier") for key in nn_instance.str_to_activations_converter.keys(): assert RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations=key, output_activation=key, dropout=0.0000001, initialiser="xavier") def test_all_initialisers_work(): """Tests that all initialisers get accepted""" nn_instance = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], hidden_activations="relu", output_activation=None, dropout=0.0000001, initialiser="xavier") for key in nn_instance.str_to_initialiser_converter.keys(): assert RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], dropout=0.0000001, initialiser=key) def test_output_shapes(): """Tests whether network outputs of correct shape""" rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 3]], hidden_activations="relu", initialiser="xavier") output = rnn(X) assert output.shape == (N, 3) rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], hidden_activations="relu", initialiser="xavier", return_final_seq_only=False) output = rnn(X) assert output.shape == (N, 3, 7) rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["lstm", 3]], hidden_activations="relu", initialiser="xavier") output = rnn(X) assert output.shape == (N, 3) rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["lstm", 7]], hidden_activations="relu", initialiser="xavier", return_final_seq_only=False) output = rnn(X) assert output.shape == (N, 3, 7) def test_return_final_seq_user_input_valid(): """Checks whether network only accepts a valid boolean value for return_final_seq_only""" for valid_case in [True, False]: assert RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], hidden_activations="relu", initialiser="xavier", return_final_seq_only=valid_case) for invalid_case in [[True], 22, [1, 3], (True, False), (5, False)]: with pytest.raises(AssertionError): print(invalid_case) RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], hidden_activations="relu", initialiser="xavier", return_final_seq_only=invalid_case) def test_embedding_layers(): """Tests whether create_embedding_layers_info method works correctly""" for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)): nn_instance = RNN( layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], columns_of_data_to_be_embedded=[0, 1], embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]]) for layer in nn_instance.embedding_layers: assert isinstance(layer, tf.keras.layers.Embedding) assert len(nn_instance.embedding_layers) == 2 assert nn_instance.embedding_layers[0].input_dim == embedding_in_dim_1 assert nn_instance.embedding_layers[0].output_dim == embedding_out_dim_1 assert nn_instance.embedding_layers[1].input_dim == embedding_in_dim_2 assert nn_instance.embedding_layers[1].output_dim == embedding_out_dim_2 def test_incorporate_embeddings(): """Tests the method incorporate_embeddings""" X_new = X X_new[:, [0, 2], :] = tf.round(X_new[:, [0, 2], :]) nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], columns_of_data_to_be_embedded=[0, 2], embedding_dimensions=[[50, 3], [55, 4]]) out = nn_instance.incorporate_embeddings(X) assert out.shape == (N, 3, 10) nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], columns_of_data_to_be_embedded=[0, 1, 2], embedding_dimensions=[[50, 3], [55, 4], [55, 4]]) out = nn_instance.incorporate_embeddings(X) assert out.shape == (N, 3, 13) nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], columns_of_data_to_be_embedded=[2], embedding_dimensions=[[150, 30]]) out = nn_instance.incorporate_embeddings(X) assert out.shape == (N, 3, 34) def test_embedding_network_can_solve_simple_problem(): """Tests whether network can solve simple problem using embeddings""" X = (np.random.random((N, 4, 5)) - 0.5) * 5.0 + 20.0 y = (X[:, :, 0] >= 25) * (X[:, :, 1] <= 25) * 1.0 nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], columns_of_data_to_be_embedded=[0, 1], embedding_dimensions=[[50, 3], [55, 3]]) assert solves_simple_problem(X, y, nn_instance) nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], columns_of_data_to_be_embedded=[1], embedding_dimensions=[[55, 3]]) assert solves_simple_problem(X, y, nn_instance) nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], columns_of_data_to_be_embedded=[1, 3, 0], embedding_dimensions=[[55, 3], [55, 5], [55, 2]]) assert solves_simple_problem(X, y, nn_instance) def test_print_model_summary(): nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]], columns_of_data_to_be_embedded=[2], embedding_dimensions=[[150, 30]]) nn_instance.print_model_summary((64, 11, 11)) def test_output_heads_error_catching(): """Tests that having multiple output heads catches errors from user inputs""" output_dims_that_should_break = [["linear", 2, 2, "SAME", "conv", 3, 4, "SAME"], [[["lstm", 3], ["gru", 4]]], [[2, 8]], [-33, 33, 33, 33, 33]] for output_dim in output_dims_that_should_break: with pytest.raises(AssertionError): RNN(layers_info=[["gru", 20], ["lstm", 8], output_dim], hidden_activations="relu", output_activation="relu") output_activations_that_should_break = ["relu", ["relu"], ["relu", "softmax"]] for output_activation in output_activations_that_should_break: with pytest.raises(AssertionError): RNN(layers_info=[["gru", 20], ["lstm", 8], [["linear", 5], ["linear", 2], ["linear", 5]]], hidden_activations="relu", output_activation=output_activation) def test_output_head_layers(): """Tests whether the output head layers get created properly""" for output_dim in [[["linear", 3],["linear", 9]], [["linear", 4], ["linear", 20]], [["linear", 1], ["linear", 1]]]: nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], output_dim], hidden_activations="relu", output_activation=["softmax", None]) assert nn_instance.output_layers[0].units == output_dim[0][1] assert nn_instance.output_layers[1].units == output_dim[1][1] def test_output_head_activations_work(): """Tests that output head activations work properly""" output_dim = [["linear", 5], ["linear", 10], ["linear", 3]] nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], output_dim], hidden_activations="relu", output_activation=["softmax", None, "relu"]) x = np.random.random((20, 10, 4)) * -20.0 x = x.astype('float32') out = nn_instance(x) assert out.shape == (20, 18) sums = tf.reduce_sum(out[:, :5], axis=1) sums_others = tf.reduce_sum(out[:, 5:], axis=1) sums_others_2 = tf.reduce_sum(out[:, 5:15], axis=1) sums_others_3 = tf.reduce_sum(out[:, 15:18], axis=1) for row in range(out.shape[0]): assert tf.math.equal(np.round(sums[row], 4), 1.0), sums[row] assert not tf.math.equal(np.round(sums_others[row], 4), 1.0), np.round(sums_others[row], 4) assert not tf.math.equal(np.round(sums_others_2[row], 4), 1.0), np.round(sums_others_2[row], 4) assert not tf.math.equal(np.round(sums_others_3[row], 4), 1.0), np.round(sums_others_3[row], 4) for col in range(3): assert out[row, 15 + col] >= 0.0, out[row, 15 + col] def test_output_head_shapes_correct(): """Tests that the output shape of network is correct when using multiple outpout heads""" N = 20 X = np.random.random((N, 10, 4)) * -20.0 X = X.astype('float32') for _ in range(25): nn_instance = RNN( layers_info=[["gru", 20], ["lstm", 8], ["linear", 1], ["linear", 12]], hidden_activations="relu") out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 12 for output_dim in [[ ["linear", 10], ["linear", 4], ["linear", 6]], [["linear", 3], ["linear", 8], ["linear", 9]]]: nn_instance = RNN( layers_info=[["gru", 20], ["lstm", 8], ["linear", 1], ["linear", 12], output_dim], hidden_activations="relu", output_activation=["softmax", None, "relu"]) out = nn_instance(X) assert out.shape[0] == N assert out.shape[1] == 20