Showing preview only (266K chars total). Download the full file or copy to clipboard to get everything.
Repository: p-christ/nn_builder
Branch: master
Commit: d58182e579c9
Files: 30
Total size: 254.4 KB
Directory structure:
gitextract_38pgry39/
├── .gitignore
├── .travis.yml
├── LICENSE
├── README.md
├── miscellaneous/
│ ├── material_for_readme/
│ │ ├── README.md
│ │ ├── nn_builder_code
│ │ ├── nn_builder_tf_code
│ │ ├── non_nn_builder_code
│ │ └── non_nn_builder_tf_code
│ └── testreadme.rst
├── nn_builder/
│ ├── Overall_Base_Network.py
│ ├── __init__.py
│ ├── pytorch/
│ │ ├── Base_Network.py
│ │ ├── CNN.py
│ │ ├── NN.py
│ │ ├── RNN.py
│ │ └── __init__.py
│ └── tensorflow/
│ ├── Base_Network.py
│ ├── CNN.py
│ ├── NN.py
│ ├── RNN.py
│ └── __init__.py
├── requirements.txt
├── setup.py
└── tests/
├── pytorch_tests/
│ ├── test_pytorch_CNN.py
│ ├── test_pytorch_NN.py
│ └── test_pytorch_RNN.py
└── tensorflow_tests/
├── test_tf_CNN.py
├── test_tf_NN.py
└── test_tf_RNN.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
*.pyc
*cache
*.idea
*__pycache__
*venv
*nn_builder.egg-info
*build
*dist
.DS_Store
*to_do_list
*.ipynb_checkpoints
================================================
FILE: .travis.yml
================================================
language:
python
python: 3.7
dist: xenial
sudo: true
install:
- pip install -r requirements.txt -q
- pip install sklearn
- pip install torchvision
script:
- export PYTHONPATH="$PYTHONPATH:$PWD"
- export PYTHONPATH="${PYTHONPATH}:/home/travis/build/p-christ/nn_builder/tests"
- export PYTHONPATH="${PYTHONPATH}:/home/travis/build/p-christ/nn_builder"
- python -m pytest /home/travis/build/p-christ/nn_builder/tests/
================================================
FILE: LICENSE
================================================
Copyright (c) 2018 The Python Packaging Authority
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
[](https://pepy.tech/project/nn-builder)  [](https://github.com/dwyl/esta/issues)
**nn_builder lets you build neural networks with less boilerplate code**. You specify the type of network you want and it builds it.
### Install
`pip install nn_builder`
### Support
| Network Type | **NN** | **CNN** | **RNN** |
| ------- | ------- | ------- | ------- |
| PyTorch | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| TensorFlow 2.0 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
### Examples
On the left is how you can create the PyTorch neural network on the right in only 1 line of code using nn_builder:
Similarly for TensorFlow on the left is how you can create the CNN on the right in only 1 line of code using nn_builder:
### Usage
See this [colab notebook](https://colab.research.google.com/drive/1UdMT3aVSV0L5Rq11nyLHxMSVtTVZryhW) for lots of examples of how to use the module.
3 types of PyTorch and TensorFlow network are currently supported: NN, CNN and RNN. Each network takes the following arguments:
| Field | Description | Default |
| :---: | :----------: | :---: |
| *input_dim*| Dimension of the input into the network. See below for more detail. Not needed for Tensorflow. | N/A |
| *layers_info* | List to indicate the layers of the network you want. Exact requirements depend on network type, see below for more detail | N/A |
| *output_activation* | String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads | No activation |
| *hidden_activations* | String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer), default is ReLU and for example "tanh" would have tanh applied on all hidden layer activations | ReLU after every hidden layer |
| *dropout* | Float to indicate what dropout probability you want applied after each hidden layer | 0 |
| *initialiser* | String to indicate which initialiser you want used to initialise all the parameters | PyTorch & TF Default |
| *batch_norm* | Boolean to indicate whether you want batch norm applied to the output of every hidden layer | False |
| *columns of_data_to_be_embedded* | List to indicate the column numbers of the data that you want to be put through an embedding layer before being fed through the hidden layers of the network | No embeddings |
| *embedding_dimensions* | If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list of the form: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...] | No embeddings |
| *y_range* | Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks | No range |
| *random_seed* | Integer to indicate the random seed you want to use | 0 |
| *return_final_seq_only* | Only needed for RNN. Boolean to indicate whether you only want to return the output for the final timestep (True) or if you want to return the output for all timesteps (False) | True |
Each network type has slightly different requirements for **input_dim** and **layers_info** as explained below:
---
### 1. NN
* **input_dim**: # Features in PyTorch, not needed for TensorFlow
* **layers_info**: List of integers to indicate number of hidden units you want per linear layer.
* For example:
```
from nn_builder.pytorch.NN import NN
model = NN(input_dim=5, layers_info=[10, 10, 1], output_activation=None, hidden_activations="relu",
dropout=0.0, initialiser="xavier", batch_norm=False)
```
---
### 2. CNN
* **input_dim**: (# Channels, Height, Width) in PyTorch, not needed for TensorFlow
* **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a CNN can be one of these 4 forms:
* ["conv", channels, kernel size, stride, padding]
* ["maxpool", kernel size, stride, padding]
* ["avgpool", kernel size, stride, padding]
* ["linear", units]
* For a PyTorch network kernel size, stride, padding and units must be integers. For TensorFlow they must all be integers except for padding which must be one of {“valid”, “same”}
* For example:
```
from nn_builder.pytorch.CNN import CNN
model = CNN(input_dim=(3, 64, 64),
layers_info=[["conv", 32, 3, 1, 0], ["maxpool", 2, 2, 0],
["conv", 64, 3, 1, 2], ["avgpool", 2, 2, 0],
["linear", 10]],
hidden_activations="relu", output_activation="softmax", dropout=0.0,
initialiser="xavier", batch_norm=True)
```
---
### 3. RNN
* **input_dim**: # Features in PyTorch, not needed for TensorFlow
* **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a CNN can be one of these 4 forms:
* ["lstm", units]
* ["gru", units]
* ["linear", units]
* For example:
```
from nn_builder.pytorch.CNN import CNN
model = RNN(input_dim=5, layers_info=[["gru", 50], ["lstm", 10], ["linear", 2]],
hidden_activations="relu", output_activation="softmax",
batch_norm=False, dropout=0.0, initialiser="xavier")
```
---
## Contributing
Anyone is very welcome to contribute via a pull request. Please see the [issues](https://github.com/p-christ/nn_builder/issues)
page for ideas on the best areas to contribute to and try to:
1. Add tests to the tests folder that cover any code you write
1. Write comments for every function
1. Create a colab notebook demonstrating how any extra functionality you created works
To help you remember things you learn about machine learning in general checkout [Gizmo](https://gizmo.ai/landing/github_links)
================================================
FILE: miscellaneous/material_for_readme/README.md
================================================
[](https://pepy.tech/project/nn-builder)  [](https://github.com/dwyl/esta/issues)
**nn_builder lets you build neural networks without boilerplate code**. You specify the type of network you want and it builds it.
### Install
`pip install nn_builder`
### Support
| Network Type | **NN** | **CNN** | **RNN** |
| ------- | ------- | ------- | ------- |
| PyTorch | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| TensorFlow 2.0 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
### Examples
On the left is how you can create the PyTorch neural network on the right in only 1 line of code using nn_builder:
+-----------------------------------------------------------+----------------------------------------------------------+
| **Building a simple NN with nn_builder** | **Building a simple NN without nn_builder** |
+-----------------------------------------------------------+----------------------------------------------------------+
| .. code:: python | .. code:: python |
| # With nn_builder | # Without nn_builder |
| from nn_builder.pytorch.NN import NN | import torch.nn as nn |
| model = NN(input_dim=25, | class NN(nn.Module): |
| layers=[150, 100, 50, 50, 50, 50, 5], | def __init__(self): |
| output_activation="softmax", dropout=0.5, | nn.Module.__init__(self) |
| initialiser="xavier", batch_norm=True) | self.fc1 = nn.Linear(25, 150) |
| | self.fc2 = nn.Linear(150, 100) |
| | self.fc3 = nn.Linear(100, 50) |
| | self.fc4 = nn.Linear(50, 50) |
| | self.fc5 = nn.Linear(50, 50) |
| | self.fc6 = nn.Linear(50, 50) |
| | self.fc7 = nn.Linear(50, 5) |
| | |
| | self.bn1 = nn.BatchNorm1d(150) |
| | self.bn2 = nn.BatchNorm1d(100) |
| | self.bn3 = nn.BatchNorm1d(50) |
| | self.bn4 = nn.BatchNorm1d(50) |
| | self.bn5 = nn.BatchNorm1d(50) |
| | self.bn6 = nn.BatchNorm1d(50) |
| | |
| | self.dropout = nn.Dropout(p=0.5) |
| | for params in [self.fc1, self.fc2, self.fc3, |
| | self.fc4, self.fc5, self.fc6, |
| | self.fc7]: |
| | nn.init.xavier_uniform_(params.weight) |
| | def forward(self, x): |
| | x = self.dropout(self.bn1(nn.ReLU()(self.fc1(x)))) |
| | x = self.dropout(self.bn2(nn.ReLU()(self.fc2(x)))) |
| | x = self.dropout(self.bn3(nn.ReLU()(self.fc3(x)))) |
| | x = self.dropout(self.bn4(nn.ReLU()(self.fc4(x)))) |
| | x = self.dropout(self.bn5(nn.ReLU()(self.fc5(x)))) |
| | x = self.dropout(self.bn6(nn.ReLU()(self.fc6(x)))) |
| | x = self.fc7(x) |
| | x = nn.Softmax(dim=1)(x) |
| | return x |
| | |
| | model = NN() |
+-----------------------------------------------------------+----------------------------------------------------------+
Similarly for TensorFlow on the left is how you can create the CNN on the right in only 1 line of code using nn_builder:
### Usage
See this [colab notebook](https://colab.research.google.com/drive/1UdMT3aVSV0L5Rq11nyLHxMSVtTVZryhW) for lots of examples of how to use the module.
3 types of PyTorch and TensorFlow network are currently supported: NN, CNN and RNN. Each network takes the following arguments:
| Field | Description | Default |
| :---: | :----------: | :---: |
| *input_dim*| Dimension of the input into the network. See below for more detail. Not needed for Tensorflow. | N/A |
| *layers_info* | List to indicate the layers of the network you want. Exact requirements depend on network type, see below for more detail | N/A |
| *output_activation* | String to indicate the activation function you want the output to go through. Provide a list of strings if you want multiple output heads | No activation |
| *hidden_activations* | String or list of string to indicate the activations you want used on the output of hidden layers (not including the output layer), default is ReLU and for example "tanh" would have tanh applied on all hidden layer activations | ReLU after every hidden layer |
| *dropout* | Float to indicate what dropout probability you want applied after each hidden layer | 0 |
| *initialiser* | String to indicate which initialiser you want used to initialise all the parameters | PyTorch & TF Default |
| *batch_norm* | Boolean to indicate whether you want batch norm applied to the output of every hidden layer | False |
| *columns of_data_to_be_embedded* | List to indicate the column numbers of the data that you want to be put through an embedding layer before being fed through the hidden layers of the network | No embeddings |
| *embedding_dimensions* | If you have categorical variables you want embedded before flowing through the network then you specify the embedding dimensions here with a list of the form: [ [embedding_input_dim_1, embedding_output_dim_1], [embedding_input_dim_2, embedding_output_dim_2] ...] | No embeddings |
| *y_range* | Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the output values to in regression tasks | No range |
| *random_seed* | Integer to indicate the random seed you want to use | 0 |
| *return_final_seq_only* | Only needed for RNN. Boolean to indicate whether you only want to return the output for the final timestep (True) or if you want to return the output for all timesteps (False) | True |
Each network type has slightly different requirements for **input_dim** and **layers_info** as explained below:
---
##### 1. NN
* **input_dim**: # Features in PyTorch, not needed for TensorFlow
* **layers_info**: List of integers to indicate number of hidden units you want per linear layer.
* For example:
```
from nn_builder.pytorch.NN import NN
model = NN(input_dim=5, layers_info=[10, 10, 1], output_activation=None, hidden_activations="relu",
dropout=0.0, initialiser="xavier", batch_norm=False)
```
---
##### 2. CNN
* **input_dim**: (# Channels, Height, Width) in PyTorch, not needed for TensorFlow
* **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a CNN can be one of these 4 forms:
* ["conv", channels, kernel size, stride, padding]
* ["maxpool", kernel size, stride, padding]
* ["avgpool", kernel size, stride, padding]
* ["linear", units]
* For a PyTorch network kernel size, stride, padding and units must be integers. For TensorFlow they must all be integers except for padding which must be one of {“valid”, “same”}
* For example:
```
from nn_builder.pytorch.CNN import CNN
model = CNN(input_dim=(3, 64, 64),
layers_info=[["conv", 32, 3, 1, 0], ["maxpool", 2, 2, 0],
["conv", 64, 3, 1, 2], ["avgpool", 2, 2, 0],
["linear", 10]],
hidden_activations="relu", output_activation="softmax", dropout=0.0,
initialiser="xavier", batch_norm=True)
```
---
##### 3. RNN
* **input_dim**: # Features in PyTorch, not needed for TensorFlow
* **layers_info**: We expect the field *layers_info* to be a list of lists indicating the size and type of layers that you want. Each layer in a CNN can be one of these 4 forms:
* ["lstm", units]
* ["gru", units]
* ["linear", units]
* For example:
```
from nn_builder.pytorch.CNN import CNN
model = RNN(input_dim=5, layers_info=[["gru", 50], ["lstm", 10], ["linear", 2]],
hidden_activations="relu", output_activation="softmax",
batch_norm=False, dropout=0.0, initialiser="xavier")
```
---
## Contributing
Together we can make something that is useful for thousands of people. Anyone is very welcome to contribute via a pull request. Please see the [issues](https://github.com/p-christ/nn_builder/issues)
page for ideas on the best areas to contribute to and try to:
1. Add tests to the tests folder that cover any code you write
1. Write comments for every function
1. Create a colab notebook demonstrating how any extra functionality you created works
================================================
FILE: miscellaneous/material_for_readme/nn_builder_code
================================================
# With nn_builder
from nn_builder.pytorch.NN import NN
model = NN(input_dim=25, layers=[150, 100, 50, 50, 50, 50, 5],
output_activation="softmax", dropout=0.5, initialiser="xavier",
batch_norm=True)
================================================
FILE: miscellaneous/material_for_readme/nn_builder_tf_code
================================================
#With nn_builder
from nn_builder.tensorflow.CNN import CNN
model=CNN(layers_info=[["conv", 32, 3, 1, "valid"],["maxpool", 3, 3, "valid"],
["conv", 64, 3, 1, "valid"],["maxpool", 3, 2, "valid"],
["conv", 128, 3, 1, "valid"],["maxpool", 3, 1, "valid"],
["conv", 256, 3, 1, "valid"],["avgpool", 3, 1, "valid"],
["linear", 64], ["linear", 10]],
hidden_activations="relu", output_activation="softmax",
dropout=0.2, initialiser="glorot_normal", batch_norm=False)
================================================
FILE: miscellaneous/material_for_readme/non_nn_builder_code
================================================
# Without nn_builder
import torch.nn as nn
class NN(nn.Module):
def __init__(self):
nn.Module.__init__(self)
self.fc1 = nn.Linear(25, 150)
self.fc2 = nn.Linear(150, 100)
self.fc3 = nn.Linear(100, 50)
self.fc4 = nn.Linear(50, 50)
self.fc5 = nn.Linear(50, 50)
self.fc6 = nn.Linear(50, 50)
self.fc7 = nn.Linear(50, 5)
self.bn1 = nn.BatchNorm1d(150)
self.bn2 = nn.BatchNorm1d(100)
self.bn3 = nn.BatchNorm1d(50)
self.bn4 = nn.BatchNorm1d(50)
self.bn5 = nn.BatchNorm1d(50)
self.bn6 = nn.BatchNorm1d(50)
self.dropout = nn.Dropout(p=0.5)
for params in [self.fc1, self.fc2, self.fc3, self.fc4, self.fc5,
self.fc6, self.fc7]:
nn.init.xavier_uniform_(params.weight)
def forward(self, x):
x = self.dropout(self.bn1(nn.ReLU()(self.fc1(x))))
x = self.dropout(self.bn2(nn.ReLU()(self.fc2(x))))
x = self.dropout(self.bn3(nn.ReLU()(self.fc3(x))))
x = self.dropout(self.bn4(nn.ReLU()(self.fc4(x))))
x = self.dropout(self.bn5(nn.ReLU()(self.fc5(x))))
x = self.dropout(self.bn6(nn.ReLU()(self.fc6(x))))
x = self.fc7(x)
x = nn.Softmax(dim=1)(x)
return x
model = NN()
================================================
FILE: miscellaneous/material_for_readme/non_nn_builder_tf_code
================================================
#Without nn_builder
import tensorflow as tf
from tensorflow.keras import Model, activations
from tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, \
MaxPool2D, AveragePooling2D
class CNN(Model):
def __init__(self):
Model.__init__(self)
self.conv1 = Conv2D(filters=32, kernel_size=3, strides=1,
padding="valid", activation="relu",
kernel_initializer="glorot_normal")
self.conv2 = Conv2D(filters=64, kernel_size=3, strides=1,
padding="valid", activation="relu",
kernel_initializer="glorot_normal")
self.conv3 = Conv2D(filters=128, kernel_size=3, strides=1,
padding="valid", activation="relu",
kernel_initializer="glorot_normal")
self.conv4 = Conv2D(filters=256, kernel_size=3, strides=1,
padding="valid", activation="relu",
kernel_initializer="glorot_normal")
self.maxpool1 = MaxPool2D(pool_size=3, strides=3, padding="valid")
self.maxpool2 = MaxPool2D(pool_size=3, strides=2, padding="valid")
self.maxpool3 = MaxPool2D(pool_size=3, strides=1, padding="valid")
self.avgpool = AveragePooling2D(pool_size=3, strides=1, padding="valid")
self.linear1 = Dense(64, activation="relu")
self.linear2 = Dense(10, activation="softmax")
self.dropout = tf.keras.layers.Dropout(rate=0.2)
def call(self, x):
x = self.dropout(self.maxpool1(self.conv1(x)))
x = self.dropout(self.maxpool2(self.conv2(x)))
x = self.dropout(self.maxpool3(self.conv3(x)))
x = self.dropout(self.avgpool(self.conv4(x)))
x = Flatten()(x)
x = self.dropout(self.linear1(x))
x = self.linear2(x)
return x
model = CNN()
================================================
FILE: miscellaneous/testreadme.rst
================================================
+-----------------------------------------------------------+----------------------------------------------------------+
| **Building a simple NN with nn_builder** | **Building a simple NN without nn_builder** |
+-----------------------------------------------------------+----------------------------------------------------------+
| .. code:: python | .. code:: python |
| # With nn_builder | # Without nn_builder |
| from nn_builder.pytorch.NN import NN | import torch.nn as nn |
| model = NN(input_dim=25, | class NN(nn.Module): |
| layers=[150, 100, 50, 50, 50, 50, 5], | def __init__(self): |
| output_activation="softmax", dropout=0.5, | nn.Module.__init__(self) |
| initialiser="xavier", batch_norm=True) | self.fc1 = nn.Linear(25, 150) |
| | self.fc2 = nn.Linear(150, 100) |
| | self.fc3 = nn.Linear(100, 50) |
| | self.fc4 = nn.Linear(50, 50) |
| | self.fc5 = nn.Linear(50, 50) |
| | self.fc6 = nn.Linear(50, 50) |
| | self.fc7 = nn.Linear(50, 5) |
| | |
| | self.bn1 = nn.BatchNorm1d(150) |
| | self.bn2 = nn.BatchNorm1d(100) |
| | self.bn3 = nn.BatchNorm1d(50) |
| | self.bn4 = nn.BatchNorm1d(50) |
| | self.bn5 = nn.BatchNorm1d(50) |
| | self.bn6 = nn.BatchNorm1d(50) |
| | |
| | self.dropout = nn.Dropout(p=0.5) |
| | for params in [self.fc1, self.fc2, self.fc3, |
| | self.fc4, self.fc5, self.fc6, |
| | self.fc7]: |
| | nn.init.xavier_uniform_(params.weight) |
| | def forward(self, x): |
| | x = self.dropout(self.bn1(nn.ReLU()(self.fc1(x)))) |
| | x = self.dropout(self.bn2(nn.ReLU()(self.fc2(x)))) |
| | x = self.dropout(self.bn3(nn.ReLU()(self.fc3(x)))) |
| | x = self.dropout(self.bn4(nn.ReLU()(self.fc4(x)))) |
| | x = self.dropout(self.bn5(nn.ReLU()(self.fc5(x)))) |
| | x = self.dropout(self.bn6(nn.ReLU()(self.fc6(x)))) |
| | x = self.fc7(x) |
| | x = nn.Softmax(dim=1)(x) |
| | return x |
| | |
| | model = NN() |
+-----------------------------------------------------------+----------------------------------------------------------+
+------------------------------------------------+--------------------------------------------+
| **Script to train an SVM on the iris dataset** | **The same script as a Sacred experiment** |
+------------------------------------------------+--------------------------------------------+
| .. code:: python | .. code:: python |
| # With nn_builder | |
| from nn_builder.pytorch.NN import NN | from numpy.random import permutation |
| | from sklearn import svm, datasets |
| | from sacred import Experiment |
| | ex = Experiment('iris_rbf_svm') |
| | |
| | @ex.config |
| | def cfg(): |
| C = 1.0 | C = 1.0 |
| gamma = 0.7 | gamma = 0.7 |
| | |
| | @ex.automain |
| | def run(C, gamma): |
| iris = datasets.load_iris() | iris = datasets.load_iris() |
| perm = permutation(iris.target.size) | per = permutation(iris.target.size) |
| iris.data = iris.data[perm] | iris.data = iris.data[per] |
| iris.target = iris.target[perm] | iris.target = iris.target[per] |
| clf = svm.SVC(C, 'rbf', gamma=gamma) | clf = svm.SVC(C, 'rbf', gamma=gamma) |
| clf.fit(iris.data[:90], | clf.fit(iris.data[:90], |
| iris.target[:90]) | iris.target[:90]) |
| print(clf.score(iris.data[90:], | return clf.score(iris.data[90:], |
| iris.target[90:])) | iris.target[90:]) |
+------------------------------------------------+--------------------------------------------+
# With nn_builder
model = NN(input_dim=25, layers=[150, 100, 50, 50, 50, 50, 5],
output_activation="softmax", dropout=0.5, initialiser="xavier",
batch_norm=True)
================================================
FILE: nn_builder/Overall_Base_Network.py
================================================
from abc import ABC, abstractmethod
class Overall_Base_Network(ABC):
def __init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm,
y_range, random_seed):
self.set_all_random_seeds(random_seed)
self.input_dim = input_dim
self.layers_info = layers_info
self.hidden_activations = hidden_activations
self.output_activation = output_activation
self.dropout = dropout
self.initialiser = initialiser
self.batch_norm = batch_norm
self.y_range = y_range
self.str_to_activations_converter = self.create_str_to_activations_converter()
self.str_to_initialiser_converter = self.create_str_to_initialiser_converter()
self.check_all_user_inputs_valid()
self.initialiser_function = self.str_to_initialiser_converter[initialiser.lower()]
self.hidden_layers = self.create_hidden_layers()
self.output_layers = self.create_output_layers()
self.dropout_layer = self.create_dropout_layer()
if self.batch_norm: self.batch_norm_layers = self.create_batch_norm_layers()
@abstractmethod
def check_all_user_inputs_valid(self):
"""Checks that all the user inputs were valid"""
raise NotImplementedError
@abstractmethod
def create_hidden_layers(self):
"""Creates the hidden layers in the network"""
raise NotImplementedError
@abstractmethod
def create_output_layers(self):
"""Creates the output layers in the network"""
raise NotImplementedError
@abstractmethod
def create_batch_norm_layers(self):
"""Creates the batch norm layers in the network"""
raise NotImplementedError
@abstractmethod
def create_dropout_layer(self):
"""Creates the dropout layers in the network"""
raise NotImplementedError
@abstractmethod
def print_model_summary(self):
"""Prints a summary of the model"""
raise NotImplementedError
@abstractmethod
def set_all_random_seeds(self, random_seed):
"""Sets all random seeds"""
raise NotImplementedError
def check_NN_layers_valid(self):
"""Checks that user input for hidden_units is valid"""
assert isinstance(self.layers_info, list), "hidden_units must be a list"
list_error_msg = "neurons must be a list of integers"
integer_error_msg = "Every element of hidden_units must be 1 or higher"
activation_error_msg = "The number of output activations provided should match the number of output layers"
for neurons in self.layers_info[:-1]:
assert isinstance(neurons, int), list_error_msg
assert neurons > 0, integer_error_msg
output_layer = self.layers_info[-1]
if isinstance(output_layer, list):
assert len(output_layer) == len(self.output_activation), activation_error_msg
for output_dim in output_layer:
assert isinstance(output_dim, int), list_error_msg
assert output_dim > 0, integer_error_msg
else:
assert isinstance(self.output_activation, str) or self.output_activation is None, activation_error_msg
assert isinstance(output_layer, int), list_error_msg
assert output_layer > 0, integer_error_msg
def check_NN_input_dim_valid(self):
"""Checks that user input for input_dim is valid"""
assert isinstance(self.input_dim, int), "input_dim must be an integer"
assert self.input_dim > 0, "input_dim must be 1 or higher"
def check_activations_valid(self):
"""Checks that user input for hidden_activations and output_activation is valid"""
valid_activations_strings = self.str_to_activations_converter.keys()
if self.output_activation is None: self.output_activation = "None"
if isinstance(self.output_activation, list):
for activation in self.output_activation:
if activation is not None:
assert activation.lower() in set(valid_activations_strings), "Output activations must be string from list {}".format(valid_activations_strings)
else:
assert self.output_activation.lower() in set(valid_activations_strings), "Output activation must be string from list {}".format(valid_activations_strings)
assert isinstance(self.hidden_activations, str) or isinstance(self.hidden_activations, list), "hidden_activations must be a string or a list of strings"
if isinstance(self.hidden_activations, str):
assert self.hidden_activations.lower() in set(valid_activations_strings), "hidden_activations must be from list {}".format(valid_activations_strings)
elif isinstance(self.hidden_activations, list):
assert len(self.hidden_activations) == len(self.layers_info), "if hidden_activations is a list then you must provide 1 activation per hidden layer"
for activation in self.hidden_activations:
assert isinstance(activation, str), "hidden_activations must be a string or list of strings"
assert activation.lower() in set(valid_activations_strings), "each element in hidden_activations must be from list {}".format(valid_activations_strings)
def check_embedding_dimensions_valid(self):
"""Checks that user input for embedding_dimensions is valid"""
assert isinstance(self.embedding_dimensions, list), "embedding_dimensions must be a list"
for embedding_dim in self.embedding_dimensions:
assert len(embedding_dim) == 2 and isinstance(embedding_dim, list), \
"Each element of embedding_dimensions must be of form (input_dim, output_dim)"
def check_y_range_values_valid(self):
"""Checks that user input for y_range is valid"""
if self.y_range:
assert isinstance(self.y_range, tuple) and len(self.y_range) == 2, "y_range must be a tuple of 2 floats or integers"
for elem in range(2):
assert isinstance(self.y_range[elem], float) or isinstance(self.y_range[elem], int), "y_range must be a tuple of 2 floats or integers"
assert self.y_range[0] <= self.y_range[1], "y_range's first element must be smaller than the second element"
def check_timesteps_to_output_valid(self):
"""Checks that user input for timesteps_to_output is valid"""
assert self.timesteps_to_output in ["all", "last"]
def check_initialiser_valid(self):
"""Checks that user input for initialiser is valid"""
valid_initialisers = set(self.str_to_initialiser_converter.keys())
assert isinstance(self.initialiser, str), "initialiser must be a string from list {}".format(valid_initialisers)
assert self.initialiser.lower() in valid_initialisers, "initialiser must be from list {}".format(valid_initialisers)
def check_return_final_seq_only_valid(self):
"""Checks whether user input for return_final_seq_only is a boolean and therefore valid. Only relevant for RNNs"""
assert isinstance(self.return_final_seq_only, bool)
def get_activation(self, activations, ix=None):
"""Gets the activation function"""
if isinstance(activations, list):
return self.str_to_activations_converter[str(activations[ix]).lower()]
return self.str_to_activations_converter[str(activations).lower()]
================================================
FILE: nn_builder/__init__.py
================================================
================================================
FILE: nn_builder/pytorch/Base_Network.py
================================================
import random
import numpy as np
import torch
import torch.nn as nn
from nn_builder.Overall_Base_Network import Overall_Base_Network
from abc import ABC, abstractmethod
class Base_Network(Overall_Base_Network, ABC):
"""Base class for PyTorch neural network classes"""
def __init__(self, input_dim, layers_info, output_activation,
hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed):
self.str_to_activations_converter = self.create_str_to_activations_converter()
self.str_to_initialiser_converter = self.create_str_to_initialiser_converter()
super().__init__(input_dim, layers_info, output_activation,
hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed)
self.initialise_all_parameters()
# Flag we use to run checks on the input data into forward the first time it is entered
self.checked_forward_input_data_once = False
@abstractmethod
def initialise_all_parameters(self):
"""Initialises all the parameters of the network"""
raise NotImplementedError
@abstractmethod
def forward(self, input_data):
"""Runs a forward pass of the network"""
raise NotImplementedError
@abstractmethod
def check_input_data_into_forward_once(self, input_data):
"""Checks the input data into the network is of the right form. Only runs the first time data is provided
otherwise would slow down training too much"""
raise NotImplementedError
def set_all_random_seeds(self, random_seed):
"""Sets all possible random seeds so results can be reproduced"""
torch.backends.cudnn.deterministic = True
torch.manual_seed(random_seed)
random.seed(random_seed)
np.random.seed(random_seed)
if torch.cuda.is_available(): torch.cuda.manual_seed_all(random_seed)
def create_str_to_activations_converter(self):
"""Creates a dictionary which converts strings to activations"""
str_to_activations_converter = {"elu": nn.ELU(), "hardshrink": nn.Hardshrink(), "hardtanh": nn.Hardtanh(),
"leakyrelu": nn.LeakyReLU(), "logsigmoid": nn.LogSigmoid(), "prelu": nn.PReLU(),
"relu": nn.ReLU(), "relu6": nn.ReLU6(), "rrelu": nn.RReLU(), "selu": nn.SELU(),
"sigmoid": nn.Sigmoid(), "softplus": nn.Softplus(), "logsoftmax": nn.LogSoftmax(),
"softshrink": nn.Softshrink(), "softsign": nn.Softsign(), "tanh": nn.Tanh(),
"tanhshrink": nn.Tanhshrink(), "softmin": nn.Softmin(), "softmax": nn.Softmax(dim=1),
"none": None}
return str_to_activations_converter
def create_str_to_initialiser_converter(self):
"""Creates a dictionary which converts strings to initialiser"""
str_to_initialiser_converter = {"uniform": nn.init.uniform_, "normal": nn.init.normal_,
"eye": nn.init.eye_,
"xavier_uniform": nn.init.xavier_uniform_, "xavier": nn.init.xavier_uniform_,
"xavier_normal": nn.init.xavier_normal_,
"kaiming_uniform": nn.init.kaiming_uniform_, "kaiming": nn.init.kaiming_uniform_,
"kaiming_normal": nn.init.kaiming_normal_, "he": nn.init.kaiming_normal_,
"orthogonal": nn.init.orthogonal_, "default": "use_default"}
return str_to_initialiser_converter
def create_dropout_layer(self):
"""Creates a dropout layer"""
return nn.Dropout(p=self.dropout)
def create_embedding_layers(self):
"""Creates the embedding layers in the network"""
embedding_layers = nn.ModuleList([])
for embedding_dimension in self.embedding_dimensions:
input_dim, output_dim = embedding_dimension
embedding_layers.extend([nn.Embedding(input_dim, output_dim)])
return embedding_layers
def initialise_parameters(self, parameters_list):
"""Initialises the list of parameters given"""
initialiser = self.str_to_initialiser_converter[self.initialiser.lower()]
if initialiser != "use_default":
for parameters in parameters_list:
if type(parameters) == nn.Linear or type(parameters) == nn.Conv2d:
initialiser(parameters.weight)
elif type(parameters) in [nn.LSTM, nn.RNN, nn.GRU]:
initialiser(parameters.weight_hh_l0)
initialiser(parameters.weight_ih_l0)
def flatten_tensor(self, tensor):
"""Flattens a tensor of shape (a, b, c, d, ...) into shape (a, b * c * d * .. )"""
return tensor.reshape(tensor.shape[0], -1)
def print_model_summary(self):
print(self)
================================================
FILE: nn_builder/pytorch/CNN.py
================================================
import torch
import torch.nn as nn
import numpy as np
from nn_builder.pytorch.Base_Network import Base_Network
class CNN(nn.Module, Base_Network):
"""Creates a PyTorch convolutional neural network
Args:
- input_dim: Tuple of integers to indicate the (channels, height, width) dimension of the input
- layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be
one of these 6 forms:
- ["conv", channels, kernel_size, stride, padding]
- ["maxpool", kernel_size, stride, padding]
- ["avgpool", kernel_size, stride, padding]
- ["adaptivemaxpool", output height, output width]
- ["adaptiveavgpool", output height, output width]
- ["linear", out]
- output_activation: String to indicate the activation function you want the output to go through. Provide a list of
strings if you want multiple output heads
- hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers
(not including the output layer). Default is ReLU.
- dropout: Float to indicate what dropout probability you want applied after each hidden layer
- initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch
initialisers are supported. PyTorch's default initialisation is the default.
- batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False
- y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the
output values to in regression tasks. Default is no range restriction
- random_seed: Integer to indicate the random seed you want to use
NOTE that this class' forward method expects input data in the form: (batch, channels, height, width)
"""
def __init__(self, input_dim, layers_info, output_activation=None,
hidden_activations="relu", dropout=0.0, initialiser="default", batch_norm=False,
y_range= (), random_seed=0, converted_from_tf_model=False):
nn.Module.__init__(self)
self.valid_cnn_hidden_layer_types = {'conv', 'maxpool', 'avgpool', 'adaptivemaxpool', 'adaptiveavgpool', 'linear'}
self.valid_layer_types_with_no_parameters = [nn.MaxPool2d, nn.AvgPool2d, nn.AdaptiveAvgPool2d, nn.AdaptiveMaxPool2d]
Base_Network.__init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser,
batch_norm, y_range, random_seed)
self.converted_from_tf_model = converted_from_tf_model
def flatten_tensor(self, tensor):
"""Flattens a tensor of shape (a, b, c, d, ...) into shape (a, b * c * d * .. )"""
if self.converted_from_tf_model:
tensor = tensor.permute(0, 2, 3, 1).contiguous()
tensor = tensor.view(tensor.size(0), -1)
else:
tensor = tensor.reshape(tensor.shape[0], -1)
return tensor
def check_all_user_inputs_valid(self):
"""Checks that all the user inputs were valid"""
self.check_CNN_input_dim_valid()
self.check_CNN_layers_valid()
self.check_activations_valid()
self.check_initialiser_valid()
self.check_y_range_values_valid()
def check_CNN_input_dim_valid(self):
"""Checks that the CNN input dim valid"""
error_msg = "input_dim must be a tuple of 3 integers indicating (channels, height, width)"
assert isinstance(self.input_dim, tuple), error_msg
for element in self.input_dim: assert isinstance(element, int) and element > 0
def check_CNN_layers_valid(self):
"""Checks that the user inputs for cnn_hidden_layers were valid. cnn_hidden_layers must be a list of layers where
each layer must be of one of these forms:
- ["conv", channels, kernel_size, stride, padding]
- ["maxpool", kernel_size, stride, padding]
- ["avgpool", kernel_size, stride, padding]
- ["adaptivemaxpool", output height, output width]
- ["adaptiveavgpool", output height, output width]
- ["linear", out]
"""
error_msg_layer_type = "First element in a layer specification must be one of {}".format(self.valid_cnn_hidden_layer_types)
error_msg_conv_layer = """Conv layer must be of form ['conv', channels, kernel_size, stride, padding] where the
final 4 elements are non-negative integers"""
error_msg_maxpool_layer = """Maxpool layer must be of form ['maxpool', kernel_size, stride, padding] where the
final 2 elements are non-negative integers"""
error_msg_avgpool_layer = """Avgpool layer must be of form ['avgpool', kernel_size, stride, padding] where the
final 2 elements are non-negative integers"""
error_msg_adaptivemaxpool_layer = """Adaptivemaxpool layer must be of form ['adaptivemaxpool', output height, output width]"""
error_msg_adaptiveavgpool_layer = """Adaptiveavgpool layer must be of form ['adaptiveavgpool', output height, output width]"""
error_msg_linear_layer = """Linear layer must be of form ['linear', out] where out is a non-negative integers"""
assert isinstance(self.layers_info, list), "layers must be a list"
all_layers = self.layers_info[:-1]
output_layer = self.layers_info[-1]
assert isinstance(output_layer, list), "layers must be a list"
if isinstance(output_layer[0], list):
assert len(output_layer) == len(
self.output_activation), "Number of output activations must equal number of output heads"
for layer in output_layer:
all_layers.append(layer)
assert layer[0].lower() == "linear", "Final layer must be linear"
else:
all_layers.append(output_layer)
assert isinstance(output_layer[0], str), error_msg_layer_type
assert output_layer[0].lower() == "linear", "Final layer must be linear"
for layer in all_layers:
assert isinstance(layer, list), "Each layer must be a list"
assert isinstance(layer[0], str), error_msg_layer_type
layer_type_name = layer[0].lower()
assert layer_type_name in self.valid_cnn_hidden_layer_types, "Layer name {} not valid, use one of {}".format(layer_type_name, self.valid_cnn_hidden_layer_types)
if layer_type_name == "conv":
assert len(layer) == 5, error_msg_conv_layer
for ix in range(3): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0, error_msg_conv_layer
assert isinstance(layer[4], int) and layer[4] >= 0, error_msg_conv_layer
elif layer_type_name == "maxpool":
assert len(layer) == 4, error_msg_maxpool_layer
for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_maxpool_layer
if layer[1] != layer[2]: print("NOTE that your maxpool kernel size {} isn't the same as your stride {}".format(layer[1], layer[2]))
assert isinstance(layer[3], int) and layer[3] >= 0, error_msg_conv_layer
elif layer_type_name == "avgpool":
assert len(layer) == 4, error_msg_avgpool_layer
for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_avgpool_layer
assert isinstance(layer[3], int) and layer[3] >= 0, error_msg_conv_layer
if layer[1] != layer[2]:print("NOTE that your avgpool kernel size {} isn't the same as your stride {}".format(layer[1], layer[2]))
elif layer_type_name == "adaptivemaxpool":
assert len(layer) == 3, error_msg_adaptivemaxpool_layer
for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_adaptivemaxpool_layer
elif layer_type_name == "adaptiveavgpool":
assert len(layer) == 3, error_msg_adaptiveavgpool_layer
for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[
ix + 1] > 0, error_msg_adaptiveavgpool_layer
elif layer_type_name == "linear":
assert len(layer) == 2, error_msg_linear_layer
for ix in range(1): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0
else:
raise ValueError("Invalid layer name")
rest_must_be_linear = False
for ix, layer in enumerate(all_layers):
if rest_must_be_linear: assert layer[0].lower() == "linear", "If have linear layers then they must come at end"
if layer[0].lower() == "linear":
rest_must_be_linear = True
def create_hidden_layers(self):
"""Creates the hidden layers in the network"""
cnn_hidden_layers = nn.ModuleList([])
input_dim = self.input_dim
for layer in self.layers_info[:-1]:
input_dim = self.create_and_append_layer(input_dim, layer, cnn_hidden_layers)
self.input_dim_into_final_layer = input_dim
return cnn_hidden_layers
def create_and_append_layer(self, input_dim, layer, list_to_append_layer_to):
"""Creates and appends a layer to the list provided"""
layer_name = layer[0].lower()
assert layer_name in self.valid_cnn_hidden_layer_types, "Layer name {} not valid, use one of {}".format(
layer_name, self.valid_cnn_hidden_layer_types)
if layer_name == "conv":
list_to_append_layer_to.extend([nn.Conv2d(in_channels=input_dim[0], out_channels=layer[1], kernel_size=layer[2],
stride=layer[3], padding=layer[4])])
elif layer_name == "maxpool":
list_to_append_layer_to.extend([nn.MaxPool2d(kernel_size=layer[1],
stride=layer[2], padding=layer[3])])
elif layer_name == "avgpool":
list_to_append_layer_to.extend([nn.AvgPool2d(kernel_size=layer[1],
stride=layer[2], padding=layer[3])])
elif layer_name == "adaptivemaxpool":
list_to_append_layer_to.extend([nn.AdaptiveMaxPool2d(output_size=(layer[1], layer[2]))])
elif layer_name == "adaptiveavgpool":
list_to_append_layer_to.extend([nn.AdaptiveAvgPool2d(output_size=(layer[1], layer[2]))])
elif layer_name == "linear":
if isinstance(input_dim, tuple): input_dim = np.prod(np.array(input_dim))
list_to_append_layer_to.extend([nn.Linear(in_features=input_dim, out_features=layer[1])])
else:
raise ValueError("Wrong layer name")
input_dim = self.calculate_new_dimensions(input_dim, layer)
return input_dim
def calculate_new_dimensions(self, input_dim, layer):
"""Calculates the new dimensions of the data after passing through a type of layer"""
layer_name = layer[0].lower()
if layer_name == "conv":
new_channels = layer[1]
kernel, stride, padding = layer[2], layer[3], layer[4]
new_height = int((input_dim[1] - kernel + 2*padding)/stride) + 1
new_width = int((input_dim[2] - kernel + 2 * padding) / stride) + 1
output_dim = (new_channels, new_height, new_width)
elif layer_name in ["maxpool", "avgpool"]:
new_channels = input_dim[0]
kernel, stride, padding = layer[1], layer[2], layer[3]
new_height = int((input_dim[1] - kernel + 2*padding)/stride) + 1
new_width = int((input_dim[2] - kernel + 2 * padding) / stride) + 1
output_dim = (new_channels, new_height, new_width)
elif layer_name in ["adaptivemaxpool", "adaptiveavgpool"]:
new_channels = input_dim[0]
output_dim = (new_channels, layer[1], layer[2])
elif layer_name == "linear":
output_dim = layer[1]
return output_dim
def create_output_layers(self):
"""Creates the output layers in the network"""
output_layers = nn.ModuleList([])
input_dim = self.input_dim_into_final_layer
if not isinstance(self.layers_info[-1][0], list) : self.layers_info[-1] = [self.layers_info[-1]]
for output_layer in self.layers_info[-1]:
self.create_and_append_layer(input_dim, output_layer, output_layers)
return output_layers
def initialise_all_parameters(self):
"""Initialises the parameters in the linear and embedding layers"""
initialisable_layers = [layer for layer in self.hidden_layers if not type(layer) in self.valid_layer_types_with_no_parameters]
self.initialise_parameters(nn.ModuleList(initialisable_layers))
output_initialisable_layers = [layer for layer in self.output_layers if
not type(layer) in self.valid_layer_types_with_no_parameters]
self.initialise_parameters(output_initialisable_layers)
def create_batch_norm_layers(self):
"""Creates the batch norm layers in the network"""
batch_norm_layers = nn.ModuleList([])
for layer in self.layers_info[:-1]:
layer_type = layer[0].lower()
if layer_type == "conv":
batch_norm_layers.extend([nn.BatchNorm2d(num_features=layer[1])])
elif layer_type == "linear":
batch_norm_layers.extend([nn.BatchNorm1d(num_features=layer[1])])
return batch_norm_layers
def forward(self, x):
"""Forward pass for the network. Note that it expects input data in the form (Batch, Channels, Height, Width)"""
if not self.checked_forward_input_data_once: self.check_input_data_into_forward_once(x)
x = self.process_hidden_layers(x)
out = self.process_output_layers(x)
if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*nn.Sigmoid()(out)
return out
def check_input_data_into_forward_once(self, x):
"""Checks the input data into forward is of the right format. Then sets a flag indicating that this has happened once
so that we don't keep checking as this would slow down the model too much"""
assert len(x.shape) == 4, "x should have the shape (batch_size, channel, height, width)"
assert x.shape[1:] == self.input_dim, "Input data must be of shape (channels, height, width) that you provided, not of shape {}".format(x.shape[1:])
self.checked_forward_input_data_once = True #So that it doesn't check again
def process_hidden_layers(self, x):
"""Puts the data x through all the hidden layers"""
flattened=False
valid_batch_norm_layer_ix = 0
for layer_ix, layer in enumerate(self.hidden_layers):
if type(layer) in self.valid_layer_types_with_no_parameters:
x = layer(x)
else:
if type(layer) == nn.Linear and not flattened:
x = self.flatten_tensor(x)
flattened = True
x = self.get_activation(self.hidden_activations, layer_ix)(layer(x))
if self.batch_norm:
x = self.batch_norm_layers[valid_batch_norm_layer_ix](x)
valid_batch_norm_layer_ix += 1
if self.dropout != 0.0: x = self.dropout_layer(x)
if not flattened: x = self.flatten_tensor(x)
return x
def process_output_layers(self, x):
"""Puts the data x through all the output layers"""
out = None
for output_layer_ix, output_layer in enumerate(self.output_layers):
activation = self.get_activation(self.output_activation, output_layer_ix)
temp_output = output_layer(x)
if activation is not None: temp_output = activation(temp_output)
if out is None: out = temp_output
else: out = torch.cat((out, temp_output), dim=1)
return out
================================================
FILE: nn_builder/pytorch/NN.py
================================================
import torch
import numpy as np
import torch.nn as nn
from nn_builder.pytorch.Base_Network import Base_Network
class NN(nn.Module, Base_Network):
"""Creates a PyTorch neural network
Args:
- input_dim: Integer to indicate the dimension of the input into the network
- layers_info: List of integers to indicate the width and number of linear layers you want in your network,
e.g. [5, 8, 1] would produce a network with 3 linear layers of width 5, 8 and then 1
- hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers
(not including the output layer). Default is ReLU.
- output_activation: String to indicate the activation function you want the output to go through. Provide a list of
strings if you want multiple output heads
- dropout: Float to indicate what dropout probability you want applied after each hidden layer
- initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch
initialisers are supported. PyTorch's default initialisation is the default.
- batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False
- columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer
before being fed through the other layers of the network. Default option is no embeddings
- embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then
you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1],
[embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings
- y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the
output values to in regression tasks. Default is no range restriction
- random_seed: Integer to indicate the random seed you want to use
"""
def __init__(self, input_dim, layers_info, output_activation=None,
hidden_activations="relu", dropout=0.0, initialiser="default", batch_norm=False,
columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (), random_seed=0):
nn.Module.__init__(self)
self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0
self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded
self.embedding_dimensions = embedding_dimensions
self.embedding_layers = self.create_embedding_layers()
Base_Network.__init__(self, input_dim, layers_info, output_activation, hidden_activations, dropout, initialiser,
batch_norm, y_range, random_seed)
def check_all_user_inputs_valid(self):
"""Checks that all the user inputs were valid"""
self.check_NN_input_dim_valid()
self.check_NN_layers_valid()
self.check_activations_valid()
self.check_embedding_dimensions_valid()
self.check_initialiser_valid()
self.check_y_range_values_valid()
def create_hidden_layers(self):
"""Creates the linear layers in the network"""
linear_layers = nn.ModuleList([])
input_dim = int(self.input_dim - len(self.embedding_dimensions) + np.sum([output_dims[1] for output_dims in self.embedding_dimensions]))
for hidden_unit in self.layers_info[:-1]:
linear_layers.extend([nn.Linear(input_dim, hidden_unit)])
input_dim = hidden_unit
return linear_layers
def create_output_layers(self):
"""Creates the output layers in the network"""
output_layers = nn.ModuleList([])
if len(self.layers_info) >= 2: input_dim = self.layers_info[-2]
else: input_dim = self.input_dim
if not isinstance(self.layers_info[-1], list): output_layer = [self.layers_info[-1]]
else: output_layer = self.layers_info[-1]
for output_dim in output_layer:
output_layers.extend([nn.Linear(input_dim, output_dim)])
return output_layers
def create_batch_norm_layers(self):
"""Creates the batch norm layers in the network"""
batch_norm_layers = nn.ModuleList([nn.BatchNorm1d(num_features=hidden_unit) for hidden_unit in self.layers_info[:-1]])
return batch_norm_layers
def initialise_all_parameters(self):
"""Initialises the parameters in the linear and embedding layers"""
self.initialise_parameters(self.hidden_layers)
self.initialise_parameters(self.output_layers)
self.initialise_parameters(self.embedding_layers)
def forward(self, x):
"""Forward pass for the network"""
if not self.checked_forward_input_data_once: self.check_input_data_into_forward_once(x)
if self.embedding_to_occur: x = self.incorporate_embeddings(x)
x = self.process_hidden_layers(x)
out = self.process_output_layers(x)
if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*nn.Sigmoid()(out)
return out
def check_input_data_into_forward_once(self, x):
"""Checks the input data into forward is of the right format. Then sets a flag indicating that this has happened once
so that we don't keep checking as this would slow down the model too much"""
for embedding_dim in self.columns_of_data_to_be_embedded:
data = x[:, embedding_dim]
data_long = data.long()
assert all(data_long >= 0), "All data to be embedded must be integers 0 and above -- {}".format(data_long)
assert torch.sum(abs(data.float() - data_long.float())) < 0.0001, """Data columns to be embedded should be integer
values 0 and above to represent the different
classes"""
if self.input_dim > len(self.columns_of_data_to_be_embedded):
assert isinstance(x, torch.FloatTensor) or isinstance(x, torch.cuda.FloatTensor), "Input data must be a float tensor"
assert len(x.shape) == 2, "X should be a 2-dimensional tensor: {}".format(x.shape)
self.checked_forward_input_data_once = True #So that it doesn't check again
def incorporate_embeddings(self, x):
"""Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready
to then be put through the linear layers"""
all_embedded_data = []
for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded):
data = x[:, embedding_var].long()
embedded_data = self.embedding_layers[embedding_layer_ix](data)
all_embedded_data.append(embedded_data)
all_embedded_data = torch.cat(tuple(all_embedded_data), dim=1)
x = torch.cat((x[:, [col for col in range(x.shape[1]) if col not in self.columns_of_data_to_be_embedded]].float(), all_embedded_data), dim=1)
return x
def process_hidden_layers(self, x):
"""Puts the data x through all the hidden layers"""
for layer_ix, linear_layer in enumerate(self.hidden_layers):
x = self.get_activation(self.hidden_activations, layer_ix)(linear_layer(x))
if self.batch_norm: x = self.batch_norm_layers[layer_ix](x)
if self.dropout != 0.0: x = self.dropout_layer(x)
return x
def process_output_layers(self, x):
"""Puts the data x through all the output layers"""
out = None
for output_layer_ix, output_layer in enumerate(self.output_layers):
activation = self.get_activation(self.output_activation, output_layer_ix)
temp_output = output_layer(x)
if activation is not None: temp_output = activation(temp_output)
if out is None: out = temp_output
else: out = torch.cat((out, temp_output), dim=1)
return out
================================================
FILE: nn_builder/pytorch/RNN.py
================================================
import torch
import torch.nn as nn
import numpy as np
from nn_builder.pytorch.Base_Network import Base_Network
class RNN(nn.Module, Base_Network):
"""Creates a PyTorch recurrent neural network
Args:
- input_dim: Integer to indicate the dimension of the input into the network
- layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be
one of these 3 forms:
- ["lstm", hidden_units]
- ["gru", hidden_units]
- ["linear", hidden_units]
- hidden_activations: String or list of string to indicate the activations you want used on the output of linear hidden layers
(not including the output layer). Default is ReLU.
- output_activation: String to indicate the activation function you want the output to go through. Provide a list of
strings if you want multiple output heads
- dropout: Float to indicate what dropout probability you want applied after each hidden layer
- initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch
initialisers are supported. PyTorch's default initialisation is the default.
- batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False
- columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer
before being fed through the other layers of the network. Default option is no embeddings
- embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then
you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1],
[embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings
- y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the
output values to in regression tasks. Default is no range restriction
- return_final_seq_only: Boolean to indicate whether you only want to return the output for the final timestep (True)
or if you want to return the output for all timesteps (False)
- random_seed: Integer to indicate the random seed you want to use
NOTE that this class' forward method expects input data in the form: (batch, sequence length, features)
"""
def __init__(self, input_dim, layers_info, output_activation=None,
hidden_activations="relu", dropout=0.0, initialiser="default", batch_norm=False,
columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (),
return_final_seq_only=True, random_seed=0):
nn.Module.__init__(self)
self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0
self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded
self.embedding_dimensions = embedding_dimensions
self.embedding_layers = self.create_embedding_layers()
self.return_final_seq_only = return_final_seq_only
self.valid_RNN_hidden_layer_types = {"linear", "gru", "lstm"}
Base_Network.__init__(self, input_dim, layers_info, output_activation,
hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed)
def check_all_user_inputs_valid(self):
"""Checks that all the user inputs were valid"""
self.check_NN_input_dim_valid()
self.check_RNN_layers_valid()
self.check_activations_valid()
self.check_embedding_dimensions_valid()
self.check_initialiser_valid()
self.check_y_range_values_valid()
self.check_return_final_seq_only_valid()
def check_RNN_layers_valid(self):
"""Checks that layers provided by user are valid"""
error_msg_layer_type = "First element in a layer specification must be one of {}".format(self.valid_RNN_hidden_layer_types)
error_msg_layer_form = "Layer must be of form [layer_name, hidden_units]"
error_msg_layer_list = "Layers must be provided as a list"
error_msg_output_heads = "Number of output activations must equal number of output heads"
assert isinstance(self.layers_info, list), error_msg_layer_list
all_layers = self.layers_info[:-1]
output_layer = self.layers_info[-1]
assert isinstance(output_layer, list), error_msg_layer_list
if isinstance(output_layer[0], list):
assert len(output_layer) == len(
self.output_activation), error_msg_output_heads
for layer in output_layer:
all_layers.append(layer)
else:
assert not isinstance(self.output_activation, list) or len(
self.output_activation) == 1, error_msg_output_heads
all_layers.append(output_layer)
rest_must_be_linear = False
for layer in all_layers:
assert isinstance(layer, list), "Each layer must be a list"
assert isinstance(layer[0], str), error_msg_layer_type
layer_type_name = layer[0].lower()
assert layer_type_name in self.valid_RNN_hidden_layer_types, "Layer name {} not valid, use one of {}".format(
layer_type_name, self.valid_RNN_hidden_layer_types)
assert isinstance(layer[1], int), error_msg_layer_form
assert layer[1] > 0, "Must have hidden_units >= 1"
assert len(layer) == 2, error_msg_layer_form
if rest_must_be_linear: assert layer[0].lower() == "linear", "If have linear layers then they must come at end"
if layer_type_name == "linear": rest_must_be_linear = True
def create_hidden_layers(self):
"""Creates the hidden layers in the network"""
RNN_hidden_layers = nn.ModuleList([])
input_dim = int(self.input_dim - len(self.embedding_dimensions) + np.sum(
[output_dims[1] for output_dims in self.embedding_dimensions]))
for layer in self.layers_info[:-1]:
input_dim = self.create_and_append_layer(input_dim, layer, RNN_hidden_layers)
self.input_dim_into_final_layer = input_dim
return RNN_hidden_layers
def create_and_append_layer(self, input_dim, layer, RNN_hidden_layers):
layer_type_name = layer[0].lower()
hidden_size = layer[1]
if layer_type_name == "lstm":
RNN_hidden_layers.extend([nn.LSTM(input_size=input_dim, hidden_size=hidden_size, batch_first=True)])
elif layer_type_name == "gru":
RNN_hidden_layers.extend(
[nn.GRU(input_size=input_dim, hidden_size=hidden_size, batch_first=True)])
elif layer_type_name == "linear":
RNN_hidden_layers.extend([nn.Linear(input_dim, hidden_size)])
else:
raise ValueError("Wrong layer names")
input_dim = hidden_size
return input_dim
def create_output_layers(self):
"""Creates the output layers in the network"""
output_layers = nn.ModuleList([])
input_dim = self.input_dim_into_final_layer
if not isinstance(self.layers_info[-1][0], list): self.layers_info[-1] = [self.layers_info[-1]]
for output_layer in self.layers_info[-1]:
self.create_and_append_layer(input_dim, output_layer, output_layers)
return output_layers
def initialise_all_parameters(self):
"""Initialises the parameters in the linear and embedding layers"""
self.initialise_parameters(self.hidden_layers)
self.initialise_parameters(self.output_layers)
self.initialise_parameters(self.embedding_layers)
def create_batch_norm_layers(self):
"""Creates the batch norm layers in the network"""
batch_norm_layers = nn.ModuleList([nn.BatchNorm1d(num_features=layer[1]) for layer in self.layers_info[:-1]])
return batch_norm_layers
def get_activation(self, activations, ix=None):
"""Gets the activation function"""
if isinstance(activations, list):
activation = self.str_to_activations_converter[str(activations[ix]).lower()]
else:
activation = self.str_to_activations_converter[str(activations).lower()]
return activation
def forward(self, x):
"""Forward pass for the network. Note that it expects input data in the form (batch, seq length, features)"""
if not self.checked_forward_input_data_once: self.check_input_data_into_forward_once(x)
batch_size, seq_length, data_dimension = x.shape
if self.embedding_to_occur: x = self.incorporate_embeddings(x, batch_size, seq_length)
x = self.process_hidden_layers(x, batch_size, seq_length)
out = self.process_output_layers(x, batch_size, seq_length)
if self.return_final_seq_only: out = out[:, -1, :]
if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*nn.Sigmoid()(out)
return out
def check_input_data_into_forward_once(self, x):
"""Checks the input data into forward is of the right format. Then sets a flag indicating that this has happened once
so that we don't keep checking as this would slow down the model too much"""
assert len(x.shape) == 3, "x should have the shape (batch_size, sequence_length, dimension)"
assert x.shape[2] == self.input_dim, "x must have the same dimension as the input_dim you provided"
for embedding_dim in self.columns_of_data_to_be_embedded:
data = x[:, :, embedding_dim]
data = data.contiguous().view(-1, 1)
data_long = data.long()
assert all(data_long >= 0), "All data to be embedded must be integers 0 and above -- {}".format(data_long)
assert torch.sum(abs(data.float() - data_long.float())) < 0.0001, """Data columns to be embedded should be integer
values 0 and above to represent the different
classes"""
if self.input_dim > len(self.columns_of_data_to_be_embedded):
assert isinstance(x, torch.FloatTensor) or isinstance(x, torch.cuda.FloatTensor), "Input data must be a float tensor"
self.checked_forward_input_data_once = True #So that it doesn't check again
def incorporate_embeddings(self, x, batch_size, seq_length):
"""Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready
to then be put through the hidden layers"""
all_embedded_data = []
for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded):
data = x[:, :, embedding_var].long()
data = data.contiguous().view(batch_size * seq_length, -1)
embedded_data = self.embedding_layers[embedding_layer_ix](data)
embedded_data = embedded_data.view(batch_size, seq_length, -1)
all_embedded_data.append(embedded_data)
all_embedded_data = torch.cat(tuple(all_embedded_data), dim=2)
non_embedded_data = x[:, :, [col for col in range(x.shape[2]) if col not in self.columns_of_data_to_be_embedded]].float()
x = torch.cat((non_embedded_data, all_embedded_data), dim=2)
return x
def process_hidden_layers(self, x, batch_size, seq_length):
"""Puts the data x through all the hidden layers"""
for layer_ix, layer in enumerate(self.hidden_layers):
if type(layer) == nn.Linear:
x = x.contiguous().view(batch_size * seq_length, -1)
activation = self.get_activation(self.hidden_activations, layer_ix)
x = activation(layer(x))
x = x.view(batch_size, seq_length, layer.out_features)
else:
x = layer(x)
x = x[0] #because we only want to keep the output and not the hidden states
if self.batch_norm:
x.transpose_(1, 2)
x = self.batch_norm_layers[layer_ix](x)
x.transpose_(1, 2)
if self.dropout != 0.0: x = self.dropout_layer(x)
return x
def process_output_layers(self, x, batch_size, seq_length):
"""Puts the data x through all the output layers"""
out = None
for output_layer_ix, output_layer in enumerate(self.output_layers):
activation = self.get_activation(self.output_activation, output_layer_ix)
if type(output_layer) == nn.Linear:
x = x.contiguous().view(batch_size * seq_length, -1)
temp_output = output_layer(x)
if activation is not None:
temp_output = activation(temp_output)
temp_output = temp_output.view(batch_size, seq_length, -1)
x = x.view(batch_size, seq_length, -1)
else:
temp_output = output_layer(x)
temp_output = temp_output[0]
if activation is not None:
if type(activation) == nn.Softmax:
temp_output = temp_output.contiguous().view(batch_size * seq_length, -1)
temp_output = activation(temp_output)
temp_output = temp_output.view(batch_size, seq_length, -1)
else:
temp_output = activation(temp_output)
if out is None: out = temp_output
else: out = torch.cat((out, temp_output), dim=2)
return out
================================================
FILE: nn_builder/pytorch/__init__.py
================================================
================================================
FILE: nn_builder/tensorflow/Base_Network.py
================================================
from tensorflow.keras.layers import BatchNormalization
from nn_builder.Overall_Base_Network import Overall_Base_Network
import tensorflow.keras.activations as activations
import tensorflow.keras.initializers as initializers
import numpy as np
import random
import tensorflow as tf
from abc import ABC, abstractmethod
class Base_Network(Overall_Base_Network, ABC):
"""Base class for TensorFlow neural network classes"""
def __init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser, batch_norm, y_range,
random_seed, input_dim):
if input_dim is not None: print("You don't need to provide input_dim for a tensorflow network")
super().__init__(None, layers_info, output_activation,
hidden_activations, dropout, initialiser, batch_norm, y_range, random_seed)
@abstractmethod
def call(self, x, training=True):
"""Runs a forward pass of the tensorflow model"""
raise NotImplementedError
@abstractmethod
def create_and_append_layer(self, layer, list_to_append_layer_to, activation, output_layer=False):
"""Creates a layer and appends it to the provided list"""
raise NotImplementedError
def set_all_random_seeds(self, random_seed):
"""Sets all possible random seeds so results can be reproduced"""
np.random.seed(random_seed)
tf.random.set_seed(random_seed)
random.seed(random_seed)
def create_str_to_activations_converter(self):
"""Creates a dictionary which converts strings to activations"""
str_to_activations_converter = {"elu": activations.elu, "exponential": activations.exponential,
"hard_sigmoid": activations.hard_sigmoid, "linear": activations.linear,
"relu": activations.relu, "selu": activations.selu, "sigmoid": activations.sigmoid,
"softmax": activations.softmax, "softplus": activations.softplus,
"softsign": activations.softsign, "tanh": activations.tanh, "none": activations.linear}
return str_to_activations_converter
def create_str_to_initialiser_converter(self):
"""Creates a dictionary which converts strings to initialiser"""
str_to_initialiser_converter = {"glorot_normal": initializers.glorot_normal, "glorot_uniform": initializers.glorot_uniform,
"xavier_normal": initializers.glorot_normal, "xavier_uniform": initializers.glorot_uniform,
"xavier": initializers.glorot_uniform, "he_normal": initializers.he_normal(),
"he_uniform": initializers.he_uniform(), "lecun_normal": initializers.lecun_normal(),
"lecun_uniform": initializers.lecun_uniform(), "truncated_normal": initializers.TruncatedNormal,
"variance_scaling": initializers.VarianceScaling, "default": initializers.glorot_uniform}
return str_to_initialiser_converter
def create_dropout_layer(self):
"""Creates a dropout layer"""
return tf.keras.layers.Dropout(rate=self.dropout)
def create_hidden_layers(self):
"""Creates the hidden layers in the network"""
hidden_layers = []
for layer_ix, layer in enumerate(self.layers_info[:-1]):
activation = self.get_activation(self.hidden_activations, layer_ix)
self.create_and_append_layer(layer, hidden_layers, activation, output_layer=False)
return hidden_layers
def create_output_layers(self):
"""Creates the output layers in the network"""
output_layers = []
network_type = type(self).__name__
if network_type in ["CNN", "RNN"]:
if not isinstance(self.layers_info[-1][0], list): self.layers_info[-1] = [self.layers_info[-1]]
elif network_type == "NN":
if isinstance(self.layers_info[-1], int): self.layers_info[-1] = [self.layers_info[-1]]
else:
raise ValueError("Network type not recognised")
for output_layer_ix, output_layer in enumerate(self.layers_info[-1]):
activation = self.get_activation(self.output_activation, output_layer_ix)
self.create_and_append_layer(output_layer, output_layers, activation, output_layer=True)
return output_layers
def create_embedding_layers(self):
"""Creates the embedding layers in the network"""
embedding_layers = []
for embedding_dimension in self.embedding_dimensions:
input_dim, output_dim = embedding_dimension
embedding_layers.extend([tf.keras.layers.Embedding(input_dim, output_dim)])
return embedding_layers
def create_batch_norm_layers(self):
"""Creates the batch norm layers in the network"""
batch_norm_layers = []
for layer in self.layers_info[:-1]:
batch_norm_layers.extend([BatchNormalization()])
return batch_norm_layers
def print_model_summary(self, input_shape=None):
assert input_shape is not None, "Must provide the input_shape parameter as a tuple"
self.build(input_shape=input_shape)
self.dropout_layer.build(input_shape)
self.summary()
================================================
FILE: nn_builder/tensorflow/CNN.py
================================================
import numpy as np
from tensorflow.keras import Model, activations
from tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, BatchNormalization, MaxPool2D, AveragePooling2D
from nn_builder.tensorflow.Base_Network import Base_Network
import tensorflow as tf
class CNN(Model, Base_Network):
"""Creates a PyTorch convolutional neural network
Args:
- layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be
one of these 4 forms:
- ["conv", channels, kernel_size, stride, padding]
- ["maxpool", kernel_size, stride, padding]
- ["avgpool", kernel_size, stride, padding]
- ["linear", out]
where all variables are integers except for padding which must be either "same" or "valid"
- output_activation: String to indicate the activation function you want the output to go through. Provide a list of
strings if you want multiple output heads
- hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers
(not including the output layer). Default is ReLU.
- dropout: Float to indicate what dropout probability you want applied after each hidden layer
- initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch
initialisers are supported. PyTorch's default initialisation is the default.
- batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False
- y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the
output values to in regression tasks. Default is no range restriction
- random_seed: Integer to indicate the random seed you want to use
NOTE that this class' call method expects input data in the form: (batch, channels, height, width)
"""
def __init__(self, layers_info, output_activation=None, hidden_activations="relu", dropout= 0.0, initialiser="default",
batch_norm=False, y_range=(), random_seed=0, input_dim=None):
Model.__init__(self)
self.valid_cnn_hidden_layer_types = {'conv', 'maxpool', 'avgpool', 'linear'}
self.valid_layer_types_with_no_parameters = (MaxPool2D, AveragePooling2D)
Base_Network.__init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser,
batch_norm, y_range, random_seed, input_dim)
def check_all_user_inputs_valid(self):
"""Checks that all the user inputs were valid"""
self.check_CNN_layers_valid()
self.check_activations_valid()
self.check_initialiser_valid()
self.check_y_range_values_valid()
def check_CNN_layers_valid(self):
"""Checks that the user inputs for cnn_hidden_layers were valid. cnn_hidden_layers must be a list of layers where
each layer must be of one of these forms:
- ["conv", channels, kernel_size, stride, padding]
- ["maxpool", kernel_size, stride, padding]
- ["avgpool", kernel_size, stride, padding]
- ["linear", out]
where all variables must be integers except padding which must be "valid" or "same"
"""
error_msg_layer_type = "First element in a layer specification must be one of {}".format(self.valid_cnn_hidden_layer_types)
error_msg_conv_layer = """Conv layer must be of form ['conv', channels, kernel_size, stride, padding] where the
variables are all non-negative integers except padding which must be either "valid" or "same"""
error_msg_maxpool_layer = """Maxpool layer must be of form ['maxpool', kernel_size, stride, padding] where the
variables are all non-negative integers except padding which must be either "valid" or "same"""
error_msg_avgpool_layer = """Avgpool layer must be of form ['avgpool', kernel_size, stride, padding] where the
variables are all non-negative integers except padding which must be either "valid" or "same"""
error_msg_linear_layer = """Linear layer must be of form ['linear', out] where out is a non-negative integers"""
assert isinstance(self.layers_info, list), "layers must be a list"
all_layers = self.layers_info[:-1]
output_layer = self.layers_info[-1]
assert isinstance(output_layer, list), "layers must be a list"
if isinstance(output_layer[0], list):
assert len(output_layer) == len(self.output_activation), "Number of output activations must equal number of output heads"
for layer in output_layer:
all_layers.append(layer)
assert isinstance(layer[0], str), error_msg_layer_type
assert layer[0].lower() == "linear", "Final layer must be linear"
else:
all_layers.append(output_layer)
assert isinstance(output_layer[0], str), error_msg_layer_type
assert output_layer[0].lower() == "linear", "Final layer must be linear"
for layer in all_layers:
assert isinstance(layer, list), "Each layer must be a list"
assert isinstance(layer[0], str), error_msg_layer_type
layer_type_name = layer[0].lower()
assert layer_type_name in self.valid_cnn_hidden_layer_types, "Layer name {} not valid, use one of {}".format(layer_type_name, self.valid_cnn_hidden_layer_types)
if layer_type_name == "conv":
assert len(layer) == 5, error_msg_conv_layer
for ix in range(3): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0, error_msg_conv_layer
assert isinstance(layer[4], str) and layer[4].lower() in ["valid", "same"], error_msg_conv_layer
elif layer_type_name == "maxpool":
assert len(layer) == 4, error_msg_maxpool_layer
for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_maxpool_layer
if layer[1] != layer[2]: print("NOTE that your maxpool kernel size {} isn't the same as your stride {}".format(layer[1], layer[2]))
assert isinstance(layer[3], str) and layer[3].lower() in ["valid", "same"], error_msg_maxpool_layer
elif layer_type_name == "avgpool":
assert len(layer) == 4, error_msg_avgpool_layer
for ix in range(2): assert isinstance(layer[ix + 1], int) and layer[ix + 1] > 0, error_msg_avgpool_layer
assert isinstance(layer[3], str) and layer[3].lower() in ["valid", "same"], error_msg_avgpool_layer
if layer[1] != layer[2]:print("NOTE that your avgpool kernel size {} isn't the same as your stride {}".format(layer[1], layer[2]))
elif layer_type_name == "linear":
assert len(layer) == 2, error_msg_linear_layer
for ix in range(1): assert isinstance(layer[ix+1], int) and layer[ix+1] > 0
else:
raise ValueError("Invalid layer name")
rest_must_be_linear = False
for ix, layer in enumerate(all_layers):
if rest_must_be_linear: assert layer[0].lower() == "linear", "If have linear layers then they must come at end"
if layer[0].lower() == "linear":
rest_must_be_linear = True
def create_and_append_layer(self, layer, list_to_append_layer_to, activation=None, output_layer=False):
"""Creates and appends a layer to the list provided"""
layer_name = layer[0].lower()
assert layer_name in self.valid_cnn_hidden_layer_types, "Layer name {} not valid, use one of {}".format(
layer_name, self.valid_cnn_hidden_layer_types)
if layer_name == "conv":
list_to_append_layer_to.extend([Conv2D(filters=layer[1], kernel_size=layer[2],
strides=layer[3], padding=layer[4], activation=activation,
kernel_initializer=self.initialiser_function)])
elif layer_name == "maxpool":
list_to_append_layer_to.extend([MaxPool2D(pool_size=(layer[1], layer[1]),
strides=(layer[2], layer[2]), padding=layer[3])])
elif layer_name == "avgpool":
list_to_append_layer_to.extend([AveragePooling2D(pool_size=(layer[1], layer[1]),
strides=(layer[2], layer[2]), padding=layer[3])])
elif layer_name == "linear":
list_to_append_layer_to.extend([Dense(layer[1], activation=activation, kernel_initializer=self.initialiser_function)])
else:
raise ValueError("Wrong layer name")
def create_batch_norm_layers(self):
"""Creates the batch norm layers in the network"""
batch_norm_layers = []
for layer in self.layers_info[:-1]:
layer_type = layer[0].lower()
if layer_type in ["conv", "linear"]:
batch_norm_layers.extend([BatchNormalization()])
return batch_norm_layers
def call(self, x, training=True):
"""Forward pass for the network. Note that it expects input data in the form (Batch, Height, Width, Channels)"""
x = self.process_hidden_layers(x, training)
out = self.process_output_layers(x)
if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0]) * activations.sigmoid(out)
return out
def process_hidden_layers(self, x, training):
"""Puts the data x through all the hidden layers"""
flattened=False
training = training or training is None
valid_batch_norm_layer_ix = 0
for layer_ix, layer in enumerate(self.hidden_layers):
if type(layer) in self.valid_layer_types_with_no_parameters:
x = layer(x)
else:
if type(layer) == Dense and not flattened:
x = Flatten()(x)
flattened = True
x = layer(x)
if self.batch_norm:
x = self.batch_norm_layers[valid_batch_norm_layer_ix](x, training=False)
valid_batch_norm_layer_ix += 1
if self.dropout != 0.0 and training: x = self.dropout_layer(x)
if not flattened: x = Flatten()(x)
return x
def process_output_layers(self, x):
"""Puts the data x through all the output layers"""
out = None
for output_layer_ix, output_layer in enumerate(self.output_layers):
temp_output = output_layer(x)
if out is None: out = temp_output
else: out = Concatenate(axis=1)([out, temp_output])
return out
================================================
FILE: nn_builder/tensorflow/NN.py
================================================
import tensorflow as tf
from tensorflow.keras import Model, activations
import numpy as np
from tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, BatchNormalization
from nn_builder.tensorflow.Base_Network import Base_Network
class NN(Model, Base_Network):
"""Creates a PyTorch neural network
Args:
- layers_info: List of integers to indicate the width and number of linear layers you want in your network
- hidden_activations: String or list of string to indicate the activations you want used on the output of hidden layers
(not including the output layer). Default is ReLU.
- output_activation: String to indicate the activation function you want the output to go through. Provide a list of
strings if you want multiple output heads
- dropout: Float to indicate what dropout probability you want applied after each hidden layer
- initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch
initialisers are supported. PyTorch's default initialisation is the default.
- batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False
- columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer
before being fed through the other layers of the network. Default option is no embeddings
- embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then
you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1],
[embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings
- y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the
output values to in regression tasks. Default is no range restriction
- random_seed: Integer to indicate the random seed you want to use
"""
def __init__(self, layers_info, output_activation=None, hidden_activations="relu", dropout=0.0, initialiser="default",
batch_norm=False, columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (), random_seed=0,
input_dim=None):
Model.__init__(self)
self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0
self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded
self.embedding_dimensions = embedding_dimensions
self.embedding_layers = self.create_embedding_layers()
Base_Network.__init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser,
batch_norm, y_range, random_seed, input_dim)
def check_all_user_inputs_valid(self):
"""Checks that all the user inputs were valid"""
self.check_NN_layers_valid()
self.check_activations_valid()
self.check_embedding_dimensions_valid()
self.check_initialiser_valid()
self.check_y_range_values_valid()
def create_and_append_layer(self, layer, list_to_append_layer_to, activation=None, output_layer=False):
"""Creates and appends a layer to the list provided"""
list_to_append_layer_to.extend([Dense(layer, activation=activation, kernel_initializer=self.initialiser_function)])
def call(self, x, training=True):
if self.embedding_to_occur: x = self.incorporate_embeddings(x)
x = self.process_hidden_layers(x, training)
out = self.process_output_layers(x)
if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0])*activations.sigmoid(out)
return out
def incorporate_embeddings(self, x):
"""Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready
to then be put through the hidden layers"""
all_embedded_data = []
for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded):
data = x[:, embedding_var]
embedded_data = self.embedding_layers[embedding_layer_ix](data)
all_embedded_data.append(embedded_data)
if len(all_embedded_data) > 1: all_embedded_data = Concatenate(axis=1)(all_embedded_data)
else: all_embedded_data = all_embedded_data[0]
non_embedded_columns = [col for col in range(x.shape[1]) if col not in self.columns_of_data_to_be_embedded]
if len(non_embedded_columns) > 0:
x = tf.gather(x, non_embedded_columns, axis=1)
x = Concatenate(axis=1)([tf.dtypes.cast(x, float), all_embedded_data])
else: x = all_embedded_data
return x
def process_hidden_layers(self, x, training):
"""Puts the data x through all the hidden layers"""
for layer_ix, linear_layer in enumerate(self.hidden_layers):
x = linear_layer(x)
if self.batch_norm: x = self.batch_norm_layers[layer_ix](x, training=False)
if self.dropout != 0.0 and (training or training is None):
x = self.dropout_layer(x)
return x
def process_output_layers(self, x):
"""Puts the data x through all the output layers"""
out = None
for output_layer_ix, output_layer in enumerate(self.output_layers):
temp_output = output_layer(x)
if out is None: out = temp_output
else:
out = Concatenate(axis=1)(inputs=[out, temp_output])
return out
================================================
FILE: nn_builder/tensorflow/RNN.py
================================================
import numpy as np
import tensorflow as tf
from tensorflow.keras import Model, activations
from tensorflow.keras.layers import Dense, Concatenate, GRU, LSTM
from nn_builder.tensorflow.Base_Network import Base_Network
class RNN(Model, Base_Network):
"""Creates a TensorFlow recurrent neural network
Args:
- layers_info: List of layer specifications to specify the hidden layers of the network. Each element of the list must be
one of these 3 forms:
- ["lstm", hidden_units]
- ["gru", hidden_units]
- ["linear", hidden_units]
- hidden_activations: String or list of string to indicate the activations you want used on the output of linear hidden layers
(not including the output layer). Default is ReLU.
- output_activation: String to indicate the activation function you want the output to go through. Provide a list of
strings if you want multiple output heads
- dropout: Float to indicate what dropout probability you want applied after each hidden layer
- initialiser: String to indicate which initialiser you want used to initialise all the parameters. All PyTorch
initialisers are supported. PyTorch's default initialisation is the default.
- batch_norm: Boolean to indicate whether you want batch norm applied to the output of every hidden layer. Default is False
- columns_of_data_to_be_embedded: List to indicate the columns numbers of the data that you want to be put through an embedding layer
before being fed through the other layers of the network. Default option is no embeddings
- embedding_dimensions: If you have categorical variables you want embedded before flowing through the network then
you specify the embedding dimensions here with a list like so: [ [embedding_input_dim_1, embedding_output_dim_1],
[embedding_input_dim_2, embedding_output_dim_2] ...]. Default is no embeddings
- y_range: Tuple of float or integers of the form (y_lower, y_upper) indicating the range you want to restrict the
output values to in regression tasks. Default is no range restriction
- return_final_seq_only: Boolean to indicate whether you only want to return the output for the final timestep (True)
or if you want to return the output for all timesteps (False)
- random_seed: Integer to indicate the random seed you want to use
NOTE that this class' call method expects input data in the form: (batch, sequence length, features)
"""
def __init__(self, layers_info, output_activation=None, hidden_activations="relu", dropout=0.0, initialiser="default",
batch_norm=False, columns_of_data_to_be_embedded=[], embedding_dimensions=[], y_range= (),
return_final_seq_only=True, random_seed=0, input_dim=None):
Model.__init__(self)
self.embedding_to_occur = len(columns_of_data_to_be_embedded) > 0
self.columns_of_data_to_be_embedded = columns_of_data_to_be_embedded
self.embedding_dimensions = embedding_dimensions
self.embedding_layers = self.create_embedding_layers()
self.return_final_seq_only = return_final_seq_only
self.valid_RNN_hidden_layer_types = {"linear", "gru", "lstm"}
Base_Network.__init__(self, layers_info, output_activation, hidden_activations, dropout, initialiser,
batch_norm, y_range, random_seed, input_dim)
def check_all_user_inputs_valid(self):
"""Checks that all the user inputs were valid"""
self.check_RNN_layers_valid()
self.check_activations_valid()
self.check_embedding_dimensions_valid()
self.check_initialiser_valid()
self.check_y_range_values_valid()
self.check_return_final_seq_only_valid()
def check_RNN_layers_valid(self):
"""Checks that layers provided by user are valid"""
error_msg_layer_type = "First element in a layer specification must be one of {}".format(self.valid_RNN_hidden_layer_types)
error_msg_layer_form = "Layer must be of form [layer_name, hidden_units]"
error_msg_layer_list = "Layers must be provided as a list"
error_msg_output_heads = "Number of output activations must equal number of output heads"
assert isinstance(self.layers_info, list), error_msg_layer_list
all_layers = self.layers_info[:-1]
output_layer = self.layers_info[-1]
assert isinstance(output_layer, list), error_msg_layer_list
if isinstance(output_layer[0], list):
assert len(output_layer) == len(
self.output_activation), error_msg_output_heads
for layer in output_layer:
all_layers.append(layer)
else:
assert not isinstance(self.output_activation, list) or len(self.output_activation) == 1, error_msg_output_heads
all_layers.append(output_layer)
rest_must_be_linear = False
for layer in all_layers:
assert isinstance(layer, list), "Each layer must be a list"
assert isinstance(layer[0], str), error_msg_layer_type
layer_type_name = layer[0].lower()
assert layer_type_name in self.valid_RNN_hidden_layer_types, "Layer name {} not valid, use one of {}".format(
layer_type_name, self.valid_RNN_hidden_layer_types)
assert isinstance(layer[1], int), error_msg_layer_form
assert layer[1] > 0, "Must have hidden_units >= 1"
assert len(layer) == 2, error_msg_layer_form
if rest_must_be_linear: assert layer[0].lower() == "linear", "If have linear layers then they must come at end"
if layer_type_name == "linear": rest_must_be_linear = True
def create_and_append_layer(self, layer, rnn_hidden_layers, activation, output_layer=False):
layer_type_name = layer[0].lower()
hidden_size = layer[1]
if output_layer and self.return_final_seq_only: return_sequences = False
else: return_sequences = True
if layer_type_name == "lstm":
rnn_hidden_layers.extend([LSTM(units=hidden_size, kernel_initializer=self.initialiser_function,
return_sequences=return_sequences)])
elif layer_type_name == "gru":
rnn_hidden_layers.extend([GRU(units=hidden_size, kernel_initializer=self.initialiser_function,
return_sequences=return_sequences)])
elif layer_type_name == "linear":
rnn_hidden_layers.extend(
[Dense(units=hidden_size, activation=activation, kernel_initializer=self.initialiser_function)])
else:
raise ValueError("Wrong layer names")
input_dim = hidden_size
return input_dim
def call(self, x, training=True):
"""Forward pass for the network. Note that it expects input data in the form (batch, seq length, features)"""
if self.embedding_to_occur: x = self.incorporate_embeddings(x)
training = training or training is None
x, restricted_to_final_seq = self.process_hidden_layers(x, training)
out = self.process_output_layers(x, restricted_to_final_seq)
if self.y_range: out = self.y_range[0] + (self.y_range[1] - self.y_range[0]) * activations.sigmoid(out)
return out
def incorporate_embeddings(self, x):
"""Puts relevant data through embedding layers and then concatenates the result with the rest of the data ready
to then be put through the hidden layers"""
all_embedded_data = []
for embedding_layer_ix, embedding_var in enumerate(self.columns_of_data_to_be_embedded):
data = x[:, :, embedding_var]
embedded_data = self.embedding_layers[embedding_layer_ix](data)
all_embedded_data.append(embedded_data)
if len(all_embedded_data) > 1: all_embedded_data = Concatenate(axis=2)(all_embedded_data)
else: all_embedded_data = all_embedded_data[0]
non_embedded_columns = [col for col in range(x.shape[2]) if col not in self.columns_of_data_to_be_embedded]
if len(non_embedded_columns) > 0:
x = tf.gather(x, non_embedded_columns, axis=2)
x = Concatenate(axis=2)([tf.dtypes.cast(x, float), all_embedded_data])
else: x = all_embedded_data
return x
def process_hidden_layers(self, x, training):
"""Puts the data x through all the hidden layers"""
restricted_to_final_seq = False
for layer_ix, layer in enumerate(self.hidden_layers):
if type(layer) == Dense:
if self.return_final_seq_only and not restricted_to_final_seq:
x = x[:, -1, :]
restricted_to_final_seq = True
x = layer(x)
else:
x = layer(x)
if self.batch_norm:
x = self.batch_norm_layers[layer_ix](x, training=False)
if self.dropout != 0.0 and training: x = self.dropout_layer(x)
return x, restricted_to_final_seq
def process_output_layers(self, x, restricted_to_final_seq):
"""Puts the data x through all the output layers"""
out = None
for output_layer_ix, output_layer in enumerate(self.output_layers):
if type(output_layer) == Dense:
if self.return_final_seq_only and not restricted_to_final_seq:
x = x[:, -1, :]
restricted_to_final_seq = True
temp_output = output_layer(x)
else:
temp_output = output_layer(x)
activation = self.get_activation(self.output_activation, output_layer_ix)
temp_output = activation(temp_output)
if out is None: out = temp_output
else:
if restricted_to_final_seq: dim = 1
else: dim = 2
out = Concatenate(axis=dim)([out, temp_output])
return out
================================================
FILE: nn_builder/tensorflow/__init__.py
================================================
================================================
FILE: requirements.txt
================================================
tensorflow==2.0.0a0
torch==1.0.1.post2
torchvision==0.2.2.post3
numpy==1.16.2
setuptools==40.8.0
pytest==4.4.0
================================================
FILE: setup.py
================================================
import setuptools
import sys
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="nn_builder",
version="1.0.5",
author="Petros Christodoulou",
author_email="p.christodoulou2@gmail.com",
description="Build neural networks in 1 line",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/p-christ/nn_builder",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
install_requires=["tensorflow==2.0.0a0" if ( sys.platform.startswith("mac") or sys.platform.startswith("darwin")) else "tensorflow-gpu==2.0.0a0"]
)
================================================
FILE: tests/pytorch_tests/test_pytorch_CNN.py
================================================
# Run from home directory with python -m pytest tests
import shutil
import pytest
import torch
import random
import numpy as np
import torch.nn as nn
from nn_builder.pytorch.CNN import CNN
import torch.optim as optim
from torchvision import datasets, transforms
N = 250
X = torch.randn((N, 1, 5, 5))
X[0:125, 0, 3, 3] += 20.0
y = X[:, 0, 3, 3] > 5.0
y = y.float()
def test_user_hidden_layers_input_rejections():
"""Tests whether network rejects invalid hidden_layers inputted from user"""
inputs_that_should_fail = [['maxpool', 33, 22, 33], [['a']], [[222, 222, 222, 222]], [["conv", 2, 2, -1]], [["conv", 2, 2]], [["conv", 2, 2, 55, 999, 33]],
[["maxpool", 33, 33]], [["maxpool", -1, 33]], [["maxpool", 33]], [["maxpoolX", 1, 33]],
[["cosnv", 2, 2]], [["avgpool", 33, 33, 333, 99]], [["avgpool", -1, 33]], [["avgpool", 33]], [["avgpoolX", 1, 33]],
[["adaptivemaxpool", 33, 33, 333, 33]], [["adaptivemaxpool", 2]], [["adaptivemaxpool", 33]], [["adaptivemaxpoolX"]],
[["adaptiveavgpool", 33, 33, 333, 11]], [["adaptiveavgpool", 2]], [["adaptiveavgpool", 33]],
[["adaptiveavgpoolX"]], [["linear", 40, -2]], [["lineafr", 40, 2]]]
for input in inputs_that_should_fail:
print(input)
with pytest.raises(AssertionError):
CNN(input_dim=1, layers_info=input, hidden_activations="relu",
output_activation="relu")
def test_user_hidden_layers_input_acceptances():
"""Tests whether network rejects invalid hidden_layers inputted from user"""
inputs_that_should_work = [[["conv", 2, 2, 3331, 2]], [["CONV", 2, 2, 3331, 22]], [["ConV", 2, 2, 3331, 1]],
[["maxpool", 2, 2, 3331]], [["MAXPOOL", 2, 2, 3331]], [["MaXpOOL", 2, 2, 3331]],
[["avgpool", 2, 2, 3331]], [["AVGPOOL", 2, 2, 3331]], [["avGpOOL", 2, 2, 3331]],
[["adaptiveavgpool", 3, 22]], [["ADAPTIVEAVGpOOL", 1, 33]], [["ADAPTIVEaVGPOOL", 3, 6]],
[["adaptivemaxpool", 4, 66]], [["ADAPTIVEMAXpOOL", 2, 2]], [["ADAPTIVEmaXPOOL", 3, 3]],
[["adaptivemaxpool", 3, 3]], [["ADAPTIVEMAXpOOL", 3, 1]], [["linear", 40]], [["lineaR", 2]],
[["LINEAR", 2]]]
for ix, input in enumerate(inputs_that_should_work):
input.append(["linear", 5])
CNN(input_dim=(1, 1, 1), layers_info=input, hidden_activations="relu",
output_activation="relu")
def test_hidden_layers_created_correctly():
"""Tests that create_hidden_layers works correctly"""
layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34],
["adaptiveavgpool", 23, 44], ["linear", 22], ["linear", 2222], ["linear", 5]]
cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations="relu",
output_activation="relu")
assert type(cnn.hidden_layers[0]) == nn.Conv2d
assert cnn.hidden_layers[0].in_channels == 3
assert cnn.hidden_layers[0].out_channels == 2
assert cnn.hidden_layers[0].kernel_size == (4, 4)
assert cnn.hidden_layers[0].stride == (3, 3)
assert cnn.hidden_layers[0].padding == (2, 2)
assert type(cnn.hidden_layers[1]) == nn.MaxPool2d
assert cnn.hidden_layers[1].kernel_size == 3
assert cnn.hidden_layers[1].stride == 4
assert cnn.hidden_layers[1].padding == 2
assert type(cnn.hidden_layers[2]) == nn.AvgPool2d
assert cnn.hidden_layers[2].kernel_size == 32
assert cnn.hidden_layers[2].stride == 42
assert cnn.hidden_layers[2].padding == 22
assert type(cnn.hidden_layers[3]) == nn.AdaptiveMaxPool2d
assert cnn.hidden_layers[3].output_size == (3, 34)
assert type(cnn.hidden_layers[4]) == nn.AdaptiveAvgPool2d
assert cnn.hidden_layers[4].output_size == (23, 44)
assert type(cnn.hidden_layers[5]) == nn.Linear
assert cnn.hidden_layers[5].in_features == 2024
assert cnn.hidden_layers[5].out_features == 22
assert type(cnn.hidden_layers[6]) == nn.Linear
assert cnn.hidden_layers[6].in_features == 22
assert cnn.hidden_layers[6].out_features == 2222
assert type(cnn.output_layers[0]) == nn.Linear
assert cnn.output_layers[0].in_features == 2222
assert cnn.output_layers[0].out_features == 5
def test_output_layers_created_correctly():
"""Tests that create_output_layers works correctly"""
layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34],
["adaptiveavgpool", 23, 44], ["linear", 22], ["linear", 2222], ["linear", 2]]
cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations="relu", output_activation="relu")
assert cnn.output_layers[0].in_features == 2222
assert cnn.output_layers[0].out_features == 2
layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34],
["adaptiveavgpool", 23, 44], ["linear", 7]]
cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations="relu",
output_activation="relu")
assert cnn.output_layers[0].in_features == 23 * 44 * 2
assert cnn.output_layers[0].out_features == 7
layers = [["conv", 5, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34], ["linear", 6]]
cnn = CNN(input_dim=(3, 10, 10), layers_info=layers, hidden_activations="relu",
output_activation="relu")
assert cnn.output_layers[0].in_features == 3 * 34 * 5
assert cnn.output_layers[0].out_features == 6
layers = [["conv", 5, 4, 3, 2], ["maxpool", 3, 4, 2], ["avgpool", 32, 42, 22], ["adaptivemaxpool", 3, 34],
[["linear", 6], ["linear", 22]]]
cnn = CNN(input_dim=(3, 1000, 1000), layers_info=layers, hidden_activations="relu",
output_activation=["softmax", None])
assert cnn.output_layers[0].in_features == 3 * 34 * 5
assert cnn.output_layers[0].out_features == 6
assert cnn.output_layers[1].in_features == 3 * 34 * 5
assert cnn.output_layers[1].out_features == 22
def test_output_dim_user_input():
"""Tests whether network rejects an invalid output_dim input from user"""
inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
CNN(input_dim=(2, 10, 10), layers_info=[2, input_value], hidden_activations="relu", output_activation="relu")
with pytest.raises(AssertionError):
CNN(input_dim=(2, 10, 10), layers_info=input_value, hidden_activations="relu", output_activation="relu")
def test_activations_user_input():
"""Tests whether network rejects an invalid hidden_activations or output_activation from user"""
inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
CNN(input_dim=(2, 10, 10), layers_info=[["conv", 2, 2, 3331, 2]], hidden_activations=input_value,
output_activation="relu")
CNN(input_dim=(2, 10, 10), layers_info=[["conv", 2, 2, 3331, 2]], hidden_activations="relu",
output_activation=input_value)
def test_initialiser_user_input():
"""Tests whether network rejects an invalid initialiser from user"""
inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
CNN(input_dim=(2, 10, 10), layers_info=[["conv", 2, 2, 3331, 2]], hidden_activations="relu",
output_activation="relu", initialiser=input_value)
CNN(layers_info=[["conv", 2, 2, 3331, 2], ["linear", 3]], hidden_activations="relu",
output_activation="relu", initialiser="xavier", input_dim=(2, 10, 10))
def test_batch_norm_layers():
"""Tests whether batch_norm_layers method works correctly"""
layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["adaptivemaxpool", 3, 34], ["linear", 5]]
cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10),
output_activation="relu", initialiser="xavier", batch_norm=False)
layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["adaptivemaxpool", 3, 34], ["linear", 5]]
cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10),
output_activation="relu", initialiser="xavier", batch_norm=True)
assert len(cnn.batch_norm_layers) == 1
assert cnn.batch_norm_layers[0].num_features == 2
layers = [["conv", 2, 4, 3, 2], ["maxpool", 3, 4, 2], ["conv", 12, 4, 3, 2], ["adaptivemaxpool", 3, 34], ["linear", 22], ["linear", 55]]
cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10),
output_activation="relu", initialiser="xavier", batch_norm=True)
assert len(cnn.batch_norm_layers) == 3
assert cnn.batch_norm_layers[0].num_features == 2
assert cnn.batch_norm_layers[1].num_features == 12
assert cnn.batch_norm_layers[2].num_features == 22
def test_linear_layers_acceptance():
"""Tests that only accepts linear layers of correct shape"""
layers_that_shouldnt_work = [[["linear", 2, 5]], [["linear", 2, 5, 5]], [["linear"]], [["linear", 2], ["linear", 5, 4]],
["linear", 0], ["linear", -5]]
for layers in layers_that_shouldnt_work:
with pytest.raises(AssertionError):
cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10),
output_activation="relu", initialiser="xavier", batch_norm=True)
layers_that_should_work = [[["linear", 44], ["linear", 2]], [["linear", 22]]]
for layer in layers_that_should_work:
assert CNN(layers_info=layer, hidden_activations="relu", input_dim=(2, 10, 10),
output_activation="relu", initialiser="xavier", batch_norm=True)
def test_linear_layers_only_come_at_end():
"""Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they
don't only come at the end"""
layers = [["conv", 2, 4, 3, 2], ["linear", 55], ["maxpool", 3, 4, 2], ["adaptivemaxpool", 3, 34]]
with pytest.raises(AssertionError):
cnn = CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10),
output_activation="relu", initialiser="xavier", batch_norm=True)
layers = [["conv", 2, 4, 3, 2], ["linear", 55]]
assert CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10),
output_activation="relu", initialiser="xavier", batch_norm=True)
layers = [["conv", 2, 4, 3, 2], ["linear", 55], ["linear", 55], ["linear", 55]]
assert CNN(layers_info=layers, hidden_activations="relu", input_dim=(2, 10, 10),
output_activation="relu", initialiser="xavier", batch_norm=True)
def test_output_activation():
"""Tests whether network outputs data that has gone through correct activation function"""
RANDOM_ITERATIONS = 20
input_dim = (5, 100, 100)
for _ in range(RANDOM_ITERATIONS):
data = torch.randn((1, *input_dim))
CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 50]],
hidden_activations="relu", input_dim=input_dim,
output_activation="relu", initialiser="xavier")
out = CNN_instance.forward(data)
assert all(out.squeeze() >= 0)
CNN_instance = CNN(layers_info=[["conv", 2, 20, 1, 0], ["linear", 5]],
hidden_activations="relu", input_dim=input_dim,
output_activation="relu", initialiser="xavier")
out = CNN_instance.forward(data)
assert all(out.squeeze() >= 0)
CNN_instance = CNN(layers_info=[["conv", 5, 20, 1, 0], ["linear", 5]],
hidden_activations="relu", input_dim=input_dim,
output_activation="relu", initialiser="xavier")
out = CNN_instance.forward(data)
assert all(out.squeeze() >= 0)
CNN_instance = CNN(layers_info=[["conv", 5, 20, 1, 0], ["linear", 22]],
hidden_activations="relu", input_dim=input_dim,
output_activation="sigmoid", initialiser="xavier")
out = CNN_instance.forward(data)
assert all(out.squeeze() >= 0)
assert all(out.squeeze() <= 1)
assert round(torch.sum(out.squeeze()).item(), 3) != 1.0
CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 5]],
hidden_activations="relu", input_dim=input_dim,
output_activation="softmax", initialiser="xavier")
out = CNN_instance.forward(data)
assert all(out.squeeze() >= 0)
assert all(out.squeeze() <= 1)
assert round(torch.sum(out.squeeze()).item(), 3) == 1.0
CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 5]],
hidden_activations="relu", input_dim=input_dim,
initialiser="xavier")
out = CNN_instance.forward(data)
assert not all(out.squeeze() >= 0)
assert not round(torch.sum(out.squeeze()).item(), 3) == 1.0
def test_y_range():
"""Tests whether setting a y range works correctly"""
for _ in range(100):
val1 = random.random() - 3.0*random.random()
val2 = random.random() + 2.0*random.random()
lower_bound = min(val1, val2)
upper_bound = max(val1, val2)
CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 5]],
hidden_activations="relu", y_range=(lower_bound, upper_bound),
initialiser="xavier", input_dim=(1, 20, 20))
random_data = torch.randn((10, 1, 20, 20))
out = CNN_instance.forward(random_data)
assert torch.sum(out > lower_bound).item() == 10*5, "lower {} vs. {} ".format(lower_bound, out)
assert torch.sum(out < upper_bound).item() == 10*5, "upper {} vs. {} ".format(upper_bound, out)
def test_deals_with_None_activation():
"""Tests whether is able to handle user inputting None as output activation"""
assert CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 5]],
hidden_activations="relu", output_activation=None,
initialiser="xavier", input_dim=(5, 5, 5))
def test_check_input_data_into_forward_once():
"""Tests that check_input_data_into_forward_once method only runs once"""
CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2], ["linear", 6]],
hidden_activations="relu", input_dim=(4, 2, 5),
output_activation="relu", initialiser="xavier")
data_not_to_throw_error = torch.randn((1, 4, 2, 5))
data_to_throw_error = torch.randn((1, 2, 20, 20))
with pytest.raises(AssertionError):
CNN_instance.forward(data_to_throw_error)
with pytest.raises(RuntimeError):
CNN_instance.forward(data_not_to_throw_error)
CNN_instance.forward(data_to_throw_error)
def test_y_range_user_input():
"""Tests whether network rejects invalid y_range inputs"""
invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)]
for y_range_value in invalid_y_range_inputs:
with pytest.raises(AssertionError):
print(y_range_value)
CNN_instance = CNN(layers_info=[["conv", 2, 2, 1, 2], ["adaptivemaxpool", 2, 2]],
hidden_activations="relu", y_range=y_range_value, input_dim=(2, 2, 2),
initialiser="xavier")
def test_model_trains():
"""Tests whether a small range of networks can solve a simple task"""
for output_activation in ["sigmoid", "None"]:
CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]], input_dim=(1, 5, 5),
hidden_activations="relu", output_activation=output_activation,
initialiser="xavier")
assert solves_simple_problem(X, y, CNN_instance)
z = X[:, 0:1, 3:4, 3:4] > 5.0
z = torch.cat([z ==1, z==0], dim=1).float()
z = z.squeeze(-1).squeeze(-1)
CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["linear", 2]], input_dim=(1, 5, 5),
hidden_activations="relu", output_activation="softmax", dropout=0.01,
initialiser="xavier")
assert solves_simple_problem(X, z, CNN_instance)
CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["linear", 1]], input_dim=(1, 5, 5),
hidden_activations="relu", output_activation=None,
initialiser="xavier")
assert solves_simple_problem(X, y, CNN_instance)
CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["linear", 1]], input_dim=(1, 5, 5),
hidden_activations="relu", output_activation=None,
initialiser="xavier", batch_norm=True)
assert solves_simple_problem(X, y, CNN_instance)
CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["maxpool", 1, 1, 0], ["linear", 1]], input_dim=(1, 5, 5),
hidden_activations="relu", output_activation=None,
initialiser="xavier")
assert solves_simple_problem(X, y, CNN_instance)
CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["avgpool", 1, 1, 0], ["linear", 1]], input_dim=(1, 5, 5),
hidden_activations="relu", output_activation=None,
initialiser="xavier")
assert solves_simple_problem(X, y, CNN_instance)
CNN_instance = CNN(layers_info=[["conv", 5, 3, 1, 0], ["adaptivemaxpool", 2, 2], ["linear", 1]], input_dim=(1, 5, 5),
hidden_activations="relu", output_activation=None,
initialiser="xavier")
assert solves_simple_problem(X, y, CNN_instance)
CNN_instance = CNN(layers_info=[["conv", 5, 3, 1, 0], ["adaptiveavgpool", 2, 2], ["linear", 1]], input_dim=(1, 5, 5),
hidden_activations="relu", output_activation=None,
initialiser="xavier")
assert solves_simple_problem(X, y, CNN_instance)
def test_model_trains_linear_layer():
"""Tests whether a small range of networks can solve a simple task"""
CNN_instance = CNN(layers_info=[["conv", 5, 3, 1, 0], ["linear", 5], ["linear", 5], ["linear", 1]],
input_dim=(1, 5, 5),
hidden_activations="relu", output_activation="sigmoid",
initialiser="xavier")
assert solves_simple_problem(X, y, CNN_instance)
CNN_instance = CNN(layers_info=[["linear", 5], ["linear", 5], ["linear", 1]], input_dim=(1, 5, 5),
hidden_activations="relu", output_activation="sigmoid",
initialiser="xavier")
assert solves_simple_problem(X, y, CNN_instance)
def test_max_pool_working():
"""Tests whether max pool layers work properly"""
N = 250
X = torch.randn((N, 1, 8, 8))
X[0:125, 0, 3, 3] = 999.99
CNN_instance = CNN(layers_info=[["maxpool", 2, 2, 0], ["maxpool", 2, 2, 0], ["maxpool", 2, 2, 0], ["linear", 1]],
hidden_activations="relu", input_dim=(1, 8, 8),
initialiser="xavier")
assert CNN_instance(X).shape == (N, 1)
def test_dropout():
"""Tests whether dropout layer reads in probability correctly"""
CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]],
hidden_activations="relu", output_activation="sigmoid", dropout=0.9999,
initialiser="xavier", input_dim=(1, 5, 5))
assert CNN_instance.dropout_layer.p == 0.9999
assert not solves_simple_problem(X, y, CNN_instance)
CNN_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]],
hidden_activations="relu", output_activation=None, dropout=0.0000001,
initialiser="xavier", input_dim=(1, 5, 5))
assert CNN_instance.dropout_layer.p == 0.0000001
assert solves_simple_problem(X, y, CNN_instance)
def solves_simple_problem(X, y, nn_instance):
"""Checks if a given network is able to solve a simple problem"""
optimizer = optim.Adam(nn_instance.parameters(), lr=0.15)
for ix in range(800):
out = nn_instance.forward(X)
loss = torch.sum((out.squeeze() - y) ** 2) / N
optimizer.zero_grad()
loss.backward()
optimizer.step()
print("LOSS ", loss)
return loss < 0.1
def test_MNIST_progress():
"""Tests that network made using CNN module can make progress on MNIST"""
batch_size = 128
train_loader = torch.utils.data.DataLoader(
datasets.MNIST(root="input/", train=True, download=True,
transform=transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))
])),
batch_size=batch_size, shuffle=True)
for batch_norm in [True, False]:
cnn = CNN(layers_info=[["conv", 20, 5, 1, 0],
["maxpool", 2, 2, 0],
["conv", 50, 5, 1, 0],
["maxpool", 2, 2, 0],
["linear", 500], ["linear", 10]], hidden_activations="relu",
output_activation="softmax", initialiser="xavier", input_dim=(1, 28, 28), batch_norm=batch_norm)
loss_fn = nn.CrossEntropyLoss()
optimizer = optim.Adam(cnn.parameters(), lr=0.001)
ix = 0
accuracies = []
for data, target in train_loader:
ix += 1
output = cnn(data)
loss = loss_fn(output, target)
optimizer.zero_grad()
loss.backward()
optimizer.step()
pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability
correct = pred.eq(target.view_as(pred)).sum().item()
print("Accuracy {}".format(correct / batch_size))
accuracies.append(correct / batch_size)
if ix > 200:
break
assert accuracies[-1] > 0.7, "Accuracy not good enough {}".format(accuracies[-1])
shutil.rmtree("input/", ignore_errors=False, onerror=None)
def test_all_activations_work():
"""Tests that all activations get accepted"""
nn_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]],
dropout=0.0000001,
initialiser="xavier", input_dim=(1, 5, 5))
for key in nn_instance.str_to_activations_converter.keys():
if key == "none": hidden_key = "relu"
else: hidden_key = key
model = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]],
hidden_activations=hidden_key, output_activation=key, dropout=0.0000001,
initialiser="xavier", input_dim=(1, 5, 5))
model(X)
def test_all_initialisers_work():
"""Tests that all initialisers get accepted"""
nn_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]],
dropout=0.0000001,
initialiser="xavier", input_dim=(1, 5, 5))
for key in nn_instance.str_to_initialiser_converter.keys():
if key != "eye":
model = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]],
dropout=0.0000001,
initialiser=key, input_dim=(1, 5, 5))
model(X)
def test_print_model_summary():
nn_instance = CNN(layers_info=[["conv", 25, 5, 1, 0], ["adaptivemaxpool", 1, 1], ["linear", 1]],
dropout=0.0000001,
initialiser="xavier", input_dim=(1, 5, 5))
nn_instance.print_model_summary()
def test_output_heads_error_catching():
"""Tests that having multiple output heads catches errors from user inputs"""
output_dims_that_should_break = [["linear", 2, 2, "SAME", "conv", 3, 4, "SAME"], [[["conv", 3, 2, "same"], ["linear", 4]]],
[[2, 8]], [-33, 33, 33, 33, 33]]
for output_dim in output_dims_that_should_break:
with pytest.raises(AssertionError):
CNN(input_dim=(12, 12, 3), layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1], output_dim],
hidden_activations="relu", output_activation="relu")
output_activations_that_should_break = ["relu", ["relu"], ["relu", "softmax"]]
for output_activation in output_activations_that_should_break:
with pytest.raises(AssertionError):
CNN(input_dim=(12, 12, 3), layers_info=[["conv", 25, 5, 1, "valid"], ["conv", 25, 5, 1, "valid"], ["linear", 1],
[["linear", 4], ["linear", 10], ["linear", 4]]],
hidden_activations="relu", output_activation=output_activation)
def test_output_head_layers():
"""Tests whether the output head layers get created properly"""
for output_dim in [[["linear", 3],["linear", 9]], [["linear", 4], ["linear", 20]], [["linear", 1], ["linear", 1]]]:
nn_instance = CNN(input_dim=(12, 12, 3), layers_info=[["conv", 25, 5, 1, 2], ["conv", 25, 5, 1, 3], ["linear", 5], output_dim],
hidden_activations="relu", output_activation=["softmax", None])
assert nn_instance.output_layers[0].out_features == output_dim[0][1]
assert nn_instance.output_layers[0].in_features == 5
assert nn_instance.output_layers[1].out_features == output_dim[1][1]
assert nn_instance.output_layers[1].in_features == 5
def test_output_head_activations_work():
"""Tests that output head activations work properly"""
output_dim = [["linear", 5], ["linear", 10], ["linear", 3]]
nn_instance = CNN(input_dim=(12, 12, 3), layers_info=[["conv", 3, 2, 1, 1], ["conv", 3, 1, 1, 1], ["linear", 1], output_dim],
hidden_activations="relu", output_activation=["softmax", None, "relu"])
x = torch.randn((20, 12, 12, 3)) * -20.0
out = nn_instance(x)
assert out.shape == (20, 18)
sums = torch.sum(out[:, :5], dim=1).detach().numpy()
sums_others = torch.sum(out[:, 5:], dim=1).detach().numpy()
sums_others_2 = torch.sum(out[:, 5:15], dim=1).detach().numpy()
sums_others_3 = torch.sum(out[:, 15:18], dim=1).detach().numpy()
for row in range(out.shape[0]):
assert np.round(sums[row], 4) == 1.0, sums[row]
assert not np.round(sums_others[row], 4) == 1.0, sums_others[row]
assert not np.round(sums_others_2[row], 4) == 1.0, sums_others_2[row]
assert not np.round(sums_others_3[row], 4) == 1.0, sums_others_3[row]
for col in range(3):
assert out[row, 15 + col] >= 0.0, out[row, 15 + col]
def test_output_head_shapes_correct():
"""Tests that the output shape of network is correct when using multiple outpout heads"""
N = 20
X = torch.randn((N, 25, 25, 2))
for _ in range(25):
nn_instance = CNN(input_dim=(25, 25, 2),
layers_info=[["conv", 25, 2, 1, 1], ["conv", 2, 5, 1, 1], ["linear", 1], ["linear", 12]],
hidden_activations="relu")
out = nn_instance(X)
assert out.shape[0] == N
assert out.shape[1] == 12
for output_dim in [[ ["linear", 10], ["linear", 4], ["linear", 6]], [["linear", 3], ["linear", 8], ["linear", 9]]]:
nn_instance = CNN(input_dim=(25, 25, 2),
layers_info=[["conv", 25, 1, 1, 2], ["conv", 25, 5, 1, 1], ["linear", 1], output_dim],
hidden_activations="relu", output_activation=["softmax", None, "relu"])
out = nn_instance(X)
assert out.shape[0] == N
assert out.shape[1] == 20
================================================
FILE: tests/pytorch_tests/test_pytorch_NN.py
================================================
# Run from home directory with python -m pytest tests
import copy
import shutil
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
import pytest
import torch
import random
import numpy as np
import torch.nn as nn
import torch.optim as optim
from nn_builder.pytorch.NN import NN
from sklearn.utils import shuffle
N = 250
X = torch.randn((N, 5))
X[:, [2, 4]] += 10.0
y = X[:, 0] > 0
y = y.float()
def test_linear_hidden_units_user_input():
"""Tests whether network rejects an invalid linear_hidden_units input from user"""
inputs_that_should_fail = ["a", ["a", "b"], [2, 4, "ss"], [-2], 2]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
NN(input_dim=2, layers_info=input_value, hidden_activations="relu", output_activation="relu")
def test_input_dim_output_dim_user_input():
"""Tests whether network rejects an invalid input_dim from user"""
inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
NN(input_dim=input_value, layers_info=[2], hidden_activations="relu", output_activation="relu")
def test_activations_user_input():
"""Tests whether network rejects an invalid hidden_activations or output_activation from user"""
inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
NN(input_dim=2, layers_info=[2], hidden_activations=input_value,
output_activation="relu")
NN(input_dim=2, layers_info=[2], hidden_activations="relu",
output_activation=input_value)
def test_initialiser_user_input():
"""Tests whether network rejects an invalid initialiser from user"""
inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
NN(input_dim=2, layers_info=[2], hidden_activations="relu",
output_activation="relu", initialiser=input_value)
NN(input_dim=2, layers_info=[2], hidden_activations="relu",
output_activation="relu", initialiser="xavier")
def test_output_shape_correct():
"""Tests whether network returns output of the right shape"""
input_dims = [x for x in range(1, 3)]
output_dims = [x for x in range(4, 6)]
linear_hidden_units_options = [ [2, 3, 4], [2, 9, 1], [55, 55, 55, 234, 15]]
for input_dim, output_dim, linear_hidden_units in zip(input_dims, output_dims, linear_hidden_units_options):
linear_hidden_units.append(output_dim)
nn_instance = NN(input_dim=input_dim, layers_info=linear_hidden_units, hidden_activations="relu",
output_activation="relu", initialiser="xavier")
data = torch.randn((25, input_dim))
output = nn_instance.forward(data)
assert output.shape == (25, output_dim)
def test_output_activation():
"""Tests whether network outputs data that has gone through correct activation function"""
RANDOM_ITERATIONS = 20
for _ in range(RANDOM_ITERATIONS):
data = torch.randn((1, 100))
nn_instance = NN(input_dim=100, layers_info=[5, 5, 5],
hidden_activations="relu",
output_activation="relu", initialiser="xavier")
out = nn_instance.forward(data)
assert all(out.squeeze() >= 0)
nn_instance = NN(input_dim=100, layers_info=[5, 5, 5],
hidden_activations="relu",
output_activation="sigmoid", initialiser="xavier")
out = nn_instance.forward(data)
assert all(out.squeeze() >= 0)
assert all(out.squeeze() <= 1)
nn_instance = NN(input_dim=100, layers_info=[5, 5, 5],
hidden_activations="relu",
output_activation="softmax", initialiser="xavier")
out = nn_instance.forward(data)
assert all(out.squeeze() >= 0)
assert all(out.squeeze() <= 1)
assert round(torch.sum(out.squeeze()).item(), 3) == 1.0
nn_instance = NN(input_dim=100, layers_info=[5, 5, 5],
hidden_activations="relu",
)
out = nn_instance.forward(data)
assert not all(out.squeeze() >= 0)
assert not round(torch.sum(out.squeeze()).item(), 3) == 1.0
def test_linear_layers():
"""Tests whether create_hidden_layers method works correctly"""
for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]):
hidden_units.append(output_dim)
nn_instance = NN(input_dim=input_dim, layers_info=hidden_units,
hidden_activations="relu",
output_activation="relu", initialiser="xavier")
for layer in nn_instance.hidden_layers:
assert isinstance(layer, nn.Linear)
assert nn_instance.hidden_layers[0].in_features == input_dim
assert nn_instance.hidden_layers[0].out_features == hidden_units[0]
assert nn_instance.hidden_layers[1].in_features == hidden_units[0]
assert nn_instance.hidden_layers[1].out_features == hidden_units[1]
assert nn_instance.hidden_layers[2].in_features == hidden_units[1]
assert nn_instance.hidden_layers[2].out_features == hidden_units[2]
assert len(nn_instance.hidden_layers) == 3
def test_embedding_layers():
"""Tests whether create_embedding_layers method works correctly"""
for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)):
nn_instance = NN(input_dim=5, layers_info=[5],
embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]])
for layer in nn_instance.embedding_layers:
assert isinstance(layer, nn.Embedding)
assert len(nn_instance.embedding_layers) == 2
assert nn_instance.embedding_layers[0].num_embeddings == embedding_in_dim_1
assert nn_instance.embedding_layers[0].embedding_dim == embedding_out_dim_1
assert nn_instance.embedding_layers[1].num_embeddings == embedding_in_dim_2
assert nn_instance.embedding_layers[1].embedding_dim == embedding_out_dim_2
def test_non_integer_embeddings_rejected():
"""Tests whether an error is raised if user tries to provide non-integer data to be embedded"""
with pytest.raises(AssertionError):
nn_instance = NN(input_dim=5, layers_info=[5],
columns_of_data_to_be_embedded=[2, 4],
embedding_dimensions=[[50, 3],
[55, 4]])
out = nn_instance.forward(X)
def test_incorporate_embeddings():
"""Tests the method incorporate_embeddings"""
X_new = X
X_new[:, [2, 4]] = torch.round(X_new[:, [2, 4]])
nn_instance = NN(input_dim=5, layers_info=[5],
columns_of_data_to_be_embedded=[2, 4],
embedding_dimensions=[[50, 3],
[55, 4]])
out = nn_instance.incorporate_embeddings(X)
assert out.shape == (N, X.shape[1]+3+4-2)
def test_embedding_network_can_solve_simple_problem():
"""Tests whether network can solve simple problem using embeddings"""
X = torch.randn(N, 2) * 5.0 + 20.0
y = (X[:, 0] >= 20) * (X[:, 1] <= 20)
X = X.long()
nn_instance = NN(input_dim=2, layers_info=[5, 1],
columns_of_data_to_be_embedded=[0, 1],
embedding_dimensions=[[50, 3],
[55, 3]])
assert solves_simple_problem(X, y.float(), nn_instance)
def test_batch_norm_layers():
"""Tests whether batch_norm_layers method works correctly"""
for input_dim, output_dim, hidden_units in zip( range(5, 8), range(9, 12), [[2, 9, 2], [3, 5, 6], [9, 12, 2]]):
hidden_units.append(output_dim)
nn_instance = NN(input_dim=input_dim, layers_info=hidden_units,
hidden_activations="relu", batch_norm=True,
output_activation="relu", initialiser="xavier")
for layer in nn_instance.batch_norm_layers:
assert isinstance(layer, nn.BatchNorm1d)
assert len(nn_instance.batch_norm_layers) == len(hidden_units) - 1
assert nn_instance.batch_norm_layers[0].num_features == hidden_units[0]
assert nn_instance.batch_norm_layers[1].num_features == hidden_units[1]
assert nn_instance.batch_norm_layers[2].num_features == hidden_units[2]
def test_model_trains():
"""Tests whether a small range of networks can solve a simple task"""
for output_activation in ["sigmoid", "None"]:
nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 10, 1],
output_activation=output_activation, dropout=0.01, batch_norm=True)
assert solves_simple_problem(X, y, nn_instance)
z = X[:, 0:1] > 0
z = torch.cat([z ==1, z==0], dim=1).float()
nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 10, 2],
output_activation="softmax", dropout=0.01, batch_norm=True)
assert solves_simple_problem(X, z, nn_instance)
def solves_simple_problem(X, y, nn_instance):
"""Checks if a given network is able to solve a simple problem"""
optimizer = optim.Adam(nn_instance.parameters(), lr=0.15)
for ix in range(800):
out = nn_instance.forward(X)
loss = torch.sum((out.squeeze() - y) ** 2) / N
optimizer.zero_grad()
loss.backward()
optimizer.step()
return loss < 0.1
def test_dropout():
"""Tests whether dropout layer reads in probability correctly"""
nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999)
assert nn_instance.dropout_layer.p == 0.9999
assert not solves_simple_problem(X, y, nn_instance)
nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.00001)
assert solves_simple_problem(X, y, nn_instance)
def test_y_range_user_input():
"""Tests whether network rejects invalid y_range inputs"""
invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)]
for y_range_value in invalid_y_range_inputs:
with pytest.raises(AssertionError):
print(y_range_value)
nn_instance = NN(input_dim=5, layers_info=[10, 10, 3],
y_range=y_range_value)
def test_y_range():
"""Tests whether setting a y range works correctly"""
for _ in range(100):
val1 = random.random() - 3.0*random.random()
val2 = random.random() + 2.0*random.random()
lower_bound = min(val1, val2)
upper_bound = max(val1, val2)
nn_instance = NN(input_dim=5, layers_info=[10, 10, 3], y_range=(lower_bound, upper_bound))
random_data = torch.randn(15, 5)
out = nn_instance.forward(random_data)
assert torch.sum(out > lower_bound).item() == 3*15, "lower {} vs. {} ".format(lower_bound, out)
assert torch.sum(out < upper_bound).item() == 3*15, "upper {} vs. {} ".format(upper_bound, out)
def test_deals_with_None_activation():
"""Tests whether is able to handle user inputting None as output activation"""
assert NN(input_dim=5, layers_info=[10, 10, 3], output_activation=None)
def test_check_input_data_into_forward_once():
"""Tests that check_input_data_into_forward_once method only runs once"""
data_to_throw_error = torch.randn(N, 2)
X = torch.randn(N, 2) * 5.0 + 20.0
y = (X[:, 0] >= 20) * (X[:, 1] <= 20)
X = X.long()
nn_instance = NN(input_dim=2, layers_info=[5, 1],
columns_of_data_to_be_embedded=[0, 1],
embedding_dimensions=[[50, 3],
[55, 3]])
with pytest.raises(AssertionError):
nn_instance.forward(data_to_throw_error)
with pytest.raises(RuntimeError):
nn_instance.forward(X)
nn_instance.forward(data_to_throw_error)
def test_all_activations_work():
"""Tests that all activations get accepted"""
nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999)
for key in nn_instance.str_to_activations_converter.keys():
if key == "none": hidden_key = "relu"
else: hidden_key = key
model = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999, hidden_activations=hidden_key, output_activation=key)
model(X)
def test_all_initialisers_work():
"""Tests that all initialisers get accepted"""
nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999)
for key in nn_instance.str_to_initialiser_converter.keys():
model = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999, initialiser=key)
model(X)
def test_print_model_summary():
nn_instance = NN(input_dim=X.shape[1], layers_info=[10, 10, 1], dropout=0.9999)
nn_instance.print_model_summary()
def test_output_heads_error_catching():
"""Tests that having multiple output heads catches errors from user inputs"""
output_dims_that_should_break = [[[2, 8]], [-33, 33, 33, 33, 33]]
for output_dim in output_dims_that_should_break:
with pytest.raises(AssertionError):
NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations="relu",
output_activation="relu")
output_activations_that_should_break = ["relu", ["relu"], ["relu", "softmax"]]
for output_activation in output_activations_that_should_break:
with pytest.raises(AssertionError):
NN(input_dim=2, layers_info=[4, 7, 9, [4, 6, 1]], hidden_activations="relu",
output_activation=output_activation)
def test_output_head_layers():
"""Tests whether the output head layers get created properly"""
for output_dim in [[3, 9], [4, 20], [1, 1]]:
nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations="relu",
output_activation=["softmax", None])
assert nn_instance.output_layers[0].in_features == 9
assert nn_instance.output_layers[1].in_features == 9
assert nn_instance.output_layers[0].out_features == output_dim[0]
assert nn_instance.output_layers[1].out_features == output_dim[1]
def test_output_head_activations_work():
"""Tests that output head activations work properly"""
nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, [5, 10, 3]], hidden_activations="relu",
output_activation=["softmax", None, "relu"])
x = torch.randn((20, 2)) * -20.0
out = nn_instance.forward(x)
assert torch.allclose(torch.sum(out[:, :5], dim=1), torch.Tensor([1.0]))
assert not torch.allclose(torch.sum(out[:, 5:], dim=1), torch.Tensor([1.0]))
for row in range(out.shape[0]):
assert all(out[row, -3:] >= 0)
def test_output_head_shapes_correct():
"""Tests that the output shape of network is correct when using multiple outpout heads"""
N = 20
X = torch.randn((N, 2))
for _ in range(25):
output_dim = random.randint(1, 100)
nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations="relu")
out = nn_instance(X)
assert out.shape[0] == N
assert out.shape[1] == output_dim
for output_dim in [[3, 9, 5, 3], [5, 5, 5, 5], [2, 1, 1, 16]]:
nn_instance = NN(input_dim=2, layers_info=[4, 7, 9, output_dim], hidden_activations="relu",
output_activation=["softmax", None, None, "relu"])
out = nn_instance(X)
assert out.shape[0] == N
assert out.shape[1] == 20
def train_on_boston_housing(model, X, y):
# Compile the model
loss_fn = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.08)
X = torch.Tensor(X)
y = torch.Tensor(y)
for _ in range(1000):
output = model(X)
loss = loss_fn(output.squeeze(-1), y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
print(loss)
return loss
def test_boston_housing_progress():
"""Tests that network made using NN module can make progress on boston housing dataset"""
boston = load_boston()
X, y = (boston.data, boston.target)
# Normalise the data
mean = np.mean(X, axis=0)
std = np.std(X, axis=0)
X = (X - mean) / std
model = NN(layers_info=[30, 10, 1], input_dim=13,
hidden_activations="relu", output_activation=None, dropout=0.0,
initialiser="xavier", batch_norm=True, y_range=(4.5, 55.0))
result = train_on_boston_housing(model, X, y)
assert result < 15
model = NN(layers_info=[15, 15, 15, 15, 1], input_dim=13, random_seed=52,
hidden_activations="relu", output_activation=None, dropout=0.0,
initialiser="xavier", batch_norm=False)
result = train_on_boston_housing(model, X, y)
assert result < 15
model = NN(layers_info=[30, 10, 1], input_dim=13,
hidden_activations="relu", output_activation=None, dropout=0.0,
initialiser="xavier", batch_norm=True)
result = train_on_boston_housing(model, X, y)
assert result < 15
model = NN(layers_info=[15, 15, 15, 1], input_dim=13,
hidden_activations="relu", output_activation=None, dropout=0.05,
initialiser="xavier", batch_norm=False)
result = train_on_boston_housing(model, X, y)
assert result < 15
================================================
FILE: tests/pytorch_tests/test_pytorch_RNN.py
================================================
# Run from home directory with python -m pytest tests
import copy
import pytest
import random
import numpy as np
import torch.nn as nn
import torch
from nn_builder.pytorch.RNN import RNN
import torch.optim as optim
N = 250
X = torch.randn((N, 5, 15))
X[0:125, 0, 3] += 20.0
y = X[:, 0, 3] > 5.0
y = y.float()
def test_user_hidden_layers_input_rejections():
"""Tests whether network rejects invalid hidden_layers inputted from user"""
inputs_that_should_fail = [[["linearr", 33]], [["linear", 12, 33]], [["gru", 2, 33]], [["lstm", 2, 33]], [["lstmr", 33]],
[["gruu", 33]], [["gru", 33], ["xxx", 33]], [["linear", 33], ["gru", 12], ["gru", 33]] ]
for input in inputs_that_should_fail:
with pytest.raises(AssertionError):
RNN(input_dim=1, layers_info=input, hidden_activations="relu",
output_activation="relu")
def test_user_hidden_layers_input_acceptances():
"""Tests whether network rejects invalid hidden_layers inputted from user"""
inputs_that_should_work = [[["linear", 33]], [["linear", 12]], [["gru", 2]], [["lstm", 2]], [["lstm", 1]],
[["gru", 330]], [["gru", 33], ["linear", 2]] ]
for input in inputs_that_should_work:
assert RNN(input_dim=1, layers_info=input, hidden_activations="relu",
output_activation="relu")
def test_hidden_layers_created_correctly():
"""Tests that create_hidden_layers works correctly"""
layers = [["gru", 25], ["lstm", 23], ["linear", 5], ["linear", 10]]
rnn = RNN(input_dim=5, layers_info=layers, hidden_activations="relu",
output_activation="relu")
assert type(rnn.hidden_layers[0]) == nn.GRU
assert rnn.hidden_layers[0].input_size == 5
assert rnn.hidden_layers[0].hidden_size == 25
assert type(rnn.hidden_layers[1]) == nn.LSTM
assert rnn.hidden_layers[1].input_size == 25
assert rnn.hidden_layers[1].hidden_size == 23
assert type(rnn.hidden_layers[2]) == nn.Linear
assert rnn.hidden_layers[2].in_features == 23
assert rnn.hidden_layers[2].out_features == 5
assert type(rnn.output_layers[0]) == nn.Linear
assert rnn.output_layers[0].in_features == 5
assert rnn.output_layers[0].out_features == 10
def test_output_layers_created_correctly():
"""Tests that create_output_layers works correctly"""
layers = [["gru", 25], ["lstm", 23], ["linear", 5], ["linear", 10]]
rnn = RNN(input_dim=5, layers_info=layers, hidden_activations="relu", output_activation="relu")
assert rnn.output_layers[0].in_features == 5
assert rnn.output_layers[0].out_features == 10
layers = [["gru", 25], ["lstm", 23], ["lstm", 10]]
rnn = RNN(input_dim=5, layers_info=layers, hidden_activations="relu",
output_activation="relu")
assert rnn.output_layers[0].input_size == 23
assert rnn.output_layers[0].hidden_size == 10
layers = [["gru", 25], ["lstm", 23], [["lstm", 10], ["linear", 15]]]
rnn = RNN(input_dim=5, layers_info=layers, hidden_activations="relu",
output_activation=["relu", "softmax"])
assert rnn.output_layers[0].input_size == 23
assert rnn.output_layers[0].hidden_size == 10
assert rnn.output_layers[1].in_features == 23
assert rnn.output_layers[1].out_features == 15
def test_output_dim_user_input():
"""Tests whether network rejects an invalid output_dim input from user"""
inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
RNN(input_dim=3, layers_info=[2, input_value], hidden_activations="relu", output_activation="relu")
with pytest.raises(AssertionError):
RNN(input_dim=6, layers_info=input_value, hidden_activations="relu", output_activation="relu")
def test_activations_user_input():
"""Tests whether network rejects an invalid hidden_activations or output_activation from user"""
inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
RNN(input_dim=4, layers_info=[["linear", 2]], hidden_activations=input_value,
output_activation="relu")
RNN(input_dim=4, layers_info=[["linear", 2]], hidden_activations="relu",
output_activation=input_value)
def test_initialiser_user_input():
"""Tests whether network rejects an invalid initialiser from user"""
inputs_that_should_fail = [-1, "aa", ["dd"], [2], 0, 2.5, {2}, "Xavier_"]
for input_value in inputs_that_should_fail:
with pytest.raises(AssertionError):
RNN(input_dim=4, layers_info=[["linear", 2]], hidden_activations="relu",
output_activation="relu", initialiser=input_value)
RNN(layers_info=[["linear", 2], ["linear", 2]], hidden_activations="relu",
output_activation="relu", initialiser="xavier", input_dim=4)
def test_batch_norm_layers():
"""Tests whether batch_norm_layers method works correctly"""
layers = [["gru", 20], ["lstm", 3], ["linear", 4], ["linear", 10]]
rnn = RNN(layers_info=layers, hidden_activations="relu", input_dim=5,
output_activation="relu", initialiser="xavier", batch_norm=True)
assert len(rnn.batch_norm_layers) == 3
assert rnn.batch_norm_layers[0].num_features == 20
assert rnn.batch_norm_layers[1].num_features == 3
assert rnn.batch_norm_layers[2].num_features == 4
def test_linear_layers_only_come_at_end():
"""Tests that it throws an error if user tries to provide list of hidden layers that include linear layers where they
don't only come at the end"""
layers = [["gru", 20], ["linear", 4], ["lstm", 3], ["linear", 10]]
with pytest.raises(AssertionError):
rnn = RNN(layers_info=layers, hidden_activations="relu", input_dim=4,
output_activation="relu", initialiser="xavier", batch_norm=True)
layers = [["gru", 20], ["lstm", 3], ["linear", 4], ["linear", 10]]
assert RNN(layers_info=layers, hidden_activations="relu", input_dim=4,
output_activation="relu", initialiser="xavier", batch_norm=True)
def test_output_activation():
"""Tests whether network outputs data that has gone through correct activation function"""
RANDOM_ITERATIONS = 20
input_dim = 100
for _ in range(RANDOM_ITERATIONS):
data = torch.randn((25, 10, 100))
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]],
hidden_activations="relu", input_dim=input_dim,
output_activation="relu", initialiser="xavier", batch_norm=True)
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5]],
hidden_activations="relu", input_dim=input_dim,
output_activation="relu", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]],
hidden_activations="relu", input_dim=input_dim,
output_activation="relu", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]],
hidden_activations="relu", input_dim=input_dim,
output_activation="sigmoid", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
assert all(out.reshape(1, -1).squeeze() <= 1)
summed_result = torch.sum(out, dim=1)
assert all(summed_result.reshape(1, -1).squeeze() != 1.0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]],
hidden_activations="relu", input_dim=input_dim,
output_activation="softmax", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
assert all(out.reshape(1, -1).squeeze() <= 1)
summed_result = torch.sum(out, dim=1)
summed_result = summed_result.reshape(1, -1).squeeze()
summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))
assert all( summed_result == 1.0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]],
hidden_activations="relu", input_dim=input_dim,
output_activation="softmax", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
assert all(out.reshape(1, -1).squeeze() <= 1)
summed_result = torch.sum(out, dim=1)
summed_result = summed_result.reshape(1, -1).squeeze()
summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))
assert all( summed_result == 1.0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]],
hidden_activations="relu", input_dim=input_dim,
initialiser="xavier")
out = RNN_instance.forward(data)
assert not all(out.reshape(1, -1).squeeze() >= 0)
assert not all(out.reshape(1, -1).squeeze() <= 0)
summed_result = torch.sum(out, dim=1)
summed_result = summed_result.reshape(1, -1).squeeze()
summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))
assert not all(summed_result == 1.0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25], ["linear", 8]],
hidden_activations="relu", input_dim=input_dim,
initialiser="xavier")
out = RNN_instance.forward(data)
assert not all(out.reshape(1, -1).squeeze() >= 0)
assert not all(out.reshape(1, -1).squeeze() <= 0)
summed_result = torch.sum(out, dim=1)
summed_result = summed_result.reshape(1, -1).squeeze()
summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))
assert not all( summed_result == 1.0)
def test_output_activation_return_return_final_seq_only_off():
"""Tests whether network outputs data that has gone through correct activation function"""
RANDOM_ITERATIONS = 20
input_dim = 100
for _ in range(RANDOM_ITERATIONS):
data = torch.randn((25, 10, 100))
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]],
hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False,
output_activation="relu", initialiser="xavier", batch_norm=True)
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5]],
hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False,
output_activation="relu", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]],
hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False,
output_activation="relu", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]],
hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False,
output_activation="sigmoid", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
assert all(out.reshape(1, -1).squeeze() <= 1)
summed_result = torch.sum(out, dim=2)
assert all(summed_result.reshape(1, -1).squeeze() != 1.0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["linear", 10], ["linear", 3]],
hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False,
output_activation="softmax", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
assert all(out.reshape(1, -1).squeeze() <= 1)
summed_result = torch.sum(out, dim=2)
summed_result = summed_result.reshape(1, -1).squeeze()
summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))
assert all( summed_result == 1.0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]],
hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False,
output_activation="softmax", initialiser="xavier")
out = RNN_instance.forward(data)
assert all(out.reshape(1, -1).squeeze() >= 0)
assert all(out.reshape(1, -1).squeeze() <= 1)
summed_result = torch.sum(out, dim=2)
summed_result = summed_result.reshape(1, -1).squeeze()
summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))
assert all( summed_result == 1.0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]],
hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False,
initialiser="xavier")
out = RNN_instance.forward(data)
assert not all(out.reshape(1, -1).squeeze() >= 0)
assert not all(out.reshape(1, -1).squeeze() <= 0)
summed_result = torch.sum(out, dim=2)
summed_result = summed_result.reshape(1, -1).squeeze()
summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))
assert not all( summed_result == 1.0)
RNN_instance = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25], ["linear", 8]],
hidden_activations="relu", input_dim=input_dim, return_final_seq_only=False,
initialiser="xavier")
out = RNN_instance.forward(data)
assert not all(out.reshape(1, -1).squeeze() >= 0)
assert not all(out.reshape(1, -1).squeeze() <= 0)
summed_result = torch.sum(out, dim=2)
summed_result = summed_result.reshape(1, -1).squeeze()
summed_result = torch.round( (summed_result * 10 ** 5) / (10 ** 5))
assert not all( summed_result == 1.0)
def test_y_range():
"""Tests whether setting a y range works correctly"""
for _ in range(100):
val1 = random.random() - 3.0*random.random()
val2 = random.random() + 2.0*random.random()
lower_bound = min(val1, val2)
upper_bound = max(val1, val2)
rnn = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]],
hidden_activations="relu", y_range=(lower_bound, upper_bound),
initialiser="xavier", input_dim=22)
random_data = torch.randn((10, 11, 22))
out = rnn.forward(random_data)
out = out.reshape(1, -1).squeeze()
assert torch.sum(out > lower_bound).item() == 25*10, "lower {} vs. {} ".format(lower_bound, out)
assert torch.sum(out < upper_bound).item() == 25*10, "upper {} vs. {} ".format(upper_bound, out)
def test_deals_with_None_activation():
"""Tests whether is able to handle user inputting None as output activation"""
assert RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]],
hidden_activations="relu", output_activation=None,
initialiser="xavier", input_dim=5)
def test_check_input_data_into_forward_once():
"""Tests that check_input_data_into_forward_once method only runs once"""
rnn = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]],
hidden_activations="relu", input_dim=5,
output_activation="relu", initialiser="xavier")
data_not_to_throw_error = torch.randn((1, 4, 5))
data_to_throw_error = torch.randn((1, 2, 20))
with pytest.raises(AssertionError):
rnn.forward(data_to_throw_error)
with pytest.raises(RuntimeError):
rnn.forward(data_not_to_throw_error)
rnn.forward(data_to_throw_error)
def test_y_range_user_input():
"""Tests whether network rejects invalid y_range inputs"""
invalid_y_range_inputs = [ (4, 1), (2, 4, 8), [2, 4], (np.array(2.0), 6.9)]
for y_range_value in invalid_y_range_inputs:
with pytest.raises(AssertionError):
print(y_range_value)
rnn = RNN(layers_info=[["lstm", 20], ["gru", 5], ["lstm", 25]],
hidden_activations="relu", y_range=y_range_value, input_dim=5,
initialiser="xavier")
def solves_simple_problem(X, y, nn_instance):
"""Checks if a given network is able to solve a simple problem"""
optimizer = optim.Adam(nn_instance.parameters(), lr=0.15)
for ix in range(800):
out = nn_instance.forward(X)
loss = torch.sum((out.squeeze() - y) ** 2) / N
optimizer.zero_grad()
loss.backward()
optimizer.step()
print("LOSS ", loss)
return loss < 0.1
def test_model_trains():
"""Tests whether a small range of networks can solve a simple task"""
for output_activation in ["sigmoid", "None"]:
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15,
hidden_activations="relu", output_activation=output_activation,
initialiser="xavier")
assert solves_simple_problem(X, y, rnn)
z = X[:, 0:1, 3:4] > 5.0
z = torch.cat([z ==1, z==0], dim=1).float()
z = z.squeeze(-1).squeeze(-1)
rnn = RNN(layers_info=[["gru", 20], ["lstm", 2]], input_dim=15,
hidden_activations="relu", output_activation="softmax", dropout=0.01,
initialiser="xavier")
assert solves_simple_problem(X, z, rnn)
rnn = RNN(layers_info=[["lstm", 20], ["linear", 1]], input_dim=15,
hidden_activations="relu", output_activation=None,
initialiser="xavier")
assert solves_simple_problem(X, y, rnn)
rnn = RNN(layers_info=[["lstm", 20], ["linear", 20], ["linear", 1]], input_dim=15,
hidden_activations="relu", output_activation=None,
initialiser="xavier", batch_norm=True)
assert solves_simple_problem(X, y, rnn)
rnn = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]], input_dim=15,
hidden_activations="relu", output_activation=None,
initialiser="xavier")
assert solves_simple_problem(X, y, rnn)
def test_error_when_provide_negative_data_for_embedding():
"""Tests that it raises an error if we try to do an embedding on negative data"""
N = 250
X = torch.randn((N, 5, 15))
X[0:125, 0, 3] += 20.0
y = X[:, 0, 3] > 5.0
y = y.float()
with pytest.raises(AssertionError):
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15,
hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[0],
embedding_dimensions=[[200, 5]], initialiser="xavier")
assert solves_simple_problem(X, y, rnn)
X[:, :, 0] = abs(X[:, :, 0]).long()
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15,
hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[0],
embedding_dimensions=[[200, 5]], initialiser="xavier")
assert solves_simple_problem(X, y, rnn)
def test_embedding_layers():
"""Tests whether create_embedding_layers method works correctly"""
for embedding_in_dim_1, embedding_out_dim_1, embedding_in_dim_2, embedding_out_dim_2 in zip(range(5, 8), range(3, 6), range(1, 4), range(24, 27)):
nn_instance = RNN(input_dim=15, layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]],
embedding_dimensions =[[embedding_in_dim_1, embedding_out_dim_1], [embedding_in_dim_2, embedding_out_dim_2]])
for layer in nn_instance.embedding_layers:
assert isinstance(layer, nn.Embedding)
assert len(nn_instance.embedding_layers) == 2
assert nn_instance.embedding_layers[0].num_embeddings == embedding_in_dim_1
assert nn_instance.embedding_layers[0].embedding_dim == embedding_out_dim_1
assert nn_instance.embedding_layers[1].num_embeddings == embedding_in_dim_2
assert nn_instance.embedding_layers[1].embedding_dim == embedding_out_dim_2
def test_model_trains_with_embeddings():
"""Tests that model trains when using embeddings"""
N = 250
X = torch.randn((N, 5, 15))
X[0:125, 0, 3] += 20.0
y = X[:, 0, 3] > 5.0
y = y.float()
Z = copy.deepcopy(X)
Z[:, :, 0] = abs(Z[:, :, 0]).long()
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15,
hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[0],
embedding_dimensions=[[200, 5]], initialiser="xavier")
assert solves_simple_problem(Z, y, rnn)
Z = copy.deepcopy(X)
Z[:, :, 0:2] = abs(Z[:, :, 0:2]).long()
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15,
hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[0, 1],
embedding_dimensions=[[200, 5], [50, 3]], initialiser="xavier")
assert solves_simple_problem(Z, y, rnn)
Z = copy.deepcopy(X)
Z[:, :, 3] = abs(Z[:, :, 3]).long()
Z[:, :, 6] = abs(Z[:, :, 6]).long()
Z[:, :, 4] = abs(Z[:, :, 4]).long()
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 1]], input_dim=15,
hidden_activations="relu", output_activation="sigmoid", columns_of_data_to_be_embedded=[3, 6, 4],
embedding_dimensions=[[200, 5], [50, 3], [50, 12]], initialiser="xavier")
assert solves_simple_problem(Z, y, rnn)
def test_dropout():
"""Tests whether dropout layer reads in probability correctly"""
rnn = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]],
hidden_activations="relu", output_activation="sigmoid", dropout=0.9999,
initialiser="xavier", input_dim=15)
assert rnn.dropout_layer.p == 0.9999
assert not solves_simple_problem(X, y, rnn)
rnn = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]],
hidden_activations="relu", output_activation=None, dropout=0.0000001,
initialiser="xavier", input_dim=15)
assert rnn.dropout_layer.p == 0.0000001
assert solves_simple_problem(X, y, rnn)
def test_all_activations_work():
"""Tests that all activations get accepted"""
nn_instance = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]],
hidden_activations="relu", output_activation=None, dropout=0.0000001,
initialiser="xavier", input_dim=15)
for key in nn_instance.str_to_activations_converter.keys():
if key == "none": hidden_key = "relu"
else: hidden_key = key
model = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]],
hidden_activations=hidden_key, output_activation=key, dropout=0.0000001,
initialiser="xavier", input_dim=15)
model(X)
def test_all_initialisers_work():
"""Tests that all initialisers get accepted"""
nn_instance = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]],
hidden_activations="relu", output_activation=None, dropout=0.0000001,
initialiser="xavier", input_dim=15)
for key in nn_instance.str_to_initialiser_converter.keys():
model = RNN(layers_info=[["lstm", 20], ["gru", 10], ["linear", 20], ["linear", 1]],
dropout=0.0000001,
initialiser=key, input_dim=15)
model(X)
def test_output_shapes():
"""Tests whether network outputs of correct shape"""
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 3]],
hidden_activations="relu", initialiser="xavier", input_dim=15)
output = rnn(X)
assert output.shape == (N, 3)
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]],
hidden_activations="relu", initialiser="xavier", return_final_seq_only=False, input_dim=15)
output = rnn(X)
assert output.shape == (N, 5, 7)
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["lstm", 3]],
hidden_activations="relu", initialiser="xavier", input_dim=15)
output = rnn(X)
assert output.shape == (N, 3)
rnn = RNN(layers_info=[["gru", 20], ["lstm", 8], ["lstm", 7]],
hidden_activations="relu", initialiser="xavier", return_final_seq_only=False, input_dim=15)
output = rnn(X)
assert output.shape == (N, 5, 7)
def test_return_final_seq_user_input_valid():
"""Checks whether network only accepts a valid boolean value for return_final_seq_only"""
for valid_case in [True, False]:
assert RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]],
hidden_activations="relu", initialiser="xavier", return_final_seq_only=valid_case, input_dim=15)
for invalid_case in [[True], 22, [1, 3], (True, False), (5, False)]:
with pytest.raises(AssertionError):
print(invalid_case)
RNN(layers_info=[["gru", 20], ["lstm", 8], ["linear", 7]],
hidden_activations="relu", initialiser="xavier", return_final_seq_only=invalid_case, input_dim=15)
def test_print_model_summary():
nn_instance = RNN(layers_info=[["gru", 20], ["lstm", 8], ["lstm", 7]],
hidden_activations="relu", initialiser="xavier", return_final_seq_only=False, input_dim=15)
nn_instance.print_model_summary()
def test_output_heads_error_catching():
"""Tests that having multiple output heads catches errors from user inputs"""
output_dims_that_should_break = [["linear", 2, 2, "SAME", "conv", 3, 4, "SAME"], [[["lstm", 3], ["gru", 4]]],
[[2, 8]], [-33, 33, 33, 33, 33]]
for output_dim in output_dims_that_should_break:
with pytest.raises(AssertionError):
RNN(input_dim=5, layers_info=[["gru", 20], ["lstm", 8], output_dim],
hidden_activations="relu", output_activation="relu")
output_activations_that_should_break = ["relu", ["relu"], ["relu", "softmax"]]
for output_activation in output_activations_that_should_break:
with pytest.raises(AssertionError):
RNN(input_dim=5, layers_info=[["gru", 20], ["lstm", 8], [["linear", 5], ["linear", 2], ["linear", 5]]],
hidden_activations="relu", output_activation=output_activation)
def test_output_head_layers():
"""Tests whether the output head layers get created properly"""
for output_dim in [[["linear", 3],["linear", 9]], [["linear", 4], ["linear", 20]], [["linear", 1], ["linear", 1]]]:
nn_instance = RNN(input_dim=5, layers_info=[["gru", 20], ["lstm", 8], output_dim],
hidden_activations="relu", output_activation=["softmax", None])
assert nn_instance.output_layers[0].out_features == output_dim[0][1]
assert nn_instance.output_layers[0].in_features == 8
assert nn_instance.output_layers[1].out_features == output_dim[1][1]
assert nn_instance.output_layers[1].in_features == 8
def test_output_head_activations_work():
"""Tests that output head activations work properly"""
output_dim = [["linear", 5], ["linear", 10], ["linear", 3]]
nn_instance = RNN(input_dim=5, layers_info=[["gru", 20], ["lstm", 8], output_dim],
hidden_activations="relu", output_activation=["softmax", None, "relu"])
x = torch.randn((20, 12, 5)) * -20.0
out = nn_instance(x)
assert out.shape == (20, 18)
sums = torch.sum(out[:, :5], dim=1).detach().numpy()
sums_others = torch.sum(out[:, 5:], dim=1).detach().numpy()
sums_others_2 = torch.sum(out[:, 5:15], dim=1).detach().numpy()
sums_others_3 = torch.sum(out[:, 15:18], dim=1).detach().numpy()
for row in range(out.shape[0]):
assert np.round(sums[row], 4) == 1.0, sums[row]
assert not np.round(sums_others[row], 4) == 1.0, sums_others[row]
assert not np.round(sums_others_2[row], 4) == 1.0, sums_others_2[row]
assert not np.round(sums_others_3[row], 4) == 1.0, sums_others_3[row]
for col in range(3):
assert out[row, 15 + col] >= 0.0, out[row, 15 + col]
def test_output_head_shapes_correct():
"""Tests that the output shape of network is correct when using multiple outpout heads"""
N = 20
X = torch.randn((N, 10, 4)) * -20.0
for _ in range(25):
nn_instance = RNN(input_dim=4,
layers_info=[["gru", 20], ["lstm", 8], ["linear", 1], ["linear", 12]],
hidden_activations="relu")
out = nn_instance(X)
assert out.shape[0] == N
assert out.shape[1] == 12
for output_dim in [[ ["linear", 10], ["linear", 4], ["linear", 6]], [["linear", 3], ["linear", 8], ["linear", 9]]]:
nn_instance = RNN(input_dim=4,
layers_info=[["gru", 20], ["lstm", 8], ["linear", 1], ["linear", 12], output_dim],
hidden_activations="relu", output_activation=["softmax", None, "relu"])
out = nn_instance(X)
assert out.shape[0] == N
assert out.shape[1] == 20
================================================
FILE: tests/tensorflow_tests/test_tf_CNN.py
================================================
# Run from home directory with python -m pytest tests
import shutil
import pytest
import random
import numpy as np
import tensorflow as tf
import torch.nn as nn
from nn_builder.tensorflow.CNN import CNN
from tensorflow.keras.layers import Dense, Flatten, Conv2D, Concatenate, BatchNormalization, MaxPool2D, AveragePooling2D
N = 250
X = np.random.random((N, 5, 5, 1))
X[0:125, 3, 3, 0] += 20.0
y = X[:, 3, 3, 0] > 5.0
def test_user_hidden_layers_input_rejections():
"""Tests whether network rejects invalid hidden_layers inputted from user"""
inputs_that_should_fail = [ ('maxpool', 3, 3, 3) , ['maxpool', 33, 22, 33], [['a']], [[222, 222, 222, 222]], [["conv", 2, 2, -1]], [["conv", 2, 2]], [["conv", 2, 2, 55, 999, 33]],
[["maxpool", 33, 33]], [["maxpool", -1, 33]], [["maxpool", 33]], [["maxpoolX", 1, 33]],
[["cosnv", 2, 2]], [["avgpool", 33, 33, 333, 99]], [["avgpool", -1, 33]], [["avgpool", 33]], [["avgpoolX", 1, 33]],
[["adaptivemaxpool", 33, 33, 333, 33]], [["adaptivemaxpool", 2]], [["adaptivemaxpool", 33]], [["adaptivemaxpoolX"]],
[["adaptiveavgpool", 33, 33, 333, 11]], [["adaptiveavgpool", 2]], [["adaptiveavgpool", 33]],
[["adaptiveavgpoolX"]], [["linear", 40, -2]], [["lineafr", 40, 2]]]
for input in inputs_that_should_fail:
print(input)
with pytest.raises(AssertionError):
CNN(layers_info=input, hidden_activations="relu",
output_activation="relu")
def test_user_hidden_layers_input_acceptances():
"""Tests whether network rejects invalid hidden_layers inputted from user"""
inputs_that_should_work = [[["conv", 2, 2, 3331, "VALID"]], [["CONV", 2, 2, 3331, "SAME"]], [["ConV", 2, 2, 3331, "valid"]],
[["maxpool", 2, 2, "same"]], [["MAXPOOL", 2, 2, "Valid"]], [["MaXpOOL", 2, 2, "SAme"]],
[["avgpool", 2, 2, "saME"]], [["AVGPOOL", 2, 2, "vaLID"]], [["avGpOOL", 2, 2, "same"]],
[["linear", 40]], [["lineaR", 2]], [["LINEAR", 2]]]
for ix, input in enumerate(inputs_that_should_work):
input.append(["linear", 5])
CNN(layers_info=input, hidden_activations="relu",
output_activation="relu")
def test_hidden_layers_created_correctly():
"""Tests that create_hidden_layers works correctly"""
layers = [["conv", 2, 4, 3, "same"], ["maxpool", 3, 4, "vaLID"], ["avgpool", 32, 42, "vaLID"],
["linear", 22], ["linear", 2222], ["linear", 5]]
cnn = CNN(layers_info=layers, hidden_activations="relu",
output_activation="relu")
assert type(cnn.hidden_layers[0]) == Conv2D
assert cnn.hidden_layers[0].filters == 2
assert cnn.hidden_layers[0].kernel_size == (4, 4)
assert cnn.hidden_layers[0].strides == (3, 3)
assert cnn.hidden_layers[0].padding == "same"
assert type(cnn.hidden_layers[1]) == MaxPool2D
assert cnn.hidden_layers[1].pool_size == (3, 3)
assert cnn.hidden_layers[1].strides == (4, 4)
assert cnn.hidden_layers[1].padding == "valid"
assert type(cnn.hi
gitextract_38pgry39/
├── .gitignore
├── .travis.yml
├── LICENSE
├── README.md
├── miscellaneous/
│ ├── material_for_readme/
│ │ ├── README.md
│ │ ├── nn_builder_code
│ │ ├── nn_builder_tf_code
│ │ ├── non_nn_builder_code
│ │ └── non_nn_builder_tf_code
│ └── testreadme.rst
├── nn_builder/
│ ├── Overall_Base_Network.py
│ ├── __init__.py
│ ├── pytorch/
│ │ ├── Base_Network.py
│ │ ├── CNN.py
│ │ ├── NN.py
│ │ ├── RNN.py
│ │ └── __init__.py
│ └── tensorflow/
│ ├── Base_Network.py
│ ├── CNN.py
│ ├── NN.py
│ ├── RNN.py
│ └── __init__.py
├── requirements.txt
├── setup.py
└── tests/
├── pytorch_tests/
│ ├── test_pytorch_CNN.py
│ ├── test_pytorch_NN.py
│ └── test_pytorch_RNN.py
└── tensorflow_tests/
├── test_tf_CNN.py
├── test_tf_NN.py
└── test_tf_RNN.py
SYMBOL INDEX (282 symbols across 15 files)
FILE: nn_builder/Overall_Base_Network.py
class Overall_Base_Network (line 3) | class Overall_Base_Network(ABC):
method __init__ (line 5) | def __init__(self, input_dim, layers_info, output_activation, hidden_a...
method check_all_user_inputs_valid (line 32) | def check_all_user_inputs_valid(self):
method create_hidden_layers (line 37) | def create_hidden_layers(self):
method create_output_layers (line 42) | def create_output_layers(self):
method create_batch_norm_layers (line 47) | def create_batch_norm_layers(self):
method create_dropout_layer (line 52) | def create_dropout_layer(self):
method print_model_summary (line 57) | def print_model_summary(self):
method set_all_random_seeds (line 62) | def set_all_random_seeds(self, random_seed):
method check_NN_layers_valid (line 66) | def check_NN_layers_valid(self):
method check_NN_input_dim_valid (line 86) | def check_NN_input_dim_valid(self):
method check_activations_valid (line 91) | def check_activations_valid(self):
method check_embedding_dimensions_valid (line 110) | def check_embedding_dimensions_valid(self):
method check_y_range_values_valid (line 117) | def check_y_range_values_valid(self):
method check_timesteps_to_output_valid (line 125) | def check_timesteps_to_output_valid(self):
method check_initialiser_valid (line 129) | def check_initialiser_valid(self):
method check_return_final_seq_only_valid (line 135) | def check_return_final_seq_only_valid(self):
method get_activation (line 139) | def get_activation(self, activations, ix=None):
FILE: nn_builder/pytorch/Base_Network.py
class Base_Network (line 8) | class Base_Network(Overall_Base_Network, ABC):
method __init__ (line 10) | def __init__(self, input_dim, layers_info, output_activation,
method initialise_all_parameters (line 21) | def initialise_all_parameters(self):
method forward (line 26) | def forward(self, input_data):
method check_input_data_into_forward_once (line 31) | def check_input_data_into_forward_once(self, input_data):
method set_all_random_seeds (line 36) | def set_all_random_seeds(self, random_seed):
method create_str_to_activations_converter (line 44) | def create_str_to_activations_converter(self):
method create_str_to_initialiser_converter (line 55) | def create_str_to_initialiser_converter(self):
method create_dropout_layer (line 66) | def create_dropout_layer(self):
method create_embedding_layers (line 70) | def create_embedding_layers(self):
method initialise_parameters (line 78) | def initialise_parameters(self, parameters_list):
method flatten_tensor (line 89) | def flatten_tensor(self, tensor):
method print_model_summary (line 93) | def print_model_summary(self):
FILE: nn_builder/pytorch/CNN.py
class CNN (line 6) | class CNN(nn.Module, Base_Network):
method __init__ (line 32) | def __init__(self, input_dim, layers_info, output_activation=None,
method flatten_tensor (line 42) | def flatten_tensor(self, tensor):
method check_all_user_inputs_valid (line 51) | def check_all_user_inputs_valid(self):
method check_CNN_input_dim_valid (line 59) | def check_CNN_input_dim_valid(self):
method check_CNN_layers_valid (line 65) | def check_CNN_layers_valid(self):
method create_hidden_layers (line 139) | def create_hidden_layers(self):
method create_and_append_layer (line 148) | def create_and_append_layer(self, input_dim, layer, list_to_append_lay...
method calculate_new_dimensions (line 174) | def calculate_new_dimensions(self, input_dim, layer):
method create_output_layers (line 196) | def create_output_layers(self):
method initialise_all_parameters (line 205) | def initialise_all_parameters(self):
method create_batch_norm_layers (line 213) | def create_batch_norm_layers(self):
method forward (line 224) | def forward(self, x):
method check_input_data_into_forward_once (line 232) | def check_input_data_into_forward_once(self, x):
method process_hidden_layers (line 239) | def process_hidden_layers(self, x):
method process_output_layers (line 258) | def process_output_layers(self, x):
FILE: nn_builder/pytorch/NN.py
class NN (line 6) | class NN(nn.Module, Base_Network):
method __init__ (line 29) | def __init__(self, input_dim, layers_info, output_activation=None,
method check_all_user_inputs_valid (line 40) | def check_all_user_inputs_valid(self):
method create_hidden_layers (line 49) | def create_hidden_layers(self):
method create_output_layers (line 58) | def create_output_layers(self):
method create_batch_norm_layers (line 69) | def create_batch_norm_layers(self):
method initialise_all_parameters (line 74) | def initialise_all_parameters(self):
method forward (line 80) | def forward(self, x):
method check_input_data_into_forward_once (line 89) | def check_input_data_into_forward_once(self, x):
method incorporate_embeddings (line 104) | def incorporate_embeddings(self, x):
method process_hidden_layers (line 116) | def process_hidden_layers(self, x):
method process_output_layers (line 124) | def process_output_layers(self, x):
FILE: nn_builder/pytorch/RNN.py
class RNN (line 6) | class RNN(nn.Module, Base_Network):
method __init__ (line 37) | def __init__(self, input_dim, layers_info, output_activation=None,
method check_all_user_inputs_valid (line 51) | def check_all_user_inputs_valid(self):
method check_RNN_layers_valid (line 61) | def check_RNN_layers_valid(self):
method create_hidden_layers (line 98) | def create_hidden_layers(self):
method create_and_append_layer (line 108) | def create_and_append_layer(self, input_dim, layer, RNN_hidden_layers):
method create_output_layers (line 123) | def create_output_layers(self):
method initialise_all_parameters (line 132) | def initialise_all_parameters(self):
method create_batch_norm_layers (line 138) | def create_batch_norm_layers(self):
method get_activation (line 143) | def get_activation(self, activations, ix=None):
method forward (line 151) | def forward(self, x):
method check_input_data_into_forward_once (line 162) | def check_input_data_into_forward_once(self, x):
method incorporate_embeddings (line 179) | def incorporate_embeddings(self, x, batch_size, seq_length):
method process_hidden_layers (line 194) | def process_hidden_layers(self, x, batch_size, seq_length):
method process_output_layers (line 212) | def process_output_layers(self, x, batch_size, seq_length):
FILE: nn_builder/tensorflow/Base_Network.py
class Base_Network (line 10) | class Base_Network(Overall_Base_Network, ABC):
method __init__ (line 12) | def __init__(self, layers_info, output_activation, hidden_activations,...
method call (line 19) | def call(self, x, training=True):
method create_and_append_layer (line 24) | def create_and_append_layer(self, layer, list_to_append_layer_to, acti...
method set_all_random_seeds (line 28) | def set_all_random_seeds(self, random_seed):
method create_str_to_activations_converter (line 34) | def create_str_to_activations_converter(self):
method create_str_to_initialiser_converter (line 43) | def create_str_to_initialiser_converter(self):
method create_dropout_layer (line 53) | def create_dropout_layer(self):
method create_hidden_layers (line 57) | def create_hidden_layers(self):
method create_output_layers (line 65) | def create_output_layers(self):
method create_embedding_layers (line 80) | def create_embedding_layers(self):
method create_batch_norm_layers (line 88) | def create_batch_norm_layers(self):
method print_model_summary (line 95) | def print_model_summary(self, input_shape=None):
FILE: nn_builder/tensorflow/CNN.py
class CNN (line 7) | class CNN(Model, Base_Network):
method __init__ (line 31) | def __init__(self, layers_info, output_activation=None, hidden_activat...
method check_all_user_inputs_valid (line 39) | def check_all_user_inputs_valid(self):
method check_CNN_layers_valid (line 46) | def check_CNN_layers_valid(self):
method create_and_append_layer (line 111) | def create_and_append_layer(self, layer, list_to_append_layer_to, acti...
method create_batch_norm_layers (line 131) | def create_batch_norm_layers(self):
method call (line 140) | def call(self, x, training=True):
method process_hidden_layers (line 147) | def process_hidden_layers(self, x, training):
method process_output_layers (line 167) | def process_output_layers(self, x):
FILE: nn_builder/tensorflow/NN.py
class NN (line 8) | class NN(Model, Base_Network):
method __init__ (line 29) | def __init__(self, layers_info, output_activation=None, hidden_activat...
method check_all_user_inputs_valid (line 40) | def check_all_user_inputs_valid(self):
method create_and_append_layer (line 48) | def create_and_append_layer(self, layer, list_to_append_layer_to, acti...
method call (line 52) | def call(self, x, training=True):
method incorporate_embeddings (line 59) | def incorporate_embeddings(self, x):
method process_hidden_layers (line 76) | def process_hidden_layers(self, x, training):
method process_output_layers (line 85) | def process_output_layers(self, x):
FILE: nn_builder/tensorflow/RNN.py
class RNN (line 7) | class RNN(Model, Base_Network):
method __init__ (line 36) | def __init__(self, layers_info, output_activation=None, hidden_activat...
method check_all_user_inputs_valid (line 49) | def check_all_user_inputs_valid(self):
method check_RNN_layers_valid (line 58) | def check_RNN_layers_valid(self):
method create_and_append_layer (line 94) | def create_and_append_layer(self, layer, rnn_hidden_layers, activation...
method call (line 113) | def call(self, x, training=True):
method incorporate_embeddings (line 122) | def incorporate_embeddings(self, x):
method process_hidden_layers (line 139) | def process_hidden_layers(self, x, training):
method process_output_layers (line 155) | def process_output_layers(self, x, restricted_to_final_seq):
FILE: tests/pytorch_tests/test_pytorch_CNN.py
function test_user_hidden_layers_input_rejections (line 18) | def test_user_hidden_layers_input_rejections():
function test_user_hidden_layers_input_acceptances (line 32) | def test_user_hidden_layers_input_acceptances():
function test_hidden_layers_created_correctly (line 46) | def test_hidden_layers_created_correctly():
function test_output_layers_created_correctly (line 90) | def test_output_layers_created_correctly():
function test_output_dim_user_input (line 130) | def test_output_dim_user_input():
function test_activations_user_input (line 139) | def test_activations_user_input():
function test_initialiser_user_input (line 149) | def test_initialiser_user_input():
function test_batch_norm_layers (line 160) | def test_batch_norm_layers():
function test_linear_layers_acceptance (line 181) | def test_linear_layers_acceptance():
function test_linear_layers_only_come_at_end (line 194) | def test_linear_layers_only_come_at_end():
function test_output_activation (line 210) | def test_output_activation():
function test_y_range (line 258) | def test_y_range():
function test_deals_with_None_activation (line 273) | def test_deals_with_None_activation():
function test_check_input_data_into_forward_once (line 279) | def test_check_input_data_into_forward_once():
function test_y_range_user_input (line 294) | def test_y_range_user_input():
function test_model_trains (line 304) | def test_model_trains():
function test_model_trains_linear_layer (line 350) | def test_model_trains_linear_layer():
function test_max_pool_working (line 363) | def test_max_pool_working():
function test_dropout (line 373) | def test_dropout():
function solves_simple_problem (line 387) | def solves_simple_problem(X, y, nn_instance):
function test_MNIST_progress (line 399) | def test_MNIST_progress():
function test_all_activations_work (line 446) | def test_all_activations_work():
function test_all_initialisers_work (line 459) | def test_all_initialisers_work():
function test_print_model_summary (line 471) | def test_print_model_summary():
function test_output_heads_error_catching (line 477) | def test_output_heads_error_catching():
function test_output_head_layers (line 493) | def test_output_head_layers():
function test_output_head_activations_work (line 503) | def test_output_head_activations_work():
function test_output_head_shapes_correct (line 528) | def test_output_head_shapes_correct():
FILE: tests/pytorch_tests/test_pytorch_NN.py
function test_linear_hidden_units_user_input (line 21) | def test_linear_hidden_units_user_input():
function test_input_dim_output_dim_user_input (line 28) | def test_input_dim_output_dim_user_input():
function test_activations_user_input (line 35) | def test_activations_user_input():
function test_initialiser_user_input (line 45) | def test_initialiser_user_input():
function test_output_shape_correct (line 55) | def test_output_shape_correct():
function test_output_activation (line 68) | def test_output_activation():
function test_linear_layers (line 101) | def test_linear_layers():
function test_embedding_layers (line 118) | def test_embedding_layers():
function test_non_integer_embeddings_rejected (line 131) | def test_non_integer_embeddings_rejected():
function test_incorporate_embeddings (line 140) | def test_incorporate_embeddings():
function test_embedding_network_can_solve_simple_problem (line 151) | def test_embedding_network_can_solve_simple_problem():
function test_batch_norm_layers (line 162) | def test_batch_norm_layers():
function test_model_trains (line 176) | def test_model_trains():
function solves_simple_problem (line 188) | def solves_simple_problem(X, y, nn_instance):
function test_dropout (line 199) | def test_dropout():
function test_y_range_user_input (line 207) | def test_y_range_user_input():
function test_y_range (line 216) | def test_y_range():
function test_deals_with_None_activation (line 229) | def test_deals_with_None_activation():
function test_check_input_data_into_forward_once (line 233) | def test_check_input_data_into_forward_once():
function test_all_activations_work (line 249) | def test_all_activations_work():
function test_all_initialisers_work (line 258) | def test_all_initialisers_work():
function test_print_model_summary (line 265) | def test_print_model_summary():
function test_output_heads_error_catching (line 269) | def test_output_heads_error_catching():
function test_output_head_layers (line 283) | def test_output_head_layers():
function test_output_head_activations_work (line 293) | def test_output_head_activations_work():
function test_output_head_shapes_correct (line 305) | def test_output_head_shapes_correct():
function train_on_boston_housing (line 323) | def train_on_boston_housing(model, X, y):
function test_boston_housing_progress (line 340) | def test_boston_housing_progress():
FILE: tests/pytorch_tests/test_pytorch_RNN.py
function test_user_hidden_layers_input_rejections (line 17) | def test_user_hidden_layers_input_rejections():
function test_user_hidden_layers_input_acceptances (line 26) | def test_user_hidden_layers_input_acceptances():
function test_hidden_layers_created_correctly (line 35) | def test_hidden_layers_created_correctly():
function test_output_layers_created_correctly (line 59) | def test_output_layers_created_correctly():
function test_output_dim_user_input (line 86) | def test_output_dim_user_input():
function test_activations_user_input (line 95) | def test_activations_user_input():
function test_initialiser_user_input (line 105) | def test_initialiser_user_input():
function test_batch_norm_layers (line 116) | def test_batch_norm_layers():
function test_linear_layers_only_come_at_end (line 126) | def test_linear_layers_only_come_at_end():
function test_output_activation (line 138) | def test_output_activation():
function test_output_activation_return_return_final_seq_only_off (line 218) | def test_output_activation_return_return_final_seq_only_off():
function test_y_range (line 300) | def test_y_range():
function test_deals_with_None_activation (line 316) | def test_deals_with_None_activation():
function test_check_input_data_into_forward_once (line 322) | def test_check_input_data_into_forward_once():
function test_y_range_user_input (line 337) | def test_y_range_user_input():
function solves_simple_problem (line 347) | def solves_simple_problem(X, y, nn_instance):
function test_model_trains (line 359) | def test_model_trains():
function test_error_when_provide_negative_data_for_embedding (line 390) | def test_error_when_provide_negative_data_for_embedding():
function test_embedding_layers (line 409) | def test_embedding_layers():
function test_model_trains_with_embeddings (line 422) | def test_model_trains_with_embeddings():
function test_dropout (line 453) | def test_dropout():
function test_all_activations_work (line 467) | def test_all_activations_work():
function test_all_initialisers_work (line 480) | def test_all_initialisers_work():
function test_output_shapes (line 491) | def test_output_shapes():
function test_return_final_seq_user_input_valid (line 513) | def test_return_final_seq_user_input_valid():
function test_print_model_summary (line 525) | def test_print_model_summary():
function test_output_heads_error_catching (line 531) | def test_output_heads_error_catching():
function test_output_head_layers (line 545) | def test_output_head_layers():
function test_output_head_activations_work (line 555) | def test_output_head_activations_work():
function test_output_head_shapes_correct (line 579) | def test_output_head_shapes_correct():
FILE: tests/tensorflow_tests/test_tf_CNN.py
function test_user_hidden_layers_input_rejections (line 17) | def test_user_hidden_layers_input_rejections():
function test_user_hidden_layers_input_acceptances (line 31) | def test_user_hidden_layers_input_acceptances():
function test_hidden_layers_created_correctly (line 43) | def test_hidden_layers_created_correctly():
function test_output_layers_created_correctly (line 77) | def test_output_layers_created_correctly():
function test_output_dim_user_input (line 102) | def test_output_dim_user_input():
function test_activations_user_input (line 111) | def test_activations_user_input():
function test_initialiser_user_input (line 121) | def test_initialiser_user_input():
function test_batch_norm_layers (line 132) | def test_batch_norm_layers():
function test_linear_layers_acceptance (line 151) | def test_linear_layers_acceptance():
function test_linear_layers_only_come_at_end (line 164) | def test_linear_layers_only_come_at_end():
function test_output_activation (line 180) | def test_output_activation():
function test_y_range (line 229) | def test_y_range():
function test_deals_with_None_activation (line 244) | def test_deals_with_None_activation():
function test_y_range_user_input (line 250) | def test_y_range_user_input():
function test_model_trains (line 260) | def test_model_trains():
function test_model_trains_part_2 (line 270) | def test_model_trains_part_2():
function solves_simple_problem (line 311) | def solves_simple_problem(X, y, nn_instance):
function test_model_trains_linear_layer (line 321) | def test_model_trains_linear_layer():
function test_max_pool_working (line 333) | def test_max_pool_working():
function test_dropout (line 343) | def test_dropout():
function test_MNIST_progress (line 356) | def test_MNIST_progress():
function test_all_activations_work (line 413) | def test_all_activations_work():
function test_all_initialisers_work (line 424) | def test_all_initialisers_work():
function test_print_model_summary (line 436) | def test_print_model_summary():
function test_output_heads_error_catching (line 442) | def test_output_heads_error_catching():
function test_output_head_layers (line 457) | def test_output_head_layers():
function test_output_head_activations_work (line 465) | def test_output_head_activations_work():
function test_output_head_shapes_correct (line 490) | def test_output_head_shapes_correct():
FILE: tests/tensorflow_tests/test_tf_NN.py
function test_linear_hidden_units_user_input (line 16) | def test_linear_hidden_units_user_input():
function test_activations_user_input (line 23) | def test_activations_user_input():
function test_initialiser_user_input (line 33) | def test_initialiser_user_input():
function test_output_shape_correct (line 43) | def test_output_shape_correct():
function test_output_activation (line 56) | def test_output_activation():
function test_linear_layers_info (line 89) | def test_linear_layers_info():
function test_embedding_layers (line 113) | def test_embedding_layers():
function test_incorporate_embeddings (line 126) | def test_incorporate_embeddings():
function test_embedding_network_can_solve_simple_problem (line 137) | def test_embedding_network_can_solve_simple_problem():
function test_batch_norm_layers_info (line 147) | def test_batch_norm_layers_info():
function test_model_trains (line 158) | def test_model_trains():
function solves_simple_problem (line 170) | def solves_simple_problem(X, y, nn_instance):
function test_dropout (line 179) | def test_dropout():
function test_y_range_user_input (line 188) | def test_y_range_user_input():
function test_y_range (line 197) | def test_y_range():
function test_deals_with_None_activation (line 210) | def test_deals_with_None_activation():
function test_all_activations_work (line 214) | def test_all_activations_work():
function test_all_initialisers_work (line 221) | def test_all_initialisers_work():
function test_print_model_summary (line 228) | def test_print_model_summary():
function test_output_heads_error_catching (line 232) | def test_output_heads_error_catching():
function test_output_head_layers (line 246) | def test_output_head_layers():
function test_output_head_activations_work (line 254) | def test_output_head_activations_work():
function test_output_head_shapes_correct (line 278) | def test_output_head_shapes_correct():
function test_boston_housing_progress (line 296) | def test_boston_housing_progress():
FILE: tests/tensorflow_tests/test_tf_RNN.py
function test_user_hidden_layers_input_rejections (line 16) | def test_user_hidden_layers_input_rejections():
function test_user_hidden_layers_input_acceptances (line 25) | def test_user_hidden_layers_input_acceptances():
function test_hidden_layers_created_correctly (line 34) | def test_hidden_layers_created_correctly():
function test_output_layers_created_correctly (line 54) | def test_output_layers_created_correctly():
function test_output_dim_user_input (line 72) | def test_output_dim_user_input():
function test_activations_user_input (line 81) | def test_activations_user_input():
function test_initialiser_user_input (line 91) | def test_initialiser_user_input():
function test_batch_norm_layers (line 102) | def test_batch_norm_layers():
function test_linear_layers_only_come_at_end (line 111) | def test_linear_layers_only_come_at_end():
function test_output_activation (line 123) | def test_output_activation():
function test_output_activation (line 195) | def test_output_activation():
function test_y_range (line 266) | def test_y_range():
function test_deals_with_None_activation (line 281) | def test_deals_with_None_activation():
function test_y_range_user_input (line 287) | def test_y_range_user_input():
function solves_simple_problem (line 298) | def solves_simple_problem(X, y, nn_instance):
function test_model_trains (line 309) | def test_model_trains():
function test_model_trains_part_2 (line 318) | def test_model_trains_part_2():
function test_model_trains_with_batch_norm (line 339) | def test_model_trains_with_batch_norm():
function test_dropout (line 346) | def test_dropout():
function test_all_activations_work (line 360) | def test_all_activations_work():
function test_all_initialisers_work (line 370) | def test_all_initialisers_work():
function test_output_shapes (line 380) | def test_output_shapes():
function test_return_final_seq_user_input_valid (line 402) | def test_return_final_seq_user_input_valid():
function test_embedding_layers (line 414) | def test_embedding_layers():
function test_incorporate_embeddings (line 427) | def test_incorporate_embeddings():
function test_embedding_network_can_solve_simple_problem (line 449) | def test_embedding_network_can_solve_simple_problem():
function test_print_model_summary (line 470) | def test_print_model_summary():
function test_output_heads_error_catching (line 476) | def test_output_heads_error_catching():
function test_output_head_layers (line 491) | def test_output_head_layers():
function test_output_head_activations_work (line 499) | def test_output_head_activations_work():
function test_output_head_shapes_correct (line 525) | def test_output_head_shapes_correct():
Condensed preview — 30 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (273K chars).
[
{
"path": ".gitignore",
"chars": 114,
"preview": "*.pyc\n*cache\n*.idea\n*__pycache__\n*venv\n*nn_builder.egg-info\n*build\n*dist\n.DS_Store\n*to_do_list\n*.ipynb_checkpoints"
},
{
"path": ".travis.yml",
"chars": 445,
"preview": "language:\n python\n\npython: 3.7\ndist: xenial\nsudo: true\n\ninstall:\n - pip install -r requirements.txt -q\n - pip install"
},
{
"path": "LICENSE",
"chars": 1073,
"preview": "Copyright (c) 2018 The Python Packaging Authority\n\nPermission is hereby granted, free of charge, to any person obtaining"
},
{
"path": "README.md",
"chars": 6457,
"preview": "[](https://pepy.tech/project/nn-builder) ![Image](https://travis-ci.org/"
},
{
"path": "miscellaneous/material_for_readme/README.md",
"chars": 11559,
"preview": "[](https://pepy.tech/project/nn-builder) ![Image](https://travis-ci.org/"
},
{
"path": "miscellaneous/material_for_readme/nn_builder_code",
"chars": 222,
"preview": "# With nn_builder\nfrom nn_builder.pytorch.NN import NN\n\nmodel = NN(input_dim=25, layers=[150, 100, 50, 50, 50, 50, 5],\n "
},
{
"path": "miscellaneous/material_for_readme/nn_builder_tf_code",
"chars": 573,
"preview": "#With nn_builder\nfrom nn_builder.tensorflow.CNN import CNN\nmodel=CNN(layers_info=[[\"conv\", 32, 3, 1, \"valid\"],[\"maxpool\""
},
{
"path": "miscellaneous/material_for_readme/non_nn_builder_code",
"chars": 1307,
"preview": "# Without nn_builder\nimport torch.nn as nn\n\nclass NN(nn.Module):\n def __init__(self):\n nn.Module.__init__(self"
},
{
"path": "miscellaneous/material_for_readme/non_nn_builder_tf_code",
"chars": 1924,
"preview": "#Without nn_builder\nimport tensorflow as tf\nfrom tensorflow.keras import Model, activations\nfrom tensorflow.keras.layers"
},
{
"path": "miscellaneous/testreadme.rst",
"chars": 7930,
"preview": "\n+-----------------------------------------------------------+----------------------------------------------------------"
},
{
"path": "nn_builder/Overall_Base_Network.py",
"chars": 7467,
"preview": "from abc import ABC, abstractmethod\n\nclass Overall_Base_Network(ABC):\n\n def __init__(self, input_dim, layers_info, ou"
},
{
"path": "nn_builder/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "nn_builder/pytorch/Base_Network.py",
"chars": 5027,
"preview": "import random\nimport numpy as np\nimport torch\nimport torch.nn as nn\nfrom nn_builder.Overall_Base_Network import Overall_"
},
{
"path": "nn_builder/pytorch/CNN.py",
"chars": 16419,
"preview": "import torch\nimport torch.nn as nn\nimport numpy as np\nfrom nn_builder.pytorch.Base_Network import Base_Network\n\nclass CN"
},
{
"path": "nn_builder/pytorch/NN.py",
"chars": 8341,
"preview": "import torch\nimport numpy as np\nimport torch.nn as nn\nfrom nn_builder.pytorch.Base_Network import Base_Network\n\nclass NN"
},
{
"path": "nn_builder/pytorch/RNN.py",
"chars": 13965,
"preview": "import torch\nimport torch.nn as nn\nimport numpy as np\nfrom nn_builder.pytorch.Base_Network import Base_Network\n\nclass RN"
},
{
"path": "nn_builder/pytorch/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "nn_builder/tensorflow/Base_Network.py",
"chars": 5369,
"preview": "from tensorflow.keras.layers import BatchNormalization\nfrom nn_builder.Overall_Base_Network import Overall_Base_Network\n"
},
{
"path": "nn_builder/tensorflow/CNN.py",
"chars": 11071,
"preview": "import numpy as np\nfrom tensorflow.keras import Model, activations\nfrom tensorflow.keras.layers import Dense, Flatten, C"
},
{
"path": "nn_builder/tensorflow/NN.py",
"chars": 5789,
"preview": "import tensorflow as tf\nfrom tensorflow.keras import Model, activations\nimport numpy as np\nfrom tensorflow.keras.layers "
},
{
"path": "nn_builder/tensorflow/RNN.py",
"chars": 10271,
"preview": "import numpy as np\nimport tensorflow as tf\nfrom tensorflow.keras import Model, activations\nfrom tensorflow.keras.layers "
},
{
"path": "nn_builder/tensorflow/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "requirements.txt",
"chars": 110,
"preview": "tensorflow==2.0.0a0\ntorch==1.0.1.post2\ntorchvision==0.2.2.post3\nnumpy==1.16.2\nsetuptools==40.8.0\npytest==4.4.0"
},
{
"path": "setup.py",
"chars": 794,
"preview": "import setuptools\nimport sys\n\nwith open(\"README.md\", \"r\") as fh:\n long_description = fh.read()\n\nsetuptools.setup(\n "
},
{
"path": "tests/pytorch_tests/test_pytorch_CNN.py",
"chars": 28621,
"preview": "# Run from home directory with python -m pytest tests\nimport shutil\nimport pytest\nimport torch\nimport random\nimport nump"
},
{
"path": "tests/pytorch_tests/test_pytorch_NN.py",
"chars": 17765,
"preview": "# Run from home directory with python -m pytest tests\nimport copy\nimport shutil\nfrom sklearn.datasets import load_boston"
},
{
"path": "tests/pytorch_tests/test_pytorch_RNN.py",
"chars": 30350,
"preview": "# Run from home directory with python -m pytest tests\nimport copy\nimport pytest\nimport random\nimport numpy as np\nimport "
},
{
"path": "tests/tensorflow_tests/test_tf_CNN.py",
"chars": 25379,
"preview": "# Run from home directory with python -m pytest tests\nimport shutil\nimport pytest\nimport random\nimport numpy as np\nimpor"
},
{
"path": "tests/tensorflow_tests/test_tf_NN.py",
"chars": 15874,
"preview": "# Run from home directory with python -m pytest tests\nimport pytest\nimport copy\nimport random\nimport numpy as np\nimport "
},
{
"path": "tests/tensorflow_tests/test_tf_RNN.py",
"chars": 26285,
"preview": "# Run from home directory with python -m pytest tests\nimport pytest\nimport random\nimport tensorflow as tf\nimport numpy a"
}
]
About this extraction
This page contains the full source code of the p-christ/nn_builder GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 30 files (254.4 KB), approximately 64.4k tokens, and a symbol index with 282 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.