Repository: ugent-korea/pytorch-unet-segmentation
Branch: master
Commit: 69169d86126e
Files: 34
Total size: 72.4 KB
Directory structure:
gitextract_h5owzdln/
├── .gitignore
├── LICENSE
├── README.md
├── readme_images/
│ ├── bright_10
│ ├── c_lb
│ ├── c_lt
│ ├── c_rb
│ ├── c_rt
│ ├── description
│ ├── division_matrix
│ ├── elastic_1
│ ├── file_name_description
│ ├── final_concate
│ ├── flip_both
│ ├── flip_hori
│ ├── flip_vert
│ ├── gn_10
│ ├── gn_100
│ ├── gn_50
│ ├── original_image
│ ├── un_100
│ ├── un_50
│ └── uniform_10
└── src/
├── accuracy.py
├── advanced_model.py
├── dataset.py
├── main.py
├── mean_std.py
├── modules.py
├── post_processing.py
├── pre_processing.py
├── result_visualization.py
├── save_history.py
└── simple_model.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*.csv
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2018 UGent Korea
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# pytorch-unet-segmentation
**Members** : PyeongEun Kim, JuHyung Lee, MiJeong Lee
**Supervisors** : Utku Ozbulak, Wesley De Neve
## Description
This project aims to implement biomedical image segmentation with the use of U-Net model. The below image briefly explains the output we want:
The dataset we used is Transmission Electron Microscopy (ssTEM) data set of the Drosophila first instar larva ventral nerve cord (VNC), which is dowloaded from [ISBI Challenge: Segmentation of of neural structures in EM stacks](http://brainiac2.mit.edu/isbi_challenge/home)
The dataset contains 30 images (.png) of size 512x512 for each train, train-labels and test.
## Table of Content
* [Dataset](#dataset)
* [Preprocessing](#preprocessing)
* [Model](#model)
* [Loss function](#lossfunction)
* [Post-processing](#postprocessing)
* [Results](#results)
* [Dependency](#dependency)
* [References](#references)
## Dataset
```ruby
class SEMDataTrain(Dataset):
def __init__(self, image_path, mask_path, in_size=572, out_size=388):
"""
Args:
image_path (str): the path where the image is located
mask_path (str): the path where the mask is located
option (str): decide which dataset to import
"""
# All file names
# Lists of image path and list of labels
# Calculate len
# Calculate mean and stdev
def __getitem__(self, index):
"""Get specific data corresponding to the index
Args:
index (int): index of the data
Returns:
Tensor: specific data on index which is converted to Tensor
"""
"""
# GET IMAGE
"""
#Augmentation on image
# Flip
# Gaussian_noise
# Uniform_noise
# Brightness
# Elastic distort {0: distort, 1:no distort}
# Crop the image
# Pad the image
# Sanity Check for Cropped image
# Normalize the image
# Add additional dimension
# Convert numpy array to tensor
#Augmentation on mask
# Flip same way with image
# Elastic distort same way with image
# Crop the same part that was cropped on image
# Sanity Check
# Normalize the mask to 0 and 1
# Add additional dimension
# Convert numpy array to tensor
return (img_as_tensor, msk_as_tensor)
def __len__(self):
"""
Returns:
length (int): length of the data
"""
```
## Preprocessing
We preprocessed the images for data augmentation. Following preprocessing are :
* Flip
* Gaussian noise
* Uniform noise
* Brightness
* Elastic deformation
* Crop
* Pad
#### Image Augmentation
Original Image
| Image |
| Flip |
Vertical |
Horizontal |
Both |
| Gaussian noise |
Standard Deviation: 10 |
Standard Deviation: 50 |
Standard Deviation: 100 |
| Uniform noise |
Intensity: 10 |
Intensity: 50 |
Intensity: 100 |
| Brightness |
Intensity: 10 |
Intensity: 20 |
Intensity: 30 |
| Elastic deformation |
Random Deformation: 1 |
Random Deformation: 2 |
Random Deformation: 3 |
#### Crop and Pad
| Crop |
Left Bottom |
Left Top |
Right Bottom |
Right Top |
Padding process is compulsory after the cropping process as the image has to fit the input size of the U-Net model.
In terms of the padding method, **symmetric padding** was done in which the pad is the reflection of the vector mirrored along the edge of the array. We selected the symmetric padding over several other padding options because it reduces the loss the most.
To help with observation, a  'yellow border' is added around the original image: outside the border indicates symmetric padding whereas inside indicates the original image.
| Pad |
Left Bottom |
Left Top |
Right bottom |
Right Top |
## Model
#### Architecture
We have same structure as U-Net Model architecture but we made a small modification to make the model smaller.
## Loss function
We used a loss function where pixel-wise softmax is combined with cross entropy.
#### Softmax
.png)
#### Cross entropy
.png)
## Post-processing
In attempt of reducing the loss, we did a post-processing on the prediction results. We applied the concept of watershed segmentation in order to point out the certain foreground regions and remove regions in the prediction image which seem to be noises.
The numbered images in the figure above indicates the stpes we took in the post-processing. To name those steps in slightly more detail:
* 1. Convertion into grayscale
* 2. Conversion into binary image
* 3. Morphological transformation: Closing
* 4. Determination of the certain background
* 5. Calculation of the distance
* 6. Determination of the certain foreground
* 7. Determination of the unknown region
* 8. Application of watershed
* 9. Determination of the final result
### Conversion into grayscale
The first step is there just in case the input image has more than 1 color channel (e.g. RGB image has 3 channels)
### Conversion into binary image
Convert the gray-scale image into binary image by processing the image with a threshold value: pixels equal to or lower than 127 will be pushed down to 0 and greater will be pushed up to 255. Such process is compulsory as later transformation processes takes in binary images.
### Morphological transformation: Closing.
We used **morphologyEX()** function in cv2 module which removes black noises (background) within white regions (foreground).
### Determination of the certain background
We used **dilate()** function in cv2 module which emphasizes/increases the white region (foreground). By doing so, we connect detached white regions together - for example, connecting detached cell membranes together - to make ensure the background region.
### Caculation of the distance
This step labels the foreground with a color code:  red color indicates farthest from the background while  blue color indicates closest to the background.
### Determination of the foreground
Now that we have an idea of how far the foreground is from the background, we apply a threshold value to decide which part could surely be the foreground.
The threshold value is the maximum distance (calculated from the previous step) multiplied by a hyper-parameter that we have to manually tune. The greater the hyper-parameter value, the greater the threshold value, and therefore we will get less area of certain foreground.
### Determination of the unknown region
From previous steps, we determined sure foreground and background regions. The rest will be classified as *'unknown'* regions.
### Label the foreground: markers
We applied **connectedComponents()** function from the cv2 module on the foreground to label the foreground regions with color to distinguish different foreground objects. We named it as a 'marker'.
### Application of watershed and Determination of the final result
After applying **watershed()** function from cv2 module on the marker, we obtained an array of -1, 1, and many others.
* -1 = Border region that distinguishes foreground and background
* 1 = Background region
To see the result, we created a clean white page of the same size with the input image. then we copied all the values from the watershed result to the white page except 1, which means that we excluded the background.
## Results
| Optimizer |
Learning Rate |
Lowest Loss |
Epoch |
Highest Accuracy |
Epoch |
| SGD |
0.001 |
0.196972 |
1445 |
0.921032 |
1855 |
| 0.005 |
0.205802 |
1815 |
0.918425 |
1795 |
| 0.01 |
0.193328 |
450 |
0.922908 |
450 |
| RMS_prop |
0.0001 |
0.203431 |
185 |
0.924543 |
230 |
| 0.0002 |
0.193456 |
270 |
0.926245 |
500 |
| 0.001 |
0.268246 |
1655 |
0.882229 |
1915 |
| Adam |
0.0001 |
0.194180 |
140 |
0.924470 |
300 |
| 0.0005 |
0.185212 |
135 |
0.925519 |
135 |
| 0.001 |
0.222277 |
165 |
0.912364 |
180 |
We chose the best learning rate that fits the optimizer based on **how fast the model converges to the lowest error**. In other word, the learning rate should make model to reach optimal solution in shortest epoch repeated. However, the intersting fact was that the epochs of lowest loss and highest accuracy were not corresponding. This might be due to the nature of loss function (Loss function is log scale, thus an extreme deviation might occur). For example, if the softmax probability of one pixel is 0.001, then the -log(0.001) would be 1000 which is a huge value that contributes to loss.
For consistency, we chose to focus on accuracy as our criterion of correctness of model.
| Accuracy and Loss Graph |
 |
 |
 |
SGD
(lr=0.01,momentum=0.99) |
RMS prop
(lr=0.0002) |
Adam
(lr=0.0005) |
We used two different optimizers (SGD, RMS PROP, and Adam). In case of SGD the momentum is manually set (0.99) whereas in case of other optimizers (RMS Prop and Adam) it is calculated automatically.
### Model Downloads
Model trained with SGD can be downloaded via **dropbox**:
https://www.dropbox.com/s/ge9654nhgv1namr/model_epoch_2290.pwf?dl=0
Model trained with RMS prop can be downloaded via **dropbox**:
https://www.dropbox.com/s/cdwltzhbs3tiiwb/model_epoch_440.pwf?dl=0
Model trained with Adam can be downloaded via **dropbox**:
https://www.dropbox.com/s/tpch6u41jrdgswk/model_epoch_100.pwf?dl=0
### Example
Input Image
| Results comparsion |
 |
 |
 |
 |
| original image mask |
RMS prop optimizer
(Accuracy 92.48 %) |
SGD optimizer
(Accuracy 91.52 %) |
Adam optimizer
(Accuracy 92.55 %) |
## Dependency
Following modules are used in the project:
* python >= 3.6
* numpy >= 1.14.5
* torch >= 0.4.0
* PIL >= 5.2.0
* scipy >= 1.1.0
* matplotlib >= 2.2.2
## References :
[1] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation, http://arxiv.org/pdf/1505.04597.pdf
[2] P.Y. Simard, D. Steinkraus, J.C. Platt. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis, http://cognitivemedium.com/assets/rmnist/Simard.pdf
================================================
FILE: readme_images/description
================================================
z_original: original image
bright_x: brightness increased by x
elastic_x: different elastic
flip_x: flip on x way
gaus_x: gaussian noise added with mean of 0 and std of x
uniform_x: uniform noise added with lower bound of -x and upper bound of x
================================================
FILE: readme_images/file_name_description
================================================
file name explanation
p: padded image
c: cropped image
lb: left bottom
lt: left top
rb: right bottom
rt: right top
================================================
FILE: src/accuracy.py
================================================
#from post_processing import *
import numpy as np
from PIL import Image
import glob as gl
import numpy as np
from PIL import Image
import torch
def accuracy_check(mask, prediction):
ims = [mask, prediction]
np_ims = []
for item in ims:
if 'str' in str(type(item)):
item = np.array(Image.open(item))
elif 'PIL' in str(type(item)):
item = np.array(item)
elif 'torch' in str(type(item)):
item = item.numpy()
np_ims.append(item)
compare = np.equal(np_ims[0], np_ims[1])
accuracy = np.sum(compare)
return accuracy/len(np_ims[0].flatten())
def accuracy_check_for_batch(masks, predictions, batch_size):
total_acc = 0
for index in range(batch_size):
total_acc += accuracy_check(masks[index], predictions[index])
return total_acc/batch_size
"""
def accuracy_compare(prediction_folder, true_mask_folder):
''' Output average accuracy of all prediction results and their corresponding true masks.
Args
prediction_folder : folder of the prediction results
true_mask_folder : folder of the corresponding true masks
Returns
a tuple of (original_accuracy, posprocess_accuracy)
'''
# Bring in the images
all_prediction = gl.glob(prediction_folder)
all_mask = gl.glob(true_mask_folder)
# Initiation
num_files = len(all_prediction)
count = 0
postprocess_acc = 0
original_acc = 0
while count != num_files:
# Prepare the arrays to be further processed.
prediction_processed = postprocess(all_prediction[count])
prediction_image = Image.open(all_prediction[count])
mask = Image.open(all_mask[count])
# converting the PIL variables into numpy array
prediction_np = np.asarray(prediction_image)
mask_np = np.asarray(mask)
# Calculate the accuracy of original and postprocessed image
postprocess_acc += accuracy_check(mask_np, prediction_processed)
original_acc += accuracy_check(mask_np, prediction_np)
# check individual accuracy
print(str(count) + 'th post acc:', accuracy_check(mask_np, prediction_processed))
print(str(count) + 'th original acc:', accuracy_check(mask_np, prediction_np))
# Move onto the next prediction/mask image
count += 1
# Average of all the accuracies
postprocess_acc = postprocess_acc / num_files
original_acc = original_acc / num_files
return (original_acc, postprocess_acc)
"""
# Experimenting
if __name__ == '__main__':
'''
predictions = 'result/*.png'
masks = '../data/val/masks/*.png'
result = accuracy_compare(predictions, masks)
print('Original Result :', result[0])
print('Postprocess result :', result[1])
'''
================================================
FILE: src/advanced_model.py
================================================
# full assembly of the sub-parts to form the complete net
import torch
import torch.nn as nn
from torch.autograd import Variable
import numpy as np
from PIL import Image
from torch.nn.functional import sigmoid
class Double_conv(nn.Module):
'''(conv => ReLU) * 2 => MaxPool2d'''
def __init__(self, in_ch, out_ch):
"""
Args:
in_ch(int) : input channel
out_ch(int) : output channel
"""
super(Double_conv, self).__init__()
self.conv = nn.Sequential(
nn.Conv2d(in_ch, out_ch, 3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(out_ch, out_ch, 3, padding=0, stride=1),
nn.ReLU(inplace=True)
)
def forward(self, x):
x = self.conv(x)
return x
class Conv_down(nn.Module):
'''(conv => ReLU) * 2 => MaxPool2d'''
def __init__(self, in_ch, out_ch):
"""
Args:
in_ch(int) : input channel
out_ch(int) : output channel
"""
super(Conv_down, self).__init__()
self.conv = Double_conv(in_ch, out_ch)
self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
def forward(self, x):
x = self.conv(x)
pool_x = self.pool(x)
return pool_x, x
class Conv_up(nn.Module):
'''(conv => ReLU) * 2 => MaxPool2d'''
def __init__(self, in_ch, out_ch):
"""
Args:
in_ch(int) : input channel
out_ch(int) : output channel
"""
super(Conv_up, self).__init__()
self.up = nn.ConvTranspose2d(in_ch, out_ch, kernel_size=2, stride=2)
self.conv = Double_conv(in_ch, out_ch)
def forward(self, x1, x2):
x1 = self.up(x1)
x1_dim = x1.size()[2]
x2 = extract_img(x1_dim, x2)
x1 = torch.cat((x1, x2), dim=1)
x1 = self.conv(x1)
return x1
def extract_img(size, in_tensor):
"""
Args:
size(int) : size of cut
in_tensor(tensor) : tensor to be cut
"""
dim1, dim2 = in_tensor.size()[2:]
in_tensor = in_tensor[:, :, int((dim1-size)/2):int((dim1+size)/2),
int((dim2-size)/2):int((dim2+size)/2)]
return in_tensor
class CleanU_Net(nn.Module):
def __init__(self, in_channels, out_channels):
super(CleanU_Net, self).__init__()
self.Conv_down1 = Conv_down(in_channels, 64)
self.Conv_down2 = Conv_down(64, 128)
self.Conv_down3 = Conv_down(128, 256)
self.Conv_down4 = Conv_down(256, 512)
self.Conv_down5 = Conv_down(512, 1024)
self.Conv_up1 = Conv_up(1024, 512)
self.Conv_up2 = Conv_up(512, 256)
self.Conv_up3 = Conv_up(256, 128)
self.Conv_up4 = Conv_up(128, 64)
self.Conv_out = nn.Conv2d(64, out_channels, 1, padding=0, stride=1)
#self.Conv_final = nn.Conv2d(out_channels, out_channels, 1, padding=0, stride=1)
def forward(self, x):
x, conv1 = self.Conv_down1(x)
#print("dConv1 => down1|", x.shape)
x, conv2 = self.Conv_down2(x)
#print("dConv2 => down2|", x.shape)
x, conv3 = self.Conv_down3(x)
#print("dConv3 => down3|", x.shape)
x, conv4 = self.Conv_down4(x)
#print("dConv4 => down4|", x.shape)
_, x = self.Conv_down5(x)
#print("dConv5|", x.shape)
x = self.Conv_up1(x, conv4)
#print("up1 => uConv1|", x.shape)
x = self.Conv_up2(x, conv3)
#print("up2 => uConv2|", x.shape)
x = self.Conv_up3(x, conv2)
#print("up3 => uConv3|", x.shape)
x = self.Conv_up4(x, conv1)
x = self.Conv_out(x)
#x = self.Conv_final(x)
return x
if __name__ == "__main__":
# A full forward pass
im = torch.randn(1, 1, 572, 572)
model = CleanU_Net(1, 2)
x = model(im)
print(x.shape)
del model
del x
# print(x.shape)
================================================
FILE: src/dataset.py
================================================
import numpy as np
from PIL import Image
import glob
import torch
import torch.nn as nn
from torch.autograd import Variable
from random import randint
from torch.utils.data.dataset import Dataset
from pre_processing import *
from mean_std import *
Training_MEAN = 0.4911
Training_STDEV = 0.1658
class SEMDataTrain(Dataset):
def __init__(self, image_path, mask_path, in_size=572, out_size=388):
"""
Args:
image_path (str): the path where the image is located
mask_path (str): the path where the mask is located
option (str): decide which dataset to import
"""
# all file names
self.mask_arr = glob.glob(str(mask_path) + "/*")
self.image_arr = glob.glob(str(image_path) + str("/*"))
self.in_size, self.out_size = in_size, out_size
# Calculate len
self.data_len = len(self.mask_arr)
# calculate mean and stdev
def __getitem__(self, index):
"""Get specific data corresponding to the index
Args:
index (int): index of the data
Returns:
Tensor: specific data on index which is converted to Tensor
"""
"""
# GET IMAGE
"""
single_image_name = self.image_arr[index]
img_as_img = Image.open(single_image_name)
# img_as_img.show()
img_as_np = np.asarray(img_as_img)
# Augmentation
# flip {0: vertical, 1: horizontal, 2: both, 3: none}
flip_num = randint(0, 3)
img_as_np = flip(img_as_np, flip_num)
# Noise Determine {0: Gaussian_noise, 1: uniform_noise
if randint(0, 1):
# Gaussian_noise
gaus_sd, gaus_mean = randint(0, 20), 0
img_as_np = add_gaussian_noise(img_as_np, gaus_mean, gaus_sd)
else:
# uniform_noise
l_bound, u_bound = randint(-20, 0), randint(0, 20)
img_as_np = add_uniform_noise(img_as_np, l_bound, u_bound)
# Brightness
pix_add = randint(-20, 20)
img_as_np = change_brightness(img_as_np, pix_add)
# Elastic distort {0: distort, 1:no distort}
sigma = randint(6, 12)
# sigma = 4, alpha = 34
img_as_np, seed = add_elastic_transform(img_as_np, alpha=34, sigma=sigma, pad_size=20)
# Crop the image
img_height, img_width = img_as_np.shape[0], img_as_np.shape[1]
pad_size = int((self.in_size - self.out_size)/2)
img_as_np = np.pad(img_as_np, pad_size, mode="symmetric")
y_loc, x_loc = randint(0, img_height-self.out_size), randint(0, img_width-self.out_size)
img_as_np = cropping(img_as_np, crop_size=self.in_size, dim1=y_loc, dim2=x_loc)
'''
# Sanity Check for image
img1 = Image.fromarray(img_as_np)
img1.show()
'''
# Normalize the image
img_as_np = normalization2(img_as_np, max=1, min=0)
img_as_np = np.expand_dims(img_as_np, axis=0) # add additional dimension
img_as_tensor = torch.from_numpy(img_as_np).float() # Convert numpy array to tensor
"""
# GET MASK
"""
single_mask_name = self.mask_arr[index]
msk_as_img = Image.open(single_mask_name)
# msk_as_img.show()
msk_as_np = np.asarray(msk_as_img)
# flip the mask with respect to image
msk_as_np = flip(msk_as_np, flip_num)
# elastic_transform of mask with respect to image
# sigma = 4, alpha = 34, seed = from image transformation
msk_as_np, _ = add_elastic_transform(
msk_as_np, alpha=34, sigma=sigma, seed=seed, pad_size=20)
msk_as_np = approximate_image(msk_as_np) # images only with 0 and 255
# Crop the mask
msk_as_np = cropping(msk_as_np, crop_size=self.out_size, dim1=y_loc, dim2=x_loc)
'''
# Sanity Check for mask
img2 = Image.fromarray(msk_as_np)
img2.show()
'''
# Normalize mask to only 0 and 1
msk_as_np = msk_as_np/255
# msk_as_np = np.expand_dims(msk_as_np, axis=0) # add additional dimension
msk_as_tensor = torch.from_numpy(msk_as_np).long() # Convert numpy array to tensor
return (img_as_tensor, msk_as_tensor)
def __len__(self):
"""
Returns:
length (int): length of the data
"""
return self.data_len
class SEMDataVal(Dataset):
def __init__(self, image_path, mask_path, in_size=572, out_size=388):
'''
Args:
image_path = path where test images are located
mask_path = path where test masks are located
'''
# paths to all images and masks
self.mask_arr = glob.glob(str(mask_path) + str("/*"))
self.image_arr = glob.glob(str(image_path) + str("/*"))
self.in_size = in_size
self.out_size = out_size
self.data_len = len(self.mask_arr)
def __getitem__(self, index):
"""Get specific data corresponding to the index
Args:
index : an integer variable that calls (indext)th image in the
path
Returns:
Tensor: 4 cropped data on index which is converted to Tensor
"""
single_image = self.image_arr[index]
img_as_img = Image.open(single_image)
# img_as_img.show()
# Convert the image into numpy array
img_as_np = np.asarray(img_as_img)
# Make 4 cropped image (in numpy array form) using values calculated above
# Cropped images will also have paddings to fit the model.
pad_size = int((self.in_size - self.out_size)/2)
img_as_np = np.pad(img_as_np, pad_size, mode="symmetric")
img_as_np = multi_cropping(img_as_np,
crop_size=self.in_size,
crop_num1=2, crop_num2=2)
# Empty list that will be filled in with arrays converted to tensor
processed_list = []
for array in img_as_np:
# SANITY CHECK: SEE THE CROPPED & PADDED IMAGES
#array_image = Image.fromarray(array)
# Normalize the cropped arrays
img_to_add = normalization2(array, max=1, min=0)
# Convert normalized array into tensor
processed_list.append(img_to_add)
img_as_tensor = torch.Tensor(processed_list)
# return tensor of 4 cropped images
# top left, top right, bottom left, bottom right respectively.
"""
# GET MASK
"""
single_mask_name = self.mask_arr[index]
msk_as_img = Image.open(single_mask_name)
# msk_as_img.show()
msk_as_np = np.asarray(msk_as_img)
# Normalize mask to only 0 and 1
msk_as_np = multi_cropping(msk_as_np,
crop_size=self.out_size,
crop_num1=2, crop_num2=2)
msk_as_np = msk_as_np/255
# msk_as_np = np.expand_dims(msk_as_np, axis=0) # add additional dimension
msk_as_tensor = torch.from_numpy(msk_as_np).long() # Convert numpy array to tensor
original_msk = torch.from_numpy(np.asarray(msk_as_img))
return (img_as_tensor, msk_as_tensor, original_msk)
def __len__(self):
return self.data_len
class SEMDataTest(Dataset):
def __init__(self, image_path, in_size=572, out_size=388):
'''
Args:
image_path = path where test images are located
mask_path = path where test masks are located
'''
# paths to all images and masks
self.image_arr = glob.glob(str(image_path) + str("/*"))
self.in_size = in_size
self.out_size = out_size
self.data_len = len(self.image_arr)
def __getitem__(self, index):
'''Get specific data corresponding to the index
Args:
index: an integer variable that calls(indext)th image in the
path
Returns:
Tensor: 4 cropped data on index which is converted to Tensor
'''
single_image = self.image_arr[index]
img_as_img = Image.open(single_image)
# img_as_img.show()
# Convert the image into numpy array
img_as_np = np.asarray(img_as_img)
pad_size = int((self.in_size - self.out_size)/2)
img_as_np = np.pad(img_as_np, pad_size, mode="symmetric")
img_as_np = multi_cropping(img_as_np,
crop_size=self.in_size,
crop_num1=2, crop_num2=2)
# Empty list that will be filled in with arrays converted to tensor
processed_list = []
for array in img_as_np:
# SANITY CHECK: SEE THE PADDED AND CROPPED IMAGES
# array_image = Image.fromarray(array)
# Normalize the cropped arrays
img_to_add = normalization2(array, max=1, min=0)
# Convert normalized array into tensor
processed_list.append(img_to_add)
img_as_tensor = torch.Tensor(processed_list)
# return tensor of 4 cropped images
# top left, top right, bottom left, bottom right respectively.
return img_as_tensor
def __len__(self):
return self.data_len
if __name__ == "__main__":
SEM_train = SEMDataTrain(
'../data/train/images', '../data/train/masks')
SEM_test = SEMDataTest(
'../data/test/images/', '../data/test/masks')
SEM_val = SEMDataVal('../data/val/images', '../data/val/masks')
imag_1, msk = SEM_train.__getitem__(0)
================================================
FILE: src/main.py
================================================
from advanced_model import CleanU_Net
from dataset import *
import torch
import torch.nn as nn
from torch.autograd import Variable
import numpy as np
from PIL import Image
from modules import *
from save_history import *
if __name__ == "__main__":
# Dataset begin
SEM_train = SEMDataTrain(
'../data/train/images', '../data/train/masks')
# TO DO: finish test data loading
SEM_test = SEMDataTest(
'../data/test/images/')
SEM_val = SEMDataVal(
'../data/val/images', '../data/val/masks')
# Dataset end
# Dataloader begins
SEM_train_load = \
torch.utils.data.DataLoader(dataset=SEM_train,
num_workers=16, batch_size=2, shuffle=True)
SEM_val_load = \
torch.utils.data.DataLoader(dataset=SEM_val,
num_workers=3, batch_size=1, shuffle=True)
SEM_test_load = \
torch.utils.data.DataLoader(dataset=SEM_test,
num_workers=3, batch_size=1, shuffle=False)
# Dataloader end
# Model
model = CleanU_Net(in_channels=1, out_channels=2)
#model = CleanU_Net()
model = torch.nn.DataParallel(model, device_ids=list(
range(torch.cuda.device_count()))).cuda()
# Loss function
criterion = nn.CrossEntropyLoss()
# Optimizerd
optimizer = torch.optim.RMSprop(model.module.parameters(), lr=0.001)
# Parameters
epoch_start = 0
epoch_end = 2000
# Saving History to csv
header = ['epoch', 'train loss', 'train acc', 'val loss', 'val acc']
save_file_name = "../history/RMS/history_RMS3.csv"
save_dir = "../history/RMS"
# Saving images and models directories
model_save_dir = "../history/RMS/saved_models3"
image_save_path = "../history/RMS/result_images3"
# Train
print("Initializing Training!")
for i in range(epoch_start, epoch_end):
# train the model
train_model(model, SEM_train_load, criterion, optimizer)
train_acc, train_loss = get_loss_train(model, SEM_train_load, criterion)
#train_loss = train_loss / len(SEM_train)
print('Epoch', str(i+1), 'Train loss:', train_loss, "Train acc", train_acc)
# Validation every 5 epoch
if (i+1) % 5 == 0:
val_acc, val_loss = validate_model(
model, SEM_val_load, criterion, i+1, True, image_save_path)
print('Val loss:', val_loss, "val acc:", val_acc)
values = [i+1, train_loss, train_acc, val_loss, val_acc]
export_history(header, values, save_dir, save_file_name)
if (i+1) % 100 == 0: # save model every 10 epoch
save_models(model, model_save_dir, i+1)
"""
# Test
print("generate test prediction")
test_model("../history/RMS/saved_models/model_epoch_440.pwf",
SEM_test_load, 440, "../history/RMS/result_images_test")
"""
================================================
FILE: src/mean_std.py
================================================
import numpy as np
from PIL import Image
import glob
def normalize_image(image):
"""
Args:
image : a string of name of image file
Return:
image_asarray : numpy array of the image
that is normalized by being divided by 255
"""
img_opened = Image.open(image)
img_asarray = np.asarray(img_opened)
img_asarray = img_asarray / 255
return img_asarray
def find_mean(image_path):
"""
Args:
image_path : pathway of all images
Return :
mean : mean value of all the images
"""
all_images = glob.glob(str(image_path) + str("/*"))
num_images = len(all_images)
mean_sum = 0
for image in all_images:
img_asarray = normalize_image(image)
individual_mean = np.mean(img_asarray)
mean_sum += individual_mean
# Divide the sum of all values by the number of images present
mean = mean_sum / num_images
return mean
def find_stdev(image_path):
"""
Args:
image_path : pathway of all images
Return :
stdev : standard deviation of all pixels
"""
# Initiation
all_images = glob.glob(str(image_path) + str("/*"))
num_images = len(all_images)
# Recall mean value from function above: def Mean(path)
mean_value = find_mean(image_path)
std_sum = 0
for image in all_images:
img_asarray = normalize_image(image)
individual_stdev = np.std(img_asarray)
std_sum += individual_stdev
std = std_sum / num_images
return std
# Experimenting
if __name__ == '__main__':
image_path = '../data/train/images'
print('for training images,')
print('mean:', find_mean(image_path))
print('stdev:', find_stdev(image_path))
================================================
FILE: src/modules.py
================================================
import numpy as np
from PIL import Image
import torch
import torch.nn as nn
from torch.autograd import Variable
from dataset import *
import torch.nn as nn
from accuracy import accuracy_check, accuracy_check_for_batch
import csv
import os
def train_model(model, data_train, criterion, optimizer):
"""Train the model and report validation error with training error
Args:
model: the model to be trained
criterion: loss function
data_train (DataLoader): training dataset
"""
model.train()
for batch, (images, masks) in enumerate(data_train):
images = Variable(images.cuda())
masks = Variable(masks.cuda())
outputs = model(images)
# print(masks.shape, outputs.shape)
loss = criterion(outputs, masks)
optimizer.zero_grad()
loss.backward()
# Update weights
optimizer.step()
# total_loss = get_loss_train(model, data_train, criterion)
def get_loss_train(model, data_train, criterion):
"""
Calculate loss over train set
"""
model.eval()
total_acc = 0
total_loss = 0
for batch, (images, masks) in enumerate(data_train):
with torch.no_grad():
images = Variable(images.cuda())
masks = Variable(masks.cuda())
outputs = model(images)
loss = criterion(outputs, masks)
preds = torch.argmax(outputs, dim=1).float()
acc = accuracy_check_for_batch(masks.cpu(), preds.cpu(), images.size()[0])
total_acc = total_acc + acc
total_loss = total_loss + loss.cpu().item()
return total_acc/(batch+1), total_loss/(batch + 1)
def validate_model(model, data_val, criterion, epoch, make_prediction=True, save_folder_name='prediction'):
"""
Validation run
"""
# calculating validation loss
total_val_loss = 0
total_val_acc = 0
for batch, (images_v, masks_v, original_msk) in enumerate(data_val):
stacked_img = torch.Tensor([]).cuda()
for index in range(images_v.size()[1]):
with torch.no_grad():
image_v = Variable(images_v[:, index, :, :].unsqueeze(0).cuda())
mask_v = Variable(masks_v[:, index, :, :].squeeze(1).cuda())
# print(image_v.shape, mask_v.shape)
output_v = model(image_v)
total_val_loss = total_val_loss + criterion(output_v, mask_v).cpu().item()
# print('out', output_v.shape)
output_v = torch.argmax(output_v, dim=1).float()
stacked_img = torch.cat((stacked_img, output_v))
if make_prediction:
im_name = batch # TODO: Change this to real image name so we know
pred_msk = save_prediction_image(stacked_img, im_name, epoch, save_folder_name)
acc_val = accuracy_check(original_msk, pred_msk)
total_val_acc = total_val_acc + acc_val
return total_val_acc/(batch + 1), total_val_loss/((batch + 1)*4)
def test_model(model_path, data_test, epoch, save_folder_name='prediction'):
"""
Test run
"""
model = torch.load(model_path)
model = torch.nn.DataParallel(model, device_ids=list(
range(torch.cuda.device_count()))).cuda()
model.eval()
for batch, (images_t) in enumerate(data_test):
stacked_img = torch.Tensor([]).cuda()
for index in range(images_t.size()[1]):
with torch.no_grad():
image_t = Variable(images_t[:, index, :, :].unsqueeze(0).cuda())
# print(image_v.shape, mask_v.shape)
output_t = model(image_t)
output_t = torch.argmax(output_t, dim=1).float()
stacked_img = torch.cat((stacked_img, output_t))
im_name = batch # TODO: Change this to real image name so we know
_ = save_prediction_image(stacked_img, im_name, epoch, save_folder_name)
print("Finish Prediction!")
def save_prediction_image(stacked_img, im_name, epoch, save_folder_name="result_images", save_im=True):
"""save images to save_path
Args:
stacked_img (numpy): stacked cropped images
save_folder_name (str): saving folder name
"""
div_arr = division_array(388, 2, 2, 512, 512)
img_cont = image_concatenate(stacked_img.cpu().data.numpy(), 2, 2, 512, 512)
img_cont = polarize((img_cont)/div_arr)*255
img_cont_np = img_cont.astype('uint8')
img_cont = Image.fromarray(img_cont_np)
# organize images in every epoch
desired_path = save_folder_name + '/epoch_' + str(epoch) + '/'
# Create the path if it does not exist
if not os.path.exists(desired_path):
os.makedirs(desired_path)
# Save Image!
export_name = str(im_name) + '.png'
img_cont.save(desired_path + export_name)
return img_cont_np
def polarize(img):
''' Polarize the value to zero and one
Args:
img (numpy): numpy array of image to be polarized
return:
img (numpy): numpy array only with zero and one
'''
img[img >= 0.5] = 1
img[img < 0.5] = 0
return img
"""
def test_SEM(model, data_test, folder_to_save):
'''Test the model with test dataset
Args:
model: model to be tested
data_test (DataLoader): test dataset
folder_to_save (str): path that the predictions would be saved
'''
for i, (images) in enumerate(data_test):
print(images)
stacked_img = torch.Tensor([])
for j in range(images.size()[1]):
image = Variable(images[:, j, :, :].unsqueeze(0).cuda())
output = model(image.cuda())
print(output)
print("size", output.size())
output = torch.argmax(output, dim=1).float()
print("size", output.size())
stacked_img = torch.cat((stacked_img, output))
div_arr = division_array(388, 2, 2, 512, 512)
print(stacked_img.size())
img_cont = image_concatenate(stacked_img.data.numpy(), 2, 2, 512, 512)
final_img = (img_cont*255/div_arr)
print(final_img)
final_img = final_img.astype("uint8")
break
return final_img
"""
if __name__ == '__main__':
SEM_train = SEMDataTrain(
'../data/train/images', '../data/train/masks')
SEM_train_load = torch.utils.data.DataLoader(dataset=SEM_train,
num_workers=3, batch_size=10, shuffle=True)
get_loss_train()
================================================
FILE: src/post_processing.py
================================================
import numpy as np
from matplotlib import pyplot as plt
def postprocess(image_path):
''' postprocessing of the prediction output
Args
image_path : path of the image
Returns
watershed_grayscale : numpy array of postprocessed image (in grayscale)
'''
# Bring in the image
img_original = cv2.imread(image_path)
img = cv2.imread(image_path)
# In case the input image has 3 channels (RGB), convert to 1 channel (grayscale)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# Use threshold => Image will have values either 0 or 255 (black or white)
ret, bin_image = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)
# Remove Hole or noise through the use of opening, closing in Morphology module
kernel = np.ones((1, 1), np.uint8)
kernel1 = np.ones((3, 3), np.uint8)
# remove noise in
closing = cv2.morphologyEx(bin_image, cv2.MORPH_CLOSE, kernel, iterations=1)
# make clear distinction of the background
# Incerease/emphasize the white region.
sure_bg = cv2.dilate(closing, kernel1, iterations=1)
# calculate the distance to the closest zero pixel for each pixel of the source.
# Adjust the threshold value with respect to the maximum distance. Lower threshold, more information.
dist_transform = cv2.distanceTransform(closing, cv2.DIST_L2, 5)
ret, sure_fg = cv2.threshold(dist_transform, 0.2*dist_transform.max(), 255, 0)
sure_fg = np.uint8(sure_fg)
# Unknown is the region of background with foreground excluded.
unknown = cv2.subtract(sure_bg, sure_fg)
# labelling on the foreground.
ret, markers = cv2.connectedComponents(sure_fg)
markers_plus1 = markers + 1
markers_plus1[unknown == 255] = 0
# Appy watershed and label the borders
markers_watershed = cv2.watershed(img, markers_plus1)
# See the watershed result in a clear white page.
img_x, img_y = img_original.shape[0], img_original.shape[1] # 512x512
white, white_color = np.zeros((img_x, img_y, 3)), np.zeros((img_x, img_y, 3))
white += 255
white_color += 255
# 1 in markers_watershed indicate the background value
# label everything not indicated as background value
white[markers_watershed != 1] = [0, 0, 0] # grayscale version
white_color[markers_watershed != 1] = [255, 0, 0] # RGB version
# Convert to numpy array for later processing
white_np = np.asarray(white) # 512x512x3
watershed_grayscale = white_np.transpose(2, 0, 1)[0, :, :] # convert to 1 channel (grayscale)
img[markers_watershed != 1] = [255, 0, 0]
return watershed_grayscale
'''
Visualizing all the intermediate processes
images = [img_original, gray,bin_image, closing, sure_bg, dist_transform, sure_fg, unknown, markers, markers_watershed, white_color, white, img]
titles = ['Original', '1. Grayscale','2. Binary','3. Closing','Sure BG','Distance','Sure FG','Unknown','Markers', 'Markers_Watershed','Result', 'Result gray','Result Overlapped']
CMAP = [None, 'gray', 'gray','gray','gray',None,'gray','gray',None, None, None, None,'gray']
for i in range(len(images)):
plt.subplot(4,4,i+1),plt.imshow(images[i], cmap=CMAP[i]),plt.title(titles[i]),plt.xticks([]),plt.yticks([])
plt.show()
'''
if __name__ == '__main__':
from PIL import Image
print(postprocess('../data/train/masks/25.png'))
================================================
FILE: src/pre_processing.py
================================================
import numpy as np
from scipy.ndimage.interpolation import map_coordinates
from scipy.ndimage.filters import gaussian_filter
from random import randint
def add_elastic_transform(image, alpha, sigma, pad_size=30, seed=None):
"""
Args:
image : numpy array of image
alpha : α is a scaling factor
sigma : σ is an elasticity coefficient
random_state = random integer
Return :
image : elastically transformed numpy array of image
"""
image_size = int(image.shape[0])
image = np.pad(image, pad_size, mode="symmetric")
if seed is None:
seed = randint(1, 100)
random_state = np.random.RandomState(seed)
else:
random_state = np.random.RandomState(seed)
shape = image.shape
dx = gaussian_filter((random_state.rand(*shape) * 2 - 1),
sigma, mode="constant", cval=0) * alpha
dy = gaussian_filter((random_state.rand(*shape) * 2 - 1),
sigma, mode="constant", cval=0) * alpha
x, y = np.meshgrid(np.arange(shape[1]), np.arange(shape[0]))
indices = np.reshape(y+dy, (-1, 1)), np.reshape(x+dx, (-1, 1))
return cropping(map_coordinates(image, indices, order=1).reshape(shape), 512, pad_size, pad_size), seed
def flip(image, option_value):
"""
Args:
image : numpy array of image
option_value = random integer between 0 to 3
Return :
image : numpy array of flipped image
"""
if option_value == 0:
# vertical
image = np.flip(image, option_value)
elif option_value == 1:
# horizontal
image = np.flip(image, option_value)
elif option_value == 2:
# horizontally and vertically flip
image = np.flip(image, 0)
image = np.flip(image, 1)
else:
image = image
# no effect
return image
def add_gaussian_noise(image, mean=0, std=1):
"""
Args:
image : numpy array of image
mean : pixel mean of image
standard deviation : pixel standard deviation of image
Return :
image : numpy array of image with gaussian noise added
"""
gaus_noise = np.random.normal(mean, std, image.shape)
image = image.astype("int16")
noise_img = image + gaus_noise
image = ceil_floor_image(image)
return noise_img
def add_uniform_noise(image, low=-10, high=10):
"""
Args:
image : numpy array of image
low : lower boundary of output interval
high : upper boundary of output interval
Return :
image : numpy array of image with uniform noise added
"""
uni_noise = np.random.uniform(low, high, image.shape)
image = image.astype("int16")
noise_img = image + uni_noise
image = ceil_floor_image(image)
return noise_img
def change_brightness(image, value):
"""
Args:
image : numpy array of image
value : brightness
Return :
image : numpy array of image with brightness added
"""
image = image.astype("int16")
image = image + value
image = ceil_floor_image(image)
return image
def ceil_floor_image(image):
"""
Args:
image : numpy array of image in datatype int16
Return :
image : numpy array of image in datatype uint8 with ceilling(maximum 255) and flooring(minimum 0)
"""
image[image > 255] = 255
image[image < 0] = 0
image = image.astype("uint8")
return image
def approximate_image(image):
"""
Args:
image : numpy array of image in datatype int16
Return :
image : numpy array of image in datatype uint8 only with 255 and 0
"""
image[image > 127.5] = 255
image[image < 127.5] = 0
image = image.astype("uint8")
return image
def normalization1(image, mean, std):
""" Normalization using mean and std
Args :
image : numpy array of image
mean :
Return :
image : numpy array of image with values turned into standard scores
"""
image = image / 255 # values will lie between 0 and 1.
image = (image - mean) / std
return image
def normalization2(image, max, min):
"""Normalization to range of [min, max]
Args :
image : numpy array of image
mean :
Return :
image : numpy array of image with values turned into standard scores
"""
image_new = (image - np.min(image))*(max - min)/(np.max(image)-np.min(image)) + min
return image_new
def stride_size(image_len, crop_num, crop_size):
"""return stride size
Args :
image_len(int) : length of one size of image (width or height)
crop_num(int) : number of crop in certain direction
crop_size(int) : size of crop
Return :
stride_size(int) : stride size
"""
return int((image_len - crop_size)/(crop_num - 1))
def multi_cropping(image, crop_size, crop_num1, crop_num2):
"""crop the image and pad it to in_size
Args :
images : numpy arrays of images
crop_size(int) : size of cropped image
crop_num2 (int) : number of crop in horizontal way
crop_num1 (int) : number of crop in vertical way
Return :
cropped_imgs : numpy arrays of stacked images
"""
img_height, img_width = image.shape[0], image.shape[1]
assert crop_size*crop_num1 >= img_width and crop_size * \
crop_num2 >= img_height, "Whole image cannot be sufficiently expressed"
assert crop_num1 <= img_width - crop_size + 1 and crop_num2 <= img_height - \
crop_size + 1, "Too many number of crops"
cropped_imgs = []
# int((img_height - crop_size)/(crop_num1 - 1))
dim1_stride = stride_size(img_height, crop_num1, crop_size)
# int((img_width - crop_size)/(crop_num2 - 1))
dim2_stride = stride_size(img_width, crop_num2, crop_size)
for i in range(crop_num1):
for j in range(crop_num2):
cropped_imgs.append(cropping(image, crop_size,
dim1_stride*i, dim2_stride*j))
return np.asarray(cropped_imgs)
# IT IS NOT USED FOR PAD AND CROP DATA OPERATION
# IF YOU WANT TO USE CROP AND PAD USE THIS FUNCTION
"""
def multi_padding(images, in_size, out_size, mode):
'''Pad the images to in_size
Args :
images : numpy array of images (CxHxW)
in_size(int) : the input_size of model (512)
out_size(int) : the output_size of model (388)
mode(str) : mode of padding
Return :
padded_imgs: numpy arrays of padded images
'''
pad_size = int((in_size - out_size)/2)
padded_imgs = []
for num in range(images.shape[0]):
padded_imgs.append(add_padding(images[num], in_size, out_size, mode=mode))
return np.asarray(padded_imgs)
"""
def cropping(image, crop_size, dim1, dim2):
"""crop the image and pad it to in_size
Args :
images : numpy array of images
crop_size(int) : size of cropped image
dim1(int) : vertical location of crop
dim2(int) : horizontal location of crop
Return :
cropped_img: numpy array of cropped image
"""
cropped_img = image[dim1:dim1+crop_size, dim2:dim2+crop_size]
return cropped_img
def add_padding(image, in_size, out_size, mode):
"""Pad the image to in_size
Args :
images : numpy array of images
in_size(int) : the input_size of model
out_size(int) : the output_size of model
mode(str) : mode of padding
Return :
padded_img: numpy array of padded image
"""
pad_size = int((in_size - out_size)/2)
padded_img = np.pad(image, pad_size, mode=mode)
return padded_img
def division_array(crop_size, crop_num1, crop_num2, dim1, dim2):
"""Make division array
Args :
crop_size(int) : size of cropped image
crop_num2 (int) : number of crop in horizontal way
crop_num1 (int) : number of crop in vertical way
dim1(int) : vertical size of output
dim2(int) : horizontal size_of_output
Return :
div_array : numpy array of numbers of 1,2,4
"""
div_array = np.zeros([dim1, dim2]) # make division array
one_array = np.ones([crop_size, crop_size]) # one array to be added to div_array
dim1_stride = stride_size(dim1, crop_num1, crop_size) # vertical stride
dim2_stride = stride_size(dim2, crop_num2, crop_size) # horizontal stride
for i in range(crop_num1):
for j in range(crop_num2):
# add ones to div_array at specific position
div_array[dim1_stride*i:dim1_stride*i + crop_size,
dim2_stride*j:dim2_stride*j + crop_size] += one_array
return div_array
def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
"""concatenate images
Args :
image : output images (should be square)
crop_num2 (int) : number of crop in horizontal way (2)
crop_num1 (int) : number of crop in vertical way (2)
dim1(int) : vertical size of output (512)
dim2(int) : horizontal size_of_output (512)
Return :
div_array : numpy arrays of numbers of 1,2,4
"""
crop_size = image.shape[1] # size of crop
empty_array = np.zeros([dim1, dim2]).astype("float64") # to make sure no overflow
dim1_stride = stride_size(dim1, crop_num1, crop_size) # vertical stride
dim2_stride = stride_size(dim2, crop_num2, crop_size) # horizontal stride
index = 0
for i in range(crop_num1):
for j in range(crop_num2):
# add image to empty_array at specific position
empty_array[dim1_stride*i:dim1_stride*i + crop_size,
dim2_stride*j:dim2_stride*j + crop_size] += image[index]
index += 1
return empty_array
if __name__ == "__main__":
from PIL import Image
b = Image.open("../data/train/images/14.png")
c = Image.open("../data/train/masks/14.png")
original = np.array(b)
originall = np.array(c)
original_norm = normalization(original, max=1, min=0)
print(original_norm)
b = Image.open("../readme_images/original.png")
original = np.array(b)
"""
original1 = add_gaussian_noise(original, 0, 100)
original1 = Image.fromarray(original1)
original1.show()
"""
original1 = add_uniform_noise(original, -100, 100)
original1 = Image.fromarray(original1)
original1.show()
"""
original1 = change_brightness(original, 50)
original1 = Image.fromarray(original1)
original1.show()
original1 = add_elastic_transform(original, 10, 4, 1)[0]
original1 = Image.fromarray(original1)
original1.show()
"""
================================================
FILE: src/result_visualization.py
================================================
import matplotlib.pyplot as plt
import matplotlib as mpl
import pandas as pd
import numpy as np
'''
For members who did not yet install the module "matplotlib",
python3 -mpip install -U matplotlib
installation of tkinter is a prerequisite. If you do not have it,
sudo apt install python3.6-tk
Now you won't have problems running this python file.
'''
def plotloss(csvfile):
'''
Args
csvfile: name of the csv file
Returns
graph_loss: trend of loss values over epoch
'''
# Bring in the csv file
loss_values = pd.read_csv(csvfile)
# Initiation
epoch = loss_values.iloc[:, 0]
tr_loss = loss_values.iloc[:, 1]
tr_acc = loss_values.iloc[:, 2]
val_loss = np.asarray(loss_values.iloc[:, 3])
val_acc = np.asarray(loss_values.iloc[:, 4])
# Reduce the volume of data
epoch_skip = epoch[::5]
tr_loss_skip = tr_loss[::5]
tr_acc_skip = tr_acc[::5]
val_loss_skip = val_loss[::5]
val_acc_skip = val_acc[::5]
fig, ax1 = plt.subplots(figsize=(8, 6))
ax2 = ax1.twinx()
# Label and color the axes
ax1.set_xlabel('Epoch', fontsize=16)
ax1.set_ylabel('Loss', fontsize=16, color='black')
ax2.set_ylabel('Accuracy', fontsize=16, color='black')
# Plot valid/train losses
ax1.plot(epoch_skip, tr_loss_skip, linewidth=2,
ls='--', color='#c92508', label='Train loss')
ax1.plot(epoch_skip, val_loss_skip, linewidth=2,
color='#c92508', label='Validation loss')
ax1.spines['left'].set_color('#f23d1d')
# Coloring the ticks
for label in ax1.get_yticklabels():
label.set_color('#c92508')
label.set_size(12)
# Plot valid/trian accuracy
ax2.plot(epoch_skip, tr_acc_skip, linewidth=2, ls='--',
color='#2348ff', label='Train Accuracy')
ax2.plot(epoch_skip, val_acc_skip, linewidth=2,
color='#2348ff', label='Validation Accuracy')
ax2.spines['right'].set_color('#2348ff')
# Coloring the ticks
for label in ax2.get_yticklabels():
label.set_color('#2348ff')
label.set_size(12)
# Manually setting the y-axis ticks
yticks = [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1]
ax1.set_yticks(yticks)
ax2.set_yticks(yticks)
for label in ax1.get_xticklabels():
label.set_size(12)
# Modification of the overall graph
fig.legend(ncol=4, loc=9, fontsize=12)
plt.xlim(xmin=0)
ax2.set_ylim(ymax=1, ymin=0)
ax1.set_ylim(ymax=1, ymin=0)
plt.xlabel('epochs')
plt.title("Adam optimizer", weight="bold")
plt.grid(True, axis='y')
# return train_loss, valid_loss
if __name__ == '__main__':
file = '../history/csv/Adam.csv'
#file = '../history/SGD/history_SGD4.csv'
plt.show(plotloss(file))
================================================
FILE: src/save_history.py
================================================
import os
import csv
import torch
def export_history(header, value, folder, file_name):
""" export data to csv format
Args:
header (list): headers of the column
value (list): values of correspoding column
folder (list): folder path
file_name: file name with path
"""
# if folder does not exists make folder
if not os.path.exists(folder):
os.makedirs(folder)
file_existence = os.path.isfile(file_name)
# if there is no file make file
if file_existence == False:
file = open(file_name, 'w', newline='')
writer = csv.writer(file)
writer.writerow(header)
writer.writerow(value)
# if there is file overwrite
else:
file = open(file_name, 'a', newline='')
writer = csv.writer(file)
writer.writerow(value)
# close file when it is done with writing
file.close()
def save_models(model, path, epoch):
"""Save model to given path
Args:
model: model to be saved
path: path that the model would be saved
epoch: the epoch the model finished training
"""
if not os.path.exists(path):
os.makedirs(path)
torch.save(model, path+"/model_epoch_{0}.pwf".format(epoch))
================================================
FILE: src/simple_model.py
================================================
import torch
import torch.nn as nn
from torch.autograd import Variable
import numpy as np
from PIL import Image
from torch.nn.functional import sigmoid
class CleanU_Net(nn.Module):
def __init__(self):
super(CleanU_Net, self).__init__()
# Conv block 1 - Down 1
self.conv1_block = nn.Sequential(
nn.Conv2d(in_channels=1, out_channels=32,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(in_channels=32, out_channels=32,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
)
self.max1 = nn.MaxPool2d(kernel_size=2, stride=2)
# Conv block 2 - Down 2
self.conv2_block = nn.Sequential(
nn.Conv2d(in_channels=32, out_channels=64,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(in_channels=64, out_channels=64,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
)
self.max2 = nn.MaxPool2d(kernel_size=2, stride=2)
# Conv block 3 - Down 3
self.conv3_block = nn.Sequential(
nn.Conv2d(in_channels=64, out_channels=128,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(in_channels=128, out_channels=128,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
)
self.max3 = nn.MaxPool2d(kernel_size=2, stride=2)
# Conv block 4 - Down 4
self.conv4_block = nn.Sequential(
nn.Conv2d(in_channels=128, out_channels=256,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(in_channels=256, out_channels=256,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
)
self.max4 = nn.MaxPool2d(kernel_size=2, stride=2)
# Conv block 5 - Down 5
self.conv5_block = nn.Sequential(
nn.Conv2d(in_channels=256, out_channels=512,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(in_channels=512, out_channels=512,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
)
# Up 1
self.up_1 = nn.ConvTranspose2d(in_channels=512, out_channels=256, kernel_size=2, stride=2)
# Up Conv block 1
self.conv_up_1 = nn.Sequential(
nn.Conv2d(in_channels=512, out_channels=256,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(in_channels=256, out_channels=256,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
)
# Up 2
self.up_2 = nn.ConvTranspose2d(in_channels=256, out_channels=128, kernel_size=2, stride=2)
# Up Conv block 2
self.conv_up_2 = nn.Sequential(
nn.Conv2d(in_channels=256, out_channels=128,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(in_channels=128, out_channels=128,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
)
# Up 3
self.up_3 = nn.ConvTranspose2d(in_channels=128, out_channels=64, kernel_size=2, stride=2)
# Up Conv block 3
self.conv_up_3 = nn.Sequential(
nn.Conv2d(in_channels=128, out_channels=64,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(in_channels=64, out_channels=64,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
)
# Up 4
self.up_4 = nn.ConvTranspose2d(in_channels=64, out_channels=32, kernel_size=2, stride=2)
# Up Conv block 4
self.conv_up_4 = nn.Sequential(
nn.Conv2d(in_channels=64, out_channels=32,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
nn.Conv2d(in_channels=32, out_channels=32,
kernel_size=3, padding=0, stride=1),
nn.ReLU(inplace=True),
)
# Final output
self.conv_final = nn.Conv2d(in_channels=32, out_channels=2,
kernel_size=1, padding=0, stride=1)
def forward(self, x):
# print('input', x.shape)
# Down 1
x = self.conv1_block(x)
# print('after conv1', x.shape)
conv1_out = x # Save out1
conv1_dim = x.shape[2]
x = self.max1(x)
# print('before conv2', x.shape)
# Down 2
x = self.conv2_block(x)
# print('after conv2', x.shape)
conv2_out = x
conv2_dim = x.shape[2]
x = self.max2(x)
# print('before conv3', x.shape)
# Down 3
x = self.conv3_block(x)
# print('after conv3', x.shape)
conv3_out = x
conv3_dim = x.shape[2]
x = self.max3(x)
# print('before conv4', x.shape)
# Down 4
x = self.conv4_block(x)
# print('after conv5', x.shape)
conv4_out = x
conv4_dim = x.shape[2]
x = self.max4(x)
# Midpoint
x = self.conv5_block(x)
# Up 1
x = self.up_1(x)
# print('up_1', x.shape)
lower = int((conv4_dim - x.shape[2]) / 2)
upper = int(conv4_dim - lower)
conv4_out_modified = conv4_out[:, :, lower:upper, lower:upper]
x = torch.cat([x, conv4_out_modified], dim=1)
# print('after cat_1', x.shape)
x = self.conv_up_1(x)
# print('after conv_1', x.shape)
# Up 2
x = self.up_2(x)
# print('up_2', x.shape)
lower = int((conv3_dim - x.shape[2]) / 2)
upper = int(conv3_dim - lower)
conv3_out_modified = conv3_out[:, :, lower:upper, lower:upper]
x = torch.cat([x, conv3_out_modified], dim=1)
# print('after cat_2', x.shape)
x = self.conv_up_2(x)
# print('after conv_2', x.shape)
# Up 3
x = self.up_3(x)
# print('up_3', x.shape)
lower = int((conv2_dim - x.shape[2]) / 2)
upper = int(conv2_dim - lower)
conv2_out_modified = conv2_out[:, :, lower:upper, lower:upper]
x = torch.cat([x, conv2_out_modified], dim=1)
# print('after cat_3', x.shape)
x = self.conv_up_3(x)
# print('after conv_3', x.shape)
# Up 4
x = self.up_4(x)
# print('up_4', x.shape)
lower = int((conv1_dim - x.shape[2]) / 2)
upper = int(conv1_dim - lower)
conv1_out_modified = conv1_out[:, :, lower:upper, lower:upper]
x = torch.cat([x, conv1_out_modified], dim=1)
# print('after cat_4', x.shape)
x = self.conv_up_4(x)
# print('after conv_4', x.shape)
# Final output
x = self.conv_final(x)
return x
if __name__ == "__main__":
# A full forward pass
im = torch.randn(1, 1, 572, 572)
model = CleanU_Net()
x = model(im)
# print(x.shape)
del model
del x
# print(x.shape)