[ { "path": ".gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*.csv\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2018 UGent Korea\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "# pytorch-unet-segmentation\n\n**Members** : PyeongEun Kim, JuHyung Lee, MiJeong Lee \n\n**Supervisors** : Utku Ozbulak, Wesley De Neve\n\n## Description\n\nThis project aims to implement biomedical image segmentation with the use of U-Net model. The below image briefly explains the output we want:\n\n

\n\n\n\nThe dataset we used is Transmission Electron Microscopy (ssTEM) data set of the Drosophila first instar larva ventral nerve cord (VNC), which is dowloaded from [ISBI Challenge: Segmentation of of neural structures in EM stacks](http://brainiac2.mit.edu/isbi_challenge/home)\n\nThe dataset contains 30 images (.png) of size 512x512 for each train, train-labels and test.\n\n\n## Table of Content\n\n* [Dataset](#dataset)\n\n* [Preprocessing](#preprocessing)\n\n* [Model](#model)\n\n* [Loss function](#lossfunction)\n\n* [Post-processing](#postprocessing)\n\n* [Results](#results)\n\n* [Dependency](#dependency)\n\n* [References](#references)\n\n\n\n## Dataset \n\n```ruby\nclass SEMDataTrain(Dataset):\n\n def __init__(self, image_path, mask_path, in_size=572, out_size=388):\n \"\"\"\n Args:\n image_path (str): the path where the image is located\n mask_path (str): the path where the mask is located\n option (str): decide which dataset to import\n \"\"\"\n # All file names\n\t# Lists of image path and list of labels\n # Calculate len\n # Calculate mean and stdev\n\n def __getitem__(self, index):\n \"\"\"Get specific data corresponding to the index\n Args:\n index (int): index of the data\n\n Returns:\n Tensor: specific data on index which is converted to Tensor\n \"\"\"\n \"\"\"\n # GET IMAGE\n \"\"\"\n #Augmentation on image\n # Flip \n # Gaussian_noise\n # Uniform_noise\n # Brightness\n # Elastic distort {0: distort, 1:no distort}\n # Crop the image\n # Pad the image\n # Sanity Check for Cropped image\n # Normalize the image\n\n # Add additional dimension\n # Convert numpy array to tensor\n \n #Augmentation on mask\n # Flip same way with image\n # Elastic distort same way with image\n # Crop the same part that was cropped on image\n # Sanity Check\n # Normalize the mask to 0 and 1\n \n # Add additional dimension\n # Convert numpy array to tensor\n\n return (img_as_tensor, msk_as_tensor)\n\n def __len__(self):\n \"\"\"\n Returns:\n length (int): length of the data\n \"\"\"\n\n```\n\n## Preprocessing \n\nWe preprocessed the images for data augmentation. Following preprocessing are :\n * Flip\n * Gaussian noise\n * Uniform noise\n * Brightness\n * Elastic deformation\n * Crop\n * Pad \n \n#### Image Augmentation\n\n\n

\n
Original Image\n

\n\n\n\n\t \n \t\t\n\t\t\n\t\t\n\t\t\t \n\t\t\t \n\t\t\t\n\t\t\t\n\t\t\n \t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n \t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\n
Image
Flip
Vertical
Horizontal
Both
Gaussian noise
Standard Deviation: 10
Standard Deviation: 50
Standard Deviation: 100
Uniform noise
Intensity: 10
Intensity: 50
Intensity: 100
Brightness
Intensity: 10
Intensity: 20
Intensity: 30
Elastic deformation
Random Deformation: 1
Random Deformation: 2
Random Deformation: 3
\n\n#### Crop and Pad\n\n\n\t \n \t\t\n\t \n\t\t\n\t\t\t\n\t\t\t \n\t\t\t\n\t\t\t \n\t\t\n \t\t\n\t\n
Crop

Left Bottom
Left Top
Right Bottom
Right Top
\n\nPadding process is compulsory after the cropping process as the image has to fit the input size of the U-Net model. \n\nIn terms of the padding method, **symmetric padding** was done in which the pad is the reflection of the vector mirrored along the edge of the array. We selected the symmetric padding over several other padding options because it reduces the loss the most. \n\nTo help with observation, a ![#ffff00](https://placehold.it/15/ffff00/000000?text=+) 'yellow border' is added around the original image: outside the border indicates symmetric padding whereas inside indicates the original image.\n\n\n\t \n \t\t\n\t \n\t\t\n\t\t\t\n\t\t\t \n\t\t\t\n\t\t\t \n\t\t\n \t\t\n\t\n
Pad

Left Bottom
Left Top
Right bottom
Right Top
\n\n\n## Model \n\n#### Architecture\n\nWe have same structure as U-Net Model architecture but we made a small modification to make the model smaller.\n\n![image](https://github.com/ugent-korea/pytorch-unet-segmentation/blob/master/readme_images/UNet_custom_parameter.png)\n\n## Loss function \n\nWe used a loss function where pixel-wise softmax is combined with cross entropy.\n\n#### Softmax\n![image](https://github.com/ugent-korea/pytorch-unet-segmentation/blob/master/readme_images/softmax(1).png)\n\n#### Cross entropy\n![image](https://github.com/ugent-korea/pytorch-unet-segmentation/blob/master/readme_images/cross%20entropy(1).png)\n\n## Post-processing \nIn attempt of reducing the loss, we did a post-processing on the prediction results. We applied the concept of watershed segmentation in order to point out the certain foreground regions and remove regions in the prediction image which seem to be noises.\n\n![postprocessing](https://github.com/ugent-korea/pytorch-unet-segmentation/blob/master/readme_images/postprocess.png)\n\nThe numbered images in the figure above indicates the stpes we took in the post-processing. To name those steps in slightly more detail:\n\n\t* 1. Convertion into grayscale\n\t* 2. Conversion into binary image\n\t* 3. Morphological transformation: Closing\n\t* 4. Determination of the certain background\n\t* 5. Calculation of the distance\n\t* 6. Determination of the certain foreground\n\t* 7. Determination of the unknown region\n\t* 8. Application of watershed\n\t* 9. Determination of the final result\n\n### Conversion into grayscale \n\nThe first step is there just in case the input image has more than 1 color channel (e.g. RGB image has 3 channels) \n\n### Conversion into binary image\n\nConvert the gray-scale image into binary image by processing the image with a threshold value: pixels equal to or lower than 127 will be pushed down to 0 and greater will be pushed up to 255. Such process is compulsory as later transformation processes takes in binary images.\n\n### Morphological transformation: Closing.\n\nWe used **morphologyEX()** function in cv2 module which removes black noises (background) within white regions (foreground).\n\t\n### Determination of the certain background\n\nWe used **dilate()** function in cv2 module which emphasizes/increases the white region (foreground). By doing so, we connect detached white regions together - for example, connecting detached cell membranes together - to make ensure the background region.\n\n### Caculation of the distance\n\nThis step labels the foreground with a color code: ![#ff0000](https://placehold.it/15/ff0000/000000?text=+) red color indicates farthest from the background while ![#003bff](https://placehold.it/15/003bff/000000?text=+) blue color indicates closest to the background.\n\n### Determination of the foreground\n\nNow that we have an idea of how far the foreground is from the background, we apply a threshold value to decide which part could surely be the foreground.\n\nThe threshold value is the maximum distance (calculated from the previous step) multiplied by a hyper-parameter that we have to manually tune. The greater the hyper-parameter value, the greater the threshold value, and therefore we will get less area of certain foreground. \n\n### Determination of the unknown region\n\nFrom previous steps, we determined sure foreground and background regions. The rest will be classified as *'unknown'* regions.\n\n### Label the foreground: markers\n\nWe applied **connectedComponents()** function from the cv2 module on the foreground to label the foreground regions with color to distinguish different foreground objects. We named it as a 'marker'.\n\n### Application of watershed and Determination of the final result\n\nAfter applying **watershed()** function from cv2 module on the marker, we obtained an array of -1, 1, and many others. \n\n\t* -1 = Border region that distinguishes foreground and background\n\t* 1 = Background region\n\nTo see the result, we created a clean white page of the same size with the input image. then we copied all the values from the watershed result to the white page except 1, which means that we excluded the background.\n\n## Results \n\n\n\t \n\t\t\n\t \t\n\t \t\n\t \t\n\t\t\n\t \t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\t\n
OptimizerLearning RateLowest LossEpochHighest AccuracyEpoch
SGD0.0010.19697214450.9210321855
0.0050.20580218150.9184251795
0.010.1933284500.922908450
RMS_prop0.00010.2034311850.924543230
0.00020.1934562700.926245500
0.0010.26824616550.8822291915
Adam0.00010.1941801400.924470300
0.00050.1852121350.925519135
0.0010.2222771650.912364180
\n\n\nWe chose the best learning rate that fits the optimizer based on **how fast the model converges to the lowest error**. In other word, the learning rate should make model to reach optimal solution in shortest epoch repeated. However, the intersting fact was that the epochs of lowest loss and highest accuracy were not corresponding. This might be due to the nature of loss function (Loss function is log scale, thus an extreme deviation might occur). For example, if the softmax probability of one pixel is 0.001, then the -log(0.001) would be 1000 which is a huge value that contributes to loss.\nFor consistency, we chose to focus on accuracy as our criterion of correctness of model. \n\n\n\n\n\t \n \t\t\n\t \n\t\t\n\t\t\t \n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n \t\t\n\t\n
Accuracy and Loss Graph
SGD
(lr=0.01,momentum=0.99)		RMS prop
(lr=0.0002)		Adam
(lr=0.0005)
\nWe used two different optimizers (SGD, RMS PROP, and Adam). In case of SGD the momentum is manually set (0.99) whereas in case of other optimizers (RMS Prop and Adam) it is calculated automatically. \n\n### Model Downloads\n\nModel trained with SGD can be downloaded via **dropbox**:\nhttps://www.dropbox.com/s/ge9654nhgv1namr/model_epoch_2290.pwf?dl=0\n\n\nModel trained with RMS prop can be downloaded via **dropbox**:\nhttps://www.dropbox.com/s/cdwltzhbs3tiiwb/model_epoch_440.pwf?dl=0\n\n\nModel trained with Adam can be downloaded via **dropbox**:\nhttps://www.dropbox.com/s/tpch6u41jrdgswk/model_epoch_100.pwf?dl=0\n\n\n\n\n### Example\n\n

\n
Input Image\n

\n\n\n\t \n \t\t\n\t \n\t\t\n\t\t\t\n\t\t\t\n\t\t\t \n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n \t\t\n\t\n
Results comparsion
original image maskRMS prop optimizer
(Accuracy 92.48 %)		SGD optimizer
(Accuracy 91.52 %)		Adam optimizer
(Accuracy 92.55 %)
\n\n## Dependency \n\nFollowing modules are used in the project:\n\n * python >= 3.6\n * numpy >= 1.14.5\n * torch >= 0.4.0\n * PIL >= 5.2.0\n * scipy >= 1.1.0\n * matplotlib >= 2.2.2\n \n\n## References :\n\n[1] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation, http://arxiv.org/pdf/1505.04597.pdf\n\n[2] P.Y. Simard, D. Steinkraus, J.C. Platt. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis, http://cognitivemedium.com/assets/rmnist/Simard.pdf\n" }, { "path": "readme_images/description", "content": "z_original: original image\nbright_x: brightness increased by x\nelastic_x: different elastic \nflip_x: flip on x way\ngaus_x: gaussian noise added with mean of 0 and std of x\nuniform_x: uniform noise added with lower bound of -x and upper bound of x\n\n" }, { "path": "readme_images/file_name_description", "content": "file name explanation\n\np: padded image\nc: cropped image\nlb: left bottom\nlt: left top\nrb: right bottom\nrt: right top\n\n" }, { "path": "src/accuracy.py", "content": "#from post_processing import *\nimport numpy as np\nfrom PIL import Image\nimport glob as gl\nimport numpy as np\nfrom PIL import Image\nimport torch\n\n\ndef accuracy_check(mask, prediction):\n ims = [mask, prediction]\n np_ims = []\n for item in ims:\n if 'str' in str(type(item)):\n item = np.array(Image.open(item))\n elif 'PIL' in str(type(item)):\n item = np.array(item)\n elif 'torch' in str(type(item)):\n item = item.numpy()\n np_ims.append(item)\n\n compare = np.equal(np_ims[0], np_ims[1])\n accuracy = np.sum(compare)\n\n return accuracy/len(np_ims[0].flatten())\n\n\ndef accuracy_check_for_batch(masks, predictions, batch_size):\n total_acc = 0\n for index in range(batch_size):\n total_acc += accuracy_check(masks[index], predictions[index])\n return total_acc/batch_size\n\n\n\"\"\"\ndef accuracy_compare(prediction_folder, true_mask_folder):\n ''' Output average accuracy of all prediction results and their corresponding true masks.\n Args\n prediction_folder : folder of the prediction results\n true_mask_folder : folder of the corresponding true masks\n Returns\n a tuple of (original_accuracy, posprocess_accuracy)\n '''\n\n # Bring in the images\n all_prediction = gl.glob(prediction_folder)\n all_mask = gl.glob(true_mask_folder)\n\n # Initiation\n num_files = len(all_prediction)\n count = 0\n postprocess_acc = 0\n original_acc = 0\n\n while count != num_files:\n\n # Prepare the arrays to be further processed.\n prediction_processed = postprocess(all_prediction[count])\n prediction_image = Image.open(all_prediction[count])\n mask = Image.open(all_mask[count])\n\n # converting the PIL variables into numpy array\n prediction_np = np.asarray(prediction_image)\n mask_np = np.asarray(mask)\n\n # Calculate the accuracy of original and postprocessed image\n postprocess_acc += accuracy_check(mask_np, prediction_processed)\n original_acc += accuracy_check(mask_np, prediction_np)\n # check individual accuracy\n print(str(count) + 'th post acc:', accuracy_check(mask_np, prediction_processed))\n print(str(count) + 'th original acc:', accuracy_check(mask_np, prediction_np))\n\n # Move onto the next prediction/mask image\n count += 1\n\n # Average of all the accuracies\n postprocess_acc = postprocess_acc / num_files\n original_acc = original_acc / num_files\n\n return (original_acc, postprocess_acc)\n\"\"\"\n\n# Experimenting\nif __name__ == '__main__':\n '''\n predictions = 'result/*.png'\n masks = '../data/val/masks/*.png'\n\n result = accuracy_compare(predictions, masks)\n print('Original Result :', result[0])\n print('Postprocess result :', result[1])\n '''\n" }, { "path": "src/advanced_model.py", "content": "# full assembly of the sub-parts to form the complete net\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nimport numpy as np\nfrom PIL import Image\nfrom torch.nn.functional import sigmoid\n\n\nclass Double_conv(nn.Module):\n\n '''(conv => ReLU) * 2 => MaxPool2d'''\n\n def __init__(self, in_ch, out_ch):\n \"\"\"\n Args:\n in_ch(int) : input channel\n out_ch(int) : output channel\n \"\"\"\n super(Double_conv, self).__init__()\n self.conv = nn.Sequential(\n nn.Conv2d(in_ch, out_ch, 3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(out_ch, out_ch, 3, padding=0, stride=1),\n nn.ReLU(inplace=True)\n )\n\n def forward(self, x):\n x = self.conv(x)\n return x\n\n\nclass Conv_down(nn.Module):\n\n '''(conv => ReLU) * 2 => MaxPool2d'''\n\n def __init__(self, in_ch, out_ch):\n \"\"\"\n Args:\n in_ch(int) : input channel\n out_ch(int) : output channel\n \"\"\"\n super(Conv_down, self).__init__()\n self.conv = Double_conv(in_ch, out_ch)\n self.pool = nn.MaxPool2d(kernel_size=2, stride=2)\n\n def forward(self, x):\n x = self.conv(x)\n pool_x = self.pool(x)\n return pool_x, x\n\n\nclass Conv_up(nn.Module):\n\n '''(conv => ReLU) * 2 => MaxPool2d'''\n\n def __init__(self, in_ch, out_ch):\n \"\"\"\n Args:\n in_ch(int) : input channel\n out_ch(int) : output channel\n \"\"\"\n super(Conv_up, self).__init__()\n self.up = nn.ConvTranspose2d(in_ch, out_ch, kernel_size=2, stride=2)\n self.conv = Double_conv(in_ch, out_ch)\n\n def forward(self, x1, x2):\n x1 = self.up(x1)\n x1_dim = x1.size()[2]\n x2 = extract_img(x1_dim, x2)\n x1 = torch.cat((x1, x2), dim=1)\n x1 = self.conv(x1)\n return x1\n\n\ndef extract_img(size, in_tensor):\n \"\"\"\n Args:\n size(int) : size of cut\n in_tensor(tensor) : tensor to be cut\n \"\"\"\n dim1, dim2 = in_tensor.size()[2:]\n in_tensor = in_tensor[:, :, int((dim1-size)/2):int((dim1+size)/2),\n int((dim2-size)/2):int((dim2+size)/2)]\n return in_tensor\n\n\nclass CleanU_Net(nn.Module):\n def __init__(self, in_channels, out_channels):\n super(CleanU_Net, self).__init__()\n self.Conv_down1 = Conv_down(in_channels, 64)\n self.Conv_down2 = Conv_down(64, 128)\n self.Conv_down3 = Conv_down(128, 256)\n self.Conv_down4 = Conv_down(256, 512)\n self.Conv_down5 = Conv_down(512, 1024)\n self.Conv_up1 = Conv_up(1024, 512)\n self.Conv_up2 = Conv_up(512, 256)\n self.Conv_up3 = Conv_up(256, 128)\n self.Conv_up4 = Conv_up(128, 64)\n self.Conv_out = nn.Conv2d(64, out_channels, 1, padding=0, stride=1)\n #self.Conv_final = nn.Conv2d(out_channels, out_channels, 1, padding=0, stride=1)\n\n def forward(self, x):\n\n x, conv1 = self.Conv_down1(x)\n #print(\"dConv1 => down1|\", x.shape)\n x, conv2 = self.Conv_down2(x)\n #print(\"dConv2 => down2|\", x.shape)\n x, conv3 = self.Conv_down3(x)\n #print(\"dConv3 => down3|\", x.shape)\n x, conv4 = self.Conv_down4(x)\n #print(\"dConv4 => down4|\", x.shape)\n _, x = self.Conv_down5(x)\n #print(\"dConv5|\", x.shape)\n x = self.Conv_up1(x, conv4)\n #print(\"up1 => uConv1|\", x.shape)\n x = self.Conv_up2(x, conv3)\n #print(\"up2 => uConv2|\", x.shape)\n x = self.Conv_up3(x, conv2)\n #print(\"up3 => uConv3|\", x.shape)\n x = self.Conv_up4(x, conv1)\n x = self.Conv_out(x)\n #x = self.Conv_final(x)\n\n return x\n\n\nif __name__ == \"__main__\":\n # A full forward pass\n im = torch.randn(1, 1, 572, 572)\n model = CleanU_Net(1, 2)\n x = model(im)\n print(x.shape)\n del model\n del x\n # print(x.shape)\n" }, { "path": "src/dataset.py", "content": "import numpy as np\nfrom PIL import Image\nimport glob\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nfrom random import randint\nfrom torch.utils.data.dataset import Dataset\nfrom pre_processing import *\nfrom mean_std import *\n\nTraining_MEAN = 0.4911\nTraining_STDEV = 0.1658\n\n\nclass SEMDataTrain(Dataset):\n\n def __init__(self, image_path, mask_path, in_size=572, out_size=388):\n \"\"\"\n Args:\n image_path (str): the path where the image is located\n mask_path (str): the path where the mask is located\n option (str): decide which dataset to import\n \"\"\"\n # all file names\n self.mask_arr = glob.glob(str(mask_path) + \"/*\")\n self.image_arr = glob.glob(str(image_path) + str(\"/*\"))\n self.in_size, self.out_size = in_size, out_size\n # Calculate len\n self.data_len = len(self.mask_arr)\n # calculate mean and stdev\n\n def __getitem__(self, index):\n \"\"\"Get specific data corresponding to the index\n Args:\n index (int): index of the data\n Returns:\n Tensor: specific data on index which is converted to Tensor\n \"\"\"\n \"\"\"\n # GET IMAGE\n \"\"\"\n single_image_name = self.image_arr[index]\n img_as_img = Image.open(single_image_name)\n # img_as_img.show()\n img_as_np = np.asarray(img_as_img)\n\n # Augmentation\n # flip {0: vertical, 1: horizontal, 2: both, 3: none}\n flip_num = randint(0, 3)\n img_as_np = flip(img_as_np, flip_num)\n\n # Noise Determine {0: Gaussian_noise, 1: uniform_noise\n if randint(0, 1):\n # Gaussian_noise\n gaus_sd, gaus_mean = randint(0, 20), 0\n img_as_np = add_gaussian_noise(img_as_np, gaus_mean, gaus_sd)\n else:\n # uniform_noise\n l_bound, u_bound = randint(-20, 0), randint(0, 20)\n img_as_np = add_uniform_noise(img_as_np, l_bound, u_bound)\n\n # Brightness\n pix_add = randint(-20, 20)\n img_as_np = change_brightness(img_as_np, pix_add)\n\n # Elastic distort {0: distort, 1:no distort}\n sigma = randint(6, 12)\n # sigma = 4, alpha = 34\n img_as_np, seed = add_elastic_transform(img_as_np, alpha=34, sigma=sigma, pad_size=20)\n\n # Crop the image\n img_height, img_width = img_as_np.shape[0], img_as_np.shape[1]\n pad_size = int((self.in_size - self.out_size)/2)\n img_as_np = np.pad(img_as_np, pad_size, mode=\"symmetric\")\n y_loc, x_loc = randint(0, img_height-self.out_size), randint(0, img_width-self.out_size)\n img_as_np = cropping(img_as_np, crop_size=self.in_size, dim1=y_loc, dim2=x_loc)\n '''\n # Sanity Check for image\n img1 = Image.fromarray(img_as_np)\n img1.show()\n '''\n # Normalize the image\n img_as_np = normalization2(img_as_np, max=1, min=0)\n img_as_np = np.expand_dims(img_as_np, axis=0) # add additional dimension\n img_as_tensor = torch.from_numpy(img_as_np).float() # Convert numpy array to tensor\n\n \"\"\"\n # GET MASK\n \"\"\"\n single_mask_name = self.mask_arr[index]\n msk_as_img = Image.open(single_mask_name)\n # msk_as_img.show()\n msk_as_np = np.asarray(msk_as_img)\n\n # flip the mask with respect to image\n msk_as_np = flip(msk_as_np, flip_num)\n\n # elastic_transform of mask with respect to image\n\n # sigma = 4, alpha = 34, seed = from image transformation\n msk_as_np, _ = add_elastic_transform(\n msk_as_np, alpha=34, sigma=sigma, seed=seed, pad_size=20)\n msk_as_np = approximate_image(msk_as_np) # images only with 0 and 255\n\n # Crop the mask\n msk_as_np = cropping(msk_as_np, crop_size=self.out_size, dim1=y_loc, dim2=x_loc)\n '''\n # Sanity Check for mask\n img2 = Image.fromarray(msk_as_np)\n img2.show()\n '''\n\n # Normalize mask to only 0 and 1\n msk_as_np = msk_as_np/255\n # msk_as_np = np.expand_dims(msk_as_np, axis=0) # add additional dimension\n msk_as_tensor = torch.from_numpy(msk_as_np).long() # Convert numpy array to tensor\n\n return (img_as_tensor, msk_as_tensor)\n\n def __len__(self):\n \"\"\"\n Returns:\n length (int): length of the data\n \"\"\"\n return self.data_len\n\n\nclass SEMDataVal(Dataset):\n def __init__(self, image_path, mask_path, in_size=572, out_size=388):\n '''\n Args:\n image_path = path where test images are located\n mask_path = path where test masks are located\n '''\n # paths to all images and masks\n self.mask_arr = glob.glob(str(mask_path) + str(\"/*\"))\n self.image_arr = glob.glob(str(image_path) + str(\"/*\"))\n self.in_size = in_size\n self.out_size = out_size\n self.data_len = len(self.mask_arr)\n\n def __getitem__(self, index):\n \"\"\"Get specific data corresponding to the index\n Args:\n index : an integer variable that calls (indext)th image in the\n path\n Returns:\n Tensor: 4 cropped data on index which is converted to Tensor\n \"\"\"\n single_image = self.image_arr[index]\n img_as_img = Image.open(single_image)\n # img_as_img.show()\n # Convert the image into numpy array\n img_as_np = np.asarray(img_as_img)\n\n # Make 4 cropped image (in numpy array form) using values calculated above\n # Cropped images will also have paddings to fit the model.\n pad_size = int((self.in_size - self.out_size)/2)\n img_as_np = np.pad(img_as_np, pad_size, mode=\"symmetric\")\n img_as_np = multi_cropping(img_as_np,\n crop_size=self.in_size,\n crop_num1=2, crop_num2=2)\n\n # Empty list that will be filled in with arrays converted to tensor\n processed_list = []\n\n for array in img_as_np:\n\n # SANITY CHECK: SEE THE CROPPED & PADDED IMAGES\n #array_image = Image.fromarray(array)\n\n # Normalize the cropped arrays\n img_to_add = normalization2(array, max=1, min=0)\n # Convert normalized array into tensor\n processed_list.append(img_to_add)\n\n img_as_tensor = torch.Tensor(processed_list)\n # return tensor of 4 cropped images\n # top left, top right, bottom left, bottom right respectively.\n\n \"\"\"\n # GET MASK\n \"\"\"\n single_mask_name = self.mask_arr[index]\n msk_as_img = Image.open(single_mask_name)\n # msk_as_img.show()\n msk_as_np = np.asarray(msk_as_img)\n # Normalize mask to only 0 and 1\n msk_as_np = multi_cropping(msk_as_np,\n crop_size=self.out_size,\n crop_num1=2, crop_num2=2)\n\n msk_as_np = msk_as_np/255\n\n # msk_as_np = np.expand_dims(msk_as_np, axis=0) # add additional dimension\n msk_as_tensor = torch.from_numpy(msk_as_np).long() # Convert numpy array to tensor\n original_msk = torch.from_numpy(np.asarray(msk_as_img))\n return (img_as_tensor, msk_as_tensor, original_msk)\n\n def __len__(self):\n\n return self.data_len\n\n\nclass SEMDataTest(Dataset):\n\n def __init__(self, image_path, in_size=572, out_size=388):\n '''\n Args:\n image_path = path where test images are located\n mask_path = path where test masks are located\n '''\n # paths to all images and masks\n\n self.image_arr = glob.glob(str(image_path) + str(\"/*\"))\n self.in_size = in_size\n self.out_size = out_size\n self.data_len = len(self.image_arr)\n\n def __getitem__(self, index):\n '''Get specific data corresponding to the index\n Args:\n index: an integer variable that calls(indext)th image in the\n path\n Returns:\n Tensor: 4 cropped data on index which is converted to Tensor\n '''\n\n single_image = self.image_arr[index]\n img_as_img = Image.open(single_image)\n # img_as_img.show()\n # Convert the image into numpy array\n img_as_np = np.asarray(img_as_img)\n\n pad_size = int((self.in_size - self.out_size)/2)\n img_as_np = np.pad(img_as_np, pad_size, mode=\"symmetric\")\n img_as_np = multi_cropping(img_as_np,\n crop_size=self.in_size,\n crop_num1=2, crop_num2=2)\n\n # Empty list that will be filled in with arrays converted to tensor\n processed_list = []\n\n for array in img_as_np:\n\n # SANITY CHECK: SEE THE PADDED AND CROPPED IMAGES\n # array_image = Image.fromarray(array)\n\n # Normalize the cropped arrays\n img_to_add = normalization2(array, max=1, min=0)\n # Convert normalized array into tensor\n processed_list.append(img_to_add)\n\n img_as_tensor = torch.Tensor(processed_list)\n # return tensor of 4 cropped images\n # top left, top right, bottom left, bottom right respectively.\n return img_as_tensor\n\n def __len__(self):\n\n return self.data_len\n\n\nif __name__ == \"__main__\":\n\n SEM_train = SEMDataTrain(\n '../data/train/images', '../data/train/masks')\n SEM_test = SEMDataTest(\n '../data/test/images/', '../data/test/masks')\n SEM_val = SEMDataVal('../data/val/images', '../data/val/masks')\n\n imag_1, msk = SEM_train.__getitem__(0)\n" }, { "path": "src/main.py", "content": "from advanced_model import CleanU_Net\nfrom dataset import *\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nimport numpy as np\nfrom PIL import Image\nfrom modules import *\nfrom save_history import *\n\n\nif __name__ == \"__main__\":\n # Dataset begin\n SEM_train = SEMDataTrain(\n '../data/train/images', '../data/train/masks')\n\n # TO DO: finish test data loading\n SEM_test = SEMDataTest(\n '../data/test/images/')\n SEM_val = SEMDataVal(\n '../data/val/images', '../data/val/masks')\n # Dataset end\n\n # Dataloader begins\n SEM_train_load = \\\n torch.utils.data.DataLoader(dataset=SEM_train,\n num_workers=16, batch_size=2, shuffle=True)\n SEM_val_load = \\\n torch.utils.data.DataLoader(dataset=SEM_val,\n num_workers=3, batch_size=1, shuffle=True)\n\n SEM_test_load = \\\n torch.utils.data.DataLoader(dataset=SEM_test,\n num_workers=3, batch_size=1, shuffle=False)\n\n # Dataloader end\n\n # Model\n model = CleanU_Net(in_channels=1, out_channels=2)\n #model = CleanU_Net()\n model = torch.nn.DataParallel(model, device_ids=list(\n range(torch.cuda.device_count()))).cuda()\n\n # Loss function\n criterion = nn.CrossEntropyLoss()\n\n # Optimizerd\n optimizer = torch.optim.RMSprop(model.module.parameters(), lr=0.001)\n\n # Parameters\n epoch_start = 0\n epoch_end = 2000\n\n # Saving History to csv\n header = ['epoch', 'train loss', 'train acc', 'val loss', 'val acc']\n save_file_name = \"../history/RMS/history_RMS3.csv\"\n save_dir = \"../history/RMS\"\n\n # Saving images and models directories\n model_save_dir = \"../history/RMS/saved_models3\"\n image_save_path = \"../history/RMS/result_images3\"\n\n # Train\n print(\"Initializing Training!\")\n for i in range(epoch_start, epoch_end):\n # train the model\n train_model(model, SEM_train_load, criterion, optimizer)\n train_acc, train_loss = get_loss_train(model, SEM_train_load, criterion)\n\n #train_loss = train_loss / len(SEM_train)\n print('Epoch', str(i+1), 'Train loss:', train_loss, \"Train acc\", train_acc)\n\n # Validation every 5 epoch\n if (i+1) % 5 == 0:\n val_acc, val_loss = validate_model(\n model, SEM_val_load, criterion, i+1, True, image_save_path)\n print('Val loss:', val_loss, \"val acc:\", val_acc)\n values = [i+1, train_loss, train_acc, val_loss, val_acc]\n export_history(header, values, save_dir, save_file_name)\n\n if (i+1) % 100 == 0: # save model every 10 epoch\n save_models(model, model_save_dir, i+1)\n\n\"\"\"\n# Test\nprint(\"generate test prediction\")\ntest_model(\"../history/RMS/saved_models/model_epoch_440.pwf\",\n SEM_test_load, 440, \"../history/RMS/result_images_test\")\n\"\"\"\n" }, { "path": "src/mean_std.py", "content": "import numpy as np\nfrom PIL import Image\nimport glob\n\n\ndef normalize_image(image):\n \"\"\"\n Args:\n image : a string of name of image file\n Return:\n image_asarray : numpy array of the image\n that is normalized by being divided by 255\n \"\"\"\n\n img_opened = Image.open(image)\n img_asarray = np.asarray(img_opened)\n img_asarray = img_asarray / 255\n\n return img_asarray\n\n\ndef find_mean(image_path):\n \"\"\"\n Args:\n image_path : pathway of all images\n Return :\n mean : mean value of all the images\n \"\"\"\n all_images = glob.glob(str(image_path) + str(\"/*\"))\n num_images = len(all_images)\n mean_sum = 0\n\n for image in all_images:\n img_asarray = normalize_image(image)\n individual_mean = np.mean(img_asarray)\n mean_sum += individual_mean\n\n # Divide the sum of all values by the number of images present\n mean = mean_sum / num_images\n\n return mean\n\n\ndef find_stdev(image_path):\n \"\"\"\n Args:\n image_path : pathway of all images\n Return :\n stdev : standard deviation of all pixels\n \"\"\"\n # Initiation\n all_images = glob.glob(str(image_path) + str(\"/*\"))\n num_images = len(all_images)\n\n # Recall mean value from function above: def Mean(path)\n mean_value = find_mean(image_path)\n std_sum = 0\n\n for image in all_images:\n img_asarray = normalize_image(image)\n individual_stdev = np.std(img_asarray)\n std_sum += individual_stdev\n\n std = std_sum / num_images\n\n return std\n\n\n# Experimenting\nif __name__ == '__main__':\n image_path = '../data/train/images'\n\n print('for training images,')\n print('mean:', find_mean(image_path))\n print('stdev:', find_stdev(image_path))\n" }, { "path": "src/modules.py", "content": "import numpy as np\nfrom PIL import Image\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nfrom dataset import *\nimport torch.nn as nn\nfrom accuracy import accuracy_check, accuracy_check_for_batch\nimport csv\nimport os\n\n\ndef train_model(model, data_train, criterion, optimizer):\n \"\"\"Train the model and report validation error with training error\n Args:\n model: the model to be trained\n criterion: loss function\n data_train (DataLoader): training dataset\n \"\"\"\n model.train()\n for batch, (images, masks) in enumerate(data_train):\n images = Variable(images.cuda())\n masks = Variable(masks.cuda())\n outputs = model(images)\n # print(masks.shape, outputs.shape)\n loss = criterion(outputs, masks)\n optimizer.zero_grad()\n loss.backward()\n # Update weights\n optimizer.step()\n # total_loss = get_loss_train(model, data_train, criterion)\n\n\ndef get_loss_train(model, data_train, criterion):\n \"\"\"\n Calculate loss over train set\n \"\"\"\n model.eval()\n total_acc = 0\n total_loss = 0\n for batch, (images, masks) in enumerate(data_train):\n with torch.no_grad():\n images = Variable(images.cuda())\n masks = Variable(masks.cuda())\n outputs = model(images)\n loss = criterion(outputs, masks)\n preds = torch.argmax(outputs, dim=1).float()\n acc = accuracy_check_for_batch(masks.cpu(), preds.cpu(), images.size()[0])\n total_acc = total_acc + acc\n total_loss = total_loss + loss.cpu().item()\n return total_acc/(batch+1), total_loss/(batch + 1)\n\n\ndef validate_model(model, data_val, criterion, epoch, make_prediction=True, save_folder_name='prediction'):\n \"\"\"\n Validation run\n \"\"\"\n # calculating validation loss\n total_val_loss = 0\n total_val_acc = 0\n for batch, (images_v, masks_v, original_msk) in enumerate(data_val):\n stacked_img = torch.Tensor([]).cuda()\n for index in range(images_v.size()[1]):\n with torch.no_grad():\n image_v = Variable(images_v[:, index, :, :].unsqueeze(0).cuda())\n mask_v = Variable(masks_v[:, index, :, :].squeeze(1).cuda())\n # print(image_v.shape, mask_v.shape)\n output_v = model(image_v)\n total_val_loss = total_val_loss + criterion(output_v, mask_v).cpu().item()\n # print('out', output_v.shape)\n output_v = torch.argmax(output_v, dim=1).float()\n stacked_img = torch.cat((stacked_img, output_v))\n if make_prediction:\n im_name = batch # TODO: Change this to real image name so we know\n pred_msk = save_prediction_image(stacked_img, im_name, epoch, save_folder_name)\n acc_val = accuracy_check(original_msk, pred_msk)\n total_val_acc = total_val_acc + acc_val\n\n return total_val_acc/(batch + 1), total_val_loss/((batch + 1)*4)\n\n\ndef test_model(model_path, data_test, epoch, save_folder_name='prediction'):\n \"\"\"\n Test run\n \"\"\"\n model = torch.load(model_path)\n model = torch.nn.DataParallel(model, device_ids=list(\n range(torch.cuda.device_count()))).cuda()\n model.eval()\n for batch, (images_t) in enumerate(data_test):\n stacked_img = torch.Tensor([]).cuda()\n for index in range(images_t.size()[1]):\n with torch.no_grad():\n image_t = Variable(images_t[:, index, :, :].unsqueeze(0).cuda())\n # print(image_v.shape, mask_v.shape)\n output_t = model(image_t)\n output_t = torch.argmax(output_t, dim=1).float()\n stacked_img = torch.cat((stacked_img, output_t))\n im_name = batch # TODO: Change this to real image name so we know\n _ = save_prediction_image(stacked_img, im_name, epoch, save_folder_name)\n print(\"Finish Prediction!\")\n\n\ndef save_prediction_image(stacked_img, im_name, epoch, save_folder_name=\"result_images\", save_im=True):\n \"\"\"save images to save_path\n Args:\n stacked_img (numpy): stacked cropped images\n save_folder_name (str): saving folder name\n \"\"\"\n div_arr = division_array(388, 2, 2, 512, 512)\n img_cont = image_concatenate(stacked_img.cpu().data.numpy(), 2, 2, 512, 512)\n img_cont = polarize((img_cont)/div_arr)*255\n img_cont_np = img_cont.astype('uint8')\n img_cont = Image.fromarray(img_cont_np)\n # organize images in every epoch\n desired_path = save_folder_name + '/epoch_' + str(epoch) + '/'\n # Create the path if it does not exist\n if not os.path.exists(desired_path):\n os.makedirs(desired_path)\n # Save Image!\n export_name = str(im_name) + '.png'\n img_cont.save(desired_path + export_name)\n return img_cont_np\n\n\ndef polarize(img):\n ''' Polarize the value to zero and one\n Args:\n img (numpy): numpy array of image to be polarized\n return:\n img (numpy): numpy array only with zero and one\n '''\n img[img >= 0.5] = 1\n img[img < 0.5] = 0\n return img\n\n\n\"\"\"\ndef test_SEM(model, data_test, folder_to_save):\n '''Test the model with test dataset\n Args:\n model: model to be tested\n data_test (DataLoader): test dataset\n folder_to_save (str): path that the predictions would be saved\n '''\n for i, (images) in enumerate(data_test):\n\n print(images)\n stacked_img = torch.Tensor([])\n for j in range(images.size()[1]):\n image = Variable(images[:, j, :, :].unsqueeze(0).cuda())\n output = model(image.cuda())\n print(output)\n print(\"size\", output.size())\n output = torch.argmax(output, dim=1).float()\n print(\"size\", output.size())\n stacked_img = torch.cat((stacked_img, output))\n div_arr = division_array(388, 2, 2, 512, 512)\n print(stacked_img.size())\n img_cont = image_concatenate(stacked_img.data.numpy(), 2, 2, 512, 512)\n final_img = (img_cont*255/div_arr)\n print(final_img)\n final_img = final_img.astype(\"uint8\")\n break\n return final_img\n\"\"\"\n\n\nif __name__ == '__main__':\n SEM_train = SEMDataTrain(\n '../data/train/images', '../data/train/masks')\n SEM_train_load = torch.utils.data.DataLoader(dataset=SEM_train,\n num_workers=3, batch_size=10, shuffle=True)\n get_loss_train()\n" }, { "path": "src/post_processing.py", "content": "import numpy as np\nfrom matplotlib import pyplot as plt\n\n\ndef postprocess(image_path):\n ''' postprocessing of the prediction output\n Args\n image_path : path of the image\n Returns\n watershed_grayscale : numpy array of postprocessed image (in grayscale)\n '''\n\n # Bring in the image\n img_original = cv2.imread(image_path)\n img = cv2.imread(image_path)\n\n # In case the input image has 3 channels (RGB), convert to 1 channel (grayscale)\n gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n\n # Use threshold => Image will have values either 0 or 255 (black or white)\n ret, bin_image = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)\n\n # Remove Hole or noise through the use of opening, closing in Morphology module\n kernel = np.ones((1, 1), np.uint8)\n kernel1 = np.ones((3, 3), np.uint8)\n\n # remove noise in\n closing = cv2.morphologyEx(bin_image, cv2.MORPH_CLOSE, kernel, iterations=1)\n\n # make clear distinction of the background\n # Incerease/emphasize the white region.\n sure_bg = cv2.dilate(closing, kernel1, iterations=1)\n\n # calculate the distance to the closest zero pixel for each pixel of the source.\n # Adjust the threshold value with respect to the maximum distance. Lower threshold, more information.\n dist_transform = cv2.distanceTransform(closing, cv2.DIST_L2, 5)\n ret, sure_fg = cv2.threshold(dist_transform, 0.2*dist_transform.max(), 255, 0)\n sure_fg = np.uint8(sure_fg)\n\n # Unknown is the region of background with foreground excluded.\n unknown = cv2.subtract(sure_bg, sure_fg)\n\n # labelling on the foreground.\n ret, markers = cv2.connectedComponents(sure_fg)\n markers_plus1 = markers + 1\n markers_plus1[unknown == 255] = 0\n\n # Appy watershed and label the borders\n markers_watershed = cv2.watershed(img, markers_plus1)\n\n # See the watershed result in a clear white page.\n img_x, img_y = img_original.shape[0], img_original.shape[1] # 512x512\n white, white_color = np.zeros((img_x, img_y, 3)), np.zeros((img_x, img_y, 3))\n white += 255\n white_color += 255\n # 1 in markers_watershed indicate the background value\n # label everything not indicated as background value\n white[markers_watershed != 1] = [0, 0, 0] # grayscale version\n white_color[markers_watershed != 1] = [255, 0, 0] # RGB version\n\n # Convert to numpy array for later processing\n white_np = np.asarray(white) # 512x512x3\n watershed_grayscale = white_np.transpose(2, 0, 1)[0, :, :] # convert to 1 channel (grayscale)\n img[markers_watershed != 1] = [255, 0, 0]\n\n return watershed_grayscale\n\n '''\n Visualizing all the intermediate processes\n\n images = [img_original, gray,bin_image, closing, sure_bg, dist_transform, sure_fg, unknown, markers, markers_watershed, white_color, white, img]\n titles = ['Original', '1. Grayscale','2. Binary','3. Closing','Sure BG','Distance','Sure FG','Unknown','Markers', 'Markers_Watershed','Result', 'Result gray','Result Overlapped']\n CMAP = [None, 'gray', 'gray','gray','gray',None,'gray','gray',None, None, None, None,'gray']\n\n\n for i in range(len(images)):\n plt.subplot(4,4,i+1),plt.imshow(images[i], cmap=CMAP[i]),plt.title(titles[i]),plt.xticks([]),plt.yticks([])\n\n plt.show()\n '''\n\n\nif __name__ == '__main__':\n from PIL import Image\n\n print(postprocess('../data/train/masks/25.png'))\n" }, { "path": "src/pre_processing.py", "content": "import numpy as np\nfrom scipy.ndimage.interpolation import map_coordinates\nfrom scipy.ndimage.filters import gaussian_filter\nfrom random import randint\n\n\ndef add_elastic_transform(image, alpha, sigma, pad_size=30, seed=None):\n \"\"\"\n Args:\n image : numpy array of image\n alpha : α is a scaling factor\n sigma : σ is an elasticity coefficient\n random_state = random integer\n Return :\n image : elastically transformed numpy array of image\n \"\"\"\n image_size = int(image.shape[0])\n image = np.pad(image, pad_size, mode=\"symmetric\")\n if seed is None:\n seed = randint(1, 100)\n random_state = np.random.RandomState(seed)\n else:\n random_state = np.random.RandomState(seed)\n shape = image.shape\n dx = gaussian_filter((random_state.rand(*shape) * 2 - 1),\n sigma, mode=\"constant\", cval=0) * alpha\n dy = gaussian_filter((random_state.rand(*shape) * 2 - 1),\n sigma, mode=\"constant\", cval=0) * alpha\n\n x, y = np.meshgrid(np.arange(shape[1]), np.arange(shape[0]))\n indices = np.reshape(y+dy, (-1, 1)), np.reshape(x+dx, (-1, 1))\n return cropping(map_coordinates(image, indices, order=1).reshape(shape), 512, pad_size, pad_size), seed\n\n\ndef flip(image, option_value):\n \"\"\"\n Args:\n image : numpy array of image\n option_value = random integer between 0 to 3\n Return :\n image : numpy array of flipped image\n \"\"\"\n if option_value == 0:\n # vertical\n image = np.flip(image, option_value)\n elif option_value == 1:\n # horizontal\n image = np.flip(image, option_value)\n elif option_value == 2:\n # horizontally and vertically flip\n image = np.flip(image, 0)\n image = np.flip(image, 1)\n else:\n image = image\n # no effect\n return image\n\n\ndef add_gaussian_noise(image, mean=0, std=1):\n \"\"\"\n Args:\n image : numpy array of image\n mean : pixel mean of image\n standard deviation : pixel standard deviation of image\n Return :\n image : numpy array of image with gaussian noise added\n \"\"\"\n gaus_noise = np.random.normal(mean, std, image.shape)\n image = image.astype(\"int16\")\n noise_img = image + gaus_noise\n image = ceil_floor_image(image)\n return noise_img\n\n\ndef add_uniform_noise(image, low=-10, high=10):\n \"\"\"\n Args:\n image : numpy array of image\n low : lower boundary of output interval\n high : upper boundary of output interval\n Return :\n image : numpy array of image with uniform noise added\n \"\"\"\n uni_noise = np.random.uniform(low, high, image.shape)\n image = image.astype(\"int16\")\n noise_img = image + uni_noise\n image = ceil_floor_image(image)\n return noise_img\n\n\ndef change_brightness(image, value):\n \"\"\"\n Args:\n image : numpy array of image\n value : brightness\n Return :\n image : numpy array of image with brightness added\n \"\"\"\n image = image.astype(\"int16\")\n image = image + value\n image = ceil_floor_image(image)\n return image\n\n\ndef ceil_floor_image(image):\n \"\"\"\n Args:\n image : numpy array of image in datatype int16\n Return :\n image : numpy array of image in datatype uint8 with ceilling(maximum 255) and flooring(minimum 0)\n \"\"\"\n image[image > 255] = 255\n image[image < 0] = 0\n image = image.astype(\"uint8\")\n return image\n\n\ndef approximate_image(image):\n \"\"\"\n Args:\n image : numpy array of image in datatype int16\n Return :\n image : numpy array of image in datatype uint8 only with 255 and 0\n \"\"\"\n image[image > 127.5] = 255\n image[image < 127.5] = 0\n image = image.astype(\"uint8\")\n return image\n\n\ndef normalization1(image, mean, std):\n \"\"\" Normalization using mean and std\n Args :\n image : numpy array of image\n mean :\n Return :\n image : numpy array of image with values turned into standard scores\n \"\"\"\n\n image = image / 255 # values will lie between 0 and 1.\n image = (image - mean) / std\n\n return image\n\n\ndef normalization2(image, max, min):\n \"\"\"Normalization to range of [min, max]\n Args :\n image : numpy array of image\n mean :\n Return :\n image : numpy array of image with values turned into standard scores\n \"\"\"\n image_new = (image - np.min(image))*(max - min)/(np.max(image)-np.min(image)) + min\n return image_new\n\n\ndef stride_size(image_len, crop_num, crop_size):\n \"\"\"return stride size\n Args :\n image_len(int) : length of one size of image (width or height)\n crop_num(int) : number of crop in certain direction\n crop_size(int) : size of crop\n Return :\n stride_size(int) : stride size\n \"\"\"\n return int((image_len - crop_size)/(crop_num - 1))\n\n\ndef multi_cropping(image, crop_size, crop_num1, crop_num2):\n \"\"\"crop the image and pad it to in_size\n Args :\n images : numpy arrays of images\n crop_size(int) : size of cropped image\n crop_num2 (int) : number of crop in horizontal way\n crop_num1 (int) : number of crop in vertical way\n Return :\n cropped_imgs : numpy arrays of stacked images\n \"\"\"\n\n img_height, img_width = image.shape[0], image.shape[1]\n assert crop_size*crop_num1 >= img_width and crop_size * \\\n crop_num2 >= img_height, \"Whole image cannot be sufficiently expressed\"\n assert crop_num1 <= img_width - crop_size + 1 and crop_num2 <= img_height - \\\n crop_size + 1, \"Too many number of crops\"\n\n cropped_imgs = []\n # int((img_height - crop_size)/(crop_num1 - 1))\n dim1_stride = stride_size(img_height, crop_num1, crop_size)\n # int((img_width - crop_size)/(crop_num2 - 1))\n dim2_stride = stride_size(img_width, crop_num2, crop_size)\n for i in range(crop_num1):\n for j in range(crop_num2):\n cropped_imgs.append(cropping(image, crop_size,\n dim1_stride*i, dim2_stride*j))\n return np.asarray(cropped_imgs)\n\n\n# IT IS NOT USED FOR PAD AND CROP DATA OPERATION\n# IF YOU WANT TO USE CROP AND PAD USE THIS FUNCTION\n\"\"\"\ndef multi_padding(images, in_size, out_size, mode):\n '''Pad the images to in_size\n Args :\n images : numpy array of images (CxHxW)\n in_size(int) : the input_size of model (512)\n out_size(int) : the output_size of model (388)\n mode(str) : mode of padding\n Return :\n padded_imgs: numpy arrays of padded images\n '''\n pad_size = int((in_size - out_size)/2)\n padded_imgs = []\n for num in range(images.shape[0]):\n padded_imgs.append(add_padding(images[num], in_size, out_size, mode=mode))\n return np.asarray(padded_imgs)\n\n\"\"\"\n\n\ndef cropping(image, crop_size, dim1, dim2):\n \"\"\"crop the image and pad it to in_size\n Args :\n images : numpy array of images\n crop_size(int) : size of cropped image\n dim1(int) : vertical location of crop\n dim2(int) : horizontal location of crop\n Return :\n cropped_img: numpy array of cropped image\n \"\"\"\n cropped_img = image[dim1:dim1+crop_size, dim2:dim2+crop_size]\n return cropped_img\n\n\ndef add_padding(image, in_size, out_size, mode):\n \"\"\"Pad the image to in_size\n Args :\n images : numpy array of images\n in_size(int) : the input_size of model\n out_size(int) : the output_size of model\n mode(str) : mode of padding\n Return :\n padded_img: numpy array of padded image\n \"\"\"\n pad_size = int((in_size - out_size)/2)\n padded_img = np.pad(image, pad_size, mode=mode)\n return padded_img\n\n\ndef division_array(crop_size, crop_num1, crop_num2, dim1, dim2):\n \"\"\"Make division array\n Args :\n crop_size(int) : size of cropped image\n crop_num2 (int) : number of crop in horizontal way\n crop_num1 (int) : number of crop in vertical way\n dim1(int) : vertical size of output\n dim2(int) : horizontal size_of_output\n Return :\n div_array : numpy array of numbers of 1,2,4\n \"\"\"\n div_array = np.zeros([dim1, dim2]) # make division array\n one_array = np.ones([crop_size, crop_size]) # one array to be added to div_array\n dim1_stride = stride_size(dim1, crop_num1, crop_size) # vertical stride\n dim2_stride = stride_size(dim2, crop_num2, crop_size) # horizontal stride\n for i in range(crop_num1):\n for j in range(crop_num2):\n # add ones to div_array at specific position\n div_array[dim1_stride*i:dim1_stride*i + crop_size,\n dim2_stride*j:dim2_stride*j + crop_size] += one_array\n return div_array\n\n\ndef image_concatenate(image, crop_num1, crop_num2, dim1, dim2):\n \"\"\"concatenate images\n Args :\n image : output images (should be square)\n crop_num2 (int) : number of crop in horizontal way (2)\n crop_num1 (int) : number of crop in vertical way (2)\n dim1(int) : vertical size of output (512)\n dim2(int) : horizontal size_of_output (512)\n Return :\n div_array : numpy arrays of numbers of 1,2,4\n \"\"\"\n crop_size = image.shape[1] # size of crop\n empty_array = np.zeros([dim1, dim2]).astype(\"float64\") # to make sure no overflow\n dim1_stride = stride_size(dim1, crop_num1, crop_size) # vertical stride\n dim2_stride = stride_size(dim2, crop_num2, crop_size) # horizontal stride\n index = 0\n for i in range(crop_num1):\n for j in range(crop_num2):\n # add image to empty_array at specific position\n empty_array[dim1_stride*i:dim1_stride*i + crop_size,\n dim2_stride*j:dim2_stride*j + crop_size] += image[index]\n index += 1\n return empty_array\n\n\nif __name__ == \"__main__\":\n from PIL import Image\n\n b = Image.open(\"../data/train/images/14.png\")\n c = Image.open(\"../data/train/masks/14.png\")\n\n original = np.array(b)\n originall = np.array(c)\n original_norm = normalization(original, max=1, min=0)\n print(original_norm)\n\n b = Image.open(\"../readme_images/original.png\")\n original = np.array(b)\n \"\"\"\n original1 = add_gaussian_noise(original, 0, 100)\n original1 = Image.fromarray(original1)\n original1.show()\n \"\"\"\n original1 = add_uniform_noise(original, -100, 100)\n original1 = Image.fromarray(original1)\n original1.show()\n \"\"\"\n original1 = change_brightness(original, 50)\n original1 = Image.fromarray(original1)\n original1.show()\n original1 = add_elastic_transform(original, 10, 4, 1)[0]\n original1 = Image.fromarray(original1)\n original1.show()\n \"\"\"\n" }, { "path": "src/result_visualization.py", "content": "\nimport matplotlib.pyplot as plt\nimport matplotlib as mpl\nimport pandas as pd\nimport numpy as np\n\n'''\nFor members who did not yet install the module \"matplotlib\",\n python3 -mpip install -U matplotlib\ninstallation of tkinter is a prerequisite. If you do not have it,\n sudo apt install python3.6-tk\nNow you won't have problems running this python file.\n'''\n\n\ndef plotloss(csvfile):\n '''\n Args\n csvfile: name of the csv file\n Returns\n graph_loss: trend of loss values over epoch\n '''\n\n # Bring in the csv file\n loss_values = pd.read_csv(csvfile)\n\n # Initiation\n epoch = loss_values.iloc[:, 0]\n tr_loss = loss_values.iloc[:, 1]\n tr_acc = loss_values.iloc[:, 2]\n val_loss = np.asarray(loss_values.iloc[:, 3])\n val_acc = np.asarray(loss_values.iloc[:, 4])\n\n # Reduce the volume of data\n epoch_skip = epoch[::5]\n tr_loss_skip = tr_loss[::5]\n tr_acc_skip = tr_acc[::5]\n val_loss_skip = val_loss[::5]\n val_acc_skip = val_acc[::5]\n\n fig, ax1 = plt.subplots(figsize=(8, 6))\n ax2 = ax1.twinx()\n\n # Label and color the axes\n ax1.set_xlabel('Epoch', fontsize=16)\n ax1.set_ylabel('Loss', fontsize=16, color='black')\n ax2.set_ylabel('Accuracy', fontsize=16, color='black')\n\n # Plot valid/train losses\n ax1.plot(epoch_skip, tr_loss_skip, linewidth=2,\n ls='--', color='#c92508', label='Train loss')\n ax1.plot(epoch_skip, val_loss_skip, linewidth=2,\n color='#c92508', label='Validation loss')\n ax1.spines['left'].set_color('#f23d1d')\n # Coloring the ticks\n for label in ax1.get_yticklabels():\n label.set_color('#c92508')\n label.set_size(12)\n\n # Plot valid/trian accuracy\n ax2.plot(epoch_skip, tr_acc_skip, linewidth=2, ls='--',\n color='#2348ff', label='Train Accuracy')\n ax2.plot(epoch_skip, val_acc_skip, linewidth=2,\n color='#2348ff', label='Validation Accuracy')\n ax2.spines['right'].set_color('#2348ff')\n # Coloring the ticks\n for label in ax2.get_yticklabels():\n label.set_color('#2348ff')\n label.set_size(12)\n\n # Manually setting the y-axis ticks\n yticks = [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1]\n ax1.set_yticks(yticks)\n ax2.set_yticks(yticks)\n\n for label in ax1.get_xticklabels():\n label.set_size(12)\n\n # Modification of the overall graph\n fig.legend(ncol=4, loc=9, fontsize=12)\n plt.xlim(xmin=0)\n ax2.set_ylim(ymax=1, ymin=0)\n ax1.set_ylim(ymax=1, ymin=0)\n plt.xlabel('epochs')\n plt.title(\"Adam optimizer\", weight=\"bold\")\n plt.grid(True, axis='y')\n\n # return train_loss, valid_loss\n\n\nif __name__ == '__main__':\n file = '../history/csv/Adam.csv'\n #file = '../history/SGD/history_SGD4.csv'\n plt.show(plotloss(file))\n" }, { "path": "src/save_history.py", "content": "import os\nimport csv\nimport torch\n\n\ndef export_history(header, value, folder, file_name):\n \"\"\" export data to csv format\n Args:\n header (list): headers of the column\n value (list): values of correspoding column\n folder (list): folder path\n file_name: file name with path\n \"\"\"\n # if folder does not exists make folder\n if not os.path.exists(folder):\n os.makedirs(folder)\n\n file_existence = os.path.isfile(file_name)\n\n # if there is no file make file\n if file_existence == False:\n file = open(file_name, 'w', newline='')\n writer = csv.writer(file)\n writer.writerow(header)\n writer.writerow(value)\n # if there is file overwrite\n else:\n file = open(file_name, 'a', newline='')\n writer = csv.writer(file)\n writer.writerow(value)\n # close file when it is done with writing\n file.close()\n\n\ndef save_models(model, path, epoch):\n \"\"\"Save model to given path\n Args:\n model: model to be saved\n path: path that the model would be saved\n epoch: the epoch the model finished training\n \"\"\"\n if not os.path.exists(path):\n os.makedirs(path)\n torch.save(model, path+\"/model_epoch_{0}.pwf\".format(epoch))\n" }, { "path": "src/simple_model.py", "content": "import torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nimport numpy as np\nfrom PIL import Image\nfrom torch.nn.functional import sigmoid\n\n\nclass CleanU_Net(nn.Module):\n\n def __init__(self):\n\n super(CleanU_Net, self).__init__()\n\n # Conv block 1 - Down 1\n self.conv1_block = nn.Sequential(\n nn.Conv2d(in_channels=1, out_channels=32,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(in_channels=32, out_channels=32,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n )\n self.max1 = nn.MaxPool2d(kernel_size=2, stride=2)\n\n # Conv block 2 - Down 2\n self.conv2_block = nn.Sequential(\n nn.Conv2d(in_channels=32, out_channels=64,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(in_channels=64, out_channels=64,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n )\n self.max2 = nn.MaxPool2d(kernel_size=2, stride=2)\n\n # Conv block 3 - Down 3\n self.conv3_block = nn.Sequential(\n nn.Conv2d(in_channels=64, out_channels=128,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(in_channels=128, out_channels=128,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n )\n self.max3 = nn.MaxPool2d(kernel_size=2, stride=2)\n\n # Conv block 4 - Down 4\n self.conv4_block = nn.Sequential(\n nn.Conv2d(in_channels=128, out_channels=256,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(in_channels=256, out_channels=256,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n )\n self.max4 = nn.MaxPool2d(kernel_size=2, stride=2)\n\n # Conv block 5 - Down 5\n self.conv5_block = nn.Sequential(\n nn.Conv2d(in_channels=256, out_channels=512,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(in_channels=512, out_channels=512,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n )\n\n # Up 1\n self.up_1 = nn.ConvTranspose2d(in_channels=512, out_channels=256, kernel_size=2, stride=2)\n\n # Up Conv block 1\n self.conv_up_1 = nn.Sequential(\n nn.Conv2d(in_channels=512, out_channels=256,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(in_channels=256, out_channels=256,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n )\n\n # Up 2\n self.up_2 = nn.ConvTranspose2d(in_channels=256, out_channels=128, kernel_size=2, stride=2)\n\n # Up Conv block 2\n self.conv_up_2 = nn.Sequential(\n nn.Conv2d(in_channels=256, out_channels=128,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(in_channels=128, out_channels=128,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n )\n\n # Up 3\n self.up_3 = nn.ConvTranspose2d(in_channels=128, out_channels=64, kernel_size=2, stride=2)\n\n # Up Conv block 3\n self.conv_up_3 = nn.Sequential(\n nn.Conv2d(in_channels=128, out_channels=64,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(in_channels=64, out_channels=64,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n )\n\n # Up 4\n self.up_4 = nn.ConvTranspose2d(in_channels=64, out_channels=32, kernel_size=2, stride=2)\n\n # Up Conv block 4\n self.conv_up_4 = nn.Sequential(\n nn.Conv2d(in_channels=64, out_channels=32,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n nn.Conv2d(in_channels=32, out_channels=32,\n kernel_size=3, padding=0, stride=1),\n nn.ReLU(inplace=True),\n )\n\n # Final output\n self.conv_final = nn.Conv2d(in_channels=32, out_channels=2,\n kernel_size=1, padding=0, stride=1)\n\n def forward(self, x):\n # print('input', x.shape)\n\n # Down 1\n x = self.conv1_block(x)\n # print('after conv1', x.shape)\n conv1_out = x # Save out1\n conv1_dim = x.shape[2]\n x = self.max1(x)\n # print('before conv2', x.shape)\n\n # Down 2\n x = self.conv2_block(x)\n # print('after conv2', x.shape)\n conv2_out = x\n conv2_dim = x.shape[2]\n x = self.max2(x)\n # print('before conv3', x.shape)\n\n # Down 3\n x = self.conv3_block(x)\n # print('after conv3', x.shape)\n conv3_out = x\n conv3_dim = x.shape[2]\n x = self.max3(x)\n # print('before conv4', x.shape)\n\n # Down 4\n x = self.conv4_block(x)\n # print('after conv5', x.shape)\n conv4_out = x\n conv4_dim = x.shape[2]\n x = self.max4(x)\n\n # Midpoint\n x = self.conv5_block(x)\n\n # Up 1\n x = self.up_1(x)\n # print('up_1', x.shape)\n lower = int((conv4_dim - x.shape[2]) / 2)\n upper = int(conv4_dim - lower)\n conv4_out_modified = conv4_out[:, :, lower:upper, lower:upper]\n x = torch.cat([x, conv4_out_modified], dim=1)\n # print('after cat_1', x.shape)\n x = self.conv_up_1(x)\n # print('after conv_1', x.shape)\n\n # Up 2\n x = self.up_2(x)\n # print('up_2', x.shape)\n lower = int((conv3_dim - x.shape[2]) / 2)\n upper = int(conv3_dim - lower)\n conv3_out_modified = conv3_out[:, :, lower:upper, lower:upper]\n x = torch.cat([x, conv3_out_modified], dim=1)\n # print('after cat_2', x.shape)\n x = self.conv_up_2(x)\n # print('after conv_2', x.shape)\n\n # Up 3\n x = self.up_3(x)\n # print('up_3', x.shape)\n lower = int((conv2_dim - x.shape[2]) / 2)\n upper = int(conv2_dim - lower)\n conv2_out_modified = conv2_out[:, :, lower:upper, lower:upper]\n x = torch.cat([x, conv2_out_modified], dim=1)\n # print('after cat_3', x.shape)\n x = self.conv_up_3(x)\n # print('after conv_3', x.shape)\n\n # Up 4\n x = self.up_4(x)\n # print('up_4', x.shape)\n lower = int((conv1_dim - x.shape[2]) / 2)\n upper = int(conv1_dim - lower)\n conv1_out_modified = conv1_out[:, :, lower:upper, lower:upper]\n x = torch.cat([x, conv1_out_modified], dim=1)\n # print('after cat_4', x.shape)\n x = self.conv_up_4(x)\n # print('after conv_4', x.shape)\n\n # Final output\n x = self.conv_final(x)\n\n return x\n\n\nif __name__ == \"__main__\":\n # A full forward pass\n im = torch.randn(1, 1, 572, 572)\n model = CleanU_Net()\n x = model(im)\n # print(x.shape)\n del model\n del x\n # print(x.shape)\n" } ]