[ { "path": ".gitignore", "content": ".env/\nmodel/\n__pycache__/\n.pyc\n" }, { "path": "README.md", "content": "# Project Defude\n\n

\n\t\"Project\n\t\"Project\n\t\"Project\n

\n\n*Synthetic Defocusing using Monocular Depth Estimation*\n\nGraded blurring of an image based on how far a point is from focus using just a single image with no extra data.\nTo achieve this we generate a depth map for the image using machine learning.\nCreate blurred versions of the image in levels based on how far a point is from the depth of focus.\nStitch the different blurred images to create the final image with the selected point fully focussed and points further away become more blurred.\n\n## How to run\nYou'll need to setup Tensorflow v1 and OpenCV for the bare minimum run.\n\n> **_NOTE:_** Setting up Tensorflow v1 can be tricky, hence it is recommeneded to use a prebuilt docker image.\n> ```bash\n> docker pull tensorflow/tensorflow:1.0.0-py3\n> ```\n> GPU specific images are also available.\n\nInstall a trained model for depth estimation.\n```sh\nsh ./depth/get_model.sh model_kitti depth/\n```\nThis will download the model which will give the best results for the sample in the repo.\n\nCall main.py with model path and image path\n```sh\npython main.py --model_path /path/to/model --image_path /path/to/image\n```\n\nThe image will load up. Clicking anywhere sets that as the point of focus, regenerating the image.\n\n*I've actually pushed in the depth data for the sample images, so you can actually run the program without tensorflow or a trained model for the sample images :)*\n\n\n### To use our nice GUI\nInstall dependencies from requirements.txt\n\nRun the gui version as\n```sh\npython gui.py --model_path /path/to/model\n```\nGui version requires some additional packages. Install as necessary.\n\n## Depth Estimation\nWe have used the method based on the paper Unsupervised Monocular Depth Estimation with Left-Right Consistency. You can find more about their amazing paper [here](http://visual.cs.ucl.ac.uk/pubs/monoDepth/).\nThey train a machine learning model to generate a depth map using just a single image.\n\n### How the machine learing model is trained in simple terms:\nThe model is trained on a large set of stereo(left-right) images to generate a right image from a given left image.\nGenerating the right image, the model is learning internally about the depth of various points in the image.\n\nOnce the model is trained. We can give it a simple image and it will be able to generate a depth map for that image.\n\nYou can learn more about their amazing project on [github](https://github.com/mrharicot/monodepth).\nTheir trained models give way better results than what we can so we are using that.\nThe depth estimation code we have is a minimal stripped down version just to run the model.\n\n\n## The Team\nThis was done as a final year project by [Haritha Paul](https://github.com/haritha1997), [Navin Mohan](https://github.com/navin-mohan), [Roshan V](https://github.com/ros-han) and me.\n\nLeave a star if you liked the project. :)\n" }, { "path": "defocus/defocus.py", "content": "import argparse\nimport copy\nimport cv2\nimport numpy as np\nimport os\n\nparser = argparse.ArgumentParser(description='Defocus using depth map.')\n\nparser.add_argument('--image_path', type=str, help='path to input image', default='../images/sample2.png')\nparser.add_argument('--blur_method', type=str, help='The type of blur to be applied', default='gaussian')\n\nargs = parser.parse_args()\n\n# This class holds functions that handle defocusing of an image using a depth map\n# It takes in the path to the image to be defocused and the blur method to be used for defocusing\n# It requires the depth map to be saved in the same folder as the image, with the suffix and extension _disp.npy\n# Parameters:\n# image_path: Path to the image which is to be defocused\n# blur_method: The blur function to be used for blurring.\n# Available values are: gaussian, avg_blur, median, bilateral\nclass DefocuserObject():\n\n def __init__(self, image_path = \"../images/sample2.png\", blur_method = \"gaussian\"):\n self.img_name = os.path.basename(image_path).split('.')[0]\n self.img_ext = os.path.basename(image_path).split('.')[-1]\n self.img_dir = os.path.dirname(image_path)\n self.blur_method = blur_method\n\n # Lambda functions to call the appropriate blur function according to set method\n # All the functions takes 2 arguments: the image and the kernel size to be used in the function\n # As the kernel size increases the amount of blur increases\n self.blur_function = {\n 'avg_blur': lambda img,ker_size: cv2.blur(img, (ker_size,ker_size)),\n 'gaussian': lambda img,ker_size: cv2.GaussianBlur(img, (ker_size,ker_size), 0),\n 'median': lambda img,ker_size: cv2.medianBlur(img, ker_size),\n 'bilateral': lambda img,ker_size: cv2.bilateralFilter(img, ker_size, 75, 75),\n }\n\n\n self.depth_data = np.load(os.path.join(self.img_dir, self.img_name + \"_disp.npy\"))\n self.img = cv2.imread(os.path.join(self.img_dir, self.img_name + \".\" + self.img_ext))\n self.blur_imgs = []\n\n # The blurred versions of the images can be precalculated\n self.blur_images()\n\n # Normalizes the depth values based on the depth of focus\n # The depth of focus is saved in point_of_focus\n # norm_depth_data holds the final normalized depth of focus\n # 0 value corresponds to the point which is focused\n # 1 value corresponds to the point furthest from the point of focus\n def normalize_pof(self):\n self.norm_depth_data = self.depth_data - self.point_of_focus\n self.norm_depth_data = np.abs(self.norm_depth_data)\n self.norm_depth_data = self.norm_depth_data / self.norm_depth_data.max()\n\n # Mouse callback function\n # The point of the click is taken as the point of focus and defocusing is performed around it\n def depth_callback(self, event, x, y, flags, param):\n if event == cv2.EVENT_LBUTTONDOWN:\n self.point_of_focus = self.depth_data[y][x]\n self.defocus_with_pof()\n\n # The depth map is normalized around the point of focus after which the defocusing of the image is done by sectioning and masking\n def defocus_with_pof(self):\n print(\"Point of focus: \", self.point_of_focus)\n self.normalize_pof()\n print(\"Normalized depth data around point of focus\")\n\n section_size = 1 / len(self.blur_imgs)\n final_image = np.zeros(self.img.shape)\n\n for index, blur_img in enumerate(self.blur_imgs):\n mask = (index*section_size <= self.norm_depth_data) & (self.norm_depth_data < (index+1)*section_size)\n masked_img = copy.deepcopy(blur_img)\n # Applying mask on copy of blurred image\n masked_img[mask==0] = [0, 0, 0]\n final_image = final_image + masked_img\n\n final_image = np.uint8(final_image)\n # cv2.imshow(\"Final\", final_image)\n cv2.imwrite(os.path.join(self.img_dir, self.img_name + \"_defocus.png\"), final_image)\n cv2.imshow(\"Project Defude\", final_image)\n\n\n def set_pof_from_coord(self, norm_x, norm_y):\n h, w = self.depth_data.shape[:2]\n self.point_of_focus = self.depth_data[int(h * norm_y)][int(w * norm_x)]\n self.defocus_with_pof()\n\n\n # Creates a window to display the original image\n # The callback function is attached to this window\n def view_image_for_blur(self):\n cv2.namedWindow(\"Project Defude\", flags = (cv2.WINDOW_GUI_NORMAL | cv2.WINDOW_AUTOSIZE))\n cv2.setMouseCallback(\"Project Defude\", self.depth_callback)\n\n cv2.imshow(\"Project Defude\", self.img)\n\n # cv2.namedWindow(\"Project Defude\", flags=cv2.WINDOW_GUI_NORMAL)\n while(cv2.waitKey() != 27):\n pass\n cv2.destroyAllWindows()\n\n # Generated different blurred versions of the original image\n # The blurred versions are stored in the list blur_imgs\n def blur_images(self):\n print(\"Generating blurred versions for image\")\n self.blur_imgs.append(self.img)\n for ker_size in range(5, 22, 4):\n self.blur_imgs.append(self.blur_function[self.blur_method](self.img, ker_size))\n\n # Show blurred images\n # for index, blur_img in enumerate(self.blur_imgs):\n # cv2.imshow(\"blur \" + str(index), blur_img)\n\n\nif __name__ == \"__main__\":\n PATH = args.image_path\n BLUR = args.blur_method\n\n defocuser = DefocuserObject(image_path = PATH, blur_method = BLUR)\n # defocuser.set_pof_from_coord(0.9, 0.9)\n defocuser.view_image_for_blur()\n" }, { "path": "depth/README.md", "content": "# Monodepth\nTensorflow implementation of unsupervised single image depth prediction using a convolutional neural network.\n\n

\n \"monodepth\"\n

\n\n**Unsupervised Monocular Depth Estimation with Left-Right Consistency**\n[Clément Godard](http://www0.cs.ucl.ac.uk/staff/C.Godard/), [Oisin Mac Aodha](http://vision.caltech.edu/~macaodha/) and [Gabriel J. Brostow](http://www0.cs.ucl.ac.uk/staff/g.brostow/)\nCVPR 2017\n\nFor more details:\n[github](http://visual.cs.ucl.ac.uk/pubs/monoDepth/),\n[project page](http://visual.cs.ucl.ac.uk/pubs/monoDepth/),\n[arXiv](https://arxiv.org/abs/1609.03677)\n\n\n\nThis code is a minimal stripped down version of the original monodepth code using OpenCV backend.\n\nAll ownership and rights to this code and it's implementation belong to the original project owners and their organization.\nPlease check the [original repository](https://github.com/mrharicot/monodepth) and their [LICENSE](https://github.com/mrharicot/monodepth/blob/master/LICENSE) file for more information.\n" }, { "path": "depth/average_gradients.py", "content": "# Copyright 2015 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom __future__ import absolute_import, division, print_function\nimport tensorflow as tf\n\ndef average_gradients(tower_grads):\n\n average_grads = []\n for grad_and_vars in zip(*tower_grads):\n # Note that each grad_and_vars looks like the following:\n # ((grad0_gpu0, var0_gpu0), ... , (grad0_gpuN, var0_gpuN))\n grads = []\n for g, _ in grad_and_vars:\n # Add 0 dimension to the gradients to represent the tower.\n expanded_g = tf.expand_dims(g, 0)\n\n # Append on a 'tower' dimension which we will average over below.\n grads.append(expanded_g)\n\n # Average over the 'tower' dimension.\n grad = tf.concat(axis=0, values=grads)\n grad = tf.reduce_mean(grad, 0)\n\n # Keep in mind that the Variables are redundant because they are shared\n # across towers. So .. we will just return the first tower's pointer to\n # the Variable.\n v = grad_and_vars[0][1]\n grad_and_var = (grad, v)\n average_grads.append(grad_and_var)\n return average_grads\n" }, { "path": "depth/bilinear_sampler.py", "content": "# Copyright 2016 The TensorFlow Authors. All Rights Reserved.\n# Copyright 2017 Modifications Clement Godard.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom __future__ import absolute_import, division, print_function\nimport tensorflow as tf\n\ndef bilinear_sampler_1d_h(input_images, x_offset, wrap_mode='border', name='bilinear_sampler', **kwargs):\n def _repeat(x, n_repeats):\n with tf.variable_scope('_repeat'):\n rep = tf.tile(tf.expand_dims(x, 1), [1, n_repeats])\n return tf.reshape(rep, [-1])\n\n def _interpolate(im, x, y):\n with tf.variable_scope('_interpolate'):\n\n # handle both texture border types\n _edge_size = 0\n if _wrap_mode == 'border':\n _edge_size = 1\n im = tf.pad(im, [[0, 0], [1, 1], [1, 1], [0, 0]], mode='CONSTANT')\n x = x + _edge_size\n y = y + _edge_size\n elif _wrap_mode == 'edge':\n _edge_size = 0\n else:\n return None\n\n x = tf.clip_by_value(x, 0.0, _width_f - 1 + 2 * _edge_size)\n\n x0_f = tf.floor(x)\n y0_f = tf.floor(y)\n x1_f = x0_f + 1\n\n x0 = tf.cast(x0_f, tf.int32)\n y0 = tf.cast(y0_f, tf.int32)\n x1 = tf.cast(tf.minimum(x1_f, _width_f - 1 + 2 * _edge_size), tf.int32)\n\n dim2 = (_width + 2 * _edge_size)\n dim1 = (_width + 2 * _edge_size) * (_height + 2 * _edge_size)\n base = _repeat(tf.range(_num_batch) * dim1, _height * _width)\n base_y0 = base + y0 * dim2\n idx_l = base_y0 + x0\n idx_r = base_y0 + x1\n\n im_flat = tf.reshape(im, tf.stack([-1, _num_channels]))\n\n pix_l = tf.gather(im_flat, idx_l)\n pix_r = tf.gather(im_flat, idx_r)\n\n weight_l = tf.expand_dims(x1_f - x, 1)\n weight_r = tf.expand_dims(x - x0_f, 1)\n\n return weight_l * pix_l + weight_r * pix_r\n\n def _transform(input_images, x_offset):\n with tf.variable_scope('transform'):\n # grid of (x_t, y_t, 1), eq (1) in ref [1]\n x_t, y_t = tf.meshgrid(tf.linspace(0.0, _width_f - 1.0, _width),\n tf.linspace(0.0 , _height_f - 1.0 , _height))\n\n x_t_flat = tf.reshape(x_t, (1, -1))\n y_t_flat = tf.reshape(y_t, (1, -1))\n\n x_t_flat = tf.tile(x_t_flat, tf.stack([_num_batch, 1]))\n y_t_flat = tf.tile(y_t_flat, tf.stack([_num_batch, 1]))\n\n x_t_flat = tf.reshape(x_t_flat, [-1])\n y_t_flat = tf.reshape(y_t_flat, [-1])\n\n x_t_flat = x_t_flat + tf.reshape(x_offset, [-1]) * _width_f\n\n input_transformed = _interpolate(input_images, x_t_flat, y_t_flat)\n\n output = tf.reshape(\n input_transformed, tf.stack([_num_batch, _height, _width, _num_channels]))\n return output\n\n with tf.variable_scope(name):\n _num_batch = tf.shape(input_images)[0]\n _height = tf.shape(input_images)[1]\n _width = tf.shape(input_images)[2]\n _num_channels = tf.shape(input_images)[3]\n\n _height_f = tf.cast(_height, tf.float32)\n _width_f = tf.cast(_width, tf.float32)\n\n _wrap_mode = wrap_mode\n\n output = _transform(input_images, x_offset)\n return output\n" }, { "path": "depth/depth_dataloader.py", "content": "# Copyright UCL Business plc 2017. Patent Pending. All rights reserved.\n#\n# The MonoDepth Software is licensed under the terms of the UCLB ACP-A licence\n# which allows for non-commercial use only, the full terms of which are made\n# available in the LICENSE file.\n#\n# For any other use of the software not covered by the UCLB ACP-A Licence,\n# please contact info@uclb.com\n\n\"\"\"Depth data loader.\n\"\"\"\n\nimport tensorflow as tf\n\ndef string_length_tf(t):\n return tf.py_func(len, [t], [tf.int64])\n\nclass DepthDataloader(object):\n \"\"\"Depth dataloader\"\"\"\n\n def __init__(self, data_path, filenames_file, params, mode):\n self.data_path = data_path\n self.params = params\n self.mode = mode\n\n self.left_image_batch = None\n self.right_image_batch = None\n\n input_queue = tf.train.string_input_producer([filenames_file], shuffle=False)\n line_reader = tf.TextLineReader()\n _, line = line_reader.read(input_queue)\n\n split_line = tf.string_split([line]).values\n\n # we load only one image for test\n if mode == 'test':\n left_image_path = tf.string_join([self.data_path, split_line[0]])\n left_image_o = self.read_image(left_image_path)\n\n if mode == 'train':\n # randomly flip images\n do_flip = tf.random_uniform([], 0, 1)\n left_image = tf.cond(do_flip > 0.5, lambda: tf.image.flip_left_right(right_image_o), lambda: left_image_o)\n right_image = tf.cond(do_flip > 0.5, lambda: tf.image.flip_left_right(left_image_o), lambda: right_image_o)\n\n # randomly augment images\n do_augment = tf.random_uniform([], 0, 1)\n left_image, right_image = tf.cond(do_augment > 0.5, lambda: self.augment_image_pair(left_image, right_image), lambda: (left_image, right_image))\n\n left_image.set_shape( [None, None, 3])\n right_image.set_shape([None, None, 3])\n\n # capacity = min_after_dequeue + (num_threads + a small safety margin) * batch_size\n min_after_dequeue = 2048\n capacity = min_after_dequeue + 4 * params.batch_size\n self.left_image_batch, self.right_image_batch = tf.train.shuffle_batch([left_image, right_image],\n params.batch_size, capacity, min_after_dequeue, params.num_threads)\n\n elif mode == 'test':\n self.left_image_batch = tf.stack([left_image_o, tf.image.flip_left_right(left_image_o)], 0)\n self.left_image_batch.set_shape( [2, None, None, 3])\n\n def augment_image_pair(self, left_image, right_image):\n # randomly shift gamma\n random_gamma = tf.random_uniform([], 0.8, 1.2)\n left_image_aug = left_image ** random_gamma\n right_image_aug = right_image ** random_gamma\n\n # randomly shift brightness\n random_brightness = tf.random_uniform([], 0.5, 2.0)\n left_image_aug = left_image_aug * random_brightness\n right_image_aug = right_image_aug * random_brightness\n\n # randomly shift color\n random_colors = tf.random_uniform([3], 0.8, 1.2)\n white = tf.ones([tf.shape(left_image)[0], tf.shape(left_image)[1]])\n color_image = tf.stack([white * random_colors[i] for i in range(3)], axis=2)\n left_image_aug *= color_image\n right_image_aug *= color_image\n\n # saturate\n left_image_aug = tf.clip_by_value(left_image_aug, 0, 1)\n right_image_aug = tf.clip_by_value(right_image_aug, 0, 1)\n\n return left_image_aug, right_image_aug\n\n def read_image(self, image_path):\n # tf.decode_image does not return the image size, this is an ugly workaround to handle both jpeg and png\n path_length = string_length_tf(image_path)[0]\n file_extension = tf.substr(image_path, path_length - 3, 3)\n file_cond = tf.equal(file_extension, 'jpg')\n\n image = tf.cond(file_cond, lambda: tf.image.decode_jpeg(tf.read_file(image_path)), lambda: tf.image.decode_png(tf.read_file(image_path)))\n\n image = tf.image.convert_image_dtype(image, tf.float32)\n image = tf.image.resize_images(image, [self.params.height, self.params.width], tf.image.ResizeMethod.AREA)\n\n return image\n" }, { "path": "depth/depth_model.py", "content": "# Copyright UCL Business plc 2017. Patent Pending. All rights reserved.\n#\n# The MonoDepth Software is licensed under the terms of the UCLB ACP-A licence\n# which allows for non-commercial use only, the full terms of which are made\n# available in the LICENSE file.\n#\n# For any other use of the software not covered by the UCLB ACP-A Licence,\n# please contact info@uclb.com\n\n\"\"\"Fully convolutional model for monocular depth estimation\n by Clement Godard, Oisin Mac Aodha and Gabriel J. Brostow\n http://visual.cs.ucl.ac.uk/pubs/monoDepth/\n\"\"\"\n\nfrom __future__ import absolute_import, division, print_function\nfrom collections import namedtuple\n\nimport numpy as np\nimport tensorflow as tf\nimport tensorflow.contrib.slim as slim\n\nfrom bilinear_sampler import *\n\ndepth_parameters = namedtuple('parameters',\n 'encoder, '\n 'height, width, '\n 'batch_size, '\n 'num_threads, '\n 'num_epochs, '\n 'wrap_mode, '\n 'use_deconv, '\n 'alpha_image_loss, '\n 'disp_gradient_loss_weight, '\n 'lr_loss_weight, '\n 'full_summary')\n\nclass DepthModel(object):\n \"\"\"depth model\"\"\"\n\n def __init__(self, params, mode, left, right, reuse_variables=None, model_index=0):\n self.params = params\n self.mode = mode\n self.left = left\n self.right = right\n self.model_collection = ['model_' + str(model_index)]\n\n self.reuse_variables = reuse_variables\n\n self.build_model()\n self.build_outputs()\n\n if self.mode == 'test':\n return\n\n self.build_losses()\n self.build_summaries()\n\n def gradient_x(self, img):\n gx = img[:,:,:-1,:] - img[:,:,1:,:]\n return gx\n\n def gradient_y(self, img):\n gy = img[:,:-1,:,:] - img[:,1:,:,:]\n return gy\n\n def upsample_nn(self, x, ratio):\n s = tf.shape(x)\n h = s[1]\n w = s[2]\n return tf.image.resize_nearest_neighbor(x, [h * ratio, w * ratio])\n\n def scale_pyramid(self, img, num_scales):\n scaled_imgs = [img]\n s = tf.shape(img)\n h = s[1]\n w = s[2]\n for i in range(num_scales - 1):\n ratio = 2 ** (i + 1)\n nh = h // ratio\n nw = w // ratio\n scaled_imgs.append(tf.image.resize_area(img, [nh, nw]))\n return scaled_imgs\n\n def generate_image_left(self, img, disp):\n return bilinear_sampler_1d_h(img, -disp)\n\n def generate_image_right(self, img, disp):\n return bilinear_sampler_1d_h(img, disp)\n\n def SSIM(self, x, y):\n C1 = 0.01 ** 2\n C2 = 0.03 ** 2\n\n mu_x = slim.avg_pool2d(x, 3, 1, 'VALID')\n mu_y = slim.avg_pool2d(y, 3, 1, 'VALID')\n\n sigma_x = slim.avg_pool2d(x ** 2, 3, 1, 'VALID') - mu_x ** 2\n sigma_y = slim.avg_pool2d(y ** 2, 3, 1, 'VALID') - mu_y ** 2\n sigma_xy = slim.avg_pool2d(x * y , 3, 1, 'VALID') - mu_x * mu_y\n\n SSIM_n = (2 * mu_x * mu_y + C1) * (2 * sigma_xy + C2)\n SSIM_d = (mu_x ** 2 + mu_y ** 2 + C1) * (sigma_x + sigma_y + C2)\n\n SSIM = SSIM_n / SSIM_d\n\n return tf.clip_by_value((1 - SSIM) / 2, 0, 1)\n\n def get_disparity_smoothness(self, disp, pyramid):\n disp_gradients_x = [self.gradient_x(d) for d in disp]\n disp_gradients_y = [self.gradient_y(d) for d in disp]\n\n image_gradients_x = [self.gradient_x(img) for img in pyramid]\n image_gradients_y = [self.gradient_y(img) for img in pyramid]\n\n weights_x = [tf.exp(-tf.reduce_mean(tf.abs(g), 3, keep_dims=True)) for g in image_gradients_x]\n weights_y = [tf.exp(-tf.reduce_mean(tf.abs(g), 3, keep_dims=True)) for g in image_gradients_y]\n\n smoothness_x = [disp_gradients_x[i] * weights_x[i] for i in range(4)]\n smoothness_y = [disp_gradients_y[i] * weights_y[i] for i in range(4)]\n return smoothness_x + smoothness_y\n\n def get_disp(self, x):\n disp = 0.3 * self.conv(x, 2, 3, 1, tf.nn.sigmoid)\n return disp\n\n def conv(self, x, num_out_layers, kernel_size, stride, activation_fn=tf.nn.elu):\n p = np.floor((kernel_size - 1) / 2).astype(np.int32)\n p_x = tf.pad(x, [[0, 0], [p, p], [p, p], [0, 0]])\n return slim.conv2d(p_x, num_out_layers, kernel_size, stride, 'VALID', activation_fn=activation_fn)\n\n def conv_block(self, x, num_out_layers, kernel_size):\n conv1 = self.conv(x, num_out_layers, kernel_size, 1)\n conv2 = self.conv(conv1, num_out_layers, kernel_size, 2)\n return conv2\n\n def maxpool(self, x, kernel_size):\n p = np.floor((kernel_size - 1) / 2).astype(np.int32)\n p_x = tf.pad(x, [[0, 0], [p, p], [p, p], [0, 0]])\n return slim.max_pool2d(p_x, kernel_size)\n\n def resconv(self, x, num_layers, stride):\n do_proj = tf.shape(x)[3] != num_layers or stride == 2\n shortcut = []\n conv1 = self.conv(x, num_layers, 1, 1)\n conv2 = self.conv(conv1, num_layers, 3, stride)\n conv3 = self.conv(conv2, 4 * num_layers, 1, 1, None)\n if do_proj:\n shortcut = self.conv(x, 4 * num_layers, 1, stride, None)\n else:\n shortcut = x\n return tf.nn.elu(conv3 + shortcut)\n\n def resblock(self, x, num_layers, num_blocks):\n out = x\n for i in range(num_blocks - 1):\n out = self.resconv(out, num_layers, 1)\n out = self.resconv(out, num_layers, 2)\n return out\n\n def upconv(self, x, num_out_layers, kernel_size, scale):\n upsample = self.upsample_nn(x, scale)\n conv = self.conv(upsample, num_out_layers, kernel_size, 1)\n return conv\n\n def deconv(self, x, num_out_layers, kernel_size, scale):\n p_x = tf.pad(x, [[0, 0], [1, 1], [1, 1], [0, 0]])\n conv = slim.conv2d_transpose(p_x, num_out_layers, kernel_size, scale, 'SAME')\n return conv[:,3:-1,3:-1,:]\n\n def build_resnet50(self):\n #set convenience functions\n conv = self.conv\n if self.params.use_deconv:\n upconv = self.deconv\n else:\n upconv = self.upconv\n\n with tf.variable_scope('encoder'):\n conv1 = conv(self.model_input, 64, 7, 2) # H/2 - 64D\n pool1 = self.maxpool(conv1, 3) # H/4 - 64D\n conv2 = self.resblock(pool1, 64, 3) # H/8 - 256D\n conv3 = self.resblock(conv2, 128, 4) # H/16 - 512D\n conv4 = self.resblock(conv3, 256, 6) # H/32 - 1024D\n conv5 = self.resblock(conv4, 512, 3) # H/64 - 2048D\n\n with tf.variable_scope('skips'):\n skip1 = conv1\n skip2 = pool1\n skip3 = conv2\n skip4 = conv3\n skip5 = conv4\n\n # DECODING\n with tf.variable_scope('decoder'):\n upconv6 = upconv(conv5, 512, 3, 2) #H/32\n concat6 = tf.concat([upconv6, skip5], 3)\n iconv6 = conv(concat6, 512, 3, 1)\n\n upconv5 = upconv(iconv6, 256, 3, 2) #H/16\n concat5 = tf.concat([upconv5, skip4], 3)\n iconv5 = conv(concat5, 256, 3, 1)\n\n upconv4 = upconv(iconv5, 128, 3, 2) #H/8\n concat4 = tf.concat([upconv4, skip3], 3)\n iconv4 = conv(concat4, 128, 3, 1)\n self.disp4 = self.get_disp(iconv4)\n udisp4 = self.upsample_nn(self.disp4, 2)\n\n upconv3 = upconv(iconv4, 64, 3, 2) #H/4\n concat3 = tf.concat([upconv3, skip2, udisp4], 3)\n iconv3 = conv(concat3, 64, 3, 1)\n self.disp3 = self.get_disp(iconv3)\n udisp3 = self.upsample_nn(self.disp3, 2)\n\n upconv2 = upconv(iconv3, 32, 3, 2) #H/2\n concat2 = tf.concat([upconv2, skip1, udisp3], 3)\n iconv2 = conv(concat2, 32, 3, 1)\n self.disp2 = self.get_disp(iconv2)\n udisp2 = self.upsample_nn(self.disp2, 2)\n\n upconv1 = upconv(iconv2, 16, 3, 2) #H\n concat1 = tf.concat([upconv1, udisp2], 3)\n iconv1 = conv(concat1, 16, 3, 1)\n self.disp1 = self.get_disp(iconv1)\n\n def build_model(self):\n with slim.arg_scope([slim.conv2d, slim.conv2d_transpose], activation_fn=tf.nn.elu):\n with tf.variable_scope('model', reuse=self.reuse_variables):\n\n self.left_pyramid = self.scale_pyramid(self.left, 4)\n if self.mode == 'train':\n self.right_pyramid = self.scale_pyramid(self.right, 4)\n\n self.model_input = self.left\n\n #build model\n if self.params.encoder == 'resnet50':\n self.build_resnet50()\n else:\n return None\n\n def build_outputs(self):\n # STORE DISPARITIES\n with tf.variable_scope('disparities'):\n self.disp_est = [self.disp1, self.disp2, self.disp3, self.disp4]\n self.disp_left_est = [tf.expand_dims(d[:,:,:,0], 3) for d in self.disp_est]\n self.disp_right_est = [tf.expand_dims(d[:,:,:,1], 3) for d in self.disp_est]\n\n if self.mode == 'test':\n return\n\n # GENERATE IMAGES\n with tf.variable_scope('images'):\n self.left_est = [self.generate_image_left(self.right_pyramid[i], self.disp_left_est[i]) for i in range(4)]\n self.right_est = [self.generate_image_right(self.left_pyramid[i], self.disp_right_est[i]) for i in range(4)]\n\n # LR CONSISTENCY\n with tf.variable_scope('left-right'):\n self.right_to_left_disp = [self.generate_image_left(self.disp_right_est[i], self.disp_left_est[i]) for i in range(4)]\n self.left_to_right_disp = [self.generate_image_right(self.disp_left_est[i], self.disp_right_est[i]) for i in range(4)]\n\n # DISPARITY SMOOTHNESS\n with tf.variable_scope('smoothness'):\n self.disp_left_smoothness = self.get_disparity_smoothness(self.disp_left_est, self.left_pyramid)\n self.disp_right_smoothness = self.get_disparity_smoothness(self.disp_right_est, self.right_pyramid)\n\n def build_losses(self):\n with tf.variable_scope('losses', reuse=self.reuse_variables):\n # IMAGE RECONSTRUCTION\n # L1\n self.l1_left = [tf.abs( self.left_est[i] - self.left_pyramid[i]) for i in range(4)]\n self.l1_reconstruction_loss_left = [tf.reduce_mean(l) for l in self.l1_left]\n self.l1_right = [tf.abs(self.right_est[i] - self.right_pyramid[i]) for i in range(4)]\n self.l1_reconstruction_loss_right = [tf.reduce_mean(l) for l in self.l1_right]\n\n # SSIM\n self.ssim_left = [self.SSIM( self.left_est[i], self.left_pyramid[i]) for i in range(4)]\n self.ssim_loss_left = [tf.reduce_mean(s) for s in self.ssim_left]\n self.ssim_right = [self.SSIM(self.right_est[i], self.right_pyramid[i]) for i in range(4)]\n self.ssim_loss_right = [tf.reduce_mean(s) for s in self.ssim_right]\n\n # WEIGTHED SUM\n self.image_loss_right = [self.params.alpha_image_loss * self.ssim_loss_right[i] + (1 - self.params.alpha_image_loss) * self.l1_reconstruction_loss_right[i] for i in range(4)]\n self.image_loss_left = [self.params.alpha_image_loss * self.ssim_loss_left[i] + (1 - self.params.alpha_image_loss) * self.l1_reconstruction_loss_left[i] for i in range(4)]\n self.image_loss = tf.add_n(self.image_loss_left + self.image_loss_right)\n\n # DISPARITY SMOOTHNESS\n self.disp_left_loss = [tf.reduce_mean(tf.abs(self.disp_left_smoothness[i])) / 2 ** i for i in range(4)]\n self.disp_right_loss = [tf.reduce_mean(tf.abs(self.disp_right_smoothness[i])) / 2 ** i for i in range(4)]\n self.disp_gradient_loss = tf.add_n(self.disp_left_loss + self.disp_right_loss)\n\n # LR CONSISTENCY\n self.lr_left_loss = [tf.reduce_mean(tf.abs(self.right_to_left_disp[i] - self.disp_left_est[i])) for i in range(4)]\n self.lr_right_loss = [tf.reduce_mean(tf.abs(self.left_to_right_disp[i] - self.disp_right_est[i])) for i in range(4)]\n self.lr_loss = tf.add_n(self.lr_left_loss + self.lr_right_loss)\n\n # TOTAL LOSS\n self.total_loss = self.image_loss + self.params.disp_gradient_loss_weight * self.disp_gradient_loss + self.params.lr_loss_weight * self.lr_loss\n\n def build_summaries(self):\n # SUMMARIES\n with tf.device('/cpu:0'):\n for i in range(4):\n tf.summary.scalar('ssim_loss_' + str(i), self.ssim_loss_left[i] + self.ssim_loss_right[i], collections=self.model_collection)\n tf.summary.scalar('l1_loss_' + str(i), self.l1_reconstruction_loss_left[i] + self.l1_reconstruction_loss_right[i], collections=self.model_collection)\n tf.summary.scalar('image_loss_' + str(i), self.image_loss_left[i] + self.image_loss_right[i], collections=self.model_collection)\n tf.summary.scalar('disp_gradient_loss_' + str(i), self.disp_left_loss[i] + self.disp_right_loss[i], collections=self.model_collection)\n tf.summary.scalar('lr_loss_' + str(i), self.lr_left_loss[i] + self.lr_right_loss[i], collections=self.model_collection)\n tf.summary.image('disp_left_est_' + str(i), self.disp_left_est[i], max_outputs=4, collections=self.model_collection)\n tf.summary.image('disp_right_est_' + str(i), self.disp_right_est[i], max_outputs=4, collections=self.model_collection)\n" }, { "path": "depth/depth_simple.py", "content": "# Copyright UCL Business plc 2017. Patent Pending. All rights reserved.\n#\n# The MonoDepth Software is licensed under the terms of the UCLB ACP-A licence\n# which allows for non-commercial use only, the full terms of which are made\n# available in the LICENSE file.\n#\n# For any other use of the software not covered by the UCLB ACP-A Licence,\n# please contact info@uclb.com\n\n# only keep warnings and errors\nimport os\nos.environ['TF_CPP_MIN_LOG_LEVEL']='0'\n\nimport numpy as np\nimport argparse\nimport re\nimport time\nimport tensorflow as tf\nimport tensorflow.contrib.slim as slim\nimport cv2\nimport matplotlib\nmatplotlib.use('agg')\nimport matplotlib.pyplot as plt\n\n\nfrom depth_model import *\nfrom depth_dataloader import *\nfrom average_gradients import *\n\nparser = argparse.ArgumentParser(description='Depth TensorFlow implementation.')\n\nparser.add_argument('--encoder', type=str, help='type of encoder, resnet50', default='resnet50')\nparser.add_argument('--image_path', type=str, help='path to the image', required=True)\nparser.add_argument('--checkpoint_path', type=str, help='path to a specific checkpoint to load', required=True)\nparser.add_argument('--input_height', type=int, help='input height', default=256)\nparser.add_argument('--input_width', type=int, help='input width', default=512)\n\nargs = parser.parse_args()\n\ndef post_process_disparity(disp):\n _, h, w = disp.shape\n l_disp = disp[0,:,:]\n r_disp = np.fliplr(disp[1,:,:])\n m_disp = 0.5 * (l_disp + r_disp)\n l, _ = np.meshgrid(np.linspace(0, 1, w), np.linspace(0, 1, h))\n l_mask = 1.0 - np.clip(20 * (l - 0.05), 0, 1)\n r_mask = np.fliplr(l_mask)\n return r_mask * l_disp + l_mask * r_disp + (1.0 - l_mask - r_mask) * m_disp\n\ndef test_simple(params):\n \"\"\"Test function.\"\"\"\n\n left = tf.placeholder(tf.float32, [2, args.input_height, args.input_width, 3])\n model = DepthModel(params, \"test\", left, None)\n\n input_image = cv2.imread(args.image_path)\n input_image = cv2.cvtColor(input_image, cv2.COLOR_BGR2RGB)\n original_height, original_width, num_channels = input_image.shape\n input_image = cv2.resize(input_image, dsize = (args.input_width, args.input_height))\n input_image = input_image.astype(np.float32) / 255\n input_images = np.stack((input_image, np.fliplr(input_image)), 0)\n\n # SESSION\n config = tf.ConfigProto(allow_soft_placement=True)\n sess = tf.Session(config=config)\n\n # SAVER\n train_saver = tf.train.Saver()\n\n # INIT\n sess.run(tf.global_variables_initializer())\n sess.run(tf.local_variables_initializer())\n coordinator = tf.train.Coordinator()\n\n # RESTORE\n restore_path = args.checkpoint_path.split(\".\")[0]\n train_saver.restore(sess, restore_path)\n\n disp = sess.run(model.disp_left_est[0], feed_dict={left: input_images})\n disp_pp = post_process_disparity(disp.squeeze()).astype(np.float32)\n\n output_directory = os.path.dirname(args.image_path)\n output_name = os.path.splitext(os.path.basename(args.image_path))[0]\n\n # np.save(os.path.join(output_directory, \"{}_disp.npy\".format(output_name)), disp_pp)\n disp_to_img = cv2.resize(disp_pp.squeeze(), dsize = (original_width, original_height))\n np.save(os.path.join(output_directory, \"{}_disp.npy\".format(output_name)), disp_to_img)\n plt.imsave(os.path.join(output_directory, \"{}_disp.png\".format(output_name)), disp_to_img, cmap='plasma')\n\n print('done!')\n\ndef main(_):\n\n params = depth_parameters(\n encoder=args.encoder,\n height=args.input_height,\n width=args.input_width,\n batch_size=2,\n num_threads=1,\n num_epochs=1,\n wrap_mode=\"border\",\n use_deconv=False,\n alpha_image_loss=0,\n disp_gradient_loss_weight=0,\n lr_loss_weight=0,\n full_summary=False)\n\n test_simple(params)\n\nif __name__ == '__main__':\n tf.app.run()\n" }, { "path": "depth/get_model.sh", "content": "model_name=$1\noutput_location=$2\n\nfilename=$model_name.zip\n\nurl=http://visual.cs.ucl.ac.uk/pubs/monoDepth/models/$filename\n\noutput_file=$output_location/$filename\n\necho \"Downloading $model_name\"\nwget -nc $url -O $output_file\nunzip $output_file -d $output_location\nrm $output_file\n" }, { "path": "gui.py", "content": "import gi\n\ngi.require_version('Gtk','3.0')\n\nfrom gi.repository import Gtk, Gdk, GdkPixbuf\nimport threading\nfrom functools import partial\nimport os\nfrom defocus.defocus import DefocuserObject\nimport cv2\n\nclass DefudeGui(object):\n\n def __init__(self,checkpoint_path,glade_file='ui-stepper.glade'):\n self.builder = Gtk.Builder()\n\n # load the glade file describing the UI\n self.builder.add_from_file(glade_file)\n\n # connect the event handlers\n self.builder.connect_signals(self)\n\n # all the available windows\n self.main_window = self.builder.get_object('main-window')\n self.about_dialog = self.builder.get_object('about-dialog')\n self.help_dialog = self.builder.get_object('help-dialog')\n\n\n # some ui elements\n self.input_image_drop = self.builder.get_object('input-image-drop')\n self.select_input_image_picker = self.builder.get_object('select-image-file-picker')\n self.step_stack = self.builder.get_object('main-step-stack')\n self.image_drop_dest = self.builder.get_object('image-drag-drop-dest')\n self.input_image_spinner = self.builder.get_object('input-image-spinner')\n self.input_image_status_line = self.builder.get_object('input-image-status')\n self.input_image_delete_btn = self.builder.get_object('delete-input-img-btn')\n self.input_image_next_btn = self.builder.get_object('input-image-next-btn')\n self.depth_map_image = self.builder.get_object('depth-map-image')\n self.pof_image = self.builder.get_object('pick-pof-image')\n self.pof_status_line = self.builder.get_object('pof-status-line')\n self.pof_status_spinner = self.builder.get_object('pof-status-spinner')\n self.result_image = self.builder.get_object('result-image')\n self.header_bar = self.builder.get_object('main-header-bar')\n\n page_ids = (\n 'start-page',\n 'select-image-page',\n 'depth-map-preview',\n 'pick-pof',\n 'result-page'\n )\n\n self.pages = tuple(\n self.builder.get_object(page_id)\n for page_id in page_ids\n )\n\n # some useful constants\n self.IMAGE_WIDTH = 600\n self.CHECKPOINT_PATH = checkpoint_path\n self.STATUS_MESSAGES = {\n 'depth_est_running': 'Estimating depthmap...',\n 'defocus_running': 'Performing defocusing...',\n 'idle': '',\n 'resizing': 'Loading the image...'\n }\n self.DEFAULT_INPUT_IMAGE = 'assets/drag-and-drop.png'\n self.WINDOW_TITLE = 'Synthetic Defocusing and Depth Estimation Tool'\n\n # state variables\n self.INPUT_IMAGE_PATH = None\n self.INPUT_IMAGE_SIZE = None\n self.DEPTH_MAP_PATH = None\n self.DEFOCUS_IMAGE_PATH = None\n self.CURRENT_STACK_PAGE = 0\n\n # set the image as a drag drop destination\n self.image_drop_dest.drag_dest_set(\n # do all the default stuff\n Gtk.DestDefaults.ALL,\n # enforce target\n [Gtk.TargetEntry.new('text/plain',Gtk.TargetFlags(4), 129)],\n Gdk.DragAction.COPY\n )\n\n def _next_page(self):\n if self.CURRENT_STACK_PAGE < len(self.pages) - 1:\n self.CURRENT_STACK_PAGE += 1\n self.step_stack.set_visible_child(self.pages[self.CURRENT_STACK_PAGE])\n\n def _prev_page(self):\n if self.CURRENT_STACK_PAGE > 0:\n self.CURRENT_STACK_PAGE -= 1\n self.step_stack.set_visible_child(self.pages[self.CURRENT_STACK_PAGE])\n\n def _set_input_image_impl(self, img_path):\n spinner = self.input_image_spinner\n status_line = self.input_image_status_line\n self.INPUT_IMAGE_PATH = img_path\n img,w,h = self._resize_image(img_path,return_size=True)\n self.INPUT_IMAGE_SIZE = (w,h)\n self.input_image_drop.set_from_pixbuf(img)\n self.select_input_image_picker.hide()\n\n spinner.stop()\n status_line.set_label(self.STATUS_MESSAGES['idle'])\n self.input_image_delete_btn.set_sensitive(True)\n self.input_image_next_btn.set_sensitive(True)\n\n def _set_input_image(self, img_path):\n\n spinner = self.input_image_spinner\n status_line = self.input_image_status_line\n spinner.start()\n status_line.set_label(self.STATUS_MESSAGES['resizing'])\n\n thread = threading.Thread(target=partial(self._set_input_image_impl,img_path))\n thread.daemon = True\n thread.start()\n\n\n def _unset_input_image(self):\n self.INPUT_IMAGE_PATH = None\n self.INPUT_IMAGE_SIZE = None\n self.input_image_drop.set_from_file(self.DEFAULT_INPUT_IMAGE)\n self.select_input_image_picker.show()\n self.input_image_next_btn.set_sensitive(False)\n\n def _resize_image(self,path,size=None,return_size=False):\n img = GdkPixbuf.Pixbuf.new_from_file(path)\n if size is None:\n h = img.get_height()\n w = img.get_width()\n\n ar = h / float(w)\n size = (self.IMAGE_WIDTH,int(self.IMAGE_WIDTH*ar))\n\n resized = img.scale_simple(size[0],size[1],GdkPixbuf.InterpType.BILINEAR)\n\n if return_size:\n return resized,size[0],size[1]\n\n return resized\n\n def _estimate_depthmap_impl(self):\n os.system(\"python ./depth/depth_simple.py --checkpoint_path \" + self.CHECKPOINT_PATH + \" --image_path \" + self.INPUT_IMAGE_PATH)\n self.DEPTH_MAP_PATH = os.path.join(os.path.dirname(self.INPUT_IMAGE_PATH), os.path.basename(self.INPUT_IMAGE_PATH).split('.')[0]) + '_disp.png'\n\n depthmap_img = self._resize_image(self.DEPTH_MAP_PATH)\n self.depth_map_image.set_from_pixbuf(depthmap_img)\n\n self.input_image_spinner.stop()\n self.input_image_status_line.set_label(self.STATUS_MESSAGES['idle'])\n self._next_page()\n self.input_image_next_btn.set_sensitive(True)\n self.input_image_delete_btn.set_sensitive(True)\n\n\n\n def _estimate_depthmap(self):\n self.input_image_next_btn.set_sensitive(False)\n self.input_image_delete_btn.set_sensitive(False)\n self.input_image_spinner.start()\n self.input_image_status_line.set_label(self.STATUS_MESSAGES['depth_est_running'])\n\n thread = threading.Thread(target=self._estimate_depthmap_impl)\n thread.daemon = True\n thread.start()\n\n def _defocus_image_impl(self, x_norm, y_norm):\n defocusser = DefocuserObject(self.INPUT_IMAGE_PATH)\n defocusser.set_pof_from_coord(x_norm,y_norm)\n self.DEFOCUS_IMAGE_PATH = os.path.join(os.path.dirname(self.INPUT_IMAGE_PATH), os.path.basename(self.INPUT_IMAGE_PATH).split('.')[0]) + '_defocus.png'\n img = self._resize_image(self.DEFOCUS_IMAGE_PATH)\n self.result_image.set_from_pixbuf(img)\n self.pof_status_spinner.stop()\n self.pof_status_line.set_label(self.STATUS_MESSAGES['idle'])\n self._next_page()\n\n def _defocus_image(self, x_norm, y_norm):\n self.pof_status_spinner.start()\n self.pof_status_line.set_label(self.STATUS_MESSAGES['defocus_running'])\n thread = threading.Thread(target=partial(self._defocus_image_impl,x_norm,y_norm))\n thread.daemon = True\n thread.start()\n\n def _save(self, src_filename):\n dialog = Gtk.FileChooserDialog(\"Save as\",self.main_window,Gtk.FileChooserAction.SAVE,(Gtk.STOCK_SAVE,Gtk.ResponseType.OK,Gtk.STOCK_CANCEL,Gtk.ResponseType.CANCEL))\n response = dialog.run()\n if response == Gtk.ResponseType.OK:\n img = cv2.imread(src_filename)\n cv2.imwrite(dialog.get_filename(),img)\n dialog.destroy()\n\n def _cleanup(self):\n file_list = (\n self.DEFOCUS_IMAGE_PATH,\n self.DEPTH_MAP_PATH,\n )\n\n for file in file_list:\n if file:\n if os.path.isfile(file):\n os.remove(file)\n\n\n def show(self):\n self.main_window.show_all()\n Gtk.main()\n\n def onDestroy(self, *args):\n Gtk.main_quit()\n # self._cleanup()\n\n def onStartPageNext(self, *args):\n self._next_page()\n\n def onBack(self, *args):\n self._prev_page()\n\n def onAbout(self, *args):\n self.about_dialog.run()\n self.about_dialog.hide()\n\n def onHelp(self, *args):\n self.help_dialog.run()\n self.help_dialog.hide()\n\n def onImagePickerSet(self, *args):\n input_file_name = args[0].get_filename()\n self._set_input_image(input_file_name)\n\n def onImageDrop(self, *args):\n filename = args[4].get_text()[7:-1]\n self._set_input_image(filename)\n\n def onDeleteInputImage(self, *args):\n self._unset_input_image()\n self.input_image_delete_btn.set_sensitive(False)\n\n def onInputImageNextBtn(self, *args):\n self._estimate_depthmap()\n\n def onDepthMapNextBtn(self, *args):\n img = self.input_image_drop.get_pixbuf()\n self.pof_image.set_from_pixbuf(img)\n self._next_page()\n\n def onDepthMapSave(self, *args):\n self._save(self.DEPTH_MAP_PATH)\n\n def onPofPick(self, *args):\n event = args[1]\n x,y = event.x, event.y\n x_norm = x / self.INPUT_IMAGE_SIZE[0]\n y_norm = y / self.INPUT_IMAGE_SIZE[1]\n print(\"x_norm: {} y_norm: {}\".format(x_norm,y_norm))\n self._defocus_image(x_norm,y_norm)\n\n def onResultSave(self, *args):\n self._save(self.DEFOCUS_IMAGE_PATH)\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser(description='Synthetic Defocussing Using Depth Estimation')\n\n parser.add_argument('--model_path', type=str, help='path to saved model', required=True)\n\n args = parser.parse_args()\n model_path = os.path.abspath(args.model_path)\n\n gui = DefudeGui(model_path)\n gui.show()\n" }, { "path": "main.py", "content": "import argparse\nimport os\n\nparser = argparse.ArgumentParser(description='Synthetic Defocussing Using Depth Estimation')\n\nparser.add_argument('--image_path', type=str, help='path to input image', default='images/sample2.png')\nparser.add_argument('--model_path', type=str, help='path to saved model', default='blah')\nparser.add_argument('--blur_method', type=str, help='the type of blur to be applied', default='gaussian')\n\nargs = parser.parse_args()\n\nimg_path = os.path.abspath(args.image_path)\nmodel_path = os.path.abspath(args.model_path)\nblur_method = args.blur_method\n\nos.system(\"python ./depth/depth_simple.py --model_path \" + model_path + \" --image_path \" + img_path)\nos.system(\"python ./defocus/defocus.py --image_path \" + img_path + \" --blur_method \" + blur_method)\n" }, { "path": "preprocessing.py", "content": "# Project: Synthetic Defocusing using Unsupervised Monocular Depth Estimation\n# This file contains the function for preprocessing stage\n\n# Input: Image file of any image format extension\n# Output: Preprocessed image for input to the model\n\nimport cv2\nimport numpy as np\nimport glob # Used for file access\n\n# Performs preprocessing functions on the given image\n# Params:\n# img: Image in OpenCV image type (numpy.ndarray)\n# Returns: Preprocessed image in OpenCV image type\ndef preprocess(img):\n # Denoising\n denoised_img = cv2.fastNlMeansDenoisingColored(img)\n\n # Conversion to Grayscale\n # gray_img = cv2.cvtColor(denoised_img, cv2.COLOR_BGR2GRAY)\n\n # Sharpening\n kernel = np.array([[0,-1,0], [-1,5,-1], [0,-1,0]])\n sharpen_img = cv2.filter2D(denoised_img, -1, kernel)\n\n # Histogram Equalization\n equ_img = cv2.equalizeHist(sharpen_img)\n clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))\n clahe = cv2.createCLAHE()\n equ_img = clahe.apply(sharpen_img)\n\n # Resizing\n resized_img = cv2.resize(sharpen_img, (512,256), cv2.INTER_AREA)\n\n prep_img = resized_img\n return prep_img\n\n# Run the preprocessing functions on a single image with before and after\n# Escape Key terminates the windows\n# The output of the file can be written to a file by setting output param\n# Params\n# file: Filename of the image as string\n# waitTime: How long the image should be shown in ms\n# output: Write the preprocessed file here\n# Returns: false if terminated using Esc key else true\ndef preprocess_single(file, waitTime = 0, output = \"\"):\n img = cv2.imread(file)\n prep_img = preprocess(img)\n\n if output:\n cv2.imwrite(output, prep_img)\n\n cv2.imshow(\"Original Image\", img)\n cv2.imshow(\"Preprocessed Image\", prep_img)\n\n if cv2.waitKey(waitTime) == 27:\n cv2.destroyAllWindows()\n return 0\n cv2.destroyAllWindows()\n\n# Performs preprocessing function on multiple images in a folder\n# Params:\n# folder_path: Folder name which holds the images with ending slash\ndef preprocess_multiple(folder_path):\n folder_path = folder_path + \".png\"\n for file in glob.glob(folder_path):\n preprocess_single(file, 2000)\n\n\nif __name__ == \"__main__\":\n preprocess_single(\"./images/sample0.png\", output=\"./sample0_pre.png\")\n\n" }, { "path": "requirements.txt", "content": "cycler==0.10.0\nkiwisolver==1.1.0\nmatplotlib==3.0.3\nnumpy==1.16.3\nopencv-python==4.1.0.25\nprotobuf==3.7.1\npycairo==1.18.1\nPyGObject==3.32.1\npyparsing==2.4.0\npython-dateutil==2.8.0\nsix==1.12.0\ntensorflow==1.15.4\n" }, { "path": "screenshot/README.md", "content": "## Our beautiful UI :D\n\n![defude_home](defude_home.png)\n![defude_select_image](defude_select_image.png)\n![defude_estimating_depth](defude_estim_depth.png)\n![defude_depthmap](defude_depthmap.png)\n![defude_defocused](defude_defocused.png)\n" }, { "path": "ui-stepper.glade", "content": "\n\n\n \n \n \n *.png\n *.jpg\n *.jpeg\n \n \n \n 600\n -1\n False\n \n \n True\n False\n Synthetic Defocusing and Depth Estimation Tool\n False\n True\n \n \n \n True\n True\n True\n True\n \n \n \n True\n False\n gtk-go-back\n \n \n \n \n \n \n \n \n True\n False\n 10\n vertical\n \n \n True\n False\n 15\n 15\n 500\n slide-left-right\n \n \n True\n False\n 1\n vertical\n \n \n True\n False\n assets/aperture-logo.png\n \n \n False\n True\n 12\n 0\n \n \n \n \n True\n False\n <big><b>Synthetic Defocusing and Depth Estimation Tool</b></big>\n True\n \n \n False\n True\n 1\n \n \n \n \n True\n False\n v1.0.1\n \n \n \n \n \n False\n True\n 7\n 2\n \n \n \n \n True\n False\n 20\n 10\n Click <b>Next</b> to start\n True\n \n \n False\n True\n 3\n \n \n \n \n Next\n True\n True\n True\n center\n center\n True\n \n \n \n True\n True\n 4\n \n \n \n \n True\n False\n \n \n gtk-help\n True\n True\n True\n True\n True\n \n \n \n False\n True\n 0\n \n \n \n \n gtk-about\n True\n True\n True\n end\n True\n right\n True\n \n \n \n True\n True\n 1\n \n \n \n \n False\n True\n 5\n \n \n \n \n Start\n start\n \n \n \n \n True\n False\n vertical\n \n \n True\n False\n center\n 10\n <big>Select the input image</big>\n True\n \n \n False\n True\n 0\n \n \n \n \n True\n False\n \n \n \n True\n False\n assets/drag-and-drop.png\n \n \n True\n 1\n \n \n \n \n True\n False\n center\n center\n 12\n 48\n imagefilefilter\n \n \n \n \n \n \n False\n True\n 25\n 1\n \n \n \n \n True\n False\n \n \n True\n False\n \n \n False\n True\n 0\n \n \n \n \n True\n False\n 5\n \n \n False\n True\n 1\n \n \n \n \n Continue\n True\n False\n True\n True\n end\n \n \n \n True\n True\n 2\n \n \n \n \n gtk-delete\n True\n False\n True\n True\n end\n 7\n True\n right\n True\n \n \n \n False\n True\n 3\n \n \n \n \n False\n True\n end\n 2\n \n \n \n \n page1\n page1\n 1\n \n \n \n \n True\n False\n vertical\n \n \n True\n False\n <big> Estimated Depthmap </big>\n True\n \n \n False\n True\n 7\n 0\n \n \n \n \n True\n False\n center\n center\n gtk-missing-image\n \n \n True\n True\n 1\n \n \n \n \n True\n False\n Click <b>Next</b> to continue\n True\n \n \n False\n True\n 2\n \n \n \n \n True\n False\n \n \n gtk-save\n True\n True\n True\n True\n right\n True\n \n \n \n False\n True\n 0\n \n \n \n \n Next\n True\n True\n True\n end\n \n \n \n True\n True\n 1\n \n \n \n \n False\n True\n 3\n \n \n \n \n page0\n page0\n 2\n \n \n \n \n True\n False\n vertical\n \n \n True\n False\n <big> Pick a Point of Focus </big>\n True\n \n \n False\n True\n 7\n 0\n \n \n \n \n True\n False\n Click anywhere in the image to set it as the <b>point of focus</b>.\n True\n \n \n False\n True\n 1\n \n \n \n \n True\n False\n \n \n \n True\n False\n center\n center\n gtk-missing-image\n \n \n \n \n False\n True\n 19\n 2\n \n \n \n \n True\n False\n \n \n True\n False\n \n \n False\n True\n 0\n \n \n \n \n True\n False\n \n \n False\n True\n 1\n \n \n \n \n False\n True\n 3\n \n \n \n \n page2\n page2\n 3\n \n \n \n \n True\n False\n vertical\n \n \n True\n False\n <big> Defocused Image </big>\n True\n \n \n False\n True\n 8\n 0\n \n \n \n \n True\n False\n center\n center\n gtk-missing-image\n \n \n True\n True\n 1\n \n \n \n \n gtk-save-as\n True\n True\n True\n end\n True\n right\n True\n \n \n \n False\n True\n 2\n \n \n \n \n page3\n page3\n 4\n \n \n \n \n True\n True\n 1\n \n \n \n \n \n \n False\n About\n 320\n dialog\n main-window\n \n \n \n \n \n False\n vertical\n 2\n \n \n False\n end\n \n \n \n \n \n \n \n \n False\n False\n 0\n \n \n \n \n True\n False\n vertical\n \n \n True\n False\n assets/aperture-logo.png\n \n \n False\n True\n 14\n 0\n \n \n \n \n True\n False\n <big><b>Sythetic Defocusing using Unsupervised Monocular Depth Estimation </b></big>\n True\n center\n True\n \n \n False\n True\n 10\n 1\n \n \n \n \n True\n False\n v1.0.1\n \n \n \n \n \n False\n True\n 11\n 2\n \n \n \n \n True\n False\n Synthetic Defocusing and Depth Estimation Tool allows you to generate a shallow depth-of-field image from an all-in-focus RGB image. The tool generates a depth map from an input image of your choice. You can click on the image to create a point of focus. \n center\n True\n 40\n 40\n \n \n False\n True\n 3\n \n \n \n \n False\n True\n 1\n \n \n \n \n \n \n False\n Help\n 320\n dialog\n main-window\n \n \n \n \n \n False\n vertical\n 2\n \n \n False\n center\n \n \n \n \n \n \n \n \n False\n False\n 1\n \n \n \n \n True\n False\n 5\n 5\n vertical\n \n \n True\n False\n <big> Help </big>\n True\n \n \n False\n True\n 6\n 0\n \n \n \n \n True\n False\n start\n <b>Usage:</b>\n True\n 0.029999999329447746\n \n \n False\n True\n 1\n \n \n \n \n True\n False\n 1. Click <b>Next</b> to begin.\n\n2. Browse to the required image or Drag and Drop the image file.\n\n3. Click <b>Delete</b> if you wish to change the image selected. Otherwise, click <b>Continue</b>.\n\n4. If you wish to save the depth map, click <b>Save As</b>.\n\n5. Click <b>Next</b> to continue.\n\n6. Click on the region in the image which you want to focus.\n\n7. Save the defocused image using <b>Save As</b> button.\n\n8. Choose the desired location to save and hit <b>Save</b>.\n True\n True\n 35\n 0\n \n \n False\n True\n 9\n 2\n \n \n \n \n True\n True\n 0\n \n \n \n \n \n\n" }, { "path": "ui.glade", "content": "\n\n\n \n \n False\n \n \n \n \n \n \n True\n False\n vertical\n 9\n top\n \n \n True\n False\n 11\n True\n \n \n True\n False\n vertical\n 10\n \n \n True\n False\n Original Imge\n \n \n False\n False\n 0\n \n \n \n \n True\n False\n 10\n True\n bottom\n \n \n True\n False\n Input Image:\n \n \n False\n True\n 0\n \n \n \n \n True\n False\n \n \n \n \n False\n True\n 6\n 1\n \n \n \n \n False\n True\n 1\n \n \n \n \n False\n True\n 0\n \n \n \n \n True\n False\n vertical\n 10\n \n \n True\n False\n Processed Image\n \n \n False\n True\n 0\n \n \n \n \n True\n False\n 20\n \n \n Save\n True\n True\n True\n \n \n True\n False\n 10\n 0\n \n \n \n \n True\n False\n 0\n 0\n \n Depth Map\n Defocussed Image\n \n \n \n \n True\n False\n 6\n 1\n \n \n \n \n False\n True\n 1\n \n \n \n \n False\n True\n 1\n \n \n \n \n False\n True\n 6\n 0\n \n \n \n \n True\n False\n immediate\n 6\n \n \n True\n False\n \n \n \n True\n False\n gtk-missing-image\n \n \n \n \n True\n True\n 5\n 0\n \n \n \n \n True\n False\n vertical\n \n \n False\n True\n 1\n \n \n \n \n True\n False\n gtk-missing-image\n \n \n True\n True\n 5\n 2\n \n \n \n \n False\n True\n 1\n \n \n \n \n True\n False\n True\n bottom\n \n \n True\n False\n Done!\n \n \n False\n True\n 0\n \n \n \n \n True\n False\n \n \n False\n False\n 1\n \n \n \n \n \n \n \n False\n False\n 5\n 2\n \n \n \n \n \n\n" } ]