[ { "path": "README.md", "content": "## Analysis of Hand Segmentation in the Wild\n\n### Abstract\nA large number of works in egocentric vision have concentrated on action and object recognition. Detection and segmentation of hands in first-person videos, however, has less been explored. For many applications in this domain, it is necessary to accurately segment not only hands of the camera wearer but also the hands of others with whom he is interacting. Here, we take an in-depth look at the hand segmentation problem. In the quest for robust hand segmentation methods, we evaluated the performance of the state of the art semantic segmentation methods, off the shelf and fine-tuned, on existing datasets. We fine-tune RefineNet, a leading semantic segmentation method, for hand segmentation and find that it does much better than the best contenders. Existing hand segmentation datasets are collected in the laboratory settings. To overcome this limitation, we contribute by collecting two new datasets: a) EgoYouTubeHands including egocentric videos containing hands in the wild, and b) HandOverFace to analyze the performance of our models in presence of similar appearance occlusions.\nWe further explore whether conditional random fields can help refine generated hand segmentations. To demonstrate the benefit of accurate hand maps, we train a CNN for hand-based activity recognition and achieve higher accuracy when a CNN was trained using hand maps produced by the fine-tuned RefineNet. Finally, we annotate a subset of the EgoHands dataset for fine-grained action recognition and show that an accuracy of 58.6% can be achieved by just looking at a single hand pose which is much better than the chance level (12.5%).\n\n* [[Paper]](http://openaccess.thecvf.com/content_cvpr_2018/papers/Urooj_Analysis_of_Hand_CVPR_2018_paper.pdf)\n* [[Project]](https://aurooj.github.io/Hand-Segmentation-in-the-Wild/)\n\n### Code\nWe have uploaded the additional files needed to train, test and evaluate our models' performance. Code for multiscale evaluation is also provided. See the folder ```refinenet_files```.\n\nTo test the models:\n* you will need to download the refinenet code from their [github repository](https://github.com/guosheng/refinenet). \n* Copy the files provided in ```refinenet_files``` folder to ```refinenet/main``` folder. \n* Place the refinenet-based hand segmentation model (see Models section) in ```refinenet/model_trained``` folder.\n* For instance, to test the model trained on EgoHands dataset, copy the ```refinenet_res101_egohands.mat``` file in ```refinenet/model_trained``` folder. Set the path to test images folder in ```demo_refinenet_test_example_egohands.m``` and run the script.\n* The demo code is the same from the original refinenet demo files except minor changes. \n\n\n### Models\nYou can download our refinenet-based hand segmentation models using the links given below:\n\n* [refinenet_res101_egohands.mat](https://drive.google.com/file/d/1u7yGIafopsn_w-RHGt1wzO-8XgmL-1zu/view?usp=sharing)\n* [refinenet_res101_eyth.mat](https://drive.google.com/file/d/12HRYXdHWOGkl71QqUdlijCq2w2ARa6-M/view?usp=sharing)\n* [refinenet_res101_gtea.mat](https://drive.google.com/file/d/1yCnpTpBuBF8wYoM4_E1o8dAWjFS0BkxM/view?usp=sharing)\n* [refinenet_res101_hof.mat](https://drive.google.com/file/d/1AOY8EQ9LRNYFusgFxHEhE_fAifNdsayh/view?usp=sharing)\n\n### Datasets\nWe used 4 hand segmentation datasets in our work, two of them(EgoYouTubeHands and HandOverFace datasets) are collected as part of our contribution:\n* [EgoHands dataset](http://vision.soic.indiana.edu/projects/egohands/)\n* EgoYouTubeHands(EYTH) dataset [[download]](https://drive.google.com/file/d/1EwjJx-V-Gq7NZtfiT6LZPLGXD2HN--qT/view?usp=sharing)\n* [GTEA dataset](http://www.cbi.gatech.edu/fpv/)\n* HandOverFace(HOF) dataset [[download]](https://drive.google.com/open?id=1hHUvINGICvOGcaDgA5zMbzAIUv7ewDd3)\n\n#### Warning!\nThanks to [Rafael Redondo Tejedor](https://github.com/valillon) who pointed out some minor mistakes in the dataset:\n* For HandOverFace dataset, 216.jpg and 221.jpg images are actually GIFs in the original size folder.\n* There were minor annotations errors for the following images in xml files: 10.jpg and 225.jpg which were pointed out and corrected by Rafael Redondo Tejedor.\n* Current link to the dataset has updated xml files for the above mentioned annotation errors.\n\n#### NEW!\n02/23/2021! Hands masks for GTEA (cropped till wrist) has been uploaded under [![data](https://github.com/aurooj/Hand-Segmentation-in-the-Wild/tree/master/data) directory. \n\nLinks to the videos used for EYTH dataset are given below. Each video is 3-6 minutes long. We cleaned the dataset before annotation and discarded unnecessary frames (e.g., frames containing text or if hands were out of view for a long time, etc). \n\n[![vid4](http://img.youtube.com/vi/dYZm7jB9YA4/0.jpg)](https://www.youtube.com/watch?v=dYZm7jB9YA4&feature=youtu.be&hd=1 \"vid4\")\n[![vid6](http://img.youtube.com/vi/5RTJ4dymKfo/0.jpg)](https://www.youtube.com/watch?v=5RTJ4dymKfo&feature=youtu.be&hd=1 \"vid6\")\n[![vid9](http://img.youtube.com/vi/vG9vfjdcmRw/0.jpg)](https://www.youtube.com/watch?v=vG9vfjdcmRw&feature=youtu.be&hd=1 \"vid9\")\n\n\n#### NEW!\nTest set for HandOverFace dataset is uploaded [here](https://drive.google.com/file/d/1-OmtqYBVmAstCzOKz8xpatw6lk9hUm--/view?usp=sharing).\n\nExample images from EgoYouTubeHands dataset:\n![EYTH](images/eyth.jpg)\n\nExample images from HandOverFace dataset:\n![HOF](images/hof.jpg)\n\n* **EgoHands+** dataset:\nTo study fine-level action recognition, we provide additional annotations for a subset of EgoHands dataset. You can find more details [here](https://github.com/aurooj/Hand-Segmentation-in-the-Wild/blob/master/egohands%2B.md) and download the dataset from this [download](https://drive.google.com/file/d/1WwGNsOhjk3hIKEnDoCKplFvMNroMCxtZ/view?usp=sharing) link.\n\n## Results\n\n### Qualitative Results\nHand segmentation results for all datasets:\n![All datasets:](images/crfs.jpg)\n\n### CVPR Poster\n![cvpr-poster](images/cvpr2018-AUK.jpg)\n\n### License\nWe only provide the annotations for the videos in EYTH and images in HOF. The copyright to the videos belong to YouTube, and images are collected from internet. The license for our work is in the license.txt file. \n\n### Acknowledgements\nWe would like to thank undergraduate students Cristopher Matos, and Jose-Valentin Sera-Josef, and MS student Shiven Goyal for helping us in data annotations. \n\n### Citation\nIf this work and/or datasets is useful for your research, please cite our paper.\n\n\n\n### Questions?\nPlease contact 'aishaurooj@gmail.com'\n\n\n\n\n\n\n\n" }, { "path": "_config.yml", "content": "theme: jekyll-theme-hacker" }, { "path": "data/readme.txt", "content": "gtea_cropped.zip has the cropped hand masks for GTEA train and test set. (see section 3.1.3 in the paper for details)\nWe used labelme tool to draw bounding boxes around hand masks provided by GTEA authors, and then kept the mask pixels inside the bounding boxes only for our use.\n\n" }, { "path": "egohands+.md", "content": "To study fine-level action recognition, we labeled a subset of video frames from [Egohands](http://vision.soic.indiana.edu/projects/egohands/) dataset at hand-level. \nWe annotated a small subset of 8 videos (800 frames),\n2 from each coarse-level activity for outdoors (courtyard) in\nthe EgoHands dataset at hand-pose level. We labeled each\nhand pose with one of the following 16 activities: **holding,\npicking, placing, resting, moving, replacing, thinking,\npulling, pushing, stacking, adjusting, matching, pressing,\nhighfive, pointing,** and **catching**. Ambiguous hand poses\nare annotated with multiple possible labels(e.g., picking and\nplacing are sometimes difficult to be inferred at a single\nframe-level).\n\n![EgoHands+](images/egohands+.png) \n\nWe have annotated hands with action labels in two settings: \n* coarse hand maps: where we just outlined the hand boundaries\n* fine hand maps: where we take extra care to outline details about fingers as much as possible.\n\nWe used [LabelMe](http://labelme.csail.mit.edu/Release3.0/) for annotations. Annotation files are in .xml format and we provide matlab scripts(based on labelme-toolbox api) to parse these annotations and generate hand masks. However, one can easily write his/her own script using LabelMe toolbox to manipulate these annotations as per their need. \n\nEach \"hand\" object is labeled with attributes in the following format: \n\n'hand_type,actions,object,subject'\n\n* 'hand_type' is which hand it is ('left' or 'right'), respective to the person the hand belongs to\n* 'actions' is all of the actions it looks like the hand is doing based on that single frame. If it's an ambiguous action, we annotate it with multiple possible actions, separated with commas. Ex:\n * 1 action: left,picking,cards,other\n * 2 actions: left,picking,placing,cards,other\n* 'object' is one of these 4 objects:\n * cards\n * jenga_block\n * chess_piece\n * puzzle_piece\n \nIf the hand is not manipulating any objects, we simply put an underscore '_' separated by commas. Ex:\nleft,resting,_,first_person\n* 'subject' is either 'first_person' or 'other'\n\nEach \"object\" is labeled and named as one of the following:\n * cards\n * jenga_block\n * chess_piece\n * puzzle_piece\n\nIn the attributes, we labeled who is manipulating the object (either 'first_person' or 'other'). If they are both manipulating the same object, we just put both separated by a comma ('first_person,other')\n\n## Usage\n1. Install LabelMe MATLAB toolbox as instructed [here](http://labelme2.csail.mit.edu/Release3.0/browserTools/php/matlab_toolbox.php).\n2. Download [EgoHands+](https://1drv.ms/u/s!AtxSFigVVA5JhNtsRdvgmxvB2c1rPg).\n3. We have borrowed some code from EgoHands dataset's [page](http://vision.soic.indiana.edu/projects/egohands/) already uploaded here. For their complete API you can refer to the original project. \n4. Place our matlab_scripts in the labelme toolbox folder.\n5. Setup paths for the directory with EgoHands+ dataset and destination directory.\n6. Run ```load_generate_gt_imgs_hands_objects2.m``` to generate masks for hands+objects setup. This would also generate a text file with images along with their labels.\n7. Run ```load_generate_gt_imgs_objects_only.m``` to generate masks for objects only setup.\n8. Run ```load_generate_gt_imgs2.m``` to generate masks for hands only setup.\n\n" }, { "path": "license.txt", "content": "MIT License\n\nCopyright (c) 2018 UCF CRCV\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "matlab_scripts/getObjectForHand.m", "content": "function [ obj ] = getObjectForHand( hand_obj, all_obj )\r\n% getObjectForHand this function returns object which is being handled by a\r\n% specific hand.\r\n% Input: hand_obj, all_obj\r\n% Output: obj: Object being handled by hand_obj\r\nis_both_hands = 0;\r\nattributes = hand_obj.attributes;\r\n\r\nhand_type = getHandType(attributes);\r\n\r\nhand_idx = vec2ind(hand_type'); % get index for hand type from one-hot vector\r\nobj_type = getObjectType(attributes);\r\nobject_exists = ~strcmp(obj_type,'_');\r\nswitch hand_idx\r\n case 1 % other_left hand\r\n %return object being handled by this hand, if any\r\n if(object_exists)\r\n datarow = getObject(all_obj, obj_type,'other','left'); \r\n \r\n end\r\n\r\n case 2 % other_right hand\r\n %return object being handled by this hand, if any\r\n if(object_exists)\r\n datarow = getObject(all_obj, obj_type, 'other', 'right'); \r\n\r\n end\r\n \r\n case 3 % my_left hand\r\n %return object being handled by this hand, if any\r\n if(object_exists)\r\n datarow = getObject(all_obj, obj_type,'first_person','left'); \r\n end\r\n \r\n case 4 % my_right hand\r\n %return object being handled by this hand, if any\r\n if(object_exists)\r\n datarow = getObject(all_obj, obj_type,'first_person','right'); \r\n end\r\n \r\n otherwise\r\n disp('Error: Invalid hand type.');\r\nend\r\nif(object_exists)\r\n \r\n for i = 1:size(datarow,2)\r\n poly = datarow(i).polygon; \r\n poly = [poly.x poly.y];\r\n obj{i} = reshapeAreaCoords(poly);\r\n end\r\nelse\r\n obj = cell(0)\r\nend\r\n\r\nend\r\n\r\nfunction [type] = getHandType(hand_attr) \r\n % this function returns hand type based on hand attributes in one hot\r\n % encoding format. output will be of format [is_other_left is_other_right is_my_left is_my_right]\r\n % Sample: [0 0 0 1] means hand type is first person's right hand.\r\n is_other_left = ~isempty(regexp(hand_attr,'left','once')) && ~isempty(regexp(hand_attr,'other','once'));\r\n is_other_right = ~isempty(regexp(hand_attr,'right','once')) && ~isempty(regexp(hand_attr,'other','once'));\r\n is_my_left = ~isempty(regexp(hand_attr,'left','once')) && ~isempty(regexp(hand_attr,'first_person','once'));\r\n is_my_right = ~isempty(regexp(hand_attr,'right','once')) && ~isempty(regexp(hand_attr,'first_person','once'));\r\n \r\n type = [is_other_left is_other_right is_my_left is_my_right];\r\nend\r\n\r\nfunction [obj_type] = getObjectType(hand_attr)\r\n% this function returns object type when input is hand attributes\r\n% object type can be 'cards', 'puzzle_piece', 'chess_piece', 'jenga_block' or '_'\r\n% where '_' means no object is being handled by this hand.\r\nattr_parts = strsplit(hand_attr,',');\r\nsize_of_attributes_parts = size(attr_parts);\r\nobj_type = attr_parts(end-1);\r\n\r\nend\r\n\r\nfunction [action] = getActionType(hand_attr)\r\n%this function returns action type when input is hand attributes\r\n% action type can be one of the 16 action classes\r\nattr_parts = strsplit(hand_attr,',');\r\naction = attr_parts(2:end-2);\r\nend\r\n\r\nfunction [out_rows] = getObject(objects, object_name,hand_type,hand_side)\r\n%works for only string values \r\n out_rows = struct();\r\n %fieldvalue\r\n for i=1:length(objects)\r\n %if object is same as object type and hand is also same as hand\r\n %type\r\n if(strcmp(objects(i).name,char(object_name)) && ~isempty(regexp(objects(i).attributes,hand_type,'once')) && ~isempty(regexp(objects(i).attributes,hand_side,'once')))\r\n\r\n out_rows = struct(objects(i));\r\n %% to do: check if the object is for the correct hand(may be check some common coordinates )\r\n end\r\n end\r\n \r\nend\r\n\r\n\r\n" }, { "path": "matlab_scripts/load_generate_gt_imgs2.m", "content": "%%%%%%%%%%this script produces masked images for hands only when hands\r\n%%%%%%%%%%are annotated with details.%%%%%%%%%%%%%%%%%%%%\r\n%%%%This script also visualizes the output\r\n%%%%%%%% images written to the output directory. \r\n%%%%%%%% Comment out all imshow() statements if you want to speedup the\r\n%%%%%%%% execution of this script.%%%%%%%%%%%%%%%%%%%%\r\n\r\nclear all; close all;\r\n\r\nfolders = {'cards_courtyard_h_s','cards_courtyard_s_h','chess_courtyard_h_s'...\r\n ,'chess_courtyard_s_h','jenga_courtyard_b_h','jenga_courtyard_h_b',...\r\n 'puzzle_courtyard_b_s','puzzle_courtyard_s_b'};\r\n\r\nDIR = 'path/to/output/directory';\r\nSRC_DIR = 'path/to/folder/with/annotations/';\r\n\r\n\r\nfileID = fopen('annotations.txt','w');%open a file for writing annotations to it.\r\n\r\nfor id = 1:length(folders)\r\nHOMEANNOTATIONS = fullfile(SRC_DIR,folders(id));\r\n\r\nD = LMdatabase(HOMEANNOTATIONS{1});\r\n\r\nfilters = strsplit(char(folders(id)),'_');%split folder name into components like cards,courtyard, s,h\r\nactivity = upper(char(filters(1))); loc = upper(char(filters(2))); actor1 = upper(char(filters(3))); actor2 = upper(char(filters(4)));\r\n% vid = getMetaBy('Location',loc,'Activity',activity,'Viewer',actor1,'Partner',actor2);\r\n% frames = vid.labelled_frames;\r\n\r\nif ~exist(char(fullfile(DIR,folders(id))), 'dir')\r\n mkdir(char(fullfile(DIR,folders(id))));\r\nend\r\n\r\nfor idx = 1:length(D)\r\n try\r\n objects = D(idx).annotation.object;\r\n img_name = D(idx).annotation.filename;\r\n img = imread(fullfile(SRC_DIR,char(folders(id)),'\\Images\\users\\shivengoyal',char(folders(id)),D(idx).annotation.filename));\r\n [row,col,ch]=size(img);\r\n% both = both + isBothHands(objects);\r\n for obj_id = 1:length(objects)\r\n if(~isempty(regexp(objects(obj_id).name,'hand','once')))\r\n attributes = objects(obj_id).attributes;\r\n if(~isempty(regexp(attributes,'left','once')) && ~isempty(regexp(attributes,'other','once')))\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n ur_left_img = bsxfun(@times, img, cast(BW, 'like', img));\r\n %ur_left_img = getSegImageForEachHand(vid,idx,img,'your_left',[]);\r\n imshow(ur_left_img);\r\n\r\n if(~isempty(ur_left_img))\r\n fn = strcat(img_name(1:end-4),'_ul.jpg');\r\n imwrite(ur_left_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n elseif(~isempty(regexp(attributes,'right','once')) && ~isempty(regexp(attributes,'other','once')))\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n ur_right_img = bsxfun(@times, img, cast(BW, 'like', img));\r\n %ur_right_img = getSegImageForEachHand(vid,idx,img,'your_right',[]);\r\n imshow(ur_right_img);\r\n \r\n if(~isempty(ur_right_img))\r\n fn = strcat(img_name(1:end-4),'_ur.jpg');\r\n imwrite(ur_right_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n elseif(~isempty(regexp(attributes,'left','once')) && ~isempty(regexp(attributes,'first_person','once')))\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n my_left_img = bsxfun(@times, img, cast(BW, 'like', img));\r\n %my_left_img = getSegImageForEachHand(vid,idx,img,'my_left',[]);\r\n imshow(my_left_img);\r\n \r\n if(~isempty(my_left_img))\r\n fn = strcat(img_name(1:end-4),'_ml.jpg');\r\n imwrite(my_left_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n elseif(~isempty(regexp(attributes,'right','once')) && ~isempty(regexp(attributes,'first_person','once')))\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n my_right_img = bsxfun(@times, img, cast(BW, 'like', img));\r\n %my_right_img = getSegImageForEachHand(vid,idx,img,'my_right',[]);\r\n imshow(my_right_img);\r\n \r\n if(~isempty(my_right_img))\r\n fn = strcat(img_name(1:end-4),'_mr.jpg');\r\n imwrite(my_right_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n \r\n \r\n attributes = strsplit(char(attributes),',');\r\n for i = 1:length(attributes)\r\n n = char(attributes(i));\r\n switch n\r\n case 'holding'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 0'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'picking'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 1'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'placing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 2'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'resting'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 3'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'moving'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 4'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'replacing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 5'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'thinking'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 6'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pulling'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 7'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pushing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 8'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'stacking'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 9'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'adjusting'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 10'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'matching'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 11'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pressing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 12'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'highfive'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 13'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pointing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 14'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'catching'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 15'));\r\n fprintf(fileID,'%s\\n',str);\r\n otherwise\r\n disp(n)\r\n end\r\n \r\n end\r\n\r\n end\r\n end\r\n catch ME\r\n msg = ME.message\r\n continue;\r\n end\r\n \r\n \r\nend\r\n\r\nend\r\nfclose(fileID);" }, { "path": "matlab_scripts/load_generate_gt_imgs_hands_objects2.m", "content": "%%%%%%%%%%this script produces masked images for hand_objects when hands\r\n%%%%%%%%%%are annotated detailed. This script also visualizes the output\r\n%%%%%%%%%%%%%%%%%%%% images written to the output directory. \r\n%%%%%%%% Comment out all imshow() statements if you want to speedup the\r\n%%%%%%%% execution of this script.\r\n\r\nclear all; close all;\r\n\r\nfolders = {'cards_courtyard_h_s','cards_courtyard_s_h','chess_courtyard_h_s',...\r\n 'chess_courtyard_s_h','jenga_courtyard_b_h','jenga_courtyard_h_b',...\r\n 'puzzle_courtyard_b_s','puzzle_courtyard_s_b'};\r\n\r\n%DIR to save your resulting images\r\nDIR = 'path\\to\\save\\output\\images\\';\r\nSRC_DIR = 'path\\to\\egohands\\fine\\annotations\\folders\\';\r\n\r\nfileID = fopen('inputfile_fineannotations.txt','w');%open a file for writing image paths along with annotations to it.\r\n\r\n% iterating over each folder\r\nfor id = 1:length(folders)\r\n \r\nHOMEANNOTATIONS = fullfile(SRC_DIR,folders(id));\r\n\r\nD = LMdatabase(HOMEANNOTATIONS{1});%read annotations for each video using Labelme library function\r\n\r\nfilters = strsplit(char(folders(id)),'_');%split folder name into components like cards,courtyard, s,h\r\nactivity = upper(char(filters(1))); loc = upper(char(filters(2))); actor1 = upper(char(filters(3))); actor2 = upper(char(filters(4)));\r\n\r\n% if DIR for output images doesn't exist already, then create it.\r\nif ~exist(char(fullfile(DIR,folders(id))), 'dir')\r\n mkdir(char(fullfile(DIR,folders(id))));\r\nend\r\n\r\n%for each image\r\nfor idx = 1:length(D)\r\n \r\n %get all annotated objects in that image\r\n objects = D(idx).annotation.object; \r\n img_name = D(idx).annotation.filename;\r\n img = imread(fullfile(SRC_DIR,char(folders(id)),'\\Images\\users\\shivengoyal',char(folders(id)),D(idx).annotation.filename));\r\n [row,col,ch]=size(img);\r\n \r\n for obj_id = 1:length(objects)\r\n \r\n try\r\n %if the object is a hand\r\n if(~isempty(regexp(objects(obj_id).name,'hand','once')))\r\n \r\n %get the object in that hand\r\n obj = getObjectForHand(objects(obj_id),objects);\r\n \r\n %read the attributes for that hand object\r\n attributes = objects(obj_id).attributes;\r\n %if the hand is left hand of third person\r\n if(~isempty(regexp(attributes,'left','once')) && ~isempty(regexp(attributes,'other','once')))\r\n if(~isempty(obj)) %if hand is manipulating some object\r\n for i = 1:length(obj)\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n BW = BW+poly2mask(double(obj{i}(1:2:end)),double(obj{i}(2:2:end)),row,col);\r\n ur_left_img = bsxfun(@times, img, cast(BW, 'like', img));\r\n imshow(BW)\r\n imshow(ur_left_img) \r\n \r\n if(~isempty(ur_left_img))\r\n %write image in destination folder\r\n fn = strcat(img_name(1:end-4),'_',num2str(i),'_ul.jpg');%filename for image\r\n imwrite(ur_left_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n else %if hand is not manipulating any object\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n ur_left_img = bsxfun(@times, img, cast(BW, 'like', img)); \r\n imshow(ur_left_img);\r\n\r\n if(~isempty(ur_left_img))\r\n %write image in destination folder\r\n fn = strcat(img_name(1:end-4),'_ul.jpg');%filename for image\r\n imwrite(ur_left_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n %if the hand is right hand of third person\r\n elseif(~isempty(regexp(attributes,'right','once')) && ~isempty(regexp(attributes,'other','once')))\r\n if(~isempty(obj))%if hand is manipulating some object\r\n for i = 1:length(obj)\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n BW = BW+poly2mask(double(obj{i}(1:2:end)),double(obj{i}(2:2:end)),row,col);\r\n ur_right_img = bsxfun(@times, img, cast(BW, 'like', img)); \r\n\r\n imshow(ur_right_img);\r\n\r\n if(~isempty(ur_right_img))\r\n fn = strcat(img_name(1:end-4),'_',num2str(i),'_ur.jpg');\r\n imwrite(ur_right_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n else %if hand does not have any object in it\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n ur_right_img = bsxfun(@times, img, cast(BW, 'like', img)); \r\n imshow(ur_right_img);\r\n\r\n if(~isempty(ur_right_img))\r\n fn = strcat(img_name(1:end-4),'_ur.jpg');\r\n imwrite(ur_right_img,char(fullfile(DIR,folders(id),fn)));\r\n end \r\n end\r\n %if the hand is left hand of first person\r\n elseif(~isempty(regexp(attributes,'left','once')) && ~isempty(regexp(attributes,'first_person','once')))\r\n if(~isempty(obj))%if hand is manipulating some object\r\n for i = 1:length(obj) \r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n BW = BW+poly2mask(double(obj{i}(1:2:end)),double(obj{i}(2:2:end)),row,col);\r\n my_left_img = bsxfun(@times, img, cast(BW, 'like', img));\r\n \r\n imshow(my_left_img);\r\n\r\n if(~isempty(my_left_img))\r\n fn = strcat(img_name(1:end-4),'_',num2str(i),'_ml.jpg');\r\n imwrite(my_left_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n else %if hand is not manipulating any object\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n my_left_img = bsxfun(@times, img, cast(BW, 'like', img)); \r\n imshow(my_left_img);\r\n\r\n if(~isempty(my_left_img))\r\n fn = strcat(img_name(1:end-4),'_ml.jpg');\r\n imwrite(my_left_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n %if the hand is \"right\" hand of first person\r\n elseif(~isempty(regexp(attributes,'right','once')) && ~isempty(regexp(attributes,'first_person','once')))\r\n if(~isempty(obj))%if hand is manipulating some object\r\n for i = 1:length(obj)\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n BW = BW+poly2mask(double(obj{i}(1:2:end)),double(obj{i}(2:2:end)),row,col);\r\n my_right_img = bsxfun(@times, img, cast(BW, 'like', img)); \r\n% \r\n imshow(my_right_img);\r\n\r\n if(~isempty(my_right_img))\r\n fn = strcat(img_name(1:end-4),'_',num2str(i),'_mr.jpg');\r\n imwrite(my_right_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n else %if hand is not manipulating any object\r\n poly = objects(obj_id).polygon;\r\n BW = poly2mask(double(poly.x),double(poly.y), row, col);\r\n my_right_img = bsxfun(@times, img, cast(BW, 'like', img)); \r\n imshow(my_right_img);\r\n\r\n if(~isempty(my_right_img))\r\n fn = strcat(img_name(1:end-4),'_mr.jpg');\r\n imwrite(my_right_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n end\r\n \r\n %write the path/to/img with its label in txt file which will be used\r\n %as an input to the caffe network for training and testing\r\n attributes = strsplit(char(attributes),',');\r\n \r\n % assign a label for corresponding action e.g., if\r\n % attributes of hand has action 'holding', then label is 0.\r\n for i = 1:length(attributes)\r\n n = char(attributes(i));\r\n switch n\r\n case 'holding'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 0'));\r\n fprintf(fileID,'%s\\n',str);%write to file\r\n case 'picking'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 1'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'placing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 2'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'resting'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 3'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'moving'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 4'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'replacing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 5'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'thinking'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 6'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pulling'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 7'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pushing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 8'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'stacking'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 9'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'adjusting'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 10'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'matching'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 11'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pressing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 12'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'highfive'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 13'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pointing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 14'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'catching'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 15'));\r\n fprintf(fileID,'%s\\n',str); \r\n otherwise\r\n disp(n)%displays non-actions attributes, useful for debugging\r\n end\r\n \r\n end\r\n\r\n end\r\n catch ME\r\n msg = ME.message\r\n ME.stack.name \r\n ME.stack.line\r\n continue;\r\n end\r\n end\r\n \r\n \r\n \r\n \r\nend\r\n\r\nend\r\nfclose(fileID);" }, { "path": "matlab_scripts/load_generate_gt_imgs_objects_only.m", "content": "%%%%%%%%%%this script produces masked images for objects bring handled by\r\n%%%%%%%%%%hands.This script also visualizes the output\r\n%%%%%%%%%%%%%%%%%%%% images written to the output directory. \r\n%%%%%%%% Comment out all imshow() statements if you want to speedup the\r\n%%%%%%%% execution of this script.%%%%%%%%%%%%%%%%%%%%\r\n\r\nclear all; close all;\r\n\r\nfolders = {'cards_courtyard_h_s','cards_courtyard_s_h','chess_courtyard_h_s'...\r\n ,'chess_courtyard_s_h','jenga_courtyard_b_h','jenga_courtyard_h_b',...\r\n 'puzzle_courtyard_b_s','puzzle_courtyard_s_b'};\r\n\r\n%DIR to save your resulting images\r\nDIR = 'path\\to\\save\\output\\images\\';\r\nSRC_DIR = 'path\\to\\egohands\\fine\\annotations\\folders\\';\r\n\r\nfileID = fopen('inputfile_fineannot.txt','w');%open a file for writing annotations to it.\r\n\r\n% iterating over each folder\r\nfor id = 1:length(folders)\r\n \r\nHOMEANNOTATIONS = fullfile(SRC_DIR,folders(id));\r\n\r\nD = LMdatabase(HOMEANNOTATIONS{1});%read annotations for each video using Labelme library function\r\n\r\nfilters = strsplit(char(folders(id)),'_');%split folder name into components like cards,courtyard, s,h\r\nactivity = upper(char(filters(1))); loc = upper(char(filters(2))); actor1 = upper(char(filters(3))); actor2 = upper(char(filters(4)));\r\n\r\n%get labelled frames from original dataset using these filters\r\nvid = getMetaBy('Location',loc,'Activity',activity,'Viewer',actor1,'Partner',actor2);\r\nframes = vid.labelled_frames;\r\n\r\n% if DIR for output images doesn't exist already, then create it.\r\nif ~exist(char(fullfile(DIR,folders(id))), 'dir')\r\n mkdir(char(fullfile(DIR,folders(id))));\r\nend\r\n\r\n%for each image\r\nfor idx = 1:length(D)\r\n \r\n %get all annotated objects in that image\r\n objects = D(idx).annotation.object; \r\n img_name = D(idx).annotation.filename;\r\n img = imread(fullfile(SRC_DIR,char(folders(id)),'\\Images\\users\\shivengoyal',char(folders(id)),D(idx).annotation.filename));\r\n [row,col,ch]=size(img);\r\n \r\n for obj_id = 1:length(objects)\r\n \r\n try\r\n %if the object is a hand\r\n if(~isempty(regexp(objects(obj_id).name,'hand','once')))\r\n \r\n %get the object in that hand\r\n obj = getObjectForHand(objects(obj_id),objects);\r\n \r\n %read the attributes for that hand object\r\n attributes = objects(obj_id).attributes;\r\n %if the hand is left hand of third person\r\n if(~isempty(regexp(attributes,'left','once')) && ~isempty(regexp(attributes,'other','once')))\r\n if(~isempty(obj)) %if hand is manipulating some object\r\n for i = 1:length(obj)\r\n BW = poly2mask(double(obj{i}(1:2:end)),double(obj{i}(2:2:end)),row,col);\r\n ur_left_img = bsxfun(@times, img, cast(BW, 'like', img));\r\n imshow(BW)\r\n imshow(ur_left_img) \r\n \r\n if(~isempty(ur_left_img))\r\n %write image in destination folder\r\n fn = strcat(img_name(1:end-4),'_',num2str(i),'_ul.jpg');%filename for image\r\n imwrite(ur_left_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n end\r\n %if the hand is right hand of third person\r\n elseif(~isempty(regexp(attributes,'right','once')) && ~isempty(regexp(attributes,'other','once')))\r\n if(~isempty(obj))%if hand is manipulating some object\r\n for i = 1:length(obj)\r\n BW = poly2mask(double(obj{i}(1:2:end)),double(obj{i}(2:2:end)),row,col);\r\n ur_right_img = bsxfun(@times, img, cast(BW, 'like', img)); \r\n \r\n imshow(ur_right_img);\r\n\r\n if(~isempty(ur_right_img))\r\n fn = strcat(img_name(1:end-4),'_',num2str(i),'_ur.jpg');\r\n imwrite(ur_right_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n end\r\n %if the hand is left hand of first person\r\n elseif(~isempty(regexp(attributes,'left','once')) && ~isempty(regexp(attributes,'first_person','once')))\r\n if(~isempty(obj))%if hand is manipulating some object\r\n for i = 1:length(obj) \r\n BW = poly2mask(double(obj{i}(1:2:end)),double(obj{i}(2:2:end)),row,col);\r\n my_left_img = bsxfun(@times, img, cast(BW, 'like', img));\r\n \r\n imshow(my_left_img);\r\n\r\n if(~isempty(my_left_img))\r\n fn = strcat(img_name(1:end-4),'_',num2str(i),'_ml.jpg');\r\n imwrite(my_left_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n end\r\n %if the hand is right hand of first person\r\n elseif(~isempty(regexp(attributes,'right','once')) && ~isempty(regexp(attributes,'first_person','once')))\r\n if(~isempty(obj))%if hand is manipulating some object\r\n for i = 1:length(obj)\r\n BW = poly2mask(double(obj{i}(1:2:end)),double(obj{i}(2:2:end)),row,col);\r\n my_right_img = bsxfun(@times, img, cast(BW, 'like', img)); \r\n \r\n imshow(my_right_img);\r\n\r\n if(~isempty(my_right_img))\r\n fn = strcat(img_name(1:end-4),'_',num2str(i),'_mr.jpg');\r\n imwrite(my_right_img,char(fullfile(DIR,folders(id),fn)));\r\n end\r\n end\r\n end\r\n end\r\n \r\n %write the path/to/img with its label in txt file which will be used\r\n %as an input to the caffe network for training and testing\r\n attributes = strsplit(char(attributes),',');\r\n \r\n % assign a label for corresponding action like if\r\n % attributes of hand has action 'holding', then label is 0.\r\n for i = 1:length(attributes)\r\n n = char(attributes(i));\r\n if(~isempty(obj))\r\n switch n\r\n case 'holding'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 0'));\r\n fprintf(fileID,'%s\\n',str);%write to file\r\n case 'picking'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 1'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'placing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 2'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'resting'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 3'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'moving'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 4'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'replacing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 5'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'thinking'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 6'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pulling'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 7'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pushing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 8'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'stacking'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 9'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'adjusting'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 10'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'matching'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 11'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pressing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 12'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'highfive'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 13'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'pointing'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 14'));\r\n fprintf(fileID,'%s\\n',str);\r\n case 'catching'\r\n str = char(strcat(fullfile(DIR,folders(id),fn),' 15'));\r\n fprintf(fileID,'%s\\n',str);\r\n otherwise\r\n disp(n)%displays non-actions attributes, useful for debugging\r\n end\r\n end \r\n end\r\n\r\n end\r\n catch ME\r\n msg = ME.message\r\n ME.stack.name \r\n ME.stack.line\r\n continue;\r\n end\r\n end \r\n \r\n \r\n \r\nend\r\n\r\nend\r\nfclose(fileID);" }, { "path": "matlab_scripts/reshapeAreaCoords.m", "content": "function shape2 = reshapeAreaCoords(shape)\n\tshape2 = zeros(1, 2*length(shape));\n\tshape2(1:2:end) = shape(:,1)';\n\tshape2(2:2:end) = shape(:,2)';\nend" }, { "path": "refinenet_files/computeAccuracyEYTH.m", "content": "close all; clear all;\n\ngt_dir = '/home/zxi/refinenet/datasets/eyth/SegmentationClass/testset';\npred_dir = '/home/zxi/refinenet/cache_data/Multiscale_evaluation_refinenet/eyth_on_eyth';\nim_dir = '/home/zxi/refinenet/datasets/eyth/JPEGImages/testset';\n\ndir1 = dir(fullfile(gt_dir));\n%remove '.' and '..' from directories\ndir1=dir1(~ismember({dir1.name},{'.','..'}));\n\nidx = 1;\nnum_of_frames = 258;\n\nIOU = zeros(1,num_of_frames,'double');\nPrecision = zeros(1,num_of_frames,'double');\nRecall = zeros(1,num_of_frames,'double');\n\nfor i = 1:length(dir1)\n vid_name=dir1(i).name; \n \n %read frames from folder\n img_files = dir(fullfile(gt_dir,vid_name));\n %remove '.' and '..' from directories\n img_files=img_files(~ismember({img_files.name},{'.','..'}));\n\n for j = 1:length(img_files)\n gt_fr_name = img_files(j).name;\n im = imread(fullfile(im_dir,vid_name,strcat(gt_fr_name(1:end-4),'.jpg')));\n gt_map = imread(fullfile(gt_dir,dir1(i).name,gt_fr_name));\n\n %now get predicted output for the ground truth\n pred_frame_path = fullfile(pred_dir,vid_name, 'predict_result_mask',strcat(gt_fr_name(1:end-4),'.png')); \n try\n pred_im = imread(pred_frame_path); \n if(size(pred_im) ~= size(gt_map))\n pred_im = imresize(pred_im,size(gt_map));\n end\n catch\n continue;\n end\n [iou,prec,recall] = getIOU(pred_im,gt_map,1);\n if(iou==0) \n num_of_frames = num_of_frames - 1;\n end\n \n IOU(idx)=iou;\n Precision(idx)=prec;\n Recall(idx)=recall; \n idx = idx+1;\n end \nend\n\n\n%fprintf(1, 'The mean Intersection_over_Union for cards videos is %d\\n The mean Intersection_over_Union for chess videos is %d\\n The mean Intersection_over_Union for jenga videos is %d\\n The mean Intersection_over_Union for puzzle videos is %d\\n '...\n% ,mean_cards_IOU,mean_chess_IOU,mean_jenga_IOU,mean_puzzle_IOU);\n\nmean_IOU = sum(IOU(:))/num_of_frames\nmean_prec = sum(Precision(:))/num_of_frames\nmean_recall = sum(Recall(:))/num_of_frames\nfprintf(1, 'The mean Intersection_over_Union for all videos is %d\\n',mean_IOU);\n\n\n" }, { "path": "refinenet_files/computeAccuracyEgoHands.m", "content": "close all; clear all;\n\ngt_dir = '/path/to/refinenet/datasets/EgoHands/SegmentationClass/test';\npred_dir = '/path/to/refinenet/cache_data/EgoHands/';\nim_dir = '/path/to/refinenet/datasets/EgoHands';\n\ndir1 = dir(fullfile(gt_dir));\n%remove '.' and '..' from directories\ndir1=dir1(~ismember({dir1.name},{'.','..'}));\n\nidx = 1;\n num_of_frames = 800; \n\nIOU = zeros(1,num_of_frames,'double');\nPrecision = zeros(1,num_of_frames,'double');\nRecall = zeros(1,num_of_frames,'double');\n\nfor i = 1:length(dir1)\n vid_name=dir1(i).name \n \n %read frames from folder\n img_files = dir(fullfile(gt_dir,vid_name));\n %remove '.' and '..' from directories\n img_files=img_files(~ismember({img_files.name},{'.','..'}));\n\n for j = 1:length(img_files)\n gt_fr_name = img_files(j).name;\n \n gt_map = imread(fullfile(gt_dir,dir1(i).name,gt_fr_name));\n\n %now get predicted output for the ground truth\n pred_frame_path = fullfile(pred_dir,vid_name,strcat(gt_fr_name(1:end-4),'.png')); \n\n \n pred_im = double(imread(pred_frame_path)); \n if(size(pred_im) ~= size(gt_map))\n gt_map = imresize(gt_map,size(pred_im)); \n end\n \n if(~islogical(gt_map))\n gt_map = imbinarize(gt_map);\n end\n\n pred_im = pred_im./255;\n \n pred_im = imbinarize(pred_im);\n\n [iou,prec,recall] = getIOU(pred_im, gt_map, 1);\n\n \n IOU(idx)=iou;\n Precision(idx)=prec;\n Recall(idx)=recall; \n idx = idx+1;\n end \nend\n\n\nmean_IOU = sum(IOU(:))/num_of_frames\nmean_prec = sum(Precision(:))/num_of_frames\nmean_recall = sum(Recall(:))/num_of_frames\nfprintf(1, 'The mean Intersection_over_Union for all videos is %d\\n',mean_IOU);\n\n\n" }, { "path": "refinenet_files/computeAccuracyGTEA.m", "content": "close all; clear all;\n\ngt_dir = '/home/zxi/refinenet/datasets/gtea/SegmentationClass/testset';\n\npred_dir = '/home/zxi/refinenet/cache_data/Multiscale_evaluation_refinenet/gtea/test/predict_result_mask';\n\nim_dir = '/home/zxi/refinenet/datasets/gtea/JPEGImages/testset';\n\n\ndir1 = dir(fullfile(pred_dir));\n\n%remove '.' and '..' from directories\ndir1=dir1(~ismember({dir1.name},{'.','..'}));\n\nnum_of_frames = length(dir1);\n\nIOU = zeros(1,num_of_frames,'double');\nPrecision = zeros(1,num_of_frames,'double');\nRecall = zeros(1,num_of_frames,'double');\n\nfor j = 1:length(dir1)\n gt_fr_name = dir1(j).name;\n im = imread(fullfile(im_dir,strcat(gt_fr_name(1:end-4),'.jpg')));\n gt_map = imread(fullfile(gt_dir,strcat(gt_fr_name(1:end-4),'.jpg')));\n \n %now get predicted output for the ground truth\n pred_frame_path = fullfile(pred_dir,strcat(gt_fr_name(1:end-4),'.png'));\n \n \n pred_im = imread(pred_frame_path); \n \n %if(~(size(gt_map)~=size(pred_im)))\n gt_map = imresize(gt_map,size(pred_im));\n % end\n \n if(~islogical(gt_map))\n gt_map = imbinarize(gt_map);\n end\n [iou,prec,recall] = getIOU(pred_im,gt_map,1); \n \n \n IOU(j)=iou;\n Precision(j)=prec;\n Recall(j)=recall; \n \nend \n\nmean_IOU = sum(IOU(:))/num_of_frames\nmean_prec = sum(Precision(:))/num_of_frames\nmean_recall = sum(Recall(:))/num_of_frames\nfprintf(1, 'The mean Intersection_over_Union for all images is %d\\n',mean_IOU);\n\n\n" }, { "path": "refinenet_files/computeAccuracyHoF.m", "content": "close all; clear all;\n\ngt_dir = '/home/zxi/refinenet/datasets/ADE20K_hand_images/SegmentationClass/test';\npred_dir = '/home/zxi/refinenet/cache_data/Multiscale_evaluation_refinenet/ade_on_ade/test/predict_result_mask';\nim_dir = '/home/zxi/refinenet/datasets/ADE20K_hand_images/JPEGImages/test';\n\n\ndir1 = dir(fullfile(pred_dir));\n%remove '.' and '..' from directories\ndir1=dir1(~ismember({dir1.name},{'.','..'}));\n\nidx = 1;\n num_of_frames = length(dir1);\n\n\nIOU = zeros(1,num_of_frames,'double');\nPrecision = zeros(1,num_of_frames,'double');\nRecall = zeros(1,num_of_frames,'double');\n\nfor j = 1:length(dir1)\n gt_fr_name = dir1(j).name;\n im = imread(fullfile(im_dir,strcat(gt_fr_name(1:end-4),'.jpg')));\n gt_map = imread(fullfile(gt_dir,dir1(j).name));\n \n %now get predicted output for the ground truth\n pred_frame_path = fullfile(pred_dir,strcat(gt_fr_name(1:end-4),'.png')); \n \n pred_im = imread(pred_frame_path); \n \n if(~islogical(gt_map))\n gt_map = imbinarize(gt_map); %works in matlab 2017\n %gt_map = im2bw(gt_map,0.1); %use this line for matlab version < 2017\n end\n [iou,prec,recall,mcr] = getIOU(pred_im,gt_map,1);\n \n \n IOU(j)=iou;\n Precision(j)=prec;\n Recall(j)=recall; \n \nend \n\n\n\nmean_IOU = sum(IOU(:))/num_of_frames\nmean_prec = sum(Precision(:))/num_of_frames\nmean_recall = sum(Recall(:))/num_of_frames\nfprintf(1, 'The mean Intersection_over_Union for all videos is %d\\n',mean_IOU);\n\n\n" }, { "path": "refinenet_files/demo_refinenet_test_example_egohands.m", "content": "\n% Author: Guosheng Lin (guosheng.lin@gmail.com)\n\n% perform segmentation prediction on user provided images.\n% specify the location of your images, e.g., using the following \n% folder which contains serveral example images:\n% ds_config.img_data_dir='../datasets/custom_data';\n\n\nfunction demo_refinenet_test_example_egohands()\n\n%%% testset\n folders = {'CARDS_COURTYARD_B_T','CARDS_OFFICE_S_B','CHESS_COURTYARD_B_T','CHESS_LIVINGROOM_T_H','JENGA_LIVINGROOM_S_T','JENGA_OFFICE_H_T','PUZZLE_COURTYARD_H_T','PUZZLE_LIVINGROOM_T_B'};\n\n%%% valset\n%folders = {'JENGA_COURTYARD_T_S','CHESS_COURTYARD_H_S','PUZZLE_OFFICE_S_T','CARDS_LIVINGROOM_S_H'}\nfor i = 1:length(folders)\n rng('shuffle');\n\n addpath('./my_utils');\n dir_matConvNet='../libs/matconvnet/matlab';\n run(fullfile(dir_matConvNet, 'vl_setupnn.m'));\n\n run_config=[];\n ds_config=[];\n\n run_config.use_gpu=true;\n % run_config.use_gpu=false;\n run_config.gpu_idx=1;\n\n % result dir:\n result_name=[char(folders(i))];\n \n result_dir=fullfile('../cache_data', 'egohands', result_name);\n\n\n % the folder that contains testing images:\n ds_config.img_data_dir=strcat('/home/zxi/refinenet/datasets/EgoHands/JPEGImages/',char(folders(i))); \n\n\n % using a trained model which is trained on VOC 2012\n % run_config.trained_model_path='../model_trained/refinenet_res101_voc2012.mat';\n % ds_config.class_info=gen_class_info_voc();\n\n\n % using the object parsing model\n run_config.trained_model_path='../model_trained/refinenet_res101_egohands.mat';\n ds_config.class_info=gen_class_info_ego();\n\n\n % for voc trained model, control the size of input images\n run_config.input_img_short_edge_min=450;\n run_config.input_img_short_edge_max=600;\n\n % set the input image scales, useful for multi-scale evaluation\n % e.g. using multiple scale settings (1.0 0.8 0.6) and average the resulting score maps.\n run_config.input_img_scale=1.0;\n\n\n run_config.gen_net_opts_fn=@gen_net_opts_model_type1;\n\n\n run_config.run_evaonly=true;\n ds_config.use_custom_data=true;\n ds_config.use_dummy_gt=true;\n run_config.use_dummy_gt=ds_config.use_dummy_gt;\n\n\n ds_config.ds_name='tmp_data';\n run_config.root_cache_dir=result_dir;\n mkdir_notexist(run_config.root_cache_dir);\n\n run_config.model_name=result_name;\n\n diary_dir=run_config.root_cache_dir;\n mkdir_notexist(diary_dir);\n diary(fullfile(diary_dir, 'output.txt'));\n diary on\n\n\n run_dir_name=fileparts(mfilename('fullpath'));\n [~, run_dir_name]=fileparts(run_dir_name);\n run_config.run_dir_name=run_dir_name;\n run_config.run_file_name=mfilename();\n\n ds_info=gen_dataset_info(ds_config);\n my_diary_flush();\n\n train_opts=run_config.gen_net_opts_fn(run_config, ds_info.class_info);\n\n\n imdb=my_gen_imdb(train_opts, ds_info);\n\n data_norm_info=[];\n data_norm_info.image_mean=128;\n\n imdb.ref.data_norm_info=data_norm_info;\n\n if run_config.use_gpu\n gpu_num=gpuDeviceCount;\n if gpu_num>=1\n gpuDevice(run_config.gpu_idx);\n else\n error('no gpu found!');\n end\n end\n\n [net_config, net_exp_info]=prepare_running_model(train_opts);\n\n my_net_tool(train_opts, imdb, net_config, net_exp_info);\n\n\n fprintf('\\n\\n--------------------------------------------------\\n\\n');\n disp('results are saved in:');\n disp(run_config.root_cache_dir);\n\n\n my_diary_flush();\n diary off\n\n end\nend\n\n\n" }, { "path": "refinenet_files/demo_refinenet_test_example_eyth.m", "content": "\n% Author: Guosheng Lin (guosheng.lin@gmail.com)\n\n% perform segmentation prediction on user provided images.\n% specify the location of your images, e.g., using the following \n% folder which contains serveral example images:\n% ds_config.img_data_dir='../datasets/custom_data';\n\n\nfunction demo_refinenet_test_example_eyth()\n folders = {'vid4','vid6','vid9'};\n\n for i = 1:length(folders)\n rng('shuffle');\n\n addpath('./my_utils');\n dir_matConvNet='../libs/matconvnet/matlab';\n run(fullfile(dir_matConvNet, 'vl_setupnn.m'));\n\n\n run_config=[];\n ds_config=[];\n\n run_config.use_gpu=true;\n % run_config.use_gpu=false;\n run_config.gpu_idx=1;\n\n\n\n % result dir:\n result_name=[char(folders(i))]\n \n result_dir=fullfile('../cache_data', 'eyth_on_eyth_val/val_1', result_name);\n\n\n % the folder that contains testing images:\n ds_config.img_data_dir=strcat('../datasets/eyth/JPEGImages/test/',char(folders(i)));\n \n\n % using a trained model which is trained on VOC 2012\n % run_config.trained_model_path='../model_trained/refinenet_res101_voc2012.mat';\n % ds_config.class_info=gen_class_info_voc();\n\n\n % using the object parsing model\n run_config.trained_model_path='../model_trained/refinenet_res101_eyth.mat';\n ds_config.class_info=gen_class_info_ego();\n\n\n % for voc trained model, control the size of input images\n run_config.input_img_short_edge_min=450;\n run_config.input_img_short_edge_max=600;\n\n % set the input image scales, useful for multi-scale evaluation\n % e.g. using multiple scale settings (1.0 0.8 0.6) and average the resulting score maps.\n run_config.input_img_scale=1.0;\n\n\n run_config.gen_net_opts_fn=@gen_net_opts_model_type1;\n\n\n run_config.run_evaonly=true;\n ds_config.use_custom_data=true;\n ds_config.use_dummy_gt=true;\n run_config.use_dummy_gt=ds_config.use_dummy_gt;\n\n\n ds_config.ds_name='tmp_data';\n run_config.root_cache_dir=result_dir;\n mkdir_notexist(run_config.root_cache_dir);\n\n run_config.model_name=result_name;\n\n diary_dir=run_config.root_cache_dir;\n mkdir_notexist(diary_dir);\n diary(fullfile(diary_dir, 'output.txt'));\n diary on\n\n\n run_dir_name=fileparts(mfilename('fullpath'));\n [~, run_dir_name]=fileparts(run_dir_name);\n run_config.run_dir_name=run_dir_name;\n run_config.run_file_name=mfilename();\n\n ds_info=gen_dataset_info(ds_config);\n my_diary_flush();\n\n train_opts=run_config.gen_net_opts_fn(run_config, ds_info.class_info);\n\n\n imdb=my_gen_imdb(train_opts, ds_info);\n\n data_norm_info=[];\n data_norm_info.image_mean=128;\n\n imdb.ref.data_norm_info=data_norm_info;\n\n if run_config.use_gpu\n gpu_num=gpuDeviceCount;\n if gpu_num>=1\n gpuDevice(run_config.gpu_idx);\n else\n error('no gpu found!');\n end\n end\n\n [net_config, net_exp_info]=prepare_running_model(train_opts);\n\n my_net_tool(train_opts, imdb, net_config, net_exp_info);\n\n\n fprintf('\\n\\n--------------------------------------------------\\n\\n');\n disp('results are saved in:');\n disp(run_config.root_cache_dir);\n\n\n my_diary_flush();\n diary off\n\n end\nend\n\n\n" }, { "path": "refinenet_files/demo_refinenet_test_example_gtea.m", "content": "\n% Author: Guosheng Lin (guosheng.lin@gmail.com)\n\n% perform segmentation prediction on user provided images.\n% specify the location of your images, e.g., using the following \n% folder which contains serveral example images:\n% ds_config.img_data_dir='../datasets/custom_data';\n\n\nfunction demo_refinenet_test_example_gtea()\n\nrng('shuffle');\n\naddpath('./my_utils');\ndir_matConvNet='../libs/matconvnet/matlab';\nrun(fullfile(dir_matConvNet, 'vl_setupnn.m'));\n\n\nrun_config=[];\nds_config=[];\n\n%run_config.use_gpu=true;\nrun_config.use_gpu=false;\nrun_config.gpu_idx=1;\n\n\n\n% result dir:\n% result_name=[char(folders(i))]%['result_' datestr(now, 'YYYYmmDDHHMMSS') '_evaonly_custom_data' char(folders(i))]\nresult_name='val_1.0';\nresult_dir=fullfile('../cache_data', 'gtea_on_gtea', result_name);\n\n\n% the folder that contains testing images:\n%ds_config.img_data_dir=strcat('../datasets/EgoHands/Testset/',char(folders(i)));\nds_config.img_data_dir='../datasets/gtea/JPEGImages/val';\n\n\n% using a trained model which is trained on VOC 2012\n% run_config.trained_model_path='../model_trained/refinenet_res101_voc2012.mat';\n% ds_config.class_info=gen_class_info_voc();\n\n\n% using the object parsing model\nrun_config.trained_model_path='../model_trained/refinenet_res101_gtea.mat';\nds_config.class_info=gen_class_info_ego();\n\n\n% for voc trained model, control the size of input images\nrun_config.input_img_short_edge_min=450;\nrun_config.input_img_short_edge_max=600;\n\n% set the input image scales, useful for multi-scale evaluation\n% e.g. using multiple scale settings (1.0 0.8 0.6) and average the resulting score maps.\nrun_config.input_img_scale=1.0;\n\n\nrun_config.gen_net_opts_fn=@gen_net_opts_model_type1;\n\n\nrun_config.run_evaonly=true;\nds_config.use_custom_data=true;\nds_config.use_dummy_gt=true;\nrun_config.use_dummy_gt=ds_config.use_dummy_gt;\n\n\nds_config.ds_name='tmp_data';\nrun_config.root_cache_dir=result_dir;\nmkdir_notexist(run_config.root_cache_dir);\n\nrun_config.model_name=result_name;\n\ndiary_dir=run_config.root_cache_dir;\nmkdir_notexist(diary_dir);\ndiary(fullfile(diary_dir, 'output.txt'));\ndiary on\n\n\nrun_dir_name=fileparts(mfilename('fullpath'));\n[~, run_dir_name]=fileparts(run_dir_name);\nrun_config.run_dir_name=run_dir_name;\nrun_config.run_file_name=mfilename();\n\nds_info=gen_dataset_info(ds_config);\nmy_diary_flush();\n\ntrain_opts=run_config.gen_net_opts_fn(run_config, ds_info.class_info);\n\n\nimdb=my_gen_imdb(train_opts, ds_info);\n\ndata_norm_info=[];\ndata_norm_info.image_mean=128;\n\nimdb.ref.data_norm_info=data_norm_info;\n\nif run_config.use_gpu\n\tgpu_num=gpuDeviceCount;\n\tif gpu_num>=1\n\t\tgpuDevice(run_config.gpu_idx);\n else\n error('no gpu found!');\n\tend\nend\n\n[net_config, net_exp_info]=prepare_running_model(train_opts);\n\nmy_net_tool(train_opts, imdb, net_config, net_exp_info);\n\n\nfprintf('\\n\\n--------------------------------------------------\\n\\n');\ndisp('results are saved in:');\ndisp(run_config.root_cache_dir);\n\n\nmy_diary_flush();\ndiary off\n\n\nend\n\n\n" }, { "path": "refinenet_files/demo_refinenet_test_example_hof.m", "content": "\n% Author: Guosheng Lin (guosheng.lin@gmail.com)\n\n% perform segmentation prediction on user provided images in HandOnFace dataset\n% specify the location of your images, e.g., using the following \n% folder which contains serveral example images:\n% ds_config.img_data_dir='../datasets/custom_data';\n\n\nfunction demo_refinenet_test_example_hof()\n\nrng('shuffle');\n\naddpath('./my_utils');\ndir_matConvNet='../libs/matconvnet/matlab';\nrun(fullfile(dir_matConvNet, 'vl_setupnn.m'));\n\n\nrun_config=[];\nds_config=[];\n\nrun_config.use_gpu=true;\n% run_config.use_gpu=false;\nrun_config.gpu_idx=1;\n\n\n\n% result dir:\nresult_name='test_1';\n\nresult_dir=fullfile('../cache_data', 'hof', result_name);\n\n\n% the folder that contains testing images:\nds_config.img_data_dir=strcat('../datasets/hof/test');\n%ds_config.img_data_dir='../datasets/EgoHands_Temporarily_Removed_Folders/TestImages';\n\n\n% using a trained model which is trained on VOC 2012\n% run_config.trained_model_path='../model_trained/refinenet_res101_voc2012.mat';\n% ds_config.class_info=gen_class_info_voc();\n\n\n% using the object parsing model\nrun_config.trained_model_path='../model_trained/refinenet_res101_hof.mat';\nds_config.class_info=gen_class_info_ego();\n\n% for voc trained model, control the size of input images\nrun_config.input_img_short_edge_min=450;\nrun_config.input_img_short_edge_max=600;\n\n% set the input image scales, useful for multi-scale evaluation\n% e.g. using multiple scale settings (1.0 0.8 0.6) and average the resulting score maps.\nrun_config.input_img_scale=1.0;\n\nrun_config.gen_net_opts_fn=@gen_net_opts_model_type1;\n\n\nrun_config.run_evaonly=true;\nds_config.use_custom_data=true;\nds_config.use_dummy_gt=true;\nrun_config.use_dummy_gt=ds_config.use_dummy_gt;\n\n\nds_config.ds_name='tmp_data';\nrun_config.root_cache_dir=result_dir;\nmkdir_notexist(run_config.root_cache_dir);\n\nrun_config.model_name=result_name;\n\ndiary_dir=run_config.root_cache_dir;\nmkdir_notexist(diary_dir);\ndiary(fullfile(diary_dir, 'output.txt'));\ndiary on\n\n\nrun_dir_name=fileparts(mfilename('fullpath'));\n[~, run_dir_name]=fileparts(run_dir_name);\nrun_config.run_dir_name=run_dir_name;\nrun_config.run_file_name=mfilename();\n\nds_info=gen_dataset_info(ds_config);\nmy_diary_flush();\n\ntrain_opts=run_config.gen_net_opts_fn(run_config, ds_info.class_info);\n\n\nimdb=my_gen_imdb(train_opts, ds_info);\n\ndata_norm_info=[];\ndata_norm_info.image_mean=128;\n\nimdb.ref.data_norm_info=data_norm_info;\n\nif run_config.use_gpu\n\tgpu_num=gpuDeviceCount;\n\tif gpu_num>=1\n\t\tgpuDevice(run_config.gpu_idx);\n else\n error('no gpu found!');\n\tend\nend\n\n[net_config, net_exp_info]=prepare_running_model(train_opts);\n\nmy_net_tool(train_opts, imdb, net_config, net_exp_info);\n\n\nfprintf('\\n\\n--------------------------------------------------\\n\\n');\ndisp('results are saved in:');\ndisp(run_config.root_cache_dir);\n\n\nmy_diary_flush();\ndiary off\n\n\nend\n\n\n" }, { "path": "refinenet_files/gen_class_info_ego.m", "content": "\nfunction class_info=gen_class_info_ego()\n\nclass_info=[];\n\nclass_info.class_names = { 'background', 'hand','void'}'; \n \n\nvoid_class_value=255;\nclass_info.class_label_values=uint8([0:1 void_class_value]);\nclass_info.background_label_value=uint8(0);\nclass_info.void_label_values=uint8(void_class_value);\n\n% addpath ../libs/VOCdevkit_2012/VOCcode\nclass_info.mask_cmap = VOClabelcolormap(256);\n\nclass_info=process_class_info(class_info);\n\nend" }, { "path": "refinenet_files/gen_class_info_gtea.m", "content": "\nfunction class_info=gen_class_info_gtea()\n\nclass_info=[];\n\nclass_info.class_names = { 'background', 'hand','void'}'; \n \n\nvoid_class_value=255;\nclass_info.class_label_values=uint8([0:1 void_class_value]);\nclass_info.background_label_value=uint8(0);\nclass_info.void_label_values=uint8(void_class_value);\n\n% addpath ../libs/VOCdevkit_2012/VOCcode\nclass_info.mask_cmap = VOClabelcolormap(256);\n\nclass_info=process_class_info(class_info);\n\nend" }, { "path": "refinenet_files/getIOU.m", "content": "function [ IOU, Precision, Recall,MCR ] = getIOU( pred_map,gt_map,get_other_scores)\n%this function computes IOU when given a prediction map and respective\n%ground truth map. Both pred_map and gt_map should be 2D binary maps.\n\n% INPUT: pred_map: a binary image prediction map\n% gt_map: a binay image ground truth map\n% get_other_scores: An optional flag which by default is zero.\n% Returns Precision, Recall and MCR(miss classification rate) scores if set to 1.\n% OUTPUT: IOU: Intersection over Union score which should be a scalar\n% number between the range [0,1].\nIOU=0; Precision=0; Recall=0;MCR=0;\nif(nargin <2)\n error('input_example : pred_map and gt_map are required inputs');\nend\nif(nargin == 2 && nargin < 3)\n get_other_scores = 0;\nend\n %compute true_positives\n true_pos = (pred_map.*gt_map);\n \n num_pxls = size(pred_map,1)*size(pred_map,2);\n \n TP_count = sum(true_pos(:));\n \n %compute false_positives\n FP_count = sum(pred_map(:)) - TP_count;\n \n %compute false negatives\n FN_count = sum(gt_map(:)) - TP_count; \n \n if(TP_count~=0 || (TP_count~=0 && FN_count~=0))\n IOU = (TP_count/(TP_count+FP_count+FN_count));\n if(get_other_scores)\n Precision = (TP_count/(TP_count+FP_count));\n Recall = (TP_count/(TP_count+FN_count));\n MCR = (FP_count+FN_count)/num_pxls;\n else\n Precision = 0;\n Recall = 0;\n MCR = 0;\n end\n end\n\nend\n\n" }, { "path": "refinenet_files/multiscale_evaluation.m", "content": "%this script generates final prediction maps after averaging the scoremaps\n%across different scales. \n% This code assumes that the prediction scoremaps have already been \n% generated for the mentioned destination folder with following naming\n% convention: _ e.g. egohands_0.6, egohands_0.8 and\n% egohands_1.0 provided that the scales are same as mentioned in this file.\n% e.g. if scales = [0.8 1.0], then this script will only look for folders\n% with _0.8 and _1.0 respectively.\nclose all; clear all;\nsrc_dir = '/home/zxi/refinenet/cache_data/gtea_on_gtea/'\ndest_dir = '/home/zxi/Desktop/research/densecrf_on_all_datasets/gtea-valset/'\n\nscales = [0.6 0.8 1.0];\n\nimg_size = [216 384];% for Egohands\n%img_size = [405 720]; % for GTEA\n\n\nfolder_name = 'val';\n\nfolders = dir(fullfile(src_dir,strcat(folder_name,'_',num2str(scales(1),1))));\n\n%remove '.' and '..' from directories\nfolders=folders(~ismember({folders.name},{'.','..'}));\n\n%folders = {};\n\nfor i = 1:length(folders)\n %for each file, read scoremap from its respective folder for all scales\n % take the average scoremap, and get prediction map for that.\n %write it back to destination folder\n folder = strcat(folder_name,'_',num2str(scales(1),1));\n path_to_sc_maps = fullfile(src_dir,folder,folders(i).name,'predict_result_full');\n \n files = dir(path_to_sc_maps)\n %remove '.' and '..' from directories\n files=files(~ismember({files.name},{'.','..'}));\n\n for j = 1:length(files)\n path_to_one_map = fullfile(path_to_sc_maps,files(j).name);\n sc_map1 = load(path_to_one_map);\n sc_map1 = double(sc_map1.data_obj.score_map);\n \n score_map_size=size(sc_map1);\n score_map_size=score_map_size(1:2);\n \n if any(img_size~=score_map_size)\n sc_map1=log(sc_map1);\n sc_map1=max(sc_map1, -20);\n sc_map1=my_resize(single(sc_map1), img_size);\n sc_map1=exp(sc_map1);\n end \n \n sum_sc = double(sc_map1); %initialize with first score_map\n \n for k = 2:length(scales) \n folder = strcat(folder_name,'_',num2str(scales(k),1));\n path_to_sc_maps = fullfile(src_dir,folder,folders(i).name,'predict_result_full');\n path_to_one_map = fullfile(path_to_sc_maps,files(j).name);\n \n sc_map = load(path_to_one_map); \n sc_map = double(sc_map.data_obj.score_map);\n \n score_map_size=size(sc_map);\n score_map_size=score_map_size(1:2);\n \n if any(img_size~=score_map_size)\n sc_map=log(sc_map);\n sc_map=max(sc_map, -20);\n sc_map=my_resize(single(sc_map), img_size);\n sc_map=exp(sc_map);\n end\n \n sum_sc = sum_sc + double(sc_map);\n end\n meansc_map = sum_sc/length(scales);%average scoremap over multiple scales\n imshow(meansc_map)\n \n file_name = files(j).name;\n file_name = strcat(file_name(1:end-4),'.png');\n \n %uncomment if want to save scoremaps\n path_to_dest_scmaps = fullfile(dest_dir,'score_maps',folders(i).name);\n if(~exist(path_to_dest_scmaps,'dir'))\n mkdir(path_to_dest_scmaps);\n end\n map_to_write = gather(meansc_map);\n save(fullfile(path_to_dest_scmaps,files(j).name),'map_to_write');\n \n [~, predict_mask]=max(meansc_map,[],3);\n% imshow(predict_mask,[])\n \n class_info = gen_class_info_ego();\n predict_mask_data=class_info.class_label_values(predict_mask);\n %%unique(predict_mask_data) %uncomment for debug only\n path_to_dest = fullfile(dest_dir,folders(i).name,'predict_result_mask');\n if(~exist(path_to_dest,'dir'))\n mkdir(path_to_dest)\n end\n \n imwrite(logical(predict_mask_data),fullfile(path_to_dest,file_name));\n end \nend\n\n\n\n\n" }, { "path": "refinenet_files/my_gen_ds_info_egohands.m", "content": "\n\n\nfunction ds_info=my_gen_ds_info_egohands(ds_config)\n\n\n\nds_dir=fullfile('../datasets', 'EgoHands');\n\ntrain_idx_file=fullfile(ds_dir, 'ImageSets/Segmentation/train.txt');\nfid=fopen(train_idx_file);\ntrain_file_names=textscan(fid, '%s');\ntrain_file_names=train_file_names{1};\nfclose(fid);\n\n\nval_idx_file=fullfile(ds_dir, 'ImageSets/Segmentation/val.txt');\nfid=fopen(val_idx_file);\nval_file_names=textscan(fid, '%s');\nval_file_names=val_file_names{1};\nfclose(fid);\n\n\ntrain_num=length(train_file_names);\nimg_names=cat(1, train_file_names, val_file_names);\nimg_num=length(img_names);\n\nimg_files=cell(img_num, 1);\nmask_files=cell(img_num, 1);\n\nfor t_idx=1:img_num\n file_name=img_names{t_idx};\n img_files{t_idx}=[file_name(1:end-4) '.jpg'];\n mask_file_name = strrep(file_name,'JPEGImages','SegmentationClass');\n mask_files{t_idx}=[mask_file_name(1:end-4) '.png'];\n \nend\n\ntrain_idxes=1:train_num;\nval_idxes=train_num+1:img_num;\n\n\nds_info=[];\n\nds_info.img_names=img_names;\nds_info.img_files=img_files;\nds_info.mask_files=mask_files;\n\nds_info.train_idxes=uint32(train_idxes');\nds_info.test_idxes=uint32(val_idxes');\n\ndisp('here::')\nimg_dir=fullfile(ds_dir, 'JPEGImages')\nmask_dir=fullfile(ds_dir, 'SegmentationClass')\n\n\ndata_dirs=[];\ndata_dirs{1}=img_dir;\ndata_dirs{2}=mask_dir;\n\ndata_dir_idxes_img=zeros([img_num 1], 'uint8')+1;\ndata_dir_idxes_mask=zeros([img_num 1], 'uint8')+2;\n\nds_info.data_dir_idxes_img=data_dir_idxes_img;\nds_info.data_dir_idxes_mask=data_dir_idxes_mask;\nds_info.data_dirs=data_dirs;\nds_info.ds_dir=ds_dir;\n\nds_info.class_info=gen_class_info_ego();\n\nds_info.ds_name='EgoHands';\n\nds_info=process_ds_info_classification(ds_info, ds_config);\n\nend\n\n\n" }, { "path": "refinenet_files/my_gen_ds_info_eyth.m", "content": "\n\n\nfunction ds_info=my_gen_ds_info_eyth(ds_config)\n\n\n\nds_dir=fullfile('../datasets', 'eyth');\n\ntrain_idx_file=fullfile(ds_dir, 'ImageSets/Segmentation/train.txt');\nfid=fopen(train_idx_file);\ntrain_file_names=textscan(fid, '%s');\ntrain_file_names=train_file_names{1};\nfclose(fid);\n\n\nval_idx_file=fullfile(ds_dir, 'ImageSets/Segmentation/val.txt');\nfid=fopen(val_idx_file);\nval_file_names=textscan(fid, '%s');\nval_file_names=val_file_names{1};\nfclose(fid);\n\n\ntrain_num=length(train_file_names);\nimg_names=cat(1, train_file_names, val_file_names);\nimg_num=length(img_names);\n\nimg_files=cell(img_num, 1);\nmask_files=cell(img_num, 1);\n\nfor t_idx=1:img_num\n file_name=img_names{t_idx};\n img_files{t_idx}=[file_name(1:end-4) '.jpg'];\n mask_file_name = strrep(file_name,'JPEGImages','SegmentationClass');\n mask_files{t_idx}=[mask_file_name(1:end-4) '.png'];\n \nend\n\ntrain_idxes=1:train_num;\nval_idxes=train_num+1:img_num;\n\n\nds_info=[];\n\nds_info.img_names=img_names;\nds_info.img_files=img_files;\nds_info.mask_files=mask_files;\n\nds_info.train_idxes=uint32(train_idxes');\nds_info.test_idxes=uint32(val_idxes');\n\ndisp('here::')\nimg_dir=fullfile(ds_dir, 'JPEGImages')\nmask_dir=fullfile(ds_dir, 'SegmentationClass')\n\n\ndata_dirs=[];\ndata_dirs{1}=img_dir;\ndata_dirs{2}=mask_dir;\n\ndata_dir_idxes_img=zeros([img_num 1], 'uint8')+1;\ndata_dir_idxes_mask=zeros([img_num 1], 'uint8')+2;\n\nds_info.data_dir_idxes_img=data_dir_idxes_img;\nds_info.data_dir_idxes_mask=data_dir_idxes_mask;\nds_info.data_dirs=data_dirs;\nds_info.ds_dir=ds_dir;\n\nds_info.class_info=gen_class_info_ego();\n\nds_info.ds_name='eyth';\n\nds_info=process_ds_info_classification(ds_info, ds_config);\n\nend\n\n\n" }, { "path": "refinenet_files/my_gen_ds_info_gtea.m", "content": "\n\n\nfunction ds_info=my_gen_ds_info_gtea(ds_config)\n\n\n\nds_dir=fullfile('../datasets', 'gtea');\n\ntrain_idx_file=fullfile(ds_dir, 'ImageSets/Segmentation/train.txt');\nfid=fopen(train_idx_file);\ntrain_file_names=textscan(fid, '%s');\ntrain_file_names=train_file_names{1};\nfclose(fid);\n\n\nval_idx_file=fullfile(ds_dir, 'ImageSets/Segmentation/val.txt');\nfid=fopen(val_idx_file);\nval_file_names=textscan(fid, '%s');\nval_file_names=val_file_names{1};\nfclose(fid);\n\n\ntrain_num=length(train_file_names);\nimg_names=cat(1, train_file_names, val_file_names);\nimg_num=length(img_names);\n\nimg_files=cell(img_num, 1);\nmask_files=cell(img_num, 1);\n\nfor t_idx=1:img_num\n file_name=img_names{t_idx};\n img_files{t_idx}=[file_name(1:end-4) '.jpg'];\n mask_file_name = strrep(file_name,'JPEGImages','SegmentationClass');\n mask_files{t_idx}=[mask_file_name(1:end-4) '.png'];\n \nend\n\ntrain_idxes=1:train_num;\nval_idxes=train_num+1:img_num;\n\n\nds_info=[];\n\nds_info.img_names=img_names;\nds_info.img_files=img_files;\nds_info.mask_files=mask_files;\n\nds_info.train_idxes=uint32(train_idxes');\nds_info.test_idxes=uint32(val_idxes');\n\ndisp('here::')\nimg_dir=fullfile(ds_dir, 'JPEGImages')\nmask_dir=fullfile(ds_dir, 'SegmentationClass')\n\n\ndata_dirs=[];\ndata_dirs{1}=img_dir;\ndata_dirs{2}=mask_dir;\n\ndata_dir_idxes_img=zeros([img_num 1], 'uint8')+1;\ndata_dir_idxes_mask=zeros([img_num 1], 'uint8')+2;\n\nds_info.data_dir_idxes_img=data_dir_idxes_img;\nds_info.data_dir_idxes_mask=data_dir_idxes_mask;\nds_info.data_dirs=data_dirs;\nds_info.ds_dir=ds_dir;\n\nds_info.class_info=gen_class_info_ego();\n\nds_info.ds_name='gtea';\n\nds_info=process_ds_info_classification(ds_info, ds_config);\n\nend\n\n\n" }, { "path": "refinenet_files/my_gen_ds_info_hof.m", "content": "\n\n\nfunction ds_info=my_gen_ds_info_hof(ds_config)\n\n\n\nds_dir=fullfile('../datasets', 'hof');\n\ntrain_idx_file=fullfile(ds_dir, 'ImageSets/Segmentation/train.txt');\nfid=fopen(train_idx_file);\ntrain_file_names=textscan(fid, '%s');\ntrain_file_names=train_file_names{1};\nfclose(fid);\n\n\nval_idx_file=fullfile(ds_dir, 'ImageSets/Segmentation/val.txt');\nfid=fopen(val_idx_file);\nval_file_names=textscan(fid, '%s');\nval_file_names=val_file_names{1};\nfclose(fid);\n\n\ntrain_num=length(train_file_names);\nimg_names=cat(1, train_file_names, val_file_names);\nimg_num=length(img_names);\n\nimg_files=cell(img_num, 1);\nmask_files=cell(img_num, 1);\n\nfor t_idx=1:img_num\n file_name=img_names{t_idx};\n img_files{t_idx}=[file_name(1:end-4) '.jpg'];\n mask_file_name = strrep(file_name,'JPEGImages','SegmentationClass');\n mask_files{t_idx}=[mask_file_name(1:end-4) '.png'];\n \nend\n\ntrain_idxes=1:train_num;\nval_idxes=train_num+1:img_num;\n\n\nds_info=[];\n\nds_info.img_names=img_names;\nds_info.img_files=img_files;\nds_info.mask_files=mask_files;\n\nds_info.train_idxes=uint32(train_idxes');\nds_info.test_idxes=uint32(val_idxes');\n\ndisp('here::')\nimg_dir=fullfile(ds_dir, 'JPEGImages')\nmask_dir=fullfile(ds_dir, 'SegmentationClass')\n\n\ndata_dirs=[];\ndata_dirs{1}=img_dir;\ndata_dirs{2}=mask_dir;\n\ndata_dir_idxes_img=zeros([img_num 1], 'uint8')+1;\ndata_dir_idxes_mask=zeros([img_num 1], 'uint8')+2;\n\nds_info.data_dir_idxes_img=data_dir_idxes_img;\nds_info.data_dir_idxes_mask=data_dir_idxes_mask;\nds_info.data_dirs=data_dirs;\nds_info.ds_dir=ds_dir;\n\nds_info.class_info=gen_class_info_ego();\n\nds_info.ds_name='gtea';\n\nds_info=process_ds_info_classification(ds_info, ds_config);\n\nend\n\n\n" } ]