Repository: AnTao97/PointCloudDatasets
Branch: master
Commit: 54b1bfd4f53a
Files: 5
Total size: 19.1 KB
Directory structure:
gitextract_c1bfgm6a/
├── .gitignore
├── LICENSE
├── README.md
├── dataset.py
└── visualize.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.DS_Store
*/.DS_Store
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2019 An Tao
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# Point Cloud Datasets
This repository provides ShapeNetCore.v2, ShapeNetPart, ModelNet40 and ModelNet10 datasets in HDF5 format. For each shape in these datasets, we use farthest point sampling algorithm to uniformly sample 2,048 points from shape surface. All points are then centered and scaled. We follow the train/val/test split in official documents.
We also provide code to load and visualize our datasets with PyTorch 1.2 and Python 3.7. See `dataset.py` and run it to have a try.
To visualize, run `visualize.py` to generate XML file and use [Mitsuba](https://www.mitsuba-renderer.org/index.html) to render it. Our code is from this [repo](https://github.com/zekunhao1995/PointFlowRenderer).
## Download link:
- ShapeNetCore.v2 (0.98G) [[TsinghuaCloud]](https://cloud.tsinghua.edu.cn/f/06a3c383dc474179b97d/) [[BaiduDisk]](https://pan.baidu.com/s/154As2kzHZczMipuoZIc0kg)
- ShapeNetPart (338M) [[TsinghuaCloud]](https://cloud.tsinghua.edu.cn/f/c25d94e163454196a26b/) [[BaiduDisk]](https://pan.baidu.com/s/1yi4bMVBE2mV8NqVRtNLoqw)
- ModelNet40 (194M) [[TsinghuaCloud]](https://cloud.tsinghua.edu.cn/f/b3d9fe3e2a514def8097/) [[BaiduDisk]](https://pan.baidu.com/s/1NQZgN8tvHVqQntxefcdVAg)
- ModelNet10 (72.5M) [[TsinghuaCloud]](https://cloud.tsinghua.edu.cn/f/5414376f6afd41ce9b6d/) [[BaiduDisk]](https://pan.baidu.com/s/1tfnKQ_yg3SfIgyLSwQ2E0g)
## ShapeNetCore.v2
ShapeNetCore.v2 datset contains 51,127 pre-aligned shapes from 55 categories, which are split into 35,708 (70%) for training, 5,158 (10%) shapes for validation and 10,261 (20%) shapes for testing. In official document there should be 51,190 shapes in total, but 63 shapes are missing in original downloaded ShapeNetCore.v2 dataset from [here](https://www.shapenet.org/download/shapenetcore).
The 55 categories include: `airplane`, `bag`, `basket`, `bathtub`, `bed`, `bench`, `birdhouse`, `bookshelf`, `bottle`, `bowl`, `bus`, `cabinet`, `camera`, `can`, `cap`, `car`, `cellphone`, `chair`, `clock`, `dishwasher`, `earphone`, `faucet`, `file`, `guitar`, `helmet`, `jar`, `keyboard`, `knife`, `lamp`, `laptop`, `mailbox`, `microphone`, `microwave`, `monitor`, `motorcycle`, `mug`, `piano`, `pillow`, `pistol`, `pot`, `printer`, `remote_control`, `rifle`, `rocket`, `skateboard`, `sofa`, `speaker`, `stove`, `table`, `telephone`, `tin_can`, `tower`, `train`, `vessel`, `washer`.
Some visualized point clouds in our ShapeNetCore.v2 dataset:
earphone lamp tower
## ShapeNetPart
ShapeNetPart dataset contains 16,881 pre-aligned shapes from 16 categories, annotated with 50 segmentation parts in total. Most object categories are labeled with two to five segmentation parts. There are 12,137 (70%) shapes for training, 1,870 (10%) shapes for validation, and 2,874 (20%) shapes for testing. We also pack the segementation label in our dataset. The link for official dataset is [here](https://shapenet.cs.stanford.edu/media/shapenet_part_seg_hdf5_data.zip).
The 16 categories include: `airplane`, `bag`, `cap`, `car`, `chair`, `earphone`, `guitar`, `knife`, `lamp`, `laptop`, `motorbike`, `mug`, `pistol`, `rocket`, `skateboard`, `table`.
Although ShapeNetPart is made from ShapeNetCore, the number of points per shape in official ShapeNetPart dataset is not very large and sometimes less than 2,048. Thus the uniform sampling quality of our ShapeNetPart dataset is lower than our ShapeNetCore.v2 dataset.
In this dataset, we change segmentation label for each point into range 0~49 according to its category. You can find a index mapping list in `dataset.py`.
Some visualized point clouds in our ShapeNetPart dataset:
airplane table chair
## ModelNet40
ModelNet40 dataset contains 12,311 pre-aligned shapes from 40 categories, which are split into 9,843 (80%) for training and 2,468 (20%) for testing. The link for official dataset is [here](http://3dvision.princeton.edu/projects/2014/3DShapeNets/ModelNet10.zip).
The 40 categories include: `airplane`, `bathtub`, `bed`, `bench`, `bookshelf`, `bottle`, `bowl`, `car`, `chair`, `cone`, `cup`, `curtain`, `desk`, `door`, `dresser`, `flower_pot`, `glass_box`, `guitar`, `keyboard`, `lamp`, `laptop`, `mantel`, `monitor`, `night_stand`, `person`, `piano`, `plant`, `radio`, `range_hood`, `sink`, `sofa`, `stairs`, `stool`, `table`, `tent`, `toilet`, `tv_stand`, `vase`, `wardrobe`, `xbox`.
**Note**: The widely used 2,048 points sampled ModelNet40 dataset ([link](https://shapenet.cs.stanford.edu/media/modelnet40_ply_hdf5_2048.zip)) only contains 9,840 shapes for training, not 9,843 in official. Our ModelNet40 dataset fixs this problem and can substitute the above mentioned dataset perfectly.
Some visualized point clouds in our ModelNet40 dataset:
vase bookshelf plant
## ModelNet10
ModelNet10 dataset is a part of ModelNet40 dataset, containing 4,899 pre-aligned shapes from 10 categories. There are 3,991 (80%) shapes for training and 908 (20%) shapes for testing. The link for official dataset is [here](http://modelnet.cs.princeton.edu/ModelNet40.zip).
The 10 categories include: `bathtub`, `bed`, `chair`, `desk`, `dresser`, `monitor`, `night_stand`, `sofa`, `table`, `toilet`.
## Dataset performance
Repos below use our datasets:
- [antao97/UnsupervisedPointCloudReconstruction](https://github.com/antao97/UnsupervisedPointCloudReconstruction)
- coming soon ...
#### Reference repos:
- [charlesq34/pointnet](https://github.com/charlesq34/pointnet)
- [charlesq34/pointnet2](https://github.com/charlesq34/pointnet2)
- [stevenygd/PointFlow](https://github.com/stevenygd/PointFlow)
- [zekunhao1995/PointFlowRenderer](https://github.com/zekunhao1995/PointFlowRenderer)
- [WangYueFt/dgcnn](https://github.com/WangYueFt/dgcnn)
================================================
FILE: dataset.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
@Author: An Tao
@Contact: ta19@mails.tsinghua.edu.cn
@File: dataset.py
@Time: 2020/1/2 10:26 AM
"""
import os
import torch
import json
import h5py
from glob import glob
import numpy as np
import torch.utils.data as data
shapenetpart_cat2id = {'airplane': 0, 'bag': 1, 'cap': 2, 'car': 3, 'chair': 4,
'earphone': 5, 'guitar': 6, 'knife': 7, 'lamp': 8, 'laptop': 9,
'motor': 10, 'mug': 11, 'pistol': 12, 'rocket': 13, 'skateboard': 14, 'table': 15}
shapenetpart_seg_num = [4, 2, 2, 4, 4, 3, 3, 2, 4, 2, 6, 2, 3, 3, 3, 3]
shapenetpart_seg_start_index = [0, 4, 6, 8, 12, 16, 19, 22, 24, 28, 30, 36, 38, 41, 44, 47]
def translate_pointcloud(pointcloud):
xyz1 = np.random.uniform(low=2./3., high=3./2., size=[3])
xyz2 = np.random.uniform(low=-0.2, high=0.2, size=[3])
translated_pointcloud = np.add(np.multiply(pointcloud, xyz1), xyz2).astype('float32')
return translated_pointcloud
def jitter_pointcloud(pointcloud, sigma=0.01, clip=0.02):
N, C = pointcloud.shape
pointcloud += np.clip(sigma * np.random.randn(N, C), -1*clip, clip)
return pointcloud
def rotate_pointcloud(pointcloud):
theta = np.pi*2 * np.random.rand()
rotation_matrix = np.array([[np.cos(theta), -np.sin(theta)],[np.sin(theta), np.cos(theta)]])
pointcloud[:,[0,2]] = pointcloud[:,[0,2]].dot(rotation_matrix) # random rotation (x,z)
return pointcloud
class Dataset(data.Dataset):
def __init__(self, root, dataset_name='modelnet40', class_choice=None,
num_points=2048, split='train', load_name=True, load_file=True,
segmentation=False, random_rotate=False, random_jitter=False,
random_translate=False):
assert dataset_name.lower() in ['shapenetcorev2', 'shapenetpart',
'modelnet10', 'modelnet40', 'shapenetpartpart']
assert num_points <= 2048
if dataset_name in ['shapenetcorev2', 'shapenetpart', 'shapenetpartpart']:
assert split.lower() in ['train', 'test', 'val', 'trainval', 'all']
else:
assert split.lower() in ['train', 'test', 'all']
if dataset_name not in ['shapenetpart'] and segmentation == True:
raise AssertionError
self.root = os.path.join(root, dataset_name + '_hdf5_2048')
self.dataset_name = dataset_name
self.class_choice = class_choice
self.num_points = num_points
self.split = split
self.load_name = load_name
self.load_file = load_file
self.segmentation = segmentation
self.random_rotate = random_rotate
self.random_jitter = random_jitter
self.random_translate = random_translate
self.path_h5py_all = []
self.path_name_all = []
self.path_file_all = []
if self.split in ['train', 'trainval', 'all']:
self.get_path('train')
if self.dataset_name in ['shapenetcorev2', 'shapenetpart', 'shapenetpartpart']:
if self.split in ['val', 'trainval', 'all']:
self.get_path('val')
if self.split in ['test', 'all']:
self.get_path('test')
data, label, seg = self.load_h5py(self.path_h5py_all)
if self.load_name or self.class_choice != None:
self.name = np.array(self.load_json(self.path_name_all)) # load label name
if self.load_file:
self.file = np.array(self.load_json(self.path_file_all)) # load file name
self.data = np.concatenate(data, axis=0)
self.label = np.concatenate(label, axis=0)
if self.segmentation:
self.seg = np.concatenate(seg, axis=0)
if self.class_choice != None:
indices = (self.name == class_choice)
self.data = self.data[indices]
self.label = self.label[indices]
self.name = self.name[indices]
if self.segmentation:
self.seg = self.seg[indices]
id_choice = shapenetpart_cat2id[class_choice]
self.seg_num_all = shapenetpart_seg_num[id_choice]
self.seg_start_index = shapenetpart_seg_start_index[id_choice]
if self.load_file:
self.file = self.file[indices]
elif self.segmentation:
self.seg_num_all = 50
self.seg_start_index = 0
def get_path(self, type):
path_h5py = os.path.join(self.root, '%s*.h5'%type)
paths = glob(path_h5py)
paths_sort = [os.path.join(self.root, type + str(i) + '.h5') for i in range(len(paths))]
self.path_h5py_all += paths_sort
if self.load_name:
paths_json = [os.path.join(self.root, type + str(i) + '_id2name.json') for i in range(len(paths))]
self.path_name_all += paths_json
if self.load_file:
paths_json = [os.path.join(self.root, type + str(i) + '_id2file.json') for i in range(len(paths))]
self.path_file_all += paths_json
return
def load_h5py(self, path):
all_data = []
all_label = []
all_seg = []
for h5_name in path:
f = h5py.File(h5_name, 'r+')
data = f['data'][:].astype('float32')
label = f['label'][:].astype('int64')
if self.segmentation:
seg = f['seg'][:].astype('int64')
f.close()
all_data.append(data)
all_label.append(label)
if self.segmentation:
all_seg.append(seg)
return all_data, all_label, all_seg
def load_json(self, path):
all_data = []
for json_name in path:
j = open(json_name, 'r+')
data = json.load(j)
all_data += data
return all_data
def __getitem__(self, item):
point_set = self.data[item][:self.num_points]
label = self.label[item]
if self.load_name:
name = self.name[item] # get label name
if self.load_file:
file = self.file[item] # get file name
if self.random_rotate:
point_set = rotate_pointcloud(point_set)
if self.random_jitter:
point_set = jitter_pointcloud(point_set)
if self.random_translate:
point_set = translate_pointcloud(point_set)
# convert numpy array to pytorch Tensor
point_set = torch.from_numpy(point_set)
label = torch.from_numpy(np.array([label]).astype(np.int64))
label = label.squeeze(0)
if self.segmentation:
seg = self.seg[item]
seg = torch.from_numpy(seg)
return point_set, label, seg, name, file
else:
return point_set, label, name, file
def __len__(self):
return self.data.shape[0]
if __name__ == '__main__':
root = os.getcwd()
# choose dataset name from 'shapenetcorev2', 'shapenetpart', 'modelnet40' and 'modelnet10'
dataset_name = 'shapenetcorev2'
# choose split type from 'train', 'test', 'all', 'trainval' and 'val'
# only shapenetcorev2 and shapenetpart dataset support 'trainval' and 'val'
split = 'train'
d = Dataset(root=root, dataset_name=dataset_name, num_points=2048, split=split)
print("datasize:", d.__len__())
item = 0
ps, lb, n, f = d[item]
print(ps.size(), ps.type(), lb.size(), lb.type(), n, f)
================================================
FILE: visualize.py
================================================
import os
import numpy as np
def standardize_bbox(pcl, points_per_object):
pt_indices = np.random.choice(pcl.shape[0], points_per_object, replace=False)
np.random.shuffle(pt_indices)
pcl = pcl[pt_indices] # n by 3
mins = np.amin(pcl, axis=0)
maxs = np.amax(pcl, axis=0)
center = ( mins + maxs ) / 2.
scale = np.amax(maxs-mins)
print("Center: {}, Scale: {}".format(center, scale))
result = ((pcl - center)/scale).astype(np.float32) # [-0.5, 0.5]
return result
xml_head = \
"""
"""
xml_ball_segment = \
"""
"""
xml_tail = \
"""