Showing preview only (477K chars total). Download the full file or copy to clipboard to get everything.
Repository: Nicholasli1995/EvoSkeleton
Branch: master
Commit: b2b355f4c1fa
Files: 81
Total size: 451.1 KB
Directory structure:
gitextract_6ign6rd0/
├── .gitignore
├── LICENSE
├── README.md
├── data/
│ └── .gitignore
├── docs/
│ ├── 2DHPE.md
│ ├── ANNOTATOR.md
│ ├── DATASET.md
│ ├── HHR.md
│ ├── TRAINING.md
│ └── Zoo.md
├── examples/
│ ├── h36m2Dpose/
│ │ └── cfgs.yaml
│ ├── h36m2Dpose.py
│ └── inference.py
├── libs/
│ ├── __init__.py
│ ├── annotator/
│ │ ├── __init__.py
│ │ ├── angle.py
│ │ ├── fit_3d.py
│ │ ├── smpl-spec-list.txt
│ │ ├── smpl_webuser/
│ │ │ ├── LICENSE.txt
│ │ │ ├── README.txt
│ │ │ ├── __init__.py
│ │ │ ├── hello_world/
│ │ │ │ ├── hello_smpl.py
│ │ │ │ └── render_smpl.py
│ │ │ ├── lbs.py
│ │ │ ├── posemapper.py
│ │ │ ├── serialization.py
│ │ │ └── verts.py
│ │ └── smplify/
│ │ ├── __init__.py
│ │ ├── fit_3d.py
│ │ ├── lib/
│ │ │ ├── __init__.py
│ │ │ ├── capsule_body.py
│ │ │ ├── capsule_ch.py
│ │ │ ├── max_mixture_prior.py
│ │ │ ├── robustifiers.py
│ │ │ └── sphere_collisions.py
│ │ └── render_model.py
│ ├── dataset/
│ │ ├── __init__.py
│ │ └── h36m/
│ │ ├── __init__.py
│ │ ├── cameras.py
│ │ ├── data_utils.py
│ │ ├── h36m_pose.py
│ │ ├── pose_dataset.py
│ │ └── pth_dataset.py
│ ├── evolution/
│ │ ├── __init__.py
│ │ ├── genetic.py
│ │ └── parameter.py
│ ├── hhr/
│ │ ├── __init__.py
│ │ ├── config/
│ │ │ ├── __init__.py
│ │ │ ├── default.py
│ │ │ └── models.py
│ │ ├── core/
│ │ │ ├── __init__.py
│ │ │ ├── evaluate.py
│ │ │ ├── function.py
│ │ │ ├── inference.py
│ │ │ └── loss.py
│ │ └── utils/
│ │ ├── __init__.py
│ │ ├── transforms.py
│ │ ├── utils.py
│ │ └── vis.py
│ ├── model/
│ │ ├── __init__.py
│ │ ├── model.py
│ │ ├── pose_hrnet.py
│ │ └── pose_resnet.py
│ ├── optimizer/
│ │ └── __init__.py
│ ├── parser/
│ │ ├── __init__.py
│ │ └── parse.py
│ ├── skeleton/
│ │ ├── __init__.py
│ │ └── anglelimits.py
│ ├── trainer/
│ │ ├── __init__.py
│ │ └── trainer.py
│ ├── utils/
│ │ ├── __init__.py
│ │ └── utils.py
│ └── visualization/
│ ├── __init__.py
│ └── viz.py
├── resources/
│ └── .gitignore
├── spec-list.txt
└── tools/
├── 2Dto3Dnet.py
├── annotate_2D.py
├── annotate_3D.py
├── evolve.py
└── imgto2Dnet.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
#/*
**/__pycache__
.spyproject/
/libs/annotator/fitted
/libs/annotator/smplify/models
/model
/examples/imgs
/examples/example_annot.npy
/examples/example_model.th
/examples/stats.npy
/examples/h36m2Dpose/final_state.pth
*.yml
*.log
*.ini
*.bak
*.pth
*.csv
*.pyc
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2020 Shichao Li
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
[](https://paperswithcode.com/sota/weakly-supervised-3d-human-pose-estimation-on?p=cascaded-deep-monocular-3d-human-pose-1)
[](https://paperswithcode.com/sota/monocular-3d-human-pose-estimation-on-human3?p=cascaded-deep-monocular-3d-human-pose-1)
[](https://paperswithcode.com/sota/3d-human-pose-estimation-on-human36m?p=cascaded-deep-monocular-3d-human-pose-1)
# EvoSkeleton
This is the project website containing relevant files for the CVPR 2020 paper "Cascaded deep monocular 3D human pose estimation with evolutionary training data". The usage and instructions are organized into several parts serving distinct purposes. Please visit the corresponding sub-page for details. For Q&A, go to [discussions](https://github.com/Nicholasli1995/EvoSkeleton/discussions). If you believe there is a technical problem, submit to [issues](https://github.com/Nicholasli1995/EvoSkeleton/issues).
News:
(2021-04-08): Release v-1.0. The support for pre-trained models is strengthened. More details have been added to the supplementary material.
## Cascaded 2D-to-3D Lifting
[This sub-page](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/TRAINING.md) details how to train a cascaded model to lift 2D key-points to 3D skeletons on H36M.
If you do not want to prepare synthetic data and train the model by yourself, you can access an examplar pre-trained model [here](https://drive.google.com/file/d/158oCTK-9Y8Bl9qxidoHcXfqfeeA7qT93/view?usp=sharing) and follow the instructions in the [document](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/TRAINING.md). This model can be used for in-the-wild inference as well as reproducing the results on MPI-INF-3DHP. The evaluation metric for MPI-INF-3DHP can be accessed [here](https://github.com/chenxuluo/OriNet-demo/tree/master/src/test_util).
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/architecture.jpg"/>
</p>
Performance on H36M ([Link to pre-trained models](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/Zoo.md))
| Protocol \#1| Avg.|Dir. | Disc| Eat| Greet| Phone| Photo | Pose | Purch.| Sit| SitD.| Smoke| Wait| WalkD.| Walk | WalkT.|
|-------------------------------------------------------------|------------------|------------------|---------------|------------------|---------------|---------------|------|---------------|------------------|------------------|---------------|---------------|---------------|---------------|---------------|---------------|
| [Martinez](https://github.com/una-dinosauria/3d-pose-baseline) et al. (ICCV'17) |62.9| 51.8 | 56.2| 58.1| 59.0 | 69.5 | 78.4| 55.2 | 58.1 | 74.0 | 94.6| 62.3 | 59.1 | 65.1 | 49.5 | 52.4 |
| Ours (S15678) |**49.7**|**45.6**|**44.6**|**49.3**|**49.3**|**52.5**|**58.5**|**46.4**|**44.3**|**53.8**|**67.5**|**49.4**|**46.1**|**52.5**|**41.4**|**44.4**|
| Protocol \#2| Avg.|Dir. | Disc| Eat| Greet| Phone| Photo | Pose | Purch.| Sit| SitD.| Smoke| Wait| WalkD.| Walk | WalkT.|
|-------------------------------------------------------------|------------------|------------------|---------------|------------------|---------------|---------------|------|---------------|------------------|------------------|---------------|---------------|---------------|---------------|---------------|---------------|
| [Martinez](https://github.com/una-dinosauria/3d-pose-baseline) et al. (ICCV'17) |47.7| 39.5 | 43.2 | 46.4 | 47.0 | 51.0| 56.0 | 41.4 | 40.6 | 56.5 | 69.4 | 49.2 | 45.0 | 49.5 | 38.0 | 43.1 |
| Ours (S15678) |**37.7** |**34.2**|**34.6**|**37.3**|**39.3**|**38.5**|**45.6**|**34.5**|**32.7**|**40.5**|**51.3**|**37.7**|**35.4**|**39.9**|**29.9**|**34.5**|
## Hierarchical Human Representation and Data Synthesis
[This sub-page](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/HHR.md) gives instructions on how to use the 3D skeleton model and how the evolution algorithm can be used to discover novel data.
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/hierarchical.jpg" width="394" height="243" />
</p>
## 2D Human Pose Estimation on H3.6M
[This page](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/2DHPE.md) shows how to perform 2D human pose estimation on Human 3.6M dataset with the pre-trained high-resolution heatmap regression model. The highly accurate 2D joint predictions may benefit your 3D human pose estimation project.
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/h36m2dpose2.png" width="789" height="208"/>
</p>
| Method | Parameters| FLOPs|Average Joint Localization Error (pixels) |
| ------------------------- | ---------------| --------------| --------------|
| CPN (CVPR' 18) | -|-| 5.4 |
| Ours (HRN + U + S) |63.6M| 32.9G | **4.4** |
## Dataset: Unconstrained 3D Pose in the Wild
[This sub-page](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/DATASET.md) describs the newly collected dataset Unconstrained 3D Human Pose in the Wild (U3DPW) and gives instructions on how to download it.
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/U3DPW.png"/>
</p>
## Interactive Annotation Tool
[This sub-page](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/ANNOTATOR.md) provides usage of an annotation tool that can be used to label 2D and 3D skeleton for an input image. U3DPW was obtained created with this tool and this tool may help increasing the scale of 3D annotation for in-the-wild images.
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/tool.gif" width="531" height="291"/>
</p>
## Environment
- Python 3.6
- Numpy 1.16
- PyTorch 1.0.1
- CUDA 9
For a complete list of other python packages, please refer to [spec-list.txt](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/spec-list.txt). The recommended environment manager is Anaconda, which can create an environment using the provided spec-list. Certain tool in this project may need other specified environment, which is detailed in its corresponding page.
## License
A MIT license is used for this repository. However, certain third-party dataset (Human 3.6M) and tool (SMPLify) are subject to their respective licenses and may not grant commercial use.
## Citation
Please star this repository and cite the following paper in your publications if it helps your research:
@InProceedings{Li_2020_CVPR,
author = {Li, Shichao and Ke, Lei and Pratama, Kevin and Tai, Yu-Wing and Tang, Chi-Keung and Cheng, Kwang-Ting},
title = {Cascaded Deep Monocular 3D Human Pose Estimation With Evolutionary Training Data},
booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}
Link to the paper:
[Cascaded Deep Monocular 3D Human Pose Estimation With Evolutionary Training Data](https://arxiv.org/abs/2006.07778)
Link to the oral presentation video:
[Youtube](https://www.youtube.com/watch?v=erYymlWw2bo)
================================================
FILE: data/.gitignore
================================================
# Ignore everything in this directory
*
# Except this file
!.gitignore
================================================
FILE: docs/2DHPE.md
================================================
The pre-trained model can be downloaded from [here](https://drive.google.com/file/d/1NjQFCz0GdS7oIdYrK5ouxEI07wYYh4r8/view?usp=sharing) and placed under "${EvoSkeleton}/examples/h36m2Dpose" folder. Your directory should look like this:
```
${EvoSkeleton}
├── examples
├── h36m2Dpose
├── cropped (prepared testing images from Human 3.6M)
├── cfgs.yaml (configuration file)
├── final_state.pth (pre-trained high-resolution heatmap regression model)
```
Then run h36m2Dpose.py at ${EvoSkeleton}/examples
```bash
python h36m2Dpose.py
```
You should expect to see results like [this](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/h36m2dpose2.png). I only uploaded a few example images since I cannot upload the whole video due to the license requirement. For your own images, you should crop the humans and prepare your data accordingly.
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/h36m2dpose.png" width="924" height="506"/>
</p>
================================================
FILE: docs/ANNOTATOR.md
================================================
The annotator is composed of three parts:
1. 2D annotation: interactively annotate 2D key-points for RGB images
2. 3D parameter fitting: obtain coarse 3D skeleton fitting results based on SMPLify.
3. 3D annotation: interactively modify 3D parameters.
## 2D Keypoints Annotation
Users can annotate 2D Keypoints of images by running the script `annotate_2d.py` under ${EvoSkeleton}/tools.
```bash
python annotate_2d.py -d DATASET_PATH
```
DATASET_PATH is the path to the folder containing images.
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/annotator_2d.gif"/>
</p>
Users can annotate 2D Keypoints in the following order by clicking on the image:
Right Ankle, Right Knee, Right Hip, Left Hip, Left Knee, Left Ankle, Right Wrist, Right Elbow, Right Shoulder,
Left Shoulder, Left Elbow, Left Wrist, Neck, Head top, Spine, Thorax, Nose
Other keyborad short-cuts are:
Press Q to exit the tool.
Press N to go to the next image.
Press Z to save the annotation.
Press C to erase all of the assigned keypoints from the image and start over.
## Coarse 3D Keypoints Estimation
Manually annotating 3D skeleton from scratch is time-consuming, thus we use a tool to obtain an initial 3D pose estimation given 2D annotation. Any method that outputs 3D pose inference given 2D key-points can be employed.
Here we use SMPLify to estimate coarse 3D skeleton. You need to set up a Python 2.7 environment where a [spec-list](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/libs/annotator/smpl-spec-list.txt) can be used as your reference if you use Anaconda. Then you need to install chumpy and opendr using pip:
```bash
pip install chumpy
pip install opendr
```
After setting up the environment, you need to download the SMPL model files [here](https://drive.google.com/drive/folders/12qJQP-h4E43FkgE74tybQUjeP_pnqAor?usp=sharing) and organize your project files as follows:
```
${EvoSkeleton}
├── libs
├── annotator
├── smplify
├── models
├── basicModel_neutral_lbs_10_207_0_v1.0.0.pkl
├── gmm_08.pkl
├── fit_3d.py
```
Then one can run fit_3d under ${EvoSkeleton}/libs/annotator/fit_3d.py to fit the SMPL model
```bash
python fit_3d.py -dataset_dir DATASET_PATH -model_dir MODEL_PATH
```
DATASET_PATH is the path to the folder containing the annotated 2D key-point file "annotation.npy".
MODEL_PATH is the path to the used SMPL model (for example, basicModel_neutral_lbs_10_207_0_v1.0.0.pkl). There are other available models depending on the gender of the subject.
The fitting process can be shown during running and the file annotation.npy will be updated with 3D parameters.
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/fitted.png"/>
</p>
## 3D Skeleton Interactive Annotation
After you have obtained fitted parameters for your dataset, you can modify the 3D parameters interactively with this tool. If you just want to try this tool without finishing the above steps, you can play with the pre-fitted parameters [here](https://drive.google.com/file/d/1OJokg844KDpRG3YQsZNpXwFXlVF8iOH5/view?usp=sharing) for U3DPW images. To start the tool, go to ${EvoSkeleton}/tools and run
```bash
python annotate_3D.py -dataset_dir DATASET_PATH
```
DATASET_PATH is the path to the folder containing the fitted parameters "fitted.npy". The tool will select one unannotated image, display it and start the interactive process. You can modify the global orientation as well as the limb orientation of the 3D skeleton. A 2D image with projected key-points will be plotted on-line so that you can check if your annotation is reasonable or not.
Some important keyborad short-cuts are:
Press number keys (2-9) to select which bone vector to rotate.
Press 0 so that the 3D skeleton will rotate as a whole.
Press arrow keys (up and down) to rotate the bone vector.
Press "m" to save an updated annotation file.
Other keyboard inputs are detailed in annotate_3D.py
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/tool.gif"/>
</p>
================================================
FILE: docs/DATASET.md
================================================
## Download
You can access the dataset [here](https://drive.google.com/file/d/1JRJuL69J0drZOAUT8VDK5ywmxt-Gm7s-/view?usp=sharing).
## Folder Structure
- imgs - Contains the collected images
- annotation.npy - Contains the pose annotation
```
${U3DPW}
├── imgs
├── annotation.npy
```
## Annotation
The annotation file is a Python dictionary that has the following format:
p2d is a numpy array of shape (num_keypoints, 2) that stores the image coordinates of the 2D key-points. Each row in the array stores (x, y) coordinate of the corresponding key-point. These key-points are re-annotated with a style similar to that of Human 3.6M, and can be accessed through key 'h36m'.
lsp is a boolean flag that indicates whether the image is collected from [Leeds Sport Pose dataset](https://sam.johnson.io/research/lsp.html) or not.
```
{
'image_name1':{'p2d':array1, 'lsp':True/False, 'h36m':array2},
'image_name2':{'p2d':array3, 'lsp':True/False, 'h36m':array4},
...
}
```
## Key-point Semantics
The name of the Human 3.6M style key-points are:
| Index | Keypoint |
|---|-------------|
| 0 | Hip Center |
| 1 | Right Hip |
| 2 | Right Knee |
| 3 | Right Ankle |
| 4 | Left Hip |
| 5 | Left Knee |
| 6 | Left Ankle |
| 7 | Spine |
| 8 | Thorax |
| 9 | Neck |
| 10 | Head Top |
| 11 | Left SHoulder |
| 12 | Left Elbow |
| 13 | Left Wrist |
| 14 | Right Shoulder |
| 15 | Right Elbow|
| 16 | Right Wrist|
================================================
FILE: docs/HHR.md
================================================
## Data Preparation
Please prepare data as instructed in the model training [sub-page](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/TRAINING.md).
The training data need to downloaded from [here](https://drive.google.com/drive/folders/1zyW8ryGXLq4bumWnVGUROpDNdubWUExg?usp=sharing) and placed under "${EvoSkeleton}/data" folder:
```
${EvoSkeleton}
├── data
├── human3.6M
├── your downloaded files
```
## Model Preparation
During data space exploration, a function that evaluates the validity of 3D skeletons is used. This function is parametrized with a model propsosed by Ijaz Akhter in CVPR 2015. You need to download the "constraints" folder from [here](https://drive.google.com/drive/folders/1MUcR9oBNUpTAJ7YUWdyVLKCQW874FszI?usp=sharing) which contains the model parameters and place them under "${EvoSkeleton}/resources" folder:
```
${EvoSkeleton}
├── recources
├── constraints
```
## Dataset Evolution
To evolve from a population of 3D skeleton (default to Human 3.6M data), go to "${EvoSkeleton}/tools" folder and run
```bash
python evolve.py -generate True
```
### Controling the Initial Population
To reproduce the experiments in different settings, you need to specify the choice of initial population.
For weakly-supervised experiments, you should only start with subject 1 (S1) data (a subset of H36M training data) as follows
```bash
python evolve.py -generate True -WS True -SS "S1"
```
You can even start with extremly scarce data (e.g., 1 percent of S1 data) as follows
```bash
python evolve.py -generate True -WS True -SS "0.01S1"
```
After finished dataset evolution, you can use the saved file for [training](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/TRAINING.md) to see how dataset evolution might help improve model generalization especially when the initial population is scarce.
## Reference
@inproceedings{akhter2015pose,
title={Pose-conditioned joint angle limits for 3D human pose reconstruction},
author={Akhter, Ijaz and Black, Michael J},
booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
pages={1446--1455},
year={2015}
}
================================================
FILE: docs/TRAINING.md
================================================
## Data Preparation
Similar to other repositories ([SimpleBaseline](https://github.com/una-dinosauria/3d-pose-baseline), [TemporalConvolution](https://github.com/facebookresearch/VideoPose3D)) on training 2D-to-3D networks, we provide pre-processed 2D detections, camera parameters and 3D poses for training. The 2D detections are produced by our modified high-resolution model, while the camera parameters and the 3D poses are taken from [SimpleBaseline](https://github.com/una-dinosauria/3d-pose-baseline).
The training data need to downloaded from [here](https://drive.google.com/drive/folders/1zyW8ryGXLq4bumWnVGUROpDNdubWUExg?usp=sharing) and placed under "${EvoSkeleton}/data" folder. Your directory should look like this:
```
${EvoSkeleton}
├── data
├── human3.6M
├── cameras.npy (Camera parameters provided by Human 3.6M)
├── threeDPose_train.npy (3D skeletons from Human 3.6M training split)
├── threeDPose_test.npy (3D skeletons from Human 3.6M test split)
├── twoDPose_HRN_train.npy (2D key-point detections obtained from the heatmap regression model for the training split)
├── twoDPose_HRN_test.npy (2D key-point detections obtained from the heatmap regression model for the test split)
```
## Weakly-Supervised Experiments on Human 3.6M Dataset
To compare with other weakly-supervised methods, only a subset of training data (e.g., subject 1 data) is used to simulate an environment where data is scarce.
To perform training, go to ./tools and run
```bash
python 2Dto3Dnet.py -train True -num_stages 2 -ws True -ws_name "S1"
```
This command performs training on synthetic 2D key-points to remove the influence of the heatmap regression model, whose results correspond to P1* in the performance table. S1 stands for subject 1 data. "num_stages" specify the number of deep learners used in the cascade.
To train on real detections obtained by the high-resolution heatmap regression model, run
```bash
python 2Dto3Dnet.py -train True -num_stages 2 -ws True -ws_name "S1" -twoD_source "HRN"
```
To train on evolved dataset, you need to specify the path to the evolved data as
```bash
python 2Dto3Dnet.py -train True -num_stages 2 -ws True -ws_name "S1" -twoD_source "HRN/synthetic" -evolved_path "YourDataPath"
```
See [this page](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/HHR.md) on how to evolve a dataset.
After data augmentation using the evolved data, we noticed the model generalization improves significantly despite the initial population size is small. Other methods utilize multi-view or temporal consistency instead of data augmentation to supervise deep models when data is scarce. Compared to them, we achieve state-of-the-art performance by synthesizing new data to supervise the deep model. P1 and P2 refers to the two protocols used for calculating average MPJPE over all 15 actions in H36M.
| Method | Avg. MPJPE (P1) | Avg. MPJPE (P2) |
| ------------------------- | --------------- | --------------- |
| Rhodin et al. (CVPR' 18) | - | 64.6 |
| Kocabas et al. (CVPR' 19) | 65.3 | 57.2 |
| Pavllo et al. (CVPR' 19) | 64.7 | - |
| Li et al. (ICCV' 19) | 88.8 | 66.5 |
| Ours | **60.8** | **46.2** |
## Fully-Supervised Experiments on Human 3.6M Dataset
To train on real detections obtained by the high-resolution heatmap regression model, run
```bash
python 2Dto3Dnet.py -train True -num_stages 2 -twoD_source "HRN"
```
To train on evolved dataset, you need to specify the path to the evolved data as
```bash
python 2Dto3Dnet.py -train True -num_stages 3 -num_blocks 3 -twoD_source "HRN/synthetic" -evolved_path "YourDataPath"
```
Here we increase model capacity with "-num_stages 3 -num_blocks 3" since the training data size is much larger (if you evolve enough generations).
While the improvement using data evolution is less obvious in fully-supervised setting compared with weakly-supervised setting, our cascaded model still achieved competitive performance compared with other 2D-to-3D lifting models.
| Method | Avg. MPJPE (P1) | Avg. MPJPE (P2) |
| -------------------------- | --------------- | --------------- |
| [Martinez et al.](https://github.com/una-dinosauria/3d-pose-baseline) (ICCV' 17) | 62.9 | 47.7 |
| Yang et al. (CVPR' 18) | 58.6 | **37.7** |
| Zhao et al. (CVPR' 19) | 57.6 | - |
| Sharma et al. (CVPR' 19) | 58.0 | 40.9 |
| Moon et al. (ICCV' 19) | 54.4 | - |
| Ours | **50.9** | 38.0 |
## Inference Example
If you only want to use a pre-trained model to conduct inference on in-the-wild images (skipping data synthesis and model training), you can download the sample images and a pre-trained checkpoint [here](https://drive.google.com/file/d/158oCTK-9Y8Bl9qxidoHcXfqfeeA7qT93/view?usp=sharing). Un-zip the downloaded file to "${EvoSkeleton}/examples" folder and your directory should look like this:
```
${EvoSkeleton}
├── examples
├── imgs (sample images)
├── example_annot.npy (2D key-points for the samples)
├── example_model.th (pre-trained model)
├── stats.npy (model statistics)
├── inference.py
```
Then you can run the following command at "${EvoSkeleton}/examples" to perform inference
```bash
python inference.py
```
<p align="center">
<img src="https://github.com/Nicholasli1995/EvoSkeleton/blob/master/imgs/example.png"/>
</p>
================================================
FILE: docs/Zoo.md
================================================
This page presents model performance on H36M under various settings. Pre-trained models and instructions for reproduction can also be found.
## Fully-supervised Setting (S15678)
[Download our pre-trained model](https://drive.google.com/drive/folders/1IRKUWrnheD03Dj30LLGlh_LLT1CK6dr5?usp=sharing)
[Download our pre-evolved data](https://drive.google.com/drive/folders/1FKFkmTJQcEdrCvZOSc8cF5OTTjFvyLav?usp=sharing)
Inference command:
```bash
python 2Dto3Dnet.py -evaluate True -twoD_source "HRN" -ckpt_dir "YourMODELPath"
```
Training command ([Docs](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/TRAINING.md)):
```bash
python 2Dto3Dnet.py -train True -num_stages 3 -num_blocks 3 -twoD_source "HRN" -evolved_path "YourDataPath"
```
Data synthesis command ([Docs](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/HHR.md)):
```bash
python evolve.py -SS "S15678" -T 1.5 -SD "YourDataPath" -generate True
```
MPJPE (P1) for each action under fully-supervised setting is shown in the table below.
| Protocol \#1 | Dir. | Disc | Eat | Greet | Phone | Photo | Pose | Purch. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT. | Avg. |
|-------------------------------------------------------------|------------------|------------------|---------------|------------------|---------------|---------------|------|---------------|------------------|------------------|---------------|---------------|---------------|---------------|---------------|---------------|
| [Martinez](https://github.com/una-dinosauria/3d-pose-baseline) et al. (ICCV'17) | 51.8 | 56.2 | 58.1 | 59.0 | 69.5 | 78.4 | 55.2 | 58.1 | 74.0 | 94.6 | 62.3 | 59.1 | 65.1 | 49.5 | 52.4 | 62.9 |
| [Fang](https://arxiv.org/abs/1710.06513) et al. (AAAI'18) | 50.1 | 54.3 | 57.0 | 57.1 | 66.6 | 73.3 | 53.4 | 55.7 | 72.8 | 88.6 | 60.3 | 57.7 | 62.7 | 47.5 | 50.6 | 60.4 |
| [Yang](https://arxiv.org/abs/1803.09722) et al. (CVPR'18) | 51.5 | 58.9 | 50.4 | 57.0 | 62.1 | 65.4 | 49.8 | 52.7 | 69.2 | 85.2 | 57.4 | 58.4 | 43.6 | 60.1 | 47.7 | 58.6 |
| [Pavlakos](https://github.com/geopavlakos/ordinal-pose3d) et al. (CVPR'18) | 48.5 | 54.4 | 54.4 | 52.0 | 59.4 | 65.3 | 49.9 | 52.9 | 65.8 | 71.1 | 56.6 | 52.9 | 60.9 | 44.7 | 47.8 | 56.2 |
| [Lee](https://openaccess.thecvf.com/content_ECCV_2018/papers/Kyoungoh_Lee_Propagating_LSTM_3D_ECCV_2018_paper.pdf) et al. (ECCV'18) | 40.2 | 49.2 | 47.8 | 52.6 | 50.1 | 75.0 | 50.2 | 43.0 | 55.8 | 73.9 | 54.1 | 55.6 | 58.2 | 43.3 | 43.3 | 52.8 |
| [Zhao](https://arxiv.org/abs/1904.03345) et al. (CVPR'19) | 47.3 | 60.7 | 51.4 | 60.5 | 61.1 | 49.9 | 47.3 | 68.1 | 86.2 | 55.0 | 67.8 | 61.0 | 42.1 | 60.6 | 45.3 | 57.6 |
| [Sharma](https://arxiv.org/abs/1904.01324) et al. (ICCV'19) | 48.6 | 54.5 | 54.2 | 55.7 | 62.6 | 72.0 | 50.5 | 54.3 | 70.0 | 78.3 | 58.1 | 55.4 | 61.4 | 45.2 | 49.7 | 58.0 |
| [Moon](https://github.com/mks0601/3DMPPE_POSENET_RELEASE) et al. (ICCV'19) | 51.5 | 56.8 | 51.2 | 52.2 | 55.2 | 47.7 | 50.9 | 63.3 | 69.9 | 54.2 | 57.4 | 50.4 | 42.5 | 57.5 | 47.7 | 54.4 |
| [Liu](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123550324.pdf) et al. (ECCV'20) | 46.3 | 52.2 | 47.3 | 50.7 | 55.5 | 67.1 | 49.2 | 46.0 | 60.4 | 71.1 | 51.5 | 50.1 | 54.5 | 40.3 | 43.7 | 52.4 |
| Ours (S15678) |45.6|44.6|49.3|49.3|52.5|58.5|46.4|44.3|53.8|67.5|49.4|46.1|52.5|41.4|44.4| 49.7 |
MPJPE (P2) for each action under fully-supervised setting is shown in the table below.
| Protocol \#2 | Dir. | Disc | Eat | Greet | Phone | Photo | Pose | Purch. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT. | Avg. |
|-------------------------------------------------------------|------------------|------------------|---------------|------------------|---------------|---------------|------|---------------|------------------|------------------|---------------|---------------|---------------|---------------|---------------|---------------|
| [Martinez](https://github.com/una-dinosauria/3d-pose-baseline) et al. (ICCV'17) | 39.5 | 43.2 | 46.4 | 47.0 | 51.0 | 56.0 | 41.4 | 40.6 | 56.5 | 69.4 | 49.2 | 45.0 | 49.5 | 38.0 | 43.1 | 47.7 |
| [Fang](https://arxiv.org/abs/1710.06513) et al. (AAAI'18) | 38.2 | 41.7 | 43.7 | 44.9 | 48.5 | 55.3 | 40.2 | 38.2 | 54.5 | 64.4 | 47.2 | 44.3 | 47.3 | 36.7 | 41.7 | 45.7 |
| [Pavlakos](https://github.com/geopavlakos/ordinal-pose3d) et al. (CVPR'18) | 34.7 | 39.8 | 41.8 | 38.6 | 42.5 | 47.5 | 38.0 | 36.6 | 50.7 | 56.8 | 42.6 | 39.6 | 43.9 | 32.1 | 36.5 | 41.8 |
| [Yang](https://arxiv.org/abs/1803.09722) et al. (CVPR'18) | 26.9 | 30.9 | 36.3 | 39.9 | 43.9 | 47.4 | 28.8 | 29.4 | 36.9 | 58.4 | 41.5 | 30.5 | 29.5 | 42.5 | 32.2 | 37.7 |
| [Sharma](https://arxiv.org/abs/1904.01324) et al. (ICCV'19) | 35.3 | 35.9 | 45.8 | 42.0 | 40.9 | 52.6 | 36.9 | 35.8 | 43.5 | 51.9 | 44.3 | 38.8 | 45.5 | 29.4 | 34.3 | 40.9 |
| [Cai](https://openaccess.thecvf.com/content_ICCV_2019/papers/Cai_Exploiting_Spatial-Temporal_Relationships_for_3D_Pose_Estimation_via_Graph_Convolutional_ICCV_2019_paper.pdf) et al. (ICCV'19) | 35.7 | 37.8 | 36.9 | 40.7 | 39.6 | 45.2 | 37.4 | 34.5 | 46.9 | 50.1 | 40.5 | 36.1 | 41.0 | 29.6 | 33.2 | 39.0 |
| [Liu](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123550324.pdf) et al. (ECCV'20) | 35.9 | 40.0 | 38.0 | 41.5 | 42.5 | 51.4 | 37.8 | 36.0 | 48.6 | 56.6 | 41.8 | 38.3 | 42.7 | 31.7 | 36.2 | 41.2 |
| Ours (S15678) |34.2|34.6|37.3|39.3|38.5|45.6|34.5|32.7|40.5|51.3|37.7|35.4|39.9|29.9|34.5| 37.7 |
## Weakly-supervised Setting (S1)
[Download our pre-trained model](https://drive.google.com/drive/folders/1PZoiizPKeoFTsvnFKIxaRDNbyb0Csx50?usp=sharing)
[Download our pre-evolved data](https://drive.google.com/drive/folders/1nTW2CCCT_sbJ1CejhuiQLTgDDU5sJjZj?usp=sharing)
Inference command:
```bash
python 2Dto3Dnet.py -evaluate True -twoD_source "HRN" -ckpt_dir "YourMODELPath"
```
Training command ([Docs](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/TRAINING.md)):
```bash
python 2Dto3Dnet.py -train True -num_stages 2 -ws True -ws_name "S1" -twoD_source "HRN" -evolved_path "YourDataPath"
```
Data synthesis command ([Docs](https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/HHR.md)):
```bash
python evolve.py -generate True -WS True -SS "S1"
```
MPJPE (P1) for each action under weakly-supervised setting is shown in the table below.
| Protocol \#1 | Dir. | Disc | Eat | Greet | Phone | Photo | Pose | Purch. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT. | Avg. |
|--------------------------------------------------------|------------------|------------------|---------------|------------------|---------------|------------------|------|---------------|------------------|-------|---------------|---------------|---------------|---------------|---------------|---------------|
| [Kocabas](https://arxiv.org/abs/1903.02330) et al. (CVPR'19) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 65.3 |
| [Pavllo](https://arxiv.org/abs/1811.11742) et al. (CVPR'19) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 64.7 |
| [Li](https://openaccess.thecvf.com/content_ICCV_2019/papers/Li_On_Boosting_Single-Frame_3D_Human_Pose_Estimation_via_Monocular_Videos_ICCV_2019_paper.pdf) et al. (ICCV'19) | 70.4 | 83.6 | 76.6 | 78.0 | 85.4 | 106.1 | 72.2 | 103.0 | 115.8 | 165.0 | 82.4 | 74.3 | 94.6 | 60.1 | 70.6 | 88.8 |
| Ours (S1) | 52.8 | 56.6 | 54.0 | 57.5 | 62.8 | 72.0 | 55.0 | 61.3 | 65.8 | 80.7 | 58.9 | 56.7 | 69.7 | 51.6 | 57.2 | 60.8 |
MPJPE (P2) for each action under fully-supervised setting is shown in the table below.
| Protocol \#2 | Dir. | Disc | Eat | Greet | Phone | Photo | Pose | Purch. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT. | Avg. |
|--------------------------------------------------------|------------------|------------------|---------------|------------------|---------------|------------------|------|---------------|------------------|-------|---------------|---------------|---------------|---------------|---------------|---------------|
| [Rhodin](https://arxiv.org/abs/1803.04775) et al. (CVPR'18) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 64.6 |
| [Kocabas](https://arxiv.org/abs/1903.02330) et al. (CVPR'19) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 57.2 |
| [Li](https://openaccess.thecvf.com/content_ICCV_2019/papers/Li_On_Boosting_Single-Frame_3D_Human_Pose_Estimation_via_Monocular_Videos_ICCV_2019_paper.pdf) et al. (ICCV'19) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 66.5 |
| Ours (S1) | 40.2 | 43.4 | 41.9| 46.1 | 48.2 | 55.1 | 42.8 | 42.6 | 49.6 | 61.1 | 44.5 | 43.2 | 51.5 | 38.1 | 44.4 | 46.2 |
================================================
FILE: examples/h36m2Dpose/cfgs.yaml
================================================
CUDNN:
BENCHMARK: true
DETERMINISTIC: false
ENABLED: true
GPUS: (0,)
MODEL:
NAME: pose_hrnet
NUM_JOINTS: 17
TARGET_TYPE: 'coordinate'
IMAGE_SIZE:
- 288
- 384
HEATMAP_SIZE:
- 288
- 384
EXTRA:
PRETRAINED_LAYERS:
- 'conv1'
- 'bn1'
- 'conv2'
- 'bn2'
- 'layer1'
- 'transition1'
- 'stage2'
- 'transition2'
- 'stage3'
- 'transition3'
- 'stage4'
FINAL_CONV_KERNEL: 1
STAGE2:
NUM_MODULES: 1
NUM_BRANCHES: 2
BLOCK: BASIC
NUM_BLOCKS:
- 4
- 4
NUM_CHANNELS:
- 48
- 96
FUSE_METHOD: SUM
STAGE3:
NUM_MODULES: 4
NUM_BRANCHES: 3
BLOCK: BASIC
NUM_BLOCKS:
- 4
- 4
- 4
NUM_CHANNELS:
- 48
- 96
- 192
FUSE_METHOD: SUM
STAGE4:
NUM_MODULES: 3
NUM_BRANCHES: 4
BLOCK: BASIC
NUM_BLOCKS:
- 4
- 4
- 4
- 4
NUM_CHANNELS:
- 48
- 96
- 192
- 384
FUSE_METHOD: SUM
================================================
FILE: examples/h36m2Dpose.py
================================================
"""
Examplar code showing how to use pre-trained heatmap regression model H() to
perform 2D pose estimation on Human 3.6M images.
"""
import sys
sys.path.append("../")
from libs.hhr.config import cfg
from libs.hhr.config import update_config
from libs.hhr.utils.utils import get_model_summary
from libs.model.pose_hrnet import get_pose_net
from libs.hhr.utils.transforms import get_affine_transform
from libs.hhr.core.loss import get_max_preds_soft_pt
import torch
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torchvision.transforms as transforms
import argparse
import os
import logging
import cv2
import numpy as np
import matplotlib.pyplot as plt
pose_connection = np.array([[0,1], [1,2], [2,3], [0,4], [4,5], [5,6], [0,7], [7,8],
[8,9], [9,10], [8,11], [11,12], [12,13], [8,14], [14,15],
[15,16]], dtype=np.int
)
I = pose_connection[:, 0]
J = pose_connection[:, 1]
pose_color = np.array([[255, 0, 0], [255, 85, 0], [255, 170, 0], [255, 255, 0],
[170, 255, 0], [85, 255, 0], [0, 255, 0], [0, 255, 85],
[0, 255, 170], [0, 255, 255], [0, 170, 255], [0, 85, 255],
[0, 0, 255], [85, 0, 255], [170, 0, 255], [255, 0, 255]
])/255.
re_order = [3, 12, 14, 16, 11, 13, 15, 1, 2, 0, 4, 5, 7, 9, 6, 8, 10]
def show2Dpose(vals, ax, lcolor="#3498db", rcolor="#e74c3c", add_labels=False):
for i in np.arange( len(I) ):
x, y = [np.array([vals[I[i], j], vals[J[i], j]]) for j in range(2)]
ax.plot(x, y, c=pose_color[i])
def parse_args():
parser = argparse.ArgumentParser(description='2D pose estimation example')
parser.add_argument('--cfg',
help='configuration file',
default='./h36m2Dpose/cfgs.yaml',
type=str)
parser.add_argument('--data_path',
help='path to pre-processed testing images',
default='./h36m2Dpose/cropped',
type=str)
args = parser.parse_args()
return args
def xywh2cs(x, y, w, h, aspect_ratio=0.75):
center = np.zeros((2), dtype=np.float32)
center[0] = x + w * 0.5
center[1] = y + h * 0.5
if w > aspect_ratio * h:
h = w * 1.0 / aspect_ratio
elif w < aspect_ratio * h:
w = h * aspect_ratio
scale = np.array([w * 1.0 / 200, h * 1.0 / 200], dtype=np.float32)
return center, scale
def gather_inputs(args, logger, image_size = (288, 384)):
root = args.data_path
img_names = os.listdir(root)
normalize = transforms.Normalize(
mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]
)
transform = transforms.Compose([
transforms.ToTensor(),
normalize,
])
inputs = []
# these testing images were cropped from videos of subject 9 and 11
for name in img_names:
pass
image_file = os.path.join(root, name)
data_numpy = cv2.imread(image_file, 1 | 128)
data_numpy = cv2.cvtColor(data_numpy, cv2.COLOR_BGR2RGB)
if data_numpy is None:
logger.error('=> fail to read {}'.format(image_file))
raise ValueError('Fail to read {}'.format(image_file))
c, s = xywh2cs(0, 0, data_numpy.shape[1], data_numpy.shape[0])
r = 0
trans = get_affine_transform(c, s, r, image_size)
input = cv2.warpAffine(
data_numpy,
trans,
(image_size[0], image_size[1]),
flags=cv2.INTER_LINEAR)
inputs.append(transform(input).unsqueeze(0))
return torch.cat(inputs)
def unnormalize(tensor):
img = tensor.data.cpu().numpy()
mean = np.array([0.485, 0.456, 0.406]).reshape(3, 1, 1)
std = np.array([0.229, 0.224, 0.225]).reshape(3, 1, 1)
img = np.transpose((((img * std) + mean) * 255).astype(np.uint8), (1, 2, 0))
return img
def visualize(inputs, model):
output = model(inputs)
pred, max_vals = get_max_preds_soft_pt(output)
pred = pred.data.cpu().numpy()
pred = pred[:, re_order, :]
plt.figure()
for i in range(len(pred)):
ax = plt.subplot(1, len(pred), i+1)
ax.imshow(unnormalize(inputs[i]))
ax.plot(pred[i][:, 0], pred[i][:, 1], 'ro')
show2Dpose(pred[i], ax)
return
def main():
args = parse_args()
update_config(cfg, args)
logging.basicConfig(format='%(asctime)-15s %(message)s')
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# cudnn related setting
cudnn.benchmark = cfg.CUDNN.BENCHMARK
torch.backends.cudnn.deterministic = cfg.CUDNN.DETERMINISTIC
torch.backends.cudnn.enabled = cfg.CUDNN.ENABLED
model = get_pose_net(cfg, is_train=False)
# load the pre-trained weights
if not os.path.exists('./h36m2Dpose/final_state.pth'):
logger.info('Please download the pre-trained model first.')
return
checkpoint = torch.load('./h36m2Dpose/final_state.pth')
model.load_state_dict(checkpoint)
dump_input = torch.rand(
(1, 3, cfg.MODEL.IMAGE_SIZE[1], cfg.MODEL.IMAGE_SIZE[0])
)
logger.info(get_model_summary(model, dump_input))
# modify the configuration file for multiple GPUs
model = torch.nn.DataParallel(model, device_ids=cfg.GPUS).cuda()
inputs = gather_inputs(args, logger)
visualize(inputs, model)
if __name__ == '__main__':
main()
================================================
FILE: examples/inference.py
================================================
"""
Am examplar script showing inference on the newly collected images in U3DPW.
"""
import sys
sys.path.append("../")
import libs.model.model as libm
from libs.dataset.h36m.data_utils import unNormalizeData
import torch
import numpy as np
import imageio
import matplotlib.pyplot as plt
num_joints = 16
gt_3d = False
pose_connection = [[0,1], [1,2], [2,3], [0,4], [4,5], [5,6], [0,7], [7,8],
[8,9], [9,10], [8,11], [11,12], [12,13], [8, 14], [14, 15], [15,16]]
# 16 out of 17 key-points are used as inputs in this examplar model
re_order_indices= [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16]
# paths
data_dic_path = './example_annot.npy'
model_path = './example_model.th'
stats = np.load('./stats.npy', allow_pickle=True).item()
dim_used_2d = stats['dim_use_2d']
mean_2d = stats['mean_2d']
std_2d = stats['std_2d']
# load the checkpoint and statistics
ckpt = torch.load(model_path)
data_dic = np.load(data_dic_path, allow_pickle=True).item()
# initialize the model
cascade = libm.get_cascade()
input_size = 32
output_size = 48
for stage_id in range(2):
# initialize a single deep learner
stage_model = libm.get_model(stage_id + 1,
refine_3d=False,
norm_twoD=False,
num_blocks=2,
input_size=input_size,
output_size=output_size,
linear_size=1024,
dropout=0.5,
leaky=False)
cascade.append(stage_model)
cascade.load_state_dict(ckpt)
cascade.eval()
# process and show total_to_show examples
count = 0
total_to_show = 10
def draw_skeleton(ax, skeleton, gt=False, add_index=True):
for segment_idx in range(len(pose_connection)):
point1_idx = pose_connection[segment_idx][0]
point2_idx = pose_connection[segment_idx][1]
point1 = skeleton[point1_idx]
point2 = skeleton[point2_idx]
color = 'k' if gt else 'r'
plt.plot([int(point1[0]),int(point2[0])],
[int(point1[1]),int(point2[1])],
c=color,
linewidth=2)
if add_index:
for (idx, re_order_idx) in enumerate(re_order_indices):
plt.text(skeleton[re_order_idx][0],
skeleton[re_order_idx][1],
str(idx+1),
color='b'
)
return
def normalize(skeleton, re_order=None):
norm_skel = skeleton.copy()
if re_order is not None:
norm_skel = norm_skel[re_order].reshape(32)
norm_skel = norm_skel.reshape(16, 2)
mean_x = np.mean(norm_skel[:,0])
std_x = np.std(norm_skel[:,0])
mean_y = np.mean(norm_skel[:,1])
std_y = np.std(norm_skel[:,1])
denominator = (0.5*(std_x + std_y))
norm_skel[:,0] = (norm_skel[:,0] - mean_x)/denominator
norm_skel[:,1] = (norm_skel[:,1] - mean_y)/denominator
norm_skel = norm_skel.reshape(32)
return norm_skel
def get_pred(cascade, data):
"""
Get prediction from a cascaded model
"""
# forward pass to get prediction for the first stage
num_stages = len(cascade)
# for legacy code that does not have the num_blocks attribute
for i in range(len(cascade)):
cascade[i].num_blocks = len(cascade[i].res_blocks)
prediction = cascade[0](data)
# prediction for later stages
for stage_idx in range(1, num_stages):
prediction += cascade[stage_idx](data)
return prediction
def show3Dpose(channels,
ax,
lcolor="#3498db",
rcolor="#e74c3c",
add_labels=True,
gt=False,
pred=False
):
vals = np.reshape( channels, (32, -1) )
I = np.array([1,2,3,1,7,8,1, 13,14,15,14,18,19,14,26,27])-1 # start points
J = np.array([2,3,4,7,8,9,13,14,15,16,18,19,20,26,27,28])-1 # end points
LR = np.array([1,1,1,0,0,0,0, 0, 0, 0, 0, 0, 0, 1, 1, 1], dtype=bool)
# Make connection matrix
for i in np.arange( len(I) ):
x, y, z = [np.array( [vals[I[i], j], vals[J[i], j]] ) for j in range(3)]
if gt or pred:
color = 'k' if gt else 'r'
ax.plot(x,y, z, lw=2, c=color)
else:
ax.plot(x,y, z, lw=2, c=lcolor if LR[i] else rcolor)
RADIUS = 750 # space around the subject
xroot, yroot, zroot = vals[0,0], vals[0,1], vals[0,2]
ax.set_xlim3d([-RADIUS+xroot, RADIUS+xroot])
ax.set_zlim3d([-RADIUS+zroot, RADIUS+zroot])
ax.set_ylim3d([-RADIUS+yroot, RADIUS+yroot])
if add_labels:
ax.set_xlabel("x")
ax.set_ylabel("z")
ax.set_zlabel("y")
ax.set_aspect('equal')
# Get rid of the panes (actually, make them white)
white = (1.0, 1.0, 1.0, 0.0)
ax.w_xaxis.set_pane_color(white)
ax.w_yaxis.set_pane_color(white)
# Get rid of the lines in 3d
ax.w_xaxis.line.set_color(white)
ax.w_yaxis.line.set_color(white)
ax.w_zaxis.line.set_color(white)
ax.invert_zaxis()
return
def re_order(skeleton):
skeleton = skeleton.copy().reshape(-1,3)
# permute the order of x,y,z axis
skeleton[:,[0,1,2]] = skeleton[:, [0,2,1]]
return skeleton.reshape(96)
def plot_3d_ax(ax,
elev,
azim,
pred,
title=None
):
ax.view_init(elev=elev, azim=azim)
show3Dpose(re_order(pred), ax)
plt.title(title)
return
def adjust_figure(left = 0,
right = 1,
bottom = 0.01,
top = 0.95,
wspace = 0,
hspace = 0.4
):
plt.subplots_adjust(left, bottom, right, top, wspace, hspace)
return
for image_name in data_dic.keys():
image_path = './imgs/' + image_name
img = imageio.imread(image_path)
f = plt.figure(figsize=(9, 3))
ax1 = plt.subplot(131)
ax1.imshow(img)
plt.title('Input image')
ax2 = plt.subplot(132)
plt.title('2D key-point inputs: {:d}*2'.format(num_joints))
ax2.set_aspect('equal')
ax2.invert_yaxis()
skeleton_pred = None
skeleton_2d = data_dic[image_name]['p2d']
# The order for the 2D keypoints is:
# 'Hip', 'RHip', 'RKnee', 'RFoot', 'LHip', 'LKnee', 'LFoot', 'Spine',
# 'Thorax', 'Neck/Nose', 'Head', 'LShoulder', 'LElbow', 'LWrist', 'RShoulder'
# 'RElbow', 'RWrist'
draw_skeleton(ax2, skeleton_2d, gt=True)
plt.plot(skeleton_2d[:,0], skeleton_2d[:,1], 'ro', 2)
# Nose was not used for this examplar model
norm_ske_gt = normalize(skeleton_2d, re_order_indices).reshape(1,-1)
pred = get_pred(cascade, torch.from_numpy(norm_ske_gt.astype(np.float32)))
pred = unNormalizeData(pred.data.numpy(),
stats['mean_3d'],
stats['std_3d'],
stats['dim_ignore_3d']
)
ax3 = plt.subplot(133, projection='3d')
plot_3d_ax(ax=ax3,
pred=pred,
elev=10.,
azim=-90,
title='3D prediction'
)
adjust_figure(left = 0.05,
right = 0.95,
bottom = 0.08,
top = 0.92,
wspace = 0.3,
hspace = 0.3
)
count += 1
if count >= total_to_show:
break
================================================
FILE: libs/__init__.py
================================================
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Empty file.
"""
================================================
FILE: libs/annotator/__init__.py
================================================
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Empty file.
"""
#import libs.dataset.h36m
================================================
FILE: libs/annotator/angle.py
================================================
import numpy as np
import scipy.io as sio
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
bone_name = {
1: 'thorax to head top',
2: 'left shoulder to left elbow',
3: 'left elbow to left wrist',
4: 'right shoulder to right elbow',
5: 'right elbow to right wrist',
6: 'left hip to left knee',
7: 'left knee to left ankle',
8: 'right hip to right knee',
9: 'right knee to right ankle'
}
static_pose_path = "/media/nicholas/Database/Github/EvoSkeleton/resources/constraints/staticPose.npy"
static_pose = np.load(static_pose_path, allow_pickle=True).item()
di = static_pose['di']
a = static_pose['a'].reshape(3)
di_indices = {2:5, 4:2, 6:13, 8:9}
nt_parent_indices = [13, 17, 18, 25, 26, 6, 7, 1, 2]
nt_child_indices = [15, 18, 19, 26, 27, 7, 8, 2, 3]
def gram_schmidt_columns(X):
B = np.zeros(X.shape)
B[:, 0] = (1/np.linalg.norm(X[:, 0]))*X[:, 0]
for i in range(1, 3):
v = X[:, i]
U = B[:, 0:i] # subspace basis which has already been orthonormalized
pc = U.T @ v # orthogonal projection coefficients of v onto U
p = U@pc
v = v - p
if np.linalg.norm(v) < 2e-16:
# vectors are not linearly independent!
raise ValueError
else:
v = normalize(v)
B[:, i] = v
return B
def normalize(vector):
return vector/np.linalg.norm(vector)
def get_basis1(skeleton):
"""
get local coordinate system
"""
# compute the vector from the left shoulder to the right shoulder
left_shoulder = skeleton[17]
right_shoulder = skeleton[25]
v1 = normalize(right_shoulder - left_shoulder)
# compute the backbone vector from the thorax to the spine
thorax = skeleton[13]
spine = skeleton[12]
v2 = normalize(spine - thorax)
# v3 is the cross product of v1 and v2 (front-facing vector for upper-body)
v3 = normalize(np.cross(v1, v2))
return v1, v2, v3
def get_normal(x1, a, x):
nth = 1e-4
# x and a are parallel
if np.linalg.norm(x - a) < nth or np.linalg.norm(x + a) < nth:
n = np.cross(x, x1)
flag = True
else:
n = np.cross(a, x)
flag = False
return normalize(n), flag
def to_spherical(xyz):
"""
convert from Cartisian coordinate to spherical coordinate
theta: [-pi, pi]
phi: [-pi/2, pi/2]
note that xyz should be float number
"""
# return in r, phi, and theta (elevation angle from z axis down)
return_value = np.zeros(xyz.shape, dtype=xyz.dtype)
xy = xyz[:,0]**2 + xyz[:,1]**2
return_value[:,0] = np.sqrt(xy + xyz[:,2]**2) # r
return_value[:,1] = np.arctan2(xyz[:,1], xyz[:,0]) # theta
return_value[:,2] = np.arctan2(xyz[:,2], np.sqrt(xy)) # phi
return return_value
def to_xyz(rthetaphi):
"""
convert from spherical coordinate to Cartisian coordinate
theta: [0, 2*pi] or [-pi, pi]
phi: [-pi/2, pi/2]
"""
return_value = np.zeros(rthetaphi.shape, dtype=rthetaphi.dtype)
sintheta = np.sin(rthetaphi[:,1])
costheta = np.cos(rthetaphi[:,1])
sinphi = np.sin(rthetaphi[:,2])
cosphi = np.cos(rthetaphi[:,2])
return_value[:,0] = rthetaphi[:,0]*costheta*cosphi # x
return_value[:,1] = rthetaphi[:,0]*sintheta*cosphi # y
return_value[:,2] = rthetaphi[:,0]*sinphi #z
return return_value
================================================
FILE: libs/annotator/fit_3d.py
================================================
"""
Obtain 3D skeleton with 2D key-points as inputs using SMPLify
Please run this script in Python 2 environment for now.
TODO: transfer this tool to Python 3.
"""
from smpl_webuser.serialization import load_model
from smplify.fit_3d import run_single_fit
# you can use multi-processing to fit the images in parallel
#from multiprocessing import Pool
import matplotlib.pyplot as plt
import numpy as np
import argparse
import cv2
import os
# whether to use the regularization terms
sph_regs = None
# number of blend shapes to use
n_betas = 1
# focal length of the camera
flength = 5000
pix_thsh = 25
scale_factor = 1
# viewpoints for rendering
do_degrees = [0.]
def main(opt):
model = load_model(opt.model_dir)
annotation_path = os.path.join(opt.dataset_dir, 'annotation.npy')
assert os.path.exists(annotation_path), "Please prepare the 2D annotation first."
annotation = np.load(annotation_path, allow_pickle=True).item()
if opt.save_image and not os.path.exists(os.path.join(opt.dataset_dir, 'fitted')):
os.makedirs(os.path.join(opt.dataset_dir, 'fitted'))
for (image_name, annots) in annotation.items():
assert 'p2d' in annots, "The image must be annotated with 2D key-points"
joints_2d = annots['p2d']
# use 14 key-points for model fitting
# one may adjust this number by considering different 2D-3D corespondence
if joints_2d.shape[0] > 14:
joints_2d = joints_2d[:14, :]
# Prepare fitting parameters
filling_list = range(12) + [13]
conf = np.zeros(joints_2d.shape[0])
conf[filling_list] = 1.0
img = cv2.imread(os.path.join(opt.dataset_dir, image_name))
# Run single fit
params, vis = run_single_fit(
img,
joints_2d,
conf,
model,
regs=sph_regs,
n_betas=n_betas,
flength=flength,
pix_thsh=pix_thsh,
scale_factor=scale_factor,
viz=opt.viz,
do_degrees=do_degrees
)
# Show result then close after 1 second
f = plt.figure(figsize=(6, 3))
plt.subplot(121)
plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
for di, deg in enumerate(do_degrees):
plt.subplot(122)
plt.cla()
plt.imshow(vis[di])
plt.draw()
plt.pause(1)
# record the params
annotation[image_name]['fitting_params'] = params
# save fitted image
if opt.save_image:
image_name = image_name.replace(".jpg", ".png")
img_save_path = os.path.join(opt.dataset_dir, 'fitted', image_name)
f.savefig(img_save_path)
plt.close(f)
print("fitted image saved at ", img_save_path)
# save the annotation dictionary with fitted parameters
np.save(os.path.join(opt.dataset_dir, "fitted.npy"), annotation)
print('3D prameters saved at ' + os.path.join(opt.dataset_dir, "fitted.npy"))
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='2D Annotation')
parser.add_argument('-dataset_dir', type=str)
parser.add_argument('-model_dir', type=str)
parser.add_argument('-save_image',default=True, type=bool)
# visualize intermeadiate results
parser.add_argument('-viz',default=False, type=bool)
opt = parser.parse_args()
main(opt)
================================================
FILE: libs/annotator/smpl-spec-list.txt
================================================
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
@EXPLICIT
https://repo.anaconda.com/pkgs/main/linux-64/_libgcc_mutex-0.1-main.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/blas-1.0-mkl.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ca-certificates-2020.6.24-0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/intel-openmp-2020.1-217.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libgfortran-ng-7.3.0-hdf63c60_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libstdcxx-ng-9.1.0-hdf63c60_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pandoc-2.9.2.1-0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libgcc-ng-9.1.0-hdf63c60_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/mkl-2020.1-217.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/expat-2.2.9-he6710b0_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/icu-58.2-he6710b0_3.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/jpeg-9b-h024ee3a_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libffi-3.3-he6710b0_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libsodium-1.0.18-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libspatialindex-1.9.3-he6710b0_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libuuid-1.0.3-h1bed415_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libxcb-1.14-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ncurses-6.2-he6710b0_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/openssl-1.1.1g-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pcre-8.44-he6710b0_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pixman-0.40.0-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/xz-5.2.5-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/yaml-0.1.7-had09818_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/zlib-1.2.11-h7b6447c_3.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/glib-2.65.0-h3eb4bd4_0.tar.bz2
https://repo.anaconda.com/pkgs/free/linux-64/hdf5-1.8.17-2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libedit-3.1.20191231-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libpng-1.6.37-hbc83047_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libxml2-2.9.10-he19cac6_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/readline-8.0-h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/tk-8.6.10-hbc83047_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/zeromq-4.3.2-he6710b0_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/zstd-1.3.7-h0b5b093_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/dbus-1.13.16-hb2f20db_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/freetype-2.10.2-h5ab3b9f_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/gstreamer-1.14.0-hb31296c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libtiff-4.0.10-h2733197_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/sqlite-3.32.3-h62c20be_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/fontconfig-2.13.0-h9420a91_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/gst-plugins-base-1.14.0-hbbd80ab_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/python-2.7.18-h15b4118_1.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/alabaster-0.7.12-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/argh-0.26.2-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/asn1crypto-1.3.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/atomicwrites-1.4.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/attrs-19.3.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/backports-1.0-py_2.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/backports_abc-0.5-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/cairo-1.14.12-h8948797_3.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/certifi-2019.11.28-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/chardet-3.0.4-py27_1003.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/cloudpickle-1.2.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/contextlib2-0.6.0.post1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/decorator-4.4.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/defusedxml-0.6.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/diff-match-patch-20181111-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/docutils-0.15.2-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/enum34-1.1.6-py27_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/functools32-3.2.3.2-py27_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/future-0.18.2-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/futures-3.3.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/idna-2.10-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/imagesize-1.2.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/ipaddress-1.0.23-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ipython_genutils-0.2.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/lazy-object-proxy-1.4.3-py27h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/markupsafe-1.1.1-py27h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/mccabe-0.6.1-py27_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/mistune-0.8.4-py27h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pandocfilters-1.4.2-py27_1.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/parso-0.5.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/pathtools-0.1.2-py_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/psutil-5.6.7-py27h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ptyprocess-0.6.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pycodestyle-2.5.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/pycparser-2.20-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pyflakes-2.1.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/pyparsing-2.4.7-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pysocks-1.7.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/pytz-2020.1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/pyxdg-0.26-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pyyaml-5.2-py27h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pyzmq-18.1.0-py27he6710b0_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/qdarkstyle-2.8.1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/qt-5.9.7-h5867ecd_1.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/qtpy-1.9.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/rope-0.17.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/rtree-0.8.3-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/scandir-1.10.0-py27h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/selectors2-2.0.1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/simplegeneric-0.8.1-py27_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/sip-4.19.8-py27hf484d3e_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/six-1.15.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/snowballstemmer-2.0.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/sortedcontainers-2.2.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/sphinxcontrib-1.0-py27_1.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/testpath-0.4.4-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/typing-3.7.4.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ujson-1.35-py27h14c3975_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/wcwidth-0.2.5-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/webencodings-0.5.1-py27_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/wrapt-1.11.2-py27h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/yapf-0.30.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/autopep8-1.4.4-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/babel-2.8.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/backports.shutil_get_terminal_size-1.0.0-py27_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/cffi-1.14.0-py27he30daa8_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/configparser-4.0.2-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/free/linux-64/harfbuzz-0.9.39-1.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/intervaltree-3.0.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/jedi-0.14.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/mkl-service-2.3.0-py27he904b0f_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/more-itertools-5.0.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/packaging-20.4-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pathlib2-2.3.5-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pexpect-4.7.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pyqt-5.9.2-py27h05f1152_2.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pyrsistent-0.15.6-py27h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/python-dateutil-2.8.1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/python-jsonrpc-server-0.3.4-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/qtawesome-0.7.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/setuptools-44.0.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/singledispatch-3.4.0.3-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/sphinxcontrib-websupport-1.2.3-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/traitlets-4.3.3-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/watchdog-0.9.0-py27_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/wurlitzer-2.0.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/backports.functools_lru_cache-1.6.1-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/bleach-3.1.5-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/cryptography-2.8-py27h1ba5d50_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/entrypoints-0.3-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/jinja2-2.11.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/jupyter_core-4.6.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/numpy-base-1.16.6-py27hde5b4d6_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pickleshare-0.7.5-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pydocstyle-3.0.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/pygments-2.5.2-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/tornado-5.1.1-py27h7b6447c_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/wheel-0.33.6-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/zipp-0.6.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/astroid-1.6.5-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/flake8-3.7.9-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/importlib_metadata-1.3.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/isort-4.3.21-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/jupyter_client-5.3.4-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pip-19.3.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/prompt_toolkit-1.0.15-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pyopenssl-19.1.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ipython-5.8.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/jsonschema-3.2.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pluggy-0.13.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/pylint-1.9.2-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/urllib3-1.25.7-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ipykernel-4.10.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/nbformat-4.4.0-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/python-language-server-0.31.2-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/requests-2.24.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/nbconvert-5.6.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/qtconsole-4.7.5-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/sphinx-1.8.5-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/spyder-kernels-1.8.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/numpydoc-1.0.0-py_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/spyder-4.0.1-py27_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/mkl_fft-1.0.15-py27ha843d7b_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/mkl_random-1.1.0-py27hd6b4f25_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/numpy-1.16.6-py27hbc911f0_0.tar.bz2
https://conda.anaconda.org/menpo/linux-64/opencv3-3.1.0-py27_0.tar.bz2
================================================
FILE: libs/annotator/smpl_webuser/LICENSE.txt
================================================
Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the SMPL body model and software, (the "Model"), including 3D meshes, blend weights blend shapes, textures, software, scripts, and animations. By downloading and/or using the Model, you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Model.
Ownership
The Model has been developed at the Max Planck Institute for Intelligent Systems (hereinafter "MPI") and is owned by and proprietary material of the Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (hereinafter “MPG”; MPI and MPG hereinafter collectively “Max-Planck”).
License Grant
Max-Planck grants you a non-exclusive, non-transferable, free of charge right:
To download the Model and use it on computers owned, leased or otherwise controlled by you and/or your organisation;
To use the Model for the sole purpose of performing non-commercial scientific research, non-commercial education, or non-commercial artistic projects.
Any other use, in particular any use for commercial purposes, is prohibited. This includes, without limitation, incorporation in a commercial product, use in a commercial service, as training data for a commercial product, for commercial ergonomic analysis (e.g. product design, architectural design, etc.), or production of other artifacts for commercial purposes including, for example, web services, movies, television programs, mobile applications, or video games. The Model may not be used for pornographic purposes or to generate pornographic material whether commercial or not. This license also prohibits the use of the Model to train methods/algorithms/neural networks/etc. for commercial use of any kind. The Model may not be reproduced, modified and/or made available in any form to any third party without Max-Planck’s prior written permission. By downloading the Model, you agree not to reverse engineer it.
Disclaimer of Representations and Warranties
You expressly acknowledge and agree that the Model results from basic research, is provided “AS IS”, may contain errors, and that any use of the Model is at your sole risk. MAX-PLANCK MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE MODEL, NEITHER EXPRESS NOR IMPLIED, AND THE ABSENCE OF ANY LEGAL OR ACTUAL DEFECTS, WHETHER DISCOVERABLE OR NOT. Specifically, and not to limit the foregoing, Max-Planck makes no representations or warranties (i) regarding the merchantability or fitness for a particular purpose of the Model, (ii) that the use of the Model will not infringe any patents, copyrights or other intellectual property rights of a third party, and (iii) that the use of the Model will not cause any damage of any kind to you or a third party.
Limitation of Liability
Under no circumstances shall Max-Planck be liable for any incidental, special, indirect or consequential damages arising out of or relating to this license, including but not limited to, any lost profits, business interruption, loss of programs or other data, or all other commercial damages or losses, even if advised of the possibility thereof.
No Maintenance Services
You understand and agree that Max-Planck is under no obligation to provide either maintenance services, update services, notices of latent defects, or corrections of defects with regard to the Model. Max-Planck nevertheless reserves the right to update, modify, or discontinue the Model at any time.
Publication with SMPL
You agree to cite the most recent paper describing the model as specified on the download website. This website lists the most up to date bibliographic information on the about page.
Media projects with SMPL
When using SMPL in a media project please give credit to Max Planck Institute for Intelligent Systems. For example: SMPL was used for character animation courtesy of the Max Planck Institute for Intelligent Systems.
Commercial licensing opportunities
For commercial use in the fields of medicine, psychology, and biomechanics, please contact ps-license@tue.mpg.de.
For commercial use in all other fields please contact Body Labs Inc at smpl@bodylabs.com
================================================
FILE: libs/annotator/smpl_webuser/README.txt
================================================
License:
========
To learn about SMPL, please visit our website: http://smpl.is.tue.mpg
You can find the SMPL paper at: http://files.is.tue.mpg.de/black/papers/SMPL2015.pdf
Visit our downloads page to download some sample animation files (FBX), and python code:
http://smpl.is.tue.mpg/downloads
For comments or questions, please email us at: smpl@tuebingen.mpg.de
System Requirements:
====================
Operating system: OSX, Linux
Python Dependencies:
- Numpy & Scipy [http://www.scipy.org/scipylib/download.html]
- Chumpy [https://github.com/mattloper/chumpy]
- OpenCV [http://opencv.org/downloads.html]
Getting Started:
================
1. Extract the Code:
--------------------
Extract the 'smpl.zip' file to your home directory (or any other location you wish)
2. Set the PYTHONPATH:
----------------------
We need to update the PYTHONPATH environment variable so that the system knows how to find the SMPL code. Add the following lines to your ~/.bash_profile file (create it if it doesn't exist; Linux users might have ~/.bashrc file instead), replacing ~/smpl with the location where you extracted the smpl.zip file:
SMPL_LOCATION=~/smpl
export PYTHONPATH=$PYTHONPATH:$SMPL_LOCATION
Open a new terminal window to check if the python path has been updated by typing the following:
> echo $PYTHONPATH
3. Run the Hello World scripts:
-------------------------------
In the new Terminal window, navigate to the smpl/smpl_webuser/hello_world directory. You can run the hello world scripts now by typing the following:
> python hello_smpl.py
OR
> python render_smpl.py
Note:
Both of these scripts will require the dependencies listed above. The scripts are provided as a sample to help you get started.
================================================
FILE: libs/annotator/smpl_webuser/__init__.py
================================================
'''
Copyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPL Model license here http://smpl.is.tue.mpg.de/license
More information about SMPL is available here http://smpl.is.tue.mpg.
For comments or questions, please email us at: smpl@tuebingen.mpg.de
About this file:
================
This is an initialization file to help python look for submodules in this directory.
'''
================================================
FILE: libs/annotator/smpl_webuser/hello_world/hello_smpl.py
================================================
'''
Copyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPL Model license here http://smpl.is.tue.mpg.de/license
More information about SMPL is available here http://smpl.is.tue.mpg.
For comments or questions, please email us at: smpl@tuebingen.mpg.de
Please Note:
============
This is a demo version of the script for driving the SMPL model with python.
We would be happy to receive comments, help and suggestions on improving this code
and in making it available on more platforms.
System Requirements:
====================
Operating system: OSX, Linux
Python Dependencies:
- Numpy & Scipy [http://www.scipy.org/scipylib/download.html]
- Chumpy [https://github.com/mattloper/chumpy]
About the Script:
=================
This script demonstrates a few basic functions to help users get started with using
the SMPL model. The code shows how to:
- Load the SMPL model
- Edit pose & shape parameters of the model to create a new body in a new pose
- Save the resulting body as a mesh in .OBJ format
Running the Hello World code:
=============================
Inside Terminal, navigate to the smpl/webuser/hello_world directory. You can run
the hello world script now by typing the following:
> python hello_smpl.py
'''
from smpl_webuser.serialization import load_model
import numpy as np
## Load SMPL model (here we load the female model)
## Make sure path is correct
m = load_model( '../../models/basicModel_f_lbs_10_207_0_v1.0.0.pkl' )
## Assign random pose and shape parameters
m.pose[:] = np.random.rand(m.pose.size) * .2
m.betas[:] = np.random.rand(m.betas.size) * .03
## Write to an .obj file
outmesh_path = './hello_smpl.obj'
with open( outmesh_path, 'w') as fp:
for v in m.r:
fp.write( 'v %f %f %f\n' % ( v[0], v[1], v[2]) )
for f in m.f+1: # Faces are 1-based, not 0-based in obj files
fp.write( 'f %d %d %d\n' % (f[0], f[1], f[2]) )
## Print message
print '..Output mesh saved to: ', outmesh_path
================================================
FILE: libs/annotator/smpl_webuser/hello_world/render_smpl.py
================================================
'''
Copyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPL Model license here http://smpl.is.tue.mpg.de/license
More information about SMPL is available here http://smpl.is.tue.mpg.
For comments or questions, please email us at: smpl@tuebingen.mpg.de
Please Note:
============
This is a demo version of the script for driving the SMPL model with python.
We would be happy to receive comments, help and suggestions on improving this code
and in making it available on more platforms.
System Requirements:
====================
Operating system: OSX, Linux
Python Dependencies:
- Numpy & Scipy [http://www.scipy.org/scipylib/download.html]
- Chumpy [https://github.com/mattloper/chumpy]
- OpenCV [http://opencv.org/downloads.html]
--> (alternatively: matplotlib [http://matplotlib.org/downloads.html])
About the Script:
=================
This script demonstrates loading the smpl model and rendering it using OpenDR
to render and OpenCV to display (or alternatively matplotlib can also be used
for display, as shown in commented code below).
This code shows how to:
- Load the SMPL model
- Edit pose & shape parameters of the model to create a new body in a new pose
- Create an OpenDR scene (with a basic renderer, camera & light)
- Render the scene using OpenCV / matplotlib
Running the Hello World code:
=============================
Inside Terminal, navigate to the smpl/webuser/hello_world directory. You can run
the hello world script now by typing the following:
> python render_smpl.py
'''
import numpy as np
from opendr.renderer import ColoredRenderer
from opendr.lighting import LambertianPointLight
from opendr.camera import ProjectPoints
from smpl_webuser.serialization import load_model
## Load SMPL model (here we load the female model)
m = load_model('../../models/basicModel_f_lbs_10_207_0_v1.0.0.pkl')
## Assign random pose and shape parameters
m.pose[:] = np.random.rand(m.pose.size) * .2
m.betas[:] = np.random.rand(m.betas.size) * .03
m.pose[0] = np.pi
## Create OpenDR renderer
rn = ColoredRenderer()
## Assign attributes to renderer
w, h = (640, 480)
rn.camera = ProjectPoints(v=m, rt=np.zeros(3), t=np.array([0, 0, 2.]), f=np.array([w,w])/2., c=np.array([w,h])/2., k=np.zeros(5))
rn.frustum = {'near': 1., 'far': 10., 'width': w, 'height': h}
rn.set(v=m, f=m.f, bgcolor=np.zeros(3))
## Construct point light source
rn.vc = LambertianPointLight(
f=m.f,
v=rn.v,
num_verts=len(m),
light_pos=np.array([-1000,-1000,-2000]),
vc=np.ones_like(m)*.9,
light_color=np.array([1., 1., 1.]))
## Show it using OpenCV
import cv2
cv2.imshow('render_SMPL', rn.r)
print ('..Print any key while on the display window')
cv2.waitKey(0)
cv2.destroyAllWindows()
## Could also use matplotlib to display
# import matplotlib.pyplot as plt
# plt.ion()
# plt.imshow(rn.r)
# plt.show()
# import pdb; pdb.set_trace()
================================================
FILE: libs/annotator/smpl_webuser/lbs.py
================================================
'''
Copyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPL Model license here http://smpl.is.tue.mpg.de/license
More information about SMPL is available here http://smpl.is.tue.mpg.
For comments or questions, please email us at: smpl@tuebingen.mpg.de
About this file:
================
This file defines linear blend skinning for the SMPL loader which
defines the effect of bones and blendshapes on the vertices of the template mesh.
Modules included:
- global_rigid_transformation:
computes global rotation & translation of the model
- verts_core: [overloaded function inherited from verts.verts_core]
computes the blending of joint-influences for each vertex based on type of skinning
'''
from smpl_webuser.posemapper import posemap
import chumpy
import numpy as np
def global_rigid_transformation(pose, J, kintree_table, xp):
results = {}
pose = pose.reshape((-1,3))
id_to_col = {kintree_table[1,i] : i for i in range(kintree_table.shape[1])}
parent = {i : id_to_col[kintree_table[0,i]] for i in range(1, kintree_table.shape[1])}
if xp == chumpy:
from smpl_webuser.posemapper import Rodrigues
rodrigues = lambda x : Rodrigues(x)
else:
import cv2
rodrigues = lambda x : cv2.Rodrigues(x)[0]
with_zeros = lambda x : xp.vstack((x, xp.array([[0.0, 0.0, 0.0, 1.0]])))
results[0] = with_zeros(xp.hstack((rodrigues(pose[0,:]), J[0,:].reshape((3,1)))))
for i in range(1, kintree_table.shape[1]):
results[i] = results[parent[i]].dot(with_zeros(xp.hstack((
rodrigues(pose[i,:]),
((J[i,:] - J[parent[i],:]).reshape((3,1)))
))))
pack = lambda x : xp.hstack([np.zeros((4, 3)), x.reshape((4,1))])
results = [results[i] for i in sorted(results.keys())]
results_global = results
if True:
results2 = [results[i] - (pack(
results[i].dot(xp.concatenate( ( (J[i,:]), 0 ) )))
) for i in range(len(results))]
results = results2
result = xp.dstack(results)
return result, results_global
def verts_core(pose, v, J, weights, kintree_table, want_Jtr=False, xp=chumpy):
A, A_global = global_rigid_transformation(pose, J, kintree_table, xp)
T = A.dot(weights.T)
rest_shape_h = xp.vstack((v.T, np.ones((1, v.shape[0]))))
v =(T[:,0,:] * rest_shape_h[0, :].reshape((1, -1)) +
T[:,1,:] * rest_shape_h[1, :].reshape((1, -1)) +
T[:,2,:] * rest_shape_h[2, :].reshape((1, -1)) +
T[:,3,:] * rest_shape_h[3, :].reshape((1, -1))).T
v = v[:,:3]
if not want_Jtr:
return v
Jtr = xp.vstack([g[:3,3] for g in A_global])
return (v, Jtr)
================================================
FILE: libs/annotator/smpl_webuser/posemapper.py
================================================
'''
Copyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPL Model license here http://smpl.is.tue.mpg.de/license
More information about SMPL is available here http://smpl.is.tue.mpg.
For comments or questions, please email us at: smpl@tuebingen.mpg.de
About this file:
================
This module defines the mapping of joint-angles to pose-blendshapes.
Modules included:
- posemap:
computes the joint-to-pose blend shape mapping given a mapping type as input
'''
import chumpy as ch
import numpy as np
import cv2
class Rodrigues(ch.Ch):
dterms = 'rt'
def compute_r(self):
return cv2.Rodrigues(self.rt.r)[0]
def compute_dr_wrt(self, wrt):
if wrt is self.rt:
return cv2.Rodrigues(self.rt.r)[1].T
def lrotmin(p):
if isinstance(p, np.ndarray):
p = p.ravel()[3:]
return np.concatenate([(cv2.Rodrigues(np.array(pp))[0]-np.eye(3)).ravel() for pp in p.reshape((-1,3))]).ravel()
if p.ndim != 2 or p.shape[1] != 3:
p = p.reshape((-1,3))
p = p[1:]
return ch.concatenate([(Rodrigues(pp)-ch.eye(3)).ravel() for pp in p]).ravel()
def posemap(s):
if s == 'lrotmin':
return lrotmin
else:
raise Exception('Unknown posemapping: %s' % (str(s),))
================================================
FILE: libs/annotator/smpl_webuser/serialization.py
================================================
'''
Copyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPL Model license here http://smpl.is.tue.mpg.de/license
More information about SMPL is available here http://smpl.is.tue.mpg.
For comments or questions, please email us at: smpl@tuebingen.mpg.de
About this file:
================
This file defines the serialization functions of the SMPL model.
Modules included:
- save_model:
saves the SMPL model to a given file location as a .pkl file
- load_model:
loads the SMPL model from a given file location (i.e. a .pkl file location),
or a dictionary object.
'''
__all__ = ['load_model', 'save_model']
import numpy as np
import pickle
import chumpy as ch
from chumpy.ch import MatVecMult
from smpl_webuser.posemapper import posemap
from smpl_webuser.verts import verts_core
def save_model(model, fname):
m0 = model
trainer_dict = {'v_template': np.asarray(m0.v_template),'J': np.asarray(m0.J),'weights': np.asarray(m0.weights),'kintree_table': m0.kintree_table,'f': m0.f, 'bs_type': m0.bs_type, 'posedirs': np.asarray(m0.posedirs)}
if hasattr(model, 'J_regressor'):
trainer_dict['J_regressor'] = m0.J_regressor
if hasattr(model, 'J_regressor_prior'):
trainer_dict['J_regressor_prior'] = m0.J_regressor_prior
if hasattr(model, 'weights_prior'):
trainer_dict['weights_prior'] = m0.weights_prior
if hasattr(model, 'shapedirs'):
trainer_dict['shapedirs'] = m0.shapedirs
if hasattr(model, 'vert_sym_idxs'):
trainer_dict['vert_sym_idxs'] = m0.vert_sym_idxs
if hasattr(model, 'bs_style'):
trainer_dict['bs_style'] = model.bs_style
else:
trainer_dict['bs_style'] = 'lbs'
pickle.dump(trainer_dict, open(fname, 'w'), -1)
def backwards_compatibility_replacements(dd):
# replacements
if 'default_v' in dd:
dd['v_template'] = dd['default_v']
del dd['default_v']
if 'template_v' in dd:
dd['v_template'] = dd['template_v']
del dd['template_v']
if 'joint_regressor' in dd:
dd['J_regressor'] = dd['joint_regressor']
del dd['joint_regressor']
if 'blendshapes' in dd:
dd['posedirs'] = dd['blendshapes']
del dd['blendshapes']
if 'J' not in dd:
dd['J'] = dd['joints']
del dd['joints']
# defaults
if 'bs_style' not in dd:
dd['bs_style'] = 'lbs'
def ready_arguments(fname_or_dict):
if not isinstance(fname_or_dict, dict):
dd = pickle.load(open(fname_or_dict, 'rb'))
# dd = pickle.load(open(fname_or_dict, 'rb'), encoding='latin1')
else:
dd = fname_or_dict
backwards_compatibility_replacements(dd)
want_shapemodel = 'shapedirs' in dd
nposeparms = dd['kintree_table'].shape[1]*3
if 'trans' not in dd:
dd['trans'] = np.zeros(3)
if 'pose' not in dd:
dd['pose'] = np.zeros(nposeparms)
if 'shapedirs' in dd and 'betas' not in dd:
dd['betas'] = np.zeros(dd['shapedirs'].shape[-1])
for s in ['v_template', 'weights', 'posedirs', 'pose', 'trans', 'shapedirs', 'betas', 'J']:
if (s in dd) and not hasattr(dd[s], 'dterms'):
dd[s] = ch.array(dd[s])
if want_shapemodel:
dd['v_shaped'] = dd['shapedirs'].dot(dd['betas'])+dd['v_template']
v_shaped = dd['v_shaped']
J_tmpx = MatVecMult(dd['J_regressor'], v_shaped[:,0])
J_tmpy = MatVecMult(dd['J_regressor'], v_shaped[:,1])
J_tmpz = MatVecMult(dd['J_regressor'], v_shaped[:,2])
dd['J'] = ch.vstack((J_tmpx, J_tmpy, J_tmpz)).T
dd['v_posed'] = v_shaped + dd['posedirs'].dot(posemap(dd['bs_type'])(dd['pose']))
else:
dd['v_posed'] = dd['v_template'] + dd['posedirs'].dot(posemap(dd['bs_type'])(dd['pose']))
return dd
def load_model(fname_or_dict):
dd = ready_arguments(fname_or_dict)
args = {
'pose': dd['pose'],
'v': dd['v_posed'],
'J': dd['J'],
'weights': dd['weights'],
'kintree_table': dd['kintree_table'],
'xp': ch,
'want_Jtr': True,
'bs_style': dd['bs_style']
}
result, Jtr = verts_core(**args)
result = result + dd['trans'].reshape((1,3))
result.J_transformed = Jtr + dd['trans'].reshape((1,3))
for k, v in dd.items():
setattr(result, k, v)
return result
================================================
FILE: libs/annotator/smpl_webuser/verts.py
================================================
'''
Copyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPL Model license here http://smpl.is.tue.mpg.de/license
More information about SMPL is available here http://smpl.is.tue.mpg.
For comments or questions, please email us at: smpl@tuebingen.mpg.de
About this file:
================
This file defines the basic skinning modules for the SMPL loader which
defines the effect of bones and blendshapes on the vertices of the template mesh.
Modules included:
- verts_decorated:
creates an instance of the SMPL model which inherits model attributes from another
SMPL model.
- verts_core: [overloaded function inherited by lbs.verts_core]
computes the blending of joint-influences for each vertex based on type of skinning
'''
import chumpy
import smpl_webuser.lbs as lbs
from smpl_webuser.posemapper import posemap
import scipy.sparse as sp
from chumpy.ch import MatVecMult
def ischumpy(x): return hasattr(x, 'dterms')
def verts_decorated(trans, pose,
v_template, J, weights, kintree_table, bs_style, f,
bs_type=None, posedirs=None, betas=None, shapedirs=None, want_Jtr=False):
for which in [trans, pose, v_template, weights, posedirs, betas, shapedirs]:
if which is not None:
assert ischumpy(which)
v = v_template
if shapedirs is not None:
if betas is None:
betas = chumpy.zeros(shapedirs.shape[-1])
v_shaped = v + shapedirs.dot(betas)
else:
v_shaped = v
if posedirs is not None:
v_posed = v_shaped + posedirs.dot(posemap(bs_type)(pose))
else:
v_posed = v_shaped
v = v_posed
if sp.issparse(J):
regressor = J
J_tmpx = MatVecMult(regressor, v_shaped[:,0])
J_tmpy = MatVecMult(regressor, v_shaped[:,1])
J_tmpz = MatVecMult(regressor, v_shaped[:,2])
J = chumpy.vstack((J_tmpx, J_tmpy, J_tmpz)).T
else:
assert(ischumpy(J))
assert(bs_style=='lbs')
result, Jtr = lbs.verts_core(pose, v, J, weights, kintree_table, want_Jtr=True, xp=chumpy)
tr = trans.reshape((1,3))
result = result + tr
Jtr = Jtr + tr
result.trans = trans
result.f = f
result.pose = pose
result.v_template = v_template
result.J = J
result.weights = weights
result.kintree_table = kintree_table
result.bs_style = bs_style
result.bs_type =bs_type
if posedirs is not None:
result.posedirs = posedirs
result.v_posed = v_posed
if shapedirs is not None:
result.shapedirs = shapedirs
result.betas = betas
result.v_shaped = v_shaped
if want_Jtr:
result.J_transformed = Jtr
return result
def verts_core(pose, v, J, weights, kintree_table, bs_style, want_Jtr=False, xp=chumpy):
if xp == chumpy:
assert(hasattr(pose, 'dterms'))
assert(hasattr(v, 'dterms'))
assert(hasattr(J, 'dterms'))
assert(hasattr(weights, 'dterms'))
assert(bs_style=='lbs')
result = lbs.verts_core(pose, v, J, weights, kintree_table, want_Jtr, xp)
return result
================================================
FILE: libs/annotator/smplify/__init__.py
================================================
================================================
FILE: libs/annotator/smplify/fit_3d.py
================================================
"""
Copyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPLify license here:
http://smplify.is.tue.mpg.de/license
About this Script:
============
This is a demo version of the algorithm implemented in the paper,
which fits the SMPL body model to the image given the joint detections.
The code is organized to be run on the LSP dataset.
See README to see how to download images and the detected joints.
"""
from os.path import join, exists, abspath, dirname
from os import makedirs
import logging
import pickle
from time import time
from glob import glob
import argparse
import cv2
import numpy as np
import chumpy as ch
from opendr.camera import ProjectPoints
from .lib.robustifiers import GMOf
from .lib.sphere_collisions import SphereCollisions
from .lib.max_mixture_prior import MaxMixtureCompletePrior
from .render_model import render_model
from smpl_webuser.serialization import load_model
from smpl_webuser.lbs import global_rigid_transformation
from smpl_webuser.verts import verts_decorated
_LOGGER = logging.getLogger(__name__)
# Mapping from LSP joints to SMPL joints.
# 0 Right ankle 8
# 1 Right knee 5
# 2 Right hip 2
# 3 Left hip 1
# 4 Left knee 4
# 5 Left ankle 7
# 6 Right wrist 21
# 7 Right elbow 19
# 8 Right shoulder 17
# 9 Left shoulder 16
# 10 Left elbow 18
# 11 Left wrist 20
# 12 Neck -
# 13 Head top added
# --------------------Camera estimation --------------------
def guess_init(model, focal_length, j2d, init_pose):
"""Initialize the camera translation via triangle similarity, by using the torso joints .
:param model: SMPL model
:param focal_length: camera focal length (kept fixed)
:param j2d: 14x2 array of CNN joints
:param init_pose: 72D vector of pose parameters used for initialization (kept fixed)
:returns: 3D vector corresponding to the estimated camera translation
"""
cids = np.arange(0, 12)
# map from LSP to SMPL joints
j2d_here = j2d[cids]
smpl_ids = [8, 5, 2, 1, 4, 7, 21, 19, 17, 16, 18, 20]
opt_pose = ch.array(init_pose)
(_, A_global) = global_rigid_transformation(
opt_pose, model.J, model.kintree_table, xp=ch)
Jtr = ch.vstack([g[:3, 3] for g in A_global])
Jtr = Jtr[smpl_ids].r
# 9 is L shoulder, 3 is L hip
# 8 is R shoulder, 2 is R hip
diff3d = np.array([Jtr[9] - Jtr[3], Jtr[8] - Jtr[2]])
mean_height3d = np.mean(np.sqrt(np.sum(diff3d**2, axis=1)))
diff2d = np.array([j2d_here[9] - j2d_here[3], j2d_here[8] - j2d_here[2]])
mean_height2d = np.mean(np.sqrt(np.sum(diff2d**2, axis=1)))
est_d = focal_length * (mean_height3d / mean_height2d)
# just set the z value
init_t = np.array([0., 0., est_d])
return init_t
def initialize_camera(model,
j2d,
img,
init_pose,
flength=5000.,
pix_thsh=25.,
viz=False):
"""Initialize camera translation and body orientation
:param model: SMPL model
:param j2d: 14x2 array of CNN joints
:param img: h x w x 3 image
:param init_pose: 72D vector of pose parameters used for initialization
:param flength: camera focal length (kept fixed)
:param pix_thsh: threshold (in pixel), if the distance between shoulder joints in 2D
is lower than pix_thsh, the body orientation as ambiguous (so a fit is run on both
the estimated one and its flip)
:param viz: boolean, if True enables visualization during optimization
:returns: a tuple containing the estimated camera,
a boolean deciding if both the optimized body orientation and its flip should be considered,
3D vector for the body orientation
"""
# optimize camera translation and body orientation based on torso joints
# LSP torso ids:
# 2=right hip, 3=left hip, 8=right shoulder, 9=left shoulder
torso_cids = [2, 3, 8, 9]
# corresponding SMPL torso ids
torso_smpl_ids = [2, 1, 17, 16]
center = np.array([img.shape[1] / 2, img.shape[0] / 2])
# initialize camera rotation
rt = ch.zeros(3)
# initialize camera translation
_LOGGER.info('initializing translation via similar triangles')
init_t = guess_init(model, flength, j2d, init_pose)
t = ch.array(init_t)
# check how close the shoulder joints are
try_both_orient = np.linalg.norm(j2d[8] - j2d[9]) < pix_thsh
opt_pose = ch.array(init_pose)
(_, A_global) = global_rigid_transformation(
opt_pose, model.J, model.kintree_table, xp=ch)
Jtr = ch.vstack([g[:3, 3] for g in A_global])
# initialize the camera
cam = ProjectPoints(
f=np.array([flength, flength]), rt=rt, t=t, k=np.zeros(5), c=center)
# we are going to project the SMPL joints
cam.v = Jtr
if viz:
viz_img = img.copy()
# draw the target (CNN) joints
for coord in np.around(j2d).astype(int):
if (coord[0] < img.shape[1] and coord[0] >= 0 and
coord[1] < img.shape[0] and coord[1] >= 0):
cv2.circle(viz_img, tuple(coord), 3, [0, 255, 0])
import matplotlib.pyplot as plt
plt.ion()
# draw optimized joints at each iteration
def on_step(_):
"""Draw a visualization."""
plt.figure(1, figsize=(5, 5))
plt.subplot(1, 1, 1)
viz_img = img.copy()
for coord in np.around(cam.r[torso_smpl_ids]).astype(int):
if (coord[0] < viz_img.shape[1] and coord[0] >= 0 and
coord[1] < viz_img.shape[0] and coord[1] >= 0):
cv2.circle(viz_img, tuple(coord), 3, [0, 0, 255])
plt.imshow(viz_img[:, :, ::-1])
plt.draw()
plt.show()
plt.pause(1e-3)
else:
on_step = None
# optimize for camera translation and body orientation
free_variables = [cam.t, opt_pose[:3]]
ch.minimize(
# data term defined over torso joints...
{'cam': j2d[torso_cids] - cam[torso_smpl_ids],
# ...plus a regularizer for the camera translation
'cam_t': 1e2 * (cam.t[2] - init_t[2])},
x0=free_variables,
method='dogleg',
callback=on_step,
options={'maxiter': 100,
'e_3': .0001,
# disp set to 1 enables verbose output from the optimizer
'disp': 0})
if viz:
plt.ioff()
return (cam, try_both_orient, opt_pose[:3].r)
# --------------------Core optimization --------------------
def optimize_on_joints(j2d,
model,
cam,
img,
prior,
try_both_orient,
body_orient,
n_betas=10,
regs=None,
conf=None,
viz=False):
"""Fit the model to the given set of joints, given the estimated camera
:param j2d: 14x2 array of CNN joints
:param model: SMPL model
:param cam: estimated camera
:param img: h x w x 3 image
:param prior: mixture of gaussians pose prior
:param try_both_orient: boolean, if True both body_orient and its flip are considered for the fit
:param body_orient: 3D vector, initialization for the body orientation
:param n_betas: number of shape coefficients considered during optimization
:param regs: regressors for capsules' axis and radius, if not None enables the interpenetration error term
:param conf: 14D vector storing the confidence values from the CNN
:param viz: boolean, if True enables visualization during optimization
:returns: a tuple containing the optimized model, its joints projected on image space, the camera translation
"""
t0 = time()
# define the mapping LSP joints -> SMPL joints
# cids are joints ids for LSP:
cids = range(12) + [13]
# joint ids for SMPL
# SMPL does not have a joint for head, instead we use a vertex for the head
# and append it later.
smpl_ids = [8, 5, 2, 1, 4, 7, 21, 19, 17, 16, 18, 20]
# the vertex id for the joint corresponding to the head
head_id = 411
# weights assigned to each joint during optimization;
# the definition of hips in SMPL and LSP is significantly different so set
# their weights to zero
base_weights = np.array(
[1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=np.float64)
if try_both_orient:
flipped_orient = cv2.Rodrigues(body_orient)[0].dot(
cv2.Rodrigues(np.array([0., np.pi, 0]))[0])
flipped_orient = cv2.Rodrigues(flipped_orient)[0].ravel()
orientations = [body_orient, flipped_orient]
else:
orientations = [body_orient]
if try_both_orient:
# store here the final error for both orientations,
# and pick the orientation resulting in the lowest error
errors = []
svs = []
cams = []
for o_id, orient in enumerate(orientations):
# initialize the shape to the mean shape in the SMPL training set
betas = ch.zeros(n_betas)
# initialize the pose by using the optimized body orientation and the
# pose prior
init_pose = np.hstack((orient, prior.weights.dot(prior.means)))
# instantiate the model:
# verts_decorated allows us to define how many
# shape coefficients (directions) we want to consider (here, n_betas)
sv = verts_decorated(
trans=ch.zeros(3),
pose=ch.array(init_pose),
v_template=model.v_template,
J=model.J_regressor,
betas=betas,
shapedirs=model.shapedirs[:, :, :n_betas],
weights=model.weights,
kintree_table=model.kintree_table,
bs_style=model.bs_style,
f=model.f,
bs_type=model.bs_type,
posedirs=model.posedirs)
# make the SMPL joints depend on betas
Jdirs = np.dstack([model.J_regressor.dot(model.shapedirs[:, :, i])
for i in range(len(betas))])
J_onbetas = ch.array(Jdirs).dot(betas) + model.J_regressor.dot(
model.v_template.r)
# get joint positions as a function of model pose, betas and trans
(_, A_global) = global_rigid_transformation(
sv.pose, J_onbetas, model.kintree_table, xp=ch)
Jtr = ch.vstack([g[:3, 3] for g in A_global]) + sv.trans
# add the head joint, corresponding to a vertex...
Jtr = ch.vstack((Jtr, sv[head_id]))
# ... and add the joint id to the list
if o_id == 0:
smpl_ids.append(len(Jtr) - 1)
# update the weights using confidence values
weights = base_weights * conf[
cids] if conf is not None else base_weights
# project SMPL joints on the image plane using the estimated camera
cam.v = Jtr
# data term: distance between observed and estimated joints in 2D
obj_j2d = lambda w, sigma: (
w * weights.reshape((-1, 1)) * GMOf((j2d[cids] - cam[smpl_ids]), sigma))
# mixture of gaussians pose prior
pprior = lambda w: w * prior(sv.pose)
# joint angles pose prior, defined over a subset of pose parameters:
# 55: left elbow, 90deg bend at -np.pi/2
# 58: right elbow, 90deg bend at np.pi/2
# 12: left knee, 90deg bend at np.pi/2
# 15: right knee, 90deg bend at np.pi/2
alpha = 10
my_exp = lambda x: alpha * ch.exp(x)
obj_angle = lambda w: w * ch.concatenate([my_exp(sv.pose[55]), my_exp(-sv.pose[
58]), my_exp(-sv.pose[12]), my_exp(-sv.pose[15])])
if viz:
import matplotlib.pyplot as plt
plt.ion()
def on_step(_):
"""Create visualization."""
plt.figure(1, figsize=(10, 10))
plt.subplot(1, 2, 1)
# show optimized joints in 2D
tmp_img = img.copy()
for coord, target_coord in zip(
np.around(cam.r[smpl_ids]).astype(int),
np.around(j2d[cids]).astype(int)):
if (coord[0] < tmp_img.shape[1] and coord[0] >= 0 and
coord[1] < tmp_img.shape[0] and coord[1] >= 0):
cv2.circle(tmp_img, tuple(coord), 3, [0, 0, 255])
if (target_coord[0] < tmp_img.shape[1] and
target_coord[0] >= 0 and
target_coord[1] < tmp_img.shape[0] and
target_coord[1] >= 0):
cv2.circle(tmp_img, tuple(target_coord), 3,
[0, 255, 0])
plt.imshow(tmp_img[:, :, ::-1])
plt.draw()
plt.show()
plt.pause(1e-2)
on_step(_)
else:
on_step = None
if regs is not None:
# interpenetration term
sp = SphereCollisions(
pose=sv.pose, betas=sv.betas, model=model, regs=regs)
sp.no_hands = True
# weight configuration used in the paper, with joints + confidence values from the CNN
# (all the weights used in the code were obtained via grid search, see the paper for more details)
# the first list contains the weights for the pose priors,
# the second list contains the weights for the shape prior
opt_weights = zip([4.04 * 1e2, 4.04 * 1e2, 57.4, 4.78],
[1e2, 5 * 1e1, 1e1, .5 * 1e1])
# run the optimization in 4 stages, progressively decreasing the
# weights for the priors
for stage, (w, wbetas) in enumerate(opt_weights):
_LOGGER.info('stage %01d', stage)
objs = {}
objs['j2d'] = obj_j2d(1., 100)
objs['pose'] = pprior(w)
objs['pose_exp'] = obj_angle(0.317 * w)
objs['betas'] = wbetas * betas
if regs is not None:
objs['sph_coll'] = 1e3 * sp
ch.minimize(
objs,
x0=[sv.betas, sv.pose],
method='dogleg',
callback=on_step,
options={'maxiter': 100,
'e_3': .0001,
'disp': 0})
t1 = time()
_LOGGER.info('elapsed %.05f', (t1 - t0))
if try_both_orient:
errors.append((objs['j2d'].r**2).sum())
svs.append(sv)
cams.append(cam)
if try_both_orient and errors[0] > errors[1]:
choose_id = 1
else:
choose_id = 0
if viz:
plt.ioff()
return (svs[choose_id], cams[choose_id].r, cams[choose_id].t.r, Jtr)
def run_single_fit(img,
j2d,
conf,
model,
regs=None,
n_betas=10,
flength=5000.,
pix_thsh=25.,
scale_factor=1,
viz=False,
do_degrees=None):
"""Run the fit for one specific image.
:param img: h x w x 3 image
:param j2d: 14x2 array of CNN joints
:param conf: 14D vector storing the confidence values from the CNN
:param model: SMPL model
:param regs: regressors for capsules' axis and radius, if not None enables the interpenetration error term
:param n_betas: number of shape coefficients considered during optimization
:param flength: camera focal length (kept fixed during optimization)
:param pix_thsh: threshold (in pixel), if the distance between shoulder joints in 2D
is lower than pix_thsh, the body orientation as ambiguous (so a fit is run on both
the estimated one and its flip)
:param scale_factor: int, rescale the image (for LSP, slightly greater images -- 2x -- help obtain better fits)
:param viz: boolean, if True enables visualization during optimization
:param do_degrees: list of degrees in azimuth to render the final fit when saving results
:returns: a tuple containing camera/model parameters and images with rendered fits
"""
if do_degrees is None:
do_degrees = []
# create the pose prior (GMM over CMU)
prior = MaxMixtureCompletePrior(n_gaussians=8).get_gmm_prior()
# get the mean pose as our initial pose
init_pose = np.hstack((np.zeros(3), prior.weights.dot(prior.means)))
if scale_factor != 1:
img = cv2.resize(img, (img.shape[1] * scale_factor,
img.shape[0] * scale_factor))
j2d[:, 0] *= scale_factor
j2d[:, 1] *= scale_factor
# estimate the camera parameters
(cam, try_both_orient, body_orient) = initialize_camera(
model,
j2d,
img,
init_pose,
flength=flength,
pix_thsh=pix_thsh,
viz=viz)
# fit
(sv, opt_j2d, t, v) = optimize_on_joints(
j2d,
model,
cam,
img,
prior,
try_both_orient,
body_orient,
n_betas=n_betas,
conf=conf,
viz=viz,
regs=regs, )
h = img.shape[0]
w = img.shape[1]
dist = np.abs(cam.t.r[2] - np.mean(sv.r, axis=0)[2])
images = []
orig_v = sv.r
for deg in do_degrees:
if deg != 0:
aroundy = cv2.Rodrigues(np.array([0, np.radians(deg), 0]))[0]
center = orig_v.mean(axis=0)
new_v = np.dot((orig_v - center), aroundy)
verts = new_v + center
else:
verts = orig_v
# now render
im = (render_model(
verts, model.f, w, h, cam, far=20 + dist) * 255.).astype('uint8')
images.append(im)
# return fit parameters
# .r converts a chumpy array into numpy array
params = {'cam_t': cam.t.r,
'f': cam.f.r,
'v': v.r,
'pose': sv.pose.r,
'betas': sv.betas.r}
return params, images
def main(base_dir,
out_dir,
use_interpenetration=True,
n_betas=10,
flength=5000.,
pix_thsh=25.,
use_neutral=False,
viz=True):
"""Set up paths to image and joint data, saves results.
:param base_dir: folder containing LSP images and data
:param out_dir: output folder
:param use_interpenetration: boolean, if True enables the interpenetration term
:param n_betas: number of shape coefficients considered during optimization
:param flength: camera focal length (an estimate)
:param pix_thsh: threshold (in pixel), if the distance between shoulder joints in 2D
is lower than pix_thsh, the body orientation as ambiguous (so a fit is run on both
the estimated one and its flip)
:param use_neutral: boolean, if True enables uses the neutral gender SMPL model
:param viz: boolean, if True enables visualization during optimization
"""
img_dir = join(abspath(base_dir), 'images/lsp')
data_dir = join(abspath(base_dir), 'results/lsp')
if not exists(out_dir):
makedirs(out_dir)
# Render degrees: List of degrees in azimuth to render the final fit.
# Note that rendering many views can take a while.
do_degrees = [0.]
sph_regs = None
if not use_neutral:
_LOGGER.info("Reading genders...")
# File storing information about gender in LSP
with open(join(data_dir, 'lsp_gender.csv')) as f:
genders = f.readlines()
model_female = load_model(MODEL_FEMALE_PATH)
model_male = load_model(MODEL_MALE_PATH)
if use_interpenetration:
sph_regs_male = np.load(SPH_REGS_MALE_PATH)
sph_regs_female = np.load(SPH_REGS_FEMALE_PATH)
else:
gender = 'neutral'
model = load_model(MODEL_NEUTRAL_PATH)
if use_interpenetration:
sph_regs = np.load(SPH_REGS_NEUTRAL_PATH)
# Load joints
est = np.load(join(data_dir, 'est_joints.npz'))['est_joints']
# Load images
img_paths = sorted(glob(join(img_dir, '*[0-9].jpg')))
for ind, img_path in enumerate(img_paths):
out_path = '%s/%04d.pkl' % (out_dir, ind)
if not exists(out_path):
_LOGGER.info('Fitting 3D body on `%s` (saving to `%s`).', img_path,
out_path)
img = cv2.imread(img_path)
if img.ndim == 2:
_LOGGER.warn("The image is grayscale!")
img = np.dstack((img, img, img))
# [x-y, keypoints, idx]
joints = est[:2, :, ind].T
conf = est[2, :, ind]
if not use_neutral:
gender = 'male' if int(genders[ind]) == 0 else 'female'
if gender == 'female':
model = model_female
if use_interpenetration:
sph_regs = sph_regs_female
elif gender == 'male':
model = model_male
if use_interpenetration:
sph_regs = sph_regs_male
params, vis = run_single_fit(
img,
joints,
conf,
model,
regs=sph_regs,
n_betas=n_betas,
flength=flength,
pix_thsh=pix_thsh,
scale_factor=2,
viz=viz,
do_degrees=do_degrees)
if viz:
import matplotlib.pyplot as plt
plt.ion()
plt.show()
plt.subplot(121)
plt.imshow(img[:, :, ::-1])
if do_degrees is not None:
print(do_degrees)
for di, deg in enumerate(do_degrees):
plt.subplot(122)
plt.cla()
plt.imshow(vis[di])
plt.draw()
plt.title('%d deg' % deg)
plt.pause(1)
raw_input('Press any key to continue...')
with open(out_path, 'w') as outf:
pickle.dump(params, outf)
print(do_degrees)
# This only saves the first rendering.
if do_degrees is not None:
cv2.imwrite(out_path.replace('.pkl', '.png'), vis[0])
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
parser = argparse.ArgumentParser(description='run SMPLify on LSP dataset')
parser.add_argument(
'base_dir',
default='/scratch1/projects/smplify_public/',
nargs='?',
help="Directory that contains images/lsp and results/lps , i.e."
"the directory you untared smplify_code.tar.gz")
parser.add_argument(
'--out_dir',
default='/home/kp-kepra/Github/smplify_public',
type=str,
help='Where results will be saved, default is /tmp/smplify_lsp')
parser.add_argument(
'--no_interpenetration',
default=False,
action='store_true',
help="Using this flag removes the interpenetration term, which speeds"
"up optimization at the expense of possible interpenetration.")
parser.add_argument(
'--gender_neutral',
default=False,
action='store_true',
help="Using this flag always uses the neutral SMPL model, otherwise "
"gender specified SMPL models are used.")
parser.add_argument(
'--n_betas',
default=10,
type=int,
help="Specify the number of shape coefficients to use.")
parser.add_argument(
'--flength',
default=5000,
type=float,
help="Specify value of focal length.")
parser.add_argument(
'--side_view_thsh',
default=25,
type=float,
help="This is thresholding value that determines whether the human is captured in a side view. If the pixel distance between the shoulders is less than this value, two initializations of SMPL fits are tried.")
parser.add_argument(
'--viz',
default=False,
action='store_true',
help="Turns on visualization of intermediate optimization steps "
"and final results.")
args = parser.parse_args()
use_interpenetration = not args.no_interpenetration
if not use_interpenetration:
_LOGGER.info('Not using interpenetration term.')
if args.gender_neutral:
_LOGGER.info('Using gender neutral model.')
# Set up paths & load models.
# Assumes 'models' in the 'code/' directory where this file is in.
MODEL_DIR = join(abspath(dirname(__file__)), 'models')
# Model paths:
MODEL_NEUTRAL_PATH = join(
MODEL_DIR, 'basicModel_neutral_lbs_10_207_0_v1.0.0.pkl')
MODEL_FEMALE_PATH = join(
MODEL_DIR, 'basicModel_f_lbs_10_207_0_v1.0.0.pkl')
MODEL_MALE_PATH = join(MODEL_DIR,
'basicmodel_m_lbs_10_207_0_v1.0.0.pkl')
if use_interpenetration:
# paths to the npz files storing the regressors for capsules
SPH_REGS_NEUTRAL_PATH = join(MODEL_DIR,
'regressors_locked_normalized_hybrid.npz')
SPH_REGS_FEMALE_PATH = join(MODEL_DIR,
'regressors_locked_normalized_female.npz')
SPH_REGS_MALE_PATH = join(MODEL_DIR,
'regressors_locked_normalized_male.npz')
main(args.base_dir, args.out_dir, use_interpenetration, args.n_betas,
args.flength, args.side_view_thsh, args.gender_neutral, args.viz)
================================================
FILE: libs/annotator/smplify/lib/__init__.py
================================================
================================================
FILE: libs/annotator/smplify/lib/capsule_body.py
================================================
"""
Copyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPLify license here:
http://smplify.is.tue.mpg.de/license
This script implements an approximation of the body by means of capsules (20 in total).
Capsules can be further simplified into spheres (with centers along the capsule axis and
radius corresponding to the capsule radius) to efficiently compute an interpenetration error term
(as in sphere_collisions.py).
"""
import numpy as np
import chumpy as ch
import scipy.sparse as sp
from .capsule_ch import Capsule
joint2name = ['pelvis', 'leftThigh', 'rightThigh', 'spine', 'leftCalf',
'rightCalf', 'spine1', 'leftFoot', 'rightFoot', 'spine2', 'neck',
'leftShoulder', 'rightShoulder', 'head', 'leftUpperArm',
'rightUpperArm', 'leftForeArm', 'rightForeArm', 'leftHand',
'rightHand']
# the orientation of each capsule
rots0 = ch.asarray(
[[0, 0, np.pi / 2], [0, 0, np.pi], [0, 0, np.pi], [0, 0, np.pi / 2],
[0, 0, np.pi], [0, 0, np.pi], [0, 0, np.pi / 2], [np.pi / 2, 0, 0],
[np.pi / 2, 0, 0], [0, 0, np.pi / 2], [0, 0, 0], [0, 0, -np.pi / 2],
[0, 0, np.pi / 2], [0, 0, 0], [0, 0, -np.pi / 2], [0, 0, np.pi / 2],
[0, 0, -np.pi / 2], [0, 0, np.pi / 2], [0, 0, -np.pi / 2],
[0, 0, np.pi / 2]])
# groups hands and fingers, feet and toes
# each comment line provides the body part corresonding to the capsule
# and the corresponding id
mujoco2segm = [[0], # hip 0
[1], # leftThigh 1
[2], # rightThigh 2
[3], # spine 3
[4], # leftCalf 4
[5], # rightCalf 5
[6], # spine1 6
[7, 10], # leftFoot + leftToes 7
[8, 11], # rightFoot + rightToes 8
[9], # spine2 9
[12], # neck 10
[13], # leftShoulder 11
[14], # rightShoulder 12
[15], # head 13
[16], # leftUpperArm 14
[17], # rightUpperArm 15
[18], # leftForeArm 16
[19], # rightForeArm 17
[20, 22], # leftHand + leftFingers 18
[21, 23]] # rightHand + rightFingers 19
# sets pairs of ids, corresponding to capsules that should not
# penetrate each other
collisions = [
[0, 16], # hip and leftForeArm
[0, 17], # hip and rightForeArm
[0, 18], # hip and leftHand
[0, 19], # hip and rightHand
[3, 16], # spine and leftForeArm
[3, 17], # spine and rightForeArm
[3, 18], # spine and leftHand
[3, 19], # spine and rightHand
[4, 5], # leftCalf and rightCalf
[6, 16], # spine1 and leftForeArm
[6, 17], # spine1 and rightForeArm
[6, 18], # spine1 and leftHand
[6, 19], # spine1 and rightHand
[7, 5], # leftFoot and rightCalf
[8, 7], # rightFoot and leftFoot
[8, 4], # rightFoot and leftCalf
[9, 16], # spine2 and leftForeArm
[9, 17], # spine2 and rightForeArm
[9, 18], # spine2 and leftHand
[9, 19], # spine2 and rightHand
[11, 16], # leftShoulder and leftForeArm
[12, 17], # rightShoulder and rightForeArm
[18, 19], # leftHand and rightHand
]
def get_capsules(model, wrt_betas=None, length_regs=None, rad_regs=None):
from opendr.geometry import Rodrigues
if length_regs is not None:
n_shape_dofs = length_regs.shape[0] - 1
else:
n_shape_dofs = model.betas.r.size
segm = np.argmax(model.weights_prior, axis=1)
J_off = ch.zeros((len(joint2name), 3))
rots = rots0.copy()
mujoco_t_mid = [0, 3, 6, 9]
if wrt_betas is not None:
# if we want to differentiate wrt betas (shape), we must have the
# regressors...
assert (length_regs is not None and rad_regs is not None)
# ... and betas must be a chumpy object
assert (hasattr(wrt_betas, 'dterms'))
pad = ch.concatenate(
(wrt_betas, ch.zeros(n_shape_dofs - len(wrt_betas)), ch.ones(1)))
lengths = pad.dot(length_regs)
rads = pad.dot(rad_regs)
else:
lengths = ch.ones(len(joint2name))
rads = ch.ones(len(joint2name))
betas = wrt_betas if wrt_betas is not None else model.betas
n_betas = len(betas)
# the joint regressors are the original, pre-optimized ones
# (middle of the part frontier)
myJ_regressor = model.J_regressor_prior
myJ0 = ch.vstack(
(ch.ch.MatVecMult(myJ_regressor, model.v_template[:, 0] +
model.shapedirs[:, :, :n_betas].dot(betas)[:, 0]),
ch.ch.MatVecMult(myJ_regressor, model.v_template[:, 1] +
model.shapedirs[:, :, :n_betas].dot(betas)[:, 1]),
ch.ch.MatVecMult(myJ_regressor, model.v_template[:, 2] +
model.shapedirs[:, :, :n_betas].dot(betas)[:, 2]))).T
# with small adjustments for hips, spine and feet
myJ = ch.vstack(
[ch.concatenate([myJ0[0, 0], (
.6 * myJ0[0, 1] + .2 * myJ0[1, 1] + .2 * myJ0[2, 1]), myJ0[9, 2]]),
ch.vstack([myJ0[i] for i in range(1, 7)]), ch.concatenate(
[myJ0[7, 0], (1.1 * myJ0[7, 1] - .1 * myJ0[4, 1]), myJ0[7, 2]]),
ch.concatenate(
[myJ0[8, 0], (1.1 * myJ0[8, 1] - .1 * myJ0[5, 1]), myJ0[8, 2]]),
ch.concatenate(
[myJ0[9, 0], myJ0[9, 1], (.2 * myJ0[9, 2] + .8 * myJ0[12, 2])]),
ch.vstack([myJ0[i] for i in range(10, 24)])])
capsules = []
# create one capsule per mujoco joint
for ijoint, segms in enumerate(mujoco2segm):
if wrt_betas is None:
vidxs = np.asarray([segm == k for k in segms]).any(axis=0)
verts = model.v_template[vidxs].r
dims = (verts.max(axis=0) - verts.min(axis=0))
rads[ijoint] = .5 * ((dims[(np.argmax(dims) + 1) % 3] + dims[(
np.argmax(dims) + 2) % 3]) / 4.)
lengths[ijoint] = max(dims) - 2. * rads[ijoint].r
# the core joints are different, since the capsule is not in the joint
# but in the middle
if ijoint in mujoco_t_mid:
len_offset = ch.vstack([ch.zeros(1), ch.abs(lengths[ijoint]) / 2.,
ch.zeros(1)]).reshape(3, 1)
caps = Capsule(
(J_off[ijoint] + myJ[mujoco2segm[ijoint][0]]).reshape(
3, 1) - Rodrigues(rots[ijoint]).dot(len_offset),
rots[ijoint], rads[ijoint], lengths[ijoint])
else:
caps = Capsule(
(J_off[ijoint] + myJ[mujoco2segm[ijoint][0]]).reshape(3, 1),
rots[ijoint], rads[ijoint], lengths[ijoint])
caps.id = ijoint
capsules.append(caps)
return capsules
def set_sphere_centers(capsule, floor=True):
if floor:
n_spheres = int(np.floor(capsule.length.r / (2 * capsule.rad.r) - 1))
else:
n_spheres = int(np.ceil(capsule.length.r / (2 * capsule.rad.r) - 1))
# remove "redundant" spheres for right and left thigh...
if capsule.id == 1 or capsule.id == 2:
centers = [capsule.axis[1].r]
# ... and right and left upper arm
elif capsule.id == 14 or capsule.id == 15:
if n_spheres >= 1:
centers = []
else:
centers = [capsule.axis[1].r]
else:
centers = [capsule.axis[0].r, capsule.axis[1].r]
if n_spheres >= 1:
step = capsule.length.r / (n_spheres + 1)
for i in xrange(n_spheres):
centers.append(capsule.axis[0].r + (capsule.axis[
1].r - capsule.axis[0].r) * step * (i + 1) / capsule.length.r)
capsule.centers = centers
return capsule.centers
def capsule_dist(capsule0, capsule1, alpha=.3, increase_hand=True):
range0 = range(capsule0.center_id,
capsule0.center_id + len(capsule0.centers))
range1 = range(capsule1.center_id,
capsule1.center_id + len(capsule1.centers))
cnt0 = ch.concatenate([[cid] * len(range1) for cid in range0])
cnt1 = ch.concatenate([range1] * len(range0))
if increase_hand:
if (capsule0.id == 18) or (capsule0.id == 19) or (
capsule1.id == 18) or (capsule1.id == 19):
dst = (alpha * 1.2 * capsule0.rad.r)**2 + (alpha * 1.2 *
capsule1.rad.r)**2
else:
dst = (alpha * capsule0.rad.r)**2 + (alpha * capsule1.rad.r)**2
else:
dst = (alpha * capsule0.rad.r)**2 + (alpha * capsule1.rad.r)**2
radiuss = np.hstack([dst] * len(cnt0)).squeeze()
return (cnt0, cnt1, radiuss)
def get_capsule_bweights(vs):
# "blend" weights for the capsule. They are binary
rows = np.arange(vs.shape[0])
cols = np.tile(np.hstack((range(10), range(12, 22))), (52, 1)).T.ravel()
data = np.ones(vs.shape[0])
caps_weights = np.asarray(
sp.csc_matrix(
(data, (rows, cols)), shape=(vs.shape[0], 24)).todense())
return caps_weights
def get_sphere_bweights(sph_vs, capsules):
rows = np.arange(sph_vs.shape[0])
cols = []
for cps, w in zip(capsules, range(10) + range(12, 22)):
cols.append([w] * len(cps.centers))
cols = np.hstack(cols)
data = np.ones(sph_vs.shape[0])
sph_weights = np.asarray(
sp.csc_matrix(
(data, (rows, cols)), shape=(sph_vs.shape[0], 24)).todense())
return sph_weights
================================================
FILE: libs/annotator/smplify/lib/capsule_ch.py
================================================
"""
Copyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPLify license here:
http://smplify.is.tue.mpg.de/license
This script implements a Capsule object, used in the body approximation implemented
in capsule_body.py. Capsule sizes depend on body shape (and are differentiable with respect to it).
Capsules are the basis to compute an approximation based on spheres, used to compute efficiently
the interpenetration error term in sphere_collisions.py.
"""
import numpy as np
import chumpy as ch
from opendr.geometry import Rodrigues
# faces for the capsules. Useful only for visualization purposes
cap_f = np.asarray(
[[0, 7, 6], [1, 7, 9], [0, 6, 11], [0, 11, 13], [0, 13, 10], [1, 9, 16],
[2, 8, 18], [3, 12, 20], [4, 14, 22], [5, 15, 24], [1, 16, 19],
[2, 18, 21], [3, 20, 23], [4, 22, 25], [5, 24, 17], [16, 17, 26],
[22, 23, 32], [48, 18, 28], [49, 20, 30], [24, 25, 34], [25, 22, 50],
[28, 19, 47], [30, 21, 48], [32, 23, 49], [17, 24, 51], [26, 17, 51],
[34, 25, 50], [23, 20, 49], [21, 18, 48], [19, 16, 47], [51, 24, 34],
[24, 15, 25], [15, 4, 25], [50, 22, 32], [22, 14, 23], [14, 3, 23],
[20, 21, 30], [20, 12, 21], [12, 2, 21], [18, 19, 28], [18, 8, 19],
[8, 1, 19], [47, 16, 26], [16, 9, 17], [9, 5, 17], [10, 15, 5],
[10, 13, 15], [13, 4, 15], [13, 14, 4], [13, 11, 14], [11, 3, 14],
[11, 12, 3], [11, 6, 12], [6, 2, 12], [9, 10, 5], [9, 7, 10], [7, 0, 10],
[6, 8, 2], [6, 7, 8], [7, 1, 8], [29, 36, 41], [31, 37, 44], [33, 38, 45],
[35, 39, 46], [27, 40, 42], [42, 46, 43], [42, 40, 46], [40, 35, 46],
[46, 45, 43], [46, 39, 45], [39, 33, 45], [45, 44, 43], [45, 38, 44],
[38, 31, 44], [44, 41, 43], [44, 37, 41], [37, 29, 41], [41, 42, 43],
[41, 36, 42], [36, 27, 42], [26, 40, 27], [26, 51, 40], [51, 35, 40],
[34, 39, 35], [34, 50, 39], [50, 33, 39], [32, 38, 33], [32, 49, 38],
[49, 31, 38], [30, 37, 31], [30, 48, 37], [48, 29, 37], [28, 36, 29],
[28, 47, 36], [47, 27, 36], [51, 34, 35], [50, 32, 33], [49, 30, 31],
[48, 28, 29], [47, 26, 27]])
elev = np.asarray(
[0., 0.5535673, 1.01721871, 0., -1.01721871, -0.5535673, 0.52359324,
0.31415301, 0.94246863, 0., -0.31415301, 0., 0.52359547, -0.52359324,
-0.52359547, -0.94246863, 0.31415501, -0.31415501, 1.57079633, 0.94247719,
0.31415501, 0.94247719, -0.94247719, -0.31415501, -0.94247719,
-1.57079633, -0.31415624, 0., 0.94248124, 1.01722122, 0.94247396,
0.55356579, -0.31415377, -0.55356579, -1.57079233, -1.01722122,
0.52359706, 0.94246791, 0., -0.94246791, -0.52359706, 0.52359371, 0., 0.,
0.31415246, -0.31415246, -0.52359371, 0.31415624, 1.57079233, 0.31415377,
-0.94247396, -0.94248124])
az = np.asarray(
[-1.57079633, -0.55358064, -2.12435586, -2.67794236, -2.12435586,
-0.55358064, -1.7595018, -1.10715248, -1.10714872, -0.55357999,
-1.10715248, -2.12436911, -2.48922865, -1.7595018, -2.48922865,
-1.10714872, 0., 0., 0., 0., 3.14159265, 3.14159265, 3.14159265,
3.14159265, 0., 0., 0., 0.46365119, 0., 1.01724226, 3.14159265,
2.58801549, 3.14159265, 2.58801549, 3.14159265, 1.01724226, 0.6523668,
2.03445078, 2.58801476, 2.03445078, 0.6523668, 1.38209652, 1.01722642,
1.57080033, 2.03444394, 2.03444394, 1.38209652, 0., 3.14159265,
3.14159265, 3.14159265, 0.])
# vertices for the capsules
v = np.vstack(
[np.cos(az) * np.cos(elev), np.sin(az) * np.cos(elev), np.sin(elev)]).T
class Capsule(object):
def __init__(self, t, rod, rad, length):
assert (hasattr(t, 'dterms'))
# the translation should be a chumpy object (differentiable wrt shape)
self.t = t # translation of the axis
self.rod = rod # rotation of the axis in Rodrigues form
# the radius should be a chumpy object (differentiable wrt shape)
assert (hasattr(rad, 'dterms'))
self.rad = rad # radius of the capsule
# the length should be a chumpy object (differentiable wrt shape)
assert (hasattr(length, 'dterms'))
self.length = length # length of the axis
axis0 = ch.vstack([0, ch.abs(self.length), 0])
self.axis = ch.vstack((t.T, (t + Rodrigues(rod).dot(axis0)).T))
v0 = ch.hstack([v[:26].T * rad, (v[26:].T * rad) + axis0])
self.v = ((t + Rodrigues(rod).dot(v0)).T)
self.set_sphere_centers()
def set_sphere_centers(self, floor=False):
# sphere centers are evenly spaced along the capsule axis length
if floor:
n_spheres = int(np.floor(self.length / (2 * self.rad) - 1))
else:
n_spheres = int(np.ceil(self.length / (2 * self.rad) - 1))
centers = [self.axis[0].r, self.axis[1].r]
if n_spheres >= 1:
step = self.length.r / (n_spheres + 1)
for i in xrange(n_spheres):
centers.append(self.axis[0].r + (self.axis[1].r - self.axis[
0].r) * step * (i + 1) / self.length.r)
self.centers = centers
================================================
FILE: libs/annotator/smplify/lib/max_mixture_prior.py
================================================
"""
Copyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPLify license here:
http://smplify.is.tue.mpg.de/license
This script implements the pose prior based on a mixture of Gaussians.
To simplify the log-likelihood computation, the sum in the mixture of Gaussians
is approximated by a max operator (see the paper for more details).
"""
import os
import numpy as np
import chumpy as ch
class MaxMixtureComplete(ch.Ch):
"""Define the MaxMixture class."""
# x is the input vector we want to evaluate the prior on;
# means, precs and weights are the parameters of the mixture
dterms = 'x'
terms = 'means', 'precs', 'weights'
def on_changed(self, which):
# on_changed is called before any call to r or dr_wrt,
# therefore it can be used also for initialization
# setup means, precs and loglikelihood expressions
if 'means' in which or 'precs' in which or 'weights' in which:
# This is just the mahalanobis part.
self.loglikelihoods = [np.sqrt(0.5) * (self.x - m).dot(s)
for m, s in zip(self.means, self.precs)]
if 'x' in which:
self.min_component_idx = np.argmin(
[(logl**2).sum().r[0] - np.log(w[0])
for logl, w in zip(self.loglikelihoods, self.weights)])
def compute_r(self):
min_w = self.weights[self.min_component_idx]
# Add the sqrt(-log(weights))
return ch.concatenate((self.loglikelihoods[self.min_component_idx].r,
np.sqrt(-np.log(min_w))))
def compute_dr_wrt(self, wrt):
# the call to dr_wrt returns a jacobian 69 x 72,
# when wrt has 72 elements (pose vector)
# here we intercept the call and return a 70 x 72 matrix,
# with an additional row of zeroes (these are the jacobian
# entries corresponding to sqrt(-log(weights))
import scipy.sparse as sp
dr = self.loglikelihoods[self.min_component_idx].dr_wrt(wrt)
if dr is not None:
# extract rows, cols and data, and return a new matrix with
# the same values but 1 additional row
Is, Js, Vs = sp.find(dr)
dr = sp.csc_matrix(
(Vs, (Is, Js)), shape=(dr.shape[0] + 1, dr.shape[1]))
return dr
class MaxMixtureCompleteWrapper(object):
"""Convenience wrapper to match interface spec."""
def __init__(self, means, precs, weights, prefix):
self.means = means
self.precs = precs # Already "sqrt"ed
self.weights = weights
self.prefix = prefix
def __call__(self, x):
# wrapping since __call__ couldn't be defined directly for a chumpy
# object
return (MaxMixtureComplete(
x=x[self.prefix:],
means=self.means,
precs=self.precs,
weights=self.weights))
class MaxMixtureCompletePrior(object):
"""Prior density estimation."""
def __init__(self, n_gaussians=8, prefix=3):
self.n_gaussians = n_gaussians
self.prefix = prefix
self.prior = self.create_prior_from_cmu()
def create_prior_from_cmu(self):
"""Load the gmm from the CMU motion database."""
from os.path import dirname
import cPickle as pickle
with open(
os.path.join(
dirname(dirname(__file__)), 'models', 'gmm_%02d.pkl' %
self.n_gaussians)) as f:
gmm = pickle.load(f)
precs = ch.asarray([np.linalg.inv(cov) for cov in gmm['covars']])
chols = ch.asarray([np.linalg.cholesky(prec) for prec in precs])
# The constant term:
sqrdets = np.array([(np.sqrt(np.linalg.det(c)))
for c in gmm['covars']])
const = (2 * np.pi)**(69 / 2.)
self.weights = ch.asarray(gmm['weights'] / (const *
(sqrdets / sqrdets.min())))
return (MaxMixtureCompleteWrapper(
means=gmm['means'],
precs=chols,
weights=self.weights,
prefix=self.prefix))
def get_gmm_prior(self):
"""Getter implementation."""
return self.prior
================================================
FILE: libs/annotator/smplify/lib/robustifiers.py
================================================
"""
Copyright 2016 Max Planck Society, Matthew Loper. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPLify license here:
http://smplify.is.tue.mpg.de/license
This script implements the Geman-McClure robustifier as chumpy object.
"""
#!/usr/bin/env python
import numpy as np
import scipy
import scipy.sparse as sp
from chumpy import Ch
__all__ = ['GMOf']
def GMOf(x, sigma):
"""Given x and sigma in some units (say mm),
returns robustified values (in same units),
by making use of the Geman-McClure robustifier."""
result = SignedSqrt(x=GMOfInternal(x=x, sigma=sigma))
return result
class SignedSqrt(Ch):
dterms = ('x', )
terms = ()
def compute_r(self):
return np.sqrt(np.abs(self.x.r)) * np.sign(self.x.r)
def compute_dr_wrt(self, wrt):
if wrt is self.x:
result = (.5 / np.sqrt(np.abs(self.x.r)))
result = np.nan_to_num(result)
result *= (self.x.r != 0).astype(np.uint32)
return sp.spdiags(result.ravel(), [0], self.x.r.size,
self.x.r.size)
class GMOfInternal(Ch):
dterms = 'x', 'sigma'
def on_changed(self, which):
if 'sigma' in which:
assert (self.sigma.r > 0)
if 'x' in which:
self.squared_input = self.x.r**2.
def compute_r(self):
return (self.sigma.r**2 *
(self.squared_input /
(self.sigma.r**2 + self.squared_input))) * np.sign(self.x.r)
def compute_dr_wrt(self, wrt):
if wrt is not self.x and wrt is not self.sigma:
return None
squared_input = self.squared_input
result = []
if wrt is self.x:
dx = self.sigma.r**2 / (self.sigma.r**2 + squared_input
) - self.sigma.r**2 * (squared_input / (
self.sigma.r**2 + squared_input)**2)
dx = 2 * self.x.r * dx
result.append(
scipy.sparse.spdiags(
(dx * np.sign(self.x.r)).ravel(), [0],
self.x.r.size,
self.x.r.size,
format='csc'))
if wrt is self.sigma:
ds = 2 * self.sigma.r * (squared_input / (
self.sigma.r**2 + squared_input)) - 2 * self.sigma.r**3 * (
squared_input / (self.sigma.r**2 + squared_input)**2)
result.append(
scipy.sparse.spdiags(
(ds * np.sign(self.x.r)).ravel(), [0],
self.x.r.size,
self.x.r.size,
format='csc'))
if len(result) == 1:
return result[0]
else:
return np.sum(result).tocsc()
================================================
FILE: libs/annotator/smplify/lib/sphere_collisions.py
================================================
"""
Copyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPLify license here:
http://smplify.is.tue.mpg.de/license
This script implements the interpenetration error term. It uses the capsule
approximation provided in capsule_body.py, and is differentiable with respect
to body shape and pose.
"""
import numpy as np
import chumpy as ch
from smpl_webuser.lbs import verts_core
from .capsule_body import get_capsules, set_sphere_centers,\
get_sphere_bweights, collisions,\
capsule_dist
class SphereCollisions(ch.Ch):
dterms = ('pose', 'betas')
terms = ('regs', 'model')
def update_capsules_and_centers(self):
centers = [set_sphere_centers(capsule) for capsule in self.capsules]
count = 0
for capsule in self.capsules:
capsule.center_id = count
count += len(capsule.centers)
self.sph_vs = ch.vstack(centers)
self.sph_weights = get_sphere_bweights(self.sph_vs, self.capsules)
self.ids0 = []
self.ids1 = []
self.radiuss = []
self.caps_pairs = []
for collision in collisions:
if hasattr(self, 'no_hands'):
(id0, id1, rd) = capsule_dist(
self.capsules[collision[0]],
self.capsules[collision[1]],
increase_hand=False)
else:
(id0, id1, rd) = capsule_dist(self.capsules[collision[0]],
self.capsules[collision[1]])
self.ids0.append(id0.r)
self.ids1.append(id1.r)
self.radiuss.append(rd)
self.caps_pairs.append(['%02d_%02d' % (collision[0], collision[1])]
* len(id0))
self.ids0 = np.concatenate(self.ids0).astype(int)
self.ids1 = np.concatenate(self.ids1).astype(int)
self.radiuss = np.concatenate(self.radiuss)
self.caps_pairs = np.concatenate(self.caps_pairs)
assert (self.caps_pairs.size == self.ids0.size)
assert (self.radiuss.size == self.ids0.size)
def update_pose(self):
self.sph_v = verts_core(
self.pose,
self.sph_vs,
self.model.J,
self.sph_weights,
self.model.kintree_table,
want_Jtr=False)
def get_objective(self):
return ch.sum((ch.exp(
-((ch.sum((self.sph_v[self.ids0] - self.sph_v[self.ids1])**2,
axis=1)) / (self.radiuss)) / 2.))**2)**.5
def compute_r(self):
return self.get_objective().r
def compute_dr_wrt(self, wrt):
# we consider derivatives only with respect to pose here,
# to avoid bias towards thin body shapes
if wrt is self.pose:
return self.get_objective().dr_wrt(wrt)
def on_changed(self, which):
if 'regs' in which:
self.length_regs = self.regs['betas2lens']
self.rad_regs = self.regs['betas2rads']
if 'betas' in which:
if not hasattr(self, 'capsules'):
self.capsules = get_capsules(
self.model,
wrt_betas=self.betas,
length_regs=self.length_regs,
rad_regs=self.rad_regs)
self.update_capsules_and_centers()
if 'pose' in which:
self.update_pose()
================================================
FILE: libs/annotator/smplify/render_model.py
================================================
"""
Copyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.
This software is provided for research purposes only.
By using this software you agree to the terms of the SMPLify license here:
http://smplify.is.tue.mpg.de/license
Utility script for rendering the SMPL model using OpenDR.
"""
import numpy as np
from opendr.camera import ProjectPoints
from opendr.renderer import ColoredRenderer
from opendr.lighting import LambertianPointLight
import cv2
colors = {
'pink': [.7, .7, .9],
'neutral': [.9, .9, .8],
'capsule': [.7, .75, .5],
'yellow': [.5, .7, .75],
}
def _create_renderer(w=640,
h=480,
rt=np.zeros(3),
t=np.zeros(3),
f=None,
c=None,
k=None,
near=.5,
far=10.):
f = np.array([w, w]) / 2. if f is None else f
c = np.array([w, h]) / 2. if c is None else c
k = np.zeros(5) if k is None else k
rn = ColoredRenderer()
rn.camera = ProjectPoints(rt=rt, t=t, f=f, c=c, k=k)
rn.frustum = {'near': near, 'far': far, 'height': h, 'width': w}
return rn
def _rotateY(points, angle):
"""Rotate the points by a specified angle."""
ry = np.array([
[np.cos(angle), 0., np.sin(angle)], [0., 1., 0.],
[-np.sin(angle), 0., np.cos(angle)]
])
return np.dot(points, ry)
def simple_renderer(rn, verts, faces, yrot=np.radians(120)):
# Rendered model color
color = colors['pink']
rn.set(v=verts, f=faces, vc=color, bgcolor=np.ones(3))
albedo = rn.vc
# Construct Back Light (on back right corner)
rn.vc = LambertianPointLight(
f=rn.f,
v=rn.v,
num_verts=len(rn.v),
light_pos=_rotateY(np.array([-200, -100, -100]), yrot),
vc=albedo,
light_color=np.array([1, 1, 1]))
# Construct Left Light
rn.vc += LambertianPointLight(
f=rn.f,
v=rn.v,
num_verts=len(rn.v),
light_pos=_rotateY(np.array([800, 10, 300]), yrot),
vc=albedo,
light_color=np.array([1, 1, 1]))
# Construct Right Light
rn.vc += LambertianPointLight(
f=rn.f,
v=rn.v,
num_verts=len(rn.v),
light_pos=_rotateY(np.array([-500, 500, 1000]), yrot),
vc=albedo,
light_color=np.array([.7, .7, .7]))
return rn.r
def get_alpha(imtmp, bgval=1.):
h, w = imtmp.shape[:2]
alpha = (~np.all(imtmp == bgval, axis=2)).astype(imtmp.dtype)
b_channel, g_channel, r_channel = cv2.split(imtmp)
im_RGBA = cv2.merge(
(b_channel, g_channel, r_channel, alpha.astype(imtmp.dtype)))
return im_RGBA
def render_model(verts, faces, w, h, cam, near=0.5, far=25, img=None):
rn = _create_renderer(
w=w, h=h, near=near, far=far, rt=cam.rt, t=cam.t, f=cam.f, c=cam.c)
# Uses img as background, otherwise white background.
if img is not None:
rn.background_image = img / 255. if img.max() > 1 else img
imtmp = simple_renderer(rn, verts, faces)
# If white bg, make transparent.
if img is None:
imtmp = get_alpha(imtmp)
return imtmp
================================================
FILE: libs/dataset/__init__.py
================================================
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Empty file.
"""
#import libs.dataset.h36m
================================================
FILE: libs/dataset/h36m/__init__.py
================================================
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
================================================
FILE: libs/dataset/h36m/cameras.py
================================================
"""
Utilities to deal with the cameras of human3.6m.
Reference: https://github.com/una-dinosauria/3d-pose-baseline
"""
import numpy as np
# import h5py
def project_point_radial( P, R, T, f, c, k, p ):
"""
Project points from 3d to 2d using camera parameters
including radial and tangential distortion
Args
P: Nx3 points in world coordinates
R: 3x3 Camera rotation matrix
T: 3x1 Camera translation parameters
f: (scalar) Camera focal length
c: 2x1 Camera center
k: 3x1 Camera radial distortion coefficients
p: 2x1 Camera tangential distortion coefficients
Returns
Proj: Nx2 points in pixel space
D: 1xN depth of each point in camera space
radial: 1xN radial distortion per point
tan: 1xN tangential distortion per point
r2: 1xN squared radius of the projected points before distortion
"""
# P is a matrix of 3-dimensional points
assert len(P.shape) == 2
assert P.shape[1] == 3
N = P.shape[0]
X = R.dot( P.T - T ) # rotate and translate
XX = X[:2,:] / X[2,:]
r2 = XX[0,:]**2 + XX[1,:]**2
radial = 1 + np.einsum( 'ij,ij->j', np.tile(k,(1, N)),
np.array([r2, r2**2, r2**3])
);
tan = p[0]*XX[1,:] + p[1]*XX[0,:]
XXX = (XX * np.tile(radial + tan,(2,1))
+ np.outer(np.array([p[1], p[0]]).reshape(-1), r2))
Proj = (f * XXX) + c
Proj = Proj.T
D = X[2,]
return Proj, D, radial, tan, r2
def world_to_camera_frame(P, R, T):
"""
Convert points from world to camera coordinates
Args
P: Nx3 3d points in world coordinates
R: 3x3 Camera rotation matrix
T: 3x1 Camera translation parameters
Returns
X_cam: Nx3 3d points in camera coordinates
"""
assert len(P.shape) == 2
assert P.shape[1] == 3
X_cam = R.dot( P.T - T ) # rotate and translate
return X_cam.T
def camera_to_world_frame(P, R, T):
"""
Inverse of world_to_camera_frame
Args
P: Nx3 points in camera coordinates
R: 3x3 Camera rotation matrix
T: 3x1 Camera translation parameters
Returns
X_cam: Nx3 points in world coordinates
"""
assert len(P.shape) == 2
assert P.shape[1] == 3
X_cam = R.T.dot( P.T ) + T # rotate and translate
return X_cam.T
def load_camera_params( hf, path ):
"""
Load h36m camera parameters
Args
hf: hdf5 open file with h36m cameras data
path: path or key inside hf to the camera we are interested in
Returns
R: 3x3 Camera rotation matrix
T: 3x1 Camera translation parameters
f: (scalar) Camera focal length
c: 2x1 Camera center
k: 3x1 Camera radial distortion coefficients
p: 2x1 Camera tangential distortion coefficients
name: String with camera id
"""
R = hf[ path.format('R') ][:]
R = R.T
T = hf[ path.format('T') ][:]
f = hf[ path.format('f') ][:]
c = hf[ path.format('c') ][:]
k = hf[ path.format('k') ][:]
p = hf[ path.format('p') ][:]
name = hf[ path.format('Name') ][:]
name = "".join( [chr(item) for item in name] )
return R, T, f, c, k, p, name
# def load_cameras( bpath='cameras.h5', subjects=[1,5,6,7,8,9,11] ):
# """
# Loads the cameras of h36m
# Args
# bpath: path to hdf5 file with h36m camera data
# subjects: List of ints representing the subject IDs for which cameras
# are requested
# Returns
# rcams: dictionary of 4 tuples per subject ID containing its camera
# parameters for the 4 h36m cams
# """
# rcams = {}
# with h5py.File(bpath,'r') as hf:
# for s in subjects:
# for c in range(4): # There are 4 cameras in human3.6m
# string = 'subject%d/camera%d/{0}' % (s,c+1)
# rcams[(s, c+1)] = load_camera_params(hf, string)
# return rcams
================================================
FILE: libs/dataset/h36m/data_utils.py
================================================
"""
Utility functions for dealing with Human3.6M dataset.
Some functions are adapted from https://github.com/una-dinosauria/3d-pose-baseline
"""
import os
import numpy as np
import copy
import logging
import matplotlib.pyplot as plt
import torch
from mpl_toolkits.mplot3d import Axes3D
import libs.dataset.h36m.cameras as cameras
import libs.dataset.h36m.pth_dataset as dataset
# Human3.6m IDs for training and testing
TRAIN_SUBJECTS = [1, 5, 6, 7, 8]
TEST_SUBJECTS = [9, 11]
# Use camera coordinate system
camera_frame = True
# Joint names in H3.6M -- data has 32 joints, but only 17 that move;
# these are the indices.
H36M_NAMES = ['']*32
H36M_NAMES[0] = 'Hip'
H36M_NAMES[1] = 'RHip'
H36M_NAMES[2] = 'RKnee'
H36M_NAMES[3] = 'RFoot'
H36M_NAMES[6] = 'LHip'
H36M_NAMES[7] = 'LKnee'
H36M_NAMES[8] = 'LFoot'
H36M_NAMES[12] = 'Spine'
H36M_NAMES[13] = 'Thorax'
H36M_NAMES[14] = 'Neck/Nose'
H36M_NAMES[15] = 'Head'
H36M_NAMES[17] = 'LShoulder'
H36M_NAMES[18] = 'LElbow'
H36M_NAMES[19] = 'LWrist'
H36M_NAMES[25] = 'RShoulder'
H36M_NAMES[26] = 'RElbow'
H36M_NAMES[27] = 'RWrist'
parent_indices = np.array([0, 1, 2, 0, 6, 7, 0, 12, 13, 14, 13, 17, 18, 13, 25, 26])
children_indices = np.array([1, 2, 3, 6, 7, 8, 12, 13, 14, 15, 17, 18, 19, 25, 26, 27])
# Stacked Hourglass produces 16 joints. These are the names.
SH_NAMES = ['']*16
SH_NAMES[0] = 'RFoot'
SH_NAMES[1] = 'RKnee'
SH_NAMES[2] = 'RHip'
SH_NAMES[3] = 'LHip'
SH_NAMES[4] = 'LKnee'
SH_NAMES[5] = 'LFoot'
SH_NAMES[6] = 'Hip'
SH_NAMES[7] = 'Spine'
SH_NAMES[8] = 'Thorax'
SH_NAMES[9] = 'Head'
SH_NAMES[10] = 'RWrist'
SH_NAMES[11] = 'RElbow'
SH_NAMES[12] = 'RShoulder'
SH_NAMES[13] = 'LShoulder'
SH_NAMES[14] = 'LElbow'
SH_NAMES[15] = 'LWrist'
# The .h5 suffix in pose sequence name is just inherited from the original
# naming convention. The '.sh' suffix means stacked hourglass key-point detector
# used in previous works. Here we just use '.sh' to represent key-points obtained
# from any heat-map regression model. We used high-resolution net instead of
# stacked-hourglass model.
def load_ckpt(opt):
cascade = torch.load(os.path.join(opt.ckpt_dir, 'model.th'))
stats = np.load(os.path.join(opt.ckpt_dir, 'stats.npy'), allow_pickle=True).item()
if opt.cuda:
cascade.cuda()
return cascade, stats
def list_remove(list_a, list_b):
"""
Fine all elements of a list A that does not exist in list B.
Args
list_a: list A
list_b: list B
Returns
list_c: result
"""
list_c = []
for item in list_a:
if item not in list_b:
list_c.append(item)
return list_c
def add_virtual_cams(cams, visualize=False):
"""
Deprecated. Add virtual cameras.
"""
# add more cameras to the scene
#R, T, f, c, k, p, name = cams[ (1,1) ]
# plot the position of human subjects
old_cam_num = 4
def add_coordinate_system(ax, origin, system, length=300, new=False):
# draw a coordinate system at a specified origin
origin = origin.reshape(3, 1)
start_points = np.repeat(origin, 3, axis=1)
# system: [v1, v2, v3]
end_points = start_points + system*length
color = ['g', 'y', 'k'] # color for v1, v2 and v3
if new:
color = ['b', 'r', 'g']
def get_args(start_points, end_points):
x = [start_points[0], end_points[0]]
y = [start_points[1], end_points[1]]
z = [start_points[2], end_points[2]]
return x, y, z
for i in range(3):
x, y, z = get_args(start_points[:,i], end_points[:,i])
ax.plot(x, y, z, lw=2, c=color[i])
return
def get_new_camera(system, center, rotation = [0,0,90.]):
from scipy.spatial.transform import Rotation as Rotation
center = center.reshape(3, 1)
start_points = np.repeat(center, 3, axis=1)
end_points = start_points + system
r = Rotation.from_euler('xyz', rotation, degrees=True)
start_points_new = r.as_dcm() @ start_points
end_points_new = r.as_dcm() @ end_points
new_system = [(end_points_new[:,i] - start_points_new[:,i]).reshape(3,1) for i in range(3)]
new_system = np.hstack(new_system)
return new_system, start_points_new[:,0]
# the new cameras are added by rotating one existing camera
# TODO: more rotations
new_cams = cams.copy()
for key in cams.keys():
subject, camera_idx = key
if camera_idx != 1: # only rotate the first camera
continue
R, T, f, c, k, p, name = cams[key]
angles = [80., 130., 270., 320.]
for angle_idx in range(len(angles)):
angle = angles[angle_idx]
new_R, new_T = get_new_camera(R.T, T, [0., 0., angle])
new_cams[(subject, old_cam_num + angle_idx + 1)]\
= (new_R.T, new_T.reshape(3,1), f, c, k, p, name+'new'+str(angle_idx+1))
# visualize cameras used
if visualize:
train_set_3d = np.load('../data/human3.6M/h36m/numpy/threeDPose_train.npy').item()
test_set_3d = np.load('../data/human3.6M/h36m/numpy/threeDPose_test.npy').item()
hips_train = np.vstack(list(train_set_3d.values()))
hips_test = np.vstack(list(test_set_3d.values()))
ax = plt.subplot(111, projection='3d')
chosen = np.random.choice(len(hips_train), 1000, replace=False)
chosen_hips = hips_train[chosen, :3]
ax.plot(chosen_hips[:,0], chosen_hips[:,1], chosen_hips[:,2], 'bo')
chosen = np.random.choice(len(hips_test), 1000, replace=False)
chosen_hips = hips_test[chosen, :3]
ax.plot(chosen_hips[:,0], chosen_hips[:,1], chosen_hips[:,2], 'ro')
ax.set_xlabel("x");ax.set_ylabel("y");ax.set_zlabel("z")
plt.title('Blue dots: Hip positions in the h36m training set. \
Red dots: testing set. \
Old camera coordinates: x-green, y-yellow, z-black \
New camera coordinates: x-blue, y-red, z-green')
plt.pause(0.1)
for key in new_cams.keys():
R, T, f, c, k, p, name = new_cams[key]
# R gives camera basis vectors row-by-row, T gives camera center
if 'new' in name:
new = True
else:
new = False
add_coordinate_system(ax, T, R.T, new=new)
RADIUS = 3000 # space around the subject
xroot, yroot, zroot = 0., 0., 500.
ax.set_xlim3d([-RADIUS+xroot, RADIUS+xroot])
ax.set_zlim3d([-RADIUS+zroot, RADIUS+zroot])
ax.set_ylim3d([-RADIUS+yroot, RADIUS+yroot])
ax.set_aspect("equal")
return new_cams
def down_sample_training_data(train_dict, opt):
"""
Down-sample the training data.
Args
train_dict: python dictionary contraining the training data
opt: experiment options
Returns
train_dict/sampled_dict: a dictionary containing a subset of training data
"""
if opt.ws_name in ['S1', 'S15', 'S156']:
sub_list = [int(opt.ws_name[i]) for i in range(1, len(opt.ws_name))]
keys_to_delete = []
for key in train_dict.keys():
if key[0] not in sub_list:
keys_to_delete.append(key)
for key in keys_to_delete:
del train_dict[key]
return train_dict
elif opt.ws_name in ['0.001S1','0.01S1', '0.05S1', '0.1S1', '0.5S1']:
ratio = float(opt.ws_name.split('S')[0])
# randomly sample a portion of 3D data
sampled_dict = {}
for key in train_dict.keys():
if key[0] != 1:
continue
total = len(train_dict[key])
sampled_num = int(ratio*total)
chosen_indices = np.random.choice(total, sampled_num, replace=False)
sampled_dict[key] = train_dict[key][chosen_indices].copy()
return sampled_dict
else:
raise ValueError('Unknown experiment setting.')
def get_train_dict_3d(opt):
"""
Get the training 3d skeletons as a Python dictionary.
Args
opt: experiment options
Returns
train_dict_3d: a dictionary containing training 3d poses
"""
if not opt.train:
return None
dict_path = os.path.join(opt.data_dir, 'threeDPose_train.npy')
#=========================================================================#
# For real 2D detections, the down-sampling and data augmentation
# are done later in get_train_dict_2d
if opt.twoD_source != 'synthetic':
train_dict_3d = np.load(dict_path, allow_pickle=True).item()
return train_dict_3d
#=========================================================================#
# For synthetic 2D detections (For Protocol P1*), the down-sampling is
# performed here and the data augmentation is assumed to be already done
if opt.evolved_path is not None:
# the data is pre-augmented
train_dict_3d = np.load(opt.evolved_path, allow_pickle=True).item()
elif opt.ws:
# raw training data from Human 3.6M (S15678)
# Down-sample the raw data to simulate an environment with scarce
# training data, which is used in weakly-supervised experiments
train_dict_3d = np.load(dict_path, allow_pickle=True).item()
train_dict_3d = down_sample_training_data(train_dict_3d, opt)
else:
# raw training data from Human 3.6M (S15678)
train_dict_3d = np.load(dict_path, allow_pickle=True).item()
return train_dict_3d
def get_test_dict_3d(opt):
"""
Get the testing 3d skeletons as a Python dictionary.
Args
opt: experiment options
Returns
test_dict_3d: a dictionary containing testing 3d poses
"""
if opt.test_source == 'h36m':
# for h36m
dict_path = os.path.join(opt.data_dir, 'threeDPose_test.npy')
test_dict_3d = np.load(dict_path, allow_pickle=True).item()
else:
raise NotImplementedError
return test_dict_3d
def get_dict_2d(train_dict_3d, test_dict_3d, rcams, ncams, opt):
"""
Prepare 2D training and testing data as Python dictionaries.
Args
train_dict_3d: dictionary containing training 3d poses
test_dict_3d: dictionary containing testing 3d poses
rcams: camera parameters
ncams: number of camera to use
opt: experiment options
Returns
train_dict_2d: a dictionary containing training 2d poses
test_dict_2d: a dictionary containing testing 2d poses
train_dict_3d: the dictionary containing training 3d poses, which may be
updated
"""
if opt.twoD_source == 'synthetic':
# project the 3D key-points to 2D ones
# This type of key-points is used to validate the performance of
# 2D-to-3D networks and the noise of 2D key-point detector is ignored.
# In fact, these 2D key-points are used as ground-truth to train the
# first stage of TAG-net.
if opt.virtual_cams:
ncams *= 2
if opt.train:
train_dict_2d = project_to_cameras(train_dict_3d, rcams, ncams=ncams)
else:
train_dict_2d = None
test_dict_2d = project_to_cameras(test_dict_3d, rcams, ncams=ncams)
elif opt.twoD_source == 'HRN':
# The 2D key-point detections obtained by the heatmap regression model.
# The model uses high-resolution net as backbone and pixel-shuffle super-resolution
# to regress high-resolution heatmaps.
if opt.train:
train_dict_2d = np.load(os.path.join(opt.data_dir, 'twoDPose_HRN_train.npy'), allow_pickle=True).item()
else:
train_dict_2d = None
test_dict_2d = np.load(os.path.join(opt.data_dir, 'twoDPose_HRN_test.npy'), allow_pickle=True).item()
def delete(dic, actions):
keys_to_delete = []
for key in dic.keys():
sub, act, name = key
if act not in actions:
keys_to_delete.append(key)
for key in keys_to_delete:
del dic[key]
return dic
def replace(dic, temp):
for key in dic.keys():
sub, act, name = key
temp_key = (sub, act, name[:-3])
synthetic = temp[temp_key]
assert len(dic[key]) == len(synthetic)
indices = np.random.choice(len(synthetic), int(0.5*len(synthetic)), replace=False)
dic[key][indices] = synthetic[indices].copy()
return dic
# # weakly-supervised experiment
def remove_keys(dic, name_list):
keys_to_delete = []
for key in dic.keys():
if key[0] not in name_list:
keys_to_delete.append(key)
for key in keys_to_delete:
del dic[key]
return dic
# down-sample the data for weakly-supervised experiment
if opt.ws and opt.ws_name in ['S1', 'S15', 'S156']:
sub_list = [int(opt.ws_name[i]) for i in range(1, len(opt.ws_name))]
remove_keys(train_dict_3d, sub_list)
if train_dict_2d is not None:
remove_keys(train_dict_2d, sub_list)
# data augmentation with evolved data
if opt.evolved_path is not None:
evolved_dict_3d = np.load(opt.evolved_path, allow_pickle=True).item()
evolved_dict_2d = project_to_cameras(evolved_dict_3d, rcams, ncams=ncams)
# combine the synthetic 2D-3D pair with the real 2D-3D pair
train_dict_3d = {**train_dict_3d, **evolved_dict_3d}
train_dict_2d = {**train_dict_2d, **evolved_dict_2d}
return train_dict_2d, test_dict_2d, train_dict_3d
def prepare_data_dict(rcams,
opt,
ncams=4,
predict_14=False,
use_nose=True
):
"""
Prepare 2D and 3D data as Python dictionaries.
Args
rcams: camera parameters
opt: experiment options
ncams: number of camera to use
predict_14: whether to predict 14 joints or not
use_nose: whether to use nose joint or not
Returns
data_dic: a dictionary containing training and testing data
data_stats: statistics computed from training data
"""
assert opt.twoD_source in ['synthetic', 'HRN'], 'Unknown 2D key-point type.'
data_dic = {}
# get 3D skeleton data
train_dict_3d = get_train_dict_3d(opt)
test_dict_3d = get_test_dict_3d(opt)
# get 2D key-point data
train_dict_2d, test_dict_2d, train_dict_3d = get_dict_2d(train_dict_3d,
test_dict_3d,
rcams,
ncams,
opt
)
# compute normalization statistics and normalize the 2D data
if opt.train:
complete_train_2d = copy.deepcopy(np.vstack(list(train_dict_2d.values())))
data_mean_2d, data_std_2d, dim_to_ignore_2d, dim_to_use_2d = \
normalization_stats(complete_train_2d,
dim=2,
norm_twoD=opt.norm_twoD,
use_nose=use_nose
)
data_dic['train_set_2d'] = normalize_data(train_dict_2d,
data_mean_2d,
data_std_2d,
dim_to_use_2d,
norm_single=opt.norm_single
)
else:
_, data_stats = load_ckpt(opt)
data_mean_2d, data_std_2d = data_stats['mean_2d'], data_stats['std_2d']
dim_to_use_2d = data_stats['dim_use_2d']
data_dic['test_set_2d'] = normalize_data(test_dict_2d,
data_mean_2d,
data_std_2d,
dim_to_use_2d,
norm_single=opt.norm_single
)
# The 3D joint position is represented in the world coordinate,
# which is converted to camera coordinate system as the regression target
if opt.train:
train_dict_3d = transform_world_to_camera(train_dict_3d, rcams, ncams=ncams)
# apply 3d post-processing (centering around root)
train_dict_3d, train_root_positions = postprocess_3d(train_dict_3d)
test_dict_3d = transform_world_to_camera(test_dict_3d, rcams, ncams=ncams)
test_dict_3d, test_root_positions = postprocess_3d(test_dict_3d)
if opt.train:
# compute normalization statistics and normalize the 3D data
complete_train_3d = copy.deepcopy(np.vstack(list(train_dict_3d.values())))
data_mean_3d, data_std_3d, dim_to_ignore_3d, dim_to_use_3d =\
normalization_stats(complete_train_3d, dim=3, predict_14=predict_14)
data_dic['train_set_3d'] = normalize_data(train_dict_3d,
data_mean_3d,
data_std_3d,
dim_to_use_3d
)
# some joints are not used during training
dim_use_2d = list_remove([i for i in range(len(data_mean_2d))],
list(dim_to_ignore_2d))
dim_use_3d = list_remove([i for i in range(len(data_mean_3d))],
list(dim_to_ignore_3d))
# assemble a dictionary for data statistics
data_stats = {'mean_2d':data_mean_2d,
'std_2d':data_std_2d,
'mean_3d':data_mean_3d,
'std_3d':data_std_3d,
'dim_ignore_2d':dim_to_ignore_2d,
'dim_ignore_3d':dim_to_ignore_3d,
'dim_use_2d':dim_use_2d,
'dim_use_3d':dim_use_3d
}
else:
data_mean_3d, data_std_3d = data_stats['mean_3d'], data_stats['std_3d']
dim_to_use_3d = data_stats['dim_use_3d']
data_dic['test_set_3d'] = normalize_data(test_dict_3d,
data_mean_3d,
data_std_3d,
dim_to_use_3d
)
return data_dic, data_stats
def select_action(dic_2d, dic_3d, action, twoD_source):
"""
Construct sub-dictionaries by specifying which action to use
Args
dic_2d: dictionary containing 2d poses
dic_3d: dictionary containing 3d poses
action: the action to use
twoD_source: how the key-points are generated (synthetic or real)
Returns
dic_2d_action: sub-dictionary containing 2d poses for the specified action
dic_3d_action: sub-dictionary containing 3d poses for the specified action
"""
dic_2d_action = {}
dic_3d_action = {}
for key in dic_2d.keys():
if key[1] == action:
dic_2d_action[key] = dic_2d[key].copy()
if twoD_source == 'synthetic':
key3d = key
else:
key3d = (key[0], key[1], key[2][:-3])
dic_3d_action[key3d] = dic_3d[key3d].copy()
return dic_2d_action, dic_3d_action
def split_action(dic_2d, dic_3d, actions, camera_frame, opt, input_size, output_size):
"""
Generate a list of datasets for each action.
Args
dic_2d: dictionary containing 2d poses
dic_3d: dictionary containing 3d poses
actions: list of defined actions
camera_frame: use camera coordinate system
opt: experiment options
input_size: input vector length
output_size: output vector length
Returns
action_dataset_list: a list of datasets where each element correspond
to one action
"""
action_dataset_list = []
for act_id in range(len(actions)):
action = actions[act_id]
dic_2d_action, dic_3d_action = select_action(dic_2d, dic_3d, action, opt.twoD_source)
eval_input, eval_output = get_all_data(dic_2d_action,
dic_3d_action,
camera_frame,
norm_twoD=opt.norm_twoD,
input_size=input_size,
output_size=output_size)
action_dataset = dataset.PoseDataset(eval_input,
eval_output,
'eval',
action_name=action,
refine_3d=opt.refine_3d)
action_dataset_list.append(action_dataset)
return action_dataset_list
def normalization_stats(complete_data,
dim,
predict_14=False,
norm_twoD=False,
use_nose=False
):
"""
Computes normalization statistics: mean and stdev, dimensions used and ignored
Args
complete_data: nxd np array with poses
dim. integer={2,3} dimensionality of the data
predict_14. boolean. Whether to use only 14 joints
use_nose: whether to use nose or not
Returns
data_mean: np vector with the mean of the data
data_std: np vector with the standard deviation of the data
dimensions_to_ignore: list of dimensions not used in the model
dimensions_to_use: list of dimensions used in the model
"""
if not dim in [2,3]:
raise(ValueError, 'dim must be 2 or 3')
data_mean = np.mean(complete_data, axis=0)
data_std = np.std(complete_data, axis=0)
# Encodes which 17 (or 14) 2d-3d pairs we are predicting
dimensions_to_ignore = []
if dim == 2:
if not use_nose:
dimensions_to_use = np.where(np.array([x != '' and x != 'Neck/Nose' for x in H36M_NAMES]))[0]
else:
dimensions_to_use = np.where(np.array([x != '' for x in H36M_NAMES]))[0]
if norm_twoD:
dimensions_to_use = np.delete(dimensions_to_use, 0)
dimensions_to_use = np.sort(np.hstack((dimensions_to_use*2,
dimensions_to_use*2+1)))
dimensions_to_ignore = np.delete(np.arange(len(H36M_NAMES)*2),
dimensions_to_use)
else:
dimensions_to_use = np.where(np.array([x != '' for x in H36M_NAMES]))[0]
# hip is deleted
# spine and neck are also deleted if predict_14
dimensions_to_use = np.delete(dimensions_to_use, [0,7,9] if predict_14 else 0)
dimensions_to_use = np.sort(np.hstack((dimensions_to_use*3,
dimensions_to_use*3+1,
dimensions_to_use*3+2)))
dimensions_to_ignore = np.delete(np.arange(len(H36M_NAMES)*3),
dimensions_to_use)
return data_mean, data_std, dimensions_to_ignore, dimensions_to_use
def transform_world_to_camera(poses_set, cams, ncams=4):
"""
Transform 3d poses from world coordinate to camera coordinate system
Args
poses_set: dictionary with 3d poses
cams: dictionary with cameras
ncams: number of cameras per subject
Return:
t3d_camera: dictionary with 3d poses in camera coordinate
"""
t3d_camera = {}
for t3dk in sorted(poses_set.keys()):
subj, action, seqname = t3dk
t3d_world = poses_set[t3dk]
for c in range(ncams):
R, T, f, c, k, p, name = cams[(subj, c+1)]
camera_coord = cameras.world_to_camera_frame(np.reshape(t3d_world, [-1, 3]), R, T)
camera_coord = np.reshape(camera_coord, [-1, len(H36M_NAMES)*3])
sname = seqname[:-3]+"."+name+".h5" # e.g.: Waiting 1.58860488.h5
t3d_camera[(subj, action, sname)] = camera_coord
return t3d_camera
def normalize_data(data, data_mean, data_std, dim_to_use, norm_single=False):
"""
Normalizes a dictionary of poses
Args
data: dictionary where values are
data_mean: np vector with the mean of the data
data_std: np vector with the standard deviation of the data
dim_to_use: list of dimensions to keep in the data
norm_single: whether to perform normalization independently for each
sample
Returns
data_out: dictionary with same keys as data, but values have been normalized
"""
data_out = {}
for key in data.keys():
data[ key ] = data[ key ][ :, dim_to_use ]
if norm_single:
# does not use statistics over the whole dataset
temp = data[key]
temp = temp.reshape(len(temp), -1, 2)
mean_x = np.mean(temp[:,:,0], axis=1).reshape(len(temp), 1)
std_x = np.std(temp[:,:,0], axis=1)
mean_y = np.mean(temp[:,:,1], axis=1).reshape(len(temp), 1)
std_y = np.std(temp[:,:,1], axis=1)
denominator = (0.5*(std_x + std_y)).reshape(len(std_x), 1)
temp[:,:,0] = (temp[:,:,0] - mean_x)/denominator
temp[:,:,1] = (temp[:,:,1] - mean_y)/denominator
data_out[key] = temp.reshape(len(temp), -1)
else:
mu = data_mean[dim_to_use]
stddev = data_std[dim_to_use]
data_out[ key ] = np.divide( (data[key] - mu), stddev )
return data_out
def unNormalizeData(normalized_data, data_mean, data_std, dimensions_to_ignore):
"""
Un-normalizes a matrix whose mean has been substracted and that has been
divided by standard deviation. Some dimensions might also be missing.
Args
normalized_data: nxd matrix to unnormalize
data_mean: np vector with the mean of the data
data_std: np vector with the standard deviation of the data
dimensions_to_ignore: list of dimensions that were removed from the original data
Returns
orig_data: the unnormalized data
"""
T = normalized_data.shape[0] # batch size
D = data_mean.shape[0] # dimensionality
orig_data = np.zeros((T, D), dtype=np.float32)
dimensions_to_use = np.array([dim for dim in range(D)
if dim not in dimensions_to_ignore])
orig_data[:, dimensions_to_use] = normalized_data
# multiply times stdev and add the mean
stdMat = data_std.reshape((1, D))
stdMat = np.repeat(stdMat, T, axis=0)
meanMat = data_mean.reshape((1, D))
meanMat = np.repeat(meanMat, T, axis=0)
orig_data = np.multiply(orig_data, stdMat) + meanMat
return orig_data
def define_actions(action):
"""
Given an action string, returns a list of corresponding actions.
Args
action: String. either "all" or one of the h36m actions
Returns
actions: List of strings. Actions to use.
Raises
ValueError: if the action is not a valid action in Human 3.6M
"""
actions = ["Directions",
"Discussion",
"Eating",
"Greeting",
"Phoning",
"Photo",
"Posing",
"Purchases",
"Sitting",
"SittingDown",
"Smoking",
"Waiting",
"WalkDog",
"Walking",
"WalkTogether"
]
if action == "All" or action == "all":
return actions
if not action in actions:
raise( ValueError, "Unrecognized action: %s" % action )
return [action]
def project_to_cameras(poses_set, cams, ncams=4):
"""
Project 3d poses using camera parameters
Args
poses_set: dictionary containing 3d poses
cams: dictionary containing camera parameters
ncams: number of cameras per subject
Returns
t2d: dictionary with 2d poses
"""
t2d = {}
for t3dk in sorted(poses_set.keys()):
subj, a, seqname = t3dk
t3d = poses_set[t3dk]
for cam in range(ncams):
R, T, f, c, k, p, name = cams[(subj, cam+1)]
pts2d, _, _, _, _ = cameras.project_point_radial(np.reshape(t3d, [-1, 3]), R, T, f, c, k, p)
pts2d = np.reshape(pts2d, [-1, len(H36M_NAMES)*2])
sname = seqname[:-3] + "." + name + ".h5" # e.g.: Waiting 1.58860488.h5
t2d[ (subj, a, sname) ] = pts2d
return t2d
def postprocess_3d(poses_set):
"""
Center 3d points around root
Args
poses_set: dictionary with 3d data
Returns
poses_set: dictionary with 3d data centred around root (center hip) joint
root_positions: dictionary with the original 3d position of each pose
"""
root_positions = {}
for k in poses_set.keys():
# Keep track of the global position
root_positions[k] = copy.deepcopy(poses_set[k][:,:3])
# Remove the root from the 3d position
poses = poses_set[k]
poses = poses - np.tile( poses[:,:3], [1, len(H36M_NAMES)] )
poses_set[k] = poses
return poses_set, root_positions
def postprocess_2d(poses_set):
"""
Center 2d points around root
Args
poses_set: dictionary with 2d data
Returns
poses_set: dictionary with 2d data centred around root (center hip) joint
root_positions: dictionary with the original 2d position of each pose
"""
root_positions = {}
for k in poses_set.keys():
# Keep track of the global position
root_positions[k] = copy.deepcopy(poses_set[k][:,:2])
# Remove the root from the 3d position
poses = poses_set[k]
poses = poses - np.tile( poses[:,:2], [1, len(H36M_NAMES)] )
poses_set[k] = poses
return poses_set, root_positions
def get_all_data(data_x,
data_y,
camera_frame,
norm_twoD=False,
input_size=32,
output_size=48
):
"""
Obtain numpy arrays for network inputs/outputs
Args
data_x: dictionary with 2d inputs
data_y: dictionary with 3d expected outputs
camera_frame: whether the 3d data is in camera coordinates
input_size: input vector length for each sample
output_size: output vector length for each sample
Returns
encoder_inputs: numpy array for the input data
decoder_outputs: numpy array for the output data
"""
if norm_twoD:
input_size -= 2
# Figure out how many frames we have
n = 0
for key2d in data_x.keys():
n2d, _ = data_x[ key2d ].shape
n = n + n2d
encoder_inputs = np.zeros((n, input_size), dtype=np.float32)
decoder_outputs = np.zeros((n, output_size), dtype=np.float32)
# Put all the data into big arrays
idx = 0
for key2d in data_x.keys():
(subj, b, fname) = key2d
# keys should be the same if 3d is in camera coordinates
key3d = key2d if (camera_frame) else (subj, b, '{0}.h5'.format(fname.split('.')[0]))
# '-sh' suffix means detected key-points are used
key3d = (subj, b, fname[:-3]) if fname.endswith('-sh') and camera_frame else key3d
n2d, _ = data_x[ key2d ].shape
encoder_inputs[idx:idx+n2d, :] = data_x[ key2d ]
decoder_outputs[idx:idx+n2d, :] = data_y[ key3d ]
idx = idx + n2d
return encoder_inputs, decoder_outputs
def prepare_dataset(opt):
"""
Prepare PyTorch dataset objects used for training 2D-to-3D deep network
Args
opt: experiment options
Returns
train_dataset: training dataset as PyTorch dataset object
eval_dataset: evaluation dataset as PyTorch dataset object
data_stats: dataset statistics computed from the training dataset
action_eval_list: a list of evaluation dataset objects where each
corresponds to one action
"""
# get relevant paths
data_dir = opt.data_dir
cameras_path = os.path.join(data_dir, 'cameras.npy')
# By default, all actions are used
actions = define_actions(opt.actions)
# load camera parameters to project 3D skeleton
rcams = np.load(cameras_path, allow_pickle=True).item()
# produce more camera views by adding virtual cameras if needed
if opt.virtual_cams:
rcams = add_virtual_cams(rcams)
# first prepare Python dictionary containing 2D and 3D data
data_dic, data_stats = prepare_data_dict(rcams,
opt,
predict_14=False
)
input_size = len(data_stats['dim_use_2d'])
output_size = len(data_stats['dim_use_3d'])
if opt.train:
# convert Python dictionary to numpy array
train_input, train_output = get_all_data(data_dic['train_set_2d'],
data_dic['train_set_3d'],
camera_frame,
norm_twoD=opt.norm_twoD,
input_size=input_size,
output_size=output_size
)
# The Numpy arrays are finally used to initialize the dataset objects
train_dataset = dataset.PoseDataset(train_input,
train_output,
'train',
refine_3d = opt.refine_3d
)
else:
train_dataset = None
eval_input, eval_output = get_all_data(data_dic['test_set_2d'],
data_dic['test_set_3d'],
camera_frame,
norm_twoD=opt.norm_twoD,
input_size=input_size,
output_size=output_size
)
eval_dataset = dataset.PoseDataset(eval_input,
eval_output,
'eval',
refine_3d = opt.refine_3d
)
# Create a list of dataset objects for action-wise evaluation
action_eval_list = split_action(data_dic['test_set_2d'],
data_dic['test_set_3d'],
actions,
camera_frame,
opt,
input_size=input_size,
output_size=output_size
)
return train_dataset, eval_dataset, data_stats, action_eval_list
================================================
FILE: libs/dataset/h36m/h36m_pose.py
================================================
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# ------------------------------------------------------------------------------
# Copyright (c) Microsoft
# Licensed under the MIT License.
# Written by Bin Xiao (Bin.Xiao@microsoft.com)
# Modified by Shichao Li (nicholas.li@connect.ust.hk)
# ------------------------------------------------------------------------------
import numpy as np
import copy
import torch
import cv2
import random
from libs.dataset.h36m.pose_dataset import JointsDataset
from libs.hhr.utils.transforms import get_affine_transform
from libs.hhr.utils.transforms import affine_transform
from libs.hhr.utils.transforms import fliplr_joints
import logging
logger = logging.getLogger(__name__)
## Human 3.6M dataset class
class H36MDataset(JointsDataset):
'''
COCO annotation:
"keypoints": {
0: "nose",
1: "left_eye",
2: "right_eye",
3: "left_ear",
4: "right_ear",
5: "left_shoulder",
6: "right_shoulder",
7: "left_elbow",
8: "right_elbow",
9: "left_wrist",
10: "right_wrist",
11: "left_hip",
12: "right_hip",
13: "left_knee",
14: "right_knee",
15: "left_ankle",
16: "right_ankle"
},
"skeleton": [
[16,14],[14,12],[17,15],[15,13],[12,13],[6,12],[7,13], [6,7],[6,8],
[7,9],[8,10],[9,11],[2,3],[1,2],[1,3],[2,4],[3,5],[4,6],[5,7]]
H36M annotation:
H36M_NAMES[0] = 'Hip'
H36M_NAMES[1] = 'RHip'
H36M_NAMES[2] = 'RKnee'
H36M_NAMES[3] = 'RFoot'
H36M_NAMES[4] = 'LHip'
H36M_NAMES[5] = 'LKnee'
H36M_NAMES[6] = 'LFoot'
H36M_NAMES[7] = 'Spine'
H36M_NAMES[8] = 'Thorax'
H36M_NAMES[9] = 'Neck/Nose'
H36M_NAMES[10] = 'Head'
H36M_NAMES[11] = 'LShoulder'
H36M_NAMES[12] = 'LElbow'
H36M_NAMES[13] = 'LWrist'
H36M_NAMES[14] = 'RShoulder'
H36M_NAMES[15] = 'RElbow'
H36M_NAMES[16] = 'RWrist'
"skeleton": [
[0,1], [1,2], [2,3], [0,4], [4,5], [5,6], [0,7], [7,8],
[8,9], [9,10], [8,11], [11,12], [12,13], [8,14], [14,15], [15,16]]
permutation from H36M to COCO:
[9, 7, 8, 0, 10, 11, 14, 12, 15, 13, 16, 4, 1, 5, 2, 6, 3]
permutation to get back:
'''
def __init__(self, cfg, is_train, annot_path, transform=None):
super().__init__(cfg, is_train, transform)
self.nms_thre = cfg.TEST.NMS_THRE
self.image_thre = cfg.TEST.IMAGE_THRE
self.soft_nms = cfg.TEST.SOFT_NMS
self.oks_thre = cfg.TEST.OKS_THRE
self.in_vis_thre = cfg.TEST.IN_VIS_THRE
self.image_width = cfg.MODEL.IMAGE_SIZE[0]
self.image_height = cfg.MODEL.IMAGE_SIZE[1]
self.loss_type = cfg.MODEL.TARGET_TYPE
self.aspect_ratio = self.image_width * 1.0 / self.image_height
self.pixel_std = 200
# path to pre-processed annotation
self.annot_path = annot_path
self.num_joints = 17
self.flip_pairs = [[5, 6], [7, 8], [9, 10], [11, 12], [13, 14], [15, 16]]
self.parent_ids = None
self.upper_body_ids = (0, 1, 2, 4, 5, 6, 7, 8, 9, 10)
self.lower_body_ids = (3, 11, 12, 13, 14, 15, 16)
self.joints_weight = np.ones((self.num_joints,1), np.float32)
self.joints_weight[[7,8,13,14]] = 1.2
self.joints_weight[[9,10,15,16]] = 1.5
## permute joint order for fine-tuning purpose
self.fine_tune_re_order = [9, 7, 8, 0, 10, 11, 14, 12, 15, 13, 16, 4, 1, 5, 2, 6, 3]
self.ratio = float(cfg.MODEL.IMAGE_SIZE[0]/cfg.MODEL.HEATMAP_SIZE[0])
self.db = self._get_db()
logging.info('=> total annotation for images: {}'.format(len(self.db)))
if is_train and cfg.DATASET.SELECT_DATA:
self.db = self.select_data(self.db)
logging.info('=> load {} samples'.format(len(self.db)))
def _get_db(self):
gt_db = np.load(self.annot_path)
# permute joints
for record in gt_db:
record['p_2d'] = record['p_2d'][self.fine_tune_re_order, :]
return gt_db
def get_weights(self):
weights = []
for sample_idx in range(len(self.db)):
path = self.db[sample_idx]['path']
if 'S6' in path or 'S8' in path:
weights.append(1.5)
else:
weights.append(1.0)
return weights
def _box2cs(self, box):
x, y, w, h = box[:4]
return self._xywh2cs(x, y, w, h)
def _xywh2cs(self, x, y, w, h):
center = np.zeros((2), dtype=np.float32)
center[0] = x + w * 0.5
center[1] = y + h * 0.5
if w > self.aspect_ratio * h:
h = w * 1.0 / self.aspect_ratio
elif w < self.aspect_ratio * h:
w = h * self.aspect_ratio
scale = np.array(
[w * 1.0 / self.pixel_std, h * 1.0 / self.pixel_std],
dtype=np.float32)
return center, scale
def __getitem__(self, idx):
db_rec = copy.deepcopy(self.db[idx])
image_file = db_rec['path']
# data_numpy = cv2.imread(
# image_file, cv2.IMREAD_COLOR | cv2.IMREAD_IGNORE_ORIENTATION
# )
# opencv 3
data_numpy = cv2.imread(
image_file, 1 | 128
)
if self.color_rgb:
data_numpy = cv2.cvtColor(data_numpy, cv2.COLOR_BGR2RGB)
if data_numpy is None:
logger.error('=> fail to read {}'.format(image_file))
raise
gitextract_6ign6rd0/
├── .gitignore
├── LICENSE
├── README.md
├── data/
│ └── .gitignore
├── docs/
│ ├── 2DHPE.md
│ ├── ANNOTATOR.md
│ ├── DATASET.md
│ ├── HHR.md
│ ├── TRAINING.md
│ └── Zoo.md
├── examples/
│ ├── h36m2Dpose/
│ │ └── cfgs.yaml
│ ├── h36m2Dpose.py
│ └── inference.py
├── libs/
│ ├── __init__.py
│ ├── annotator/
│ │ ├── __init__.py
│ │ ├── angle.py
│ │ ├── fit_3d.py
│ │ ├── smpl-spec-list.txt
│ │ ├── smpl_webuser/
│ │ │ ├── LICENSE.txt
│ │ │ ├── README.txt
│ │ │ ├── __init__.py
│ │ │ ├── hello_world/
│ │ │ │ ├── hello_smpl.py
│ │ │ │ └── render_smpl.py
│ │ │ ├── lbs.py
│ │ │ ├── posemapper.py
│ │ │ ├── serialization.py
│ │ │ └── verts.py
│ │ └── smplify/
│ │ ├── __init__.py
│ │ ├── fit_3d.py
│ │ ├── lib/
│ │ │ ├── __init__.py
│ │ │ ├── capsule_body.py
│ │ │ ├── capsule_ch.py
│ │ │ ├── max_mixture_prior.py
│ │ │ ├── robustifiers.py
│ │ │ └── sphere_collisions.py
│ │ └── render_model.py
│ ├── dataset/
│ │ ├── __init__.py
│ │ └── h36m/
│ │ ├── __init__.py
│ │ ├── cameras.py
│ │ ├── data_utils.py
│ │ ├── h36m_pose.py
│ │ ├── pose_dataset.py
│ │ └── pth_dataset.py
│ ├── evolution/
│ │ ├── __init__.py
│ │ ├── genetic.py
│ │ └── parameter.py
│ ├── hhr/
│ │ ├── __init__.py
│ │ ├── config/
│ │ │ ├── __init__.py
│ │ │ ├── default.py
│ │ │ └── models.py
│ │ ├── core/
│ │ │ ├── __init__.py
│ │ │ ├── evaluate.py
│ │ │ ├── function.py
│ │ │ ├── inference.py
│ │ │ └── loss.py
│ │ └── utils/
│ │ ├── __init__.py
│ │ ├── transforms.py
│ │ ├── utils.py
│ │ └── vis.py
│ ├── model/
│ │ ├── __init__.py
│ │ ├── model.py
│ │ ├── pose_hrnet.py
│ │ └── pose_resnet.py
│ ├── optimizer/
│ │ └── __init__.py
│ ├── parser/
│ │ ├── __init__.py
│ │ └── parse.py
│ ├── skeleton/
│ │ ├── __init__.py
│ │ └── anglelimits.py
│ ├── trainer/
│ │ ├── __init__.py
│ │ └── trainer.py
│ ├── utils/
│ │ ├── __init__.py
│ │ └── utils.py
│ └── visualization/
│ ├── __init__.py
│ └── viz.py
├── resources/
│ └── .gitignore
├── spec-list.txt
└── tools/
├── 2Dto3Dnet.py
├── annotate_2D.py
├── annotate_3D.py
├── evolve.py
└── imgto2Dnet.py
SYMBOL INDEX (336 symbols across 43 files)
FILE: examples/h36m2Dpose.py
function show2Dpose (line 41) | def show2Dpose(vals, ax, lcolor="#3498db", rcolor="#e74c3c", add_labels=...
function parse_args (line 46) | def parse_args():
function xywh2cs (line 60) | def xywh2cs(x, y, w, h, aspect_ratio=0.75):
function gather_inputs (line 73) | def gather_inputs(args, logger, image_size = (288, 384)):
function unnormalize (line 105) | def unnormalize(tensor):
function visualize (line 112) | def visualize(inputs, model):
function main (line 125) | def main():
FILE: examples/inference.py
function draw_skeleton (line 54) | def draw_skeleton(ax, skeleton, gt=False, add_index=True):
function normalize (line 74) | def normalize(skeleton, re_order=None):
function get_pred (line 89) | def get_pred(cascade, data):
function show3Dpose (line 104) | def show3Dpose(channels,
function re_order (line 145) | def re_order(skeleton):
function plot_3d_ax (line 151) | def plot_3d_ax(ax,
function adjust_figure (line 162) | def adjust_figure(left = 0,
FILE: libs/annotator/angle.py
function gram_schmidt_columns (line 27) | def gram_schmidt_columns(X):
function normalize (line 44) | def normalize(vector):
function get_basis1 (line 47) | def get_basis1(skeleton):
function get_normal (line 63) | def get_normal(x1, a, x):
function to_spherical (line 74) | def to_spherical(xyz):
function to_xyz (line 90) | def to_xyz(rthetaphi):
FILE: libs/annotator/fit_3d.py
function main (line 29) | def main(opt):
FILE: libs/annotator/smpl_webuser/lbs.py
function global_rigid_transformation (line 27) | def global_rigid_transformation(pose, J, kintree_table, xp):
function verts_core (line 63) | def verts_core(pose, v, J, weights, kintree_table, want_Jtr=False, xp=ch...
FILE: libs/annotator/smpl_webuser/posemapper.py
class Rodrigues (line 25) | class Rodrigues(ch.Ch):
method compute_r (line 28) | def compute_r(self):
method compute_dr_wrt (line 31) | def compute_dr_wrt(self, wrt):
function lrotmin (line 36) | def lrotmin(p):
function posemap (line 45) | def posemap(s):
FILE: libs/annotator/smpl_webuser/serialization.py
function save_model (line 32) | def save_model(model, fname):
function backwards_compatibility_replacements (line 52) | def backwards_compatibility_replacements(dd):
function ready_arguments (line 77) | def ready_arguments(fname_or_dict):
function load_model (line 116) | def load_model(fname_or_dict):
FILE: libs/annotator/smpl_webuser/verts.py
function ischumpy (line 30) | def ischumpy(x): return hasattr(x, 'dterms')
function verts_decorated (line 32) | def verts_decorated(trans, pose,
function verts_core (line 92) | def verts_core(pose, v, J, weights, kintree_table, bs_style, want_Jtr=Fa...
FILE: libs/annotator/smplify/fit_3d.py
function guess_init (line 59) | def guess_init(model, focal_length, j2d, init_pose):
function initialize_camera (line 92) | def initialize_camera(model,
function optimize_on_joints (line 193) | def optimize_on_joints(j2d,
function run_single_fit (line 402) | def run_single_fit(img,
function main (line 499) | def main(base_dir,
FILE: libs/annotator/smplify/lib/capsule_body.py
function get_capsules (line 87) | def get_capsules(model, wrt_betas=None, length_regs=None, rad_regs=None):
function set_sphere_centers (line 161) | def set_sphere_centers(capsule, floor=True):
function capsule_dist (line 189) | def capsule_dist(capsule0, capsule1, alpha=.3, increase_hand=True):
function get_capsule_bweights (line 209) | def get_capsule_bweights(vs):
function get_sphere_bweights (line 220) | def get_sphere_bweights(sph_vs, capsules):
FILE: libs/annotator/smplify/lib/capsule_ch.py
class Capsule (line 67) | class Capsule(object):
method __init__ (line 69) | def __init__(self, t, rod, rad, length):
method set_sphere_centers (line 86) | def set_sphere_centers(self, floor=False):
FILE: libs/annotator/smplify/lib/max_mixture_prior.py
class MaxMixtureComplete (line 17) | class MaxMixtureComplete(ch.Ch):
method on_changed (line 24) | def on_changed(self, which):
method compute_r (line 38) | def compute_r(self):
method compute_dr_wrt (line 44) | def compute_dr_wrt(self, wrt):
class MaxMixtureCompleteWrapper (line 63) | class MaxMixtureCompleteWrapper(object):
method __init__ (line 66) | def __init__(self, means, precs, weights, prefix):
method __call__ (line 72) | def __call__(self, x):
class MaxMixtureCompletePrior (line 82) | class MaxMixtureCompletePrior(object):
method __init__ (line 85) | def __init__(self, n_gaussians=8, prefix=3):
method create_prior_from_cmu (line 90) | def create_prior_from_cmu(self):
method get_gmm_prior (line 117) | def get_gmm_prior(self):
FILE: libs/annotator/smplify/lib/robustifiers.py
function GMOf (line 19) | def GMOf(x, sigma):
class SignedSqrt (line 28) | class SignedSqrt(Ch):
method compute_r (line 32) | def compute_r(self):
method compute_dr_wrt (line 35) | def compute_dr_wrt(self, wrt):
class GMOfInternal (line 44) | class GMOfInternal(Ch):
method on_changed (line 47) | def on_changed(self, which):
method compute_r (line 54) | def compute_r(self):
method compute_dr_wrt (line 59) | def compute_dr_wrt(self, wrt):
FILE: libs/annotator/smplify/lib/sphere_collisions.py
class SphereCollisions (line 21) | class SphereCollisions(ch.Ch):
method update_capsules_and_centers (line 25) | def update_capsules_and_centers(self):
method update_pose (line 60) | def update_pose(self):
method get_objective (line 69) | def get_objective(self):
method compute_r (line 74) | def compute_r(self):
method compute_dr_wrt (line 77) | def compute_dr_wrt(self, wrt):
method on_changed (line 83) | def on_changed(self, which):
FILE: libs/annotator/smplify/render_model.py
function _create_renderer (line 24) | def _create_renderer(w=640,
function _rotateY (line 45) | def _rotateY(points, angle):
function simple_renderer (line 54) | def simple_renderer(rn, verts, faces, yrot=np.radians(120)):
function get_alpha (line 93) | def get_alpha(imtmp, bgval=1.):
function render_model (line 104) | def render_model(verts, faces, w, h, cam, near=0.5, far=25, img=None):
FILE: libs/dataset/h36m/cameras.py
function project_point_radial (line 8) | def project_point_radial( P, R, T, f, c, k, p ):
function world_to_camera_frame (line 51) | def world_to_camera_frame(P, R, T):
function camera_to_world_frame (line 70) | def camera_to_world_frame(P, R, T):
function load_camera_params (line 89) | def load_camera_params( hf, path ):
FILE: libs/dataset/h36m/data_utils.py
function load_ckpt (line 72) | def load_ckpt(opt):
function list_remove (line 79) | def list_remove(list_a, list_b):
function add_virtual_cams (line 95) | def add_virtual_cams(cams, visualize=False):
function down_sample_training_data (line 184) | def down_sample_training_data(train_dict, opt):
function get_train_dict_3d (line 218) | def get_train_dict_3d(opt):
function get_test_dict_3d (line 253) | def get_test_dict_3d(opt):
function get_dict_2d (line 270) | def get_dict_2d(train_dict_3d, test_dict_3d, rcams, ncams, opt):
function prepare_data_dict (line 354) | def prepare_data_dict(rcams,
function select_action (line 455) | def select_action(dic_2d, dic_3d, action, twoD_source):
function split_action (line 480) | def split_action(dic_2d, dic_3d, actions, camera_frame, opt, input_size,...
function normalization_stats (line 514) | def normalization_stats(complete_data,
function transform_world_to_camera (line 563) | def transform_world_to_camera(poses_set, cams, ncams=4):
function normalize_data (line 585) | def normalize_data(data, data_mean, data_std, dim_to_use, norm_single=Fa...
function unNormalizeData (line 620) | def unNormalizeData(normalized_data, data_mean, data_std, dimensions_to_...
function define_actions (line 647) | def define_actions(action):
function project_to_cameras (line 683) | def project_to_cameras(poses_set, cams, ncams=4):
function postprocess_3d (line 706) | def postprocess_3d(poses_set):
function postprocess_2d (line 726) | def postprocess_2d(poses_set):
function get_all_data (line 746) | def get_all_data(data_x,
function prepare_dataset (line 793) | def prepare_dataset(opt):
FILE: libs/dataset/h36m/h36m_pose.py
class H36MDataset (line 26) | class H36MDataset(JointsDataset):
method __init__ (line 78) | def __init__(self, cfg, is_train, annot_path, transform=None):
method _get_db (line 114) | def _get_db(self):
method get_weights (line 121) | def get_weights(self):
method _box2cs (line 131) | def _box2cs(self, box):
method _xywh2cs (line 135) | def _xywh2cs(self, x, y, w, h):
method __getitem__ (line 150) | def __getitem__(self, idx):
method generate_target (line 239) | def generate_target(self, joints, joints_vis):
FILE: libs/dataset/h36m/pose_dataset.py
class JointsDataset (line 29) | class JointsDataset(Dataset):
method __init__ (line 30) | def __init__(self, cfg, is_train, root=None, image_set=None, transform...
method _get_db (line 60) | def _get_db(self):
method evaluate (line 63) | def evaluate(self, cfg, preds, output_dir, *args, **kwargs):
method half_body_transform (line 66) | def half_body_transform(self, joints, joints_vis):
method __len__ (line 111) | def __len__(self,):
method __getitem__ (line 114) | def __getitem__(self, idx):
method select_data (line 201) | def select_data(self, db):
method generate_target (line 234) | def generate_target(self, joints, joints_vis):
FILE: libs/dataset/h36m/pth_dataset.py
function normalize (line 8) | def normalize(vec):
function unNormalizeData (line 12) | def unNormalizeData(normalized_data, data_mean, data_std, dimensions_to_...
class PoseDataset (line 42) | class PoseDataset(torch.utils.data.Dataset):
method __init__ (line 43) | def __init__(self, array_2d, array_3d, split, action_name=None, refine...
method __len__ (line 61) | def __len__(self):
method __getitem__ (line 64) | def __getitem__(self, idx):
method set_stage (line 72) | def set_stage(self, stage_idx):
method stage_update (line 76) | def stage_update(self, model, stats, opt, verbose=False):
FILE: libs/evolution/genetic.py
function get_subtree (line 94) | def get_subtree(joint_idx, children_idx):
function swap_bones (line 108) | def swap_bones(bones_father, bones_mother, root_idx):
function get_bone_length (line 116) | def get_bone_length(skeleton):
function get_h36m_bone_length (line 124) | def get_h36m_bone_length(visualize=True):
function get_random_rotation (line 158) | def get_random_rotation(sigma=60.):
function rotate_bone_random (line 169) | def rotate_bone_random(bone, sigma=10.):
function rotate_pose_random (line 174) | def rotate_pose_random(pose=None, sigma=60.):
function re_order (line 188) | def re_order(skeleton):
function set_z (line 192) | def set_z(pose, target):
function modify_pose (line 201) | def modify_pose(skeleton, local_bones, bone_length, ro=False):
function exploration (line 210) | def exploration(father, mother, opt, post_processing=True):
function show3Dpose (line 281) | def show3Dpose(channels,
function choose_best (line 362) | def choose_best(population, fraction = 0.02, method='random'):
function project_to_cameras (line 377) | def project_to_cameras(poses, cams):
function transform_world_to_camera (line 393) | def transform_world_to_camera(poses, cams):
function normalize (line 406) | def normalize(data, mean=None, std=None):
function unnormalize (line 416) | def unnormalize(data, mean, std):
function postprocess_3d (line 419) | def postprocess_3d( poses):
function calc_errors (line 422) | def calc_errors(pred_poses, gt_poses, protocol='mpjpe'):
function to_numpy (line 435) | def to_numpy(tensor):
function get_prediction (line 438) | def get_prediction(cascade, data):
function get_score (line 449) | def get_score(model_file, candidates):
function active_select (line 491) | def active_select(model_file, candidates, ratio):
function cast_to_float (line 501) | def cast_to_float(dic, dtype=np.float32):
function xyz2spherical (line 507) | def xyz2spherical(xyz):
function spherical2xyz (line 517) | def spherical2xyz(rphitheta):
function position_to_angle (line 540) | def position_to_angle(skeletons):
function angle_to_position (line 560) | def angle_to_position(rphitheta, skeletons):
function mutate_bone_length (line 573) | def mutate_bone_length(population, opt, gen_idx, method='simple'):
function one_iteration (line 613) | def one_iteration(population, opt, model_file=None):
function get_save_path (line 646) | def get_save_path(opt, gen_idx):
function split_and_save (line 656) | def split_and_save(final_poses, parameters, gen_idx):
function save_results (line 668) | def save_results(poses, opt, gen_idx):
function evolution (line 676) | def evolution(initial_population, opt, model_file=None):
FILE: libs/evolution/parameter.py
function parse_arg (line 6) | def parse_arg():
FILE: libs/hhr/config/default.py
function update_config (line 128) | def update_config(cfg, args):
FILE: libs/hhr/core/evaluate.py
function calc_dists (line 17) | def calc_dists(preds, target, normalize):
function dist_acc (line 32) | def dist_acc(dists, thr=0.5):
function accuracy (line 42) | def accuracy(output, target, hm_type='gaussian', thr=0.5):
function get_distance (line 77) | def get_distance(gt, pred):
function accuracy_pixel (line 85) | def accuracy_pixel(output, meta_data, image_size = (288.0, 384.0), arg_m...
FILE: libs/hhr/core/function.py
function test_transformation (line 27) | def test_transformation(meta_data, image_size=(288, 384)):
function train (line 49) | def train(config, train_loader, model, criterion, optimizer, epoch,
function validate (line 116) | def validate(config, val_loader, val_dataset, model, criterion, output_dir,
function validate_pixel (line 234) | def validate_pixel(config, val_loader, val_dataset, model, criterion, ou...
function _print_name_value (line 334) | def _print_name_value(name_value, full_arch_name):
class AverageMeter (line 354) | class AverageMeter(object):
method __init__ (line 356) | def __init__(self):
method reset (line 359) | def reset(self):
method update (line 365) | def update(self, val, n=1):
FILE: libs/hhr/core/inference.py
function get_max_preds (line 19) | def get_max_preds(batch_heatmaps):
function get_final_preds (line 50) | def get_final_preds(config, batch_heatmaps, center, scale):
function get_max_preds_soft (line 82) | def get_max_preds_soft(batch_heatmaps):
FILE: libs/hhr/core/loss.py
class JointsMSELoss (line 18) | class JointsMSELoss(nn.Module):
method __init__ (line 19) | def __init__(self, use_target_weight):
method forward (line 24) | def forward(self, output, target, target_weight):
class JointsOHKMMSELoss (line 45) | class JointsOHKMMSELoss(nn.Module):
method __init__ (line 46) | def __init__(self, use_target_weight, topk=8):
method ohkm (line 52) | def ohkm(self, loss):
method forward (line 64) | def forward(self, output, target, target_weight):
function get_max_preds_soft_pt (line 90) | def get_max_preds_soft_pt(batch_heatmaps):
class JointsCoordinateLoss (line 116) | class JointsCoordinateLoss(nn.Module):
method __init__ (line 117) | def __init__(self, use_target_weight, loss_type='sl1', image_size=(384...
method forward (line 124) | def forward(self, output, target, target_weight):
class WingLoss (line 139) | class WingLoss(nn.Module):
method __init__ (line 140) | def __init__(self, use_target_weight, width=5, curvature=0.5, image_si...
method forward (line 147) | def forward(self, output, target, target_weight):
FILE: libs/hhr/utils/transforms.py
function flip_back (line 16) | def flip_back(output_flipped, matched_parts):
function fliplr_joints (line 33) | def fliplr_joints(joints, joints_vis, width, matched_parts):
function transform_preds (line 50) | def transform_preds(coords, center, scale, output_size):
function get_affine_transform (line 58) | def get_affine_transform(
function affine_transform (line 93) | def affine_transform(pt, t):
function affine_transform_modified (line 98) | def affine_transform_modified(pts, t):
function get_3rd_point (line 104) | def get_3rd_point(a, b):
function get_dir (line 109) | def get_dir(src_point, rot_rad):
function crop (line 119) | def crop(img, center, scale, output_size, rot=0):
FILE: libs/hhr/utils/utils.py
function create_logger (line 23) | def create_logger(cfg, cfg_name, phase='train'):
function get_optimizer (line 61) | def get_optimizer(cfg, model):
function save_checkpoint (line 80) | def save_checkpoint(states, is_best, output_dir,
function get_model_summary (line 88) | def get_model_summary(model, *input_tensors, item_length=26, verbose=Fal...
FILE: libs/hhr/utils/vis.py
function save_batch_image_with_joints (line 21) | def save_batch_image_with_joints(batch_image, batch_joints, batch_joints...
function save_batch_heatmaps (line 55) | def save_batch_heatmaps(batch_image, batch_heatmaps, file_name,
function save_debug_images (line 120) | def save_debug_images(config, input, meta, target, joints_pred, output,
FILE: libs/model/model.py
class ResidualBlock (line 8) | class ResidualBlock(nn.Module):
method __init__ (line 12) | def __init__(self, linear_size, p_dropout=0.5, kaiming=False, leaky=Fa...
method forward (line 31) | def forward(self, x):
class FCModel (line 46) | class FCModel(nn.Module):
method __init__ (line 47) | def __init__(self,
method forward (line 100) | def forward(self, x):
method get_representation (line 105) | def get_representation(self, x):
function get_model (line 119) | def get_model(stage_id,
function prepare_optim (line 141) | def prepare_optim(model, opt):
function get_cascade (line 162) | def get_cascade():
FILE: libs/model/pose_hrnet.py
function conv3x3 (line 22) | def conv3x3(in_planes, out_planes, stride=1):
class BasicBlock (line 28) | class BasicBlock(nn.Module):
method __init__ (line 31) | def __init__(self, inplanes, planes, stride=1, downsample=None):
method forward (line 41) | def forward(self, x):
class Bottleneck (line 60) | class Bottleneck(nn.Module):
method __init__ (line 63) | def __init__(self, inplanes, planes, stride=1, downsample=None):
method forward (line 78) | def forward(self, x):
class HighResolutionModule (line 101) | class HighResolutionModule(nn.Module):
method __init__ (line 102) | def __init__(self, num_branches, blocks, num_blocks, num_inchannels,
method _check_branches (line 119) | def _check_branches(self, num_branches, blocks, num_blocks,
method _make_one_branch (line 139) | def _make_one_branch(self, branch_index, block, num_blocks, num_channels,
method _make_branches (line 177) | def _make_branches(self, num_branches, block, num_blocks, num_channels):
method _make_fuse_layers (line 187) | def _make_fuse_layers(self):
method get_num_inchannels (line 244) | def get_num_inchannels(self):
method forward (line 247) | def forward(self, x):
class PoseHighResolutionNet (line 274) | class PoseHighResolutionNet(nn.Module):
method __init__ (line 276) | def __init__(self, cfg, **kwargs):
method _make_transition_layer (line 341) | def _make_transition_layer(
method _make_layer (line 382) | def _make_layer(self, block, planes, blocks, stride=1):
method _make_stage (line 401) | def _make_stage(self, layer_config, num_inchannels,
method forward (line 433) | def forward(self, x):
method init_weights (line 472) | def init_weights(self, pretrained=''):
method load_my_state_dict (line 504) | def load_my_state_dict(self, state_dict):
function get_pose_net (line 512) | def get_pose_net(cfg, is_train, **kwargs):
FILE: libs/model/pose_resnet.py
function conv3x3 (line 22) | def conv3x3(in_planes, out_planes, stride=1):
class BasicBlock (line 30) | class BasicBlock(nn.Module):
method __init__ (line 33) | def __init__(self, inplanes, planes, stride=1, downsample=None):
method forward (line 43) | def forward(self, x):
class Bottleneck (line 62) | class Bottleneck(nn.Module):
method __init__ (line 65) | def __init__(self, inplanes, planes, stride=1, downsample=None):
method forward (line 80) | def forward(self, x):
class PoseResNet (line 103) | class PoseResNet(nn.Module):
method __init__ (line 105) | def __init__(self, block, layers, cfg, **kwargs):
method _make_layer (line 136) | def _make_layer(self, block, planes, blocks, stride=1):
method _get_deconv_cfg (line 153) | def _get_deconv_cfg(self, deconv_kernel, index):
method _make_deconv_layer (line 166) | def _make_deconv_layer(self, num_layers, num_filters, num_kernels):
method forward (line 193) | def forward(self, x):
method init_weights (line 209) | def init_weights(self, pretrained=''):
function get_pose_net (line 261) | def get_pose_net(cfg, is_train, **kwargs):
FILE: libs/parser/parse.py
function parse_arg (line 3) | def parse_arg():
FILE: libs/skeleton/anglelimits.py
function is_valid_local (line 105) | def is_valid_local(skeleton_local, return_ang = False):
function is_valid (line 151) | def is_valid(skeleton, return_ang = False, camera = False):
function normalize (line 173) | def normalize(vector):
function to_spherical (line 179) | def to_spherical(xyz):
function to_xyz (line 194) | def to_xyz(rthetaphi):
function test_coordinate_conversion (line 210) | def test_coordinate_conversion():
function gram_schmidt_columns (line 225) | def gram_schmidt_columns(X):
function direction_check (line 245) | def direction_check(system, v1, v2, v3):
function get_normal (line 254) | def get_normal(x1, a, x):
function get_basis1 (line 268) | def get_basis1(skeleton):
function to_local (line 285) | def to_local(skeleton):
function to_global (line 354) | def to_global(skeleton, bones_local, cache=False):
function test_global_local_conversion (line 420) | def test_global_local_conversion():
function show3Dpose (line 437) | def show3Dpose(channels, ax, lcolor="#3498db", rcolor="#e74c3c", add_lab...
function get_histogram2d (line 489) | def get_histogram2d(angles, validmap=None, smooth=True):
function sample_from_histogram (line 505) | def sample_from_histogram(histogram,
function histogram_transform (line 537) | def histogram_transform(histogram, gamma):
function smooth_histogram2d (line 547) | def smooth_histogram2d(data):
function decorate_axis (line 578) | def decorate_axis(ax, title=None):
function adjust_figure (line 585) | def adjust_figure(left = 0,
function plot_relative_poses (line 595) | def plot_relative_poses(skeletons, cameras=None):
function plot_distribution (line 622) | def plot_distribution(H_temp,
function get_skeleton (line 688) | def get_skeleton(bones, pose, bone_length=template_bone_lengths):
function test_get_skeleton (line 698) | def test_get_skeleton():
function grow_from_torso (line 712) | def grow_from_torso(poses, all_angles, cache=False):
function test_grow_from_torso (line 741) | def test_grow_from_torso():
function sample_lower_limbs (line 759) | def sample_lower_limbs(angles, bin_size=3):
function sample_upper_limbs (line 785) | def sample_upper_limbs(angles, sample_num):
FILE: libs/trainer/trainer.py
function train_cascade (line 13) | def train_cascade(train_dataset, eval_dataset, stats, action_eval_list, ...
function evaluate_cascade (line 73) | def evaluate_cascade(cascade,
function logger_print (line 117) | def logger_print(epoch,
function train (line 136) | def train(train_dataset,
function evaluate_action_wise (line 220) | def evaluate_action_wise(dataset_list, model, stats, opt):
function align_skeleton (line 253) | def align_skeleton(skeletons_pred, skeletons_gt, num_of_joints):
function evaluate (line 268) | def evaluate(eval_dataset,
FILE: libs/utils/utils.py
function save_ckpt (line 22) | def save_ckpt(opt, record, stats):
function load_ckpt (line 42) | def load_ckpt(opt):
function list_remove (line 49) | def list_remove(list_a, list_b):
function get_all_data (line 57) | def get_all_data(data_x, data_y, camera_frame):
function adjust_figure (line 94) | def adjust_figure(left = 0, right = 1, bottom = 0.01, top = 0.95,
function visualize (line 99) | def visualize(eval_dataset, model, stats, opt, save=False, save_dir=None):
function temp_visualize (line 162) | def temp_visualize(eval_dataset, model, stats, opt):
function visualize_cascade (line 197) | def visualize_cascade(eval_dataset, cascade, stats, opt, save=False, sav...
function compute_similarity_transform (line 296) | def compute_similarity_transform(X, Y, compute_optimal_scale=False):
FILE: libs/visualization/viz.py
function show3Dpose (line 12) | def show3Dpose(channels, ax, lcolor="#3498db", rcolor="#e74c3c", add_lab...
function show2Dpose (line 81) | def show2Dpose(channels, ax, lcolor="#3498db", rcolor="#e74c3c", add_lab...
FILE: tools/2Dto3Dnet.py
function main (line 20) | def main():
FILE: tools/annotate_2D.py
function initialization (line 54) | def initialization(opt):
function plot_image (line 67) | def plot_image(img_path):
function onclick (line 78) | def onclick(event):
function onrelease (line 82) | def onrelease(event):
function save_results (line 102) | def save_results():
function onkey (line 106) | def onkey(event):
function main (line 134) | def main(opt):
FILE: tools/annotate_3D.py
function press (line 79) | def press(event):
function show3Dpose (line 126) | def show3Dpose(channels,
function to_local (line 178) | def to_local(skeleton):
function update_line (line 254) | def update_line(line_idx, parent_idx, child_idx):
function update_global (line 272) | def update_global(angle_idx):
function rotate_global (line 287) | def rotate_global(rot):
function update_skeleton (line 302) | def update_skeleton(angle_idx, key_name):
function update_projection (line 354) | def update_projection():
function save_skeleton (line 370) | def save_skeleton():
function visualize (line 379) | def visualize(pose, skeleton, img):
function create_python3_file (line 410) | def create_python3_file(opt):
function main (line 424) | def main(opt):
FILE: tools/evolve.py
function cast_to_float (line 17) | def cast_to_float(dic, dtype=np.float32):
function random_rotation (line 23) | def random_rotation(pose, sigma=360):
function initialize_population (line 34) | def initialize_population(data_dic, opt):
function initialize_model_file (line 69) | def initialize_model_file(opt):
function split_and_save (line 81) | def split_and_save(evolved_population):
function visualize (line 108) | def visualize(initial_population, evolved_population):
function main (line 136) | def main():
FILE: tools/imgto2Dnet.py
function parse_args (line 48) | def parse_args():
function main (line 83) | def main():
Condensed preview — 81 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (480K chars).
[
{
"path": ".gitignore",
"chars": 264,
"preview": "#/*\n**/__pycache__\n.spyproject/\n/libs/annotator/fitted\n/libs/annotator/smplify/models\n/model\n/examples/imgs\n/examples/ex"
},
{
"path": "LICENSE",
"chars": 1067,
"preview": "MIT License\n\nCopyright (c) 2020 Shichao Li\n\nPermission is hereby granted, free of charge, to any person obtaining a copy"
},
{
"path": "README.md",
"chars": 7699,
"preview": "[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/cascaded-deep-monocular-3d-human-pose-1"
},
{
"path": "data/.gitignore",
"chars": 71,
"preview": "# Ignore everything in this directory\n*\n# Except this file\n!.gitignore\n"
},
{
"path": "docs/2DHPE.md",
"chars": 1046,
"preview": "The pre-trained model can be downloaded from [here](https://drive.google.com/file/d/1NjQFCz0GdS7oIdYrK5ouxEI07wYYh4r8/vi"
},
{
"path": "docs/ANNOTATOR.md",
"chars": 4186,
"preview": "The annotator is composed of three parts:\n1. 2D annotation: interactively annotate 2D key-points for RGB images\n2. 3D pa"
},
{
"path": "docs/DATASET.md",
"chars": 1451,
"preview": "## Download\nYou can access the dataset [here](https://drive.google.com/file/d/1JRJuL69J0drZOAUT8VDK5ywmxt-Gm7s-/view?usp"
},
{
"path": "docs/HHR.md",
"chars": 2233,
"preview": "## Data Preparation\nPlease prepare data as instructed in the model training [sub-page](https://github.com/Nicholasli1995"
},
{
"path": "docs/TRAINING.md",
"chars": 5699,
"preview": "## Data Preparation\nSimilar to other repositories ([SimpleBaseline](https://github.com/una-dinosauria/3d-pose-baseline),"
},
{
"path": "docs/Zoo.md",
"chars": 12952,
"preview": "This page presents model performance on H36M under various settings. Pre-trained models and instructions for reproductio"
},
{
"path": "examples/h36m2Dpose/cfgs.yaml",
"chars": 1035,
"preview": "CUDNN:\n BENCHMARK: true\n DETERMINISTIC: false\n ENABLED: true\nGPUS: (0,)\n\nMODEL:\n NAME: pose_hrnet\n NUM_JOINTS: 17\n "
},
{
"path": "examples/h36m2Dpose.py",
"chars": 5530,
"preview": "\"\"\"\nExamplar code showing how to use pre-trained heatmap regression model H() to \nperform 2D pose estimation on Human 3."
},
{
"path": "examples/inference.py",
"chars": 7540,
"preview": "\"\"\"\nAm examplar script showing inference on the newly collected images in U3DPW. \n\"\"\"\n\nimport sys\nsys.path.append(\"../\")"
},
{
"path": "libs/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/annotator/__init__.py",
"chars": 94,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n#import libs.dataset.h36m\n\n"
},
{
"path": "libs/annotator/angle.py",
"chars": 3356,
"preview": "import numpy as np \nimport scipy.io as sio \nimport matplotlib.pyplot as plt \nfrom mpl_toolkits.mplot3d import Axes3D \n\nb"
},
{
"path": "libs/annotator/fit_3d.py",
"chars": 3447,
"preview": "\"\"\"\nObtain 3D skeleton with 2D key-points as inputs using SMPLify\nPlease run this script in Python 2 environment for now"
},
{
"path": "libs/annotator/smpl-spec-list.txt",
"chars": 12917,
"preview": "# This file may be used to create an environment using:\n# $ conda create --name <env> --file <this file>\n# platform: lin"
},
{
"path": "libs/annotator/smpl_webuser/LICENSE.txt",
"chars": 4290,
"preview": "Please read carefully the following terms and conditions and any accompanying documentation before you download and/or u"
},
{
"path": "libs/annotator/smpl_webuser/README.txt",
"chars": 1743,
"preview": "License:\n========\nTo learn about SMPL, please visit our website: http://smpl.is.tue.mpg\nYou can find the SMPL paper at: "
},
{
"path": "libs/annotator/smpl_webuser/__init__.py",
"chars": 534,
"preview": "'''\nCopyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.\nThis software i"
},
{
"path": "libs/annotator/smpl_webuser/hello_world/hello_smpl.py",
"chars": 2115,
"preview": "'''\nCopyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.\nThis software i"
},
{
"path": "libs/annotator/smpl_webuser/hello_world/render_smpl.py",
"chars": 3028,
"preview": "'''\nCopyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.\nThis software i"
},
{
"path": "libs/annotator/smpl_webuser/lbs.py",
"chars": 2857,
"preview": "'''\nCopyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.\nThis software i"
},
{
"path": "libs/annotator/smpl_webuser/posemapper.py",
"chars": 1423,
"preview": "'''\nCopyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.\nThis software i"
},
{
"path": "libs/annotator/smpl_webuser/serialization.py",
"chars": 4549,
"preview": "'''\nCopyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.\nThis software i"
},
{
"path": "libs/annotator/smpl_webuser/verts.py",
"chars": 3286,
"preview": "'''\nCopyright 2015 Matthew Loper, Naureen Mahmood and the Max Planck Gesellschaft. All rights reserved.\nThis software i"
},
{
"path": "libs/annotator/smplify/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "libs/annotator/smplify/fit_3d.py",
"chars": 25926,
"preview": "\"\"\"\nCopyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.\nThis software is provided fo"
},
{
"path": "libs/annotator/smplify/lib/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "libs/annotator/smplify/lib/capsule_body.py",
"chars": 9518,
"preview": "\"\"\"\nCopyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.\nThis software is provided fo"
},
{
"path": "libs/annotator/smplify/lib/capsule_ch.py",
"chars": 5146,
"preview": "\"\"\"\nCopyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.\nThis software is provided fo"
},
{
"path": "libs/annotator/smplify/lib/max_mixture_prior.py",
"chars": 4380,
"preview": "\"\"\"\nCopyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.\nThis software is provided fo"
},
{
"path": "libs/annotator/smplify/lib/robustifiers.py",
"chars": 2841,
"preview": "\"\"\"\nCopyright 2016 Max Planck Society, Matthew Loper. All rights reserved.\nThis software is provided for research purpos"
},
{
"path": "libs/annotator/smplify/lib/sphere_collisions.py",
"chars": 3509,
"preview": "\"\"\"\nCopyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.\nThis software is provided fo"
},
{
"path": "libs/annotator/smplify/render_model.py",
"chars": 3206,
"preview": "\"\"\"\nCopyright 2016 Max Planck Society, Federica Bogo, Angjoo Kanazawa. All rights reserved.\nThis software is provided fo"
},
{
"path": "libs/dataset/__init__.py",
"chars": 94,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n#import libs.dataset.h36m\n\n"
},
{
"path": "libs/dataset/h36m/__init__.py",
"chars": 48,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\n"
},
{
"path": "libs/dataset/h36m/cameras.py",
"chars": 4056,
"preview": "\"\"\"\nUtilities to deal with the cameras of human3.6m.\nReference: https://github.com/una-dinosauria/3d-pose-baseline \n\"\"\"\n"
},
{
"path": "libs/dataset/h36m/data_utils.py",
"chars": 35620,
"preview": "\"\"\"\nUtility functions for dealing with Human3.6M dataset.\nSome functions are adapted from https://github.com/una-dinosau"
},
{
"path": "libs/dataset/h36m/h36m_pose.py",
"chars": 10922,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n# -----------------------------------------------------------------------"
},
{
"path": "libs/dataset/h36m/pose_dataset.py",
"chars": 10237,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/dataset/h36m/pth_dataset.py",
"chars": 5428,
"preview": "import logging\n\nimport torch\nimport torch.utils.data\nimport torch.nn.functional as F\nimport numpy as np\n\ndef normalize(v"
},
{
"path": "libs/evolution/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/evolution/genetic.py",
"chars": 26699,
"preview": "\"\"\"\nUtility functions for genetic evolution.\n\"\"\"\nimport libs.dataset.h36m.cameras as cameras\nfrom libs.skeleton.anglelim"
},
{
"path": "libs/evolution/parameter.py",
"chars": 3640,
"preview": "\"\"\"\nArguments and hyper-parameters used in dataset evolution. \n\"\"\"\nimport argparse\n\ndef parse_arg():\n parser = argpar"
},
{
"path": "libs/hhr/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/hhr/config/__init__.py",
"chars": 369,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/hhr/config/default.py",
"chars": 3732,
"preview": "\n# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed u"
},
{
"path": "libs/hhr/config/models.py",
"chars": 2118,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/hhr/core/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/hhr/core/evaluate.py",
"chars": 4663,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/hhr/core/function.py",
"chars": 13682,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/hhr/core/inference.py",
"chars": 4267,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/hhr/core/loss.py",
"chars": 6439,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/hhr/utils/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "libs/hhr/utils/transforms.py",
"chars": 3837,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/hhr/utils/utils.py",
"chars": 7094,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/hhr/utils/vis.py",
"chars": 5116,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/model/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/model/model.py",
"chars": 5272,
"preview": "\"\"\"\nFully-connected residual network as a single deep learner.\n\"\"\"\n\nimport torch.nn as nn\nimport torch\n\nclass ResidualBl"
},
{
"path": "libs/model/pose_hrnet.py",
"chars": 18833,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/model/pose_resnet.py",
"chars": 9460,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
},
{
"path": "libs/optimizer/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/parser/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/parser/parse.py",
"chars": 6195,
"preview": "import argparse\n\ndef parse_arg():\n parser = argparse.ArgumentParser(description='2Dto3Dnet.py')\n ## paths\n pars"
},
{
"path": "libs/skeleton/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/skeleton/anglelimits.py",
"chars": 30962,
"preview": "\"\"\"\nUtility functions for the hierarchical human representation. \nA Python implementation for pose-conditioned joint ang"
},
{
"path": "libs/trainer/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/trainer/trainer.py",
"chars": 14730,
"preview": "\"\"\"\nUtility functions for cascaded model training and evaluation.\n\"\"\"\nimport libs.dataset.h36m.data_utils as data_utils\n"
},
{
"path": "libs/utils/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/utils/utils.py",
"chars": 13086,
"preview": "\"\"\"\nUtility functions.\n\"\"\"\nimport libs.dataset.h36m.data_utils as data_utils\nimport libs.dataset.h36m.cameras as cameras"
},
{
"path": "libs/visualization/__init__.py",
"chars": 69,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nEmpty file.\n\"\"\"\n\n\n"
},
{
"path": "libs/visualization/viz.py",
"chars": 3831,
"preview": "\"\"\"\nFunctions to visualize human poses\n\"\"\"\n\nimport matplotlib.pyplot as plt\nimport libs.dataset.h36m.data_utils as data_"
},
{
"path": "resources/.gitignore",
"chars": 71,
"preview": "# Ignore everything in this directory\n*\n# Except this file\n!.gitignore\n"
},
{
"path": "spec-list.txt",
"chars": 24574,
"preview": "# This file may be used to create an environment using:\n# $ conda create --name <env> --file <this file>\n# platform: lin"
},
{
"path": "tools/2Dto3Dnet.py",
"chars": 2083,
"preview": "\"\"\"\n3D pose estimation based on 2D key-point coordinates as inputs.\nAuthor: Shichao Li\nEmail: nicholas.li@connect.ust.hk"
},
{
"path": "tools/annotate_2D.py",
"chars": 5013,
"preview": "\"\"\"\nAnnotate 2D key-points interactively.\nPress Q to exit the tool.\nPress C to remove the annotation.\nPress N to go to t"
},
{
"path": "tools/annotate_3D.py",
"chars": 15942,
"preview": "\"\"\"\nInteractive annotation tool for 3D human pose estimation.\nGiven an image and a coarse 3D skeleton estimation, the us"
},
{
"path": "tools/evolve.py",
"chars": 6145,
"preview": "\"\"\"\nEvolution of 3D human skeleton.\nauthor: Nicholas Li\ncontact: nicholas.li@connect.ust.hk\n\"\"\"\nimport sys\nsys.path.appe"
},
{
"path": "tools/imgto2Dnet.py",
"chars": 7962,
"preview": "# ------------------------------------------------------------------------------\n# Copyright (c) Microsoft\n# Licensed un"
}
]
About this extraction
This page contains the full source code of the Nicholasli1995/EvoSkeleton GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 81 files (451.1 KB), approximately 129.4k tokens, and a symbol index with 336 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.