Repository: simonalexanderson/ListenDenoiseAction Branch: main Commit: 9b0885b69991 Files: 65 Total size: 238.9 KB Directory structure: gitextract_9cw0czva/ ├── .gitignore ├── Dockerfile ├── LICENSE ├── README.md ├── data/ │ └── motorica_dance/ │ ├── audio18_features.txt │ ├── audio29_features.txt │ ├── ch0_spec_beatact_features.txt │ ├── dance_styles.txt │ ├── dance_test_files.txt │ ├── dance_train_files.txt │ ├── data_pipe.expmap_30fps.sav │ ├── gen_files.txt │ ├── kthstreet_gKR_sFM_cAll_d01_mKR_ch01_chargedcableupyour_001_00.audio29_30fps.pkl │ ├── kthstreet_gKR_sFM_cAll_d01_mKR_ch01_chargedcableupyour_001_00.expmap_30fps.pkl │ ├── kthstreet_gKR_sFM_cAll_d01_mKR_ch01_chargedcableupyour_001_01.audio29_30fps.pkl │ ├── kthstreet_gKR_sFM_cAll_d01_mKR_ch01_chargedcableupyour_001_01.expmap_30fps.pkl │ ├── kthstreet_gLH_sFM_cAll_d02_mLH_ch01_lala_001_00.audio29_30fps.pkl │ ├── kthstreet_gLH_sFM_cAll_d02_mLH_ch01_lala_001_00.expmap_30fps.pkl │ ├── kthstreet_gLH_sFM_cAll_d02_mLH_ch01_lala_001_01.audio29_30fps.pkl │ ├── kthstreet_gLH_sFM_cAll_d02_mLH_ch01_lala_001_01.expmap_30fps.pkl │ ├── kthstreet_gLO_sFM_cAll_d02_mLO_ch01_arethafranklinrocksteady_002_00.audio29_30fps.pkl │ ├── kthstreet_gLO_sFM_cAll_d02_mLO_ch01_arethafranklinrocksteady_002_00.expmap_30fps.pkl │ ├── kthstreet_gLO_sFM_cAll_d02_mLO_ch01_arethafranklinrocksteady_002_01.audio29_30fps.pkl │ ├── kthstreet_gLO_sFM_cAll_d02_mLO_ch01_arethafranklinrocksteady_002_01.expmap_30fps.pkl │ ├── kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bombom_002_00.audio29_30fps.pkl │ ├── kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bombom_002_00.expmap_30fps.pkl │ ├── kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bombom_002_01.audio29_30fps.pkl │ ├── kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bombom_002_01.expmap_30fps.pkl │ ├── kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bombom_001_00.audio29_30fps.pkl │ ├── kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bombom_001_00.expmap_30fps.pkl │ ├── kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bombom_001_01.audio29_30fps.pkl │ ├── kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bombom_001_01.expmap_30fps.pkl │ └── pose_features.expmap.txt ├── experiments/ │ ├── dance_LDA-U.sh │ ├── dance_LDA.sh │ └── dance_mix_experts.sh ├── hparams/ │ ├── diffusion_dance_LDA-U.yaml │ └── diffusion_dance_LDA.yaml ├── models/ │ ├── BaseModel.py │ ├── LightningModel.py │ ├── nn.py │ └── transformer/ │ ├── tisa_transformer.py │ └── tisa_v2.py ├── pretrained_models/ │ ├── .gitkeep │ └── README.md ├── pymo/ │ ├── Pivots.py │ ├── Quaternions.py │ ├── __init__.py │ ├── data.py │ ├── features.py │ ├── parsers.py │ ├── preprocessing.py │ ├── rotation_tools.py │ ├── viz_tools.py │ └── writers.py ├── requirements.txt ├── run_docker.sh ├── synthesize.py ├── train.py └── utils/ ├── cut_wav.py ├── download_from_youtube.py ├── duplicate_features.sh ├── hparams.py ├── logging_mixin.py └── motion_dataset.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ results/* **/__pycache__ ================================================ FILE: Dockerfile ================================================ FROM nvcr.io/nvidia/pytorch:22.10-py3 ENV DEBIAN_FRONTEND=noninteractive WORKDIR /workspace RUN apt-get update RUN apt-get install rename RUN apt-get install -y ffmpeg COPY requirements.txt /tmp RUN pip install -r /tmp/requirements.txt ================================================ FILE: LICENSE ================================================ License and Copyright Information --------------------------------- The contents of this repository may not be used for any purpose other than academic research. It is free to use for research purposes by academic institutes, companies, and individuals. Use for commercial purposes is not permitted without prior written consent from Motorica AB. If you are interested in using the codebase, pretrained models or the dataset for commercial purposes or non-research purposes, please contact us at info@motorica.ai in advance. Unauthorised redistribution is prohibited without written approval. Please include the following citations in any preprints and publications that use this repository. @article{alexanderson2023listen, title={Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models}, author={Alexanderson, Simon and Nagy, Rajmund and Beskow, Jonas and Henter, Gustav Eje}, journal={ACM Trans. Graph.}, volume={42}, number={4}, pages={1--20}, doi={10.1145/3592458}, year={2023} } The code for translation-invariant self-attention (TISA) was written by Ulme Wennberg. Please cite their ACL 2021 article if you use this code. ================================================ FILE: README.md ================================================ # Listen, Denoise, Action! This repository provides code and models for the paper [Listen, denoise, action! Audio-driven motion synthesis with diffusion models](https://arxiv.org/abs/2211.09707). Please watch the following video for an introduction to our work: * [SIGGRAPH 2023 presentation](https://youtu.be/Qfd2EpzWgok) For video samples and a general overview, please see [our project page](https://www.speech.kth.se/research/listen-denoise-action/). For the new dance dataset with high-quality mocap, please see [the Motorica Dance Dataset](https://github.com/simonalexanderson/MotoricaDanceDataset/). ## Installation We provide a Docker file and `requirements.txt` for installation using a Docker image or Conda. ### Installation using Conda ``` conda install python=3.9 conda install -c conda-forge mpi4py mpich pip install -r requirements.txt ``` ## Dance synthesis demo ### Data and pretrained models Please [download our pretrained dance models here](https://zenodo.org/record/8156769) and move them to the `pretrained_models` folder. We include processed music inputs from the test dataset in the `data` folder for generating dances from the model. ### Synthesis scripts You can use the following shell scripts for reproducing the dance user studies in the paper: ``` ./experiments/dance_LDA.sh ./experiments/dance_LDA-U.sh ``` To try out locomotion synthesis, please go to https://www.motorica.ai/. ## Training data The four main training datasets from our SIGGRAPH 2023 paper are available online: * [The Trinity Speech Gesture Dataset](https://trinityspeechgesture.scss.tcd.ie/) * [The ZEGGS dataset](https://github.com/ubisoft/ubisoft-laforge-ZeroEGGS) * [The 100STYLE dataset](https://www.ianxmason.com/100style/) * [The Motorica Dance Dataset](https://github.com/simonalexanderson/MotoricaDanceDataset/), a new dataset with high-quality dance mocap released together with our paper ## License and copyright information The contents of this repository may not be used for any purpose other than academic research. It is free to use for research purposes by academic institutes, companies, and individuals. Use for commercial purposes is not permitted without prior written consent from Motorica AB. If you are interested in using the codebase, pretrained models or the dataset for commercial purposes or non-research purposes, please contact us at info@motorica.ai in advance. Unauthorised redistribution is prohibited without written approval. ### Attribution Please include the following citations in any preprints and publications that use this repository. ``` @article{alexanderson2023listen, title={Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models}, author={Alexanderson, Simon and Nagy, Rajmund and Beskow, Jonas and Henter, Gustav Eje}, year={2023} issue_date={August 2023}, publisher={ACM}, volume={42}, number={4}, doi={10.1145/3592458}, journal={ACM Trans. Graph.}, articleno={44}, numpages={20}, pages={44:1--44:20} } ``` The [code for translation-invariant self-attention](https://github.com/ulmewennberg/tisa) (TISA) was written by [Ulme Wennberg](https://www.kth.se/profile/ulme). Please cite [the correspoding ACL 2021 article](https://aclanthology.org/2021.acl-short.18) if you use this code. ================================================ FILE: data/motorica_dance/audio18_features.txt ================================================ MFCC_0 MFCC_1 MFCC_2 Chroma_0 Chroma_1 Chroma_2 Chroma_3 Chroma_4 Chroma_5 Chroma_6 Chroma_7 Chroma_8 Chroma_9 Chroma_10 Chroma_11 Spectralflux_0 Beatactivation_0 Beat_0 ================================================ FILE: data/motorica_dance/audio29_features.txt ================================================ MFCC_0 MFCC_1 MFCC_2 MFCC_3 MFCC_4 MFCC_5 MFCC_6 MFCC_7 MFCC_8 MFCC_9 MFCC_10 MFCC_11 MFCC_12 MFCC_13 MFCC_14 MFCC_15 MFCC_16 MFCC_17 MFCC_18 MFCC_19 Chroma_0 Chroma_1 Chroma_2 Chroma_3 Chroma_4 Chroma_5 Spectralflux_0 Beatactivation_0 Beat_0 ================================================ FILE: data/motorica_dance/ch0_spec_beatact_features.txt ================================================ Chroma_0 Spectralflux_0 Beatactivation_0 ================================================ FILE: data/motorica_dance/dance_styles.txt ================================================ gCA gCH gJZ gKR gLH gLO gPO gTP ================================================ FILE: data/motorica_dance/dance_test_files.txt ================================================ kthjazz_gCH_sFM_cAll_d02_mCH_ch01_whitemanpaulandhisorchestraloisiana_006_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_whitemanpaulandhisorchestraloisiana_006_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_bennygoodmansugarfootstomp_003_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_bennygoodmansugarfootstomp_003_01 kthjazz_gTP_sFM_sngl_d02_015_00 kthjazz_gTP_sFM_sngl_d02_015_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch24_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch24_01 kthstreet_gKR_sFM_cAll_d01_mKR_ch01_chargedcableupyour_001_00 kthstreet_gKR_sFM_cAll_d01_mKR_ch01_chargedcableupyour_001_01 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_thisisit_001_00 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_thisisit_001_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_lala_001_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_lala_001_01 kthstreet_gLO_sFM_cAll_d02_mLO_ch01_arethafranklinrocksteady_002_00 kthstreet_gLO_sFM_cAll_d02_mLO_ch01_arethafranklinrocksteady_002_01 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bombom_002_00 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bombom_002_01 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bombom_001_00 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bombom_001_01 ================================================ FILE: data/motorica_dance/dance_train_files.txt ================================================ kthjazz_gCH_sFM_cAll_d02_mCH_ch01_beatlestreetwashboardbandfortyandtight_003_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_beatlestreetwashboardbandfortyandtight_003_00_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_beatlestreetwashboardbandfortyandtight_003_01 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_beatlestreetwashboardbandfortyandtight_003_01_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_charlestonchaserswabashblues_004_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_charlestonchaserswabashblues_004_00_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_charlestonchaserswabashblues_004_01 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_charlestonchaserswabashblues_004_01_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_lloydkeatingandhismusicturnontheheat_010_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_lloydkeatingandhismusicturnontheheat_010_00_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_lloydkeatingandhismusicturnontheheat_010_01 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_lloydkeatingandhismusicturnontheheat_010_01_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_royalgardenblues_002_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_royalgardenblues_002_00_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_royalgardenblues_002_01 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_royalgardenblues_002_01_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_tedlewisgladragdoll_007_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_tedlewisgladragdoll_007_00_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_tedlewisgladragdoll_007_01 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_tedlewisgladragdoll_007_01_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_therythmiceightkansascitykitty_001_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_therythmiceightkansascitykitty_001_00_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_therythmiceightkansascitykitty_001_01 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_therythmiceightkansascitykitty_001_01_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_therythmiceightumthaumthadadada_009_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_therythmiceightumthaumthadadada_009_00_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_therythmiceightumthaumthadadada_009_01 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_therythmiceightumthaumthadadada_009_01_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_thesavoyorpheansfivefoottwoeyesofblue_008_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_thesavoyorpheansfivefoottwoeyesofblue_008_00_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_thesavoyorpheansfivefoottwoeyesofblue_008_01 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_thesavoyorpheansfivefoottwoeyesofblue_008_01_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_thesavoyorpheansthecharleston_005_00 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_thesavoyorpheansthecharleston_005_00_mirrored kthjazz_gCH_sFM_cAll_d02_mCH_ch01_thesavoyorpheansthecharleston_005_01 kthjazz_gCH_sFM_cAll_d02_mCH_ch01_thesavoyorpheansthecharleston_005_01_mirrored kthjazz_gCH_sFM_sngl_d01_007_00 kthjazz_gCH_sFM_sngl_d01_007_00_mirrored kthjazz_gCH_sFM_sngl_d01_007_01 kthjazz_gCH_sFM_sngl_d01_007_01_mirrored kthjazz_gCH_sFM_sngl_d01_008_00 kthjazz_gCH_sFM_sngl_d01_008_00_mirrored kthjazz_gCH_sFM_sngl_d01_008_01 kthjazz_gCH_sFM_sngl_d01_008_01_mirrored kthjazz_gCH_sFM_sngl_d01_009_00 kthjazz_gCH_sFM_sngl_d01_009_00_mirrored kthjazz_gCH_sFM_sngl_d01_009_01 kthjazz_gCH_sFM_sngl_d01_009_01_mirrored kthjazz_gCH_sFM_sngl_d01_019_00 kthjazz_gCH_sFM_sngl_d01_019_00_mirrored kthjazz_gCH_sFM_sngl_d01_019_01 kthjazz_gCH_sFM_sngl_d01_019_01_mirrored kthjazz_gCH_sFM_sngl_d02_005_00 kthjazz_gCH_sFM_sngl_d02_005_00_mirrored kthjazz_gCH_sFM_sngl_d02_005_01 kthjazz_gCH_sFM_sngl_d02_005_01_mirrored kthjazz_gCH_sFM_sngl_d02_006_00 kthjazz_gCH_sFM_sngl_d02_006_00_mirrored kthjazz_gCH_sFM_sngl_d02_006_01 kthjazz_gCH_sFM_sngl_d02_006_01_mirrored kthjazz_gCH_sFM_sngl_d02_018_00 kthjazz_gCH_sFM_sngl_d02_018_00_mirrored kthjazz_gCH_sFM_sngl_d02_018_01 kthjazz_gCH_sFM_sngl_d02_018_01_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_bennygoodmanairmailspecial_002_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_bennygoodmanairmailspecial_002_00_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_bennygoodmanairmailspecial_002_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_bennygoodmanairmailspecial_002_01_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_budpowellboundingwithbud_006_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_budpowellboundingwithbud_006_00_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_budpowellboundingwithbud_006_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_budpowellboundingwithbud_006_01_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_countbasieshortygeorge_007_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_countbasieshortygeorge_007_00_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_countbasieshortygeorge_007_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_countbasieshortygeorge_007_01_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dizzygillespiegroovinghigh_004_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dizzygillespiegroovinghigh_004_00_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dizzygillespiegroovinghigh_004_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dizzygillespiegroovinghigh_004_01_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dizzygillespieohbopshbam_005_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dizzygillespieohbopshbam_005_00_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dizzygillespieohbopshbam_005_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dizzygillespieohbopshbam_005_01_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dukeellingtontaketheatrain_001_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dukeellingtontaketheatrain_001_00_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dukeellingtontaketheatrain_001_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_dukeellingtontaketheatrain_001_01_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_glennmillerinthemood_003_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_glennmillerinthemood_003_00_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_glennmillerinthemood_003_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_glennmillerinthemood_003_01_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_jimmyluncefordfordancersonly_004_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_jimmyluncefordfordancersonly_004_00_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_jimmyluncefordfordancersonly_004_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_jimmyluncefordfordancersonly_004_01_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_lionelhamptonflyinghome_005_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_lionelhamptonflyinghome_005_00_mirrored kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_lionelhamptonflyinghome_005_01 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_lionelhamptonflyinghome_005_01_mirrored kthjazz_gJZ_sFM_sngl_d01_001_00 kthjazz_gJZ_sFM_sngl_d01_001_00_mirrored kthjazz_gJZ_sFM_sngl_d01_001_01 kthjazz_gJZ_sFM_sngl_d01_001_01_mirrored kthjazz_gJZ_sFM_sngl_d01_002_00 kthjazz_gJZ_sFM_sngl_d01_002_00_mirrored kthjazz_gJZ_sFM_sngl_d01_002_01 kthjazz_gJZ_sFM_sngl_d01_002_01_mirrored kthjazz_gJZ_sFM_sngl_d01_020_00 kthjazz_gJZ_sFM_sngl_d01_020_00_mirrored kthjazz_gJZ_sFM_sngl_d01_020_01 kthjazz_gJZ_sFM_sngl_d01_020_01_mirrored kthjazz_gJZ_sFM_sngl_d01_021_00 kthjazz_gJZ_sFM_sngl_d01_021_00_mirrored kthjazz_gJZ_sFM_sngl_d01_021_01 kthjazz_gJZ_sFM_sngl_d01_021_01_mirrored kthjazz_gJZ_sFM_sngl_d02_003_00 kthjazz_gJZ_sFM_sngl_d02_003_00_mirrored kthjazz_gJZ_sFM_sngl_d02_003_01 kthjazz_gJZ_sFM_sngl_d02_003_01_mirrored kthjazz_gJZ_sFM_sngl_d02_004_00 kthjazz_gJZ_sFM_sngl_d02_004_00_mirrored kthjazz_gJZ_sFM_sngl_d02_004_01 kthjazz_gJZ_sFM_sngl_d02_004_01_mirrored kthjazz_gTP_sFM_sngl_d01_010_00 kthjazz_gTP_sFM_sngl_d01_010_00_mirrored kthjazz_gTP_sFM_sngl_d01_010_01 kthjazz_gTP_sFM_sngl_d01_010_01_mirrored kthjazz_gTP_sFM_sngl_d02_014_00 kthjazz_gTP_sFM_sngl_d02_014_00_mirrored kthjazz_gTP_sFM_sngl_d02_014_01 kthjazz_gTP_sFM_sngl_d02_014_01_mirrored kthjazz_gTP_sFM_sngl_d02_016_00 kthjazz_gTP_sFM_sngl_d02_016_00_mirrored kthjazz_gTP_sFM_sngl_d02_016_01 kthjazz_gTP_sFM_sngl_d02_016_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch0_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch0_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch0_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch0_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch10_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch10_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch10_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch10_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch13_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch13_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch13_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch13_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch14_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch14_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch14_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch14_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch15_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch15_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch15_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch15_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch16_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch16_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch16_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch16_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch17_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch17_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch17_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch17_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch19_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch19_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch19_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch19_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch1_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch1_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch1_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch1_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch20_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch20_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch20_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch20_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch22_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch22_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch22_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch22_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch24_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch24_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch24_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch24_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch28_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch28_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch28_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch28_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch29_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch29_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch29_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch29_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch2_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch2_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch2_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch2_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch30_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch30_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch30_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch30_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch3_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch3_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch3_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch3_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch4_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch4_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch4_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch4_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch5_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch5_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch5_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch5_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch6_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch6_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch6_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch6_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch8_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch8_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch8_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch8_01_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch9_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch9_00_mirrored kthmisc_gCA_sFM_cAll_d01_mCA_ch9_01 kthmisc_gCA_sFM_cAll_d01_mCA_ch9_01_mirrored kthstreet_gKR_sFM_cAll_d01_mKR1_ch01_00 kthstreet_gKR_sFM_cAll_d01_mKR1_ch01_00_mirrored kthstreet_gKR_sFM_cAll_d01_mKR1_ch01_01 kthstreet_gKR_sFM_cAll_d01_mKR1_ch01_01_mirrored kthstreet_gKR_sFM_cAll_d01_mKR3_ch03_00 kthstreet_gKR_sFM_cAll_d01_mKR3_ch03_00_mirrored kthstreet_gKR_sFM_cAll_d01_mKR3_ch03_01 kthstreet_gKR_sFM_cAll_d01_mKR3_ch03_01_mirrored kthstreet_gKR_sFM_cAll_d01_mKR4_ch04_00 kthstreet_gKR_sFM_cAll_d01_mKR4_ch04_00_mirrored kthstreet_gKR_sFM_cAll_d01_mKR4_ch04_01 kthstreet_gKR_sFM_cAll_d01_mKR4_ch04_01_mirrored kthstreet_gKR_sFM_cAll_d01_mKR5_ch05_00 kthstreet_gKR_sFM_cAll_d01_mKR5_ch05_00_mirrored kthstreet_gKR_sFM_cAll_d01_mKR5_ch05_01 kthstreet_gKR_sFM_cAll_d01_mKR5_ch05_01_mirrored kthstreet_gKR_sFM_cAll_d01_mKR6_ch06_00 kthstreet_gKR_sFM_cAll_d01_mKR6_ch06_00_mirrored kthstreet_gKR_sFM_cAll_d01_mKR6_ch06_01 kthstreet_gKR_sFM_cAll_d01_mKR6_ch06_01_mirrored kthstreet_gKR_sFM_cAll_d01_mKR7_ch07_00 kthstreet_gKR_sFM_cAll_d01_mKR7_ch07_00_mirrored kthstreet_gKR_sFM_cAll_d01_mKR7_ch07_01 kthstreet_gKR_sFM_cAll_d01_mKR7_ch07_01_mirrored kthstreet_gKR_sFM_cAll_d01_mKR_ch01_1stafterthecaos_000_00 kthstreet_gKR_sFM_cAll_d01_mKR_ch01_1stafterthecaos_000_00_mirrored kthstreet_gKR_sFM_cAll_d01_mKR_ch01_1stafterthecaos_000_01 kthstreet_gKR_sFM_cAll_d01_mKR_ch01_1stafterthecaos_000_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH11_ch11_00 kthstreet_gLH_sFM_cAll_d01_mLH11_ch11_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH11_ch11_01 kthstreet_gLH_sFM_cAll_d01_mLH11_ch11_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH12_ch12_00 kthstreet_gLH_sFM_cAll_d01_mLH12_ch12_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH12_ch12_01 kthstreet_gLH_sFM_cAll_d01_mLH12_ch12_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH13_ch13_00 kthstreet_gLH_sFM_cAll_d01_mLH13_ch13_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH13_ch13_01 kthstreet_gLH_sFM_cAll_d01_mLH13_ch13_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH14_ch14_00 kthstreet_gLH_sFM_cAll_d01_mLH14_ch14_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH14_ch14_01 kthstreet_gLH_sFM_cAll_d01_mLH14_ch14_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH15_ch15_00 kthstreet_gLH_sFM_cAll_d01_mLH15_ch15_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH15_ch15_01 kthstreet_gLH_sFM_cAll_d01_mLH15_ch15_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH1_ch01_00 kthstreet_gLH_sFM_cAll_d01_mLH1_ch01_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH1_ch01_01 kthstreet_gLH_sFM_cAll_d01_mLH1_ch01_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH2_ch02_00 kthstreet_gLH_sFM_cAll_d01_mLH2_ch02_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH2_ch02_01 kthstreet_gLH_sFM_cAll_d01_mLH2_ch02_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH3_ch03_00 kthstreet_gLH_sFM_cAll_d01_mLH3_ch03_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH3_ch03_01 kthstreet_gLH_sFM_cAll_d01_mLH3_ch03_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH4_ch04_00 kthstreet_gLH_sFM_cAll_d01_mLH4_ch04_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH4_ch04_01 kthstreet_gLH_sFM_cAll_d01_mLH4_ch04_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH6_ch06_00 kthstreet_gLH_sFM_cAll_d01_mLH6_ch06_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH6_ch06_01 kthstreet_gLH_sFM_cAll_d01_mLH6_ch06_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH7_ch07_00 kthstreet_gLH_sFM_cAll_d01_mLH7_ch07_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH7_ch07_01 kthstreet_gLH_sFM_cAll_d01_mLH7_ch07_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH8_ch08_00 kthstreet_gLH_sFM_cAll_d01_mLH8_ch08_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH8_ch08_01 kthstreet_gLH_sFM_cAll_d01_mLH8_ch08_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH9_ch09_00 kthstreet_gLH_sFM_cAll_d01_mLH9_ch09_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH9_ch09_01 kthstreet_gLH_sFM_cAll_d01_mLH9_ch09_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_danstrams_00 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_danstrams_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_danstrams_01 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_danstrams_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_feelingsvstech_001_00 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_feelingsvstech_001_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_feelingsvstech_001_01 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_feelingsvstech_001_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_imhere20_005_00 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_imhere20_005_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_imhere20_005_01 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_imhere20_005_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_jesusbreak_001_00 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_jesusbreak_001_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_jesusbreak_001_01 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_jesusbreak_001_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_vibeness108bpm_001_00 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_vibeness108bpm_001_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_vibeness108bpm_001_01 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_vibeness108bpm_001_01_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_yougonnaregretit_003_00 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_yougonnaregretit_003_00_mirrored kthstreet_gLH_sFM_cAll_d01_mLH_ch01_yougonnaregretit_003_01 kthstreet_gLH_sFM_cAll_d01_mLH_ch01_yougonnaregretit_003_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_basketboll_002_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_basketboll_002_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_basketboll_002_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_basketboll_002_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_danstrams_000_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_danstrams_000_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_danstrams_000_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_danstrams_000_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_digit_001_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_digit_001_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_digit_001_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_digit_001_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_glitter_001_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_glitter_001_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_glitter_001_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_glitter_001_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_hitmewobble_001_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_hitmewobble_001_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_hitmewobble_001_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_hitmewobble_001_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_luco100bpm_002_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_luco100bpm_002_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_luco100bpm_002_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_luco100bpm_002_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_opdirt_002_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_opdirt_002_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_opdirt_002_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_opdirt_002_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_thisisit_001_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_thisisit_001_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_thisisit_001_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_thisisit_001_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_yougonnaregretit_002_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_yougonnaregretit_002_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_yougonnaregretit_002_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_yougonnaregretit_002_01_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_yourhead108bpm_001_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_yourhead108bpm_001_00_mirrored kthstreet_gLH_sFM_cAll_d02_mLH_ch01_yourhead108bpm_001_01 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_yourhead108bpm_001_01_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_boogiewonderland_001_00 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_boogiewonderland_001_00_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_boogiewonderland_001_01 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_boogiewonderland_001_01_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_dennisscorpiocoffey_001_00 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_dennisscorpiocoffey_001_00_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_dennisscorpiocoffey_001_01 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_dennisscorpiocoffey_001_01_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_funk_001_00 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_funk_001_00_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_funk_001_01 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_funk_001_01_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_jamesbrownpapasgotabrandnewbag_001_00 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_jamesbrownpapasgotabrandnewbag_001_00_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_jamesbrownpapasgotabrandnewbag_001_01 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_jamesbrownpapasgotabrandnewbag_001_01_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_princesexydanser_001_00 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_princesexydanser_001_00_mirrored kthstreet_gLO_sFM_cAll_d01_mLO_ch01_princesexydanser_001_01 kthstreet_gLO_sFM_cAll_d01_mLO_ch01_princesexydanser_001_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO1_ch01_00 kthstreet_gPO_sFM_cAll_d01_mPO1_ch01_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO1_ch01_01 kthstreet_gPO_sFM_cAll_d01_mPO1_ch01_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO2_ch02_00 kthstreet_gPO_sFM_cAll_d01_mPO2_ch02_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO2_ch02_01 kthstreet_gPO_sFM_cAll_d01_mPO2_ch02_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO3_ch03_00 kthstreet_gPO_sFM_cAll_d01_mPO3_ch03_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO3_ch03_01 kthstreet_gPO_sFM_cAll_d01_mPO3_ch03_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO4_ch04_00 kthstreet_gPO_sFM_cAll_d01_mPO4_ch04_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO4_ch04_01 kthstreet_gPO_sFM_cAll_d01_mPO4_ch04_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO5_ch05_00 kthstreet_gPO_sFM_cAll_d01_mPO5_ch05_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO5_ch05_01 kthstreet_gPO_sFM_cAll_d01_mPO5_ch05_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO6_ch06_00 kthstreet_gPO_sFM_cAll_d01_mPO6_ch06_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO6_ch06_01 kthstreet_gPO_sFM_cAll_d01_mPO6_ch06_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO7_ch07_00 kthstreet_gPO_sFM_cAll_d01_mPO7_ch07_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO7_ch07_01 kthstreet_gPO_sFM_cAll_d01_mPO7_ch07_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bbonthespot_003_00 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bbonthespot_003_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bbonthespot_003_01 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_bbonthespot_003_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO_ch01_heropop_002_00 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_heropop_002_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO_ch01_heropop_002_01 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_heropop_002_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO_ch01_justus2_001_00 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_justus2_001_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO_ch01_justus2_001_01 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_justus2_001_01_mirrored kthstreet_gPO_sFM_cAll_d01_mPO_ch01_luco100bpm_001_00 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_luco100bpm_001_00_mirrored kthstreet_gPO_sFM_cAll_d01_mPO_ch01_luco100bpm_001_01 kthstreet_gPO_sFM_cAll_d01_mPO_ch01_luco100bpm_001_01_mirrored kthstreet_gPO_sFM_cAll_d02_mPO_ch01_atombounce_001_00 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_atombounce_001_00_mirrored kthstreet_gPO_sFM_cAll_d02_mPO_ch01_atombounce_001_01 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_atombounce_001_01_mirrored kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bbonthespot_004_00 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bbonthespot_004_00_mirrored kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bbonthespot_004_01 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bbonthespot_004_01_mirrored kthstreet_gPO_sFM_cAll_d02_mPO_ch01_funk_002_00 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_funk_002_00_mirrored kthstreet_gPO_sFM_cAll_d02_mPO_ch01_funk_002_01 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_funk_002_01_mirrored kthstreet_gPO_sFM_cAll_d02_mPO_ch01_justus2_002_00 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_justus2_002_00_mirrored kthstreet_gPO_sFM_cAll_d02_mPO_ch01_justus2_002_01 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_justus2_002_01_mirrored ================================================ FILE: data/motorica_dance/gen_files.txt ================================================ kthjazz_gCH_sFM_cAll_d02_mCH_ch01_whitemanpaulandhisorchestraloisiana_006_00 kthjazz_gJZ_sFM_cAll_d02_mJZ_ch01_bennygoodmansugarfootstomp_003_00 kthmisc_gCA_sFM_cAll_d01_mCA_ch24_00 kthstreet_gKR_sFM_cAll_d01_mKR_ch01_chargedcableupyour_001_00 kthstreet_gLH_sFM_cAll_d02_mLH_ch01_lala_001_00 kthstreet_gLO_sFM_cAll_d02_mLO_ch01_arethafranklinrocksteady_002_00 kthstreet_gPO_sFM_cAll_d02_mPO_ch01_bombom_001_00 ================================================ FILE: data/motorica_dance/pose_features.expmap.txt ================================================ RightFoot_alpha RightFoot_beta RightFoot_gamma RightLeg_alpha RightLeg_beta RightLeg_gamma RightUpLeg_alpha RightUpLeg_beta RightUpLeg_gamma LeftFoot_alpha LeftFoot_beta LeftFoot_gamma LeftLeg_alpha LeftLeg_beta LeftLeg_gamma LeftUpLeg_alpha LeftUpLeg_beta LeftUpLeg_gamma RightHand_alpha RightHand_beta RightHand_gamma RightForeArm_alpha RightForeArm_beta RightForeArm_gamma RightArm_alpha RightArm_beta RightArm_gamma RightShoulder_alpha RightShoulder_beta RightShoulder_gamma LeftHand_alpha LeftHand_beta LeftHand_gamma LeftForeArm_alpha LeftForeArm_beta LeftForeArm_gamma LeftArm_alpha LeftArm_beta LeftArm_gamma LeftShoulder_alpha LeftShoulder_beta LeftShoulder_gamma Head_alpha Head_beta Head_gamma Neck_alpha Neck_beta Neck_gamma Spine1_alpha Spine1_beta Spine1_gamma Spine_alpha Spine_beta Spine_gamma Hips_alpha Hips_beta Hips_gamma Hips_Yposition reference_dXposition reference_dZposition reference_dYrotation ================================================ FILE: experiments/dance_LDA-U.sh ================================================ #!/bin/bash checkpoint=pretrained_models/dance_LDA-U.ckpt dest_dir=results/generated/dance_LDA-U if [ ! -d "${dest_dir}" ]; then mkdir -p "${dest_dir}" fi data_dir=data/motorica_dance wav_dir=data/motorica_dance basenames=$(cat "${data_dir}/gen_files.txt") start=0 seed=150 fps=30 trim_s=0 length_s=10 trim=$((trim_s*fps)) length=$((length_s*fps)) fixed_seed=false gpu="cuda:0" render_video=true for wavfile in $basenames; do start=0 for postfix in 0 1 2 3 4 5 6 7 8 9 10 11 do input_file=${wavfile}.audio29_${fps}fps.pkl output_file=${wavfile::-3}_${postfix}_${style} echo "start=${start}, len=${length}, postfix=${postfix}, seed=${seed}" python synthesize.py --checkpoints="${checkpoint}" --data_dirs="${data_dir}" --input_files="${input_file}" --start=${start} --end=${length} --seed=${seed} --postfix=${postfix} --trim=${trim} --dest_dir=${dest_dir} --gpu=${gpu} --video=${render_video} --outfile=${output_file} if [ "$fixed_seed" != "true" ]; then seed=$((seed+1)) fi echo seed=$seed python utils/cut_wav.py ${wav_dir}/${wavfile::-3}.wav $(((start+trim)/fps)) $(((start+length-trim)/fps)) ${postfix} ${dest_dir} if [ "$render_video" == "true" ]; then ffmpeg -y -i ${dest_dir}/${output_file}.mp4 -i ${dest_dir}/${wavfile::-3}_${postfix}.wav ${dest_dir}/${output_file}_audio.mp4 rm ${dest_dir}/${output_file}.mp4 fi start=$((start+length)) done done ================================================ FILE: experiments/dance_LDA.sh ================================================ #!/bin/bash checkpoint=pretrained_models/dance_LDA.ckpt dest_dir=results/generated/dance_LDA if [ ! -d "${dest_dir}" ]; then mkdir -p "${dest_dir}" fi data_dir=data/motorica_dance wav_dir=data/motorica_dance basenames=$(cat "${data_dir}/gen_files.txt") start=0 seed=150 fps=30 trim_s=0 length_s=10 trim=$((trim_s*fps)) length=$((length_s*fps)) fixed_seed=false gpu="cuda:0" render_video=true for wavfile in $basenames; do start=0 style=$(echo $wavfile | awk -F "_" '{print $2}') #Coherent style parsed from file-name for postfix in 0 1 2 3 4 5 6 7 8 9 10 11 do input_file=${wavfile}.audio29_${fps}fps.pkl output_file=${wavfile::-3}_${postfix}_${style} echo "start=${start}, len=${length}, postfix=${postfix}, seed=${seed}" python synthesize.py --checkpoints="${checkpoint}" --data_dirs="${data_dir}" --input_files="${input_file}" --styles="${style}" --start=${start} --end=${length} --seed=${seed} --postfix=${postfix} --trim=${trim} --dest_dir=${dest_dir} --gpu=${gpu} --video=${render_video} --outfile=${output_file} if [ "$fixed_seed" != "true" ]; then seed=$((seed+1)) fi echo seed=$seed python utils/cut_wav.py ${wav_dir}/${wavfile::-3}.wav $(((start+trim)/fps)) $(((start+length-trim)/fps)) ${postfix} ${dest_dir} if [ "$render_video" == "true" ]; then ffmpeg -y -i ${dest_dir}/${output_file}.mp4 -i ${dest_dir}/${wavfile::-3}_${postfix}.wav ${dest_dir}/${output_file}_audio.mp4 rm ${dest_dir}/${output_file}.mp4 fi start=$((start+length)) done done ================================================ FILE: experiments/dance_mix_experts.sh ================================================ #!/bin/bash ### Mixing N models/conditionings to generate a mix of styles. 1. Model A trained w.o. style conditioning 2. Model B with style S1 ### 3. Model B with style S2. The sampling will be done with e=e0 + g1*(e1-e0) + g2*(e2-e0) + ... ### gn are guidance factors. E.g. g1=1, g2=0 => style S1 g1=0,g2=1 => S2, g1=1,g2=1 => mix of styles # # Dance checkpoint1=pretrained_models/dance_LDA-U.ckpt checkpoint2=pretrained_models/dance_LDA.ckpt checkpoint3=${checkpoint2} dest_dir=results/dance_mix_experts if [ ! -d "${dest_dir}" ]; then mkdir -p "${dest_dir}" fi data_dir=data/motorica_dance basenames=$(cat "${data_dir}/gen_files.txt") # Different guidance factors for mixing models guidance_factors_lst=("1.0,1.0" "0.5,0.5" "0.25,1.0" "1.0,0.25") style=None,gJZ,gLO start=0 seed=150 fps=30 trim_s=0 length_s=10 trim=$((trim_s*fps)) length=$((length_s*fps)) fixed_seed=false gpu="cuda:0" render_video=true for wavfile in $basenames; do start=0 for postfix in 0 do for guidance_factors in ${guidance_factors_lst[@]}; do input_file=${wavfile}.audio29_${fps}fps.pkl input_file2=${input_file} input_file3=${input_file} output_file=${wavfile::-3}_${postfix}_${style}_${guidance_factors} echo Generating motion from ${input_file} to ${dest_dir}/${output_file} echo "start=${start}, len=${length}, postfix=${postfix}, seed=${seed}" python synthesize.py --checkpoints="${checkpoint1},${checkpoint2},${checkpoint3}" --data_dirs="${data_dir},${data_dir},${data_dir}" --input_files="${input_file},${input_file2},${input_file3}" --styles="${style}" --start=${start} --end=${length} --trim=${trim} --seed=${seed} --postfix=${postfix} --dest_dir=${dest_dir} --gf=${guidance_factors} --gpu=${gpu} --outfile=${output_file} --video=${render_video} if [ "$fixed_seed" != "true" ]; then seed=$((seed+1)) fi echo seed=$seed python utils/cut_wav.py ${data_dir}/${wavfile::-3}.wav $(((start+trim)/fps)) $(((start+length-trim)/fps)) ${postfix} ${dest_dir} ffmpeg -y -i ${dest_dir}/${output_file}.mp4 -i ${dest_dir}/${wavfile::-3}_${postfix}.wav ${dest_dir}/${output_file}_audio.mp4 rm ${dest_dir}/${output_file}.mp4 start=$((start+length)) postfix=$((postfix+1)) done done done ================================================ FILE: hparams/diffusion_dance_LDA-U.yaml ================================================ Data: segment_length: 150 style_index: [1] trim_edges: 300 traindata_filename: dance_train_files_kth.txt testdata_filename: dance_test_files_kth.txt input_modality: audio35_30fps output_modality: expmap_30fps input_feats_file: ch0_spec_beatact_features.txt datapipe_filename: data_pipe.expmap_30fps.sav timestretch_prob: 0.2 timestretch_factor: 0.1 Diffusion: name: tisa #tisa|conv residual_layers: 20 residual_channels: 256 embedding_dim: 512 args: tisa: num_blocks: 2 num_heads: 8 activation: relu dropout: 0.1 norm: LN d_ff: 1024 seq_len: 150 use_v2: true use_preln: false bias: false dilation_cycle: [0,1,2] conv: dilation_cycle_length: 10 unconditional: false noise_schedule_start: 0.01 noise_schedule_end: 0.7 n_noise_schedule: 150 Infer: eps: 1 seq_len: 25 Optim: Schedule: args: lambda: val: 10 multiplicative: val: 10 step: gamma: 0.99995 step_size: 10 name: step warm_up: 3000 args: adam: betas: - 0.9 - 0.999 eps: 1.0e-08 rmsprop: eps: 1.0e-08 sgd: momentum: 0.9 name: adam Validation: render: true apply_dropout: false render_every_n_epochs: 1 max_render_clips: 10 gen_synth_ctrl: false lr: 0.0006 batch_size: 80 num_dataloader_workers: 1 pruning_amount: 0.0 quantization: false Trainer: accelerator: gpu devices: [2] accumulate_grad_batches: 1 default_root_dir: results/training/dance_LDA-U gradient_clip_val: 25 deterministic: false fast_dev_run: false max_epochs: 10 min_epochs: 1 precision: 32 resume_from_checkpoint: null ================================================ FILE: hparams/diffusion_dance_LDA.yaml ================================================ Data: segment_length: 150 style_index: [1] trim_edges: 300 styles_file: dance_styles_kth.txt traindata_filename: dance_train_files_kth.txt testdata_filename: dance_test_files_kth.txt input_modality: audio35_30fps output_modality: expmap_30fps input_feats_file: ch0_spec_beatact_features.txt datapipe_filename: data_pipe.expmap_30fps.sav timestretch_prob: 0.2 timestretch_factor: 0.1 Diffusion: name: tisa #tisa|conv residual_layers: 20 residual_channels: 256 embedding_dim: 512 args: tisa: num_blocks: 2 num_heads: 8 activation: relu dropout: 0.1 norm: LN d_ff: 1024 seq_len: 150 use_preln: false bias: false dilation_cycle: [0,1,2] conv: dilation_cycle_length: 10 unconditional: false noise_schedule_start: 0.01 noise_schedule_end: 0.7 n_noise_schedule: 150 Infer: eps: 1 seq_len: 25 Optim: Schedule: args: lambda: val: 10 multiplicative: val: 10 step: gamma: 0.99995 step_size: 10 name: step warm_up: 3000 args: adam: betas: - 0.9 - 0.999 eps: 1.0e-08 rmsprop: eps: 1.0e-08 sgd: momentum: 0.9 name: adam Validation: render: true apply_dropout: false render_every_n_epochs: 1 max_render_clips: 10 gen_synth_ctrl: false lr: 0.0006 batch_size: 80 num_dataloader_workers: 1 pruning_amount: 0.0 quantization: false Trainer: accelerator: gpu devices: [2] accumulate_grad_batches: 1 default_root_dir: results/training/dance_LDA gradient_clip_val: 25 deterministic: false fast_dev_run: false max_epochs: 10 min_epochs: 1 precision: 32 resume_from_checkpoint: null ================================================ FILE: models/BaseModel.py ================================================ # Copyright 2023 Motorica AB, Inc. All Rights Reserved. import os import torch import numpy as np from torch.optim import SGD, Adam, RMSprop from torch.optim.lr_scheduler import LambdaLR, MultiplicativeLR, StepLR, CosineAnnealingWarmRestarts from pytorch_lightning import LightningModule from utils.logging_mixin import LoggingMixin from typing import Tuple, Optional, Union, Dict from argparse import Namespace class BaseModel(LoggingMixin, LightningModule): def __init__(self, conf: Optional[Union[Dict, Namespace]] = None, **kwargs): super().__init__() self.save_hyperparameters(conf) scalers = self.hparams.Data["scalers"] #Get input and output scalers from hparams input_means=np.array([]) input_stds=np.array([]) if scalers["in_scaler"] is not None: input_means = scalers["in_scaler"].mean_ input_stds = scalers["in_scaler"].scale_ self.input_means = torch.from_numpy(input_means) self.input_scales = torch.from_numpy(input_stds) self.output_means = torch.from_numpy(scalers["out_scaler"].mean_) self.output_scales = torch.from_numpy(scalers["out_scaler"].scale_) def get_scalers(self): return self.hparams.Data["scalers"] # standarize input def standardizeInput(self, input_tensor): return ((input_tensor - self.input_means.type_as(input_tensor)) / self.input_scales.type_as(input_tensor)) # standarize output def standardizeOutput(self, output_tensor): return ((output_tensor - self.output_means.type_as(output_tensor)) / self.output_scales.type_as(output_tensor)) # Add scale and means to output def destandardizeInput(self, input_tensor): return (input_tensor * self.input_scales.type_as(input_tensor) + self.input_means.type_as(input_tensor)) # Add scale and means to output def destandardizeOutput(self, predictions): return (predictions * self.output_scales.type_as(predictions) + self.output_means.type_as(predictions)) def configure_optimizers(self): lr_params = self.hparams.Optim optim_args = lr_params["args"][lr_params["name"]] optimizers = {"adam": Adam, "sgd": SGD, "rmsprop": RMSprop} # Define optimizer optimizer = optimizers[lr_params["name"]]( self.parameters(), lr=self.hparams.lr, **optim_args ) # Define Learning Rate Scheduling def lambda1(val): return lambda epoch: epoch // val sched_params = self.hparams.Optim["Schedule"] sched_name = sched_params["name"] if not sched_name: return optimizer sched_args = sched_params["args"][sched_name] if sched_name == "step": scheduler = StepLR(optimizer, **sched_args) elif sched_name == "multiplicative": scheduler = MultiplicativeLR( optimizer, lr_lambda=[lambda1(sched_args["val"])] ) elif sched_name == "lambda": scheduler = LambdaLR(optimizer, lr_lambda=[lambda1(sched_args["val"])]) elif sched_name == "cos_warm": scheduler = CosineAnnealingWarmRestarts(optimizer, T_0=sched_args["T_0"]) else: raise NotImplementedError("Unimplemented Scheduler!") #return [optimizer], [scheduler] return { "optimizer": optimizer, "lr_scheduler": { "scheduler": scheduler, "interval": "step", }, } # learning rate warm-up def on_before_optimizer_step(self, optimizer, optimizer_idx): lr = self.hparams.lr #warm up lr warm_up = self.hparams.Optim["Schedule"]["warm_up"] if self.trainer.global_step < warm_up: lr_scale = min(1.0, float(self.trainer.global_step + 1) / warm_up) lr *= lr_scale for pg in optimizer.param_groups: pg["lr"] = lr ================================================ FILE: models/LightningModel.py ================================================ # Copyright 2023 Motorica AB, Inc. All Rights Reserved. import os import sys import numpy as np import torch from torch import nn, Tensor from torch.nn import functional as F from models.BaseModel import BaseModel from models.nn import LDA from typing import Tuple, Optional, Union, Dict from argparse import Namespace class LitLDA(BaseModel): def __init__(self, conf, **kwargs): super().__init__(conf) self.input_dim = 0 self.style_dim = 0 if self.hparams.Data["scalers"]["in_scaler"] is not None: self.input_dim = self.hparams.Data["scalers"]["in_scaler"].mean_.shape[0] if self.hparams.Data["scalers"]["style_scaler"] is not None: self.style_dim = self.hparams.Data["scalers"]["style_scaler"].mean_.shape[0] self.pose_dim = self.hparams.Data["scalers"]["out_scaler"].mean_.shape[0] self.g_cond_dim = self.style_dim self.unconditional = self.input_dim == 0 n_timesteps = self.hparams.Data["segment_length"] diff_params = self.hparams.Diffusion beta_min = diff_params["noise_schedule_start"] beta_max = diff_params["noise_schedule_end"] self.n_noise_schedule = diff_params["n_noise_schedule"] self.noise_schedule_name = "linear" self.noise_schedule = torch.linspace(beta_min, beta_max, self.n_noise_schedule) self.noise_level = torch.cumprod(1 - self.noise_schedule, dim=0) nn_name = diff_params["name"] nn_args = diff_params["args"][nn_name] self.diffusion_model = LDA(self.pose_dim, self.hparams.Diffusion["residual_layers"], self.hparams.Diffusion["residual_channels"], self.hparams.Diffusion["embedding_dim"], self.input_dim, self.g_cond_dim, self.n_noise_schedule, nn_name, nn_args) self.loss_fn = nn.MSELoss() def get_input_dim(self): return self.input_dim def get_style_dim(self): return self.style_dim def get_pose_dim(self): return self.pose_dim def diffusion(self, poses, t): N, T, C = poses.shape noise = torch.randn_like(poses) noise_scale = self.noise_level.type_as(noise)[t].unsqueeze(1).unsqueeze(2).repeat(1,T,C) noise_scale_sqrt = noise_scale**0.5 noisy_poses = noise_scale_sqrt * poses + (1.0 - noise_scale)**0.5 * noise return noisy_poses, noise def forward(self, batch): ctrl, global_cond, poses =batch N, T, C = poses.shape num_noisesteps = self.n_noise_schedule t = torch.randint(0, num_noisesteps, [N], device=poses.device) noise = torch.randn_like(poses) noise_scale = self.noise_level.type_as(noise)[t].unsqueeze(1).unsqueeze(2).repeat(1,T,C) noise_scale_sqrt = noise_scale**0.5 noisy_poses = noise_scale_sqrt * poses + (1.0 - noise_scale)**0.5 * noise noisy_poses, noise = self.diffusion(poses, t) predicted = self.diffusion_model(noisy_poses, ctrl, global_cond, t) loss = self.loss_fn(noise, predicted.squeeze(1)) return loss def training_step(self, batch, batch_idx): loss = self(batch) self.log('Loss/train', loss, sync_dist=True) return loss def validation_step(self, batch, batch_idx): loss = self(batch) self.log('val_loss', loss, prog_bar=True, sync_dist=True) if (self.trainer.global_step > 0 and batch_idx==0 and (self.trainer.current_epoch == self.trainer.max_epochs-1 or self.trainer.current_epoch % self.hparams.Validation["render_every_n_epochs"]==0)): # Log results for the validation data self.synthesize_and_log(batch, "val") output = {"val_loss": loss} return output def validation_epoch_end(self, outputs): avg_loss = torch.stack([x["val_loss"] for x in outputs]).mean() self.log('Loss/val', avg_loss, sync_dist=True) def synthesize_and_log(self, batch, log_prefix): ctrl, g_cond, _ =batch clips = self.synthesize(ctrl, g_cond) self.log_jerk(clips[:,:,:self.pose_dim], log_prefix) file_name = f"{self.current_epoch}_{self.global_step}_{log_prefix}" self.log_results(clips.cpu().detach().numpy(), file_name, log_prefix, render_video=False) def test_step(self, batch, batch_idx): loss = self(batch) self.synthesize_and_log(batch, "test") output = {"test_loss": loss} return output def synthesize(self, ctrl, global_cond): print("synthesize") training_noise_schedule = self.noise_schedule.to(ctrl.device) inference_noise_schedule = training_noise_schedule talpha = 1 - training_noise_schedule talpha_cum = torch.cumprod(talpha, dim=0) beta = inference_noise_schedule alpha = 1 - beta alpha_cum = torch.cumprod(alpha, dim=0) T = [] for s in range(len(inference_noise_schedule)): for t in range(len(training_noise_schedule) - 1): if talpha_cum[t+1] <= alpha_cum[s] <= talpha_cum[t]: twiddle = (talpha_cum[t]**0.5 - alpha_cum[s]**0.5) / (talpha_cum[t]**0.5 - talpha_cum[t+1]**0.5) T.append(t + twiddle) break if len(ctrl.shape) == 2:# Expand rank 2 tensors by adding a batch dimension. ctrl = ctrl.unsqueeze(0) global_cond = global_cond.unsqueeze(0) poses = torch.randn(ctrl.shape[0], ctrl.shape[1], self.pose_dim, device=ctrl.device) nbatch = poses.size(0) noise_scale = (alpha_cum**0.5).type_as(poses).unsqueeze(1) for n in range(len(alpha) - 1, -1, -1): c1 = 1 / alpha[n]**0.5 c2 = beta[n] / (1 - alpha_cum[n])**0.5 poses = c1 * (poses - c2 * self.diffusion_model(poses, ctrl, global_cond, T[n].unsqueeze(-1)).squeeze(1)) if n > 0: noise = torch.randn_like(poses) sigma = ((1.0 - alpha_cum[n-1]) / (1.0 - alpha_cum[n]) * beta[n])**0.5 poses += sigma * noise anim_clip = self.destandardizeOutput(poses) if not self.unconditional: out_ctrl = self.destandardizeInput(ctrl) anim_clip = torch.cat((anim_clip, out_ctrl), dim=2) return anim_clip ================================================ FILE: models/nn.py ================================================ # Copyright 2023 Motorica AB, Inc. All Rights Reserved. import torch import torch.nn as nn import torch.nn.functional as F from models.transformer.tisa_transformer import TisaTransformer from math import sqrt class Conv1dLayer(nn.Module): def __init__(self, in_channels, out_channels, kernel_size, padding=0, dilation=1): super().__init__() self.conv1d = nn.Conv1d(in_channels, out_channels, kernel_size=kernel_size, padding=padding, dilation=dilation) nn.init.kaiming_normal_(self.conv1d.weight) def forward(self, x): return self.conv1d(x.permute(0,2,1)).permute(0,2,1) def silu(x): return x * torch.sigmoid(x) class DiffusionEmbedding(nn.Module): def __init__(self, max_steps, in_channels, hidden_channels): super().__init__() self.in_channels = in_channels self.register_buffer('embedding', self._build_embedding(max_steps), persistent=False) self.projection1 = nn.Linear(in_channels, hidden_channels) self.projection2 = nn.Linear(hidden_channels, hidden_channels) def forward(self, diffusion_step): if diffusion_step.dtype in [torch.int32, torch.int64]: x = self.embedding[diffusion_step] else: x = self._lerp_embedding(diffusion_step) x = self.projection1(x) x = silu(x) x = self.projection2(x) x = silu(x) return x def _lerp_embedding(self, t): low_idx = torch.floor(t).long() high_idx = torch.ceil(t).long() low = self.embedding[low_idx] high = self.embedding[high_idx] return low + (high - low) * (t - low_idx) def _build_embedding(self, max_steps): steps = torch.arange(max_steps).unsqueeze(1) # [T,1] dims = torch.arange(64).unsqueeze(0) # [1,64] table = steps * 10.0**(dims * 4.0 / 63.0) # [T,64] table = torch.cat([torch.sin(table), torch.cos(table)], dim=1) return table class ResidualBlock(nn.Module): def __init__(self, residual_channels, embedding_dim, l_cond_dim, nn_name, nn_args, index): super().__init__() if nn_name=="tisa": dilation_cycle = nn_args["dilation_cycle"] dilation=dilation_cycle[(index % len(dilation_cycle))] self.nn = TisaTransformer(residual_channels, 2 * residual_channels, d_model=residual_channels, num_blocks=nn_args["num_blocks"], num_heads=nn_args["num_heads"], activation=nn_args["activation"], norm=nn_args["norm"], drop_prob=nn_args["dropout"], d_ff=nn_args["d_ff"], seqlen=nn_args["seq_len"], use_preln=nn_args["use_preln"], bias=nn_args["bias"], dilation=dilation) elif nn_name=="conv": dilation=2**(index % nn_args["dilation_cycle_length"]) self.nn = Conv1dLayer(residual_channels, 2 * residual_channels, 3, padding=dilation, dilation=dilation) else: raise ValueError(f"Unknown nn_name: {nn_name}") self.l_cond_dim = l_cond_dim self.diffusion_projection = nn.Linear(embedding_dim, residual_channels) self.local_cond_projection = nn.Linear(l_cond_dim, residual_channels) self.output_projection = Conv1dLayer(residual_channels, 2 * residual_channels, 1) self.residual_channels = residual_channels def forward(self, x, diffusion_step, local_cond): diffusion_step = self.diffusion_projection(diffusion_step).unsqueeze(1) y = x + diffusion_step if self.l_cond_dim > 0: y += self.local_cond_projection(local_cond) y = self.nn(y).squeeze(-1) gate, filter = torch.chunk(y, 2, dim=2) y = torch.sigmoid(gate) * torch.tanh(filter) y = self.output_projection(y) residual, skip = torch.chunk(y, 2, dim=2) return (x + residual) / sqrt(2.0), skip class LDA(nn.Module): def __init__(self, pose_dim, residual_layers, residual_channels, embedding_dim, l_cond_dim, g_cond_dim, n_noise_schedule, nn_name, nn_args): super().__init__() self.input_projection = Conv1dLayer(pose_dim, residual_channels, 1) self.diffusion_embedding = DiffusionEmbedding(n_noise_schedule, 128, embedding_dim) self.residual_layers = nn.ModuleList([ ResidualBlock(residual_channels, embedding_dim, l_cond_dim + g_cond_dim, nn_name, nn_args, i) for i in range(residual_layers) ]) self.skip_projection = Conv1dLayer(residual_channels, residual_channels, 1) self.output_projection = Conv1dLayer(residual_channels, pose_dim, 1) nn.init.zeros_(self.output_projection.conv1d.weight) self.l_cond_dim = l_cond_dim self.g_cond_dim = g_cond_dim def forward(self, x, local_cond, global_cond, diffusion_step): x = self.input_projection(x) x = F.relu(x) diffusion_step = self.diffusion_embedding(diffusion_step) if self.g_cond_dim > 0: local_cond=torch.cat((local_cond, global_cond), dim=2) skip = None i=1 for layer in self.residual_layers: x, skip_connection = layer(x, diffusion_step, local_cond) skip = skip_connection if skip is None else skip_connection + skip if skip is not None: x = skip / sqrt(len(self.residual_layers)) x = self.skip_projection(x) x = F.relu(x) x = self.output_projection(x) return x ================================================ FILE: models/transformer/tisa_transformer.py ================================================ # Copyright 2023 Motorica AB, Inc. All Rights Reserved. import torch import torch.nn as nn import torch.nn.functional as F from models.transformer.tisa_v2 import TisaV2 class TisaTransformer(nn.Module): def __init__( self, in_channels, out_channels, d_model, num_blocks, num_heads, activation, norm, drop_prob, d_ff=2048, tisa_num_kernels=21, seqlen=128, use_preln=False, bias=False, dilation=1 ): super(TisaTransformer, self).__init__() self.in_proj = nn.Linear(in_channels, d_model) self.attention_blocks = nn.ModuleList( [ AttnBlock(d_model, num_heads, activation, norm, drop_prob, tisa_num_kernels, seqlen, use_preln, d_ff, bias=bias, dilation=dilation) for _ in range(num_blocks) ] ) self.out_proj = nn.Linear(d_model, out_channels) def forward(self, x): x = self.in_proj(x) for layer in self.attention_blocks: x = layer(x) x = self.out_proj(x) return x class Norm(nn.Module): def __init__(self, d_model, eps = 1e-6): super().__init__() self.size = d_model # create two learnable parameters to calibrate normalisation self.alpha = nn.Parameter(torch.ones(self.size)) self.bias = nn.Parameter(torch.zeros(self.size)) self.eps = eps def forward(self, x): norm = self.alpha * (x - x.mean(dim=-1, keepdim=True)) \ / (x.std(dim=-1, keepdim=True) + self.eps) + self.bias return norm class AttnBlock(nn.Module): def __init__(self, d_model, num_heads, activation, norm, drop_prob, tisa_num_kernels, seqlen, use_preln=False, d_ff=2048, bias=False, dilation=1): super(AttnBlock, self).__init__() self.use_preln = use_preln self.attn = GatedAttn(d_model, num_heads=num_heads, activation=activation, seqlen=seqlen, drop_prob=drop_prob, tisa_num_kernels=tisa_num_kernels) if (dilation>0): self.ff = ConvLayer(d_model, d_ff, activation=activation, dilation=dilation, dropout=drop_prob, bias=bias) else: self.ff = FeedForward(d_model, d_ff, activation=activation, dropout=drop_prob, bias=bias) if norm == "T5": self.norm_1 = T5LayerNorm(d_model) self.norm_2 = T5LayerNorm(d_model) elif norm == "LN": self.norm_1 = nn.LayerNorm(d_model) self.norm_2 = nn.LayerNorm(d_model) else: raise ValueError(f"unknown norm: {norm}") self.dropout_1 = nn.Dropout(drop_prob) self.dropout_2 = nn.Dropout(drop_prob) def forward(self, x): if self.use_preln: x = self.dropout_1(self.attn(self.norm_2(x))) + x else: x = self.dropout_1(self.attn(x)) + x x = self.norm_2(x) if self.use_preln: x = self.dropout_2(self.ff(self.norm_1(x))) + x else: x = self.dropout_2(self.ff(x)) + x x = self.norm_1(x) return x class T5LayerNorm(nn.Module): def __init__(self, hidden_size, eps=1e-6): """ Construct a layernorm module in the T5 style. No bias and no subtraction of mean. """ super().__init__() self.weight = nn.Parameter(torch.ones(hidden_size)) self.variance_epsilon = eps def forward(self, hidden_states): # T5 uses a layer_norm which only scales and doesn't shift, which is also known as Root Mean # Square Layer Normalization https://arxiv.org/abs/1910.07467 thus varience is calculated # w/o mean and there is no bias. Additionally we want to make sure that the accumulation for # half-precision inputs is done in fp32 variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) return self.weight * hidden_states class ConvLayer(nn.Module): def __init__(self, d_model, d_ff=2048, activation="relu", dilation=1, dropout = 0.1, bias=False): super().__init__() # We set d_ff as a default to 2048 self.conv_1 = nn.Conv1d(d_model, d_ff, 3, padding=dilation, dilation=dilation, bias=bias) self.dropout = nn.Dropout(dropout) self.conv_2 = nn.Conv1d(d_ff, d_model, 3, padding=dilation, dilation=dilation, bias=bias) if activation=="relu": self.act = nn.ReLU() elif activation=="gelu": self.act = nn.GELU() def forward(self, x): x = self.dropout(self.act(self.conv_1(x.permute(0,2,1)))) x = self.conv_2(x) return x.permute(0,2,1) class FeedForward(nn.Module): def __init__(self, d_model, d_ff=2048, activation="RELU", dropout = 0.1, bias=False): super().__init__() # We set d_ff as a default to 2048 self.linear_1 = nn.Linear(d_model, d_ff, bias=bias) self.dropout = nn.Dropout(dropout) self.linear_2 = nn.Linear(d_ff, d_model, bias=bias) if activation=="relu": self.act = nn.ReLU() elif activation=="gelu": self.act = nn.GELU() def forward(self, x): x = self.dropout(self.act(self.linear_1(x))) x = self.linear_2(x) return x class GatedAttn(nn.Module): """Gated Multi-Head Self-Attention Block Based on the paper: "Attention Is All You Need" by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin (https://arxiv.org/abs/1706.03762). Args: d_model (int): Number of channels in the input. num_heads (int): Number of attention heads. drop_prob (float): Dropout probability. """ def __init__(self, d_model, num_heads=4, activation="RELU", seqlen=128, drop_prob=0.0, tisa_num_kernels=21): super(GatedAttn, self).__init__() assert d_model%num_heads==0, f"num_heads ({num_heads}) is not evenly divisible by d_model ({d_model})" self.d_model = d_model self.num_heads = num_heads if drop_prob>0: self.dropout = nn.Dropout(drop_prob) else: self.dropout = None self.in_proj = nn.Linear(d_model, 3 * d_model, bias=False) self.gate = nn.Linear(d_model, 2 * d_model) self.key_depth_per_head = torch.tensor(self.d_model / self.num_heads) self.position_scorer = TisaV2(self.num_heads, tisa_num_kernels=tisa_num_kernels, tisa_dropout_prob=drop_prob, num_position_agnostic_heads=0, max_field_view=seqlen//2, min_field_view=5) self.position_scorer._init_weights() def forward(self, x): b, h, c = x.size() _, seq_len, num_channels = x.size() # Compute q, k, v memory, query = torch.split(self.in_proj(x), (2 * c, c), dim=-1) q = self.split_last_dim(query, self.num_heads) k, v = [ self.split_last_dim(tensor, self.num_heads) for tensor in torch.split(memory, self.d_model, dim=2) ] # Compute attention and reshape x = self.dot_product_attention(q, k, v) x = self.combine_last_two_dim(x.permute(0, 2, 1, 3)) x = self.gate(x) a, b = x.chunk(2, dim=-1) x = a * torch.sigmoid(b) return x def dot_product_attention(self, q, k, v): """Dot-product attention. Args: q (torch.Tensor): Queries of shape (batch, heads, length_q, depth_k) k (torch.Tensor): Keys of shape (batch, heads, length_kv, depth_k) v (torch.Tensor): Values of shape (batch, heads, length_kv, depth_v) bias (bool): Use bias for attention. Returns: attn (torch.Tensor): Output of attention mechanism. """ weights = torch.matmul(q, k.permute(0, 1, 3, 2)) weights /= torch.sqrt(self.key_depth_per_head) seq_len = weights.shape[-1] weights += self.position_scorer(seq_len) weights = F.softmax(weights, dim=-1) if self.dropout is not None: weights = self.dropout(weights) attn = torch.matmul(weights, v) return attn @staticmethod def split_last_dim(x, n): """Reshape x so that the last dimension becomes two dimensions. The first of these two dimensions is n. Args: x (torch.Tensor): Tensor with shape (..., m) n (int): Size of second-to-last dimension. Returns: ret (torch.Tensor): Resulting tensor with shape (..., n, m/n) """ #import pdb;pdb.set_trace() old_shape = list(x.size()) last = old_shape[-1] if last is not None: new_shape = old_shape[:-1] + [n] + [last // n] else: new_shape = old_shape[:-1] + [n] ret = x.view(new_shape) return ret.permute(0, 2, 1, 3) @staticmethod def combine_last_two_dim(x): """Merge the last two dimensions of `x`. Args: x (torch.Tensor): Tensor with shape (..., m, n) Returns: ret (torch.Tensor): Resulting tensor with shape (..., m * n) """ old_shape = list(x.size()) a, b = old_shape[-2:] new_shape = old_shape[:-2] + [a * b] ret = x.contiguous().view(new_shape) return ret ================================================ FILE: models/transformer/tisa_v2.py ================================================ # Copyright 2023 Ulme Wennberg, Inc. All Rights Reserved. import torch from torch import nn from typing import List import math class TisaV2(nn.Module): def __init__(self, num_attention_heads, tisa_num_kernels, tisa_dropout_prob, num_position_agnostic_heads, max_field_view, min_field_view, p_eps = 1e-8): super().__init__() self.num_attention_heads = num_attention_heads self.num_kernels = tisa_num_kernels self.tisa_dropout_prob = tisa_dropout_prob self.num_position_agnostic_heads = num_position_agnostic_heads self.max_field_view = max_field_view self.min_field_view = min_field_view self.p_eps = p_eps self.eps = 1e-8 self.offsets = nn.Parameter( torch.zeros(1, self.num_kernels, self.num_attention_heads, 1, 1) ) self.amplitudes = nn.Parameter( torch.zeros(1, self.num_kernels, self.num_attention_heads, 1, 1) ) self.sharpness = nn.Parameter( torch.zeros(1, self.num_kernels, self.num_attention_heads, 1, 1) ) self.bias = nn.Parameter(torch.zeros(1, self.num_attention_heads, 1, 1)) self.dropout = nn.Dropout(self.tisa_dropout_prob) self.num_position_agnostic_heads = self.num_position_agnostic_heads self.num_position_aware_heads = ( self.num_attention_heads - self.num_position_agnostic_heads ) self.position_agnostic_heads = torch.arange( self.num_attention_heads - self.num_position_agnostic_heads, self.num_attention_heads, ) assert 0 < self.p_eps < 1 """ exp ( - field_view * m) = p_eps - log(p_eps) = field_view * m m = - log(p_eps) / field_view """ self.one_side_min_field_view = self.min_field_view / 2 self.one_side_max_field_view = self.max_field_view / 2 self.first_slope = -math.log(self.p_eps) / self.one_side_min_field_view self.last_slope = -math.log(self.p_eps) / self.one_side_max_field_view self.slopes = nn.Parameter( ( self.first_slope * (self.last_slope / self.first_slope) ** ( torch.arange(self.num_attention_heads) / (self.num_position_aware_heads - 1) ) ).reshape(self.num_attention_heads, 1, 1), requires_grad=False, ) # Disable exponential decay for position agnostic heads self.slopes[self.position_agnostic_heads, 0, 0] = 0.0 def create_relative_offsets(self, seq_len): """Creates offsets for all the relative distances between -seq_len + 1 to seq_len - 1.""" return ( torch.arange(-seq_len, seq_len + 1, device=self.offsets.device) .unsqueeze(0) .unsqueeze(0) ) def forward(self, dim=-1, skip_apply_dropout=False): """Computes the translation-invariant positional contribution to the attention matrix in the self-attention module of transformer models.""" indices_from = torch.arange(dim, device=self.offsets.device).unsqueeze(0).unsqueeze(0) indices_to = indices_from if not self.num_kernels: return torch.zeros( (self.num_attention_heads, indices_from.shape[-1], indices_to.shape[-1]) ) params = (indices_to.unsqueeze(-1) - indices_from.unsqueeze(-2)).unsqueeze(-4) exponential_decay_arguments = -params.abs() * self.slopes params = params.unsqueeze(-4) - self.offsets params = params / self.sharpness params = self.amplitudes.abs() * torch.sigmoid(self.amplitudes.sign() * params) if self.training and not skip_apply_dropout: params = self.dropout(params) params = params.sum(dim=-4) # Make final dimensions completely position agnostic params[:, :, self.position_agnostic_heads] = 0.0 params += self.eps + self.bias.abs() params = torch.log(params) params = params + exponential_decay_arguments return params.squeeze(0) def _init_weights(self): """Initialize the weights""" torch.nn.init.normal_(self.offsets, mean=0.0, std=15.0) torch.nn.init.normal_(self.amplitudes, mean=0.0, std=0.01) self.sharpness.data.fill_(5.0) self.bias.data.fill_(1.0) ================================================ FILE: pretrained_models/.gitkeep ================================================ ================================================ FILE: pretrained_models/README.md ================================================ Please [download our pretrained dance models here](https://zenodo.org/record/8156769) and place them in this folder. ================================================ FILE: pymo/Pivots.py ================================================ import numpy as np from pymo.Quaternions import Quaternions class Pivots: """ Pivots is an ndarray of angular rotations This wrapper provides some functions for working with pivots. These are particularly useful as a number of atomic operations (such as adding or subtracting) cannot be achieved using the standard arithmatic and need to be defined differently to work correctly """ def __init__(self, ps): self.ps = np.array(ps) def __str__(self): return "Pivots("+ str(self.ps) + ")" def __repr__(self): return "Pivots("+ repr(self.ps) + ")" def __add__(self, other): return Pivots(np.arctan2(np.sin(self.ps + other.ps), np.cos(self.ps + other.ps))) def __sub__(self, other): return Pivots(np.arctan2(np.sin(self.ps - other.ps), np.cos(self.ps - other.ps))) def __mul__(self, other): return Pivots(self.ps * other.ps) def __div__(self, other): return Pivots(self.ps / other.ps) def __mod__(self, other): return Pivots(self.ps % other.ps) def __pow__(self, other): return Pivots(self.ps ** other.ps) def __lt__(self, other): return self.ps < other.ps def __le__(self, other): return self.ps <= other.ps def __eq__(self, other): return self.ps == other.ps def __ne__(self, other): return self.ps != other.ps def __ge__(self, other): return self.ps >= other.ps def __gt__(self, other): return self.ps > other.ps def __abs__(self): return Pivots(abs(self.ps)) def __neg__(self): return Pivots(-self.ps) def __iter__(self): return iter(self.ps) def __len__(self): return len(self.ps) def __getitem__(self, k): return Pivots(self.ps[k]) def __setitem__(self, k, v): self.ps[k] = v.ps def _ellipsis(self): return tuple(map(lambda x: slice(None), self.shape)) def quaternions(self, plane='xz'): fa = self._ellipsis() axises = np.ones(self.ps.shape + (3,)) axises[fa + ("xyz".index(plane[0]),)] = 0.0 axises[fa + ("xyz".index(plane[1]),)] = 0.0 return Quaternions.from_angle_axis(self.ps, axises) def directions(self, plane='xz'): dirs = np.zeros((len(self.ps), 3)) dirs["xyz".index(plane[0])] = np.sin(self.ps) dirs["xyz".index(plane[1])] = np.cos(self.ps) return dirs def normalized(self): xs = np.copy(self.ps) while np.any(xs > np.pi): xs[xs > np.pi] = xs[xs > np.pi] - 2 * np.pi while np.any(xs < -np.pi): xs[xs < -np.pi] = xs[xs < -np.pi] + 2 * np.pi return Pivots(xs) def interpolate(self, ws): dir = np.average(self.directions, weights=ws, axis=0) return np.arctan2(dir[2], dir[0]) def copy(self): return Pivots(np.copy(self.ps)) @property def shape(self): return self.ps.shape @classmethod def from_quaternions(cls, qs, forward='z', plane='xz'): ds = np.zeros(qs.shape + (3,)) ds[...,'xyz'.index(forward)] = 1.0 return Pivots.from_directions(qs * ds, plane=plane) @classmethod def from_directions(cls, ds, plane='xz'): ys = ds[...,'xyz'.index(plane[0])] xs = ds[...,'xyz'.index(plane[1])] return Pivots(np.arctan2(ys, xs)) ================================================ FILE: pymo/Quaternions.py ================================================ import numpy as np class Quaternions: """ Quaternions is a wrapper around a numpy ndarray that allows it to act as if it were an narray of a quaternion data type. Therefore addition, subtraction, multiplication, division, negation, absolute, are all defined in terms of quaternion operations such as quaternion multiplication. This allows for much neater code and many routines which conceptually do the same thing to be written in the same way for point data and for rotation data. The Quaternions class has been desgined such that it should support broadcasting and slicing in all of the usual ways. """ def __init__(self, qs): if isinstance(qs, np.ndarray): if len(qs.shape) == 1: qs = np.array([qs]) self.qs = qs return if isinstance(qs, Quaternions): self.qs = qs.qs return raise TypeError('Quaternions must be constructed from iterable, numpy array, or Quaternions, not %s' % type(qs)) def __str__(self): return "Quaternions("+ str(self.qs) + ")" def __repr__(self): return "Quaternions("+ repr(self.qs) + ")" """ Helper Methods for Broadcasting and Data extraction """ @classmethod def _broadcast(cls, sqs, oqs, scalar=False): if isinstance(oqs, float): return sqs, oqs * np.ones(sqs.shape[:-1]) ss = np.array(sqs.shape) if not scalar else np.array(sqs.shape[:-1]) os = np.array(oqs.shape) if len(ss) != len(os): raise TypeError('Quaternions cannot broadcast together shapes %s and %s' % (sqs.shape, oqs.shape)) if np.all(ss == os): return sqs, oqs if not np.all((ss == os) | (os == np.ones(len(os))) | (ss == np.ones(len(ss)))): raise TypeError('Quaternions cannot broadcast together shapes %s and %s' % (sqs.shape, oqs.shape)) sqsn, oqsn = sqs.copy(), oqs.copy() for a in np.where(ss == 1)[0]: sqsn = sqsn.repeat(os[a], axis=a) for a in np.where(os == 1)[0]: oqsn = oqsn.repeat(ss[a], axis=a) return sqsn, oqsn """ Adding Quaterions is just Defined as Multiplication """ def __add__(self, other): return self * other def __sub__(self, other): return self / other """ Quaterion Multiplication """ def __mul__(self, other): """ Quaternion multiplication has three main methods. When multiplying a Quaternions array by Quaternions normal quaternion multiplication is performed. When multiplying a Quaternions array by a vector array of the same shape, where the last axis is 3, it is assumed to be a Quaternion by 3D-Vector multiplication and the 3D-Vectors are rotated in space by the Quaternions. When multipplying a Quaternions array by a scalar or vector of different shape it is assumed to be a Quaternions by Scalars multiplication and the Quaternions are scaled using Slerp and the identity quaternions. """ """ If Quaternions type do Quaternions * Quaternions """ if isinstance(other, Quaternions): sqs, oqs = Quaternions._broadcast(self.qs, other.qs) q0 = sqs[...,0]; q1 = sqs[...,1]; q2 = sqs[...,2]; q3 = sqs[...,3]; r0 = oqs[...,0]; r1 = oqs[...,1]; r2 = oqs[...,2]; r3 = oqs[...,3]; qs = np.empty(sqs.shape) qs[...,0] = r0 * q0 - r1 * q1 - r2 * q2 - r3 * q3 qs[...,1] = r0 * q1 + r1 * q0 - r2 * q3 + r3 * q2 qs[...,2] = r0 * q2 + r1 * q3 + r2 * q0 - r3 * q1 qs[...,3] = r0 * q3 - r1 * q2 + r2 * q1 + r3 * q0 return Quaternions(qs) """ If array type do Quaternions * Vectors """ if isinstance(other, np.ndarray) and other.shape[-1] == 3: vs = Quaternions(np.concatenate([np.zeros(other.shape[:-1] + (1,)), other], axis=-1)) return (self * (vs * -self)).imaginaries """ If float do Quaternions * Scalars """ if isinstance(other, np.ndarray) or isinstance(other, float): return Quaternions.slerp(Quaternions.id_like(self), self, other) raise TypeError('Cannot multiply/add Quaternions with type %s' % str(type(other))) def __div__(self, other): """ When a Quaternion type is supplied, division is defined as multiplication by the inverse of that Quaternion. When a scalar or vector is supplied it is defined as multiplicaion of one over the supplied value. Essentially a scaling. """ if isinstance(other, Quaternions): return self * (-other) if isinstance(other, np.ndarray): return self * (1.0 / other) if isinstance(other, float): return self * (1.0 / other) raise TypeError('Cannot divide/subtract Quaternions with type %s' + str(type(other))) def __eq__(self, other): return self.qs == other.qs def __ne__(self, other): return self.qs != other.qs def __neg__(self): """ Invert Quaternions """ return Quaternions(self.qs * np.array([[1, -1, -1, -1]])) def __abs__(self): """ Unify Quaternions To Single Pole """ qabs = self.normalized().copy() top = np.sum(( qabs.qs) * np.array([1,0,0,0]), axis=-1) bot = np.sum((-qabs.qs) * np.array([1,0,0,0]), axis=-1) qabs.qs[top < bot] = -qabs.qs[top < bot] return qabs def __iter__(self): return iter(self.qs) def __len__(self): return len(self.qs) def __getitem__(self, k): return Quaternions(self.qs[k]) def __setitem__(self, k, v): self.qs[k] = v.qs @property def lengths(self): return np.sum(self.qs**2.0, axis=-1)**0.5 @property def reals(self): return self.qs[...,0] @property def imaginaries(self): return self.qs[...,1:4] @property def shape(self): return self.qs.shape[:-1] def repeat(self, n, **kwargs): return Quaternions(self.qs.repeat(n, **kwargs)) def normalized(self): return Quaternions(self.qs / self.lengths[...,np.newaxis]) def log(self): norm = abs(self.normalized()) imgs = norm.imaginaries lens = np.sqrt(np.sum(imgs**2, axis=-1)) lens = np.arctan2(lens, norm.reals) / (lens + 1e-10) return imgs * lens[...,np.newaxis] def constrained(self, axis): rl = self.reals im = np.sum(axis * self.imaginaries, axis=-1) t1 = -2 * np.arctan2(rl, im) + np.pi t2 = -2 * np.arctan2(rl, im) - np.pi top = Quaternions.exp(axis[np.newaxis] * (t1[:,np.newaxis] / 2.0)) bot = Quaternions.exp(axis[np.newaxis] * (t2[:,np.newaxis] / 2.0)) img = self.dot(top) > self.dot(bot) ret = top.copy() ret[ img] = top[ img] ret[~img] = bot[~img] return ret def constrained_x(self): return self.constrained(np.array([1,0,0])) def constrained_y(self): return self.constrained(np.array([0,1,0])) def constrained_z(self): return self.constrained(np.array([0,0,1])) def dot(self, q): return np.sum(self.qs * q.qs, axis=-1) def copy(self): return Quaternions(np.copy(self.qs)) def reshape(self, s): self.qs.reshape(s) return self def interpolate(self, ws): return Quaternions.exp(np.average(abs(self).log, axis=0, weights=ws)) def euler(self, order='xyz'): q = self.normalized().qs q0 = q[...,0] q1 = q[...,1] q2 = q[...,2] q3 = q[...,3] es = np.zeros(self.shape + (3,)) if order == 'xyz': es[...,0] = np.arctan2(2 * (q0 * q1 + q2 * q3), 1 - 2 * (q1 * q1 + q2 * q2)) es[...,1] = np.arcsin((2 * (q0 * q2 - q3 * q1)).clip(-1,1)) es[...,2] = np.arctan2(2 * (q0 * q3 + q1 * q2), 1 - 2 * (q2 * q2 + q3 * q3)) elif order == 'yzx': es[...,0] = np.arctan2(2 * (q1 * q0 - q2 * q3), -q1 * q1 + q2 * q2 - q3 * q3 + q0 * q0) es[...,1] = np.arctan2(2 * (q2 * q0 - q1 * q3), q1 * q1 - q2 * q2 - q3 * q3 + q0 * q0) es[...,2] = np.arcsin((2 * (q1 * q2 + q3 * q0)).clip(-1,1)) else: raise NotImplementedError('Cannot convert from ordering %s' % order) """ # These conversion don't appear to work correctly for Maya. # http://bediyap.com/programming/convert-quaternion-to-euler-rotations/ if order == 'xyz': es[...,0] = np.arctan2(2 * (q0 * q3 - q1 * q2), q0 * q0 + q1 * q1 - q2 * q2 - q3 * q3) es[...,1] = np.arcsin((2 * (q1 * q3 + q0 * q2)).clip(-1,1)) es[...,2] = np.arctan2(2 * (q0 * q1 - q2 * q3), q0 * q0 - q1 * q1 - q2 * q2 + q3 * q3) elif order == 'yzx': es[...,0] = np.arctan2(2 * (q0 * q1 - q2 * q3), q0 * q0 - q1 * q1 + q2 * q2 - q3 * q3) es[...,1] = np.arcsin((2 * (q1 * q2 + q0 * q3)).clip(-1,1)) es[...,2] = np.arctan2(2 * (q0 * q2 - q1 * q3), q0 * q0 + q1 * q1 - q2 * q2 - q3 * q3) elif order == 'zxy': es[...,0] = np.arctan2(2 * (q0 * q2 - q1 * q3), q0 * q0 - q1 * q1 - q2 * q2 + q3 * q3) es[...,1] = np.arcsin((2 * (q0 * q1 + q2 * q3)).clip(-1,1)) es[...,2] = np.arctan2(2 * (q0 * q3 - q1 * q2), q0 * q0 - q1 * q1 + q2 * q2 - q3 * q3) elif order == 'xzy': es[...,0] = np.arctan2(2 * (q0 * q2 + q1 * q3), q0 * q0 + q1 * q1 - q2 * q2 - q3 * q3) es[...,1] = np.arcsin((2 * (q0 * q3 - q1 * q2)).clip(-1,1)) es[...,2] = np.arctan2(2 * (q0 * q1 + q2 * q3), q0 * q0 - q1 * q1 + q2 * q2 - q3 * q3) elif order == 'yxz': es[...,0] = np.arctan2(2 * (q1 * q2 + q0 * q3), q0 * q0 - q1 * q1 + q2 * q2 - q3 * q3) es[...,1] = np.arcsin((2 * (q0 * q1 - q2 * q3)).clip(-1,1)) es[...,2] = np.arctan2(2 * (q1 * q3 + q0 * q2), q0 * q0 - q1 * q1 - q2 * q2 + q3 * q3) elif order == 'zyx': es[...,0] = np.arctan2(2 * (q0 * q1 + q2 * q3), q0 * q0 - q1 * q1 - q2 * q2 + q3 * q3) es[...,1] = np.arcsin((2 * (q0 * q2 - q1 * q3)).clip(-1,1)) es[...,2] = np.arctan2(2 * (q0 * q3 + q1 * q2), q0 * q0 + q1 * q1 - q2 * q2 - q3 * q3) else: raise KeyError('Unknown ordering %s' % order) """ # https://github.com/ehsan/ogre/blob/master/OgreMain/src/OgreMatrix3.cpp # Use this class and convert from matrix return es def average(self): if len(self.shape) == 1: import numpy.core.umath_tests as ut system = ut.matrix_multiply(self.qs[:,:,np.newaxis], self.qs[:,np.newaxis,:]).sum(axis=0) w, v = np.linalg.eigh(system) qiT_dot_qref = (self.qs[:,:,np.newaxis] * v[np.newaxis,:,:]).sum(axis=1) return Quaternions(v[:,np.argmin((1.-qiT_dot_qref**2).sum(axis=0))]) else: raise NotImplementedError('Cannot average multi-dimensionsal Quaternions') def angle_axis(self): norm = self.normalized() s = np.sqrt(1 - (norm.reals**2.0)) s[s == 0] = 0.001 angles = 2.0 * np.arccos(norm.reals) axis = norm.imaginaries / s[...,np.newaxis] return angles, axis def transforms(self): qw = self.qs[...,0] qx = self.qs[...,1] qy = self.qs[...,2] qz = self.qs[...,3] x2 = qx + qx; y2 = qy + qy; z2 = qz + qz; xx = qx * x2; yy = qy * y2; wx = qw * x2; xy = qx * y2; yz = qy * z2; wy = qw * y2; xz = qx * z2; zz = qz * z2; wz = qw * z2; m = np.empty(self.shape + (3,3)) m[...,0,0] = 1.0 - (yy + zz) m[...,0,1] = xy - wz m[...,0,2] = xz + wy m[...,1,0] = xy + wz m[...,1,1] = 1.0 - (xx + zz) m[...,1,2] = yz - wx m[...,2,0] = xz - wy m[...,2,1] = yz + wx m[...,2,2] = 1.0 - (xx + yy) return m def ravel(self): return self.qs.ravel() @classmethod def id(cls, n): if isinstance(n, tuple): qs = np.zeros(n + (4,)) qs[...,0] = 1.0 return Quaternions(qs) if isinstance(n, int) or isinstance(n, long): qs = np.zeros((n,4)) qs[:,0] = 1.0 return Quaternions(qs) raise TypeError('Cannot Construct Quaternion from %s type' % str(type(n))) @classmethod def id_like(cls, a): qs = np.zeros(a.shape + (4,)) qs[...,0] = 1.0 return Quaternions(qs) @classmethod def exp(cls, ws): ts = np.sum(ws**2.0, axis=-1)**0.5 ts[ts == 0] = 0.001 ls = np.sin(ts) / ts qs = np.empty(ws.shape[:-1] + (4,)) qs[...,0] = np.cos(ts) qs[...,1] = ws[...,0] * ls qs[...,2] = ws[...,1] * ls qs[...,3] = ws[...,2] * ls return Quaternions(qs).normalized() @classmethod def slerp(cls, q0s, q1s, a): fst, snd = cls._broadcast(q0s.qs, q1s.qs) fst, a = cls._broadcast(fst, a, scalar=True) snd, a = cls._broadcast(snd, a, scalar=True) len = np.sum(fst * snd, axis=-1) neg = len < 0.0 len[neg] = -len[neg] snd[neg] = -snd[neg] amount0 = np.zeros(a.shape) amount1 = np.zeros(a.shape) linear = (1.0 - len) < 0.01 omegas = np.arccos(len[~linear]) sinoms = np.sin(omegas) amount0[ linear] = 1.0 - a[linear] amount1[ linear] = a[linear] amount0[~linear] = np.sin((1.0 - a[~linear]) * omegas) / sinoms amount1[~linear] = np.sin( a[~linear] * omegas) / sinoms return Quaternions( amount0[...,np.newaxis] * fst + amount1[...,np.newaxis] * snd) @classmethod def between(cls, v0s, v1s): a = np.cross(v0s, v1s) w = np.sqrt((v0s**2).sum(axis=-1) * (v1s**2).sum(axis=-1)) + (v0s * v1s).sum(axis=-1) return Quaternions(np.concatenate([w[...,np.newaxis], a], axis=-1)).normalized() @classmethod def from_angle_axis(cls, angles, axis): axis = axis / (np.sqrt(np.sum(axis**2, axis=-1)) + 1e-10)[...,np.newaxis] sines = np.sin(angles / 2.0)[...,np.newaxis] cosines = np.cos(angles / 2.0)[...,np.newaxis] return Quaternions(np.concatenate([cosines, axis * sines], axis=-1)) @classmethod def from_euler(cls, es, order='xyz', world=False): axis = { 'x' : np.array([1,0,0]), 'y' : np.array([0,1,0]), 'z' : np.array([0,0,1]), } q0s = Quaternions.from_angle_axis(es[...,0], axis[order[0]]) q1s = Quaternions.from_angle_axis(es[...,1], axis[order[1]]) q2s = Quaternions.from_angle_axis(es[...,2], axis[order[2]]) return (q2s * (q1s * q0s)) if world else (q0s * (q1s * q2s)) @classmethod def from_transforms(cls, ts): d0, d1, d2 = ts[...,0,0], ts[...,1,1], ts[...,2,2] q0 = ( d0 + d1 + d2 + 1.0) / 4.0 q1 = ( d0 - d1 - d2 + 1.0) / 4.0 q2 = (-d0 + d1 - d2 + 1.0) / 4.0 q3 = (-d0 - d1 + d2 + 1.0) / 4.0 q0 = np.sqrt(q0.clip(0,None)) q1 = np.sqrt(q1.clip(0,None)) q2 = np.sqrt(q2.clip(0,None)) q3 = np.sqrt(q3.clip(0,None)) c0 = (q0 >= q1) & (q0 >= q2) & (q0 >= q3) c1 = (q1 >= q0) & (q1 >= q2) & (q1 >= q3) c2 = (q2 >= q0) & (q2 >= q1) & (q2 >= q3) c3 = (q3 >= q0) & (q3 >= q1) & (q3 >= q2) q1[c0] *= np.sign(ts[c0,2,1] - ts[c0,1,2]) q2[c0] *= np.sign(ts[c0,0,2] - ts[c0,2,0]) q3[c0] *= np.sign(ts[c0,1,0] - ts[c0,0,1]) q0[c1] *= np.sign(ts[c1,2,1] - ts[c1,1,2]) q2[c1] *= np.sign(ts[c1,1,0] + ts[c1,0,1]) q3[c1] *= np.sign(ts[c1,0,2] + ts[c1,2,0]) q0[c2] *= np.sign(ts[c2,0,2] - ts[c2,2,0]) q1[c2] *= np.sign(ts[c2,1,0] + ts[c2,0,1]) q3[c2] *= np.sign(ts[c2,2,1] + ts[c2,1,2]) q0[c3] *= np.sign(ts[c3,1,0] - ts[c3,0,1]) q1[c3] *= np.sign(ts[c3,2,0] + ts[c3,0,2]) q2[c3] *= np.sign(ts[c3,2,1] + ts[c3,1,2]) qs = np.empty(ts.shape[:-2] + (4,)) qs[...,0] = q0 qs[...,1] = q1 qs[...,2] = q2 qs[...,3] = q3 return cls(qs) ================================================ FILE: pymo/__init__.py ================================================ ================================================ FILE: pymo/data.py ================================================ import numpy as np class Joint(): def __init__(self, name, parent=None, children=None): self.name = name self.parent = parent self.children = children class MocapData(): def __init__(self): self.skeleton = {} self.values = None self.channel_names = [] self.framerate = 0.0 self.root_name = '' self.take_name = '' def traverse(self, j=None): stack = [self.root_name] while stack: joint = stack.pop() yield joint for c in self.skeleton[joint]['children']: stack.append(c) def clone(self): import copy new_data = MocapData() new_data.skeleton = copy.deepcopy(self.skeleton) new_data.values = copy.deepcopy(self.values) new_data.channel_names = copy.deepcopy(self.channel_names) new_data.root_name = copy.deepcopy(self.root_name) new_data.framerate = copy.deepcopy(self.framerate) if hasattr(self,'take_name'): new_data.take_name = copy.deepcopy(self.take_name) return new_data def get_all_channels(self): '''Returns all of the channels parsed from the file as a 2D numpy array''' frames = [f[1] for f in self.values] return np.asarray([[channel[2] for channel in frame] for frame in frames]) def get_skeleton_tree(self): tree = [] root_key = [j for j in self.skeleton if self.skeleton[j]['parent']==None][0] root_joint = Joint(root_key) def get_empty_channels(self): #TODO pass def get_constant_channels(self): #TODO pass ================================================ FILE: pymo/features.py ================================================ ''' A set of mocap feature extraction functions Created by Omid Alemi | Nov 17 2017 ''' import numpy as np import pandas as pd import peakutils import matplotlib.pyplot as plt def get_foot_contact_idxs(signal, t=0.02, min_dist=120): up_idxs = peakutils.indexes(signal, thres=t/max(signal), min_dist=min_dist) down_idxs = peakutils.indexes(-signal, thres=t/min(signal), min_dist=min_dist) return [up_idxs, down_idxs] def create_foot_contact_signal(mocap_track, col_name, start=1, t=0.02, min_dist=120): signal = mocap_track.values[col_name].values idxs = get_foot_contact_idxs(signal, t, min_dist) step_signal = [] c = start for f in range(len(signal)): if f in idxs[1]: c = 0 elif f in idxs[0]: c = 1 step_signal.append(c) return step_signal def plot_foot_up_down(mocap_track, col_name, t=0.02, min_dist=120): signal = mocap_track.values[col_name].values idxs = get_foot_contact_idxs(signal, t, min_dist) plt.plot(mocap_track.values.index, signal) plt.plot(mocap_track.values.index[idxs[0]], signal[idxs[0]], 'ro') plt.plot(mocap_track.values.index[idxs[1]], signal[idxs[1]], 'go') ================================================ FILE: pymo/parsers.py ================================================ ''' BVH Parser Class By Omid Alemi Created: June 12, 2017 Based on: https://gist.github.com/johnfredcee/2007503 ''' import os import re import numpy as np from pymo.data import Joint, MocapData class BVHScanner(): ''' A wrapper class for re.Scanner ''' def __init__(self): def identifier(scanner, token): return 'IDENT', token def operator(scanner, token): return 'OPERATOR', token def digit(scanner, token): return 'DIGIT', token def open_brace(scanner, token): return 'OPEN_BRACE', token def close_brace(scanner, token): return 'CLOSE_BRACE', token self.scanner = re.Scanner([ (r'[a-zA-Z_]\w*', identifier), #(r'-*[0-9]+(\.[0-9]+)?', digit), # won't work for .34 #(r'[-+]?[0-9]*\.?[0-9]+', digit), # won't work for 4.56e-2 #(r'[-+]?[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?', digit), (r'-*[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?', digit), (r'}', close_brace), (r'}', close_brace), (r'{', open_brace), (r':', None), (r'\s+', None) ]) def scan(self, stuff): return self.scanner.scan(stuff) class BVHParser(): ''' A class to parse a BVH file. Extracts the skeleton and channel values ''' def __init__(self, filename=None): self.reset() def reset(self): self._skeleton = {} self.bone_context = [] self._motion_channels = [] self._motions = [] self.current_token = 0 self.framerate = 0.0 self.root_name = '' self.scanner = BVHScanner() self.data = MocapData() def parse(self, filename, start=0, stop=-1): self.reset() with open(filename, 'r') as bvh_file: raw_contents = bvh_file.read() tokens, remainder = self.scanner.scan(raw_contents) self._parse_hierarchy(tokens) self.current_token = self.current_token + 1 self._parse_motion(tokens, start, stop) self.data.skeleton = self._skeleton self.data.channel_names = self._motion_channels self.data.values = self._to_DataFrame() self.data.root_name = self.root_name self.data.framerate = self.framerate self.data.take_name = os.path.basename(os.path.splitext(filename)[0]) return self.data def _to_DataFrame(self): '''Returns all of the channels parsed from the file as a pandas DataFrame''' import pandas as pd time_index = pd.to_timedelta([f[0] for f in self._motions], unit='s') frames = [f[1] for f in self._motions] channels = np.asarray([[channel[2] for channel in frame] for frame in frames]) column_names = ['%s_%s'%(c[0], c[1]) for c in self._motion_channels] return pd.DataFrame(data=channels, index=time_index, columns=column_names) def _new_bone(self, parent, name): bone = {'parent': parent, 'channels': [], 'offsets': [], 'order': '','children': []} return bone def _push_bone_context(self,name): self.bone_context.append(name) def _get_bone_context(self): return self.bone_context[len(self.bone_context)-1] def _pop_bone_context(self): self.bone_context = self.bone_context[:-1] return self.bone_context[len(self.bone_context)-1] def _read_offset(self, bvh, token_index): if bvh[token_index] != ('IDENT', 'OFFSET'): return None, None token_index = token_index + 1 offsets = [0.0] * 3 for i in range(3): offsets[i] = float(bvh[token_index][1]) token_index = token_index + 1 return offsets, token_index def _read_channels(self, bvh, token_index): if bvh[token_index] != ('IDENT', 'CHANNELS'): return None, None token_index = token_index + 1 channel_count = int(bvh[token_index][1]) token_index = token_index + 1 channels = [""] * channel_count order = "" for i in range(channel_count): channels[i] = bvh[token_index][1] token_index = token_index + 1 if(channels[i] == "Xrotation" or channels[i]== "Yrotation" or channels[i]== "Zrotation"): order += channels[i][0] else : order = "" return channels, token_index, order def _parse_joint(self, bvh, token_index): end_site = False joint_id = bvh[token_index][1] token_index = token_index + 1 joint_name = bvh[token_index][1] token_index = token_index + 1 parent_name = self._get_bone_context() if (joint_id == "End"): joint_name = parent_name+ '_Nub' end_site = True joint = self._new_bone(parent_name, joint_name) if bvh[token_index][0] != 'OPEN_BRACE': print('Was expecting brance, got ', bvh[token_index]) return None token_index = token_index + 1 offsets, token_index = self._read_offset(bvh, token_index) joint['offsets'] = offsets if not end_site: channels, token_index, order = self._read_channels(bvh, token_index) joint['channels'] = channels joint['order'] = order for channel in channels: self._motion_channels.append((joint_name, channel)) self._skeleton[joint_name] = joint self._skeleton[parent_name]['children'].append(joint_name) while (bvh[token_index][0] == 'IDENT' and bvh[token_index][1] == 'JOINT') or (bvh[token_index][0] == 'IDENT' and bvh[token_index][1] == 'End'): self._push_bone_context(joint_name) token_index = self._parse_joint(bvh, token_index) self._pop_bone_context() if bvh[token_index][0] == 'CLOSE_BRACE': return token_index + 1 print('Unexpected token ', bvh[token_index]) def _parse_hierarchy(self, bvh): self.current_token = 0 if bvh[self.current_token] != ('IDENT', 'HIERARCHY'): return None self.current_token = self.current_token + 1 if bvh[self.current_token] != ('IDENT', 'ROOT'): return None self.current_token = self.current_token + 1 if bvh[self.current_token][0] != 'IDENT': return None root_name = bvh[self.current_token][1] root_bone = self._new_bone(None, root_name) self.current_token = self.current_token + 2 #skipping open brace offsets, self.current_token = self._read_offset(bvh, self.current_token) channels, self.current_token, order = self._read_channels(bvh, self.current_token) root_bone['offsets'] = offsets root_bone['channels'] = channels root_bone['order'] = order self._skeleton[root_name] = root_bone self._push_bone_context(root_name) for channel in channels: self._motion_channels.append((root_name, channel)) while bvh[self.current_token][1] == 'JOINT' or bvh[self.current_token][1] == 'End': self.current_token = self._parse_joint(bvh, self.current_token) self.root_name = root_name def _parse_motion(self, bvh, start, stop): if bvh[self.current_token][0] != 'IDENT': print('Unexpected text') return None if bvh[self.current_token][1] != 'MOTION': print('No motion section') return None self.current_token = self.current_token + 1 if bvh[self.current_token][1] != 'Frames': return None self.current_token = self.current_token + 1 frame_count = int(bvh[self.current_token][1]) if stop<0 or stop>frame_count: stop = frame_count assert(start>=0) assert(start=start: self._motions[idx] = (frame_time, channel_values) frame_time = frame_time + frame_rate idx+=1 ================================================ FILE: pymo/preprocessing.py ================================================ ''' Preprocessing Tranformers Based on sci-kit's API By Omid Alemi Created on June 12, 2017 ''' import copy import pandas as pd import numpy as np import transforms3d as t3d import scipy.ndimage.filters as filters from scipy.spatial.transform import Rotation as R from scipy import signal, interpolate from sklearn.base import BaseEstimator, TransformerMixin from sklearn.pipeline import Pipeline from pymo.rotation_tools import Rotation, euler2expmap, euler2expmap2, expmap2euler, euler_reorder, unroll, euler2vectors, vectors2euler from pymo.Quaternions import Quaternions from pymo.Pivots import Pivots class MocapParameterizer(BaseEstimator, TransformerMixin): def __init__(self, param_type = 'euler', ref_pose=None): ''' param_type = {'euler', 'quat', 'expmap', 'position', 'expmap2pos'} ''' self.param_type = param_type if (ref_pose is not None): self.ref_pose = self._to_quat(ref_pose)[0] else: self.ref_pose = None def fit(self, X, y=None): return self def transform(self, X, y=None): #print("MocapParameterizer: " + self.param_type) if self.param_type == 'euler': return X elif self.param_type == 'expmap': if self.ref_pose is None: return self._to_expmap(X) else: return self._to_expmap2(X) elif self.param_type == 'vectors': return self._euler_to_vectors(X) elif self.param_type == 'quat': return self._to_quat(X) elif self.param_type == 'position': return self._to_pos(X) elif self.param_type == 'expmap2pos': return self._expmap_to_pos(X) else: raise 'param types: euler, quat, expmap, position, expmap2pos' # return X def inverse_transform(self, X, copy=None): if self.param_type == 'euler': return X elif self.param_type == 'expmap': if self.ref_pose is None: return self._expmap_to_euler(X) else: return self._expmap_to_euler2(X) elif self.param_type == 'vectors': return self._vectors_to_euler(X) elif self.param_type == 'quat': return self._quat_to_euler(X) elif self.param_type == 'position': # raise 'positions 2 eulers is not supported' print('positions 2 eulers is not supported') return X else: raise 'param types: euler, quat, expmap, position' def _to_quat(self, X): '''Converts joints rotations in quaternions''' Q = [] for track in X: channels = [] titles = [] euler_df = track.values # Create a new DataFrame to store the exponential map rep quat_df = euler_df.copy() # List the columns that contain rotation channels rot_cols = [c for c in euler_df.columns if ('rotation' in c and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton if 'Nub' not in joint) for joint in joints: rot_order = track.skeleton[joint]['order'] # Get the rotation columns that belong to this joint rc = euler_df[[c for c in rot_cols if joint in c]] r1_col = '%s_%srotation'%(joint, rot_order[0]) r2_col = '%s_%srotation'%(joint, rot_order[1]) r3_col = '%s_%srotation'%(joint, rot_order[2]) # Make sure the columns are organized in xyz order if rc.shape[1] < 3: euler_values = np.zeros((euler_df.shape[0], 3)) rot_order = "XYZ" else: euler_values = np.pi/180.0*np.transpose(np.array([track.values[r1_col], track.values[r2_col], track.values[r3_col]])) quat_df.drop([r1_col, r2_col, r3_col], axis=1, inplace=True) quats = Quaternions.from_euler(np.asarray(euler_values), order=rot_order.lower(), world=False) # Create the corresponding columns in the new DataFrame quat_df['%s_qWrotation'%joint] = pd.Series(data=[e[0] for e in quats], index=quat_df.index) quat_df['%s_qXrotation'%joint] = pd.Series(data=[e[1] for e in quats], index=quat_df.index) quat_df['%s_qYrotation'%joint] = pd.Series(data=[e[2] for e in quats], index=quat_df.index) quat_df['%s_qZrotation'%joint] = pd.Series(data=[e[3] for e in quats], index=quat_df.index) new_track = track.clone() new_track.values = quat_df Q.append(new_track) return Q def _quat_to_euler(self, X): Q = [] for track in X: channels = [] titles = [] quat_df = track.values # Create a new DataFrame to store the exponential map rep #euler_df = pd.DataFrame(index=exp_df.index) euler_df = quat_df.copy() # List the columns that contain rotation channels quat_params = [c for c in quat_df.columns if ( any(p in c for p in ['qWrotation','qXrotation','qYrotation','qZrotation']) and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton if 'Nub' not in joint) for joint in joints: r = quat_df[[c for c in quat_params if joint in c]] # Get the columns that belong to this joint euler_df.drop(['%s_qWrotation'%joint, '%s_qXrotation'%joint, '%s_qYrotation'%joint, '%s_qZrotation'%joint], axis=1, inplace=True) quat = [[f[1]['%s_qWrotation'%joint], f[1]['%s_qXrotation'%joint], f[1]['%s_qYrotation'%joint], f[1]['%s_qZrotation'%joint]] for f in r.iterrows()] # Make sure the columsn are organized in xyz order quats=Quaternions(np.asarray(quat)) euler_rots = 180/np.pi*quats.euler() track.skeleton[joint]['order'] = 'ZYX' rot_order = track.skeleton[joint]['order'] #euler_rots = [Rotation(f, 'expmap').to_euler(True, rot_order) for f in expmap] # Convert the exp maps to eulers #euler_rots = [expmap2euler(f, rot_order, True) for f in expmap] # Convert the exp maps to eulers # Create the corresponding columns in the new DataFrame euler_df['%s_%srotation'%(joint, rot_order[2])] = pd.Series(data=[e[0] for e in euler_rots], index=euler_df.index) euler_df['%s_%srotation'%(joint, rot_order[1])] = pd.Series(data=[e[1] for e in euler_rots], index=euler_df.index) euler_df['%s_%srotation'%(joint, rot_order[0])] = pd.Series(data=[e[2] for e in euler_rots], index=euler_df.index) new_track = track.clone() new_track.values = euler_df Q.append(new_track) return Q def _to_pos(self, X): '''Converts joints rotations in Euler angles to joint positions''' Q = [] for track in X: channels = [] titles = [] euler_df = track.values # Create a new DataFrame to store the exponential map rep pos_df = pd.DataFrame(index=euler_df.index) # List the columns that contain rotation channels rot_cols = [c for c in euler_df.columns if ('rotation' in c)] # List the columns that contain position channels pos_cols = [c for c in euler_df.columns if ('position' in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton) tree_data = {} for joint in track.traverse(): parent = track.skeleton[joint]['parent'] rot_order = track.skeleton[joint]['order'] #print("rot_order:" + joint + " :" + rot_order) # Get the rotation columns that belong to this joint rc = euler_df[[c for c in rot_cols if joint in c]] # Get the position columns that belong to this joint pc = euler_df[[c for c in pos_cols if joint in c]] # Make sure the columns are organized in xyz order if rc.shape[1] < 3: euler_values = np.zeros((euler_df.shape[0], 3)) rot_order = "XYZ" else: euler_values = np.pi/180.0*np.transpose(np.array([track.values['%s_%srotation'%(joint, rot_order[0])], track.values['%s_%srotation'%(joint, rot_order[1])], track.values['%s_%srotation'%(joint, rot_order[2])]])) if pc.shape[1] < 3: pos_values = np.asarray([[0,0,0] for f in pc.iterrows()]) else: pos_values =np.asarray([[f[1]['%s_Xposition'%joint], f[1]['%s_Yposition'%joint], f[1]['%s_Zposition'%joint]] for f in pc.iterrows()]) quats = Quaternions.from_euler(np.asarray(euler_values), order=rot_order.lower(), world=False) tree_data[joint]=[ [], # to store the rotation matrix [] # to store the calculated position ] if track.root_name == joint: tree_data[joint][0] = quats#rotmats # tree_data[joint][1] = np.add(pos_values, track.skeleton[joint]['offsets']) tree_data[joint][1] = pos_values else: # for every frame i, multiply this joint's rotmat to the rotmat of its parent tree_data[joint][0] = tree_data[parent][0]*quats# np.matmul(rotmats, tree_data[parent][0]) # add the position channel to the offset and store it in k, for every frame i k = pos_values + np.asarray(track.skeleton[joint]['offsets']) # multiply k to the rotmat of the parent for every frame i q = tree_data[parent][0]*k #np.matmul(k.reshape(k.shape[0],1,3), tree_data[parent][0]) # add q to the position of the parent, for every frame i tree_data[joint][1] = tree_data[parent][1] + q #q.reshape(k.shape[0],3) + tree_data[parent][1] # Create the corresponding columns in the new DataFrame df = pd.DataFrame(data=tree_data[joint][1], index=pos_df.index, columns=['%s_Xposition'%joint, '%s_Yposition'%joint, '%s_Zposition'%joint]) pos_df = pd.concat((pos_df, df), axis=1) new_track = track.clone() new_track.values = pos_df Q.append(new_track) return Q def _expmap2rot(self, expmap): theta = np.linalg.norm(expmap, axis=1, keepdims=True) nz = np.nonzero(theta)[0] expmap[nz,:] = expmap[nz,:]/theta[nz] nrows=expmap.shape[0] x = expmap[:,0] y = expmap[:,1] z = expmap[:,2] s = np.sin(theta*0.5).reshape(nrows) c = np.cos(theta*0.5).reshape(nrows) rotmats = np.zeros((nrows, 3, 3)) rotmats[:,0,0] = 2*(x*x-1)*s*s+1 rotmats[:,0,1] = 2*x*y*s*s-2*z*c*s rotmats[:,0,2] = 2*x*z*s*s+2*y*c*s rotmats[:,1,0] = 2*x*y*s*s+2*z*c*s rotmats[:,1,1] = 2*(y*y-1)*s*s+1 rotmats[:,1,2] = 2*y*z*s*s-2*x*c*s rotmats[:,2,0] = 2*x*z*s*s-2*y*c*s rotmats[:,2,1] = 2*y*z*s*s+2*x*c*s rotmats[:,2,2] = 2*(z*z-1)*s*s+1 return rotmats def _expmap_to_pos(self, X): '''Converts joints rotations in expmap notation to joint positions''' Q = [] for track in X: channels = [] titles = [] exp_df = track.values # Create a new DataFrame to store the exponential map rep pos_df = pd.DataFrame(index=exp_df.index) # List the columns that contain rotation channels exp_params = [c for c in exp_df.columns if ( any(p in c for p in ['alpha', 'beta','gamma']) and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton) tree_data = {} for joint in track.traverse(): parent = track.skeleton[joint]['parent'] if 'Nub' not in joint: r = exp_df[[c for c in exp_params if joint in c]] # Get the columns that belong to this joint expmap = r.values #expmap = [[f[1]['%s_alpha'%joint], f[1]['%s_beta'%joint], f[1]['%s_gamma'%joint]] for f in r.iterrows()] else: expmap = np.zeros((exp_df.shape[0], 3)) # Convert the eulers to rotation matrices #rotmats = np.asarray([Rotation(f, 'expmap').rotmat for f in expmap]) #angs = np.linalg.norm(expmap,axis=1, keepdims=True) rotmats = self._expmap2rot(expmap) tree_data[joint]=[ [], # to store the rotation matrix [] # to store the calculated position ] pos_values = np.zeros((exp_df.shape[0], 3)) if track.root_name == joint: tree_data[joint][0] = rotmats # tree_data[joint][1] = np.add(pos_values, track.skeleton[joint]['offsets']) tree_data[joint][1] = pos_values else: # for every frame i, multiply this joint's rotmat to the rotmat of its parent tree_data[joint][0] = np.matmul(rotmats, tree_data[parent][0]) # add the position channel to the offset and store it in k, for every frame i k = pos_values + track.skeleton[joint]['offsets'] # multiply k to the rotmat of the parent for every frame i q = np.matmul(k.reshape(k.shape[0],1,3), tree_data[parent][0]) # add q to the position of the parent, for every frame i tree_data[joint][1] = q.reshape(k.shape[0],3) + tree_data[parent][1] # Create the corresponding columns in the new DataFrame df = pd.DataFrame(data=tree_data[joint][1], index=pos_df.index, columns=['%s_Xposition'%joint, '%s_Yposition'%joint, '%s_Zposition'%joint]) pos_df = pd.concat((pos_df, df), axis=1) new_track = track.clone() new_track.values = pos_df Q.append(new_track) return Q def _to_expmap(self, X): '''Converts Euler angles to Exponential Maps''' Q = [] for track in X: channels = [] titles = [] euler_df = track.values # Create a new DataFrame to store the exponential map rep exp_df = euler_df.copy()# pd.DataFrame(index=euler_df.index) # List the columns that contain rotation channels rots = [c for c in euler_df.columns if ('rotation' in c and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton if 'Nub' not in joint) for joint in joints: #print(joint) r = euler_df[[c for c in rots if joint in c]] # Get the columns that belong to this joint rot_order = track.skeleton[joint]['order'] r1_col = '%s_%srotation'%(joint, rot_order[0]) r2_col = '%s_%srotation'%(joint, rot_order[1]) r3_col = '%s_%srotation'%(joint, rot_order[2]) exp_df.drop([r1_col, r2_col, r3_col], axis=1, inplace=True) euler = np.transpose(np.array([r[r1_col], r[r2_col], r[r3_col]])) #exps = [Rotation(f, 'euler', from_deg=True, order=rot_order).to_expmap() for f in euler] # Convert the eulers to exp maps exps = unroll(np.array([euler2expmap(f, rot_order, True) for f in euler])) # Convert the exp maps to eulers #exps = euler2expmap2(euler, rot_order, True) # Convert the eulers to exp maps # Create the corresponding columns in the new DataFrame exp_df.insert(loc=0, column='%s_gamma'%joint, value=pd.Series(data=[e[2] for e in exps], index=exp_df.index)) exp_df.insert(loc=0, column='%s_beta'%joint, value=pd.Series(data=[e[1] for e in exps], index=exp_df.index)) exp_df.insert(loc=0, column='%s_alpha'%joint, value=pd.Series(data=[e[0] for e in exps], index=exp_df.index)) #print(exp_df.columns) new_track = track.clone() new_track.values = exp_df Q.append(new_track) return Q def _expmap_to_euler(self, X): Q = [] for track in X: channels = [] titles = [] exp_df = track.values # Create a new DataFrame to store the exponential map rep euler_df = exp_df.copy() # List the columns that contain rotation channels exp_params = [c for c in exp_df.columns if ( any(p in c for p in ['alpha', 'beta','gamma']) and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton if 'Nub' not in joint) for joint in joints: r = exp_df[[c for c in exp_params if joint in c]] # Get the columns that belong to this joint jt_alpha = '%s_alpha'%joint jt_beta = '%s_beta'%joint jt_gamma = '%s_gamma'%joint euler_df.drop([jt_alpha, jt_beta, jt_gamma], axis=1, inplace=True) expmap = np.transpose(np.array([track.values[jt_alpha], track.values[jt_beta], track.values[jt_gamma]])) rot_order = track.skeleton[joint]['order'] euler_rots = np.array(R.from_rotvec(expmap).as_euler(rot_order, degrees=True)) #euler_rots = [expmap2euler(f, rot_order, True) for f in expmap] # Convert the exp maps to eulers # Create the corresponding columns in the new DataFrame euler_df['%s_%srotation'%(joint, rot_order[0])] = pd.Series(data=[e[0] for e in euler_rots], index=euler_df.index) euler_df['%s_%srotation'%(joint, rot_order[1])] = pd.Series(data=[e[1] for e in euler_rots], index=euler_df.index) euler_df['%s_%srotation'%(joint, rot_order[2])] = pd.Series(data=[e[2] for e in euler_rots], index=euler_df.index) new_track = track.clone() new_track.values = euler_df Q.append(new_track) return Q def _to_expmap2(self, X): '''Converts Euler angles to Exponential Maps''' Q = [] for track in X: channels = [] titles = [] euler_df = track.values # Create a new DataFrame to store the exponential map rep exp_df = euler_df.copy()# pd.DataFrame(index=euler_df.index) # Copy the root positions into the new DataFrame #rxp = '%s_Xposition'%track.root_name #ryp = '%s_Yposition'%track.root_name #rzp = '%s_Zposition'%track.root_name #exp_df[rxp] = pd.Series(data=euler_df[rxp], index=exp_df.index) #exp_df[ryp] = pd.Series(data=euler_df[ryp], index=exp_df.index) #exp_df[rzp] = pd.Series(data=euler_df[rzp], index=exp_df.index) # List the columns that contain rotation channels rots = [c for c in euler_df.columns if ('rotation' in c and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton if 'Nub' not in joint) for joint in joints: r = euler_df[[c for c in rots if joint in c]] # Get the columns that belong to this joint rot_order = track.skeleton[joint]['order'] # Get the rotation columns that belong to this joint rc = euler_df[[c for c in rots if joint in c]] r1_col = '%s_%srotation'%(joint, rot_order[0]) r2_col = '%s_%srotation'%(joint, rot_order[1]) r3_col = '%s_%srotation'%(joint, rot_order[2]) # Make sure the columns are organized in xyz order #print("joint:" + str(joint) + " rot_order:" + str(rot_order)) if rc.shape[1] < 3: euler_values = np.zeros((euler_df.shape[0], 3)) rot_order = "XYZ" else: euler_values = np.pi/180.0*np.transpose(np.array([track.values[r1_col], track.values[r2_col], track.values[r3_col]])) quats = Quaternions.from_euler(np.asarray(euler_values), order=rot_order.lower(), world=False) #exps = [Rotation(f, 'euler', from_deg=True, order=rot_order).to_expmap() for f in euler] # Convert the eulers to exp maps #exps = unroll(np.array([euler2expmap(f, rot_order, True) for f in euler])) # Convert the exp maps to eulers #exps = euler2expmap2(euler, rot_order, True) # Convert the eulers to exp maps # Create the corresponding columns in the new DataFrame if (self.ref_pose is not None): q1_col = '%s_qWrotation'%(joint) q2_col = '%s_qXrotation'%(joint) q3_col = '%s_qYrotation'%(joint) q4_col = '%s_qZrotation'%(joint) ref_q = Quaternions(np.asarray([[f[1][q1_col], f[1][q2_col], f[1][q3_col], f[1][q4_col]] for f in self.ref_pose.values.iterrows()])) #print("ref_q:" + str(ref_q.shape)) ref_q = ref_q[0,:] quats=(-ref_q)*quats angles, axis = quats.angle_axis() aa = np.where(angles>np.pi) angles[aa] = angles[aa]-2*np.pi #exps = unroll(angles[:,None]*axis) exps = angles[:,None]*axis #print(f"{joint}: {str(exps[0,:])}") #exps = np.array([quat2expmap(f) for f in quats]) exp_df.drop([r1_col, r2_col, r3_col], axis=1, inplace=True) exp_df.insert(loc=0, column='%s_gamma'%joint, value=pd.Series(data=[e[2] for e in exps], index=exp_df.index)) exp_df.insert(loc=0, column='%s_beta'%joint, value=pd.Series(data=[e[1] for e in exps], index=exp_df.index)) exp_df.insert(loc=0, column='%s_alpha'%joint, value=pd.Series(data=[e[0] for e in exps], index=exp_df.index)) #print(exp_df.columns) new_track = track.clone() new_track.values = exp_df Q.append(new_track) return Q def _expmap_to_euler2(self, X): Q = [] for track in X: channels = [] titles = [] exp_df = track.values # Create a new DataFrame to store the exponential map rep #euler_df = pd.DataFrame(index=exp_df.index) euler_df = exp_df.copy() # Copy the root positions into the new DataFrame #rxp = '%s_Xposition'%track.root_name #ryp = '%s_Yposition'%track.root_name #rzp = '%s_Zposition'%track.root_name #euler_df[rxp] = pd.Series(data=exp_df[rxp], index=euler_df.index) #euler_df[ryp] = pd.Series(data=exp_df[ryp], index=euler_df.index) #euler_df[rzp] = pd.Series(data=exp_df[rzp], index=euler_df.index) # List the columns that contain rotation channels exp_params = [c for c in exp_df.columns if ( any(p in c for p in ['alpha', 'beta','gamma']) and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton if 'Nub' not in joint) for joint in joints: r = exp_df[[c for c in exp_params if joint in c]] # Get the columns that belong to this joint euler_df.drop(['%s_alpha'%joint, '%s_beta'%joint, '%s_gamma'%joint], axis=1, inplace=True) expmap = [[f[1]['%s_alpha'%joint], f[1]['%s_beta'%joint], f[1]['%s_gamma'%joint]] for f in r.iterrows()] # Make sure the columsn are organized in xyz order angs = np.linalg.norm(expmap, axis=1) quats=Quaternions.from_angle_axis(angs, expmap/(np.tile(angs[:, None]+1e-10, (1,3)))) if (self.ref_pose is not None): q1_col = '%s_qWrotation'%(joint) q2_col = '%s_qXrotation'%(joint) q3_col = '%s_qYrotation'%(joint) q4_col = '%s_qZrotation'%(joint) ref_q = Quaternions(np.asarray([[f[1][q1_col], f[1][q2_col], f[1][q3_col], f[1][q4_col]] for f in self.ref_pose.values.iterrows()])) #print("ref_q:" + str(ref_q.shape)) ref_q = ref_q[0,:] quats=ref_q*quats euler_rots = 180/np.pi*quats.euler() track.skeleton[joint]['order'] = 'ZYX' rot_order = track.skeleton[joint]['order'] #euler_rots = [Rotation(f, 'expmap').to_euler(True, rot_order) for f in expmap] # Convert the exp maps to eulers #euler_rots = [expmap2euler(f, rot_order, True) for f in expmap] # Convert the exp maps to eulers # Create the corresponding columns in the new DataFrame euler_df['%s_%srotation'%(joint, rot_order[2])] = pd.Series(data=[e[0] for e in euler_rots], index=euler_df.index) euler_df['%s_%srotation'%(joint, rot_order[1])] = pd.Series(data=[e[1] for e in euler_rots], index=euler_df.index) euler_df['%s_%srotation'%(joint, rot_order[0])] = pd.Series(data=[e[2] for e in euler_rots], index=euler_df.index) new_track = track.clone() new_track.values = euler_df Q.append(new_track) return Q def _euler_to_vectors(self, X): '''Converts Euler angles to Up and Fwd vectors''' Q = [] for track in X: channels = [] titles = [] euler_df = track.values # Create a new DataFrame to store the exponential map rep vec_df = euler_df.copy()# pd.DataFrame(index=euler_df.index) # List the columns that contain rotation channels rots = [c for c in euler_df.columns if ('rotation' in c and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton if 'Nub' not in joint) for joint in joints: #print(joint) r = euler_df[[c for c in rots if joint in c]] # Get the columns that belong to this joint rot_order = track.skeleton[joint]['order'] r1_col = '%s_%srotation'%(joint, rot_order[0]) r2_col = '%s_%srotation'%(joint, rot_order[1]) r3_col = '%s_%srotation'%(joint, rot_order[2]) vec_df.drop([r1_col, r2_col, r3_col], axis=1, inplace=True) euler = [[f[1][r1_col], f[1][r2_col], f[1][r3_col]] for f in r.iterrows()] vectors = np.array([euler2vectors(f, rot_order, True) for f in euler]) vec_df.insert(loc=0, column='%s_xUp'%joint, value=pd.Series(data=[e[0] for e in vectors], index=vec_df.index)) vec_df.insert(loc=0, column='%s_yUp'%joint, value=pd.Series(data=[e[1] for e in vectors], index=vec_df.index)) vec_df.insert(loc=0, column='%s_zUp'%joint, value=pd.Series(data=[e[2] for e in vectors], index=vec_df.index)) vec_df.insert(loc=0, column='%s_xFwd'%joint, value=pd.Series(data=[e[3] for e in vectors], index=vec_df.index)) vec_df.insert(loc=0, column='%s_yFwd'%joint, value=pd.Series(data=[e[4] for e in vectors], index=vec_df.index)) vec_df.insert(loc=0, column='%s_zFwd'%joint, value=pd.Series(data=[e[5] for e in vectors], index=vec_df.index)) #print(exp_df.columns) new_track = track.clone() new_track.values = vec_df Q.append(new_track) return Q def _vectors_to_euler(self, X): '''Converts Up and Fwd vectors to Euler angles''' Q = [] for track in X: channels = [] titles = [] vec_df = track.values # Create a new DataFrame to store the exponential map rep #euler_df = pd.DataFrame(index=exp_df.index) euler_df = vec_df.copy() # List the columns that contain rotation channels vec_params = [c for c in vec_df.columns if ( any(p in c for p in ['xUp', 'yUp','zUp','xFwd', 'yFwd','zFwd']) and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton if 'Nub' not in joint) for joint in joints: r = vec_df[[c for c in vec_params if joint in c]] # Get the columns that belong to this joint euler_df.drop(['%s_xUp'%joint, '%s_yUp'%joint, '%s_zUp'%joint, '%s_xFwd'%joint, '%s_yFwd'%joint, '%s_zFwd'%joint], axis=1, inplace=True) vectors = [[f[1]['%s_xUp'%joint], f[1]['%s_yUp'%joint], f[1]['%s_zUp'%joint], f[1]['%s_xFwd'%joint], f[1]['%s_yFwd'%joint], f[1]['%s_zFwd'%joint]] for f in r.iterrows()] # Make sure the columsn are organized in xyz order rot_order = track.skeleton[joint]['order'] euler_rots = [vectors2euler(f, rot_order, True) for f in vectors] # Create the corresponding columns in the new DataFrame euler_df['%s_%srotation'%(joint, rot_order[0])] = pd.Series(data=[e[0] for e in euler_rots], index=euler_df.index) euler_df['%s_%srotation'%(joint, rot_order[1])] = pd.Series(data=[e[1] for e in euler_rots], index=euler_df.index) euler_df['%s_%srotation'%(joint, rot_order[2])] = pd.Series(data=[e[2] for e in euler_rots], index=euler_df.index) new_track = track.clone() new_track.values = euler_df Q.append(new_track) return Q class Mirror(BaseEstimator, TransformerMixin): def __init__(self, axis="X", append=True): """ Mirrors the data """ self.axis = axis self.append = append def fit(self, X, y=None): return self def transform(self, X, y=None): #print("Mirror: " + self.axis) Q = [] if self.append: for track in X: Q.append(track) for track in X: channels = [] titles = [] if self.axis == "X": signs = np.array([1,-1,-1]) if self.axis == "Y": signs = np.array([-1,1,-1]) if self.axis == "Z": signs = np.array([-1,-1,1]) euler_df = track.values # Create a new DataFrame to store the exponential map rep new_df = pd.DataFrame(index=euler_df.index) # Copy the root positions into the new DataFrame rxp = '%s_Xposition'%track.root_name ryp = '%s_Yposition'%track.root_name rzp = '%s_Zposition'%track.root_name new_df[rxp] = pd.Series(data=-signs[0]*euler_df[rxp], index=new_df.index) new_df[ryp] = pd.Series(data=-signs[1]*euler_df[ryp], index=new_df.index) new_df[rzp] = pd.Series(data=-signs[2]*euler_df[rzp], index=new_df.index) # List the columns that contain rotation channels rots = [c for c in euler_df.columns if ('rotation' in c and 'Nub' not in c)] #lft_rots = [c for c in euler_df.columns if ('Left' in c and 'rotation' in c and 'Nub' not in c)] #rgt_rots = [c for c in euler_df.columns if ('Right' in c and 'rotation' in c and 'Nub' not in c)] lft_joints = (joint for joint in track.skeleton if 'Left' in joint and 'Nub' not in joint) rgt_joints = (joint for joint in track.skeleton if 'Right' in joint and 'Nub' not in joint) new_track = track.clone() for lft_joint in lft_joints: #lr = euler_df[[c for c in rots if lft_joint + "_" in c]] #rot_order = track.skeleton[lft_joint]['order'] #lft_eulers = [[f[1]['%s_Xrotation'%lft_joint], f[1]['%s_Yrotation'%lft_joint], f[1]['%s_Zrotation'%lft_joint]] for f in lr.iterrows()] rgt_joint = lft_joint.replace('Left', 'Right') #rr = euler_df[[c for c in rots if rgt_joint + "_" in c]] #rot_order = track.skeleton[rgt_joint]['order'] # rgt_eulers = [[f[1]['%s_Xrotation'%rgt_joint], f[1]['%s_Yrotation'%rgt_joint], f[1]['%s_Zrotation'%rgt_joint]] for f in rr.iterrows()] # Create the corresponding columns in the new DataFrame new_df['%s_Xrotation'%lft_joint] = pd.Series(data=signs[0]*track.values['%s_Xrotation'%rgt_joint], index=new_df.index) new_df['%s_Yrotation'%lft_joint] = pd.Series(data=signs[1]*track.values['%s_Yrotation'%rgt_joint], index=new_df.index) new_df['%s_Zrotation'%lft_joint] = pd.Series(data=signs[2]*track.values['%s_Zrotation'%rgt_joint], index=new_df.index) new_df['%s_Xrotation'%rgt_joint] = pd.Series(data=signs[0]*track.values['%s_Xrotation'%lft_joint], index=new_df.index) new_df['%s_Yrotation'%rgt_joint] = pd.Series(data=signs[1]*track.values['%s_Yrotation'%lft_joint], index=new_df.index) new_df['%s_Zrotation'%rgt_joint] = pd.Series(data=signs[2]*track.values['%s_Zrotation'%lft_joint], index=new_df.index) # List the joints that are not left or right, i.e. are on the trunk joints = (joint for joint in track.skeleton if 'Nub' not in joint and 'Left' not in joint and 'Right' not in joint) for joint in joints: #r = euler_df[[c for c in rots if joint in c]] # Get the columns that belong to this joint #rot_order = track.skeleton[joint]['order'] #eulers = [[f[1]['%s_Xrotation'%joint], f[1]['%s_Yrotation'%joint], f[1]['%s_Zrotation'%joint]] for f in r.iterrows()] # Create the corresponding columns in the new DataFrame new_df['%s_Xrotation'%joint] = pd.Series(data=signs[0]*track.values['%s_Xrotation'%joint], index=new_df.index) new_df['%s_Yrotation'%joint] = pd.Series(data=signs[1]*track.values['%s_Yrotation'%joint], index=new_df.index) new_df['%s_Zrotation'%joint] = pd.Series(data=signs[2]*track.values['%s_Zrotation'%joint], index=new_df.index) new_track.values = new_df new_track.take_name = track.take_name + "_mirrored" Q.append(new_track) return Q def inverse_transform(self, X, copy=None, start_pos=None): return X class EulerReorder(BaseEstimator, TransformerMixin): def __init__(self, new_order): """ Add a """ self.new_order = new_order def fit(self, X, y=None): self.orig_skeleton = copy.deepcopy(X[0].skeleton) return self def transform(self, X, y=None): #print("EulerReorder") Q = [] for track in X: channels = [] titles = [] euler_df = track.values # Create a new DataFrame to store the exponential map rep #new_df = pd.DataFrame(index=euler_df.index) new_df = euler_df.copy() # Copy the root positions into the new DataFrame rxp = '%s_Xposition'%track.root_name ryp = '%s_Yposition'%track.root_name rzp = '%s_Zposition'%track.root_name new_df[rxp] = pd.Series(data=euler_df[rxp], index=new_df.index) new_df[ryp] = pd.Series(data=euler_df[ryp], index=new_df.index) new_df[rzp] = pd.Series(data=euler_df[rzp], index=new_df.index) # List the columns that contain rotation channels rots = [c for c in euler_df.columns if ('rotation' in c and 'Nub' not in c)] # List the joints that are not end sites, i.e., have channels joints = (joint for joint in track.skeleton if 'Nub' not in joint) new_track = track.clone() for joint in joints: r = euler_df[[c for c in rots if joint in c]] # Get the columns that belong to this joint rot_order = track.skeleton[joint]['order'] r1_col = '%s_%srotation'%(joint, rot_order[0]) r2_col = '%s_%srotation'%(joint, rot_order[1]) r3_col = '%s_%srotation'%(joint, rot_order[2]) #euler = [[f[1][r1_col], f[1][r2_col], f[1][r3_col]] for f in r.iterrows()] euler = np.transpose(np.array([r[r1_col], r[r2_col], r[r3_col]])) #euler = [[f[1]['%s_Xrotation'%(joint)], f[1]['%s_Yrotation'%(joint)], f[1]['%s_Zrotation'%(joint)]] for f in r.iterrows()] new_euler = [euler_reorder(f, rot_order, self.new_order, True) for f in euler] #new_euler = euler_reorder2(np.array(euler), rot_order, self.new_order, True) # Create the corresponding columns in the new DataFrame new_df['%s_%srotation'%(joint, self.new_order[0])] = pd.Series(data=[e[0] for e in new_euler], index=new_df.index) new_df['%s_%srotation'%(joint, self.new_order[1])] = pd.Series(data=[e[1] for e in new_euler], index=new_df.index) new_df['%s_%srotation'%(joint, self.new_order[2])] = pd.Series(data=[e[2] for e in new_euler], index=new_df.index) new_track.skeleton[joint]['order'] = self.new_order new_track.values = new_df Q.append(new_track) return Q def inverse_transform(self, X, copy=None, start_pos=None): return X class JointSelector(BaseEstimator, TransformerMixin): ''' Allows for filtering the mocap data to include only the selected joints ''' def __init__(self, joints, include_root=False): self.joints = joints self.include_root = include_root def fit(self, X, y=None): selected_joints = [] selected_channels = [] if self.include_root: selected_joints.append(X[0].root_name) selected_joints.extend(self.joints) for joint_name in selected_joints: if joint_name.endswith("_Nub"): selected_channels.extend([o for o in X[0].values.columns if (joint_name + "_") in o]) else: selected_channels.extend([o for o in X[0].values.columns if (joint_name + "_") in o and 'Nub' not in o]) self.selected_joints = selected_joints self.selected_channels = selected_channels self.not_selected = X[0].values.columns.difference(selected_channels) self.not_selected_values = {c:X[0].values[c].values[0] for c in self.not_selected} self.orig_skeleton = X[0].skeleton return self def transform(self, X, y=None): #print("JointSelector") Q = [] for track in X: t2 = track.clone() for key in track.skeleton.keys(): if key not in self.selected_joints: t2.skeleton.pop(key) t2.values = track.values[self.selected_channels] for key in t2.skeleton.keys(): to_remove = list(set(t2.skeleton[key]['children']) - set(self.selected_joints)) [t2.skeleton[key]['children'].remove(c) for c in to_remove] Q.append(t2) return Q def inverse_transform(self, X, copy=None): Q = [] for track in X: t2 = track.clone() skeleton = self.orig_skeleton for key in track.skeleton.keys(): skeleton[key]['order']=track.skeleton[key]['order'] t2.skeleton = skeleton for d in self.not_selected: t2.values[d] = self.not_selected_values[d] Q.append(t2) return Q class Numpyfier(BaseEstimator, TransformerMixin): ''' Just converts the values in a MocapData object into a numpy array Useful for the final stage of a pipeline before training ''' def __init__(self): pass def fit(self, X, y=None): self.org_mocap_ = X[0].clone() self.org_mocap_.values.drop(self.org_mocap_.values.index, inplace=True) return self def transform(self, X, y=None): #print("Numpyfier") Q = [] for track in X: Q.append(track.values.values) #print("Numpyfier:" + str(track.values.columns)) return np.array(Q) def inverse_transform(self, X, copy=None): Q = [] for track in X: new_mocap = self.org_mocap_.clone() time_index = pd.to_timedelta([f for f in range(track.shape[0])], unit='s')*self.org_mocap_.framerate new_df = pd.DataFrame(data=track, index=time_index, columns=self.org_mocap_.values.columns) new_mocap.values = new_df Q.append(new_mocap) return Q class Slicer(BaseEstimator, TransformerMixin): ''' Slice the data into intervals of equal size ''' def __init__(self, window_size, overlap=0.5): self.window_size = window_size self.overlap = overlap pass def fit(self, X, y=None): self.org_mocap_ = X[0].clone() self.org_mocap_.values.drop(self.org_mocap_.values.index, inplace=True) return self def transform(self, X, y=None): #print("Slicer") Q = [] for track in X: vals = track.values.values nframes = vals.shape[0] overlap_frames = (int)(self.overlap*self.window_size) n_sequences = (nframes-overlap_frames)//(self.window_size-overlap_frames) if n_sequences>0: y = np.zeros((n_sequences, self.window_size, vals.shape[1])) # extract sequences from the input data for i in range(0,n_sequences): frameIdx = (self.window_size-overlap_frames) * i Q.append(vals[frameIdx:frameIdx+self.window_size,:]) return np.array(Q) def inverse_transform(self, X, copy=None): Q = [] for track in X: new_mocap = self.org_mocap_.clone() time_index = pd.to_timedelta([f for f in range(track.shape[0])], unit='s') new_df = pd.DataFrame(data=track, index=time_index, columns=self.org_mocap_.values.columns) new_mocap.values = new_df Q.append(new_mocap) return Q class RootTransformer(BaseEstimator, TransformerMixin): def __init__(self, method, hips_axis_order="XYZ", position_smoothing=0, rotation_smoothing=0, separate_root=True): """ Accepted methods: abdolute_translation_deltas pos_rot_deltas """ self.method = method self.position_smoothing=position_smoothing self.rotation_smoothing=rotation_smoothing self.separate_root = separate_root self.hips_axis_order = hips_axis_order # relative rotation from the hips awis the the x-side, y-up, z-forward convention rot_mat = np.zeros((3,3)) for i in range(3): ax_i = ord(hips_axis_order[i])-ord("X") rot_mat[i,ax_i]=1 self.root_rotation_offset = Quaternions.from_transforms(rot_mat[np.newaxis, :, :]) self.hips_side_axis = -rot_mat[0,:] def fit(self, X, y=None): return self def transform(self, X, y=None): #print("RootTransformer") Q = [] for track in X: if self.method == 'abdolute_translation_deltas': new_df = track.values.copy() xpcol = '%s_Xposition'%track.root_name ypcol = '%s_Yposition'%track.root_name zpcol = '%s_Zposition'%track.root_name dxpcol = '%s_dXposition'%track.root_name dzpcol = '%s_dZposition'%track.root_name x=track.values[xpcol].copy() z=track.values[zpcol].copy() if self.position_smoothing>0: x_sm = filters.gaussian_filter1d(x, self.position_smoothing, axis=0, mode='nearest') z_sm = filters.gaussian_filter1d(z, self.position_smoothing, axis=0, mode='nearest') dx = pd.Series(data=x_sm, index=new_df.index).diff() dz = pd.Series(data=z_sm, index=new_df.index).diff() new_df[xpcol] = x-x_sm new_df[zpcol] = z-z_sm else: dx = x.diff() dz = z.diff() new_df.drop([xpcol, zpcol], axis=1, inplace=True) dx[0] = dx[1] dz[0] = dz[1] new_df[dxpcol] = dx new_df[dzpcol] = dz new_track = track.clone() new_track.values = new_df # end of abdolute_translation_deltas elif self.method == 'pos_rot_deltas': new_track = track.clone() # Absolute columns xp_col = '%s_Xposition'%track.root_name yp_col = '%s_Yposition'%track.root_name zp_col = '%s_Zposition'%track.root_name #rot_order = track.skeleton[track.root_name]['order'] #%(joint, rot_order[0]) rot_order = track.skeleton[track.root_name]['order'] r1_col = '%s_%srotation'%(track.root_name, rot_order[0]) r2_col = '%s_%srotation'%(track.root_name, rot_order[1]) r3_col = '%s_%srotation'%(track.root_name, rot_order[2]) # Delta columns # dxp_col = '%s_dXposition'%track.root_name # dzp_col = '%s_dZposition'%track.root_name # dxr_col = '%s_dXrotation'%track.root_name # dyr_col = '%s_dYrotation'%track.root_name # dzr_col = '%s_dZrotation'%track.root_name dxp_col = 'reference_dXposition' dzp_col = 'reference_dZposition' dxr_col = 'reference_dXrotation' dyr_col = 'reference_dYrotation' dzr_col = 'reference_dZrotation' positions = np.transpose(np.array([track.values[xp_col], track.values[yp_col], track.values[zp_col]])) rotations = np.pi/180.0*np.transpose(np.array([track.values[r1_col], track.values[r2_col], track.values[r3_col]])) """ Get Trajectory and smooth it""" trajectory_filterwidth = self.position_smoothing reference = positions.copy()*np.array([1,0,1]) if trajectory_filterwidth>0: reference = filters.gaussian_filter1d(reference, trajectory_filterwidth, axis=0, mode='nearest') """ Get Root Velocity """ velocity = np.diff(reference, axis=0) velocity = np.vstack((velocity[0,:], velocity)) """ Remove Root Translation """ positions = positions-reference """ Get Forward Direction along the x-z plane, assuming character is facig z-forward """ #forward = [Rotation(f, 'euler', from_deg=True, order=rot_order).rotmat[:,2] for f in rotations] # get the z-axis of the rotation matrix, assuming character is facig z-forward #print("order:" + rot_order.lower()) quats = Quaternions.from_euler(rotations, order=rot_order.lower(), world=False) #forward = quats*np.array([[0,0,1]]) #forward[:,1] = 0 side_dirs = quats*self.hips_side_axis forward = np.cross(np.array([[0,1,0]]), side_dirs) """ Smooth Forward Direction """ direction_filterwidth = self.rotation_smoothing if direction_filterwidth>0: forward = filters.gaussian_filter1d(forward, direction_filterwidth, axis=0, mode='nearest') forward = forward / np.sqrt((forward**2).sum(axis=-1))[...,np.newaxis] """ Remove Y Rotation """ target = np.array([[0,0,1]]).repeat(len(forward), axis=0) rotation = Quaternions.between(target, forward)[:,np.newaxis] positions = (-rotation[:,0]) * positions #new_rotations = (-rotation[:,0]) * quats new_rotations = (-self.root_rotation_offset) * (-rotation[:,0]) * quats """ Get Root Rotation """ #print(rotation[:,0]) velocity = (-rotation[:,0]) * velocity rvelocity = Pivots.from_quaternions(rotation[1:] * -rotation[:-1]).ps rvelocity = np.vstack((rvelocity[0], rvelocity)) eulers = np.array([t3d.euler.quat2euler(q, axes=('s'+rot_order.lower()[::-1]))[::-1] for q in new_rotations])*180.0/np.pi new_df = track.values.copy() root_pos_x = pd.Series(data=positions[:,0], index=new_df.index) root_pos_y = pd.Series(data=positions[:,1], index=new_df.index) root_pos_z = pd.Series(data=positions[:,2], index=new_df.index) root_pos_x_diff = pd.Series(data=velocity[:,0], index=new_df.index) root_pos_z_diff = pd.Series(data=velocity[:,2], index=new_df.index) root_rot_1 = pd.Series(data=eulers[:,0], index=new_df.index) root_rot_2 = pd.Series(data=eulers[:,1], index=new_df.index) root_rot_3 = pd.Series(data=eulers[:,2], index=new_df.index) root_rot_y_diff = pd.Series(data=rvelocity[:,0], index=new_df.index) #new_df.drop([xr_col, yr_col, zr_col, xp_col, zp_col], axis=1, inplace=True) new_df[xp_col] = root_pos_x new_df[yp_col] = root_pos_y new_df[zp_col] = root_pos_z new_df[dxp_col] = root_pos_x_diff new_df[dzp_col] = root_pos_z_diff new_df[r1_col] = root_rot_1 new_df[r2_col] = root_rot_2 new_df[r3_col] = root_rot_3 #new_df[dxr_col] = root_rot_x_diff new_df[dyr_col] = root_rot_y_diff #new_df[dzr_col] = root_rot_z_diff new_track.values = new_df elif self.method == 'pos_xyz_rot_deltas': new_track = track.clone() # Absolute columns xp_col = '%s_Xposition'%track.root_name yp_col = '%s_Yposition'%track.root_name zp_col = '%s_Zposition'%track.root_name #rot_order = track.skeleton[track.root_name]['order'] #%(joint, rot_order[0]) rot_order = track.skeleton[track.root_name]['order'] r1_col = '%s_%srotation'%(track.root_name, rot_order[0]) r2_col = '%s_%srotation'%(track.root_name, rot_order[1]) r3_col = '%s_%srotation'%(track.root_name, rot_order[2]) # Delta columns # dxp_col = '%s_dXposition'%track.root_name # dzp_col = '%s_dZposition'%track.root_name # dxr_col = '%s_dXrotation'%track.root_name # dyr_col = '%s_dYrotation'%track.root_name # dzr_col = '%s_dZrotation'%track.root_name dxp_col = 'reference_dXposition' dyp_col = 'reference_dYposition' dzp_col = 'reference_dZposition' dxr_col = 'reference_dXrotation' dyr_col = 'reference_dYrotation' dzr_col = 'reference_dZrotation' positions = np.transpose(np.array([track.values[xp_col], track.values[yp_col], track.values[zp_col]])) rotations = np.pi/180.0*np.transpose(np.array([track.values[r1_col], track.values[r2_col], track.values[r3_col]])) """ Get Trajectory and smooth it""" trajectory_filterwidth = self.position_smoothing #reference = positions.copy()*np.array([1,0,1]) if trajectory_filterwidth>0: reference = filters.gaussian_filter1d(positions, trajectory_filterwidth, axis=0, mode='nearest') """ Get Root Velocity """ velocity = np.diff(reference, axis=0) velocity = np.vstack((velocity[0,:], velocity)) """ Remove Root Translation """ positions = positions-reference """ Get Forward Direction along the x-z plane, assuming character is facig z-forward """ #forward = [Rotation(f, 'euler', from_deg=True, order=rot_order).rotmat[:,2] for f in rotations] # get the z-axis of the rotation matrix, assuming character is facig z-forward #print("order:" + rot_order.lower()) quats = Quaternions.from_euler(rotations, order=rot_order.lower(), world=False) #calculate the hips forward directions given in global cordinates #side_ax = np.zeros((1,3)) #side_ax[0,self.hips_side_axis]=1 #side_dirs = quats*side_ax side_dirs = quats*self.hips_side_axis forward = np.cross(np.array([[0,1,0]]), side_dirs) """ Smooth Forward Direction """ direction_filterwidth = self.rotation_smoothing if direction_filterwidth>0: forward = filters.gaussian_filter1d(forward, direction_filterwidth, axis=0, mode='nearest') # make unit vector forward = forward / np.sqrt((forward**2).sum(axis=-1))[...,np.newaxis] """ Remove Y Rotation """ target = np.array([[0,0,1]]).repeat(len(forward), axis=0) rotation = Quaternions.between(target, forward)[:,np.newaxis] positions = (-rotation[:,0]) * positions new_rotations = (-self.root_rotation_offset) * (-rotation[:,0]) * quats """ Get Root Rotation """ #print(rotation[:,0]) velocity = (-rotation[:,0]) * velocity rvelocity = Pivots.from_quaternions(rotation[1:] * -rotation[:-1]).ps rvelocity = np.vstack((rvelocity[0], rvelocity)) eulers = np.array([t3d.euler.quat2euler(q, axes=('s'+rot_order.lower()[::-1]))[::-1] for q in new_rotations])*180.0/np.pi new_df = track.values.copy() root_pos_x = pd.Series(data=positions[:,0], index=new_df.index) root_pos_y = pd.Series(data=positions[:,1], index=new_df.index) root_pos_z = pd.Series(data=positions[:,2], index=new_df.index) root_pos_x_diff = pd.Series(data=velocity[:,0], index=new_df.index) root_pos_y_diff = pd.Series(data=velocity[:,1], index=new_df.index) root_pos_z_diff = pd.Series(data=velocity[:,2], index=new_df.index) root_rot_1 = pd.Series(data=eulers[:,0], index=new_df.index) root_rot_2 = pd.Series(data=eulers[:,1], index=new_df.index) root_rot_3 = pd.Series(data=eulers[:,2], index=new_df.index) root_rot_y_diff = pd.Series(data=rvelocity[:,0], index=new_df.index) #new_df.drop([xr_col, yr_col, zr_col, xp_col, zp_col], axis=1, inplace=True) new_df[xp_col] = root_pos_x new_df[yp_col] = root_pos_y new_df[zp_col] = root_pos_z new_df[dxp_col] = root_pos_x_diff new_df[dyp_col] = root_pos_y_diff new_df[dzp_col] = root_pos_z_diff new_df[r1_col] = root_rot_1 new_df[r2_col] = root_rot_2 new_df[r3_col] = root_rot_3 #new_df[dxr_col] = root_rot_x_diff new_df[dyr_col] = root_rot_y_diff #new_df[dzr_col] = root_rot_z_diff new_track.values = new_df elif self.method == 'hip_centric': new_track = track.clone() # Absolute columns xp_col = '%s_Xposition'%track.root_name yp_col = '%s_Yposition'%track.root_name zp_col = '%s_Zposition'%track.root_name xr_col = '%s_Xrotation'%track.root_name yr_col = '%s_Yrotation'%track.root_name zr_col = '%s_Zrotation'%track.root_name new_df = track.values.copy() all_zeros = np.zeros(track.values[xp_col].values.shape) new_df[xp_col] = pd.Series(data=all_zeros, index=new_df.index) new_df[yp_col] = pd.Series(data=all_zeros, index=new_df.index) new_df[zp_col] = pd.Series(data=all_zeros, index=new_df.index) new_df[zp_col] = pd.Series(data=all_zeros, index=new_df.index) new_df[xr_col] = pd.Series(data=all_zeros, index=new_df.index) new_df[yr_col] = pd.Series(data=all_zeros, index=new_df.index) new_df[zr_col] = pd.Series(data=all_zeros, index=new_df.index) new_track.values = new_df #print(new_track.values.columns) Q.append(new_track) return Q def inverse_transform(self, X, copy=None, start_pos=None): Q = [] #TODO: simplify this implementation startx = 0 startz = 0 if start_pos is not None: startx, startz = start_pos for track in X: new_track = track.clone() if self.method == 'abdolute_translation_deltas': new_df = new_track.values xpcol = '%s_Xposition'%track.root_name ypcol = '%s_Yposition'%track.root_name zpcol = '%s_Zposition'%track.root_name dxpcol = '%s_dXposition'%track.root_name dzpcol = '%s_dZposition'%track.root_name dx = track.values[dxpcol].values dz = track.values[dzpcol].values recx = [startx] recz = [startz] for i in range(dx.shape[0]-1): recx.append(recx[i]+dx[i+1]) recz.append(recz[i]+dz[i+1]) # recx = [recx[i]+dx[i+1] for i in range(dx.shape[0]-1)] # recz = [recz[i]+dz[i+1] for i in range(dz.shape[0]-1)] # recx = dx[:-1] + dx[1:] # recz = dz[:-1] + dz[1:] if self.position_smoothing > 0: new_df[xpcol] = pd.Series(data=new_df[xpcol]+recx, index=new_df.index) new_df[zpcol] = pd.Series(data=new_df[zpcol]+recz, index=new_df.index) else: new_df[xpcol] = pd.Series(data=recx, index=new_df.index) new_df[zpcol] = pd.Series(data=recz, index=new_df.index) new_df.drop([dxpcol, dzpcol], axis=1, inplace=True) new_track.values = new_df # end of abdolute_translation_deltas elif self.method == 'pos_rot_deltas': # Absolute columns rot_order = track.skeleton[track.root_name]['order'] xp_col = '%s_Xposition'%track.root_name yp_col = '%s_Yposition'%track.root_name zp_col = '%s_Zposition'%track.root_name xr_col = '%s_Xrotation'%track.root_name yr_col = '%s_Yrotation'%track.root_name zr_col = '%s_Zrotation'%track.root_name r1_col = '%s_%srotation'%(track.root_name, rot_order[0]) r2_col = '%s_%srotation'%(track.root_name, rot_order[1]) r3_col = '%s_%srotation'%(track.root_name, rot_order[2]) # Delta columns # dxp_col = '%s_dXposition'%track.root_name # dzp_col = '%s_dZposition'%track.root_name # dyr_col = '%s_dYrotation'%track.root_name dxp_col = 'reference_dXposition' dzp_col = 'reference_dZposition' dyr_col = 'reference_dYrotation' positions = np.transpose(np.array([track.values[xp_col], track.values[yp_col], track.values[zp_col]])) rotations = np.pi/180.0*np.transpose(np.array([track.values[r1_col], track.values[r2_col], track.values[r3_col]])) quats = Quaternions.from_euler(rotations, order=rot_order.lower(), world=False) new_df = track.values.copy() dx = track.values[dxp_col].values dz = track.values[dzp_col].values dry = track.values[dyr_col].values #rec_p = np.array([startx, 0, startz])+positions[0,:] rec_ry = Quaternions.id(quats.shape[0]) rec_xp = [0] rec_zp = [0] #rec_r = Quaternions.id(quats.shape[0]) for i in range(dx.shape[0]-1): #print(dry[i]) q_y = Quaternions.from_angle_axis(np.array(dry[i+1]), np.array([0,1,0])) rec_ry[i+1] = q_y*rec_ry[i] #print("dx: + " + str(dx[i+1])) dp = rec_ry[i+1]*np.array([dx[i+1], 0, dz[i+1]]) rec_xp.append(rec_xp[i]+dp[0,0]) rec_zp.append(rec_zp[i]+dp[0,2]) if self.separate_root: qq = quats xx = positions[:,0] zz = positions[:,2] else: qq = rec_ry*self.root_rotation_offset*quats pp = rec_ry*positions xx = rec_xp + pp[:,0] zz = rec_zp + pp[:,2] eulers = np.array([t3d.euler.quat2euler(q, axes=('s'+rot_order.lower()[::-1]))[::-1] for q in qq])*180.0/np.pi new_df = track.values.copy() root_rot_1 = pd.Series(data=eulers[:,0], index=new_df.index) root_rot_2 = pd.Series(data=eulers[:,1], index=new_df.index) root_rot_3 = pd.Series(data=eulers[:,2], index=new_df.index) new_df[xp_col] = pd.Series(data=xx, index=new_df.index) new_df[zp_col] = pd.Series(data=zz, index=new_df.index) new_df[r1_col] = pd.Series(data=root_rot_1, index=new_df.index) new_df[r2_col] = pd.Series(data=root_rot_2, index=new_df.index) new_df[r3_col] = pd.Series(data=root_rot_3, index=new_df.index) if self.separate_root: ref_rot_order="ZXY" new_df["reference_Xposition"] = pd.Series(data=rec_xp, index=new_df.index) new_df["reference_Zposition"] = pd.Series(data=rec_zp, index=new_df.index) eulers_ry = np.array([t3d.euler.quat2euler(q, axes=('s'+ref_rot_order.lower()[::-1]))[::-1] for q in rec_ry])*180.0/np.pi new_df["reference_Yrotation"] = pd.Series(data=eulers_ry[:,ref_rot_order.find('Y')], index=new_df.index) new_df.drop([dyr_col, dxp_col, dzp_col], axis=1, inplace=True) new_track.values = new_df elif self.method == 'pos_xyz_rot_deltas': # Absolute columns rot_order = track.skeleton[track.root_name]['order'] xp_col = '%s_Xposition'%track.root_name yp_col = '%s_Yposition'%track.root_name zp_col = '%s_Zposition'%track.root_name xr_col = '%s_Xrotation'%track.root_name yr_col = '%s_Yrotation'%track.root_name zr_col = '%s_Zrotation'%track.root_name r1_col = '%s_%srotation'%(track.root_name, rot_order[0]) r2_col = '%s_%srotation'%(track.root_name, rot_order[1]) r3_col = '%s_%srotation'%(track.root_name, rot_order[2]) # Delta columns # dxp_col = '%s_dXposition'%track.root_name # dzp_col = '%s_dZposition'%track.root_name # dyr_col = '%s_dYrotation'%track.root_name dxp_col = 'reference_dXposition' dyp_col = 'reference_dYposition' dzp_col = 'reference_dZposition' dyr_col = 'reference_dYrotation' positions = np.transpose(np.array([track.values[xp_col], track.values[yp_col], track.values[zp_col]])) rotations = np.pi/180.0*np.transpose(np.array([track.values[r1_col], track.values[r2_col], track.values[r3_col]])) quats = Quaternions.from_euler(rotations, order=rot_order.lower(), world=False) new_df = track.values.copy() dx = track.values[dxp_col].values dy = track.values[dyp_col].values dz = track.values[dzp_col].values dry = track.values[dyr_col].values #rec_p = np.array([startx, 0, startz])+positions[0,:] rec_ry = Quaternions.id(quats.shape[0]) rec_xp = [0] rec_yp = [0] rec_zp = [0] #rec_r = Quaternions.id(quats.shape[0]) for i in range(dx.shape[0]-1): #print(dry[i]) q_y = Quaternions.from_angle_axis(np.array(dry[i+1]), np.array([0,1,0])) rec_ry[i+1] = q_y*rec_ry[i] #print("dx: + " + str(dx[i+1])) dp = rec_ry[i+1]*np.array([dx[i+1], dy[i+1], dz[i+1]]) rec_xp.append(rec_xp[i]+dp[0,0]) rec_yp.append(rec_yp[i]+dp[0,1]) rec_zp.append(rec_zp[i]+dp[0,2]) if self.separate_root: qq = quats xx = positions[:,0] yy = positions[:,1] zz = positions[:,2] else: qq = rec_ry*self.root_rotation_offset*quats pp = rec_ry*positions xx = rec_xp + pp[:,0] yy = rec_yp + pp[:,1] zz = rec_zp + pp[:,2] eulers = np.array([t3d.euler.quat2euler(q, axes=('s'+rot_order.lower()[::-1]))[::-1] for q in qq])*180.0/np.pi new_df = track.values.copy() root_rot_1 = pd.Series(data=eulers[:,0], index=new_df.index) root_rot_2 = pd.Series(data=eulers[:,1], index=new_df.index) root_rot_3 = pd.Series(data=eulers[:,2], index=new_df.index) new_df[xp_col] = pd.Series(data=xx, index=new_df.index) new_df[yp_col] = pd.Series(data=yy, index=new_df.index) new_df[zp_col] = pd.Series(data=zz, index=new_df.index) new_df[r1_col] = pd.Series(data=root_rot_1, index=new_df.index) new_df[r2_col] = pd.Series(data=root_rot_2, index=new_df.index) new_df[r3_col] = pd.Series(data=root_rot_3, index=new_df.index) if self.separate_root: new_df["reference_Xposition"] = pd.Series(data=rec_xp, index=new_df.index) new_df["reference_Yposition"] = pd.Series(data=rec_yp, index=new_df.index) new_df["reference_Zposition"] = pd.Series(data=rec_zp, index=new_df.index) eulers_ry = np.array([t3d.euler.quat2euler(q, axes=('s'+rot_order.lower()[::-1]))[::-1] for q in rec_ry])*180.0/np.pi new_df["reference_Yrotation"] = pd.Series(data=eulers_ry[:,rot_order.find('Y')], index=new_df.index) new_df.drop([dyr_col, dxp_col, dyp_col, dzp_col], axis=1, inplace=True) new_track.values = new_df #print(new_track.values.columns) Q.append(new_track) return Q class RootCentricPositionNormalizer(BaseEstimator, TransformerMixin): def __init__(self): pass def fit(self, X, y=None): return self def transform(self, X, y=None): Q = [] for track in X: new_track = track.clone() rxp = '%s_Xposition'%track.root_name ryp = '%s_Yposition'%track.root_name rzp = '%s_Zposition'%track.root_name projected_root_pos = track.values[[rxp, ryp, rzp]] projected_root_pos.loc[:,ryp] = 0 # we want the root's projection on the floor plane as the ref new_df = pd.DataFrame(index=track.values.index) all_but_root = [joint for joint in track.skeleton if track.root_name not in joint] # all_but_root = [joint for joint in track.skeleton] for joint in all_but_root: new_df['%s_Xposition'%joint] = pd.Series(data=track.values['%s_Xposition'%joint]-projected_root_pos[rxp], index=new_df.index) new_df['%s_Yposition'%joint] = pd.Series(data=track.values['%s_Yposition'%joint]-projected_root_pos[ryp], index=new_df.index) new_df['%s_Zposition'%joint] = pd.Series(data=track.values['%s_Zposition'%joint]-projected_root_pos[rzp], index=new_df.index) # keep the root as it is now new_df[rxp] = track.values[rxp] new_df[ryp] = track.values[ryp] new_df[rzp] = track.values[rzp] new_track.values = new_df Q.append(new_track) return Q def inverse_transform(self, X, copy=None): Q = [] for track in X: new_track = track.clone() rxp = '%s_Xposition'%track.root_name ryp = '%s_Yposition'%track.root_name rzp = '%s_Zposition'%track.root_name projected_root_pos = track.values[[rxp, ryp, rzp]] projected_root_pos.loc[:,ryp] = 0 # we want the root's projection on the floor plane as the ref new_df = pd.DataFrame(index=track.values.index) for joint in track.skeleton: new_df['%s_Xposition'%joint] = pd.Series(data=track.values['%s_Xposition'%joint]+projected_root_pos[rxp], index=new_df.index) new_df['%s_Yposition'%joint] = pd.Series(data=track.values['%s_Yposition'%joint]+projected_root_pos[ryp], index=new_df.index) new_df['%s_Zposition'%joint] = pd.Series(data=track.values['%s_Zposition'%joint]+projected_root_pos[rzp], index=new_df.index) new_track.values = new_df Q.append(new_track) return Q class Flattener(BaseEstimator, TransformerMixin): def __init__(self): pass def fit(self, X, y=None): return self def transform(self, X, y=None): return np.concatenate(X, axis=0) class ConstantsRemover(BaseEstimator, TransformerMixin): ''' For now it just looks at the first track ''' def __init__(self, eps = 1e-6): self.eps = eps def fit(self, X, y=None): stds = X[0].values.std() cols = X[0].values.columns.values self.const_dims_ = [c for c in cols if (stds[c] < self.eps).any()] self.const_values_ = {c:X[0].values[c].values[0] for c in cols if (stds[c] < self.eps).any()} return self def transform(self, X, y=None): Q = [] for track in X: t2 = track.clone() #for key in t2.skeleton.keys(): # if key in self.ConstDims_: # t2.skeleton.pop(key) #print(track.values.columns.difference(self.const_dims_)) t2.values.drop(self.const_dims_, axis=1, inplace=True) #t2.values = track.values[track.values.columns.difference(self.const_dims_)] Q.append(t2) return Q def inverse_transform(self, X, copy=None): Q = [] for track in X: t2 = track.clone() for d in self.const_dims_: t2.values[d] = self.const_values_[d] # t2.values.assign(d=pd.Series(data=self.const_values_[d], index = t2.values.index)) Q.append(t2) return Q class ListStandardScaler(BaseEstimator, TransformerMixin): def __init__(self, is_DataFrame=False): self.is_DataFrame = is_DataFrame def fit(self, X, y=None): if self.is_DataFrame: X_train_flat = np.concatenate([m.values for m in X], axis=0) else: X_train_flat = np.concatenate([m for m in X], axis=0) self.data_mean_ = np.mean(X_train_flat, axis=0) self.data_std_ = np.std(X_train_flat, axis=0) return self def transform(self, X, y=None): Q = [] for track in X: if self.is_DataFrame: normalized_track = track.copy() normalized_track.values = (track.values - self.data_mean_) / self.data_std_ else: normalized_track = (track - self.data_mean_) / self.data_std_ Q.append(normalized_track) if self.is_DataFrame: return Q else: return np.array(Q) def inverse_transform(self, X, copy=None): Q = [] for track in X: if self.is_DataFrame: unnormalized_track = track.copy() unnormalized_track.values = (track.values * self.data_std_) + self.data_mean_ else: unnormalized_track = (track * self.data_std_) + self.data_mean_ Q.append(unnormalized_track) if self.is_DataFrame: return Q else: return np.array(Q) class ListMinMaxScaler(BaseEstimator, TransformerMixin): def __init__(self, is_DataFrame=False): self.is_DataFrame = is_DataFrame def fit(self, X, y=None): if self.is_DataFrame: X_train_flat = np.concatenate([m.values for m in X], axis=0) else: X_train_flat = np.concatenate([m for m in X], axis=0) self.data_max_ = np.max(X_train_flat, axis=0) self.data_min_ = np.min(X_train_flat, axis=0) return self def transform(self, X, y=None): Q = [] for track in X: if self.is_DataFrame: normalized_track = track.copy() normalized_track.values = (track.values - self.data_min_) / (self.data_max_ - self.data_min_) else: normalized_track = (track - self.data_min_) / (self.data_max_ - self.data_min_) Q.append(normalized_track) if self.is_DataFrame: return Q else: return np.array(Q) def inverse_transform(self, X, copy=None): Q = [] for track in X: if self.is_DataFrame: unnormalized_track = track.copy() unnormalized_track.values = (track.values * (self.data_max_ - self.data_min_)) + self.data_min_ else: unnormalized_track = (track * (self.data_max_ - self.data_min_)) + self.data_min_ Q.append(unnormalized_track) if self.is_DataFrame: return Q else: return np.array(Q) class Resampler(BaseEstimator, TransformerMixin): def __init__(self, fps, method='cubic'): ''' Method to resample a pandas dataframe to a different framerate. NOTE: Pandas resampling is quit unintuitive when resampling to odd framerates using interpolation. Thus we do it in this complex way. ''' self.tgt_frametime = 1.0/fps self.method = method def fit(self, X, y=None): #print("Resampling to tgt_frametime: " + str(self.tgt_frametime)) self.orig_frametime=X[0].framerate return self def resample_dataframe(self, df, frametime, method='cubic'): #Create a time index for the resampled data rate = str(round(1.0e9*frametime))+'N' time_index = df.resample(rate).indices #reindex the old data. This will turn all non-matching indices to NAN tmp = df.reindex(time_index) #merge with the old data and sort tmp = pd.concat([df, tmp]).sort_index() #remove duplicate time indices. Then fill the NAN values using interpolation tmp=tmp[~tmp.index.duplicated(keep='first')].interpolate(method=method) #return the values using the resampled indices return tmp.loc[list(time_index)] def resample_df(self, df, new_frametime, old_frametime, mode='cubic'): #Create a time index for the resampled data data = df.values nframes = data.shape[0] nframes_new = round(nframes*old_frametime/new_frametime) x = np.arange(0, nframes)/(nframes-1) xnew = np.arange(0, nframes_new)/(nframes_new-1) data_out = np.zeros((nframes_new, data.shape[1])) for jj in range(data.shape[1]): y = data[:,jj] f = interpolate.interp1d(x, y, bounds_error=False, kind=mode, fill_value='extrapolate') data_out[:,jj] = f(xnew) time_index = pd.to_timedelta([f for f in range(xnew.shape[0])], unit='s')*new_frametime out = pd.DataFrame(data=data_out, index=time_index, columns=df.columns) #Scale root deltas to match new frame-rate sc = nframes/nframes_new rootdelta_cols = [c for c in df.columns if ('reference_d' in c)] out[rootdelta_cols]*=sc return out # def resample_poly_df(self, df, new_frametime, old_frametime): # old_fps = round(1/old_frametime) # new_fps = round(1/new_frametime) # lcm = np.lcm(old_fps, new_fps) # up = lcm//old_fps # down = lcm//new_fps # new_vals = signal.resample_poly(df.values, up, down, padtype='line') # time_index = pd.to_timedelta([f for f in range(new_vals.shape[0])], unit='s')*new_frametime # new_df = pd.DataFrame(data=new_vals, index=time_index, columns=df.columns) # return new_df def transform(self, X, y=None): Q = [] for track in X: new_track = track.clone() # if self.method=="resample_poly": # new_track.values = self.resample_poly_df(track.values, self.tgt_frametime, track.framerate) # else: new_track.values = self.resample_df(track.values, self.tgt_frametime, track.framerate, self.method) #new_track.values = self.resample_dataframe(track.values, self.tgt_frametime, method=self.method) new_track.framerate = self.tgt_frametime Q.append(new_track) return Q def inverse_transform(self, X, copy=None): Q = [] for track in X: new_track = track.clone() #new_track.values = self.resample_dataframe(track.values, self.orig_frametime, method=self.method) if self.method=="resample_poly": new_track.values = self.resample_poly_df(track.values, self.orig_frametime, track.framerate) else: new_track.values = self.resample_df(track.values, self.orig_frametime, track.framerate, self.method) new_track.framerate = self.orig_frametime Q.append(new_track) return Q class DownSampler(BaseEstimator, TransformerMixin): def __init__(self, tgt_fps, keep_all=False): self.tgt_fps = tgt_fps self.keep_all = keep_all def fit(self, X, y=None): return self def transform(self, X, y=None): Q = [] for track in X: orig_fps=round(1.0/track.framerate) rate = orig_fps//self.tgt_fps if orig_fps%self.tgt_fps!=0: print("error orig_fps (" + str(orig_fps) + ") is not dividable with tgt_fps (" + str(self.tgt_fps) + ")") else: print("downsampling with rate: " + str(rate)) #print(track.values.size) for ii in range(0,rate): new_track = track.clone() if self.keep_all: new_track.take_name = new_track.take_name + "_" + str(ii).zfill(2) new_track.values = track.values[ii::rate].copy() #print(new_track.values.size) #new_track = track[0:-1:self.rate] new_track.framerate = 1.0/self.tgt_fps Q.append(new_track) if not self.keep_all: break return Q def inverse_transform(self, X, copy=None): return X class ReverseTime(BaseEstimator, TransformerMixin): def __init__(self, append=True): self.append = append def fit(self, X, y=None): return self def transform(self, X, y=None): #print("ReverseTime") Q = [] if self.append: for track in X: Q.append(track) for track in X: new_track = track.clone() new_track.values = track.values[-1::-1] new_track.values.index=new_track.values.index[0]-new_track.values.index Q.append(new_track) return Q def inverse_transform(self, X, copy=None): return X class ListFeatureUnion(BaseEstimator, TransformerMixin): def __init__(self, processors): self.processors = processors def fit(self, X, y=None): assert(y is None) for proc in self.processors: if isinstance(proc, Pipeline): #Loop steps and run fit on each. This is necessary since #running fit on a Pipeline runs fit_transform on all steps #and not only fit. for step in proc.steps: step[1].fit(X) else: proc.fit(X) return self def transform(self, X, y=None): assert(y is None) #print("ListFeatureUnion") Q = [] idx=0 for proc in self.processors: Z = proc.transform(X) if idx==0: Q = Z else: assert(len(Q)==len(Z)) for idx2,track in enumerate(Z): Q[idx2].values = pd.concat([Q[idx2].values,Z[idx2].values], axis=1) idx += 1 return Q def inverse_transform(self, X, y=None): return X class RollingStatsCalculator(BaseEstimator, TransformerMixin): ''' Creates a causal mean and std filter with a rolling window of length win (based on using prev and current values) ''' def __init__(self, win): self.win = win def fit(self, X, y=None): return self def transform(self, X, y=None): #print("RollingStatsCalculator: " + str(self.win)) Q = [] for track in X: new_track = track.clone() mean_df = track.values.rolling(window=self.win).mean() std_df = track.values.rolling(window=self.win).std() # rolling.mean results in Nans in start seq. Here we fill these win = min(self.win, new_track.values.shape[0]) for i in range(1,win): mm=track.values[:i].rolling(window=i).mean() ss=track.values[:i].rolling(window=i).std() mean_df.iloc[i-1] = mm.iloc[i-1] std_df.iloc[i-1] = ss.iloc[i-1] std_df.iloc[0] = std_df.iloc[1] # Append to new_track.values=pd.concat([mean_df.add_suffix('_mean'), std_df.add_suffix('_std')], axis=1) Q.append(new_track) return Q def inverse_transform(self, X, copy=None): return X class FeatureCounter(BaseEstimator, TransformerMixin): def __init__(self): pass def fit(self, X, y=None): self.n_features = len(X[0].values.columns) return self def transform(self, X, y=None): return X def inverse_transform(self, X, copy=None): return X #TODO: JointsSelector (x) #TODO: SegmentMaker #TODO: DynamicFeaturesAdder #TODO: ShapeFeaturesAdder #TODO: DataFrameNumpier (x) class TemplateTransform(BaseEstimator, TransformerMixin): def __init__(self): pass def fit(self, X, y=None): return self def transform(self, X, y=None): return X ================================================ FILE: pymo/rotation_tools.py ================================================ ''' Tools for Manipulating and Converting 3D Rotations By Omid Alemi Created: June 12, 2017 Adapted from that matlab file... ''' import math import numpy as np import transforms3d as t3d from pymo.Quaternions import Quaternions def deg2rad(x): return x/180*math.pi def rad2deg(x): return x/math.pi*180 def unroll(rots): new_rot = rots.copy() ang0 = np.linalg.norm(rots, axis=1) + 1e-8 idx = np.where(ang0>np.pi)[0] ax = rots/np.tile(ang0[:,None], (1,3)) ang1=ang0-2*np.pi alt_rot = ax*np.tile(ang1[:,None], (1,3)) new_rot[idx] = alt_rot[idx] return new_rot def unroll_1(rots): new_rots = rots.copy() # Compute angles and alternative rotation angles angs = np.linalg.norm(rots, axis=1) alt_angs=2*np.pi-angs #find discontinuities d_angs = np.diff(angs, axis=0) d_angs2 = alt_angs[1:]-angs[:-1] swps = np.where(np.abs(d_angs2)i', rots[:-1,:], rots[1:,:]) #ax = rots/np.tile(angs[:, None], (1,3)) #d_ax = np.linalg.norm(np.diff(ax, axis=0), axis=1) alt_angs=2*np.pi-angs #find discontinuities d_angs = np.diff(angs, axis=0) d_angs2 = alt_angs[1:]-angs[:-1] # FIXME should check if dot product is <0 not norm d_ax swps = np.where((dotprod<-1))[0] #swps = np.where((np.abs(d_ax)>0.5))[0] #swps = np.where(np.abs(d_angs2) 1.0e-10: vector = rot / theta else: vector = np.array([1.,0.,0.]) theta=0.0 eul = t3d.euler.axangle2euler(vector, theta, 'r' + order.lower()) if use_deg: return np.rad2deg(eul) else: return eul def euler2vectors(rot, order='XYZ', use_deg=False): if use_deg: rot = np.deg2rad(rot) # order = "r" + (order.lower())[::-1] # try both r and s # rotation_matrix = np.transpose(t3d.euler.euler2mat(rot[0], rot[1], rot[2], axes=order)) rotation_matrix = t3d.euler.euler2mat(rot[0], rot[1], rot[2], 'r' + order.lower()) y_vector_x_value = rotation_matrix[0][1] y_vector_y_value = rotation_matrix[1][1] y_vector_z_value = rotation_matrix[2][1] z_vector_x_value = rotation_matrix[0][2] z_vector_y_value = rotation_matrix[1][2] z_vector_z_value = rotation_matrix[2][2] return y_vector_x_value, y_vector_y_value, y_vector_z_value, z_vector_x_value, z_vector_y_value, z_vector_z_value def vectors2euler(axises, order='XYZ', use_deg=False): # create rotation matrix y_vector = [axises[0], axises[1], axises[2]] z_vector = [axises[3], axises[4], axises[5]] x_vector = np.cross(y_vector, z_vector) R = np.column_stack((x_vector, y_vector, z_vector)) # orthogonalize vectors u,s,vt = np.linalg.svd(R); R = np.matmul(u, vt); # to euler eul = t3d.euler.mat2euler(R, 'r' + order.lower()) if use_deg: return np.rad2deg(eul) else: return eul class Rotation(): def __init__(self,rot, param_type, **params): self.rotmat = [] if param_type == 'euler': self._from_euler(rot[0],rot[1],rot[2], params) elif param_type == 'expmap': self._from_expmap(rot[0], rot[1], rot[2], params) def _from_euler(self, alpha, beta, gamma, params): '''Expecting degress''' if params['from_deg']==True: alpha = deg2rad(alpha) beta = deg2rad(beta) gamma = deg2rad(gamma) order = "s" + ((params['order']).lower())[::-1] # Quaternions.from_euler() self.rotmat = np.transpose(t3d.euler.euler2mat(gamma, beta , alpha, axes=order)) # ca = math.cos(alpha) # cb = math.cos(beta) # cg = math.cos(gamma) # sa = math.sin(alpha) # sb = math.sin(beta) # sg = math.sin(gamma) # # Rx = np.asarray([[1, 0, 0], # [0, ca, sa], # [0, -sa, ca] # ]) # # Ry = np.asarray([[cb, 0, -sb], # [0, 1, 0], # [sb, 0, cb]]) # # Rz = np.asarray([[cg, sg, 0], # [-sg, cg, 0], # [0, 0, 1]]) # # self.rotmat = np.eye(3) # # order = params['order'] # for i in range(0,len(order)): # if order[i]=='X': # self.rotmat = np.matmul(Rx, self.rotmat) # elif order[i]=='Y': # self.rotmat = np.matmul(Ry, self.rotmat) # elif order[i]=='Z': # self.rotmat = np.matmul(Rz, self.rotmat) # else: # print('unknown rotation axis: ' + order[i]) # # # self.rotmat = np.matmul(np.matmul(Rz, Ry), Rx) # print ("------" + "TRUE") # print (self.rotmat) def _from_expmap(self, alpha, beta, gamma, params): if (alpha == 0 and beta == 0 and gamma == 0): self.rotmat = np.eye(3) return #TODO: Check exp map params theta = np.linalg.norm([alpha, beta, gamma]) expmap = [alpha, beta, gamma] / theta x = expmap[0] y = expmap[1] z = expmap[2] s = math.sin(theta/2) c = math.cos(theta/2) self.rotmat = np.asarray([ [2*(x**2-1)*s**2+1, 2*x*y*s**2-2*z*c*s, 2*x*z*s**2+2*y*c*s], [2*x*y*s**2+2*z*c*s, 2*(y**2-1)*s**2+1, 2*y*z*s**2-2*x*c*s], [2*x*z*s**2-2*y*c*s, 2*y*z*s**2+2*x*c*s , 2*(z**2-1)*s**2+1] ]) def get_euler_axis(self): R = self.rotmat theta = math.acos((self.rotmat.trace() - 1) / 2) axis = np.asarray([R[2,1] - R[1,2], R[0,2] - R[2,0], R[1,0] - R[0,1]]) axis = axis/(2*math.sin(theta)) return theta, axis def to_expmap(self): axis, theta = t3d.axangles.mat2axangle(self.rotmat, unit_thresh=1e-05) # theta, axis = self.get_euler_axis() rot_arr = theta * axis if np.isnan(rot_arr).any(): rot_arr = [0, 0, 0] return rot_arr def to_euler(self, use_deg=False, order='xyz'): order = "s" + order.lower() eulers = t3d.euler.mat2euler(np.transpose(self.rotmat), axes=order) return eulers[::-1] # eulers = np.zeros((2, 3)) # # if np.absolute(np.absolute(self.rotmat[2, 0]) - 1) < 1e-12: # #GIMBAL LOCK! # print('Gimbal') # if np.absolute(self.rotmat[2, 0]) - 1 < 1e-12: # eulers[:,0] = math.atan2(-self.rotmat[0,1], -self.rotmat[0,2]) # eulers[:,1] = -math.pi/2 # else: # eulers[:,0] = math.atan2(self.rotmat[0,1], -elf.rotmat[0,2]) # eulers[:,1] = math.pi/2 # # return eulers # # theta = - math.asin(self.rotmat[2,0]) # theta2 = math.pi - theta # # # psi1, psi2 # eulers[0,0] = math.atan2(self.rotmat[2,1]/math.cos(theta), self.rotmat[2,2]/math.cos(theta)) # eulers[1,0] = math.atan2(self.rotmat[2,1]/math.cos(theta2), self.rotmat[2,2]/math.cos(theta2)) # # # theta1, theta2 # eulers[0,1] = theta # eulers[1,1] = theta2 # # # phi1, phi2 # eulers[0,2] = math.atan2(self.rotmat[1,0]/math.cos(theta), self.rotmat[0,0]/math.cos(theta)) # eulers[1,2] = math.atan2(self.rotmat[1,0]/math.cos(theta2), self.rotmat[0,0]/math.cos(theta2)) # if use_deg: eulers = rad2deg(eulers) return eulers def to_quat(self): #TODO pass def __str__(self): return "Rotation Matrix: \n " + self.rotmat.__str__() ================================================ FILE: pymo/viz_tools.py ================================================ import pandas as pd import numpy as np import matplotlib.animation as animation import matplotlib.colors as colors import matplotlib.patheffects as pe import matplotlib.pyplot as plt #import IPython import os def save_fig(fig_id, tight_layout=True): if tight_layout: plt.tight_layout() plt.savefig(fig_id + '.png', format='png', dpi=300) def draw_stickfigure(mocap_track, frame, data=None, joints=None, draw_names=False, ax=None, figsize=(8,8)): if ax is None: fig = plt.figure(figsize=figsize) ax = fig.add_subplot(111) if joints is None: joints_to_draw = mocap_track.skeleton.keys() else: joints_to_draw = joints if data is None: df = mocap_track.values else: df = data for joint in joints_to_draw: ax.scatter(x=df['%s_Xposition'%joint][frame], y=df['%s_Yposition'%joint][frame], alpha=0.6, c='b', marker='o') parent_x = df['%s_Xposition'%joint][frame] parent_y = df['%s_Yposition'%joint][frame] children_to_draw = [c for c in mocap_track.skeleton[joint]['children'] if c in joints_to_draw] for c in children_to_draw: child_x = df['%s_Xposition'%c][frame] child_y = df['%s_Yposition'%c][frame] ax.plot([parent_x, child_x], [parent_y, child_y], 'k-', lw=2) if draw_names: ax.annotate(joint, (df['%s_Xposition'%joint][frame] + 0.1, df['%s_Yposition'%joint][frame] + 0.1)) return ax def draw_stickfigure3d(mocap_track, frame, data=None, joints=None, draw_names=False, ax=None, figsize=(8,8)): from mpl_toolkits.mplot3d import Axes3D if ax is None: fig = plt.figure(figsize=figsize) ax = fig.add_subplot(111, projection='3d') if joints is None: joints_to_draw = mocap_track.skeleton.keys() else: joints_to_draw = joints if data is None: df = mocap_track.values else: df = data for joint in joints_to_draw: parent_x = df['%s_Xposition'%joint][frame] parent_y = df['%s_Zposition'%joint][frame] parent_z = df['%s_Yposition'%joint][frame] # ^ In mocaps, Y is the up-right axis ax.scatter(xs=parent_x, ys=parent_y, zs=parent_z, alpha=0.6, c='b', marker='o') children_to_draw = [c for c in mocap_track.skeleton[joint]['children'] if c in joints_to_draw] for c in children_to_draw: child_x = df['%s_Xposition'%c][frame] child_y = df['%s_Zposition'%c][frame] child_z = df['%s_Yposition'%c][frame] # ^ In mocaps, Y is the up-right axis ax.plot([parent_x, child_x], [parent_y, child_y], [parent_z, child_z], 'k-', lw=2, c='black') if draw_names: ax.text(x=parent_x + 0.1, y=parent_y + 0.1, z=parent_z + 0.1, s=joint, color='rgba(0,0,0,0.9') return ax def sketch_move(mocap_track, data=None, ax=None, figsize=(16,8)): if ax is None: fig = plt.figure(figsize=figsize) ax = fig.add_subplot(111) if data is None: data = mocap_track.values for frame in range(0, data.shape[0], 4): # draw_stickfigure(mocap_track, f, data=data, ax=ax) for joint in mocap_track.skeleton.keys(): children_to_draw = [c for c in mocap_track.skeleton[joint]['children']] parent_x = data['%s_Xposition'%joint][frame] parent_y = data['%s_Yposition'%joint][frame] frame_alpha = frame/data.shape[0] for c in children_to_draw: child_x = data['%s_Xposition'%c][frame] child_y = data['%s_Yposition'%c][frame] ax.plot([parent_x, child_x], [parent_y, child_y], '-', lw=1, color='gray', alpha=frame_alpha) def render_mp4(mocap_track, filename, data=None, ax=None, axis_scale=50, elev=45, azim=45): if ax is None: fig = plt.figure(figsize=(10,10)) ax = fig.add_subplot(111, projection='3d') ax.set_xlim3d(-axis_scale, axis_scale) ax.set_zlim3d( 0, axis_scale) ax.set_ylim3d(-axis_scale, axis_scale) ax.grid(True) ax.set_axis_off() ax.view_init(elev=elev, azim=azim) xs = np.linspace(-200, 200, 50) ys = np.linspace(-200, 200, 50) X, Y = np.meshgrid(xs, ys) Z = np.zeros(X.shape) wframe = ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2, color='grey',lw=0.2) # fig = plt.figure(figsize=figsize) # ax = fig.add_subplot(111) if data is None: data = mocap_track.values fps=int(np.round(1/mocap_track.framerate)) lines=[] lines.append([plt.plot([0,0], [0,0], [0,0], color='red', lw=2, path_effects=[pe.Stroke(linewidth=3, foreground='black'), pe.Normal()])[0] for _ in range(len(mocap_track.skeleton.keys()))]) def animate(frame): changed = [] j=0 for joint in mocap_track.skeleton.keys(): children_to_draw = [c for c in mocap_track.skeleton[joint]['children']] parent_x = data['%s_Xposition'%joint][frame] parent_y = data['%s_Yposition'%joint][frame] parent_z = data['%s_Zposition'%joint][frame] #frame_alpha = frame/data.shape[0] for c in children_to_draw: child_x = data['%s_Xposition'%c][frame] child_y = data['%s_Yposition'%c][frame] child_z = data['%s_Zposition'%c][frame] lines[0][j].set_data(np.array([[child_x, parent_x],[-child_z,-parent_z]])) lines[0][j].set_3d_properties(np.array([ child_y,parent_y])) changed += lines j+=1 return changed plt.tight_layout() ani = animation.FuncAnimation(fig, animate, np.arange(data.shape[0]), interval=1000/fps) if filename != None: ani.save(filename, fps=fps, bitrate=13934) ani.event_source.stop() del ani plt.close() try: plt.show() plt.save() except AttributeError as e: pass def viz_cnn_filter(feature_to_viz, mocap_track, data, gap=25): fig = plt.figure(figsize=(16,4)) ax = plt.subplot2grid((1,8),(0,0)) ax.imshow(feature_to_viz.T, aspect='auto', interpolation='nearest') ax = plt.subplot2grid((1,8),(0,1), colspan=7) for frame in range(feature_to_viz.shape[0]): frame_alpha = 0.2#frame/data.shape[0] * 2 + 0.2 for joint_i, joint in enumerate(mocap_track.skeleton.keys()): children_to_draw = [c for c in mocap_track.skeleton[joint]['children']] parent_x = data['%s_Xposition'%joint][frame] + frame * gap parent_y = data['%s_Yposition'%joint][frame] ax.scatter(x=parent_x, y=parent_y, alpha=0.6, cmap='RdBu', c=feature_to_viz[frame][joint_i] * 10000, marker='o', s = abs(feature_to_viz[frame][joint_i] * 10000)) plt.axis('off') for c in children_to_draw: child_x = data['%s_Xposition'%c][frame] + frame * gap child_y = data['%s_Yposition'%c][frame] ax.plot([parent_x, child_x], [parent_y, child_y], '-', lw=1, color='gray', alpha=frame_alpha) def print_skel(X): stack = [X.root_name] tab=0 while stack: joint = stack.pop() tab = len(stack) print('%s- %s (%s)'%('| '*tab, joint, X.skeleton[joint]['parent'])) for c in X.skeleton[joint]['children']: stack.append(c) # def nb_play_mocap_fromurl(mocap, mf, frame_time=1/30, scale=1, base_url='http://titan:8385'): # if mf == 'bvh': # bw = BVHWriter() # with open('test.bvh', 'w') as ofile: # bw.write(mocap, ofile) # filepath = '../notebooks/test.bvh' # elif mf == 'pos': # c = list(mocap.values.columns) # for cc in c: # if 'rotation' in cc: # c.remove(cc) # mocap.values.to_csv('test.csv', index=False, columns=c) # filepath = '../notebooks/test.csv' # else: # return # url = '%s/mocapplayer/player.html?data_url=%s&scale=%f&cz=200&order=xzyi&frame_time=%f'%(base_url, filepath, scale, frame_time) # iframe = '' # link = 'New Window'%url # return IPython.display.HTML(iframe+link) # def nb_play_mocap(mocap, mf, meta=None, frame_time=1/30, scale=1, camera_z=500, base_url=None): # data_template = 'var dataBuffer = `$$DATA$$`;' # data_template += 'var metadata = $$META$$;' # data_template += 'start(dataBuffer, metadata, $$CZ$$, $$SCALE$$, $$FRAMETIME$$);' # dir_path = os.path.dirname(os.path.realpath(__file__)) # if base_url is None: # base_url = os.path.join(dir_path, 'mocapplayer/playBuffer.html') # # print(dir_path) # if mf == 'bvh': # pass # elif mf == 'pos': # cols = list(mocap.values.columns) # for c in cols: # if 'rotation' in c: # cols.remove(c) # data_csv = mocap.values.to_csv(index=False, columns=cols) # if meta is not None: # lines = [','.join(item) for item in meta.astype('str')] # meta_csv = '[' + ','.join('[%s]'%l for l in lines) +']' # else: # meta_csv = '[]' # data_assigned = data_template.replace('$$DATA$$', data_csv) # data_assigned = data_assigned.replace('$$META$$', meta_csv) # data_assigned = data_assigned.replace('$$CZ$$', str(camera_z)) # data_assigned = data_assigned.replace('$$SCALE$$', str(scale)) # data_assigned = data_assigned.replace('$$FRAMETIME$$', str(frame_time)) # else: # return # with open(os.path.join(dir_path, 'mocapplayer/data.js'), 'w') as oFile: # oFile.write(data_assigned) # url = '%s?&cz=200&order=xzyi&frame_time=%f&scale=%f'%(base_url, frame_time, scale) # iframe = '' # link = 'New Window'%url # return IPython.display.HTML(iframe+link) ================================================ FILE: pymo/writers.py ================================================ import numpy as np import pandas as pd class BVHWriter(): def __init__(self): pass def write(self, X, ofile, framerate=-1, start=0, stop=-1): # Writing the skeleton info ofile.write('HIERARCHY\n') self.motions_ = [] self._printJoint(X, X.root_name, 0, ofile) if stop > 0: nframes = stop-start else: nframes = X.values.shape[0] stop = X.values.shape[0] # Writing the motion header ofile.write('MOTION\n') ofile.write('Frames: %d\n'%nframes) if framerate > 0: ofile.write('Frame Time: %f\n'%float(1.0/framerate)) else: ofile.write('Frame Time: %f\n'%X.framerate) # Writing the data self.motions_ = np.asarray(self.motions_).T lines = [" ".join(item) for item in self.motions_[start:stop].astype(str)] ofile.write("".join("%s\n"%l for l in lines)) def _printJoint(self, X, joint, tab, ofile): if X.skeleton[joint]['parent'] == None: ofile.write('ROOT %s\n'%joint) elif len(X.skeleton[joint]['children']) > 0: ofile.write('%sJOINT %s\n'%('\t'*(tab), joint)) else: ofile.write('%sEnd site\n'%('\t'*(tab))) ofile.write('%s{\n'%('\t'*(tab))) ofile.write('%sOFFSET %3.5f %3.5f %3.5f\n'%('\t'*(tab+1), X.skeleton[joint]['offsets'][0], X.skeleton[joint]['offsets'][1], X.skeleton[joint]['offsets'][2])) rot_order = X.skeleton[joint]['order'] #print("rot_order = " + rot_order) channels = X.skeleton[joint]['channels'] rot = [c for c in channels if ('rotation' in c)] pos = [c for c in channels if ('position' in c)] n_channels = len(rot) +len(pos) ch_str = '' if n_channels > 0: for ci in range(len(pos)): cn = pos[ci] self.motions_.append(np.asarray(X.values['%s_%s'%(joint,cn)].values)) ch_str = ch_str + ' ' + cn for ci in range(len(rot)): cn = '%srotation'%(rot_order[ci]) self.motions_.append(np.asarray(X.values['%s_%s'%(joint,cn)].values)) ch_str = ch_str + ' ' + cn if len(X.skeleton[joint]['children']) > 0: #ch_str = ''.join(' %s'*n_channels%tuple(channels)) ofile.write('%sCHANNELS %d%s\n' %('\t'*(tab+1), n_channels, ch_str)) for c in X.skeleton[joint]['children']: self._printJoint(X, c, tab+1, ofile) ofile.write('%s}\n'%('\t'*(tab))) ================================================ FILE: requirements.txt ================================================ numpy scipy librosa torchmetrics pytorch-lightning==1.8.3 pandas matplotlib scikit-learn==0.24.2 mpi4py joblib smplx transforms3d x-transformers madmom ftfy axial_positional_embedding einops entmax lightning-bolts loguru madgrad mido jsmin optuna ================================================ FILE: run_docker.sh ================================================ docker run -it --rm --gpus '"device=0,1,2,3,4,5,6,7"' -v $PWD:/workspace/dockers --ipc=host -v=$HOME/data:/workspace/dockers/data simonal_diffusion /bin/sh -c 'cd dockers; bash' ================================================ FILE: synthesize.py ================================================ # Copyright 2023 Motorica AB, Inc. All Rights Reserved. from os.path import join import os, sys, getopt import torch import numpy as np import pickle as pkl from pytorch_lightning import Trainer, seed_everything from utils.motion_dataset import styles2onehot, nans2zeros from models.LightningModel import LitLDA def sample_mixmodels(models, batches, guidance_factors): # asserts that the models are compatible, # i.e. they have the same number of noise steps, pose dim and pose scalers assert len(guidance_factors)==(len(models)-1), "n_guidance_factors should be eq to n_models-1" noise_sched_0 = models[0].noise_schedule o_scaler_0 = models[0].hparams["Data"]["scalers"]["out_scaler"] eps = 0.000001 for i in range(1, len(models)): # models should have same noise schedule assert torch.all(torch.abs(models[i].noise_schedule - noise_sched_0) 0: noise = torch.randn_like(poses) sigma = ((1.0 - alpha_cum[n-1]) / (1.0 - alpha_cum[n]) * beta[n])**0.5 poses += sigma * noise out_poses = models[-1].destandardizeOutput(poses) if not models[-1].unconditional: out_ctrl = models[-1].destandardizeInput(ctrl) anim_clip = torch.cat((out_poses, out_ctrl), dim=2).cpu().detach().numpy() else: anim_clip = out_poses.cpu().detach().numpy() return anim_clip def do_synthesize(models, l_conds, g_conds, file_name, postfix, trim, dest_dir, guidance_factors, gpu, render_video, outfile): nframes = l_conds[-1].size(1) device = torch.device(gpu) batches = [] for i in range(len(models)): models[i].to(device) models[i].eval() batch = l_conds[i].to(device) if len(l_conds[i])>0 else [], g_conds[i].to(device) if len(g_conds[i])>0 else [], None batches.append(batch) with torch.no_grad(): clips = sample_mixmodels(models, batches, guidance_factors) models[-1].log_results(clips[:,trim:nframes-trim,:], outfile, "", logdir=dest_dir, render_video=render_video) def nans2zeros(x): ii = np.where(np.isinf(x)) x[ii]=0 ii = np.where(np.isnan(x)) x[ii]=0 return x def get_style_vector(styles_file, style_token, nbatch, nframes): all_styles = np.loadtxt(styles_file, dtype=str) styles_onehot = styles2onehot(all_styles, style_token) styles = styles_onehot.repeat(nbatch, nframes,1) def get_cond(model, data_dir, input_file, style_token, length): # Load input features with open(join(data_dir, input_file), 'rb') as f: ctrl = pkl.load(f) ctrl = ctrl[startframe:] if endframe>0 and endframe0: file_name = file_name + "_" + log_prefix bvh_data = self.feats_to_bvh(pred_clips) nclips = len(bvh_data) framerate = np.rint(1/bvh_data[0].framerate) if self.hparams.Validation["max_render_clips"]: nclips = min(nclips, self.hparams.Validation["max_render_clips"]) self.write_bvh(bvh_data[:nclips], log_dir=logdir, name_prefix=file_name) if render_video: pos_data = self.bvh_to_pos(bvh_data) if render_video: self.render_video(pos_data[:nclips], log_dir=logdir, name_prefix=file_name) def feats_to_bvh(self, pred_clips): #import pdb;pdb.set_trace() data_pipeline = jl.load(Path(self.hparams.dataset_root) / self.hparams.Data["datapipe_filename"]) n_feats = data_pipeline["cnt"].n_features data_pipeline["root"].separate_root=False print('inverse_transform...') bvh_data=data_pipeline.inverse_transform(pred_clips[:,:,:n_feats]) return bvh_data def write_bvh(self, bvh_data, log_dir="", name_prefix=""): writer = BVHWriter() nclips = len(bvh_data) for i in range(nclips): if nclips>1: fname = f"{log_dir}/{name_prefix}_{str(i).zfill(3)}.bvh" else: fname = f"{log_dir}/{name_prefix}.bvh" print('writing:' + fname) with open(fname,'w') as f: writer.write(bvh_data[i], f) def bvh_to_pos(self, bvh_data): # convert to joint positions return MocapParameterizer('position').fit_transform(bvh_data) def render_video(self, pos_data, log_dir="", name_prefix=""): # write bvh and skeleton motion nclips = len(pos_data) for i in range(nclips): if nclips>1: fname = f"{log_dir}/{name_prefix}_{str(i).zfill(3)}" else: fname = f"{log_dir}/{name_prefix}" print('writing:' + fname + ".mp4") render_mp4(pos_data[i], fname + ".mp4", axis_scale=200) def log_jerk(self, x, log_prefix): deriv = x[:, 1:] - x[:, :-1] acc = deriv[:, 1:] - deriv[:, :-1] jerk = acc[:, 1:] - acc[:, :-1] self.log(f'{log_prefix}_jerk', torch.mean(torch.abs(jerk)), sync_dist=True) ================================================ FILE: utils/motion_dataset.py ================================================ # Copyright 2023 Motorica AB, Inc. All Rights Reserved. import os import random import argparse import json import torch import torch.utils.data import sys from pathlib import Path import numpy as np from sklearn.preprocessing import StandardScaler import pickle as pkl import pandas as pd from scipy import interpolate def concat_dataframes(x,y): # Assume data is synched on the start time if x.shape[0]0) for i in range(len(inds)): assert len(tokens) > inds[i], f"{inds[i]} out of range in {basename}" out+=tokens[inds[i]] if i0: in_feats = in_feats[trim_edges:-trim_edges] out_feats = out_feats[trim_edges:-trim_edges] n_frames=in_feats.shape[0] # global conditioning (from file naming convention) if "styles_file" in data_hparams: styles_file = Path(data_root) / data_hparams["styles_file"] all_styles = np.loadtxt(styles_file, dtype=str).tolist() styles_oh = np.tile(styles2onehot(all_styles, parse_token(files[fi], data_hparams["style_index"])),(n_frames,1)) self.n_styles = len(all_styles) else: self.n_styles = 0 #we create indexes for full length sequences here seglen=max_segment_length if n_frames >= seglen: idx_array = torch.arange(start_idx, start_idx + n_frames).unfold( 0, seglen, 1 ) data["input"].append(in_feats) data["output"].append(out_feats) if self.n_styles>0: data["styles"].append(styles_oh) indexes.append(idx_array) start_idx += n_frames #flatten vertically and make into a torch tensor data["input"]=torch.from_numpy(np.vstack(data["input"])).float() data["output"]=torch.from_numpy(np.vstack(data["output"])).float() if self.n_styles>0: data["styles"]=torch.from_numpy(np.vstack(data["styles"])).float() print(f"=== tot number of frames: {data['output'].shape[0]} =====") self.data = data indexes=torch.cat(indexes, dim=0) self.indexes = indexes[torch.randperm(indexes.size(0))] def assert_not_const(self, data): eps = 1e-6 assert((data.std(axis=0)0: style_scaler = StandardScaler() style_scaler.mean_=np.zeros(self.n_styles) style_scaler.scale_=np.ones(self.n_styles) else: style_scaler=None return {"in_scaler": in_scaler, "style_scaler": style_scaler,"out_scaler": out_scaler} def standardize(self, scalers): self.data["input"] = torch.from_numpy(scalers["in_scaler"].transform(self.data["input"])).float() self.data["output"] = torch.from_numpy(scalers["out_scaler"].transform(self.data["output"])).float() def timestretch(self, data, segment_length, factor, has_root_motion=False): if factor<1.0: #Truncate original samples and stretch return resample_data(data[:int(factor*segment_length)],segment_length, has_root_motion) elif factor>1.0: #Stretch original samples and trunkate return resample_data(data,int(factor*segment_length),has_root_motion)[:segment_length] else: # return original return data[:segment_length] def __getitem__(self, index): in_feats = self.data["input"][self.indexes[index]] out_feats = self.data["output"][self.indexes[index]] if self.timestretch_factor>0: #note that the sequences are longer than specified so we can resample faster speeds if torch.rand((1,))0: styles = self.data["styles"][self.indexes[index]] styles = styles[:self.segment_length] else: styles = [] return (in_feats, styles, out_feats) def __len__(self): return self.indexes.size(0)