Repository: younggyoseo/FastTD3 Branch: main Commit: 7acc3a3c739d Files: 25 Total size: 738.3 KB Directory structure: gitextract_7_u6euyd/ ├── .gitignore ├── .pre-commit-config.yaml ├── LICENSE ├── README.md ├── data/ │ ├── humanoidbench_result.json │ ├── isaaclab_result.json │ └── playground_result.json ├── fast_td3/ │ ├── __init__.py │ ├── environments/ │ │ ├── humanoid_bench_env.py │ │ ├── isaaclab_env.py │ │ ├── mtbench_env.py │ │ └── mujoco_playground_env.py │ ├── fast_td3.py │ ├── fast_td3_deploy.py │ ├── fast_td3_simbav2.py │ ├── fast_td3_utils.py │ ├── hyperparams.py │ ├── train.py │ ├── train_multigpu.py │ └── training_notebook.ipynb ├── requirements/ │ ├── requirements.txt │ ├── requirements_isaacgym.txt │ └── requirements_playground.txt ├── setup.py └── sim2real.md ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ models wandb *.pyc .ipynb_checkpoints fast_td3.egg-info/ ================================================ FILE: .pre-commit-config.yaml ================================================ repos: - repo: https://github.com/psf/black rev: stable hooks: - id: black ================================================ FILE: LICENSE ================================================ This software is part of the BAIR Commons HIC Repository as of calendar year 2025. -------------------------------------------------------------------------------- MIT License Copyright (c) 2025 Younggyo Seo Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- MIT License Copyright (c) 2024 LeanRL developers Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- Code in `cleanrl/ddpg_continuous_action.py` and `cleanrl/td3_continuous_action.py` are adapted from https://github.com/sfujim/TD3 MIT License Copyright (c) 2020 Scott Fujimoto Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- Code in `cleanrl/sac_continuous_action.py` is inspired and adapted from [haarnoja/sac](https://github.com/haarnoja/sac), [openai/spinningup](https://github.com/openai/spinningup), [pranz24/pytorch-soft-actor-critic](https://github.com/pranz24/pytorch-soft-actor-critic), [DLR-RM/stable-baselines3](https://github.com/DLR-RM/stable-baselines3), and [denisyarats/pytorch_sac](https://github.com/denisyarats/pytorch_sac). - [haarnoja/sac](https://github.com/haarnoja/sac/blob/8258e33633c7e37833cc39315891e77adfbe14b2/LICENSE.txt) COPYRIGHT All contributions by the University of California: Copyright (c) 2017, 2018 The Regents of the University of California (Regents) All rights reserved. All other contributions: Copyright (c) 2017, 2018, the respective contributors All rights reserved. SAC uses a shared copyright model: each contributor holds copyright over their contributions to the SAC codebase. The project versioning records all such contribution and copyright details. If a contributor wants to further mark their specific copyright on a particular contribution, they should indicate their copyright solely in the commit message of the change when it is committed. LICENSE Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. CONTRIBUTION AGREEMENT By contributing to the SAC repository through pull-request, comment, or otherwise, the contributor releases their content to the license and copyright terms herein. - [openai/spinningup](https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/LICENSE) The MIT License Copyright (c) 2018 OpenAI (http://openai.com) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - [DLR-RM/stable-baselines3](https://github.com/DLR-RM/stable-baselines3/blob/44e53ff8115e8f4bff1d5218f10c8c7d1a4cfc12/LICENSE) The MIT License Copyright (c) 2019 Antonin Raffin Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - [denisyarats/pytorch_sac](https://github.com/denisyarats/pytorch_sac/blob/81c5b536d3a1c5616b2531e446450df412a064fb/LICENSE) MIT License Copyright (c) 2019 Denis Yarats Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - [pranz24/pytorch-soft-actor-critic](https://github.com/pranz24/pytorch-soft-actor-critic/blob/master/LICENSE) MIT License Copyright (c) 2018 Pranjal Tandon Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --------------------------------------------------------------------------------- The CONTRIBUTING.md is adopted from https://github.com/entity-neural-network/incubator/blob/2a0c38b30828df78c47b0318c76a4905020618dd/CONTRIBUTING.md and https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md MIT License Copyright (c) 2021 Entity Neural Network developers Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. MIT License Copyright (c) 2020 Stable-Baselines Team Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --------------------------------------------------------------------------------- The cleanrl/ppo_continuous_action_isaacgym.py is contributed by Nvidia SPDX-FileCopyrightText: Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: MIT Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- Code in `cleanrl/qdagger_dqn_atari_impalacnn.py` and `cleanrl/qdagger_dqn_atari_jax_impalacnn.py` are adapted from https://github.com/google-research/reincarnating_rl **NOTE: the original repo did not fill out the copyright section in their license so the following copyright notice is copied as is per the license requirement. See https://github.com/google-research/reincarnating_rl/blob/a1d402f48a9f8658ca6aa0ddf416ab391745ff2c/LICENSE#L189 Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: README.md ================================================ # FastTD3 - Simple and Fast RL for Humanoid Control [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![arXiv](https://img.shields.io/badge/arXiv-2505.22642-b31b1b.svg)](https://arxiv.org/abs/2505.22642) FastTD3 is a high-performance variant of the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, optimized for complex humanoid control tasks. FastTD3 can solve various humanoid control tasks with dexterous hands from HumanoidBench in just a few hours of training. Furthermore, FastTD3 achieves similar or better wall-time-efficiency to PPO in high-dimensional control tasks from popular simulations such as IsaacLab and MuJoCo Playground. For more information, please see our [project webpage](https://younggyo.me/fast_td3) ## ❗ Updates - **[Sep/20/2025]** Added `data` directory that contains training logs used for plotting in the report. - **[Sep/17/2025]** Fixed an issue where `std_min` and `std_max` were not included in Actor config (credit: [@ningyuanz](https://github.com/ningyuanz)). - **[Aug/07/2025]** If you encounter an issue in reproducing the performance, try disabling `torch.compile`. Please use `--no_compile` in argument. - **[Jul/07/2025]** Added support for multi-GPU training! See [Multi-GPU Training](#multi-gpu-training) section for details. - **[Jul/02/2025]** Optimized codebase to speed up training around 10-30% when using a single RTX 4090 GPU. - **[Jun/20/2025]** Added support for [MTBench](https://github.com/Viraj-Joshi/MTBench) with the help of [Viraj Joshi](https://viraj-joshi.github.io/). - **[Jun/15/2025]** Added support for FastTD3 + [SimbaV2](https://dojeon-ai.github.io/SimbaV2/)! It's faster to train, and often achieves better asymptotic performance. We recommend using FastTD3 + SimbaV2 for most cases. - **[Jun/06/2025]** Thanks to [Antonin Raffin](https://araffin.github.io/) ([@araffin](https://github.com/araffin)), we fixed the issues when using `n_steps` > 1, which stabilizes training with n-step return quite a lot! - **[Jun/01/2025]** Updated the figures in the technical report to report deterministic evaluation for IsaacLab tasks. ## ✨ Features FastTD3 offers researchers a significant speedup in training complex humanoid agents. - Ready-to-go codebase with detailed instructions and pre-configured hyperparameters for each task - Support popular benchmarks: HumanoidBench, MuJoCo Playground, and IsaacLab - User-friendly features that can accelerate your research, such as rendering rollouts, torch optimizations (AMP and compile), and saving and loading checkpoints ## ⚙️ Prerequisites Before you begin, ensure you have the following installed: - Conda (for environment management) - Git LFS (Large File Storage) -- For IsaacLab - CMake -- For IsaacLab And the following system packages: ```bash sudo apt install libglfw3 libgl1-mesa-glx libosmesa6 git-lfs cmake ``` ## 📖 Installation This project requires different Conda environments for different sets of experiments. ### Common Setup First, ensure the common dependencies are installed as mentioned in the [Prerequisites](#prerequisites) section. ### Environment for HumanoidBench ```bash conda create -n fasttd3_hb -y python=3.10 conda activate fasttd3_hb pip install --editable git+https://github.com/carlosferrazza/humanoid-bench.git#egg=humanoid-bench pip install -r requirements/requirements.txt ``` ### Environment for MuJoCo Playground ```bash conda create -n fasttd3_playground -y python=3.10 conda activate fasttd3_playground pip install -r requirements/requirements_playground.txt ``` **⚠️ Note:** Our `requirements_playground.txt` specifies `Jax==0.4.35`, which we found to be stable for latest GPUs in certain tasks such as `LeapCubeReorient` or `LeapCubeRotateZAxis` **⚠️ Note:** Current FastTD3 codebase uses customized MuJoCo Playground that supports saving last observations into info dictionary. We will work on incorporating this change into official repository hopefully soon. ### Environment for IsaacLab ```bash conda create -n fasttd3_isaaclab -y python=3.10 conda activate fasttd3_isaaclab # Install IsaacLab (refer to official documentation for the latest steps) # Official Quickstart: https://isaac-sim.github.io/IsaacLab/main/source/setup/quickstart.html pip install 'isaacsim[all,extscache]==4.5.0' --extra-index-url https://pypi.nvidia.com git clone https://github.com/isaac-sim/IsaacLab.git cd IsaacLab ./isaaclab.sh --install cd .. # Install project-specific requirements pip install -r requirements/requirements.txt ``` ### Environment for MTBench MTBench does not support humanoid experiments, but is a useful multi-task benchmark with massive parallel simulation. This could be useful for users who want to use FastTD3 for their multi-task experiments. ```bash conda create -n fasttd3_mtbench -y python=3.8 # Note python version conda activate fasttd3_mtbench # Install IsaacGym -- recommend to follow instructions in https://github.com/BoosterRobotics/booster_gym ... # Install MTBench git clone https://github.com/Viraj-Joshi/MTBench.git cd MTbench pip install -e . pip install skrl cd .. # Install project-specific requirements pip install -r requirements/requirements_isaacgym.txt ``` ### (Optional) Accelerate headless GPU rendering in cloud instances In some cloud VM images the NVIDIA kernel driver is present but the user-space OpenGL/EGL/Vulkan libraries aren't, so MuJoCo falls back to CPU renderer. You can install just the NVIDIA user-space libraries (and skip rebuilding the kernel module) with: ```bash sudo apt install -y kmod sudo sh NVIDIA-Linux-x86_64-.run -s --no-kernel-module --ui=none --no-questions ``` As a rule-of-thumb, if you're running experiments and rendering is taking longer than 5 seconds, it is very likely that GPU renderer is not used. ## 🚀 Running Experiments Activate the appropriate Conda environment before running experiments. Please see `fast_td3/hyperparams.py` for information regarding hyperparameters! ### HumanoidBench Experiments ```bash conda activate fasttd3_hb # FastTD3 python fast_td3/train.py \ --env_name h1hand-hurdle-v0 \ --exp_name FastTD3 \ --render_interval 5000 \ --seed 1 # FastTD3 + SimbaV2 python fast_td3/train.py \ --env_name h1hand-hurdle-v0 \ --exp_name FastTD3 \ --render_interval 5000 \ --agent fasttd3_simbav2 \ --batch_size 8192 \ --critic_learning_rate_end 3e-5 \ --actor_learning_rate_end 3e-5 \ --weight_decay 0.0 \ --critic_hidden_dim 512 \ --critic_num_blocks 2 \ --actor_hidden_dim 256 \ --actor_num_blocks 1 \ --seed 1 ``` ### MuJoCo Playground Experiments ```bash conda activate fasttd3_playground # FastTD3 python fast_td3/train.py \ --env_name T1JoystickFlatTerrain \ --exp_name FastTD3 \ --render_interval 5000 \ --seed 1 # FastTD3 + SimbaV2 python fast_td3/train.py \ --env_name T1JoystickFlatTerrain \ --exp_name FastTD3 \ --render_interval 5000 \ --agent fasttd3_simbav2 \ --batch_size 8192 \ --critic_learning_rate_end 3e-5 \ --actor_learning_rate_end 3e-5 \ --weight_decay 0.0 \ --critic_hidden_dim 512 \ --critic_num_blocks 2 \ --actor_hidden_dim 256 \ --actor_num_blocks 1 \ --seed 1 ``` ### IsaacLab Experiments ```bash conda activate fasttd3_isaaclab # FastTD3 python fast_td3/train.py \ --env_name Isaac-Velocity-Flat-G1-v0 \ --exp_name FastTD3 \ --render_interval 0 \ --seed 1 # FastTD3 + SimbaV2 python fast_td3/train.py \ --env_name Isaac-Repose-Cube-Allegro-Direct-v0 \ --exp_name FastTD3 \ --render_interval 0 \ --agent fasttd3_simbav2 \ --batch_size 8192 \ --critic_learning_rate_end 3e-5 \ --actor_learning_rate_end 3e-5 \ --weight_decay 0.0 \ --critic_hidden_dim 512 \ --critic_num_blocks 2 \ --actor_hidden_dim 256 \ --actor_num_blocks 1 \ --seed 1 ``` ### MTBench Experiments ```bash conda activate fasttd3_mtbench # FastTD3 python fast_td3/train.py \ --env_name MTBench-meta-world-v2-mt10 \ --exp_name FastTD3 \ --render_interval 0 \ --seed 1 # FastTD3 + SimbaV2 python fast_td3/train.py \ --env_name MTBench-meta-world-v2-mt10 \ --exp_name FastTD3 \ --render_interval 0 \ --agent fasttd3_simbav2 \ --batch_size 8192 \ --critic_learning_rate_end 3e-5 \ --actor_learning_rate_end 3e-5 \ --weight_decay 0.0 \ --critic_hidden_dim 1024 \ --critic_num_blocks 2 \ --actor_hidden_dim 512 \ --actor_num_blocks 1 \ --seed 1 ``` **Quick note:** For boolean-based arguments, you can set them to False by adding `no_` in front each argument, for instance, if you want to disable Clipped Q Learning, you can specify `--no_use_cdq` in your command. ## 💡 Performance-Related Tips We used a single Nvidia A100 80GB GPU for all experiments. Here are some remarks and tips for improving performances in your setup or troubleshooting in your machine configurations. - *Sample-efficiency* tends to improve with larger `num_envs`, `num_updates`, and `batch_size`. But this comes at the cost of *Time-efficiency*. Our default settings are optimized for wall-time efficiency on a single A100 80GB GPU. If you're using a different setup, consider tuning hyperparameters accordingly. - When FastTD3 performance is stuck at local minima at the early phase of training in your experiments - First consider increasing the `num_updates`. This happens usually when the agent fails to exploit value functions. We also find higher `num_updates` tends to be helpful for relatively easier tasks or tasks with low-dimensional action spaces. - If the agent is completely stuck or much worse than your expectation, try using `num_steps=3` or disabling `use_cdq`. - For tasks that have penalty reward terms (e.g., torques, energy, action_rate, ..), consider lowering them for initial experiments, and tune the values. In some cases, curriculum learning with lower penalty terms followed by fine-tuning with stronger terms is effective. - When you encounter out-of-memory error with your GPU, our recommendation for reducing GPU usage is (i) smaller `buffer_size`, (ii) smaller `batch_size`, and then (iii) smaller `num_envs`. Because our codebase is assigning the whole replay buffer in GPU to reduce CPU-GPU transfer bottleneck, it usually has the largest GPU consumption, but usually less harmful to reduce. - Consider using `--compile_mode max-autotune` if you plan to run for many training steps. This may speed up training by up to 10% at the cost of a few additional minutes of heavy compilation. ## Multi-GPU Training We support multi-GPU training. If your machine supports multiple GPUs, or specify multiple GPUs using `CUDA_VISIBLE_DEVICES`, and run `train_multigpu.py`, it will automatically use all GPUs to scale up training. **Important:** Our multi-GPU implementation launches the **same experiment independently on each GPU** rather than distributing parameters across GPUs. This means: - Effective number of environments: `num_envs × num_gpus` - Effective batch size: `batch_size × num_gpus` - Effective buffer size: `buffer_size × num_gpus` Each GPU runs a complete copy of the training process, which scales up data collection and training throughput proportionally to the number of GPUs. For instance, running IsaacLab experiments with 4 GPUs and `num_envs=1024` will end up in similar results as experiments with 1 GPU with `num_envs=4096`. ## 🛝 Playing with the FastTD3 training A Jupyter notebook (`training_notebook.ipynb`) is available to help you get started with: - Training FastTD3 agents. - Loading pre-trained models. - Visualizing agent behavior. - Potentially, re-training or fine-tuning models. ## 🤖 Sim-to-Real RL with FastTD3 We provide the [walkthrough](sim2real.md) for training deployable policies with FastTD3. ## Contributing We welcome contributions! Please feel free to submit issues and pull requests. ## License This project is licensed under the MIT License -- see the [LICENSE](LICENSE) file for details. Note that the repository relies on third-party libraries subject to their respective licenses. ## Acknowledgements This codebase builds upon [LeanRL](https://github.com/pytorch-labs/LeanRL) framework. We would like to thank people who have helped throughout the project: - We thank [Kevin Zakka](https://kzakka.com/) for the help in setting up MuJoCo Playground. - We thank [Changyeon Kim](https://changyeon.site/) for testing the early version of this codebase ## Citations ### FastTD3 ```bibtex @article{seo2025fasttd3, title={FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control}, author={Seo, Younggyo and Sferrazza, Carmelo and Geng, Haoran and Nauman, Michal and Yin, Zhao-Heng and Abbeel, Pieter}, journal={arXiv preprint arXiv:2505.22642}, year={2025} } ``` ### TD3 ```bibtex @inproceedings{fujimoto2018addressing, title={Addressing function approximation error in actor-critic methods}, author={Fujimoto, Scott and Hoof, Herke and Meger, David}, booktitle={International conference on machine learning}, pages={1587--1596}, year={2018}, organization={PMLR} } ``` ### SimbaV2 ```bibtex @article{lee2025hyperspherical, title={Hyperspherical normalization for scalable deep reinforcement learning}, author={Lee, Hojoon and Lee, Youngdo and Seno, Takuma and Kim, Donghu and Stone, Peter and Choo, Jaegul}, journal={arXiv preprint arXiv:2502.15280}, year={2025} } ``` ### LeanRL Following the [LeanRL](https://github.com/pytorch-labs/LeanRL)'s recommendation, we put CleanRL's bibtex here: ```bibtex @article{huang2022cleanrl, author = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo}, title = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms}, journal = {Journal of Machine Learning Research}, year = {2022}, volume = {23}, number = {274}, pages = {1--18}, url = {http://jmlr.org/papers/v23/21-1342.html} } ``` ### Parallel Q-Learning (PQL) ```bibtex @inproceedings{li2023parallel, title={Parallel $ Q $-Learning: Scaling Off-policy Reinforcement Learning under Massively Parallel Simulation}, author={Li, Zechu and Chen, Tao and Hong, Zhang-Wei and Ajay, Anurag and Agrawal, Pulkit}, booktitle={International Conference on Machine Learning}, pages={19440--19459}, year={2023}, organization={PMLR} } ``` ### HumanoidBench ```bibtex @inproceedings{sferrazza2024humanoidbench, title={Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation}, author={Sferrazza, Carmelo and Huang, Dun-Ming and Lin, Xingyu and Lee, Youngwoon and Abbeel, Pieter}, booktitle={Robotics: Science and Systems}, year={2024} } ``` ### MuJoCo Playground ```bibtex @article{zakka2025mujoco, title={MuJoCo Playground}, author={Zakka, Kevin and Tabanpour, Baruch and Liao, Qiayuan and Haiderbhai, Mustafa and Holt, Samuel and Luo, Jing Yuan and Allshire, Arthur and Frey, Erik and Sreenath, Koushil and Kahrs, Lueder A and others}, journal={arXiv preprint arXiv:2502.08844}, year={2025} } ``` ### IsaacLab ```bibtex @article{mittal2023orbit, author={Mittal, Mayank and Yu, Calvin and Yu, Qinxi and Liu, Jingzhou and Rudin, Nikita and Hoeller, David and Yuan, Jia Lin and Singh, Ritvik and Guo, Yunrong and Mazhar, Hammad and Mandlekar, Ajay and Babich, Buck and State, Gavriel and Hutter, Marco and Garg, Animesh}, journal={IEEE Robotics and Automation Letters}, title={Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments}, year={2023}, volume={8}, number={6}, pages={3740-3747}, doi={10.1109/LRA.2023.3270034} } ``` ### MTBench ```bibtex @inproceedings{ joshi2025benchmarking, title={Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks}, author={Viraj Joshi and Zifan Xu and Bo Liu and Peter Stone and Amy Zhang}, booktitle={Reinforcement Learning Conference}, year={2025}, url={https://openreview.net/forum?id=z0MM0y20I2} } ``` ### Getting SAC to Work on a Massive Parallel Simulator ```bibtex @article{raffin2025isaacsim, title = "Getting SAC to Work on a Massive Parallel Simulator: An RL Journey With Off-Policy Algorithms", author = "Raffin, Antonin", journal = "araffin.github.io", year = "2025", month = "Feb", url = "https://araffin.github.io/post/sac-massive-sim/" } ``` ### Speeding Up SAC with Massively Parallel Simulation ```bibtex @article{shukla2025fastsac, title = "Speeding Up SAC with Massively Parallel Simulation", author = "Shukla, Arth", journal = "https://arthshukla.substack.com", year = "2025", month = "Mar", url = "https://arthshukla.substack.com/p/speeding-up-sac-with-massively-parallel" } ``` ================================================ FILE: data/humanoidbench_result.json ================================================ { "h1hand_walk": { "time": [ 0.0, 327.4388405844, 654.8776811688, 982.3165217531999, 1309.7553623376, 1637.194202922, 1964.6330435063999, 2292.0718840908, 2619.5107246752, 2946.9495652596, 3274.388405844 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0 ], "return": [ 0.0, 18.961605707804363, 21.807273546854656, 48.724927266438804, 94.65358352661133, 178.67118326822916, 501.64170328776044, 711.7367757161459, 901.3774210611979, 928.8246256510416, 928.2600301106771 ], "return_min": [ 0.0, 14.858372081304717, 12.96401166123347, 41.67681180605566, 54.95595528974483, 94.37760201041398, 279.43906161791847, 545.4619319542776, 877.1935439686172, 921.6425265540179, 906.3836415708065 ], "return_max": [ 0.0, 23.06483933430401, 30.65053543247584, 55.77304272682195, 134.35121176347783, 262.96476452604435, 723.8443449576024, 878.0116194780142, 925.5612981537786, 936.0067247480654, 950.1364186505477 ] }, "h1hand_stand": { "time": [ 0.0, 325.5565317698, 651.1130635396, 976.6695953094, 1302.2261270792, 1627.782658849, 1953.3391906188, 2278.8957223886, 2604.4522541584, 2930.0087859282003, 3255.565317698 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0 ], "return": [ 0.0, 31.879506429036457, 42.487387339274086, 53.94635899861654, 71.63724517822266, 109.86285400390625, 255.9549357096354, 628.9070638020834, 776.9771118164062, 887.54248046875, 911.9251912434896 ], "return_min": [ 0.0, 21.445549292274528, 22.163247088006734, 45.241289229102975, 64.28271961875848, 76.26589758620838, 191.1954024340911, 393.33148063655887, 656.7962055436002, 851.7788577992328, 880.8326840828662 ], "return_max": [ 0.0, 42.313463565798386, 62.81152759054144, 62.6514287681301, 78.99177073768683, 143.45981042160412, 320.7144689851797, 864.4826469676079, 897.1580180892123, 923.3061031382672, 943.017698404113 ] }, "h1hand_run": { "time": [ 0.0, 306.4421410813333, 612.8842821626666, 919.326423244, 1225.7685643253333, 1532.2107054066666, 1838.652846488, 2145.0949875693336, 2451.5371286506665, 2757.9792697320004, 3064.4214108133333, 3370.8635518946667, 3677.305692976, 3983.747834057334, 4290.189975138667, 4596.63211622 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0 ], "return": [ 0.0, 6.890847841898601, 10.521868069966635, 9.320551554361979, 17.587395985921223, 31.98662503560384, 46.11953608194987, 74.0572992960612, 121.98380025227864, 210.85086059570312, 506.30312093098956, 524.246826171875, 775.0013631184896, 811.9727172851562, 886.3514200846354, 902.0192260742188 ], "return_min": [ 0.0, 5.2453829775246215, 9.646580055146266, 6.430592906343505, 12.881361282168676, 26.82687314527471, 41.16444668808793, 62.03005710131488, 100.789247637638, 183.66015405980465, 427.02318928000045, 408.7644706774459, 737.4298733834557, 774.6940229887994, 879.065567239731, 892.5835266487527 ], "return_max": [ 0.0, 8.536312706272579, 11.397156084787003, 12.210510202380451, 22.29343068967377, 37.14637692593297, 51.07462547581181, 86.08454149080752, 143.17835286691928, 238.0415671316016, 585.5830525819787, 639.729181666304, 812.5728528535235, 849.2514115815131, 893.6372729295398, 911.4549254996848 ] }, "h1hand_reach": { "time": [ 0.0, 372.80056838400003, 745.6011367680001, 1118.401705152, 1491.2022735360001, 1864.00284192, 2236.803410304, 2609.603978688, 2982.4045470720002, 3355.2051154560004, 3728.00568384, 4100.806252224001, 4473.606820608, 4846.407388992, 5219.207957376, 5592.00852576, 5964.8090941440005, 6337.609662528001, 6710.410230912001, 7083.210799296, 7456.01136768, 7828.811936064, 8201.612504448001, 8574.413072832, 8947.213641216, 9320.0142096, 9692.814777984, 10065.615346368, 10438.415914752, 10811.216483136, 11184.01705152 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0 ], "return": [ 0.0, 941.600341796875, 1135.4913330078125, 1474.2825520833333, 2313.754109700521, 3213.2079264322915, 4587.813883463542, 5064.874186197917, 5511.294270833333, 5576.633138020833, 6172.243489583333, 7008.345377604167, 6894.714680989583, 7536.7392578125, 7494.722819010417, 7460.627278645833, 7870.114095052083, 8164.121256510417, 8070.581217447917, 8084.484700520833, 8364.873697916666, 8254.95654296875, 8171.631673177083, 8403.752604166666, 8585.269368489584, 8388.333170572916, 8449.694010416666, 8608.694986979166, 8495.815755208334, 8748.194661458334, 8146.561197916667 ], "return_min": [ 0.0, 681.8662271797355, 1021.0384348384862, 1397.7603324284514, 1976.0678889906676, 2504.978510893123, 3452.5915196733913, 4091.507234596569, 4614.513790451145, 4584.797954406994, 5541.806414474982, 6259.515364983769, 6343.318836978547, 6854.539919974239, 6980.611797856333, 6705.693438934862, 7218.966063142879, 7807.798913658527, 7519.692863858716, 7372.049219729361, 7880.456149452872, 7570.937682802555, 7367.755272289276, 7989.806520979583, 8098.439330032555, 7945.363999112206, 8019.104933439191, 8156.065950744518, 7987.549893436531, 8366.520573819495, 7681.10435210547 ], "return_max": [ 0.0, 1201.3344564140145, 1249.9442311771388, 1550.804771738215, 2651.4403304103744, 3921.4373419714602, 5723.036247253693, 6038.241137799265, 6408.074751215521, 6568.468321634672, 6802.680564691684, 7757.175390224565, 7446.110525000619, 8218.938595650761, 8008.8338401645005, 8215.561118356805, 8521.262126961288, 8520.443599362306, 8621.469571037118, 8796.920181312305, 8849.29124638046, 8938.975403134946, 8975.50807406489, 8817.69868735375, 9072.099406946612, 8831.302342033627, 8880.283087394142, 9061.324023213814, 9004.081616980136, 9129.868749097173, 8612.018043727863 ] }, "h1hand_hurdle": { "time": [ 0.0, 361.5054530611936, 723.0109061223872, 1084.5163591835806, 1446.0218122447743, 1807.5272653059678, 2169.032718367161, 2530.538171428355, 2892.0436244895486, 3253.549077550742, 3615.0545306119357, 3976.559983673129, 4338.065436734322, 4699.570889795516, 5061.07634285671, 5422.581795917904, 5784.087248979097, 6145.5927020402905, 6507.098155101484, 6868.603608162677, 7230.109061223871, 7591.614514285065, 7953.119967346258, 8314.625420407452, 8676.130873468644, 9037.636326529839, 9399.141779591033, 9760.647232652227, 10122.15268571342, 10483.658138774614, 10845.163591835808, 11206.669044897 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0, 19840000.0 ], "return": [ 0.0, 6.492611567179362, 10.588607947031656, 18.610817273457844, 29.661352157592773, 49.00843048095703, 85.58958435058594, 115.55878194173177, 173.2019500732422, 218.64263916015625, 319.6346893310547, 377.22975667317706, 496.73987833658856, 490.29189046223956, 610.6637573242188, 658.8320719401041, 689.0982869466146, 710.2075602213541, 735.3473714192709, 778.7053426106771, 762.7415568033854, 804.9999593098959, 830.8478800455729, 852.31982421875, 861.3894653320312, 814.1538696289062, 871.5404663085938, 876.4856160481771, 852.7265625, 889.6229248046875, 887.1525065104166, 886.8323160807291 ], "return_min": [ 0.0, 6.225973005334275, 8.412515126451241, 15.738143769806989, 21.76995365926382, 31.328784883872007, 56.83435619834776, 79.92347525493489, 112.07297356902194, 138.09470906365925, 216.08342306974944, 259.30040536744895, 389.9528713112492, 378.8689856869445, 480.1631532334461, 565.0675789124052, 598.9966247159923, 618.670444534875, 659.2942580856163, 729.4083980816131, 708.118056513654, 737.1563442724927, 785.5860491473233, 822.1971602106087, 853.2213237200906, 798.9233122130538, 848.562144860611, 867.3290162790252, 808.398990047507, 877.0273627520126, 880.6210154175322, 878.8996722928737 ], "return_max": [ 0.0, 6.7592501290244495, 12.764700767612071, 21.4834907771087, 37.55275065592173, 66.68807607804206, 114.34481250282411, 151.19408862852865, 234.33092657746244, 299.19056925665325, 423.18595559235996, 495.1591079789052, 603.526885361928, 601.7147952375346, 741.1643614149914, 752.596564967803, 779.1999491772369, 801.7446759078333, 811.4004847529254, 828.0022871397412, 817.3650570931168, 872.8435743472991, 876.1097109438225, 882.4424882268913, 869.5576069439719, 829.3844270447587, 894.5187877565764, 885.6422158173291, 897.054134952493, 902.2184868573624, 893.683997603301, 894.7649598685846 ] }, "h1hand_crawl": { "time": [ 0.0, 351.0455393933077, 702.0910787866154, 1053.136618179923, 1404.1821575732308, 1755.2276969665386, 2106.273236359846, 2457.318775753154, 2808.3643151464616, 3159.4098545397696, 3510.455393933077, 3861.5009333263847, 4212.546472719692, 4563.592012113 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0 ], "return": [ 0.0, 352.52854410807294, 560.5812377929688, 552.0515441894531, 642.7318115234375, 714.5783081054688, 806.6729939778646, 860.4519856770834, 892.0007731119791, 920.3323160807291, 924.2599487304688, 945.3890991210938, 948.0378011067709, 956.2367553710938 ], "return_min": [ 0.0, 324.2043638986621, 545.5137620982706, 483.4934811974669, 635.1979678248894, 683.6522397156278, 727.5537564989331, 784.8352581603768, 839.2658061728478, 887.326843488466, 905.17119209457, 927.7121326375325, 932.2496491343232, 945.4639700799313 ], "return_max": [ 0.0, 380.85272431748376, 575.6487134876669, 620.6096071814393, 650.2656552219856, 745.5043764953097, 885.7922314567961, 936.0687131937899, 944.7357400511105, 953.3377886729922, 943.3487053663675, 963.066065604655, 963.8259530792185, 967.0095406622562 ] }, "h1hand_maze": { "time": [ 0.0, 373.7336517353571, 747.4673034707142, 1121.2009552060715, 1494.9346069414285, 1868.6682586767856, 2242.401910412143, 2616.1355621475, 2989.869213882857, 3363.6028656182143, 3737.3365173535713, 4111.070169088928, 4484.803820824286, 4858.537472559643, 5232.271124295 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0 ], "return": [ 0.0, 140.10218302408853, 145.8614705403646, 147.9732869466146, 182.3691864013672, 233.79485066731772, 329.2705078125, 352.1261393229167, 357.1931559244792, 357.9311014811198, 298.9239857991536, 358.42406209309894, 350.0440165201823, 353.48012288411456, 358.6817118326823 ], "return_min": [ 0.0, 124.88956930471686, 135.93298616192584, 142.26980814418167, 163.01049749771548, 169.52534951133262, 312.57792152579833, 341.7712339158121, 350.91098197823186, 354.19958286243155, 211.5847442406885, 346.7498400009268, 342.2860426951425, 344.2272969649359, 354.4987809321401 ], "return_max": [ 0.0, 155.31479674346022, 155.78995491880335, 153.67676574904752, 201.7278753050189, 298.0643518233028, 345.96309409920167, 362.48104473002127, 363.4753298707265, 361.66262009980807, 386.26322735761875, 370.0982841852711, 357.8019903452221, 362.73294880329325, 362.8646427332245 ] }, "h1hand_sit_simple": { "time": [ 0.0, 352.13274472366663, 704.2654894473333, 1056.398234171, 1408.5309788946665, 1760.6637236183333, 2112.796468342, 2464.9292130656668, 2817.061957789333, 3169.194702513 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0 ], "return": [ 0.0, 139.5462900797526, 653.7905069986979, 863.320068359375, 937.2736612955729, 917.0751342773438, 925.4646809895834, 934.3729451497396, 931.1214192708334, 929.0094401041666 ], "return_min": [ 0.0, 39.1384600509537, 561.3652721120882, 840.7680414731517, 928.7285812113073, 882.5101152312143, 910.9714846547159, 910.6220791649206, 912.940393490759, 909.0497473763742 ], "return_max": [ 0.0, 239.95412010855148, 746.2157418853076, 885.8720952455983, 945.8187413798385, 951.6401533234732, 939.9578773244508, 958.1238111345587, 949.3024450509077, 948.9691328319591 ] }, "h1hand_sit_hard": { "time": [ 0.0, 355.06120608526663, 710.1224121705333, 1065.1836182558, 1420.2448243410665, 1775.3060304263333, 2130.3672365116, 2485.4284425968663, 2840.489648682133, 3195.5508547674003, 3550.6120608526667, 3905.673266937933, 4260.7344730232, 4615.795679108467, 4970.8568851937325, 5325.918091279, 5680.979297364266, 6036.040503449533, 6391.101709534801, 6746.162915620066, 7101.224121705333, 7456.2853277906, 7811.346533875866, 8166.407739961132, 8521.4689460464, 8876.530152131665, 9231.591358216934, 9586.6525643022, 9941.713770387465, 10296.774976472732, 10651.836182558 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0 ], "return": [ 0.0, 95.34047635396321, 195.36861419677734, 318.97853597005206, 396.64198811848956, 375.91650390625, 488.83921305338544, 548.6920267740885, 574.7945251464844, 566.6195780436198, 583.4102783203125, 590.7933756510416, 600.908457438151, 594.9339090983073, 610.3661804199219, 616.0959269205729, 610.0286865234375, 601.2849934895834, 611.8743286132812, 634.0820617675781, 632.5235493977865, 669.2791239420573, 701.1968180338541, 683.4253133138021, 706.3424275716146, 741.2285461425781, 699.800537109375, 749.2343546549479, 752.3892517089844, 734.4484252929688, 726.0768636067709 ], "return_min": [ 0.0, 2.5724704693975298, 93.58268813745721, 233.71247327846993, 368.7017851150339, 332.2081460365896, 340.63550191707344, 329.6445384869926, 351.03336679701863, 343.03801256893826, 363.99036303185073, 360.8858716647696, 369.6746990033322, 375.6881012599298, 384.3501951686797, 391.8689483225843, 385.04690037041826, 394.37185439974064, 381.77529344069296, 411.24672410522163, 405.85462333371214, 459.9114779257408, 504.43995094660033, 483.4763839716003, 499.2238858422869, 538.4201845543151, 470.58627615990247, 533.4576585016802, 541.3393913458676, 524.7727664466893, 470.0900042127373 ], "return_max": [ 0.0, 188.1084822385289, 297.15454025609745, 404.2445986616342, 424.58219112194524, 419.6248617759104, 637.0429241896975, 767.7395150611844, 798.5556834959501, 790.2011435183013, 802.8301936087743, 820.7008796373136, 832.1422158729698, 814.1797169366847, 836.3821656711641, 840.3229055185615, 835.0104726764567, 808.1981325794261, 841.9733637858695, 856.9173994299347, 859.1924754618608, 878.6467699583736, 897.9536851211079, 883.374242656004, 913.4609693009423, 944.0369077308411, 929.0147980588475, 965.0110508082156, 963.4391120721011, 944.1240841392482, 982.0637230008044 ] }, "h1hand_balance_simple": { "time": [ 0.0, 175.717594372775, 351.43518874555, 527.152783118325, 702.8703774911, 878.587971863875, 1054.30556623665, 1230.023160609425, 1405.7407549822, 1581.458349354975, 1757.17594372775, 1932.8935381005253, 2108.6111324733, 2284.328726846075, 2460.04632121885, 2635.763915591625, 2811.4815099644, 2987.1991043371754, 3162.91669870995, 3338.6342930827254, 3514.3518874555, 3690.0694818282755, 3865.7870762010507, 4041.5046705738255, 4217.2222649466, 4392.939859319375, 4568.65745369215, 4744.375048064925, 4920.0926424377, 5095.8102368104755, 5271.52783118325, 5447.245425556025, 5622.9630199288, 5798.680614301576, 5974.398208674351, 6150.115803047125, 6325.8333974199, 6501.550991792676, 6677.268586165451, 6852.986180538226, 7028.703774911 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0, 19840000.0, 20480000.0, 21120000.0, 21760000.0, 22400000.0, 23040000.0, 23680000.0, 24320000.0, 24960000.0, 25600000.0 ], "return": [ 0.0, 52.67643197377523, 77.08096504211426, 100.20963668823242, 118.43112309773763, 157.09767150878906, 208.31290181477866, 215.8346757888794, 295.65764872233075, 373.72972615559894, 395.9920959472656, 326.1277338663737, 487.8640950520833, 412.4946568806966, 481.59149678548175, 489.4783935546875, 555.3950398763021, 549.860850016276, 587.4115397135416, 623.5388387044271, 538.5026601155599, 437.4591878255208, 563.7863718668619, 591.0351969401041, 635.8733622233073, 531.7674204508463, 557.8404591878256, 567.099183400472, 504.17720794677734, 611.8459065755209, 568.1315002441406, 531.0865529378256, 719.8113403320312, 617.6780344645182, 661.9119974772135, 701.4621276855469, 612.2174580891927, 698.0676879882812, 607.3912162780762, 701.1986897786459, 788.4394226074219 ], "return_min": [ 0.0, 34.03153521489918, 67.34714304892007, 93.43179041853405, 108.3451803854909, 131.60774027010135, 186.807812935322, 121.50917666317413, 243.0843846608089, 314.6589689026109, 335.93710664802484, 199.6418475452607, 421.46378419781684, 271.96722431041565, 419.1856732133755, 384.7529814203755, 496.47070280779275, 468.9030197426357, 520.0155579034114, 555.5749497533352, 406.5754864025145, 275.34879196835936, 498.0399353927207, 550.1356761025373, 516.3616959889615, 320.4064699483651, 442.36312091556783, 326.04143673322324, 303.6744638373668, 485.79766687848047, 303.55332555045385, 466.7197913683005, 625.171113211483, 395.48405865968545, 542.6871855684739, 615.614612025825, 499.65604515073494, 548.4941266431731, 350.1081528445121, 573.6862052997205, 653.9984031207549 ], "return_max": [ 0.0, 71.32132873265128, 86.81478703530844, 106.9874829579308, 128.51706580998433, 182.58760274747678, 229.8179906942353, 310.1601749145847, 348.2309127838526, 432.80048340858696, 456.0470852465064, 452.61362018748673, 554.2644059063498, 553.0220894509775, 543.997320357588, 594.2038056889995, 614.3193769448116, 630.8186802899163, 654.8075215236719, 691.502727655519, 670.4298338286053, 599.5695836826823, 629.5328083410031, 631.9347177776709, 755.385028457653, 743.1283709533275, 673.3177974600833, 808.1569300677209, 704.6799520561879, 737.8941462725612, 832.7096749378275, 595.4533145073506, 814.4515674525795, 839.872010269351, 781.1368093859531, 787.3096433452688, 724.7788710276506, 847.6412493333894, 864.6742797116402, 828.7111742575712, 922.8804420940888 ] }, "h1hand_balance_hard": { "time": [ 0.0, 181.3574838361809, 362.7149676723618, 544.0724515085427, 725.4299353447236, 906.7874191809045, 1088.1449030170854, 1269.5023868532662, 1450.8598706894472, 1632.2173545256283, 1813.574838361809, 1994.9323221979898, 2176.289806034171, 2357.647289870352, 2539.0047737065324, 2720.3622575427135, 2901.7197413788945, 3083.0772252150755, 3264.4347090512565, 3445.792192887437, 3627.149676723618, 3808.507160559799, 3989.8646443959797, 4171.222128232161, 4352.579612068342, 4533.937095904523, 4715.294579740704, 4896.652063576884, 5078.009547413065, 5259.367031249247, 5440.724515085427, 5622.081998921608, 5803.439482757789, 5984.79696659397, 6166.154450430151, 6347.511934266332, 6528.869418102513, 6710.226901938693, 6891.584385774874, 7072.941869611055, 7254.299353447236, 7435.656837283417, 7617.014321119598, 7798.371804955779, 7979.729288791959, 8161.08677262814, 8342.444256464321, 8523.801740300501, 8705.159224136683, 8886.516707972865, 9067.874191809045, 9249.231675645227, 9430.589159481407, 9611.946643317588, 9793.304127153768, 9974.66161098995, 10156.01909482613, 10337.376578662312, 10518.734062498494, 10700.091546334672, 10881.449030170854, 11062.806514007034, 11244.163997843216, 11425.521481679396, 11606.878965515578, 11788.236449351758, 11969.59393318794, 12150.951417024122, 12332.308900860302, 12513.666384696484, 12695.023868532664, 12876.381352368846, 13057.738836205026, 13239.096320041204, 13420.453803877386, 13601.811287713566, 13783.168771549748, 13964.526255385928, 14145.88373922211, 14327.24122305829, 14508.598706894472, 14689.956190730652, 14871.313674566834, 15052.671158403016, 15234.028642239196, 15415.386126075378, 15596.743609911558, 15778.10109374774, 15959.458577583919, 16140.816061420099, 16322.17354525628, 16503.53102909246, 16684.888512928643, 16866.245996764825, 17047.603480601003, 17228.960964437185, 17410.318448273367, 17591.67593210955, 17773.03341594573, 17954.39089978191, 18135.74838361809, 18317.105867454273, 18498.463351290455, 18679.820835126633, 18861.178318962815, 19042.535802798993, 19223.893286635175, 19405.250770471357, 19586.608254307535, 19767.965738143717, 19949.3232219799, 20130.68070581608, 20312.03818965226, 20493.39567348844, 20674.753157324623, 20856.110641160805, 21037.468124996987, 21218.825608833165, 21400.183092669344, 21581.540576505526, 21762.898060341708, 21944.25554417789, 22125.613028014068, 22306.97051185025, 22488.32799568643, 22669.685479522614, 22851.042963358792, 23032.400447194974, 23213.757931031156, 23395.115414867338, 23576.472898703516, 23757.830382539698, 23939.18786637588, 24120.54535021206, 24301.902834048244, 24483.260317884422, 24664.617801720604, 24845.975285556786, 25027.332769392968, 25208.690253229146, 25390.047737065328, 25571.40522090151, 25752.762704737692, 25934.12018857387, 26115.477672410052, 26296.83515624623, 26478.19264008241, 26659.55012391859, 26840.907607754772, 27022.265091590954, 27203.622575427133, 27384.980059263315, 27566.337543099497, 27747.69502693568, 27929.052510771857, 28110.40999460804, 28291.76747844422, 28473.124962280403, 28654.48244611658, 28835.839929952763, 29017.197413788945, 29198.554897625127, 29379.912381461305, 29561.269865297487, 29742.62734913367, 29923.98483296985, 30105.342316806033, 30286.69980064221, 30468.057284478393, 30649.414768314575, 30830.772252150757, 31012.129735986935, 31193.487219823117, 31374.8447036593, 31556.20218749548, 31737.55967133166, 31918.917155167837, 32100.27463900402, 32281.632122840198, 32462.98960667638, 32644.34709051256, 32825.70457434874, 33007.06205818492, 33188.4195420211, 33369.777025857285, 33551.13450969347, 33732.49199352965, 33913.84947736583, 34095.206961202006, 34276.56444503819, 34457.92192887437, 34639.27941271055, 34820.63689654673, 35001.994380382916, 35183.3518642191, 35364.70934805528, 35546.06683189146, 35727.424315727636, 35908.78179956382, 36090.1392834 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0, 19840000.0, 20480000.0, 21120000.0, 21760000.0, 22400000.0, 23040000.0, 23680000.0, 24320000.0, 24960000.0, 25600000.0, 26240000.0, 26880000.0, 27520000.0, 28160000.0, 28800000.0, 29440000.0, 30080000.0, 30720000.0, 31360000.0, 32000000.0, 32640000.0, 33280000.0, 33920000.0, 34560000.0, 35200000.0, 35840000.0, 36480000.0, 37120000.0, 37760000.0, 38400000.0, 39040000.0, 39680000.0, 40320000.0, 40960000.0, 41600000.0, 42240000.0, 42880000.0, 43520000.0, 44160000.0, 44800000.0, 45440000.0, 46080000.0, 46720000.0, 47360000.0, 48000000.0, 48640000.0, 49280000.0, 49920000.0, 50560000.0, 51200000.0, 51840000.0, 52480000.0, 53120000.0, 53760000.0, 54400000.0, 55040000.0, 55680000.0, 56320000.0, 56960000.0, 57600000.0, 58240000.0, 58880000.0, 59520000.0, 60160000.0, 60800000.0, 61440000.0, 62080000.0, 62720000.0, 63360000.0, 64000000.0, 64640000.0, 65280000.0, 65920000.0, 66560000.0, 67200000.0, 67840000.0, 68480000.0, 69120000.0, 69760000.0, 70400000.0, 71040000.0, 71680000.0, 72320000.0, 72960000.0, 73600000.0, 74240000.0, 74880000.0, 75520000.0, 76160000.0, 76800000.0, 77440000.0, 78080000.0, 78720000.0, 79360000.0, 80000000.0, 80640000.0, 81280000.0, 81920000.0, 82560000.0, 83200000.0, 83840000.0, 84480000.0, 85120000.0, 85760000.0, 86400000.0, 87040000.0, 87680000.0, 88320000.0, 88960000.0, 89600000.0, 90240000.0, 90880000.0, 91520000.0, 92160000.0, 92800000.0, 93440000.0, 94080000.0, 94720000.0, 95360000.0, 96000000.0, 96640000.0, 97280000.0, 97920000.0, 98560000.0, 99200000.0, 99840000.0, 100480000.0, 101120000.0, 101760000.0, 102400000.0, 103040000.0, 103680000.0, 104320000.0, 104960000.0, 105600000.0, 106240000.0, 106880000.0, 107520000.0, 108160000.0, 108800000.0, 109440000.0, 110080000.0, 110720000.0, 111360000.0, 112000000.0, 112640000.0, 113280000.0, 113920000.0, 114560000.0, 115200000.0, 115840000.0, 116480000.0, 117120000.0, 117760000.0, 118400000.0, 119040000.0, 119680000.0, 120320000.0, 120960000.0, 121600000.0, 122240000.0, 122880000.0, 123520000.0, 124160000.0, 124800000.0, 125440000.0, 126080000.0, 126720000.0, 127360000.0 ], "return": [ 0.0, 38.196160634358726, 58.770947774251304, 62.07254155476888, 67.4906005859375, 70.74937947591145, 74.7848612467448, 77.75097147623698, 79.08344014485677, 80.12428538004558, 84.40036265055339, 84.25612894694011, 89.841064453125, 86.2732442220052, 92.54730733235677, 95.28934987386067, 95.8080342610677, 97.19317372639973, 107.09626007080078, 110.65089670817058, 114.67183176676433, 110.9939193725586, 112.79729715983073, 113.6125971476237, 116.34737141927083, 126.48547871907552, 133.89915466308594, 109.39595286051433, 134.32386779785156, 115.43318176269531, 143.85650126139322, 119.99575297037761, 132.2745564778646, 135.0002670288086, 187.80313873291016, 159.67255147298178, 146.672732035319, 158.2981389363607, 136.65037282307944, 128.09097798665366, 169.46741231282553, 161.30030314127603, 164.15653228759766, 137.65259297688803, 136.9373321533203, 206.19168599446616, 159.12822469075522, 179.7809041341146, 172.49272664388022, 250.76256306966147, 194.66214497884116, 195.08185323079428, 235.58951314290366, 166.8591283162435, 238.91442362467447, 245.55697631835938, 207.1026153564453, 255.12977600097656, 217.94174194335938, 166.34827168782553, 149.34825642903647, 183.07278696695963, 218.01471964518228, 235.31849161783853, 195.28186543782553, 210.81011454264322, 170.80133565266928, 211.78978983561197, 218.24620564778647, 197.42310587565103, 182.58748881022134, 152.4922637939453, 197.9974594116211, 201.38032023111978, 190.21263122558594, 240.82181294759116, 286.7570495605469, 188.37657165527344, 187.61556498209634, 203.70641072591147, 269.580815633138, 177.3887939453125, 252.92713419596353, 267.0267028808594, 240.3121795654297, 239.61591593424478, 207.05109151204428, 235.18618774414062, 177.18890889485678, 202.33190409342447, 160.5045623779297, 185.72308349609375, 188.78704833984375, 277.3094177246094, 272.63533782958984, 224.07214864095053, 232.78923543294272, 184.01846822102866, 225.31009419759116, 225.0649668375651, 304.5735168457031, 210.20008341471353, 201.0149180094401, 243.43156941731772, 215.55621337890625, 195.11417643229166, 200.63154093424478, 250.17939249674478, 166.86117553710938, 239.04779052734375, 202.9445546468099, 230.7897491455078, 200.33321634928384, 166.21000162760416, 196.61333719889322, 262.797119140625, 172.99969991048178, 223.8022206624349, 207.91385396321616, 194.3153839111328, 213.40963745117188, 237.738525390625, 221.23444112141928, 197.73790486653647, 180.7091064453125, 154.77271016438803, 241.7831827799479, 224.94592793782553, 185.94915771484375, 201.07649739583334, 219.7017618815104, 167.22635904947916, 170.35607401529947, 163.76718393961588, 225.6471710205078, 183.0110321044922, 192.79330444335938, 243.40941365559897, 207.83230590820312, 187.21705118815103, 254.02437337239584, 182.39125061035156, 186.51834615071616, 198.46961466471353, 173.48248799641928, 169.77811686197916, 194.39813232421875, 229.33489481608072, 197.5059356689453, 195.6003621419271, 193.1749471028646, 201.3477986653646, 149.3585459391276, 160.55817159016928, 262.8863830566406, 194.60286458333334, 155.88604227701822, 155.73231506347656, 185.68701171875, 232.66948445638022, 191.00763956705728, 184.53072102864584, 238.246337890625, 183.43255615234375, 208.6761220296224, 198.46814473470053, 182.61422729492188, 219.0441640218099, 272.5106150309245, 228.05343627929688, 161.55077107747397, 202.0955810546875, 226.70318094889322, 252.42306518554688, 248.66568501790366, 242.6952362060547, 189.46682739257812, 151.86629740397134, 180.39802042643228, 214.47354125976562, 190.78692626953125, 205.54802958170572, 258.9932454427083, 253.63004557291666, 211.98404439290366, 211.90413411458334, 191.16903686523438, 219.9814249674479, 173.84603881835938, 207.7472915649414, 268.8358154296875, 219.39684041341147, 195.6284434000651, 185.613037109375, 184.69058227539062, 175.6501668294271, 221.70868937174478, 179.950927734375, 196.78107706705728, 243.99713134765625 ], "return_min": [ 0.0, 34.45428217803158, 55.41737184120804, 61.41869831871158, 66.43249621420325, 69.62003501316896, 73.62000081838235, 74.60206213497288, 76.32229807409914, 77.34785807783203, 81.51511554114175, 79.76769534189077, 85.01361415766722, 81.90230639389048, 89.69955042492953, 90.6178239151312, 88.02047685300616, 96.07391744623853, 90.61242490649848, 101.45223290545653, 95.02559927014411, 100.60899040578748, 95.09311835848013, 97.23245698211139, 94.93065587379392, 96.36739335426385, 105.43736833701234, 97.00441945236639, 98.35959134250493, 98.28940804777696, 92.67936441640799, 96.88782462350221, 104.22682035772064, 118.07620764500636, 122.03861283666923, 77.70721064239504, 123.05194666169182, 90.49265642797565, 109.53681253072628, 100.79331677933696, 132.82121169040914, 118.10119067918774, 128.9584653469691, 130.8071722988806, 134.49261515372942, 92.08135912203397, 124.19046377359581, 98.81203426788495, 112.60570730312568, 191.6848021718945, 169.2558995325841, 164.44067226472907, 162.58134839465347, 101.12928192118886, 150.588229166145, 186.23761923215693, 150.1613026052525, 164.03645279537557, 155.29520585208192, 156.8134730705425, 110.8007256006957, 89.45774057195948, 183.66333080851112, 182.8045774261979, 147.439039041815, 192.50448931520535, 153.81440899229477, 159.37387265718547, 156.8847591896903, 140.9422189870913, 133.79386046844553, 121.2208783359017, 117.78878466109666, 162.0891325977859, 142.91152152327456, 145.66650485354245, 181.36210061214183, 172.0549260337872, 129.77998980595356, 184.15597889815282, 226.4602045321107, 139.90753198856368, 212.27614858542546, 258.7678969036728, 128.43190294932754, 155.87710678035796, 124.28137596113937, 212.97156265828514, 144.41096078962323, 124.89275940718964, 141.10913428610715, 166.79807120535114, 182.2678255457014, 174.8131193671947, 117.43064036520028, 186.94050341879202, 159.0381560034768, 155.60826522621846, 183.84612565643005, 137.57668363877298, 271.9786662677094, 160.17609619913318, 146.13779258497067, 175.88971848479008, 173.54095007784315, 134.59203053917872, 166.65528346229607, 180.4588111039666, 145.9365899929819, 151.56289219435445, 133.81831701051803, 191.35533174986944, 157.67126588854296, 130.87317243361662, 178.3746080438651, 229.5139109960802, 149.1997992438169, 160.45883006247314, 160.1759590836749, 125.17163497894785, 175.2298130949002, 187.0507229628084, 199.16419394083135, 141.84808456873083, 120.67866534173005, 141.746042432658, 220.1337018131785, 158.57603198020888, 148.14785461734908, 184.8981647429553, 148.06760980619828, 122.53421916003592, 130.70419158023753, 119.64487618156373, 154.3790765581566, 152.53780110145092, 178.43447367032533, 194.91483762218263, 149.96549429029432, 154.4239946844543, 169.34695408115252, 175.13164904117792, 134.66237652341292, 172.8040557403638, 153.92361892495038, 132.77755809968215, 158.73452883959555, 178.48066951373008, 158.17790159002567, 184.34224660737877, 151.50959380362832, 181.05786787718358, 128.44826901669964, 135.85806372404804, 224.19958239539173, 159.54753154225168, 139.97302598768363, 136.08539724634934, 169.96330045285345, 214.40018491853647, 138.97055888555082, 174.08251732565873, 188.24750174073284, 154.5361398840479, 147.65977634084902, 150.8830394081585, 148.54773711493283, 195.06914279689272, 249.31286395949382, 148.8014394989193, 128.945207717944, 171.02724765762608, 164.33448317784035, 173.62293096123312, 211.63861571735728, 163.8461708101563, 170.99422189105155, 127.6976483325336, 166.47019281749013, 182.54807929613827, 124.39573212425708, 162.95733679973253, 218.6751066764981, 200.5109498309493, 169.39523590474533, 139.30340693319937, 164.06790645736223, 182.94378495233704, 142.95961881523903, 146.49638746199017, 233.57057465778408, 165.27324945760324, 171.61117351294516, 142.4697787158965, 160.95787142437774, 163.002228594134, 167.43590048100222, 160.83785550209495, 151.80602190528384, 205.61825791496244 ], "return_max": [ 0.0, 41.93803909068587, 62.12452370729457, 62.72638479082619, 68.54870495767175, 71.87872393865395, 75.94972167510724, 80.89988081750109, 81.8445822156144, 82.90071268225913, 87.28560975996503, 88.74456255198945, 94.66851474858278, 90.64418205011992, 95.395064239784, 99.96087583259015, 103.59559166912925, 98.31243000656094, 123.58009523510309, 119.84956051088463, 134.31806426338454, 121.3788483393297, 130.50147596118134, 129.99273731313602, 137.76408696474772, 156.6035640838872, 162.36094098915953, 121.78748626866226, 170.2881442531982, 132.57695547761367, 195.03363810637845, 143.103681317253, 160.32229259800854, 151.92432641261084, 253.56766462915107, 241.63789230356852, 170.29351740894617, 226.10362144474573, 163.7639331154326, 155.38863919397033, 206.11361293524192, 204.49941560336433, 199.35459922822622, 144.49801365489546, 139.3820491529112, 320.30201286689834, 194.06598560791463, 260.74977400034425, 232.37974598463475, 309.84032396742845, 220.06839042509822, 225.7230341968595, 308.5976778911538, 232.58897471129814, 327.2406180832039, 304.8763334045618, 264.0439281076381, 346.22309920657756, 280.5882780346368, 175.88307030510856, 187.89578725737724, 276.68783336195975, 252.36610848185344, 287.83240580947916, 243.12469183383607, 229.1157397700811, 187.7882623130438, 264.20570701403847, 279.60765210588266, 253.90399276421076, 231.38111715199716, 183.7636492519889, 278.20613416214553, 240.67150786445367, 237.51374092789732, 335.97712104163986, 392.1519985089519, 204.69821727675966, 245.45114015823913, 223.2568425536701, 312.7014267341653, 214.87005590206132, 293.5781198065016, 275.28550885804594, 352.19245618153184, 323.3547250881316, 289.8208070629492, 257.4008128299961, 209.96685700009033, 279.7710487796593, 179.89999046975223, 204.64809578683636, 195.3062711339861, 379.80571608202405, 427.84003529397944, 261.203793863109, 306.54031486240865, 212.42867121583885, 266.77406273875226, 312.5532500363572, 337.16836742369685, 260.2240706302939, 255.89204343390952, 310.97342034984536, 257.57147667996935, 255.6363223254046, 234.6077984061935, 319.8999738895229, 187.78576108123684, 326.53268886033305, 272.0707922831018, 270.2241665411462, 242.99516681002473, 201.5468308215917, 214.85206635392134, 296.0803272851698, 196.79960057714666, 287.1456112623967, 255.6517488427574, 263.4591328433178, 251.58946180744354, 288.42632781844156, 243.3046883020072, 253.6277251643421, 240.73954754889496, 167.79937789611807, 263.4326637467173, 291.3158238954422, 223.75046081233842, 217.2548300487114, 291.33591395682254, 211.9184989389224, 210.0079564503614, 207.889491697668, 296.915265482859, 213.48426310753345, 207.15213521639342, 291.9039896890153, 265.69911752611193, 220.01010769184776, 338.70179266363914, 189.6508521795252, 238.3743157780194, 224.13517358906327, 193.04135706788819, 206.77867562427616, 230.06173580884195, 280.18912011843133, 236.83396974786496, 206.85847767647542, 234.84030040210087, 221.6377294535456, 170.26882286155555, 185.25827945629052, 301.5731837178895, 229.658197624415, 171.7990585663528, 175.37923288060378, 201.41072298464655, 250.93878399422397, 243.04472024856375, 194.97892473163296, 288.24517404051716, 212.3289724206396, 269.6924677183958, 246.05325006124255, 216.68071747491092, 243.0191852467271, 295.70836610235517, 307.30543305967444, 194.15633443700395, 233.16391445174892, 289.0718787199461, 331.2231994098606, 285.69275431845006, 321.54430160195307, 207.9394328941047, 176.0349464754091, 194.32584803537443, 246.39900322339298, 257.17812041480545, 248.1387223636789, 299.3113842089185, 306.749141314884, 254.57285288106198, 284.5048612959673, 218.27016727310652, 257.0190649825588, 204.73245882147972, 268.9981956678926, 304.10105620159095, 273.5204313692197, 219.64571328718503, 228.7562955028535, 208.4232931264035, 188.29810506472018, 275.98147826248737, 199.06399996665505, 241.75613222883072, 282.37600478035006 ] }, "h1hand_stair": { "time": [ 0.0, 387.40268051806663, 774.8053610361333, 1162.2080415542, 1549.6107220722665, 1937.013402590333, 2324.4160831084, 2711.818763626466, 3099.221444144533, 3486.6241246626, 3874.026805180666, 4261.4294856987335, 4648.8321662168, 5036.234846734867, 5423.637527252932, 5811.040207771, 6198.442888289066, 6585.845568807133, 6973.2482493252, 7360.650929843266, 7748.053610361332, 8135.4562908794, 8522.858971397467, 8910.261651915533, 9297.6643324336, 9685.067012951667, 10072.469693469733, 10459.872373987799, 10847.275054505864, 11234.677735023932, 11622.080415542 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0 ], "return": [ 0.0, 23.89271863301595, 34.71700096130371, 60.09582392374674, 73.17438507080078, 151.34715779622397, 224.9233856201172, 271.56304423014325, 316.3719889322917, 371.5347391764323, 382.11834716796875, 427.6658426920573, 422.29393513997394, 487.94639078776044, 474.13771565755206, 548.6403503417969, 592.8809407552084, 625.5245564778646, 660.0660400390625, 652.6300048828125, 654.2977498372396, 667.5510660807291, 689.77783203125, 690.3555501302084, 692.7436726888021, 714.7565511067709, 700.9408976236979, 730.7681477864584, 744.6813557942709, 703.8641560872396, 689.6199951171875 ], "return_min": [ 0.0, 15.890752302265494, 29.305759036971846, 45.07623410366991, 64.0256023149527, 108.29035638002946, 168.69137872905253, 235.03404534904712, 268.15860258395946, 351.4744388725385, 372.16143158111464, 411.6028455251883, 342.44605901687726, 425.6015045319289, 438.91894798410647, 466.1055596018814, 559.5168597506888, 596.1932083138497, 610.9713671046555, 592.2205757048379, 609.8229299953937, 623.5360991628968, 650.7099879959824, 643.435216307602, 640.6507078943843, 684.2973876188141, 674.313749484506, 702.2090532268096, 720.8903004478105, 644.9576463371507, 627.955726274373 ], "return_max": [ 0.0, 31.894684963766405, 40.12824288563557, 75.11541374382357, 82.32316782664886, 194.40395921241847, 281.15539251118184, 308.0920431112394, 364.5853752806239, 391.59503948032614, 392.07526275482286, 443.72883985892634, 502.1418112630706, 550.2912770435919, 509.35648333099766, 631.1751410817124, 626.2450217597279, 654.8559046418795, 709.1607129734695, 713.0394340607871, 698.7725696790856, 711.5660329985615, 728.8456760665176, 737.2758839528148, 744.83663748322, 745.2157145947276, 727.5680457628897, 759.3272423461071, 768.4724111407312, 762.7706658373286, 751.284263960002 ] }, "h1hand_slide": { "time": [ 0.0, 309.2993865171333, 618.5987730342666, 927.8981595514, 1237.1975460685333, 1546.4969325856666, 1855.7963191028, 2165.0957056199336, 2474.3950921370665, 2783.6944786542, 3092.9938651713333, 3402.293251688467, 3711.5926382056, 4020.8920247227334, 4330.191411239867, 4639.490797757, 4948.790184274133, 5258.089570791267, 5567.3889573084, 5876.688343825534, 6185.9877303426665, 6495.2871168598, 6804.586503376934, 7113.885889894067, 7423.1852764112, 7732.484662928333, 8041.784049445467, 8351.0834359626, 8660.382822479734, 8969.682208996868, 9278.981595514 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0 ], "return": [ 0.0, 25.91161886850993, 46.50613911946615, 69.50710932413737, 125.7647476196289, 170.76056416829428, 238.53360493977866, 265.1405283610026, 287.49407958984375, 301.4013264973958, 266.9022521972656, 286.79445393880206, 311.7269694010417, 334.2634684244792, 393.5237731933594, 413.90247599283856, 468.1215108235677, 478.6412353515625, 521.7706604003906, 597.3247273763021, 682.8544311523438, 815.7204182942709, 869.786865234375, 870.6855061848959, 924.8785807291666, 915.4677937825521, 915.6895751953125, 901.403076171875, 910.4904378255209, 935.8196411132812, 904.0150146484375 ], "return_min": [ 0.0, 17.31227752983613, 41.81182919734604, 57.14489937214505, 96.5149593569322, 163.1428876300595, 222.31254684790358, 231.41575879824072, 255.4749652490164, 243.84204384859126, 249.15111161639513, 265.7686367908954, 283.11169371305834, 310.17930949833135, 345.48766050051773, 399.79618209856903, 452.1210687697244, 462.9765296109677, 495.99326194394064, 558.9139653990013, 668.4022813113013, 795.2447221786368, 830.8790856292156, 823.1121698948539, 915.6907611174105, 893.0342590385319, 869.1971213119441, 878.6902612410832, 894.2164153448738, 921.1641786848095, 859.1733242686221 ], "return_max": [ 0.0, 34.51096020718373, 51.20044904158625, 81.8693192761297, 155.01453588232562, 178.37824070652906, 254.75466303165373, 298.86529792376456, 319.5131939306711, 358.9606091462004, 284.6533927781361, 307.8202710867087, 340.342245089025, 358.347627350627, 441.559885886201, 428.0087698871081, 484.121952877411, 494.3059410921573, 547.5480588568406, 635.7354893536029, 697.3065809933862, 836.1961144099049, 908.6946448395344, 918.2588424749379, 934.0664003409228, 937.9013285265723, 962.1820290786809, 924.1158911026668, 926.764460306168, 950.475103541753, 948.8567050282529 ] }, "h1hand_pole": { "time": [ 0.0, 385.46620238036667, 770.9324047607333, 1156.3986071411002, 1541.8648095214667, 1927.331011901833, 2312.7972142822005, 2698.2634166625667, 3083.7296190429333, 3469.1958214233, 3854.662023803666, 4240.128226184033, 4625.594428564401, 5011.060630944767, 5396.526833325133, 5781.9930357055, 6167.459238085867, 6552.925440466233, 6938.3916428466, 7323.857845226967, 7709.324047607332, 8094.7902499877, 8480.256452368067, 8865.722654748433, 9251.188857128802, 9636.655059509167, 10022.121261889533, 10407.5874642699, 10793.053666650267, 11178.519869030633, 11563.986071411 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0 ], "return": [ 0.0, 86.94035212198894, 315.2128092447917, 259.81410725911456, 438.83277384440106, 453.3154805501302, 472.5972188313802, 600.6257120768229, 602.0543416341146, 604.5667724609375, 625.5578002929688, 697.9258626302084, 730.2250773111979, 770.3667805989584, 756.2709757486979, 817.2851765950521, 814.6104939778646, 834.628662109375, 836.6917317708334, 836.28759765625, 802.5843302408854, 828.5123494466146, 889.4725545247396, 927.0976359049479, 918.7555135091146, 918.3529663085938, 929.7622884114584, 927.1256306966146, 924.934814453125, 935.3818562825521, 867.1444905598959 ], "return_min": [ 0.0, 34.151917472845525, 169.25709369786907, 195.91231211958, 309.209006533762, 333.5828818070501, 457.3834555761806, 538.0321487054368, 556.7912159567086, 546.9315820056133, 562.6627573317008, 625.2555533079764, 698.4326425070107, 708.3732902267687, 687.8694938402527, 787.8280779660413, 789.6780869058325, 772.5260295300255, 792.1538079552, 783.306104588348, 739.7921613627637, 715.3343740103686, 847.4289290942377, 887.3489508402633, 877.1328123681296, 863.362262954571, 880.872949690255, 877.8235128123039, 882.9430126525199, 900.4706760458522, 743.7438399173179 ], "return_max": [ 0.0, 139.72878677113235, 461.1685247917143, 323.7159023986491, 568.4565411550401, 573.0480792932102, 487.81098208657977, 663.2192754482089, 647.3174673115207, 662.2019629162617, 688.4528432542367, 770.5961719524404, 762.0175121153851, 832.360270971148, 824.672457657143, 846.742275224063, 839.5429010498967, 896.7312946887245, 881.2296555864667, 889.269090724152, 865.376499119007, 941.6903248828606, 931.5161799552416, 966.8463209696324, 960.3782146500996, 973.3436696626165, 978.6516271326617, 976.4277485809254, 966.9266162537301, 970.293036519252, 990.5451412024738 ] }, "h1hand_push": { "time": [ 0.0, 312.6167574498032, 625.2335148996063, 937.8502723494095, 1250.4670297992127, 1563.083787249016, 1875.700544698819, 2188.3173021486223, 2500.9340595984254, 2813.5508170482285, 3126.167574498032, 3438.784331947835, 3751.401089397638, 4064.0178468474414, 4376.634604297245, 4689.251361747049, 5001.868119196851, 5314.484876646654, 5627.101634096457, 5939.718391546261, 6252.335148996064, 6564.951906445867, 6877.56866389567, 7190.1854213454735, 7502.802178795276, 7815.418936245081, 8128.035693694883, 8440.652451144686, 8753.26920859449, 9065.885966044292, 9378.502723494097, 9691.119480943898, 10003.736238393702, 10316.352995843505, 10628.969753293308, 10941.586510743113, 11254.203268192914, 11566.820025642717, 11879.436783092522, 12192.053540542325, 12504.670297992128, 12817.287055441931, 13129.903812891735, 13442.520570341538, 13755.13732779134, 14067.754085241144, 14380.370842690947, 14692.987600140748, 15005.604357590551, 15318.221115040358, 15630.837872490161, 15943.454629939963, 16256.071387389766, 16568.68814483957, 16881.304902289372, 17193.921659739175, 17506.53841718898, 17819.15517463878, 18131.771932088584, 18444.38868953839, 18757.005446988194, 19069.622204437994, 19382.238961887797, 19694.8557193376, 20007.472476787403, 20320.089234237206, 20632.70599168701, 20945.322749136812, 21257.939506586616, 21570.55626403642, 21883.173021486225, 22195.78977893603, 22508.406536385828, 22821.02329383563, 23133.640051285434, 23446.25680873524, 23758.873566185044, 24071.490323634847, 24384.10708108465, 24696.723838534454, 25009.340595984257, 25321.95735343406, 25634.574110883863, 25947.190868333666, 26259.80762578347, 26572.424383233272, 26885.041140683075, 27197.65789813288, 27510.27465558268, 27822.891413032485, 28135.508170482288, 28448.12492793209, 28760.741685381894, 29073.358442831694, 29385.975200281497, 29698.5919577313, 30011.208715181103, 30323.825472630913, 30636.442230080716, 30949.05898753052, 31261.675744980323, 31574.292502430126, 31886.909259879925, 32199.52601732973, 32512.14277477953, 32824.759532229335, 33137.37628967914, 33449.99304712894, 33762.609804578744, 34075.22656202855, 34387.84331947835, 34700.46007692815, 35013.07683437796, 35325.69359182776, 35638.31034927756, 35950.927106727366, 36263.54386417717, 36576.16062162697, 36888.77737907678, 37201.394136526585, 37514.01089397639, 37826.62765142619, 38139.24440887599, 38451.86116632579, 38764.477923775594, 39077.0946812254, 39389.7114386752, 39702.328196125, 40014.944953574806, 40327.56171102461, 40640.17846847441, 40952.795225924216, 41265.41198337402, 41578.02874082382, 41890.645498273625, 42203.26225572343, 42515.87901317323, 42828.495770623034, 43141.11252807284, 43453.72928552265, 43766.34604297245, 44078.962800422254, 44391.57955787206, 44704.19631532185, 45016.813072771656, 45329.42983022146, 45642.04658767126, 45954.663345121066, 46267.28010257087, 46579.89686002067, 46892.51361747048, 47205.130374920285, 47517.74713237009, 47830.36388981989, 48142.980647269695, 48455.5974047195, 48768.2141621693, 49080.830919619104, 49393.44767706891, 49706.06443451871, 50018.68119196851, 50331.29794941832, 50643.91470686812, 50956.53146431792, 51269.148221767726, 51581.76497921753, 51894.38173666733, 52206.998494117135, 52519.61525156694, 52832.23200901674, 53144.848766466544, 53457.46552391635, 53770.08228136615, 54082.699038815954, 54395.31579626576, 54707.93255371556, 55020.54931116536, 55333.166068615166, 55645.78282606497, 55958.39958351477, 56271.016340964576, 56583.63309841438, 56896.24985586418, 57208.866613313985, 57521.48337076379, 57834.10012821359, 58146.71688566339, 58459.33364311319, 58771.95040056299 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0, 19840000.0, 20480000.0, 21120000.0, 21760000.0, 22400000.0, 23040000.0, 23680000.0, 24320000.0, 24960000.0, 25600000.0, 26240000.0, 26880000.0, 27520000.0, 28160000.0, 28800000.0, 29440000.0, 30080000.0, 30720000.0, 31360000.0, 32000000.0, 32640000.0, 33280000.0, 33920000.0, 34560000.0, 35200000.0, 35840000.0, 36480000.0, 37120000.0, 37760000.0, 38400000.0, 39040000.0, 39680000.0, 40320000.0, 40960000.0, 41600000.0, 42240000.0, 42880000.0, 43520000.0, 44160000.0, 44800000.0, 45440000.0, 46080000.0, 46720000.0, 47360000.0, 48000000.0, 48640000.0, 49280000.0, 49920000.0, 50560000.0, 51200000.0, 51840000.0, 52480000.0, 53120000.0, 53760000.0, 54400000.0, 55040000.0, 55680000.0, 56320000.0, 56960000.0, 57600000.0, 58240000.0, 58880000.0, 59520000.0, 60160000.0, 60800000.0, 61440000.0, 62080000.0, 62720000.0, 63360000.0, 64000000.0, 64640000.0, 65280000.0, 65920000.0, 66560000.0, 67200000.0, 67840000.0, 68480000.0, 69120000.0, 69760000.0, 70400000.0, 71040000.0, 71680000.0, 72320000.0, 72960000.0, 73600000.0, 74240000.0, 74880000.0, 75520000.0, 76160000.0, 76800000.0, 77440000.0, 78080000.0, 78720000.0, 79360000.0, 80000000.0, 80640000.0, 81280000.0, 81920000.0, 82560000.0, 83200000.0, 83840000.0, 84480000.0, 85120000.0, 85760000.0, 86400000.0, 87040000.0, 87680000.0, 88320000.0, 88960000.0, 89600000.0, 90240000.0, 90880000.0, 91520000.0, 92160000.0, 92800000.0, 93440000.0, 94080000.0, 94720000.0, 95360000.0, 96000000.0, 96640000.0, 97280000.0, 97920000.0, 98560000.0, 99200000.0, 99840000.0, 100480000.0, 101120000.0, 101760000.0, 102400000.0, 103040000.0, 103680000.0, 104320000.0, 104960000.0, 105600000.0, 106240000.0, 106880000.0, 107520000.0, 108160000.0, 108800000.0, 109440000.0, 110080000.0, 110720000.0, 111360000.0, 112000000.0, 112640000.0, 113280000.0, 113920000.0, 114560000.0, 115200000.0, 115840000.0, 116480000.0, 117120000.0, 117760000.0, 118400000.0, 119040000.0, 119680000.0, 120320000.0 ], "return": [ 0.0, -465.70371500651044, -209.66437276204428, -215.35665893554688, -79.81609725952148, -97.29651769002278, -171.10615030924478, -58.52375920613607, 35.39680035909017, 67.48361078898112, 17.984004974365234, 46.02268600463867, 106.70371754964192, 52.096457163492836, 87.57223892211914, 60.093360900878906, 202.40312703450522, 35.78864034016927, 77.78755187988281, 199.13873799641928, 161.0906524658203, -6.331198374430339, 227.29449462890625, 225.0338363647461, 311.14427693684894, 233.39718119303384, 389.0839029947917, 355.971435546875, 262.3876698811849, 351.2383626302083, 315.8093973795573, 400.12449137369794, 398.5304260253906, 408.3233947753906, 473.21187082926434, 377.5498504638672, 485.5910339355469, 400.6451822916667, 458.1425069173177, 439.06847127278644, 326.82744852701825, 403.84735616048175, 480.24944559733075, 495.53029378255206, 483.7826232910156, 533.0876261393229, 472.01613362630206, 496.1668395996094, 512.2647094726562, 514.5851847330729, 470.1879475911458, 544.6070658365885, 360.79221598307294, 496.0656331380208, 504.4897104899089, 546.812978108724, 533.8811543782552, 535.4517415364584, 471.8639500935872, 557.7514444986979, 629.8241882324219, 534.7682189941406, 511.1558583577474, 564.1106160481771, 494.4066518147786, 652.6817423502604, 521.490966796875, 520.7968419392904, 562.0110270182291, 456.7043991088867, 605.8467203776041, 640.2967529296875, 603.9107055664062, 574.7682291666666, 611.0949910481771, 631.2831217447916, 605.4075927734375, 546.4880421956381, 651.6212768554688, 625.2835998535156, 702.7826131184896, 600.3868408203125, 652.3099568684896, 490.0066731770833, 593.0624084472656, 541.475596110026, 587.3904622395834, 612.0767822265625, 577.2430826822916, 571.9033610026041, 625.8723347981771, 488.1465759277344, 592.8579711914062, 510.37445576985675, 523.4727376302084, 603.7941080729166, 547.8638305664062, 583.2688598632812, 570.062510172526, 527.5540873209635, 582.3120727539062, 656.2349243164062, 508.43341064453125, 607.9522298177084, 621.880859375, 636.3435770670573, 665.6842244466146, 588.9774169921875, 636.5202026367188, 662.1762288411459, 561.2283732096354, 598.4204813639323, 546.9343566894531, 626.6971232096354, 652.875732421875, 466.8674825032552, 524.4137674967448, 582.6043701171875, 534.4446411132812, 614.892588297526, 673.6151529947916, 547.5692138671875, 635.2989908854166, 611.1135660807291, 512.3421630859375, 652.3751627604166, 667.9252522786459, 649.40771484375, 579.6252848307291, 612.9027201334635, 615.0734354654948, 541.9854431152344, 587.8748372395834, 592.4884033203125, 755.3507893880209, 699.7528279622396, 717.8043619791666, 616.4128214518229, 610.3042602539062, 576.1018473307291, 669.9233601888021, 662.5718587239584, 601.7003580729166, 619.9717000325521, 616.17236328125, 647.919189453125, 720.2536214192709, 676.1502685546875, 638.7950236002604, 725.2923787434896, 708.4577229817709, 657.550038655599, 606.4547119140625, 720.4335225423177, 638.5685424804688, 644.5047810872396, 675.3748372395834, 545.7748819986979, 775.0440673828125, 696.5569864908854, 684.9332377115885, 675.4559529622396, 626.1231486002604, 670.3891398111979, 633.9583841959635, 662.5113728841146, 625.4878743489584, 723.1873982747396, 747.1727294921875, 709.0388793945312, 723.6645100911459, 625.6194051106771, 593.6909993489584, 727.9166056315104, 689.2951049804688, 630.6175944010416, 730.9366251627604, 689.8072102864584, 696.1072387695312, 611.1871439615885, 745.7859090169271, 636.9237670898438, 793.2918090820312, 752.7328694661459, 767.5932413736979, 738.7565307617188, 687.819091796875, 690.4653930664062, 769.5697631835938 ], "return_min": [ 0.0, -651.7377904873346, -218.19490330067313, -230.40671483223093, -105.68158272497024, -210.8956213842916, -233.1911752316335, -161.58889381026177, 26.55360975511062, 4.289192049179803, -94.47579688203697, 4.849184562816511, 69.07344829711946, -22.40429315765919, 45.086476580251066, -54.71491217125336, 125.38167282396118, -98.6124742090824, -38.373892937135324, 145.39119983240533, 134.49273414913256, -59.93287233128325, 171.49033469732623, 133.5127446996148, 206.89851474999222, 177.69275565837683, 239.91640177783498, 218.16699335525163, 125.28320106593608, 177.69565455876588, 91.22711231593178, 174.67941277233874, 224.61917253986607, 176.72156917446412, 178.20974585184655, 162.2428865098692, 308.5236004576717, 101.43215919483697, 205.59336156515505, 218.07057710774632, 76.95591916101498, 102.58860226185669, 256.5846713119894, 279.8230541236833, 288.91774124081064, 285.3080260060623, 376.76426537108176, 292.3935583245744, 229.1453409203956, 294.52281767361126, 302.78698861571456, 356.1194735219518, 69.45947274408792, 383.9554921423231, 298.3401702958506, 321.4518209660676, 315.3049143709103, 346.0651825103665, 151.19494125741988, 357.1257353560337, 390.8240639442132, 348.42316335307174, 298.91965969292187, 462.3853769182599, 251.6649901787817, 479.78588898797443, 397.877011500767, 215.596613915609, 323.2564485248071, 186.6042427741824, 426.8233101559147, 362.0302869053308, 404.98571774036856, 359.2847078708686, 484.5054633019436, 493.2501176605723, 411.44129488977313, 256.5276751004272, 454.29501648725363, 413.8562263314633, 573.720797985042, 416.46692181466364, 467.0122085781444, 258.0562843972043, 395.2243701392048, 368.5233705526356, 452.636496904995, 464.5032990675227, 436.79629123776, 374.69121493690363, 466.7821493928696, 263.51951290052034, 429.315411959895, 304.9247130991566, 419.90325565387303, 432.2364118278375, 338.1089545341084, 364.2132868278239, 393.04972714186283, 359.9071598690796, 533.942556029381, 468.2753658460946, 261.8154182211118, 522.5536947167411, 557.584211053626, 510.90186706032495, 524.9047980503793, 487.37465424235245, 467.86209848079557, 545.6308974446529, 377.0755931355628, 422.04626971453774, 406.3920283244486, 444.8626825076674, 540.525108305069, 420.5410566062819, 489.81667849244985, 441.43925917296247, 289.6483089231782, 516.6815783827745, 591.9147876948792, 376.4286871094273, 534.641886307782, 388.33717493137976, 322.51834109876495, 523.6089601382292, 557.0252517096708, 537.192194230057, 479.63953220680753, 489.9297848053796, 456.3615203978124, 452.74782656646664, 437.8910113628925, 444.34476831068963, 622.8334792297339, 544.8195929605185, 599.7323369153378, 512.7902955981406, 406.1789579431253, 452.78874036721595, 582.1919827487237, 594.584856475092, 518.5465718482806, 598.0693065627131, 472.0681568588566, 485.4507956994522, 590.2059351368265, 586.9153657898868, 589.259669009053, 635.5611082804211, 623.3885305260387, 452.57285763522157, 565.053967031291, 539.3486244195427, 504.29379132207185, 558.5882700286743, 508.762833637914, 356.62391796920565, 629.1636777085658, 556.0355628400487, 444.2008257553892, 524.9059378555166, 404.75813990024164, 517.9978694633127, 513.871824831959, 612.3382451960599, 516.9922775142898, 680.363477847471, 627.5686018024139, 600.6839979902597, 614.6357271298568, 381.38851880783363, 525.7235252030711, 636.9343357735295, 617.0474412446536, 520.9154402643403, 551.0778985607511, 616.0456681349963, 608.2566427500684, 506.0818447569322, 715.3050290539916, 469.6640181908483, 710.4375439178509, 677.2274044637059, 715.0402021473672, 669.0276233195295, 590.3275579897734, 592.3499843504039, 654.5575435877125 ], "return_max": [ 0.0, -279.6696395256863, -201.13384222341543, -200.30660303886282, -53.95061179407273, 16.30258600424601, -109.02112538685606, 44.54137539798962, 44.23999096306972, 130.67802952878245, 130.44380683076744, 87.19618744646084, 144.3339868021644, 126.59720748464485, 130.05800126398722, 174.90163397301117, 279.42458124504924, 170.18975488942095, 193.94899669690096, 252.88627616043323, 187.68857078250807, 47.27047558242258, 283.09865456048624, 316.5549280298774, 415.39003912370566, 289.10160672769086, 538.2514042117484, 493.77587773849837, 399.4921386964337, 524.7810707016507, 540.3916824431828, 625.5695699750571, 572.4416795109152, 639.9252203763172, 768.2139958066821, 592.8568144178652, 662.658467413422, 699.8582053884963, 710.6916522694803, 660.0663654378266, 576.6989778930215, 705.1061100591069, 703.9142198826721, 711.2375334414209, 678.6475053412206, 780.8672262725835, 567.2680018815224, 699.9401208746443, 795.3840780249169, 734.6475517925345, 637.5889065665771, 733.0946581512252, 652.1249592220579, 608.1757741337185, 710.6392506839671, 772.1741352513804, 752.4573943856002, 724.8383005625502, 792.5329589297546, 758.377153641362, 868.8243125206305, 721.1132746352096, 723.3920570225729, 665.8358551780943, 737.1483134507755, 825.5775957125463, 645.104922092983, 825.9970699629719, 800.7656055116511, 726.804555443591, 784.8701305992936, 918.5632189540443, 802.835693392444, 790.2517504624647, 737.6845187944107, 769.316125829011, 799.3738906571018, 836.448409290849, 848.9475372236839, 836.710973375568, 831.8444282519372, 784.3067598259613, 837.6077051588348, 721.9570619569623, 790.9004467553265, 714.4278216674164, 722.1444275741717, 759.6502653856023, 717.6898741268233, 769.1155070683046, 784.9625202034847, 712.7736389549484, 756.4005304229174, 715.824198440557, 627.0422196065438, 775.3518043179957, 757.6187065987041, 802.3244328987387, 747.0752932031892, 695.2010147728474, 630.6815894784315, 844.1944827867179, 755.0514030679507, 693.3507649186756, 686.177507696374, 761.7852870737895, 806.4636508428499, 690.5801797420226, 805.1783067926419, 778.7215602376389, 745.3811532837079, 774.7946930133268, 687.4766850544577, 808.5315639116034, 765.226356538681, 513.1939084002285, 559.0108565010396, 723.7694810614125, 779.2409733033843, 713.1035982122775, 755.315518294704, 718.7097406249477, 735.9560954630513, 833.8899572300785, 702.16598507311, 781.141365382604, 778.825252847621, 761.623235457443, 679.6110374546507, 735.8756554615474, 773.785350533177, 631.2230596640021, 737.8586631162742, 740.6320383299353, 887.8680995463078, 854.6860629639607, 835.8763870429955, 720.0353473055052, 814.4295625646872, 699.4149542942423, 757.6547376288805, 730.5588609728247, 684.8541442975527, 641.8740935023911, 760.2765697036434, 810.3875832067978, 850.3013077017152, 765.3851713194882, 688.3303781914677, 815.0236492065582, 793.526915437503, 862.5272196759764, 647.855456796834, 901.5184206650928, 772.8432936388656, 730.421292145805, 841.9868408412527, 734.9258460281901, 920.9244570570592, 837.0784101417221, 925.6656496677879, 826.0059680689626, 847.4881573002791, 822.7804101590831, 754.044943559968, 712.6845005721693, 733.983471183627, 766.0113187020082, 866.7768571819611, 817.3937607988028, 832.6932930524349, 869.8502914135206, 661.6584734948457, 818.8988754894913, 761.542768716284, 740.319748537743, 910.7953517647696, 763.5687524379205, 783.9578347889941, 716.2924431662449, 776.2667889798627, 804.1835159888392, 876.1460742462116, 828.2383344685859, 820.1462806000286, 808.485438203908, 785.3106256039766, 788.5808017824086, 884.581982779475 ] }, "h1hand_cabinet": { "time": [ 0.0, 482.0103869639, 964.0207739278, 1446.0311608917, 1928.0415478556, 2410.0519348195, 2892.0623217834, 3374.0727087472997, 3856.0830957112, 4338.0934826751, 4820.103869639, 5302.114256602899, 5784.1246435668, 6266.1350305307, 6748.145417494599, 7230.155804458499, 7712.1661914224, 8194.1765783863, 8676.1869653502, 9158.197352314099, 9640.207739278 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0 ], "return": [ 0.0, 48.33211135864258, 41.21469751993815, 59.57719930013021, 75.67262268066406, 56.34848594665527, 81.90048472086589, 80.5737673441569, 81.59727350870769, 73.00228627522786, 88.43795522054036, 141.72611236572266, 114.48175557454427, 148.1658706665039, 112.07423400878906, 155.38145955403647, 130.3720957438151, 148.42277018229166, 155.57244873046875, 131.13333638509116, 188.2349853515625 ], "return_min": [ 0.0, 35.922316004492785, 28.06647614718972, 37.90235693360684, 44.75038850207668, 36.40830307855872, 61.24509401202533, 63.87340072139955, 61.64813090949677, 32.291284324269796, 66.49554026294187, 118.72944532889935, 74.6534328246156, 99.11059146542257, 80.17006584290101, 130.34182639754522, 98.22037494029969, 132.83954440041933, 144.75589448674302, 102.16349649420323, 136.03014741728717 ], "return_max": [ 0.0, 60.74190671279237, 54.36291889268658, 81.25204166665358, 106.59485685925145, 76.28866881475183, 102.55587542970645, 97.27413396691426, 101.5464161079186, 113.71328822618592, 110.38037017813885, 164.72277940254597, 154.31007832447293, 197.22114986758524, 143.97840217467711, 180.42109271052772, 162.5238165473305, 164.005995964164, 166.38900297419448, 160.10317627597908, 240.43982328583783 ] }, "h1hand_door": { "time": [ 0.0, 474.1971419775, 948.394283955, 1422.5914259325, 1896.78856791, 2370.9857098875, 2845.182851865, 3319.3799938425, 3793.57713582, 4267.7742777975, 4741.971419775, 5216.1685617525, 5690.36570373, 6164.5628457075, 6638.759987685, 7112.9571296625, 7587.15427164, 8061.3514136175, 8535.548555595, 9009.7456975725, 9483.94283955 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0 ], "return": [ 0.0, 113.92080434163411, 225.69124348958334, 249.13692220052084, 276.68773396809894, 287.51111857096356, 287.70652262369794, 298.76854451497394, 303.0335388183594, 305.23622639973956, 311.20665486653644, 312.6994222005208, 316.5781656901042, 321.4351806640625, 323.1549580891927, 328.4767761230469, 331.2190246582031, 332.8224283854167, 333.1351013183594, 330.49583943684894, 331.7462666829427 ], "return_min": [ 0.0, 83.71072308019035, 208.12097190097407, 239.92148547332886, 268.6843518582533, 280.06576497283703, 282.01917265202314, 295.1558484227942, 298.3069802027448, 300.2730247321967, 303.5733979888646, 300.2010388008713, 303.3949109736265, 311.38583689703665, 308.21839791204366, 310.94946877443516, 310.7380146256782, 314.98049407945916, 316.6951429319435, 312.89852900846034, 317.0532202601144 ], "return_max": [ 0.0, 144.13088560307787, 243.26151507819262, 258.3523589277128, 284.69111607794457, 294.9564721690901, 293.3938725953727, 302.3812406071537, 307.76009743397395, 310.19942806728244, 318.8399117442083, 325.19780560017034, 329.76142040658186, 331.48452443108835, 338.0915182663417, 346.0040834716586, 351.70003469072805, 350.6643626913742, 349.57505970477524, 348.09314986523754, 346.43931310577096 ] }, "h1hand_truck": { "time": [ 0.0, 356.2344639081333, 712.4689278162666, 1068.7033917244, 1424.9378556325332, 1781.1723195406666, 2137.4067834488, 2493.6412473569335, 2849.8757112650665, 3206.1101751732, 3562.3446390813333, 3918.5791029894667, 4274.8135668976, 4631.048030805733, 4987.282494713867, 5343.516958622, 5699.751422530133, 6055.985886438267, 6412.2203503464, 6768.454814254534, 7124.689278162667, 7480.9237420708, 7837.158205978933, 8193.392669887067, 8549.6271337952, 8905.861597703333, 9262.096061611466, 9618.3305255196, 9974.564989427734, 10330.799453335867, 10687.033917244, 11043.268381152133, 11399.502845060266, 11755.7373089684, 12111.971772876534, 12468.206236784667, 12824.4407006928, 13180.675164600932, 13536.909628509067, 13893.1440924172, 14249.378556325333, 14605.613020233466, 14961.8474841416, 15318.081948049734, 15674.316411957867, 16030.550875866, 16386.785339774135, 16743.019803682266, 17099.2542675904, 17455.48873149853, 17811.723195406666, 18167.957659314798, 18524.192123222932, 18880.426587131067, 19236.6610510392, 19592.89551494733, 19949.129978855468, 20305.364442763603, 20661.598906671734, 21017.833370579865, 21374.067834488 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0, 19840000.0, 20480000.0, 21120000.0, 21760000.0, 22400000.0, 23040000.0, 23680000.0, 24320000.0, 24960000.0, 25600000.0, 26240000.0, 26880000.0, 27520000.0, 28160000.0, 28800000.0, 29440000.0, 30080000.0, 30720000.0, 31360000.0, 32000000.0, 32640000.0, 33280000.0, 33920000.0, 34560000.0, 35200000.0, 35840000.0, 36480000.0, 37120000.0, 37760000.0, 38400000.0 ], "return": [ 0.0, 744.8859558105469, 796.5100860595703, 910.9717559814453, 975.1925201416016, 1047.324951171875, 1099.5369873046875, 1148.0682067871094, 1201.0522766113281, 1233.505615234375, 1255.5220031738281, 1290.6919860839844, 1311.6923828125, 1338.3526000976562, 1359.3213195800781, 1323.95654296875, 1365.9458312988281, 1372.3952026367188, 1371.4238586425781, 1380.5548706054688, 1387.2607421875, 1377.4356384277344, 1405.5323791503906, 1398.5776977539062, 1386.4181823730469, 1399.1890563964844, 1414.0697021484375, 1413.0535888671875, 1426.5853576660156, 1436.8981628417969, 1447.1106567382812, 1452.8192138671875, 1458.5155639648438, 1482.8174133300781, 1488.4780883789062, 1509.2735900878906, 1529.7430114746094, 1542.8193664550781, 1523.0067443847656, 1534.5915832519531, 1547.8734130859375, 1558.2222290039062, 1559.4790344238281, 1521.8999328613281, 1579.9296264648438, 1568.0743103027344, 1592.0011596679688, 1607.4393005371094, 1601.7018432617188, 1628.3120422363281, 1632.6797180175781, 1637.2363586425781, 1640.4541015625, 1648.7554626464844, 1637.6644287109375, 1660.0960693359375, 1661.5935363769531, 1654.3347778320312, 1661.7125244140625, 1665.4105834960938, 1686.1488037109375 ], "return_min": [ 0.0, 728.8708482176052, 762.0585259369802, 843.073411732124, 889.465099012336, 998.3990660977275, 1063.9706960921915, 1116.6410384966878, 1161.0117972409382, 1195.7086321791687, 1226.7667966679335, 1246.3428404936096, 1272.6613742372551, 1296.5317436158239, 1329.0087405178306, 1273.7531020924337, 1335.3961578228586, 1342.6771680749928, 1338.9942289987519, 1354.398859758841, 1352.2997701371219, 1322.036434338705, 1362.2402065457197, 1348.2375712433736, 1328.8611382153329, 1353.1487389210438, 1362.2909765642328, 1378.9425298602578, 1373.9676375490837, 1387.940960280719, 1394.5923024990384, 1392.6599873879725, 1397.4042979437356, 1416.8406937577124, 1436.3985979041825, 1451.1102549614811, 1462.6237026144347, 1480.086465539093, 1446.3469016733263, 1463.1031736967145, 1480.826246455255, 1507.2520940243835, 1484.954383969524, 1427.5705673711657, 1514.958677668042, 1495.3212565792096, 1524.5236672425701, 1493.734800162037, 1512.489867922995, 1521.317076862614, 1501.4009697149443, 1512.9679254210876, 1523.2808539998027, 1516.759711486079, 1521.9907509338339, 1541.900269758106, 1540.4093855281021, 1540.2998088326399, 1549.106670304241, 1539.2902805167103, 1557.2959035745841 ], "return_max": [ 0.0, 760.9010634034886, 830.9616461821604, 978.8701002307666, 1060.9199412708672, 1096.2508362460223, 1135.1032785171835, 1179.495375077531, 1241.092755981718, 1271.3025982895813, 1284.2772096797228, 1335.0411316743591, 1350.7233913877449, 1380.1734565794886, 1389.6338986423257, 1374.1599838450663, 1396.4955047747976, 1402.1132371984447, 1403.8534882864044, 1406.7108814520966, 1422.2217142378781, 1432.8348425167637, 1448.8245517550615, 1448.9178242644389, 1443.975226530761, 1445.229373871925, 1465.8484277326422, 1447.1646478741172, 1479.2030777829475, 1485.8553654028747, 1499.629010977524, 1512.9784403464025, 1519.6268299859519, 1548.7941329024438, 1540.55757885363, 1567.4369252143001, 1596.862320334784, 1605.5522673710632, 1599.666587096205, 1606.0799928071917, 1614.92057971662, 1609.192363983429, 1634.0036848781322, 1616.2292983514906, 1644.9005752616456, 1640.8273640262591, 1659.4786520933674, 1721.1438009121819, 1690.9138186004425, 1735.3070076100423, 1763.958466320212, 1761.5047918640687, 1757.6273491251973, 1780.7512138068898, 1753.3381064880411, 1778.291868913769, 1782.7776872258041, 1768.3697468314226, 1774.318378523884, 1791.5308864754772, 1815.0017038472909 ] }, "h1hand_cube": { "time": [ 0.0, 383.7138327153, 767.4276654306, 1151.1414981459, 1534.8553308612, 1918.5691635765002, 2302.2829962918, 2685.9968290071, 3069.7106617224, 3453.4244944377, 3837.1383271530003, 4220.8521598683, 4604.5659925836, 4988.2798252989, 5371.9936580142, 5755.7074907295, 6139.4213234448, 6523.1351561600995, 6906.8489888754, 7290.5628215907, 7674.276654306001 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0 ], "return": [ 0.0, 13.080046653747559, 17.67292849222819, 21.029696782430012, 22.918834050496418, 24.50208346048991, 26.14434051513672, 35.192575454711914, 47.09134292602539, 60.91011047363281, 78.56637700398763, 158.25643920898438, 161.40049743652344, 135.81275049845377, 156.24981689453125, 225.83556111653647, 166.00001017252603, 213.50342814127603, 196.95799255371094, 205.33394368489584, 221.19602966308594 ], "return_min": [ 0.0, 11.60227465184281, 16.627264717234375, 20.016001116689015, 22.27803754584949, 23.950662436438055, 23.630521485969602, 23.826143545159375, 31.20715640142832, 43.19868665817663, 46.30510159872897, 130.83772424309208, 124.81023014354234, 58.897895471155806, 106.57978648931135, 219.94269343957157, 148.87724150076664, 193.22438191225328, 179.88170809603196, 178.60399498737277, 210.30042873135127 ], "return_max": [ 0.0, 14.557818655652307, 18.718592267222007, 22.04339244817101, 23.559630555143347, 25.053504484541765, 28.658159544303835, 46.55900736426445, 62.97552945062246, 78.621534289089, 110.82765240924628, 185.67515417487667, 197.99076472950452, 212.72760552575173, 205.91984729975115, 231.72842879350137, 183.12277884428542, 233.7824743702988, 214.03427701138992, 232.0638923824189, 232.0916305948206 ] }, "h1hand_bookshelf_simple": { "time": [ 0.0, 607.6057830167999, 1215.2115660335999, 1822.8173490504, 2430.4231320671997, 3038.028915084, 3645.6346981008, 4253.2404811176, 4860.8462641343995, 5468.4520471512, 6076.057830168, 6683.663613184801, 7291.2693962016, 7898.875179218399, 8506.4809622352, 9114.086745252, 9721.692528268799, 10329.298311285598, 10936.9040943024, 11544.5098773192, 12152.115660336 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0 ], "return": [ 0.0, 72.59014129638672, 632.5331013997396, 482.05072021484375, 417.5148289998372, 487.9495900472005, 716.89990234375, 602.3393147786459, 723.6810099283854, 746.5069580078125, 680.5458170572916, 555.5288289388021, 762.16845703125, 761.0180053710938, 791.8561401367188, 807.6997273763021, 812.0292765299479, 816.4413655598959, 816.0659993489584, 819.2063598632812, 821.7978922526041 ], "return_min": [ 0.0, 31.586202163025966, 591.3560061650909, 333.63710951300334, 182.78757131870665, 272.4163623170391, 704.8062688039831, 466.3347552851721, 701.2404371259585, 723.2593912345505, 616.9144495010835, 362.17470111103154, 737.8159270729626, 749.7294135850707, 778.164806689675, 801.360166533209, 804.7945123252052, 799.3652871282668, 800.0447951141915, 800.9554924523965, 804.5781225227375 ], "return_max": [ 0.0, 113.59408042974746, 673.7101966343884, 630.4643309166842, 652.2420866809678, 703.4828177773619, 728.9935358835169, 738.3438742721196, 746.1215827308123, 769.7545247810745, 744.1771846134998, 748.8829567665728, 786.5209869895374, 772.3065971571168, 805.5474735837626, 814.0392882193952, 819.2640407346905, 833.5174439915249, 832.0872035837252, 837.457227274166, 839.0176619824707 ] }, "h1hand_bookshelf_hard": { "time": [ 0.0, 543.7358112938999, 1087.4716225877999, 1631.2074338817, 2174.9432451755997, 2718.6790564694998, 3262.4148677634, 3806.1506790573, 4349.886490351199, 4893.6223016450995, 5437.3581129389995, 5981.0939242329005, 6524.8297355268, 7068.5655468207, 7612.3013581146, 8156.0371694085, 8699.772980702399, 9243.5087919963, 9787.244603290199, 10330.9804145841, 10874.716225877999, 11418.452037171899, 11962.187848465801, 12505.9236597597, 13049.6594710536, 13593.395282347501, 14137.1310936414, 14680.8669049353, 15224.6027162292, 15768.3385275231, 16312.074338817 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0 ], "return": [ 0.0, 115.04310989379883, 414.3397623697917, 392.4563496907552, 384.3321024576823, 445.73602294921875, 444.00734456380206, 507.5265604654948, 502.8351338704427, 563.632080078125, 479.8868408203125, 540.4158935546875, 504.34425862630206, 576.1414082845052, 565.2438761393229, 577.5482991536459, 557.5013427734375, 589.2844645182291, 630.3101806640625, 653.0324300130209, 625.0805460611979, 712.5912882486979, 699.0077311197916, 691.9513346354166, 665.5681762695312, 712.416259765625, 694.9320068359375, 723.7599080403646, 704.8312174479166, 718.2322794596354, 723.0150553385416 ], "return_min": [ 0.0, 43.59245386852548, 332.4218818003097, 299.83306064986766, 259.7826089424531, 356.7439063459728, 399.02623029313395, 478.43961173592857, 480.8452493513443, 529.9966373726622, 431.6378949090861, 494.5170640027713, 422.71565001450017, 521.9929747241146, 534.9803965407618, 513.9383974775631, 536.6718386033316, 543.0112313037442, 612.9851065450109, 646.1570432133583, 611.3883743554222, 687.0628606877813, 676.3860733803567, 672.6699943456148, 627.8278602083814, 698.0179459658565, 658.5159895805781, 707.7512852510698, 689.8015943037292, 706.8565054687476, 705.8978014554126 ], "return_max": [ 0.0, 186.4937659190722, 496.2576429392737, 485.0796387316427, 508.8815959729115, 534.7281395524647, 488.9884588344702, 536.613509195061, 524.8250183895411, 597.2675227835878, 528.1357867315388, 586.3147231066037, 585.972867238104, 630.2898418448959, 595.5073557378839, 641.1582008297287, 578.3308469435434, 635.557697732714, 647.6352547831141, 659.9078168126834, 638.7727177669735, 738.1197158096145, 721.6293888592265, 711.2326749252185, 703.3084923306811, 726.8145735653935, 731.3480240912969, 739.7685308296594, 719.8608405921041, 729.6080534505231, 740.1323092216707 ] }, "h1hand_basketball": { "time": [ 0.0, 302.93849988592, 605.87699977184, 908.8154996577599, 1211.75399954368, 1514.6924994295998, 1817.6309993155198, 2120.56949920144, 2423.50799908736, 2726.44649897328, 3029.3849988591996, 3332.32349874512, 3635.2619986310397, 3938.20049851696, 4241.13899840288, 4544.0774982888, 4847.01599817472, 5149.95449806064, 5452.89299794656, 5755.83149783248, 6058.769997718399, 6361.708497604321, 6664.64699749024, 6967.58549737616, 7270.523997262079, 7573.462497147999, 7876.40099703392, 8179.33949691984, 8482.27799680576, 8785.216496691679, 9088.1549965776, 9391.09349646352, 9694.03199634944, 9996.970496235359, 10299.90899612128, 10602.847496007198, 10905.78599589312, 11208.72449577904, 11511.66299566496, 11814.60149555088, 12117.539995436799, 12420.478495322719, 12723.416995208641, 13026.35549509456, 13329.29399498048, 13632.232494866399, 13935.17099475232, 14238.10949463824, 14541.047994524159, 14843.98649441008, 15146.924994295998 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0, 19840000.0, 20480000.0, 21120000.0, 21760000.0, 22400000.0, 23040000.0, 23680000.0, 24320000.0, 24960000.0, 25600000.0, 26240000.0, 26880000.0, 27520000.0, 28160000.0, 28800000.0, 29440000.0, 30080000.0, 30720000.0, 31360000.0, 32000000.0 ], "return": [ 0.0, 15.71937624613444, 20.09008280436198, 28.070403416951496, 37.13821029663086, 48.140185038248696, 80.68272908528645, 124.40814971923828, 126.71758778889973, 161.22200520833334, 202.98166147867838, 204.3378473917643, 293.4285481770833, 315.3683675130208, 264.8266194661458, 278.62744140625, 291.4298909505208, 324.5673472086589, 380.23256429036456, 394.94849650065106, 298.89170837402344, 292.83021036783856, 401.73631795247394, 375.27061971028644, 386.8580322265625, 303.90362548828125, 453.88189697265625, 383.60216267903644, 323.7223765055339, 264.7662048339844, 407.06358846028644, 348.29319254557294, 415.5738118489583, 385.26619720458984, 416.16399637858075, 491.7182312011719, 459.73570760091144, 295.7273661295573, 305.8920593261719, 537.0791524251302, 566.8707377115885, 511.0398864746094, 371.42539469401044, 502.34424845377606, 437.50067138671875, 296.82629648844403, 467.6920878092448, 347.05420939127606, 577.7560831705729, 553.7951253255209, 528.9782918294271 ], "return_min": [ 0.0, 14.283715681857537, 16.631100081870674, 24.850849036315818, 25.682231879991985, 40.36236009161064, 62.133789054744106, 84.60940729497763, 81.26086178598888, 90.6152240448114, 104.67304686229286, 94.84658618782778, 198.80811217235924, 260.1644753571341, 154.88538218147525, 223.0036764790103, 261.8258052334307, 207.3873303370363, 286.2910705112556, 381.83799263632784, 245.92185692409578, 247.45123796782846, 364.1263718288159, 353.57045366003666, 364.68906452491575, 209.5885532905373, 390.04425163109977, 314.08592505260197, 213.68833766761594, 214.40978111414398, 329.92800186753425, 229.6240291107526, 337.0526110087683, 199.91591366084674, 268.50189022268603, 431.10496585660894, 434.0950085790193, 121.5927500915191, 259.7480611862015, 473.60034355255095, 467.59375605440545, 416.2188627932545, 249.31592931518838, 411.0434362283788, 261.3459217634894, 176.60671178439534, 300.33125904403, 221.02134041025272, 518.1882874852209, 470.53052483418503, 443.62534877420796 ], "return_max": [ 0.0, 17.155036810411342, 23.549065526853287, 31.289957797587174, 48.59418871326973, 55.91800998488675, 99.2316691158288, 164.20689214349892, 172.1743137918106, 231.8287863718553, 301.29027609506386, 313.82910859570086, 388.0489841818074, 370.5722596689075, 374.76785675081635, 334.2512063334897, 321.03397666761094, 441.74736408028144, 474.1740580694735, 408.0590003649743, 351.8615598239511, 338.2091827678487, 439.346264076132, 396.9707857605362, 409.02699992820925, 398.21869768602517, 517.7195423142127, 453.1184003054709, 433.7564153434518, 315.12262855382477, 484.19917505303863, 466.9623559803933, 494.09501268914835, 570.616480748333, 563.8261025344755, 552.3314965457348, 485.3764066228036, 469.86198216759556, 352.03605746614227, 600.5579612977095, 666.1477193687715, 605.8609101559642, 493.5348600728325, 593.6450606791733, 613.6554210099481, 417.04588119249274, 635.0529165744597, 473.0870783722994, 637.3238788559248, 637.0597258168567, 614.3312348846463 ] }, "h1hand_window": { "time": [ 0.0, 375.65864049644, 751.31728099288, 1126.97592148932, 1502.63456198576, 1878.2932024822, 2253.95184297864, 2629.61048347508, 3005.26912397152, 3380.9277644679596, 3756.5864049644, 4132.24504546084, 4507.90368595728, 4883.56232645372, 5259.22096695016, 5634.8796074466, 6010.53824794304, 6386.196888439479, 6761.855528935919, 7137.51416943236, 7513.1728099288, 7888.831450425239, 8264.49009092168, 8640.14873141812, 9015.80737191456, 9391.466012411, 9767.12465290744, 10142.78329340388, 10518.44193390032, 10894.10057439676, 11269.7592148932, 11645.41785538964, 12021.07649588608, 12396.73513638252, 12772.393776878958, 13148.052417375398, 13523.711057871838, 13899.369698368278, 14275.02833886472, 14650.68697936116, 15026.3456198576, 15402.00426035404, 15777.662900850479, 16153.321541346919, 16528.98018184336, 16904.6388223398, 17280.29746283624, 17655.95610333268, 18031.61474382912, 18407.27338432556, 18782.932024822 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0, 19840000.0, 20480000.0, 21120000.0, 21760000.0, 22400000.0, 23040000.0, 23680000.0, 24320000.0, 24960000.0, 25600000.0, 26240000.0, 26880000.0, 27520000.0, 28160000.0, 28800000.0, 29440000.0, 30080000.0, 30720000.0, 31360000.0, 32000000.0 ], "return": [ 0.0, 9.344390233357748, 52.42061551411947, 165.07393646240234, 269.02171834309894, 336.8362223307292, 303.15257771809894, 354.14097086588544, 428.8276774088542, 233.65752283732095, 354.2726542154948, 446.46274820963544, 386.8551534016927, 488.44984944661456, 527.6175537109375, 499.61435953776044, 520.1758015950521, 533.7708740234375, 517.2260945638021, 615.2128092447916, 586.9806111653646, 631.5748697916666, 625.1402994791666, 522.4225260416666, 661.9009399414062, 621.3815714518229, 593.1093546549479, 609.5584716796875, 645.6303304036459, 606.4934692382812, 629.7203369140625, 619.4031575520834, 616.0424601236979, 619.2044474283854, 614.5390421549479, 615.4394938151041, 560.7685038248698, 474.59820556640625, 470.9849739074707, 668.3463948567709, 682.6567586263021, 464.4889322916667, 636.9969889322916, 627.0051981608073, 670.6390177408854, 633.2178141276041, 626.3368123372396, 667.2381591796875, 550.3759155273438, 638.7027587890625, 613.3283081054688 ], "return_min": [ 0.0, 7.793400754588625, 26.974647549456396, 96.90356411186578, 228.96287551323448, 309.31272125827473, 286.3705291802933, 326.8671206873417, 392.2532160264861, 84.46841388823327, 304.0752715628187, 367.63128755485377, 198.75912330174802, 418.10049013236585, 495.99406990234695, 466.59337810178283, 426.0956847450194, 528.027321153031, 427.74345783940487, 588.4670108977814, 572.1403582244072, 621.8291590940654, 580.3812827338901, 411.5124914178696, 624.5023420369643, 602.0108924461694, 561.3460675594077, 584.0818549693116, 585.6577994132044, 524.2934694401896, 556.6414078895589, 597.6325510119733, 578.4353616396595, 535.9672354385903, 548.0507414116869, 510.63892367992145, 495.25338547915845, 273.7842564520836, 166.61370163419144, 594.1346175295835, 654.6452652746367, 289.90572041974156, 555.2077996580139, 529.1499181682586, 639.152777939848, 553.0639730276913, 538.3629737339184, 595.1634994947292, 330.96572081605984, 596.1039158315787, 502.0886536727603 ], "return_max": [ 0.0, 10.895379712126871, 77.86658347878254, 233.24430881293893, 309.0805611729634, 364.35972340318364, 319.93462625590456, 381.4148210444292, 465.4021387912223, 382.84663178640864, 404.4700368681709, 525.2942088644171, 574.9511835016374, 558.7992087608633, 559.241037519528, 532.635340973738, 614.2559184450848, 539.514426893844, 606.7087312881994, 641.9586075918019, 601.820864106322, 641.3205804892679, 669.8993162244432, 633.3325606654637, 699.2995378458482, 640.7522504574764, 624.872641750488, 635.0350883900634, 705.6028613940873, 688.6934690363729, 702.7992659385661, 641.1737640921934, 653.6495586077363, 702.4416594181805, 681.0273428982089, 720.2400639502868, 626.2836221705811, 675.4121546807289, 775.35624618075, 742.5581721839583, 710.6682519779675, 639.0721441635918, 718.7861782065694, 724.8604781533559, 702.1252575419228, 713.371655227517, 714.3106509405609, 739.3128188646458, 769.7861102386277, 681.3016017465463, 724.5679625381772 ] }, "h1hand_spoon": { "time": [ 0.0, 322.2585738862, 644.5171477724, 966.7757216586, 1289.0342955448, 1611.292869431, 1933.5514433172, 2255.8100172034, 2578.0685910896, 2900.3271649758, 3222.585738862 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0 ], "return": [ 0.0, 122.02554003397624, 186.70377604166666, 261.64577229817706, 300.80299886067706, 328.19361368815106, 339.11742146809894, 356.2308349609375, 356.4991963704427, 361.5694986979167, 369.06699625651044 ], "return_min": [ 0.0, 34.19382261198194, 91.68614757119323, 220.42312853910647, 278.48433900289217, 320.38500523770017, 325.4230361788222, 331.20817004445007, 354.7223789896127, 349.114697321542, 364.1910965162153 ], "return_max": [ 0.0, 209.85725745597054, 281.7214045121401, 302.86841605724766, 323.12165871846196, 336.00222213860195, 352.8118067573757, 381.25349987742493, 358.27601375127267, 374.0243000742914, 373.9428959968056 ] }, "h1hand_package": { "time": [ 0.0, 340.7822499229, 681.5644998458, 1022.3467497687001, 1363.1289996916, 1703.9112496145, 2044.6934995374002, 2385.4757494603, 2726.2579993832, 3067.0402493061, 3407.822499229, 3748.6047491519, 4089.3869990748003, 4430.1692489977, 4770.9514989206, 5111.7337488435005, 5452.5159987664, 5793.2982486893, 6134.0804986122, 6474.862748535101, 6815.644998458 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0 ], "return": [ 0.0, -8347.962727864584, -7953.91357421875, -7639.56884765625, -7962.0322265625, -8035.910807291667, -7031.6279296875, -7409.0107421875, -6930.870279947917, -7567.949869791667, -8071.77685546875, -6989.133951822917, -7464.599609375, -7308.195963541667, -7369.9765625, -7056.262532552083, -7344.883463541667, -6283.265950520833, -7419.428873697917, -7106.29931640625, -7487.24951171875 ], "return_min": [ 0.0, -9291.683999288682, -8680.473541162266, -7798.53664407206, -8377.834608732313, -8635.369594101, -7305.690398586479, -7625.490297935072, -7398.502723648596, -7928.4496191424105, -8369.074711589812, -7316.171947275012, -8169.863535365709, -7698.752542894817, -7491.752086731574, -7174.744292881087, -7563.93196904456, -6491.269627424492, -7715.537928880829, -7283.65067627879, -7745.076350527821 ], "return_max": [ 0.0, -7404.241456440486, -7227.353607275234, -7480.60105124044, -7546.229844392687, -7436.452020482336, -6757.565460788521, -7192.531186439928, -6463.237836247238, -7207.4501204409235, -7774.478999347688, -6662.095956370822, -6759.335683384291, -6917.6393841885165, -7248.201038268426, -6937.780772223079, -7125.834958038774, -6075.262273617174, -7123.319818515005, -6928.94795653371, -7229.422672909679 ] }, "h1hand_powerlift": { "time": [ 0.0, 330.8597732899667, 661.7195465799334, 992.5793198699, 1323.4390931598668, 1654.2988664498334, 1985.1586397398, 2316.0184130297666, 2646.8781863197337, 2977.7379596097003, 3308.597732899667, 3639.4575061896335, 3970.3172794796, 4301.177052769567, 4632.036826059533, 4962.896599349499, 5293.756372639467, 5624.616145929434, 5955.475919219401, 6286.335692509367, 6617.195465799334, 6948.055239089301, 7278.915012379267, 7609.774785669233, 7940.6345589592, 8271.494332249167, 8602.354105539134, 8933.2138788291, 9264.073652119067, 9594.933425409034, 9925.793198698999 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0, 13440000.0, 14080000.0, 14720000.0, 15360000.0, 16000000.0, 16640000.0, 17280000.0, 17920000.0, 18560000.0, 19200000.0 ], "return": [ 0.0, 113.51462300618489, 121.9887212117513, 114.68843332926433, 136.49109395345053, 133.07645670572916, 150.48657735188803, 147.2944081624349, 160.43098958333334, 160.48809814453125, 171.36279296875, 173.65789794921875, 175.5028839111328, 205.1602579752604, 244.93697102864584, 211.19544474283853, 298.95766194661456, 304.22186279296875, 269.803716023763, 286.91443888346356, 311.9042460123698, 286.8816223144531, 310.73230997721356, 300.2124938964844, 296.96824137369794, 301.94849650065106, 325.45009358723956, 312.3852844238281, 329.8019307454427, 309.90221150716144, 324.70689900716144 ], "return_min": [ 0.0, 98.60086063853973, 114.55728915382741, 95.01551471352975, 131.31232655463052, 132.52894643766788, 145.7905631000291, 133.60270472550508, 157.63405153985775, 159.3252445285353, 161.9094348876847, 170.73693424256538, 159.80489922254142, 193.6200625634648, 219.08587047631957, 191.63435289037298, 270.29798851974755, 286.12178639622397, 237.38621982654573, 260.68960010790164, 304.83943049232437, 273.19842889543116, 303.7634891685769, 274.43171223168747, 285.8977429185736, 278.42592868276324, 313.4876563980677, 295.64470743599946, 324.5820886809485, 283.7021672876653, 312.297612561625 ], "return_max": [ 0.0, 128.42838537383005, 129.42015326967518, 134.3613519449989, 141.66986135227054, 133.62396697379043, 155.18259160374697, 160.98611159936473, 163.22792762680893, 161.6509517605272, 180.8161510498153, 176.57886165587212, 191.2008685997242, 216.700453387056, 270.7880715809721, 230.75653659530408, 327.6173353734816, 322.32193918971353, 302.2212122209803, 313.1392776590255, 318.96906153241525, 300.5648157334751, 317.70113078585024, 325.9932755612813, 308.03873982882226, 325.4710643185389, 337.4125307764114, 329.1258614116568, 335.02177280993686, 336.1022557266576, 337.1161854526979 ] }, "h1hand_room": { "time": [ 0.0, 465.0374399698, 930.0748799396, 1395.1123199094, 1860.1497598792, 2325.187199849, 2790.2246398188, 3255.2620797886, 3720.2995197584, 4185.3369597282, 4650.374399698, 5115.4118396678, 5580.4492796376, 6045.4867196074, 6510.5241595772, 6975.561599547, 7440.5990395168, 7905.6364794866, 8370.6739194564, 8835.711359426201, 9300.748799396 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0 ], "return": [ 0.0, 13.283664067586264, 16.206466674804688, 15.741510391235352, 17.81355921427409, 17.636091868082683, 20.84869448343913, 21.28005091349284, 29.286340077718098, 36.723029454549156, 55.41912968953451, 85.37582143147786, 111.54248301188152, 156.0373992919922, 161.06229146321616, 165.90640258789062, 175.1955108642578, 181.9734141031901, 174.88094584147134, 183.4038848876953, 176.96692911783853 ], "return_min": [ 0.0, 10.70380901646449, 15.550924139783536, 13.495577650634877, 15.432328173343322, 15.783839514520572, 17.184290172785886, 20.759621257856587, 14.243299280874174, 26.23099362535826, 21.424434200721606, 46.457612262779534, 76.395700351806, 142.86692395305536, 149.31479438123523, 150.45741972383985, 170.95027780126063, 173.86462173894353, 169.2223202556131, 179.6609978535562, 165.3778977118618 ], "return_max": [ 0.0, 15.863519118708037, 16.86200920982584, 17.98744313183583, 20.194790255204857, 19.488344221644795, 24.51309879409237, 21.800480569129093, 44.32938087456202, 47.21506528374005, 89.41382517834741, 124.29403060017619, 146.68926567195703, 169.207874630929, 172.80978854519708, 181.3553854519414, 179.440743927255, 190.08220646743666, 180.53957142732958, 187.14677192183441, 188.55596052381526 ] }, "h1hand_insert_small": { "time": [ 0.0, 315.23139559215, 630.4627911843, 945.6941867764501, 1260.9255823686, 1576.15697796075, 1891.3883735529002, 2206.6197691450498, 2521.8511647372, 2837.08256032935, 3152.3139559215, 3467.5453515136496, 3782.7767471058005, 4098.0081426979505, 4413.2395382900995, 4728.47093388225, 5043.7023294744, 5358.93372506655, 5674.1651206587, 5989.39651625085, 6304.627911843 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0 ], "return": [ 0.0, 25.803948720296223, 65.2731081644694, 110.42922719319661, 124.65948994954427, 151.74050903320312, 158.40826924641928, 164.9043426513672, 167.84961954752603, 170.26196797688803, 165.7256317138672, 150.51225789388022, 159.7950897216797, 169.25322469075522, 170.7041982014974, 162.28902689615884, 171.52276102701822, 173.49874369303384, 164.4512736002604, 162.95905049641928, 171.48112996419272 ], "return_min": [ 0.0, 16.259126578045933, 27.20639444461152, 103.35436633108236, 114.25403470486306, 144.04370182263935, 153.86552855398466, 161.79437375930542, 166.41660146284676, 168.28469510106487, 161.68870399170868, 121.0084021069551, 141.81689245511308, 166.0688410087526, 167.85661523998058, 155.22548705200677, 168.48752737496002, 171.0301362849067, 156.70273979774515, 149.2293200021275, 169.0827938103662 ], "return_max": [ 0.0, 35.348770862546516, 103.33982188432729, 117.50408805531086, 135.0649451942255, 159.4373162437669, 162.9510099388539, 168.01431154342896, 169.2826376322053, 172.2392408527112, 169.7625594360257, 180.01611368080532, 177.7732869882463, 172.43760837275784, 173.55178116301423, 169.35256674031092, 174.55799467907642, 175.967351101161, 172.19980740277566, 176.68878099071105, 173.87946611801922 ] }, "h1hand_insert_normal": { "time": [ 0.0, 353.9304402593, 707.8608805186, 1061.7913207779, 1415.7217610372, 1769.6522012965002, 2123.5826415558, 2477.5130818151, 2831.4435220744, 3185.3739623336996, 3539.3044025930003, 3893.2348428523, 4247.1652831116, 4601.0957233709, 4955.0261636302, 5308.9566038895, 5662.8870441488, 6016.8174844080995, 6370.747924667399, 6724.6783649267, 7078.608805186001 ], "env_step": [ 0.0, 640000.0, 1280000.0, 1920000.0, 2560000.0, 3200000.0, 3840000.0, 4480000.0, 5120000.0, 5760000.0, 6400000.0, 7040000.0, 7680000.0, 8320000.0, 8960000.0, 9600000.0, 10240000.0, 10880000.0, 11520000.0, 12160000.0, 12800000.0 ], "return": [ 0.0, 24.10168711344401, 54.070421854654946, 99.81527709960938, 135.47543080647787, 130.91323852539062, 152.1724065144857, 162.59126790364584, 164.9534657796224, 168.67947896321616, 168.1095987955729, 168.2154744466146, 174.81605529785156, 176.54434204101562, 179.82152811686197, 179.19852193196616, 182.65686543782553, 184.3785146077474, 187.4399668375651, 185.9150187174479, 191.98021443684897 ], "return_min": [ 0.0, 6.769507504418215, 38.73171288128932, 81.49871730558638, 113.11107571013979, 99.97948537079725, 130.3948495447914, 159.98263262262466, 163.80269122995844, 166.81903211046978, 163.45338885046095, 160.9069368380094, 169.2923240593455, 169.42940571808268, 168.66802504056085, 164.02913103172816, 165.1513139904467, 169.5900424483381, 170.5853867351582, 166.09199645001894, 172.27076893820444 ], "return_max": [ 0.0, 41.43386672246981, 69.40913082802058, 118.13183689363237, 157.83978590281595, 161.846991679984, 173.94996348417996, 165.19990318466702, 166.10424032928637, 170.53992581596253, 172.76580874068486, 175.5240120552198, 180.33978653635762, 183.65927836394857, 190.9750311931631, 194.36791283220415, 200.16241688520435, 199.16698676715671, 204.29454693997198, 205.73804098487687, 211.6896599354935 ] } } ================================================ FILE: data/isaaclab_result.json ================================================ { "Isaac-Velocity-Flat-G1-v0": { "FastTD3": { "time": [ 0.0, 426.030303030303, 852.060606060606, 1278.090909090909, 1704.121212121212, 2130.151515151515, 2556.181818181818, 2982.212121212121, 3408.242424242424, 3834.2727272727275, 4260.30303030303, 4686.333333333333, 5112.363636363636, 5538.393939393939, 5964.424242424242, 6390.454545454545, 6816.484848484848, 7242.515151515152, 7668.545454545455, 8094.575757575758, 8520.60606060606, 8946.636363636364, 9372.666666666666, 9798.69696969697, 10224.727272727272, 10650.757575757576, 11076.787878787878, 11502.818181818182, 11928.848484848484, 12354.878787878788, 12780.90909090909, 13206.939393939394, 13632.969696969696, 14059.0 ], "env_step": [ 0, 20480000.0, 40960000.0, 61440000.0, 81920000.0, 102400000.0, 122880000.0, 143360000.0, 163840000.0, 184320000.0, 204800000.0, 225280000.0, 245760000.0, 266240000.0, 286720000.0, 307200000.0, 327680000.0, 348160000.0, 368640000.0, 389120000.0, 409600000.0, 430080000.0, 450560000.0, 471040000.0, 491520000.0, 512000000.0, 532480000.0, 552960000.0, 573440000.0, 593920000.0, 614400000.0, 634880000.0, 655360000.0, 675840000.0 ], "return": [ 0.0, 8.399159749348959, 9.912333170572916, 14.84921375910441, 18.012757539749146, 22.504486719767254, 27.125051498413086, 28.701950073242188, 30.197755813598633, 31.469010670979817, 32.14909235636393, 32.57625071207682, 32.17204030354818, 32.481005350748696, 30.93257013956706, 32.5035769144694, 32.88990084330241, 31.94870122273763, 32.73742167154948, 32.58588981628418, 32.65592384338379, 30.25904655456543, 32.79962031046549, 32.60353914896647, 32.388885498046875, 32.74258168538412, 30.75256411234538, 32.113457997639976, 30.90188471476237, 30.882394790649414, 31.751652399698894, 32.93280283610026, 32.285691579182945, 32.544677734375 ], "return_min": [ 0.0, -0.4342708013469121, 0.15131576165535776, 3.056790615527323, 4.690397268276504, 13.184447519334512, 21.25279365145474, 22.70949840620638, 25.157422436625925, 27.96643827593808, 29.69758316302085, 30.56877892180152, 29.76787859060772, 30.696770112276443, 28.644023779533903, 30.465339016626782, 30.92349475362472, 29.35783947935273, 30.895364911595788, 31.14117205982312, 31.353408955374725, 27.429461255702815, 31.54753402208512, 31.30661763069416, 31.116279961417973, 31.529114984883087, 29.048792317035463, 30.636109312623667, 29.16179985474695, 29.983837058445577, 29.91569292592529, 32.48186031918378, 31.36364585744209, 32.12861466393829 ], "return_max": [ 0.0, 17.23259030004483, 19.673350579490474, 26.641636902681498, 31.335117811221785, 31.824525920199996, 32.99730934537143, 34.694401740277996, 35.238089190571344, 34.97158306602155, 34.60060154970701, 34.58372250235212, 34.57620201648864, 34.265240589220944, 33.221116499600214, 34.54181481231202, 34.85630693298009, 34.539562966122524, 34.57947843150316, 34.03060757274524, 33.95843873139285, 33.08863185342805, 34.05170659884587, 33.900460667238775, 33.66149103467578, 33.95604838588515, 32.456335907655294, 33.590806682656286, 32.6419695747778, 31.78095252285325, 33.5876118734725, 33.383745353016735, 33.207737300923796, 32.96074080481171 ] }, "null": { "time": [ 0.0, 4.206, 8.412, 12.618, 16.824, 21.03, 25.236, 29.442, 33.648, 37.854, 42.06, 46.266, 50.472, 54.678, 58.884, 63.09, 67.296, 71.502, 75.708, 79.914, 84.12, 88.326, 92.532, 96.738, 100.944, 105.15, 109.356, 113.562, 117.768, 121.974, 126.18, 130.386, 134.592, 138.798, 143.004, 147.21, 151.416, 155.622, 159.828, 164.034, 168.24, 172.446, 176.652, 180.858, 185.064, 189.27, 193.476, 197.682, 201.888, 206.094, 210.3, 214.506, 218.712, 222.918, 227.124, 231.33, 235.536, 239.742, 243.948, 248.154, 252.36, 256.566, 260.772, 264.978, 269.184, 273.39, 277.596, 281.802, 286.008, 290.214, 294.42, 298.626, 302.832, 307.038, 311.244, 315.45, 319.656, 323.862, 328.068, 332.274, 336.48, 340.686, 344.892, 349.098, 353.304, 357.51, 361.716, 365.922, 370.128, 374.334, 378.54, 382.746, 386.952, 391.158, 395.364, 399.57, 403.776, 407.982, 412.188, 416.394, 420.6, 424.806, 429.012, 433.218, 437.424, 441.63, 445.836, 450.042, 454.248, 458.454, 462.66, 466.866, 471.072, 475.278, 479.484, 483.69, 487.896, 492.102, 496.308, 500.514, 504.72, 508.926, 513.132, 517.338, 521.544, 525.75, 529.956, 534.162, 538.368, 542.574, 546.78, 550.986, 555.192, 559.398, 563.604, 567.81, 572.016, 576.222, 580.428, 584.634, 588.84, 593.046, 597.252, 601.458, 605.664, 609.87, 614.076, 618.282, 622.488, 626.694, 630.9, 635.106, 639.312, 643.518, 647.724, 651.93, 656.136, 660.342, 664.548, 668.754, 672.96, 677.166, 681.372, 685.578, 689.784, 693.99, 698.196, 702.402, 706.608, 710.814, 715.02, 719.226, 723.432, 727.638, 731.844, 736.05, 740.256, 744.462, 748.668, 752.874, 757.08, 761.286, 765.492, 769.698, 773.904, 778.11, 782.316, 786.522, 790.728, 794.934, 799.14, 803.346, 807.552, 811.758, 815.964, 820.17, 824.376, 828.582, 832.788, 836.994, 841.2, 845.406, 849.612, 853.818, 858.024, 862.23, 866.436, 870.642, 874.848, 879.054, 883.26, 887.466, 891.672, 895.878, 900.084, 904.29, 908.496, 912.702, 916.908, 921.114, 925.32, 929.526, 933.732, 937.938, 942.144, 946.35, 950.556, 954.762, 958.968, 963.174, 967.38, 971.586, 975.792, 979.998, 984.204, 988.41, 992.616, 996.822, 1001.028, 1005.234, 1009.44, 1013.646, 1017.852, 1022.058, 1026.264, 1030.47, 1034.676, 1038.882, 1043.088, 1047.294, 1051.5, 1055.706, 1059.912, 1064.118, 1068.324, 1072.53, 1076.736, 1080.942, 1085.148, 1089.354, 1093.56, 1097.766, 1101.972, 1106.178, 1110.384, 1114.59, 1118.796, 1123.002, 1127.208, 1131.414, 1135.62, 1139.826, 1144.032, 1148.238, 1152.444, 1156.65, 1160.856, 1165.062, 1169.268, 1173.474, 1177.68, 1181.886, 1186.092, 1190.298, 1194.504, 1198.71, 1202.916, 1207.122, 1211.328, 1215.534, 1219.74, 1223.946, 1228.152, 1232.358, 1236.564, 1240.77, 1244.976, 1249.182, 1253.388, 1257.594, 1261.8, 1266.006, 1270.212, 1274.418, 1278.624, 1282.83, 1287.036, 1291.242, 1295.448, 1299.654, 1303.86, 1308.066, 1312.272, 1316.478, 1320.684, 1324.89, 1329.096, 1333.302, 1337.508, 1341.714, 1345.92, 1350.126, 1354.332, 1358.538, 1362.744, 1366.95, 1371.156, 1375.362, 1379.568, 1383.774, 1387.98, 1392.186, 1396.392, 1400.598, 1404.804, 1409.01, 1413.216, 1417.422, 1421.628, 1425.834, 1430.04, 1434.246, 1438.452, 1442.658, 1446.864, 1451.07, 1455.276, 1459.482, 1463.688, 1467.894, 1472.1, 1476.306, 1480.512, 1484.718, 1488.924, 1493.13, 1497.336, 1501.542, 1505.748, 1509.954, 1514.16, 1518.366, 1522.572, 1526.778, 1530.984, 1535.19, 1539.396, 1543.602, 1547.808, 1552.014, 1556.22, 1560.426, 1564.632, 1568.838, 1573.044, 1577.25, 1581.456, 1585.662, 1589.868, 1594.074, 1598.28, 1602.486, 1606.692, 1610.898, 1615.104, 1619.31, 1623.516, 1627.722, 1631.928, 1636.134, 1640.34, 1644.546, 1648.752, 1652.958, 1657.164, 1661.37, 1665.576, 1669.782, 1673.988, 1678.194, 1682.4, 1686.606, 1690.812, 1695.018, 1699.224, 1703.43, 1707.636, 1711.842, 1716.048, 1720.254, 1724.46, 1728.666, 1732.872, 1737.078, 1741.284, 1745.49, 1749.696, 1753.902, 1758.108, 1762.314, 1766.52, 1770.726, 1774.932, 1779.138, 1783.344, 1787.55, 1791.756, 1795.962, 1800.168, 1804.374, 1808.58, 1812.786, 1816.992, 1821.198, 1825.404, 1829.61, 1833.816, 1838.022, 1842.228, 1846.434, 1850.64, 1854.846, 1859.052, 1863.258, 1867.464, 1871.67, 1875.876, 1880.082, 1884.288, 1888.494, 1892.7, 1896.906, 1901.112, 1905.318, 1909.524, 1913.73, 1917.936, 1922.142, 1926.348, 1930.554, 1934.76, 1938.966, 1943.172, 1947.378, 1951.584, 1955.79, 1959.996, 1964.202, 1968.408, 1972.614, 1976.82, 1981.026, 1985.232, 1989.438, 1993.644, 1997.85, 2002.056, 2006.262, 2010.468, 2014.674, 2018.88, 2023.086, 2027.292, 2031.498, 2035.704, 2039.91, 2044.116, 2048.322, 2052.528, 2056.734, 2060.94, 2065.146, 2069.352, 2073.558, 2077.764, 2081.97, 2086.176, 2090.382, 2094.588, 2098.794, 2103.0 ], "env_step": [ 0, 294912, 393216, 491520, 589824, 786432, 884736, 983040, 1081344, 1376256, 1671168, 1867776, 1966080, 2260992, 2457600, 2850816, 3145728, 3342336, 3735552, 3833856, 4521984, 5013504, 5210112, 5308416, 5603328, 5898240, 6094848, 6389760, 6488064, 6684672, 7274496, 7372800, 7569408, 7667712, 7962624, 8749056, 9043968, 9633792, 9830400, 9928704, 10223616, 10321920, 10518528, 10715136, 10813440, 11010048, 11206656, 11599872, 11698176, 12582912, 12976128, 13271040, 13664256, 13762560, 13860864, 14057472, 14155776, 14352384, 14843904, 15237120, 15335424, 15630336, 15728640, 16416768, 16613376, 17104896, 17301504, 17793024, 17891328, 17989632, 18087936, 18284544, 19169280, 19267584, 19365888, 19562496, 19857408, 19955712, 20545536, 20643840, 20742144, 20938752, 21037056, 21331968, 21430272, 21528576, 21626880, 22216704, 22315008, 22413312, 23101440, 23199744, 23691264, 24084480, 24772608, 24969216, 25165824, 25559040, 25853952, 25952256, 26247168, 26345472, 26443776, 26935296, 27033600, 27328512, 27623424, 27918336, 28213248, 28311552, 28409856, 28704768, 28901376, 29294592, 30375936, 30572544, 30670848, 31260672, 31457280, 31555584, 31752192, 32047104, 32440320, 32538624, 32636928, 32833536, 33030144, 33619968, 33718272, 33816576, 33914880, 35192832, 35389440, 35487744, 35586048, 36274176, 36372480, 36470784, 36667392, 37158912, 37650432, 38043648, 38731776, 38928384, 39223296, 39616512, 39714816, 40009728, 40206336, 40894464, 41091072, 41189376, 41680896, 41779200, 41975808, 42172416, 42270720, 42369024, 42467328, 42958848, 43155456, 43352064, 43548672, 43745280, 43941888, 45023232, 45711360, 46104576, 46202880, 46301184, 46792704, 47480832, 47579136, 47677440, 47874048, 48168960, 48758784, 49053696, 49250304, 49348608, 49446912, 49643520, 49938432, 50331648, 50429952, 50626560, 51019776, 52002816, 52297728, 52887552, 53084160, 53182464, 53477376, 53772288, 53870592, 54067200, 54460416, 54558720, 55640064, 55934976, 56328192, 56426496, 56623104, 57212928, 58884096, 59375616, 59670528, 60260352, 60850176, 60948480, 61734912, 61833216, 62029824, 62521344, 62619648, 63406080, 63504384, 63799296, 64094208, 64290816, 64585728, 65077248, 65372160, 65470464, 65667072, 65863680, 66453504, 66650112, 66846720, 67239936, 67436544, 68517888, 68812800, 69304320, 69402624, 69894144, 70287360, 70385664, 70483968, 70582272, 70975488, 71073792, 71172096, 71270400, 71663616, 71958528, 72253440, 72548352, 72646656, 72843264, 73039872, 73236480, 73334784, 73728000, 74219520, 74317824, 74514432, 74907648, 75005952, 75399168, 75595776, 75988992, 76382208, 76578816, 76677120, 77070336, 77365248, 77561856, 77758464, 78053376, 78544896, 78643200, 78839808, 78938112, 79429632, 79724544, 79822848, 79921152, 80019456, 80510976, 80707584, 81297408, 81494016, 81690624, 82378752, 82870272, 84049920, 84148224, 84443136, 85917696, 86212608, 86310912, 86409216, 86507520, 86605824, 86704128, 86802432, 86999040, 87097344, 87392256, 87490560, 87588864, 87785472, 88276992, 88768512, 88866816, 89063424, 89456640, 89554944, 89849856, 89948160, 90046464, 90144768, 90439680, 90931200, 91029504, 91127808, 91619328, 91815936, 92110848, 92209152, 92504064, 92798976, 93388800, 93487104, 93683712, 93880320, 93978624, 94470144, 95256576, 95354880, 95649792, 95748096, 95846400, 96141312, 96239616, 96337920, 96829440, 96927744, 97320960, 97517568, 97615872, 97714176, 98304000, 98697216, 99090432, 99287040, 99385344, 99778560, 99876864, 100073472, 100171776, 100368384, 100564992, 100958208, 101154816, 101253120, 101744640, 101842944, 102137856, 102334464, 102432768, 102825984, 102924288, 103120896, 103219200, 103710720, 103809024, 103907328, 104103936, 104398848, 105185280, 105381888, 105480192, 105971712, 106070016, 106659840, 106758144, 106856448, 106954752, 107053056, 107249664, 107347968, 107446272, 107839488, 108232704, 108429312, 108822528, 109314048, 109707264, 109805568, 110100480, 110198784, 110788608, 110985216, 111083520, 111280128, 111378432, 111575040, 112164864, 112656384, 112754688, 112852992, 113541120, 114229248, 114327552, 114524160, 114622464, 114917376, 115015680, 115212288, 115900416, 116195328, 116490240, 117080064, 117178368, 117374976, 117768192, 118456320, 119734272, 120127488, 120324096, 120422400, 121012224, 121405440, 121602048, 121700352, 122191872, 122388480, 123076608, 123568128, 123666432, 124256256, 124354560, 124649472, 124747776, 124944384, 125239296, 125337600, 125632512, 125927424, 126418944, 126517248, 126713856, 127401984, 128188416, 128286720, 128581632, 128876544, 129564672, 130056192, 131137536, 131334144, 131530752, 132120576, 132218880, 132513792, 133005312, 133890048, 134086656, 134184960, 134774784, 135364608, 135561216, 136249344, 136347648, 136544256, 136740864, 137527296, 137625600, 138215424, 138412032, 138608640, 139001856, 139100160, 139198464, 139296768, 139395072, 139493376, 139984896, 140181504, 140279808, 141066240, 141656064, 141852672, 142049280, 142540800, 142737408, 143032320, 143228928, 143818752, 143917056, 144113664, 144211968, 144900096, 145391616, 146178048, 146276352, 146374656, 146767872, 146964480 ], "return": [ 0.0, -4.244887483882465, -6.062188769976298, -5.946233414014181, -5.934175154368082, -5.931420200665792, -5.926097372372946, -5.890506885846456, -5.845593395233155, -5.745785452524821, -5.6373031155268345, -5.537198613484701, -5.437902429898579, -5.354564905166626, -5.315855964024862, -5.305097824732463, -5.233588047027588, -5.21569048722585, -5.142245790163676, -5.169298741022746, -5.144494528770447, -5.092627458572388, -5.067925599416097, -5.0944153579076135, -5.075692962010701, -5.080377626419067, -5.094903222719828, -5.125536025365193, -5.115845588048299, -5.232694799105326, -5.326773929595947, -5.540815292994181, -5.565265787442525, -5.767993720372519, -6.0354754002889, -6.422926289240519, -6.6024469566345205, -7.196066060066222, -7.374582826296488, -7.664462378819784, -8.25899796485901, -8.364871314366658, -8.789411764144896, -8.797979049682617, -9.422961320877077, -9.468292330106099, -9.801179609298705, -9.881031123797099, -10.184216540654502, -10.079860796928406, -9.81778653462728, -9.665794699986776, -9.04004868030548, -7.917503507932028, -6.944561026891073, -6.476619219779969, -6.198991281986236, -5.729001940091451, -4.624928730726242, -3.9058311102787653, -3.415297171473503, -3.0760760113100214, -2.5505499145636956, -1.384848330305734, -0.2722055218198026, 0.37754478539495423, 0.9910718053362021, 1.9497537283304458, 2.0353074125107375, 2.7150473963220914, 3.57238434004287, 4.393549767136574, 5.22253050327301, 5.594326151609422, 5.980829033851624, 6.248111243247986, 6.301437652111052, 6.819287878672282, 7.668490665753683, 7.9056279023488365, 8.528459188143414, 8.9157959318161, 9.004425404866536, 9.240382947921754, 10.005460894107818, 10.388603275616964, 10.506964916388194, 11.041883727709452, 11.13649615128835, 11.725313129425048, 12.456134204069775, 12.757048015594483, 13.031923011144002, 13.684242639541628, 14.265029729207356, 14.391678101221721, 14.820610736211142, 15.028353486855826, 15.586650098164876, 16.098702065149944, 16.480080706278486, 16.47846872806549, 16.60223277727763, 16.996064742406208, 16.962005354563395, 17.414829864501954, 17.879654438495635, 18.075921009381613, 18.59587174097697, 18.768117705980938, 19.066952754656473, 19.140114987691245, 19.385992137591046, 19.82112768173218, 20.07494811375936, 20.38481765429179, 20.64120300292969, 20.944377085367837, 21.046505676905316, 21.219120070139567, 21.259444128672282, 21.437351309458418, 21.490676719347636, 21.614972871144612, 21.73858602523804, 21.836027539571123, 21.938353093465167, 22.12277064641317, 22.096696147918703, 22.379330412546793, 22.3636573155721, 22.667728048960367, 22.804457734425863, 22.989772860209147, 22.81014023224513, 22.752846819559732, 22.87626227140427, 23.268132722377782, 23.450477498372397, 23.49831963221232, 23.642868830362957, 23.749657942454018, 23.829254946708676, 23.89071097215017, 23.95269861221313, 23.80049831072489, 23.97088320493698, 24.035058199564617, 23.987095406850177, 24.143753560384113, 24.161667313575744, 24.20282023747762, 24.515907077789308, 24.48859402338664, 24.65326049168905, 24.695785293579103, 24.68093146642049, 24.64084663550059, 24.774629052480062, 24.721785335540773, 24.80927684466044, 24.64293538570404, 24.611320269902546, 24.890653877258302, 25.03125279744466, 25.04939617792765, 25.056911741892495, 25.012861070632937, 25.006108971436817, 25.19971148808797, 25.244278527895606, 25.243163836797077, 25.20921319325765, 25.25394191265106, 25.238686189651492, 25.24318765004476, 25.405525760650633, 25.469966049194337, 25.493687133789063, 25.4357274723053, 25.59193923314412, 25.664110005696614, 25.589963239034017, 25.78192579905192, 25.539709254900615, 25.59874127229055, 25.71211055119832, 25.83485471089681, 25.849721884727483, 25.78178211212158, 25.905887908935544, 26.03030003229777, 25.978182222048442, 26.085983327229815, 26.114759820302325, 26.203529485066735, 26.11474272966385, 26.242303619384767, 26.12391219774882, 25.88732498884201, 26.3737934366862, 26.40230457305908, 26.32623062769572, 26.447194538116452, 26.39948532104492, 26.48303238550822, 26.58627212524414, 26.62195227940877, 26.305497008959453, 26.5571306848526, 26.676981925964355, 26.715099360148113, 26.7491853205363, 26.77513189315796, 26.77031274795532, 26.699470434188843, 26.697296330134076, 26.761435826619465, 26.773332411448163, 26.817168051401776, 26.812312013308212, 26.730486590067546, 26.980856755574546, 26.768192443052925, 26.75973533630371, 26.726683316230776, 26.629371005694072, 26.737455078760785, 27.04755596160889, 26.996712977091473, 27.15843843460083, 27.056537364323933, 27.07920110066732, 27.063170181910195, 26.999029286702477, 26.95050971984863, 27.028388187090556, 27.162973842620847, 27.03351854960124, 26.927428197860717, 27.040847080548605, 27.197179768880208, 27.07626317977905, 27.213608862559, 27.158017387390135, 27.26690890630086, 27.162100307146705, 27.21130386193593, 27.205319883028665, 27.24621044794718, 27.22654726187388, 27.380611095428467, 27.297355159123736, 27.317657216389975, 27.35010398864746, 27.37887262980143, 27.204053241411845, 27.41668393452962, 27.38345681508382, 27.39409785588582, 27.458933315277097, 27.5122291692098, 27.437938969930016, 27.426795717875166, 27.50698534965515, 27.471194496154784, 27.481666673024495, 27.474861726760864, 27.534579620361328, 27.485984223683676, 27.63851408004761, 27.601514314015706, 27.468947197596233, 27.463055524826046, 27.63535165786743, 27.540319763819372, 27.61072525024414, 27.618154637018836, 27.742102012634277, 27.669274317423504, 27.588603134155274, 27.560866799354557, 27.558601020177207, 27.617678836186727, 27.334658497174576, 27.753905919392903, 27.710659249623617, 27.830444075266524, 27.792324670155846, 27.715141557057695, 27.80116782506307, 27.822862389882403, 27.925147520701092, 27.828692003885905, 27.774182446797692, 27.82865638097127, 27.87010919570923, 27.88524030049642, 27.780755894978842, 27.869576110839844, 27.8847115389506, 27.88112828572591, 27.95226758956909, 27.9632572110494, 27.98185961405436, 27.900285018285114, 27.749773004849754, 27.95440636952718, 27.897387890815736, 27.774074436823526, 27.851776852607728, 27.88595195611318, 27.957040735880536, 28.01861265818278, 28.072463569641116, 27.927986801465355, 28.054341169993084, 28.04882469813029, 28.08775482813517, 28.013588581085205, 27.971034762064615, 27.939692214330037, 27.961951595942182, 27.91120104789734, 27.912012211481727, 27.946552640597023, 28.13232330322266, 28.155323626200357, 28.21534303029378, 28.105512018203736, 28.12738781611124, 28.18438720703125, 28.129107367197673, 28.202164802551266, 28.183705285390218, 28.11675905227661, 28.143270753224694, 28.115149936676023, 27.962960600852966, 28.082834936777747, 28.17713045756022, 28.071439940134685, 28.036277154286704, 28.015016555786133, 27.74030689318975, 28.143797952334086, 28.083258043924967, 28.143300120035807, 28.22578639984131, 28.189498087565102, 28.28595506032308, 28.153611634572346, 28.271268889109294, 28.1683358446757, 28.215656159718833, 28.146134676933286, 28.001276264190675, 27.983188803990686, 27.911292719841004, 28.1467857392629, 28.130204054514564, 28.11707345326742, 28.18377244313558, 28.046776949564617, 28.17007304509481, 28.181098283131917, 28.141192302703857, 28.183331216176352, 28.045621960957845, 28.206850026448564, 28.094409753481546, 28.100912955602013, 28.095957455635073, 28.08301568508148, 27.989888830184935, 27.994859463373817, 28.122985763549806, 28.221680157979325, 28.20320109049479, 28.07082467556, 28.08045124053955, 28.234146792093913, 28.24658082962036, 28.167981278101603, 28.19567700703939, 28.181134967803956, 28.086012459596002, 28.175039056142168, 28.172370891571045, 28.007196400960286, 28.25803017298381, 28.184969975153606, 28.25297903696696, 28.171354506810506, 28.238894797960914, 28.285828113555908, 27.95717629591624, 28.18456648508708, 28.211361389160157, 28.155993226369223, 28.133868006070454, 28.174914951324464, 28.19500195185343, 28.166361468633017, 27.803325554529824, 27.919510776996614, 28.102442877292635, 28.161700401306152, 28.24873664855957, 28.270584068298344, 28.14114588101705, 28.200696325302122, 28.242460556030277, 28.08400403817495, 28.200781688690185, 28.170070962905882, 28.197558924357097, 28.121695067087813, 27.968536138534546, 28.11471136728922, 28.243860149383547, 28.14787934621175, 28.204611466725666, 28.119738744099934, 28.031441315015158, 27.99424771149953, 28.260030975341795, 28.055403108596803, 28.15147446632385, 28.236702639261882, 28.21238270441691, 28.137156661351522, 28.181380929946897, 28.15697519938151, 28.175251019795738, 28.244781970977783, 28.278363844553628, 28.293849919637044, 28.27718382517497, 28.177998784383135, 28.196655069986978, 28.221141552925115, 28.249198462168376, 28.194035822550457, 28.266193669637047, 28.199925409952797, 28.2498019917806, 28.1529238319397, 28.29481091181437, 28.126179259618127, 28.218130435943603, 28.13479091326396, 28.208430185317994, 28.240509795347847, 28.256366844177247, 28.31536688486735, 28.207336940765384, 28.345287787119545, 28.27900697072347, 27.943899685541783, 27.870618518193563, 28.060490986506142, 28.1964542388916, 28.145226942698162, 28.134871492385866, 28.11652568499247, 28.205403124491372, 28.25891178131103, 28.224900798797606, 28.18699549039205, 28.245219739278156, 27.95408559640249, 28.04483809630076, 28.184945538838704, 28.124226392110188, 28.126491333643596, 28.222030410766603, 28.277731647491454, 28.204816850026447, 28.326336828867596, 28.27530563990275, 28.337517782847087, 28.272301177978516, 28.169770604769386, 28.10294930299123, 28.11485408782959, 28.196449371973674, 28.230150807698564, 28.27503276189168, 28.39388942718506, 28.159951156775154, 28.471651503245038, 28.29022908528646, 28.39702470143636, 28.33332508722941, 28.344238732655842, 28.264009912808735, 28.330615717569987, 28.14671749909719, 28.083581444422403, 28.162535241444903 ], "return_min": [ 0.0, -6.643669384840878, -6.182021783308077, -5.991857294835561, -6.01209477450521, -5.9483798211704855, -5.999782261850072, -5.916902296139923, -5.886733559999383, -5.808524865018686, -5.767673070588338, -5.656879782210418, -5.516316668851651, -5.4007619174378085, -5.372779064718162, -5.3428969204069, -5.253152293681659, -5.233328113111429, -5.161262128656727, -5.182595984476208, -5.185557630198057, -5.129352151903135, -5.092037379164627, -5.131577815601254, -5.093579424569167, -5.093719788086104, -5.128770606785492, -5.164726254562823, -5.158627605712601, -5.348331700371459, -5.433969219755715, -5.692475061982907, -5.661098991347332, -6.019793748866466, -6.3892772787774526, -6.618471670782988, -6.93623582771524, -7.783379441949263, -7.772743356571364, -8.374310306500192, -8.383183613218646, -8.739301357337679, -9.011180633645404, -9.167613726500848, -9.546097485889467, -9.830183506287723, -10.401676641230377, -10.383695511002582, -10.622962101795936, -10.417617680809428, -10.212604744756401, -10.430187174747314, -10.060513521247342, -8.386746572083084, -7.483890396900349, -6.769170298951451, -6.563200095765459, -6.363015656832963, -5.314321983518964, -4.321166917783934, -3.8361483530772995, -3.3285683618568154, -2.997812024835769, -2.1293781494344417, -0.5802740736090363, -0.2522531694659173, 0.44993455353870715, 1.1269001717729363, 1.2624771886189383, 2.242960541089087, 2.4963479947590113, 3.0845735431593404, 3.9279124633414924, 4.763775894213173, 5.049968982159641, 5.389919698911818, 5.48248416097672, 6.229347854311914, 6.521668988071955, 6.659136206112889, 7.427314258358753, 7.959931882256278, 8.028050845313581, 8.24168761513716, 9.345870248729108, 9.602207436283068, 9.661819458045514, 10.36769504799724, 10.479651576379243, 10.918097477003705, 11.49926771582258, 11.726100767383969, 12.017986155957812, 13.050080426953622, 13.706833546223123, 13.771967969075195, 13.830807728673413, 14.210788794303577, 14.74650412027818, 15.154901166861874, 15.427889498076768, 15.445720007976437, 15.42139226603878, 15.934492457847714, 15.834368998414789, 16.39698386013562, 16.947691591248272, 16.807375863238427, 17.612806679277735, 17.842858557229146, 17.8214759601972, 17.97937647179555, 18.227829245354723, 18.7644055402232, 19.325366890517444, 19.559545910473766, 19.630990604684314, 20.052528700425622, 20.07969423110993, 20.24338282719667, 20.316218292810056, 20.54437276683007, 20.61879250350746, 20.737601855773278, 20.842643896008965, 20.960295805924897, 21.05675283045511, 21.38516463895171, 21.29067986749454, 21.551385627548925, 21.53450581143342, 22.106338660705283, 22.277585385218103, 22.343014598444217, 22.336837392663952, 21.891858139567134, 21.873949149291946, 22.42005171861463, 22.791150529503188, 22.91568567873898, 23.103328667240632, 23.274285177687627, 23.381478304066786, 23.500560268083092, 23.567086975556506, 23.514825951037885, 23.69441051835588, 23.67786951187934, 23.70868130951907, 23.82123286847361, 23.921364629682458, 23.864665666102454, 24.240071066132046, 24.171078317636052, 24.294435932577716, 24.315215024411746, 24.429722664880202, 24.309891323003807, 24.358354925393265, 24.293342169139848, 24.524251045267125, 24.410533051064895, 24.259007268772475, 24.68031418108018, 24.6622536418055, 24.7900871663593, 24.839123934450377, 24.604940968948597, 24.678304904444722, 24.83199777097151, 24.893924012460136, 25.025322863275832, 25.006304955399667, 24.927603177713035, 24.870433418062774, 24.8206808457444, 25.042394627792078, 25.12909070901515, 25.127701577023146, 24.909385724601925, 25.26391796949775, 25.36489524550621, 25.262896738352463, 25.4065724808923, 25.235586585718753, 25.25165776554294, 25.21133184941777, 25.382838458944565, 25.291931799688506, 25.27953037736325, 25.45371787444146, 25.53356545650795, 25.659911682303985, 25.61246706370948, 25.67222761174437, 25.72419865409862, 25.691071648603913, 25.770755480440684, 25.503872557466057, 25.09439470110851, 25.94571608744351, 25.953143316854963, 25.715659588443316, 25.933383297310503, 25.944354522810634, 26.08021549674279, 26.162014760788978, 26.199008962618258, 25.834289088400073, 25.974639007126676, 26.167551596823426, 26.250286619672007, 26.287575915584835, 26.28088931506064, 26.365906335455993, 26.081784122112968, 26.129397360813545, 26.259239788514925, 26.150394042457304, 26.291752596593632, 26.313943131522063, 26.13410198199206, 26.55317314286874, 26.216099134936506, 26.380888321629403, 26.255848946469214, 25.980031272338643, 26.045063632168027, 26.6120241458033, 26.665698145218652, 26.699119141354032, 26.569793952169537, 26.58967941825657, 26.739340476993025, 26.330084638992602, 26.42094707654985, 26.519086340714452, 26.68647767657777, 26.502398302370402, 26.35836957803332, 26.500074767531167, 26.756825197135797, 26.61465102248258, 26.883696729797684, 26.721789269674048, 26.79997215000162, 26.507070394509714, 26.713321001102937, 26.675860686076554, 26.789316392947274, 26.734816868011468, 26.925030375521263, 26.78185686006352, 26.828669365171432, 26.898265564136672, 26.997670098960125, 26.58346725765291, 26.98360963815846, 26.949306773332108, 26.866127011603215, 26.904331611316476, 27.018117921387102, 26.913934278398166, 26.942785909195926, 27.0572525358541, 26.85422753198156, 26.802436434084658, 26.912846528111146, 26.953078963074674, 26.871379332530296, 27.133054445759683, 27.06329828105225, 26.86718681382005, 26.784242612220062, 27.129768449971873, 26.899663029998653, 27.092409754576504, 26.943686185842243, 27.126004030091234, 27.075997107782985, 27.00478223988364, 26.954881553754195, 27.128120737317083, 27.26391501026856, 26.62878925470602, 27.312878507222614, 27.209355420020994, 27.262608107937105, 27.206222758165307, 27.0120863356424, 27.192331535808393, 27.290430172994057, 27.383470471744634, 27.256051277576322, 27.192774668712882, 27.236724746496336, 27.286242265193405, 27.33987699486662, 27.134929513717836, 27.33610965800927, 27.342853607308985, 27.230874367930866, 27.32615844944727, 27.32991611463294, 27.49000317352162, 27.216721211062875, 27.2486755565496, 27.509170514821, 27.211931598018147, 27.182387761535466, 27.24112250621991, 27.222793413598858, 27.38562263124806, 27.445904418318488, 27.447368426403077, 27.37308873644017, 27.44656335043385, 27.377184098808225, 27.547445306617423, 27.4595219676506, 27.53880062124529, 27.625638695312645, 27.5723796241114, 27.136062660248417, 27.226319032746265, 27.29633140116729, 27.631505760426187, 27.674954931232374, 27.728625374929177, 27.591904928550058, 27.576594539421116, 27.65714041002535, 27.646333039333086, 27.665661961035934, 27.68377053604231, 27.692063291596124, 27.649428242882358, 27.534870946917117, 27.32500127104943, 27.567980196771007, 27.681384583879268, 27.433536908880416, 27.378598361090592, 27.700985322336567, 27.209901228897266, 27.678789740723424, 27.723840546913753, 27.614518330308616, 27.688873802362004, 27.62059429477935, 27.801427670180885, 27.670245048845366, 27.763636926151346, 27.70675009152503, 27.64940661773334, 27.4335089027518, 27.349631559524077, 27.39953333193096, 27.055275052523694, 27.54209773920211, 27.62560556671412, 27.71488282497128, 27.664155463406473, 27.47051615110866, 27.654433053969747, 27.63343809662031, 27.606025816266033, 27.609445142855385, 27.528261681132612, 27.64719501683486, 27.55462395513759, 27.4532317127279, 27.543209949117035, 27.420699812464836, 27.296464677297084, 27.363041088328057, 27.564845376439095, 27.647633517497994, 27.649257684453776, 27.564422745631916, 27.479429303596643, 27.70740389878346, 27.75087319581631, 27.67755023901446, 27.605301469431296, 27.607315751349333, 27.5710917990152, 27.589205013009703, 27.651066017265375, 27.462911639913305, 27.790245977010994, 27.575784425550502, 27.761994983293686, 27.607845196094743, 27.911244281136735, 27.74855794849233, 27.75745937747287, 27.75661492897514, 27.675667899653263, 27.699200621330775, 27.54465793316485, 27.661892369297806, 27.6970472029542, 27.52901228320329, 26.743230879922507, 26.990685311555517, 27.34539146443403, 27.5018051654596, 27.644637825924267, 27.723576723331842, 27.55805454031313, 27.690556460208317, 27.597992245732584, 27.4626199942016, 27.627549588268174, 27.67728166542562, 27.596896737593983, 27.434762114958083, 27.08431125696573, 27.458241416131465, 27.57888616087306, 27.43995994393739, 27.518333940197355, 27.390749467406465, 27.35386087119415, 27.525564231159255, 27.602347628922445, 27.23495997563261, 27.416147599069753, 27.50096654763, 27.62275519246325, 27.400365606658994, 27.433065480233566, 27.42234033103539, 27.51588385631785, 27.61102157868808, 27.629601057877036, 27.71414576486763, 27.638847101139316, 27.447860172670634, 27.59945093674042, 27.515394478263605, 27.5893211887083, 27.587017995854602, 27.57611290440029, 27.546331425135435, 27.559619073896844, 27.412269601066168, 27.691375433619754, 27.555580339779763, 27.673951594656256, 27.609451507297706, 27.621196011145194, 27.506598606210353, 27.707851219599547, 27.818420700582276, 27.51418845748706, 27.7430355760454, 27.723090472019116, 27.478483040480732, 27.317902256390436, 27.372293722132902, 27.686639453016497, 27.65690428848761, 27.395440888850807, 27.52792304343521, 27.52742960163641, 27.675669095177042, 27.758840469215265, 27.58115012938146, 27.81812255636417, 27.190757018784822, 27.375990397421095, 27.75719466939146, 27.623855557976576, 27.58026164356946, 27.749223344499686, 27.89389499473171, 27.608558293475895, 27.98247433110001, 27.800560273039114, 27.871598481486938, 27.67552564582866, 27.637531189824664, 27.672534330671002, 27.537895380241416, 27.7179898811072, 27.71181110280576, 27.75002013063051, 27.917199121877417, 27.612882484909445, 28.020438773362084, 27.785153456248054, 27.8962429311074, 27.862730599851073, 27.906056862893415, 27.8207261742784, 27.763832370520205, 27.582263225940892, 27.412546214292902, 27.39258011828865 ], "return_max": [ 0.0, -1.8461055829240518, -5.942355756644519, -5.9006095331928, -5.856255534230954, -5.914460580161098, -5.85241248289582, -5.864111475552989, -5.804453230466926, -5.6830460400309555, -5.506933160465331, -5.4175174447589844, -5.359488190945507, -5.308367892895443, -5.2589328633315615, -5.267298729058026, -5.214023800373518, -5.198052861340272, -5.123229451670625, -5.156001497569283, -5.103431427342837, -5.055902765241641, -5.043813819667567, -5.057252900213973, -5.057806499452236, -5.06703546475203, -5.061035838654163, -5.086345796167563, -5.073063570383997, -5.117057897839193, -5.219578639436179, -5.389155524005456, -5.469432583537717, -5.516193691878572, -5.681673521800347, -6.22738090769805, -6.268658085553801, -6.608752678183182, -6.976422296021613, -6.954614451139375, -8.134812316499373, -7.990441271395638, -8.567642894644388, -8.428344372864386, -9.299825155864687, -9.106401153924475, -9.200682577367033, -9.378366736591616, -9.745470979513067, -9.742103913047384, -9.422968324498157, -8.901402225226239, -8.019583839363618, -7.448260443780973, -6.405231656881797, -6.1840681406084865, -5.834782468207012, -5.094988223349939, -3.93553547793352, -3.490495302773597, -2.994445989869707, -2.8235836607632274, -2.103287804291622, -0.6403185111770264, 0.035863029969431026, 1.0073427402558257, 1.532209057133697, 2.7726072848879553, 2.8081376364025368, 3.1871342515550958, 4.6484206853267285, 5.7025259911138075, 6.517148543204527, 6.4248764090056705, 6.911689085543606, 7.106302787584154, 7.120391143245384, 7.409227903032649, 8.81531234343541, 9.152119598584784, 9.629604117928073, 9.871659981375924, 9.980799964419491, 10.239078280706348, 10.665051539486528, 11.17499911495086, 11.352110374730874, 11.716072407421665, 11.793340726197457, 12.532528781846391, 13.41300069231697, 13.787995263804998, 14.045859866330192, 14.318404852129634, 14.823225912191589, 15.011388233368248, 15.810413743748871, 15.845918179408075, 16.42679607605157, 17.042502963438015, 17.5322719144802, 17.511217448154543, 17.783073288516484, 18.057637026964702, 18.089641710712, 18.432675868868287, 18.811617285742997, 19.3444661555248, 19.578936802676207, 19.69337685473273, 20.312429549115745, 20.30085350358694, 20.54415502982737, 20.877849823241156, 20.824529337001273, 21.210089398109815, 21.651415401175065, 21.83622547031005, 22.013317122700702, 22.194857313082466, 22.202669964534508, 22.330329852086766, 22.362560935187812, 22.492343886515947, 22.634528154467112, 22.71175927321735, 22.819953356475224, 22.860376653874628, 22.902712428342866, 23.20727519754466, 23.192808819710784, 23.22911743721545, 23.331330083633624, 23.636531121974077, 23.283443071826305, 23.61383549955233, 23.87857539351659, 24.116213726140934, 24.109804467241606, 24.080953585685656, 24.182408993485282, 24.22503070722041, 24.277031589350567, 24.28086167621725, 24.338310248869753, 24.086170670411896, 24.247355891518083, 24.392246887249893, 24.265509504181285, 24.466274252294614, 24.40196999746903, 24.540974808852788, 24.79174308944657, 24.806109729137226, 25.012085050800383, 25.07635556274646, 24.932140267960776, 24.971801947997374, 25.19090317956686, 25.150228501941697, 25.09430264405376, 24.875337720343182, 24.963633271032617, 25.100993573436423, 25.400251953083824, 25.308705189496003, 25.274699549334613, 25.420781172317277, 25.33391303842891, 25.567425205204433, 25.594633043331076, 25.46100481031832, 25.412121431115633, 25.58028064758909, 25.60693896124021, 25.66569445434512, 25.76865689350919, 25.810841389373525, 25.85967269055498, 25.962069220008672, 25.91996049679049, 25.963324765887016, 25.91702973971557, 26.157279117211537, 25.843831924082476, 25.94582477903816, 26.212889252978872, 26.286870962849054, 26.40751196976646, 26.284033846879908, 26.35805794342963, 26.52703460808759, 26.2964527617929, 26.55949959075015, 26.55729202886028, 26.68286031603485, 26.538413810723785, 26.71385175832885, 26.74395183803158, 26.68025527657551, 26.801870785928887, 26.851465829263198, 26.93680166694812, 26.961005778922402, 26.854616119279207, 26.88584927427365, 27.0105294896993, 27.044895596199282, 26.776704929518832, 27.13962236257852, 27.186412255105285, 27.179912100624218, 27.210794725487762, 27.269374471255276, 27.174719160454647, 27.31715674626472, 27.265195299454607, 27.263631864724005, 27.39627078043902, 27.34258350620992, 27.31068089509436, 27.32687119814303, 27.408540368280352, 27.320285751169344, 27.138582350978016, 27.197517685992338, 27.2787107390495, 27.429846525353543, 27.48308777741448, 27.327727808964294, 27.617757727847625, 27.54328077647833, 27.56872278307807, 27.386999886827365, 27.667973934412352, 27.480072363147407, 27.53769003346666, 27.639470008663924, 27.564638796832078, 27.496486817688115, 27.581619393566044, 27.63753434062462, 27.537875337075523, 27.543520995320314, 27.594245505106223, 27.7338456626001, 27.817130219783696, 27.709286722768926, 27.734779079980775, 27.703104502947088, 27.71827765573629, 27.83619181533567, 27.81285345818395, 27.806645067608518, 27.80194241315825, 27.76007516064274, 27.82463922517078, 27.84975823090078, 27.817606856835535, 27.922068700168428, 28.013535019237718, 28.006340417032497, 27.961943661461866, 27.910805526554405, 27.9567181634562, 28.088161460328006, 28.160896911964333, 28.03687692541058, 28.11608027764798, 28.100589114837057, 28.143973714335537, 28.13973034697916, 28.070707581372414, 28.14186843743203, 28.14093486576299, 28.180976497640092, 28.129040745911777, 28.29262308819543, 28.35819999517732, 28.262551527064023, 28.17242402842691, 28.16685204495492, 27.98908130303733, 27.971442662104895, 28.04052773964313, 28.194933331563192, 28.21196307922624, 28.398280042595943, 28.378426582146385, 28.41819677847299, 28.41000411431775, 28.355294606770748, 28.46682456965755, 28.401332730195488, 28.3555902248825, 28.420588015446206, 28.453976126225054, 28.43060360612622, 28.42658227623985, 28.403042563670418, 28.426569470592217, 28.531382203520955, 28.57837672969091, 28.596598307465857, 28.4737160545871, 28.583848825507353, 28.250870453149908, 28.39964222423336, 28.582844183613325, 28.365761112111585, 28.462431198995546, 28.5491104986275, 28.52845884051301, 28.59132089804707, 28.697558712879154, 28.48288486649054, 28.66211898955232, 28.720465297452353, 28.62806434965292, 28.567655194519812, 28.40326890288394, 28.25374573334743, 28.351523567772965, 28.686339435546262, 28.59770539021719, 28.596773880026756, 28.63314084601913, 28.63569232116834, 28.702060685658385, 28.619119107857415, 28.678181092801367, 28.71163400403715, 28.61188169506226, 28.7386676440666, 28.683640034738126, 28.541454812957095, 28.63711326356703, 28.69542892643493, 28.600919930656502, 28.597689676784487, 28.672876331241174, 28.709342971388953, 28.693955947482817, 28.329047789235698, 28.270712557482234, 28.60880616394475, 28.442675540936182, 28.672081909762998, 28.76269899732062, 28.758401880350853, 28.770482450465273, 28.636978220299326, 28.778900852067242, 28.629921597826367, 28.781905701704325, 28.858760451114772, 28.652920968857273, 28.56684427605041, 28.767310387158314, 28.75147373932369, 28.634802542315008, 28.51926408156356, 28.70338942286469, 28.62303774802057, 28.685713036219873, 28.728758469643523, 28.67635878914168, 28.75721728949732, 28.56298224078308, 28.766505036062266, 28.634195551825503, 28.748594198476127, 28.64870496215311, 28.745331557698126, 28.683312983072785, 28.626677838419578, 28.681126150660518, 28.795726798460656, 28.757144496535805, 28.577226605488086, 28.681473177482456, 28.760889685404365, 28.74228846342441, 28.658412317188745, 28.786052544647482, 28.75495418425858, 28.600933120176805, 28.760873099274633, 28.693675765876716, 28.551481162007267, 28.725814368956623, 28.79415552475671, 28.743963090640232, 28.734863817526268, 28.566545314785092, 28.823098278619486, 28.156893214359613, 28.612518041199017, 28.74705487866705, 28.61278583140767, 28.72307807897606, 28.68793753335112, 28.692956700752664, 28.803710654062744, 28.86342022913714, 28.84833624243771, 28.85949429015124, 28.821595637152704, 28.852835471194876, 28.817591413264847, 28.72423722172097, 28.710836190395927, 28.88692886632797, 28.7053880821483, 28.774013789112196, 28.662860260386143, 28.79822111112021, 28.808628019217544, 28.85276102010336, 28.771181318446978, 28.908834137894033, 28.85579874848611, 28.890888993253977, 28.848728020793402, 28.709021758836165, 28.462931191839804, 28.917714321761146, 28.875846241560996, 28.88680133357795, 28.972438730893764, 28.802010216370572, 28.87394771604405, 28.92969637966023, 28.89161006772763, 28.834618183273626, 28.878542363267485, 28.92712663123022, 28.873554074406456, 28.915520549210623, 28.908137396095636, 28.793859203233534, 28.926888627586624, 28.909075735628452, 28.801053649246313, 28.956274434873805, 28.85351939477016, 28.939984909664357, 28.89357806281323, 28.898246390008985, 28.69677817945649, 28.76230927723095, 28.660130319230213, 28.795664359490793, 28.97442098448534, 28.804882468754947, 28.81231306915242, 28.90048542404371, 28.94753999819369, 28.83492346942782, 28.409316330602834, 28.42333477999669, 28.748688250879383, 28.706269024766705, 28.633549596908715, 28.874302095920925, 28.705128326549733, 28.883376647346335, 28.84215446744502, 28.690961128379946, 28.79284085140264, 28.67231692219214, 28.717414174020156, 28.713685795180428, 28.61269640828595, 28.6245972262438, 28.67272102371773, 28.69483747703352, 28.661568300251197, 28.801075406577, 28.67019932663518, 28.750051006766387, 28.803437084207236, 28.869076710128372, 28.70201001971411, 28.53336427531146, 28.69181279541776, 28.674908862840148, 28.748490512591367, 28.80004539315285, 28.8705797324927, 28.707019828640863, 28.922864233127992, 28.795304714324864, 28.897806471765325, 28.80391957460775, 28.78242060241827, 28.70729365133907, 28.89739906461977, 28.711171772253486, 28.754616674551905, 28.932490364601158 ] } }, "Isaac-Velocity-Rough-G1-v0": { "FastTD3": { "time": [ 0.0, 473.25, 946.5, 1419.75, 1893.0, 2366.25, 2839.5, 3312.75, 3786.0 ], "env_step": [ 0, 20480000.0, 40960000.0, 61440000.0, 81920000.0, 102400000.0, 122880000.0, 143360000.0, 163840000.0 ], "return": [ 0.0, 18.40044911702474, 27.28994305928548, 33.946146647135414, 35.17380905151367, 36.433878580729164, 37.3122189839681, 38.439432779947914, 37.77874247233073 ], "return_min": [ 0.0, 17.450108886655634, 25.337943364273244, 32.46642651481381, 32.07014682726799, 34.59316721803161, 34.879719062140424, 37.541568810964186, 36.28540258902592 ], "return_max": [ 0.0, 19.350789347393842, 29.241942754297717, 35.42586677945702, 38.27747127575935, 38.27458994342672, 39.74471890579578, 39.33729674893164, 39.27208235563553 ] }, "null": { "time": [ 0.0, 9.782, 19.564, 29.346, 39.128, 48.91, 58.692, 68.474, 78.256, 88.038, 97.82, 107.602, 117.384, 127.166, 136.948, 146.73, 156.512, 166.294, 176.076, 185.858, 195.64, 205.422, 215.204, 224.986, 234.768, 244.55, 254.332, 264.114, 273.896, 283.678, 293.46, 303.242, 313.024, 322.806, 332.588, 342.37, 352.152, 361.934, 371.716, 381.498, 391.28, 401.062, 410.844, 420.626, 430.408, 440.19, 449.972, 459.754, 469.536, 479.318, 489.1, 498.882, 508.664, 518.446, 528.228, 538.01, 547.792, 557.574, 567.356, 577.138, 586.92, 596.702, 606.484, 616.266, 626.048, 635.83, 645.612, 655.394, 665.176, 674.958, 684.74, 694.522, 704.304, 714.086, 723.868, 733.65, 743.432, 753.214, 762.996, 772.778, 782.56, 792.342, 802.124, 811.906, 821.688, 831.47, 841.252, 851.034, 860.816, 870.598, 880.38, 890.162, 899.944, 909.726, 919.508, 929.29, 939.072, 948.854, 958.636, 968.418, 978.2, 987.982, 997.764, 1007.546, 1017.328, 1027.11, 1036.892, 1046.674, 1056.456, 1066.238, 1076.02, 1085.802, 1095.584, 1105.366, 1115.148, 1124.93, 1134.712, 1144.494, 1154.276, 1164.058, 1173.84, 1183.622, 1193.404, 1203.186, 1212.968, 1222.75, 1232.532, 1242.314, 1252.096, 1261.878, 1271.66, 1281.442, 1291.224, 1301.006, 1310.788, 1320.57, 1330.352, 1340.134, 1349.916, 1359.698, 1369.48, 1379.262, 1389.044, 1398.826, 1408.608, 1418.39, 1428.172, 1437.954, 1447.736, 1457.518, 1467.3, 1477.082, 1486.864, 1496.646, 1506.428, 1516.21, 1525.992, 1535.774, 1545.556, 1555.338, 1565.12, 1574.902, 1584.684, 1594.466, 1604.248, 1614.03, 1623.812, 1633.594, 1643.376, 1653.158, 1662.94, 1672.722, 1682.504, 1692.286, 1702.068, 1711.85, 1721.632, 1731.414, 1741.196, 1750.978, 1760.76, 1770.542, 1780.324, 1790.106, 1799.888, 1809.67, 1819.452, 1829.234, 1839.016, 1848.798, 1858.58, 1868.362, 1878.144, 1887.926, 1897.708, 1907.49, 1917.272, 1927.054, 1936.836, 1946.618, 1956.4, 1966.182, 1975.964, 1985.746, 1995.528, 2005.31, 2015.092, 2024.874, 2034.656, 2044.438, 2054.22, 2064.002, 2073.784, 2083.566, 2093.348, 2103.13, 2112.912, 2122.694, 2132.476, 2142.258, 2152.04, 2161.822, 2171.604, 2181.386, 2191.168, 2200.95, 2210.732, 2220.514, 2230.296, 2240.078, 2249.86, 2259.642, 2269.424, 2279.206, 2288.988, 2298.77, 2308.552, 2318.334, 2328.116, 2337.898, 2347.68, 2357.462, 2367.244, 2377.026, 2386.808, 2396.59, 2406.372, 2416.154, 2425.936, 2435.718, 2445.5, 2455.282, 2465.064, 2474.846, 2484.628, 2494.41, 2504.192, 2513.974, 2523.756, 2533.538, 2543.32, 2553.102, 2562.884, 2572.666, 2582.448, 2592.23, 2602.012, 2611.794, 2621.576, 2631.358, 2641.14, 2650.922, 2660.704, 2670.486, 2680.268, 2690.05, 2699.832, 2709.614, 2719.396, 2729.178, 2738.96, 2748.742, 2758.524, 2768.306, 2778.088, 2787.87, 2797.652, 2807.434, 2817.216, 2826.998, 2836.78, 2846.562, 2856.344, 2866.126, 2875.908, 2885.69, 2895.472, 2905.254, 2915.036, 2924.818, 2934.6, 2944.382, 2954.164, 2963.946, 2973.728, 2983.51, 2993.292, 3003.074, 3012.856, 3022.638, 3032.42, 3042.202, 3051.984, 3061.766, 3071.548, 3081.33, 3091.112, 3100.894, 3110.676, 3120.458, 3130.24, 3140.022, 3149.804, 3159.586, 3169.368, 3179.15, 3188.932, 3198.714, 3208.496, 3218.278, 3228.06, 3237.842, 3247.624, 3257.406, 3267.188, 3276.97, 3286.752, 3296.534, 3306.316, 3316.098, 3325.88, 3335.662, 3345.444, 3355.226, 3365.008, 3374.79, 3384.572, 3394.354, 3404.136, 3413.918, 3423.7, 3433.482, 3443.264, 3453.046, 3462.828, 3472.61, 3482.392, 3492.174, 3501.956, 3511.738, 3521.52, 3531.302, 3541.084, 3550.866, 3560.648, 3570.43, 3580.212, 3589.994, 3599.776, 3609.558, 3619.34, 3629.122, 3638.904, 3648.686, 3658.468, 3668.25, 3678.032, 3687.814, 3697.596, 3707.378, 3717.16, 3726.942, 3736.724, 3746.506, 3756.288, 3766.07, 3775.852, 3785.634, 3795.416, 3805.198, 3814.98, 3824.762, 3834.544, 3844.326, 3854.108, 3863.89, 3873.672, 3883.454, 3893.236, 3903.018, 3912.8, 3922.582, 3932.364, 3942.146, 3951.928, 3961.71, 3971.492, 3981.274, 3991.056, 4000.838, 4010.62, 4020.402, 4030.184, 4039.966, 4049.748, 4059.53, 4069.312, 4079.094, 4088.876, 4098.658, 4108.44, 4118.222, 4128.004, 4137.786, 4147.568, 4157.35, 4167.132, 4176.914, 4186.696, 4196.478, 4206.26, 4216.042, 4225.824, 4235.606, 4245.388, 4255.17, 4264.952, 4274.734, 4284.516, 4294.298, 4304.08, 4313.862, 4323.644, 4333.426, 4343.208, 4352.99, 4362.772, 4372.554, 4382.336, 4392.118, 4401.9, 4411.682, 4421.464, 4431.246, 4441.028, 4450.81, 4460.592, 4470.374, 4480.156, 4489.938, 4499.72, 4509.502, 4519.284, 4529.066, 4538.848, 4548.63, 4558.412, 4568.194, 4577.976, 4587.758, 4597.54, 4607.322, 4617.104, 4626.886, 4636.668, 4646.45, 4656.232, 4666.014, 4675.796, 4685.578, 4695.36, 4705.142, 4714.924, 4724.706, 4734.488, 4744.27, 4754.052, 4763.834, 4773.616, 4783.398, 4793.18, 4802.962, 4812.744, 4822.526, 4832.308, 4842.09, 4851.872, 4861.654, 4871.436, 4881.218, 4891.0 ], "env_step": [ 0, 2064384, 2654208, 3833856, 4128768, 5308416, 5701632, 5898240, 5996544, 6586368, 7471104, 7667712, 7864320, 10125312, 10321920, 10420224, 11010048, 11108352, 11304960, 11698176, 12582912, 12976128, 13565952, 13860864, 14057472, 14352384, 15335424, 15925248, 17301504, 18481152, 19759104, 19857408, 21528576, 21921792, 22020096, 22216704, 22609920, 22806528, 23592960, 24379392, 24576000, 24772608, 26443776, 26542080, 28508160, 29294592, 29687808, 30375936, 30474240, 31358976, 31653888, 31948800, 32342016, 32735232, 33325056, 34701312, 34996224, 35487744, 37060608, 37748736, 40108032, 40402944, 40501248, 40796160, 41582592, 42860544, 42958848, 43253760, 44531712, 44826624, 44924928, 45023232, 45711360, 45809664, 46202880, 46301184, 46989312, 47185920, 47579136, 47775744, 48365568, 50135040, 50528256, 50724864, 52199424, 53182464, 53379072, 53673984, 54067200, 54558720, 55050240, 56328192, 56721408, 57311232, 57507840, 58982400, 59768832, 60260352, 62128128, 62324736, 62619648, 63012864, 63111168, 63307776, 65077248, 65175552, 65372160, 65863680, 66355200, 66945024, 67436544, 68517888, 69697536, 69894144, 70483968, 71368704, 71467008, 71565312, 72548352, 74121216, 74416128, 75005952, 75300864, 75399168, 76185600, 76775424, 79527936, 79921152, 80314368, 80805888, 82182144, 83558400, 83656704, 83951616, 84541440, 84738048, 85524480, 85622784, 86409216, 86802432, 87392256, 87490560, 87588864, 87687168, 87785472, 87883776, 88375296, 88965120, 89161728, 89260032, 89849856, 90341376, 91422720, 91815936, 91914240, 92209152, 92700672, 93290496, 93487104, 94175232, 94371840, 94961664, 95059968, 95354880, 95748096, 95944704, 96239616, 96436224, 96632832, 98107392, 98304000, 98500608, 98697216, 98893824, 99090432, 99385344, 100073472, 100270080, 100761600, 101056512, 101842944, 102039552, 103219200, 104005632, 104693760, 104890368, 105086976, 105676800, 106266624, 107544576, 108724224, 109510656, 110198784, 110297088, 110788608, 111083520, 111476736, 111771648, 112656384, 114524160, 114622464, 116883456, 117080064, 117276672, 117374976, 117964800, 118358016, 118849536, 119046144, 119537664, 122486784, 122585088, 122978304, 123863040, 124059648, 124846080, 125927424, 126124032, 126615552, 127205376, 127500288, 129466368, 130351104, 130547712, 132710400, 133300224, 133791744, 134086656, 134873088, 135168000, 135757824, 136544256, 137723904, 138018816, 139001856, 139395072, 140574720, 140771328, 141656064, 141950976, 142245888, 142540800, 142639104, 143228928, 144113664, 144211968, 144310272, 145195008, 145981440, 146374656, 147554304, 147652608, 147947520, 149028864, 150110208, 150601728, 150798336, 151388160, 152469504, 152567808, 152666112, 153255936, 153845760, 154238976, 154435584, 156106752, 156303360, 157384704, 157581312, 157974528, 158564352, 159645696, 160628736, 161021952, 161120256, 162791424, 164265984, 165249024, 165937152, 166232064, 166526976, 166920192, 167018496, 167116800, 167313408, 167510016, 167804928, 167903232, 168198144, 168296448, 168886272, 169181184, 169279488, 170950656, 171343872, 171835392, 172720128, 173899776, 174391296, 174784512, 175177728, 175767552, 175964160, 176259072, 176357376, 176455680, 176553984, 177438720, 177635328, 178028544, 178520064, 178618368, 178913280, 179109888, 179404800, 180387840, 180977664, 182157312, 182353920, 184418304, 185204736, 185303040, 185892864, 186679296, 187072512, 187170816, 187564032, 187662336, 188841984, 189136896, 189628416, 190119936, 191004672, 192479232, 192774144, 193069056, 193855488, 194641920, 194740224, 195723264, 196313088, 197296128, 198475776, 199163904, 201031680, 201424896, 201916416, 202113024, 202604544, 202997760, 204079104, 205750272, 206241792, 206635008, 206929920, 207618048, 207716352, 209092608, 209879040, 210272256, 210370560, 210763776, 210862080, 211746816, 211943424, 213811200, 214892544, 216858624, 217055232, 217743360, 218628096, 218923008, 219217920, 219316224, 219414528, 219906048, 221577216, 222461952, 223739904, 223838208, 224428032, 224821248, 225116160, 226000896, 226983936, 227278848, 227475456, 228556800, 228851712, 228950016, 229244928, 230129664, 230227968, 230817792, 231309312, 231899136, 232783872, 233078784, 233275392, 233373696, 233766912, 234848256, 235536384, 237109248, 237305856, 237404160, 237699072, 238190592, 238780416, 239468544, 240451584, 240746496, 242122752, 242221056, 243105792, 243400704, 244187136, 245268480, 245563392, 245760000, 245858304, 246841344, 247136256, 247529472, 247922688, 249004032, 249200640, 249495552, 250773504, 250970112, 252641280, 253427712, 253526016, 255688704, 256868352, 257359872, 258342912, 258637824, 259620864, 259719168, 260210688, 260603904, 261095424, 261292032, 261685248, 262176768, 263651328, 263749632, 265224192, 265912320, 266010624, 266108928, 266502144, 266797056, 268271616, 268959744, 269844480, 270237696, 270925824, 271024128, 271319040, 271417344, 271515648, 272007168, 272302080, 272498688, 273186816, 274268160, 274366464, 274563072, 274759680, 275644416, 276529152, 278102016, 278495232, 280559616, 280756224, 280854528, 281542656, 282230784, 282820608, 283115520, 283705344, 284786688, 284884992, 285769728, 286457856, 286654464, 286851072, 286949376, 288423936, 288522240, 290586624, 290881536, 291471360, 292159488, 292552704, 292749312, 293437440, 294027264, 294420480 ], "return": [ 0.0, -5.6881421136856085, -5.5180525922775265, -5.366918431917827, -5.325606632232667, -5.190784207979838, -5.038557925224304, -4.9782898871103916, -4.919183246294657, -4.905417499542236, -4.753361112276713, -4.771880612373352, -4.804907844861348, -5.218108415603638, -5.452975128491719, -5.459520988464355, -5.728807779947917, -6.293901631037394, -6.285780429840088, -7.291187626520792, -7.811659456888834, -7.735299712816873, -7.949773249626159, -8.23085147778193, -8.023751058578492, -7.658783731460571, -7.70408089876175, -7.493596713145574, -6.802546091179053, -6.02310235808293, -5.458186463266611, -4.963028981884321, -4.4428239252169925, -3.7779746002703902, -3.5969980679887037, -3.5128461730852725, -2.607398334058623, -1.886164164994067, -2.217925976102706, -1.715551134304454, -1.1327023444014288, -0.2374029954833287, 0.16780850405494366, 0.42279706716537485, 1.096491112957398, 1.278363335741063, 1.5731980310132105, 1.7684402238329249, 2.3747797200921923, 2.706895433639487, 3.013346167653799, 2.790748157600562, 3.2253397530379395, 3.0303994294193886, 3.191454022967567, 3.9127346877443294, 4.167857200068732, 4.537999338172376, 4.813405925780535, 5.180030785401662, 5.09416914695253, 5.239751322865486, 5.4623762649297705, 5.56985650156935, 5.447801009019216, 5.8951625898480415, 5.910035707950591, 6.284413039684296, 5.8277599749962485, 5.393726678490638, 5.226986187398434, 5.363927096525828, 6.0447085696458815, 6.022956263919671, 5.796698542435965, 5.830887189706167, 6.911331553782026, 6.52881139477094, 6.529581413269042, 5.943345656866828, 6.304789720376333, 6.2125239910682035, 6.334142784873644, 6.165451030731202, 6.199949367642403, 6.212140672504901, 6.043669728835424, 6.123819414041936, 6.673933910727501, 6.168897040734688, 5.858888900627693, 6.185403854250907, 6.2324638844529785, 6.3189695264895755, 5.429789771735668, 5.867755074054003, 5.9698123177016775, 6.2788647194703415, 6.108241503039996, 6.194222606966893, 5.6370969935754935, 5.860745827356975, 5.627046625390649, 5.9018260749181115, 5.5154518715043865, 5.629180502370001, 5.838229054013888, 5.671999172481397, 5.293166875243187, 5.526465571175019, 5.529769248962403, 5.878975370128949, 5.542135804857438, 4.932354421118895, 5.269491405735414, 5.4556330054998385, 5.799737800732255, 5.829726587931315, 5.968212634523709, 5.584529157479604, 5.912035822945957, 5.757281076312065, 6.098358916342259, 6.190662302970885, 6.169422122637431, 6.15962440357233, 6.283701638529698, 6.239324719111124, 6.476538527409236, 6.383258503476779, 6.16934560338656, 6.73461858527114, 6.619796413257718, 7.001251819133759, 6.74785814344883, 6.963990779990951, 7.004205753803254, 6.493811826705933, 6.724569251736004, 6.9595607940355935, 7.377430408398311, 7.391885064045589, 7.427037688096365, 7.614314963817596, 7.211228293528159, 7.1433285713195795, 7.39753812789917, 8.07717345317205, 7.722901459336281, 7.50667692820231, 7.629707463184992, 7.9553125536441796, 8.055245854854585, 7.869338385264079, 7.858060536384582, 8.241977485815683, 8.327039570013682, 8.404392083485922, 8.476991500854492, 8.417776710490386, 8.851075089772543, 8.378894633452099, 8.568723888397217, 8.807038741906483, 8.106232002973558, 9.009860292275748, 8.951988192399343, 8.541935040950776, 8.760698347091676, 9.00345556338628, 9.382102955182392, 9.348882626692456, 9.577981095314026, 9.366613285541534, 9.75678734322389, 9.434628633260727, 9.696422396500905, 9.92948784351349, 10.117260546286902, 10.056760675112406, 9.992847542762755, 10.470781871875127, 10.06678469657898, 10.482964709599813, 10.941076284249624, 10.520150343577066, 10.739750129381816, 10.971128408114117, 10.290067638556163, 11.149061189492542, 11.524184123675028, 11.633825011253357, 11.01133109887441, 11.827612968285878, 11.392092748483023, 11.764327395757041, 11.70821429570516, 11.733774793148042, 12.074558056990305, 11.616416401068369, 11.907227954069773, 12.560407905578614, 12.36894993464152, 12.091517403125764, 12.747315545082094, 12.558328438599903, 12.342031540075938, 12.96610297203064, 12.804973107973737, 12.630402886867524, 13.08221505800883, 13.075428587595622, 13.208603329658509, 12.54065419991811, 13.22744600137075, 13.474049444993335, 13.782562464872996, 13.70558207432429, 13.664181585311889, 13.819338383674621, 14.275129564603171, 14.416780421733856, 13.759321572780609, 14.259545510609945, 14.518339014053346, 14.214087019761402, 14.85833446741104, 14.629418731530507, 14.74431406259537, 14.71238970677058, 15.07072572628657, 14.685916913350423, 14.83047775665919, 14.68453181187312, 14.993108842372893, 15.29401432434718, 15.640210219224295, 15.341570507685345, 15.120413868427278, 15.916163510481516, 15.719678219159443, 15.838848996957141, 15.478066034317017, 15.486124574343364, 15.805122829278309, 15.930275870958965, 15.642760404745738, 16.525526213645936, 16.194698879718782, 16.2467432085673, 16.22657354513804, 16.767331422170002, 16.556936605771384, 16.73060949246089, 16.508015065987905, 16.879581843217213, 16.68470219373703, 16.331015097300213, 17.1932918492953, 17.274409716129302, 17.57423019170761, 17.57290551662445, 17.42105441729228, 17.225453462600708, 17.133760628700255, 17.36790020386378, 17.69404323577881, 17.407497894763946, 17.805010588963825, 17.68197588523229, 17.84539549589157, 17.520849231878916, 17.575091780821484, 18.307277234395347, 17.58764229933421, 17.703922248681383, 18.099228777885436, 18.280002871354423, 17.969462502002717, 18.29429830869039, 18.262509983380635, 18.301710220972698, 18.00693675438563, 18.199882732232414, 18.125358258883157, 18.519552324612935, 17.990541600386305, 18.569027196566264, 19.178397612571718, 18.605370388031005, 18.528695232868195, 18.21270858923594, 18.763140827814738, 18.657966167926787, 18.7651668437322, 18.790467648506166, 18.698187935352326, 18.84629309415817, 19.124954300721484, 18.934716939926147, 18.991049098173775, 19.78828253428141, 19.01165625333786, 18.79346280097961, 19.511103755633037, 19.065227732658386, 19.421975706418355, 19.211517736117045, 18.913628775278728, 19.251926871935524, 19.194325835704806, 19.08125954469045, 19.474459347724917, 19.06925933043162, 19.73238619248072, 19.493218274116515, 19.7938659175237, 20.104998388290408, 20.18069859266281, 20.11826090415319, 19.140096864700315, 19.855108708540598, 19.700749299526212, 20.264274895191193, 20.147352376778922, 19.73275642077128, 19.819068230787913, 20.011384061177573, 20.02887071212133, 20.07820870399475, 19.71256620407104, 19.83414210796356, 20.391562454700473, 20.002346297105152, 20.657952380180358, 20.258645397027333, 20.86577610492706, 20.674734957218167, 20.29822519461314, 20.035884976387024, 20.60575511932373, 19.918341357707977, 20.589110221862796, 20.634883064428966, 20.818063621520995, 20.816770624319712, 20.601503771940866, 20.502998891671496, 21.004771229426066, 20.736558705965678, 21.25983631690343, 21.269482579231262, 21.138610611756643, 20.938844652970634, 20.940916749636333, 21.20807293574015, 20.98027428865433, 21.046181119283045, 21.32889621814092, 21.05542133967082, 21.157513887882235, 21.59118415673574, 21.552832601865134, 21.455512606302893, 21.077143347263334, 21.365520514647162, 21.446350107192995, 21.994762991269425, 22.047921177546186, 21.408621869087217, 21.42010664701462, 21.38248774766922, 21.897231921354926, 21.796455845832824, 22.00268397172292, 21.341353923479716, 22.258859420617423, 21.87750239610672, 22.012093653678892, 22.11572533210119, 22.070228363672893, 21.90432560443878, 21.88233465353648, 21.868809100786848, 22.011972271601355, 21.727508841355643, 22.31119052171707, 22.12301502863566, 22.116760693391168, 22.425632123152415, 22.14697431723277, 22.17153032461802, 22.668648889859515, 22.57969790617625, 22.410574590365087, 22.001601606210073, 21.817777061462404, 22.229211839834846, 22.151013066768645, 22.43489840984344, 22.271503342787426, 21.90029133637746, 22.330113309224444, 22.955066961447397, 21.912163372834527, 22.186071017583213, 22.38241572300593, 22.47269917170207, 22.804330983956657, 22.403881865342456, 22.417255460421245, 22.50846732934316, 22.436016081174216, 22.510304973125457, 22.433432060082755, 22.458809146881105, 22.923311399618786, 22.47686696211497, 22.402629868189496, 22.616998944282532, 22.77077663580577, 22.905668086210884, 22.748367921511335, 22.205995389620465, 22.907470565636952, 22.8850816377004, 22.782681437333423, 22.988625747362775, 23.00508107582728, 22.886291638215383, 22.84144187609355, 23.04126522064209, 23.382145015398663, 22.824609417120616, 23.142978974183396, 23.523871717453, 23.504992637634277, 23.42341079155604, 23.660274498462673, 23.1919059920311, 23.286985676288605, 23.697115731239318, 23.35182003657023, 23.776714529196422, 23.88528642098109, 23.77788284619649, 23.53571937243144, 23.429716286659243, 23.593821457227076, 23.6001250298818, 24.131974957784013, 23.962793831030528, 23.7850101741155, 23.81359586238861, 23.871057611306508, 23.643791459401445, 23.712499034404757, 24.161554266611734, 24.0301451587677, 24.00370688756307, 24.046240677833556, 24.134283207257592, 23.779774011770883, 24.4099445605278, 24.606129792531334, 24.044053098360695, 24.392736365000406, 24.15341828107834, 24.489894301096598, 24.132509734630585, 24.23703903357188, 23.809118733406066, 24.21431887785594, 24.55282241980235, 24.52900070508321, 24.679437781969707, 24.63161761045456, 24.952858759562176, 24.57767979303996, 24.37862017631531, 24.57373389085134, 24.632339243094126, 24.571167072455086, 24.859973764419554, 24.99239943822225, 24.529701237678527, 24.770756055514017, 24.961983086268106, 24.882949465115868, 24.77448396841685, 25.166602612336476, 25.016854931513468, 25.510536619822187, 25.088715693155923, 25.07602248350779, 25.324519255956016, 24.968988726933798, 25.13067001819611, 25.30761011282603, 24.8148122882843, 24.975149160226184 ], "return_min": [ 0.0, -5.847385083804003, -5.7513117592921645, -5.748779817936011, -5.728821428054697, -5.559080329213756, -5.3636133316322265, -5.302210290559181, -5.251333814058662, -5.194289546721069, -4.8739384283057285, -4.842630461534321, -4.9151453492215245, -5.843299921732156, -6.3329078817675795, -6.219128552891097, -6.481890776537796, -6.802709925873941, -6.810748831780586, -8.054595643731085, -8.723781704520437, -8.580540940926463, -8.575722894918862, -9.220375253800077, -8.827859993796615, -8.938540268579562, -9.537985418663462, -9.839037998870781, -9.260156556661503, -8.357282452989063, -8.319941544401631, -7.793454754107815, -7.859164931975359, -6.597464244259629, -6.4995116443969785, -6.33825469881865, -5.805293320023743, -5.415640063856721, -5.345139400172988, -5.100149198443003, -4.775884997246164, -3.901154927530331, -3.5430011561290904, -2.929028364818739, -2.4767586717301917, -1.8265270804764393, -1.6176343798866804, -1.5852250643868577, -0.7284182326456308, -0.44979643620616727, -0.2577023392133886, -0.23194055700112637, 0.07253895591730597, -0.15651519592880758, 0.7055207887916808, 1.6360583348580673, 1.9420826897659431, 2.473776496516727, 3.0785349575773653, 3.9844120773625447, 3.441431652994045, 3.620430832636923, 4.100560387951199, 3.930868276930429, 4.550770792569821, 4.336923129658898, 4.599446487157081, 5.373622367155295, 5.176054051043158, 4.666238701619989, 3.951051968725962, 4.258857327615415, 5.173113526915158, 5.073355247708585, 4.371943558498154, 4.54636012262562, 5.808916391380994, 5.551197830898223, 5.34759574306736, 5.0277504587736, 5.306229214765686, 5.073083524975472, 5.491268758513117, 5.176724814298337, 5.573712277277728, 5.888213169839131, 5.000021083204806, 5.256854984893558, 5.665206042050714, 5.461860319919166, 5.175830911510187, 5.179716016850846, 5.417536568039349, 5.359956086682328, 4.491544768033436, 5.161442954113744, 5.31481914992216, 5.355294443273641, 5.662462702702301, 5.6259969021338865, 4.447962590494374, 5.167120718722103, 4.842777828917456, 5.4036097300877675, 4.706745617468813, 4.8632407453674995, 5.030492081517986, 5.312459879968527, 4.742696950444269, 4.907279705908248, 4.9511379993283064, 4.982742827487655, 4.956854429855504, 4.301009798833486, 4.838506825396664, 4.992781515720493, 5.290940471382674, 5.14348241722087, 5.579096637106402, 5.168243772759865, 5.5843474503410535, 4.996058413060161, 5.761075259812566, 6.067030634385872, 5.710991308717619, 5.431620018314759, 5.867838818613855, 5.807803655471879, 6.06987462127813, 6.016871892402834, 5.381815690518922, 6.535831944081462, 5.8180260636282695, 6.419818415073631, 6.423334427947558, 6.251856128022345, 6.676194921207104, 5.896413605071722, 6.127789900798275, 6.504789139261375, 7.125364290761334, 6.82233672990498, 7.183872097738285, 7.573518841732584, 6.668143500018438, 6.800361653251276, 7.354062609207972, 7.814342488447513, 7.348284773553699, 7.027655732334908, 7.164310953205494, 7.324909554131923, 7.79977177857154, 7.507403710366009, 7.445613236257481, 7.949163965051599, 8.029579861760306, 8.049631732191134, 7.98473374738377, 8.09692933562954, 8.595150747693317, 7.889534769486059, 8.182050033875075, 8.543661957454564, 7.83448305630662, 8.854549640247985, 8.734680818048101, 8.03914865755524, 8.188374096405493, 8.70408444962382, 9.08980827510886, 8.676364682762179, 9.001981073268468, 9.080738296785063, 9.163175043725404, 8.826958364232896, 9.212786567279501, 9.491640818533266, 9.813652886857588, 9.663235392269925, 8.831743708382604, 9.444130141965037, 9.063594161553546, 9.684780684589581, 10.580158786557131, 9.732795426778067, 9.653901282414537, 10.326423482399091, 9.520840913025431, 10.355890488911198, 10.635577274552382, 10.414180755942379, 10.261674546782004, 11.18722024808477, 10.622391593094918, 10.994410335019602, 11.134014403610646, 10.664608004781611, 11.602299322770435, 10.367783660472115, 10.797210054066554, 11.815822853246, 11.480982140330964, 11.09412850857745, 11.98628863883241, 12.067312099605639, 11.848993921764682, 12.27060637972741, 11.958689462390439, 12.120182752071292, 12.259304748296094, 12.537474673178647, 12.453123537891061, 11.77096552979382, 12.56915001066292, 12.472330111732534, 13.177987976754496, 12.936951414977278, 12.52047362077546, 13.183660561178558, 12.95994516279078, 13.693192051103107, 12.795102644449155, 13.118756348528736, 13.149097145512771, 13.6082577654462, 13.899241828559406, 14.07114478898924, 13.950021395740468, 13.671012076467038, 14.427718747037764, 14.21831097874148, 14.149740627413518, 14.239057837801768, 14.322750650080883, 15.077346698203964, 14.881819807588435, 14.711101975059336, 14.653175754415862, 15.045519200395086, 14.859055954390353, 15.135940821200311, 15.028333599354786, 14.868744466774464, 15.226525571946476, 15.31503408437216, 14.7364282155483, 15.393723435935847, 15.221118103469363, 15.473782593262934, 15.38283804151289, 15.710453887247304, 15.186448409186136, 15.756452784424319, 15.430849440579916, 15.735974655408505, 15.385461297489352, 15.333921868051087, 16.501653952635927, 16.599003846243647, 16.893195703478842, 16.799448832304318, 16.33393092320314, 16.538116462206915, 16.23932883789974, 16.2392471269894, 16.859290630715464, 16.366514700387274, 16.96167137928475, 16.83021949534363, 16.93184653675907, 16.409533092068592, 16.91301269277289, 17.66389706800658, 16.52550778385674, 17.459393479765613, 17.50406546273049, 17.832717112063488, 16.734378055630582, 17.583295864709555, 17.237196356311454, 17.522937823995925, 17.06484780577894, 17.512659281323586, 17.696821198647786, 17.04809714010114, 17.227966220191867, 18.19230346714556, 18.37196814867198, 17.557956314796975, 17.552119087203025, 17.0894637013022, 18.043170483807394, 17.935874151751484, 17.921508832666053, 18.489324193422146, 18.065332946132315, 18.56311195761933, 18.541413772237288, 18.322821215118474, 18.471863873121876, 19.07608052741781, 18.177953258137197, 18.69541772898632, 18.49791617502517, 18.566899710949116, 18.869836981612593, 18.515412419249127, 18.150184516257696, 18.774430720238293, 18.46886497647636, 18.34975519370064, 18.962458767076292, 18.417420290694835, 18.93657658048281, 18.87946945833734, 19.222341574933896, 18.911153114649522, 19.422345302867388, 19.256796418085795, 18.48861765420574, 19.39348874173251, 18.686595302739825, 19.865326190725906, 19.023511220811436, 19.48042699746907, 19.081363475407006, 19.168077325829774, 18.979960703588816, 18.960901993379544, 19.085202579524726, 18.999560969030274, 19.711606769702964, 18.959488752069426, 20.172440615902723, 19.29982897043603, 20.30167667282796, 20.204167564783216, 19.55270459351527, 19.489361101608488, 19.868227551195588, 19.63629873639086, 19.727042384361628, 19.650460912172175, 20.304064722344762, 20.530433882082416, 19.815647233599815, 20.10753847428461, 20.4396515278063, 20.306096779316388, 20.61319684217262, 20.79427306573855, 20.50030667587243, 20.6337940539087, 20.444683167378784, 20.72371245122131, 20.144890882169804, 20.36948294609023, 20.206079840247245, 20.502400121874505, 20.651558224409296, 21.28869615462403, 21.300729820560587, 20.8884738709366, 20.363674394481194, 21.263709922926456, 21.190644948408696, 21.67965625019779, 21.672157253802666, 20.828498976146875, 20.533388979541037, 20.787073606771063, 21.3215631638706, 21.45988745682205, 21.8200930560308, 20.78817167231566, 21.655696960233414, 21.714002809028038, 21.866736774087958, 21.496232647602827, 21.514589951285405, 21.652090762772087, 21.473138258549696, 21.755693431891835, 21.255858480721358, 21.143571918920127, 22.029002663461362, 21.64474379663598, 21.940025206676346, 22.165775704622984, 22.04905783589783, 21.59529946870306, 22.254059528916766, 22.043365619383987, 22.1456767933878, 21.618004891876733, 21.03774232298335, 21.839062277998448, 21.76475316203705, 21.911048228507198, 22.079604134598345, 21.752365020342125, 21.822249546209413, 22.35343603606794, 21.148138818179262, 21.85643268659679, 22.121406400168098, 22.085430097085883, 22.653894448701333, 21.977986147875118, 21.921911937890975, 22.297120078238695, 22.066393283390376, 21.940620483870475, 22.235573940667088, 22.283730896453243, 22.634673881619737, 22.184573864106852, 22.203408337928114, 22.389830706449683, 22.43042728311132, 22.446823918539195, 22.044060302395227, 21.844482147140617, 22.77543008355557, 22.599038030903625, 22.57242381413829, 22.555253775533405, 22.48516432179059, 22.47232352416741, 22.425519482395455, 22.42933711405455, 22.754756160177056, 22.481979415628512, 22.614384920296025, 23.406009885583067, 22.81771388369111, 23.29872607532363, 23.416096087931436, 22.965717407618822, 22.929384674771033, 23.174763366150852, 23.146237793971597, 23.323793709515748, 23.595784120655967, 23.306981854777174, 23.198029117791275, 22.63874471439163, 23.15110418633385, 23.11791809286721, 23.598176818256775, 23.558965419102005, 23.276426473355116, 23.379731300823266, 23.43913676383632, 23.488234499067303, 23.282094444838254, 23.726945372178676, 23.525413020785443, 23.754440998475417, 23.436217257418015, 23.829225328503867, 22.961255437753874, 23.969318481298654, 24.117145710800806, 23.861927483847584, 23.966705747354556, 23.85984075903328, 24.153896945938225, 23.342786406280453, 23.429045950457304, 23.186800448251468, 23.67177143368521, 24.023554060603047, 24.078081394249793, 24.15116228716688, 24.29868128398168, 24.37892108053382, 24.16634732334588, 23.78905968944604, 23.99682474205593, 24.238707910184257, 23.613639579649146, 23.82335433971291, 24.63875722333858, 24.03743156203353, 24.080775971265105, 24.315146075914914, 24.590043843689905, 24.216200725563397, 24.76149381518966, 24.876524565210026, 24.85678605216847, 24.49169589110832, 24.59421602505435, 24.40681221171436, 24.375352695958586, 24.33337034631693, 25.01862482532535, 24.24420512893595, 23.777740104262747 ], "return_max": [ 0.0, -5.528899143567214, -5.2847934252628885, -4.985057045899643, -4.922391836410636, -4.82248808674592, -4.713502518816382, -4.654369483661602, -4.587032678530652, -4.616545452363403, -4.632783796247698, -4.701130763212384, -4.694670340501172, -4.592916909475119, -4.573042375215858, -4.6999134240376135, -4.975724783358038, -5.7850933362008465, -5.76081202789959, -6.5277796093105005, -6.89953720925723, -6.890058484707282, -7.323823604333456, -7.241327701763785, -7.219642123360369, -6.379027194341581, -5.8701763788600365, -5.148155427420367, -4.344935625696603, -3.688922263176798, -2.5964313821315916, -2.1326032096608274, -1.0264829184586266, -0.9584849562811515, -0.6944844915804289, -0.6874376473518944, 0.590496651906498, 1.6433117338685868, 0.9092874479675763, 1.6690469298340955, 2.510480308443307, 3.426348936563674, 3.8786181642389774, 3.7746224991494883, 4.669740897644988, 4.3832537519585655, 4.764030441913102, 5.122105512052707, 5.477977672830015, 5.863587303485141, 6.284394674520987, 5.813436872202251, 6.378140550158573, 6.217314054767584, 5.677387257143454, 6.189411040630592, 6.393631710371521, 6.602222179828026, 6.548276893983705, 6.37564949344078, 6.746906640911015, 6.85907181309405, 6.824192141908342, 7.208844726208271, 6.34483122546861, 7.453402050037185, 7.220624928744101, 7.195203712213297, 6.479465898949339, 6.121214655361287, 6.502920406070905, 6.468996865436241, 6.9163036123766055, 6.972557280130757, 7.221453526373775, 7.115414256786714, 8.013746716183057, 7.506424958643656, 7.711567083470725, 6.858940854960057, 7.30335022598698, 7.351964457160935, 7.1770168112341715, 7.154177247164066, 6.8261864580070775, 6.536068175170672, 7.087318374466043, 6.9907838431903135, 7.682661779404288, 6.875933761550209, 6.541946889745199, 7.191091691650969, 7.047391200866608, 7.277982966296823, 6.368034775437899, 6.574067193994262, 6.624805485481195, 7.202434995667042, 6.5540203033776905, 6.762448311799899, 6.826231396656613, 6.554370935991846, 6.411315421863841, 6.400042419748456, 6.32415812553996, 6.395120259372502, 6.6459660265097895, 6.0315384649942665, 5.843636800042105, 6.14565143644179, 6.108400498596499, 6.775207912770243, 6.127417179859371, 5.563699043404305, 5.700475986074164, 5.918484495279184, 6.308535130081836, 6.51597075864176, 6.357328631941017, 6.000814542199343, 6.239724195550861, 6.518503739563969, 6.435642572871952, 6.314293971555899, 6.627852936557242, 6.887628788829901, 6.699564458445542, 6.670845782750369, 6.883202433540342, 6.749645114550724, 6.956875516254199, 6.933405226460819, 7.421566762887166, 7.582685223193887, 7.072381858950101, 7.676125431959557, 7.332216586399404, 7.091210048340144, 7.3213486026737336, 7.414332448809812, 7.629496526035288, 7.961433398186197, 7.670203278454445, 7.6551110859026075, 7.754313087037881, 7.486295489387883, 7.441013646590369, 8.340004417896585, 8.097518145118864, 7.985698124069713, 8.095103973164491, 8.585715553156435, 8.31071993113763, 8.231273060162149, 8.270507836511683, 8.534791006579768, 8.624499278267058, 8.75915243478071, 8.969249254325213, 8.738624085351233, 9.106999431851769, 8.868254497418137, 8.95539774291936, 9.070415526358403, 8.377980949640495, 9.165170944303512, 9.169295566750584, 9.044721424346312, 9.333022597777859, 9.30282667714874, 9.674397635255923, 10.021400570622733, 10.153981117359585, 9.652488274298006, 10.350399642722374, 10.04229890228856, 10.18005822572231, 10.367334868493714, 10.420868205716216, 10.450285957954886, 11.153951377142906, 11.497433601785218, 11.069975231604413, 11.281148734610044, 11.301993781942116, 11.307505260376066, 11.825598976349095, 11.615833333829142, 11.059294364086895, 11.942231890073886, 12.412790972797673, 12.853469266564336, 11.760987650966815, 12.468005688486986, 12.161793903871128, 12.53424445649448, 12.282414187799674, 12.802941581514473, 12.546816791210176, 12.865049141664622, 13.017245854072993, 13.304992957911228, 13.256917728952077, 13.088906297674077, 13.508342451331778, 13.049344777594166, 12.835069158387194, 13.661599564333871, 13.651256753557035, 13.140623021663755, 13.905125367721565, 13.613382502012598, 13.964083121425956, 13.310342870042401, 13.885741992078579, 14.475768778254137, 14.387136952991495, 14.474212733671303, 14.807889549848317, 14.455016206170685, 15.590313966415563, 15.140368792364605, 14.723540501112062, 15.400334672691153, 15.887580882593921, 14.819916274076604, 15.817427106262675, 15.187692674071773, 15.538606729450272, 15.753767337074121, 15.713732705535376, 15.153522847959367, 15.511214885904863, 15.130005785944471, 15.663467034664903, 15.510681950490394, 16.398600630860155, 15.972039040311355, 15.587651982438693, 16.786807820567944, 16.580300483928532, 16.54175717271397, 15.927798469279248, 16.103504681912266, 16.38372008661014, 16.545517657545773, 16.549092593943175, 17.657328991356025, 17.168279655968202, 17.019703823871666, 17.070309048763193, 17.8242089570927, 17.927424802356633, 17.70476620049746, 17.585180691395895, 18.02318903102592, 17.983943089984713, 17.328108326549337, 17.884929745954672, 17.949815586014957, 18.255264679936378, 18.34636220094458, 18.508177911381416, 17.9127904629945, 18.02819241950077, 18.49655328073816, 18.528795840842157, 18.448481089140618, 18.6483497986429, 18.53373227512095, 18.758944455024068, 18.63216537168924, 18.237170868870077, 18.950657400784113, 18.64977681481168, 17.948451017597154, 18.69439209304038, 18.727288630645358, 19.20454694837485, 19.005300752671225, 19.287823610449816, 19.08048261794947, 18.949025702992323, 18.887106183141242, 18.553895319118528, 19.991007509124728, 18.753116980580742, 18.945750925986967, 19.984827076471458, 19.652784461265036, 19.505271378533365, 19.33595347716968, 19.48311117182208, 19.38005818410209, 19.608824854798346, 19.091611103590186, 19.331042924572337, 19.12947423069701, 19.70849482920568, 19.54661266473382, 19.510234323225674, 20.500484541145013, 19.845359248538525, 18.891507872972902, 20.524291336240903, 19.563555754367655, 19.974114431224116, 19.907623052984963, 19.67707303429976, 19.729423023632755, 19.91978669493325, 19.81276389568026, 19.98645992837354, 19.721098370168406, 20.52819580447863, 20.10696708989569, 20.365390260113504, 21.298843661931294, 20.939051882458234, 20.979725390220583, 19.79157607519489, 20.316728675348685, 20.7149032963126, 20.66322359965648, 21.27119353274641, 19.98508584407349, 20.55677298616882, 20.85469079652537, 21.07778072065384, 21.195515414609957, 20.339929828617358, 20.66872324689685, 21.071518139697982, 21.04520384214088, 21.143464144457994, 21.217461823618635, 21.429875537026163, 21.145302349653118, 21.043745795711008, 20.58240885116556, 21.343282687451875, 20.200383979025094, 21.451178059363965, 21.619305216685756, 21.332062520697228, 21.10310736655701, 21.387360310281917, 20.89845930905838, 21.56989093104583, 21.167020632614967, 21.906475791634243, 21.744692092723973, 21.776914547640857, 21.243895252032566, 21.437150331893882, 21.692433420258993, 21.815657695138853, 21.72287929247586, 22.451712596034593, 21.608442557467132, 21.663469551355174, 21.893672158847448, 21.80493538316968, 22.022551341669185, 21.790612300045474, 21.46733110636787, 21.702055265977293, 22.30986973234106, 22.423685101289706, 21.988744762027558, 22.306824314488203, 21.977901888567377, 22.472900678839252, 22.1330242348436, 22.18527488741504, 21.894536174643772, 22.862021881001432, 22.041001983185403, 22.157450533269827, 22.73521801659955, 22.62586677606038, 22.156560446105477, 22.29153104852326, 21.98192476968186, 22.768086062481352, 22.31144576379116, 22.593378379972776, 22.60128626063534, 22.29349618010599, 22.685488541681845, 22.244890798567713, 22.747761180532976, 23.083238250802264, 23.116030192968513, 22.675472387342374, 22.385198320543413, 22.59781179994146, 22.619361401671245, 22.53727297150024, 22.958748591179685, 22.463402550976507, 22.048217652412795, 22.837977072239475, 23.556697886826854, 22.676187927489792, 22.515709348569636, 22.643425045843763, 22.85996824631826, 22.95476751921198, 22.829777582809793, 22.912598982951515, 22.719814580447622, 22.805638878958057, 23.07998946238044, 22.63129017949842, 22.633887397308968, 23.211948917617836, 22.769160060123085, 22.60185139845088, 22.84416718211538, 23.11112598850022, 23.364512253882573, 23.452675540627443, 22.567508632100314, 23.039511047718335, 23.171125244497173, 22.992939060528556, 23.421997719192145, 23.52499782986397, 23.300259752263354, 23.257364269791648, 23.653193327229626, 24.00953387062027, 23.16723941861272, 23.671573028070767, 23.641733549322936, 24.192271391577446, 23.548095507788453, 23.90445290899391, 23.418094576443377, 23.644586677806178, 24.219468096327784, 23.557402279168866, 24.229635348877096, 24.17478872130621, 24.248783837615807, 23.873409627071606, 24.220687858926855, 24.036538728120302, 24.082331966896387, 24.665773097311252, 24.36662224295905, 24.29359387487588, 24.247460423953953, 24.302978458776696, 23.799348419735587, 24.14290362397126, 24.596163161044792, 24.534877296749954, 24.252972776650722, 24.656264098249096, 24.439341086011318, 24.598292585787892, 24.85057063975695, 25.095113874261862, 24.226178712873807, 24.818766982646256, 24.446995803123404, 24.82589165625497, 24.922233062980716, 25.045032116686453, 24.431437018560665, 24.756866322026674, 25.082090779001653, 24.979920015916626, 25.207713276772534, 24.964553936927437, 25.52679643859053, 24.989012262734036, 24.968180663184576, 25.15064303964675, 25.025970576003996, 25.528694565261027, 25.896593189126197, 25.34604165310592, 25.021970913323525, 25.46073613976293, 25.608820096621297, 25.17585508654183, 25.3327672112703, 25.571711409483292, 25.15718529781691, 26.164287187475903, 25.685735495203524, 25.557828941961226, 26.24222630019767, 25.56262475790901, 25.92796969007529, 25.59659540032671, 25.385419447632653, 26.17255821618962 ] } }, "Isaac-Repose-Cube-Shadow-Direct-v0": { "FastTD3": { "time": [ 0.0, 330.68918918918916, 661.3783783783783, 992.0675675675676, 1322.7567567567567, 1653.445945945946, 1984.1351351351352, 2314.824324324324, 2645.5135135135133, 2976.2027027027025, 3306.891891891892, 3637.5810810810813, 3968.2702702702704, 4298.959459459459, 4629.648648648648, 4960.3378378378375, 5291.027027027027, 5621.716216216216, 5952.405405405405, 6283.094594594595, 6613.783783783784, 6944.472972972973, 7275.1621621621625, 7605.851351351352, 7936.540540540541, 8267.22972972973, 8597.918918918918, 8928.608108108108, 9259.297297297297, 9589.986486486487, 9920.675675675675, 10251.364864864865, 10582.054054054053, 10912.743243243243, 11243.432432432432, 11574.121621621622, 11904.81081081081, 12235.5, 12566.18918918919, 12896.878378378378, 13227.567567567568, 13558.256756756757, 13888.945945945947, 14219.635135135135, 14550.324324324325, 14881.013513513513, 15211.702702702703, 15542.391891891892, 15873.081081081082, 16203.77027027027, 16534.45945945946, 16865.14864864865, 17195.837837837837, 17526.527027027027, 17857.216216216217, 18187.905405405407, 18518.594594594593, 18849.283783783783, 19179.972972972973, 19510.662162162163, 19841.35135135135, 20172.04054054054, 20502.72972972973, 20833.41891891892, 21164.108108108107, 21494.797297297297, 21825.486486486487, 22156.175675675677, 22486.864864864863, 22817.554054054053, 23148.243243243243, 23478.932432432433, 23809.62162162162, 24140.31081081081, 24471.0 ], "env_step": [ 0, 20480000.0, 40960000.0, 61440000.0, 81920000.0, 102400000.0, 122880000.0, 143360000.0, 163840000.0, 184320000.0, 204800000.0, 225280000.0, 245760000.0, 266240000.0, 286720000.0, 307200000.0, 327680000.0, 348160000.0, 368640000.0, 389120000.0, 409600000.0, 430080000.0, 450560000.0, 471040000.0, 491520000.0, 512000000.0, 532480000.0, 552960000.0, 573440000.0, 593920000.0, 614400000.0, 634880000.0, 655360000.0, 675840000.0, 696320000.0, 716800000.0, 737280000.0, 757760000.0, 778240000.0, 798720000.0, 819200000.0, 839680000.0, 860160000.0, 880640000.0, 901120000.0, 921600000.0, 942080000.0, 962560000.0, 983040000.0, 1003520000.0, 1024000000.0, 1044480000.0, 1064960000.0, 1085440000.0, 1105920000.0, 1126400000.0, 1146880000.0, 1167360000.0, 1187840000.0, 1208320000.0, 1228800000.0, 1249280000.0, 1269760000.0, 1290240000.0, 1310720000.0, 1331200000.0, 1351680000.0, 1372160000.0, 1392640000.0, 1413120000.0, 1433600000.0, 1454080000.0, 1474560000.0, 1495040000.0, 1515520000.0 ], "return": [ 0.0, 382.9488016764323, 3325.7025553385415, 5983.295735677083, 7569.175944010417, 8402.328776041666, 8895.064453125, 9108.5126953125, 9351.854166666666, 9457.5625, 9600.7568359375, 9676.693684895834, 9690.444010416666, 9769.830729166666, 9926.238606770834, 9924.583658854166, 9927.7548828125, 9980.768880208334, 10041.905598958334, 10035.136067708334, 10072.431640625, 10218.255533854166, 10219.3720703125, 10140.8955078125, 10195.677734375, 10264.537760416666, 10275.744466145834, 10346.177734375, 10295.432291666666, 10356.6630859375, 10349.065755208334, 10308.400716145834, 10315.165364583334, 10341.683268229166, 10255.400716145834, 10431.327799479166, 10433.752604166666, 10452.818684895834, 10408.553385416666, 10441.5556640625, 10436.6494140625, 10437.878255208334, 10553.256510416666, 10468.160481770834, 10498.621419270834, 10488.468424479166, 10543.875651041666, 10547.175455729166, 10521.928059895834, 10505.654947916666, 10564.725260416666, 10582.248046875, 10550.1796875, 10497.415690104166, 10528.411458333334, 10508.069986979166, 10514.443033854166, 10597.4140625, 10602.786783854166, 10624.103515625, 10591.981119791666, 10728.904622395834, 10649.1533203125, 10522.616536458334, 10581.818033854166, 10641.3125, 10679.970703125, 10570.703125, 10605.524739583334, 10627.1142578125, 10650.056966145834, 10705.324869791666, 10595.880859375, 10693.9228515625, 10580.137044270834 ], "return_min": [ 0.0, 359.41059655147757, 2949.856590942412, 5776.541036955849, 7395.004803407598, 8248.132074280249, 8828.972871967186, 8958.659157152422, 9303.277804746735, 9412.228024899145, 9468.185128938925, 9569.948018047084, 9621.435716430296, 9699.129445877223, 9816.969484996353, 9769.014548711579, 9800.333162033327, 9923.72243183005, 10007.683047995282, 10021.136837813956, 10014.480328880763, 10184.788847313212, 10133.861623316618, 10067.86992411191, 10167.232640558608, 10193.601492220183, 10155.176689746373, 10245.377468169529, 10250.017587387996, 10272.533376427831, 10282.273485433534, 10234.121208538385, 10256.477859592766, 10269.329184206632, 10134.271242258248, 10374.883823081536, 10396.353590283215, 10322.026966766209, 10373.607044323328, 10369.999796834647, 10364.3489023407, 10367.793788095061, 10471.49904773048, 10392.126377871275, 10455.048923201117, 10359.731093856988, 10457.410493575546, 10457.966017755905, 10424.317180105267, 10399.902885386984, 10495.50258255835, 10495.068429035728, 10444.166750515762, 10321.152995551243, 10445.516438211675, 10460.457192528489, 10451.086833492674, 10475.111650363446, 10487.424596133797, 10618.090402098596, 10435.907537302917, 10626.064610097928, 10564.51645361193, 10429.779904509182, 10460.358847702342, 10615.205139810138, 10551.641075842563, 10454.799861652631, 10486.209465501834, 10502.08255684319, 10509.412334038356, 10596.807083458754, 10527.152332367086, 10618.574418628925, 10455.838224661382 ], "return_max": [ 0.0, 406.48700680138705, 3701.548519734671, 6190.050434398317, 7743.347084613236, 8556.525477803083, 8961.156034282814, 9258.366233472578, 9400.430528586598, 9502.896975100855, 9733.328542936075, 9783.439351744584, 9759.452304403036, 9840.53201245611, 10035.507728545315, 10080.152768996753, 10055.176603591673, 10037.815328586617, 10076.128149921386, 10049.135297602712, 10130.382952369237, 10251.72222039512, 10304.882517308382, 10213.92109151309, 10224.122828191392, 10335.47402861315, 10396.312242545295, 10446.978000580471, 10340.846995945336, 10440.792795447169, 10415.858024983134, 10382.680223753283, 10373.852869573902, 10414.0373522517, 10376.53019003342, 10487.771775876796, 10471.151618050117, 10583.61040302546, 10443.499726510005, 10513.111531290353, 10508.9499257843, 10507.962722321607, 10635.013973102852, 10544.194585670393, 10542.193915340551, 10617.205755101344, 10630.340808507786, 10636.384893702427, 10619.538939686401, 10611.407010446348, 10633.947938274981, 10669.427664714272, 10656.192624484238, 10673.678384657089, 10611.306478454993, 10555.682781429843, 10577.799234215658, 10719.716474636554, 10718.148971574536, 10630.116629151404, 10748.054702280415, 10831.74463469374, 10733.79018701307, 10615.453168407486, 10703.27722000599, 10667.419860189862, 10808.300330407437, 10686.606388347369, 10724.840013664834, 10752.14595878181, 10790.701598253312, 10813.842656124578, 10664.609386382914, 10769.271284496075, 10704.435863880286 ] }, "null": { "time": [ 0.0, 27.434, 54.868, 82.302, 109.736, 137.17, 164.604, 192.038, 219.472, 246.906, 274.34, 301.774, 329.208, 356.642, 384.076, 411.51, 438.944, 466.378, 493.812, 521.246, 548.68, 576.114, 603.548, 630.982, 658.416, 685.85, 713.284, 740.718, 768.152, 795.586, 823.02, 850.454, 877.888, 905.322, 932.756, 960.19, 987.624, 1015.058, 1042.492, 1069.926, 1097.36, 1124.794, 1152.228, 1179.662, 1207.096, 1234.53, 1261.964, 1289.398, 1316.832, 1344.266, 1371.7, 1399.134, 1426.568, 1454.002, 1481.436, 1508.87, 1536.304, 1563.738, 1591.172, 1618.606, 1646.04, 1673.474, 1700.908, 1728.342, 1755.776, 1783.21, 1810.644, 1838.078, 1865.512, 1892.946, 1920.38, 1947.814, 1975.248, 2002.682, 2030.116, 2057.55, 2084.984, 2112.418, 2139.852, 2167.286, 2194.72, 2222.154, 2249.588, 2277.022, 2304.456, 2331.89, 2359.324, 2386.758, 2414.192, 2441.626, 2469.06, 2496.494, 2523.928, 2551.362, 2578.796, 2606.23, 2633.664, 2661.098, 2688.532, 2715.966, 2743.4, 2770.834, 2798.268, 2825.702, 2853.136, 2880.57, 2908.004, 2935.438, 2962.872, 2990.306, 3017.74, 3045.174, 3072.608, 3100.042, 3127.476, 3154.91, 3182.344, 3209.778, 3237.212, 3264.646, 3292.08, 3319.514, 3346.948, 3374.382, 3401.816, 3429.25, 3456.684, 3484.118, 3511.552, 3538.986, 3566.42, 3593.854, 3621.288, 3648.722, 3676.156, 3703.59, 3731.024, 3758.458, 3785.892, 3813.326, 3840.76, 3868.194, 3895.628, 3923.062, 3950.496, 3977.93, 4005.364, 4032.798, 4060.232, 4087.666, 4115.1, 4142.534, 4169.968, 4197.402, 4224.836, 4252.27, 4279.704, 4307.138, 4334.572, 4362.006, 4389.44, 4416.874, 4444.308, 4471.742, 4499.176, 4526.61, 4554.044, 4581.478, 4608.912, 4636.346, 4663.78, 4691.214, 4718.648, 4746.082, 4773.516, 4800.95, 4828.384, 4855.818, 4883.252, 4910.686, 4938.12, 4965.554, 4992.988, 5020.422, 5047.856, 5075.29, 5102.724, 5130.158, 5157.592, 5185.026, 5212.46, 5239.894, 5267.328, 5294.762, 5322.196, 5349.63, 5377.064, 5404.498, 5431.932, 5459.366, 5486.8, 5514.234, 5541.668, 5569.102, 5596.536, 5623.97, 5651.404, 5678.838, 5706.272, 5733.706, 5761.14, 5788.574, 5816.008, 5843.442, 5870.876, 5898.31, 5925.744, 5953.178, 5980.612, 6008.046, 6035.48, 6062.914, 6090.348, 6117.782, 6145.216, 6172.65, 6200.084, 6227.518, 6254.952, 6282.386, 6309.82, 6337.254, 6364.688, 6392.122, 6419.556, 6446.99, 6474.424, 6501.858, 6529.292, 6556.726, 6584.16, 6611.594, 6639.028, 6666.462, 6693.896, 6721.33, 6748.764, 6776.198, 6803.632, 6831.066, 6858.5, 6885.934, 6913.368, 6940.802, 6968.236, 6995.67, 7023.104, 7050.538, 7077.972, 7105.406, 7132.84, 7160.274, 7187.708, 7215.142, 7242.576, 7270.01, 7297.444, 7324.878, 7352.312, 7379.746, 7407.18, 7434.614, 7462.048, 7489.482, 7516.916, 7544.35, 7571.784, 7599.218, 7626.652, 7654.086, 7681.52, 7708.954, 7736.388, 7763.822, 7791.256, 7818.69, 7846.124, 7873.558, 7900.992, 7928.426, 7955.86, 7983.294, 8010.728, 8038.162, 8065.596, 8093.03, 8120.464, 8147.898, 8175.332, 8202.766, 8230.2, 8257.634, 8285.068, 8312.502, 8339.936, 8367.37, 8394.804, 8422.238, 8449.672, 8477.106, 8504.54, 8531.974, 8559.408, 8586.842, 8614.276, 8641.71, 8669.144, 8696.578, 8724.012, 8751.446, 8778.88, 8806.314, 8833.748, 8861.182, 8888.616, 8916.05, 8943.484, 8970.918, 8998.352, 9025.786, 9053.22, 9080.654, 9108.088, 9135.522, 9162.956, 9190.39, 9217.824, 9245.258, 9272.692, 9300.126, 9327.56, 9354.994, 9382.428, 9409.862, 9437.296, 9464.73, 9492.164, 9519.598, 9547.032, 9574.466, 9601.9, 9629.334, 9656.768, 9684.202, 9711.636, 9739.07, 9766.504, 9793.938, 9821.372, 9848.806, 9876.24, 9903.674, 9931.108, 9958.542, 9985.976, 10013.41, 10040.844, 10068.278, 10095.712, 10123.146, 10150.58, 10178.014, 10205.448, 10232.882, 10260.316, 10287.75, 10315.184, 10342.618, 10370.052, 10397.486, 10424.92, 10452.354, 10479.788, 10507.222, 10534.656, 10562.09, 10589.524, 10616.958, 10644.392, 10671.826, 10699.26, 10726.694, 10754.128, 10781.562, 10808.996, 10836.43, 10863.864, 10891.298, 10918.732, 10946.166, 10973.6, 11001.034, 11028.468, 11055.902, 11083.336, 11110.77, 11138.204, 11165.638, 11193.072, 11220.506, 11247.94, 11275.374, 11302.808, 11330.242, 11357.676, 11385.11, 11412.544, 11439.978, 11467.412, 11494.846, 11522.28, 11549.714, 11577.148, 11604.582, 11632.016, 11659.45, 11686.884, 11714.318, 11741.752, 11769.186, 11796.62, 11824.054, 11851.488, 11878.922, 11906.356, 11933.79, 11961.224, 11988.658, 12016.092, 12043.526, 12070.96, 12098.394, 12125.828, 12153.262, 12180.696, 12208.13, 12235.564, 12262.998, 12290.432, 12317.866, 12345.3, 12372.734, 12400.168, 12427.602, 12455.036, 12482.47, 12509.904, 12537.338, 12564.772, 12592.206, 12619.64, 12647.074, 12674.508, 12701.942, 12729.376, 12756.81, 12784.244, 12811.678, 12839.112, 12866.546, 12893.98, 12921.414, 12948.848, 12976.282, 13003.716, 13031.15, 13058.584, 13086.018, 13113.452, 13140.886, 13168.32, 13195.754, 13223.188, 13250.622, 13278.056, 13305.49, 13332.924, 13360.358, 13387.792, 13415.226, 13442.66, 13470.094, 13497.528, 13524.962, 13552.396, 13579.83, 13607.264, 13634.698, 13662.132, 13689.566, 13717.0 ], "env_step": [ 0, 1441792, 2883584, 6422528, 11141120, 12976128, 13238272, 13631488, 21757952, 21889024, 24248320, 32112640, 35782656, 39976960, 46137344, 47185920, 51511296, 52297728, 54525952, 55705600, 57409536, 59244544, 59899904, 65142784, 73531392, 73793536, 74055680, 80740352, 87031808, 88473600, 88735744, 91488256, 95551488, 96731136, 103022592, 103284736, 104595456, 108527616, 110493696, 111280128, 120061952, 121503744, 125173760, 130154496, 130285568, 131858432, 135659520, 141295616, 144048128, 147587072, 152567808, 153616384, 156237824, 156499968, 157417472, 158990336, 160432128, 162267136, 163315712, 163708928, 167510016, 170393600, 170655744, 172621824, 173801472, 175374336, 176816128, 178388992, 182190080, 182845440, 184156160, 185860096, 187695104, 187957248, 190447616, 192806912, 192937984, 198967296, 202375168, 203423744, 206176256, 209453056, 220463104, 221380608, 221511680, 222298112, 223870976, 226361344, 228720640, 232259584, 233570304, 240779264, 252444672, 253755392, 254935040, 266076160, 268697600, 269090816, 270794752, 272498688, 274202624, 274333696, 278921216, 284688384, 287834112, 289538048, 296222720, 296484864, 300023808, 301072384, 309592064, 310378496, 313393152, 315228160, 315621376, 316014592, 319029248, 319160320, 319684608, 322568192, 324403200, 325058560, 325582848, 329646080, 331087872, 331743232, 333053952, 335020032, 336461824, 343015424, 345767936, 346030080, 348127232, 348651520, 353370112, 356384768, 358219776, 362938368, 369098752, 374210560, 375652352, 376438784, 379846656, 380502016, 385875968, 386793472, 391118848, 394395648, 395051008, 396492800, 399376384, 403570688, 405405696, 406847488, 407240704, 408944640, 409862144, 411172864, 417333248, 426639360, 431226880, 433717248, 433979392, 436469760, 439091200, 439353344, 440532992, 440926208, 442630144, 445513728, 447873024, 462684160, 465698816, 466092032, 467927040, 468975616, 469106688, 469368832, 470024192, 477888512, 478937088, 481558528, 482869248, 486146048, 490995712, 492044288, 493617152, 494272512, 496762880, 497549312, 497942528, 498860032, 500039680, 504365056, 510656512, 511967232, 513409024, 515244032, 516423680, 517865472, 518389760, 520355840, 520880128, 524288000, 532938752, 534249472, 537395200, 537526272, 537919488, 538181632, 540278784, 544735232, 547094528, 548143104, 549847040, 551288832, 563216384, 564264960, 567017472, 567672832, 569114624, 570163200, 570687488, 571211776, 572129280, 575537152, 581566464, 583008256, 594804736, 596639744, 597557248, 603193344, 606601216, 607387648, 609353728, 610533376, 612368384, 615120896, 616038400, 617480192, 619970560, 622067712, 627965952, 628359168, 628490240, 629932032, 633733120, 636092416, 638189568, 638713856, 639893504, 642121728, 643694592, 649592832, 650510336, 653262848, 660340736, 662831104, 662962176, 664535040, 664928256, 666501120, 666632192, 669384704, 682229760, 683147264, 683409408, 686817280, 688652288, 691142656, 694943744, 695599104, 695730176, 708837376, 712376320, 713293824, 714604544, 715784192, 719716352, 720109568, 721813504, 722337792, 725090304, 726138880, 741998592, 747896832, 757989376, 760610816, 761135104, 762183680, 762707968, 762970112, 764149760, 768868352, 775028736, 782893056, 783286272, 786432000, 788529152, 789184512, 798097408, 801243136, 804388864, 809238528, 810156032, 810287104, 810549248, 812646400, 821035008, 821821440, 821952512, 826015744, 826540032, 827064320, 830734336, 833224704, 834928640, 835977216, 836501504, 837550080, 839516160, 844234752, 845152256, 845283328, 848035840, 848953344, 853934080, 858259456, 861929472, 865992704, 866779136, 866910208, 870055936, 874905600, 875560960, 876216320, 876347392, 877527040, 880279552, 884080640, 886571008, 889716736, 892993536, 893911040, 895614976, 899284992, 899809280, 903872512, 905707520, 908984320, 917897216, 919076864, 921436160, 928251904, 929431552, 929693696, 933101568, 933756928, 933888000, 934412288, 937426944, 940965888, 941096960, 942014464, 944766976, 945815552, 949354496, 953679872, 954204160, 955121664, 955645952, 958791680, 959578112, 961019904, 962199552, 963641344, 971112448, 978321408, 978452480, 991952896, 995622912, 998113280, 1012662272, 1013055488, 1013317632, 1014497280, 1014890496, 1021968384, 1027866624, 1028784128, 1030488064, 1033371648, 1034944512, 1035993088, 1037434880, 1040187392, 1040711680, 1044512768, 1049231360, 1050411008, 1050673152, 1054605312, 1056309248, 1057619968, 1058275328, 1062731776, 1064173568, 1065353216, 1068236800, 1087504384, 1091567616, 1093402624, 1099300864, 1104150528, 1105068032, 1105723392, 1105985536, 1106771968, 1107034112, 1111752704, 1114243072, 1114898432, 1115947008, 1117650944, 1118175232, 1118568448, 1118961664, 1120796672, 1124728832, 1126170624, 1126432768, 1127612416, 1129447424, 1130102784, 1134166016, 1138229248, 1138753536, 1139015680, 1140064256, 1140195328, 1140850688, 1143472128, 1144782848, 1152647168, 1157890048, 1160249344, 1160773632, 1163264000, 1168113664, 1171521536, 1175715840, 1176109056, 1177157632, 1177288704, 1178861568, 1180696576, 1181483008, 1184235520, 1186201600, 1186463744, 1186856960, 1193541632, 1199046656, 1207566336, 1212547072, 1220935680, 1231421440, 1232338944, 1234173952, 1234829312, 1236402176, 1237975040, 1241513984, 1247281152, 1249247232, 1260519424, 1260650496, 1260912640, 1265106944, 1266024448, 1266679808, 1267466240, 1267597312, 1270743040, 1271267328, 1274281984, 1275592704, 1279918080, 1281490944, 1282670592, 1282932736, 1283063808, 1286995968, 1289224192, 1294073856, 1294467072, 1294729216, 1295253504, 1302331392, 1303117824, 1308622848 ], "return": [ 0.0, 5.236283680185677, 102.00346561616661, 135.75974839633702, 217.9203052608967, 274.8097470428944, 280.9905634007454, 299.1841593341827, 320.27578971481324, 432.8313995814324, 447.9990608375072, 584.088671979189, 699.7305512046814, 878.0466557469368, 1019.8984030554295, 1002.1022647475005, 1077.601406442404, 1216.5086271886826, 1344.7774969083666, 1438.1453649656773, 1527.9062448343634, 1598.799950851202, 1724.6216583614346, 1799.3939088706968, 1875.027166177392, 1938.843445581436, 1990.5927048807146, 2046.135383161068, 2154.9252163674832, 2328.1289598841663, 2251.4501824004647, 2331.8998207616805, 2466.500129586458, 2404.7145573153493, 2583.091887678146, 2663.2294233992097, 2617.075662468195, 2711.554448129177, 2837.6077219905856, 2781.6240523614883, 2832.8479318242075, 2925.1859030513765, 2881.0791418402196, 3072.1628783736232, 2981.710749827385, 3235.595831243992, 3119.2624001512527, 3235.403108201504, 3242.492927195132, 3126.3234116115573, 3290.1375757427218, 3258.472759049416, 3309.0437541728015, 3313.4140671682353, 3432.629024629116, 3499.5061689734457, 3488.21505269146, 3543.742109605789, 3562.137778020858, 3650.741189673424, 3654.8448517746924, 3567.923550039291, 3579.626926459789, 3639.0841007270815, 3597.3003351850507, 3683.1284365525244, 3722.5531045532234, 3735.0221378602982, 3698.455936802864, 3723.6692370138167, 3766.410806018829, 3831.3407022600177, 3729.1116201820378, 3781.410314613342, 3944.3908525352476, 3908.5141741094594, 3913.3661565508837, 3948.811793651581, 3896.9180006217953, 3912.389421329498, 3971.1608142261503, 3826.356984316826, 4092.6435207028394, 4122.543938989639, 4019.705203544617, 4088.232182945013, 3999.3917568225866, 3998.7158728313443, 4117.989953351974, 4136.921306218624, 4137.75292036438, 4140.776959014892, 4129.997290261269, 4199.470038002968, 4129.829729902267, 4184.820166660308, 4154.497287431716, 4140.801594711304, 4223.778666554451, 4230.720368682862, 4335.208910928726, 4320.587505472183, 4279.5891404767035, 4372.656096599103, 4213.4231868834495, 4271.961008087159, 4305.597461331367, 4328.0909260568615, 4414.040591306687, 4414.380168487549, 4329.981696447373, 4426.483426190853, 4378.840885532379, 4353.974874118805, 4407.2164109535215, 4410.12230081749, 4318.541604546071, 4506.371503078461, 4457.252771657944, 4510.632806407929, 4467.406568637848, 4413.122340072632, 4427.9682405395515, 4320.175341485977, 4497.8624214315405, 4578.464052150726, 4498.392016014099, 4478.23864899826, 4521.771654136657, 4512.799011329651, 4485.047106071472, 4448.143535078049, 4536.469063568115, 4482.167032763004, 4461.829647455215, 4554.236630702972, 4481.976551002503, 4594.135609493256, 4576.309885606766, 4533.668542879104, 4593.2214253082275, 4597.23950636673, 4558.801636837006, 4625.431267238617, 4764.541600046157, 4643.302667323589, 4543.5092151412955, 4569.829136842728, 4691.1080415382385, 4614.56410823822, 4712.355486083985, 4627.576166591644, 4757.001649238587, 4754.4851669654845, 4628.6072655735015, 4665.539616868973, 4598.129826223374, 4725.939671382905, 4612.051948223114, 4712.190674921036, 4647.940960597992, 4706.48961642456, 4784.072993350982, 4740.444469493867, 4751.443206726074, 4626.085614406586, 4828.636562986374, 4830.9071949653635, 4888.817148040771, 4799.730001449585, 4803.863621171951, 4859.824459934234, 4789.379549774169, 4779.189342220307, 4762.207045719147, 4787.940696491241, 4791.509553516387, 4779.069134399413, 4833.9439957618715, 4925.727799835206, 4792.0102535820015, 4803.0068109359745, 4868.051529243469, 4796.148416542053, 4738.990540931702, 4707.000625541687, 4889.161682231903, 4860.448832752228, 4938.442969347, 4888.24940235138, 4962.321301048278, 4835.466524139405, 4850.661842075348, 4886.989757774352, 4896.520996544838, 4915.5101806182865, 4850.981204780578, 4777.362599323273, 4771.4760668907165, 4934.89265607071, 4974.275241218567, 4727.025384067535, 4763.875608978271, 4932.105666618347, 4788.006986774444, 4963.5356856842045, 4975.0135208892825, 4855.027117713928, 4849.923748680115, 4722.474940223694, 4853.057108760834, 4936.012247238159, 4823.776972335816, 4981.17521820259, 4858.281689052582, 4863.421200771332, 4954.844341381073, 4927.6710865173345, 4907.429337055206, 4927.645297805787, 4883.977951461792, 4842.6132472763065, 4913.963537078857, 4923.129209091187, 4895.095619495392, 4766.5804323959355, 4873.155703453063, 4971.745718772889, 4847.09561015892, 5030.245681488037, 4974.522681049347, 4934.959723098755, 5034.568326183319, 4819.898979125976, 5042.744435153962, 5037.858683376312, 4964.680161628724, 5014.581190307617, 4910.504225143433, 5046.842536304474, 4956.216326187134, 4993.427708992004, 5011.969644168854, 5013.753195144653, 5012.076093692779, 5007.031513072967, 4977.757820075988, 4918.5938954772955, 5027.504535961151, 5029.7741360549935, 5097.065293293, 5031.401218261719, 5118.090171936035, 4980.158747291565, 5087.3497388954165, 4997.21746483612, 5049.89016718483, 5052.48202145338, 4998.701796848297, 4999.208668247224, 4886.12294643402, 5026.853234748841, 4984.986180892944, 4995.789886810303, 5111.35556067276, 4959.245058212281, 5026.2039459877005, 5023.929447631836, 5109.746718658448, 5074.376200294495, 5043.018234619141, 4994.723159210205, 5033.370203350068, 4994.979048362731, 5050.698421276093, 4970.567650695801, 5142.736372253418, 5206.550512470245, 4957.855958114624, 5110.360929222106, 4998.667523242951, 5110.177480525972, 5062.622328849792, 4984.220653633118, 5052.763286636353, 5011.880573486328, 4979.447981903077, 5115.19174407959, 5032.8902960281375, 4902.994464187623, 5029.657383789063, 5048.658683044434, 5028.910795043946, 4909.834355426789, 4911.035541084289, 5085.230002365113, 5012.631169448853, 5113.218371482849, 5160.849894561767, 5167.585114463806, 5095.002285575867, 5172.043252494812, 4730.221512008667, 4915.24465864563, 4942.3944827651985, 5003.76211641693, 5107.787829620362, 5158.120981674195, 4982.206357467652, 5077.543367221832, 4966.418243301392, 5100.223124847412, 4913.418493736267, 4970.639212806702, 5088.850089622498, 5057.40702806759, 5014.9243317947385, 5020.501419372559, 4948.997114341736, 5052.762624259949, 5032.860914924621, 5054.642137229919, 4985.218594497681, 5033.177627876282, 5066.468012580872, 5092.042655509949, 5112.60638760376, 5105.836066616058, 5162.45666355896, 5147.268983337402, 5008.682528160095, 5037.204464515687, 5168.267694854736, 5033.9065711059575, 5066.976074508666, 5196.279511566162, 5093.294189910889, 5123.048552562714, 5061.140061874389, 5010.1269136505125, 5056.934467567444, 5041.947351013184, 4975.713252967835, 5049.654645835876, 4962.487987426758, 5010.011297958375, 4965.694996154785, 4945.400082908631, 5028.638770584106, 4975.87925856781, 5109.705345397949, 5117.070566131592, 5078.346026298523, 5131.825363212586, 5016.282830970764, 5116.730377670288, 5131.330189277649, 4940.001313926697, 5057.009353103637, 5024.000999679566, 5004.674883605958, 5155.328939086914, 5043.259490737915, 5130.654353294372, 5165.902773330688, 5011.435378509521, 5092.643320243836, 5026.348706832885, 4994.030126960754, 5007.96946346283, 4934.862240600586, 5056.537859802245, 5079.81287551117, 5065.511166244507, 5100.1370931091315, 5057.302691589355, 5006.724618148803, 4968.527329902649, 5003.266977737427, 5129.967458953857, 5059.039959884643, 4958.431204872131, 5055.456167999268, 4921.641325973511, 5108.192240783692, 5031.896564933777, 5050.812078147888, 5052.395368637085, 5183.077568122863, 5005.689804779053, 5095.717711608887, 4981.185117408752, 5037.46228678894, 5057.316320030212, 4998.430526168823, 4994.4445332031255, 4908.080332427979, 5160.852257247925, 4942.386556732178, 4977.80005708313, 4999.5668801116935, 5007.867194953918, 4988.167340057373, 5165.724187385559, 5148.104109390259, 4887.678792877197, 5108.603537368775, 5116.676084091187, 5039.772944686889, 5130.552591812134, 5052.179794326782, 5101.057332191468, 5050.516229759216, 5028.3649343185425, 5132.270079299927, 5086.8566754608155, 5134.982005775452, 5245.009395843505, 5210.982839965821, 5080.490185150146, 4979.681203170776, 5064.320907157898, 5119.925566406249, 5057.879759536743, 5102.072025909424, 5031.215700546264, 5019.1513743896485, 5086.748718887329, 5121.011468109131, 4908.945695343017, 5117.348231658935, 4964.579453414916, 4975.248231781006, 4773.869886505127, 5031.6520500183115, 5036.789762901307, 5068.320360412597, 5040.455801651001, 5126.994954360962, 5097.743839080811, 5033.6443821945195, 5011.765733551025, 5089.503886322022, 5084.362827529907, 5128.805971763611, 4963.232673416138, 4993.680429214478, 5017.836319595337, 4977.133294456481, 5038.540714752197, 4958.906798588752, 4985.946089561463, 5016.781194366455, 5072.254841171265, 4999.833683868407, 5056.062237426758, 5017.829210861206, 5101.638251464844, 4970.891885879517, 5022.780729690552, 5060.428104476929, 4972.043661186219, 5053.362578033448, 5031.0058487396245, 4934.2498914031985, 5110.03906829834, 4950.636026023864, 4875.610343353272, 4976.717636108398, 5058.674133804321, 5123.543363433839, 5034.8297764587405, 5080.173645599365, 5067.5694812011725, 5051.071068786621, 5086.857314743042, 5068.077889968872, 5049.054003616333, 5146.766270660401, 4886.542053894043, 5008.992813140869, 4992.11653074646, 4966.736727882385, 5051.737206115723, 5065.692213798522, 5092.993588012695, 5089.436850601196, 5091.006365074158, 4989.2801470794675, 4894.385232406616, 4994.612436340332, 5031.422633369445, 5046.974989196778, 5007.333500549317, 4881.702481468201, 4976.65591708374, 4950.086229003907, 5119.464210418701, 4998.693757247925, 4954.041024032593, 5041.931761985778 ], "return_min": [ 0.0, -19.14411247696058, -121.16046486277807, -111.67575958425746, -64.1436141826434, -17.015392825342985, -22.940368605642163, -3.986633963480813, 16.45188670459504, 6.712078908638375, 23.85082779635968, 86.26909947922843, 2.3765774458348687, 44.0580940604425, 51.74676530953434, 94.09635517895913, 133.5802702677007, 118.54627781556928, 230.43897668253862, 149.94855524555805, 164.4603191881199, 294.78872684605767, 387.44056976074944, 404.0936291535736, 441.647768772071, 467.92491306047145, 497.5399510083257, 528.3376665133649, 568.4442803620686, 593.6743880036522, 638.3961789571385, 646.1292398391315, 726.1854735064071, 760.678543075496, 813.7057680064918, 811.1814907472349, 806.9400518326188, 885.9346247741264, 974.0043227807846, 943.5976664970865, 970.6241361595141, 1023.7487332620542, 1004.06703673836, 1058.59503174645, 1038.537140980236, 1112.7265178803164, 1108.2450790246705, 1162.0126130055623, 1140.1430752161418, 1127.3474259063005, 1209.4124047888185, 1227.0148951328126, 1254.5907390533848, 1259.2118933961092, 1304.3821379264937, 1341.8510404397261, 1334.1638840733267, 1387.7054534142812, 1388.493792272047, 1406.9151574162347, 1411.3257125421574, 1402.4307723224233, 1365.9277026352697, 1423.9381890492577, 1436.6417972550985, 1526.7958849973243, 1490.2378308979005, 1524.4179032350703, 1508.60452650845, 1571.3854383959533, 1671.377841916741, 1602.3149013798893, 1606.342585436565, 1610.0229047545126, 1687.299419306462, 1692.328175549469, 1700.26400600844, 1733.556046192612, 1698.785655133484, 1726.406747475267, 1778.1068786005849, 1740.454015587948, 1771.6557512560412, 1776.77397064449, 1775.8776709323652, 1776.3228369000517, 1792.5858563221318, 1774.7354070048045, 1800.8583504193862, 1816.794544878237, 1823.984431059709, 1896.8176100192845, 1897.2129955425148, 1853.6753946156741, 1835.3116948934676, 1941.9333416585014, 1910.544436016929, 1880.463058796557, 1867.5256232219508, 1878.544475364241, 1931.870927414434, 1973.7462233945084, 1920.783859465787, 1999.0382888402146, 1938.3399576500383, 1955.910168943888, 1939.1607659918436, 1953.6593699900172, 2012.3137686933655, 2057.846870538191, 1998.6500577518632, 2024.7936330469051, 2016.4967539773697, 2084.3824818781127, 2068.8351000636003, 2058.411771772281, 1999.7633880198582, 2131.007436038327, 2067.0917258319623, 2048.4664728104117, 2079.7951743338467, 2095.6410366821565, 2080.528611406117, 2095.2630976565965, 2191.3286345660763, 2185.63603676565, 2150.691076199537, 2136.198161291987, 2188.951956508231, 2180.4250874574514, 2169.7256105084693, 2154.997344072441, 2213.5608371490653, 2210.2927707706976, 2153.256393948539, 2235.717427827676, 2194.490043243389, 2264.51700837704, 2173.7039562347713, 2173.804066878409, 2244.1961785150534, 2234.912982382214, 2225.976085695263, 2242.2668054749147, 2329.0418085188944, 2305.4428641971654, 2223.966626169306, 2318.948059745077, 2283.2458759037136, 2248.311774422755, 2284.8161171890733, 2323.620721687716, 2276.211854106748, 2374.589542585502, 2369.418336175175, 2328.881011003339, 2348.1115426429883, 2378.4947609359847, 2333.725166696244, 2269.919633914457, 2325.4606795781906, 2343.6328501169664, 2430.4667807822657, 2421.2777490354506, 2364.3030357536536, 2371.5663799555314, 2438.1595922788797, 2407.628486342669, 2448.5114309384257, 2427.3816029917816, 2427.8041511026536, 2426.303159435416, 2396.4504228395253, 2464.8680439436444, 2531.8481655959376, 2459.342234503615, 2429.461720649014, 2443.596154596039, 2490.218512085239, 2537.18964757073, 2481.942569079252, 2489.58739873738, 2544.8305272005277, 2434.2977132683172, 2546.201156270302, 2481.0448799529127, 2508.2578790292987, 2506.982385477666, 2543.4425662020126, 2526.6750452729334, 2535.2401693878924, 2420.4033001284233, 2529.9318837139185, 2464.7313204897723, 2455.062815013207, 2499.7594748315646, 2596.718039013545, 2498.3330112661365, 2501.0188623383533, 2599.0403583265006, 2607.1146976611967, 2439.0988796102047, 2526.370944260332, 2576.28989670563, 2511.707113022177, 2583.7177867994187, 2596.082599644324, 2554.6982037986077, 2529.74288248526, 2557.2585855898205, 2548.3530088858483, 2645.7672715369263, 2530.241552844906, 2654.4528062686954, 2586.362923416898, 2559.3633733143765, 2657.8307338445657, 2559.81400870566, 2592.6661708658503, 2602.55990188937, 2538.8301559972024, 2526.818866221739, 2585.3523332974496, 2554.961569267399, 2618.709741830747, 2605.355065797593, 2533.366643731315, 2603.488801614006, 2623.9244136739503, 2604.5956721182524, 2582.0186102727416, 2488.195110192989, 2678.5429039689307, 2629.859266587499, 2681.822785369794, 2673.388099328283, 2631.2053841972024, 2683.1040794644287, 2658.8158941097913, 2625.0378821215463, 2618.371798162222, 2614.0333241682824, 2716.2779532738177, 2715.674905375105, 2608.987316665234, 2690.6974059137247, 2701.197499910661, 2603.51085242943, 2619.8276987693166, 2693.6704249315353, 2693.6398598498545, 2643.599226087435, 2706.0174013939577, 2683.98035596259, 2644.515138217409, 2647.779150677115, 2714.6857252242958, 2667.5344769851313, 2743.8487219943686, 2703.994709429201, 2652.7106753653225, 2687.259255540809, 2689.8341723747, 2702.3172134851657, 2690.619898558037, 2690.20159914257, 2731.2882072746784, 2724.5897084520916, 2793.4886357198225, 2786.153795751157, 2752.4751036105013, 2727.4717608288397, 2731.200797820936, 2731.4992825409627, 2743.995331826182, 2726.718163468208, 2791.2976764991945, 2873.345345243874, 2689.543749833033, 2698.1604599580855, 2717.14020210079, 2802.7555602870093, 2662.787734710689, 2733.5043919293325, 2703.9124319961306, 2754.7889111106833, 2688.629707960803, 2799.943871259781, 2761.855562336642, 2702.762397210271, 2766.4033078959274, 2746.765063949556, 2703.0380361993293, 2677.6219217340554, 2678.0154902951144, 2730.7819082837295, 2675.980785801202, 2769.8035378145146, 2751.2892434738205, 2766.09438145931, 2838.9154928833927, 2846.060846359105, 2756.848066909525, 2682.546426218882, 2775.3927221554013, 2737.9923712545597, 2805.0775929216393, 2799.4732623761683, 2733.8937176473405, 2831.7684297898013, 2685.4213352603974, 2788.835019436861, 2732.175812834023, 2747.4124471873465, 2795.2861736509262, 2762.6546285933678, 2802.362123603685, 2773.0122277888067, 2786.386136822623, 2719.684115864333, 2696.0847392216106, 2729.5592388262608, 2749.530823626408, 2816.6736458246664, 2804.7249710771875, 2906.6257654562905, 2849.055185139825, 2792.8976333702376, 2830.1563676677547, 2862.338290963305, 2752.3066554958996, 2740.868848732521, 2835.277596142402, 2798.7544825399314, 2851.664996594126, 2876.4141655376375, 2769.61919969799, 2793.185397097183, 2854.2205718089917, 2805.046206408135, 2856.312368522653, 2771.9291952890603, 2783.2311234464614, 2829.6024385859, 2722.508448202175, 2786.129787489262, 2754.204349514493, 2758.812791317388, 2734.931164664823, 2790.801199668432, 2825.8404851348037, 2785.41183130212, 2766.1152077464644, 2820.4563379891065, 2794.0924159205956, 2832.681822424446, 2778.744410842281, 2748.4515817251945, 2846.286648490259, 2762.645957826851, 2739.2733696640116, 2887.434781031315, 2838.746726928752, 2857.8076543181005, 2829.741328850746, 2731.8624506729475, 2809.194877964095, 2835.639991983081, 2774.2901464782617, 2812.890136567504, 2782.458862314639, 2735.865533018469, 2815.348098915428, 2805.9535033565508, 2890.763881300486, 2848.1588593900888, 2833.473149169562, 2787.2073612284817, 2795.9358155848436, 2941.596198936999, 2833.9879722064406, 2796.524481611566, 2784.61578363909, 2767.3057501974986, 2814.1997308881514, 2819.9634090838476, 2822.0487443886145, 2846.601672510604, 2847.951669243141, 2773.4273529098605, 2791.740390265389, 2862.543566549477, 2796.9048118651504, 2821.260893453963, 2841.843039744259, 2848.5834357883973, 2798.8378551984883, 2897.1628474778718, 2777.0892532593875, 2747.3025608182193, 2884.5188328905856, 2805.6558784292642, 2873.661394292537, 2907.3751075268256, 2893.325621942176, 2747.0951412659056, 2896.585398318787, 2813.064564004543, 2844.616281909878, 2863.9378095266284, 2856.5618401472434, 2872.102570091827, 2940.6296522612597, 2785.87743103792, 2890.401273625524, 2806.926249812004, 2800.5105874257015, 2934.333051948449, 2861.8742741290494, 2786.2141788882514, 2807.6331076032425, 2877.682585162351, 2907.9847977626364, 2928.9358913186143, 2935.2800486093333, 2792.3580558379067, 2857.7672668995774, 2844.386286244289, 2918.7957425843156, 2863.844621429787, 2858.7225352463993, 2867.658150496989, 2812.5856823295476, 2752.0894152911765, 2799.511261192273, 2832.7366098620546, 2876.003235962489, 2833.2982203386555, 2864.5488689222852, 2829.771138514546, 2840.602454991052, 2866.285854070573, 2882.464991637063, 2938.8973156252555, 2898.082892254109, 2697.687819643127, 2899.413144777709, 2874.372861938143, 2847.475277422531, 2900.2066438553893, 2825.795888362836, 2839.4757763190655, 2826.958567352137, 2876.644334335877, 2818.6940827226636, 2912.043561819364, 2818.903337286878, 2960.633181371939, 2771.5312069174724, 2832.1639711050643, 2896.1997997205617, 2841.616172046931, 2965.283974920245, 2835.2115810197247, 2839.6763936115753, 2905.8665082890457, 2816.410968088986, 2821.6035482367897, 2840.024586564322, 2852.1119461610997, 2958.3247730333446, 2904.3779799890126, 2910.740656152374, 2904.1957497786334, 2868.9946188248155, 2958.624178704684, 2896.141774405019, 2875.93737045916, 2916.411133918859, 2736.410530263688, 2833.689915605084, 2823.6615401054737, 2844.3455128566343, 2896.0081045476854, 2898.4796963828626, 2872.743197070322, 2888.9444025657967, 2874.813433677915, 2855.044082230879, 2902.575874793305, 2956.667727405208, 2946.343838304082, 2819.605031631867, 2926.1013529820198, 2804.995174389852, 2939.3965595095196, 2869.770046678058, 2920.6278785991376, 2891.905681590925, 2900.453764117853, 2908.1465660190843 ], "return_max": [ 0.0, 29.61667983733193, 325.16739609511126, 383.1952563769315, 499.98422470443677, 566.6348869111318, 584.9214954071329, 602.3549526318462, 624.0996927250314, 858.9507202542263, 872.1472938786548, 1081.9082444791497, 1397.084524963528, 1712.0352174334312, 1988.0500408013247, 1910.1081743160419, 2021.6225426171072, 2314.470976561796, 2459.1160171341944, 2726.3421746857966, 2891.352170480607, 2902.8111748563465, 3061.8027469621197, 3194.69418858782, 3308.406563582713, 3409.7619781024005, 3483.6454587531034, 3563.9330998087707, 3741.406152372898, 4062.5835317646806, 3864.5041858437908, 4017.6704016842295, 4206.814785666509, 4048.7505715552024, 4352.4780073498005, 4515.277356051185, 4427.2112731037705, 4537.174271484228, 4701.2111212003865, 4619.65043822589, 4695.071727488901, 4826.623072840699, 4758.091246942079, 5085.730725000796, 4924.884358674533, 5358.465144607668, 5130.279721277835, 5308.793603397446, 5344.8427791741215, 5125.299397316814, 5370.8627466966245, 5289.930622966019, 5363.496769292218, 5367.616240940361, 5560.875911331738, 5657.161297507166, 5642.266221309594, 5699.778765797297, 5735.78176376967, 5894.5672219306125, 5898.363991007227, 5733.416327756158, 5793.326150284309, 5854.230012404905, 5757.958873115003, 5839.460988107725, 5954.868378208546, 5945.626372485526, 5888.307347097278, 5875.95303563168, 5861.443770120917, 6060.366503140146, 5851.8806549275105, 5952.7977244721715, 6201.482285764034, 6124.70017266945, 6126.468307093328, 6164.067541110549, 6095.050346110107, 6098.3720951837295, 6164.214749851715, 5912.259953045705, 6413.631290149638, 6468.313907334788, 6263.532736156869, 6400.141528989974, 6206.197657323041, 6222.696338657885, 6435.121556284561, 6457.04806755901, 6451.521409669051, 6384.736308010499, 6362.781584980023, 6545.264681390262, 6424.347764911066, 6427.706991662115, 6398.450138846503, 6401.14013062605, 6580.031709886951, 6582.896262001483, 6738.546894443019, 6667.428787549858, 6638.39442148762, 6746.273904357991, 6488.506416116861, 6588.011847230429, 6672.034156670891, 6702.522482123706, 6815.767413920008, 6770.913466436907, 6661.313335142882, 6828.1732193348, 6741.185017087388, 6623.567266359498, 6745.597721843443, 6761.832829862699, 6637.319821072283, 6881.735570118594, 6847.413817483926, 6972.799140005446, 6855.017962941849, 6730.603643463108, 6775.407869672987, 6545.0875853153575, 6804.396208297005, 6971.292067535803, 6846.092955828661, 6820.279136704533, 6854.591351765083, 6845.17293520185, 6800.3686016344745, 6741.289726083658, 6859.377289987166, 6754.0412947553095, 6770.402900961892, 6872.755833578269, 6769.463058761616, 6923.7542106094725, 6978.915814978762, 6893.5330188798, 6942.246672101402, 6959.566030351246, 6891.627187978749, 7008.595729002319, 7200.041391573421, 6981.162470450013, 6863.0518041132855, 6820.71021394038, 7098.970207172763, 6980.816442053685, 7139.894854978897, 6931.531611495573, 7237.791444370425, 7134.3807913454675, 6887.796194971828, 7002.198222734607, 6848.14810980376, 7073.384581829825, 6890.378729749984, 7154.461715927615, 6970.4212416177925, 7069.346382732154, 7137.679205919698, 7059.611189952282, 7138.583377698495, 6880.60484885764, 7219.113533693868, 7254.185903588058, 7329.122865143116, 7172.078399907388, 7179.9230912412495, 7293.345760433052, 7182.308676708813, 7093.510640496968, 6992.565925842356, 7116.539158478868, 7153.557386383761, 7114.542114202787, 7177.669479438504, 7314.265952099681, 7102.077938084751, 7116.426223134569, 7191.272531286411, 7157.9991198157895, 6931.779925593101, 6932.956371130462, 7270.065485434507, 7213.915280026789, 7333.443372491987, 7249.823759429826, 7389.402432708663, 7250.529748150387, 7171.391800436779, 7309.248195058932, 7337.97917807647, 7331.260886405009, 7105.244370547611, 7056.392187380408, 7041.93327144308, 7270.7449538149185, 7341.435784775938, 7014.951888524865, 7001.38027369621, 7287.921436531064, 7064.306860526711, 7343.35358456899, 7353.944442134241, 7155.35603162925, 7170.10461487497, 6887.691294857567, 7157.76120863582, 7226.257222939392, 7117.3123918267265, 7307.8976301364855, 7130.200454688265, 7167.479028228288, 7251.85794891758, 7295.528164329009, 7222.1925032445615, 7252.730693722204, 7229.125746926382, 7158.407628330874, 7242.574740860266, 7291.296848914973, 7171.481497160037, 6927.805798994278, 7212.944763174812, 7340.002635931771, 7070.266806643889, 7455.895690857822, 7367.026751825952, 7381.72433600452, 7390.593748397708, 7009.938691664454, 7403.66608493813, 7402.329267424341, 7298.154939060245, 7346.058301150806, 7162.192556177075, 7468.647190487402, 7294.060854212045, 7372.8220938157265, 7307.66133506389, 7311.831484914201, 7415.164870720324, 7323.365620232209, 7254.318140241316, 7233.676938525161, 7435.1813731529855, 7365.877847178452, 7500.490726736145, 7419.203210436002, 7530.162942478113, 7276.337138620541, 7530.184339573424, 7346.655778995124, 7385.094609145364, 7437.42956592163, 7253.554871702225, 7294.422627065246, 7119.535217502718, 7366.4472139568725, 7280.138189411187, 7289.26256013544, 7532.0912227874815, 7228.2885172819915, 7321.119684700723, 7323.269186811582, 7426.004801597073, 7362.598604837833, 7333.561365627782, 7261.974557591571, 7335.5396088791995, 7258.4588141845, 7357.401510726004, 7214.417137923394, 7494.175068007642, 7539.755679696616, 7226.1681663962145, 7522.561398486127, 7280.1948443851115, 7417.599400764935, 7462.456922988895, 7234.9369153369025, 7401.6141412765755, 7268.972235861973, 7270.26625584535, 7430.439616899398, 7303.925029719633, 7103.226531164974, 7292.911459682198, 7350.552302139312, 7354.783553888562, 7142.046789119524, 7144.055591873464, 7439.678096446497, 7349.2815530965045, 7456.633205151184, 7570.410545649713, 7569.0758474683025, 7351.089078268342, 7498.025658630519, 6703.594957107808, 7147.942891072378, 7109.396243374996, 7269.531861579301, 7410.498066319084, 7516.768700972221, 7230.518997287963, 7323.318304653863, 7247.415151342387, 7411.611230257962, 7094.6611746385115, 7193.865978426057, 7382.41400559407, 7352.159427541812, 7227.486539985792, 7267.990610956312, 7111.6080918608495, 7385.841132655565, 7369.637090627632, 7379.725035633577, 7220.906365368954, 7249.681609927897, 7328.211054084556, 7277.459545563608, 7376.157590067694, 7418.774499861878, 7494.756959450166, 7432.199675711498, 7265.05840082429, 7333.5400802988515, 7501.25779356707, 7269.058659671984, 7282.287152423206, 7516.144857594687, 7416.969180123788, 7452.911708028245, 7268.059551939787, 7215.20762089289, 7257.556566612235, 7311.965506737308, 7168.195382489208, 7269.706853085852, 7202.467526651341, 7233.892808427487, 7177.185642795077, 7131.987374499873, 7322.346376503389, 7160.957317467188, 7393.5702056610935, 7448.729300961065, 7390.5768448505805, 7443.194388436066, 7238.4732460209325, 7400.7789329161305, 7483.915967713017, 7131.551046128199, 7267.732057717016, 7285.356041532281, 7270.076397547904, 7423.223097142512, 7247.772254547079, 7403.501052270644, 7502.064217810631, 7291.008306346094, 7376.091762523576, 7217.057421682689, 7213.770107443247, 7203.048790358156, 7087.265618886533, 7377.210186586022, 7344.277652106912, 7325.068829132463, 7309.510304917777, 7266.446523788622, 7179.976087128043, 7149.847298576817, 7210.598139890009, 7318.3387189707155, 7284.091947562846, 7120.337928132696, 7326.296552359445, 7075.976901749524, 7402.184750679233, 7243.829720783706, 7279.575411907163, 7258.189064763566, 7518.203467002586, 7237.952256648245, 7399.695032952384, 7099.826668268028, 7278.0197617127305, 7293.371746606462, 7155.0180125933875, 7140.305630617853, 7017.322809657469, 7424.541667017978, 7107.683860204968, 7208.29755334804, 7114.614927332801, 7210.078511478572, 7102.6732858222085, 7424.073267244293, 7402.882596838341, 7028.26244448849, 7320.621676418762, 7420.28760417783, 7234.9296074639, 7397.167374097639, 7247.79774850632, 7330.012094291109, 7160.402807257173, 7270.852437599166, 7374.138884974329, 7366.787101109627, 7469.453424125203, 7555.685739738562, 7560.091405802592, 7374.766191412042, 7151.72929873831, 7250.9592291534445, 7331.866335049863, 7186.823627754872, 7268.864003209515, 7270.073345254621, 7180.53548187972, 7329.111151530369, 7323.227193633946, 6954.046769256247, 7375.973928071471, 7061.500756332844, 7137.9107812324655, 6795.650357719078, 7263.79283884435, 7240.842915940559, 7260.637484862706, 7247.613382963346, 7389.441039799638, 7365.716539647076, 7226.686309397986, 7157.245613031477, 7296.54278100698, 7229.828339434558, 7359.529051273112, 7228.777527189149, 7087.947713651247, 7161.2997772525305, 7106.791311490431, 7176.874785649005, 7092.017708814668, 7132.41640280386, 7206.603821380772, 7267.865348006653, 7180.973285014151, 7200.080913034151, 7216.755084435534, 7242.643321557749, 7170.252564841561, 7213.397488276039, 7224.656409233296, 7102.471150325507, 7141.441181146651, 7226.800116459524, 7028.823389194822, 7314.211628307634, 7084.861083958743, 6929.617138469754, 7113.410685652475, 7265.236321447543, 7288.761953834333, 7165.281572928468, 7249.606635046356, 7230.943212623712, 7233.147518748427, 7215.090450781399, 7240.014005532725, 7222.170636773506, 7377.121407401943, 7036.673577524399, 7184.2957106766535, 7160.571521387446, 7089.127942908135, 7207.4663076837605, 7232.904731214181, 7313.243978955068, 7289.929298636595, 7307.1992964704, 7123.516211928056, 6886.194590019926, 7032.557145275457, 7116.501428434808, 7274.344946761688, 7088.565648116615, 6958.40978854655, 7013.91527465796, 7030.4024113297555, 7318.300542238265, 7105.481832904925, 7007.628283947332, 7175.7169579524725 ] } }, "Isaac-Repose-Cube-Allegro-Direct-v0": { "FastTD3": { "time": [ 0.0, 487.19444444444446, 974.3888888888889, 1461.5833333333333, 1948.7777777777778, 2435.972222222222, 2923.1666666666665, 3410.3611111111113, 3897.5555555555557, 4384.75, 4871.944444444444, 5359.138888888889, 5846.333333333333, 6333.527777777777, 6820.722222222223, 7307.916666666667, 7795.111111111111, 8282.305555555555, 8769.5, 9256.694444444445, 9743.888888888889, 10231.083333333334, 10718.277777777777, 11205.472222222223, 11692.666666666666, 12179.861111111111, 12667.055555555555, 13154.25, 13641.444444444445, 14128.638888888889, 14615.833333333334, 15103.027777777777, 15590.222222222223, 16077.416666666666, 16564.61111111111, 17051.805555555555, 17539.0 ], "env_step": [ 0, 20480000.0, 40960000.0, 61440000.0, 81920000.0, 102400000.0, 122880000.0, 143360000.0, 163840000.0, 184320000.0, 204800000.0, 225280000.0, 245760000.0, 266240000.0, 286720000.0, 307200000.0, 327680000.0, 348160000.0, 368640000.0, 389120000.0, 409600000.0, 430080000.0, 450560000.0, 471040000.0, 491520000.0, 512000000.0, 532480000.0, 552960000.0, 573440000.0, 593920000.0, 614400000.0, 634880000.0, 655360000.0, 675840000.0, 696320000.0, 716800000.0, 737280000.0 ], "return": [ 0.0, 122.10221354166667, 435.0882975260417, 1421.8710123697917, 2348.2281901041665, 3163.7948404947915, 3698.861083984375, 4025.7603352864585, 4337.995442708333, 4589.761393229167, 4782.493001302083, 4890.22412109375, 5030.248046875, 5155.931477864583, 5282.59326171875, 5324.516927083333, 5457.94775390625, 5515.32958984375, 5564.52880859375, 5673.402506510417, 5739.943033854167, 5763.798502604167, 5847.9658203125, 5859.408203125, 5951.106282552083, 5957.567057291667, 6028.259602864583, 6083.15185546875, 6115.253580729167, 6183.505045572917, 6220.779947916667, 6284.12353515625, 6349.08984375, 6387.808756510417, 6371.83056640625, 6479.999348958333, 6477.792643229167 ], "return_min": [ 0.0, 112.10858028361118, 408.6617848585944, 1304.8137768436577, 2207.112529998192, 3054.42341419639, 3651.647282857428, 3973.3270545893097, 4298.076884293968, 4527.783274664672, 4740.259300399453, 4875.8400900253755, 5014.454607297292, 5066.5720863274855, 5229.377519512968, 5267.243130768987, 5421.939380663636, 5453.997084027583, 5459.271770361508, 5589.086964125713, 5681.091427120193, 5680.027501991008, 5702.091715209337, 5709.277469015035, 5830.487430823852, 5819.911405529101, 5917.187135276544, 5921.141539621999, 5981.998730611572, 5966.944434412144, 6069.596032579598, 6084.438338052321, 6161.312687749657, 6146.352347118702, 6086.806821737842, 6166.503157669386, 6221.325351242079 ], "return_max": [ 0.0, 132.09584679972215, 461.514810193489, 1538.9282478959258, 2489.343850210141, 3273.1662667931932, 3746.074885111322, 4078.1936159836073, 4377.914001122698, 4651.739511793662, 4824.726702204713, 4904.6081521621245, 5046.041486452708, 5245.2908694016805, 5335.809003924532, 5381.790723397679, 5493.956127148864, 5576.662095659917, 5669.785846825992, 5757.718048895121, 5798.794640588141, 5847.569503217326, 5993.839925415663, 6009.538937234965, 6071.725134280314, 6095.222709054233, 6139.332070452622, 6245.162171315501, 6248.508430846762, 6400.06565673369, 6371.963863253736, 6483.808732260179, 6536.866999750343, 6629.265165902132, 6656.854311074658, 6793.49554024728, 6734.259935216255 ] }, "null": { "time": [ 0.0, 33.32, 66.64, 99.96, 133.28, 166.6, 199.92, 233.24, 266.56, 299.88, 333.2, 366.52, 399.84, 433.16, 466.48, 499.8, 533.12, 566.44, 599.76, 633.08, 666.4, 699.72, 733.04, 766.36, 799.68, 833.0, 866.32, 899.64, 932.96, 966.28, 999.6, 1032.92, 1066.24, 1099.56, 1132.88, 1166.2, 1199.52, 1232.84, 1266.16, 1299.48, 1332.8, 1366.12, 1399.44, 1432.76, 1466.08, 1499.4, 1532.72, 1566.04, 1599.36, 1632.68, 1666.0, 1699.32, 1732.64, 1765.96, 1799.28, 1832.6, 1865.92, 1899.24, 1932.56, 1965.88, 1999.2, 2032.52, 2065.84, 2099.16, 2132.48, 2165.8, 2199.12, 2232.44, 2265.76, 2299.08, 2332.4, 2365.72, 2399.04, 2432.36, 2465.68, 2499.0, 2532.32, 2565.64, 2598.96, 2632.28, 2665.6, 2698.92, 2732.24, 2765.56, 2798.88, 2832.2, 2865.52, 2898.84, 2932.16, 2965.48, 2998.8, 3032.12, 3065.44, 3098.76, 3132.08, 3165.4, 3198.72, 3232.04, 3265.36, 3298.68, 3332.0, 3365.32, 3398.64, 3431.96, 3465.28, 3498.6, 3531.92, 3565.24, 3598.56, 3631.88, 3665.2, 3698.52, 3731.84, 3765.16, 3798.48, 3831.8, 3865.12, 3898.44, 3931.76, 3965.08, 3998.4, 4031.72, 4065.04, 4098.36, 4131.68, 4165.0, 4198.32, 4231.64, 4264.96, 4298.28, 4331.6, 4364.92, 4398.24, 4431.56, 4464.88, 4498.2, 4531.52, 4564.84, 4598.16, 4631.48, 4664.8, 4698.12, 4731.44, 4764.76, 4798.08, 4831.4, 4864.72, 4898.04, 4931.36, 4964.68, 4998.0, 5031.32, 5064.64, 5097.96, 5131.28, 5164.6, 5197.92, 5231.24, 5264.56, 5297.88, 5331.2, 5364.52, 5397.84, 5431.16, 5464.48, 5497.8, 5531.12, 5564.44, 5597.76, 5631.08, 5664.4, 5697.72, 5731.04, 5764.36, 5797.68, 5831.0, 5864.32, 5897.64, 5930.96, 5964.28, 5997.6, 6030.92, 6064.24, 6097.56, 6130.88, 6164.2, 6197.52, 6230.84, 6264.16, 6297.48, 6330.8, 6364.12, 6397.44, 6430.76, 6464.08, 6497.4, 6530.72, 6564.04, 6597.36, 6630.68, 6664.0, 6697.32, 6730.64, 6763.96, 6797.28, 6830.6, 6863.92, 6897.24, 6930.56, 6963.88, 6997.2, 7030.52, 7063.84, 7097.16, 7130.48, 7163.8, 7197.12, 7230.44, 7263.76, 7297.08, 7330.4, 7363.72, 7397.04, 7430.36, 7463.68, 7497.0, 7530.32, 7563.64, 7596.96, 7630.28, 7663.6, 7696.92, 7730.24, 7763.56, 7796.88, 7830.2, 7863.52, 7896.84, 7930.16, 7963.48, 7996.8, 8030.12, 8063.44, 8096.76, 8130.08, 8163.4, 8196.72, 8230.04, 8263.36, 8296.68, 8330.0, 8363.32, 8396.64, 8429.96, 8463.28, 8496.6, 8529.92, 8563.24, 8596.56, 8629.88, 8663.2, 8696.52, 8729.84, 8763.16, 8796.48, 8829.8, 8863.12, 8896.44, 8929.76, 8963.08, 8996.4, 9029.72, 9063.04, 9096.36, 9129.68, 9163.0, 9196.32, 9229.64, 9262.96, 9296.28, 9329.6, 9362.92, 9396.24, 9429.56, 9462.88, 9496.2, 9529.52, 9562.84, 9596.16, 9629.48, 9662.8, 9696.12, 9729.44, 9762.76, 9796.08, 9829.4, 9862.72, 9896.04, 9929.36, 9962.68, 9996.0, 10029.32, 10062.64, 10095.96, 10129.28, 10162.6, 10195.92, 10229.24, 10262.56, 10295.88, 10329.2, 10362.52, 10395.84, 10429.16, 10462.48, 10495.8, 10529.12, 10562.44, 10595.76, 10629.08, 10662.4, 10695.72, 10729.04, 10762.36, 10795.68, 10829.0, 10862.32, 10895.64, 10928.96, 10962.28, 10995.6, 11028.92, 11062.24, 11095.56, 11128.88, 11162.2, 11195.52, 11228.84, 11262.16, 11295.48, 11328.8, 11362.12, 11395.44, 11428.76, 11462.08, 11495.4, 11528.72, 11562.04, 11595.36, 11628.68, 11662.0, 11695.32, 11728.64, 11761.96, 11795.28, 11828.6, 11861.92, 11895.24, 11928.56, 11961.88, 11995.2, 12028.52, 12061.84, 12095.16, 12128.48, 12161.8, 12195.12, 12228.44, 12261.76, 12295.08, 12328.4, 12361.72, 12395.04, 12428.36, 12461.68, 12495.0, 12528.32, 12561.64, 12594.96, 12628.28, 12661.6, 12694.92, 12728.24, 12761.56, 12794.88, 12828.2, 12861.52, 12894.84, 12928.16, 12961.48, 12994.8, 13028.12, 13061.44, 13094.76, 13128.08, 13161.4, 13194.72, 13228.04, 13261.36, 13294.68, 13328.0, 13361.32, 13394.64, 13427.96, 13461.28, 13494.6, 13527.92, 13561.24, 13594.56, 13627.88, 13661.2, 13694.52, 13727.84, 13761.16, 13794.48, 13827.8, 13861.12, 13894.44, 13927.76, 13961.08, 13994.4, 14027.72, 14061.04, 14094.36, 14127.68, 14161.0, 14194.32, 14227.64, 14260.96, 14294.28, 14327.6, 14360.92, 14394.24, 14427.56, 14460.88, 14494.2, 14527.52, 14560.84, 14594.16, 14627.48, 14660.8, 14694.12, 14727.44, 14760.76, 14794.08, 14827.4, 14860.72, 14894.04, 14927.36, 14960.68, 14994.0, 15027.32, 15060.64, 15093.96, 15127.28, 15160.6, 15193.92, 15227.24, 15260.56, 15293.88, 15327.2, 15360.52, 15393.84, 15427.16, 15460.48, 15493.8, 15527.12, 15560.44, 15593.76, 15627.08, 15660.4, 15693.72, 15727.04, 15760.36, 15793.68, 15827.0, 15860.32, 15893.64, 15926.96, 15960.28, 15993.6, 16026.92, 16060.24, 16093.56, 16126.88, 16160.2, 16193.52, 16226.84, 16260.16, 16293.48, 16326.8, 16360.12, 16393.44, 16426.76, 16460.08, 16493.4, 16526.72, 16560.04, 16593.36, 16626.68, 16660.0 ], "env_step": [ 0, 1835008, 10878976, 11141120, 11403264, 11534336, 12189696, 12713984, 16252928, 20185088, 23199744, 25952256, 29097984, 31981568, 33554432, 33947648, 36306944, 37355520, 40763392, 45219840, 46923776, 48496640, 54919168, 58458112, 60424192, 62783488, 72351744, 73400320, 76283904, 77594624, 81788928, 82182144, 82575360, 83623936, 87293952, 88866816, 89653248, 90439680, 93454336, 94371840, 97910784, 98435072, 98697216, 100401152, 100663296, 103022592, 105512960, 115867648, 117047296, 123994112, 125173760, 125304832, 130023424, 134086656, 135790592, 136839168, 139460608, 141164544, 141819904, 143523840, 144310272, 146800640, 147062784, 150077440, 152043520, 155058176, 163840000, 165281792, 171573248, 174325760, 174456832, 175505408, 177733632, 178126848, 179961856, 181403648, 181927936, 184549376, 188088320, 193331200, 193855488, 204472320, 205651968, 206176256, 206569472, 210763776, 211156992, 214433792, 214827008, 215351296, 219676672, 221118464, 222429184, 224657408, 224919552, 226492416, 228982784, 234094592, 244187136, 245366784, 251920384, 257687552, 258080768, 260308992, 260964352, 261226496, 263979008, 270139392, 274333696, 284164096, 285868032, 295043072, 297795584, 299761664, 299892736, 303955968, 304218112, 307757056, 311820288, 316145664, 319291392, 322306048, 322699264, 323747840, 326369280, 336592896, 336855040, 339738624, 343146496, 345243648, 346947584, 347602944, 349700096, 353370112, 358744064, 359399424, 365166592, 372899840, 376963072, 378273792, 383254528, 389021696, 390463488, 391905280, 394657792, 395444224, 396361728, 406847488, 407109632, 407896064, 412745728, 417333248, 420216832, 422051840, 422313984, 424148992, 427425792, 428867584, 429654016, 430440448, 437125120, 440270848, 443940864, 448397312, 452329472, 452722688, 455606272, 457572352, 459669504, 464125952, 468320256, 469106688, 471334912, 474087424, 474349568, 475004928, 476315648, 479723520, 480903168, 481689600, 483917824, 484048896, 484573184, 485883904, 486801408, 488898560, 489553920, 492306432, 502530048, 511311872, 512884736, 519307264, 519569408, 525074432, 525205504, 529793024, 529924096, 530186240, 531496960, 533331968, 535035904, 536215552, 539885568, 540016640, 541327360, 548274176, 558628864, 560857088, 565444608, 570294272, 572260352, 573833216, 576847872, 577896448, 589824000, 591921152, 592052224, 592314368, 604241920, 605814784, 607125504, 608174080, 609484800, 610402304, 613285888, 613941248, 615120896, 620625920, 621150208, 625082368, 626786304, 628883456, 637534208, 638320640, 638713856, 639238144, 640679936, 642121728, 643039232, 643563520, 646447104, 648544256, 649592832, 652214272, 654180352, 654704640, 655884288, 656670720, 662175744, 662306816, 663224320, 668598272, 670040064, 670957568, 671875072, 673447936, 674365440, 677773312, 684982272, 686424064, 691273728, 692584448, 692846592, 695468032, 701366272, 702021632, 705036288, 705953792, 706347008, 706740224, 707133440, 714604544, 716177408, 719716352, 720633856, 720896000, 724041728, 724303872, 724697088, 725876736, 728236032, 730988544, 731512832, 734658560, 737411072, 740425728, 740950016, 742522880, 742785024, 743047168, 745013248, 745799680, 746848256, 747110400, 747896832, 752091136, 756940800, 763887616, 765984768, 766377984, 767295488, 767819776, 776077312, 778960896, 781451264, 785907712, 788660224, 792592384, 797573120, 797835264, 798621696, 799670272, 802553856, 803078144, 805044224, 807272448, 809893888, 811728896, 812384256, 814612480, 814874624, 815398912, 821035008, 828243968, 828506112, 832831488, 832962560, 835059712, 835977216, 838205440, 838860800, 842661888, 843448320, 848560128, 850132992, 850657280, 861011968, 861667328, 865730560, 870580224, 872284160, 882114560, 883425280, 883556352, 890372096, 892338176, 896139264, 900726784, 904134656, 907149312, 908984320, 909901824, 912916480, 915013632, 916979712, 918159360, 918814720, 919339008, 920256512, 921960448, 923795456, 926679040, 928514048, 935198720, 936116224, 939261952, 942276608, 942669824, 947519488, 949092352, 949223424, 951844864, 951975936, 956432384, 958398464, 961544192, 962985984, 965476352, 974782464, 974913536, 981598208, 983433216, 984743936, 985661440, 987365376, 990773248, 992608256, 996802560, 998506496, 998768640, 1002831872, 1013579776, 1014104064, 1016987648, 1017118720, 1017643008, 1018298368, 1027735552, 1027866624, 1028784128, 1033764864, 1034682368, 1046478848, 1050279936, 1054343168, 1056178176, 1058930688, 1066139648, 1067843584, 1068498944, 1070989312, 1072693248, 1072824320, 1075838976, 1076494336, 1079246848, 1081212928, 1089208320, 1093795840, 1095892992, 1097465856, 1100611584, 1103364096, 1105723392, 1106378752, 1107296256, 1108869120, 1110704128, 1111621632, 1111752704, 1112408064, 1115553792, 1115947008, 1116602368, 1117126656, 1120272384, 1126563840, 1127219200, 1131413504, 1134952448, 1135214592, 1141768192, 1151598592, 1153302528, 1154613248, 1156317184, 1156972544, 1158021120, 1162084352, 1163264000, 1163788288, 1167589376, 1172307968, 1176764416, 1179123712, 1182007296, 1184628736, 1190133760, 1193541632, 1193803776, 1196294144, 1196949504, 1199439872, 1199702016, 1210712064, 1211105280, 1212022784, 1212284928, 1215168512, 1215430656, 1217134592, 1218969600, 1222639616, 1225785344, 1226178560, 1234305024, 1238499328, 1239547904, 1242169344, 1242300416, 1245315072, 1246887936, 1259470848, 1260257280, 1264713728, 1265500160, 1269039104, 1273626624, 1277820928, 1279787008, 1280180224, 1283457024, 1284767744, 1287520256, 1289224192, 1293418496, 1296171008, 1305214976, 1306918912, 1307049984, 1309409280 ], "return": [ 0.0, -15.208062119086582, 27.985837462544442, 33.065118919610974, 63.739390674432116, 73.38403714617094, 81.76407511790593, 83.41257841587067, 94.50324208100636, 87.02840101520219, 96.66484376668932, 105.62438798487187, 103.97968959172567, 123.7255840688944, 130.92081548810003, 138.31118696252506, 143.94612252871195, 139.69248750011124, 161.23692795117697, 175.8585472281774, 158.11115952809652, 178.34551179766655, 195.95468923449516, 213.11414700110754, 189.21305813153586, 210.28728338837627, 275.5193917051951, 290.2456347779433, 328.01072924931844, 333.3376958012581, 364.03000196417173, 401.8089190347989, 394.4687174904347, 417.68140043894454, 426.2444740418593, 444.2910240530967, 434.3765281144779, 448.3476926716169, 524.3667556663355, 501.29599927862483, 509.64772761027024, 526.7205876469612, 538.6500786121686, 625.3346225706736, 607.9900910063585, 599.0540671960513, 685.6463563728333, 641.4686959608397, 668.2085113616785, 683.5394006729126, 660.5089496715864, 724.7123481567701, 721.6675941574573, 724.0352485704421, 779.669870207707, 760.2905054322878, 845.789764096737, 833.1766210738818, 924.196205678781, 830.4869041840235, 895.8986332496006, 844.670979039669, 856.6293645552795, 918.985225797097, 891.081965850989, 856.4227460622787, 907.1667261115709, 917.6800900681814, 966.6861300317446, 1008.8145752668382, 955.3873797392845, 959.3507392517726, 957.2452503728867, 1013.6096401135125, 1013.7457611942291, 1038.3559891557695, 1040.9363767464956, 1014.6830427742003, 1033.165902372996, 1059.8773714057604, 1054.6472192327183, 1128.236418112119, 1109.1773519515991, 1101.6388582499821, 1083.090611615181, 1195.6467939186095, 1115.4643084677061, 1157.6829477691651, 1120.246314201355, 1091.018876689275, 1148.0232208383084, 1154.231260531346, 1117.6589318927129, 1163.7312007880212, 1187.4146699810028, 1178.052467587789, 1234.0955010000864, 1209.0325522549947, 1230.0507287740709, 1282.1594580292701, 1234.0622125005723, 1271.1317810726166, 1225.4457596842446, 1270.9370497743287, 1265.8080000972748, 1285.0425640010833, 1273.9300214743614, 1239.6171310869852, 1258.1802626220385, 1325.5853050804137, 1297.6688449207943, 1369.975601091385, 1344.9264648564656, 1358.1098932425182, 1362.4708606147767, 1349.7314314222338, 1346.8320728611945, 1401.8542354472477, 1355.3846225547788, 1387.9516682084402, 1368.1288306109109, 1388.3219243033727, 1406.3560511612893, 1421.1573902893067, 1403.0136197344461, 1477.9739192326863, 1441.3298983828226, 1404.8318265914916, 1472.0646509361268, 1450.5080246194204, 1433.188861509959, 1486.0691950257617, 1499.02348812898, 1496.1851874955494, 1478.6511289024354, 1513.782597764333, 1502.1337704817454, 1531.7802909342447, 1463.9768899472554, 1446.483530351321, 1510.6605901590983, 1529.3808396275838, 1572.5691393343607, 1531.6558720620476, 1523.744317480723, 1494.7289040152234, 1560.7028294316926, 1585.1348492209115, 1537.608029677073, 1577.6323381169634, 1532.9813650194803, 1546.102662928899, 1572.8515084838866, 1600.6941162077585, 1576.739197918574, 1615.7113013267517, 1603.171879494985, 1576.099616212845, 1523.232181816101, 1565.5702744865418, 1609.7783724792798, 1703.4605189323427, 1615.7065868759157, 1594.436960922877, 1675.8799238904319, 1656.2481291834513, 1627.7645898501078, 1656.3225352541606, 1694.707741953532, 1655.415842622121, 1658.082688509623, 1713.2894029490153, 1588.20122118632, 1667.4801043764749, 1676.096830698649, 1687.85254655838, 1708.2096050071716, 1754.2827956136068, 1712.5367916679381, 1640.9438634514809, 1689.144550711314, 1664.0739923032124, 1683.9244037755332, 1696.7922770563762, 1768.1795618375143, 1727.5779999160768, 1736.8503835614522, 1719.941689491272, 1686.0338038508096, 1725.988723818461, 1738.6104576237994, 1740.7765754508973, 1705.1388481903075, 1670.4937730471295, 1658.214816932678, 1695.121711362203, 1792.4769620259603, 1745.4210001754761, 1760.5531140454611, 1781.594201936722, 1723.959014670054, 1671.6467679278055, 1757.5837479400634, 1741.0655566724142, 1774.9067363675433, 1751.9776115671793, 1750.5093571726482, 1795.1821841748554, 1745.589820652008, 1840.589610424042, 1717.551601155599, 1794.2092125193278, 1743.430915406545, 1779.547692756653, 1831.2570528539025, 1783.0511383438109, 1758.9477525520324, 1770.414341278076, 1782.2672223154705, 1818.8416625086466, 1744.8974377123516, 1782.1652972284953, 1861.500615870158, 1812.3222024790448, 1802.887804667155, 1756.6487368011476, 1830.3497687530519, 1783.0421785227456, 1770.1143108113608, 1762.8640670140585, 1797.7952985254924, 1739.315527311961, 1925.4324143981933, 1844.565812117259, 1859.4592933972679, 1782.93512550354, 1849.921053543091, 1896.0195062383016, 1852.6285049819944, 1870.3044067255653, 1812.883816680908, 1843.5550619125363, 1834.9702418899535, 1875.8230379358927, 1864.2732896296184, 1803.783723742167, 1781.7357353655498, 1851.7352653630576, 1828.711000213623, 1857.986676165263, 1830.465396830241, 1835.2933151626587, 1848.049705619812, 1831.3830486679078, 1903.9784999211631, 1841.445189297994, 1774.7733537991842, 1857.219704844157, 1853.6428640619913, 1827.793980585734, 1823.921475461324, 1918.2105669784548, 1919.4746598943075, 1856.3971628570555, 1857.3752022743222, 1857.5700806744892, 1898.651220486959, 1823.3593325424197, 1915.0531467819212, 1811.474196472168, 1920.302176755269, 1835.0142211023967, 1995.462091623942, 1945.341724770864, 1967.6135405858356, 1895.7059489440917, 1927.3008184973398, 1878.2231207148234, 1885.3370151138306, 1885.9712224833172, 1913.0469807688396, 1917.5231856409707, 1890.7723024622599, 1894.288302523295, 1817.398354326884, 1903.3093048095704, 1896.6997822443645, 1938.755055440267, 1874.6556552886962, 1941.1620425669353, 1896.1112930552165, 1945.2559768676758, 1922.5934006500245, 1906.1691541703542, 1938.1057643127442, 1862.2512243143717, 1945.4173630015055, 1937.4000315475462, 1943.181752243042, 1997.07544312795, 1915.3227619679772, 2009.8460191726688, 1928.3586402511598, 1915.2570816675823, 1958.9155800882975, 1925.594481455485, 1903.2088528315228, 1948.8292271677656, 1903.9699386978148, 1866.7248757425943, 1931.2256560770668, 1949.2665031051638, 1971.5855646133423, 2036.839611829122, 1957.3344420369467, 1971.0834051767986, 1930.8539343388875, 1910.7293909327188, 1963.4615940475462, 1936.6964011383054, 1920.6460943349202, 1964.1139424006142, 1939.6381277974444, 1983.1162591298423, 1969.4077895991006, 1916.6050984573365, 1989.2113303375245, 1848.2459172439576, 1952.5798476282755, 1950.3160333887736, 2034.1265435791017, 1957.3808938471477, 1975.661287091573, 1952.5201565678915, 1955.1099046198526, 1849.4640664164226, 2036.2135239664713, 1902.6708364105223, 1976.565142110189, 2000.1304290771484, 1970.1112725194296, 1999.9440422439575, 1968.6209470717113, 1991.952435569763, 1978.2394967905682, 2020.874125760396, 2021.6539390945434, 1970.0303056081136, 1953.0815971247357, 1943.7658596038818, 2006.5767294057212, 2002.7899671554562, 1949.61315633138, 1911.2338114929198, 1962.1358880996704, 2024.1895262781782, 2029.7549323145547, 1971.429596494039, 2015.836051381429, 1989.5375478617352, 1955.2756966527304, 2022.758341852824, 2020.1011663182574, 1985.6239275360106, 1995.0723993174233, 1984.3480288060507, 1990.6294145965576, 1941.2070553334554, 2012.1252802149454, 1954.0151485188799, 1966.6849535624187, 1956.8263164011635, 2009.279321454366, 1971.528699051539, 1971.0075210444131, 1974.9195355733236, 2014.5018423970541, 1963.5272241973878, 2062.865896174113, 2064.069284261068, 2032.2438666407268, 2045.7047826004027, 1994.6503591664634, 1944.566607913971, 2000.897696100871, 2038.6879258982342, 1946.1669205729165, 1967.0156407292682, 1937.578423576355, 2037.6293779500327, 1968.8212608210245, 2010.168547808329, 2016.3574635696411, 2048.5558566919963, 1975.3541211573283, 2061.4655352020263, 1937.0920262018838, 2034.1998028818764, 2086.1933650080364, 2010.8763928985597, 2010.5087301635742, 2027.8863892364504, 1981.2459899648031, 2012.5796449931465, 2036.9558761215212, 1977.7446804300944, 2063.926620445252, 2099.370090967814, 2031.5188530985515, 2079.7016563924153, 2036.4256989796959, 1980.0338309478757, 1954.092360877991, 2026.7211258061727, 2042.9697960281374, 2037.1259219614665, 2016.3184883499146, 2117.8120729827883, 1992.8279691696168, 1965.9154495493574, 1946.7313609409332, 2057.197262687683, 2033.9488484064739, 2022.2097825749715, 2064.514777202606, 2023.224939028422, 2095.5899105580647, 2000.7877479807537, 1995.3137570063275, 2066.0285115305583, 2057.512433611552, 2072.2311502329508, 2018.1963609441125, 2038.0414914067585, 2055.4949537150064, 2100.3472706476846, 2047.5307428741453, 2046.3145210266111, 2070.8851332473755, 2031.5462546221415, 2070.7231659825643, 2043.2447252909342, 2080.842988020579, 2043.9892267862954, 2013.1029631296794, 2077.845181528727, 1999.9314056777955, 2024.9522764460246, 2041.5528373972575, 2076.5959600957235, 2093.144813868205, 1954.2837760353088, 2075.3821962610878, 2055.6548400624592, 2082.33140557607, 2025.3550900268554, 2046.7053001149495, 2069.434745763143, 2030.7184680811563, 2129.1889733250937, 2045.801846707662, 2053.877966028849, 2099.7886859385176, 2071.1038087495167, 2113.2374559020996, 2142.6716127904256, 2041.0685925801592, 2022.7078050359094, 2069.1886087226867, 2069.757692337036, 2128.6494229634604, 2141.4252186838785, 2052.935356267293, 2093.082716585795, 2135.0062626902263, 2099.3032523727416, 2047.962077682813, 2055.616015446981, 2096.431504268646, 2165.0426854324337, 2071.234339103699, 2026.6400920359295, 2068.65818195343, 2060.2222735087075, 2063.7241380437213, 2022.5148383331298, 2068.477814381917, 2088.7389111423495, 2072.564161783854, 2097.643168818156, 2138.8968943659465, 2132.4746655718486, 2145.5754326375322, 2101.0678678512572, 2086.687912394206, 2057.179594141642, 2086.038907839457, 2093.507007738749, 2101.1903778711953, 2098.7112515767417, 2140.860017267863 ], "return_min": [ 0.0, -27.161190098873938, -22.596889943332886, -18.04884047175878, 34.96802557070835, 42.14144832782458, 42.18764349919466, 55.902530695070986, 77.36366604239011, 71.03807036708037, 81.47330597845952, 97.73028509544966, 97.97264168524211, 105.96846207393762, 110.22956162854999, 128.14846576261468, 108.559148752248, 123.92340874979348, 132.78944635044505, 159.42855628495957, 144.46347678148942, 157.8610229525654, 158.35143035502105, 197.7479254473252, 160.54953800824737, 188.7994952359248, 217.39242615649243, 244.56688681087465, 301.11488470145423, 256.17273889754574, 292.8235861221529, 314.92021021487477, 306.4235374628608, 325.3520406500228, 344.34044697638694, 368.11517927384756, 336.8252980067012, 333.9395555124325, 427.2549358603718, 371.37608381527093, 420.9938648544428, 389.18023388585493, 388.7416321479917, 472.426062403483, 420.6928762920395, 424.0957463022822, 507.2579861115297, 482.9738704260956, 514.3924576752916, 515.5455037645223, 505.47987268452863, 549.1951941961452, 576.1318275055664, 554.7743324166245, 578.684664797232, 592.2225573434476, 624.6161211833768, 633.7916668340124, 680.684790989311, 628.6966472885151, 661.4334250216607, 693.722886939662, 632.8989605489518, 736.02060626199, 709.4836056376417, 656.7443508943927, 726.4691582918115, 733.7766667428998, 772.0049538980603, 771.1210966295357, 687.8146653746687, 772.2877183570141, 757.2506362258152, 831.5431251001323, 764.1121868299346, 774.9296975673354, 779.6430454129754, 760.2656831857466, 837.0739666126292, 834.701324203895, 820.945902380164, 918.2923429426099, 912.5397683243333, 865.2231092682929, 851.5707559534205, 961.1212342855194, 841.2827250647913, 908.2660836637016, 853.0973778229541, 842.277914583651, 895.7067200072679, 903.8746022376938, 909.0731031912112, 899.3540128471468, 892.1310636941934, 914.4578585600933, 1000.6894379726076, 930.2474086813293, 928.192620748513, 982.6184080156437, 985.0062257377579, 972.0931290747644, 1020.8499845695702, 1043.6865484733269, 962.5154794990744, 995.2043902628145, 1016.0871129267222, 999.3151881903785, 988.0216683911464, 1076.0456266138542, 1038.413815810871, 1134.5676517335849, 1127.0582451565451, 1029.4210202300158, 1008.7807577978716, 1131.6949019203973, 1084.653325585139, 1178.7731229642109, 1113.7457074972601, 1113.0967302489153, 1074.7978853578866, 1061.731097782584, 1118.770170470866, 1100.9872796639804, 1153.3560738865303, 1179.5669987881379, 1131.3296485345286, 1115.279419330474, 1150.983972038616, 1150.3776378371845, 1091.8778342513265, 1174.5594091944524, 1144.1578655765995, 1226.8169439788194, 1206.4039871117989, 1190.3003022235698, 1144.5405407138749, 1232.5913778660001, 1154.2737017656432, 1170.5795243711475, 1220.6887059803785, 1205.7898655331019, 1351.4844213259557, 1217.0863485837851, 1227.702084485864, 1179.059482340776, 1271.70303368492, 1248.3406054138006, 1248.1741466642331, 1277.1677626249261, 1214.558260797336, 1197.5053085935224, 1285.9107737681893, 1303.3717898497944, 1226.9067480283068, 1281.0673349128906, 1270.0320762435354, 1194.3613478912507, 1213.3823682848417, 1185.1487094475583, 1310.7679868182706, 1370.9720882821025, 1268.202556716447, 1242.343764031949, 1346.877691688978, 1317.495343061811, 1314.568714119066, 1361.3424888032723, 1339.0235495620177, 1267.3943711979912, 1301.7229607501765, 1368.062464069575, 1236.568057266568, 1365.4843721048537, 1267.4246117227956, 1380.7292223209784, 1404.7881334910667, 1420.0921517039255, 1361.3780125494404, 1332.0237658028361, 1376.9324775195712, 1341.9790275047662, 1318.1740906143439, 1328.6053661732908, 1424.480831565852, 1373.1092791573649, 1374.772976307375, 1324.893876172093, 1286.3563379195516, 1346.6836717547092, 1369.9218004416691, 1330.8701694224194, 1280.8059137701707, 1254.5575411723346, 1335.9852094244845, 1364.3224842003256, 1385.5057623830635, 1380.023729056787, 1432.4021606571796, 1358.2536781852564, 1359.5747690286767, 1284.2510806986868, 1455.7353569125598, 1388.2634609261072, 1381.3655357175812, 1374.0751343798888, 1411.685858399476, 1387.5794869635397, 1375.6519713514167, 1458.1457389245174, 1345.079579841159, 1388.019191836721, 1386.2673140956695, 1373.1435768897766, 1483.1654724490288, 1419.1127589141586, 1422.6441999617675, 1394.316212442754, 1435.0119399069597, 1486.8789219870775, 1388.1124059768877, 1408.8235776500576, 1388.8400768587592, 1381.4120399968901, 1465.4034592263815, 1400.4350416301975, 1504.3549284270698, 1365.002290992044, 1419.4738236188787, 1385.7303798776138, 1380.7123758037292, 1374.325530176849, 1546.4586301789147, 1416.9020671146745, 1451.6460060492382, 1395.8395607612806, 1516.8886349128268, 1566.2220417641367, 1443.3438756018134, 1497.0517409211652, 1439.7744408874278, 1461.1655404900193, 1431.6120416100669, 1482.632285336264, 1445.61722471427, 1373.9022471610704, 1383.4778009699876, 1514.5874970761788, 1449.3897922200754, 1459.2718640980195, 1445.673988689869, 1481.867102547193, 1416.5571935452258, 1428.6376018618348, 1471.5909162454504, 1437.9651272477709, 1376.6714314274057, 1435.2779243640286, 1460.0146592348506, 1444.573448109354, 1425.1645320221062, 1589.7575798981375, 1537.6307690887043, 1481.7225802245484, 1456.5958771890605, 1430.5105954220862, 1460.8339212615228, 1442.4455362521799, 1487.3489052326277, 1450.4672045571854, 1477.3002344345819, 1395.6450137317802, 1568.0512429525927, 1491.1838236247427, 1565.7123137775534, 1484.6256077306602, 1515.8810622500755, 1472.0829419518004, 1472.7142075134593, 1505.0820518975652, 1497.5566648524955, 1498.4511328410176, 1466.658886426278, 1543.8937521195107, 1450.2672900390025, 1430.1587646486926, 1498.9568595144442, 1447.542614495117, 1447.4056109371613, 1507.8509163027904, 1455.4372825667892, 1452.9341609485696, 1498.1143864181686, 1515.1634447227061, 1468.4866831933964, 1442.4412955931375, 1527.5556465426737, 1536.0565024663938, 1514.247800150507, 1583.536500617152, 1474.5626088776535, 1509.91882996999, 1528.1334189297806, 1470.9582023660391, 1578.043905306006, 1520.5414630516445, 1441.6468159267697, 1470.1766185048043, 1482.3683791048722, 1467.4125032446327, 1524.607720528038, 1503.150773887472, 1486.3705340005833, 1559.1060328345488, 1569.8997699314025, 1520.4322467032387, 1396.8354230278119, 1453.6314931495442, 1517.509476294477, 1450.7759682630508, 1474.1624315114384, 1480.0016272160174, 1532.2355047038743, 1520.3996238733514, 1519.2821826630918, 1528.9508599691799, 1543.4562284675474, 1432.0963105506623, 1442.7383662102407, 1495.1000309817264, 1596.8286355215296, 1548.0623781368479, 1513.473194333779, 1445.623128198912, 1558.6842115517425, 1470.387713310401, 1577.066097963253, 1454.9469317209691, 1526.2121612927629, 1574.327076638486, 1594.8228845777896, 1576.1779926049167, 1483.476359836326, 1506.2643467472722, 1488.250470296554, 1560.3696060920358, 1631.5524746290182, 1494.4993851391039, 1565.1822489015794, 1449.5369646032502, 1569.2371285379234, 1563.1984140915442, 1521.9330510275392, 1544.174194066606, 1558.2547751910918, 1561.1145405838733, 1552.731538143909, 1578.208912640424, 1516.625577093806, 1628.3454945911135, 1479.4409458937757, 1580.0587191038408, 1563.635343391275, 1519.502138217722, 1513.6321589265149, 1498.35854524867, 1501.401110032914, 1544.7897019286283, 1540.6201225300201, 1497.3815390956493, 1569.4663899211246, 1512.6643267583618, 1535.621112398052, 1495.5567046492806, 1429.5073148231895, 1557.4443777879355, 1567.0962056198289, 1530.7736527134039, 1584.7302307758193, 1560.4982829956334, 1596.559082695724, 1641.3211833407408, 1517.8796661373995, 1450.5012692957116, 1517.66185857882, 1623.861862632764, 1524.60548796018, 1501.9337103757932, 1446.0298615019626, 1606.0463985254053, 1583.0037423751878, 1606.8359237011962, 1579.0443105882025, 1666.6902098590833, 1476.5094490138526, 1579.4924227062668, 1502.4046419676313, 1581.0602606282112, 1643.893308246401, 1534.4804488328232, 1537.0412616431054, 1580.3659515002705, 1496.6012652120808, 1659.6623235855254, 1633.5241005729476, 1509.08508107819, 1611.0853749851426, 1629.3447044536451, 1577.0238591286475, 1741.3789317090202, 1604.8568264986948, 1464.4603215632667, 1528.850724150617, 1601.3909147640104, 1563.432721183317, 1563.117337356342, 1534.0604064009335, 1682.1950946507377, 1589.7012207264347, 1529.3202351849368, 1562.783527089386, 1596.4480706175289, 1612.1582084256165, 1556.2473492890576, 1572.5127956006552, 1579.9003587354428, 1603.3815175879731, 1496.2912121768586, 1547.1989400956572, 1529.7008966898647, 1591.9319267874932, 1620.66586739163, 1524.8975652680986, 1596.7308901438619, 1578.509760776493, 1644.9447567025031, 1608.9701122356018, 1584.3660853819813, 1583.958221531358, 1547.7429354654662, 1588.0843985723182, 1620.043228572627, 1624.8077049713972, 1535.407323620321, 1615.0586351498025, 1677.2807431240822, 1575.592527627532, 1644.6257283034101, 1612.8801666411941, 1630.8985728703, 1615.2113297627213, 1546.6288675826943, 1664.8153291931917, 1553.9245644556117, 1572.1012552448246, 1562.6978368470532, 1604.6422087335768, 1613.4301426983723, 1637.4686336857023, 1654.803314447553, 1572.5107533638277, 1648.6667519103562, 1605.3503187941099, 1585.1654769940812, 1650.9726708678616, 1699.881191757293, 1544.4039259615927, 1650.2981905111487, 1643.155559833605, 1621.1338818819904, 1745.929102319576, 1666.0085948359579, 1633.82751805364, 1645.624210375258, 1662.397404348184, 1662.432444994714, 1590.101139389238, 1675.457243838382, 1622.1487061811008, 1713.4838525418695, 1662.250600129536, 1619.9537068312998, 1580.1483505469207, 1660.4698799430842, 1613.5100304468401, 1530.5004914938645, 1684.3522602101496, 1580.6234154752178, 1613.4683769892192, 1589.010467444366, 1694.1019780951112, 1667.2330748321283, 1708.554258745362, 1575.2434393823205, 1606.5804405155193, 1654.5279706504311, 1603.8763502603792, 1600.9958365716125, 1654.6551724197448, 1690.9247311841996, 1644.1095798874194 ], "return_max": [ 0.0, -3.254934139299225, 78.56856486842177, 84.17907831098073, 92.51075577815588, 104.6266259645173, 121.34050673661721, 110.92262613667035, 111.64281811962262, 103.01873166332402, 111.85638155491911, 113.51849087429409, 109.98673749820924, 141.4827060638512, 151.61206934765008, 148.47390816243544, 179.3330963051759, 155.461566250429, 189.68440955190889, 192.28853817139523, 171.75884227470362, 198.8300006427677, 233.55794811396927, 228.48036855488988, 217.87657825482435, 231.77507154082772, 333.6463572538978, 335.92438274501194, 354.90657379718266, 410.5026527049705, 435.23641780619056, 488.69762785472307, 482.5138975180086, 510.0107602278663, 508.14850110733164, 520.4668688323459, 531.9277582222546, 562.7558298308014, 621.4785754722992, 631.2159147419787, 598.3015903660977, 664.2609414080675, 688.5585250763455, 778.2431827378641, 795.2873057206775, 774.0123880898204, 864.034726634137, 799.9635214955838, 822.0245650480654, 851.5332975813029, 815.5380266586442, 900.229502117395, 867.2033608093482, 893.2961647242598, 980.6550756181821, 928.3584535211279, 1066.9634070100972, 1032.5615753137513, 1167.707620368251, 1032.2771610795319, 1130.3638414775405, 995.619071139676, 1080.359768561607, 1101.949845332204, 1072.6803260643364, 1056.1011412301648, 1087.8642939313304, 1101.583513393463, 1161.367306165429, 1246.5080539041408, 1222.9600941039002, 1146.4137601465309, 1157.2398645199582, 1195.6761551268926, 1263.3793355585235, 1301.7822807442035, 1302.2297080800158, 1269.100402362654, 1229.2578381333628, 1285.0534186076259, 1288.3485360852726, 1338.180493281628, 1305.8149355788648, 1338.0546072316713, 1314.6104672769413, 1430.1723535516996, 1389.645891870621, 1407.0998118746288, 1387.395250579756, 1339.759838794899, 1400.3397216693488, 1404.587918824998, 1326.2447605942145, 1428.1083887288955, 1482.6982762678122, 1441.6470766154844, 1467.5015640275653, 1487.8176958286601, 1531.9088367996287, 1581.7005080428967, 1483.1181992633867, 1570.1704330704688, 1430.041534798919, 1498.1875510753305, 1569.100520695475, 1574.880737739352, 1531.7729300220008, 1479.9190739835917, 1528.3388568529308, 1575.1249835469732, 1556.9238740307176, 1605.3835504491851, 1562.794684556386, 1686.7987662550206, 1716.160963431682, 1567.7679609240702, 1609.01082013725, 1624.9353479302845, 1597.0235376122976, 1662.8066061679651, 1661.4597758639352, 1714.9127508241615, 1693.9419318517125, 1741.327500914633, 1652.671165582362, 1776.3808396772347, 1751.3301482311167, 1694.3842338525092, 1793.1453298336376, 1750.6384114016564, 1774.4998887685915, 1797.578980857071, 1853.8891106813605, 1765.5534310122794, 1750.898270693072, 1837.2648933050964, 1859.727000249616, 1830.9692040024893, 1773.6800781288675, 1722.3875363314944, 1800.6324743378182, 1852.9718137220657, 1793.6538573427656, 1846.22539554031, 1819.7865504755823, 1810.3983256896709, 1849.7026251784653, 1921.9290930280224, 1827.041912689913, 1878.0969136090007, 1851.4044692416246, 1894.7000172642756, 1859.792243199584, 1898.0164425657226, 1926.5716478088414, 1950.3552677406128, 1936.3116827464346, 1957.8378845344391, 1833.0819953473604, 1945.9918395255254, 1908.788758140289, 2035.9489495825828, 1963.2106170353845, 1946.530157813805, 2004.8821560918857, 1995.0009153050917, 1940.9604655811495, 1951.3025817050489, 2050.391934345046, 2043.437314046251, 2014.4424162690696, 2058.5163418284556, 1939.834385106072, 1969.475836648096, 2084.7690496745026, 1994.9758707957817, 2011.6310765232765, 2088.473439523288, 2063.695570786436, 1949.8639611001256, 2001.3566239030567, 1986.1689571016586, 2049.6747169367227, 2064.9791879394616, 2111.8782921091765, 2082.046720674789, 2098.927790815529, 2114.989502810451, 2085.7112697820676, 2105.2937758822127, 2107.29911480593, 2150.682981479375, 2129.4717826104443, 2086.4300049219246, 1980.4444244408717, 2025.9209385240806, 2199.448161668857, 2110.8182712941652, 2088.7040674337427, 2204.9347256881874, 2088.3432603114315, 2059.042455156924, 2059.432138967567, 2093.8676524187213, 2168.4479370175054, 2129.88008875447, 2089.33285594582, 2202.784881386171, 2115.5276699525994, 2223.0334819235663, 2090.023622470039, 2200.3992332019347, 2100.5945167174204, 2185.9518086235294, 2179.348633258776, 2146.989517773463, 2095.2513051422975, 2146.512470113398, 2129.5225047239815, 2150.804403030216, 2101.6824694478155, 2155.5070168069333, 2334.1611548815567, 2243.2323649611994, 2140.3721501079285, 2112.8624319720975, 2156.344609079034, 2201.0820660534473, 2120.754798003843, 2139.997754150503, 2214.8782212472556, 2104.305524447073, 2304.4061986174715, 2272.2295571198433, 2267.2725807452975, 2170.030690245799, 2182.9534721733553, 2225.8169707124666, 2261.9131343621757, 2243.5570725299654, 2185.9931924743883, 2225.9445833350533, 2238.32844216984, 2269.0137905355214, 2282.929354544967, 2233.665200323264, 2179.993669761112, 2188.8830336499364, 2208.0322082071707, 2256.701488232506, 2215.256804970613, 2188.7195277781243, 2279.542217694398, 2234.128495473981, 2336.366083596876, 2244.925251348217, 2172.875276170963, 2279.161485324285, 2247.271068889132, 2211.014513062114, 2222.6784189005416, 2246.663554058772, 2301.3185506999107, 2231.0717454895625, 2258.154527359584, 2284.629565926892, 2336.468519712395, 2204.2731288326595, 2342.7573883312148, 2172.4811883871503, 2363.3041190759564, 2274.3834284730133, 2422.872940295291, 2399.499625916985, 2369.514767394118, 2306.786290157523, 2338.720574744604, 2284.3632994778463, 2297.959822714202, 2266.8603930690692, 2328.5372966851837, 2336.595238440924, 2314.8857184982417, 2244.6828529270792, 2184.5294186147653, 2376.4598449704486, 2294.442704974285, 2429.967496385417, 2301.905699640231, 2374.4731688310803, 2336.7853035436437, 2437.577792786782, 2347.0724148818804, 2297.1748636180023, 2407.724845432092, 2282.061153035606, 2363.2790794603375, 2338.7435606286986, 2372.115704335577, 2410.614385638748, 2356.082915058301, 2509.7732083753476, 2328.583861572539, 2359.5559609691254, 2339.787254870589, 2330.6474998593253, 2364.770889736276, 2427.481835830727, 2325.571498290757, 2266.037248240556, 2337.8435916260955, 2395.3822323228555, 2456.8005952261014, 2514.5731908236953, 2344.769114142491, 2421.7345636503587, 2464.872445649963, 2367.8272887158937, 2409.413711800615, 2422.61683401356, 2367.129757158402, 2448.226257585211, 2347.040750891015, 2445.8328943863335, 2419.5333965351097, 2304.259336945493, 2434.966432207502, 2264.3955239372526, 2462.4213290463103, 2405.532035795821, 2471.424451636674, 2366.6994095574473, 2437.8493798493673, 2459.417184936871, 2351.5355976879628, 2228.540419522444, 2495.3609499696895, 2350.3947411000754, 2426.9181229276155, 2425.9337815158106, 2345.3996604610697, 2423.7100918829983, 2453.7655343070965, 2477.640524392254, 2468.2285232845825, 2481.3786454287565, 2411.7554035600683, 2445.5612260771236, 2340.980945347892, 2437.9947546045137, 2443.9163302735187, 2442.381520219368, 2377.293261635221, 2278.2934289192335, 2366.017001008249, 2487.264511972483, 2506.7783264852005, 2364.650280347654, 2515.046525669052, 2350.729601132357, 2431.110447411685, 2465.4579646018074, 2476.5669892452397, 2451.745716854299, 2476.512639708332, 2470.3375123634314, 2479.857719160201, 2337.6244087382825, 2483.6304378998707, 2410.6487579421105, 2363.903517203713, 2400.988306043965, 2482.93753051068, 2447.5006934537973, 2512.5077272656367, 2392.3946933587117, 2461.9074791742796, 2396.2807956813717, 2541.0015615724064, 2567.6402855265023, 2467.9286505857294, 2450.0883818600646, 2471.4210521955274, 2438.6319465322304, 2484.1335336229217, 2453.5139891637045, 2367.7283531856533, 2432.0975710827433, 2429.126985650747, 2469.2123573746603, 2354.638779266861, 2413.501171915462, 2453.67061655108, 2430.421503524909, 2474.198793300804, 2543.438647697786, 2371.7794104361365, 2487.3393451355414, 2528.493421769672, 2487.2723369642963, 2483.976198684043, 2475.4068269726304, 2465.8907147175255, 2365.4969664007676, 2440.387651670095, 2446.404279781999, 2516.7678659053613, 2569.395477481983, 2486.0138470684556, 2418.0243810758107, 2467.994571460697, 2495.6073403324845, 2379.3339976053653, 2452.0513368483353, 2522.5068708729577, 2511.134506566591, 2498.5765702988956, 2553.429051314839, 2395.954717612799, 2402.510663913778, 2330.6791947924803, 2517.9464547578377, 2455.739488387331, 2488.1722158608854, 2556.516758804557, 2466.5495193214015, 2587.7983035281563, 2505.284283784649, 2443.4285739169977, 2602.356126371252, 2523.09294043561, 2523.7964330742716, 2511.4951566201266, 2479.352092669655, 2532.48014665352, 2555.749784592866, 2486.0913735126887, 2508.262956671241, 2557.812044963393, 2515.3495737788166, 2553.3619333928104, 2466.4462220092414, 2536.8782710697606, 2552.5711299522695, 2411.1472911095566, 2478.409619933372, 2424.270283728059, 2405.278824588639, 2470.225508153321, 2522.293347321147, 2571.0782979736887, 2361.938684487923, 2485.949063328984, 2557.3851156693067, 2592.561555907316, 2488.0123432066575, 2488.7683914963222, 2525.439348827914, 2423.96830247661, 2603.5746322026343, 2519.0929400514965, 2459.089180147342, 2594.2270530829255, 2557.042140504952, 2575.502240936338, 2585.462033823558, 2537.7332591987256, 2395.11741956067, 2495.2216576117685, 2518.381502792082, 2511.369743607345, 2616.841842531799, 2472.043194480946, 2540.541222796332, 2607.6151210322687, 2536.174059750769, 2505.823015976388, 2435.7747870555795, 2570.7143023561916, 2616.601518322998, 2480.2180780778617, 2433.326477240559, 2557.168013359939, 2459.9746670743307, 2513.9382456406024, 2514.529185172395, 2452.603368553685, 2596.854406809481, 2531.659946578489, 2606.275870191946, 2583.691810636782, 2597.716256311569, 2582.5966065297025, 2626.892296320194, 2566.795384272892, 2459.831217632853, 2568.2014654185346, 2586.0181789058856, 2547.725583322646, 2506.497771969284, 2637.6104546483066 ] } }, "Isaac-Velocity-Flat-H1-v0": { "FastTD3": { "time": [ 0.0, 306.64285714285717, 613.2857142857143, 919.9285714285714, 1226.5714285714287, 1533.2142857142858, 1839.857142857143, 2146.5, 2453.1428571428573, 2759.785714285714, 3066.4285714285716, 3373.0714285714284, 3679.714285714286, 3986.3571428571427, 4293.0 ], "env_step": [ 0, 20480000.0, 40960000.0, 61440000.0, 81920000.0, 102400000.0, 122880000.0, 143360000.0, 163840000.0, 184320000.0, 204800000.0, 225280000.0, 245760000.0, 266240000.0, 286720000.0 ], "return": [ 0.0, 17.756866772969563, 27.37510617574056, 31.223042170206707, 31.729761759440105, 32.67671267191569, 32.91831970214844, 33.834747314453125, 33.87761561075846, 33.86462910970052, 35.029486338297524, 35.26934560139974, 35.934749603271484, 35.113051096598305, 35.61514027913412 ], "return_min": [ 0.0, 14.716606314864778, 25.070346544765535, 31.068635948141047, 30.948657101492984, 31.904495164552372, 31.709390777382133, 33.20864308639823, 33.359869135298865, 33.35168347791627, 34.427511220703074, 34.53568214692868, 35.450113096166, 34.36188582932133, 35.525983562100734 ], "return_max": [ 0.0, 20.79712723107435, 29.679865806715583, 31.377448392272367, 32.51086641738723, 33.448930179279, 34.12724862691474, 34.46085154250802, 34.39536208621806, 34.37757474148478, 35.63146145589197, 36.003009055870805, 36.41938611037697, 35.86421636387528, 35.7042969961675 ] }, "null": { "time": [ 0.0, 1.938, 3.876, 5.814, 7.752, 9.69, 11.628, 13.566, 15.504, 17.442, 19.38, 21.318, 23.256, 25.194, 27.132, 29.07, 31.008, 32.946, 34.884, 36.822, 38.76, 40.698, 42.636, 44.574, 46.512, 48.45, 50.388, 52.326, 54.264, 56.202, 58.14, 60.078, 62.016, 63.954, 65.892, 67.83, 69.768, 71.706, 73.644, 75.582, 77.52, 79.458, 81.396, 83.334, 85.272, 87.21, 89.148, 91.086, 93.024, 94.962, 96.9, 98.838, 100.776, 102.714, 104.652, 106.59, 108.528, 110.466, 112.404, 114.342, 116.28, 118.218, 120.156, 122.094, 124.032, 125.97, 127.908, 129.846, 131.784, 133.722, 135.66, 137.598, 139.536, 141.474, 143.412, 145.35, 147.288, 149.226, 151.164, 153.102, 155.04, 156.978, 158.916, 160.854, 162.792, 164.73, 166.668, 168.606, 170.544, 172.482, 174.42, 176.358, 178.296, 180.234, 182.172, 184.11, 186.048, 187.986, 189.924, 191.862, 193.8, 195.738, 197.676, 199.614, 201.552, 203.49, 205.428, 207.366, 209.304, 211.242, 213.18, 215.118, 217.056, 218.994, 220.932, 222.87, 224.808, 226.746, 228.684, 230.622, 232.56, 234.498, 236.436, 238.374, 240.312, 242.25, 244.188, 246.126, 248.064, 250.002, 251.94, 253.878, 255.816, 257.754, 259.692, 261.63, 263.568, 265.506, 267.444, 269.382, 271.32, 273.258, 275.196, 277.134, 279.072, 281.01, 282.948, 284.886, 286.824, 288.762, 290.7, 292.638, 294.576, 296.514, 298.452, 300.39, 302.328, 304.266, 306.204, 308.142, 310.08, 312.018, 313.956, 315.894, 317.832, 319.77, 321.708, 323.646, 325.584, 327.522, 329.46, 331.398, 333.336, 335.274, 337.212, 339.15, 341.088, 343.026, 344.964, 346.902, 348.84, 350.778, 352.716, 354.654, 356.592, 358.53, 360.468, 362.406, 364.344, 366.282, 368.22, 370.158, 372.096, 374.034, 375.972, 377.91, 379.848, 381.786, 383.724, 385.662, 387.6, 389.538, 391.476, 393.414, 395.352, 397.29, 399.228, 401.166, 403.104, 405.042, 406.98, 408.918, 410.856, 412.794, 414.732, 416.67, 418.608, 420.546, 422.484, 424.422, 426.36, 428.298, 430.236, 432.174, 434.112, 436.05, 437.988, 439.926, 441.864, 443.802, 445.74, 447.678, 449.616, 451.554, 453.492, 455.43, 457.368, 459.306, 461.244, 463.182, 465.12, 467.058, 468.996, 470.934, 472.872, 474.81, 476.748, 478.686, 480.624, 482.562, 484.5, 486.438, 488.376, 490.314, 492.252, 494.19, 496.128, 498.066, 500.004, 501.942, 503.88, 505.818, 507.756, 509.694, 511.632, 513.57, 515.508, 517.446, 519.384, 521.322, 523.26, 525.198, 527.136, 529.074, 531.012, 532.95, 534.888, 536.826, 538.764, 540.702, 542.64, 544.578, 546.516, 548.454, 550.392, 552.33, 554.268, 556.206, 558.144, 560.082, 562.02, 563.958, 565.896, 567.834, 569.772, 571.71, 573.648, 575.586, 577.524, 579.462, 581.4, 583.338, 585.276, 587.214, 589.152, 591.09, 593.028, 594.966, 596.904, 598.842, 600.78, 602.718, 604.656, 606.594, 608.532, 610.47, 612.408, 614.346, 616.284, 618.222, 620.16, 622.098, 624.036, 625.974, 627.912, 629.85, 631.788, 633.726, 635.664, 637.602, 639.54, 641.478, 643.416, 645.354, 647.292, 649.23, 651.168, 653.106, 655.044, 656.982, 658.92, 660.858, 662.796, 664.734, 666.672, 668.61, 670.548, 672.486, 674.424, 676.362, 678.3, 680.238, 682.176, 684.114, 686.052, 687.99, 689.928, 691.866, 693.804, 695.742, 697.68, 699.618, 701.556, 703.494, 705.432, 707.37, 709.308, 711.246, 713.184, 715.122, 717.06, 718.998, 720.936, 722.874, 724.812, 726.75, 728.688, 730.626, 732.564, 734.502, 736.44, 738.378, 740.316, 742.254, 744.192, 746.13, 748.068, 750.006, 751.944, 753.882, 755.82, 757.758, 759.696, 761.634, 763.572, 765.51, 767.448, 769.386, 771.324, 773.262, 775.2, 777.138, 779.076, 781.014, 782.952, 784.89, 786.828, 788.766, 790.704, 792.642, 794.58, 796.518, 798.456, 800.394, 802.332, 804.27, 806.208, 808.146, 810.084, 812.022, 813.96, 815.898, 817.836, 819.774, 821.712, 823.65, 825.588, 827.526, 829.464, 831.402, 833.34, 835.278, 837.216, 839.154, 841.092, 843.03, 844.968, 846.906, 848.844, 850.782, 852.72, 854.658, 856.596, 858.534, 860.472, 862.41, 864.348, 866.286, 868.224, 870.162, 872.1, 874.038, 875.976, 877.914, 879.852, 881.79, 883.728, 885.666, 887.604, 889.542, 891.48, 893.418, 895.356, 897.294, 899.232, 901.17, 903.108, 905.046, 906.984, 908.922, 910.86, 912.798, 914.736, 916.674, 918.612, 920.55, 922.488, 924.426, 926.364, 928.302, 930.24, 932.178, 934.116, 936.054, 937.992, 939.93, 941.868, 943.806, 945.744, 947.682, 949.62, 951.558, 953.496, 955.434, 957.372, 959.31, 961.248, 963.186, 965.124, 967.062, 969.0 ], "env_step": [ 0, 98304, 393216, 491520, 688128, 786432, 983040, 1081344, 1179648, 1376256, 1572864, 1966080, 2457600, 2752512, 3244032, 3342336, 3735552, 4128768, 4227072, 4521984, 4620288, 5013504, 5406720, 5701632, 5799936, 5996544, 6193152, 6782976, 6979584, 7077888, 7274496, 7569408, 7667712, 7766016, 7962624, 8060928, 8159232, 8257536, 8355840, 8454144, 8552448, 8650752, 8847360, 8945664, 9043968, 9142272, 9240576, 9338880, 9437184, 9535488, 10027008, 10518528, 10813440, 10911744, 11304960, 11599872, 11698176, 11894784, 12189696, 12288000, 12582912, 13074432, 13369344, 13467648, 14057472, 14155776, 14254080, 14450688, 14647296, 14745600, 14843904, 15138816, 15335424, 15433728, 15532032, 15925248, 16023552, 16220160, 16318464, 16416768, 16711680, 16809984, 16908288, 17006592, 17203200, 17301504, 17399808, 17694720, 18087936, 18284544, 18481152, 18579456, 18677760, 18874368, 19169280, 19267584, 19464192, 19759104, 19857408, 19955712, 20348928, 20447232, 20840448, 20938752, 21135360, 21233664, 21430272, 21528576, 21823488, 21921792, 22020096, 22413312, 22708224, 23592960, 23691264, 23789568, 23887872, 23986176, 24182784, 24674304, 24772608, 24870912, 24969216, 25165824, 25460736, 25559040, 25657344, 25755648, 26050560, 26247168, 26443776, 26738688, 26836992, 26935296, 27033600, 27230208, 27623424, 28016640, 28114944, 28409856, 28508160, 28606464, 28803072, 28901376, 29097984, 29392896, 29491200, 29786112, 29982720, 30277632, 30375936, 30572544, 30670848, 30867456, 31162368, 31457280, 31555584, 31653888, 31850496, 31948800, 32636928, 32931840, 33030144, 33128448, 33325056, 33423360, 33521664, 33619968, 33914880, 34209792, 34504704, 34897920, 34996224, 35192832, 35389440, 35487744, 35586048, 35782656, 35979264, 36077568, 36372480, 36864000, 36962304, 37060608, 37257216, 37355520, 37453824, 37552128, 37650432, 37847040, 38043648, 38141952, 38338560, 38633472, 38731776, 38830080, 38928384, 39026688, 39124992, 39223296, 39419904, 39518208, 39714816, 39813120, 39911424, 40108032, 40206336, 40501248, 40599552, 40697856, 40796160, 41091072, 41189376, 41287680, 41385984, 41484288, 41582592, 41680896, 41877504, 42074112, 42467328, 42565632, 42663936, 42860544, 42958848, 43155456, 43253760, 43450368, 43548672, 43646976, 43745280, 43843584, 44138496, 44236800, 44335104, 44531712, 45023232, 45121536, 45219840, 45318144, 45514752, 45613056, 46006272, 46497792, 46596096, 46989312, 47087616, 47185920, 47284224, 47480832, 47775744, 47972352, 48758784, 48857088, 49152000, 49545216, 49840128, 50036736, 50135040, 50233344, 50626560, 50724864, 51019776, 51609600, 51904512, 52101120, 52199424, 52396032, 52494336, 52690944, 52887552, 52985856, 53182464, 53379072, 53870592, 54165504, 54263808, 54362112, 54460416, 54558720, 55050240, 55148544, 55246848, 55345152, 55443456, 55541760, 56033280, 56131584, 56426496, 56918016, 57016320, 57212928, 57409536, 57606144, 57704448, 57901056, 57999360, 58097664, 58195968, 58294272, 58490880, 58687488, 58785792, 58884096, 59080704, 59375616, 59473920, 59670528, 59867136, 59965440, 60063744, 60162048, 60260352, 60456960, 60555264, 60751872, 61046784, 61145088, 61341696, 61636608, 61931520, 62029824, 62128128, 62619648, 62717952, 62816256, 62914560, 63012864, 63111168, 63209472, 63307776, 63406080, 63504384, 63700992, 63799296, 64094208, 64192512, 64487424, 64585728, 64782336, 64880640, 64978944, 65077248, 65372160, 65470464, 65667072, 65765376, 66158592, 66256896, 66453504, 66650112, 66748416, 66846720, 67239936, 67829760, 68026368, 68419584, 68911104, 69107712, 69402624, 69500928, 69697536, 69795840, 70090752, 70287360, 70385664, 70582272, 70778880, 71368704, 71467008, 71565312, 71761920, 71958528, 72056832, 72351744, 72843264, 73039872, 73138176, 73236480, 73334784, 73433088, 73629696, 73728000, 73826304, 74022912, 74121216, 74317824, 74711040, 74809344, 74907648, 75005952, 75300864, 75399168, 75497472, 75890688, 76087296, 76382208, 76972032, 77070336, 77168640, 77266944, 77365248, 77463552, 77955072, 78544896, 79036416, 79134720, 79233024, 79527936, 79724544, 79822848, 80117760, 80314368, 80412672, 81002496, 81100800, 81395712, 81592320, 81690624, 81788928, 82280448, 82378752, 82477056, 82575360, 82968576, 83558400, 83755008, 84148224, 84344832, 84443136, 84836352, 85229568, 85327872, 85426176, 85524480, 86016000, 86409216, 86507520, 86605824, 86704128, 86802432, 86900736, 86999040, 87293952, 87588864, 87687168, 87785472, 87982080, 88080384, 88178688, 88276992, 88375296, 88473600, 88768512, 89456640, 89653248, 90046464, 90243072, 90439680, 90636288, 90734592, 90931200, 91127808, 91226112, 91422720, 91521024, 91914240, 92209152, 92307456, 92405760, 92504064, 92602368, 92897280, 92995584, 93192192, 93388800, 93487104, 93585408, 93683712, 93880320, 93978624, 94076928, 94273536, 94568448, 94961664, 95059968, 95453184, 95649792, 95748096, 96141312, 96239616, 96337920, 96632832, 96829440, 96927744, 97026048, 97124352, 97517568, 97615872, 97714176, 98009088 ], "return": [ 0.0, -1.8211565176363214, -5.450758711894353, -5.3709365749359135, -5.429918484687806, -5.405227454503378, -5.405241473515829, -5.415232059160869, -5.403986158370972, -5.387823613484701, -5.385273424784342, -5.385051577885946, -5.3242382780710855, -5.249849046071371, -5.189958149592082, -5.174289337793986, -5.123459353446961, -5.08548603216807, -5.016271096865336, -5.009552868207295, -4.954833366076151, -5.0061487213770555, -5.083908321062723, -5.117522551218669, -5.198800565401712, -5.252866994539897, -5.271668980916341, -5.396079812049866, -5.497399250666301, -5.5650003163019806, -5.655306051572164, -5.701566495895386, -5.728630830446879, -5.84670994758606, -5.9707754500706995, -6.021102941830953, -6.134735550880432, -6.214331653912862, -6.284563399950663, -6.213578990697861, -6.246251831650734, -6.054963727196058, -5.333051507820685, -5.382347539166609, -5.251159350425005, -5.389291501591603, -5.257902181896691, -5.346594299984474, -5.132231690908472, -5.119642965160311, -4.369317753314972, -2.551518141825994, -2.0427605971507727, -1.654090012193968, -0.45424261227250096, 0.30654167612083255, 0.9049611513409764, 1.4666259219994149, 1.8304008860948187, 2.1126247743951776, 2.523588375778248, 4.109312493679268, 4.620685461089015, 5.1200420312893895, 6.284545958700279, 6.4868240279083444, 6.8971409591659905, 7.536932006279627, 8.15484679798285, 8.331846399108569, 8.431811542312305, 8.856677153110503, 9.335615858236949, 9.78888226588567, 10.49026326974233, 10.818796298503875, 11.156456514994304, 12.019008894761404, 12.639701535701752, 13.121436796983083, 13.421463690598806, 13.684878850777944, 14.299875932534535, 14.655534369150798, 15.455265655517579, 15.615330264568328, 15.952052096525827, 16.416555580298105, 17.00739818652471, 17.326351670424142, 17.74453566392263, 17.81917943954468, 18.01390697956085, 18.690608501434326, 19.088207910855612, 19.423237595558167, 19.538911277453106, 19.90228245576223, 20.2472456741333, 21.422647728919983, 22.148719353675844, 22.370413347880046, 22.562456116676334, 22.841183287302652, 23.412538000742597, 23.730912761688234, 23.912889324824018, 24.026660318374635, 24.433674205144246, 24.443872968355816, 24.66891985098521, 24.82599383831024, 25.119243364334107, 25.31676951805751, 26.091290965080262, 26.447999830245973, 26.64739019076029, 26.733827587763468, 27.08850084940592, 27.234617513020833, 27.478463020324707, 27.659565181732177, 27.78176722844442, 28.15255584081014, 28.325472304026288, 28.51744202931722, 28.56866101582845, 28.694656810760502, 29.01594970703125, 29.116026492913566, 29.186051642894743, 29.267378375530242, 29.347289582093556, 29.509137900670368, 29.575214611689248, 29.494310462474825, 29.566665900548298, 29.60818713188171, 29.639113759994505, 29.53559107542038, 29.904576396942144, 30.098937937418622, 30.19250619252523, 30.24227956771851, 30.431443729400637, 30.525310249328612, 30.656077963511148, 30.735889282226562, 30.977694892883303, 30.981900826295217, 31.003630145390826, 31.060925975640615, 31.31851980845133, 31.361310170491535, 31.415920890967055, 31.278630374272666, 31.255216824213665, 31.50279481649399, 31.523300778865813, 31.72419583002726, 31.805352369944256, 31.853926067352294, 31.89563065846761, 31.944775962829592, 31.964195365905763, 32.005060046513876, 32.09825029055278, 32.20195636113485, 32.22373517990113, 32.28627243677775, 32.30005905151367, 32.32767866770426, 32.29544750054677, 32.26161565621694, 32.263696993192035, 32.373338920275366, 32.55905792236328, 32.5971901512146, 32.6483419418335, 32.577331740061446, 32.65967615763346, 32.50604391256968, 32.524837381045025, 32.453570529619846, 32.473360339800514, 32.50699528217316, 32.506180488268534, 32.38849087794622, 32.69574457168579, 32.88326619466146, 32.86996450424194, 32.8433007367452, 32.78393250147502, 32.71742320140203, 32.82113723754883, 32.95492044448852, 33.014233360290525, 32.95029862721761, 32.85574058214823, 32.63808204094568, 32.99278465588887, 33.014845549265544, 32.9728839302063, 32.93397596995035, 32.93905897140503, 32.82802639643351, 33.13534347216288, 33.17514415105184, 33.158193543752034, 33.1641508102417, 33.12397045771281, 33.163712355295814, 33.11274451891581, 33.1312927087148, 33.143696921666454, 33.16906797726949, 33.178442191282905, 32.782245856126146, 32.80315121809642, 32.75034557580948, 32.67346092542013, 32.794016884167995, 32.83101738452911, 32.991992235183716, 33.21010211626689, 33.21421767870585, 33.16107631365458, 33.34934869766235, 33.3692709350586, 33.232156912485756, 33.22941769282023, 33.397237396240236, 33.35016419092814, 33.349866479237875, 33.06404978672663, 33.11358634869257, 33.121176311969755, 33.3388022740682, 33.27921138127645, 33.40164766311646, 33.26413485129674, 33.38225418726603, 33.25256678104401, 33.34821179548899, 33.353779506683345, 33.49222758611044, 33.44428084691366, 33.51035448074341, 33.359815897146866, 33.3635134824117, 33.3504612382253, 33.269343312581384, 33.32479614496231, 33.34401448726654, 33.41709674676259, 33.360723201433814, 33.26504530906678, 33.209413143793746, 33.18987674395243, 33.44505291938781, 33.48292690277099, 33.544238618214926, 33.54053416570028, 33.54602517445882, 33.44048034191132, 33.470887223879494, 33.45609283606212, 33.60205269495646, 33.60698293685913, 33.64999708175659, 33.42452413479487, 33.21609922568003, 33.38258064905802, 33.48653169314067, 33.47724048535029, 33.42847904761632, 33.446080621083574, 33.46206173578898, 33.45817431767781, 33.49116609573364, 33.59252418518067, 33.59685064315796, 33.64596748987834, 33.60812362670899, 33.535508891741436, 33.64438659667969, 33.54005140622457, 33.40192108074824, 33.46623891750971, 33.573429889678955, 33.59540475845337, 33.57583349863688, 33.54388561884562, 33.60371445973714, 33.51048959970475, 33.26232818206151, 33.357575463453934, 33.67110483805339, 33.843606103261315, 33.799716498057045, 33.775785026550295, 33.812021497090655, 33.82620253245036, 33.807082901000975, 33.81073764165242, 33.63764720280965, 33.485613335768385, 33.65504789034525, 33.64657088915507, 33.74490778446198, 33.688238808314004, 33.65053201357524, 33.63879053115844, 33.75179484367371, 33.647144407431284, 33.66565582354864, 33.70479655663172, 33.741632859706876, 33.691917811234795, 33.69865835030873, 33.71498268286387, 33.65212141513825, 33.829328810373944, 33.85409680366516, 33.76617241859436, 33.86793506940206, 33.95968713760376, 33.957069365183514, 33.96092315673828, 33.9038141822815, 33.78778015931447, 33.802784686088565, 33.772612570126846, 33.9072968451182, 33.70608182668686, 33.72268221696218, 33.55177603403727, 33.914818897247315, 33.96780169169108, 33.953882675170895, 33.93014724095662, 33.98231894810994, 34.00627389907837, 33.90533470153809, 33.743881049950915, 33.74382516463597, 33.882653643290205, 33.64851376930873, 33.627794663111366, 33.58648926973343, 33.86447109540304, 34.00397662480672, 33.97151560465495, 33.960879128774, 34.00438674926758, 33.868241135279334, 34.008157377243045, 33.93994318962097, 33.86863050699234, 33.942175194422404, 33.71918567577998, 33.833626924355826, 33.925179407596595, 33.953650324344636, 33.954390884240475, 33.76647464593251, 33.7064058415095, 33.85868651946385, 33.66485804319382, 33.75230008284251, 33.7514514986674, 33.88039131800334, 33.99921775182088, 33.79016396840413, 33.70870203018188, 33.6108287191391, 34.0640186325709, 34.005900454521175, 34.13911101659139, 34.08947592894236, 34.20443028767904, 34.22977528889974, 33.92905995845794, 33.909010221163435, 33.91740820089976, 33.95200910568237, 34.14121877988179, 34.023240098953245, 33.9469082959493, 34.020559568405154, 34.05597455660502, 34.06671770413717, 34.19973275502523, 33.96253667593002, 33.83099495649338, 33.4363557044665, 33.69214943011602, 34.0144168249766, 34.00501637458802, 34.17815904617309, 34.2427860959371, 34.245318311055506, 34.188450533548995, 34.22512925465902, 34.15011143684387, 34.2105561765035, 34.13866239150366, 34.12881470441818, 34.110780564149216, 34.22185877482096, 34.2055456161499, 34.165921246210736, 34.14928177833557, 33.96859579165777, 33.83083196242651, 33.84601767460506, 33.855398262341815, 33.90395102659861, 33.87031110127767, 33.7126193745931, 33.87908462524414, 33.95480376561483, 34.25536927223205, 34.22514481544495, 34.27022196451823, 34.18469769001007, 34.00267321586609, 34.32991437276204, 34.30865136464437, 34.28107172648112, 34.19567479451497, 34.210505580902094, 34.00116219679514, 34.07368288199107, 34.046332208315526, 33.97249688625336, 34.081047983169555, 34.074682232538855, 34.0766791232427, 33.994779232343035, 33.904674633344015, 33.83213842709859, 34.186360073089595, 34.22037427902222, 34.04206380049387, 33.92758392572403, 33.86605194489161, 34.140858041445405, 33.71333369255066, 34.19078587214151, 34.12247137387593, 33.897727811336516, 34.093153146107994, 34.081556427478795, 34.225092843373616, 34.18845344543457, 34.03204092979431, 34.113009688059485, 33.65711380243301, 33.61201909224192, 33.91202318350474, 34.027429844538375, 34.03296609560649, 33.95445609887441, 33.90101886590322, 34.025549514293665, 33.72658471345901, 34.018566299279534, 33.888727053006484, 33.98073536554972, 33.990113363266, 34.07285011450449, 34.21228943506876, 34.15036308288574, 34.02376880645752, 33.992461017767596, 34.11304693222046, 34.16727955500285, 34.13363500595093, 33.661870261828106, 33.74703230381012, 34.067941786448166, 34.21645594914754, 34.21975947697957, 34.18952447255453, 34.21511004765828, 34.25453131357828, 34.22559996922811, 34.182696437835695, 34.18072349866231, 34.169769096374516, 34.24306580225627, 34.274355023701986, 34.28168207804362, 34.30049002329508, 34.11715464274088, 34.11595391432444, 34.07781842867533, 33.51906168699264, 33.571848865350084, 33.712630089918775 ], "return_min": [ 0.0, -4.209529659930448, -5.628355577413527, -5.454971782483552, -5.463996139367627, -5.432741154225297, -5.426888002217753, -5.436807539475557, -5.425299966313791, -5.412182911469524, -5.4104728792417385, -5.42808231221047, -5.375521999410365, -5.3230288484886135, -5.2797338855391915, -5.240507263917413, -5.19926127769744, -5.179219142521742, -5.07908284410958, -5.032903823359099, -4.969668282106384, -5.094362818841234, -5.281996883458076, -5.3920096236993045, -5.553269647597237, -5.729036621436148, -5.778985218016288, -6.0301838321880625, -6.217390434009163, -6.3835459993152615, -6.435836116763705, -6.501016284407782, -6.518235181677481, -6.641257321926678, -6.804782635415524, -6.825958611170952, -6.900034629882504, -7.004061243688619, -7.070126037618526, -6.943216421817451, -6.772073465644133, -6.451086699481175, -6.637737980770315, -7.133531965887451, -7.235613664375382, -7.620630142315411, -7.732041885493856, -7.93043079664659, -8.039439052465882, -8.052169475312821, -8.145489222359249, -6.533113114668902, -6.194064854392917, -5.821998196115499, -4.269998114915388, -3.060062102494689, -2.1766498666046306, -1.522520611941957, -1.0960023869981923, -0.8067221361568175, -0.34875043312814347, -0.24051924615993325, -0.030125249299234547, 0.6147006250639313, 1.6705156621163555, 1.8978773573042886, 2.492487197096148, 3.2078728538939396, 3.9325072151810927, 4.1495417987583405, 4.464200048218766, 4.695257760930558, 5.106385058253912, 5.80625788052178, 6.747859303388516, 7.033520115548896, 7.61384687106856, 9.121095072409364, 10.247541656339465, 11.164436568949185, 11.350218779574417, 11.715560912007636, 12.267298712031716, 12.49084894245549, 12.617885254881479, 12.860896229018945, 13.149930335939148, 13.545043920619051, 14.029415433170419, 14.37584588528084, 14.804220948047497, 14.792772486155016, 14.983113341048067, 15.783149829958043, 15.928902106852796, 16.3501276896744, 16.427507281469122, 16.841994678140587, 17.335166265053587, 19.849475169109517, 20.568988023923914, 20.92521922217223, 21.098380853856582, 21.237686399862632, 21.481328160428717, 21.861707905672635, 21.946872710964136, 22.15615356250457, 22.480182586951475, 22.439113334324347, 22.783054452309194, 22.71390440698288, 23.02964898305132, 23.057338734037277, 24.1364739924029, 24.55653930143314, 24.790787121035002, 24.863631804668003, 25.544529906995944, 25.892309377370864, 26.34160171474906, 26.628026989254643, 26.781689166899664, 26.97167769178579, 27.025717843397775, 27.279217143675528, 27.345248394639587, 27.534701973589357, 28.028468767921982, 28.12209195976707, 28.158240120581077, 28.17333204523116, 28.265092400875353, 28.47092830171811, 28.51324744549357, 28.60718218296158, 28.404079499641472, 28.433513656812547, 28.48298083773327, 28.20777281229254, 28.83806026401171, 29.095672022051513, 29.176605133554403, 29.227176012444364, 29.51045156047436, 29.55934002827877, 29.725652398442254, 29.857791131590933, 30.250474949875066, 30.097892827836176, 30.1205720272998, 30.222571934983538, 30.593196755308323, 30.67311152561452, 30.657967491293633, 30.58703137205276, 30.578546793645796, 30.716421153479097, 30.756165798926283, 31.130321696185632, 31.180588597373752, 31.250815806957338, 31.274866022851175, 31.360237967449972, 31.344060491523454, 31.399367117675258, 31.514803511527273, 31.678545248104303, 31.724162265731543, 31.718766100096886, 31.820441167887104, 31.809748956778147, 31.667314786205985, 31.665149156142952, 31.62035110631372, 31.742718209367506, 32.022972224180734, 32.04248009692774, 32.197398524591215, 32.03430305853228, 32.237484974198075, 32.100151691025054, 32.09026116382551, 31.987826541323436, 31.984515127439128, 31.933867257918262, 31.919322252655554, 31.77659611423544, 32.206594745672824, 32.45533158655721, 32.39492503630701, 32.304099971179696, 32.25262729610579, 32.41908143160763, 32.30487923098815, 32.5582240553896, 32.60160459883723, 32.431401203565294, 32.369493417243255, 32.21161546726209, 32.510942233125974, 32.55640827577592, 32.48728961375227, 32.446934766045196, 32.43471770493697, 32.20878724298071, 32.76947491920594, 32.80921502629607, 32.79006888503141, 32.781752366577294, 32.731922917885335, 32.82623178897912, 32.6462664405936, 32.63425745048648, 32.632413252321314, 32.65034100784082, 32.57447847932977, 31.77427950571114, 31.909927808121, 31.618161890837925, 31.47261618272682, 31.717342494918874, 31.838050338276254, 32.41965951145684, 32.749675875368965, 32.79157376861183, 32.70116923350964, 32.8588465396553, 32.9674953250711, 32.811215486513525, 32.717401451010154, 32.95060249716797, 32.87874931891394, 32.84635712663109, 32.143342025778594, 32.289241920374245, 32.33099643505135, 32.76786138310686, 32.72909474247403, 32.93045251612466, 32.8030939787735, 32.87775578006752, 32.70376760973, 32.81084849537366, 32.780997041413244, 32.968767569051465, 32.93412799801414, 32.98463241141619, 32.69498579548257, 33.03098130574189, 32.6656808730222, 32.56637424042163, 32.65086773570051, 32.67490052230798, 32.73366487570359, 32.709750099740184, 32.365409665479326, 32.34194315152263, 32.23646837486713, 32.75996267344055, 32.870196870809686, 33.093166501517324, 33.07097673853406, 33.106031609170124, 33.02547366270539, 33.07608886836146, 33.078466212870545, 33.07309098432332, 33.05535620081123, 33.15630201158901, 32.80892911401169, 32.493294814600226, 32.718674498279675, 32.835207144874474, 32.84555438935561, 32.760458185436136, 32.642590885496105, 32.645517500011216, 32.787408521131695, 32.803128271257215, 32.96766573915857, 33.02182458196768, 33.06764542818848, 33.04412100149526, 33.021516566445484, 33.02732691408033, 32.98825196500852, 32.751573189623244, 32.92157085654797, 33.008391144829616, 33.080099670691325, 33.06197665637975, 33.04223494190235, 33.13707536866676, 32.989384983307176, 32.478441922327704, 32.809178853591355, 33.25661037163764, 33.39445979675766, 33.405295153552395, 33.35775763402796, 33.32191098609057, 33.35619586350611, 33.299655769009085, 33.30764757278204, 32.8644155605529, 32.88229689011347, 32.91915262551825, 33.17257336994728, 33.14484868109328, 33.18764081460977, 33.158712628442274, 33.1068222921623, 33.26054504412447, 33.32513415225772, 33.372564664651954, 33.42742018698553, 33.48416863808679, 33.436095550397106, 33.126944711982034, 33.46067212999888, 33.51442993553725, 33.540470684181045, 33.46254666887379, 33.20444097781898, 33.377362989405796, 33.60318391569346, 33.57344715637073, 33.586223732418226, 33.60310619733873, 33.291651688605775, 33.25409801387752, 33.214976891334466, 33.52219575845811, 33.15037514066368, 33.15326696850124, 32.87370710954431, 33.54568352000057, 33.61704161008727, 33.62752242483598, 33.56393179559753, 33.65882495209487, 33.660115425241685, 33.67286110422097, 33.30801149644509, 33.300742198021766, 33.60665529291178, 32.859103411917665, 32.914552223565174, 32.95125362838702, 33.39018207803091, 33.61013390260663, 33.612453490604345, 33.48235920338519, 33.47339237810102, 33.191781660525834, 33.52863233851444, 33.620556089757095, 33.57614865826481, 33.58555271932166, 33.303261304921946, 33.508115073184634, 33.41952419674288, 33.43345984596744, 33.43200662672882, 33.14624532362674, 33.04859363863694, 33.29341880859891, 32.918686567270065, 33.17985651291829, 32.965548127877376, 33.10163278964282, 33.363579769434914, 32.83360904204991, 32.711381927941886, 32.70752628894768, 33.7298202959727, 33.63457119738028, 33.64793008978906, 33.56577284885361, 33.690202662460585, 33.76106324297055, 33.306808336901106, 33.32180686687506, 33.275599433915005, 33.349055974790524, 33.595424980213885, 33.42629543250327, 33.38586942035939, 33.510834643329005, 33.54593447676686, 33.538963472683065, 33.72586910080425, 33.22731798357096, 32.823112523161, 32.46374112984253, 32.526541578874195, 33.31524413846802, 33.29824960721062, 33.661277133487346, 33.743415806420096, 33.71825778666474, 33.64058746221431, 33.637353039854816, 33.47049646526259, 33.60852705127985, 33.570249232057954, 33.53589359925487, 33.51646917475235, 33.652598212485174, 33.65615853904905, 33.451178705396565, 33.434563643244886, 33.21332901949298, 32.95450014870425, 33.01047405583394, 32.76524800040569, 32.79024886950684, 32.856638338699675, 32.476051180352556, 32.92826454418564, 33.07934915126772, 33.50815317466772, 33.61016219502737, 33.67094222164079, 33.5974662406585, 33.33226295166716, 33.73789944265289, 33.661757322902666, 33.6498226778196, 33.49991993987689, 33.58044768148721, 33.09412398739152, 33.27651517632219, 33.119493786965954, 33.142376454589446, 33.61579164819622, 33.60038353053637, 33.21038260677685, 33.03290612718018, 32.79867687333837, 32.8516254366356, 33.48160374972705, 33.61470766029705, 33.37146528384432, 33.20071362052474, 33.040557209674404, 33.48061965929551, 33.157542261300485, 33.62163890121192, 33.34363454006072, 33.19473801655624, 33.314548215535616, 33.28270404732501, 33.54525858592337, 33.50697891926656, 33.417634181268, 33.572607856603355, 32.58939569769712, 32.58494234591918, 32.86144232437622, 33.05700141705454, 33.08225605251803, 33.01093363603906, 32.812148900160935, 33.29221635477259, 32.796024803236065, 33.28556066972077, 32.91004207668372, 33.14638977414718, 33.104234306250206, 33.34592715103986, 33.50639581739729, 33.60880215573404, 33.35797515442404, 33.21328473787432, 33.313652344537516, 33.448086993150646, 33.41096578399726, 32.33001209193853, 32.52109783963744, 33.29527984774868, 33.58159214390531, 33.51072681904591, 33.56232171762343, 33.51557845368608, 33.56718987218156, 33.550278943889005, 33.443680915948306, 33.45137163124387, 33.59035406831639, 33.501152248875314, 33.61012691014657, 33.62100590719376, 33.68242704066349, 33.20934171056824, 33.40098750245513, 33.59525360877404, 32.5350500516155, 32.648825969698215, 33.04150141687526 ], "return_max": [ 0.0, 0.5672166246578054, -5.273161846375179, -5.286901367388275, -5.395840830007986, -5.3777137547814595, -5.383594944813906, -5.393656578846181, -5.3826723504281535, -5.363464315499878, -5.360073970326945, -5.342020843561422, -5.272954556731806, -5.176669243654128, -5.100182413644972, -5.108071411670558, -5.047657429196482, -4.991752921814399, -4.953459349621092, -4.98620191305549, -4.939998450045919, -4.917934623912877, -4.88581975866737, -4.843035478738034, -4.8443314832061874, -4.776697367643646, -4.764352743816394, -4.761975791911669, -4.777408067323439, -4.7464546332887, -4.874775986380623, -4.90211670738299, -4.9390264792162775, -5.052162573245442, -5.136768264725875, -5.216247272490953, -5.36943647187836, -5.4246020641371056, -5.4990007622828, -5.483941559578271, -5.720430197657335, -5.6588407549109405, -4.028365034871055, -3.6311631124457673, -3.2667050364746277, -3.1579528608677947, -2.7837624782995265, -2.762757803322358, -2.2250243293510614, -2.1871164550077995, -0.5931462842706945, 1.430076831016914, 2.108543660091372, 2.5138181717275634, 3.3615128903703857, 3.673145454736354, 3.986572169286583, 4.455772455940787, 4.75680415918783, 5.031971684947173, 5.3959271846846395, 8.459144233518469, 9.271496171477263, 9.625383437514849, 10.898576255284201, 11.0757706985124, 11.301794721235833, 11.865991158665313, 12.377186380784607, 12.514150999458797, 12.399423036405844, 13.018096545290447, 13.564846658219984, 13.77150665124956, 14.232667236096145, 14.604072481458854, 14.699066158920047, 14.916922717113444, 15.03186141506404, 15.078437025016981, 15.492708601623194, 15.654196789548251, 16.332453153037356, 16.820219795846107, 18.29264605615368, 18.369764300117712, 18.754173857112505, 19.28806723997716, 19.985380939879004, 20.27685745556744, 20.684850379797762, 20.845586392934344, 21.044700618073634, 21.59806717291061, 22.247513714858428, 22.49634750144193, 22.65031527343709, 22.96257023338387, 23.159325083213012, 22.99582028873045, 23.728450683427774, 23.81560747358786, 24.026531379496085, 24.444680174742672, 25.343747841056476, 25.600117617703834, 25.8789059386839, 25.8971670742447, 26.387165823337018, 26.448632602387285, 26.554785249661222, 26.9380832696376, 27.208837745616893, 27.57620030207774, 28.046107937757622, 28.339460359058805, 28.50399326048558, 28.604023370858933, 28.6324717918159, 28.5769256486708, 28.615324325900353, 28.69110337420971, 28.781845289989175, 29.333433989834493, 29.6252267646548, 29.755666914958912, 29.792073637017314, 29.854611647931648, 30.00343064614052, 30.109961026060063, 30.21386316520841, 30.361424705829325, 30.429486763311758, 30.547347499622624, 30.637181777884926, 30.38143874198807, 30.729252301455123, 30.782860606950873, 30.79524668225574, 30.863409338548223, 30.97109252987258, 31.10220385278573, 31.208407251496055, 31.257383122992653, 31.352435898326917, 31.491280470378456, 31.586503528580042, 31.613987432862192, 31.70491483589154, 31.865908824754257, 31.88668826348185, 31.89928001629769, 32.04384286159434, 32.049508815368554, 32.17387429064048, 31.970229376492572, 31.931886854781535, 32.28916847950888, 32.290435758805344, 32.318069963868886, 32.430116142514755, 32.45703632774725, 32.51639529408404, 32.52931395820921, 32.584330240288075, 32.6107529753525, 32.681697069578284, 32.725367474165395, 32.72330809407071, 32.853778773458615, 32.77967693514024, 32.84560837863038, 32.92358021488756, 32.858082156290926, 32.90704288007035, 33.00395963118322, 33.09514362054583, 33.151900205501455, 33.099285359075786, 33.12036042159061, 33.08186734106885, 32.911936134114306, 32.95941359826454, 32.919314517916256, 32.9622055521619, 33.08012330642805, 33.09303872388151, 33.000385641657, 33.18489439769876, 33.31120080276571, 33.345003972176876, 33.3825015023107, 33.31523770684425, 33.015764971196425, 33.337395244109516, 33.35161683358744, 33.426862121743824, 33.46919605086993, 33.3419877470532, 33.064548614629274, 33.474627078651764, 33.473282822755166, 33.458478246660334, 33.42101717385551, 33.44340023787309, 33.4472655498863, 33.50121202511982, 33.541073275807605, 33.52631820247266, 33.54654925390611, 33.51601799754028, 33.50119292161251, 33.57922259723802, 33.628327966943125, 33.654980591011594, 33.68779494669816, 33.78240590323604, 33.790212206541156, 33.69637462807184, 33.88252926078104, 33.874305668113436, 33.870691273417115, 33.82398443078197, 33.56432495891059, 33.67052835716481, 33.63686158879987, 33.62098339379953, 33.8398508556694, 33.771046545046104, 33.65309833845799, 33.7414339346303, 33.8438722953125, 33.82157906294234, 33.85337583184466, 33.98475754767466, 33.9379307770109, 33.91135618888816, 33.90974316502954, 33.82932802007886, 33.87284281010825, 33.72517572381998, 33.88675259446454, 33.80136595235802, 33.885575095604324, 33.92656197195345, 34.01568760316941, 33.95443369581317, 34.036076550070625, 34.02464599881116, 33.696045659081506, 34.0352416034284, 33.97231238474114, 33.99872455422412, 34.013128452225104, 34.10052861782159, 34.011696303127444, 34.16468095265423, 34.076883136064865, 34.143285113037734, 34.13014316533508, 34.0956569347323, 33.99531073491253, 34.0100915928665, 33.986018739747514, 33.85548702111725, 33.86568557939753, 33.83371945925369, 34.1310144055896, 34.15860967290703, 34.143692151924164, 34.040119155578054, 33.93890363675984, 34.046486799836366, 34.137856241406865, 34.108926581344974, 34.096499909796506, 34.24957035667104, 34.27860597156674, 34.12894011422392, 34.179203920210064, 34.217382631202774, 34.171876704348236, 34.2242895515682, 34.17212625192271, 34.04950121703739, 34.26144627927905, 34.09185084744062, 34.05226897187323, 34.01090697847145, 34.138468634528294, 34.110709846215414, 34.089690340894016, 34.04553629578889, 34.070353550807525, 34.03159421610232, 34.046214441795314, 33.90597207331651, 34.085599304469135, 34.29275240976497, 34.194137842561695, 34.19381241907263, 34.30213200809074, 34.29620920139461, 34.314510032992864, 34.31382771052281, 34.41087884506641, 34.088929781423296, 34.39094315517225, 34.120568408362864, 34.34496688783068, 34.18883680201824, 34.1423513987082, 34.170758770154585, 34.243044643222945, 33.96915466260485, 33.95874698244533, 33.98217292627791, 33.99909708132696, 33.947740072072484, 34.27037198863543, 33.969293235728856, 33.78981289473924, 34.11818693656684, 34.245646938456524, 34.327903859369734, 34.35850714939832, 34.31619035951406, 34.340691573996295, 34.33562258105834, 34.20452216722427, 34.28390863002317, 34.35147135829961, 34.330248248919226, 34.2923979317783, 34.26178851271005, 34.29209746542311, 34.22984495853022, 34.28395427449406, 34.31856177329489, 34.28024292550581, 34.29636268631571, 34.305812944125016, 34.35243237291506, 34.137808298855205, 34.17975060345674, 34.18690813125018, 34.15865199366863, 34.43792412669979, 34.34103710265756, 34.22172491107983, 34.33876011277517, 34.39781934700681, 34.33057771870555, 34.43939905416282, 34.535381120434145, 34.544700610032834, 34.48768241597165, 34.25933028948484, 34.161112355719865, 34.29879766952315, 34.13511004663802, 34.15913877552702, 34.43083461845031, 34.47384080272183, 34.47677514175213, 34.38670396823829, 34.36421804438206, 34.423954230328796, 34.41102951911757, 34.32474365276673, 34.537354869457424, 34.65914984636386, 34.634855734206845, 34.74671889475835, 34.70602213242188, 34.51413114933052, 34.3982169691691, 34.37722971166207, 34.63029194339373, 34.613179009031114, 34.71865791289749, 34.698487334828926, 34.551311580014776, 34.49621357545181, 34.55921696788451, 34.55496223657422, 34.687012579549695, 34.62018476540322, 34.50794717153921, 34.530284493481304, 34.566014636443185, 34.59447193559127, 34.67359640924621, 34.69775536828907, 34.838877389825754, 34.40897027909047, 34.85775728135784, 34.71358951148518, 34.71178314196541, 34.69504095885884, 34.7421563854541, 34.77237883544627, 34.73631360488368, 34.81290546946322, 34.82972640842515, 34.812585301727154, 34.70707555094936, 34.721735809581496, 34.70509195354608, 34.79111933715674, 34.754932693250744, 34.88066378702491, 34.863999913426255, 34.72386256382256, 34.70716377614877, 34.68156129337618, 34.94554852427794, 35.017653183690385, 34.88398386385567, 34.94918756883365, 34.82990470630264, 34.83025837996193, 35.002585369796385, 34.84012743586253, 34.86950170739566, 34.77192913936164, 34.67308348006502, 34.92192930287119, 34.95554540638607, 34.912320775142646, 34.891429649153054, 34.84056348031698, 34.908200406198766, 34.870850587659945, 34.9731706296651, 34.802617317917274, 34.54630431814289, 34.54898093454134, 34.94297563970855, 34.956652337505886, 35.01067239334966, 34.81265141756158, 34.89111639645214, 34.82604089774738, 34.71266231714342, 34.65445423092332, 34.691546680108814, 34.8010964235953, 34.26912512380084, 34.759932843071105, 34.901308207691145, 34.600717606116795, 34.87175807668037, 34.88040880763258, 34.90492710082386, 34.86992797160258, 34.64644767832063, 34.653411519515615, 34.7248319071689, 34.639095838564664, 34.96260404263327, 34.99785827202221, 34.98367613869495, 34.897978561709756, 34.9898888316455, 34.75888267381474, 34.65714462368196, 34.751571928838295, 34.867412029329245, 34.81508095695226, 34.87599242028179, 34.79977307796912, 34.91818305274023, 34.69192401003744, 34.689562458491, 34.771637297660874, 34.91244151990341, 34.88647211685505, 34.856304227904594, 34.993728431717685, 34.972966767982804, 34.84060372514765, 34.85131975438976, 34.92879213491323, 34.81672722748563, 34.914641641630475, 34.94187275497501, 34.90092099456721, 34.92171195972308, 34.91007536608075, 34.749184124432645, 34.98497935563722, 34.9385831372574, 34.942358248893484, 34.918553005926675, 35.02496757491352, 34.830920326193755, 34.56038324857662, 34.50307332236979, 34.49487176100195, 34.38375876296229 ] } }, "Isaac-Velocity-Rough-H1-v0": { "FastTD3": { "time": [ 0.0, 364.7, 729.4, 1094.1, 1458.8, 1823.5, 2188.2, 2552.9, 2917.6, 3282.3, 3647.0 ], "env_step": [ 0, 20480000.0, 40960000.0, 61440000.0, 81920000.0, 102400000.0, 122880000.0, 143360000.0, 163840000.0, 184320000.0, 204800000.0 ], "return": [ 0.0, 0.8428554534912109, 10.589672724405924, 18.404813448588055, 20.45177714029948, 22.65296808878581, 23.864246368408203, 24.028223673502605, 23.937875747680664, 24.01940409342448, 25.180285771687824 ], "return_min": [ 0.0, -2.6961081957202064, 6.380003830010861, 15.739199224866137, 19.15368035970884, 22.281677949442358, 23.391415568764668, 23.599606396166806, 23.64279778449942, 23.07866341290395, 24.741632156502256 ], "return_max": [ 0.0, 4.381819102702629, 14.799341618800987, 21.07042767230997, 21.74987392089012, 23.02425822812926, 24.33707716805174, 24.456840950838405, 24.232953710861906, 24.96014477394501, 25.618939386873393 ] }, "null": { "time": [ 0.0, 7.016, 14.032, 21.048, 28.064, 35.08, 42.096, 49.112, 56.128, 63.144, 70.16, 77.176, 84.192, 91.208, 98.224, 105.24, 112.256, 119.272, 126.288, 133.304, 140.32, 147.336, 154.352, 161.368, 168.384, 175.4, 182.416, 189.432, 196.448, 203.464, 210.48, 217.496, 224.512, 231.528, 238.544, 245.56, 252.576, 259.592, 266.608, 273.624, 280.64, 287.656, 294.672, 301.688, 308.704, 315.72, 322.736, 329.752, 336.768, 343.784, 350.8, 357.816, 364.832, 371.848, 378.864, 385.88, 392.896, 399.912, 406.928, 413.944, 420.96, 427.976, 434.992, 442.008, 449.024, 456.04, 463.056, 470.072, 477.088, 484.104, 491.12, 498.136, 505.152, 512.168, 519.184, 526.2, 533.216, 540.232, 547.248, 554.264, 561.28, 568.296, 575.312, 582.328, 589.344, 596.36, 603.376, 610.392, 617.408, 624.424, 631.44, 638.456, 645.472, 652.488, 659.504, 666.52, 673.536, 680.552, 687.568, 694.584, 701.6, 708.616, 715.632, 722.648, 729.664, 736.68, 743.696, 750.712, 757.728, 764.744, 771.76, 778.776, 785.792, 792.808, 799.824, 806.84, 813.856, 820.872, 827.888, 834.904, 841.92, 848.936, 855.952, 862.968, 869.984, 877.0, 884.016, 891.032, 898.048, 905.064, 912.08, 919.096, 926.112, 933.128, 940.144, 947.16, 954.176, 961.192, 968.208, 975.224, 982.24, 989.256, 996.272, 1003.288, 1010.304, 1017.32, 1024.336, 1031.352, 1038.368, 1045.384, 1052.4, 1059.416, 1066.432, 1073.448, 1080.464, 1087.48, 1094.496, 1101.512, 1108.528, 1115.544, 1122.56, 1129.576, 1136.592, 1143.608, 1150.624, 1157.64, 1164.656, 1171.672, 1178.688, 1185.704, 1192.72, 1199.736, 1206.752, 1213.768, 1220.784, 1227.8, 1234.816, 1241.832, 1248.848, 1255.864, 1262.88, 1269.896, 1276.912, 1283.928, 1290.944, 1297.96, 1304.976, 1311.992, 1319.008, 1326.024, 1333.04, 1340.056, 1347.072, 1354.088, 1361.104, 1368.12, 1375.136, 1382.152, 1389.168, 1396.184, 1403.2, 1410.216, 1417.232, 1424.248, 1431.264, 1438.28, 1445.296, 1452.312, 1459.328, 1466.344, 1473.36, 1480.376, 1487.392, 1494.408, 1501.424, 1508.44, 1515.456, 1522.472, 1529.488, 1536.504, 1543.52, 1550.536, 1557.552, 1564.568, 1571.584, 1578.6, 1585.616, 1592.632, 1599.648, 1606.664, 1613.68, 1620.696, 1627.712, 1634.728, 1641.744, 1648.76, 1655.776, 1662.792, 1669.808, 1676.824, 1683.84, 1690.856, 1697.872, 1704.888, 1711.904, 1718.92, 1725.936, 1732.952, 1739.968, 1746.984, 1754.0, 1761.016, 1768.032, 1775.048, 1782.064, 1789.08, 1796.096, 1803.112, 1810.128, 1817.144, 1824.16, 1831.176, 1838.192, 1845.208, 1852.224, 1859.24, 1866.256, 1873.272, 1880.288, 1887.304, 1894.32, 1901.336, 1908.352, 1915.368, 1922.384, 1929.4, 1936.416, 1943.432, 1950.448, 1957.464, 1964.48, 1971.496, 1978.512, 1985.528, 1992.544, 1999.56, 2006.576, 2013.592, 2020.608, 2027.624, 2034.64, 2041.656, 2048.672, 2055.688, 2062.704, 2069.72, 2076.736, 2083.752, 2090.768, 2097.784, 2104.8, 2111.816, 2118.832, 2125.848, 2132.864, 2139.88, 2146.896, 2153.912, 2160.928, 2167.944, 2174.96, 2181.976, 2188.992, 2196.008, 2203.024, 2210.04, 2217.056, 2224.072, 2231.088, 2238.104, 2245.12, 2252.136, 2259.152, 2266.168, 2273.184, 2280.2, 2287.216, 2294.232, 2301.248, 2308.264, 2315.28, 2322.296, 2329.312, 2336.328, 2343.344, 2350.36, 2357.376, 2364.392, 2371.408, 2378.424, 2385.44, 2392.456, 2399.472, 2406.488, 2413.504, 2420.52, 2427.536, 2434.552, 2441.568, 2448.584, 2455.6, 2462.616, 2469.632, 2476.648, 2483.664, 2490.68, 2497.696, 2504.712, 2511.728, 2518.744, 2525.76, 2532.776, 2539.792, 2546.808, 2553.824, 2560.84, 2567.856, 2574.872, 2581.888, 2588.904, 2595.92, 2602.936, 2609.952, 2616.968, 2623.984, 2631.0, 2638.016, 2645.032, 2652.048, 2659.064, 2666.08, 2673.096, 2680.112, 2687.128, 2694.144, 2701.16, 2708.176, 2715.192, 2722.208, 2729.224, 2736.24, 2743.256, 2750.272, 2757.288, 2764.304, 2771.32, 2778.336, 2785.352, 2792.368, 2799.384, 2806.4, 2813.416, 2820.432, 2827.448, 2834.464, 2841.48, 2848.496, 2855.512, 2862.528, 2869.544, 2876.56, 2883.576, 2890.592, 2897.608, 2904.624, 2911.64, 2918.656, 2925.672, 2932.688, 2939.704, 2946.72, 2953.736, 2960.752, 2967.768, 2974.784, 2981.8, 2988.816, 2995.832, 3002.848, 3009.864, 3016.88, 3023.896, 3030.912, 3037.928, 3044.944, 3051.96, 3058.976, 3065.992, 3073.008, 3080.024, 3087.04, 3094.056, 3101.072, 3108.088, 3115.104, 3122.12, 3129.136, 3136.152, 3143.168, 3150.184, 3157.2, 3164.216, 3171.232, 3178.248, 3185.264, 3192.28, 3199.296, 3206.312, 3213.328, 3220.344, 3227.36, 3234.376, 3241.392, 3248.408, 3255.424, 3262.44, 3269.456, 3276.472, 3283.488, 3290.504, 3297.52, 3304.536, 3311.552, 3318.568, 3325.584, 3332.6, 3339.616, 3346.632, 3353.648, 3360.664, 3367.68, 3374.696, 3381.712, 3388.728, 3395.744, 3402.76, 3409.776, 3416.792, 3423.808, 3430.824, 3437.84, 3444.856, 3451.872, 3458.888, 3465.904, 3472.92, 3479.936, 3486.952, 3493.968, 3500.984, 3508.0 ], "env_step": [ 0, 491520, 786432, 1179648, 2162688, 2457600, 2654208, 3342336, 3440640, 3637248, 4128768, 5505024, 5701632, 6488064, 6684672, 6782976, 6979584, 8159232, 8355840, 9535488, 10518528, 10616832, 10911744, 11108352, 11304960, 11796480, 12288000, 12386304, 12582912, 13172736, 13369344, 14155776, 14548992, 14843904, 14942208, 15728640, 15826944, 16613376, 17694720, 18677760, 18874368, 18972672, 19267584, 20054016, 20348928, 20742144, 23101440, 23199744, 23396352, 23494656, 23986176, 24576000, 25264128, 25952256, 26148864, 26640384, 26836992, 27033600, 27131904, 27328512, 27525120, 28999680, 29687808, 29982720, 30081024, 30179328, 30375936, 30965760, 31162368, 32145408, 32440320, 32735232, 32931840, 34701312, 35684352, 36175872, 37158912, 37453824, 37945344, 38338560, 38436864, 38633472, 39321600, 40009728, 40796160, 41189376, 41680896, 41877504, 42270720, 44138496, 44236800, 45318144, 45416448, 46497792, 46596096, 46694400, 47087616, 47185920, 47579136, 47775744, 47972352, 48070656, 48562176, 49152000, 49643520, 50823168, 51609600, 53477376, 53870592, 55934976, 57704448, 60260352, 60555264, 62816256, 63700992, 64290816, 64389120, 65568768, 65765376, 67534848, 68419584, 69697536, 69894144, 70189056, 72646656, 72941568, 74907648, 75202560, 76087296, 77561856, 78544896, 78839808, 79331328, 80019456, 80412672, 80805888, 81002496, 81297408, 81494016, 81592320, 81887232, 82083840, 83361792, 84738048, 84934656, 85426176, 85622784, 86310912, 86409216, 86802432, 87293952, 88276992, 88965120, 90636288, 90832896, 91029504, 91226112, 93192192, 94076928, 94568448, 95158272, 95453184, 95944704, 96239616, 96534528, 96927744, 97320960, 98009088, 98304000, 98598912, 98992128, 99287040, 99385344, 100958208, 102039552, 102629376, 103317504, 104103936, 104398848, 104497152, 104792064, 104988672, 106070016, 106463232, 106758144, 107544576, 110690304, 111476736, 111575040, 112164864, 112558080, 112852992, 113639424, 113737728, 114327552, 115212288, 115703808, 116490240, 117473280, 118259712, 118554624, 118652928, 118849536, 118947840, 119144448, 122585088, 122880000, 123469824, 124452864, 124551168, 125042688, 126025728, 126124032, 126812160, 127008768, 128679936, 128778240, 129466368, 129957888, 130056192, 130744320, 131334144, 132612096, 132907008, 133398528, 133496832, 134873088, 136642560, 136740864, 136937472, 137428992, 137625600, 139001856, 140476416, 140574720, 141754368, 142147584, 142344192, 143228928, 143622144, 143818752, 144506880, 144900096, 145195008, 147259392, 147456000, 148045824, 148733952, 149323776, 149422080, 150896640, 151388160, 152764416, 153550848, 154730496, 155516928, 156794880, 156893184, 156991488, 158957568, 159645696, 160432128, 161021952, 161316864, 161513472, 161906688, 162299904, 162791424, 163577856, 163774464, 164560896, 165150720, 165347328, 165937152, 166133760, 167706624, 168296448, 168394752, 168493056, 169476096, 169967616, 170065920, 170164224, 170262528, 170360832, 170459136, 170557440, 170754048, 172916736, 173211648, 173506560, 173604864, 175079424, 175767552, 175964160, 176160768, 176455680, 177143808, 178323456, 178716672, 179208192, 180682752, 180879360, 181174272, 183238656, 184123392, 184221696, 185106432, 185892864, 186384384, 187465728, 188645376, 189530112, 189628416, 189726720, 189923328, 190316544, 190808064, 191397888, 191987712, 192282624, 192380928, 193658880, 194445312, 194740224, 197394432, 197492736, 198082560, 198672384, 199065600, 199655424, 200245248, 201228288, 201818112, 201916416, 202309632, 202407936, 203194368, 205848576, 206438400, 207028224, 207618048, 207912960, 208109568, 208601088, 209584128, 209682432, 209879040, 210272256, 210468864, 211845120, 212041728, 212828160, 213221376, 213909504, 214302720, 214499328, 214695936, 215384064, 215482368, 215678976, 215973888, 216072192, 216170496, 216563712, 217350144, 218136576, 218333184, 218431488, 218628096, 218726400, 219021312, 220102656, 220397568, 220495872, 220790784, 221577216, 221872128, 223051776, 223346688, 224329728, 224919552, 227082240, 227475456, 228753408, 228851712, 229933056, 230326272, 230522880, 230621184, 231112704, 232488960, 232980480, 233373696, 233472000, 234160128, 234258432, 234356736, 234651648, 234848256, 236224512, 237895680, 238092288, 238288896, 238583808, 239370240, 240254976, 241434624, 242515968, 242712576, 243400704, 244088832, 244187136, 245170176, 245268480, 245465088, 245956608, 246841344, 247431168, 247529472, 247627776, 249200640, 249987072, 250085376, 251953152, 252248064, 252837888, 253526016, 253624320, 253820928, 254017536, 254509056, 256180224, 256868352, 256966656, 257949696, 258146304, 258637824, 258834432, 259227648, 259620864, 259719168, 259915776, 261980160, 262569984, 262766592, 263454720, 264339456, 265224192, 265322496, 265814016, 265912320, 266207232, 266403840, 266797056, 266895360, 267681792, 268369920, 269746176, 270041088, 270237696, 270434304, 270729216, 271220736, 271613952, 272007168, 272203776, 273186816, 273580032, 273776640, 273973248, 274464768, 275152896, 275644416, 276037632, 277118976, 278495232, 279281664, 279478272, 279674880, 280756224, 281542656, 281935872, 283213824, 283312128, 283803648, 285474816, 285573120, 286556160, 286949376, 288718848, 289505280, 289996800, 290291712, 291864576, 292061184, 292257792, 292651008, 293535744, 293732352, 294027264 ], "return": [ 0.0, -5.422170765797297, -5.365042141278585, -5.321201788584392, -5.282661674817402, -5.279570868810018, -5.099006303151449, -4.985642132759094, -5.039014817873636, -5.05856496334076, -5.127120292981465, -5.301984883944193, -5.787255485852559, -6.492295300165812, -6.859130134582519, -6.679243364930152, -6.585359064737955, -6.638607020725806, -6.56383727868398, -5.579464244066426, -4.730856805543104, -4.85022839024663, -4.08018668456624, -3.7235442843676236, -2.5684121332814294, -2.3172017341541746, -1.8826996049284936, -1.7406521982420236, -1.3593260787675778, 2.4418186768361676, 2.5145354398091633, 3.53728093676269, 3.7485122829924027, 5.24840559720993, 5.905099849700928, 6.319293590386709, 6.558245106538137, 6.420299232800801, 7.086359784603118, 7.828594415982565, 8.177236615816753, 8.169828134377797, 8.109048763116201, 7.960519586404165, 7.632579886913301, 8.5222447570165, 9.480246132214864, 9.057840726375579, 8.536546057065328, 8.322958382765451, 8.593875254789987, 9.066962721347808, 9.4220951239268, 9.672696344852447, 9.42037155866623, 9.068991316159567, 9.227477009296416, 8.801373164653777, 9.659750367005666, 9.834801010290782, 9.733246921698251, 9.725076502164205, 9.575205939610798, 9.757781678835551, 9.515549600919087, 10.32995819648107, 10.743536387284598, 10.337343724568685, 10.239077248573302, 9.3617884238561, 9.82903660694758, 10.205209770997365, 10.302206721305849, 10.8276744333903, 11.209485573768616, 10.820306483109794, 10.813732381661731, 11.153387095133462, 11.57680031220118, 11.863303298155467, 12.104669983386993, 12.226192903518674, 12.226238845984142, 12.321544722716013, 12.010425718625386, 11.959592173894245, 12.702534379164378, 12.588660320440928, 12.730267083644867, 13.09765745639801, 13.098441476821899, 13.007480210463207, 12.896890660127005, 13.57533088763555, 14.103462287584941, 14.065941929022472, 13.571870381832122, 13.765269615650178, 13.634288456439974, 13.655183283686638, 13.933596647580465, 14.153377745151522, 13.484311122894288, 13.991338650385538, 14.217809696992239, 14.372374130090078, 14.48803258895874, 13.902689254283905, 14.008463487625122, 15.318258340358733, 14.727367254892984, 15.613157649834951, 15.31281984011332, 16.40384552160899, 16.804359534581504, 16.20739706039429, 16.820533531506857, 16.80816020488739, 17.16210698922475, 17.353576091925305, 17.84310720125834, 17.64714405377706, 17.666645275751748, 17.592155150572456, 17.889433806737262, 18.332248009840647, 18.998913583755495, 18.876606470743813, 19.217209668159484, 19.138889632225034, 18.948357582092285, 19.382302405039468, 19.401501762866975, 19.0712877702713, 19.468732802073163, 19.555791912873588, 19.585630633036295, 19.76214089870453, 19.80392255862554, 19.832017685572307, 19.642031608422595, 20.049005517959596, 20.15443610191345, 20.743607517878214, 20.581281551520032, 21.05371557156245, 20.423204619089763, 20.687201137542726, 21.01858803431193, 21.266472924550374, 21.371525878906255, 21.652281880378723, 21.299677049318948, 21.71703642209371, 21.522530011336006, 21.78698669195175, 22.034059658050538, 21.718550532658895, 21.89046088218689, 21.815517278512317, 22.191909686724347, 22.510205643971762, 22.344761204719543, 22.288901374340057, 22.643796286582944, 22.330036065578458, 22.251962281068163, 22.40083763678869, 22.013201440970104, 22.518215183417, 22.72313869396845, 22.826243489583334, 22.859257675011957, 23.06479694366455, 22.927606552441915, 22.74173924446106, 22.974355036417638, 23.50650741418203, 23.338268404006957, 22.919516321818037, 23.388370493253074, 23.163635557492572, 23.164315322240196, 23.40869496822357, 23.341194401582083, 23.485880421797432, 23.504843918482464, 23.721208157539365, 23.993358154296875, 23.909098584651947, 23.956194926897684, 24.3459050989151, 24.185533266067505, 24.134012846946717, 24.43199095249176, 24.177146474520367, 24.436910311381023, 24.54009437640508, 24.16632465362549, 24.412582944234213, 24.380601506233216, 24.468062558174136, 24.40877560297648, 24.71468245188395, 24.60827238559723, 24.740209692319237, 24.53396530866623, 24.663431402842207, 24.959084293047585, 24.908331915537516, 24.947622075080872, 24.6909338927269, 24.940318908691406, 25.16483751296997, 24.60695726076762, 25.112809979120886, 25.086977144877114, 24.773433555761972, 24.926412556966145, 24.998008828957875, 25.06103755235672, 24.86278089841207, 24.97040135542552, 25.0428440785408, 25.258395306269325, 25.210753719011944, 25.279651983579, 25.36194952170054, 25.692520167032878, 25.59342387040456, 25.554213938713076, 25.520990746816, 25.31104542016983, 25.52819642861684, 25.75904115041097, 25.72125477631887, 25.69705542564392, 25.73389612515767, 25.833797233899435, 25.576571871439615, 25.42343479792277, 25.95107562383016, 26.04738268693288, 25.781439321835837, 25.82269303480784, 25.62690583546956, 25.771284255981442, 26.133092199961343, 26.049315468470255, 26.092107651233675, 25.754562322298685, 26.115973978042604, 25.99166533311208, 26.16707617441813, 25.920664300918578, 26.06877758344014, 26.161833999951682, 26.205389365355174, 26.313472315470378, 26.228369690577185, 26.16665622870127, 26.187024978796643, 26.11963609695435, 26.04467580159505, 26.249091874758403, 26.569917356967924, 26.703698792457583, 26.39212519009908, 26.452600940068564, 26.32757260640462, 26.345079108079275, 26.594597864151, 26.496056710879007, 26.45632796843847, 26.45881276925405, 26.322761637369794, 26.41735316912333, 26.37774245738983, 26.58394108136495, 26.462264091173807, 26.70522584835688, 26.743710562388102, 26.41018081347148, 26.82354289849599, 26.584163082440696, 26.812084256807964, 26.887239375114444, 26.643198828697205, 26.901577798525494, 26.89036826769511, 27.12406840324402, 27.15825298945109, 26.974571345647174, 26.76065761725108, 26.80236604690552, 26.789501201311747, 27.014291658401486, 27.125630804697675, 27.15856007575989, 27.129440813859304, 27.022346736590066, 27.000964590708417, 26.63156608104706, 27.112519771258036, 27.229807084401447, 26.898604194323223, 26.96032204310099, 27.331697724660234, 26.889267352422078, 27.212157440185546, 27.202150770823163, 27.021636052926382, 27.016969119707742, 27.01914302031199, 27.37581664721171, 27.32651935577393, 27.31468152999878, 27.005629046758017, 27.286985462506617, 27.205869766076404, 27.2348338898023, 27.08270854314168, 27.12608816782634, 27.383097707430522, 27.283648891448976, 27.06018231074015, 27.234840971628827, 27.476725266774494, 27.707652390797932, 27.227288559277852, 27.3107603931427, 27.011848467191058, 27.36976047515869, 27.391914099057516, 27.385174859364827, 27.42166522105535, 27.630024264653525, 27.380434902509055, 27.47323336760203, 27.651270182927448, 27.44848692893982, 27.483901700973508, 27.21515275557836, 27.497027911345167, 27.572442706425985, 27.560447527567543, 27.36793696244558, 27.42006613175074, 27.493619120915735, 27.689073079427086, 27.50264252583186, 27.43011033058167, 27.64782487869263, 27.20738600730896, 27.465467980702716, 27.282119348843892, 27.665611066818233, 27.81447112083435, 27.87130407969157, 27.64211650212606, 27.7080734316508, 27.621387904485065, 27.702936627070113, 27.598064007759092, 27.722659085591633, 27.64586271524429, 27.999647210439047, 27.964578876495363, 27.837010904947917, 27.638748474121098, 27.76256209055583, 28.027448720932004, 27.709779580434162, 27.66845738093058, 27.72518820444743, 27.65982426007589, 27.579961166381835, 27.47355571269989, 27.76356635570526, 27.914256649017332, 27.394945340156553, 27.68763256072998, 27.689757191340124, 27.835959854125974, 27.943186308542888, 27.47778695344925, 27.718737327257788, 27.678874394098916, 27.8374245039622, 27.850753422578176, 27.432888934612276, 27.755647220611568, 27.72031798203786, 27.529903690020245, 27.607865818341576, 27.712867733637495, 27.99885702927907, 27.869108432133988, 27.792950577735905, 27.947116510073347, 27.803222284317016, 27.81898798942566, 27.745415393511454, 27.990131341616316, 28.14864422162374, 27.884647294680278, 27.716099699338276, 27.833606998125713, 28.014787572224932, 27.519190567334494, 27.7539791226387, 27.726883935928345, 27.769253171284998, 28.20607573827108, 27.807977307637533, 27.813251864115397, 27.978915383021036, 27.882311402956645, 28.148458941777548, 27.912579143047335, 27.93072050094604, 27.767308457692465, 28.174651209513346, 28.172889951070147, 28.343097937901813, 28.38390079498291, 28.22494049390157, 28.322374693552657, 28.16988698323568, 28.241848163604732, 28.278563607533773, 27.97595597028732, 28.320658842722576, 28.24419835249583, 28.260422398249307, 28.164151094754533, 28.21596943537394, 28.46122231165568, 28.27712041378021, 28.590335140228273, 28.311755402882895, 28.67214571317037, 28.50228882789612, 28.595824553171795, 28.50311995824178, 28.401127497355144, 28.80892203013102, 28.64995643933614, 28.63968591054281, 28.514695514043172, 28.670809841156004, 28.587760623296102, 28.8155946191152, 28.869128182729085, 28.65603721936544, 28.823503991762795, 28.95586130619049, 28.869864171346027, 28.76503180821737, 29.12373401006063, 28.938447481791183, 28.85428058942159, 28.565257782936097, 28.63410388787587, 28.925285924275716, 29.037555669148762, 28.76260529677073, 28.680053356488543, 28.91645269393921, 28.756759365399677, 28.825944765408835, 28.823520730336508, 28.880934257507324, 28.690467605590822, 28.81139595031738, 28.592766869862874, 28.87614825725555, 28.741094710032144, 29.01044699350993, 29.127510652542117, 28.686315304438278, 28.4821040948232, 28.787544911702472, 28.696746519406634, 28.59993433316549, 28.86362829049428, 28.638428608576458, 28.810294100443524, 28.65783554712931, 28.748908634185792, 28.889724987347922, 28.775232067108153, 28.812776306470237, 28.68773955980937, 28.682406692504884, 28.710577583312986, 28.798399791717532, 28.751555989583334, 28.43867567062378, 28.622998971939086, 28.5418660291036 ], "return_min": [ 0.0, -5.520963165181699, -5.398309007261076, -5.350411727864705, -5.300563709063288, -5.373167151286887, -5.244743047488927, -5.018510029574243, -5.23146215874125, -5.253785732489721, -5.446553555244509, -5.580799970293974, -6.26151833649266, -6.979740932840051, -7.505812459772718, -7.092135843312693, -7.39920407445033, -7.7452636222094355, -7.9050134285622855, -6.48987129549158, -5.654353417920056, -5.815269690086543, -4.9033229608684366, -4.364460007935154, -2.7990416494641397, -2.9495417390998684, -2.1731720117375652, -1.8570615821223686, -1.683727363340683, 1.3986503374892052, 1.578396193379838, 3.2064730591978994, 3.58337034294805, 4.303947198498285, 4.23811368968946, 4.844166448777771, 5.027521112944505, 5.054222594896141, 6.455040957762881, 6.603025119237945, 7.240161268213682, 7.445770366101291, 7.321740412287392, 6.651607739669598, 5.521256497526231, 6.277467043395728, 8.574633125834227, 7.974521417804013, 7.55285509943894, 7.299916074228484, 7.504092070112076, 7.389169200708295, 7.971362274313357, 8.948853512556664, 9.05238518322198, 8.173627072810628, 8.597720236513965, 8.279687455077365, 8.446574822859874, 8.99332873454503, 9.243581089298994, 8.82941824395879, 8.975877725575135, 9.256436597948978, 8.90494565678453, 9.57158849729407, 10.188809105616095, 9.77360360644471, 9.337660824915222, 8.051044333340174, 8.864180010138652, 8.744804824430856, 8.49640108291722, 10.067010098473812, 10.014871957505976, 9.64823068915522, 9.15242180940713, 10.082828645171604, 10.776856899541066, 10.577554678856938, 10.922870535973447, 10.903899618788715, 11.110146302869207, 11.234899446013943, 10.664980188331834, 10.61023834917109, 10.977294363326532, 11.188408920116812, 11.418770133334487, 11.80776633603974, 11.658676561647457, 11.498703604312505, 10.969441397737207, 12.203737145900309, 12.891263068433968, 12.743336304240811, 12.151705716844223, 12.767198778851697, 12.792299101983268, 12.17240064854574, 12.148095266100462, 12.432768169811734, 11.365202841854662, 12.16684822910994, 12.495021032412778, 13.26835609727346, 13.050785858650338, 12.281424819974456, 12.449084199784531, 13.625030204012711, 13.21897266304786, 14.48782746105462, 14.103399327134131, 15.519930829855001, 15.97552373557178, 15.529220441611425, 15.74885704151496, 15.016924988239797, 15.989010107035773, 16.442932816788804, 16.99508371654349, 16.53486178808719, 16.706773643149774, 16.661555294093077, 17.555512654693615, 18.112669203895777, 18.17118170021899, 18.065946631978374, 18.42934931637301, 18.676209657546792, 17.903826141385156, 18.6034161292777, 18.48889474575812, 17.758579913985656, 18.625405819750064, 19.23532065084414, 19.04211715145168, 18.877847871623374, 19.14924260166623, 18.998339729549528, 18.571789285071276, 19.417446402769226, 19.584450623471515, 20.181799769130308, 20.17112513383781, 20.51215091710531, 19.469530447820137, 20.424742202150163, 20.575052829757396, 20.784339971244243, 20.79814880451078, 21.561650860339824, 20.769088555281552, 21.583207375785705, 21.41970488153784, 21.66693042236608, 21.86798628928739, 21.316749097027397, 21.436188053015243, 21.134490993196177, 21.871763789661642, 22.315875755910508, 22.09950101356021, 22.12789918481529, 22.271840277606675, 22.060546728262114, 21.934839827743634, 22.100503142963884, 21.245196426043726, 21.869711692631675, 22.132582526880245, 22.46149806767977, 22.481397577260815, 22.770352157659595, 22.421027680957025, 22.252393067221245, 22.333097419765267, 22.991255548588548, 22.674374393471915, 22.535454298444176, 23.123628328353366, 22.69643708364719, 22.5819198826207, 22.94708603291372, 23.07638785760627, 22.74560562166574, 23.06932107625478, 23.300721860075413, 23.78194955286657, 23.523293205955508, 23.525998829282933, 23.76377698261631, 23.62537850586962, 23.864635305032632, 23.875682383296933, 23.46097954015071, 23.82543754032398, 23.761480614083627, 23.255379816835923, 23.54560044868967, 23.668944427175553, 23.832584937777096, 23.425595245832536, 24.05916452843633, 23.455597327213653, 24.038986145333908, 23.56369203440597, 24.171526792323046, 24.545470005081768, 24.484432222472666, 24.334504603248885, 23.39817352631335, 24.033054551448778, 24.482760384790556, 23.846917945112985, 24.390506037773623, 24.27431765022773, 24.061599451017777, 24.33728896947999, 24.144899762326954, 23.99540815574204, 23.57496830841703, 23.930048473254217, 23.94523741034942, 24.258384314268703, 24.37110129883553, 24.412283448059235, 24.324307472463204, 24.5954491279622, 24.899447987634463, 24.558637573616835, 24.818269162062435, 24.08092939132887, 24.275233238542537, 24.93987956500964, 24.822554121729205, 25.0395148262323, 25.136228336368635, 24.872140743082323, 24.757863129395144, 24.30237863460298, 24.730835786257032, 24.91673874558004, 25.09881997330028, 25.11666242969545, 24.928856844013023, 25.00190933529314, 24.950489014452607, 25.148173576425023, 24.952507670890725, 24.841307217586042, 25.71003983841862, 25.26704538169371, 25.224044727235064, 25.220263130793953, 25.21835000241302, 25.290933540632018, 25.181038007064192, 25.301149522880877, 25.62093761422375, 25.432852276965008, 25.342523079209098, 25.269339675082783, 24.949805414134993, 25.233401834020132, 25.576083550231427, 26.339662357104462, 25.792150727045904, 25.70682197517077, 25.807590150340896, 25.894729706751228, 25.837302977147562, 25.78135554745692, 25.68552533534089, 25.954483650340283, 25.94591635682661, 25.97817717944112, 25.75997075579898, 26.297399027884342, 25.80107864935811, 25.989724615648488, 25.984221681740333, 26.06569266026917, 26.481365883267824, 26.164847513213495, 26.603717323908448, 26.35348614473745, 26.30406394025493, 26.35116010418455, 26.23870062654923, 26.471481913277074, 26.72865827913775, 26.456025977567716, 26.509809123334087, 26.135543174587916, 26.414447583473727, 26.847739914839327, 26.986797064210627, 26.619696131921607, 26.740811445864274, 26.601182567396776, 26.63685962355374, 26.202893309498105, 26.53260006050866, 26.81746777977994, 26.35970819265938, 26.873575006909427, 26.999221390449513, 26.41511857693682, 26.750161047320116, 26.847929553691117, 26.322235246867464, 26.229498529666962, 26.46977801746138, 26.892038673702228, 26.58452566559043, 26.72107694201021, 26.389049676052693, 27.013425431358826, 26.902429688294113, 26.802812226360995, 26.385575632648816, 26.369492487494366, 26.920060112200925, 26.998724796791922, 26.68365616862227, 26.77927573299712, 27.14768838044163, 27.466859888550175, 26.89888082937872, 27.12633609842479, 26.714824263923475, 27.08128409218404, 26.98816964606872, 26.923163448198604, 26.80857466695888, 27.13636985361538, 26.978532109223686, 27.299038517001165, 27.52869762405963, 27.23234788711246, 27.250432316649512, 26.60946317727994, 27.10714554669029, 27.46608859685352, 27.26461963307082, 27.069754957120214, 27.31017870242129, 27.065506717686876, 27.195062973283548, 27.047814098820695, 27.272618294455032, 27.346293947512688, 26.68929354412294, 26.833985415803223, 26.550998418048945, 26.988439648514994, 27.333751302384293, 27.418736622351446, 26.98571833055701, 27.385679063968748, 27.334479780511817, 27.53131813266607, 27.09134479874994, 27.455784867981503, 27.108790568073573, 27.7333053434198, 27.735471393711826, 27.46263765149636, 26.99981107374089, 27.086095394451846, 27.660553475563162, 27.47148725177635, 27.462223492396287, 27.47444853382932, 27.28271224520508, 27.030280701112197, 26.953127198486797, 27.07038773963107, 27.40335757841962, 26.828224693044927, 27.455776077350375, 27.459088288760473, 27.55987625989684, 27.667771051731005, 27.306320572769852, 27.42247721506064, 27.363389568960162, 27.2417683309204, 27.16002617417197, 26.879341156026854, 27.409533505672982, 27.258284186429318, 27.13185402158389, 27.22422988865852, 27.107298046733188, 27.540015302529557, 27.252735638909673, 26.872490744618894, 27.202678057764224, 27.40268095166156, 27.28055555351517, 27.170128347919047, 27.69789697407816, 27.642380106133988, 27.332501783443227, 27.10840513223893, 27.31736408059001, 27.46624924375133, 26.64193874196349, 27.50888790519094, 27.229939764753325, 26.98551956035765, 27.744788462148396, 27.149869593429152, 27.484981796277946, 27.606668302204987, 27.255173815148787, 27.456134913716546, 27.099531472422207, 27.393945264619013, 27.25108337760589, 27.76944243898332, 27.733599140160784, 27.84646082153513, 27.933202163430472, 27.73045705324124, 28.041684326913252, 27.692758796177635, 27.746313479577946, 27.628974429062982, 27.369484454792406, 27.606020764457973, 27.75161067029577, 27.70126964798436, 27.649431270243365, 27.454127809238894, 27.842896762811133, 27.544721561443446, 27.835889016851528, 27.298496196673447, 28.035712672520734, 27.992795921610345, 28.12532764605974, 27.78349937488175, 27.672714149185374, 28.198158361700223, 28.114355241871845, 28.23306676740723, 28.19400077454502, 28.325819368964144, 28.409198407665194, 28.661559949465364, 28.534057885334462, 28.220808826082358, 28.480647238129848, 28.816288333143056, 28.823603017711733, 28.72982788893736, 28.891935847236173, 28.686997229285602, 28.75076735412921, 28.415471057558182, 28.44960991011972, 28.623370633354085, 28.8170532416532, 28.496819790952056, 28.313081375883748, 28.761926257981262, 28.570715940030333, 28.71350487001399, 28.64412790707295, 28.68453951723116, 28.415278606195162, 28.421916550091932, 28.176255415956366, 28.62514978950491, 28.498932264171632, 28.8257088770386, 29.01524324193152, 28.559692033382177, 28.30321338178444, 28.491747264076793, 28.542804594692967, 28.22614934251985, 28.106067120710527, 28.232353672890493, 28.643067026302873, 28.433486131362784, 28.47990484587543, 28.390362508890856, 28.348291888933517, 28.412395453613676, 28.177133127616443, 28.24077424802352, 28.172266605923447, 28.225951816403256, 28.400378623201224, 28.023487501372532, 28.417555112851705, 28.373084049809865 ], "return_max": [ 0.0, -5.3233783664128955, -5.331775275296094, -5.291991849304079, -5.264759640571516, -5.18597458633315, -4.953269558813971, -4.952774235943945, -4.8465674770060225, -4.863344194191798, -4.807687030718422, -5.023169797594413, -5.312992635212458, -6.004849667491572, -6.212447809392319, -6.266350886547611, -5.77151405502558, -5.531950419242177, -5.222661128805675, -4.669057192641273, -3.807360193166153, -3.8851870904067165, -3.2570504082640426, -3.082628560800093, -2.337782617098719, -1.6848617292084809, -1.5922271981194223, -1.6242428143616785, -1.0349247941944726, 3.48498701618313, 3.4506746862384885, 3.8680888143274808, 3.9136542230367555, 6.192863995921575, 7.572086009712397, 7.794420731995648, 8.088969100131768, 7.786375870705461, 7.717678611443356, 9.054163712727185, 9.114311963419823, 8.893885902654302, 8.89635711394501, 9.269431433138731, 9.74390327630037, 10.767022470637272, 10.385859138595501, 10.141160034947145, 9.520237014691716, 9.346000691302418, 9.683658439467898, 10.744756241987321, 10.872827973540243, 10.396539177148231, 9.788357934110481, 9.964355559508505, 9.857233782078868, 9.323058874230188, 10.872925911151457, 10.676273286036535, 10.222912754097509, 10.62073476036962, 10.174534153646462, 10.259126759722124, 10.126153545053644, 11.088327895668069, 11.298263668953101, 10.90108384269266, 11.140493672231383, 10.672532514372024, 10.79389320375651, 11.665614717563875, 12.108012359694477, 11.588338768306787, 12.404099190031255, 11.992382277064369, 12.475042953916333, 12.22394554509532, 12.376743724861296, 13.149051917453995, 13.286469430800539, 13.548486188248633, 13.342331389099076, 13.408189999418083, 13.35587124891894, 13.308945998617402, 14.427774395002224, 13.988911720765044, 14.041764033955246, 14.387548576756283, 14.53820639199634, 14.51625681661391, 14.824339922516803, 14.946924629370791, 15.315661506735914, 15.388547553804132, 14.992035046820021, 14.76334045244866, 14.47627781089668, 15.137965918827536, 15.719098029060468, 15.87398732049131, 15.603419403933914, 15.815829071661136, 15.9405983615717, 15.476392162906697, 15.925279319267142, 15.523953688593354, 15.567842775465712, 17.011486476704757, 16.23576184673811, 16.73848783861528, 16.52224035309251, 17.28776021336298, 17.633195333591228, 16.885573679177153, 17.892210021498755, 18.599395421534982, 18.335203871413725, 18.264219367061806, 18.691130685973192, 18.759426319466925, 18.62651690835372, 18.522755007051835, 18.22335495878091, 18.551826815785518, 19.826645467292, 19.68726630950925, 20.005070019945958, 19.601569606903276, 19.992889022799414, 20.161188680801235, 20.31410877997583, 20.383995626556946, 20.31205978439626, 19.876263174903034, 20.12914411462091, 20.646433925785686, 20.45860251558485, 20.665695641595086, 20.712273931773915, 20.680564633149967, 20.724421580355386, 21.30541526662612, 20.991437969202256, 21.59528022601959, 21.376878790359388, 20.949660072935288, 21.46212323886646, 21.748605877856505, 21.944902953301728, 21.742912900417622, 21.830265543356344, 21.850865468401715, 21.625355141134172, 21.90704296153742, 22.200133026813685, 22.120351968290393, 22.344733711358536, 22.496543563828457, 22.512055583787053, 22.704535532033017, 22.590021395878875, 22.449903563864822, 23.015752295559214, 22.599525402894802, 22.569084734392693, 22.701172130613497, 22.78120645589648, 23.166718674202322, 23.313694861056657, 23.190988911486897, 23.2371177727631, 23.359241729669506, 23.434185423926806, 23.23108542170087, 23.61561265307001, 24.02175927977551, 24.002162414542, 23.303578345191898, 23.653112658152782, 23.630834031337955, 23.746710761859692, 23.870303903533422, 23.606000945557895, 24.226155221929126, 23.94036676071015, 24.141694455003318, 24.204766755727178, 24.294903963348386, 24.386391024512434, 24.92803321521389, 24.74568802626539, 24.403390388860803, 24.988299521686585, 24.893313408890023, 25.048383082438068, 25.31870813872653, 25.07726949041506, 25.279565439778757, 25.092258585290878, 25.103540178571176, 25.391955960120423, 25.370200375331567, 25.760947443980804, 25.441433239304565, 25.504238582926487, 25.155336013361367, 25.372698581013402, 25.332231608602367, 25.56073954691286, 25.98369425914045, 25.847583265934034, 25.846914641149386, 25.366996576422252, 25.83511392046815, 25.8996366395265, 25.485267660506167, 25.5155361444523, 25.851117895588796, 26.126666948971398, 26.150593488407107, 26.01075423759682, 26.140450746732178, 26.258406298269946, 26.050406139188357, 26.147020519098767, 26.39959157093788, 26.789591206103555, 26.287399753174657, 26.549790303809317, 26.223712331569565, 26.54116144901079, 26.781159618691145, 26.578202735812297, 26.619955430908533, 26.354596025055542, 26.331563913946706, 26.79545372471655, 26.395280613484086, 26.54449096124256, 27.17131546140329, 27.17802662828572, 26.464058670371394, 26.528723639920234, 26.3249548269261, 26.540659176669745, 27.31569538547008, 26.950457360515486, 27.231707631576626, 26.66781742701133, 26.521908117666587, 26.716285284530453, 27.1101076216012, 26.621065471043202, 26.919205164467265, 27.032734459271346, 27.229740723646156, 27.32579510805988, 26.835801766930622, 26.900460180437534, 27.031526878384188, 26.969932518825914, 27.13954618905511, 27.264781915496673, 27.56375116370442, 27.067735227810704, 26.992099653152255, 27.19837990496636, 26.84755506246834, 26.795428509407323, 27.35189275115444, 27.210757874301095, 27.227130601536047, 26.963141888167815, 26.699606917912977, 26.856529158805543, 26.995514158980676, 26.87048313484556, 27.123449532989504, 27.42072708106527, 27.50319944303587, 26.75466896667379, 27.165719913724157, 27.003478651667898, 27.02045118970748, 27.42099260549144, 26.98233371713948, 27.451995492866438, 27.542035908840987, 27.776654893210967, 27.587847699764428, 27.493116713726632, 27.011506111168075, 27.46918891922312, 27.164554819149767, 27.180843401963646, 27.264464545184723, 27.697424019598174, 27.518070181854334, 27.443510905783356, 27.365069557863094, 27.06023885259601, 27.692439482007412, 27.642146389022955, 27.437500195987067, 27.047069079292555, 27.664174058870955, 27.363416127907335, 27.674153833050976, 27.55637198795521, 27.7210368589853, 27.80443970974852, 27.5685080231626, 27.859594620721193, 28.068513045957427, 27.90828611798735, 27.62220841746334, 27.56054549365441, 27.509309843858695, 27.666855553243604, 27.779841453634546, 27.88268384815831, 27.84613530266012, 27.56857298610603, 27.436708452858028, 27.690406210260534, 27.80576215310736, 27.94844489304569, 27.555696289176986, 27.49518468786061, 27.30887267045864, 27.65823685813334, 27.795658552046312, 27.84718627053105, 28.03475577515182, 28.123678675691668, 27.782337695794425, 27.647428218202897, 27.773842741795264, 27.664625970767176, 27.717371085297504, 27.820842333876776, 27.886910276000044, 27.67879681599845, 27.856275422064268, 27.666118967770945, 27.52995356108019, 27.921731524144594, 28.183083185570624, 27.957470952843025, 27.587602366708307, 27.949355809872575, 27.725478470494977, 28.09695054560221, 28.01324027963884, 28.34278248512147, 28.29519093928441, 28.323871537031692, 28.298514673695113, 28.03046779933285, 27.908296028458313, 27.874555121474156, 28.104783216768244, 27.989533303201764, 28.18293486241501, 28.265989077458293, 28.1936863592789, 28.211384158399472, 28.277685874501305, 28.43902878665981, 28.394343966300845, 27.948071909091976, 27.874691269464872, 27.97592787506554, 28.0369362749467, 28.129641631651474, 27.993984226912985, 28.456744971779454, 28.425155719615045, 27.96166598726818, 27.919489044109586, 27.920426093919776, 28.11204344835511, 28.21860156535477, 27.649253334128648, 28.014997439454937, 27.99435921923767, 28.433080677004, 28.54148067098438, 27.986436713197698, 28.101760935550153, 28.1823517776464, 27.927953358456598, 27.991501748024632, 28.3184374205418, 28.45769875602858, 28.485481225358303, 28.713410410852916, 28.69155496238247, 28.203763616972473, 28.35742042533615, 28.32070243910386, 28.282365709154472, 28.654908337113493, 28.436792805917328, 28.323794266437623, 28.349849915661416, 28.563325900698533, 28.3964423927055, 27.999070340086462, 28.223828107103365, 28.552986782212347, 28.66736301439376, 28.466085021845913, 28.14152193195285, 28.351162463837085, 28.509448990764504, 28.84078296983855, 28.725626813672463, 28.46749573727307, 28.28353353777904, 28.579859980043373, 28.61218076197951, 28.8397350542685, 28.83459942653535, 28.719423934561902, 28.603065060192062, 28.647015170293727, 28.73738284763152, 28.928152786004564, 28.582427485782233, 29.03529692098718, 28.73678603469589, 28.819575148514254, 28.6788709192657, 28.977811061508984, 29.079547860500224, 29.009519266116975, 29.344781263605018, 29.325014609092342, 29.308578753820004, 29.011781734181895, 29.06632146028385, 29.22274054160181, 29.129540845524915, 29.419685698561814, 29.185557636800436, 29.046305053678388, 28.835390253541323, 29.015800313347864, 28.76632283892701, 28.969629288765034, 29.204198480123708, 29.091265612648524, 29.166360745395743, 29.09543427923792, 28.91612532498032, 28.800235727497377, 29.355532172885088, 29.189897734296764, 28.957793824713974, 28.715044508314012, 28.81859786563202, 29.227201215197347, 29.258058096644326, 29.028390802589403, 29.047025337093338, 29.070979129897157, 28.942802790769022, 28.93838466080368, 29.002913553600067, 29.07732899778349, 28.96565660498648, 29.200875350542827, 29.009278323769383, 29.12714672500619, 28.983257155892655, 29.19518510998126, 29.239778063152716, 28.81293857549438, 28.66099480786196, 29.083342559328152, 28.8506884441203, 28.973719323811128, 29.621189460278032, 29.044503544262422, 28.977521174584176, 28.882184962895835, 29.017912422496153, 29.389087465804987, 29.20217224528279, 29.213157159326798, 29.198345992002295, 29.124039136986248, 29.248888560702525, 29.370847767031808, 29.102733355965444, 28.85386383987503, 28.828442831026468, 28.71064800839733 ] } } } ================================================ FILE: data/playground_result.json ================================================ { "G1JoystickFlatTerrain": { "PPO": { "time": [ 0.0, 213.46350627565, 426.9270125513, 640.3905188269499, 853.8540251026, 1067.31753137825, 1280.7810376538998, 1494.2445439295498, 1707.7080502052, 1921.1715564808499, 2134.6350627565, 2348.0985690321495, 2561.5620753077997, 2775.02558158345, 2988.4890878590995, 3201.9525941347497, 3415.4161004104, 3628.8796066860496, 3842.3431129616997, 4055.8066192373494, 4269.270125513 ], "env_step": [ 0, 0.0, 10649600.0, 21299200.0, 31948800.0, 42598400.0, 53248000.0, 63897600.0, 74547200.0, 85196800.0, 95846400.0, 106496000.0, 117145600.0, 127795200.0, 138444800.0, 149094400.0, 159744000.0, 170393600.0, 181043200.0, 191692800.0, 202342400.0 ], "return": [ 0.0, -3.5786768595377603, -1.73488183816274, -1.4672497908274333, -1.0114571849505107, 2.8664944966634116, 8.109216849009195, 12.15112050374349, 14.619683265686035, 16.250600814819336, 17.415241877237957, 19.337273279825848, 19.362972259521484, 20.73096466064453, 20.419541041056316, 20.557310740152996, 21.584625244140625, 22.650484720865887, 23.268614451090496, 23.964858373006184, 23.381336212158203 ], "return_min": [ 0.0, -3.633624033793396, -1.8136003478544724, -1.5840279132431287, -1.1807665772933178, 1.9221867482777824, 7.587271126532166, 10.834839595162432, 13.544779870637864, 15.040593747413778, 16.337773770375044, 18.42818295768993, 18.613121548035885, 19.99613895549954, 18.939780920828404, 19.84353432250872, 20.260494181371044, 22.000288411591143, 22.748610923303467, 22.875291623275718, 22.801813358049884 ], "return_max": [ 0.0, -3.5237296852821247, -1.6561633284710076, -1.350471668411738, -0.8421477926077037, 3.810802245049041, 8.631162571486225, 13.467401412324548, 15.694586660734206, 17.460607882224895, 18.49270998410087, 20.246363601961765, 20.112822971007084, 21.465790365789523, 21.899301161284228, 21.27108715779727, 22.908756306910206, 23.30068103014063, 23.788617978877525, 25.05442512273665, 23.960859066266522 ] }, "FastTD3": { "time": [ 0.0, 433.3400245197778, 866.6800490395556, 1300.0200735593332, 1733.3600980791111, 2166.700122598889, 2600.0401471186665, 3033.3801716384446, 3466.7201961582223, 3900.060220678 ], "env_step": [ 0, 5120000.0, 10240000.0, 15360000.0, 20480000.0, 25600000.0, 30720000.0, 35840000.0, 40960000.0, 46080000.0 ], "return": [ 0.0, 4.487485567728679, 16.74320411682129, 23.278841654459637, 27.637049357096355, 29.066091537475586, 30.403479894002277, 30.677172978719074, 30.934303283691406, 31.36344337463379 ], "return_min": [ 0.0, 2.9314155707645613, 16.25554819781311, 22.555713738285238, 27.491040766131707, 28.865730935672705, 30.389128000549142, 30.534967621827455, 30.650468173578087, 31.16558190517613 ], "return_max": [ 0.0, 6.043555564692796, 17.23086003582947, 24.001969570634035, 27.783057948061003, 29.266452139278467, 30.417831787455412, 30.819378335610693, 31.218138393804725, 31.56130484409145 ] } }, "G1JoystickRoughTerrain": { "PPO": { "time": [ 0.0, 430.02984852145, 860.0596970429, 1290.08954556435, 1720.1193940858, 2150.14924260725, 2580.1790911287, 3010.2089396501497, 3440.2387881716, 3870.26863669305, 4300.2984852145, 4730.32833373595, 5160.3581822574, 5590.38803077885, 6020.417879300299, 6450.44772782175, 6880.4775763432, 7310.507424864651, 7740.5372733861, 8170.56712190755, 8600.596970429 ], "env_step": [ 0, 0.0, 10649600.0, 21299200.0, 31948800.0, 42598400.0, 53248000.0, 63897600.0, 74547200.0, 85196800.0, 95846400.0, 106496000.0, 117145600.0, 127795200.0, 138444800.0, 149094400.0, 159744000.0, 170393600.0, 181043200.0, 191692800.0, 202342400.0 ], "return": [ 0.0, -3.6238108476003013, -1.7691378990809123, -1.5785710016886394, -1.4046337604522705, -1.2989702622095745, 0.9209129015604655, 5.599383354187012, 9.098564147949219, 10.36640707651774, 11.69961961110433, 12.815311431884766, 13.983243624369303, 14.600811004638672, 15.401162465413412, 16.72524897257487, 17.59372901916504, 17.17877769470215, 18.038379033406574, 19.438427607218426, 19.256649017333984 ], "return_min": [ 0.0, -3.6888348904174544, -1.8035371110148877, -1.6518320374166224, -1.4434835026307684, -1.3658170492283774, -0.3835055360183316, 3.5933339888782276, 8.425358069779085, 9.000362013173968, 10.886633777692076, 12.64601019994388, 13.083222241676461, 14.4930801524455, 14.552314554875068, 15.949481202226977, 16.67476213765229, 16.62720103040347, 17.008479664699678, 18.59841010178635, 19.188480973348607 ], "return_max": [ 0.0, -3.558786804783148, -1.734738687146937, -1.5053099659606564, -1.3657840182737726, -1.2321234751907715, 2.2253313391392626, 7.605432719495796, 9.771770226119353, 11.732452139861513, 12.512605444516584, 12.98461266382565, 14.883265007062144, 14.708541856831843, 16.250010375951756, 17.501016742922765, 18.512695900677787, 17.730354359000827, 19.06827840211347, 20.278445112650502, 19.32481706131936 ] }, "FastTD3": { "time": [ 0.0, 599.3992147375556, 1198.798429475111, 1798.1976442126668, 2397.596858950222, 2996.996073687778, 3596.3952884253335, 4195.794503162889, 4795.193717900444, 5394.592932638 ], "env_step": [ 0, 5120000.0, 10240000.0, 15360000.0, 20480000.0, 25600000.0, 30720000.0, 35840000.0, 40960000.0, 46080000.0 ], "return": [ 0.0, 3.2808993657430015, 13.874738057454428, 20.431355794270832, 24.597570419311523, 26.355728149414062, 27.401456197102863, 27.81001917521159, 28.1074701944987, 27.748165130615234 ], "return_min": [ 0.0, 2.8048553516036847, 13.351663266422506, 19.902439898270348, 24.322450701046584, 25.83280810079747, 27.064759880116778, 27.480682414578805, 28.029204257881428, 27.266035552178657 ], "return_max": [ 0.0, 3.756943379882318, 14.39781284848635, 20.960271690271316, 24.872690137576463, 26.878648198030653, 27.73815251408895, 28.139355935844375, 28.18573613111597, 28.23029470905181 ] } }, "T1JoystickFlatTerrain": { "PPO": { "time": [ 0.0, 156.0373824072, 312.0747648144, 468.1121472216, 624.1495296288, 780.186912036, 936.2242944432, 1092.2616768504, 1248.2990592576, 1404.3364416647998, 1560.373824072, 1716.4112064792, 1872.4485888864, 2028.4859712936, 2184.5233537008, 2340.5607361079997, 2496.5981185152, 2652.6355009224, 2808.6728833295997, 2964.7102657368, 3120.747648144 ], "env_step": [ 0, 0.0, 10649600.0, 21299200.0, 31948800.0, 42598400.0, 53248000.0, 63897600.0, 74547200.0, 85196800.0, 95846400.0, 106496000.0, 117145600.0, 127795200.0, 138444800.0, 149094400.0, 159744000.0, 170393600.0, 181043200.0, 191692800.0, 202342400.0 ], "return": [ 0.0, 0.327275812625885, 0.8326747417449951, 2.144671678543091, 9.289753913879395, 14.297985553741455, 18.105278968811035, 20.841957092285156, 25.58073329925537, 28.61104106903076, 29.105724334716797, 31.17124652862549, 31.661392211914062, 33.493133544921875, 31.919815063476562, 34.29528045654297, 34.032718658447266, 33.67773628234863, 35.22429275512695, 35.6923885345459, 34.44869613647461 ], "return_min": [ 0.0, 0.2993045151233673, 0.763859748840332, 1.678368091583252, 8.580310821533203, 14.120144844055176, 17.573272705078125, 18.319969177246094, 23.739049911499023, 26.93745994567871, 28.4345703125, 30.444316864013672, 29.525779724121094, 32.53502655029297, 31.550701141357422, 33.862396240234375, 33.20738220214844, 33.50837326049805, 34.951812744140625, 35.534149169921875, 33.58796691894531 ], "return_max": [ 0.0, 0.3552471101284027, 0.9014897346496582, 2.6109752655029297, 9.999197006225586, 14.475826263427734, 18.637285232543945, 23.36394500732422, 27.42241668701172, 30.284622192382812, 29.776878356933594, 31.898176193237305, 33.79700469970703, 34.45124053955078, 32.2889289855957, 34.72816467285156, 34.858055114746094, 33.84709930419922, 35.49677276611328, 35.85062789916992, 35.309425354003906 ] }, "FastTD3": { "time": [ 0.0, 336.3940926118889, 672.7881852237778, 1009.1822778356667, 1345.5763704475555, 1681.9704630594445, 2018.3645556713334, 2354.758648283222, 2691.152740895111, 3027.546833507 ], "env_step": [ 0, 5120000.0, 10240000.0, 15360000.0, 20480000.0, 25600000.0, 30720000.0, 35840000.0, 40960000.0, 46080000.0 ], "return": [ 0.0, 17.497735182444256, 23.35838508605957, 29.219560623168945, 32.47132873535156, 33.60111618041992, 34.1003163655599, 34.494296391805015, 34.647412618001304, 34.80020268758138 ], "return_min": [ 0.0, 14.391559420284299, 20.08936997440851, 27.716625485294703, 29.593702042397194, 30.287767241900532, 30.65928257092611, 31.33219802426975, 31.594485160912818, 31.565288339653822 ], "return_max": [ 0.0, 20.603910944604213, 26.62740019771063, 30.722495761043188, 35.34895542830593, 36.91446511893931, 37.54135016019369, 37.65639475934028, 37.70034007508979, 38.03511703550894 ] } }, "T1JoystickRoughTerrain": { "PPO": { "time": [ 0.0, 224.59065696369998, 449.18131392739997, 673.7719708911, 898.3626278547999, 1122.9532848185, 1347.5439417822, 1572.1345987459, 1796.7252557095999, 2021.3159126733, 2245.906569637, 2470.4972266007, 2695.0878835644, 2919.6785405281, 3144.2691974918, 3368.8598544554998, 3593.4505114191998, 3818.0411683828997, 4042.6318253466, 4267.2224823103, 4491.813139274 ], "env_step": [ 0, 0.0, 10649600.0, 21299200.0, 31948800.0, 42598400.0, 53248000.0, 63897600.0, 74547200.0, 85196800.0, 95846400.0, 106496000.0, 117145600.0, 127795200.0, 138444800.0, 149094400.0, 159744000.0, 170393600.0, 181043200.0, 191692800.0, 202342400.0 ], "return": [ 0.0, 0.35334428151448566, 0.8989856839179993, 1.4808813730875652, 2.67240309715271, 4.544658501942952, 5.027948538462321, 6.222784360249837, 7.626932621002197, 8.986979166666666, 9.412737210591635, 12.002973874409994, 12.44714609781901, 14.011113484700521, 15.020503044128418, 16.263832092285156, 17.344195048014324, 17.939491907755535, 19.251351674397785, 18.859820048014324, 18.648751576741535 ], "return_min": [ 0.0, 0.31176972443433465, 0.8396267360691291, 1.2663937149181501, 2.530283852092835, 4.464613059374979, 4.9106046317547305, 5.38566465616166, 5.950525768996994, 8.09050058904016, 7.377376393544287, 8.834169037903324, 9.362149864454587, 10.808991493684232, 12.488085054344273, 13.79303938444693, 16.140531526516416, 16.37832635466966, 18.58600524039474, 18.461735559677493, 17.944450849685 ], "return_max": [ 0.0, 0.39491883859463667, 0.9583446317668695, 1.6953690312569802, 2.814522342212585, 4.624703944510926, 5.145292445169912, 7.0599040643380135, 9.303339473007401, 9.883457744293173, 11.448098027638983, 15.171778710916664, 15.532142331183433, 17.21323547571681, 17.55292103391256, 18.734624800123385, 18.547858569512233, 19.50065746084141, 19.91669810840083, 19.257904536351155, 19.35305230379807 ] }, "FastTD3": { "time": [ 0.0, 368.47968223911107, 736.9593644782221, 1105.4390467173334, 1473.9187289564443, 1842.3984111955554, 2210.878093434667, 2579.3577756737777, 2947.8374579128886, 3316.317140152 ], "env_step": [ 0, 5120000.0, 10240000.0, 15360000.0, 20480000.0, 25600000.0, 30720000.0, 35840000.0, 40960000.0, 46080000.0 ], "return": [ 0.0, 9.457830508550009, 15.376138687133789, 18.36476771036784, 23.326606432596844, 26.596385320027668, 28.654596010843914, 29.314904530843098, 30.264007886250813, 30.40758482615153 ], "return_min": [ 0.0, 6.853786206432801, 13.629757692263793, 15.874774229828427, 21.370459278573758, 25.307348278642664, 26.545756306321362, 26.61948520643785, 27.26373076116701, 27.204618859278554 ], "return_max": [ 0.0, 12.061874810667216, 17.122519682003784, 20.854761190907254, 25.28275358661993, 27.885422361412672, 30.763435715366466, 32.010323855248345, 33.264285011334614, 33.61055079302451 ] } } } ================================================ FILE: fast_td3/__init__.py ================================================ """ Fast TD3 is a high-performance implementation of Twin Delayed Deep Deterministic Policy Gradient (TD3) with distributional critics for reinforcement learning. """ # Core model components from fast_td3.fast_td3 import Actor, Critic, DistributionalQNetwork from fast_td3.fast_td3_utils import EmpiricalNormalization, SimpleReplayBuffer from fast_td3.fast_td3_deploy import Policy, load_policy __all__ = [ # Core model components "Actor", "Critic", "DistributionalQNetwork", "EmpiricalNormalization", "SimpleReplayBuffer", "Policy", "load_policy", ] ================================================ FILE: fast_td3/environments/humanoid_bench_env.py ================================================ from __future__ import annotations import gymnasium as gym import humanoid_bench from gymnasium.wrappers import TimeLimit from stable_baselines3.common.vec_env import SubprocVecEnv import numpy as np import torch from loguru import logger as log # Disable all logging below CRITICAL level log.remove() log.add(lambda msg: False, level="CRITICAL") def make_env(env_name, rank, render_mode=None, seed=0): """ Utility function for multiprocessed env. :param rank: (int) index of the subprocess :param seed: (int) the inital seed for RNG """ if env_name in [ "h1hand-push-v0", "h1-push-v0", "h1hand-cube-v0", "h1cube-v0", "h1hand-basketball-v0", "h1-basketball-v0", "h1hand-kitchen-v0", "h1-kitchen-v0", ]: max_episode_steps = 500 else: max_episode_steps = 1000 def _init(): import humanoid_bench env = gym.make(env_name, render_mode=render_mode) env = TimeLimit(env, max_episode_steps=max_episode_steps) env.unwrapped.seed(seed + rank) return env return _init class HumanoidBenchEnv: """Wraps HumanoidBench environment to support parallel environments.""" def __init__(self, env_name, num_envs=1, render_mode=None, device=None): # NOTE: HumanoidBench action space is already normalized to [-1, 1] device = device or torch.device("cuda" if torch.cuda.is_available() else "cpu") self.sim_device = device self.num_envs = num_envs # Create the base environment self.envs = SubprocVecEnv( [make_env(env_name, i, render_mode=render_mode) for i in range(num_envs)] ) if env_name in [ "h1hand-push-v0", "h1-push-v0", "h1hand-cube-v0", "h1cube-v0", "h1hand-basketball-v0", "h1-basketball-v0", "h1hand-kitchen-v0", "h1-kitchen-v0", ]: self.max_episode_steps = 500 else: self.max_episode_steps = 1000 # For compatibility with MuJoCo Playground self.asymmetric_obs = False # For comptatibility with MuJoCo Playground self.num_obs = self.envs.observation_space.shape[-1] self.num_actions = self.envs.action_space.shape[-1] def reset(self): """Reset the environment.""" observations = self.envs.reset() observations = torch.from_numpy(observations).to( device=self.sim_device, dtype=torch.float ) return observations def render(self): assert ( self.num_envs == 1 ), "Currently only supports single environment rendering" return self.envs.render() def step(self, actions): assert isinstance(actions, torch.Tensor) actions = actions.cpu().numpy() observations, rewards, dones, raw_infos = self.envs.step(actions) # This will be used for getting 'true' next observations infos = dict() infos["observations"] = {"raw": {"obs": observations.copy()}} truncateds = np.zeros_like(dones) for i in range(self.num_envs): if raw_infos[i].get("TimeLimit.truncated", False): truncateds[i] = True infos["observations"]["raw"]["obs"][i] = raw_infos[i][ "terminal_observation" ] observations = torch.from_numpy(observations).to( device=self.sim_device, dtype=torch.float ) rewards = torch.from_numpy(rewards).to( device=self.sim_device, dtype=torch.float ) dones = torch.from_numpy(dones).to(device=self.sim_device) truncateds = torch.from_numpy(truncateds).to(device=self.sim_device) infos["observations"]["raw"]["obs"] = torch.from_numpy( infos["observations"]["raw"]["obs"] ).to(device=self.sim_device, dtype=torch.float) infos["time_outs"] = truncateds return observations, rewards, dones, infos ================================================ FILE: fast_td3/environments/isaaclab_env.py ================================================ from typing import Optional import gymnasium as gym import torch class IsaacLabEnv: """Wrapper for IsaacLab environments to be compatible with MuJoCo Playground""" def __init__( self, task_name: str, device: str, num_envs: int, seed: int, action_bounds: Optional[float] = None, ): from isaaclab.app import AppLauncher app_launcher = AppLauncher(headless=True, device=device) simulation_app = app_launcher.app import isaaclab_tasks from isaaclab_tasks.utils.parse_cfg import parse_env_cfg env_cfg = parse_env_cfg( task_name, device=device, num_envs=num_envs, ) env_cfg.seed = seed self.seed = seed self.envs = gym.make(task_name, cfg=env_cfg, render_mode=None) self.num_envs = self.envs.unwrapped.num_envs self.max_episode_steps = self.envs.unwrapped.max_episode_length self.action_bounds = action_bounds self.num_obs = self.envs.unwrapped.single_observation_space["policy"].shape[0] self.asymmetric_obs = "critic" in self.envs.unwrapped.single_observation_space if self.asymmetric_obs: self.num_privileged_obs = self.envs.unwrapped.single_observation_space[ "critic" ].shape[0] else: self.num_privileged_obs = 0 self.num_actions = self.envs.unwrapped.single_action_space.shape[0] def reset(self, random_start_init: bool = True) -> torch.Tensor: obs_dict, _ = self.envs.reset() # NOTE: decorrelate episode horizons like RSL‑RL if random_start_init: self.envs.unwrapped.episode_length_buf = torch.randint_like( self.envs.unwrapped.episode_length_buf, high=int(self.max_episode_steps) ) return obs_dict["policy"] def reset_with_critic_obs(self) -> tuple[torch.Tensor, torch.Tensor]: obs_dict, _ = self.envs.reset() return obs_dict["policy"], obs_dict["critic"] def step( self, actions: torch.Tensor ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, dict]: if self.action_bounds is not None: actions = torch.clamp(actions, -1.0, 1.0) * self.action_bounds obs_dict, rew, terminations, truncations, infos = self.envs.step(actions) dones = (terminations | truncations).to(dtype=torch.long) obs = obs_dict["policy"] critic_obs = obs_dict["critic"] if self.asymmetric_obs else None info_ret = {"time_outs": truncations, "observations": {"critic": critic_obs}} # NOTE: There's really no way to get the raw observations from IsaacLab # We just use the 'reset_obs' as next_obs, unfortunately. # See https://github.com/isaac-sim/IsaacLab/issues/1362 info_ret["observations"]["raw"] = { "obs": obs, "critic_obs": critic_obs, } return obs, rew, dones, info_ret def render(self): raise NotImplementedError( "We don't support rendering for IsaacLab environments" ) ================================================ FILE: fast_td3/environments/mtbench_env.py ================================================ from __future__ import annotations import torch from omegaconf import OmegaConf import isaacgym import isaacgymenvs class MTBenchEnv: def __init__( self, task_name: str, device_id: int, num_envs: int, seed: int, ): # NOTE: Currently, we only support Meta-World-v2 MT-10/MT-50 in MTBench task_config = MTBENCH_MW2_CONFIG.copy() if task_name == "meta-world-v2-mt10": # MT-10 Setup assert num_envs == 4096, "MT-10 only supports 4096 environments (for now)" self.num_tasks = 10 task_config["env"]["tasks"] = [4, 16, 17, 18, 28, 31, 38, 40, 48, 49] task_config["env"]["taskEnvCount"] = [410] * 6 + [409] * 4 elif task_name == "meta-world-v2-mt50": # MT-50 Setup self.num_tasks = 50 assert num_envs == 8192, "MT-50 only supports 8192 environments (for now)" task_config["env"]["tasks"] = list(range(50)) task_config["env"]["taskEnvCount"] = [164] * 42 + [163] * 8 # 6888 + 1304 else: raise ValueError(f"Unsupported task name: {task_name}") task_config["env"]["numEnvs"] = num_envs task_config["env"]["numObservations"] = 39 + self.num_tasks task_config["env"]["seed"] = seed # Convert dictionary to OmegaConf object env_cfg = {"task": task_config} env_cfg = OmegaConf.create(env_cfg) self.env = isaacgymenvs.make( task=env_cfg.task.name, num_envs=num_envs, sim_device=f"cuda:{device_id}", rl_device=f"cuda:{device_id}", seed=seed, headless=True, cfg=env_cfg, ) self.num_envs = num_envs self.asymmetric_obs = False self.num_obs = self.env.observation_space.shape[0] assert ( self.num_obs == 39 + self.num_tasks ), "MTBench observation space is 39 + num_tasks (one-hot vector)" self.num_privileged_obs = 0 self.num_actions = self.env.action_space.shape[0] self.max_episode_steps = self.env.max_episode_length def reset(self) -> torch.Tensor: """Reset the environment.""" # TODO: Check if we need no_grad and detach here with torch.no_grad(): # do we need this? self.env.reset_idx(torch.arange(self.num_envs, device=self.env.device)) self.env.cumulatives["rewards"][:] = 0 self.env.cumulatives["success"][:] = 0 obs_dict = self.env.reset() return obs_dict["obs"].detach() def step( self, actions: torch.Tensor ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, dict]: """Step the environment.""" assert isinstance(actions, torch.Tensor) # TODO: Check if we need no_grad and detach here with torch.no_grad(): obs_dict, rew, dones, infos = self.env.step(actions.detach()) truncations = infos["time_outs"] info_ret = {"time_outs": truncations.detach()} if "episode" in infos: info_ret["episode"] = infos["episode"] # NOTE: There's really no way to get the raw observations from IsaacGym # We just use the 'reset_obs' as next_obs, unfortunately. info_ret["observations"] = {"raw": {"obs": obs_dict["obs"].detach()}} return obs_dict["obs"].detach(), rew.detach(), dones.detach(), info_ret def render(self): raise NotImplementedError( "We don't support rendering for IsaacLab environments" ) MTBENCH_MW2_CONFIG = { "name": "meta-world-v2", "physics_engine": "physx", "env": { "numEnvs": 1, "envSpacing": 1.5, "episodeLength": 150, "enableDebugVis": False, "clipObservations": 5.0, "clipActions": 1.0, "aggregateMode": 3, "actionScale": 0.01, "resetNoise": 0.15, "tasks": [0], "taskEnvCount": [4096], "init_at_random_progress": True, "exemptedInitAtRandomProgressTasks": [], "taskEmbedding": True, "taskEmbeddingType": "one_hot", "seed": 42, "cameraRenderingInterval": 5000, "cameraWidth": 1024, "cameraHeight": 1024, "sparse_reward": False, "termination_on_success": False, "reward_scale": 1.0, "fixed": False, "numObservations": None, "numActions": 4, }, "enableCameraSensors": False, "sim": { "dt": 0.01667, "substeps": 2, "up_axis": "z", "use_gpu_pipeline": True, "gravity": [0.0, 0.0, -9.81], "physx": { "num_threads": 4, "solver_type": 1, "use_gpu": True, "num_position_iterations": 8, "num_velocity_iterations": 1, "contact_offset": 0.005, "rest_offset": 0.0, "bounce_threshold_velocity": 0.2, "max_depenetration_velocity": 1000.0, "default_buffer_size_multiplier": 10.0, "max_gpu_contact_pairs": 1048576, "num_subscenes": 4, "contact_collection": 0, }, }, "task": {"randomize": False}, } ================================================ FILE: fast_td3/environments/mujoco_playground_env.py ================================================ from mujoco_playground import registry from mujoco_playground import wrapper_torch import jax import mujoco class PlaygroundEvalEnvWrapper: def __init__( self, eval_env, max_episode_steps, env_name, num_eval_envs, seed, device_rank=None, ): """ Wrapper used for evaluation / rendering environments. Note that this is different from training environments that are wrapped with RSLRLBraxWrapper. """ self.env = eval_env self.env_name = env_name self.num_envs = num_eval_envs self.jit_reset = jax.jit(jax.vmap(self.env.reset)) self.jit_step = jax.jit(jax.vmap(self.env.step)) if isinstance(self.env.unwrapped.observation_size, dict): self.asymmetric_obs = True else: self.asymmetric_obs = False self.key = jax.random.PRNGKey(seed) if device_rank is not None: gpu_devices = jax.devices("gpu") self.key = jax.device_put(self.key, gpu_devices[device_rank]) self.key_reset = jax.random.split(self.key, num_eval_envs) self.max_episode_steps = max_episode_steps def reset(self): self.state = self.jit_reset(self.key_reset) if self.asymmetric_obs: obs = wrapper_torch._jax_to_torch(self.state.obs["state"]) else: obs = wrapper_torch._jax_to_torch(self.state.obs) return obs def step(self, actions): self.state = self.jit_step(self.state, wrapper_torch._torch_to_jax(actions)) if self.asymmetric_obs: next_obs = wrapper_torch._jax_to_torch(self.state.obs["state"]) else: next_obs = wrapper_torch._jax_to_torch(self.state.obs) rewards = wrapper_torch._jax_to_torch(self.state.reward) dones = wrapper_torch._jax_to_torch(self.state.done) return next_obs, rewards, dones, None def render_trajectory(self, trajectory): scene_option = mujoco.MjvOption() scene_option.flags[mujoco.mjtVisFlag.mjVIS_TRANSPARENT] = False scene_option.flags[mujoco.mjtVisFlag.mjVIS_PERTFORCE] = False scene_option.flags[mujoco.mjtVisFlag.mjVIS_CONTACTFORCE] = False frames = self.env.render( trajectory, camera="track" if "Joystick" in self.env_name else None, height=480, width=640, scene_option=scene_option, ) return frames def make_env( env_name, seed, num_envs, num_eval_envs, device_rank, use_tuned_reward=False, use_domain_randomization=False, use_push_randomization=False, ): # Make training environment train_env_cfg = registry.get_default_config(env_name) is_humanoid_task = env_name in [ "G1JoystickRoughTerrain", "G1JoystickFlatTerrain", "T1JoystickRoughTerrain", "T1JoystickFlatTerrain", ] if use_tuned_reward and is_humanoid_task: # NOTE: Tuned reward for G1. Used for producing Figure 7 in the paper. # Somehow it works reasonably for T1 as well. # However, see `sim2real.md` for sim-to-real RL with Booster T1 train_env_cfg.reward_config.scales.energy = -5e-5 train_env_cfg.reward_config.scales.action_rate = -1e-1 train_env_cfg.reward_config.scales.torques = -1e-3 train_env_cfg.reward_config.scales.pose = -1.0 train_env_cfg.reward_config.scales.tracking_ang_vel = 1.25 train_env_cfg.reward_config.scales.tracking_lin_vel = 1.25 train_env_cfg.reward_config.scales.feet_phase = 1.0 train_env_cfg.reward_config.scales.ang_vel_xy = -0.3 train_env_cfg.reward_config.scales.orientation = -5.0 if is_humanoid_task and not use_push_randomization: train_env_cfg.push_config.enable = False train_env_cfg.push_config.magnitude_range = [0.0, 0.0] randomizer = ( registry.get_domain_randomizer(env_name) if use_domain_randomization else None ) raw_env = registry.load(env_name, config=train_env_cfg) train_env = wrapper_torch.RSLRLBraxWrapper( raw_env, num_envs, seed, train_env_cfg.episode_length, train_env_cfg.action_repeat, randomization_fn=randomizer, device_rank=device_rank, ) # Make evaluation environment eval_env_cfg = registry.get_default_config(env_name) if is_humanoid_task and not use_push_randomization: eval_env_cfg.push_config.enable = False eval_env_cfg.push_config.magnitude_range = [0.0, 0.0] eval_env = registry.load(env_name, config=eval_env_cfg) eval_env = PlaygroundEvalEnvWrapper( eval_env, eval_env_cfg.episode_length, env_name, num_eval_envs, seed, device_rank=device_rank, ) render_env_cfg = registry.get_default_config(env_name) if is_humanoid_task and not use_push_randomization: render_env_cfg.push_config.enable = False render_env_cfg.push_config.magnitude_range = [0.0, 0.0] render_env = registry.load(env_name, config=render_env_cfg) render_env = PlaygroundEvalEnvWrapper( render_env, render_env_cfg.episode_length, env_name, 1, seed, device_rank=device_rank, ) return train_env, eval_env, render_env ================================================ FILE: fast_td3/fast_td3.py ================================================ import torch import torch.nn as nn import torch.nn.functional as F class DistributionalQNetwork(nn.Module): def __init__( self, n_obs: int, n_act: int, num_atoms: int, v_min: float, v_max: float, hidden_dim: int, device: torch.device = None, ): super().__init__() self.net = nn.Sequential( nn.Linear(n_obs + n_act, hidden_dim, device=device), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim // 2, device=device), nn.ReLU(), nn.Linear(hidden_dim // 2, hidden_dim // 4, device=device), nn.ReLU(), nn.Linear(hidden_dim // 4, num_atoms, device=device), ) self.v_min = v_min self.v_max = v_max self.num_atoms = num_atoms def forward(self, obs: torch.Tensor, actions: torch.Tensor) -> torch.Tensor: x = torch.cat([obs, actions], 1) x = self.net(x) return x def projection( self, obs: torch.Tensor, actions: torch.Tensor, rewards: torch.Tensor, bootstrap: torch.Tensor, discount: float, q_support: torch.Tensor, device: torch.device, ) -> torch.Tensor: delta_z = (self.v_max - self.v_min) / (self.num_atoms - 1) batch_size = rewards.shape[0] target_z = ( rewards.unsqueeze(1) + bootstrap.unsqueeze(1) * discount.unsqueeze(1) * q_support ) target_z = target_z.clamp(self.v_min, self.v_max) b = (target_z - self.v_min) / delta_z l = torch.floor(b).long() u = torch.ceil(b).long() is_int = (l == u) l_mask = is_int & (l > 0) u_mask = is_int & (l == 0) l = torch.where(l_mask, l - 1, l) u = torch.where(u_mask, u + 1, u) next_dist = F.softmax(self.forward(obs, actions), dim=1) proj_dist = torch.zeros_like(next_dist) offset = ( torch.linspace( 0, (batch_size - 1) * self.num_atoms, batch_size, device=device ) .unsqueeze(1) .expand(batch_size, self.num_atoms) .long() ) proj_dist.view(-1).index_add_( 0, (l + offset).view(-1), (next_dist * (u.float() - b)).view(-1) ) proj_dist.view(-1).index_add_( 0, (u + offset).view(-1), (next_dist * (b - l.float())).view(-1) ) return proj_dist class Critic(nn.Module): def __init__( self, n_obs: int, n_act: int, num_atoms: int, v_min: float, v_max: float, hidden_dim: int, device: torch.device = None, ): super().__init__() self.qnet1 = DistributionalQNetwork( n_obs=n_obs, n_act=n_act, num_atoms=num_atoms, v_min=v_min, v_max=v_max, hidden_dim=hidden_dim, device=device, ) self.qnet2 = DistributionalQNetwork( n_obs=n_obs, n_act=n_act, num_atoms=num_atoms, v_min=v_min, v_max=v_max, hidden_dim=hidden_dim, device=device, ) self.register_buffer( "q_support", torch.linspace(v_min, v_max, num_atoms, device=device) ) self.device = device def forward(self, obs: torch.Tensor, actions: torch.Tensor) -> torch.Tensor: return self.qnet1(obs, actions), self.qnet2(obs, actions) def projection( self, obs: torch.Tensor, actions: torch.Tensor, rewards: torch.Tensor, bootstrap: torch.Tensor, discount: float, ) -> torch.Tensor: """Projection operation that includes q_support directly""" q1_proj = self.qnet1.projection( obs, actions, rewards, bootstrap, discount, self.q_support, self.q_support.device, ) q2_proj = self.qnet2.projection( obs, actions, rewards, bootstrap, discount, self.q_support, self.q_support.device, ) return q1_proj, q2_proj def get_value(self, probs: torch.Tensor) -> torch.Tensor: """Calculate value from logits using support""" return torch.sum(probs * self.q_support, dim=1) class Actor(nn.Module): def __init__( self, n_obs: int, n_act: int, num_envs: int, init_scale: float, hidden_dim: int, std_min: float = 0.05, std_max: float = 0.8, device: torch.device = None, ): super().__init__() self.n_act = n_act self.net = nn.Sequential( nn.Linear(n_obs, hidden_dim, device=device), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim // 2, device=device), nn.ReLU(), nn.Linear(hidden_dim // 2, hidden_dim // 4, device=device), nn.ReLU(), ) self.fc_mu = nn.Sequential( nn.Linear(hidden_dim // 4, n_act, device=device), nn.Tanh(), ) nn.init.normal_(self.fc_mu[0].weight, 0.0, init_scale) nn.init.constant_(self.fc_mu[0].bias, 0.0) noise_scales = ( torch.rand(num_envs, 1, device=device) * (std_max - std_min) + std_min ) self.register_buffer("noise_scales", noise_scales) self.register_buffer("std_min", torch.as_tensor(std_min, device=device)) self.register_buffer("std_max", torch.as_tensor(std_max, device=device)) self.n_envs = num_envs self.device = device def forward(self, obs: torch.Tensor) -> torch.Tensor: x = obs x = self.net(x) action = self.fc_mu(x) return action def explore( self, obs: torch.Tensor, dones: torch.Tensor = None, deterministic: bool = False ) -> torch.Tensor: # If dones is provided, resample noise for environments that are done if dones is not None and dones.sum() > 0: # Generate new noise scales for done environments (one per environment) new_scales = ( torch.rand(self.n_envs, 1, device=obs.device) * (self.std_max - self.std_min) + self.std_min ) # Update only the noise scales for environments that are done dones_view = dones.view(-1, 1) > 0 self.noise_scales.copy_( torch.where(dones_view, new_scales, self.noise_scales) ) act = self(obs) if deterministic: return act noise = torch.randn_like(act) * self.noise_scales return act + noise class MultiTaskActor(Actor): def __init__(self, num_tasks: int, task_embedding_dim: int, *args, **kwargs): super().__init__(*args, **kwargs) self.num_tasks = num_tasks self.task_embedding_dim = task_embedding_dim self.task_embedding = nn.Embedding( num_tasks, task_embedding_dim, max_norm=1.0, device=self.device ) def forward(self, obs: torch.Tensor) -> torch.Tensor: # TODO: Optimize the code to be compatible with cudagraphs # Currently in-place creation of task_indices is not compatible with cudagraphs task_ids_one_hot = obs[..., -self.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) task_embeddings = self.task_embedding(task_indices) obs = torch.cat([obs[..., : -self.num_tasks], task_embeddings], dim=-1) return super().forward(obs) class MultiTaskCritic(Critic): def __init__(self, num_tasks: int, task_embedding_dim: int, *args, **kwargs): super().__init__(*args, **kwargs) self.num_tasks = num_tasks self.task_embedding_dim = task_embedding_dim self.task_embedding = nn.Embedding( num_tasks, task_embedding_dim, max_norm=1.0, device=self.device ) def forward(self, obs: torch.Tensor, actions: torch.Tensor) -> torch.Tensor: # TODO: Optimize the code to be compatible with cudagraphs # Currently in-place creation of task_indices is not compatible with cudagraphs task_ids_one_hot = obs[..., -self.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) task_embeddings = self.task_embedding(task_indices) obs = torch.cat([obs[..., : -self.num_tasks], task_embeddings], dim=-1) return super().forward(obs, actions) def projection( self, obs: torch.Tensor, actions: torch.Tensor, rewards: torch.Tensor, bootstrap: torch.Tensor, discount: float, ) -> torch.Tensor: task_ids_one_hot = obs[..., -self.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) task_embeddings = self.task_embedding(task_indices) obs = torch.cat([obs[..., : -self.num_tasks], task_embeddings], dim=-1) return super().projection(obs, actions, rewards, bootstrap, discount) ================================================ FILE: fast_td3/fast_td3_deploy.py ================================================ import math import torch import torch.nn as nn from .fast_td3_utils import EmpiricalNormalization from .fast_td3 import Actor from .fast_td3_simbav2 import Actor as ActorSimbaV2 class Policy(nn.Module): def __init__( self, n_obs: int, n_act: int, args: dict, agent: str = "fasttd3", ): super().__init__() self.args = args num_envs = args["num_envs"] init_scale = args["init_scale"] actor_hidden_dim = args["actor_hidden_dim"] actor_kwargs = dict( n_obs=n_obs, n_act=n_act, num_envs=num_envs, device="cpu", init_scale=init_scale, hidden_dim=actor_hidden_dim, ) if agent == "fasttd3": actor_cls = Actor elif agent == "fasttd3_simbav2": actor_cls = ActorSimbaV2 actor_num_blocks = args["actor_num_blocks"] actor_kwargs.pop("init_scale") actor_kwargs.update( { "scaler_init": math.sqrt(2.0 / actor_hidden_dim), "scaler_scale": math.sqrt(2.0 / actor_hidden_dim), "alpha_init": 1.0 / (actor_num_blocks + 1), "alpha_scale": 1.0 / math.sqrt(actor_hidden_dim), "expansion": 4, "c_shift": 3.0, "num_blocks": actor_num_blocks, } ) else: raise ValueError(f"Agent {agent} not supported") self.actor = actor_cls( **actor_kwargs, ) self.obs_normalizer = EmpiricalNormalization(shape=n_obs, device="cpu") self.actor.eval() self.obs_normalizer.eval() @torch.no_grad def forward(self, obs: torch.Tensor) -> torch.Tensor: norm_obs = self.obs_normalizer(obs) actions = self.actor(norm_obs) return actions @torch.no_grad def act(self, obs: torch.Tensor) -> torch.distributions.Normal: actions = self.forward(obs) return torch.distributions.Normal(actions, torch.ones_like(actions) * 1e-8) def load_policy(checkpoint_path): torch_checkpoint = torch.load( f"{checkpoint_path}", map_location="cpu", weights_only=False ) args = torch_checkpoint["args"] agent = args.get("agent", "fasttd3") if agent == "fasttd3": n_obs = torch_checkpoint["actor_state_dict"]["net.0.weight"].shape[-1] n_act = torch_checkpoint["actor_state_dict"]["fc_mu.0.weight"].shape[0] elif agent == "fasttd3_simbav2": # TODO: Too hard-coded, maybe save n_obs and n_act in the checkpoint? n_obs = ( torch_checkpoint["actor_state_dict"]["embedder.w.w.weight"].shape[-1] - 1 ) n_act = torch_checkpoint["actor_state_dict"]["predictor.mean_bias"].shape[0] else: raise ValueError(f"Agent {agent} not supported") policy = Policy( n_obs=n_obs, n_act=n_act, args=args, agent=agent, ) policy.actor.load_state_dict(torch_checkpoint["actor_state_dict"]) if len(torch_checkpoint["obs_normalizer_state"]) == 0: policy.obs_normalizer = nn.Identity() else: policy.obs_normalizer.load_state_dict(torch_checkpoint["obs_normalizer_state"]) return policy ================================================ FILE: fast_td3/fast_td3_simbav2.py ================================================ import torch import torch.nn as nn import torch.nn.functional as F import math def l2normalize( tensor: torch.Tensor, axis: int = -1, eps: float = 1e-8 ) -> torch.Tensor: """Computes L2 normalization of a tensor.""" return tensor / (torch.linalg.norm(tensor, ord=2, dim=axis, keepdim=True) + eps) class Scaler(nn.Module): """ A learnable scaling layer. """ def __init__( self, dim: int, init: float = 1.0, scale: float = 1.0, device: torch.device = None, ): super().__init__() self.scaler = nn.Parameter(torch.full((dim,), init * scale, device=device)) self.forward_scaler = init / scale def forward(self, x: torch.Tensor) -> torch.Tensor: return self.scaler.to(x.dtype) * self.forward_scaler * x class HyperDense(nn.Module): """ A dense layer without bias and with orthogonal initialization. """ def __init__(self, in_dim: int, hidden_dim: int, device: torch.device = None): super().__init__() self.w = nn.Linear(in_dim, hidden_dim, bias=False, device=device) nn.init.orthogonal_(self.w.weight, gain=1.0) def forward(self, x: torch.Tensor) -> torch.Tensor: return self.w(x) class HyperMLP(nn.Module): """ A small MLP with a specific architecture using HyperDense and Scaler. """ def __init__( self, in_dim: int, hidden_dim: int, out_dim: int, scaler_init: float, scaler_scale: float, eps: float = 1e-8, device: torch.device = None, ): super().__init__() self.w1 = HyperDense(in_dim, hidden_dim, device=device) self.scaler = Scaler(hidden_dim, scaler_init, scaler_scale, device=device) self.w2 = HyperDense(hidden_dim, out_dim, device=device) self.eps = eps def forward(self, x: torch.Tensor) -> torch.Tensor: x = self.w1(x) x = self.scaler(x) # `eps` is required to prevent zero vector. x = F.relu(x) + self.eps x = self.w2(x) x = l2normalize(x, axis=-1) return x class HyperEmbedder(nn.Module): """ Embeds input by concatenating a constant, normalizing, and applying layers. """ def __init__( self, in_dim: int, hidden_dim: int, scaler_init: float, scaler_scale: float, c_shift: float, device: torch.device = None, ): super().__init__() # The input dimension to the dense layer is in_dim + 1 self.w = HyperDense(in_dim + 1, hidden_dim, device=device) self.scaler = Scaler(hidden_dim, scaler_init, scaler_scale, device=device) self.c_shift = c_shift def forward(self, x: torch.Tensor) -> torch.Tensor: new_axis = torch.full( (*x.shape[:-1], 1), self.c_shift, device=x.device, dtype=x.dtype ) x = torch.cat([x, new_axis], dim=-1) x = l2normalize(x, axis=-1) x = self.w(x) x = self.scaler(x) x = l2normalize(x, axis=-1) return x class HyperLERPBlock(nn.Module): """ A residual block using Linear Interpolation (LERP). """ def __init__( self, hidden_dim: int, scaler_init: float, scaler_scale: float, alpha_init: float, alpha_scale: float, expansion: int = 4, device: torch.device = None, ): super().__init__() self.mlp = HyperMLP( in_dim=hidden_dim, hidden_dim=hidden_dim * expansion, out_dim=hidden_dim, scaler_init=scaler_init / math.sqrt(expansion), scaler_scale=scaler_scale / math.sqrt(expansion), device=device, ) self.alpha_scaler = Scaler( dim=hidden_dim, init=alpha_init, scale=alpha_scale, device=device, ) def forward(self, x: torch.Tensor) -> torch.Tensor: residual = x mlp_out = self.mlp(x) # The original paper uses (x - residual) but x is the residual here. # This is interpreted as alpha * (mlp_output - residual_input) x = residual + self.alpha_scaler(mlp_out - residual) x = l2normalize(x, axis=-1) return x class HyperTanhPolicy(nn.Module): """ A policy that outputs a Tanh action. """ def __init__( self, hidden_dim: int, action_dim: int, scaler_init: float, scaler_scale: float, device: torch.device = None, ): super().__init__() self.mean_w1 = HyperDense(hidden_dim, hidden_dim, device=device) self.mean_scaler = Scaler(hidden_dim, scaler_init, scaler_scale, device=device) self.mean_w2 = HyperDense(hidden_dim, action_dim, device=device) self.mean_bias = nn.Parameter(torch.zeros(action_dim, device=device)) def forward(self, x: torch.Tensor) -> torch.Tensor: # Mean path mean = self.mean_w1(x) mean = self.mean_scaler(mean) mean = self.mean_w2(mean) + self.mean_bias.to(mean.dtype) mean = torch.tanh(mean) return mean class HyperCategoricalValue(nn.Module): """ A value function that predicts a categorical distribution over a range of values. """ def __init__( self, hidden_dim: int, num_bins: int, scaler_init: float, scaler_scale: float, device: torch.device = None, ): super().__init__() self.w1 = HyperDense(hidden_dim, hidden_dim, device=device) self.scaler = Scaler(hidden_dim, scaler_init, scaler_scale, device=device) self.w2 = HyperDense(hidden_dim, num_bins, device=device) self.bias = nn.Parameter(torch.zeros(num_bins, device=device)) def forward(self, x: torch.Tensor) -> torch.Tensor: logits = self.w1(x) logits = self.scaler(logits) logits = self.w2(logits) + self.bias.to(logits.dtype) return logits class DistributionalQNetwork(nn.Module): def __init__( self, n_obs: int, n_act: int, num_atoms: int, v_min: float, v_max: float, hidden_dim: int, scaler_init: float, scaler_scale: float, alpha_init: float, alpha_scale: float, num_blocks: int, c_shift: float, expansion: int, device: torch.device = None, ): super().__init__() self.embedder = HyperEmbedder( in_dim=n_obs + n_act, hidden_dim=hidden_dim, scaler_init=scaler_init, scaler_scale=scaler_scale, c_shift=c_shift, device=device, ) self.encoder = nn.Sequential( *[ HyperLERPBlock( hidden_dim=hidden_dim, scaler_init=scaler_init, scaler_scale=scaler_scale, alpha_init=alpha_init, alpha_scale=alpha_scale, expansion=expansion, device=device, ) for _ in range(num_blocks) ] ) self.predictor = HyperCategoricalValue( hidden_dim=hidden_dim, num_bins=num_atoms, scaler_init=1.0, scaler_scale=1.0, device=device, ) self.v_min = v_min self.v_max = v_max self.num_atoms = num_atoms def forward(self, obs: torch.Tensor, actions: torch.Tensor) -> torch.Tensor: x = torch.cat([obs, actions], 1) x = self.embedder(x) x = self.encoder(x) x = self.predictor(x) return x def projection( self, obs: torch.Tensor, actions: torch.Tensor, rewards: torch.Tensor, bootstrap: torch.Tensor, discount: float, q_support: torch.Tensor, device: torch.device, ) -> torch.Tensor: delta_z = (self.v_max - self.v_min) / (self.num_atoms - 1) batch_size = rewards.shape[0] target_z = ( rewards.unsqueeze(1) + bootstrap.unsqueeze(1) * discount.unsqueeze(1) * q_support ) target_z = target_z.clamp(self.v_min, self.v_max) b = (target_z - self.v_min) / delta_z l = torch.floor(b).long() u = torch.ceil(b).long() is_int = (l == u) l_mask = is_int & (l > 0) u_mask = is_int & (l == 0) l = torch.where(l_mask, l - 1, l) u = torch.where(u_mask, u + 1, u) next_dist = F.softmax(self.forward(obs, actions), dim=1) proj_dist = torch.zeros_like(next_dist) offset = ( torch.linspace( 0, (batch_size - 1) * self.num_atoms, batch_size, device=device ) .unsqueeze(1) .expand(batch_size, self.num_atoms) .long() ) proj_dist.view(-1).index_add_( 0, (l + offset).view(-1), (next_dist * (u.float() - b)).view(-1) ) proj_dist.view(-1).index_add_( 0, (u + offset).view(-1), (next_dist * (b - l.float())).view(-1) ) return proj_dist class Critic(nn.Module): def __init__( self, n_obs: int, n_act: int, num_atoms: int, v_min: float, v_max: float, hidden_dim: int, scaler_init: float, scaler_scale: float, alpha_init: float, alpha_scale: float, num_blocks: int, c_shift: float, expansion: int, device: torch.device = None, ): super().__init__() self.qnet1 = DistributionalQNetwork( n_obs=n_obs, n_act=n_act, num_atoms=num_atoms, v_min=v_min, v_max=v_max, scaler_init=scaler_init, scaler_scale=scaler_scale, alpha_init=alpha_init, alpha_scale=alpha_scale, num_blocks=num_blocks, c_shift=c_shift, expansion=expansion, hidden_dim=hidden_dim, device=device, ) self.qnet2 = DistributionalQNetwork( n_obs=n_obs, n_act=n_act, num_atoms=num_atoms, v_min=v_min, v_max=v_max, scaler_init=scaler_init, scaler_scale=scaler_scale, alpha_init=alpha_init, alpha_scale=alpha_scale, num_blocks=num_blocks, c_shift=c_shift, expansion=expansion, hidden_dim=hidden_dim, device=device, ) self.register_buffer( "q_support", torch.linspace(v_min, v_max, num_atoms, device=device) ) self.device = device def forward(self, obs: torch.Tensor, actions: torch.Tensor) -> torch.Tensor: return self.qnet1(obs, actions), self.qnet2(obs, actions) def projection( self, obs: torch.Tensor, actions: torch.Tensor, rewards: torch.Tensor, bootstrap: torch.Tensor, discount: float, ) -> torch.Tensor: """Projection operation that includes q_support directly""" q1_proj = self.qnet1.projection( obs, actions, rewards, bootstrap, discount, self.q_support, self.q_support.device, ) q2_proj = self.qnet2.projection( obs, actions, rewards, bootstrap, discount, self.q_support, self.q_support.device, ) return q1_proj, q2_proj def get_value(self, probs: torch.Tensor) -> torch.Tensor: """Calculate value from logits using support""" return torch.sum(probs * self.q_support, dim=1) class Actor(nn.Module): def __init__( self, n_obs: int, n_act: int, num_envs: int, hidden_dim: int, scaler_init: float, scaler_scale: float, alpha_init: float, alpha_scale: float, expansion: int, c_shift: float, num_blocks: int, std_min: float = 0.05, std_max: float = 0.8, device: torch.device = None, ): super().__init__() self.n_act = n_act self.embedder = HyperEmbedder( in_dim=n_obs, hidden_dim=hidden_dim, scaler_init=scaler_init, scaler_scale=scaler_scale, c_shift=c_shift, device=device, ) self.encoder = nn.Sequential( *[ HyperLERPBlock( hidden_dim=hidden_dim, scaler_init=scaler_init, scaler_scale=scaler_scale, alpha_init=alpha_init, alpha_scale=alpha_scale, expansion=expansion, device=device, ) for _ in range(num_blocks) ] ) self.predictor = HyperTanhPolicy( hidden_dim=hidden_dim, action_dim=n_act, scaler_init=1.0, scaler_scale=1.0, device=device, ) noise_scales = ( torch.rand(num_envs, 1, device=device) * (std_max - std_min) + std_min ) self.register_buffer("noise_scales", noise_scales) self.register_buffer("std_min", torch.as_tensor(std_min, device=device)) self.register_buffer("std_max", torch.as_tensor(std_max, device=device)) self.n_envs = num_envs self.device = device def forward(self, obs: torch.Tensor) -> torch.Tensor: x = obs x = self.embedder(x) x = self.encoder(x) x = self.predictor(x) return x def explore( self, obs: torch.Tensor, dones: torch.Tensor = None, deterministic: bool = False ) -> torch.Tensor: # If dones is provided, resample noise for environments that are done if dones is not None and dones.sum() > 0: # Generate new noise scales for done environments (one per environment) new_scales = ( torch.rand(self.n_envs, 1, device=obs.device) * (self.std_max - self.std_min) + self.std_min ) # Update only the noise scales for environments that are done dones_view = dones.view(-1, 1) > 0 self.noise_scales.copy_( torch.where(dones_view, new_scales, self.noise_scales) ) act = self(obs) if deterministic: return act noise = torch.randn_like(act) * self.noise_scales return act + noise class MultiTaskActor(Actor): def __init__(self, num_tasks: int, task_embedding_dim: int, *args, **kwargs): super().__init__(*args, **kwargs) self.num_tasks = num_tasks self.task_embedding_dim = task_embedding_dim self.task_embedding = nn.Embedding( num_tasks, task_embedding_dim, max_norm=1.0, device=self.device ) def forward(self, obs: torch.Tensor) -> torch.Tensor: # TODO: Optimize the code to be compatible with cudagraphs # Currently in-place creation of task_indices is not compatible with cudagraphs task_ids_one_hot = obs[..., -self.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) task_embeddings = self.task_embedding(task_indices) obs = torch.cat([obs[..., : -self.num_tasks], task_embeddings], dim=-1) return super().forward(obs) class MultiTaskCritic(Critic): def __init__(self, num_tasks: int, task_embedding_dim: int, *args, **kwargs): super().__init__(*args, **kwargs) self.num_tasks = num_tasks self.task_embedding_dim = task_embedding_dim self.task_embedding = nn.Embedding( num_tasks, task_embedding_dim, max_norm=1.0, device=self.device ) def forward(self, obs: torch.Tensor, actions: torch.Tensor) -> torch.Tensor: # TODO: Optimize the code to be compatible with cudagraphs # Currently in-place creation of task_indices is not compatible with cudagraphs task_ids_one_hot = obs[..., -self.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) task_embeddings = self.task_embedding(task_indices) obs = torch.cat([obs[..., : -self.num_tasks], task_embeddings], dim=-1) return super().forward(obs, actions) def projection( self, obs: torch.Tensor, actions: torch.Tensor, rewards: torch.Tensor, bootstrap: torch.Tensor, discount: float, ) -> torch.Tensor: task_ids_one_hot = obs[..., -self.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) task_embeddings = self.task_embedding(task_indices) obs = torch.cat([obs[..., : -self.num_tasks], task_embeddings], dim=-1) return super().projection(obs, actions, rewards, bootstrap, discount) ================================================ FILE: fast_td3/fast_td3_utils.py ================================================ import os from typing import Optional import torch import torch.nn as nn import torch.distributed as dist from tensordict import TensorDict class SimpleReplayBuffer(nn.Module): def __init__( self, n_env: int, buffer_size: int, n_obs: int, n_act: int, n_critic_obs: int, asymmetric_obs: bool = False, playground_mode: bool = False, n_steps: int = 1, gamma: float = 0.99, device=None, ): """ A simple replay buffer that stores transitions in a circular buffer. Supports n-step returns and asymmetric observations. When playground_mode=True, critic_observations are treated as a concatenation of regular observations and privileged observations, and only the privileged part is stored to save memory. TODO (Younggyo): Refactor to split this into SimpleReplayBuffer and NStepReplayBuffer """ super().__init__() self.n_env = n_env self.buffer_size = buffer_size self.n_obs = n_obs self.n_act = n_act self.n_critic_obs = n_critic_obs self.asymmetric_obs = asymmetric_obs self.playground_mode = playground_mode and asymmetric_obs self.gamma = gamma self.n_steps = n_steps self.device = device self.observations = torch.zeros( (n_env, buffer_size, n_obs), device=device, dtype=torch.float ) self.actions = torch.zeros( (n_env, buffer_size, n_act), device=device, dtype=torch.float ) self.rewards = torch.zeros( (n_env, buffer_size), device=device, dtype=torch.float ) self.dones = torch.zeros((n_env, buffer_size), device=device, dtype=torch.long) self.truncations = torch.zeros( (n_env, buffer_size), device=device, dtype=torch.long ) self.next_observations = torch.zeros( (n_env, buffer_size, n_obs), device=device, dtype=torch.float ) if asymmetric_obs: if self.playground_mode: # Only store the privileged part of observations (n_critic_obs - n_obs) self.privileged_obs_size = n_critic_obs - n_obs self.privileged_observations = torch.zeros( (n_env, buffer_size, self.privileged_obs_size), device=device, dtype=torch.float, ) self.next_privileged_observations = torch.zeros( (n_env, buffer_size, self.privileged_obs_size), device=device, dtype=torch.float, ) else: # Store full critic observations self.critic_observations = torch.zeros( (n_env, buffer_size, n_critic_obs), device=device, dtype=torch.float ) self.next_critic_observations = torch.zeros( (n_env, buffer_size, n_critic_obs), device=device, dtype=torch.float ) self.ptr = 0 @torch.no_grad() def extend( self, tensor_dict: TensorDict, ): observations = tensor_dict["observations"] actions = tensor_dict["actions"] rewards = tensor_dict["next"]["rewards"] dones = tensor_dict["next"]["dones"] truncations = tensor_dict["next"]["truncations"] next_observations = tensor_dict["next"]["observations"] ptr = self.ptr % self.buffer_size self.observations[:, ptr] = observations self.actions[:, ptr] = actions self.rewards[:, ptr] = rewards self.dones[:, ptr] = dones self.truncations[:, ptr] = truncations self.next_observations[:, ptr] = next_observations if self.asymmetric_obs: critic_observations = tensor_dict["critic_observations"] next_critic_observations = tensor_dict["next"]["critic_observations"] if self.playground_mode: # Extract and store only the privileged part privileged_observations = critic_observations[:, self.n_obs :] next_privileged_observations = next_critic_observations[:, self.n_obs :] self.privileged_observations[:, ptr] = privileged_observations self.next_privileged_observations[:, ptr] = next_privileged_observations else: # Store full critic observations self.critic_observations[:, ptr] = critic_observations self.next_critic_observations[:, ptr] = next_critic_observations self.ptr += 1 @torch.no_grad() def sample(self, batch_size: int): # we will sample n_env * batch_size transitions if self.n_steps == 1: indices = torch.randint( 0, min(self.buffer_size, self.ptr), (self.n_env, batch_size), device=self.device, ) obs_indices = indices.unsqueeze(-1).expand(-1, -1, self.n_obs) act_indices = indices.unsqueeze(-1).expand(-1, -1, self.n_act) observations = torch.gather(self.observations, 1, obs_indices).reshape( self.n_env * batch_size, self.n_obs ) next_observations = torch.gather( self.next_observations, 1, obs_indices ).reshape(self.n_env * batch_size, self.n_obs) actions = torch.gather(self.actions, 1, act_indices).reshape( self.n_env * batch_size, self.n_act ) rewards = torch.gather(self.rewards, 1, indices).reshape( self.n_env * batch_size ) dones = torch.gather(self.dones, 1, indices).reshape( self.n_env * batch_size ) truncations = torch.gather(self.truncations, 1, indices).reshape( self.n_env * batch_size ) effective_n_steps = torch.ones_like(dones) if self.asymmetric_obs: if self.playground_mode: # Gather privileged observations priv_obs_indices = indices.unsqueeze(-1).expand( -1, -1, self.privileged_obs_size ) privileged_observations = torch.gather( self.privileged_observations, 1, priv_obs_indices ).reshape(self.n_env * batch_size, self.privileged_obs_size) next_privileged_observations = torch.gather( self.next_privileged_observations, 1, priv_obs_indices ).reshape(self.n_env * batch_size, self.privileged_obs_size) # Concatenate with regular observations to form full critic observations critic_observations = torch.cat( [observations, privileged_observations], dim=1 ) next_critic_observations = torch.cat( [next_observations, next_privileged_observations], dim=1 ) else: # Gather full critic observations critic_obs_indices = indices.unsqueeze(-1).expand( -1, -1, self.n_critic_obs ) critic_observations = torch.gather( self.critic_observations, 1, critic_obs_indices ).reshape(self.n_env * batch_size, self.n_critic_obs) next_critic_observations = torch.gather( self.next_critic_observations, 1, critic_obs_indices ).reshape(self.n_env * batch_size, self.n_critic_obs) else: # Sample base indices if self.ptr >= self.buffer_size: # When the buffer is full, there is no protection against sampling across different episodes # We avoid this by temporarily setting self.pos - 1 to truncated = True if not done # https://github.com/DLR-RM/stable-baselines3/blob/b91050ca94f8bce7a0285c91f85da518d5a26223/stable_baselines3/common/buffers.py#L857-L860 # TODO (Younggyo): Change the reference when this SB3 branch is merged current_pos = self.ptr % self.buffer_size curr_truncations = self.truncations[:, current_pos - 1].clone() self.truncations[:, current_pos - 1] = torch.logical_not( self.dones[:, current_pos - 1] ) indices = torch.randint( 0, self.buffer_size, (self.n_env, batch_size), device=self.device, ) else: # Buffer not full - ensure n-step sequence doesn't exceed valid data max_start_idx = max(1, self.ptr - self.n_steps + 1) indices = torch.randint( 0, max_start_idx, (self.n_env, batch_size), device=self.device, ) obs_indices = indices.unsqueeze(-1).expand(-1, -1, self.n_obs) act_indices = indices.unsqueeze(-1).expand(-1, -1, self.n_act) # Get base transitions observations = torch.gather(self.observations, 1, obs_indices).reshape( self.n_env * batch_size, self.n_obs ) actions = torch.gather(self.actions, 1, act_indices).reshape( self.n_env * batch_size, self.n_act ) if self.asymmetric_obs: if self.playground_mode: # Gather privileged observations priv_obs_indices = indices.unsqueeze(-1).expand( -1, -1, self.privileged_obs_size ) privileged_observations = torch.gather( self.privileged_observations, 1, priv_obs_indices ).reshape(self.n_env * batch_size, self.privileged_obs_size) # Concatenate with regular observations to form full critic observations critic_observations = torch.cat( [observations, privileged_observations], dim=1 ) else: # Gather full critic observations critic_obs_indices = indices.unsqueeze(-1).expand( -1, -1, self.n_critic_obs ) critic_observations = torch.gather( self.critic_observations, 1, critic_obs_indices ).reshape(self.n_env * batch_size, self.n_critic_obs) # Create sequential indices for each sample # This creates a [n_env, batch_size, n_step] tensor of indices seq_offsets = torch.arange(self.n_steps, device=self.device).view(1, 1, -1) all_indices = ( indices.unsqueeze(-1) + seq_offsets ) % self.buffer_size # [n_env, batch_size, n_step] # Gather all rewards and terminal flags # Using advanced indexing - result shapes: [n_env, batch_size, n_step] all_rewards = torch.gather( self.rewards.unsqueeze(-1).expand(-1, -1, self.n_steps), 1, all_indices ) all_dones = torch.gather( self.dones.unsqueeze(-1).expand(-1, -1, self.n_steps), 1, all_indices ) all_truncations = torch.gather( self.truncations.unsqueeze(-1).expand(-1, -1, self.n_steps), 1, all_indices, ) # Create masks for rewards *after* first done # This creates a cumulative product that zeroes out rewards after the first done all_dones_shifted = torch.cat( [torch.zeros_like(all_dones[:, :, :1]), all_dones[:, :, :-1]], dim=2 ) # First reward should not be masked done_masks = torch.cumprod( 1.0 - all_dones_shifted, dim=2 ) # [n_env, batch_size, n_step] effective_n_steps = done_masks.sum(2) # Create discount factors discounts = torch.pow( self.gamma, torch.arange(self.n_steps, device=self.device) ) # [n_steps] # Apply masks and discounts to rewards masked_rewards = all_rewards * done_masks # [n_env, batch_size, n_step] discounted_rewards = masked_rewards * discounts.view( 1, 1, -1 ) # [n_env, batch_size, n_step] # Sum rewards along the n_step dimension n_step_rewards = discounted_rewards.sum(dim=2) # [n_env, batch_size] # Find index of first done or truncation or last step for each sequence first_done = torch.argmax( (all_dones > 0).float(), dim=2 ) # [n_env, batch_size] first_trunc = torch.argmax( (all_truncations > 0).float(), dim=2 ) # [n_env, batch_size] # Handle case where there are no dones or truncations no_dones = all_dones.sum(dim=2) == 0 no_truncs = all_truncations.sum(dim=2) == 0 # When no dones or truncs, use the last index first_done = torch.where(no_dones, self.n_steps - 1, first_done) first_trunc = torch.where(no_truncs, self.n_steps - 1, first_trunc) # Take the minimum (first) of done or truncation final_indices = torch.minimum( first_done, first_trunc ) # [n_env, batch_size] # Create indices to gather the final next observations final_next_obs_indices = torch.gather( all_indices, 2, final_indices.unsqueeze(-1) ).squeeze( -1 ) # [n_env, batch_size] # Gather final values final_next_observations = self.next_observations.gather( 1, final_next_obs_indices.unsqueeze(-1).expand(-1, -1, self.n_obs) ) final_dones = self.dones.gather(1, final_next_obs_indices) final_truncations = self.truncations.gather(1, final_next_obs_indices) if self.asymmetric_obs: if self.playground_mode: # Gather final privileged observations final_next_privileged_observations = ( self.next_privileged_observations.gather( 1, final_next_obs_indices.unsqueeze(-1).expand( -1, -1, self.privileged_obs_size ), ) ) # Reshape for output next_privileged_observations = ( final_next_privileged_observations.reshape( self.n_env * batch_size, self.privileged_obs_size ) ) # Concatenate with next observations to form full next critic observations next_observations_reshaped = final_next_observations.reshape( self.n_env * batch_size, self.n_obs ) next_critic_observations = torch.cat( [next_observations_reshaped, next_privileged_observations], dim=1, ) else: # Gather final next critic observations directly final_next_critic_observations = ( self.next_critic_observations.gather( 1, final_next_obs_indices.unsqueeze(-1).expand( -1, -1, self.n_critic_obs ), ) ) next_critic_observations = final_next_critic_observations.reshape( self.n_env * batch_size, self.n_critic_obs ) # Reshape everything to batch dimension rewards = n_step_rewards.reshape(self.n_env * batch_size) dones = final_dones.reshape(self.n_env * batch_size) truncations = final_truncations.reshape(self.n_env * batch_size) effective_n_steps = effective_n_steps.reshape(self.n_env * batch_size) next_observations = final_next_observations.reshape( self.n_env * batch_size, self.n_obs ) out = TensorDict( { "observations": observations, "actions": actions, "next": { "rewards": rewards, "dones": dones, "truncations": truncations, "observations": next_observations, "effective_n_steps": effective_n_steps, }, }, batch_size=self.n_env * batch_size, ) if self.asymmetric_obs: out["critic_observations"] = critic_observations out["next"]["critic_observations"] = next_critic_observations if self.n_steps > 1 and self.ptr >= self.buffer_size: # Roll back the truncation flags introduced for safe sampling self.truncations[:, current_pos - 1] = curr_truncations return out class EmpiricalNormalization(nn.Module): """Normalize mean and variance of values based on empirical values.""" def __init__(self, shape, device, eps=1e-2, until=None): """Initialize EmpiricalNormalization module. Args: shape (int or tuple of int): Shape of input values except batch axis. eps (float): Small value for stability. until (int or None): If this arg is specified, the link learns input values until the sum of batch sizes exceeds it. """ super().__init__() self.eps = eps self.until = until self.device = device self.register_buffer("_mean", torch.zeros(shape).unsqueeze(0).to(device)) self.register_buffer("_var", torch.ones(shape).unsqueeze(0).to(device)) self.register_buffer("_std", torch.ones(shape).unsqueeze(0).to(device)) self.register_buffer("count", torch.tensor(0, dtype=torch.long).to(device)) @property def mean(self): return self._mean.squeeze(0).clone() @property def std(self): return self._std.squeeze(0).clone() @torch.no_grad() def forward( self, x: torch.Tensor, center: bool = True, update: bool = True ) -> torch.Tensor: if x.shape[1:] != self._mean.shape[1:]: raise ValueError( f"Expected input of shape (*,{self._mean.shape[1:]}), got {x.shape}" ) if self.training and update: self.update(x) if center: return (x - self._mean) / (self._std + self.eps) else: return x / (self._std + self.eps) @torch.jit.unused def update(self, x): if self.until is not None and self.count >= self.until: return if dist.is_available() and dist.is_initialized(): # Calculate global batch size arithmetically local_batch_size = x.shape[0] world_size = dist.get_world_size() global_batch_size = world_size * local_batch_size # Calculate the stats x_shifted = x - self._mean local_sum_shifted = torch.sum(x_shifted, dim=0, keepdim=True) local_sum_sq_shifted = torch.sum(x_shifted.pow(2), dim=0, keepdim=True) # Sync the stats across all processes stats_to_sync = torch.cat([local_sum_shifted, local_sum_sq_shifted], dim=0) dist.all_reduce(stats_to_sync, op=dist.ReduceOp.SUM) global_sum_shifted, global_sum_sq_shifted = stats_to_sync # Calculate the mean and variance of the global batch batch_mean_shifted = global_sum_shifted / global_batch_size batch_var = ( global_sum_sq_shifted / global_batch_size - batch_mean_shifted.pow(2) ) batch_mean = batch_mean_shifted + self._mean else: global_batch_size = x.shape[0] batch_mean = torch.mean(x, dim=0, keepdim=True) batch_var = torch.var(x, dim=0, keepdim=True, unbiased=False) new_count = self.count + global_batch_size # Update mean delta = batch_mean - self._mean self._mean.copy_(self._mean + delta * (global_batch_size / new_count)) # Update variance delta2 = batch_mean - self._mean m_a = self._var * self.count m_b = batch_var * global_batch_size M2 = m_a + m_b + delta2.pow(2) * (self.count * global_batch_size / new_count) self._var.copy_(M2 / new_count) self._std.copy_(self._var.sqrt()) self.count.copy_(new_count) @torch.jit.unused def inverse(self, y): return y * (self._std + self.eps) + self._mean class RewardNormalizer(nn.Module): def __init__( self, gamma: float, device: torch.device, g_max: float = 10.0, epsilon: float = 1e-8, ): super().__init__() self.register_buffer( "G", torch.zeros(1, device=device) ) # running estimate of the discounted return self.register_buffer("G_r_max", torch.zeros(1, device=device)) # running-max self.G_rms = EmpiricalNormalization(shape=1, device=device) self.gamma = gamma self.g_max = g_max self.epsilon = epsilon def _scale_reward(self, rewards: torch.Tensor) -> torch.Tensor: var_denominator = self.G_rms.std[0] + self.epsilon min_required_denominator = self.G_r_max / self.g_max denominator = torch.maximum(var_denominator, min_required_denominator) return rewards / denominator def update_stats( self, rewards: torch.Tensor, dones: torch.Tensor, ): self.G = self.gamma * (1 - dones) * self.G + rewards self.G_rms.update(self.G.view(-1, 1)) local_max = torch.max(torch.abs(self.G)) if dist.is_available() and dist.is_initialized(): dist.all_reduce(local_max, op=dist.ReduceOp.MAX) self.G_r_max = max(self.G_r_max, local_max) def forward(self, rewards: torch.Tensor) -> torch.Tensor: return self._scale_reward(rewards) class PerTaskEmpiricalNormalization(nn.Module): """Normalize mean and variance of values based on empirical values for each task.""" def __init__( self, num_tasks: int, shape: tuple, device: torch.device, eps: float = 1e-2, until: int = None, ): """ Initialize PerTaskEmpiricalNormalization module. Args: num_tasks (int): The total number of tasks. shape (int or tuple of int): Shape of input values except batch axis. eps (float): Small value for stability. until (int or None): If specified, learns until the sum of batch sizes for a specific task exceeds this value. """ super().__init__() if not isinstance(shape, tuple): shape = (shape,) self.num_tasks = num_tasks self.shape = shape self.eps = eps self.until = until self.device = device # Buffers now have a leading dimension for tasks self.register_buffer("_mean", torch.zeros(num_tasks, *shape).to(device)) self.register_buffer("_var", torch.ones(num_tasks, *shape).to(device)) self.register_buffer("_std", torch.ones(num_tasks, *shape).to(device)) self.register_buffer( "count", torch.zeros(num_tasks, dtype=torch.long).to(device) ) def forward( self, x: torch.Tensor, task_ids: torch.Tensor, center: bool = True ) -> torch.Tensor: """ Normalize the input tensor `x` using statistics for the given `task_ids`. Args: x (torch.Tensor): Input tensor of shape [num_envs, *shape]. task_ids (torch.Tensor): Tensor of task indices, shape [num_envs]. center (bool): If True, center the data by subtracting the mean. """ if x.shape[1:] != self.shape: raise ValueError(f"Expected input shape (*, {self.shape}), got {x.shape}") if x.shape[0] != task_ids.shape[0]: raise ValueError("Batch size of x and task_ids must match.") # Gather the stats for the tasks in the current batch # Reshape task_ids for broadcasting: [num_envs] -> [num_envs, 1, ...] view_shape = (task_ids.shape[0],) + (1,) * len(self.shape) task_ids_expanded = task_ids.view(view_shape).expand_as(x) mean = self._mean.gather(0, task_ids_expanded) std = self._std.gather(0, task_ids_expanded) if self.training: self.update(x, task_ids) if center: return (x - mean) / (std + self.eps) else: return x / (std + self.eps) @torch.jit.unused def update(self, x: torch.Tensor, task_ids: torch.Tensor): """Update running statistics for the tasks present in the batch.""" unique_tasks = torch.unique(task_ids) for task_id in unique_tasks: if self.until is not None and self.count[task_id] >= self.until: continue # Create a mask to select data for the current task mask = task_ids == task_id x_task = x[mask] batch_size = x_task.shape[0] if batch_size == 0: continue # Update count for this task old_count = self.count[task_id].clone() new_count = old_count + batch_size # Update mean task_mean = self._mean[task_id] batch_mean = torch.mean(x_task, dim=0) delta = batch_mean - task_mean self._mean[task_id].copy_(task_mean + (batch_size / new_count) * delta) # Update variance using Chan's parallel algorithm if old_count > 0: batch_var = torch.var(x_task, dim=0, unbiased=False) m_a = self._var[task_id] * old_count m_b = batch_var * batch_size M2 = m_a + m_b + (delta**2) * (old_count * batch_size / new_count) self._var[task_id].copy_(M2 / new_count) else: # For the first batch of this task self._var[task_id].copy_(torch.var(x_task, dim=0, unbiased=False)) self._std[task_id].copy_(torch.sqrt(self._var[task_id])) self.count[task_id].copy_(new_count) class PerTaskRewardNormalizer(nn.Module): def __init__( self, num_tasks: int, gamma: float, device: torch.device, g_max: float = 10.0, epsilon: float = 1e-8, ): """ Per-task reward normalizer, motivation comes from BRC (https://arxiv.org/abs/2505.23150v1) """ super().__init__() self.num_tasks = num_tasks self.gamma = gamma self.g_max = g_max self.epsilon = epsilon self.device = device # Per-task running estimate of the discounted return self.register_buffer("G", torch.zeros(num_tasks, device=device)) # Per-task running-max of the discounted return self.register_buffer("G_r_max", torch.zeros(num_tasks, device=device)) # Use the new per-task normalizer for the statistics of G self.G_rms = PerTaskEmpiricalNormalization( num_tasks=num_tasks, shape=(1,), device=device ) def _scale_reward( self, rewards: torch.Tensor, task_ids: torch.Tensor ) -> torch.Tensor: """ Scales rewards using per-task statistics. Args: rewards (torch.Tensor): Reward tensor, shape [num_envs]. task_ids (torch.Tensor): Task indices, shape [num_envs]. """ # Gather stats for the tasks in the batch std_for_batch = self.G_rms._std.gather(0, task_ids.unsqueeze(-1)).squeeze(-1) g_r_max_for_batch = self.G_r_max.gather(0, task_ids) var_denominator = std_for_batch + self.epsilon min_required_denominator = g_r_max_for_batch / self.g_max denominator = torch.maximum(var_denominator, min_required_denominator) # Add a small epsilon to the final denominator to prevent division by zero # in case g_r_max is also zero. return rewards / (denominator + self.epsilon) def update_stats( self, rewards: torch.Tensor, dones: torch.Tensor, task_ids: torch.Tensor ): """ Updates the running discounted return and its statistics for each task. Args: rewards (torch.Tensor): Reward tensor, shape [num_envs]. dones (torch.Tensor): Done tensor, shape [num_envs]. task_ids (torch.Tensor): Task indices, shape [num_envs]. """ if not (rewards.shape == dones.shape == task_ids.shape): raise ValueError("rewards, dones, and task_ids must have the same shape.") # === Update G (running discounted return) === # Gather the previous G values for the tasks in the batch prev_G = self.G.gather(0, task_ids) # Update G for each environment based on its own reward and done signal new_G = self.gamma * (1 - dones.float()) * prev_G + rewards # Scatter the updated G values back to the main buffer self.G.scatter_(0, task_ids, new_G) # === Update G_rms (statistics of G) === # The update function handles the per-task logic internally self.G_rms.update(new_G.unsqueeze(-1), task_ids) # === Update G_r_max (running max of |G|) === prev_G_r_max = self.G_r_max.gather(0, task_ids) # Update the max for each environment updated_G_r_max = torch.maximum(prev_G_r_max, torch.abs(new_G)) # Scatter the new maxes back to the main buffer self.G_r_max.scatter_(0, task_ids, updated_G_r_max) def forward(self, rewards: torch.Tensor, task_ids: torch.Tensor) -> torch.Tensor: """ Normalizes rewards. During training, it also updates the running statistics. Args: rewards (torch.Tensor): Reward tensor, shape [num_envs]. task_ids (torch.Tensor): Task indices, shape [num_envs]. """ return self._scale_reward(rewards, task_ids) def cpu_state(sd): # detach & move to host without locking the compute stream return {k: v.detach().to("cpu", non_blocking=True) for k, v in sd.items()} def save_params( global_step, actor, qnet, qnet_target, obs_normalizer, critic_obs_normalizer, args, save_path, ): """Save model parameters and training configuration to disk.""" def get_ddp_state_dict(model): """Get state dict from model, handling DDP wrapper if present.""" if hasattr(model, "module"): return model.module.state_dict() return model.state_dict() os.makedirs(os.path.dirname(save_path), exist_ok=True) save_dict = { "actor_state_dict": cpu_state(get_ddp_state_dict(actor)), "qnet_state_dict": cpu_state(get_ddp_state_dict(qnet)), "qnet_target_state_dict": cpu_state(get_ddp_state_dict(qnet_target)), "obs_normalizer_state": ( cpu_state(obs_normalizer.state_dict()) if hasattr(obs_normalizer, "state_dict") else None ), "critic_obs_normalizer_state": ( cpu_state(critic_obs_normalizer.state_dict()) if hasattr(critic_obs_normalizer, "state_dict") else None ), "args": vars(args), # Save all arguments "global_step": global_step, } torch.save(save_dict, save_path, _use_new_zipfile_serialization=True) print(f"Saved parameters and configuration to {save_path}") def get_ddp_state_dict(model): """Get state dict from model, handling DDP wrapper if present.""" if hasattr(model, "module"): return model.module.state_dict() return model.state_dict() def load_ddp_state_dict(model, state_dict): """Load state dict into model, handling DDP wrapper if present.""" if hasattr(model, "module"): model.module.load_state_dict(state_dict) else: model.load_state_dict(state_dict) @torch.no_grad() def mark_step(): # call this once per iteration *before* any compiled function torch.compiler.cudagraph_mark_step_begin() ================================================ FILE: fast_td3/hyperparams.py ================================================ import os from dataclasses import dataclass import tyro @dataclass class BaseArgs: # Default hyperparameters -- specifically for HumanoidBench # See MuJoCoPlaygroundArgs for default hyperparameters for MuJoCo Playground # See IsaacLabArgs for default hyperparameters for IsaacLab env_name: str = "h1hand-stand-v0" """the id of the environment""" agent: str = "fasttd3" """the agent to use: currently support [fasttd3, fasttd3_simbav2]""" seed: int = 1 """seed of the experiment""" torch_deterministic: bool = True """if toggled, `torch.backends.cudnn.deterministic=False`""" cuda: bool = True """if toggled, cuda will be enabled by default""" device_rank: int = 0 """the rank of the device""" exp_name: str = os.path.basename(__file__)[: -len(".py")] """the name of this experiment""" project: str = "FastTD3" """the project name""" use_wandb: bool = True """whether to use wandb""" checkpoint_path: str = None """the path to the checkpoint file""" num_envs: int = 128 """the number of environments to run in parallel""" num_eval_envs: int = 128 """the number of evaluation environments to run in parallel (only valid for MuJoCo Playground)""" total_timesteps: int = 150000 """total timesteps of the experiments""" critic_learning_rate: float = 3e-4 """the learning rate of the critic""" actor_learning_rate: float = 3e-4 """the learning rate for the actor""" critic_learning_rate_end: float = 3e-4 """the learning rate of the critic at the end of training""" actor_learning_rate_end: float = 3e-4 """the learning rate for the actor at the end of training""" buffer_size: int = 1024 * 50 """the replay memory buffer size""" num_steps: int = 1 """the number of steps to use for the multi-step return""" gamma: float = 0.99 """the discount factor gamma""" tau: float = 0.1 """target smoothing coefficient (default: 0.005)""" batch_size: int = 32768 """the batch size of sample from the replay memory""" policy_noise: float = 0.001 """the scale of policy noise""" std_min: float = 0.001 """the minimum scale of noise""" std_max: float = 0.4 """the maximum scale of noise""" learning_starts: int = 10 """timestep to start learning""" policy_frequency: int = 2 """the frequency of training policy (delayed)""" noise_clip: float = 0.5 """noise clip parameter of the Target Policy Smoothing Regularization""" num_updates: int = 2 """the number of updates to perform per step""" init_scale: float = 0.01 """the scale of the initial parameters""" num_atoms: int = 101 """the number of atoms""" v_min: float = -250.0 """the minimum value of the support""" v_max: float = 250.0 """the maximum value of the support""" critic_hidden_dim: int = 1024 """the hidden dimension of the critic network""" actor_hidden_dim: int = 512 """the hidden dimension of the actor network""" critic_num_blocks: int = 2 """(SimbaV2 only) the number of blocks in the critic network""" actor_num_blocks: int = 1 """(SimbaV2 only) the number of blocks in the actor network""" use_cdq: bool = True """whether to use Clipped Double Q-learning""" measure_burnin: int = 3 """Number of burn-in iterations for speed measure.""" eval_interval: int = 5000 """the interval to evaluate the model""" render_interval: int = 5000 """the interval to render the model""" compile: bool = True """whether to use torch.compile.""" compile_mode: str = "reduce-overhead" """the mode of torch.compile.""" obs_normalization: bool = True """whether to enable observation normalization""" reward_normalization: bool = False """whether to enable reward normalization""" use_grad_norm_clipping: bool = False """whether to use gradient norm clipping.""" max_grad_norm: float = 0.0 """the maximum gradient norm""" amp: bool = True """whether to use amp""" amp_dtype: str = "bf16" """the dtype of the amp""" disable_bootstrap: bool = False """Whether to disable bootstrap in the critic learning""" use_domain_randomization: bool = False """(Playground only) whether to use domain randomization""" use_push_randomization: bool = False """(Playground only) whether to use push randomization""" use_tuned_reward: bool = False """(Playground only) Use tuned reward for G1""" action_bounds: float = 1.0 """(IsaacLab only) the bounds of the action space (-action_bounds, action_bounds)""" task_embedding_dim: int = 32 """the dimension of the task embedding""" weight_decay: float = 0.1 """the weight decay of the optimizer""" save_interval: int = 5000 """the interval to save the model""" def get_args(): """ Parse command-line arguments and return the appropriate Args instance based on env_name. """ # First, parse all arguments using the base Args class base_args = tyro.cli(BaseArgs) # Map environment names to their specific Args classes # For tasks not here, default hyperparameters are used # See below links for available task list # - HumanoidBench (https://arxiv.org/abs/2403.10506) # - IsaacLab (https://isaac-sim.github.io/IsaacLab/main/source/overview/environments.html) # - MuJoCo Playground (https://arxiv.org/abs/2502.08844) env_to_args_class = { # HumanoidBench # NOTE: These tasks are not full list of HumanoidBench tasks "h1hand-reach-v0": H1HandReachArgs, "h1hand-balance-simple-v0": H1HandBalanceSimpleArgs, "h1hand-balance-hard-v0": H1HandBalanceHardArgs, "h1hand-pole-v0": H1HandPoleArgs, "h1hand-truck-v0": H1HandTruckArgs, "h1hand-maze-v0": H1HandMazeArgs, "h1hand-push-v0": H1HandPushArgs, "h1hand-basketball-v0": H1HandBasketballArgs, "h1hand-window-v0": H1HandWindowArgs, "h1hand-package-v0": H1HandPackageArgs, "h1hand-truck-v0": H1HandTruckArgs, # MuJoCo Playground # NOTE: These tasks are not full list of MuJoCo Playground tasks "G1JoystickFlatTerrain": G1JoystickFlatTerrainArgs, "G1JoystickRoughTerrain": G1JoystickRoughTerrainArgs, "T1JoystickFlatTerrain": T1JoystickFlatTerrainArgs, "T1JoystickRoughTerrain": T1JoystickRoughTerrainArgs, "LeapCubeReorient": LeapCubeReorientArgs, "LeapCubeRotateZAxis": LeapCubeRotateZAxisArgs, "Go1JoystickFlatTerrain": Go1JoystickFlatTerrainArgs, "Go1JoystickRoughTerrain": Go1JoystickRoughTerrainArgs, "Go1Getup": Go1GetupArgs, "CheetahRun": CheetahRunArgs, # NOTE: Example config for DeepMind Control Suite # IsaacLab # NOTE: These tasks are not full list of IsaacLab tasks "Isaac-Lift-Cube-Franka-v0": IsaacLiftCubeFrankaArgs, "Isaac-Open-Drawer-Franka-v0": IsaacOpenDrawerFrankaArgs, "Isaac-Velocity-Flat-H1-v0": IsaacVelocityFlatH1Args, "Isaac-Velocity-Flat-G1-v0": IsaacVelocityFlatG1Args, "Isaac-Velocity-Rough-H1-v0": IsaacVelocityRoughH1Args, "Isaac-Velocity-Rough-G1-v0": IsaacVelocityRoughG1Args, "Isaac-Repose-Cube-Allegro-Direct-v0": IsaacReposeCubeAllegroDirectArgs, "Isaac-Repose-Cube-Shadow-Direct-v0": IsaacReposeCubeShadowDirectArgs, # MTBench "MTBench-meta-world-v2-mt10": MetaWorldMT10Args, "MTBench-meta-world-v2-mt50": MetaWorldMT50Args, } # If the provided env_name has a specific Args class, use it if base_args.env_name in env_to_args_class: specific_args_class = env_to_args_class[base_args.env_name] # Re-parse with the specific class, maintaining any user overrides specific_args = tyro.cli(specific_args_class) return specific_args if base_args.env_name.startswith("h1hand-") or base_args.env_name.startswith("h1-"): # HumanoidBench specific_args = tyro.cli(HumanoidBenchArgs) elif base_args.env_name.startswith("Isaac-"): # IsaacLab specific_args = tyro.cli(IsaacLabArgs) elif base_args.env_name.startswith("MTBench-"): # MTBench specific_args = tyro.cli(MTBenchArgs) else: # MuJoCo Playground specific_args = tyro.cli(MuJoCoPlaygroundArgs) return specific_args @dataclass class HumanoidBenchArgs(BaseArgs): # See HumanoidBench (https://arxiv.org/abs/2403.10506) for available task list total_timesteps: int = 100000 @dataclass class H1HandReachArgs(HumanoidBenchArgs): env_name: str = "h1hand-reach-v0" v_min: float = -2000.0 v_max: float = 2000.0 @dataclass class H1HandBalanceSimpleArgs(HumanoidBenchArgs): env_name: str = "h1hand-balance-simple-v0" total_timesteps: int = 200000 @dataclass class H1HandBalanceHardArgs(HumanoidBenchArgs): env_name: str = "h1hand-balance-hard-v0" total_timesteps: int = 1000000 @dataclass class H1HandPoleArgs(HumanoidBenchArgs): env_name: str = "h1hand-pole-v0" total_timesteps: int = 150000 @dataclass class H1HandTruckArgs(HumanoidBenchArgs): env_name: str = "h1hand-truck-v0" total_timesteps: int = 500000 @dataclass class H1HandMazeArgs(HumanoidBenchArgs): env_name: str = "h1hand-maze-v0" v_min: float = -1000.0 v_max: float = 1000.0 @dataclass class H1HandPushArgs(HumanoidBenchArgs): env_name: str = "h1hand-push-v0" v_min: float = -1000.0 v_max: float = 1000.0 total_timesteps: int = 1000000 @dataclass class H1HandBasketballArgs(HumanoidBenchArgs): env_name: str = "h1hand-basketball-v0" v_min: float = -2000.0 v_max: float = 2000.0 total_timesteps: int = 250000 @dataclass class H1HandWindowArgs(HumanoidBenchArgs): env_name: str = "h1hand-window-v0" total_timesteps: int = 250000 @dataclass class H1HandPackageArgs(HumanoidBenchArgs): env_name: str = "h1hand-package-v0" v_min: float = -10000.0 v_max: float = 10000.0 @dataclass class H1HandTruckArgs(HumanoidBenchArgs): env_name: str = "h1hand-truck-v0" v_min: float = -1000.0 v_max: float = 1000.0 @dataclass class MuJoCoPlaygroundArgs(BaseArgs): # Default hyperparameters for many of Playground environments v_min: float = -10.0 v_max: float = 10.0 buffer_size: int = 1024 * 10 num_envs: int = 1024 num_eval_envs: int = 1024 gamma: float = 0.97 @dataclass class MTBenchArgs(BaseArgs): # Default hyperparameters for MTBench reward_normalization: bool = True v_min: float = -10.0 v_max: float = 10.0 buffer_size: int = 2048 # 2K is usually enough for MTBench num_envs: int = 4096 num_eval_envs: int = 4096 gamma: float = 0.97 num_steps: int = 8 compile_mode: str = "default" # Multi-task training is not compatible with cudagraphs @dataclass class MetaWorldMT10Args(MTBenchArgs): # This config achieves 97 ~ 98% success rate within 10k steps (15-20 mins on A100) env_name: str = "MTBench-meta-world-v2-mt10" num_envs: int = 4096 num_eval_envs: int = 4096 num_steps: int = 8 gamma: float = 0.97 compile_mode: str = "default" # Multi-task training is not compatible with cudagraphs @dataclass class MetaWorldMT50Args(MTBenchArgs): # FastTD3 + SimbaV2 achieves >90% success rate within 20k steps (80 mins on A100) # Performance further improves with more training steps, slowly. env_name: str = "MTBench-meta-world-v2-mt50" num_envs: int = 8192 num_eval_envs: int = 8192 num_steps: int = 8 gamma: float = 0.99 compile_mode: str = "default" # Multi-task training is not compatible with cudagraphs @dataclass class G1JoystickFlatTerrainArgs(MuJoCoPlaygroundArgs): env_name: str = "G1JoystickFlatTerrain" total_timesteps: int = 100000 @dataclass class G1JoystickRoughTerrainArgs(MuJoCoPlaygroundArgs): env_name: str = "G1JoystickRoughTerrain" total_timesteps: int = 100000 @dataclass class T1JoystickFlatTerrainArgs(MuJoCoPlaygroundArgs): env_name: str = "T1JoystickFlatTerrain" total_timesteps: int = 100000 @dataclass class T1JoystickRoughTerrainArgs(MuJoCoPlaygroundArgs): env_name: str = "T1JoystickRoughTerrain" total_timesteps: int = 100000 @dataclass class T1LowDofJoystickFlatTerrainArgs(MuJoCoPlaygroundArgs): env_name: str = "T1LowDofJoystickFlatTerrain" total_timesteps: int = 1000000 @dataclass class T1LowDofJoystickRoughTerrainArgs(MuJoCoPlaygroundArgs): env_name: str = "T1LowDofJoystickRoughTerrain" total_timesteps: int = 1000000 @dataclass class CheetahRunArgs(MuJoCoPlaygroundArgs): # NOTE: This config will work for most DMC tasks, though we haven't tested DMC extensively. # Future research can consider using LayerNorm as we find it sometimes works better for DMC tasks. env_name: str = "CheetahRun" num_steps: int = 3 v_min: float = -500.0 v_max: float = 500.0 std_min: float = 0.1 policy_noise: float = 0.1 @dataclass class Go1JoystickFlatTerrainArgs(MuJoCoPlaygroundArgs): env_name: str = "Go1JoystickFlatTerrain" total_timesteps: int = 50000 std_min: float = 0.2 std_max: float = 0.8 policy_noise: float = 0.2 num_updates: int = 8 @dataclass class Go1JoystickRoughTerrainArgs(MuJoCoPlaygroundArgs): env_name: str = "Go1JoystickRoughTerrain" total_timesteps: int = 50000 std_min: float = 0.2 std_max: float = 0.8 policy_noise: float = 0.2 num_updates: int = 8 @dataclass class Go1GetupArgs(MuJoCoPlaygroundArgs): env_name: str = "Go1Getup" total_timesteps: int = 50000 std_min: float = 0.2 std_max: float = 0.8 policy_noise: float = 0.2 num_updates: int = 8 @dataclass class LeapCubeReorientArgs(MuJoCoPlaygroundArgs): env_name: str = "LeapCubeReorient" num_steps: int = 3 gamma: float = 0.99 policy_noise: float = 0.2 v_min: float = -50.0 v_max: float = 50.0 use_cdq: bool = False @dataclass class LeapCubeRotateZAxisArgs(MuJoCoPlaygroundArgs): env_name: str = "LeapCubeRotateZAxis" num_steps: int = 1 policy_noise: float = 0.2 gamma: float = 0.99 v_min: float = -10.0 v_max: float = 10.0 use_cdq: bool = False @dataclass class IsaacLabArgs(BaseArgs): v_min: float = -10.0 v_max: float = 10.0 buffer_size: int = 1024 * 10 num_envs: int = 4096 num_eval_envs: int = 4096 action_bounds: float = 1.0 std_max: float = 0.4 num_atoms: int = 251 render_interval: int = 0 # IsaacLab does not support rendering in our codebase total_timesteps: int = 100000 @dataclass class IsaacLiftCubeFrankaArgs(IsaacLabArgs): # Value learning is unstable for Lift Cube task Due to brittle reward shaping # Therefore, we need to disable bootstrap from 'reset_obs' in IsaacLab # Higher UTD works better for manipulation tasks env_name: str = "Isaac-Lift-Cube-Franka-v0" num_updates: int = 8 v_min: float = -50.0 v_max: float = 50.0 std_max: float = 0.8 num_envs: int = 1024 num_eval_envs: int = 1024 action_bounds: float = 3.0 disable_bootstrap: bool = True total_timesteps: int = 20000 @dataclass class IsaacOpenDrawerFrankaArgs(IsaacLabArgs): # Higher UTD works better for manipulation tasks env_name: str = "Isaac-Open-Drawer-Franka-v0" v_min: float = -50.0 v_max: float = 50.0 num_updates: int = 8 action_bounds: float = 3.0 total_timesteps: int = 20000 @dataclass class IsaacVelocityFlatH1Args(IsaacLabArgs): env_name: str = "Isaac-Velocity-Flat-H1-v0" num_steps: int = 8 num_updates: int = 4 total_timesteps: int = 75000 @dataclass class IsaacVelocityFlatG1Args(IsaacLabArgs): env_name: str = "Isaac-Velocity-Flat-G1-v0" num_steps: int = 8 num_updates: int = 4 total_timesteps: int = 50000 @dataclass class IsaacVelocityRoughH1Args(IsaacLabArgs): env_name: str = "Isaac-Velocity-Rough-H1-v0" num_steps: int = 8 num_updates: int = 4 buffer_size: int = 1024 * 5 # To reduce memory usage total_timesteps: int = 50000 @dataclass class IsaacVelocityRoughG1Args(IsaacLabArgs): env_name: str = "Isaac-Velocity-Rough-G1-v0" num_steps: int = 8 num_updates: int = 4 buffer_size: int = 1024 * 5 # To reduce memory usage total_timesteps: int = 50000 @dataclass class IsaacReposeCubeAllegroDirectArgs(IsaacLabArgs): env_name: str = "Isaac-Repose-Cube-Allegro-Direct-v0" total_timesteps: int = 100000 v_min: float = -500.0 v_max: float = 500.0 @dataclass class IsaacReposeCubeShadowDirectArgs(IsaacLabArgs): env_name: str = "Isaac-Repose-Cube-Shadow-Direct-v0" total_timesteps: int = 100000 v_min: float = -500.0 v_max: float = 500.0 ================================================ FILE: fast_td3/train.py ================================================ import os import sys os.environ["TORCHDYNAMO_INLINE_INBUILT_NN_MODULES"] = "1" os.environ["OMP_NUM_THREADS"] = "1" if sys.platform != "darwin": os.environ["MUJOCO_GL"] = "egl" else: os.environ["MUJOCO_GL"] = "glfw" os.environ["XLA_PYTHON_CLIENT_PREALLOCATE"] = "false" os.environ["JAX_DEFAULT_MATMUL_PRECISION"] = "highest" import random import time import math import tqdm import wandb import numpy as np try: # Required for avoiding IsaacGym import error import isaacgym except ImportError: pass import torch import torch.nn as nn import torch.nn.functional as F import torch.optim as optim from torch.amp import autocast, GradScaler from tensordict import TensorDict from fast_td3_utils import ( EmpiricalNormalization, RewardNormalizer, PerTaskRewardNormalizer, SimpleReplayBuffer, save_params, mark_step, ) from hyperparams import get_args torch.set_float32_matmul_precision("high") try: import jax.numpy as jnp except ImportError: pass def main(): args = get_args() print(args) run_name = f"{args.env_name}__{args.exp_name}__{args.seed}" amp_enabled = args.amp and args.cuda and torch.cuda.is_available() amp_device_type = ( "cuda" if args.cuda and torch.cuda.is_available() else "mps" if args.cuda and torch.backends.mps.is_available() else "cpu" ) amp_dtype = torch.bfloat16 if args.amp_dtype == "bf16" else torch.float16 scaler = GradScaler(enabled=amp_enabled and amp_dtype == torch.float16) if args.use_wandb: wandb.init( project=args.project, name=run_name, config=vars(args), save_code=True, ) random.seed(args.seed) np.random.seed(args.seed) torch.manual_seed(args.seed) torch.backends.cudnn.deterministic = args.torch_deterministic if not args.cuda: device = torch.device("cpu") else: if torch.cuda.is_available(): device = torch.device(f"cuda:{args.device_rank}") elif torch.backends.mps.is_available(): device = torch.device(f"mps:{args.device_rank}") else: raise ValueError("No GPU available") print(f"Using device: {device}") if args.env_name.startswith("h1hand-") or args.env_name.startswith("h1-"): from environments.humanoid_bench_env import HumanoidBenchEnv env_type = "humanoid_bench" envs = HumanoidBenchEnv(args.env_name, args.num_envs, device=device) eval_envs = envs render_env = HumanoidBenchEnv( args.env_name, 1, render_mode="rgb_array", device=device ) elif args.env_name.startswith("Isaac-"): from environments.isaaclab_env import IsaacLabEnv env_type = "isaaclab" envs = IsaacLabEnv( args.env_name, device.type, args.num_envs, args.seed, action_bounds=args.action_bounds, ) eval_envs = envs render_env = envs elif args.env_name.startswith("MTBench-"): from environments.mtbench_env import MTBenchEnv env_name = "-".join(args.env_name.split("-")[1:]) env_type = "mtbench" envs = MTBenchEnv(env_name, args.device_rank, args.num_envs, args.seed) eval_envs = envs render_env = envs else: from environments.mujoco_playground_env import make_env # TODO: Check if re-using same envs for eval could reduce memory usage env_type = "mujoco_playground" envs, eval_envs, render_env = make_env( args.env_name, args.seed, args.num_envs, args.num_eval_envs, args.device_rank, use_tuned_reward=args.use_tuned_reward, use_domain_randomization=args.use_domain_randomization, use_push_randomization=args.use_push_randomization, ) n_act = envs.num_actions n_obs = envs.num_obs if type(envs.num_obs) == int else envs.num_obs[0] if envs.asymmetric_obs: n_critic_obs = ( envs.num_privileged_obs if type(envs.num_privileged_obs) == int else envs.num_privileged_obs[0] ) else: n_critic_obs = n_obs action_low, action_high = -1.0, 1.0 if args.obs_normalization: obs_normalizer = EmpiricalNormalization(shape=n_obs, device=device) critic_obs_normalizer = EmpiricalNormalization( shape=n_critic_obs, device=device ) else: obs_normalizer = nn.Identity() critic_obs_normalizer = nn.Identity() if args.reward_normalization: if env_type in ["mtbench"]: reward_normalizer = PerTaskRewardNormalizer( num_tasks=envs.num_tasks, gamma=args.gamma, device=device, g_max=min(abs(args.v_min), abs(args.v_max)), ) else: reward_normalizer = RewardNormalizer( gamma=args.gamma, device=device, g_max=min(abs(args.v_min), abs(args.v_max)), ) else: reward_normalizer = nn.Identity() actor_kwargs = { "n_obs": n_obs, "n_act": n_act, "num_envs": args.num_envs, "device": device, "init_scale": args.init_scale, "hidden_dim": args.actor_hidden_dim, "std_min": args.std_min, "std_max": args.std_max, } critic_kwargs = { "n_obs": n_critic_obs, "n_act": n_act, "num_atoms": args.num_atoms, "v_min": args.v_min, "v_max": args.v_max, "hidden_dim": args.critic_hidden_dim, "device": device, } if env_type == "mtbench": actor_kwargs["n_obs"] = n_obs - envs.num_tasks + args.task_embedding_dim critic_kwargs["n_obs"] = n_critic_obs - envs.num_tasks + args.task_embedding_dim actor_kwargs["num_tasks"] = envs.num_tasks actor_kwargs["task_embedding_dim"] = args.task_embedding_dim critic_kwargs["num_tasks"] = envs.num_tasks critic_kwargs["task_embedding_dim"] = args.task_embedding_dim if args.agent == "fasttd3": if env_type in ["mtbench"]: from fast_td3 import MultiTaskActor, MultiTaskCritic actor_cls = MultiTaskActor critic_cls = MultiTaskCritic else: from fast_td3 import Actor, Critic actor_cls = Actor critic_cls = Critic print("Using FastTD3") elif args.agent == "fasttd3_simbav2": if env_type in ["mtbench"]: from fast_td3_simbav2 import MultiTaskActor, MultiTaskCritic actor_cls = MultiTaskActor critic_cls = MultiTaskCritic else: from fast_td3_simbav2 import Actor, Critic actor_cls = Actor critic_cls = Critic print("Using FastTD3 + SimbaV2") actor_kwargs.pop("init_scale") actor_kwargs.update( { "scaler_init": math.sqrt(2.0 / args.actor_hidden_dim), "scaler_scale": math.sqrt(2.0 / args.actor_hidden_dim), "alpha_init": 1.0 / (args.actor_num_blocks + 1), "alpha_scale": 1.0 / math.sqrt(args.actor_hidden_dim), "expansion": 4, "c_shift": 3.0, "num_blocks": args.actor_num_blocks, } ) critic_kwargs.update( { "scaler_init": math.sqrt(2.0 / args.critic_hidden_dim), "scaler_scale": math.sqrt(2.0 / args.critic_hidden_dim), "alpha_init": 1.0 / (args.critic_num_blocks + 1), "alpha_scale": 1.0 / math.sqrt(args.critic_hidden_dim), "num_blocks": args.critic_num_blocks, "expansion": 4, "c_shift": 3.0, } ) else: raise ValueError(f"Agent {args.agent} not supported") actor = actor_cls(**actor_kwargs) if env_type in ["mtbench"]: # Python 3.8 doesn't support 'from_module' in tensordict policy = actor.explore else: from tensordict import from_module actor_detach = actor_cls(**actor_kwargs) # Copy params to actor_detach without grad from_module(actor).data.to_module(actor_detach) policy = actor_detach.explore qnet = critic_cls(**critic_kwargs) qnet_target = critic_cls(**critic_kwargs) qnet_target.load_state_dict(qnet.state_dict()) q_optimizer = optim.AdamW( list(qnet.parameters()), lr=torch.tensor(args.critic_learning_rate, device=device), weight_decay=args.weight_decay, ) actor_optimizer = optim.AdamW( list(actor.parameters()), lr=torch.tensor(args.actor_learning_rate, device=device), weight_decay=args.weight_decay, ) # Add learning rate schedulers q_scheduler = optim.lr_scheduler.CosineAnnealingLR( q_optimizer, T_max=args.total_timesteps, eta_min=torch.tensor(args.critic_learning_rate_end, device=device), ) actor_scheduler = optim.lr_scheduler.CosineAnnealingLR( actor_optimizer, T_max=args.total_timesteps, eta_min=torch.tensor(args.actor_learning_rate_end, device=device), ) rb = SimpleReplayBuffer( n_env=args.num_envs, buffer_size=args.buffer_size, n_obs=n_obs, n_act=n_act, n_critic_obs=n_critic_obs, asymmetric_obs=envs.asymmetric_obs, playground_mode=env_type == "mujoco_playground", n_steps=args.num_steps, gamma=args.gamma, device=device, ) policy_noise = args.policy_noise noise_clip = args.noise_clip def evaluate(): num_eval_envs = eval_envs.num_envs episode_returns = torch.zeros(num_eval_envs, device=device) episode_lengths = torch.zeros(num_eval_envs, device=device) done_masks = torch.zeros(num_eval_envs, dtype=torch.bool, device=device) if env_type == "isaaclab": obs = eval_envs.reset(random_start_init=False) else: obs = eval_envs.reset() # Run for a fixed number of steps for i in range(eval_envs.max_episode_steps): with torch.no_grad(), autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): obs = normalize_obs(obs, update=False) actions = actor(obs) next_obs, rewards, dones, infos = eval_envs.step(actions.float()) if env_type == "mtbench": # We only report success rate in MTBench evaluation rewards = ( infos["episode"]["success"].float() if "episode" in infos else 0.0 ) episode_returns = torch.where( ~done_masks, episode_returns + rewards, episode_returns ) episode_lengths = torch.where( ~done_masks, episode_lengths + 1, episode_lengths ) if env_type == "mtbench" and "episode" in infos: dones = dones | infos["episode"]["success"] done_masks = torch.logical_or(done_masks, dones) if done_masks.all(): break obs = next_obs return episode_returns.mean().item(), episode_lengths.mean().item() def render_with_rollout(): # Quick rollout for rendering if env_type == "humanoid_bench": obs = render_env.reset() renders = [render_env.render()] elif env_type in ["isaaclab", "mtbench"]: raise NotImplementedError( "We don't support rendering for IsaacLab and MTBench environments" ) else: obs = render_env.reset() render_env.state.info["command"] = jnp.array([[1.0, 0.0, 0.0]]) renders = [render_env.state] for i in range(render_env.max_episode_steps): with torch.no_grad(), autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): obs = normalize_obs(obs, update=False) actions = actor(obs) next_obs, _, done, _ = render_env.step(actions.float()) if env_type == "mujoco_playground": render_env.state.info["command"] = jnp.array([[1.0, 0.0, 0.0]]) if i % 2 == 0: if env_type == "humanoid_bench": renders.append(render_env.render()) else: renders.append(render_env.state) if done.any(): break obs = next_obs if env_type == "mujoco_playground": renders = render_env.render_trajectory(renders) return renders def update_main(data, logs_dict): with autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): observations = data["observations"] next_observations = data["next"]["observations"] if envs.asymmetric_obs: critic_observations = data["critic_observations"] next_critic_observations = data["next"]["critic_observations"] else: critic_observations = observations next_critic_observations = next_observations actions = data["actions"] rewards = data["next"]["rewards"] dones = data["next"]["dones"].bool() truncations = data["next"]["truncations"].bool() if args.disable_bootstrap: bootstrap = (~dones).float() else: bootstrap = (truncations | ~dones).float() clipped_noise = torch.randn_like(actions) clipped_noise = clipped_noise.mul(policy_noise).clamp( -noise_clip, noise_clip ) next_state_actions = (actor(next_observations) + clipped_noise).clamp( action_low, action_high ) discount = args.gamma ** data["next"]["effective_n_steps"] with torch.no_grad(): qf1_next_target_projected, qf2_next_target_projected = ( qnet_target.projection( next_critic_observations, next_state_actions, rewards, bootstrap, discount, ) ) qf1_next_target_value = qnet_target.get_value(qf1_next_target_projected) qf2_next_target_value = qnet_target.get_value(qf2_next_target_projected) if args.use_cdq: qf_next_target_dist = torch.where( qf1_next_target_value.unsqueeze(1) < qf2_next_target_value.unsqueeze(1), qf1_next_target_projected, qf2_next_target_projected, ) qf1_next_target_dist = qf2_next_target_dist = qf_next_target_dist else: qf1_next_target_dist, qf2_next_target_dist = ( qf1_next_target_projected, qf2_next_target_projected, ) qf1, qf2 = qnet(critic_observations, actions) qf1_loss = -torch.sum( qf1_next_target_dist * F.log_softmax(qf1, dim=1), dim=1 ).mean() qf2_loss = -torch.sum( qf2_next_target_dist * F.log_softmax(qf2, dim=1), dim=1 ).mean() qf_loss = qf1_loss + qf2_loss q_optimizer.zero_grad(set_to_none=True) scaler.scale(qf_loss).backward() scaler.unscale_(q_optimizer) if args.use_grad_norm_clipping: critic_grad_norm = torch.nn.utils.clip_grad_norm_( qnet.parameters(), max_norm=args.max_grad_norm if args.max_grad_norm > 0 else float("inf"), ) else: critic_grad_norm = torch.tensor(0.0, device=device) scaler.step(q_optimizer) scaler.update() logs_dict["critic_grad_norm"] = critic_grad_norm.detach() logs_dict["qf_loss"] = qf_loss.detach() logs_dict["qf_max"] = qf1_next_target_value.max().detach() logs_dict["qf_min"] = qf1_next_target_value.min().detach() return logs_dict def update_pol(data, logs_dict): with autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): critic_observations = ( data["critic_observations"] if envs.asymmetric_obs else data["observations"] ) qf1, qf2 = qnet(critic_observations, actor(data["observations"])) qf1_value = qnet.get_value(F.softmax(qf1, dim=1)) qf2_value = qnet.get_value(F.softmax(qf2, dim=1)) if args.use_cdq: qf_value = torch.minimum(qf1_value, qf2_value) else: qf_value = (qf1_value + qf2_value) / 2.0 actor_loss = -qf_value.mean() actor_optimizer.zero_grad(set_to_none=True) scaler.scale(actor_loss).backward() scaler.unscale_(actor_optimizer) if args.use_grad_norm_clipping: actor_grad_norm = torch.nn.utils.clip_grad_norm_( actor.parameters(), max_norm=args.max_grad_norm if args.max_grad_norm > 0 else float("inf"), ) else: actor_grad_norm = torch.tensor(0.0, device=device) scaler.step(actor_optimizer) scaler.update() logs_dict["actor_grad_norm"] = actor_grad_norm.detach() logs_dict["actor_loss"] = actor_loss.detach() return logs_dict @torch.no_grad() def soft_update(src, tgt, tau: float): src_ps = [p.data for p in src.parameters()] tgt_ps = [p.data for p in tgt.parameters()] torch._foreach_mul_(tgt_ps, 1.0 - tau) torch._foreach_add_(tgt_ps, src_ps, alpha=tau) if args.compile: compile_mode = args.compile_mode update_main = torch.compile(update_main, mode=compile_mode) update_pol = torch.compile(update_pol, mode=compile_mode) policy = torch.compile(policy, mode=None) normalize_obs = torch.compile(obs_normalizer.forward, mode=None) normalize_critic_obs = torch.compile(critic_obs_normalizer.forward, mode=None) if args.reward_normalization: update_stats = torch.compile(reward_normalizer.update_stats, mode=None) normalize_reward = torch.compile(reward_normalizer.forward, mode=None) else: normalize_obs = obs_normalizer.forward normalize_critic_obs = critic_obs_normalizer.forward if args.reward_normalization: update_stats = reward_normalizer.update_stats normalize_reward = reward_normalizer.forward if envs.asymmetric_obs: obs, critic_obs = envs.reset_with_critic_obs() critic_obs = torch.as_tensor(critic_obs, device=device, dtype=torch.float) else: obs = envs.reset() if args.checkpoint_path: # Load checkpoint if specified torch_checkpoint = torch.load( f"{args.checkpoint_path}", map_location=device, weights_only=False ) actor.load_state_dict(torch_checkpoint["actor_state_dict"]) obs_normalizer.load_state_dict(torch_checkpoint["obs_normalizer_state"]) critic_obs_normalizer.load_state_dict( torch_checkpoint["critic_obs_normalizer_state"] ) qnet.load_state_dict(torch_checkpoint["qnet_state_dict"]) qnet_target.load_state_dict(torch_checkpoint["qnet_target_state_dict"]) global_step = torch_checkpoint["global_step"] else: global_step = 0 dones = None pbar = tqdm.tqdm(total=args.total_timesteps, initial=global_step) start_time = None desc = "" while global_step < args.total_timesteps: mark_step() logs_dict = TensorDict() if ( start_time is None and global_step >= args.measure_burnin + args.learning_starts ): start_time = time.time() measure_burnin = global_step with torch.no_grad(), autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): norm_obs = normalize_obs(obs) actions = policy(obs=norm_obs, dones=dones) next_obs, rewards, dones, infos = envs.step(actions.float()) truncations = infos["time_outs"] if args.reward_normalization: if env_type == "mtbench": task_ids_one_hot = obs[..., -envs.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) update_stats(rewards, dones.float(), task_ids=task_indices) else: update_stats(rewards, dones.float()) if envs.asymmetric_obs: next_critic_obs = infos["observations"]["critic"] # Compute 'true' next_obs and next_critic_obs for saving true_next_obs = torch.where( dones[:, None] > 0, infos["observations"]["raw"]["obs"], next_obs ) if envs.asymmetric_obs: true_next_critic_obs = torch.where( dones[:, None] > 0, infos["observations"]["raw"]["critic_obs"], next_critic_obs, ) transition = TensorDict( { "observations": obs, "actions": torch.as_tensor(actions, device=device, dtype=torch.float), "next": { "observations": true_next_obs, "rewards": torch.as_tensor( rewards, device=device, dtype=torch.float ), "truncations": truncations.long(), "dones": dones.long(), }, }, batch_size=(envs.num_envs,), device=device, ) if envs.asymmetric_obs: transition["critic_observations"] = critic_obs transition["next"]["critic_observations"] = true_next_critic_obs rb.extend(transition) obs = next_obs if envs.asymmetric_obs: critic_obs = next_critic_obs if global_step > args.learning_starts: for i in range(args.num_updates): data = rb.sample(max(1, args.batch_size // args.num_envs)) data["observations"] = normalize_obs(data["observations"]) data["next"]["observations"] = normalize_obs( data["next"]["observations"] ) if envs.asymmetric_obs: data["critic_observations"] = normalize_critic_obs( data["critic_observations"] ) data["next"]["critic_observations"] = normalize_critic_obs( data["next"]["critic_observations"] ) raw_rewards = data["next"]["rewards"] if env_type in ["mtbench"] and args.reward_normalization: # Multi-task reward normalization task_ids_one_hot = data["observations"][..., -envs.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) data["next"]["rewards"] = normalize_reward( raw_rewards, task_ids=task_indices ) else: data["next"]["rewards"] = normalize_reward(raw_rewards) logs_dict = update_main(data, logs_dict) if args.num_updates > 1: if i % args.policy_frequency == 1: logs_dict = update_pol(data, logs_dict) else: if global_step % args.policy_frequency == 0: logs_dict = update_pol(data, logs_dict) soft_update(qnet, qnet_target, args.tau) if global_step % 100 == 0 and start_time is not None: speed = (global_step - measure_burnin) / (time.time() - start_time) pbar.set_description(f"{speed: 4.4f} sps, " + desc) with torch.no_grad(): logs = { "actor_loss": logs_dict["actor_loss"].mean(), "qf_loss": logs_dict["qf_loss"].mean(), "qf_max": logs_dict["qf_max"].mean(), "qf_min": logs_dict["qf_min"].mean(), "actor_grad_norm": logs_dict["actor_grad_norm"].mean(), "critic_grad_norm": logs_dict["critic_grad_norm"].mean(), "env_rewards": rewards.mean(), "buffer_rewards": raw_rewards.mean(), } if args.eval_interval > 0 and global_step % args.eval_interval == 0: print(f"Evaluating at global step {global_step}") eval_avg_return, eval_avg_length = evaluate() if env_type in ["humanoid_bench", "isaaclab", "mtbench"]: # NOTE: Hacky way of evaluating performance, but just works obs = envs.reset() logs["eval_avg_return"] = eval_avg_return logs["eval_avg_length"] = eval_avg_length if ( args.render_interval > 0 and global_step % args.render_interval == 0 ): renders = render_with_rollout() render_video = wandb.Video( np.array(renders).transpose( 0, 3, 1, 2 ), # Convert to (T, C, H, W) format fps=30, format="gif", ) logs["render_video"] = render_video if args.use_wandb: wandb.log( { "speed": speed, "frame": global_step * args.num_envs, "critic_lr": q_scheduler.get_last_lr()[0], "actor_lr": actor_scheduler.get_last_lr()[0], **logs, }, step=global_step, ) if ( args.save_interval > 0 and global_step > 0 and global_step % args.save_interval == 0 ): print(f"Saving model at global step {global_step}") save_params( global_step, actor, qnet, qnet_target, obs_normalizer, critic_obs_normalizer, args, f"models/{run_name}_{global_step}.pt", ) global_step += 1 actor_scheduler.step() q_scheduler.step() pbar.update(1) save_params( global_step, actor, qnet, qnet_target, obs_normalizer, critic_obs_normalizer, args, f"models/{run_name}_final.pt", ) if __name__ == "__main__": main() ================================================ FILE: fast_td3/train_multigpu.py ================================================ import os import sys os.environ["TORCHDYNAMO_INLINE_INBUILT_NN_MODULES"] = "1" os.environ["OMP_NUM_THREADS"] = "1" if sys.platform != "darwin": os.environ["MUJOCO_GL"] = "egl" else: os.environ["MUJOCO_GL"] = "glfw" os.environ["XLA_PYTHON_CLIENT_PREALLOCATE"] = "false" os.environ["JAX_DEFAULT_MATMUL_PRECISION"] = "highest" import random import time import math import tqdm import wandb import numpy as np try: # Required for avoiding IsaacGym import error import isaacgym except ImportError: pass import torch import torch.nn as nn import torch.nn.functional as F import torch.optim as optim from torch.amp import autocast, GradScaler from torch.nn.parallel import DistributedDataParallel as DDP import torch.multiprocessing as mp from tensordict import TensorDict from fast_td3_utils import ( EmpiricalNormalization, RewardNormalizer, PerTaskRewardNormalizer, SimpleReplayBuffer, save_params, get_ddp_state_dict, load_ddp_state_dict, mark_step, ) from hyperparams import get_args torch.set_float32_matmul_precision("high") try: import jax.numpy as jnp except ImportError: pass def setup_distributed(rank: int, world_size: int): os.environ["MASTER_ADDR"] = os.getenv("MASTER_ADDR", "localhost") os.environ["MASTER_PORT"] = os.getenv("MASTER_PORT", "12355") is_distributed = world_size > 1 if is_distributed: print( f"Initializing distributed training with rank {rank}, world size {world_size}" ) torch.distributed.init_process_group( backend="nccl", init_method="env://", world_size=world_size, rank=rank ) torch.cuda.set_device(rank) return is_distributed def main(rank: int, world_size: int): is_distributed = setup_distributed(rank, world_size) args = get_args() if rank == 0: print(args) run_name = f"{args.env_name}__{args.exp_name}__{args.seed}" amp_enabled = args.amp and args.cuda and torch.cuda.is_available() amp_device_type = ( f"cuda:{rank}" if args.cuda and torch.cuda.is_available() else "mps" if args.cuda and torch.backends.mps.is_available() else "cpu" ) amp_dtype = torch.bfloat16 if args.amp_dtype == "bf16" else torch.float16 scaler = GradScaler(enabled=amp_enabled and amp_dtype == torch.float16) if args.use_wandb and rank == 0: wandb.init( project=args.project, name=run_name, config=vars(args), save_code=True, ) # Use different seeds per rank to avoid synchronization issues random.seed(args.seed + rank) np.random.seed(args.seed + rank) torch.manual_seed(args.seed + rank) torch.backends.cudnn.deterministic = args.torch_deterministic if not args.cuda: device = torch.device("cpu") else: if torch.cuda.is_available(): device = torch.device(f"cuda:{rank}") elif torch.backends.mps.is_available(): device = torch.device(f"mps:{rank}") else: raise ValueError("No GPU available") print(f"Using device: {device}") if args.env_name.startswith("h1hand-") or args.env_name.startswith("h1-"): from environments.humanoid_bench_env import HumanoidBenchEnv env_type = "humanoid_bench" envs = HumanoidBenchEnv(args.env_name, args.num_envs, device=device) eval_envs = envs render_env = HumanoidBenchEnv( args.env_name, 1, render_mode="rgb_array", device=device ) elif args.env_name.startswith("Isaac-"): from environments.isaaclab_env import IsaacLabEnv env_type = "isaaclab" envs = IsaacLabEnv( args.env_name, f"cuda:{rank}", args.num_envs, args.seed + rank, action_bounds=args.action_bounds, ) eval_envs = envs render_env = envs elif args.env_name.startswith("MTBench-"): from environments.mtbench_env import MTBenchEnv env_name = "-".join(args.env_name.split("-")[1:]) env_type = "mtbench" envs = MTBenchEnv(env_name, rank, args.num_envs, args.seed + rank) eval_envs = envs render_env = envs else: from environments.mujoco_playground_env import make_env # TODO: Check if re-using same envs for eval could reduce memory usage env_type = "mujoco_playground" envs, eval_envs, render_env = make_env( args.env_name, args.seed + rank, args.num_envs, args.num_eval_envs, rank, use_tuned_reward=args.use_tuned_reward, use_domain_randomization=args.use_domain_randomization, use_push_randomization=args.use_push_randomization, ) n_act = envs.num_actions n_obs = envs.num_obs if type(envs.num_obs) == int else envs.num_obs[0] if envs.asymmetric_obs: n_critic_obs = ( envs.num_privileged_obs if type(envs.num_privileged_obs) == int else envs.num_privileged_obs[0] ) else: n_critic_obs = n_obs action_low, action_high = -1.0, 1.0 if args.obs_normalization: obs_normalizer = EmpiricalNormalization(shape=n_obs, device=device) critic_obs_normalizer = EmpiricalNormalization( shape=n_critic_obs, device=device ) else: obs_normalizer = nn.Identity() critic_obs_normalizer = nn.Identity() if args.reward_normalization: if env_type in ["mtbench"]: reward_normalizer = PerTaskRewardNormalizer( num_tasks=envs.num_tasks, gamma=args.gamma, device=device, g_max=min(abs(args.v_min), abs(args.v_max)), ) else: reward_normalizer = RewardNormalizer( gamma=args.gamma, device=device, g_max=min(abs(args.v_min), abs(args.v_max)), ) else: reward_normalizer = nn.Identity() actor_kwargs = { "n_obs": n_obs, "n_act": n_act, "num_envs": args.num_envs, "device": device, "init_scale": args.init_scale, "hidden_dim": args.actor_hidden_dim, "std_min": args.std_min, "std_max": args.std_max, } critic_kwargs = { "n_obs": n_critic_obs, "n_act": n_act, "num_atoms": args.num_atoms, "v_min": args.v_min, "v_max": args.v_max, "hidden_dim": args.critic_hidden_dim, "device": device, } if env_type == "mtbench": actor_kwargs["n_obs"] = n_obs - envs.num_tasks + args.task_embedding_dim critic_kwargs["n_obs"] = n_critic_obs - envs.num_tasks + args.task_embedding_dim actor_kwargs["num_tasks"] = envs.num_tasks actor_kwargs["task_embedding_dim"] = args.task_embedding_dim critic_kwargs["num_tasks"] = envs.num_tasks critic_kwargs["task_embedding_dim"] = args.task_embedding_dim if args.agent == "fasttd3": if env_type in ["mtbench"]: from fast_td3 import MultiTaskActor, MultiTaskCritic actor_cls = MultiTaskActor critic_cls = MultiTaskCritic else: from fast_td3 import Actor, Critic actor_cls = Actor critic_cls = Critic if rank == 0: print("Using FastTD3") elif args.agent == "fasttd3_simbav2": if env_type in ["mtbench"]: from fast_td3_simbav2 import MultiTaskActor, MultiTaskCritic actor_cls = MultiTaskActor critic_cls = MultiTaskCritic else: from fast_td3_simbav2 import Actor, Critic actor_cls = Actor critic_cls = Critic if rank == 0: print("Using FastTD3 + SimbaV2") actor_kwargs.pop("init_scale") actor_kwargs.update( { "scaler_init": math.sqrt(2.0 / args.actor_hidden_dim), "scaler_scale": math.sqrt(2.0 / args.actor_hidden_dim), "alpha_init": 1.0 / (args.actor_num_blocks + 1), "alpha_scale": 1.0 / math.sqrt(args.actor_hidden_dim), "expansion": 4, "c_shift": 3.0, "num_blocks": args.actor_num_blocks, } ) critic_kwargs.update( { "scaler_init": math.sqrt(2.0 / args.critic_hidden_dim), "scaler_scale": math.sqrt(2.0 / args.critic_hidden_dim), "alpha_init": 1.0 / (args.critic_num_blocks + 1), "alpha_scale": 1.0 / math.sqrt(args.critic_hidden_dim), "num_blocks": args.critic_num_blocks, "expansion": 4, "c_shift": 3.0, } ) else: raise ValueError(f"Agent {args.agent} not supported") actor = actor_cls(**actor_kwargs) if is_distributed: actor = DDP(actor, device_ids=[rank]) if env_type in ["mtbench"]: # Python 3.8 doesn't support 'from_module' in tensordict policy = actor.module.explore if hasattr(actor, "module") else actor.explore else: from tensordict import from_module actor_detach = actor_cls(**actor_kwargs) # Copy params to actor_detach without grad from_module(actor.module if hasattr(actor, "module") else actor).data.to_module( actor_detach ) policy = actor_detach.explore qnet = critic_cls(**critic_kwargs) if is_distributed: qnet = DDP(qnet, device_ids=[rank]) qnet_target = critic_cls(**critic_kwargs) # Create a separate instance qnet_target.load_state_dict(get_ddp_state_dict(qnet)) q_optimizer = optim.AdamW( list(qnet.parameters()), lr=torch.tensor(args.critic_learning_rate, device=device), weight_decay=args.weight_decay, ) actor_optimizer = optim.AdamW( list(actor.parameters()), lr=torch.tensor(args.actor_learning_rate, device=device), weight_decay=args.weight_decay, ) # Add learning rate schedulers q_scheduler = optim.lr_scheduler.CosineAnnealingLR( q_optimizer, T_max=args.total_timesteps, eta_min=torch.tensor(args.critic_learning_rate_end, device=device), ) actor_scheduler = optim.lr_scheduler.CosineAnnealingLR( actor_optimizer, T_max=args.total_timesteps, eta_min=torch.tensor(args.actor_learning_rate_end, device=device), ) rb = SimpleReplayBuffer( n_env=args.num_envs, buffer_size=args.buffer_size, n_obs=n_obs, n_act=n_act, n_critic_obs=n_critic_obs, asymmetric_obs=envs.asymmetric_obs, playground_mode=env_type == "mujoco_playground", n_steps=args.num_steps, gamma=args.gamma, device=device, ) policy_noise = args.policy_noise noise_clip = args.noise_clip def evaluate(): num_eval_envs = eval_envs.num_envs episode_returns = torch.zeros(num_eval_envs, device=device) episode_lengths = torch.zeros(num_eval_envs, device=device) done_masks = torch.zeros(num_eval_envs, dtype=torch.bool, device=device) if env_type == "isaaclab": obs = eval_envs.reset(random_start_init=False) else: obs = eval_envs.reset() # Run for a fixed number of steps for i in range(eval_envs.max_episode_steps): with torch.no_grad(), autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): obs = normalize_obs(obs, update=False) actions = actor(obs) next_obs, rewards, dones, infos = eval_envs.step(actions.float()) if env_type == "mtbench": # We only report success rate in MTBench evaluation rewards = ( infos["episode"]["success"].float() if "episode" in infos else 0.0 ) episode_returns = torch.where( ~done_masks, episode_returns + rewards, episode_returns ) episode_lengths = torch.where( ~done_masks, episode_lengths + 1, episode_lengths ) if env_type == "mtbench" and "episode" in infos: dones = dones | infos["episode"]["success"] done_masks = torch.logical_or(done_masks, dones) if done_masks.all(): break obs = next_obs return episode_returns.mean(), episode_lengths.mean() def render_with_rollout(): # Quick rollout for rendering if env_type == "humanoid_bench": obs = render_env.reset() renders = [render_env.render()] elif env_type in ["isaaclab", "mtbench"]: raise NotImplementedError( "We don't support rendering for IsaacLab and MTBench environments" ) else: obs = render_env.reset() render_env.state.info["command"] = jnp.array([[1.0, 0.0, 0.0]]) renders = [render_env.state] for i in range(render_env.max_episode_steps): with torch.no_grad(), autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): obs = normalize_obs(obs, update=False) actions = actor(obs) next_obs, _, done, _ = render_env.step(actions.float()) if env_type == "mujoco_playground": render_env.state.info["command"] = jnp.array([[1.0, 0.0, 0.0]]) if i % 2 == 0: if env_type == "humanoid_bench": renders.append(render_env.render()) else: renders.append(render_env.state) if done.any(): break obs = next_obs if env_type == "mujoco_playground": renders = render_env.render_trajectory(renders) return renders def update_main(data, logs_dict): with autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): observations = data["observations"] next_observations = data["next"]["observations"] if envs.asymmetric_obs: critic_observations = data["critic_observations"] next_critic_observations = data["next"]["critic_observations"] else: critic_observations = observations next_critic_observations = next_observations actions = data["actions"] rewards = data["next"]["rewards"] dones = data["next"]["dones"].bool() truncations = data["next"]["truncations"].bool() if args.disable_bootstrap: bootstrap = (~dones).float() else: bootstrap = (truncations | ~dones).float() clipped_noise = torch.randn_like(actions) clipped_noise = clipped_noise.mul(policy_noise).clamp( -noise_clip, noise_clip ) next_state_actions = (actor(next_observations) + clipped_noise).clamp( action_low, action_high ) discount = args.gamma ** data["next"]["effective_n_steps"] with torch.no_grad(): qf1_next_target_projected, qf2_next_target_projected = ( qnet_target.projection( next_critic_observations, next_state_actions, rewards, bootstrap, discount, ) ) qf1_next_target_value = qnet_target.get_value(qf1_next_target_projected) qf2_next_target_value = qnet_target.get_value(qf2_next_target_projected) if args.use_cdq: qf_next_target_dist = torch.where( qf1_next_target_value.unsqueeze(1) < qf2_next_target_value.unsqueeze(1), qf1_next_target_projected, qf2_next_target_projected, ) qf1_next_target_dist = qf2_next_target_dist = qf_next_target_dist else: qf1_next_target_dist, qf2_next_target_dist = ( qf1_next_target_projected, qf2_next_target_projected, ) qf1, qf2 = qnet(critic_observations, actions) qf1_loss = -torch.sum( qf1_next_target_dist * F.log_softmax(qf1, dim=1), dim=1 ).mean() qf2_loss = -torch.sum( qf2_next_target_dist * F.log_softmax(qf2, dim=1), dim=1 ).mean() qf_loss = qf1_loss + qf2_loss q_optimizer.zero_grad(set_to_none=True) scaler.scale(qf_loss).backward() scaler.unscale_(q_optimizer) if args.use_grad_norm_clipping: critic_grad_norm = torch.nn.utils.clip_grad_norm_( qnet.parameters(), max_norm=args.max_grad_norm if args.max_grad_norm > 0 else float("inf"), ) else: critic_grad_norm = torch.tensor(0.0, device=device) scaler.step(q_optimizer) scaler.update() logs_dict["critic_grad_norm"] = critic_grad_norm.detach() logs_dict["qf_loss"] = qf_loss.detach() logs_dict["qf_max"] = qf1_next_target_value.max().detach() logs_dict["qf_min"] = qf1_next_target_value.min().detach() return logs_dict def update_pol(data, logs_dict): with autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): critic_observations = ( data["critic_observations"] if envs.asymmetric_obs else data["observations"] ) qf1, qf2 = qnet(critic_observations, actor(data["observations"])) qf1_value = ( qnet.module.get_value(F.softmax(qf1, dim=1)) if hasattr(qnet, "module") else qnet.get_value(F.softmax(qf1, dim=1)) ) qf2_value = ( qnet.module.get_value(F.softmax(qf2, dim=1)) if hasattr(qnet, "module") else qnet.get_value(F.softmax(qf2, dim=1)) ) if args.use_cdq: qf_value = torch.minimum(qf1_value, qf2_value) else: qf_value = (qf1_value + qf2_value) / 2.0 actor_loss = -qf_value.mean() actor_optimizer.zero_grad(set_to_none=True) scaler.scale(actor_loss).backward() scaler.unscale_(actor_optimizer) if args.use_grad_norm_clipping: actor_grad_norm = torch.nn.utils.clip_grad_norm_( actor.parameters(), max_norm=args.max_grad_norm if args.max_grad_norm > 0 else float("inf"), ) else: actor_grad_norm = torch.tensor(0.0, device=device) scaler.step(actor_optimizer) scaler.update() logs_dict["actor_grad_norm"] = actor_grad_norm.detach() logs_dict["actor_loss"] = actor_loss.detach() return logs_dict @torch.no_grad() def soft_update(src, tgt, tau: float): # Handle DDP module by accessing .module attribute src_module = src.module if hasattr(src, "module") else src tgt_module = tgt.module if hasattr(tgt, "module") else tgt src_ps = [p.data for p in src_module.parameters()] tgt_ps = [p.data for p in tgt_module.parameters()] torch._foreach_mul_(tgt_ps, 1.0 - tau) torch._foreach_add_(tgt_ps, src_ps, alpha=tau) if args.compile: compile_mode = args.compile_mode update_main = torch.compile(update_main, mode=compile_mode) update_pol = torch.compile(update_pol, mode=compile_mode) policy = torch.compile(policy, mode=None) normalize_obs = torch.compile(obs_normalizer.forward, mode=None) normalize_critic_obs = torch.compile(critic_obs_normalizer.forward, mode=None) if args.reward_normalization: update_stats = torch.compile(reward_normalizer.update_stats, mode=None) normalize_reward = torch.compile(reward_normalizer.forward, mode=None) else: normalize_obs = obs_normalizer.forward normalize_critic_obs = critic_obs_normalizer.forward if args.reward_normalization: update_stats = reward_normalizer.update_stats normalize_reward = reward_normalizer.forward if envs.asymmetric_obs: obs, critic_obs = envs.reset_with_critic_obs() critic_obs = torch.as_tensor(critic_obs, device=device, dtype=torch.float) else: obs = envs.reset() if args.checkpoint_path: # Load checkpoint if specified torch_checkpoint = torch.load( f"{args.checkpoint_path}", map_location=device, weights_only=False ) load_ddp_state_dict(actor, torch_checkpoint["actor_state_dict"]) if torch_checkpoint["obs_normalizer_state"] is not None: obs_normalizer.load_state_dict(torch_checkpoint["obs_normalizer_state"]) if torch_checkpoint["critic_obs_normalizer_state"] is not None: critic_obs_normalizer.load_state_dict( torch_checkpoint["critic_obs_normalizer_state"] ) load_ddp_state_dict(qnet, torch_checkpoint["qnet_state_dict"]) qnet_target.load_state_dict(torch_checkpoint["qnet_target_state_dict"]) global_step = torch_checkpoint["global_step"] else: global_step = 0 dones = None pbar = tqdm.tqdm(total=args.total_timesteps, initial=global_step) start_time = None desc = "" while global_step < args.total_timesteps: mark_step() logs_dict = TensorDict() if ( start_time is None and global_step >= args.measure_burnin + args.learning_starts ): start_time = time.time() measure_burnin = global_step with torch.no_grad(), autocast( device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled ): norm_obs = normalize_obs(obs) actions = policy(obs=norm_obs, dones=dones) next_obs, rewards, dones, infos = envs.step(actions.float()) truncations = infos["time_outs"] if args.reward_normalization: if env_type == "mtbench": task_ids_one_hot = obs[..., -envs.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) update_stats(rewards, dones.float(), task_ids=task_indices) else: update_stats(rewards, dones.float()) if envs.asymmetric_obs: next_critic_obs = infos["observations"]["critic"] # Compute 'true' next_obs and next_critic_obs for saving true_next_obs = torch.where( dones[:, None] > 0, infos["observations"]["raw"]["obs"], next_obs ) if envs.asymmetric_obs: true_next_critic_obs = torch.where( dones[:, None] > 0, infos["observations"]["raw"]["critic_obs"], next_critic_obs, ) transition = TensorDict( { "observations": obs, "actions": torch.as_tensor(actions, device=device, dtype=torch.float), "next": { "observations": true_next_obs, "rewards": torch.as_tensor( rewards, device=device, dtype=torch.float ), "truncations": truncations.long(), "dones": dones.long(), }, }, batch_size=(envs.num_envs,), device=device, ) if envs.asymmetric_obs: transition["critic_observations"] = critic_obs transition["next"]["critic_observations"] = true_next_critic_obs rb.extend(transition) obs = next_obs if envs.asymmetric_obs: critic_obs = next_critic_obs if global_step > args.learning_starts: for i in range(args.num_updates): data = rb.sample(max(1, args.batch_size // args.num_envs)) data["observations"] = normalize_obs(data["observations"]) data["next"]["observations"] = normalize_obs( data["next"]["observations"] ) if envs.asymmetric_obs: data["critic_observations"] = normalize_critic_obs( data["critic_observations"] ) data["next"]["critic_observations"] = normalize_critic_obs( data["next"]["critic_observations"] ) raw_rewards = data["next"]["rewards"] if env_type in ["mtbench"] and args.reward_normalization: # Multi-task reward normalization task_ids_one_hot = data["observations"][..., -envs.num_tasks :] task_indices = torch.argmax(task_ids_one_hot, dim=1) data["next"]["rewards"] = normalize_reward( raw_rewards, task_ids=task_indices ) else: data["next"]["rewards"] = normalize_reward(raw_rewards) logs_dict = update_main(data, logs_dict) if args.num_updates > 1: if i % args.policy_frequency == 1: logs_dict = update_pol(data, logs_dict) else: if global_step % args.policy_frequency == 0: logs_dict = update_pol(data, logs_dict) soft_update(qnet, qnet_target, args.tau) if global_step % 100 == 0 and start_time is not None: speed = (global_step - measure_burnin) / (time.time() - start_time) if rank == 0: pbar.set_description(f"{speed: 4.4f} sps, " + desc) with torch.no_grad(): logs = { "actor_loss": logs_dict["actor_loss"].mean(), "qf_loss": logs_dict["qf_loss"].mean(), "qf_max": logs_dict["qf_max"].mean(), "qf_min": logs_dict["qf_min"].mean(), "actor_grad_norm": logs_dict["actor_grad_norm"].mean(), "critic_grad_norm": logs_dict["critic_grad_norm"].mean(), "env_rewards": rewards.mean(), "buffer_rewards": raw_rewards.mean(), } if args.eval_interval > 0 and global_step % args.eval_interval == 0: local_eval_avg_return, local_eval_avg_length = evaluate() eval_results = torch.tensor( [local_eval_avg_return, local_eval_avg_length], device=device, ) if is_distributed: torch.distributed.all_reduce( eval_results, op=torch.distributed.ReduceOp.AVG ) if rank == 0: global_avg_return = eval_results[0].item() global_avg_length = eval_results[1].item() print( f"Evaluating at global step {global_step}: Avg Return={global_avg_return:.2f}" ) logs["eval_avg_return"] = global_avg_return logs["eval_avg_length"] = global_avg_length if env_type in ["humanoid_bench", "isaaclab", "mtbench"]: # NOTE: Hacky way of evaluating performance, but just works obs = envs.reset() if ( args.render_interval > 0 and global_step % args.render_interval == 0 ): renders = render_with_rollout() render_video = wandb.Video( np.array(renders).transpose( 0, 3, 1, 2 ), # Convert to (T, C, H, W) format fps=30, format="gif", ) logs["render_video"] = render_video if args.use_wandb and rank == 0: wandb.log( { "speed": speed, "frame": global_step * args.num_envs, "critic_lr": q_scheduler.get_last_lr()[0], "actor_lr": actor_scheduler.get_last_lr()[0], **logs, }, step=global_step, ) if ( args.save_interval > 0 and global_step > 0 and global_step % args.save_interval == 0 and rank == 0 ): print(f"Saving model at global step {global_step}") save_params( global_step, actor, qnet, qnet_target, obs_normalizer, critic_obs_normalizer, args, f"models/{run_name}_{global_step}.pt", ) global_step += 1 actor_scheduler.step() q_scheduler.step() if rank == 0: pbar.update(1) save_params( global_step, actor, qnet, qnet_target, obs_normalizer, critic_obs_normalizer, args, f"models/{run_name}_final.pt", ) # Cleanup distributed training if is_distributed: torch.distributed.destroy_process_group() if __name__ == "__main__": world_size = torch.cuda.device_count() mp.spawn(main, args=(world_size,), nprocs=world_size) ================================================ FILE: fast_td3/training_notebook.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# FastTD3 Training Notebook\n", "\n", "Welcome! This notebook will let you execute a series of code blocks that enables you to experience how FastTD3 works -- each block will import packages, define arguments, create environments, create FastTD3 agent, and train the agent.\n", "\n", "This notebook also provide the same functionalities as `train.py` -- you can use this notebook to train your own agents, upload logs to wandb, render rollouts, and fine-tune pre-trained agents with more environment steps!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# NOTE: Set environment variables and import packages\n", "\n", "import os\n", "\n", "os.environ[\"TORCHDYNAMO_INLINE_INBUILT_NN_MODULES\"] = \"1\"\n", "os.environ[\"OMP_NUM_THREADS\"] = \"1\"\n", "if sys.platform != \"darwin\":\n", " os.environ[\"MUJOCO_GL\"] = \"egl\"\n", "else:\n", " os.environ[\"MUJOCO_GL\"] = \"glfw\"\n", "os.environ[\"XLA_PYTHON_CLIENT_PREALLOCATE\"] = \"false\"\n", "os.environ[\"JAX_DEFAULT_MATMUL_PRECISION\"] = \"highest\"\n", "\n", "import random\n", "import time\n", "\n", "import tqdm\n", "import wandb\n", "import numpy as np\n", "\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "from torch.amp import autocast, GradScaler\n", "from tensordict import TensorDict, from_module\n", "\n", "torch.set_float32_matmul_precision(\"high\")\n", "\n", "from fast_td3_utils import (\n", " EmpiricalNormalization,\n", " SimpleReplayBuffer,\n", " save_params,\n", ")\n", "\n", "from fast_td3 import Critic, Actor" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# NOTE: Set checkpoint if you want to fine-tune from existing checkpoint\n", "# e.g., set checkpoint to \"models/h1-walk-v0_notebook_experiment_30000.pt\"\n", "checkpoint_path = None" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# NOTE: Customize arguments as needed\n", "# However, IsaacLab may not work in Notebook Setup.\n", "# We recommend using HumanoidBench or MuJoCo Playground for notebook experiments.\n", "\n", "# For quick experiments, let's use a task without dexterous hands\n", "# But for your research, we recommend using `h1hand` tasks in HumanoidBench!\n", "from hyperparams import HumanoidBenchArgs\n", "\n", "args = HumanoidBenchArgs(\n", " env_name=\"h1-walk-v0\",\n", " total_timesteps=20000,\n", " render_interval=5000,\n", " eval_interval=5000,\n", ")\n", "run_name = f\"{args.env_name}_notebook_experiment\"" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# NOTE: GPU-Related Configurations\n", "\n", "amp_enabled = args.amp and args.cuda and torch.cuda.is_available()\n", "amp_device_type = (\n", " \"cuda\"\n", " if args.cuda and torch.cuda.is_available()\n", " else \"mps\" if args.cuda and torch.backends.mps.is_available() else \"cpu\"\n", ")\n", "amp_dtype = torch.bfloat16 if args.amp_dtype == \"bf16\" else torch.float16\n", "\n", "scaler = GradScaler(enabled=amp_enabled and amp_dtype == torch.float16)\n", "\n", "if not args.cuda:\n", " device = torch.device(\"cpu\")\n", "else:\n", " if torch.cuda.is_available():\n", " device = torch.device(f\"cuda:{args.device_rank}\")\n", " elif torch.backends.mps.is_available():\n", " device = torch.device(f\"mps:{args.device_rank}\")\n", " else:\n", " raise ValueError(\"No GPU available\")\n", "print(f\"Using device: {device}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# NOTE: Define Wandb if needed\n", "\n", "# Set use_wandb to True if you want to use Wandb\n", "use_wandb = True\n", "\n", "if use_wandb:\n", " wandb.init(\n", " project=\"FastTD3\",\n", " name=run_name,\n", " config=vars(args),\n", " save_code=True,\n", " )" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# NOTE: Initialize Environment and Related Variables\n", "\n", "if args.env_name.startswith(\"h1hand-\") or args.env_name.startswith(\"h1-\"):\n", " from environments.humanoid_bench_env import HumanoidBenchEnv\n", "\n", " env_type = \"humanoid_bench\"\n", " envs = HumanoidBenchEnv(args.env_name, args.num_envs, device=device)\n", " eval_envs = envs\n", " render_env = HumanoidBenchEnv(\n", " args.env_name, 1, render_mode=\"rgb_array\", device=device\n", " )\n", "elif args.env_name.startswith(\"Isaac-\"):\n", " from environments.isaaclab_env import IsaacLabEnv\n", "\n", " env_type = \"isaaclab\"\n", " envs = IsaacLabEnv(\n", " args.env_name,\n", " device.type,\n", " args.num_envs,\n", " args.seed,\n", " action_bounds=args.action_bounds,\n", " )\n", " eval_envs = envs\n", " render_envs = envs\n", "else:\n", " from environments.mujoco_playground_env import make_env\n", " import jax.numpy as jnp\n", "\n", " env_type = \"mujoco_playground\"\n", " envs, eval_envs, render_env = make_env(\n", " args.env_name,\n", " args.seed,\n", " args.num_envs,\n", " args.num_eval_envs,\n", " args.device_rank,\n", " use_tuned_reward=args.use_tuned_reward,\n", " use_domain_randomization=args.use_domain_randomization,\n", " )\n", "\n", "n_act = envs.num_actions\n", "n_obs = envs.num_obs if type(envs.num_obs) == int else envs.num_obs[0]\n", "if envs.asymmetric_obs:\n", " n_critic_obs = (\n", " envs.num_privileged_obs\n", " if type(envs.num_privileged_obs) == int\n", " else envs.num_privileged_obs[0]\n", " )\n", "else:\n", " n_critic_obs = n_obs\n", "action_low, action_high = -1.0, 1.0" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# NOTE: Initialize Normalizer, Actor, and Critic\n", "\n", "if args.obs_normalization:\n", " obs_normalizer = EmpiricalNormalization(shape=n_obs, device=device)\n", " critic_obs_normalizer = EmpiricalNormalization(shape=n_critic_obs, device=device)\n", "else:\n", " obs_normalizer = nn.Identity()\n", " critic_obs_normalizer = nn.Identity()\n", "\n", "normalize_obs = obs_normalizer.forward\n", "normalize_critic_obs = critic_obs_normalizer.forward\n", "\n", "# Actor setup\n", "actor = Actor(\n", " n_obs=n_obs,\n", " n_act=n_act,\n", " num_envs=args.num_envs,\n", " device=device,\n", " init_scale=args.init_scale,\n", " hidden_dim=args.actor_hidden_dim,\n", ")\n", "actor_detach = Actor(\n", " n_obs=n_obs,\n", " n_act=n_act,\n", " num_envs=args.num_envs,\n", " device=device,\n", " init_scale=args.init_scale,\n", " hidden_dim=args.actor_hidden_dim,\n", ")\n", "# Copy params to actor_detach without grad\n", "from_module(actor).data.to_module(actor_detach)\n", "policy = actor_detach.explore\n", "\n", "qnet = Critic(\n", " n_obs=n_critic_obs,\n", " n_act=n_act,\n", " num_atoms=args.num_atoms,\n", " v_min=args.v_min,\n", " v_max=args.v_max,\n", " hidden_dim=args.critic_hidden_dim,\n", " device=device,\n", ")\n", "qnet_target = Critic(\n", " n_obs=n_critic_obs,\n", " n_act=n_act,\n", " num_atoms=args.num_atoms,\n", " v_min=args.v_min,\n", " v_max=args.v_max,\n", " hidden_dim=args.critic_hidden_dim,\n", " device=device,\n", ")\n", "qnet_target.load_state_dict(qnet.state_dict())\n", "\n", "q_optimizer = optim.AdamW(\n", " list(qnet.parameters()),\n", " lr=args.critic_learning_rate,\n", " weight_decay=args.weight_decay,\n", ")\n", "actor_optimizer = optim.AdamW(\n", " list(actor.parameters()),\n", " lr=args.actor_learning_rate,\n", " weight_decay=args.weight_decay,\n", ")\n", "\n", "rb = SimpleReplayBuffer(\n", " n_env=args.num_envs,\n", " buffer_size=args.buffer_size,\n", " n_obs=n_obs,\n", " n_act=n_act,\n", " n_critic_obs=n_critic_obs,\n", " asymmetric_obs=envs.asymmetric_obs,\n", " playground_mode=env_type == \"mujoco_playground\",\n", " n_steps=args.num_steps,\n", " gamma=args.gamma,\n", " device=device,\n", ")" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# NOTE: Define Evaluation & Rendering Functions\n", "\n", "\n", "def evaluate():\n", " obs_normalizer.eval()\n", " num_eval_envs = eval_envs.num_envs\n", " episode_returns = torch.zeros(num_eval_envs, device=device)\n", " episode_lengths = torch.zeros(num_eval_envs, device=device)\n", " done_masks = torch.zeros(num_eval_envs, dtype=torch.bool, device=device)\n", "\n", " if env_type == \"isaaclab\":\n", " obs = eval_envs.reset(random_start_init=False)\n", " else:\n", " obs = eval_envs.reset()\n", "\n", " # Run for a fixed number of steps\n", " for _ in range(eval_envs.max_episode_steps):\n", " with torch.no_grad(), autocast(\n", " device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled\n", " ):\n", " obs = normalize_obs(obs)\n", " actions = actor(obs)\n", "\n", " next_obs, rewards, dones, _ = eval_envs.step(actions.float())\n", " episode_returns = torch.where(\n", " ~done_masks, episode_returns + rewards, episode_returns\n", " )\n", " episode_lengths = torch.where(~done_masks, episode_lengths + 1, episode_lengths)\n", " done_masks = torch.logical_or(done_masks, dones)\n", " if done_masks.all():\n", " break\n", " obs = next_obs\n", "\n", " obs_normalizer.train()\n", " return episode_returns.mean().item(), episode_lengths.mean().item()\n", "\n", "\n", "def render_with_rollout():\n", " obs_normalizer.eval()\n", "\n", " # Quick rollout for rendering\n", " if env_type == \"humanoid_bench\":\n", " obs = render_env.reset()\n", " renders = [render_env.render()]\n", " elif env_type == \"isaaclab\":\n", " raise NotImplementedError(\n", " \"We don't support rendering for IsaacLab environments\"\n", " )\n", " else:\n", " obs = render_env.reset()\n", " render_env.state.info[\"command\"] = jnp.array([[1.0, 0.0, 0.0]])\n", " renders = [render_env.state]\n", " for i in range(render_env.max_episode_steps):\n", " with torch.no_grad(), autocast(\n", " device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled\n", " ):\n", " obs = normalize_obs(obs)\n", " actions = actor(obs)\n", " next_obs, _, done, _ = render_env.step(actions.float())\n", " if env_type == \"mujoco_playground\":\n", " render_env.state.info[\"command\"] = jnp.array([[1.0, 0.0, 0.0]])\n", " if i % 2 == 0:\n", " if env_type == \"humanoid_bench\":\n", " renders.append(render_env.render())\n", " else:\n", " renders.append(render_env.state)\n", " if done.any():\n", " break\n", " obs = next_obs\n", "\n", " if env_type == \"mujoco_playground\":\n", " renders = render_env.render_trajectory(renders)\n", "\n", " obs_normalizer.train()\n", " return renders" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# NOTE: Define Update Functions\n", "\n", "policy_noise = args.policy_noise\n", "noise_clip = args.noise_clip\n", "\n", "\n", "def update_main(data, logs_dict):\n", " with autocast(device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled):\n", " observations = data[\"observations\"]\n", " next_observations = data[\"next\"][\"observations\"]\n", " if envs.asymmetric_obs:\n", " critic_observations = data[\"critic_observations\"]\n", " next_critic_observations = data[\"next\"][\"critic_observations\"]\n", " else:\n", " critic_observations = observations\n", " next_critic_observations = next_observations\n", " actions = data[\"actions\"]\n", " rewards = data[\"next\"][\"rewards\"]\n", " dones = data[\"next\"][\"dones\"].bool()\n", " truncations = data[\"next\"][\"truncations\"].bool()\n", " if args.disable_bootstrap:\n", " bootstrap = (~dones).float()\n", " else:\n", " bootstrap = (truncations | ~dones).float()\n", "\n", " clipped_noise = torch.randn_like(actions)\n", " clipped_noise = clipped_noise.mul(policy_noise).clamp(-noise_clip, noise_clip)\n", "\n", " next_state_actions = (actor(next_observations) + clipped_noise).clamp(\n", " action_low, action_high\n", " )\n", "\n", " with torch.no_grad():\n", " qf1_next_target_projected, qf2_next_target_projected = (\n", " qnet_target.projection(\n", " next_critic_observations,\n", " next_state_actions,\n", " rewards,\n", " bootstrap,\n", " args.gamma,\n", " )\n", " )\n", " qf1_next_target_value = qnet_target.get_value(qf1_next_target_projected)\n", " qf2_next_target_value = qnet_target.get_value(qf2_next_target_projected)\n", " if args.use_cdq:\n", " qf_next_target_dist = torch.where(\n", " qf1_next_target_value.unsqueeze(1)\n", " < qf2_next_target_value.unsqueeze(1),\n", " qf1_next_target_projected,\n", " qf2_next_target_projected,\n", " )\n", " qf1_next_target_dist = qf2_next_target_dist = qf_next_target_dist\n", " else:\n", " qf1_next_target_dist, qf2_next_target_dist = (\n", " qf1_next_target_projected,\n", " qf2_next_target_projected,\n", " )\n", "\n", " qf1, qf2 = qnet(critic_observations, actions)\n", " qf1_loss = -torch.sum(\n", " qf1_next_target_dist * F.log_softmax(qf1, dim=1), dim=1\n", " ).mean()\n", " qf2_loss = -torch.sum(\n", " qf2_next_target_dist * F.log_softmax(qf2, dim=1), dim=1\n", " ).mean()\n", " qf_loss = qf1_loss + qf2_loss\n", "\n", " q_optimizer.zero_grad(set_to_none=True)\n", " scaler.scale(qf_loss).backward()\n", " scaler.unscale_(q_optimizer)\n", "\n", " critic_grad_norm = torch.nn.utils.clip_grad_norm_(\n", " qnet.parameters(),\n", " max_norm=args.max_grad_norm if args.max_grad_norm > 0 else float(\"inf\"),\n", " )\n", " scaler.step(q_optimizer)\n", " scaler.update()\n", "\n", " logs_dict[\"buffer_rewards\"] = rewards.mean()\n", " logs_dict[\"critic_grad_norm\"] = critic_grad_norm.detach()\n", " logs_dict[\"qf_loss\"] = qf_loss.detach()\n", " logs_dict[\"qf_max\"] = qf1_next_target_value.max().detach()\n", " logs_dict[\"qf_min\"] = qf1_next_target_value.min().detach()\n", " return logs_dict\n", "\n", "\n", "def update_pol(data, logs_dict):\n", " with autocast(device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled):\n", " critic_observations = (\n", " data[\"critic_observations\"] if envs.asymmetric_obs else data[\"observations\"]\n", " )\n", "\n", " qf1, qf2 = qnet(critic_observations, actor(data[\"observations\"]))\n", " qf1_value = qnet.get_value(F.softmax(qf1, dim=1))\n", " qf2_value = qnet.get_value(F.softmax(qf2, dim=1))\n", " if args.use_cdq:\n", " qf_value = torch.minimum(qf1_value, qf2_value)\n", " else:\n", " qf_value = (qf1_value + qf2_value) / 2.0\n", " actor_loss = -qf_value.mean()\n", "\n", " actor_optimizer.zero_grad(set_to_none=True)\n", " scaler.scale(actor_loss).backward()\n", " scaler.unscale_(actor_optimizer)\n", " actor_grad_norm = torch.nn.utils.clip_grad_norm_(\n", " actor.parameters(),\n", " max_norm=args.max_grad_norm if args.max_grad_norm > 0 else float(\"inf\"),\n", " )\n", " scaler.step(actor_optimizer)\n", " scaler.update()\n", " logs_dict[\"actor_grad_norm\"] = actor_grad_norm.detach()\n", " logs_dict[\"actor_loss\"] = actor_loss.detach()\n", " return logs_dict" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# NOTE: Compile Functions if Needed\n", "\n", "if args.compile:\n", " mode = None\n", " update_main = torch.compile(update_main, mode=mode)\n", " update_pol = torch.compile(update_pol, mode=mode)\n", " policy = torch.compile(policy, mode=mode)\n", " normalize_obs = torch.compile(normalize_obs, mode=mode)\n", " normalize_critic_obs = torch.compile(normalize_critic_obs, mode=mode)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# NOTE: Load Checkpoint if Needed\n", "if checkpoint_path is not None:\n", " torch_checkpoint = torch.load(\n", " f\"{checkpoint_path}\", map_location=device, weights_only=False\n", " )\n", "\n", " actor.load_state_dict(torch_checkpoint[\"actor_state_dict\"])\n", " obs_normalizer.load_state_dict(torch_checkpoint[\"obs_normalizer_state\"])\n", " critic_obs_normalizer.load_state_dict(\n", " torch_checkpoint[\"critic_obs_normalizer_state\"]\n", " )\n", " qnet.load_state_dict(torch_checkpoint[\"qnet_state_dict\"])\n", " qnet_target.load_state_dict(torch_checkpoint[\"qnet_target_state_dict\"])\n", " global_step = torch_checkpoint[\"global_step\"]\n", "else:\n", " global_step = 0" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# NOTE: Utility functions for displaying videos in notebook\n", "\n", "from IPython.display import display, HTML\n", "import base64\n", "import imageio\n", "import tempfile\n", "import os\n", "\n", "\n", "def frames_to_video_html(frames, fps=30):\n", " \"\"\"\n", " Convert a list of numpy arrays to an HTML5 video element.\n", "\n", " Args:\n", " frames (list): List of numpy arrays representing video frames\n", " fps (int): Frames per second for the video\n", "\n", " Returns:\n", " HTML object containing the video element\n", " \"\"\"\n", " # Create a temporary file to store the video\n", " with tempfile.NamedTemporaryFile(suffix=\".mp4\", delete=False) as temp_file:\n", " temp_filename = temp_file.name\n", "\n", " # Save frames as video\n", " imageio.mimsave(temp_filename, frames, fps=fps)\n", "\n", " # Read the video file and encode it to base64\n", " with open(temp_filename, \"rb\") as f:\n", " video_data = f.read()\n", " video_b64 = base64.b64encode(video_data).decode(\"utf-8\")\n", "\n", " # Create HTML video element\n", " video_html = f\"\"\"\n", " \n", " \"\"\"\n", "\n", " # Clean up the temporary file\n", " os.unlink(temp_filename)\n", "\n", " return HTML(video_html)\n", "\n", "\n", "def update_video_display(frames, fps=30):\n", " \"\"\"\n", " Display video frames as an embedded HTML5 video element.\n", "\n", " Args:\n", " frames (list): List of numpy arrays representing video frames\n", " fps (int): Frames per second for the video\n", " \"\"\"\n", " video_html = frames_to_video_html(frames, fps=fps)\n", " display(video_html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# NOTE: Main Training Loop\n", "\n", "if envs.asymmetric_obs:\n", " obs, critic_obs = envs.reset_with_critic_obs()\n", " critic_obs = torch.as_tensor(critic_obs, device=device, dtype=torch.float)\n", "else:\n", " obs = envs.reset()\n", "pbar = tqdm.tqdm(total=args.total_timesteps, initial=global_step)\n", "\n", "dones = None\n", "while global_step < args.total_timesteps:\n", " logs_dict = TensorDict()\n", " with torch.no_grad(), autocast(\n", " device_type=amp_device_type, dtype=amp_dtype, enabled=amp_enabled\n", " ):\n", " norm_obs = normalize_obs(obs)\n", " actions = policy(obs=norm_obs, dones=dones)\n", "\n", " next_obs, rewards, dones, infos = envs.step(actions.float())\n", " truncations = infos[\"time_outs\"]\n", "\n", " if envs.asymmetric_obs:\n", " next_critic_obs = infos[\"observations\"][\"critic\"]\n", "\n", " # Compute 'true' next_obs and next_critic_obs for saving\n", " true_next_obs = torch.where(\n", " dones[:, None] > 0, infos[\"observations\"][\"raw\"][\"obs\"], next_obs\n", " )\n", " if envs.asymmetric_obs:\n", " true_next_critic_obs = torch.where(\n", " dones[:, None] > 0,\n", " infos[\"observations\"][\"raw\"][\"critic_obs\"],\n", " next_critic_obs,\n", " )\n", " transition = TensorDict(\n", " {\n", " \"observations\": obs,\n", " \"actions\": torch.as_tensor(actions, device=device, dtype=torch.float),\n", " \"next\": {\n", " \"observations\": true_next_obs,\n", " \"rewards\": torch.as_tensor(rewards, device=device, dtype=torch.float),\n", " \"truncations\": truncations.long(),\n", " \"dones\": dones.long(),\n", " },\n", " },\n", " batch_size=(envs.num_envs,),\n", " device=device,\n", " )\n", " if envs.asymmetric_obs:\n", " transition[\"critic_observations\"] = critic_obs\n", " transition[\"next\"][\"critic_observations\"] = true_next_critic_obs\n", "\n", " obs = next_obs\n", " if envs.asymmetric_obs:\n", " critic_obs = next_critic_obs\n", "\n", " rb.extend(transition)\n", "\n", " batch_size = args.batch_size // args.num_envs\n", " if global_step > args.learning_starts:\n", " for i in range(args.num_updates):\n", " data = rb.sample(batch_size)\n", " data[\"observations\"] = normalize_obs(data[\"observations\"])\n", " data[\"next\"][\"observations\"] = normalize_obs(data[\"next\"][\"observations\"])\n", " if envs.asymmetric_obs:\n", " data[\"critic_observations\"] = normalize_critic_obs(\n", " data[\"critic_observations\"]\n", " )\n", " data[\"next\"][\"critic_observations\"] = normalize_critic_obs(\n", " data[\"next\"][\"critic_observations\"]\n", " )\n", " logs_dict = update_main(data, logs_dict)\n", " if args.num_updates > 1:\n", " if i % args.policy_frequency == 1:\n", " logs_dict = update_pol(data, logs_dict)\n", " else:\n", " if global_step % args.policy_frequency == 0:\n", " logs_dict = update_pol(data, logs_dict)\n", "\n", " for param, target_param in zip(qnet.parameters(), qnet_target.parameters()):\n", " target_param.data.copy_(\n", " args.tau * param.data + (1 - args.tau) * target_param.data\n", " )\n", "\n", " if global_step > 0 and global_step % 100 == 0:\n", " with torch.no_grad():\n", " logs = {\n", " \"actor_loss\": logs_dict[\"actor_loss\"].mean(),\n", " \"qf_loss\": logs_dict[\"qf_loss\"].mean(),\n", " \"qf_max\": logs_dict[\"qf_max\"].mean(),\n", " \"qf_min\": logs_dict[\"qf_min\"].mean(),\n", " \"actor_grad_norm\": logs_dict[\"actor_grad_norm\"].mean(),\n", " \"critic_grad_norm\": logs_dict[\"critic_grad_norm\"].mean(),\n", " \"buffer_rewards\": logs_dict[\"buffer_rewards\"].mean(),\n", " \"env_rewards\": rewards.mean(),\n", " }\n", "\n", " if args.eval_interval > 0 and global_step % args.eval_interval == 0:\n", " eval_avg_return, eval_avg_length = evaluate()\n", " if env_type in [\"humanoid_bench\", \"isaaclab\"]:\n", " # NOTE: Hacky way of evaluating performance, but just works\n", " obs = envs.reset()\n", " logs[\"eval_avg_return\"] = eval_avg_return\n", " logs[\"eval_avg_length\"] = eval_avg_length\n", "\n", " if args.render_interval > 0 and global_step % args.render_interval == 0:\n", " renders = render_with_rollout()\n", " print_logs = {\n", " k: v.item() if isinstance(v, torch.Tensor) else v\n", " for k, v in logs.items()\n", " }\n", " for k, v in print_logs.items():\n", " print(f\"{k}: {v:.4f}\")\n", " update_video_display(renders, fps=30)\n", " if use_wandb:\n", " wandb.log(\n", " {\n", " \"render_video\": wandb.Video(\n", " np.array(renders).transpose(\n", " 0, 3, 1, 2\n", " ), # Convert to (T, C, H, W) format\n", " fps=30,\n", " format=\"gif\",\n", " )\n", " },\n", " step=global_step,\n", " )\n", " if use_wandb:\n", " wandb.log(\n", " {\n", " \"frame\": global_step * args.num_envs,\n", " **logs,\n", " },\n", " step=global_step,\n", " )\n", "\n", " if (\n", " args.save_interval > 0\n", " and global_step > 0\n", " and global_step % args.save_interval == 0\n", " ):\n", " save_params(\n", " global_step,\n", " actor,\n", " qnet,\n", " qnet_target,\n", " obs_normalizer,\n", " critic_obs_normalizer,\n", " args,\n", " f\"models/{run_name}_{global_step}.pt\",\n", " )\n", "\n", " global_step += 1\n", " pbar.update(1)\n", "\n", "save_params(\n", " global_step,\n", " actor,\n", " qnet,\n", " qnet_target,\n", " obs_normalizer,\n", " critic_obs_normalizer,\n", " args,\n", " f\"models/{run_name}_final.pt\",\n", ")" ] } ], "metadata": { "kernelspec": { "display_name": "fasttd3_hb", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.17" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: requirements/requirements.txt ================================================ gymnasium<1.0.0 jax-jumpy==1.0.0 ; python_version >= "3.8" and python_version < "3.11" matplotlib moviepy numpy<2.0 pandas protobuf pygame stable-baselines3 tqdm wandb torchrl==0.7.2 tensordict==0.7.2 tyro loguru torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124 ================================================ FILE: requirements/requirements_isaacgym.txt ================================================ gymnasium<1.0.0 jax-jumpy==1.0.0 ; python_version >= "3.8" and python_version < "3.11" matplotlib moviepy numpy<2.0 pandas protobuf pygame stable-baselines3 tqdm wandb torchrl==0.5.0 tensordict==0.5.0 tyro loguru ================================================ FILE: requirements/requirements_playground.txt ================================================ gymnasium<1.0.0 jax-jumpy==1.0.0 ; python_version >= "3.8" and python_version < "3.11" matplotlib moviepy numpy<2.0 pandas protobuf pygame stable-baselines3 tqdm wandb torchrl==0.7.2 tensordict==0.7.2 tyro loguru git+https://github.com/younggyoseo/mujoco_playground.git torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124 jax[cuda12]==0.4.35 nvidia-cublas-cu12==12.4.5.8 nvidia-cuda-cupti-cu12==12.4.127 nvidia-cuda-nvcc-cu12==12.8.93 nvidia-cuda-nvrtc-cu12==12.4.127 nvidia-cuda-runtime-cu12==12.4.127 nvidia-cudnn-cu12==9.1.0.70 nvidia-cufft-cu12==11.2.1.3 nvidia-curand-cu12==10.3.5.147 nvidia-cusolver-cu12==11.6.1.9 nvidia-cusparse-cu12==12.3.1.170 nvidia-cusparselt-cu12==0.6.2 nvidia-nccl-cu12==2.21.5 nvidia-nvjitlink-cu12==12.4.127 nvidia-nvtx-cu12==12.4.127 ================================================ FILE: setup.py ================================================ from setuptools import setup, find_packages setup( name="fast_td3", version="0.1.0", description="FastTD3 implementation", author="", author_email="", url="", packages=find_packages(), ) ================================================ FILE: sim2real.md ================================================ # Guide for Sim2Real Training & Deployment This guide provides guide to run sim-to-real experiments using FastTD3 and BoosterGym. **⚠️ Warning:** Deploying RL policies to real hardware can be sometimes very dangerous. Please make sure that you understand everything, check your policies work well in simulation, set every robot configuration correct (e.g., damping, stiffness, torque limits, etc), and follow proper safety protocols. **Simply copy-pasting commands in this README is not safe**. ## ⚙️ Prerequisites Install dependencies for Playground experiments (see `README.md`) Then, install `fast_td3` package with `pip install -e .` so you can import its classes in BoosterGym (see `fast_td3_deploy.py`). **⚠️ Note:** Our sim-to-real experiments depend on our customized MuJoCo Playground that supports `T1LowDimJoystick` tasks for 12-DOF T1 control instead of 23-DOF T1 control in `T1Joystick` tasks. ## 🚀 Training in simulation Users can train deployable policies for Booster T1 with FastTD3 using the below script: ```bash python fast_td3/train.py --env_name T1LowDimJoystickRoughTerrain --exp_name FastTD3 --use_domain_randomization --use_push_randomization --total_timesteps 1000000 --render_interval 0 --seed 2 ``` **⚠️ Note:** There is no 'guaranteed' number of training steps that can ensure safe real-world deployment. Usually, the gait becomes more stable with longer training. Please check the quality of gaits via sim-to-sim transfer, and fine-tune the policy to fix the issues. Use the checkpoints in `models` directory for sim-to-sim or sim-to-real transfer. **⚠️ Note:** We set `render_interval` to 0 to avoid dumping a lot of videos into wandb. Make sure to set it to non-zero values if you want to render videos during training. ### (Optional) 2-Stage Training For faster convergence, users can consider introducing curriculum to the training -- so that the robot first learns to walk in a flat terrain without push perturbations. For this, train policies with the below script: ```bash STAGE1_STEPS = 100000 STAGE2_STEPS = 300000 # Effective steps: 300000 - 200000 = 100000 SEED = 2 CHECKPOINT_PATH = T1LowDimJoystickFlatTerrain__FastTD3-Stage1__${SEED}_final.pt conda activate fasttd3_playground # Stage 1 training python fast_td3/train.py --env_name T1LowDimJoystickFlatTerrain --exp_name FastTD3-Stage1 --use_domain_randomization --no_use_push_randomization --total_timesteps ${STAGE1_STEPS} --render_interval 0 --seed ${SEED} # Stage 2 training python fast_td3/train.py --env_name T1LowDimJoystickRoughTerrain --exp_name FastTD3-Stage2 --use_domain_randomization --use_push_randomization --total_timesteps ${STAGE2_STEPS} --render_interval 0 --checkpoint_path ${CHECKPOINT_PATH} --seed ${SEED} ``` Again, 100K and 200K steps do not guarantee safe real-world deployment. Please check the quality of gaits via sim-to-sim transfer, and fine-tune the policy to fix the issues. Use the final checkpoint (`models/T1LowDimJoystickRoughTerrain__FastTD3-Stage2__${SEED}_final.pt`) for sim-to-sim or sim-to-real transfer. ## 🛝 Deployment with BoosterGym We use the customized version of [BoosterGym](https://github.com/BoosterRobotics/booster_gym) for deployment with FastTD3. First, clone our fork of BoosterGym. ```bash git clone https://github.com/carlosferrazza/booster_gym.git ``` Then, follow the [guide](https://github.com/carlosferrazza/booster_gym) to install dependencies for BoosterGym. ### Sim-to-Sim Transfer You can check whether the trained policy transfers to non-MJX version of MuJoCo. Use the following commands in a machine that supports rendering to test sim-to-sim transfer: ```bash cd /booster_gym # Activate your BoosterGym virtual environemnt # Launch MuJoCo simulation python play_mujoco.py --task=T1 --checkpoint= mjpython play_mujoco.py --task=T1 --checkpoint= # for Mac ``` ### Sim-to-Real Transfer First, prepare a JIT-scripted checkpoint ```python # Python snippets for JIT-scripting checkpoints import torch from fast_td3 import load_policy policy = load_policy() scripted_policy = torch.jit.script(policy) scripted_policy.save() ``` Then, deploy this JIT-scripted checkpoint by following the guide on [Booster T1 Deployment](https://github.com/carlosferrazza/booster_gym/tree/main/deploy). **⚠️ Warning:** Please double-check every value in robot configuration (`booster_gym/deploy/configs/T1.yaml`) is correctly set! If values for position control such as `damping` or `stiffness` are set differently, your robot may perform dangerous behaviors. **⚠️ Warning:** You may want to use different configuration (e.g., `damping` and `stiffness`, etc) for your own experiments. Just make sure to thoroughly test it in simulation and make sure to set the values correctly. --- 🚀 That's it! Hope everything went smoothly, and be aware of your safety.