[
{
"path": ".gitattributes",
"content": "* text=auto\nmodin/_version.py export-subst\n"
},
{
"path": ".github/ISSUE_TEMPLATE/bug-report.yaml",
"content": "name: Bug report\ndescription: Report incorrect behavior in the Modin library\ntitle: 'BUG: '\nlabels: ['bug 🦗', 'Triage 🩹']\n\nbody:\n - type: checkboxes\n id: checks\n attributes:\n label: Modin version checks\n options:\n - label: >\n I have checked that this issue has not already been reported.\n required: true\n - label: >\n I have confirmed this bug exists on the latest released version of Modin.\n required: true\n - label: >\n I have confirmed this bug exists on the main branch of Modin. (In order to do this you\n can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-main-branch).)\n - type: textarea\n id: example\n attributes:\n label: Reproducible Example\n description: >\n Please follow [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) on how to\n provide a minimal, copy-pastable example.\n placeholder: >\n import modin.pandas as pd\n\n df = pd.DataFrame(range(5))\n\n ...\n render: python\n validations:\n required: true\n - type: textarea\n id: problem\n attributes:\n label: Issue Description\n description: >\n Please provide a description of the issue shown in the reproducible example.\n validations:\n required: true\n - type: textarea\n id: expected-behavior\n attributes:\n label: Expected Behavior\n description: >\n Please describe or show a code example of the expected behavior.\n validations:\n required: true\n - type: textarea\n id: logs\n attributes:\n label: Error Logs\n description: >\n Please paste the output of any relevant error logs.\n value: >\n \n\n\n ```python-traceback\n\n\n Replace this line with the error backtrace (if applicable).\n\n\n ```\n\n\n \n - type: textarea\n id: version\n attributes:\n label: Installed Versions\n description: >\n Please paste the output of ``pd.show_versions()``\n value: >\n \n\n\n Replace this line with the output of pd.show_versions()\n\n\n \n validations:\n required: true\n"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.md",
"content": "---\nname: Feature request\nabout: Request a new API or feature implementation\ntitle: ''\nlabels: 'new feature/request 💬, Triage 🩹'\nassignees: ''\n\n---\n\n**Is your feature request related to a problem? Please describe.**\nA clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?\n"
},
{
"path": ".github/ISSUE_TEMPLATE/question.md",
"content": "---\nname: Question\nabout: You want to ask a question\ntitle: ''\nlabels: 'question ❓, Triage 🩹'\nassignees: ''\n\n---\n\n\n"
},
{
"path": ".github/PULL_REQUEST_TEMPLATE.md",
"content": "\n\n## What do these changes do?\n\n\n\n- [x] first commit message and PR title follow format outlined [here](https://modin.readthedocs.io/en/latest/development/contributing.html#commit-message-formatting)\n > **_NOTE:_** If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.\n- [ ] passes `flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py`\n- [ ] passes `black --check modin/ asv_bench/benchmarks scripts/doc_checker.py`\n- [ ] signed commit with `git commit -s` \n- [ ] Resolves #? \n- [ ] tests added and passing\n- [ ] module layout described at `docs/development/architecture.rst` is up-to-date \n"
},
{
"path": ".github/actions/mamba-env/action.yml",
"content": "name: \"Install environment using Mamba\"\ndescription: \"Prepare the environment to run Modin\"\ninputs:\n python-version:\n description: \"Python version to install\"\n default: \"3.9\"\n environment-file:\n description: \"Conda environment yml\"\n required: true\n activate-environment:\n description: \"Conda environment to activate\"\n default: \"modin\"\n\nruns:\n using: \"composite\"\n steps:\n - name: Get current week\n id: get-week\n # use current week as cache key to periodically refresh the cache,\n # as cache is based on requirements, but dependencies push\n # updated versions at some irregular pace\n run: echo \"thisweek=$(/bin/date -u '+%Y.w%W')\" >> $GITHUB_OUTPUT\n shell: bash\n - name: Cache conda\n id: cache-conda\n uses: actions/cache@v4\n with:\n path: |\n ~/conda_pkgs_dir\n ~/.cache/pip\n key:\n ${{ runner.os }}-conda-${{ steps.get-week.outputs.thisweek }}-${{ hashFiles(inputs.environment-file) }}\n - uses: conda-incubator/setup-miniconda@v3\n with:\n miniforge-variant: Miniforge3\n miniforge-version: latest\n use-mamba: true\n activate-environment: ${{ inputs.activate-environment }}\n environment-file: ${{ inputs.environment-file }}\n python-version: ${{ inputs.python-version }}\n channel-priority: strict\n # we set use-only-tar-bz2 to false in order for conda to properly find new packages to be installed\n # for more info see https://github.com/conda-incubator/setup-miniconda/issues/264\n use-only-tar-bz2: false\n - shell: bash -l {0}\n run: |\n conda run -n ${{ inputs.activate-environment }} pip install .\n conda list -n ${{ inputs.activate-environment }}\n"
},
{
"path": ".github/actions/python-only/action.yml",
"content": "name: \"Install Python only\"\ndescription: \"Prepare the environment to run simple tasks\"\ninputs:\n python-version:\n description: \"Python version to install\"\n default: \"3.9\"\n\nruns:\n using: \"composite\"\n steps:\n - uses: actions/setup-python@v5\n with:\n python-version: ${{ inputs.python-version }}\n architecture: \"x64\"\n cache: 'pip'\n"
},
{
"path": ".github/actions/run-core-tests/action.yml",
"content": "name: \"Run core Modin tests\"\ndescription: \"Run core Modin tests like dataframe or groupby\"\ninputs:\n runner:\n description: \"Runner for tests\"\n default: \"python -m pytest\"\n parallel:\n description: \"How to run tests in parallel\"\n default: \"-n 2\"\n\nruns:\n using: \"composite\"\n steps:\n - uses: ./.github/actions/run-core-tests/group_1\n with:\n runner: ${{ inputs.runner }}\n parallel: ${{ inputs.parallel }}\n - uses: ./.github/actions/run-core-tests/group_2\n with:\n runner: ${{ inputs.runner }}\n parallel: ${{ inputs.parallel }}\n - uses: ./.github/actions/run-core-tests/group_3\n with:\n runner: ${{ inputs.runner }}\n parallel: ${{ inputs.parallel }}\n - uses: ./.github/actions/run-core-tests/group_4\n with:\n runner: ${{ inputs.runner }}\n parallel: ${{ inputs.parallel }}\n"
},
{
"path": ".github/actions/run-core-tests/group_1/action.yml",
"content": "name: \"Run core Modin tests - group 1\"\ndescription: \"Run core Modin tests like dataframe or groupby\"\ninputs:\n runner:\n description: \"Runner for tests\"\n default: \"python -m pytest\"\n parallel:\n description: \"How to run tests in parallel\"\n default: \"-n 2\"\n\nruns:\n using: \"composite\"\n steps:\n - run: |\n echo \"::group::Running dataframe tests (group 1)...\"\n ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/dataframe/test_binary.py \\\n modin/tests/pandas/dataframe/test_default.py \\\n modin/tests/pandas/dataframe/test_indexing.py \\\n modin/tests/pandas/dataframe/test_iter.py\n echo \"::endgroup::\"\n shell: bash -l {0}\n"
},
{
"path": ".github/actions/run-core-tests/group_2/action.yml",
"content": "name: \"Run core Modin tests - group 2\"\ndescription: \"Run core Modin tests like dataframe or groupby\"\ninputs:\n runner:\n description: \"Runner for tests\"\n default: \"python -m pytest\"\n parallel:\n description: \"How to run tests in parallel\"\n default: \"-n 2\"\n\nruns:\n using: \"composite\"\n steps:\n - run: |\n echo \"::group::Running dataframe tests (group 2)...\"\n ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/dataframe/test_join_sort.py \\\n modin/tests/pandas/dataframe/test_reduce.py \\\n modin/tests/pandas/dataframe/test_udf.py \\\n modin/tests/pandas/dataframe/test_window.py \\\n modin/tests/pandas/dataframe/test_pickle.py \\\n modin/tests/pandas/test_repartition.py \\\n modin/tests/pandas/test_backend.py\n echo \"::endgroup::\"\n shell: bash -l {0}\n"
},
{
"path": ".github/actions/run-core-tests/group_3/action.yml",
"content": "name: \"Run core Modin tests - group 3\"\ndescription: \"Run core Modin tests like dataframe or groupby\"\ninputs:\n runner:\n description: \"Runner for tests\"\n default: \"python -m pytest\"\n parallel:\n description: \"How to run tests in parallel\"\n default: \"-n 2\"\n\nruns:\n using: \"composite\"\n steps:\n - run: |\n echo \"::group::Running tests (group 3)...\"\n ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/test_series.py \\\n modin/tests/pandas/dataframe/test_map_metadata.py\n echo \"::endgroup::\"\n shell: bash -l {0}\n - run: |\n echo \"::group::Running range-partitioning tests (group 3)...\"\n MODIN_RANGE_PARTITIONING=1 ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/test_groupby.py\n MODIN_RANGE_PARTITIONING=1 ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/test_series.py -k \"test_unique or test_nunique or drop_duplicates or test_resample\"\n MODIN_RANGE_PARTITIONING=1 ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/test_general.py -k \"test_unique\"\n MODIN_RANGE_PARTITIONING=1 ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/dataframe/test_map_metadata.py -k \"drop_duplicates\"\n MODIN_RANGE_PARTITIONING=1 ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/dataframe/test_join_sort.py -k \"merge\"\n MODIN_RANGE_PARTITIONING=1 ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/dataframe/test_default.py -k \"resample\"\n echo \"::endgroup::\"\n shell: bash -l {0}\n"
},
{
"path": ".github/actions/run-core-tests/group_4/action.yml",
"content": "name: \"Run core Modin tests - group 4\"\ndescription: \"Run core Modin tests like dataframe or groupby\"\ninputs:\n runner:\n description: \"Runner for tests\"\n default: \"python -m pytest\"\n parallel:\n description: \"How to run tests in parallel\"\n default: \"-n 2\"\n\nruns:\n using: \"composite\"\n steps:\n - run: |\n echo \"::group::Running tests (group 4)...\"\n ${{ inputs.runner }} ${{ inputs.parallel }} modin/tests/pandas/test_rolling.py \\\n modin/tests/pandas/test_expanding.py \\\n modin/tests/pandas/test_groupby.py \\\n modin/tests/pandas/test_reshape.py \\\n modin/tests/pandas/test_general.py\n echo \"::endgroup::\"\n shell: bash -l {0}\n - run: |\n echo \"::group::Running concat tests (group 4)...\"\n ${{ inputs.runner }} modin/tests/pandas/test_concat.py # Ray and Dask versions fails with -n 2\n echo \"::endgroup::\"\n shell: bash -l {0}\n"
},
{
"path": ".github/actions/upload-coverage/action.yml",
"content": "name: Upload Coverage\ndescription: Upload coverage files\n\nruns:\n using: \"composite\"\n\n steps:\n - run: |\n COVERAGE_UUID=$(python3 -c \"import uuid; print(uuid.uuid4())\")\n mv .coverage .coverage.${COVERAGE_UUID}\n echo \"COVERAGE_UUID=${COVERAGE_UUID}\" >> $GITHUB_ENV\n id: coverage-uuid\n shell: bash\n - uses: actions/upload-artifact@v4\n with:\n name: coverage-data-${{ env.COVERAGE_UUID }}\n path: .coverage*\n include-hidden-files: true\n"
},
{
"path": ".github/dependabot.yaml",
"content": "version: 2\nupdates:\n - package-ecosystem: \"github-actions\"\n directory: \"/\"\n schedule:\n interval: \"monthly\"\n groups:\n github-actions:\n patterns:\n - \"*\"\n"
},
{
"path": ".github/stale.yml",
"content": "# Number of days of inactivity before an Issue or Pull Request becomes stale\ndaysUntilStale: 365\n\n# Number of days of inactivity before an Issue or Pull Request with the stale label is closed.\n# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale.\ndaysUntilClose: 7\n\n# Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled)\nonlyLabels: []\n\n# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable\nexemptLabels: []\n\n# Set to true to ignore issues in a project (defaults to false)\nexemptProjects: false\n\n# Set to true to ignore issues in a milestone (defaults to false)\nexemptMilestones: false\n\n# Set to true to ignore issues with an assignee (defaults to false)\nexemptAssignees: false\n\n# Label to use when marking as stale\nstaleLabel: stale\n\n# Comment to post when marking as stale. Set to `false` to disable\nmarkComment: >\n This issue has been automatically marked as stale because it has not had\n recent activity. It will be closed if no further activity occurs within the next\n 7 days. Thank you for your contributions.\n\n# Comment to post when removing the stale label.\n# unmarkComment: >\n# Your comment here.\n\n# Comment to post when closing a stale Issue or Pull Request.\n closeComment: >\n Closing as stale.\n"
},
{
"path": ".github/workflows/ci-notebooks.yml",
"content": "name: ci-notebooks\non:\n pull_request:\n paths:\n - modin/**\n - examples/tutorial/**\n - .github/workflows/ci-notebooks.yml\n - setup.cfg\n - setup.py\n - requirements/env_unidist_linux.yml\nconcurrency:\n # Cancel other jobs in the same branch. We don't care whether CI passes\n # on old commits.\n group: ${{ github.workflow }}-${{ github.ref }}\n cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}\nenv:\n MODIN_GITHUB_CI: true\njobs:\n test-tutorial-notebooks:\n defaults:\n run:\n shell: bash -l {0}\n name: test tutorial notebooks\n runs-on: ubuntu-latest\n strategy:\n matrix:\n execution: [pandas_on_ray, pandas_on_dask, pandas_on_unidist]\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/python-only\n if: matrix.execution != 'pandas_on_unidist'\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: requirements/env_unidist_linux.yml\n activate-environment: modin_on_unidist\n if: matrix.execution == 'pandas_on_unidist'\n - name: Cache datasets\n uses: actions/cache@v4\n with:\n path: taxi.csv\n # update cache only if notebooks require it to be changed\n key: taxi-csv-dataset-${{ hashFiles('examples/tutorial/jupyter/**') }}\n # replace modin with . in the tutorial requirements file for `pandas_on_ray` and\n # `pandas_on_dask` since we need Modin built from sources\n - run: sed -i 's/modin/./g' examples/tutorial/jupyter/execution/${{ matrix.execution }}/requirements.txt\n if: matrix.execution != 'pandas_on_unidist'\n # install dependencies required for notebooks execution for `pandas_on_ray` and `pandas_on_dask`\n # Override modin-spreadsheet install for now\n - run: |\n pip install -r examples/tutorial/jupyter/execution/${{ matrix.execution }}/requirements.txt\n pip install git+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5\n if: matrix.execution != 'pandas_on_unidist'\n # Build Modin from sources for `pandas_on_unidist`\n - run: pip install -e .\n if: matrix.execution == 'pandas_on_unidist'\n # install test dependencies\n # NOTE: If you are changing the set of packages installed here, make sure that\n # the dev requirements match them.\n - run: pip install pytest pytest-cov black flake8 flake8-print flake8-no-implicit-concat\n if: matrix.execution != 'pandas_on_unidist'\n - run: pip install flake8-print jupyter nbformat nbconvert\n if: matrix.execution == 'pandas_on_unidist'\n - run: pip list\n if: matrix.execution != 'pandas_on_unidist'\n - run: |\n conda info\n conda list\n if: matrix.execution == 'pandas_on_unidist'\n # setup kernel configuration for `pandas_on_unidist` execution with mpi backend\n - run: python examples/tutorial/jupyter/execution/${{ matrix.execution }}/setup_kernel.py\n if: matrix.execution == 'pandas_on_unidist'\n - run: jupyter kernelspec list\n - run: |\n black --check --diff examples/tutorial/jupyter/execution/${{ matrix.execution }}/test/test_notebooks.py\n black --check --diff examples/tutorial/jupyter/execution/test/utils.py\n - run: |\n flake8 --enable=T examples/tutorial/jupyter/execution/${{ matrix.execution }}/test/test_notebooks.py\n flake8 --enable=T examples/tutorial/jupyter/execution/test/utils.py\n - run: python -m pytest examples/tutorial/jupyter/execution/${{ matrix.execution }}/test/test_notebooks.py\n"
},
{
"path": ".github/workflows/ci-required.yml",
"content": "name: ci-required\non: pull_request\nconcurrency:\n # Cancel other jobs in the same branch. We don't care whether CI passes\n # on old commits.\n group: ${{ github.workflow }}-${{ github.ref }}\n cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}\nenv:\n MODIN_GITHUB_CI: true\n\njobs:\n\n check-pr-title:\n runs-on: ubuntu-latest\n steps:\n - uses: Slashgear/action-check-pr-title@v4.3.0\n with:\n # NOTE: If you change the allowed prefixes here, update\n # the documentation about them in /docs/development/contributing.rst\n regexp: '^(?:FEAT|DOCS|FIX|REFACTOR|TEST|PERF)-#\\d+:'\n\n build-docs:\n name: build docs\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n with:\n fetch-depth: 1\n - uses: actions/setup-python@v5\n with:\n python-version: \"3.9\"\n architecture: \"x64\"\n cache: \"pip\"\n cache-dependency-path: '**/requirements-doc.txt'\n - run: pip install -r docs/requirements-doc.txt\n - run: cd docs && sphinx-build -T -E -W -b html . build\n\n lint-pydocstyle:\n name: lint (pydocstyle)\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/python-only\n # The `numpydoc` version here MUST match the versions in the dev requirements files.\n - run: pip install pytest pytest-cov pydocstyle numpydoc==1.6.0\n - run: python -m pytest scripts/test\n - run: pip install -e \".[all]\"\n - run: |\n python scripts/doc_checker.py --add-ignore=D101,D102,D103,D105 --disable-numpydoc \\\n modin/pandas/dataframe.py modin/pandas/series.py \\\n modin/pandas/groupby.py \\\n modin/pandas/series_utils.py modin/pandas/general.py \\\n modin/pandas/plotting.py modin/pandas/utils.py \\\n modin/pandas/iterator.py modin/pandas/indexing.py \\\n - run: python scripts/doc_checker.py modin/core/dataframe\n - run: python scripts/doc_checker.py modin/core/execution/dask\n - run: |\n python scripts/doc_checker.py \\\n modin/pandas/accessor.py modin/pandas/general.py \\\n modin/pandas/groupby.py modin/pandas/indexing.py \\\n modin/pandas/iterator.py modin/pandas/plotting.py \\\n modin/pandas/series_utils.py modin/pandas/utils.py \\\n modin/pandas/base.py \\\n modin/pandas/io.py \\\n asv_bench/benchmarks/utils \\\n asv_bench/benchmarks/__init__.py asv_bench/benchmarks/io/__init__.py \\\n asv_bench/benchmarks/scalability/__init__.py \\\n modin/core/io \\\n modin/pandas/series.py \\\n modin/core/execution/python \\\n modin/pandas/dataframe.py \\\n modin/config/__init__.py \\\n modin/config/__main__.py \\\n modin/config/envvars.py \\\n modin/config/pubsub.py\n - run: python scripts/doc_checker.py modin/distributed\n - run: python scripts/doc_checker.py modin/utils.py\n - run: python scripts/doc_checker.py modin/experimental/sklearn\n - run: |\n python scripts/doc_checker.py modin/experimental/xgboost/__init__.py \\\n modin/experimental/xgboost/utils.py modin/experimental/xgboost/xgboost.py \\\n modin/experimental/xgboost/xgboost_ray.py\n - run: python scripts/doc_checker.py modin/core/execution/ray\n - run: |\n python scripts/doc_checker.py modin/core/execution/dispatching/factories/factories.py \\\n modin/core/execution/dispatching/factories/dispatcher.py \\\n - run: python scripts/doc_checker.py scripts/doc_checker.py\n - run: |\n python scripts/doc_checker.py modin/experimental/pandas/io.py \\\n modin/experimental/pandas/__init__.py\n - run: python scripts/doc_checker.py modin/core/storage_formats/base\n - run: python scripts/doc_checker.py modin/core/storage_formats/pandas\n - run: python scripts/doc_checker.py modin/experimental/batch/pipeline.py\n - run: python scripts/doc_checker.py modin/logging\n\n lint-black-isort:\n name: lint (black and isort)\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/python-only\n - run: pip install black>=24.1.0 isort>=5.12\n # NOTE: keep the black command here in sync with the pre-commit hook in\n # /contributing/pre-commit\n - run: black --check --diff modin/ asv_bench/benchmarks scripts/doc_checker.py\n - run: isort . --check-only\n"
},
{
"path": ".github/workflows/ci.yml",
"content": "name: ci\non:\n pull_request:\n paths:\n # NOTE: keep these paths in sync with the paths that trigger the\n # fuzzydata Github Actions in .github/workflows/fuzzydata-test.yml\n - .github/workflows/**\n - .github/actions/**\n - '!.github/workflows/push-to-main.yml'\n - asv_bench/**\n - modin/**\n - requirements/**\n - scripts/**\n - environment-dev.yml\n - requirements-dev.txt\n - setup.cfg\n - setup.py\n - versioneer.py\n push:\n schedule:\n - cron: \"30 2 * * WED\"\n - cron: \"30 2 * * THU\"\nconcurrency:\n # Cancel other jobs in the same branch. We don't care whether CI passes\n # on old commits.\n group: ${{ github.workflow }}-${{ github.ref }}\n cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}\nenv:\n MODIN_GITHUB_CI: true\n\njobs:\n python-filter:\n runs-on: ubuntu-latest\n outputs:\n python-version: ${{ steps.choose.outputs.python-version }}\n steps:\n - id: choose\n run: |\n if [[ \"${{ github.event.schedule }}\" = \"30 2 * * WED\" ]]\n then\n echo \"python-version=3.10\" >> \"$GITHUB_OUTPUT\"\n elif [[ \"${{ github.event.schedule }}\" = \"30 2 * * THU\" ]]\n then\n echo \"python-version=3.11\" >> \"$GITHUB_OUTPUT\"\n else\n echo \"python-version=3.9\" >> \"$GITHUB_OUTPUT\"\n fi\n\n lint-mypy:\n needs: [python-filter]\n name: lint (mypy)\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/python-only\n with:\n python-version: ${{ needs.python-filter.outputs.python-version }}\n - run: pip install -r requirements-dev.txt\n - run: mypy --config-file mypy.ini\n\n lint-flake8:\n needs: [python-filter]\n name: lint (flake8)\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/python-only\n with:\n python-version: ${{ needs.python-filter.outputs.python-version }}\n # NOTE: If you are changing the set of packages installed here, make sure that\n # the dev requirements match them.\n - run: pip install flake8 flake8-print flake8-no-implicit-concat\n # NOTE: keep the flake8 command here in sync with the pre-commit hook in\n # /contributing/pre-commit\n - run: flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py\n\n test-api-and-no-engine:\n needs: [python-filter]\n name: Test API, headers and no-engine mode\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: requirements/requirements-no-engine.yml\n python-version: ${{ needs.python-filter.outputs.python-version }}\n - run: python -m pytest modin/tests/pandas/test_api.py\n - run: python -m pytest modin/tests/test_executions_api.py\n - run: python -m pytest modin/tests/test_headers.py\n - run: python -m pytest modin/tests/core/test_dispatcher.py::test_add_option\n - uses: ./.github/actions/upload-coverage\n\n test-clean-install:\n needs: [lint-flake8, python-filter]\n strategy:\n matrix:\n os:\n - ubuntu\n - windows\n runs-on: ${{ matrix.os }}-latest\n defaults:\n run:\n shell: bash -l {0}\n name: test-clean-install-${{ matrix.os }}\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/python-only\n with:\n python-version: ${{ needs.python-filter.outputs.python-version }}\n - run: python -m pip install -e \".[all]\"\n - name: Ensure Ray and Dask engines start up\n run: |\n MODIN_ENGINE=dask python -c \"import modin.pandas as pd; print(pd.DataFrame([1,2,3]))\"\n MODIN_ENGINE=ray python -c \"import modin.pandas as pd; print(pd.DataFrame([1,2,3]))\"\n - name: Ensure MPI engine start up\n # Install a working MPI implementation beforehand so mpi4py can link to it\n run: |\n sudo apt-get update\n sudo apt-get install software-properties-common\n\n sudo add-apt-repository \"deb http://archive.ubuntu.com/ubuntu jammy main universe restricted multiverse\"\n sudo add-apt-repository \"deb http://archive.ubuntu.com/ubuntu jammy-updates main universe restricted multiverse\"\n sudo add-apt-repository \"deb http://security.ubuntu.com/ubuntu jammy-security main universe restricted multiverse\"\n sudo apt-get update\n\n sudo apt-get install libmpich-dev=4.0-3 libmpich12=4.0-3 mpich=4.0-3\n python -m pip install -e \".[mpi]\"\n # mpi4py 4.1 does not work with the mpich versions above.\n # TODO(https://github.com/modin-project/modin/issues/7615): figure out\n # the correct libmpich versions for mpi4py >= 4.1\n python -m pip install \"mpi4py<4.1\"\n MODIN_ENGINE=unidist UNIDIST_BACKEND=mpi mpiexec -n 1 python -c \"import modin.pandas as pd; print(pd.DataFrame([1,2,3]))\"\n if: matrix.os == 'ubuntu'\n\n test-internals:\n needs: [lint-flake8, python-filter]\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n name: test-internals\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: environment-dev.yml\n python-version: ${{ needs.python-filter.outputs.python-version }}\n - name: Internals tests\n run: python -m pytest modin/tests/core/test_dispatcher.py\n - run: python -m pytest modin/tests/config\n - run: python -m pytest modin/tests/test_envvar_catcher.py\n - run: python -m pytest modin/tests/core/storage_formats/base/test_internals.py\n - run: python -m pytest modin/tests/core/storage_formats/pandas/test_internals.py\n - run: python -m pytest modin/tests/test_envvar_npartitions.py\n - run: python -m pytest modin/tests/test_utils.py\n - run: python -m pytest asv_bench/test/test_utils.py\n - run: python -m pytest modin/tests/interchange/dataframe_protocol/base\n - run: python -m pytest modin/tests/test_dataframe_api_standard.py\n - run: python -m pytest modin/tests/test_logging.py\n - run: python -m pytest modin/tests/test_metrics.py\n - run: python -m pytest modin/tests/pandas/extensions\n - uses: ./.github/actions/upload-coverage\n\n test-defaults:\n needs: [lint-flake8, python-filter]\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n strategy:\n matrix:\n execution: [BaseOnPython]\n env:\n MODIN_TEST_DATASET_SIZE: \"small\"\n name: Test ${{ matrix.execution }} execution, Python ${{ needs.python-filter.outputs.python-version }}\"\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: environment-dev.yml\n python-version: ${{ needs.python-filter.outputs.python-version }}\n - name: Install HDF5\n run: sudo apt update && sudo apt install -y libhdf5-dev\n - name: xgboost tests\n run: |\n # TODO(https://github.com/modin-project/modin/issues/5194): Uncap xgboost\n # when we use collective instead of rabit.\n # Per the thread https://github.com/conda-forge/miniforge/issues/513,\n # remove unused conda packages and caches to avoid `Found incorrect\n # download: joblib` error from mamba.\n mamba clean --all\n mamba install \"xgboost>=1.7.1,<2.0.0\" scikit-learn -c conda-forge\n python -m pytest modin/tests/experimental/xgboost/test_default.py --execution=${{ matrix.execution }}\n - run: python -m pytest -n 2 modin/tests/core/storage_formats/base/test_internals.py --execution=${{ matrix.execution }}\n - uses: ./.github/actions/run-core-tests\n with:\n runner: python -m pytest --execution=${{ matrix.execution }}\n - uses: ./.github/actions/upload-coverage\n\n test-asv-benchmarks:\n if: github.event_name == 'pull_request'\n needs: [lint-flake8]\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n env:\n MODIN_ENGINE: ray\n MODIN_MEMORY: 1000000000\n MODIN_TEST_DATASET_SIZE: small\n name: test-asv-benchmarks\n steps:\n - uses: actions/checkout@v4\n with:\n fetch-depth: 1\n - uses: conda-incubator/setup-miniconda@v3\n with:\n auto-activate-base: true\n activate-environment: \"\"\n miniforge-variant: Miniforge3\n miniforge-version: latest\n use-mamba: true\n - name: Running benchmarks\n run: |\n git remote add upstream https://github.com/modin-project/modin.git\n git fetch upstream\n if git diff upstream/main --name-only | grep -q \"^asv_bench/\"; then\n cd asv_bench\n\n mamba env create -f ../environment-dev.yml\n conda activate modin\n pip install ..\n\n asv machine --yes\n\n # check Modin on Ray\n asv run --quick --dry-run --python=same --strict --show-stderr --launch-method=spawn \\\n -b ^benchmarks -b ^io -b ^scalability | tee benchmarks.log\n\n # check pure pandas\n MODIN_ASV_USE_IMPL=pandas asv run --quick --dry-run --python=same --strict --show-stderr --launch-method=spawn \\\n -b ^benchmarks -b ^io | tee benchmarks.log\n else\n echo \"Benchmarks did not run, no changes detected\"\n fi\n if: always()\n\n - name: Publish benchmarks artifact\n uses: actions/upload-artifact@v4\n with:\n name: Benchmarks log\n path: asv_bench/benchmarks.log\n include-hidden-files: true\n if: failure()\n\n execution-filter:\n # Choose which executions we want to run all tests for on a pull request.\n # We always test 'native' and 'python' executions completely because they\n # are fast, but we only test ray, dask, and unidist, if we think this pull\n # request is affecting how we execute with those engines specifically.\n runs-on: ubuntu-latest\n outputs:\n ray: ${{ steps.filter.outputs.ray }}\n dask: ${{ steps.filter.outputs.dask }}\n unidist: ${{ steps.filter.outputs.unidist }}\n engines: ${{ steps.engines.outputs.engines }}\n experimental: ${{ steps.experimental.outputs.experimental }}\n steps:\n - uses: actions/checkout@v4\n - uses: dorny/paths-filter@v3\n id: filter\n with:\n filters: |\n shared: &shared\n - 'modin/core/execution/dispatching/**'\n ray:\n - *shared\n - 'modin/core/execution/ray/**'\n dask:\n - *shared\n - 'modin/core/execution/dask/**'\n unidist:\n - *shared\n - 'modin/core/execution/unidist/**'\n experimental:\n - 'modin/experimental/**'\n - uses: actions/setup-python@v5\n - id: engines\n run: |\n python -c \"import sys, json; print('engines=' + json.dumps(['python', 'native'] + (sys.argv[1] == 'true' and ['ray'] or []) + (sys.argv[2] == 'true' and ['dask'] or []) ))\" \\\n \"${{ steps.filter.outputs.ray }}\" \"${{ steps.filter.outputs.dask }}\" >> $GITHUB_OUTPUT\n\n test-all-unidist:\n needs: [lint-flake8, execution-filter, python-filter]\n if: github.event_name == 'push' || needs.execution-filter.outputs.unidist == 'true'\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n strategy:\n matrix:\n python-version: [ \"${{ needs.python-filter.outputs.python-version }}\" ]\n unidist-backend: [\"mpi\"]\n env:\n MODIN_ENGINE: \"Unidist\"\n UNIDIST_BACKEND: ${{matrix.unidist-backend}}\n # Only test reading from SQL server and postgres on ubuntu for now.\n # Eventually, we should test on Windows, too, but we will have to set up\n # the servers differently.\n MODIN_TEST_READ_FROM_SQL_SERVER: true\n MODIN_TEST_READ_FROM_POSTGRES: true\n name: test-ubuntu (engine unidist ${{matrix.unidist-backend}}, python ${{matrix.python-version}})\n services:\n moto:\n image: motoserver/moto:5.0.13\n ports:\n - 5000:5000\n env:\n AWS_ACCESS_KEY_ID: foobar_key\n AWS_SECRET_ACCESS_KEY: foobar_secret\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: requirements/env_unidist_linux.yml\n activate-environment: modin_on_unidist\n python-version: ${{matrix.python-version}}\n - name: Install HDF5\n run: sudo apt update && sudo apt install -y libhdf5-dev\n - name: Set up postgres\n # Locally, specifying port 2345:5432 works, but 2345:2345 and 5432:5432 do not. This solution is from\n # https://stackoverflow.com/questions/36415654/cant-connect-docker-postgresql-9-3\n run: |\n sudo docker pull postgres\n sudo docker run --name some-postgres -e POSTGRES_USER=sa -e POSTGRES_PASSWORD=Strong.Pwd-123 -e POSTGRES_DB=postgres -d -p 2345:5432 postgres\n - run: mpiexec -n 1 python -m pytest modin/tests/pandas/internals/test_benchmark_mode.py\n - run: mpiexec -n 1 python -m pytest modin/tests/test_partition_api.py\n - uses: ./.github/actions/run-core-tests\n with:\n runner: mpiexec -n 1 python -m pytest\n parallel: \"\"\n - run: mpiexec -n 1 python -m pytest modin/tests/numpy\n - run: chmod +x ./.github/workflows/sql_server/set_up_sql_server.sh\n - run: ./.github/workflows/sql_server/set_up_sql_server.sh\n # need an extra argument \"genv\" to set environment variables for mpiexec. We need\n # these variables to test writing to the mock s3 filesystem.\n - uses: nick-fields/retry@v3\n # to avoid issues with non-stable `to_csv` tests for unidist on MPI backend.\n # for details see: https://github.com/modin-project/modin/pull/6776\n with:\n timeout_minutes: 15\n max_attempts: 3\n command: |\n conda run --no-capture-output -n modin_on_unidist mpiexec -n 1 -genv AWS_ACCESS_KEY_ID foobar_key \\\n -genv AWS_SECRET_ACCESS_KEY foobar_secret python -m pytest modin/tests/pandas/test_io.py --verbose\n - run: |\n mpiexec -n 1 -genv AWS_ACCESS_KEY_ID foobar_key -genv AWS_SECRET_ACCESS_KEY foobar_secret \\\n python -m pytest modin/tests/experimental/test_io_exp.py\n - run: mpiexec -n 1 python -m pytest modin/tests/interchange/dataframe_protocol/test_general.py\n - run: mpiexec -n 1 python -m pytest modin/tests/interchange/dataframe_protocol/pandas/test_protocol.py\n - run: |\n python -m pip install lazy_import\n mpiexec -n 1 python -m pytest modin/tests/pandas/integrations/\n - uses: ./.github/actions/upload-coverage\n\n test-all:\n needs: [lint-flake8, execution-filter, python-filter]\n strategy:\n matrix:\n os:\n - ubuntu\n - windows\n python-version: [ \"${{ needs.python-filter.outputs.python-version }}\" ]\n # On push, run the tests for all engines. Otherwise, for pull requests,\n # only run tests for engines that depend on files changed in this PR.\n engine: ${{ fromJSON( (github.event_name == 'push' && '[\"python\", \"ray\", \"dask\", \"native\"]') || needs.execution-filter.outputs.engines ) }}\n test_task:\n - group_1\n - group_2\n - group_3\n - group_4\n exclude: # python and native engines only have one task group that contains all the tests\n - engine: \"python\"\n test_task: \"group_2\"\n - engine: \"native\"\n test_task: \"group_2\"\n - engine: \"python\"\n test_task: \"group_3\"\n - engine: \"native\"\n test_task: \"group_3\"\n - engine: \"python\"\n test_task: \"group_4\"\n - engine: \"native\"\n test_task: \"group_4\"\n runs-on: ${{ matrix.os }}-latest\n defaults:\n run:\n shell: bash -l {0}\n env:\n MODIN_ENGINE: ${{matrix.engine}}\n # Only test reading from SQL server and postgres on ubuntu for now.\n # Eventually, we should test on Windows, too, but we will have to set up\n # the servers differently.\n MODIN_TEST_READ_FROM_SQL_SERVER: ${{ matrix.os == 'ubuntu' }}\n MODIN_TEST_READ_FROM_POSTGRES: ${{ matrix.os == 'ubuntu' }}\n name: test-${{ matrix.os }} (engine ${{matrix.engine}}, python ${{matrix.python-version}}, ${{matrix.test_task}})\n services:\n # Using workaround https://github.com/actions/runner/issues/822#issuecomment-1524826092\n moto:\n # we only need moto service on Ubuntu and for group_4 task, or for native or python engine.\n image: ${{ (matrix.os == 'ubuntu' && (matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_4')) && 'motoserver/moto:5.0.13' || '' }}\n ports:\n - 5000:5000\n env:\n AWS_ACCESS_KEY_ID: foobar_key\n AWS_SECRET_ACCESS_KEY: foobar_secret\n steps:\n - name: Set native storage format\n run: echo \"MODIN_STORAGE_FORMAT=Native\" >> $GITHUB_ENV\n if: matrix.engine == 'native'\n - name: Limit ray memory\n run: echo \"MODIN_MEMORY=1000000000\" >> $GITHUB_ENV\n if: matrix.os == 'ubuntu' && matrix.engine == 'ray'\n - name: Tell Modin to use existing ray cluster\n run: echo \"MODIN_RAY_CLUSTER=True\" >> $GITHUB_ENV\n if: matrix.os == 'windows' && matrix.engine == 'ray'\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: environment-dev.yml\n python-version: ${{matrix.python-version}}\n - name: Start local ray cluster\n # Try a few times to start ray to work around\n # https://github.com/modin-project/modin/issues/4562\n uses: nick-fields/retry@v3\n with:\n timeout_minutes: 5\n max_attempts: 5\n command: ray start --head --port=6379 --object-store-memory=1000000000\n if: matrix.os == 'windows' && matrix.engine == 'ray'\n - name: Install HDF5\n run: sudo apt update && sudo apt install -y libhdf5-dev\n if: matrix.os == 'ubuntu'\n - name: Set up postgres\n # Locally, specifying port 2345:5432 works, but 2345:2345 and 5432:5432 do not. This solution is from\n # https://stackoverflow.com/questions/36415654/cant-connect-docker-postgresql-9-3\n run: |\n sudo docker pull postgres\n sudo docker run --name some-postgres -e POSTGRES_USER=sa -e POSTGRES_PASSWORD=Strong.Pwd-123 -e POSTGRES_DB=postgres -d -p 2345:5432 postgres\n if: matrix.os == 'ubuntu'\n\n # BEGIN partitioned execution tests. We run these tests along with group 1,\n # or if we are on the \"python\" engine, which only has a single group. We\n # skip these tests on the \"native\" engine, which does not use partitions.\n\n - run: python -m pytest modin/tests/pandas/internals/test_benchmark_mode.py\n if: matrix.engine != 'native' && (matrix.engine == 'python' || matrix.test_task == 'group_1')\n - run: python -m pytest modin/tests/test_partition_api.py\n # Skip this test for python because we do not define unwrap_partitions()\n # for python execution.\n if: matrix.engine != 'native' && matrix.engine != 'python' && matrix.test_task == 'group_1'\n - name: xgboost tests\n run: |\n # TODO(https://github.com/modin-project/modin/issues/5194): Uncap xgboost\n # when we use collective instead of rabit.\n mamba install \"xgboost>=1.7.1,<2.0.0\" scikit-learn -c conda-forge\n python -m pytest -n 2 \\\n modin/tests/experimental/xgboost/test_default.py \\\n modin/tests/experimental/xgboost/test_xgboost.py \\\n modin/tests/experimental/xgboost/test_dmatrix.py\n if: matrix.engine != 'native' && matrix.os != 'windows' && (matrix.engine == 'python' || matrix.test_task == 'group_1')\n - run: python -m pytest -n 2 modin/tests/experimental/test_pipeline.py\n if: matrix.engine != 'native' && (matrix.engine == 'python' || matrix.test_task == 'group_1')\n\n\n # END partitioned execution tests.\n\n\n # BEGIN test groups.\n # Run all the tests in the corresponding group for this instance of the\n # test matrix. For example, if we are in the matrix's 'group_4', run the\n # tests for 'group_4'. For each of 'native' and 'python' engines, we run\n # all tests in a single job, so we ignore the grouping.\n\n - uses: ./.github/actions/run-core-tests/group_1\n with:\n # When running with Ray engine on Windows using 2 pytest workers tests are failing in CI.\n # See https://github.com/modin-project/modin/issues/7387.\n parallel: ${{ matrix.engine == 'ray' && matrix.os == 'windows' && '-n 1' || '-n 2' }}\n if: matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_1'\n - uses: ./.github/actions/run-core-tests/group_2\n with:\n # When running with Ray engine on Windows using 2 pytest workers tests are failing in CI.\n # See https://github.com/modin-project/modin/issues/7387.\n parallel: ${{ matrix.engine == 'ray' && matrix.os == 'windows' && '-n 1' || '-n 2' }}\n if: matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_2'\n - uses: ./.github/actions/run-core-tests/group_3\n with:\n # When running with Ray engine on Windows using 2 pytest workers tests are failing in CI.\n # See https://github.com/modin-project/modin/issues/7387.\n parallel: ${{ matrix.engine == 'ray' && matrix.os == 'windows' && '-n 1' || '-n 2' }}\n if: matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_3'\n - uses: ./.github/actions/run-core-tests/group_4\n with:\n # When running with Ray engine on Windows using 2 pytest workers tests are failing in CI.\n # See https://github.com/modin-project/modin/issues/7387.\n parallel: ${{ matrix.engine == 'ray' && matrix.os == 'windows' && '-n 1' || '-n 2' }}\n if: matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_4'\n - run: python -m pytest -n 2 modin/tests/numpy\n # Native execution does not support the modin Numpy API.\n if: matrix.engine == 'python' || matrix.test_task == 'group_4'\n\n # END test groups.\n\n\n # BEGIN some tests that we run along with group 4 for engines other than\n # 'native' and 'python'. 'native' and 'python' jobs will run these tests\n # along with all other tests in a single group.\n\n - run: chmod +x ./.github/workflows/sql_server/set_up_sql_server.sh\n if: matrix.os == 'ubuntu' && (matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_4')\n - run: ./.github/workflows/sql_server/set_up_sql_server.sh\n if: matrix.os == 'ubuntu' && (matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_4')\n # Do not add parallelism (`-n` argument) here - it will cause mock S3 service to fail.\n - run: python -m pytest modin/tests/pandas/test_io.py --verbose\n timeout-minutes: 60\n if: matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_4'\n - run: python -m pytest modin/tests/experimental/test_io_exp.py\n if: matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_4'\n - run: python -m pytest modin/tests/interchange/dataframe_protocol/test_general.py\n if: matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_4'\n - run: python -m pytest modin/tests/interchange/dataframe_protocol/pandas/test_protocol.py\n if: matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_4'\n - run: python -m pytest modin/tests/polars/test_dataframe.py\n - run: |\n python -m pip install lazy_import\n python -m pytest modin/tests/pandas/integrations/\n if: matrix.engine == 'python' || matrix.engine == 'native' || matrix.test_task == 'group_4'\n\n\n # END tests that run on group 4, or in the single group for 'native' and\n # python' engines.\n\n - uses: ./.github/actions/upload-coverage\n - name: Stop local ray cluster\n run: ray stop\n if: matrix.os == 'windows' && matrix.engine == 'ray'\n\n test-sanity:\n # The \"sanity\" tests run on each pull request to test that a subset of the\n # full tests work with the slower engines (ray, dask, and unidist-MPI).\n needs: [lint-flake8, execution-filter, python-filter]\n # If we don't need to run any sanity tests, the job matrix that we generate\n # here gives a single job with all the matrix fields empty (that is, os,\n # execution, etc. are not set, so we treat them as \"\").\n # so, if the matrix is going to be empty, we need to skip this job\n # completely. This bizarre behavior is not in the official documentation,\n # of GitHub actions matrices, but someone does mention it here:\n # https://stackoverflow.com/a/77118991\n if: |\n github.event_name == 'pull_request' &&\n (\n needs.execution-filter.outputs.ray != 'true' ||\n needs.execution-filter.outputs.dask != 'true' ||\n needs.execution-filter.outputs.unidist != 'true'\n )\n strategy:\n matrix:\n os:\n - ubuntu\n - windows\n python-version: [ \"${{ needs.python-filter.outputs.python-version }}\" ]\n running-all-ray-tests: [ \"${{ needs.execution-filter.outputs.ray }}\" ]\n running-all-dask-tests: [ \"${{needs.execution-filter.outputs.dask}}\" ]\n running-all-unidist-tests: [ \"${{needs.execution-filter.outputs.unidist}}\" ]\n execution: [ray, dask, unidist]\n # If we're going to run all ray tests because we've detected a\n # change to the ray engine, we don't need to run these sanity tests\n # on ray. Likewise for dask and unidist.\n exclude:\n - running-all-ray-tests: 'true'\n execution: ray\n - running-all-dask-tests: 'true'\n execution: dask\n - running-all-unidist-tests: 'true'\n execution: unidist\n runs-on: ${{ matrix.os }}-latest\n defaults:\n run:\n shell: bash -l {0}\n env:\n MODIN_ENGINE: ${{ matrix.execution }}\n UNIDIST_BACKEND: \"mpi\"\n PARALLEL: ${{ matrix.execution != 'unidist' && matrix.os != 'windows' && '-n 2' || '' }}\n PYTEST_COMMAND: >-\n ${{\n (\n (matrix.execution == 'ray' || matrix.execution == 'dask') &&\n 'python -m pytest'\n ) ||\n (\n matrix.execution == 'unidist' &&\n 'mpiexec -n 1 -genv AWS_ACCESS_KEY_ID foobar_key -genv AWS_SECRET_ACCESS_KEY foobar_secret python -m pytest'\n ) ||\n 'UNKNOWN_PYTEST_COMMAND'\n }}\n name: test-${{ matrix.os }}-sanity (engine ${{ matrix.execution }}, python ${{matrix.python-version}})\n services:\n moto:\n image: ${{ matrix.os != 'windows' && 'motoserver/moto:5.0.13' || '' }}\n ports:\n - 5000:5000\n env:\n AWS_ACCESS_KEY_ID: foobar_key\n AWS_SECRET_ACCESS_KEY: foobar_secret\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: ${{ matrix.os == 'ubuntu' && matrix.execution == 'unidist' && 'requirements/env_unidist_linux.yml' || matrix.os == 'windows' && matrix.execution == 'unidist' && 'requirements/env_unidist_win.yml' || 'environment-dev.yml' }}\n activate-environment: ${{ matrix.execution == 'unidist' && 'modin_on_unidist' || 'modin' }}\n python-version: ${{matrix.python-version}}\n - name: Install HDF5\n run: sudo apt update && sudo apt install -y libhdf5-dev\n if: matrix.os != 'windows'\n - name: Limit ray memory\n run: echo \"MODIN_MEMORY=1000000000\" >> $GITHUB_ENV\n if: matrix.os != 'windows' && matrix.execution == 'ray'\n - name: Tell Modin to use existing ray cluster\n run: echo \"MODIN_RAY_CLUSTER=True\" >> $GITHUB_ENV\n if: matrix.os == 'windows' && matrix.execution == 'ray'\n - name: Start local ray cluster\n # Try a few times to start ray to work around\n # https://github.com/modin-project/modin/issues/4562\n uses: nick-fields/retry@v3\n with:\n timeout_minutes: 5\n max_attempts: 5\n command: ray start --head --port=6379 --object-store-memory=1000000000\n if: matrix.os == 'windows' && matrix.execution == 'ray'\n - run: MODIN_BENCHMARK_MODE=True $PYTEST_COMMAND modin/tests/pandas/internals/test_benchmark_mode.py\n - run: $PYTEST_COMMAND $PARALLEL modin/tests/test_partition_api.py\n - run: $PYTEST_COMMAND modin/tests/pandas/extensions\n - name: xgboost tests\n run: |\n # TODO(https://github.com/modin-project/modin/issues/5194): Uncap xgboost\n # when we use collective instead of rabit.\n mamba install \"xgboost>=1.7.1,<2.0.0\" scikit-learn -c conda-forge\n $PYTEST_COMMAND $PARALLEL \\\n modin/tests/experimental/xgboost/test_default.py \\\n modin/tests/experimental/xgboost/test_xgboost.py \\\n modin/tests/experimental/xgboost/test_dmatrix.py\n if: matrix.os != 'windows' && needs.execution-filter.outputs.experimental == 'true'\n - run: $PYTEST_COMMAND $PARALLEL modin/tests/experimental/test_pipeline.py\n if: matrix.os != 'windows' && matrix.execution != 'unidist' && needs.execution-filter.outputs.experimental == 'true'\n - name: \"test DF: binary, default, iter\"\n run: |\n $PYTEST_COMMAND $PARALLEL \\\n modin/tests/pandas/dataframe/test_binary.py \\\n modin/tests/pandas/dataframe/test_default.py \\\n modin/tests/pandas/dataframe/test_iter.py\n if: matrix.os != 'windows'\n - name: \"test DF: reduce, udf, window, pickle\"\n run: |\n $PYTEST_COMMAND $PARALLEL \\\n modin/tests/pandas/dataframe/test_reduce.py \\\n modin/tests/pandas/dataframe/test_udf.py \\\n modin/tests/pandas/dataframe/test_window.py \\\n modin/tests/pandas/dataframe/test_pickle.py\n if: matrix.os != 'windows'\n - run: $PYTEST_COMMAND modin/tests/pandas/test_series.py\n if: matrix.execution == 'ray'\n - run: $PYTEST_COMMAND -m \"not exclude_in_sanity\" modin/tests/pandas/test_series.py\n if: matrix.execution != 'ray'\n - run: $PYTEST_COMMAND modin/tests/pandas/dataframe/test_map_metadata.py\n if: matrix.execution == 'ray'\n - run: $PYTEST_COMMAND -m \"not exclude_in_sanity\" modin/tests/pandas/dataframe/test_map_metadata.py\n if: matrix.execution != 'ray'\n - name: \"test rolling, expanding, reshape, general, concat\"\n run: |\n $PYTEST_COMMAND $PARALLEL \\\n modin/tests/pandas/test_rolling.py \\\n modin/tests/pandas/test_expanding.py \\\n modin/tests/pandas/test_reshape.py \\\n modin/tests/pandas/test_general.py \\\n modin/tests/pandas/test_concat.py\n if: matrix.os != 'windows'\n - run: $PYTEST_COMMAND $PARALLEL modin/tests/numpy\n - run: $PYTEST_COMMAND -m \"not exclude_in_sanity\" modin/tests/pandas/test_io.py --verbose\n if: matrix.execution != 'unidist'\n - uses: nick-fields/retry@v3\n # to avoid issues with non-stable `to_csv` tests for unidist on MPI backend.\n # for details see: https://github.com/modin-project/modin/pull/6776\n with:\n timeout_minutes: 15\n max_attempts: 3\n command: conda run --no-capture-output -n modin_on_unidist $PYTEST_COMMAND -m \"not exclude_in_sanity\" modin/tests/pandas/test_io.py --verbose\n if: matrix.execution == 'unidist'\n - run: $PYTEST_COMMAND modin/tests/experimental/test_io_exp.py\n - run: $PYTEST_COMMAND $PARALLEL modin/tests/interchange/dataframe_protocol/test_general.py\n - run: $PYTEST_COMMAND $PARALLEL modin/tests/interchange/dataframe_protocol/pandas/test_protocol.py\n - name: Stop local ray cluster\n run: ray stop\n if: matrix.os == 'windows' && matrix.execution == 'ray'\n - uses: ./.github/actions/upload-coverage\n\n test-experimental:\n needs: [lint-flake8, python-filter]\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n env:\n MODIN_ENGINE: \"python\"\n MODIN_EXPERIMENTAL: \"True\"\n name: test experimental\n services:\n moto:\n image: motoserver/moto:5.0.13\n ports:\n - 5000:5000\n env:\n AWS_ACCESS_KEY_ID: foobar_key\n AWS_SECRET_ACCESS_KEY: foobar_secret\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: environment-dev.yml\n python-version: ${{ needs.python-filter.outputs.python-version }}\n - name: Install HDF5\n run: sudo apt update && sudo apt install -y libhdf5-dev\n - run: python -m pytest -n 2 modin/tests/pandas/dataframe/test_map_metadata.py\n - run: python -m pytest -n 2 modin/tests/pandas/test_series.py\n # Do not add parallelism (`-n` argument) here - it will cause mock S3 service to fail.\n - run: python -m pytest modin/tests/pandas/test_io.py --verbose\n - uses: ./.github/actions/upload-coverage\n\n test-spreadsheet:\n needs: [lint-flake8, python-filter]\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n strategy:\n matrix:\n python-version: [ \"${{ needs.python-filter.outputs.python-version }}\" ]\n engine: [\"ray\", \"dask\"]\n env:\n MODIN_EXPERIMENTAL: \"True\"\n MODIN_ENGINE: ${{matrix.engine}}\n name: test-spreadsheet (engine ${{matrix.engine}}, python ${{matrix.python-version}})\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: environment-dev.yml\n python-version: ${{matrix.python-version}}\n - run: python -m pytest modin/tests/experimental/spreadsheet/test_general.py\n\n test-native-dataframe-interoperability:\n needs: [ lint-flake8]\n if: ${{ github.event_name == 'push' || github.event_name == 'pull_request' }}\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n strategy:\n matrix:\n python-version: [\"3.9\"]\n env:\n # Test interoperability between PandasOnPython dataframes/series and\n # native dataframes/series.\n MODIN_ENGINE: \"Python\"\n name: test-native-dataframe-interoperability python ${{matrix.python-version}})\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: environment-dev.yml\n python-version: ${{matrix.python-version}}\n - run: python -m pytest modin/tests/pandas/native_df_interoperability/ -n 2\n - uses: ./.github/actions/upload-coverage\n\n merge-coverage-artifacts:\n needs: [test-internals, test-api-and-no-engine, test-defaults, test-all-unidist, test-all, test-experimental, test-sanity, test-native-dataframe-interoperability]\n if: always() # we need to run it regardless of some job being skipped, like in PR\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n steps:\n - name: Merge Artifacts\n uses: actions/upload-artifact/merge@v4\n with:\n name: coverage-data\n pattern: coverage-data-*\n include-hidden-files: true\n delete-merged: true\n\n upload-coverage:\n needs: [merge-coverage-artifacts, python-filter]\n if: always() # we need to run it regardless of some job being skipped, like in PR\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/python-only\n with:\n python-version: ${{ needs.python-filter.outputs.python-version }}\n - name: Download coverage data\n uses: actions/download-artifact@v4\n with:\n name: coverage-data\n - run: pip install coverage\n - name: Combine coverage\n run: python -m coverage combine\n - name: Generate coverage report in xml format\n run: python -m coverage xml\n - uses: codecov/codecov-action@v4\n with:\n fail_ci_if_error: ${{ github.event_name == 'push' }} # do not care about uploads in PR\n token: ${{ secrets.CODECOV_TOKEN }} # this token is available at https://app.codecov.io/account/github/modin-project/\n"
},
{
"path": ".github/workflows/codeql/codeql-config.yml",
"content": "name: \"Modin CodeQL config\"\n\npaths:\n - modin/** \n"
},
{
"path": ".github/workflows/codeql.yml",
"content": "name: \"CodeQL\"\n\non:\n push:\n branches: [ \"main\" ]\n pull_request:\n branches: [ \"main\" ]\n\nconcurrency:\n # Cancel other jobs in the same branch. We don't care whether CI passes\n # on old commits.\n group: ${{ github.workflow }}-${{ github.ref }}\n cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}\nenv:\n MODIN_GITHUB_CI: true\njobs:\n analyze:\n name: Analyze\n runs-on: ubuntu-latest\n permissions:\n actions: read\n contents: read\n security-events: write\n\n strategy:\n fail-fast: false\n matrix:\n language: [ python ]\n\n steps:\n - name: Checkout\n uses: actions/checkout@v4\n\n - name: Initialize CodeQL\n uses: github/codeql-action/init@v3\n with:\n languages: ${{ matrix.language }}\n queries: +security-and-quality\n config-file: ./.github/workflows/codeql/codeql-config.yml\n\n - name: Autobuild\n uses: github/codeql-action/autobuild@v3\n\n - name: Perform CodeQL Analysis\n uses: github/codeql-action/analyze@v3\n with:\n category: \"/language:${{ matrix.language }}\"\n"
},
{
"path": ".github/workflows/fuzzydata-test.yml",
"content": "name: fuzzy\non:\n pull_request:\n paths:\n # NOTE: keep these paths in sync with the paths that trigger the CI Github\n # Actions in .github/workflows/ci.yml\n - .github/workflows/**\n - '!.github/workflows/push-to-main.yml'\n - asv_bench/**\n - modin/**\n - requirements/**\n - scripts/**\n - environment-dev.yml\n - requirements-dev.txt\n - setup.cfg\n - setup.py\n - versioneer.py\nconcurrency:\n # Cancel other jobs in the same branch. We don't care whether CI passes\n # on old commits.\n group: ${{ github.workflow }}-${{ github.ref }}\n cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}\nenv:\n MODIN_GITHUB_CI: true\njobs:\n test-fuzzydata:\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n strategy:\n matrix:\n python-version: [\"3.9\"]\n engine: [\"ray\", \"dask\"]\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: environment-dev.yml\n python-version: ${{matrix.python-version}}\n - name: test-fuzzydata (engine ${{matrix.engine}}, python ${{matrix.python-version}})\n run: python -m pytest modin/tests/experimental/test_fuzzydata.py -Wignore::UserWarning --log-file=/tmp/fuzzydata-test-wf-${{matrix.engine}}/run.log --log-file-level=INFO\n env:\n MODIN_ENGINE: ${{matrix.engine}}\n - uses: actions/upload-artifact@v4\n if: success() || failure()\n with:\n name: fuzzydata-test-workflow-${{matrix.engine}}\n path: /tmp/fuzzydata-test-wf-${{matrix.engine}}/* # Must match output dir in test_fuzzydata.py\n if-no-files-found: error\n include-hidden-files: true\n"
},
{
"path": ".github/workflows/publish-to-pypi.yml",
"content": "name: Publish Modin wheel to PyPI\n\non:\n schedule:\n - cron: \"42 0 * * WED\"\n push:\n tags: \n - '*'\n workflow_dispatch:\n\njobs:\n build-n-publish:\n name: Build and publish Modin wheel to PyPI\n environment: release\n runs-on: ubuntu-latest\n permissions:\n id-token: write # IMPORTANT: this permission is mandatory for trusted publishing\n\n steps:\n - uses: actions/checkout@v4\n with:\n fetch-depth: 0\n fetch-tags: true\n - name: Checkout latest git tag\n run: git checkout $(git describe --tags \"$(git rev-list --tags --max-count=1)\")\n if: github.event_name == 'push'\n - name: Set up Python\n uses: actions/setup-python@v5\n with:\n python-version: \"3.9.x\"\n\n - name: Install/update tools\n run: python3 -m pip install --upgrade build wheel\n - name: Build a pure Python wheel\n run: python3 setup.py sdist bdist_wheel\n\n - uses: actions/upload-artifact@v4\n with:\n name: modin-wheel-and-source-tarball \n path: ./dist/\n include-hidden-files: true\n\n - name: Publish Modin wheel to PyPI\n if: github.event_name == 'push'\n uses: pypa/gh-action-pypi-publish@release/v1\n"
},
{
"path": ".github/workflows/push-to-main.yml",
"content": "name: push-to-main\non:\n push:\n branches:\n - main\nconcurrency:\n # Cancel other jobs in the same branch. We don't care whether CI passes\n # on old commits.\n group: ${{ github.workflow }}-${{ github.ref }}\n cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}\nenv:\n MODIN_GITHUB_CI: true\njobs:\n test-ray-master:\n runs-on: ubuntu-latest\n defaults:\n run:\n # `shell: bash -l {0}` - special way to activate modin environment\n shell: bash -l {0}\n services:\n moto:\n image: motoserver/moto:5.0.13\n ports:\n - 5000:5000\n env:\n AWS_ACCESS_KEY_ID: foobar_key\n AWS_SECRET_ACCESS_KEY: foobar_secret\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: environment-dev.yml\n - name: install Ray nightly build\n # Use --force-reinstall to always reinstall ray and its dependencies.\n # botocore isn't compatible with urllib3>=2; see #6094 for details\n run: pip install --force-reinstall \"urllib3<2\" https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp39-cp39-manylinux2014_x86_64.whl\n - name: Conda environment\n run: |\n conda info\n conda list\n - run: sudo apt update && sudo apt install -y libhdf5-dev\n - name: Run parallelizable Modin Tests\n run: >\n python -m pytest -n 2\n modin/tests/pandas/dataframe/test_binary.py\n modin/tests/pandas/dataframe/test_default.py\n modin/tests/pandas/dataframe/test_indexing.py\n modin/tests/pandas/dataframe/test_iter.py\n modin/tests/pandas/dataframe/test_join_sort.py\n modin/tests/pandas/dataframe/test_map_metadata.py\n modin/tests/pandas/dataframe/test_reduce.py\n modin/tests/pandas/dataframe/test_udf.py\n modin/tests/pandas/dataframe/test_window.py\n modin/tests/pandas/test_series.py\n modin/tests/numpy/test_array.py\n modin/tests/numpy/test_array_creation.py\n modin/tests/numpy/test_array_arithmetic.py\n modin/tests/numpy/test_array_axis_functions.py\n modin/tests/numpy/test_array_logic.py\n modin/tests/numpy/test_array_linalg.py\n modin/tests/numpy/test_array_indexing.py\n modin/tests/numpy/test_array_math.py\n modin/tests/numpy/test_array_shaping.py\n modin/tests/pandas/test_rolling.py\n modin/tests/pandas/test_expanding.py\n modin/tests/pandas/test_concat.py\n modin/tests/pandas/test_groupby.py\n modin/tests/pandas/test_reshape.py\n modin/tests/pandas/test_general.py\n - name: Run non-parallelizable Modin Tests\n run: >\n python -m pytest\n modin/tests/pandas/test_io.py\n modin/tests/experimental/test_io_exp.py\n\n test-docs:\n runs-on: ubuntu-latest\n defaults:\n run:\n shell: bash -l {0}\n name: test docs\n steps:\n - uses: actions/checkout@v4\n - uses: ./.github/actions/mamba-env\n with:\n environment-file: environment-dev.yml\n - run: sudo apt update && sudo apt install -y libhdf5-dev\n - name: Docstring URL validity check\n run: python -m pytest modin/tests/test_docstring_urls.py\n"
},
{
"path": ".github/workflows/sql_server/set_up_sql_server.sh",
"content": "# This script sets up a SQL server listening at 0.0.0.0:1234.\n\n# If any step fails, we can't set up a valid SQL server for unit tests.\nset -e\n\n# Pull the 2019 SQL server docker container image by following:\n# https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-docker?view=sql-server-ver15&pivots=cs1-powershell#pullandrun2019\nsudo docker pull mcr.microsoft.com/mssql/server:2019-latest\nsudo docker run -d --name example_sql_server -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=Strong.Pwd-123' -p 1433:1433 mcr.microsoft.com/mssql/server:2019-latest\n\n\n# Wait 10 seconds because if we don't the server typically will not be ready\n# to accept connections by the time we want to make them.\nsleep 10\n"
},
{
"path": ".gitignore",
"content": "# Byte-compiled / optimized / DLL files\r\n__pycache__/\r\n*.py[cod]\r\n*$py.class\r\n\r\n# C extensions\r\n*.so\r\n\r\n# Distribution / packaging\r\n.Python\r\nbuild/\r\ndevelop-eggs/\r\ndist/\r\ndownloads/\r\neggs/\r\n.eggs/\r\nlib/\r\nlib64/\r\nparts/\r\nsdist/\r\nvar/\r\nwheels/\r\n*.egg-info/\r\n.installed.cfg\r\n*.egg\r\nMANIFEST\r\nscripts/gh-users-cache.json\r\n\r\n# PyInstaller\r\n# Usually these files are written by a python script from a template\r\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\r\n*.manifest\r\n*.spec\r\n\r\n# Installer logs\r\npip-log.txt\r\npip-delete-this-directory.txt\r\n\r\n# Unit test / coverage reports\r\nhtmlcov/\r\n.tox/\r\n.coverage\r\n.coverage.*\r\n.cache\r\nnosetests.xml\r\ncoverage.xml\r\n*.cover\r\n.hypothesis/\r\n.pytest_cache/\r\n\r\n# Translations\r\n*.mo\r\n*.pot\r\n\r\n# Django stuff:\r\n*.log\r\nlocal_settings.py\r\ndb.sqlite3\r\n\r\n# Flask stuff:\r\ninstance/\r\n.webassets-cache\r\n\r\n# Scrapy stuff:\r\n.scrapy\r\n\r\n# Sphinx documentation\r\ndocs/_build/\r\ndocs/flow/modin/configs_help.csv\r\n\r\n# PyBuilder\r\ntarget/\r\n\r\n# Jupyter Notebook\r\n.ipynb_checkpoints\r\n\r\n# pyenv\r\n.python-version\r\n\r\n# celery beat schedule file\r\ncelerybeat-schedule\r\n\r\n# SageMath parsed files\r\n*.sage.py\r\n\r\n# Environments\r\n.env\r\n.venv\r\nenv/\r\nvenv/\r\nENV/\r\nenv.bak/\r\nvenv.bak/\r\n\r\n# Spyder project settings\r\n.spyderproject\r\n.spyproject\r\n\r\n# Rope project settings\r\n.ropeproject\r\n\r\n# mkdocs documentation\r\n/site\r\n\r\n# mypy\r\n.mypy_cache/\r\n\r\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm\r\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\r\n\r\n# User-specific stuff\r\n.idea/\r\n.idea/**/workspace.xml\r\n.idea/**/tasks.xml\r\n.idea/**/usage.statistics.xml\r\n.idea/**/dictionaries\r\n.idea/**/shelf\r\n*.DS_Store\r\n\r\n# Sensitive or high-churn files\r\n.idea/**/dataSources/\r\n.idea/**/dataSources.ids\r\n.idea/**/dataSources.local.xml\r\n.idea/**/sqlDataSources.xml\r\n.idea/**/dynamic.xml\r\n.idea/**/uiDesigner.xml\r\n.idea/**/dbnavigator.xml\r\n\r\n# Gradle\r\n.idea/**/gradle.xml\r\n.idea/**/libraries\r\n\r\n# vscode settings\r\n.vscode/\r\n\r\n# CMake\r\ncmake-build-*/\r\n\r\n# Mongo Explorer plugin\r\n.idea/**/mongoSettings.xml\r\n\r\n# File-based project format\r\n*.iws\r\n\r\n# IntelliJ\r\nout/\r\n\r\n# mpeltonen/sbt-idea plugin\r\n.idea_modules/\r\n\r\n# JIRA plugin\r\natlassian-ide-plugin.xml\r\n\r\n# Cursive Clojure plugin\r\n.idea/replstate.xml\r\n\r\n# Crashlytics plugin (for Android Studio and IntelliJ)\r\ncom_crashlytics_export_strings.xml\r\ncrashlytics.properties\r\ncrashlytics-build.properties\r\nfabric.properties\r\n\r\n# Editor-based Rest Client\r\n.idea/httpRequests\r\n\r\n# Cscope and Tags\r\ntags\r\ncscope.files\r\ncscope.out\r\n\r\n# PYTest Benchmarks\r\n.benchmarks/\r\n\r\n# Dask workspace\r\ndask-worker-space/\r\nnode_modules\r\n\r\n# Asv stuff\r\nasv_bench/.asv/\r\nasv_bench/modin/\r\n\r\n# Sublime stuff\r\n*.sublime-workspace\r\n*.sublime-project\r\n"
},
{
"path": ".readthedocs.yaml",
"content": "# .readthedocs.yaml\n# Read the Docs configuration file\n# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details\n\n# Required\nversion: 2\n\n# Set the version of Python and other tools you might need\nbuild:\n os: ubuntu-20.04\n tools:\n python: \"3.9\"\n\n# Build documentation in the docs/ directory with Sphinx\nsphinx:\n configuration: docs/conf.py\n\nformats: all\n\npython:\n install:\n - requirements: docs/requirements-doc.txt\n"
},
{
"path": "CODEOWNERS",
"content": "# These owners will be the default owners for everything in\n# the repo unless a later match takes precedence,\n* @modin-project/modin-core @devin-petersohn @mvashishtha @RehanSD @YarShev @vnlitvinov @anmyachev @dchigarev\n"
},
{
"path": "CODE_OF_CONDUCT.md",
"content": "# Contributor Covenant Code of Conduct\n\n## Our Pledge\n\nIn the interest of fostering an open and welcoming environment, we as\ncontributors and maintainers pledge to making participation in our project and\nour community a harassment-free experience for everyone, regardless of age, body\nsize, disability, ethnicity, sex characteristics, gender identity and expression,\nlevel of experience, education, socio-economic status, nationality, personal\nappearance, race, religion, or sexual identity and orientation.\n\n## Our Standards\n\nExamples of behavior that contributes to creating a positive environment\ninclude:\n\n* Using welcoming and inclusive language\n* Being respectful of differing viewpoints and experiences\n* Gracefully accepting constructive criticism\n* Focusing on what is best for the community\n* Showing empathy towards other community members\n\nExamples of unacceptable behavior by participants include:\n\n* The use of sexualized language or imagery and unwelcome sexual attention or\n advances\n* Trolling, insulting/derogatory comments, and personal or political attacks\n* Public or private harassment\n* Publishing others' private information, such as a physical or electronic\n address, without explicit permission\n* Other conduct which could reasonably be considered inappropriate in a\n professional setting\n\n## Our Responsibilities\n\nProject maintainers are responsible for clarifying the standards of acceptable\nbehavior and are expected to take appropriate and fair corrective action in\nresponse to any instances of unacceptable behavior.\n\nProject maintainers have the right and responsibility to remove, edit, or\nreject comments, commits, code, wiki edits, issues, and other contributions\nthat are not aligned to this Code of Conduct, or to ban temporarily or\npermanently any contributor for other behaviors that they deem inappropriate,\nthreatening, offensive, or harmful.\n\n## Scope\n\nThis Code of Conduct applies both within project spaces and in public spaces\nwhen an individual is representing the project or its community. Examples of\nrepresenting a project or community include using an official project e-mail\naddress, posting via an official social media account, or acting as an appointed\nrepresentative at an online or offline event. Representation of a project may be\nfurther defined and clarified by project maintainers.\n\n## Enforcement\n\nInstances of abusive, harassing, or otherwise unacceptable behavior may be\nreported by contacting the project team at conduct@gr-oss.io. All\ncomplaints will be reviewed and investigated and will result in a response that\nis deemed necessary and appropriate to the circumstances. The project team is\nobligated to maintain confidentiality with regard to the reporter of an incident.\nFurther details of specific enforcement policies may be posted separately.\n\nProject maintainers who do not follow or enforce the Code of Conduct in good\nfaith may face temporary or permanent repercussions as determined by other\nmembers of the project's leadership.\n\n## Attribution\n\nThis Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,\navailable at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html\n\n[homepage]: https://www.contributor-covenant.org\n\nFor answers to common questions about this code of conduct, see\nhttps://www.contributor-covenant.org/faq\n"
},
{
"path": "LICENSE",
"content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n\n# Certain code used and distributed in this package is forked from pandas\n# (https://github.com/pandas-dev/pandas). The pandas LICENSE\n# below applies to those certain forked components in this project:\n\nBSD 3-Clause License\n\nCopyright (c) 2008-2011, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team\nAll rights reserved.\n\nCopyright (c) 2011-2025, Open source contributors.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n\n* Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\n* Neither the name of the copyright holder nor the names of its\n contributors may be used to endorse or promote products derived from\n this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."
},
{
"path": "LICENSE_HEADER",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n"
},
{
"path": "MANIFEST.in",
"content": "include versioneer.py\ninclude modin/_version.py\ninclude modin/tests/pandas/data/*.csv\n"
},
{
"path": "NOTICE",
"content": "Modin\n\nCopyright (c) 2018-2024 Modin Developers.\n"
},
{
"path": "README.md",
"content": "
\n
Scale your pandas workflows by changing one line of code
\n\n### What is Modin?\n\nModin is a drop-in replacement for [pandas](https://github.com/pandas-dev/pandas). While pandas is\nsingle-threaded, Modin lets you instantly speed up your workflows by scaling pandas so it uses all of your\ncores. Modin works especially well on larger datasets, where pandas becomes painfully slow or runs\n[out of memory](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html).\nAlso, Modin comes with the [additional APIs](https://modin.readthedocs.io/en/latest/usage_guide/advanced_usage/index.html#additional-apis)\nto improve user experience.\n\nBy simply replacing the import statement, Modin offers users effortless speed and scale for their pandas workflows:\n\n\n\nIn the GIFs below, Modin (left) and pandas (right) perform *the same pandas operations* on a 2GB dataset. The only difference between the two notebook examples is the import statement. \n\n
\n\n
\n
\n
\n
\n\n\n
\n
\n
\n
\n\n
\n\nThe charts below show the speedup you get by replacing pandas with Modin based on the examples above. The example notebooks can be found [here](examples/jupyter). To learn more about the speedups you could get with Modin and try out some examples on your own, check out our [10-minute quickstart guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html) to try out some examples on your own!\n\n\n\n### Installation\n\n#### From PyPI\n\nModin can be installed with `pip` on Linux, Windows and MacOS:\n\n```bash\npip install \"modin[all]\" # (Recommended) Install Modin with Ray and Dask engines.\n```\n\nIf you want to install Modin with a specific engine, we recommend:\n\n```bash\npip install \"modin[ray]\" # Install Modin dependencies and Ray.\npip install \"modin[dask]\" # Install Modin dependencies and Dask.\npip install \"modin[mpi]\" # Install Modin dependencies and MPI through unidist.\n```\n\nTo get Modin on MPI through unidist (as of unidist 0.5.0) fully working\nit is required to have a working MPI implementation installed beforehand.\nOtherwise, installation of `modin[mpi]` may fail. Refer to\n[Installing with pip](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-pip)\nsection of the unidist documentation for more details about installation.\n\n**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: `ray` instead of `ray[default]`.\nThis means that the dashboard and cluster launcher are no longer installed by default.\nIf you need those, consider installing `ray[default]` along with `modin[ray]`.\n\nModin automatically detects which engine(s) you have installed and uses that for scheduling computation.\n\n#### From conda-forge\n\nInstalling from [conda forge](https://github.com/conda-forge/modin-feedstock) using `modin-all`\nwill install Modin and three engines: [Ray](https://github.com/ray-project/ray), [Dask](https://github.com/dask/dask) and\n[MPI through unidist](https://github.com/modin-project/unidist).\n\n```bash\nconda install -c conda-forge modin-all\n```\n\nEach engine can also be installed individually (and also as a combination of several engines):\n\n```bash\nconda install -c conda-forge modin-ray # Install Modin dependencies and Ray.\nconda install -c conda-forge modin-dask # Install Modin dependencies and Dask.\nconda install -c conda-forge modin-mpi # Install Modin dependencies and MPI through unidist.\n```\n\n**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: `ray-core` instead of `ray-default`.\nThis means that the dashboard and cluster launcher are no longer installed by default.\nIf you need those, consider installing `ray-default` along with `modin-ray`.\n\nRefer to\n[Installing with conda](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-conda)\nsection of the unidist documentation for more details on how to install a specific MPI implementation to run on.\n\nTo speed up conda installation we recommend using libmamba solver. To do this install it in a base environment:\n\n```bash\nconda install -n base conda-libmamba-solver\n```\n\nand then use it during istallation either like:\n\n```bash\nconda install -c conda-forge modin-ray --experimental-solver=libmamba\n```\n\nor starting from conda 22.11 and libmamba solver 22.12 versions:\n\n```bash\nconda install -c conda-forge modin-ray --solver=libmamba\n```\n\n#### Choosing a Compute Engine\n\nIf you want to choose a specific compute engine to run on, you can set the environment\nvariable `MODIN_ENGINE` and Modin will do computation with that engine:\n\n```bash\nexport MODIN_ENGINE=ray # Modin will use Ray\nexport MODIN_ENGINE=dask # Modin will use Dask\nexport MODIN_ENGINE=unidist # Modin will use Unidist\n```\n\nIf you want to choose the Unidist engine, you should set the additional environment \nvariable ``UNIDIST_BACKEND``. Currently, Modin only supports MPI through unidist:\n\n```bash\nexport UNIDIST_BACKEND=mpi # Unidist will use MPI backend\n```\n\nThis can also be done within a notebook/interpreter before you import Modin:\n\n```python\nimport modin.config as modin_cfg\nimport unidist.config as unidist_cfg\n\nmodin_cfg.Engine.put(\"ray\") # Modin will use Ray\nmodin_cfg.Engine.put(\"dask\") # Modin will use Dask\n\nmodin_cfg.Engine.put('unidist') # Modin will use Unidist\nunidist_cfg.Backend.put('mpi') # Unidist will use MPI backend\n```\n\n_Note: You should not change the engine after your first operation with Modin as it will result in undefined behavior._\n\n#### Which engine should I use?\n\nOn Linux, MacOS, and Windows you can install and use either Ray, Dask or MPI through unidist. There is no knowledge required\nto use either of these engines as Modin abstracts away all of the complexity, so feel\nfree to pick either!\n\n### Pandas API Coverage\n\n
\nSome pandas APIs are easier to implement than others, so if something is missing feel\nfree to open an issue!\n\n### More about Modin\n\nFor the complete documentation on Modin, visit our [ReadTheDocs](https://modin.readthedocs.io/en/latest/index.html) page.\n\n#### Scale your pandas workflow by changing a single line of code.\n\n_Note: In local mode (without a cluster), Modin will create and manage a local (Dask or Ray) cluster for the execution._\n\nTo use Modin, you do not need to specify how to distribute the data, or even know how many\ncores your system has. In fact, you can continue using your previous\npandas notebooks while experiencing a considerable speedup from Modin, even on a single\nmachine. Once you've changed your import statement, you're ready to use Modin just like\nyou would with pandas!\n\n#### Faster pandas, even on your laptop\n\n\n\nThe `modin.pandas` DataFrame is an extremely light-weight parallel DataFrame.\nModin transparently distributes the data and computation so that you can continue using the same pandas API\nwhile working with more data faster. Because it is so light-weight,\nModin provides speed-ups of up to 4x on a laptop with 4 physical cores.\n\nIn pandas, you are only able to use one core at a time when you are doing computation of\nany kind. With Modin, you are able to use all of the CPU cores on your machine. Even with a\ntraditionally synchronous task like `read_csv`, we see large speedups by efficiently\ndistributing the work across your entire machine.\n\n```python\nimport modin.pandas as pd\n\ndf = pd.read_csv(\"my_dataset.csv\")\n```\n\n#### Modin can handle the datasets that pandas can't \n\nOften data scientists have to switch between different tools\nfor operating on datasets of different sizes. Processing large dataframes with pandas\nis slow, and pandas does not support working with dataframes that are too large to fit\ninto the available memory. As a result, pandas workflows that work well\nfor prototyping on a few MBs of data do not scale to tens or hundreds of GBs (depending on the size\nof your machine). Modin supports operating on data that does not fit in memory, so that you can comfortably\nwork with hundreds of GBs without worrying about substantial slowdown or memory errors.\nWith [cluster](https://modin.readthedocs.io/en/latest/getting_started/using_modin/using_modin_cluster.html)\nand [out of core](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html)\nsupport, Modin is a DataFrame library with both great single-node performance and high\nscalability in a cluster.\n\n#### Modin Architecture\n\nWe designed [Modin's architecture](https://modin.readthedocs.io/en/latest/development/architecture.html)\nto be modular so we can plug in different components as they develop and improve:\n\n\n\n### Other Resources\n\n#### Getting Started with Modin\n\n- [Documentation](https://modin.readthedocs.io/en/latest/)\n- [10-min Quickstart Guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html)\n- [Examples and Tutorials](https://modin.readthedocs.io/en/latest/getting_started/examples.html)\n- [Videos and Blogposts](https://modin.readthedocs.io/en/latest/getting_started/examples.html#talks-podcasts)\n- [Benchmarking Modin](https://modin.readthedocs.io/en/latest/usage_guide/benchmarking.html)\n\n#### Modin Community\n\n- [Slack](https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA)\n- [Twitter](https://twitter.com/modin_project)\n- [Mailing List](https://groups.google.com/g/modin-dev)\n- [GitHub Issues](https://github.com/modin-project/modin/issues)\n- [StackOverflow](https://stackoverflow.com/questions/tagged/modin)\n\n#### Learn More about Modin\n\n- [Frequently Asked Questions (FAQs)](https://modin.readthedocs.io/en/latest/getting_started/faq.html)\n- [Troubleshooting Guide](https://modin.readthedocs.io/en/latest/getting_started/troubleshooting.html)\n- [Development Guide](https://modin.readthedocs.io/en/latest/development/index.html)\n- Modin is built on many years of research and development at UC Berkeley. Check out these selected papers to learn more about how Modin works:\n - [Flexible Rule-Based Decomposition and Metadata Independence in Modin](https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf) (VLDB 2021)\n - [Dataframe Systems: Theory, Architecture, and Implementation](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-193.pdf) (PhD Dissertation 2021)\n - [Towards Scalable Dataframe Systems](https://arxiv.org/pdf/2001.00888.pdf) (VLDB 2020)\n\n#### Getting Involved\n\n***`modin.pandas` is currently under active development. Requests and contributions are welcome!***\n\nFor more information on how to contribute to Modin, check out the\n[Modin Contribution Guide](https://modin.readthedocs.io/en/latest/development/contributing.html).\n\n### License\n\n[Apache License 2.0](LICENSE)\n"
},
{
"path": "asv_bench/README.md",
"content": "# Modin ASV benchmarks\n\n## Here are some scenarios in which [ASV](https://asv.readthedocs.io/en/stable/index.html) can be used:\n\n* Check the impact of the new patch on the performance of a certain set of operations:\n\n `asv continuous -f 1.05 src/main HEAD -b TimeGroupBy --launch-method=spawn`\n\n* Check for presence of errors inside of benchmarks after changing them or writing new ones:\n\n `asv run --quick --show-stderr --python=same --launch-method=spawn`\n\n* Run entire benchmark suite to get the current times:\n\n `asv run --launch-method=spawn`\n\n* Check the range of commits for performance degradation:\n\n ```\n asv run [start_hash]..[end_hash] --launch-method=spawn\n asv publish\n asv preview\n ```\n\nFor more consistent results, you may need to use the following parameters which\ndescription is in [ASV docs](https://asv.readthedocs.io/en/stable/benchmarks.html?highlight=sample_time#timing-benchmarks):\n\n* `-a sample_time=1`\n* `-a warmup_time=1`\n\n### Notes about using Modin on Ray with Asv:\n\n* `--launch-method=forkserver` is not working;\n* Each set of parameters for each test is launched in its own process, which brings\n a large overhead, since for each process redis server and other necessary processes\n from ray initialization are started and destroyed.\n\n## Adding new benchmark\n\nBasic information on writing benchmarks is present [in ASV documentation](https://asv.readthedocs.io/en/stable/writing_benchmarks.html)\n\nBenchmarks from `benchmarks/benchmarks.py`, `benchmarks/scalability/scalability_benchmarks.py` or `benchmarks/io/csv.py`\ncould be used as a starting point.\n\nRequirements:\n* the benchmark should be able to run both on Modin and on Pandas when the appropriate value\nof the environment variable `MODIN_ASV_USE_IMPL` is selected.\n* the size of the benchmark dataset should depend on the environment variable `MODIN_TEST_DATASET_SIZE`.\n\n## Changing existing benchmark\n\nIt should be remembered that the hash calculated from the benchmark source code is used to display the results.\nWhen changing the benchmark, the old results will no longer be displayed in the dashboard. In general, this is the correct\nbehavior so as not to get a situation when incomparable numbers are displayed in the dashboard.\nBut it should be noted that there could be changes in the source code when it is still correct to compare\nthe \"before\" and \"after\" versions, for example, name of a variable changed, comment added, etc.\nIn this case you must either run a new version of the benchmark for all the commits ever accounted for or manually change\nthe hash in the corresponding result files.\n\n## Pipeline for displaying results in a dashboard\n\nStep 1: checking benchmarks for validity, runs in PRs CI.\n During the test, the benchmarks are run once on small data.\n The implementation can be found in `test-asv-benchmarks` job of [ci.yml](https://github.com/modin-project/modin/blob/main/.github/workflows/ci.yml)\n\nStep 2: running benchmarks with saving the results in [modin-bench@master](https://github.com/modin-project/modin-bench).\n The launch takes place on internal server using specific TeamCity configuration.\n The description of the server can be found in the [\"Benchmark list\"](https://modin.org/modin-bench/#summarylist?sort=0&dir=asc) tab,\n on the left when you hover the mouse over the machine name. \n This step starts as scheduled (now every half hour), subject to the presence of new commits in the Modin `main` branch.\n Command to run benchmarks: `asv run HASHFILE:hashfile.txt --show-stderr --machine xeon-e5 --launch-method=spawn`.\n In the file `hashfile.txt` is the last modin commit hash.\n Writing to a `modin-bench@master` triggers 3 step of the pipeline.\n\nStep 3: converting the results to html representation, which is saved in [modin-bench@gh-pages](https://github.com/modin-project/modin-bench)\n The implementation can be found in `deploy-gh-pages` job of [push.yml](https://github.com/modin-project/modin-bench/blob/master/.github/workflows/push.yml)\n\nBasic actions for step 2:\n* setup environment variable:\n * export MODIN_TEST_DATASET=Big\n * export MODIN_CPUS=44\n* setup git client\n* prepare json file with machine description\n * This file should be placed in the user's home directory.\n * ASV does not always automatically create the file with the description of the machine correctly (e.g. due to being run in a container).\n It is recommended to create a file using [asv machine](https://asv.readthedocs.io/en/stable/commands.html?highlight=machine%20description#asv-machine) command, and manually check the result.\n [Example](https://github.com/modin-project/modin-bench/blob/master/results/xeon-e5/machine.json)\n* copy old result to folder where new result will appear\n (conflict resolution will be performed by ASV itself instead of git)\n* push performance result to modin-bench repository\n"
},
{
"path": "asv_bench/asv.conf.dask.json",
"content": "{\n // The version of the config file format. Do not change, unless\n // you know what you are doing.\n \"version\": 1,\n\n // The name of the project being benchmarked\n \"project\": \"modin\",\n\n // The project's homepage\n \"project_url\": \"https://modin.readthedocs.io/\",\n\n // The URL or local path of the source code repository for the\n // project being benchmarked\n \"repo\": \"..\",\n\n // List of branches to benchmark. If not provided, defaults to \"master\"\n // (for git) or \"default\" (for mercurial).\n \"branches\": [\"main\"],\n\n // Customizable commands for building, installing, and\n // uninstalling the project. See asv.conf.json documentation.\n \"install_command\": [\"in-dir={env_dir} python -mpip install {wheel_file}[dask]\"],\n\n // The tool to use to create environments. May be \"conda\",\n // \"virtualenv\" or other value depending on the plugins in use.\n // If missing or the empty string, the tool will be automatically\n // determined by looking for tools on the PATH environment\n // variable.\n \"environment_type\": \"conda\",\n\n // timeout in seconds for installing any dependencies in environment\n // defaults to 10 min\n \"install_timeout\": 6000,\n\n // the base URL to show a commit for the project.\n \"show_commit_url\": \"https://github.com/modin-project/modin/commit/\",\n\n // The Pythons you'd like to test against. If not provided, defaults\n // to the current version of Python used to run `asv`.\n \"pythons\": [\"3.9\"],\n\n // The list of conda channel names to be searched for benchmark\n // dependency packages in the specified order\n \"conda_channels\": [\"conda-forge\", \"defaults\"],\n\n // The directory (relative to the current directory) to cache the Python\n // environments in. If not provided, defaults to \"env\"\n \"env_dir\": \".asv/env\",\n\n // The directory (relative to the current directory) that raw benchmark\n // results are stored in. If not provided, defaults to \"results\".\n \"results_dir\": \".asv/results\",\n\n // The directory (relative to the current directory) that the html tree\n // should be written to. If not provided, defaults to \"html\".\n \"html_dir\": \".asv/html\",\n}\n"
},
{
"path": "asv_bench/asv.conf.json",
"content": "{\n // The version of the config file format. Do not change, unless\n // you know what you are doing.\n \"version\": 1,\n\n // The name of the project being benchmarked\n \"project\": \"modin\",\n\n // The project's homepage\n \"project_url\": \"https://modin.readthedocs.io/\",\n\n // The URL or local path of the source code repository for the\n // project being benchmarked\n \"repo\": \"..\",\n\n // List of branches to benchmark. If not provided, defaults to \"master\"\n // (for git) or \"default\" (for mercurial).\n \"branches\": [\"main\"],\n\n // Customizable commands for building, installing, and\n // uninstalling the project. See asv.conf.json documentation.\n \"install_command\": [\"in-dir={env_dir} python -mpip install {wheel_file}[ray]\"],\n\n // The tool to use to create environments. May be \"conda\",\n // \"virtualenv\" or other value depending on the plugins in use.\n // If missing or the empty string, the tool will be automatically\n // determined by looking for tools on the PATH environment\n // variable.\n \"environment_type\": \"conda\",\n\n // timeout in seconds for installing any dependencies in environment\n // defaults to 10 min\n \"install_timeout\": 6000,\n\n // the base URL to show a commit for the project.\n \"show_commit_url\": \"https://github.com/modin-project/modin/commit/\",\n\n // The Pythons you'd like to test against. If not provided, defaults\n // to the current version of Python used to run `asv`.\n \"pythons\": [\"3.9\"],\n\n // The list of conda channel names to be searched for benchmark\n // dependency packages in the specified order\n \"conda_channels\": [\"conda-forge\", \"defaults\"],\n\n // The directory (relative to the current directory) to cache the Python\n // environments in. If not provided, defaults to \"env\"\n \"env_dir\": \".asv/env\",\n\n // The directory (relative to the current directory) that raw benchmark\n // results are stored in. If not provided, defaults to \"results\".\n \"results_dir\": \".asv/results\",\n\n // The directory (relative to the current directory) that the html tree\n // should be written to. If not provided, defaults to \"html\".\n \"html_dir\": \".asv/html\",\n}\n"
},
{
"path": "asv_bench/asv.conf.unidist.json",
"content": "{\n // The version of the config file format. Do not change, unless\n // you know what you are doing.\n \"version\": 1,\n\n // The name of the project being benchmarked\n \"project\": \"modin\",\n\n // The project's homepage\n \"project_url\": \"https://modin.readthedocs.io/\",\n\n // The URL or local path of the source code repository for the\n // project being benchmarked\n \"repo\": \"..\",\n\n // List of branches to benchmark. If not provided, defaults to \"master\"\n // (for git) or \"default\" (for mercurial).\n \"branches\": [\"main\"],\n\n // Customizable commands for building, installing, and\n // uninstalling the project. See asv.conf.json documentation.\n \"install_command\": [\"in-dir={env_dir} python -mpip install {wheel_file}[unidist]\"],\n\n // The tool to use to create environments. May be \"conda\",\n // \"virtualenv\" or other value depending on the plugins in use.\n // If missing or the empty string, the tool will be automatically\n // determined by looking for tools on the PATH environment\n // variable.\n \"environment_type\": \"conda\",\n\n // timeout in seconds for installing any dependencies in environment\n // defaults to 10 min\n \"install_timeout\": 6000,\n\n // the base URL to show a commit for the project.\n \"show_commit_url\": \"https://github.com/modin-project/modin/commit/\",\n\n // The Pythons you'd like to test against. If not provided, defaults\n // to the current version of Python used to run `asv`.\n \"pythons\": [\"3.9\"],\n\n // The list of conda channel names to be searched for benchmark\n // dependency packages in the specified order\n \"conda_channels\": [\"conda-forge\", \"defaults\"],\n\n // The directory (relative to the current directory) to cache the Python\n // environments in. If not provided, defaults to \"env\"\n \"env_dir\": \".asv/env\",\n\n // The directory (relative to the current directory) that raw benchmark\n // results are stored in. If not provided, defaults to \"results\".\n \"results_dir\": \".asv/results\",\n\n // The directory (relative to the current directory) that the html tree\n // should be written to. If not provided, defaults to \"html\".\n \"html_dir\": \".asv/html\",\n}\n"
},
{
"path": "asv_bench/benchmarks/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin benchmarks.\"\"\"\n"
},
{
"path": "asv_bench/benchmarks/benchmarks.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"General Modin benchmarks.\"\"\"\n\n# define `MODIN_CPUS` env var to control the number of partitions\n# it should be defined before modin.pandas import (in case of using os.environ)\n\n# define `MODIN_ASV_USE_IMPL` env var to choose library for using in performance\n# measurements\n\nimport math\n\nimport numpy as np\n\nfrom .utils import (\n GROUPBY_NGROUPS,\n IMPL,\n RAND_HIGH,\n RAND_LOW,\n execute,\n gen_nan_data,\n generate_dataframe,\n get_benchmark_shapes,\n random_booleans,\n random_columns,\n random_string,\n translator_groupby_ngroups,\n)\n\n\nclass BaseTimeGroupBy:\n def setup(self, shape, ngroups=5, groupby_ncols=1):\n ngroups = translator_groupby_ngroups(ngroups, shape)\n self.df, self.groupby_columns = generate_dataframe(\n \"int\",\n *shape,\n RAND_LOW,\n RAND_HIGH,\n groupby_ncols,\n count_groups=ngroups,\n )\n\n\nclass TimeGroupByMultiColumn(BaseTimeGroupBy):\n param_names = [\"shape\", \"ngroups\", \"groupby_ncols\"]\n params = [\n get_benchmark_shapes(\"TimeGroupByMultiColumn\"),\n GROUPBY_NGROUPS,\n [6],\n ]\n\n def time_groupby_agg_quan(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).agg(\"quantile\"))\n\n def time_groupby_agg_mean(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).apply(lambda df: df.mean()))\n\n\nclass TimeGroupByDefaultAggregations(BaseTimeGroupBy):\n param_names = [\"shape\", \"ngroups\"]\n params = [\n get_benchmark_shapes(\"TimeGroupByDefaultAggregations\"),\n GROUPBY_NGROUPS,\n ]\n\n def time_groupby_count(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).count())\n\n def time_groupby_size(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).size())\n\n def time_groupby_sum(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).sum())\n\n def time_groupby_mean(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).mean())\n\n\nclass TimeGroupByDictionaryAggregation(BaseTimeGroupBy):\n param_names = [\"shape\", \"ngroups\", \"operation_type\"]\n params = [\n get_benchmark_shapes(\"TimeGroupByDictionaryAggregation\"),\n GROUPBY_NGROUPS,\n [\"reduce\", \"aggregation\"],\n ]\n operations = {\n \"reduce\": [\"sum\", \"count\", \"prod\"],\n \"aggregation\": [\"quantile\", \"std\", \"median\"],\n }\n\n def setup(self, shape, ngroups, operation_type):\n super().setup(shape, ngroups)\n self.cols_to_agg = self.df.columns[1:4]\n operations = self.operations[operation_type]\n self.agg_dict = {\n c: operations[i % len(operations)] for i, c in enumerate(self.cols_to_agg)\n }\n\n def time_groupby_dict_agg(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).agg(self.agg_dict))\n\n\nclass TimeJoin:\n param_names = [\"shapes\", \"how\", \"sort\"]\n params = [\n get_benchmark_shapes(\"TimeJoin\"),\n [\"left\", \"inner\"],\n [False],\n ]\n\n def setup(self, shapes, how, sort):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n\n def time_join(self, shapes, how, sort):\n # join dataframes on index to get the predictable shape\n execute(self.df1.join(self.df2, how=how, lsuffix=\"left_\", sort=sort))\n\n\nclass TimeJoinStringIndex:\n param_names = [\"shapes\", \"sort\"]\n params = [\n get_benchmark_shapes(\"TimeJoinStringIndex\"),\n [True, False],\n ]\n\n def setup(self, shapes, sort):\n assert shapes[0] % 100 == 0, \"implementation restriction\"\n level1 = IMPL.Index([f\"i-{i}\" for i in range(10)], dtype=object).values\n level2 = IMPL.Index(\n [f\"i-{i}\" for i in range(shapes[0] // 100)], dtype=object\n ).values\n codes1 = np.arange(10).repeat(shapes[0] // 100)\n codes2 = np.tile(np.arange(shapes[0] // 100), 10)\n index2 = IMPL.MultiIndex(levels=[level1, level2], codes=[codes1, codes2])\n self.df_multi = IMPL.DataFrame(\n np.random.randn(len(index2), 4), index=index2, columns=[\"A\", \"B\", \"C\", \"D\"]\n )\n\n self.key1 = np.tile(level1.take(codes1), 10)\n self.key2 = np.tile(level2.take(codes2), 10)\n self.df = generate_dataframe(\"int\", *shapes, RAND_LOW, RAND_HIGH)\n # just to keep source shape\n self.df = self.df.drop(columns=self.df.columns[-2:])\n self.df[\"key1\"] = self.key1\n self.df[\"key2\"] = self.key2\n execute(self.df)\n\n self.df_key1 = IMPL.DataFrame(\n np.random.randn(len(level1), 4), index=level1, columns=[\"A\", \"B\", \"C\", \"D\"]\n )\n self.df_key2 = IMPL.DataFrame(\n np.random.randn(len(level2), 4), index=level2, columns=[\"A\", \"B\", \"C\", \"D\"]\n )\n\n def time_join_dataframe_index_multi(self, shapes, sort):\n execute(self.df.join(self.df_multi, on=[\"key1\", \"key2\"], sort=sort))\n\n def time_join_dataframe_index_single_key_bigger(self, shapes, sort):\n execute(self.df.join(self.df_key2, on=\"key2\", sort=sort))\n\n def time_join_dataframe_index_single_key_small(self, shapes, sort):\n execute(self.df.join(self.df_key1, on=\"key1\", sort=sort))\n\n\nclass TimeMergeDefault:\n param_names = [\"shapes\", \"how\", \"sort\"]\n params = [\n get_benchmark_shapes(\"TimeMergeDefault\"),\n [\"left\", \"inner\"],\n [True, False],\n ]\n\n def setup(self, shapes, how, sort):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n\n def time_merge(self, shapes, how, sort):\n execute(IMPL.merge(self.df1, self.df2, how=how, sort=sort))\n\n\nclass TimeMerge:\n param_names = [\"shapes\", \"how\", \"sort\"]\n params = [\n get_benchmark_shapes(\"TimeMerge\"),\n [\"left\", \"inner\"],\n [True, False],\n ]\n\n def setup(self, shapes, how, sort):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n\n def time_merge(self, shapes, how, sort):\n # merge dataframes by index to get the predictable shape\n execute(\n self.df1.merge(\n self.df2, left_index=True, right_index=True, how=how, sort=sort\n )\n )\n\n def time_merge_dataframe_empty_right(self, shapes, how, sort):\n # Getting an empty dataframe using `iloc` should be very fast,\n # so the impact on the time of the merge operation should be negligible.\n execute(IMPL.merge(self.df1, self.df2.iloc[:0], how=how, sort=sort))\n\n def time_merge_dataframe_empty_left(self, shapes, how, sort):\n # Getting an empty dataframe using `iloc` should be very fast,\n # so the impact on the time of the merge operation should be negligible.\n execute(IMPL.merge(self.df1.iloc[:0], self.df2, how=how, sort=sort))\n\n\nclass TimeMergeCategoricals:\n param_names = [\"shapes\", \"data_type\"]\n params = [\n get_benchmark_shapes(\"MergeCategoricals\"),\n [\"object\", \"category\"],\n ]\n\n def setup(self, shapes, data_type):\n assert len(shapes) == 2\n assert shapes[1] == 2\n size = (shapes[0],)\n self.left = IMPL.DataFrame(\n {\n \"X\": np.random.choice(range(0, 10), size=size),\n \"Y\": np.random.choice([\"one\", \"two\", \"three\"], size=size),\n }\n )\n\n self.right = IMPL.DataFrame(\n {\n \"X\": np.random.choice(range(0, 10), size=size),\n \"Z\": np.random.choice([\"jjj\", \"kkk\", \"sss\"], size=size),\n }\n )\n\n if data_type == \"category\":\n self.left = self.left.assign(Y=self.left[\"Y\"].astype(\"category\"))\n execute(self.left)\n self.right = self.right.assign(Z=self.right[\"Z\"].astype(\"category\"))\n execute(self.right)\n\n def time_merge_categoricals(self, shapes, data_type):\n execute(IMPL.merge(self.left, self.right, on=\"X\"))\n\n\nclass TimeConcat:\n param_names = [\"shapes\", \"how\", \"axis\", \"ignore_index\"]\n params = [\n get_benchmark_shapes(\"TimeConcat\"),\n [\"inner\", \"outer\"],\n [0, 1],\n [True, False],\n ]\n\n def setup(self, shapes, how, axis, ignore_index):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n\n def time_concat(self, shapes, how, axis, ignore_index):\n execute(\n IMPL.concat(\n [self.df1, self.df2], axis=axis, join=how, ignore_index=ignore_index\n )\n )\n\n\nclass TimeBinaryOp:\n param_names = [\"shapes\", \"binary_op\", \"axis\"]\n params = [\n get_benchmark_shapes(\"TimeBinaryOp\"),\n [\"mul\"],\n [0, 1],\n ]\n\n def setup(self, shapes, binary_op, axis):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n self.op = getattr(self.df1, binary_op)\n\n def time_binary_op(self, shapes, binary_op, axis):\n execute(self.op(self.df2, axis=axis))\n\n\nclass TimeBinaryOpSeries:\n param_names = [\"shapes\", \"binary_op\"]\n params = [\n get_benchmark_shapes(\"TimeBinaryOpSeries\"),\n [\"mul\"],\n ]\n\n def setup(self, shapes, binary_op):\n df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n self.series1 = df1[df1.columns[0]]\n self.series2 = df2[df2.columns[0]]\n self.op = getattr(self.series1, binary_op)\n execute(self.series1)\n execute(self.series2)\n\n def time_binary_op_series(self, shapes, binary_op):\n execute(self.op(self.series2))\n\n\nclass BaseTimeSetItem:\n param_names = [\"shape\", \"item_length\", \"loc\", \"is_equal_indices\"]\n\n @staticmethod\n def get_loc(df, loc, axis, item_length):\n locs_dict = {\n \"zero\": 0,\n \"middle\": len(df.axes[axis]) // 2,\n \"last\": len(df.axes[axis]) - 1,\n }\n base_loc = locs_dict[loc]\n range_based_loc = np.arange(\n base_loc, min(len(df.axes[axis]), base_loc + item_length)\n )\n return (\n (df.axes[axis][base_loc], base_loc)\n if len(range_based_loc) == 1\n else (df.axes[axis][range_based_loc], range_based_loc)\n )\n\n def setup(self, shape, item_length, loc, is_equal_indices):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH).copy()\n self.loc, self.iloc = self.get_loc(\n self.df, loc, item_length=item_length, axis=1\n )\n\n self.item = self.df[self.loc] + 1\n self.item_raw = self.item.to_numpy()\n if not is_equal_indices:\n self.item.index = reversed(self.item.index)\n\n\nclass TimeSetItem(BaseTimeSetItem):\n params = [\n get_benchmark_shapes(\"TimeSetItem\"),\n [1],\n [\"zero\", \"middle\", \"last\"],\n [True, False],\n ]\n\n def time_setitem_qc(self, *args, **kwargs):\n self.df[self.loc] = self.item\n execute(self.df)\n\n def time_setitem_raw(self, *args, **kwargs):\n self.df[self.loc] = self.item_raw\n execute(self.df)\n\n\nclass TimeInsert(BaseTimeSetItem):\n params = [\n get_benchmark_shapes(\"TimeInsert\"),\n [1],\n [\"zero\", \"middle\", \"last\"],\n [True, False],\n ]\n\n def time_insert_qc(self, *args, **kwargs):\n self.df.insert(loc=self.iloc, column=random_string(), value=self.item)\n execute(self.df)\n\n def time_insert_raw(self, *args, **kwargs):\n self.df.insert(loc=self.iloc, column=random_string(), value=self.item_raw)\n execute(self.df)\n\n\nclass TimeArithmetic:\n param_names = [\"shape\", \"axis\"]\n params = [\n get_benchmark_shapes(\"TimeArithmetic\"),\n [0, 1],\n ]\n\n def setup(self, shape, axis):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n def time_sum(self, shape, axis):\n execute(self.df.sum(axis=axis))\n\n def time_count(self, shape, axis):\n execute(self.df.count(axis=axis))\n\n def time_median(self, shape, axis):\n execute(self.df.median(axis=axis))\n\n def time_nunique(self, shape, axis):\n execute(self.df.nunique(axis=axis))\n\n def time_apply(self, shape, axis):\n execute(self.df.apply(lambda df: df.sum(), axis=axis))\n\n def time_mean(self, shape, axis):\n execute(self.df.mean(axis=axis))\n\n def time_mode(self, shape, axis):\n execute(self.df.mode(axis=axis))\n\n def time_add(self, shape, axis):\n execute(self.df.add(2, axis=axis))\n\n def time_mul(self, shape, axis):\n execute(self.df.mul(2, axis=axis))\n\n def time_mod(self, shape, axis):\n execute(self.df.mod(2, axis=axis))\n\n def time_abs(self, shape, axis):\n execute(self.df.abs())\n\n def time_aggregate(self, shape, axis):\n execute(self.df.aggregate(lambda df: df.sum(), axis=axis))\n\n def time_is_in(self, shape, axis):\n execute(self.df.isin([0, 2]))\n\n def time_transpose(self, shape, axis):\n execute(self.df.transpose())\n\n\nclass TimeSortValues:\n param_names = [\"shape\", \"columns_number\", \"ascending_list\"]\n params = [\n get_benchmark_shapes(\"TimeSortValues\"),\n [1, 2, 10, 100],\n [False, True],\n ]\n\n def setup(self, shape, columns_number, ascending_list):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n self.columns = random_columns(self.df.columns, columns_number)\n self.ascending = (\n random_booleans(columns_number)\n if ascending_list\n else bool(random_booleans(1)[0])\n )\n\n def time_sort_values(self, shape, columns_number, ascending_list):\n execute(self.df.sort_values(self.columns, ascending=self.ascending))\n\n\nclass TimeDrop:\n param_names = [\"shape\", \"axis\", \"drop_ncols\"]\n params = [\n get_benchmark_shapes(\"TimeDrop\"),\n [0, 1],\n [1, 0.8],\n ]\n\n def setup(self, shape, axis, drop_ncols):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n drop_count = (\n int(len(self.df.axes[axis]) * drop_ncols)\n if isinstance(drop_ncols, float)\n else drop_ncols\n )\n self.labels = self.df.axes[axis][:drop_count]\n\n def time_drop(self, shape, axis, drop_ncols):\n execute(self.df.drop(self.labels, axis=axis))\n\n\nclass TimeHead:\n param_names = [\"shape\", \"head_count\"]\n params = [\n get_benchmark_shapes(\"TimeHead\"),\n [5, 0.8],\n ]\n\n def setup(self, shape, head_count):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n self.head_count = (\n int(head_count * len(self.df.index))\n if isinstance(head_count, float)\n else head_count\n )\n\n def time_head(self, shape, head_count):\n execute(self.df.head(self.head_count))\n\n\nclass TimeTail:\n param_names = [\"shape\", \"tail_count\"]\n params = [\n get_benchmark_shapes(\"TimeTail\"),\n [5, 0.8],\n ]\n\n def setup(self, shape, tail_count):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n self.tail_count = (\n int(tail_count * len(self.df.index))\n if isinstance(tail_count, float)\n else tail_count\n )\n\n def time_tail(self, shape, tail_count):\n execute(self.df.tail(self.tail_count))\n\n\nclass TimeExplode:\n param_names = [\"shape\"]\n params = [\n get_benchmark_shapes(\"TimeExplode\"),\n ]\n\n def setup(self, shape):\n self.df = generate_dataframe(\n \"int\", *shape, RAND_LOW, RAND_HIGH, gen_unique_key=True\n )\n\n def time_explode(self, shape):\n execute(self.df.explode(\"col1\"))\n\n\nclass TimeFillnaSeries:\n param_names = [\"value_type\", \"shape\", \"limit\"]\n params = [\n [\"scalar\", \"dict\", \"Series\"],\n get_benchmark_shapes(\"TimeFillnaSeries\"),\n [None, 0.8],\n ]\n\n def setup(self, value_type, shape, limit):\n self.series = gen_nan_data(*shape)\n\n if value_type == \"scalar\":\n self.value = 18.19\n elif value_type == \"dict\":\n self.value = {k: k * 1.23 for k in range(shape[0])}\n elif value_type == \"Series\":\n self.value = IMPL.Series(\n [k * 1.23 for k in range(shape[0])], index=IMPL.RangeIndex(shape[0])\n )\n else:\n assert False\n limit = int(limit * shape[0]) if limit else None\n self.kw = {\"value\": self.value, \"limit\": limit}\n\n def time_fillna(self, value_type, shape, limit):\n execute(self.series.fillna(**self.kw))\n\n def time_fillna_inplace(self, value_type, shape, limit):\n self.series.fillna(inplace=True, **self.kw)\n execute(self.series)\n\n\nclass TimeFillnaDataFrame:\n param_names = [\"value_type\", \"shape\", \"limit\"]\n params = [\n [\"scalar\", \"dict\", \"DataFrame\", \"Series\"],\n get_benchmark_shapes(\"TimeFillnaDataFrame\"),\n [None, 0.8],\n ]\n\n def setup(self, value_type, shape, limit):\n self.df = gen_nan_data(*shape)\n columns = self.df.columns\n\n if value_type == \"scalar\":\n self.value = 18.19\n elif value_type == \"dict\":\n self.value = {k: i * 1.23 for i, k in enumerate(columns)}\n elif value_type == \"Series\":\n self.value = IMPL.Series(\n [i * 1.23 for i in range(len(columns))], index=columns\n )\n elif value_type == \"DataFrame\":\n self.value = IMPL.DataFrame(\n {\n k: [i + j * 1.23 for j in range(shape[0])]\n for i, k in enumerate(columns)\n },\n index=IMPL.RangeIndex(shape[0]),\n columns=columns,\n )\n else:\n assert False\n limit = int(limit * shape[0]) if limit else None\n self.kw = {\"value\": self.value, \"limit\": limit}\n\n def time_fillna(self, value_type, shape, limit):\n execute(self.df.fillna(**self.kw))\n\n def time_fillna_inplace(self, value_type, shape, limit):\n self.df.fillna(inplace=True, **self.kw)\n execute(self.df)\n\n\nclass BaseTimeValueCounts:\n def setup(self, shape, ngroups=5, subset=1):\n ngroups = translator_groupby_ngroups(ngroups, shape)\n self.df, self.subset = generate_dataframe(\n \"int\",\n *shape,\n RAND_LOW,\n RAND_HIGH,\n groupby_ncols=subset,\n count_groups=ngroups,\n )\n\n\nclass TimeValueCountsFrame(BaseTimeValueCounts):\n param_names = [\"shape\", \"ngroups\", \"subset\"]\n params = [\n get_benchmark_shapes(\"TimeValueCountsFrame\"),\n GROUPBY_NGROUPS,\n [2, 10],\n ]\n\n def time_value_counts(self, *args, **kwargs):\n execute(self.df.value_counts(subset=self.subset))\n\n\nclass TimeValueCountsSeries(BaseTimeValueCounts):\n param_names = [\"shape\", \"ngroups\", \"bins\"]\n params = [\n get_benchmark_shapes(\"TimeValueCountsSeries\"),\n GROUPBY_NGROUPS,\n [None, 3],\n ]\n\n def setup(self, shape, ngroups, bins):\n super().setup(ngroups=ngroups, shape=shape)\n self.df = self.df[self.subset[0]]\n\n def time_value_counts(self, shape, ngroups, bins):\n execute(self.df.value_counts(bins=bins))\n\n\nclass TimeIndexing:\n param_names = [\"shape\", \"indexer_type\"]\n params = [\n get_benchmark_shapes(\"TimeIndexing\"),\n [\n \"bool_array\",\n \"bool_series\",\n \"scalar\",\n \"slice\",\n \"continuous_slice\",\n \"numpy_array_take_all_values\",\n \"python_list_take_10_values\",\n \"function\",\n ],\n ]\n\n indexer_getters = {\n \"bool_array\": lambda df: np.array([False, True] * (len(df) // 2)),\n # This boolean-Series is a projection of the source frame, it shouldn't\n # be reimported or triggered to execute:\n \"bool_series\": lambda df: df.iloc[:, 0] > 50,\n \"scalar\": lambda df: len(df) // 2,\n \"slice\": lambda df: slice(0, len(df), 2),\n \"continuous_slice\": lambda df: slice(len(df) // 2),\n \"numpy_array_take_all_values\": lambda df: np.arange(len(df)),\n \"python_list_take_10_values\": lambda df: list(range(min(10, len(df)))),\n \"function\": lambda df: (lambda df: df.index[::-2]),\n }\n\n def setup(self, shape, indexer_type):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n self.indexer = self.indexer_getters[indexer_type](self.df)\n if isinstance(self.indexer, (IMPL.Series, IMPL.DataFrame)):\n # HACK: Triggering `dtypes` meta-data computation in advance,\n # so it won't affect the `loc/iloc` time:\n self.indexer.dtypes\n\n def time_iloc(self, shape, indexer_type):\n # Pandas doesn't implement `df.iloc[series boolean_mask]` and raises an exception on it.\n # Replacing this with the semantically equivalent construction:\n if indexer_type != \"bool_series\":\n execute(self.df.iloc[self.indexer])\n else:\n execute(self.df[self.indexer])\n\n def time_loc(self, shape, indexer_type):\n execute(self.df.loc[self.indexer])\n\n\nclass TimeIndexingColumns:\n param_names = [\"shape\"]\n params = [get_benchmark_shapes(\"TimeIndexing\")]\n\n def setup(self, shape):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n self.numeric_indexer = [0, 1]\n self.labels_indexer = self.df.columns[self.numeric_indexer].tolist()\n\n def time_iloc(self, shape):\n execute(self.df.iloc[:, self.numeric_indexer])\n\n def time_loc(self, shape):\n execute(self.df.loc[:, self.labels_indexer])\n\n def time___getitem__(self, shape):\n execute(self.df[self.labels_indexer])\n\n\nclass TimeMultiIndexing:\n param_names = [\"shape\"]\n params = [get_benchmark_shapes(\"TimeMultiIndexing\")]\n\n def setup(self, shape):\n df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n index = IMPL.MultiIndex.from_product(\n [df.index[: shape[0] // 2], [\"bar\", \"foo\"]]\n )\n columns = IMPL.MultiIndex.from_product(\n [df.columns[: shape[1] // 2], [\"buz\", \"fuz\"]]\n )\n\n df.index = index\n df.columns = columns\n\n self.df = df.sort_index(axis=1)\n\n def time_multiindex_loc(self, shape):\n execute(\n self.df.loc[\n self.df.index[2] : self.df.index[-2],\n self.df.columns[2] : self.df.columns[-2],\n ]\n )\n\n\nclass TimeResetIndex:\n param_names = [\"shape\", \"drop\", \"level\"]\n params = [\n get_benchmark_shapes(\"TimeResetIndex\"),\n [False, True],\n [None, \"level_1\"],\n ]\n\n def setup(self, shape, drop, level):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n if level:\n index = IMPL.MultiIndex.from_product(\n [self.df.index[: shape[0] // 2], [\"bar\", \"foo\"]],\n names=[\"level_1\", \"level_2\"],\n )\n self.df.index = index\n\n def time_reset_index(self, shape, drop, level):\n execute(self.df.reset_index(drop=drop, level=level))\n\n\nclass TimeAstype:\n param_names = [\"shape\", \"dtype\", \"astype_ncolumns\"]\n params = [\n get_benchmark_shapes(\"TimeAstype\"),\n [\"float64\", \"category\"],\n [\"one\", \"all\"],\n ]\n\n def setup(self, shape, dtype, astype_ncolumns):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n if astype_ncolumns == \"all\":\n self.astype_arg = dtype\n elif astype_ncolumns == \"one\":\n self.astype_arg = {\"col1\": dtype}\n else:\n raise ValueError(f\"astype_ncolumns: {astype_ncolumns} isn't supported\")\n\n def time_astype(self, shape, dtype, astype_ncolumns):\n execute(self.df.astype(self.astype_arg))\n\n\nclass TimeDescribe:\n param_names = [\"shape\"]\n params = [\n get_benchmark_shapes(\"TimeDescribe\"),\n ]\n\n def setup(self, shape):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n def time_describe(self, shape):\n execute(self.df.describe())\n\n\nclass TimeProperties:\n param_names = [\"shape\"]\n params = [\n get_benchmark_shapes(\"TimeProperties\"),\n ]\n\n def setup(self, shape):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n def time_shape(self, shape):\n return self.df.shape\n\n def time_columns(self, shape):\n return self.df.columns\n\n def time_index(self, shape):\n return self.df.index\n\n\nclass TimeIndexingNumericSeries:\n param_names = [\"shape\", \"dtype\", \"index_structure\"]\n params = [\n get_benchmark_shapes(\"TimeIndexingNumericSeries\"),\n (np.int64, np.uint64, np.float64),\n (\"unique_monotonic_inc\", \"nonunique_monotonic_inc\"),\n ]\n\n def setup(self, shape, dtype, index_structure):\n N = shape[0]\n indices = {\n \"unique_monotonic_inc\": IMPL.Index(range(N), dtype=dtype),\n \"nonunique_monotonic_inc\": IMPL.Index(\n list(range(N // 100)) + [(N // 100) - 1] + list(range(N // 100, N - 1)),\n dtype=dtype,\n ),\n }\n self.data = IMPL.Series(np.random.rand(N), index=indices[index_structure])\n self.array = np.arange(N // 2)\n self.index_to_query = N // 2\n self.array_list = self.array.tolist()\n execute(self.data)\n\n def time_getitem_scalar(self, shape, index, index_structure):\n # not calling execute as execute function fails for scalar\n self.data[self.index_to_query]\n\n def time_getitem_slice(self, shape, index, index_structure):\n execute(self.data[: self.index_to_query])\n\n def time_getitem_list_like(self, shape, index, index_structure):\n execute(self.data[[self.index_to_query]])\n\n def time_getitem_array(self, shape, index, index_structure):\n execute(self.data[self.array])\n\n def time_getitem_lists(self, shape, index, index_structure):\n execute(self.data[self.array_list])\n\n def time_iloc_array(self, shape, index, index_structure):\n execute(self.data.iloc[self.array])\n\n def time_iloc_list_like(self, shape, index, index_structure):\n execute(self.data.iloc[[self.index_to_query]])\n\n def time_iloc_scalar(self, shape, index, index_structure):\n # not calling execute as execute function fails for scalar\n self.data.iloc[self.index_to_query]\n\n def time_iloc_slice(self, shape, index, index_structure):\n execute(self.data.iloc[: self.index_to_query])\n\n def time_loc_array(self, shape, index, index_structure):\n execute(self.data.loc[self.array])\n\n def time_loc_list_like(self, shape, index, index_structure):\n execute(self.data.loc[[self.index_to_query]])\n\n def time_loc_scalar(self, shape, index, index_structure):\n self.data.loc[self.index_to_query]\n\n def time_loc_slice(self, shape, index, index_structure):\n execute(self.data.loc[: self.index_to_query])\n\n\nclass TimeReindex:\n param_names = [\"shape\"]\n params = [get_benchmark_shapes(\"TimeReindex\")]\n\n def setup(self, shape):\n rows, cols = shape\n rng = IMPL.date_range(start=\"1/1/1970\", periods=rows, freq=\"1min\")\n self.df = IMPL.DataFrame(\n np.random.rand(rows, cols), index=rng, columns=range(cols)\n )\n self.df[\"foo\"] = \"bar\"\n self.rng_subset = IMPL.Index(rng[::2])\n self.df2 = IMPL.DataFrame(\n index=range(rows), data=np.random.rand(rows, cols), columns=range(cols)\n )\n level1 = IMPL.Index(\n [f\"i-{i}\" for i in range(rows // 10)], dtype=object\n ).values.repeat(10)\n level2 = np.tile(\n IMPL.Index([f\"i-{i}\" for i in range(10)], dtype=object).values, rows // 10\n )\n index = IMPL.MultiIndex.from_arrays([level1, level2])\n self.s = IMPL.Series(np.random.randn(rows), index=index)\n self.s_subset = self.s[::2]\n self.s_subset_no_cache = self.s[::2].copy()\n\n mi = IMPL.MultiIndex.from_product([rng[: len(rng) // 10], range(10)])\n self.s2 = IMPL.Series(np.random.randn(len(mi)), index=mi)\n self.s2_subset = self.s2[::2].copy()\n execute(self.df), execute(self.df2)\n execute(self.s), execute(self.s_subset)\n execute(self.s2), execute(self.s2_subset)\n execute(self.s_subset_no_cache)\n\n def time_reindex_dates(self, shape):\n execute(self.df.reindex(self.rng_subset))\n\n def time_reindex_columns(self, shape):\n execute(self.df2.reindex(columns=self.df.columns[1:5]))\n\n def time_reindex_multiindex_with_cache(self, shape):\n # MultiIndex._values gets cached (pandas specific)\n execute(self.s.reindex(self.s_subset.index))\n\n def time_reindex_multiindex_no_cache(self, shape):\n # Copy to avoid MultiIndex._values getting cached (pandas specific)\n execute(self.s.reindex(self.s_subset_no_cache.index.copy()))\n\n def time_reindex_multiindex_no_cache_dates(self, shape):\n # Copy to avoid MultiIndex._values getting cached (pandas specific)\n execute(self.s2_subset.reindex(self.s2.index.copy()))\n\n\nclass TimeReindexMethod:\n params = [\n get_benchmark_shapes(\"TimeReindexMethod\"),\n [\"pad\", \"backfill\"],\n [IMPL.date_range, IMPL.period_range],\n ]\n param_names = [\"shape\", \"method\", \"constructor\"]\n\n def setup(self, shape, method, constructor):\n N = shape[0]\n self.idx = constructor(\"1/1/2000\", periods=N, freq=\"1min\")\n self.ts = IMPL.Series(np.random.randn(N), index=self.idx)[::2]\n execute(self.ts)\n\n def time_reindex_method(self, shape, method, constructor):\n execute(self.ts.reindex(self.idx, method=method))\n\n\nclass TimeFillnaMethodSeries:\n params = [get_benchmark_shapes(\"TimeFillnaMethodSeries\"), [\"pad\", \"backfill\"]]\n param_names = [\"shape\", \"method\"]\n\n def setup(self, shape, method):\n N = shape[0]\n self.idx = IMPL.date_range(\"1/1/2000\", periods=N, freq=\"1min\")\n ts = IMPL.Series(np.random.randn(N), index=self.idx)[::2]\n self.ts_reindexed = ts.reindex(self.idx)\n self.ts_float32 = self.ts_reindexed.astype(\"float32\")\n execute(self.ts_reindexed), execute(self.ts_float32)\n\n def time_reindexed(self, shape, method):\n execute(self.ts_reindexed.fillna(method=method))\n\n def time_float_32(self, shape, method):\n execute(self.ts_float32.fillna(method=method))\n\n\nclass TimeFillnaMethodDataframe:\n params = [get_benchmark_shapes(\"TimeFillnaMethodDataframe\"), [\"pad\", \"backfill\"]]\n param_names = [\"shape\", \"method\"]\n\n def setup(self, shape, method):\n self.idx = IMPL.date_range(\"1/1/2000\", periods=shape[0], freq=\"1min\")\n df_ts = IMPL.DataFrame(np.random.randn(*shape), index=self.idx)[::2]\n self.df_ts_reindexed = df_ts.reindex(self.idx)\n self.df_ts_float32 = self.df_ts_reindexed.astype(\"float32\")\n execute(self.df_ts_reindexed), execute(self.df_ts_float32)\n\n def time_reindexed(self, shape, method):\n execute(self.df_ts_reindexed.fillna(method=method))\n\n def time_float_32(self, shape, method):\n execute(self.df_ts_float32.fillna(method=method))\n\n\nclass TimeLevelAlign:\n params = [get_benchmark_shapes(\"TimeLevelAlign\")]\n param_names = [\"shapes\"]\n\n def setup(self, shapes):\n rows, cols = shapes[0]\n rows_sqrt = round(math.sqrt(rows))\n # the new number of rows may differ from the requested (slightly, so ok)\n rows = rows_sqrt * rows_sqrt\n self.index = IMPL.MultiIndex(\n levels=[np.arange(10), np.arange(rows_sqrt), np.arange(rows_sqrt)],\n codes=[\n np.arange(10).repeat(rows),\n np.tile(np.arange(rows_sqrt).repeat(rows_sqrt), 10),\n np.tile(np.tile(np.arange(rows_sqrt), rows_sqrt), 10),\n ],\n )\n self.df1 = IMPL.DataFrame(\n np.random.randn(len(self.index), cols), index=self.index\n )\n self.df2 = IMPL.DataFrame(np.random.randn(*shapes[1]))\n execute(self.df1), execute(self.df2)\n\n def time_align_level(self, shapes):\n left, right = self.df1.align(self.df2, level=1, copy=False)\n execute(left), execute(right)\n\n def time_reindex_level(self, shapes):\n # `reindex` returns the same result here as `align`.\n # Approximately the same performance is expected.\n execute(self.df2.reindex(self.index, level=1))\n\n\nclass TimeDropDuplicatesDataframe:\n params = [get_benchmark_shapes(\"TimeDropDuplicatesDataframe\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n rows, cols = shape\n N = rows // 10\n K = 10\n data = {}\n # dataframe would have cols-1 keys(strings) and one value(int) column\n for col in range(cols - 1):\n data[\"key\" + str(col + 1)] = IMPL.Index(\n [f\"i-{i}\" for i in range(N)], dtype=object\n ).values.repeat(K)\n data[\"value\"] = np.random.randn(N * K)\n self.df = IMPL.DataFrame(data)\n execute(self.df)\n\n def time_drop_dups(self, shape):\n execute(self.df.drop_duplicates(self.df.columns[:-1]))\n\n def time_drop_dups_inplace(self, shape):\n self.df.drop_duplicates(self.df.columns[:-1], inplace=True)\n execute(self.df)\n\n\nclass TimeDropDuplicatesSeries:\n params = [get_benchmark_shapes(\"TimeDropDuplicatesSeries\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n rows = shape[0]\n self.series = IMPL.Series(\n np.tile(\n IMPL.Index([f\"i-{i}\" for i in range(rows // 10)], dtype=object).values,\n 10,\n )\n )\n execute(self.series)\n\n def time_drop_dups(self, shape):\n execute(self.series.drop_duplicates())\n\n def time_drop_dups_string(self, shape):\n self.series.drop_duplicates(inplace=True)\n execute(self.series)\n\n\nclass TimeDatetimeAccessor:\n params = [get_benchmark_shapes(\"TimeDatetimeAccessor\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.series = IMPL.Series(\n IMPL.timedelta_range(\"1 days\", periods=shape[0], freq=\"h\")\n )\n execute(self.series)\n\n def time_dt_accessor(self, shape):\n execute(self.series.dt)\n\n def time_timedelta_days(self, shape):\n execute(self.series.dt.days)\n\n def time_timedelta_seconds(self, shape):\n execute(self.series.dt.seconds)\n\n\nclass BaseCategories:\n def setup(self, shape):\n rows = shape[0]\n arr = [f\"s{i:04d}\" for i in np.random.randint(0, rows // 10, size=rows)]\n self.ts = IMPL.Series(arr).astype(\"category\")\n execute(self.ts)\n\n\nclass TimeSetCategories(BaseCategories):\n params = [get_benchmark_shapes(\"TimeSetCategories\")]\n param_names = [\"shape\"]\n\n def time_set_categories(self, shape):\n execute(self.ts.cat.set_categories(self.ts.cat.categories[::2]))\n\n\nclass TimeRemoveCategories(BaseCategories):\n params = [get_benchmark_shapes(\"TimeRemoveCategories\")]\n param_names = [\"shape\"]\n\n def time_remove_categories(self, shape):\n execute(self.ts.cat.remove_categories(self.ts.cat.categories[::2]))\n\n\nclass BaseReshape:\n def setup(self, shape):\n rows, cols = shape\n k = 10\n arrays = [\n np.arange(rows // k).repeat(k),\n np.roll(np.tile(np.arange(rows // k), k), 25),\n ]\n index = IMPL.MultiIndex.from_arrays(arrays)\n self.df = IMPL.DataFrame(np.random.randn(rows, cols), index=index)\n execute(self.df)\n\n\nclass TimeStack(BaseReshape):\n params = [get_benchmark_shapes(\"TimeStack\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n super().setup(shape)\n self.udf = self.df.unstack(1)\n execute(self.udf)\n\n def time_stack(self, shape):\n execute(self.udf.stack())\n\n\nclass TimeUnstack(BaseReshape):\n params = [get_benchmark_shapes(\"TimeUnstack\")]\n param_names = [\"shape\"]\n\n def time_unstack(self, shape):\n execute(self.df.unstack(1))\n\n\nclass TimeReplace:\n params = [get_benchmark_shapes(\"TimeReplace\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n rows, cols = shape\n self.to_replace = {i: getattr(IMPL, \"Timestamp\")(i) for i in range(rows)}\n self.df = IMPL.DataFrame(np.random.randint(rows, size=(rows, cols)))\n execute(self.df)\n\n def time_replace(self, shape):\n execute(self.df.replace(self.to_replace))\n\n\nclass TimeGroups:\n params = [get_benchmark_shapes(\"TimeGroups\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.series = IMPL.Series(np.random.randint(0, 100, size=shape[0]))\n execute(self.series)\n\n # returns a pretty dict thus not calling execute\n def time_series_groups(self, shape):\n self.series.groupby(self.series).groups\n\n # returns a dict thus not calling execute\n def time_series_indices(self, shape):\n self.series.groupby(self.series).indices\n\n\nclass TimeRepr:\n params = [get_benchmark_shapes(\"TimeRepr\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.df = IMPL.DataFrame(np.random.randn(*shape))\n execute(self.df)\n\n # returns a string thus not calling execute\n def time_repr(self, shape):\n repr(self.df)\n\n\nclass TimeMaskBool:\n params = [get_benchmark_shapes(\"TimeMaskBool\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.df = IMPL.DataFrame(np.random.randn(*shape))\n self.mask = self.df < 0\n execute(self.df), execute(self.mask)\n\n def time_frame_mask(self, shape):\n execute(self.df.mask(self.mask))\n\n\nclass TimeIsnull:\n params = [get_benchmark_shapes(\"TimeIsnull\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n sample = np.array([np.nan, 1.0])\n data = np.random.choice(sample, (shape[0], shape[1]))\n self.df = IMPL.DataFrame(data)\n execute(self.df)\n\n def time_isnull(self, shape):\n execute(IMPL.isnull(self.df))\n\n\nclass TimeDropna:\n params = ([\"all\", \"any\"], [0, 1], get_benchmark_shapes(\"TimeDropna\"))\n param_names = [\"how\", \"axis\", \"shape\"]\n\n def setup(self, how, axis, shape):\n row, col = shape\n self.df = IMPL.DataFrame(np.random.randn(row, col))\n self.df.iloc[row // 20 : row // 10, col // 3 : col // 2] = np.nan\n self.df[\"foo\"] = \"bar\"\n execute(self.df)\n\n def time_dropna(self, how, axis, shape):\n execute(self.df.dropna(how=how, axis=axis))\n\n\nclass TimeEquals:\n params = [get_benchmark_shapes(\"TimeEquals\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.df = IMPL.DataFrame(np.random.randn(*shape))\n self.df.iloc[-1, -1] = np.nan\n execute(self.df)\n\n # returns a boolean thus not calling execute\n def time_frame_float_equal(self, shape):\n self.df.equals(self.df)\n\n\nfrom .utils import setup # noqa: E402, F401\n"
},
{
"path": "asv_bench/benchmarks/io/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"IO Modin benchmarks.\"\"\"\n"
},
{
"path": "asv_bench/benchmarks/io/csv.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport numpy as np\n\nfrom ..utils import (\n ASV_USE_IMPL,\n IMPL,\n RAND_HIGH,\n RAND_LOW,\n execute,\n generate_dataframe,\n get_benchmark_shapes,\n get_shape_id,\n prepare_io_data,\n)\n\n\nclass BaseReadCsv:\n # test data file should be created only once\n def setup_cache(self, test_filename=\"io_test_file\"):\n test_filenames = prepare_io_data(\n test_filename, self.data_type, get_benchmark_shapes(self.__class__.__name__)\n )\n return test_filenames\n\n def setup(self, test_filenames, shape, *args, **kwargs):\n # ray init\n if ASV_USE_IMPL == \"modin\":\n IMPL.DataFrame([])\n self.shape_id = get_shape_id(shape)\n\n\nclass TimeReadCsvSkiprows(BaseReadCsv):\n shapes = get_benchmark_shapes(\"TimeReadCsvSkiprows\")\n skiprows_mapping = {\n \"lambda_even_rows\": lambda x: x % 2,\n \"range_uniform\": np.arange(1, shapes[0][0] // 10),\n \"range_step2\": np.arange(1, shapes[0][0], 2),\n }\n data_type = \"str_int\"\n\n param_names = [\"shape\", \"skiprows\"]\n params = [\n shapes,\n [None, \"lambda_even_rows\", \"range_uniform\", \"range_step2\"],\n ]\n\n def setup(self, test_filenames, shape, skiprows):\n super().setup(test_filenames, shape, skiprows)\n self.skiprows = self.skiprows_mapping[skiprows] if skiprows else None\n\n def time_skiprows(self, test_filenames, shape, skiprows):\n execute(IMPL.read_csv(test_filenames[self.shape_id], skiprows=self.skiprows))\n\n\nclass TimeReadCsvTrueFalseValues(BaseReadCsv):\n data_type = \"true_false_int\"\n\n param_names = [\"shape\"]\n params = [get_benchmark_shapes(\"TimeReadCsvTrueFalseValues\")]\n\n def time_true_false_values(self, test_filenames, shape):\n execute(\n IMPL.read_csv(\n test_filenames[self.shape_id],\n true_values=[\"Yes\", \"true\"],\n false_values=[\"No\", \"false\"],\n ),\n )\n\n\nclass TimeReadCsvNamesDtype:\n shapes = get_benchmark_shapes(\"TimeReadCsvNamesDtype\")\n _dtypes_params = [\"Int64\", \"Int64_Timestamp\"]\n _timestamp_columns = [\"col1\", \"col2\"]\n\n param_names = [\"shape\", \"names\", \"dtype\"]\n params = [\n shapes,\n [\"array-like\"],\n _dtypes_params,\n ]\n\n def _get_file_id(self, shape, dtype):\n return get_shape_id(shape) + dtype\n\n def _add_timestamp_columns(self, df):\n df = df.copy()\n date_column = IMPL.date_range(\"2000\", periods=df.shape[0], freq=\"ms\")\n for col in self._timestamp_columns:\n df[col] = date_column\n return df\n\n def setup_cache(self, test_filename=\"io_test_file_csv_names_dtype\"):\n # filenames with a metadata of saved dataframes\n cache = {}\n for shape in self.shapes:\n for dtype in self._dtypes_params:\n df = generate_dataframe(\n \"int\", *shape, RAND_LOW, RAND_HIGH, impl=\"pandas\"\n )\n if dtype == \"Int64_Timestamp\":\n df = self._add_timestamp_columns(df)\n\n file_id = self._get_file_id(shape, dtype)\n cache[file_id] = (\n f\"{test_filename}_{file_id}.csv\",\n df.columns.to_list(),\n df.dtypes.to_dict(),\n )\n df.to_csv(cache[file_id][0], index=False)\n return cache\n\n def setup(self, cache, shape, names, dtype):\n # ray init\n if ASV_USE_IMPL == \"modin\":\n IMPL.DataFrame([])\n file_id = self._get_file_id(shape, dtype)\n self.filename, self.names, self.dtype = cache[file_id]\n\n self.parse_dates = None\n if dtype == \"Int64_Timestamp\":\n # cached version of dtype should not change\n self.dtype = self.dtype.copy()\n for col in self._timestamp_columns:\n del self.dtype[col]\n self.parse_dates = self._timestamp_columns\n\n def time_read_csv_names_dtype(self, cache, shape, names, dtype):\n execute(\n IMPL.read_csv(\n self.filename,\n names=self.names,\n header=0,\n dtype=self.dtype,\n parse_dates=self.parse_dates,\n )\n )\n\n\nfrom ..utils import setup # noqa: E402, F401\n"
},
{
"path": "asv_bench/benchmarks/io/parquet.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nfrom ..utils import (\n ASV_USE_IMPL,\n IMPL,\n execute,\n get_benchmark_shapes,\n get_shape_id,\n prepare_io_data_parquet,\n)\n\n\nclass TimeReadParquet:\n shapes = get_benchmark_shapes(\"TimeReadParquet\")\n data_type = \"str_int\"\n\n param_names = [\"shape\"]\n params = [\n shapes,\n ]\n\n # test data file should be created only once\n def setup_cache(self, test_filename=\"io_test_file\"):\n test_filenames = prepare_io_data_parquet(\n test_filename, self.data_type, get_benchmark_shapes(self.__class__.__name__)\n )\n return test_filenames\n\n def setup(self, test_filenames, shape):\n # ray init\n if ASV_USE_IMPL == \"modin\":\n IMPL.DataFrame([])\n self.shape_id = get_shape_id(shape)\n\n def time_read_parquet(self, test_filenames, shape):\n execute(\n IMPL.read_parquet(\n test_filenames[self.shape_id],\n )\n )\n\n\nfrom ..utils import setup # noqa: E402, F401\n"
},
{
"path": "asv_bench/benchmarks/scalability/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Benchmarks measuring how Modin performance scales when MODIN_CPUS are changed.\"\"\"\n"
},
{
"path": "asv_bench/benchmarks/scalability/scalability_benchmarks.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"These benchmarks are supposed to be run only for modin, since they do not make sense for pandas.\"\"\"\n\nimport modin.pandas as pd\n\ntry:\n from modin.pandas.io import from_pandas\nexcept ImportError:\n from modin.pandas.utils import from_pandas\n\ntry:\n from modin.pandas.io import to_numpy, to_pandas\nexcept ImportError:\n try:\n from modin.utils import to_numpy, to_pandas\n except ImportError:\n # This provides compatibility with older versions of the Modin, allowing us to test old commits.\n from modin.pandas.utils import to_pandas\n\nimport pandas\n\nfrom ..utils import (\n RAND_HIGH,\n RAND_LOW,\n execute,\n gen_data,\n generate_dataframe,\n get_benchmark_shapes,\n)\n\n\nclass TimeFromPandas:\n param_names = [\"shape\", \"cpus\"]\n params = [\n get_benchmark_shapes(\"TimeFromPandas\"),\n [4, 16, 32],\n ]\n\n def setup(self, shape, cpus):\n self.data = pandas.DataFrame(gen_data(\"int\", *shape, RAND_LOW, RAND_HIGH))\n from modin.config import NPartitions\n\n NPartitions.get = lambda: cpus\n # trigger ray init\n pd.DataFrame([])\n\n def time_from_pandas(self, shape, cpus):\n execute(from_pandas(self.data))\n\n\nclass TimeToPandas:\n param_names = [\"shape\", \"cpus\"]\n params = [\n get_benchmark_shapes(\"TimeToPandas\"),\n [4, 16, 32],\n ]\n\n def setup(self, shape, cpus):\n from modin.config import NPartitions\n\n NPartitions.get = lambda: cpus\n self.data = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH, impl=\"modin\")\n\n def time_to_pandas(self, shape, cpus):\n # to_pandas is already synchronous\n to_pandas(self.data)\n\n\nclass TimeToNumPy:\n param_names = [\"shape\", \"cpus\"]\n params = [\n get_benchmark_shapes(\"TimeToNumPy\"),\n [4, 16, 32],\n ]\n\n def setup(self, shape, cpus):\n from modin.config import NPartitions\n\n NPartitions.get = lambda: cpus\n self.data = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH, impl=\"modin\")\n\n def time_to_numpy(self, shape, cpus):\n # to_numpy is already synchronous\n to_numpy(self.data)\n\n\nfrom ..utils import setup # noqa: E402, F401\n"
},
{
"path": "asv_bench/benchmarks/utils/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin benchmarks utils.\"\"\"\n\nfrom .common import (\n IMPL,\n execute,\n gen_data,\n gen_nan_data,\n generate_dataframe,\n get_shape_id,\n prepare_io_data,\n prepare_io_data_parquet,\n random_booleans,\n random_columns,\n random_string,\n setup,\n translator_groupby_ngroups,\n)\nfrom .compatibility import ASV_USE_IMPL, ASV_USE_STORAGE_FORMAT\nfrom .data_shapes import GROUPBY_NGROUPS, RAND_HIGH, RAND_LOW, get_benchmark_shapes\n\n__all__ = [\n \"ASV_USE_IMPL\",\n \"ASV_USE_STORAGE_FORMAT\",\n \"RAND_LOW\",\n \"RAND_HIGH\",\n \"GROUPBY_NGROUPS\",\n \"get_benchmark_shapes\",\n \"IMPL\",\n \"execute\",\n \"get_shape_id\",\n \"gen_data\",\n \"gen_nan_data\",\n \"generate_dataframe\",\n \"prepare_io_data\",\n \"prepare_io_data_parquet\",\n \"random_string\",\n \"random_columns\",\n \"random_booleans\",\n \"translator_groupby_ngroups\",\n \"setup\",\n]\n"
},
{
"path": "asv_bench/benchmarks/utils/common.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nThe module contains the functionality that is used when benchmarking Modin commits.\n\nIn the case of using utilities from the main Modin code, there is a chance that when\nbenchmarking old commits, the utilities changed, which in turn can unexpectedly affect\nthe performance results, hence some utility functions are duplicated here.\n\"\"\"\n\nimport logging\nimport uuid\nfrom typing import Optional, Union\n\nimport numpy as np\nimport pandas\n\nimport modin.pandas\n\nfrom .compatibility import ASV_DATASET_SIZE, ASV_USE_ENGINE, ASV_USE_IMPL\nfrom .data_shapes import RAND_HIGH, RAND_LOW\n\nPOSSIBLE_IMPL = {\n \"modin\": modin.pandas,\n \"pandas\": pandas,\n}\nIMPL = POSSIBLE_IMPL[ASV_USE_IMPL]\n\n\ndef translator_groupby_ngroups(groupby_ngroups: Union[str, int], shape: tuple) -> int:\n \"\"\"\n Translate a string representation of the number of groups, into a number.\n\n Parameters\n ----------\n groupby_ngroups : str or int\n Number of groups that will be used in `groupby` operation.\n shape : tuple\n Same as pandas.Dataframe.shape.\n\n Returns\n -------\n int\n \"\"\"\n if ASV_DATASET_SIZE == \"big\":\n if groupby_ngroups == \"huge_amount_groups\":\n return min(shape[0] // 2, 5000)\n return groupby_ngroups\n else:\n return groupby_ngroups\n\n\nclass weakdict(dict): # noqa: GL08\n __slots__ = (\"__weakref__\",)\n\n\ndata_cache = dict()\ndataframes_cache = dict()\n\n\ndef gen_nan_data(nrows: int, ncols: int) -> dict:\n \"\"\"\n Generate nan data with caching.\n\n The generated data are saved in the dictionary and on a subsequent call,\n if the keys match, saved data will be returned. Therefore, we need\n to carefully monitor the changing of saved data and make its copy if needed.\n\n Parameters\n ----------\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n\n Returns\n -------\n modin.pandas.DataFrame or pandas.DataFrame or modin.pandas.Series or pandas.Series\n DataFrame or Series with shape (nrows, ncols) or (nrows,), respectively.\n \"\"\"\n cache_key = (ASV_USE_IMPL, nrows, ncols)\n if cache_key in data_cache:\n return data_cache[cache_key]\n\n logging.info(\"Generating nan data {} rows and {} columns\".format(nrows, ncols))\n\n if ncols > 1:\n columns = [f\"col{x}\" for x in range(ncols)]\n data = IMPL.DataFrame(np.nan, index=IMPL.RangeIndex(nrows), columns=columns)\n elif ncols == 1:\n data = IMPL.Series(np.nan, index=IMPL.RangeIndex(nrows))\n else:\n assert False, \"Number of columns (ncols) should be >= 1\"\n\n data_cache[cache_key] = data\n return data\n\n\ndef gen_int_data(nrows: int, ncols: int, rand_low: int, rand_high: int) -> dict:\n \"\"\"\n Generate int data.\n\n Parameters\n ----------\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n\n Returns\n -------\n dict\n Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.\n \"\"\"\n data = {\n \"col{}\".format(i): np.random.randint(rand_low, rand_high, size=(nrows))\n for i in range(ncols)\n }\n return data\n\n\ndef gen_str_int_data(nrows: int, ncols: int, rand_low: int, rand_high: int) -> dict:\n \"\"\"\n Generate int data and string data.\n\n Parameters\n ----------\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n\n Returns\n -------\n dict\n Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.\n One of the columns with string values.\n \"\"\"\n data = gen_int_data(nrows, ncols, rand_low, rand_high).copy()\n # convert values in arbitary column to string type\n key = list(data.keys())[0]\n data[key] = [f\"str_{x}\" for x in data[key]]\n return data\n\n\ndef gen_true_false_int_data(nrows, ncols, rand_low, rand_high):\n \"\"\"\n Generate int data and string data \"true\" and \"false\" values.\n\n Parameters\n ----------\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n\n Returns\n -------\n dict\n Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.\n One half of the columns with integer values, another half - with \"true\" and\n \"false\" string values.\n \"\"\"\n data = gen_int_data(nrows // 2, ncols // 2, rand_low, rand_high)\n\n data_true_false = {\n \"tf_col{}\".format(i): np.random.choice(\n [\"Yes\", \"true\", \"No\", \"false\"], size=(nrows - nrows // 2)\n )\n for i in range(ncols - ncols // 2)\n }\n data.update(data_true_false)\n return data\n\n\ndef gen_data(\n data_type: str,\n nrows: int,\n ncols: int,\n rand_low: int,\n rand_high: int,\n) -> dict:\n \"\"\"\n Generate data with caching.\n\n The generated data are saved in the dictionary and on a subsequent call,\n if the keys match, saved data will be returned. Therefore, we need\n to carefully monitor the changing of saved data and make its copy if needed.\n\n Parameters\n ----------\n data_type : {\"int\", \"str_int\", \"true_false_int\"}\n Type of data generation.\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n\n Returns\n -------\n dict\n Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.\n\n Notes\n -----\n Returned data type depends on the `data_type` parameter in the next way:\n - `data_type`==\"int\" - all columns will be contain only integer values;\n - `data_type`==\"str_int\" some of the columns will be of string type;\n - `data_type`==\"true_false_int\" half of the columns will be filled with\n string values representing \"true\" and \"false\" values and another half - with\n integers.\n \"\"\"\n type_to_generator = {\n \"int\": gen_int_data,\n \"str_int\": gen_str_int_data,\n \"true_false_int\": gen_true_false_int_data,\n }\n cache_key = (data_type, nrows, ncols, rand_low, rand_high)\n if cache_key in data_cache:\n return data_cache[cache_key]\n\n logging.info(\n \"Generating {} data {} rows and {} columns [{}-{}]\".format(\n data_type, nrows, ncols, rand_low, rand_high\n )\n )\n assert data_type in type_to_generator\n data_generator = type_to_generator[data_type]\n\n data = data_generator(nrows, ncols, rand_low, rand_high)\n data_cache[cache_key] = weakdict(data)\n\n return data\n\n\ndef generate_dataframe(\n data_type: str,\n nrows: int,\n ncols: int,\n rand_low: int,\n rand_high: int,\n groupby_ncols: Optional[int] = None,\n count_groups: Optional[int] = None,\n gen_unique_key: bool = False,\n cache_prefix: str = None,\n impl: str = None,\n) -> Union[modin.pandas.DataFrame, pandas.DataFrame]:\n \"\"\"\n Generate DataFrame with caching.\n\n The generated dataframes are saved in the dictionary and on a subsequent call,\n if the keys match, one of the saved dataframes will be returned. Therefore, we need\n to carefully monitor that operations that change the dataframe work with its copy.\n\n Parameters\n ----------\n data_type : str\n Type of data generation;\n supported types: {\"int\", \"str_int\"}.\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n groupby_ncols : int, default: None\n Number of columns for which `groupby` will be called in the future;\n to get more stable performance results, we need to have the same number of values\n in each group every benchmarking time.\n count_groups : int, default: None\n Count of groups in groupby columns.\n gen_unique_key : bool, default: False\n Generate `col1` column where all elements are unique.\n cache_prefix : str, optional\n Prefix to add to the cache key of the requested frame.\n impl : str, optional\n Implementation used to create the dataframe;\n supported implemetations: {\"modin\", \"pandas\"}.\n\n Returns\n -------\n modin.pandas.DataFrame or pandas.DataFrame [and list]\n\n Notes\n -----\n The list of groupby columns names returns when groupby columns are generated.\n \"\"\"\n assert not (\n (groupby_ncols is None) ^ (count_groups is None)\n ), \"You must either specify both parameters 'groupby_ncols' and 'count_groups' or none of them.\"\n\n if groupby_ncols and count_groups:\n ncols -= groupby_ncols\n\n if impl is None:\n impl = ASV_USE_IMPL\n\n cache_key = (\n impl,\n data_type,\n nrows,\n ncols,\n rand_low,\n rand_high,\n groupby_ncols,\n count_groups,\n gen_unique_key,\n )\n\n if cache_prefix is not None:\n cache_key = (cache_prefix, *cache_key)\n\n if cache_key in dataframes_cache:\n return dataframes_cache[cache_key]\n\n logging.info(\n \"Allocating {} DataFrame {}: {} rows and {} columns [{}-{}]\".format(\n impl, data_type, nrows, ncols, rand_low, rand_high\n )\n )\n data = gen_data(data_type, nrows, ncols, rand_low, rand_high)\n\n if groupby_ncols and count_groups:\n groupby_columns = [f\"groupby_col{x}\" for x in range(groupby_ncols)]\n for groupby_col in groupby_columns:\n data[groupby_col] = np.tile(np.arange(count_groups), nrows // count_groups)\n\n if gen_unique_key:\n data[\"col1\"] = np.arange(nrows)\n\n df = POSSIBLE_IMPL[impl].DataFrame(data)\n\n if groupby_ncols and count_groups:\n dataframes_cache[cache_key] = df, groupby_columns\n return df, groupby_columns\n\n dataframes_cache[cache_key] = df\n return df\n\n\ndef random_string() -> str:\n \"\"\"\n Create a 36-character random string.\n\n Returns\n -------\n str\n \"\"\"\n return str(uuid.uuid4())\n\n\ndef random_columns(df_columns: list, columns_number: int) -> list:\n \"\"\"\n Pick sublist of random columns from a given sequence.\n\n Parameters\n ----------\n df_columns : list\n Columns to choose from.\n columns_number : int\n How many columns to pick.\n\n Returns\n -------\n list\n \"\"\"\n return list(np.random.choice(df_columns, size=columns_number))\n\n\ndef random_booleans(number: int) -> list:\n \"\"\"\n Create random list of booleans with `number` elements.\n\n Parameters\n ----------\n number : int\n Count of booleans in result list.\n\n Returns\n -------\n list\n \"\"\"\n return list(np.random.choice([True, False], size=number))\n\n\ndef execute(df: Union[modin.pandas.DataFrame, pandas.DataFrame]):\n \"\"\"\n Make sure the calculations are finished.\n\n Parameters\n ----------\n df : modin.pandas.DataFrame or pandas.Datarame\n DataFrame to be executed.\n \"\"\"\n if ASV_USE_IMPL == \"modin\":\n partitions = df._query_compiler._modin_frame._partitions.flatten()\n mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls\n if len(partitions) and hasattr(mgr_cls, \"wait_partitions\"):\n mgr_cls.wait_partitions(partitions)\n return\n\n # compatibility with old Modin versions\n all(\n map(\n lambda partition: partition.drain_call_queue() or True,\n partitions,\n )\n )\n if ASV_USE_ENGINE == \"ray\":\n from ray import wait\n\n all(map(lambda partition: wait([partition._data]), partitions))\n elif ASV_USE_ENGINE == \"dask\":\n from dask.distributed import wait\n\n all(map(lambda partition: wait(partition._data), partitions))\n elif ASV_USE_ENGINE == \"python\":\n pass\n\n elif ASV_USE_IMPL == \"pandas\":\n pass\n\n\ndef get_shape_id(shape: tuple) -> str:\n \"\"\"\n Join shape numbers into a string with `_` delimiters.\n\n Parameters\n ----------\n shape : tuple\n Same as pandas.Dataframe.shape.\n\n Returns\n -------\n str\n \"\"\"\n return \"_\".join([str(element) for element in shape])\n\n\ndef prepare_io_data(test_filename: str, data_type: str, shapes: list):\n \"\"\"\n Prepare data for IO tests with caching.\n\n Parameters\n ----------\n test_filename : str\n Unique file identifier that is used to distinguish data\n for different tests.\n data_type : {\"int\", \"str_int\", \"true_false_int\"}\n Type of data generation.\n shapes : list\n Data shapes to prepare.\n\n Returns\n -------\n test_filenames : dict\n Dictionary that maps dataset shape to the file on disk.\n \"\"\"\n test_filenames = {}\n for shape in shapes:\n shape_id = get_shape_id(shape)\n test_filenames[shape_id] = f\"{test_filename}_{shape_id}_{data_type}.csv\"\n df = generate_dataframe(data_type, *shape, RAND_LOW, RAND_HIGH, impl=\"pandas\")\n df.to_csv(test_filenames[shape_id], index=False)\n\n return test_filenames\n\n\ndef prepare_io_data_parquet(test_filename: str, data_type: str, shapes: list):\n \"\"\"\n Prepare data for IO tests with caching.\n\n Parameters\n ----------\n test_filename : str\n Unique file identifier that is used to distinguish data\n for different tests.\n data_type : \"str_int\"\n Type of data generation.\n shapes : list\n Data shapes to prepare.\n\n Returns\n -------\n test_filenames : dict\n Dictionary that maps dataset shape to the file on disk.\n \"\"\"\n test_filenames = {}\n for shape in shapes:\n shape_id = get_shape_id(shape)\n test_filenames[shape_id] = f\"{test_filename}_{shape_id}_{data_type}.parquet\"\n df = generate_dataframe(data_type, *shape, RAND_LOW, RAND_HIGH, impl=\"pandas\")\n df.to_parquet(test_filenames[shape_id], index=False)\n\n return test_filenames\n\n\ndef setup(*args, **kwargs): # noqa: GL08\n # This function just needs to be imported into each benchmark file to\n # set up the random seed before each function. ASV run it automatically.\n # https://asv.readthedocs.io/en/latest/writing_benchmarks.html\n np.random.seed(42)\n"
},
{
"path": "asv_bench/benchmarks/utils/compatibility.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Compatibility layer for parameters used by ASV.\"\"\"\n\nimport os\n\nimport modin.pandas as pd\n\ntry:\n from modin.config import NPartitions\n\n NPARTITIONS = NPartitions.get()\nexcept ImportError:\n NPARTITIONS = pd.DEFAULT_NPARTITIONS\n\ntry:\n from modin.config import AsvImplementation, Engine, StorageFormat, TestDatasetSize\n\n ASV_USE_IMPL = AsvImplementation.get()\n ASV_DATASET_SIZE = TestDatasetSize.get() or \"Small\"\n ASV_USE_ENGINE = Engine.get()\n ASV_USE_STORAGE_FORMAT = StorageFormat.get()\nexcept ImportError:\n # The same benchmarking code can be run for different versions of Modin, so in\n # case of an error importing important variables, we'll just use predefined values\n ASV_USE_IMPL = os.environ.get(\"MODIN_ASV_USE_IMPL\", \"modin\")\n ASV_DATASET_SIZE = os.environ.get(\"MODIN_TEST_DATASET_SIZE\", \"Small\")\n ASV_USE_ENGINE = os.environ.get(\"MODIN_ENGINE\", \"Ray\")\n ASV_USE_STORAGE_FORMAT = os.environ.get(\"MODIN_STORAGE_FORMAT\", \"Pandas\")\n\nASV_USE_IMPL = ASV_USE_IMPL.lower()\nASV_DATASET_SIZE = ASV_DATASET_SIZE.lower()\nASV_USE_ENGINE = ASV_USE_ENGINE.lower()\nASV_USE_STORAGE_FORMAT = ASV_USE_STORAGE_FORMAT.lower()\n\nassert ASV_USE_IMPL in (\"modin\", \"pandas\")\nassert ASV_DATASET_SIZE in (\"big\", \"small\")\nassert ASV_USE_ENGINE in (\"ray\", \"dask\", \"python\", \"unidist\")\nassert ASV_USE_STORAGE_FORMAT in (\"pandas\")\n"
},
{
"path": "asv_bench/benchmarks/utils/data_shapes.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Define data shapes.\"\"\"\n\nimport json\nimport os\n\nfrom .compatibility import ASV_DATASET_SIZE\n\nRAND_LOW = 0\nRAND_HIGH = 100\n\nBINARY_OP_DATA_SIZE = {\n \"big\": [\n [[5000, 5000], [5000, 5000]],\n # the case extremely inefficient\n # [[20, 500_000], [10, 1_000_000]],\n [[500_000, 20], [1_000_000, 10]],\n ],\n \"small\": [[[250, 250], [250, 250]], [[10_000, 20], [25_000, 10]]],\n}\nUNARY_OP_DATA_SIZE = {\n \"big\": [\n [5000, 5000],\n # the case extremely inefficient\n # [10, 1_000_000],\n [1_000_000, 10],\n ],\n \"small\": [[250, 250], [10_000, 10]],\n}\nSERIES_DATA_SIZE = {\n \"big\": [[100_000, 1]],\n \"small\": [[10_000, 1]],\n}\nBINARY_OP_SERIES_DATA_SIZE = {\n \"big\": [\n [[500_000, 1], [1_000_000, 1]],\n [[500_000, 1], [500_000, 1]],\n ],\n \"small\": [[[5_000, 1], [10_000, 1]]],\n}\n\n\nDEFAULT_GROUPBY_NGROUPS = {\n \"big\": [100, \"huge_amount_groups\"],\n \"small\": [5],\n}\nGROUPBY_NGROUPS = DEFAULT_GROUPBY_NGROUPS[ASV_DATASET_SIZE]\n\n_DEFAULT_CONFIG_T = [\n (\n UNARY_OP_DATA_SIZE[ASV_DATASET_SIZE],\n [\n # Pandas storage format benchmarks\n \"TimeGroupByMultiColumn\",\n \"TimeGroupByDefaultAggregations\",\n \"TimeGroupByDictionaryAggregation\",\n \"TimeSetItem\",\n \"TimeInsert\",\n \"TimeArithmetic\",\n \"TimeSortValues\",\n \"TimeDrop\",\n \"TimeHead\",\n \"TimeTail\",\n \"TimeExplode\",\n \"TimeFillna\",\n \"TimeFillnaDataFrame\",\n \"TimeValueCountsFrame\",\n \"TimeValueCountsSeries\",\n \"TimeIndexing\",\n \"TimeMultiIndexing\",\n \"TimeResetIndex\",\n \"TimeAstype\",\n \"TimeDescribe\",\n \"TimeProperties\",\n \"TimeReindex\",\n \"TimeReindexMethod\",\n \"TimeFillnaMethodDataframe\",\n \"TimeDropDuplicatesDataframe\",\n \"TimeStack\",\n \"TimeUnstack\",\n \"TimeRepr\",\n \"TimeMaskBool\",\n \"TimeIsnull\",\n \"TimeDropna\",\n \"TimeEquals\",\n # IO benchmarks\n \"TimeReadCsvSkiprows\",\n \"TimeReadCsvTrueFalseValues\",\n \"TimeReadCsvNamesDtype\",\n \"TimeReadParquet\",\n # Scalability benchmarks\n \"TimeFromPandas\",\n \"TimeToPandas\",\n \"TimeToNumPy\",\n ],\n ),\n (\n BINARY_OP_DATA_SIZE[ASV_DATASET_SIZE],\n [\n # Pandas storage format benchmarks\n \"TimeJoin\",\n \"TimeMerge\",\n \"TimeMergeDefault\",\n \"TimeConcat\",\n \"TimeAppend\",\n \"TimeBinaryOp\",\n \"TimeLevelAlign\",\n ],\n ),\n (\n SERIES_DATA_SIZE[ASV_DATASET_SIZE],\n [\n # Pandas storage format benchmarks\n \"TimeFillnaSeries\",\n \"TimeGroups\",\n \"TimeIndexingNumericSeries\",\n \"TimeFillnaMethodSeries\",\n \"TimeDatetimeAccessor\",\n \"TimeSetCategories\",\n \"TimeRemoveCategories\",\n \"TimeDropDuplicatesSeries\",\n ],\n ),\n (\n BINARY_OP_SERIES_DATA_SIZE[ASV_DATASET_SIZE],\n [\n # Pandas storage format benchmarks\n \"TimeBinaryOpSeries\",\n ],\n ),\n]\n\nDEFAULT_CONFIG = {}\nDEFAULT_CONFIG[\"MergeCategoricals\"] = (\n [[10_000, 2]] if ASV_DATASET_SIZE == \"big\" else [[1_000, 2]]\n)\nDEFAULT_CONFIG[\"TimeJoinStringIndex\"] = (\n [[100_000, 64]] if ASV_DATASET_SIZE == \"big\" else [[1_000, 4]]\n)\nDEFAULT_CONFIG[\"TimeReplace\"] = (\n [[10_000, 2]] if ASV_DATASET_SIZE == \"big\" else [[1_000, 2]]\n)\nfor config in (_DEFAULT_CONFIG_T,):\n for _shape, _names in config:\n DEFAULT_CONFIG.update({_name: _shape for _name in _names})\n\n# Correct shapes in the case when the operation ended with a timeout error\nif ASV_DATASET_SIZE == \"big\":\n DEFAULT_CONFIG[\"TimeMergeDefault\"] = [\n [[1000, 1000], [1000, 1000]],\n [[500_000, 20], [1_000_000, 10]],\n ]\n DEFAULT_CONFIG[\"TimeLevelAlign\"] = [\n [[2500, 2500], [2500, 2500]],\n [[250_000, 20], [500_000, 10]],\n ]\n DEFAULT_CONFIG[\"TimeStack\"] = [\n [1500, 1500],\n [100_000, 10],\n ]\n DEFAULT_CONFIG[\"TimeUnstack\"] = DEFAULT_CONFIG[\"TimeStack\"]\n\nCONFIG_FROM_FILE = None\n\n\ndef get_benchmark_shapes(bench_id: str):\n \"\"\"\n Get custom benchmark shapes from a json file stored in MODIN_ASV_DATASIZE_CONFIG.\n\n If `bench_id` benchmark is not found in the file, then the default value will\n be used.\n\n Parameters\n ----------\n bench_id : str\n Unique benchmark identifier that is used to get shapes.\n\n Returns\n -------\n list\n Benchmark shapes.\n \"\"\"\n global CONFIG_FROM_FILE\n if not CONFIG_FROM_FILE:\n try:\n from modin.config import AsvDataSizeConfig\n\n filename = AsvDataSizeConfig.get()\n except ImportError:\n filename = os.environ.get(\"MODIN_ASV_DATASIZE_CONFIG\", None)\n if filename:\n # should be json\n with open(filename) as _f:\n CONFIG_FROM_FILE = json.load(_f)\n\n if CONFIG_FROM_FILE and bench_id in CONFIG_FROM_FILE:\n return CONFIG_FROM_FILE[bench_id]\n return DEFAULT_CONFIG[bench_id]\n"
},
{
"path": "asv_bench/test/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n"
},
{
"path": "asv_bench/test/test_utils.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nfrom unittest.mock import Mock, mock_open, patch\n\nimport numpy as np\nimport pytest\nfrom benchmarks.utils import data_shapes, execute, get_benchmark_shapes\n\nimport modin.pandas as pd\nfrom modin.config import AsvDataSizeConfig\n\n\n@pytest.mark.parametrize(\n \"asv_config_content, result\",\n [\n (\n '{\"TimeJoin\": [[[10, 10], [15, 15]], [[11, 11], [13, 13]]], \\\n \"TimeGroupBy\": [[11, 11], [13, 13]]}',\n [\n [\n # binary shapes\n [[10, 10], [15, 15]],\n [[11, 11], [13, 13]],\n ],\n [\n # unary shapes\n [11, 11],\n [13, 13],\n ],\n ],\n ),\n ],\n)\n@patch.object(data_shapes, \"CONFIG_FROM_FILE\", new=None)\ndef test_get_benchmark_shapes(asv_config_content, result):\n AsvDataSizeConfig.put(\"mock_filename\")\n with patch(\"builtins.open\", mock_open(read_data=asv_config_content)):\n assert result[0] == get_benchmark_shapes(\"TimeJoin\")\n assert result[1] == get_benchmark_shapes(\"TimeGroupBy\")\n\n\n@pytest.mark.parametrize(\n \"asv_config_content, result\",\n [\n (\n '{\"TimeJoin\": [[[10, 10], [15, 15]]]',\n [[100, 100]],\n ),\n ],\n)\n@patch.object(data_shapes, \"CONFIG_FROM_FILE\", new=None)\ndef test_get_benchmark_shapes_default(asv_config_content, result):\n AsvDataSizeConfig.put(None)\n with patch.object(data_shapes, \"DEFAULT_CONFIG\", new={\"TimeJoin\": result}):\n assert result == get_benchmark_shapes(\"TimeJoin\")\n\n\ndef test_execute():\n df = pd.DataFrame(np.random.rand(100, 64))\n partitions = df._query_compiler._modin_frame._partitions.flatten()\n mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls\n with patch.object(mgr_cls, \"wait_partitions\", new=Mock()):\n execute(df)\n mgr_cls.wait_partitions.assert_called_once()\n assert (mgr_cls.wait_partitions.call_args[0] == partitions).all()\n"
},
{
"path": "ci/teamcity/Dockerfile.teamcity-ci",
"content": "# Create images from this container like this (in modin repo root):\n#\n# git rev-parse HEAD > ci/teamcity/git-rev\n#\n# tar cf ci/teamcity/modin.tar .\n#\n# docker build --build-arg ENVIRONMENT=environment-dev.yml -t modin-project/teamcity-ci:${BUILD_NUMBER} -f ci/teamcity/Dockerfile.teamcity-ci ci/teamcity\n\nFROM rayproject/ray:latest\n\nARG ENVIRONMENT=environment-dev.yml\n\nADD modin.tar /modin\nADD git-rev /modin/git-rev\n\nWORKDIR /modin\nRUN sudo chown -R ray /modin\n\n# Make RUN commands use `bash --login`:\nSHELL [\"/bin/bash\", \"--login\", \"-c\"]\n\n# Initialize conda in bash config files:\nRUN conda init bash\nENV PATH /home/ray/anaconda3/envs/modin/bin:$PATH\n\nRUN conda config --set channel_priority strict\nRUN conda update python -y\nRUN conda env create -f ${ENVIRONMENT}\nRUN conda install curl PyGithub\n\n# Activate the environment, and make sure it's activated:\n# The following line also removed conda initialization from\n# ~/.bashrc so conda starts complaining that it should be\n# initialized for bash. But it is necessary to do it because\n# activation is not always executed when \"docker exec\" is used\n# and then conda initialization overwrites PATH with its base\n# environment where python doesn't have any packages installed.\nRUN echo \"conda activate modin\" > ~/.bashrc\nRUN echo \"Make sure environment is activated\"\nRUN conda list -n modin\n"
},
{
"path": "ci/teamcity/build-docker.py",
"content": "import os\nimport sys\n\n\ndef execute_command(cmd):\n status = os.system(cmd)\n ec = os.WEXITSTATUS(status)\n if ec != 0:\n raise SystemExit('Command \"{}\" failed'.format(cmd))\n\n\nif sys.platform.startswith(\"linux\"):\n execute_command(\"git rev-parse HEAD > git-rev\")\n execute_command(\n \"(cd ../.. && git archive -o ci/teamcity/modin.tar $(cat ci/teamcity/git-rev))\"\n )\n base_image = \"ray-project/deploy\"\n requirements = \"requirements-dev.txt\"\n execute_command(\n \"docker build -f Dockerfile.modin-base --build-arg BASE_IMAGE={} -t modin-project/modin-base .\".format(\n base_image\n )\n )\nelse:\n raise SystemExit(\n \"TeamCity CI in Docker containers is supported only on Linux at the moment.\"\n )\n\nexecute_command(\n \"docker build -f Dockerfile.teamcity-ci --build-arg REQUIREMENTS={} -t modin-project/teamcity-ci .\".format(\n requirements\n )\n)\n\nif sys.platform.startswith(\"linux\"):\n execute_command(\"rm ./modin.tar ./git-rev\")\n"
},
{
"path": "ci/teamcity/comment_on_pr.py",
"content": "\"\"\"\nPost the comment like the following to the PR:\n```\n:robot: TeamCity test results bot :robot:\n\n\n```\n\"\"\"\n\nimport os\nimport sys\n\nfrom github import Github\n\n# Check if this is a pull request or not based on the environment variable\ntry:\n pr_id = int(os.environ[\"GITHUB_PR_NUMBER\"].split(\"/\")[-1])\nexcept Exception:\n sys.exit(0)\n\nengine = os.environ[\"MODIN_ENGINE\"]\n\nheader = \"\"\"
\n TeamCity {} test results bot
\\n\\n\"\"\".format(\n engine.title()\n)\nif engine == \"ray\":\n pytest_outputs = [\"ray_tests.log\"]\nelif engine == \"dask\":\n pytest_outputs = [\"dask_tests.log\"]\nelif engine == \"python\":\n pytest_outputs = [\"python_tests.log\"]\nelse:\n raise Exception(\"Unknown Engine, set `MODIN_ENGINE` environment variable\")\n\nfull_comment = \"\"\n# Do not include coverage info in PR comment\nsplit_by_first = (\n \"----------- coverage: platform linux, python 3.7.5-final-0 -----------\"\n)\nsplit_by_second = \"--------------------------------------------------------------------------------------\"\n\ntests_failed = False\nfor out in pytest_outputs:\n content = open(out, \"r\").read()\n full_comment += \"\".join(\n \"\".join(\n [\n i.split(split_by_first)[0],\n i.split(split_by_first)[-1].split(split_by_second)[-1],\n ]\n )\n for i in content.split(\"+ python3 -m pytest \")\n )\n tests_failed = tests_failed or (\"FAILURES\" in full_comment)\n if len(full_comment) > 65_000:\n full_comment = (\n full_comment[-65_000:] + \"\\n\\nRemaining output truncated\\n\\n\"\n )\n full_comment = \"Tests Logs\\n\\n\\n```\\n\" + full_comment\n full_comment += \"\\n```\\n\\n\\n\"\n\nif not tests_failed:\n header += '
Tests PASSed
\\n\\n'\nelse:\n header += '
Tests FAILed
\\n\\n'\n\nfull_comment = header + full_comment\n\ntoken = os.environ[\"GITHUB_TOKEN\"]\ng = Github(token)\nrepo = g.get_repo(\"modin-project/modin\")\n\npr = repo.get_pull(pr_id)\nif any(\n i.user.login == \"modin-bot\"\n and \"TeamCity {} test results bot\".format(engine).lower() in i.body.lower()\n for i in pr.get_issue_comments()\n):\n pr_comment_list = [\n i\n for i in list(pr.get_issue_comments())\n if i.user.login == \"modin-bot\"\n and \"TeamCity {} test results bot\".format(engine).lower() in i.body.lower()\n ]\n assert len(pr_comment_list) == 1, \"Too many comments from modin-bot already\"\n pr_comment_list[0].edit(full_comment)\nelse:\n pr.create_issue_comment(full_comment)\n"
},
{
"path": "codecov.yml",
"content": "comment: false\ncoverage:\n status:\n project:\n default:\n branches:\n - main\n target: 85%\n patch:\n default:\n target: 30%\n"
},
{
"path": "contributing/contributing.md",
"content": "# Modin dev onboarding\n\n1. [Set up git](https://docs.github.com/en/get-started/quickstart/set-up-git)\n1. [install anaconda](https://www.anaconda.com/products/individual#macos). Once installed,\nyou should reopen your terminal to find \"(base)\" next to your prompt: \n1. [Generate an SSH key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) for GitHub\n1. Fork the [modin repo](https://github.com/modin-project/modin) on GitHub\n1. Clone the forked repo in a local directory of your choice: \n ```\n git clone ${PATH_TO_REPO}\n ```\n where the path can be found here: \n4. Inside the cloned \"modin\" directory, add a remote branch called \"upstream\":\n ```\n git remote add upstream git@github.com:modin-project/modin.git\n ```\n where the upstream link comes from here: \n1. Fetch the upstream branch:\n ```\n git fetch upstream\n ```\n1. Set the default remote branch for your local main branch. \n ```\n git branch --set-upstream-to=upstream/main main\n ```\n1. Install modin from local source code, and install all its dependencies:\n ```\n pip install -e \".[all]\"\n ```\n1. Install ipython:\n ```\n pip install ipython\n ```\n1. If you ever want to install modin at a release version (not the editable version from your machine): \n ```\n pip install modin\n ```\n1. If you want a specific version:\n ```\n pip install modin==0.11\n ```\n1. To upgrade modin to the newest available version:\n ```\n pip install -U modin\n ```\n1. Now go back to local modin.\n ```\n pip install -e .\n ```\n1. Try out modin in ipython:\n ```\n ipython\n import modin\n modin.__version__\n ```\n You should see the Modin version, which consists of the version, the last commit number, and the last commit hash.\n\n1. List Modin versions:\n ```\n git tag\n ```\n\n1. Get a summary of a particular release:\n ```\n git tag -l --format='%(contents)' 0.11.0\n ```\n\n1. Check out the developer requirements in `requirements-dev.txt`. Install them with:\n ```\n pip install -r requirements-dev.txt\n ```\n\n1. Try a unit test:\n ```\n pytest modin/tests/pandas/test_concat.py\n ```\n\n1. [Add a GPG key](https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-new-gpg-key-to-your-github-account ) to your Modin account. Your commits need to be signed with a GPG key. For mac, you can use [Mac GPG](https://gpgtools.org/).\n\n\n1. (Optional) We recommend a few workflow settings:\n\n 1. If you use Visual Studio Code, auto-format with [black](https://black.readthedocs.io/en/stable/) every time you save changes:\n 1. Install [Microsoft's Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python)\n 1. Open your VSCode settings, in `Code -> Preferences -> Settings`.\n 1. Search for \"python formatting provider\" and select \"black\" from the dropdown menu.\n 1. Again in settings, search for \"format on save\" and enable the \"Editor: Format on Save\" option.\n 2. Add a pre-commit hook:\n 1. In your modin repository, copy [this pre-commit file](pre-commit) to `.git/hooks/pre-commit`\n 1. Every time you try to commit, git will try to run flake8 and mypy, and abort the commit if either one fails. This lets you make sure your commits pass these tests before you push to GitHub.\n 1. To bypass the pre-commit hook (e.g. if you don't want to create a pull request, or you already know your code will pass the tests), commit with the flag `--no-verify`."
},
{
"path": "contributing/pre-commit",
"content": "#!/bin/sh\n#\n# Called by \"git commit\" with no arguments. The hook should\n# exit with non-zero status after issuing an appropriate message if\n# it wants to stop the commit.\n#\n\nset -e\n\nprintf \"running black. This script will preempt the commit if black fails.\\n\"\nblack --check --diff modin/ asv_bench/benchmarks scripts/doc_checker.py\nprintf 'black passed!\\n'\n\nprintf \"running isort. This script will preempt the commit if isort fails.\\n\"\nisort . --check-only\nprintf 'isort passed!\\n'\n\nprintf \"running flake8. This script will preempt the commit if flake8 fails.\\n\"\nflake8 modin/ asv_bench/benchmarks scripts/doc_checker.py\nprintf \"flake8 passed!\\n\"\n\nprintf \"running mypy. This script will preempt the commit if mypy fails.\\n\"\nmypy --config-file mypy.ini\nprintf \"mypy passed!\\n\"\nprintf \"pre-commit hook finished!\\n\"\n"
},
{
"path": "docker/Dockerfile",
"content": "FROM continuumio/miniconda3\n\nRUN conda install -c conda-forge psutil setproctitle\nRUN pip install modin"
},
{
"path": "docs/_static/custom.js",
"content": "document.addEventListener(\"DOMContentLoaded\", function () {\n var script = document.createElement(\"script\");\n script.type = \"module\";\n script.id = \"runllm-widget-script\"\n\n script.src = \"https://widget.runllm.com\";\n\n script.setAttribute(\"runllm-keyboard-shortcut\", \"Mod+j\"); // cmd-j or ctrl-j to open the widget.\n script.setAttribute(\"runllm-name\", \"Modin\");\n script.setAttribute(\"runllm-position\", \"BOTTOM_RIGHT\");\n script.setAttribute(\"runllm-assistant-id\", \"164\");\n\n script.async = true;\n document.head.appendChild(script);\n});\n"
},
{
"path": "docs/_templates/layout.html",
"content": "{% extends \"!layout.html\" %}\n {% block footer %} {{ super() }}\n\n \n\n{% endblock %}"
},
{
"path": "docs/conf.py",
"content": "# -*- coding: utf-8 -*-\n#\n# Configuration file for the Sphinx documentation builder.\n#\n# This file does only contain a selection of the most common options. For a\n# full list see the documentation:\n# http://www.sphinx-doc.org/en/stable/config\n\nimport os\n\n# -- Project information -----------------------------------------------------\nimport sys\nimport types\n\nimport ray\n\n\n# stub ray.remote to be a no-op so it doesn't shadow docstrings\ndef noop_decorator(*args, **kwargs):\n if len(args) == 1 and len(kwargs) == 0 and callable(args[0]):\n # This is the case where the decorator is just @ray.remote without parameters.\n return args[0]\n return lambda cls_or_func: cls_or_func\n\n\nray.remote = noop_decorator\n\n# fake modules if they're missing\nfor mod_name in (\n \"xgboost\",\n \"unidist\",\n \"unidist.config\",\n):\n try:\n __import__(mod_name)\n except ImportError:\n sys.modules[mod_name] = types.ModuleType(\n mod_name, f\"fake {mod_name} for building docs\"\n )\nif not hasattr(sys.modules[\"xgboost\"], \"Booster\"):\n sys.modules[\"xgboost\"].Booster = type(\"Booster\", (object,), {})\nif not hasattr(sys.modules[\"unidist\"], \"remote\"):\n sys.modules[\"unidist\"].remote = noop_decorator\nif not hasattr(sys.modules[\"unidist\"], \"core\"):\n sys.modules[\"unidist\"].core = type(\"core\", (object,), {})\nif not hasattr(sys.modules[\"unidist\"].core, \"base\"):\n sys.modules[\"unidist\"].core.base = type(\"base\", (object,), {})\nif not hasattr(sys.modules[\"unidist\"].core.base, \"object_ref\"):\n sys.modules[\"unidist\"].core.base.object_ref = type(\"object_ref\", (object,), {})\nif not hasattr(sys.modules[\"unidist\"].core.base.object_ref, \"ObjectRef\"):\n sys.modules[\"unidist\"].core.base.object_ref.ObjectRef = type(\"ObjectRef\", (object,), {})\n\nsys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), \"..\")))\nimport modin\nfrom modin.config.__main__ import export_config_help\n\nconfigs_file_path = os.path.abspath(\n os.path.join(os.path.dirname(__file__), \"flow/modin/configs_help.csv\")\n)\n# Export configs help to create configs table in the docs/flow/modin/config.rst\nexport_config_help(configs_file_path)\n\nproject = \"Modin\"\ncopyright = \"2018-2024, Modin Developers.\"\nauthor = \"Modin contributors\"\n\n# The short X.Y version\nversion = \"{}\".format(modin.__version__)\n# The full version, including alpha/beta/rc tags\nrelease = version\n\n\n# -- General configuration ---------------------------------------------------\n\n# If your documentation needs a minimal Sphinx version, state it here.\n#\n# needs_sphinx = '1.0'\n\n# Add any Sphinx extension module names here, as strings. They can be\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\n# ones.\nextensions = [\n \"sphinx.ext.autodoc\",\n \"sphinx.ext.napoleon\",\n \"sphinx.ext.intersphinx\",\n \"sphinx.ext.todo\",\n \"sphinx.ext.mathjax\",\n \"sphinx.ext.githubpages\",\n \"sphinx.ext.graphviz\",\n \"sphinxcontrib.plantuml\",\n \"sphinx_issues\",\n]\n\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = [\"_templates\"]\n\n# The suffix(es) of source filenames.\n# You can specify multiple suffix as a list of string:\n#\n# source_suffix = ['.rst', '.md']\nsource_suffix = \".rst\"\n\n# The master toctree document.\nmaster_doc = \"index\"\n\n# The language for content autogenerated by Sphinx. Refer to documentation\n# for a list of supported languages.\n#\n# This is also used if you do content translation via gettext catalogs.\n# Usually you set \"language\" from the command line for these cases.\nlanguage = \"en\"\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\n# This pattern also affects html_static_path and html_extra_path .\nexclude_patterns = [\"_build\", \"Thumbs.db\", \".DS_Store\"]\nhtml_static_path = [\"_static\"]\nhtml_js_files = [\"custom.js\"]\n\n# The name of the Pygments (syntax highlighting) style to use.\npygments_style = \"sphinx\"\n\n\n# -- Options for HTML output -------------------------------------------------\n\n# Maps git branches to Sphinx themes\ndefault_html_theme = \"pydata_sphinx_theme\"\ncurrent_branch = \"nature\"\n\n# The theme to use for HTML and HTML Help pages. See the documentation for\n# a list of builtin themes.\n#\nhtml_theme = \"pydata_sphinx_theme\"\n\nhtml_favicon = \"img/MODIN_ver2.ico\"\n\nhtml_logo = \"img/MODIN_ver2.png\"\n\nhtml_context = {\"default_mode\": \"light\"}\n\n# Theme options are theme-specific and customize the look and feel of a theme\n# further. For a list of options available for each theme, see the\n# documentation.\n#\nhtml_theme_options = {\n \"navbar_end\": [\"navbar-icon-links\"],\n \"sidebarwidth\": 270,\n \"collapse_navigation\": False,\n \"navigation_depth\": 4,\n \"show_toc_level\": 2,\n \"github_url\": \"https://github.com/modin-project/modin\",\n \"icon_links\": [\n {\n \"name\": \"PyPI\",\n \"url\": \"https://pypi.org/project/modin\",\n \"icon\": \"fab fa-python\",\n },\n {\n \"name\": \"conda-forge\",\n \"url\": \"https://anaconda.org/conda-forge/modin\",\n \"icon\": \"fas fa-circle-notch\",\n },\n {\n \"name\": \"Join the Slack\",\n \"url\": \"https://modin.org/slack.html\",\n \"icon\": \"fab fa-slack\",\n },\n {\n \"name\": \"Mailing List\",\n \"url\": \"https://groups.google.com/forum/#!forum/modin-dev\",\n \"icon\": \"fas fa-envelope-square\",\n },\n ],\n \"navigation_with_keys\": True,\n}\n\n# Custom sidebar templates, must be a dictionary that maps document names\n# to template names.\n#\n# The default sidebars (for documents that don't match any pattern) are\n# defined by theme itself. Builtin themes are using these templates by\n# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',\n# 'searchbox.html']``.\n#\n# The default pydata_sphinx_theme sidebar templates are\n# sidebar-nav-bs.html and search-field.html.\nhtml_sidebars = {}\n\nissues_github_path = \"modin-project/modin\"\n"
},
{
"path": "docs/contact.rst",
"content": "Contact\r\n=======\r\n\r\nSlack\r\n-----\r\n\r\nJoin our `Slack`_ community to connect with Modin users and contributors,\r\ndiscuss, and ask questions about all things Modin-related.\r\n\r\nMailing List\r\n------------\r\n\r\nGeneral questions, potential contributors, and ideas can be directed to the\r\n`developer mailing list`_. It is an open Google Group, so feel free to join anytime! If\r\nyou are unsure about where to ask or post something, the mailing list is a good place to\r\nask as well.\r\n\r\nIssues\r\n------\r\n\r\nBug reports and feature requests can be directed to the issues_ page of the Modin\r\nGitHub repo.\r\n\r\n.. _Slack: https://modin.org/slack.html\r\n.. _developer mailing list: https://groups.google.com/forum/#!forum/modin-dev\r\n.. _issues: https://github.com/modin-project/modin/issues\r\n"
},
{
"path": "docs/development/architecture.rst",
"content": "System Architecture\n===================\n\nIn this section, we will lay out the overall system architecture for\nModin, as well as go into detail about the component design, implementation and\nother important details. This document also contains important reference\ninformation for those interested in contributing new functionality, bugfixes\nand enhancements.\n\nHigh-Level Architectural View\n-----------------------------\nThe diagram below outlines the general layered view to the components of Modin\nwith a short description of each major section of the documentation following.\n\n\n.. image:: /img/modin_architecture.png\n :align: center\n\nModin is logically separated into different layers that represent the hierarchy of a\ntypical Database Management System. Abstracting out each component allows us to\nindividually optimize and swap out components without affecting the rest of the system.\nWe can implement, for example, new compute kernels that are optimized for a certain type\nof data and can simply plug it in to the existing infrastructure by implementing a small\ninterface. It can still be distributed by our choice of compute engine with the\nlogic internally.\n\nSystem View\n-----------\nA top-down view of Modin’s architecture is detailed below:\n\n.. image:: /img/10000_meter.png\n :align: center\n\nThe user - Data Scientist interacts with the Modin system by sending interactive or\nbatch commands through API and Modin executes them using various execution\nengines: Ray, Dask and MPI are currently supported.\n\nSubsystem/Container View\n------------------------\nIf we click down to the next level of details we will see that inside Modin the layered\narchitecture is implemented using several interacting components:\n\n.. image:: /img/component_view.png\n :align: center\n\nFor the simplicity the other execution systems - Dask and MPI are omitted and only Ray execution is shown.\n\n* Dataframe subsystem is the backbone of the dataframe holding and query compilation. It is responsible for\n dispatching the ingress/egress to the appropriate module, getting the pandas API and calling the query\n compiler to convert calls to the internal intermediate Dataframe Algebra.\n* Data Ingress/Egress Module is working in conjunction with Dataframe and Partitions subsystem to read data\n split into partitions and send data into the appropriate node for storing.\n* Query Planner is subsystem that translates the pandas API to intermediate Dataframe Algebra representation\n DAG and performs an initial set of optimizations.\n* Query Executor is responsible for getting the Dataframe Algebra DAG, performing further optimizations based\n on a selected storage format and mapping or compiling the Dataframe Algebra DAG to and actual\n execution sequence.\n* Storage formats module is responsible for mapping the abstract operation to an actual executor call, e.g. pandas,\n custom format.\n* Orchestration subsystem is responsible for spawning and controlling the actual execution environment for the\n selected execution. It spawns the actual nodes, fires up the execution environment, e.g. Ray, monitors the state\n of executors and provides telemetry\n\nComponent View\n--------------\n\nUser queries which perform data transformation, data ingress or data egress pass through the Modin components\ndetailed below. The path the query takes is mostly similar across execution systems.\n\nData Transformation\n'''''''''''''''''''\n\n.. image:: /img/generic_data_transform.svg\n :align: center\n\nQuery Compiler\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe :ref:`Query Compiler ` receives queries from the pandas API layer. The API layer is\nresponsible for ensuring a clean input to the Query Compiler. The Query Compiler must\nhave knowledge of the compute kernels and in-memory format of the data in order to\nefficiently compile the query.\n\nThe Query Compiler is responsible for sending the compiled query to the Core Modin Dataframe.\nIn this design, the Query Compiler does not have information about where or when the\nquery will be executed, and gives the control of the partition layout to the Modin\nDataframe.\n\nIn the interest of reducing the pandas API, the Query Compiler layer closely follows the\npandas API, but cuts out a large majority of the repetition.\n\n.. _auto-switch architecture:\n\nAutomatic Engine Switching and Casting\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nQueryCompilers which are derived from QueryCompilerCaster can participate in automatic casting when\ndifferent query compilers, representing different underlying engines, are used together in a\nfunction. A relative \"cost\" of casting is used to determine which query compiler everything should\nbe moved to. Each query compiler must implement the functions, `move_to_cost`, `move_to_me_cost`, \n`max_cost` and `stay_cost` to provide information and query costs associated with different decision\npoints in cost opimization. With the exception of `max_cost` these methods need to return a \nQCCoercionCost in the range of 0-1000.\n\nThese functions have precise meanings:\n\n* `move_to_cost` is the transmission cost of moving the data, including known serialization costs\n from the perspective of that particular compiler. Colloquially, the question being asked of the\n query compiler is, \"What is the normalized cost of moving my data to the other engine?\"\n* `move_to_me_cost` is the execution cost for the data and operation on the proposed *destination*\n query compiler. Since this method is called before the data has been migrated this is a class\n method and the destination query_compiler may have very limited information on the possible cost\n after migration. Factors that may be considered here include available memory, cpu, and the\n unique characteristics of the engine. The question being asked is, \"If this data were moved to\n me, what would be the normalized execution cost to perform that operation?\"\n* `stay_cost` is the execution cost on the current query compilier ( where the data is ). The question\n asked of the query compiler is, \"If I were to keep this data on my engine, what would be the normalized\n execution cost?\"\n* `max_cost` is the maximum cost allowed by this query compiler across all data movements. This method\n sets a normalized upper bound for situations where multiple data frames from different engines all\n need to move to the same engine. The value returned by this method can exceed \n QCCoercionCost.COST_IMPOSSIBLE\n\nThere are generally two places where automatic casting is considered: When two or more DataFrames on\ndifferent engines are participating in an operation ( such as pd.concat ) or at registered functions\nfor particular engines through the `register_function_for_pre_op_switch` and \n`register_function_for_post_op_switch` methods.\n\nCore Modin Dataframe\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nAt this layer, operations can be performed lazily. Currently, Modin executes most\noperations eagerly in an attempt to behave as pandas does. Some operations, e.g.\n``transpose`` are expensive and create full copies of the data in-memory. In these\ncases, we can wait until another operation triggers computation. In the future, we plan\nto add additional query planning and laziness to Modin to ensure that queries are\nperformed efficiently.\n\nThe structure of the Core Modin Dataframe is extensible, such that any operation that could\nbe better optimized for a given execution can be overridden and optimized in that way.\n\nThis layer has a significantly reduced API from the QueryCompiler and the user-facing\nAPI. Each of these APIs represents a single way of performing a given operation or\nbehavior.\n\nCore Modin Dataframe API\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nMore documentation can be found internally in the code_. This API is not complete, but\nrepresents an overwhelming majority of operations and behaviors.\n\nThis API can be implemented by other distributed/parallel DataFrame libraries and\nplugged in to Modin as well. Create an issue_ or discuss\non our `Slack `_ for more information!\n\nThe :doc:`Core Modin Dataframe ` is responsible for the data layout and shuffling, partitioning,\nand serializing the tasks that get sent to each partition. Other implementations of the\nModin Dataframe interface will have to handle these as well.\n\nPartition Manager\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe Partition Manager can change the size and shape of the partitions based on the type\nof operation. For example, certain operations are complex and require access to an\nentire column or row. The Partition Manager can convert the block partitions to row\npartitions or column partitions. This gives Modin the flexibility to perform operations\nthat are difficult in row-only or column-only partitioning schemas.\n\nAnother important component of the Partition Manager is the serialization and shipment\nof compiled queries to the Partitions. It maintains metadata for the length and width of\neach partition, so when operations only need to operate on or extract a subset of the\ndata, it can ship those queries directly to the correct partition. This is particularly\nimportant for some operations in pandas which can accept different arguments and\noperations for different columns, e.g. ``fillna`` with a dictionary.\n\nThis abstraction separates the actual data movement and function application from the\nDataframe layer to keep the Core Dataframe API small and separately optimize the data\nmovement and metadata management.\n\nPartitions\n\"\"\"\"\"\"\"\"\"\"\n\nPartitions are responsible for managing a subset of the Dataframe. As mentioned\nbelow, the Dataframe is partitioned both row and column-wise. This gives Modin\nscalability in both directions and flexibility in data layout. There are a number of\noptimizations in Modin that are implemented in the partitions. Partitions are specific\nto the execution framework and in-memory format of the data, allowing Modin to\nexploit potential optimizations across both. These optimizations are explained\nfurther on the pages specific to the execution framework.\n\nExecution Engine\n''''''''''''''''\n\nThis layer performs computation on partitions of the data. The\nModin Dataframe is designed to work with `task parallel`_ frameworks, but integration with\ndata parallel frameworks should be possible with some effort.\n\nStorage Format\n''''''''''''''\n\nThe :doc:`storage format ` describes the in-memory partition type.\nThe base storage format in Modin is pandas. In the default case, the Modin Dataframe operates on partitions that contain ``pandas.DataFrame`` objects.\n\nData Ingress\n''''''''''''\n\n.. note::\n Data ingress operations (e.g. ``read_csv``) in Modin load data from the source into\n partitions and vice versa for data egress (e.g. ``to_csv``) operation.\n Improved performance is achieved by reading/writing in partitions in parallel.\n\nData ingress starts with a function in the pandas API layer (e.g. ``read_csv``). Then the user's\nquery is passed to the :doc:`Factory Dispatcher `,\nwhich defines a factory specific for the execution. The factory for execution contains an IO class\n(e.g. ``PandasOnRayIO``) whose responsibility is to perform a parallel read/write from/to a file.\nThis IO class contains class methods with interfaces and names that are similar to pandas IO functions\n(e.g. ``PandasOnRayIO.read_csv``). The IO class declares the Modin Dataframe and Query Compiler\nclasses specific for the execution engine and storage format to ensure the correct object is constructed.\nIt also declares IO methods that are mix-ins containing a combination of the engine-specific class for\ndeploying remote tasks, the class for parsing the given file format and the class handling the chunking\nof the format-specific file on the head node (see dispatcher classes implementation\n:doc:`details `). The output from the IO class data ingress function is\na :doc:`Modin Dataframe `.\n\n.. image:: /img/generic_data_ingress.svg\n :align: center\n\nData Egress\n'''''''''''\n\nData egress operations (e.g. ``to_csv``) are similar to data ingress operations up to\nexecution-specific IO class functions construction. Data egress functions of the IO class\nare defined slightly different from data ingress functions and created only\nspecifically for the engine since partitions already have information about its storage\nformat. Using the IO class, data is exported from partitions to the target file.\n\n.. image:: /img/generic_data_egress.svg\n :align: center\n\nSupported Execution Engines and Storage Formats\n'''''''''''''''''''''''''''''''''''''''''''''''\n\nThis is a list of execution engines and in-memory formats supported in Modin. If you\nwould like to contribute a new execution engine or in-memory format, please see the\ndocumentation page on :doc:`contributing `.\n\n- :doc:`pandas on Ray `\n - Uses the Ray_ execution framework.\n - The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.\n - For more information on the execution path, see the :doc:`pandas on Ray ` page.\n- :doc:`pandas on Dask `\n - Uses the `Dask Futures`_ execution framework.\n - The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.\n - For more information on the execution path, see the :doc:`pandas on Dask ` page.\n- :doc:`pandas on MPI `\n - Uses MPI_ through the Unidist_ execution framework.\n - The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.\n - For more information on the execution path, see the :doc:`pandas on Unidist ` page.\n- :doc:`pandas on Python `\n - Uses native python execution - mainly used for debugging.\n - The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.\n - For more information on the execution path, see the :doc:`pandas on Python ` page.\n- pandas on Snowflake\n - Uses the Snowpark Python library to transpile pandas API calls to SQL queries.\n - The storage format is the custom-defined `Snowflake` format; data remains within Snowflake warehouses until retrieved by pandas API calls.\n - For more information on pandas on Snowflake, refer to Snowflake's `documentation `_ (external link).\n\n.. _directory-tree:\n\nDataFrame Partitioning\n----------------------\n\nThe Modin DataFrame architecture follows in the footsteps of modern architectures for\ndatabase and high performance matrix systems. We chose a partitioning schema that\npartitions along both columns and rows because it gives Modin flexibility and\nscalability in both the number of columns and the number of rows. The\nfollowing figure illustrates this concept.\n\n.. image:: /img/block_partitions_diagram.png\n :align: center\n\nCurrently, the main in-memory format of each partition is a\n`pandas DataFrame`_ (:doc:`pandas storage format `).\n\nIndex\n-----\n\nWe currently use the ``pandas.Index`` object for indexing both columns and rows. In the\nfuture, we will implement a distributed, pandas-compatible Index object in order to remove\nthis scaling limitation from the system. Most workloads will not be affected by this scalability limit\nsince it only appears when operating on more than 10's of billions of columns or rows.\n**Important note**: If you are using the\ndefault index (``pandas.RangeIndex``) there is a fixed memory overhead (~200 bytes) and\nthere will be no scalability issues with the index.\n\nAPI\n---\n\nThe API is the outer-most layer that faces users. The following classes contain Modin's implementation of the pandas API:\n\n.. toctree::\n /flow/modin/pandas/base\n /flow/modin/pandas/dataframe\n /flow/modin/pandas/series\n\nModule/Class View\n-----------------\n\nModin's modules layout is shown below. Click on the links to deep dive into Modin's internal implementation\ndetails. The documentation covers most modules, with more docs being added everyday!\n\n.. parsed-literal::\n ├───.github\n ├───asv_bench\n ├───ci\n ├───docker\n ├───docs\n ├───examples\n ├───modin\n │ ├─── :doc:`config `\n | ├─── :doc:`utils `\n │ ├───core\n │ │ ├─── :doc:`dataframe `\n │ │ │ ├─── :doc:`algebra `\n │ │ │ ├─── :doc:`base `\n │ │ │ └─── :doc:`pandas `\n │ │ ├───execution\n │ │ │ ├───dask\n │ │ │ │ ├───common\n │ │ │ │ └───implementations\n │ │ │ │ └─── :doc:`pandas_on_dask `\n │ │ │ ├─── :doc:`dispatching `\n │ │ │ ├───python\n │ │ │ │ └───implementations\n │ │ │ │ └─── :doc:`pandas_on_python `\n │ │ │ ├───ray\n │ │ │ │ ├───common\n │ │ │ │ ├─── :doc:`generic `\n │ │ │ │ └───implementations\n │ │ │ │ └─── :doc:`pandas_on_ray `\n │ │ │ └───unidist\n │ │ │ ├───common\n │ │ │ ├─── :doc:`generic `\n │ │ │ └───implementations\n │ │ │ └─── :doc:`pandas_on_unidist `\n │ │ ├─── :doc:`io `\n │ │ └─── :doc:`storage_formats `\n │ │ ├─── :doc:`base `\n │ │ └─── :doc:`pandas `\n │ ├───distributed\n │ │ ├───dataframe\n │ │ │ └─── :doc:`pandas `\n │ ├─── :doc:`experimental `\n │ │ ├───core\n | | | └─── :doc:`io `\n │ │ ├─── :doc:`pandas `\n │ │ ├─── :doc:`sklearn `\n │ │ ├───spreadsheet\n │ │ ├─── :doc:`xgboost `\n │ │ └─── :doc:`batch `\n │ └───pandas\n │ ├─── :doc:`dataframe `\n │ └─── :doc:`series `\n ├───requirements\n ├───scripts\n └───stress_tests\n\n.. _pandas Dataframe: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html\n.. _Ray: https://github.com/ray-project/ray\n.. _Unidist: https://github.com/modin-project/unidist\n.. _MPI: https://www.mpi-forum.org/\n.. _code: https://github.com/modin-project/modin/blob/main/modin/core/dataframe\n.. _Dask: https://github.com/dask/dask\n.. _Dask Futures: https://docs.dask.org/en/latest/futures.html\n.. _issue: https://github.com/modin-project/modin/issues\n.. _task parallel: https://en.wikipedia.org/wiki/Task_parallelism\n.. _experimental features: /usage_guide/advanced_usage/index.html\n"
},
{
"path": "docs/development/contributing.rst",
"content": "Contributing\n============\n\nGetting Started\n---------------\n\nIf you're interested in getting involved in the development of Modin, but aren't sure\nwhere start, take a look at the issues tagged `Good first issue`_ or Documentation_.\nThese are issues that would be good for getting familiar with the codebase and better\nunderstanding some of the more complex components of the architecture. There is\ndocumentation here about the :doc:`architecture ` that you will\nwant to review in order to get started.\n\nAlso, feel free to join the discussions on the `developer mailing list`_.\n\nIf you want a quick guide to getting your development environment setup, please\nuse `the contributing instructions on GitHub`_.\n\nCertificate of Origin\n---------------------\n\nTo keep a clear track of who did what, we use a `sign-off` procedure (same requirements\nfor using the signed-off-by process as the Linux kernel has\nhttps://www.kernel.org/doc/html/v4.17/process/submitting-patches.html) on patches or pull\nrequests that are being sent. The sign-off is a simple line at the end of the explanation\nfor the patch, which certifies that you wrote it or otherwise have the right to pass it\non as an open-source patch. The rules are pretty simple: if you can certify the below:\n\nCERTIFICATE OF ORIGIN V 1.1\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\"By making a contribution to this project, I certify that:\n\n1.) The contribution was created in whole or in part by me and I have the right to\nsubmit it under the open source license indicated in the file; or\n2.) The contribution is based upon previous work that, to the best of my knowledge, is\ncovered under an appropriate open source license and I have the right under that license\nto submit that work with modifications, whether created in whole or in part by me, under\nthe same open source license (unless I am permitted to submit under a different\nlicense), as indicated in the file; or\n3.) The contribution was provided directly to me by some other person who certified (a),\n(b) or (c) and I have not modified it.\n4.) I understand and agree that this project and the contribution are public and that a\nrecord of the contribution (including all personal information I submit with it,\nincluding my sign-off) is maintained indefinitely and may be redistributed consistent\nwith this project or the open source license(s) involved.\"\n\n\n.. code-block:: bash\n\n This is my commit message\n\n Signed-off-by: Awesome Developer \n\n\nCode without a proper signoff cannot be merged into the\nmain branch. Note: You must use your real name (sorry, no pseudonyms or anonymous\ncontributions.)\n\nThe text can either be manually added to your commit body, or you can add either ``-s``\nor ``--signoff`` to your usual ``git commit`` commands:\n\n\n\n.. code-block:: bash\n\n git commit --signoff -m \"This is my commit message\"\n git commit -s -m \"This is my commit message\"\n\nThis will use your default git configuration which is found in .git/config. To change\nthis, you can use the following commands:\n\n.. code-block:: bash\n\n git config --global user.name \"Awesome Developer\"\n git config --global user.email \"awesome.developer.@example.org\"\n\nIf you have authored a commit that is missing the signed-off-by line, you can amend your\ncommits and push them to GitHub.\n\n.. code-block:: bash\n\n git commit --amend --signoff\n\nIf you've pushed your changes to GitHub already you'll need to force push your branch\nafter this with ``git push -f``.\n\nCommit Message formatting\n-------------------------\nWe request that your first commit follow a particular format, and we\n**require** that your PR title follow the format. The format is:\n\n.. code-block:: bash\n\n FEAT-#9999: Add `DataFrame.rolling` functionality, to enable rolling window operations\n\nThe ``FEAT`` component represents the type of commit. This component of the commit\nmessage can be one of the following:\n\n* FEAT: A new feature that is added\n* DOCS: Documentation improvements or updates\n* FIX: A bugfix contribution\n* REFACTOR: Moving or removing code without change in functionality\n* TEST: Test updates or improvements\n* PERF: Performance enhancements\n\nThe ``#9999`` component of the commit message should be the issue number in the Modin\nGitHub issue tracker: https://github.com/modin-project/modin/issues. This is important\nbecause it links commits to their issues.\n\nThe commit message should follow a colon (:) and be descriptive and succinct.\n\nA Modin CI job on GitHub will enforce that your pull request title follows the\nformat we suggest. Note that if you update the PR title, you have to push\nanother commit (even if it's empty) or amend your last commit for the job to\npick up the new PR title. Re-running the job in Github Actions won't work.\n\nGeneral Rules for committers\n----------------------------\n\n- Try to write a PR name as descriptive as possible.\n- Try to keep PRs as small as possible. One PR should be making one semantically atomic change.\n- Don't merge your own PRs even if you are technically able to do it.\n\nDevelopment Dependencies\n------------------------\n\nWe recommend doing development in a virtualenv or conda environment, though this decision\nis ultimately yours. You will want to run the following in order to install all of the required\ndependencies for running the tests and formatting the code:\n\n.. code-block:: bash\n\n conda env create --file environment-dev.yml\n # or\n pip install -r requirements-dev.txt\n\nCode Formatting and Lint\n------------------------\n\nWe use black_ for code formatting. Before you submit a pull request, please make sure\nthat you run the following from the project root:\n\n.. code-block:: bash\n\n black modin/ asv_bench/benchmarks scripts/doc_checker.py\n\nWe also use flake8_ to check linting errors. Running the following from the project root\nwill ensure that it passes the lint checks on Github Actions:\n\n.. code-block:: bash\n\n flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py\n\nWe test that this has been run on our `Github Actions`_ test suite. If you do this and find\nthat the tests are still failing, try updating your version of black and flake8.\n\nAdding a test\n-------------\n\nIf you find yourself fixing a bug or adding a new feature, don't forget to add a test to\nthe test suite to verify its correctness! More on testing and the layout of the tests\ncan be found in our testing documentation. We ask that you follow the existing\nstructure of the tests for ease of maintenance.\n\nRunning the tests\n-----------------\n\nTo run the entire test suite, run the following from the project root:\n\n.. code-block:: bash\n\n pytest modin/pandas/test\n\nThe test suite is very large, and may take a long time if you run every test. If you've\nonly modified a small amount of code, it may be sufficient to run a single test or some\nsubset of the test suite. In order to run a specific test run:\n\n.. code-block:: bash\n\n pytest modin/pandas/test::test_new_functionality\n\nThe entire test suite is automatically run for each pull request.\n\nPerformance measurement\n-----------------------\n\nWe use Asv_ tool for performance tracking of various Modin functionality. The results\ncan be viewed here: `Asv dashboard`_.\n\nMore information can be found in the `Asv readme`_.\n\n\nBuilding documentation\n----------------------\n\nTo build the documentation, please follow the steps below from the project root:\n\n.. code-block:: bash\n\n pip install -r docs/requirements-doc.txt\n sphinx-build -b html docs docs/build\n\nTo visualize the documentation locally, run the following from `build` folder:\n\n.. code-block:: bash\n\n python -m http.server \n # python -m http.server 1234\n\nthen open the browser at `0.0.0.0:` (e.g. `0.0.0.0:1234`).\n\nContributing a new execution framework or in-memory format\n----------------------------------------------------------\n\nIf you are interested in contributing support for a new execution framework or in-memory\nformat, please make sure you understand the :doc:`architecture ` of Modin.\n\nThe best place to start the discussion for adding a new execution framework or in-memory\nformat is the `developer mailing list`_.\n\nMore docs on this coming soon...\n\n.. _Good first issue: https://github.com/modin-project/modin/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue+%3Abeginner%3A%22\n.. _Documentation: https://github.com/modin-project/modin/issues?q=is%3Aissue+is%3Aopen+label%3A%22documentation+%3Abookmark_tabs%3A%22\n.. _black: https://github.com/ambv/black\n.. _flake8: http://flake8.pycqa.org/en/latest/\n.. _Github Actions: https://github.com/features/actions\n.. _Asv: https://github.com/airspeed-velocity/asv#airspeed-velocity\n.. _developer mailing list: https://groups.google.com/forum/#!forum/modin-dev\n.. _Asv dashboard: https://modin.org/modin-bench/#/\n.. _Asv readme: https://github.com/modin-project/modin/blob/main/asv_bench/README.md\n.. _the contributing instructions on GitHub: https://github.com/modin-project/modin/blob/main/contributing/contributing.md"
},
{
"path": "docs/development/index.rst",
"content": "Development\n===========\n\n.. toctree::\n :maxdepth: 4\n\n contributing\n architecture\n partition_api\n using_pandas_on_ray\n using_pandas_on_dask\n using_pandas_on_python\n using_pandas_on_mpi\n\n.. meta::\n :description lang=en:\n Development-specific documentation.\n"
},
{
"path": "docs/development/partition_api.rst",
"content": "Partition API in Modin\n======================\n\nWhen you are working with a :py:class:`~modin.pandas.dataframe.DataFrame`, you can unwrap its remote partitions\nto get the raw futures objects compatible with the execution engine (e.g. ``ray.ObjectRef`` for Ray).\nIn addition to unwrapping of the remote partitions we also provide an API to construct a ``modin.pandas.DataFrame``\nfrom raw futures objects.\n\nPartition IPs\n-------------\nFor finer grained placement control, Modin also provides an API to get the IP addresses of the nodes that hold each partition.\nYou can pass the partitions having needed IPs to your function. It can help with minimizing of data movement between nodes.\n\nPartition API implementations\n-----------------------------\nBy default, a :py:class:`~modin.pandas.dataframe.DataFrame` stores underlying partitions as ``pandas.DataFrame`` objects.\nYou can find the specific implementation of Modin's Partition Interface in :doc:`pandas Partition API `.\n\n.. toctree::\n :hidden:\n\n /flow/modin/distributed/dataframe/pandas\n\nRay engine\n----------\nHowever, it is worth noting that for Modin on ``Ray`` engine with ``pandas`` in-memory format IPs of the remote partitions may not match\nactual locations if the partitions are lower than 100 kB. Ray saves such objects (<= 100 kB, by default) in in-process store\nof the calling process (please, refer to `Ray documentation`_ for more information). We can't get IPs for such objects while maintaining good performance.\nSo, you should keep in mind this for unwrapping of the remote partitions with their IPs. Several options are provided to handle the case in\n``How to handle Ray objects that are lower 100 kB`` section.\n\nDask engine\n-----------\nThere is no mentioned above issue for Modin on ``Dask`` engine with ``pandas`` in-memory format because ``Dask`` saves any objects\nin the worker process that processes a function (please, refer to `Dask documentation`_ for more information).\n\nUnidist engine\n--------------\nCurrently, Modin only supports MPI through unidist. There is no mentioned above issue for\nModin on ``Unidist`` engine using ``MPI`` backend with ``pandas`` in-memory format\nbecause ``Unidist`` saves any objects in the MPI worker process that processes a function\n(please, refer to `Unidist documentation`_ for more information).\n\nHow to handle Ray objects that are lower than 100 kB\n----------------------------------------------------\n\n* If you are sure that each of the remote partitions being unwrapped is higher than 100 kB, you can just import Modin or perform ``ray.init()`` manually.\n\n* If you don't know partition sizes you can pass the option ``_system_config={\"max_direct_call_object_size\": ,}``, where ``nbytes`` is threshold for objects that will be stored in in-process store, to ``ray.init()``.\n\n* You can also start Ray as follows: ``ray start --head --system-config='{\"max_direct_call_object_size\":}'``.\n\nNote that when specifying the threshold the performance of some Modin operations may change.\n\n.. _`Ray documentation`: https://docs.ray.io/en/master/index.html#\n.. _`Dask documentation`: https://distributed.dask.org/en/latest/index.html\n.. _`Unidist documentation`: https://unidist.readthedocs.io/en/latest/index.html\n"
},
{
"path": "docs/development/using_pandas_on_dask.rst",
"content": "pandas on Dask\n==============\n\nThis section describes usage related documents for the pandas on Dask component of Modin.\n\nModin uses pandas as a primary memory format of the underlying partitions and optimizes queries\ningested from the API layer in a specific way to this format. Thus, there is no need to care of choosing it\nbut you can explicitly specify it anyway as shown below.\n\nOne of the execution engines that Modin uses is Dask. To enable the pandas on Dask execution you should set the following environment variables:\n\n.. code-block:: bash\n\n export MODIN_ENGINE=dask\n export MODIN_STORAGE_FORMAT=pandas\n\nor turn them on in source code:\n\n.. code-block:: python\n\n import modin.config as cfg\n cfg.Engine.put('dask')\n cfg.StorageFormat.put('pandas')\n\nUsing Modin on Dask locally\n---------------------------\n\nIf you want to run Modin on Dask locally using a single node, just set Modin engine to ``Dask`` and \ncontinue working with a Modin DataFrame as if it was a pandas DataFrame.\nYou can either initialize a Dask client on your own and Modin connects to the existing Dask cluster or\nallow Modin itself to initialize a Dask client.\n\n.. code-block:: python\n\n import modin.pandas as pd\n import modin.config as modin_cfg\n\n modin_cfg.Engine.put(\"dask\")\n df = pd.DataFrame(...)\n\nUsing Modin on Dask in a Cluster\n--------------------------------\n\nIf you want to run Modin on Dask in a cluster, you should set up a Dask cluster and initialize a Dask client.\nOnce the Dask client is initialized, Modin will be able to connect to it and use the Dask cluster.\n\n.. code-block:: python\n\n from distributed import Client\n import modin.pandas as pd\n import modin.config as modin_cfg\n \n # Define your cluster here\n cluster = ...\n client = Client(cluster)\n\n modin_cfg.Engine.put(\"dask\")\n df = pd.DataFrame(...)\n\nTo get more information on how to deploy and run a Dask cluster, visit the `Deploy Dask Clusters`_ page.\n\nConversion between Modin DataFrame and Dask DataFrame\n-----------------------------------------------------\n\nModin DataFrame can be converted to/from Dask DataFrame with no-copy partition conversion.\nThis allows you to take advantage of both Modin and Dask libraries for maximum performance.\n\n.. code-block:: python\n\n import modin.pandas as pd\n import modin.config as modin_cfg\n from modin.pandas.io import to_dask, from_dask\n\n modin_cfg.Engine.put(\"dask\")\n df = pd.DataFrame(...)\n\n # Convert Modin to Dask DataFrame\n dask_df = to_dask(df)\n \n # Convert Dask to Modin DataFrame\n modin_df = from_dask(dask_df)\n\n.. _Deploy Dask Clusters: https://docs.dask.org/en/stable/deploying.html\n"
},
{
"path": "docs/development/using_pandas_on_mpi.rst",
"content": "pandas on MPI through unidist\n=============================\n\nThis section describes usage related documents for the pandas on MPI through unidist component of Modin.\n\nModin uses pandas as a primary memory format of the underlying partitions and optimizes queries\ningested from the API layer in a specific way to this format. Thus, there is no need to care of choosing it\nbut you can explicitly specify it anyway as shown below.\n\nOne of the execution engines that Modin uses is MPI through unidist.\nTo enable the pandas on MPI through unidist execution you should set the following environment variables:\n\n.. code-block:: bash\n\n export MODIN_ENGINE=unidist\n export MODIN_STORAGE_FORMAT=pandas\n export UNIDIST_BACKEND=mpi\n\nor turn it on in source code:\n\n.. code-block:: python\n\n import modin.config as modin_cfg\n import unidist.config as unidist_cfg\n\n modin_cfg.Engine.put('unidist')\n modin_cfg.StorageFormat.put('pandas')\n unidist_cfg.Backend.put('mpi')\n\nTo run a python application you should use ``mpiexec -n 1 python `` command.\n\n.. code-block:: bash\n\n mpiexec -n 1 python script.py\n\nFor more information on how to run a python application with unidist on MPI backend\nplease refer to `Unidist on MPI`_ section of the unidist documentation.\n\nAs of unidist 0.5.0 there is support for a shared object store for MPI backend.\nThe feature allows to improve performance in the workloads,\nwhere workers use same data multiple times by reducing data copies.\nYou can enable the feature by setting the following environment variable:\n\n.. code-block:: bash\n\n export UNIDIST_MPI_SHARED_OBJECT_STORE=True\n\nor turn it on in source code:\n\n.. code-block:: python\n\n import unidist.config as unidist_cfg\n\n unidist_cfg.MpiSharedObjectStore.put(True)\n\n.. _`Unidist on MPI`: https://unidist.readthedocs.io/en/latest/using_unidist/unidist_on_mpi.html"
},
{
"path": "docs/development/using_pandas_on_python.rst",
"content": "pandas on Python\n================\n\nThis section describes usage related documents for the pandas on Python component of Modin.\n\nModin uses pandas as the primary memory format of the underlying partitions and optimizes queries\nfrom the API layer in a specific way to this format. Since it is a default, you do not need to specify\nthe pandas memory format, but we show how to explicitly set it below.\n\nOne of the execution engines that Modin uses is Python. This engine is sequential and used for debugging.\nTo enable the pandas on Python execution you should set the following environment variables:\n\n.. code-block:: bash\n\n export MODIN_ENGINE=python\n export MODIN_STORAGE_FORMAT=pandas\n\nor turn a debug mode on:\n\n.. code-block:: bash\n\n export MODIN_DEBUG=True\n export MODIN_STORAGE_FORMAT=pandas\n\nor do the same in source code:\n\n.. code-block:: python\n\n import modin.config as cfg\n cfg.Engine.put('python')\n cfg.StorageFormat.put('pandas')\n\n.. code-block:: python\n\n import modin.config as cfg\n cfg.IsDebug.put(True)\n cfg.StorageFormat.put('pandas')"
},
{
"path": "docs/development/using_pandas_on_ray.rst",
"content": "pandas on Ray\n=============\n\nThis section describes usage related documents for the pandas on Ray component of Modin.\n\nModin uses pandas as a primary memory format of the underlying partitions and optimizes queries\ningested from the API layer in a specific way to this format. Thus, there is no need to care of choosing it\nbut you can explicitly specify it anyway as shown below.\n\nOne of the execution engines that Modin uses is Ray. If you have Ray installed in your system,\nModin also uses it by default to distribute computations.\n\nIf you want to be explicit, you could set the following environment variables:\n\n.. code-block:: bash\n\n export MODIN_ENGINE=ray\n export MODIN_STORAGE_FORMAT=pandas\n\nor turn it on in source code:\n\n.. code-block:: python\n\n import modin.config as cfg\n cfg.Engine.put('ray')\n cfg.StorageFormat.put('pandas')\n"
},
{
"path": "docs/ecosystem.rst",
"content": "Ecosystem\n=========\n\nThere is a constantly growing number of users and packages using pandas\nto address their specific needs in data preparation, analysis and visualization.\npandas is being used ubiquitously and is a good choise to handle small-sized data.\nHowever, pandas scales poorly and is non-interactive on moderate to large datasets.\nModin provides a drop-in replacement API for pandas and scales computation across nodes and\nCPUs available. What you need to do to switch to Modin is just replace a single line of code.\n\n.. code-block:: python\n\n # import pandas as pd\n import modin.pandas as pd\n\nWhile most packages can consume a pandas DataFrame and operate it efficiently,\nthis is not the case with a Modin DataFrame due to its distributed nature.\nThus, some packages may lack support for handling Modin DataFrame(s) correctly and,\nmoreover, efficiently. Modin implements such methods as ``__array__``, ``__dataframe__``, etc.\nto facilitate other libraries to consume a Modin DataFrame. If you feel that a certain library\ncan operate efficiently with a specific format of data, it is possible to convert a Modin DataFrame\nto the format preferred.\n\nto_pandas\n---------\n\nYou can refer to `pandas ecosystem`_ page to get more details on\nwhere pandas can be used and what libraries it powers.\n\n.. code-block:: python\n\n from modin.pandas.io import to_pandas\n\n pandas_df = to_pandas(modin_df)\n\nto_numpy\n--------\n\nYou can refer to `NumPy ecosystem`_ section of NumPy documentation to get more details on\nwhere NumPy can be used and what libraries it powers.\n\n.. code-block:: python\n\n from modin.pandas.io import to_numpy\n\n numpy_arr = to_numpy(modin_df)\n\nto_ray\n------\n\nYou can refer to `Ray Data`_ page to get more details on\nwhere Ray Dataset can be used and what libraries it powers.\n\n.. code-block:: python\n\n from modin.pandas.io import to_ray\n\n ray_dataset = to_ray(modin_df)\n\nto_dask\n-------\n\nYou can refer to `Dask DataFrame`_ page to get more details on\nwhere Dask DataFrame can be used and what libraries it powers.\n\n.. code-block:: python\n\n from modin.pandas.io import to_dask\n\n dask_df = to_dask(modin_df)\n\n.. _pandas ecosystem: https://pandas.pydata.org/community/ecosystem.html\n.. _NumPy ecosystem: https://numpy.org\n.. _Ray Data: https://docs.ray.io/en/latest/data/data.html\n.. _Dask DataFrame: https://docs.dask.org/en/stable/dataframe.html\n\n"
},
{
"path": "docs/flow/modin/config.rst",
"content": ":orphan:\n\nModin Configuration Settings\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nTo adjust Modin's default behavior, you can set the value of Modin\nconfigs by setting an environment variable or by using the\n``modin.config`` API. To list all available configs in Modin, please\nrun ``python -m modin.config`` to print all\nModin configs with descriptions.\n\nPublic API\n''''''''''\n\nPotentially, the source of configs can be any, but for now only environment\nvariables are implemented. Any environment variable originate from\n:class:`~modin.config.envvars.EnvironmentVariable`, which contains most of\nthe config API implementation.\n\n.. autoclass:: modin.config.envvars.EnvironmentVariable\n :members: get, put, get_help, get_value_source, once, subscribe\n\nModin Configs List\n''''''''''''''''''\n\n.. csv-table::\n :file: configs_help.csv\n :header-rows: 1\n\nUsage Guide\n'''''''''''\n\nSee example of interaction with Modin configs below, as it can be seen config\nvalue can be set either by setting the environment variable or by using config\nAPI.\n\n.. code-block:: python\n\n import os\n\n # Setting `MODIN_ENGINE` environment variable.\n # Also can be set outside the script.\n os.environ[\"MODIN_ENGINE\"] = \"Dask\"\n\n import modin.config\n import modin.pandas as pd\n\n # Checking initially set `Engine` config,\n # which corresponds to `MODIN_ENGINE` environment\n # variable\n print(modin.config.Engine.get()) # prints 'Dask'\n\n # Checking default value of `NPartitions`\n print(modin.config.NPartitions.get()) # prints '8'\n\n # Changing value of `NPartitions`\n modin.config.NPartitions.put(16)\n print(modin.config.NPartitions.get()) # prints '16'\n\nOne can also use config variables with a context manager in order to use\nsome config only for a certain part of the code:\n\n.. code-block:: python\n\n import modin.config as cfg\n\n # Default value for this config is 'False'\n print(cfg.RangePartitioning.get()) # False\n\n # Set the config to 'True' inside of the context-manager\n with cfg.context(RangePartitioning=True):\n print(cfg.RangePartitioning.get()) # True\n df.merge(...) # will use range-partitioning impl\n\n # Once the context is over, the config gets back to its previous value\n print(cfg.RangePartitioning.get()) # False\n\n # You can also set multiple config at once when you pass a dictionary to 'cfg.context'\n print(cfg.AsyncReadMode.get()) # False\n\n with cfg.context(RangePartitioning=True, AsyncReadMode=True):\n print(cfg.RangePartitioning.get()) # True\n print(cfg.AsyncReadMode.get()) # True\n print(cfg.RangePartitioning.get()) # False\n print(cfg.AsyncReadMode.get()) # False\n"
},
{
"path": "docs/flow/modin/core/dataframe/algebra.rst",
"content": ":orphan:\n\nOperators Module Description\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nBrief description\n'''''''''''''''''\nMost of the functions that are evaluated by `QueryCompiler` can be categorized into\none of the patterns: Map, TreeReduce, Binary, Reduce, etc., called core operators. The ``modin.core.dataframe.algebra``\nmodule provides templates to easily build such types of functions. These templates\nare supposed to be used at the `QueryCompiler` level since each built function accepts\nand returns `QueryCompiler`.\n\nHigh-Level Module Overview\n''''''''''''''''''''''''''\nEach template class implements a\n``register`` method, which takes functions to apply and\ninstantiate the related template. Functions that are passed to ``register`` will be executed\nagainst converted to pandas and preprocessed in a template-specific way partition, so the function\nwould take one of the pandas object: ``pandas.DataFrame``, ``pandas.Series`` or ``pandas.DataFrameGroupbyObject``.\n\n.. note:: \n Currently, functions that are built in that way are supported only in a pandas\n storage format (i.e. can be used only in `PandasQueryCompiler`).\n\nAlgebra module provides templates for this type of function:\n\nMap operator\n-------------\nUniformly apply a function argument to each partition in parallel. \n**Note**: map function should not change the shape of the partitions.\n\n.. figure:: /img/map_evaluation.svg\n :align: center\n\nThis operator performs best when the number of partitions equals to the number of CPUs\nso that each single partition gets processed in parallel. When the number of partitions is 1.5x greater than\nthe number of CPUs, Modin applies a heuristic to join some partitions to get \"ideal\" partitioning so that\neach new partition gets processed in parallel.\n\nReduce operator\n---------------\nApplies an argument function that reduces each column or row on the specified axis into a scalar, but requires knowledge about the whole axis.\nBe aware that providing this knowledge may be expensive because the execution engine has to\nconcatenate partitions along the specified axis. Also, note that the execution engine expects\nthat the reduce function returns a one dimensional frame.\n\n.. figure:: /img/reduce_evaluation.svg\n :align: center\n\nThis operator performs best when the number of partitions (row or column partitions in depend on the specified axis)\nequals to the number of CPUs so that each single axis partition gets processed in parallel.\n\nTreeReduce operator\n-------------------\nApplies an argument function that reduces specified axis into a scalar. First applies map function to each partition\nin parallel, then concatenates resulted partitions along the specified axis and applies reduce\nfunction. In contrast with `Map function` template, here you're allowed to change partition shape\nin the map phase. Note that the execution engine expects that the reduce function returns a one dimensional frame.\n\nThis operator performs best when the number of partitions (including the initial and intermediate stages)\nequals to the number of CPUs so that each single axis partition gets processed in parallel.\n\nBinary operator\n---------------\nApplies an argument function, that takes exactly two operands (first is always `QueryCompiler`).\nIf both operands are query compilers then the execution engine broadcasts partitions of\nthe right operand to the left.\n\n.. figure:: /img/binary_evaluation.svg\n :align: center\n\n.. warning::\n To be able to do frame broadcasting, partitioning along the index axis of both frames\n has to be equal, otherwise they need to be aligned first. The execution engine will do\n it automatically but note that this requires repartitioning, which is a much \n more expensive operation than the binary function itself.\n\nThis operator performs best when both operands have identical partitioning and the number of partitions of an operand\nequals to the number of CPUs so that each single partition gets processed in parallel.\n\nFold operator\n-------------\nApplies an argument function that requires knowledge of the whole axis. Be aware that providing this knowledge may be\nexpensive because the execution engine has to concatenate partitions along the specified axis.\n\nThis operator performs best when the number of partitions (row or column partitions in depend on the specified axis)\nequals to the number of CPUs so that each single axis partition gets processed in parallel.\n\nGroupBy operator\n----------------\nEvaluates GroupBy aggregation for that type of functions that can be executed via TreeReduce approach.\nTo be able to form groups engine broadcasts ``by`` partitions to each partition of the source frame.\n\nThis operator performs best when the cardinality of ``by`` columns is low (small number of output groups).\nAt the ``Map`` stage, the operator computes the aggregation for each row partition individually, meaning,\nthat the ``Reduce`` stage takes a dataframe with the following number of rows:\n``num_groups * n_row_parts``. If the number of groups is too high, there's a risk of getting a dataframe\nwith even bigger than the initial shape at the ``Reduce`` stage.\n\nDefault-to-pandas operator\n--------------------------\nDo :doc:`fallback to pandas ` for passed function.\n\nThis operator has a performance penalty for going from a partitioned Modin DataFrame to pandas because of\nthe communication cost and single-threaded nature of pandas.\n\n\nHow to register your own function\n'''''''''''''''''''''''''''''''''\nLet's examine an example of how to use the algebra module to create your own\nnew functions.\n\nImagine you have a complex aggregation that can be implemented into a single query but\ndoesn't have any implementation in pandas API. If you know how to implement this\naggregation efficiently in a distributed frame, you may want to use one of the above described\npatterns (e.g. ``TreeReduce``).\n\nLet's implement a function that counts non-NA values for each column or row\n(``pandas.DataFrame.count``). First, we need to determine the function type.\nTreeReduce approach would be great: in a map phase, we'll count non-NA cells in each\npartition in parallel and then just sum its results in the reduce phase.\n\nTo define the TreeReduce function that does `count` + `sum` we just need to register the\nappropriate functions and then assign the result to the picked `QueryCompiler`\n(`PandasQueryCompiler` in our case):\n\n.. code-block:: python\n\n from modin.core.storage_formats import PandasQueryCompiler\n from modin.core.dataframe.algebra import TreeReduce\n\n PandasQueryCompiler.custom_count = TreeReduce.register(pandas.DataFrame.count, pandas.DataFrame.sum)\n\nThen, we want to handle it from the :py:class:`~modin.pandas.dataframe.DataFrame`, so we need to create a way to do that:\n\n.. code-block:: python\n\n import modin.pandas as pd\n\n def count_func(self, **kwargs):\n # The constructor allows you to pass in a query compiler as a keyword argument\n return self.__constructor__(query_compiler=self._query_compiler.custom_count(**kwargs))\n\n pd.DataFrame.count_custom = count_func\n\nAnd then you can use it like you usually would:\n\n.. code-block:: python\n\n df.count_custom(axis=1)\n\nMany of the `pandas` API functions can be easily implemented this way, so if you find\nout that one of your favorite function is still defaulted to pandas and decide to\ncontribute to Modin to add its implementation, you may use this example as a reference.\n"
},
{
"path": "docs/flow/modin/core/dataframe/base/dataframe.rst",
"content": "ModinDataframe\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe` is the parent class for all dataframes - regardless of what storage format they are backed by. Its purpose is to define the algebra operators that must be exposed by a dataframe.\n\nThis class exposes the dataframe algebra and is meant to be subclassed by all dataframe implementations.\nDescendants of this class implement the algebra, and act as the intermediate level\nbetween the query compiler and the underlying execution details (e.g. the conforming partition manager). The class provides\na significantly reduced set of operations that can be composed to form any pandas query.\n\nThe :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` is an example of a descendant of this class. It currently has implementations for some of the operators\nexposed in this class, and is currently being refactored to include implementations for all of the algebra operators. Please\nrefer to the :doc:`PandasDataframe documentation ` for more information.\n\nThe :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe` is independent of implementation specific details such as partitioning, storage format, or execution engine.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.dataframe.base.dataframe.dataframe.ModinDataframe\n :members:\n"
},
{
"path": "docs/flow/modin/core/dataframe/base/index.rst",
"content": "Purpose\n=======\n\nThe :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe` serves the purpose of describing and defining the :doc:`Core Dataframe Algebra `.\n\nIt is the core construction element and serves as the client for the :doc:`Modin Query Compiler`. Descendants that offer implementations execute the queries from the compiler by invoking functions over partitions via a partition manager.\n\nThe partitions and partition manager interfaces are currently implementation-specific, but may\nbe standardized in the future.\n\nThe :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe` and axis partitions are the interfaces that must be implemented by any :doc:`execution backend` in order for it to be plugged in to Modin.\nThese classes are mostly abstract, however very simple and generic enough methods like\n:py:meth:`~modin.core.dataframe.base.partitioning.BaseDataframeAxisPartition.force_materialization` can be implemented at the base level because for now we do not expect them to differ in any implementation.\n\nModinDataframe Interface\n========================\n\n* :doc:`ModinDataframe ` is an abstract class which represents the algebra operators a dataframe must expose.\n* :doc:`BaseDataframeAxisPartition ` is an abstract class, representing a joined group of partitions along some axis (either rows or labels).\n\n.. toctree::\n :hidden:\n\n dataframe\n partitioning/axis_partition\n"
},
{
"path": "docs/flow/modin/core/dataframe/base/partitioning/axis_partition.rst",
"content": "BaseDataframeAxisPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is base for any axis partition class and serves as the last level on which\noperations that were conveyed from the partition manager are being performed on an entire column or row.\n\n**Note**: ``modin.core.dataframe.base`` intentionally does not describe any particular partition interface,\nas it is the partition manager responsibility (if said partition manager is implemented), i.e. it is\ntoo low-level to be present on the base, abstract level.\n\nThe class provides an API that has to be overridden by the child classes in order to manipulate\non a list of block partitions (making up column or row partition) they store.\n\nThe procedures that use this class and its methods assume that they have some global knowledge\nabout the entire axis. This may require the implementation to use concatenation or append on the\nlist of block partitions.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.dataframe.base.partitioning.axis_partition.BaseDataframeAxisPartition\n :members:\n"
},
{
"path": "docs/flow/modin/core/dataframe/index.rst",
"content": ":orphan:\n\nCore Modin Dataframe Objects\n============================\n\nModin partitions data to scale efficiently.\nTo keep track of everything a few key classes are introduced: ``Dataframe``, ``Partition``, ``AxisPartiton`` and ``PartitionManager``.\n\n* ``Dataframe`` is the class conforming to Dataframe Algebra.\n* ``Partition`` is an element of a NxM grid which, when combined, represents the ``Dataframe``\n* ``AxisPartition`` is a joined group of ``Partition``-s along some axis (either rows or columns)\n* ``PartitionManager`` is the manager that implements the primitives used for Dataframe Algebra operations over ``Partition``-s\n\nEach :doc:`storage format `, execution engine, and each execution system (storage format + execution engine)\nmay have its own implementations of these Core Dataframe's entities.\nCurrent stable implementations are the following:\n\n* :doc:`Base ModinDataframe ` defines a common interface and algebra operators for `Dataframe` implementations.\n\nStorage format specific:\n\n* :doc:`Modin PandasDataframe ` is an implementation for any frame class of :doc:`pandas storage format `.\n\nEngine specific:\n\n* :doc:`Modin GenericRayDataframe ` is an implementation for any frame class that works on Ray execution engine.\n* :doc:`Modin GenericUnidistDataframe ` is an implementation for any frame class that works on Unidist execution engine.\n\nExecution system specific:\n\n* :doc:`Modin PandasOnRayDataframe ` is a specialization of the Core Modin Dataframe for ``PandasOnRay`` execution.\n* :doc:`Modin PandasOnDaskDataframe ` is specialization of the Core Modin Dataframe for ``PandasOnDask`` execution.\n* :doc:`Modin PandasOnPythonDataframe ` is a specialization of the Core Modin Dataframe for ``PandasOnPython`` execution.\n* :doc:`Modin PandasOnUnidistDataframe ` is a specialization of the Core Modin Dataframe for ``PandasOnUnidist`` execution.\n\n.. note::\n At the current stage of Modin development, the base interfaces of the Dataframe objects are not defined yet.\n So for now the origin of all changes in the Dataframe interfaces is the :doc:`Dataframe for pandas storage format`.\n\n.. toctree::\n :hidden:\n\n base/index\n pandas/index\n"
},
{
"path": "docs/flow/modin/core/dataframe/pandas/dataframe.rst",
"content": "PandasDataframe\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n:py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` is a direct descendant of :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe`. Its purpose is to implement the abstract interfaces for usage with all ``pandas``-based :doc:`storage formats`.\n:py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` could be inherited and augmented further by any specific implementation which needs it to take special care of some behavior or to improve performance for certain execution engine.\n\nThe class serves as the intermediate level\nbetween ``pandas`` query compiler and conforming partition manager. All queries formed\nat the query compiler layer are ingested by this class and then conveyed jointly with the stored partitions\ninto the partition manager for processing. Direct partitions manipulation by this class is prohibited except\ncases if an operation is strictly private or protected and called inside of the class only. The class provides\nsignificantly reduced set of operations that fit plenty of pandas operations.\n\nMain tasks of :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` are storage of partitions, manipulation with labels of axes and\nproviding set of methods to perform operations on the internal data.\n\nAs mentioned above, ``PandasDataframe`` shouldn't work with stored partitions directly and\nthe responsibility for modifying partitions array has to lay on :doc:`partitioning/partition_manager`. For example, method\n:meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.broadcast_apply_full_axis` redirects applying\nfunction to :meth:`~PandasDataframePartitionManager.broadcast_axis_partitions` method.\n\n``Modin PandasDataframe`` can be created from ``pandas.DataFrame``, ``pyarrow.Table``\n(methods :meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.from_pandas`,\n:meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.from_arrow` are used respectively). Also,\n``PandasDataframe`` can be converted to ``np.array``, ``pandas.DataFrame``\n(methods :meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.to_numpy`,\n:meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.to_pandas` are used respectively).\n\nManipulation with labels of axes happens using internal methods for changing labels on the new,\nadding prefixes/suffixes etc.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe\n :members:\n"
},
{
"path": "docs/flow/modin/core/dataframe/pandas/index.rst",
"content": "Modin PandasDataframe Objects\n=============================\n\n``modin.core.dataframe.pandas`` is the package which houses common implementations\nof different Modin internal classes used by most `pandas`-based :doc:`storage formats`.\n\nIt also double-serves as the full example of how to implement Modin execution backend pieces (sans the :doc:`execution part` which is absent here),\nas it implements everything an execution backend needs to be fully conformant to Modin expectations.\n\n* :doc:`PandasDataframe ` is the class conforming to Dataframe Algebra.\n* :doc:`PandasDataframePartition ` implements ``Partition`` interface holding ``pandas.DataFrame``.\n* :doc:`PandasDataframeAxisPartition ` is a joined group of ``PandasDataframePartition``-s along some axis (either rows or labels)\n* :doc:`PandasDataframePartitionManager ` is the manager that implements the primitives used for Dataframe Algebra operations over ``PandasDataframePartition``-s\n* :doc:`ModinDtypes `\n* :doc:`ModinIndex `\n\n.. toctree::\n :hidden:\n\n dataframe\n partitioning/partition\n partitioning/axis_partition\n partitioning/partition_manager\n metadata/dtypes\n metadata/index\n"
},
{
"path": "docs/flow/modin/core/dataframe/pandas/metadata/dtypes.rst",
"content": "ModinDtypes\n\"\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.dataframe.pandas.metadata.dtypes.ModinDtypes\n :members:"
},
{
"path": "docs/flow/modin/core/dataframe/pandas/metadata/index.rst",
"content": "ModinIndex\n\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.dataframe.pandas.metadata.index.ModinIndex\n :members:"
},
{
"path": "docs/flow/modin/core/dataframe/pandas/partitioning/axis_partition.rst",
"content": "PandasDataframeAxisPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class implements abstract interface methods from :py:class:`~modin.core.dataframe.base.partitioning.axis_partition.BaseDataframeAxisPartition`\ngiving the means for a sibling :doc:`partition manager` to actually work with the axis-wide partitions.\n\nThe class is base for any axis partition class of ``pandas`` storage format.\n\nSubclasses must implement ``list_of_blocks`` which represents data wrapped by the :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`\nobjects and creates something interpretable as a ``pandas.DataFrame``.\n\nSee :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.partitioning.axis_partition.PandasOnRayDataframeAxisPartition`\nfor an example on how to override/use this class when the implementation needs to be augmented.\n\nThe :py:class:`~modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition` object has an invariant that requires that this\nobject is never returned from a function. It assumes that there will always be\n``PandasDataframeAxisPartition`` object stored and structures itself accordingly.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition\n :members:\n"
},
{
"path": "docs/flow/modin/core/dataframe/pandas/partitioning/partition.rst",
"content": "PandasDataframePartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is base for any partition class of ``pandas`` storage format and serves as the last level\non which operations that were conveyed from the partition manager are being performed on an\nindividual block partition.\n\nThe class provides an API that has to be overridden by child classes in order to manipulate\non data and metadata they store.\n\nThe public API exposed by the children of this class is used in :py:class:`~modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager`.\n\nThe objects wrapped by the child classes are treated as immutable by ``PandasDataframePartitionManager`` subclasses\nand no logic for updating inplace.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition\n :members:\n"
},
{
"path": "docs/flow/modin/core/dataframe/pandas/partitioning/partition_manager.rst",
"content": "PandasDataframePartitionManager\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is base for any partition manager class of ``pandas`` storage format and serves as\nintermediate level between :doc:`Modin PandasDataframe <../dataframe>` and conforming :doc:`partition ` class.\nThe class is responsible for partitions manipulation and applying a function to individual partitions:\nblock partitions, row partitions or column partitions, i.e. the class can form axis partitions from\nblock partitions to apply a function if an operation requires access to an entire column or row.\nThe class translates frame API into partition API and also can have some preprocessing operations\ndepending on the partition type for improving performance (for example,\n:meth:`~modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager.preprocess_func`).\n\nMain task of partition manager is to keep knowledge of how partitions are stored and managed\ninternal to itself, so surrounding code could use it via lean enough API without worrying about\nimplementation details.\n\nPartition manager can apply user-passed (arbitrary) function in different modes:\n\n* block-wise (apply a function to individual block partitions):\n\n * optionally accepting partition indices along each axis\n * optionally accepting an item to be split so parts of it would be sent to each partition\n\n* along a full axis (apply a function to an entire column or row made up of block partitions when user function needs information about the whole axis)\n\nIt can also broadcast partitions from `right` to `left` when executing certain operations making\n`right` partitions available for functions executed where `left` live.\n\n..\n TODO: insert more text explaining \"broadcast\" term\n\nPartition manager also is used to create \"logical\" partitions, or :doc:`axis partitions `\nby joining existing partitions along specified axis (either rows or labels),\nand to concatenate different partition sets along given axis.\n\nIt also maintains mapping from \"external\" (end user-visible) indices along all axes to internal\nindices which are actually pairs of indices of partitions and indices inside the partitions,\nas well as manages conversion to numpy and pandas representations.\n\n\nPublic API\n----------\n\n.. autoclass:: modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/dataframe.rst",
"content": "PandasOnDaskDataframe\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is the specific implementation of the dataframe algebra for the `Dask` execution engine.\nIt serves as an intermediate level between ``pandas`` query compiler and\n:py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartitionManager`.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.dataframe.PandasOnDaskDataframe\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/index.rst",
"content": ":orphan:\n\nPandasOnDask Execution\n======================\n\nQueries that perform data transformation, data ingress or data egress using the `pandas on Dask` execution\npass through the Modin components detailed below.\n\nTo enable `pandas on Dask` execution, please refer to the usage section in :doc:`pandas on Dask `.\n\nData Transformation\n'''''''''''''''''''\n\n.. image:: /img/pandas_on_dask_data_transform.svg\n :align: center\n\nWhen a user calls any :py:class:`~modin.pandas.dataframe.DataFrame` API, a query starts forming at the `API` layer\nto be executed at the `Execution` layer. The `API` layer is responsible for processing the query appropriately,\nfor example, determining whether the final result should be a ``DataFrame`` or ``Series`` object. This layer is also responsible for sanitizing the input to the\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler`, e.g. validating a parameter from the query\nand defining specific intermediate values to provide more context to the query compiler.\nThe :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for\nprocessing the query, received from the :py:class:`~modin.pandas.dataframe.DataFrame` `API` layer,\nto determine how to apply it to a subset of the data - either cell-wise or along an axis-wise partition backed by the `pandas`\nstorage format. The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` maps the query to one of the :doc:`Core Algebra Operators ` of\nthe :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.dataframe.PandasOnDaskDataframe` which inherits\ngeneric functionality from the :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`.\n\nPandasOnDask Dataframe implementation\n-------------------------------------\n\nModin implements ``Dataframe``, ``PartitionManager``, ``AxisPartition`` and ``Partition`` classes\nspecifically for the `PandasOnDask` execution.\n\n* :doc:`PandasOnDaskDataframe `\n* :doc:`PandasOnDaskDataframePartition `\n* :doc:`PandasOnDaskDataframeVirtualPartition `\n* :doc:`PandasOnDaskDataframePartitionManager `\n\n.. toctree::\n :hidden:\n\n dataframe\n partitioning/partition\n partitioning/virtual_partition\n partitioning/partition_manager\n\n\nData Ingress\n''''''''''''\n\n.. image:: /img/pandas_on_dask_data_ingress.svg\n :align: center\n\nData Egress\n'''''''''''\n\n.. image:: /img/pandas_on_dask_data_egress.svg\n :align: center\n\n\nWhen a user calls any IO function from the ``modin.pandas.io`` module, the `API` layer queries the\n:py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` which defines a factory specific for\nthe execution, namely, the :py:class:`~modin.core.execution.dispatching.factories.factories.PandasOnDaskFactory`. The factory, in turn,\nexposes the :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO` class\nwhose responsibility is to perform a parallel read/write from/to a file.\n\nWhen reading data from a CSV file, for example, the :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO` class forwards\nthe user query to the :meth:`~modin.core.io.text.CSVDispatcher._read` method of :py:class:`~modin.core.io.text.CSVDispatcher`, where the query's parameters are preprocessed\nto check if they are supported by the execution (defaulting to pandas if they are not) and computes some metadata\ncommon for all partitions to be read. Then, the file is split into row chunks, and this data is used to launch remote tasks on the Dask workers\nvia the :meth:`~modin.core.execution.dask.common.engine_wrapper.DaskWrapper.deploy` method of :py:class:`~modin.core.execution.dask.common.engine_wrapper.DaskWrapper`.\nOn each Dask worker, the :py:class:`~modin.core.storage_formats.pandas.parsers.PandasCSVParser` parses data.\nAfter the remote tasks are finished, additional result postprocessing is performed,\nand a new query compiler with the data read is returned.\n\nWhen writing data to a CSV file, for example, the :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO` processes\nthe user query to execute it on Dask workers. Then, the :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO` asks the\n:py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskDataframe` to decompose the data into row-wise partitions\nthat will be written into the file in parallel in Dask workers.\n\n.. note::\n Currently, data egress uses default `pandas` implementation for `pandas on Dask` execution.\n"
},
{
"path": "docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.rst",
"content": "PandasOnDaskDataframePartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`,\nproviding the API to perform operations on a block partition, namely, ``pandas.DataFrame``, using Dask as the execution engine.\n\nIn addition to wrapping a ``pandas.DataFrame``, the class also holds the following metadata:\n\n* ``length`` - length of ``pandas.DataFrame`` wrapped\n* ``width`` - width of ``pandas.DataFrame`` wrapped\n* ``ip`` - node IP address that holds ``pandas.DataFrame`` wrapped\n\nAn operation on a block partition can be performed in two modes:\n\n* asynchronously_ - via :meth:`~modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartition.apply`\n* lazily_ - via :meth:`~modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartition.add_to_apply_calls`\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartition\n :members:\n\n .. _asynchronously: https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)\n .. _lazily: https://en.wikipedia.org/wiki/Lazy_evaluation\n"
},
{
"path": "docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition_manager.rst",
"content": "PandasOnDaskDataframePartitionManager\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThis class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager`\nusing Dask as the execution engine. This class is responsible for partition manipulation and applying a function to\nblock/row/column partitions.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartitionManager\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/virtual_partition.rst",
"content": "PandasOnDaskDataframeVirtualPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.virtual_partition.PandasOnDaskDataframeVirtualPartition`,\nproviding the API to perform operations on an axis (column or row) partition using Dask as the execution engine.\nThe axis partition is a wrapper over a list of block partitions that are stored in this class.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframeVirtualPartition\n :members:\n\nPandasOnDaskDataframeColumnPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframeColumnPartition\n :members:\n\nPandasOnDaskDataframeRowPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframeRowPartition\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/dispatching.rst",
"content": ":orphan:\n\n..\n TODO: add links to documentation for mentioned modules.\n\nFactories Module Description\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nBrief description\n'''''''''''''''''\nModin has several execution engines and storage formats, combining them together forms certain executions. \nCalling any :py:class:`~modin.pandas.dataframe.DataFrame` API function will end up in some execution-specific method. The responsibility of dispatching high-level API calls to\nexecution-specific function belongs to the :ref:`QueryCompiler `, which is determined at the time of the dataframe's creation by the factory of\nthe corresponding execution. The mission of this module is to route IO function calls from\nthe API level to its actual execution-specific implementations, which builds the\n`QueryCompiler` of the appropriate execution.\n\nExecution representation via Factories\n''''''''''''''''''''''''''''''''''''''\nExecution is a combination of the :doc:`storage format ` and an actual execution engine.\nFor example, ``PandasOnRay`` execution means the combination of the `pandas storage format` and `Ray` engine.\n\nEach storage format has its own :ref:`Query Compiler ` which compiles the most efficient queries\nfor the corresponding :doc:`Core Modin Dataframe ` implementation. Speaking about ``PandasOnRay``\nexecution, its Query Compiler is :doc:`PandasQueryCompiler ` and the\nDataframe implementation is :doc:`PandasDataframe `,\nwhich is general implementation for every execution of the pandas storage format. The actual implementation of ``PandasOnRay`` dataframe\nis defined by the :doc:`PandasOnRayDataframe ` class that\nextends ``PandasDataframe``.\n\nIn the scope of this module, each execution is represented with a factory class located in\n``modin/core/execution/dispatching/factories/factories.py``. Each factory contains a field that identifies the IO module of the corresponding execution. This IO module is\nresponsible for dispatching calls of IO functions to their actual implementations in the\nunderlying IO module. For more information about IO module visit :doc:`IO ` page.\n\nFactory Dispatcher\n''''''''''''''''''\nThe :py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` class provides \npublic methods whose interface corresponds to pandas IO functions, the only difference is that they return `QueryCompiler` of the\nselected storage format instead of high-level :py:class:`~modin.pandas.dataframe.DataFrame`. ``FactoryDispatcher`` is responsible for routing\nthese IO calls to the factory which represents the selected execution.\n\nSo when you call ``read_csv()`` function and your execution is ``PandasOnRay`` then the\ntrace would be the following:\n\n.. figure:: /img/factory_dispatching.svg\n :align: center\n\n``modin.pandas.read_csv`` calls ``FactoryDispatcher.read_csv``, which calls ``._read_csv``\nfunction of the factory of the selected execution, in our case it's ``PandasOnRayFactory._read_csv``,\nwhich in turn forwards this call to the actual implementation of ``read_csv`` — to the\n``PandasOnRayIO.read_csv``. The result of ``modin.pandas.read_csv`` will return a high-level Modin\nDataFrame with the appropriate `QueryCompiler` bound to it, which is responsible for\ndispatching all of the further function calls.\n\nPublic API\n''''''''''\n\n.. automodule:: modin.core.execution.dispatching.factories.factories\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/python/implementations/pandas_on_python/dataframe.rst",
"content": "PandasOnPythonDataframe\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is specific implementation of :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`\nfor `Python` execution engine. It serves as an intermediate level between\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` and\n:py:class:`~modin.core.execution.python.implementations.pandas_on_python.partitioning.partition_manager.PandasOnPythonDataframePartitionManager`.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.dataframe.dataframe.PandasOnPythonDataframe\n :members:"
},
{
"path": "docs/flow/modin/core/execution/python/implementations/pandas_on_python/index.rst",
"content": ":orphan:\n\nPandasOnPython Execution\n========================\n\nQueries that perform data transformation, data ingress or data egress using the `pandas on Python` execution\npass through the Modin components detailed below.\n\n`pandas on Python` execution is sequential and it's used for the debug purposes. To enable `pandas on Python` execution,\nplease refer to the usage section in :doc:`pandas on Python `.\n\nData Transformation\n'''''''''''''''''''\n\n.. image:: /img/pandas_on_python_data_transform.svg\n :align: center\n\nWhen a user calls any :py:class:`~modin.pandas.dataframe.DataFrame` API, a query starts forming at the `API` layer\nto be executed at the `Execution` layer. The `API` layer is responsible for processing the query appropriately,\nfor example, determining whether the final result should be a ``DataFrame`` or ``Series`` object. This layer is also responsible for sanitizing the input to the\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler`, e.g. validating a parameter from the query\nand defining specific intermediate values to provide more context to the query compiler.\nThe :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for\nprocessing the query, received from the :py:class:`~modin.pandas.dataframe.DataFrame` `API` layer,\nto determine how to apply it to a subset of the data - either cell-wise or along an axis-wise partition backed by the `pandas`\nstorage format. The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` maps the query to one of the :doc:`Core Algebra Operators ` of\nthe :py:class:`~modin.core.execution.python.implementations.pandas_on_python.dataframe.dataframe.PandasOnPythonDataframe` which inherits\ngeneric functionality from the :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`.\n\nPandasOnPython Dataframe implementation\n---------------------------------------\n\nThis page describes implementation of :doc:`Modin PandasDataframe Objects `\nspecific for `PandasOnPython` execution. Since Python engine doesn't allow computation parallelization,\noperations on partitions are performed sequentially. The absence of parallelization doesn't give any\nperformance speed-up, so ``PandasOnPython`` is used for testing purposes only.\n\n* :doc:`PandasOnPythonDataframe `\n* :doc:`PandasOnPythonDataframePartition `\n* :doc:`PandasOnPythonDataframeAxisPartition `\n* :doc:`PandasOnPythonDataframePartitionManager `\n\n.. toctree::\n :hidden:\n\n dataframe\n partitioning/partition\n partitioning/axis_partition\n partitioning/partition_manager\n\n\nData Ingress\n''''''''''''\n\n.. image:: /img/pandas_on_python_data_ingress.svg\n :align: center\n\nData Egress\n'''''''''''\n\n.. image:: /img/pandas_on_python_data_egress.svg\n :align: center\n\n\nWhen a user calls any IO function from the ``modin.pandas.io`` module, the `API` layer queries the\n:py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` which defines a factory specific for\nthe execution, namely, the :py:class:`~modin.core.execution.dispatching.factories.factories.PandasOnPythonFactory`. The factory, in turn,\nexposes the :py:class:`~modin.core.execution.python.implementations.pandas_on_python.io.PandasOnPythonIO` class\nwhose responsibility is a read/write from/to a file.\n\nWhen reading data from a CSV file, for example, the :py:class:`~modin.core.execution.python.implementations.pandas_on_python.io.io.PandasOnPythonIO` class\nreads the data using corresponding `pandas` function (``pandas.read_csv()`` in this case). After the reading is complete, a new query compiler is created from `pandas` object\nusing :py:meth:`~modin.core.execution.python.implementations.pandas_on_python.io.io.PandasOnPythonIO.from_pandas` and returned.\n\nWhen writing data to a CSV file, for example, the :py:class:`~modin.core.execution.python.implementations.pandas_on_python.io.PandasOnPythonIO` converts a query compiler\nto `pandas` object using :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.to_pandas`. After that, `pandas` writes the data to the file using\ncorresponding function (``pandas.to_csv()`` in this case)."
},
{
"path": "docs/flow/modin/core/execution/python/implementations/pandas_on_python/partitioning/axis_partition.rst",
"content": "PandasOnPythonDataframeAxisPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition`,\nproviding the API to perform operations on an axis partition, using Python\nas the execution engine. The axis partition is made up of list of block\npartitions that are stored in this class.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.virtual_partition.PandasOnPythonDataframeAxisPartition\n\nPandasOnPythonFrameColumnPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.virtual_partition.PandasOnPythonDataframeColumnPartition\n :members:\n\nPandasOnPythonFrameRowPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.virtual_partition.PandasOnPythonDataframeRowPartition\n :members:"
},
{
"path": "docs/flow/modin/core/execution/python/implementations/pandas_on_python/partitioning/partition.rst",
"content": "PandasOnPythonDataframePartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`,\nproviding the API to perform operations on a block partition using Python as the execution engine.\n\nIn addition to wrapping a ``pandas.DataFrame``, the class also holds the following metadata:\n\n* ``length`` - length of ``pandas.DataFrame`` wrapped\n* ``width`` - width of ``pandas.DataFrame`` wrapped\n\nAn operation on a block partition can be performed in two modes:\n\n* immediately via :meth:`~modin.core.execution.python.implementations.pandas_on_python.partitioning.partition.PandasOnPythonDataframePartition.apply` - \n in this case accumulated call queue and new function will be executed\n immediately.\n* lazily_ via :meth:`~modin.core.execution.python.implementations.pandas_on_python.partitioning.partition.PandasOnPythonDataframePartition.add_to_apply_calls` -\n in this case function will be added to the call queue and no computations\n will be done at the moment.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.partition.PandasOnPythonDataframePartition\n :members:\n\n .. _lazily: https://en.wikipedia.org/wiki/Lazy_evaluation"
},
{
"path": "docs/flow/modin/core/execution/python/implementations/pandas_on_python/partitioning/partition_manager.rst",
"content": "PandasOnPythonDataframePartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager`\nusing Python as the execution engine. This class is responsible for partitions manipulation and applying\na function to block/row/column partitions.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.partition_manager.PandasOnPythonDataframePartitionManager\n :members:"
},
{
"path": "docs/flow/modin/core/execution/ray/generic.rst",
"content": ":orphan:\n\nGeneric Ray-based members\n=========================\n\nObjects which are storage format agnostic but require specific Ray implementation\nare placed in ``modin.core.execution.ray.generic``.\n\nTheir purpose is to implement certain parallel I/O operations and to serve\nas a foundation for building storage format specific objects:\n\n.. autoclass:: modin.core.execution.ray.generic.io.RayIO\n :members:\n\n.. autoclass:: modin.core.execution.ray.generic.partitioning.GenericRayDataframePartitionManager\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/dataframe.rst",
"content": "PandasOnRayDataframe\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is specific implementation of :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`\nclass using Ray distributed engine. It serves as an intermediate level between\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` and\n:py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartitionManager`.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.dataframe.PandasOnRayDataframe\n :members:"
},
{
"path": "docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/index.rst",
"content": ":orphan:\n\nPandasOnRay Execution\n=====================\n\nQueries that perform data transformation, data ingress or data egress using the `pandas on Ray` execution\npass through the Modin components detailed below.\n\nTo enable `pandas on Ray` execution, please refer to the usage section in :doc:`pandas on Ray `.\n\nData Transformation\n'''''''''''''''''''\n\n.. image:: /img/pandas_on_ray_data_transform.svg\n :align: center\n\nWhen a user calls any :py:class:`~modin.pandas.dataframe.DataFrame` API, a query starts forming at the `API` layer\nto be executed at the `Execution` layer. The `API` layer is responsible for processing the query appropriately,\nfor example, determining whether the final result should be a ``DataFrame`` or ``Series`` object. This layer is also responsible for sanitizing the input to the\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler`, e.g. validating a parameter from the query\nand defining specific intermediate values to provide more context to the query compiler.\nThe :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for\nprocessing the query, received from the :py:class:`~modin.pandas.dataframe.DataFrame` `API` layer,\nto determine how to apply it to a subset of the data - either cell-wise or along an axis-wise partition backed by the `pandas`\nstorage format. The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` maps the query to one of the :doc:`Core Algebra Operators ` of\nthe :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.dataframe.PandasOnRayDataframe` which inherits\ngeneric functionality from the ``GenericRayDataframe`` and the :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`.\n\n..\n TODO: insert a link to ``GenericRayDataframe`` once we add an implementatiton of the class\n\nPandasOnRay Dataframe implementation\n------------------------------------\n\nModin implements ``Dataframe``, ``PartitionManager``, ``VirtualPartition`` (a specific kind of ``AxisPartition`` with the capability\nto combine smaller partitions into the one \"virtual\") and ``Partition`` classes specifically for the ``PandasOnRay`` execution:\n\n* :doc:`PandasOnRayDataframe `\n* :doc:`PandasOnRayDataframePartition `\n* :doc:`PandasOnRayDataframeVirtualPartition `\n* :doc:`PandasOnRayDataframePartitionManager `\n\n.. toctree::\n :hidden:\n\n dataframe\n partitioning/partition\n partitioning/axis_partition\n partitioning/partition_manager\n\nData Ingress\n''''''''''''\n\n.. image:: /img/pandas_on_ray_data_ingress.svg\n :align: center\n\nData Egress\n'''''''''''\n\n.. image:: /img/pandas_on_ray_data_egress.svg\n :align: center\n\n\nWhen a user calls any IO function from the ``modin.pandas.io`` module, the `API` layer queries the\n:py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` which defines a factory specific for\nthe execution, namely, the :py:class:`~modin.core.execution.dispatching.factories.factories.PandasOnRayFactory`. The factory, in turn,\nexposes the :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO` class\nwhose responsibility is to perform a parallel read/write from/to a file.\n\nWhen reading data from a CSV file, for example, the :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO` class forwards\nthe user query to the :meth:`~modin.core.io.text.CSVDispatcher._read` method of :py:class:`~modin.core.io.text.CSVDispatcher`, where the query's parameters are preprocessed\nto check if they are supported by the execution (defaulting to pandas if they are not) and computes some metadata\ncommon for all partitions to be read. Then, the file is split into row chunks, and this data is used to launch remote tasks on the Ray workers\nvia the :meth:`~modin.core.execution.ray.common.RayWrapper.deploy` method of :py:class:`~modin.core.execution.ray.common.RayWrapper`.\nOn each Ray worker, the :py:class:`~modin.core.storage_formats.pandas.parsers.PandasCSVParser` parses data.\nAfter the remote tasks are finished, additional result postprocessing is performed,\nand a new query compiler with the data read is returned.\n\nWhen writing data to a CSV file, for example, the :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO` processes\nthe user query to execute it on Ray workers. Then, the :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO` asks the\n:py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.dataframe.PandasOnRayDataframe` to decompose the data into row-wise partitions\nthat will be written into the file in parallel in Ray workers."
},
{
"path": "docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/axis_partition.rst",
"content": "PandasOnRayDataframeVirtualPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThis class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition`,\nproviding the API to perform operations on an axis partition, using Ray as an execution engine. The virtual partition is\na wrapper over a list of block partitions, which are stored in this class, with the capability to combine the smaller partitions into the one \"virtual\".\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframeVirtualPartition\n :members:\n\nPandasOnRayDataframeColumnPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframeColumnPartition\n :members:\n\nPandasOnRayDataframeRowPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframeRowPartition\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.rst",
"content": "PandasOnRayDataframePartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`,\nproviding the API to perform operations on a block partition, namely, ``pandas.DataFrame``, using Ray as an execution engine.\n\nIn addition to wrapping a ``pandas.DataFrame``, the class also holds the following metadata:\n\n* ``length`` - length of ``pandas.DataFrame`` wrapped\n* ``width`` - width of ``pandas.DataFrame`` wrapped\n* ``ip`` - node IP address that holds ``pandas.DataFrame`` wrapped\n\nAn operation on a block partition can be performed in two modes:\n\n* asynchronously_ - via :meth:`~modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartition.apply`\n* lazily_ - via :meth:`~modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartition.add_to_apply_calls`\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartition\n :members:\n\n.. _asynchronously: https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)\n.. _lazily: https://en.wikipedia.org/wiki/Lazy_evaluation\n"
},
{
"path": "docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.rst",
"content": "PandasOnRayDataframePartitionManager\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThis class is the specific implementation of :py:class:`~modin.core.execution.ray.generic.partitioning.GenericRayDataframePartitionManager`\nusing Ray distributed engine. This class is responsible for partition manipulation and applying a function to\nblock/row/column partitions.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartitionManager\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/unidist/generic.rst",
"content": ":orphan:\n\nGeneric Unidist-based members\n=============================\n\nObjects which are storage format agnostic but require specific Unidist implementation\nare placed in ``modin.core.execution.unidist.generic``.\n\nTheir purpose is to implement certain parallel I/O operations and to serve\nas a foundation for building storage format specific objects:\n\n.. autoclass:: modin.core.execution.unidist.generic.io.UnidistIO\n :members:\n\n.. autoclass:: modin.core.execution.unidist.generic.partitioning.GenericUnidistDataframePartitionManager\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/dataframe.rst",
"content": "PandasOnUnidistDataframe\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is specific implementation of :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`\nclass using Unidist distributed engine. It serves as an intermediate level between\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` and\n:py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartitionManager`.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.dataframe.PandasOnUnidistDataframe\n :members:"
},
{
"path": "docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/index.rst",
"content": ":orphan:\n\nPandasOnUnidist Execution\n=========================\n\nQueries that perform data transformation, data ingress or data egress using the `pandas on Unidist` execution\npass through the Modin components detailed below.\n\nTo enable `pandas on MPI through unidist` execution,\nplease refer to the usage section in :doc:`pandas on MPI through unidist `.\n\nData Transformation\n'''''''''''''''''''\n\n.. image:: /img/pandas_on_unidist_data_transform.svg\n :align: center\n\nWhen a user calls any :py:class:`~modin.pandas.dataframe.DataFrame` API, a query starts forming at the `API` layer\nto be executed at the `Execution` layer. The `API` layer is responsible for processing the query appropriately,\nfor example, determining whether the final result should be a ``DataFrame`` or ``Series`` object. This layer is also responsible for sanitizing the input to the\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler`, e.g. validating a parameter from the query\nand defining specific intermediate values to provide more context to the query compiler.\nThe :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for\nprocessing the query, received from the :py:class:`~modin.pandas.dataframe.DataFrame` `API` layer,\nto determine how to apply it to a subset of the data - either cell-wise or along an axis-wise partition backed by the `pandas`\nstorage format. The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` maps the query to one of the :doc:`Core Algebra Operators ` of\nthe :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.dataframe.PandasOnUnidistDataframe` which inherits\ngeneric functionality from the ``GenericUnidistDataframe`` and the :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`.\n\n..\n TODO: insert a link to ``GenericUnidistDataframe`` once we add an implementatiton of the class\n\nPandasOnUnidist Dataframe implementation\n----------------------------------------\n\nModin implements ``Dataframe``, ``PartitionManager``, ``VirtualPartition`` (a specific kind of ``AxisPartition`` with the capability\nto combine smaller partitions into the one \"virtual\") and ``Partition`` classes specifically for the ``PandasOnUnidist`` execution:\n\n* :doc:`PandasOnUnidistDataframe `\n* :doc:`PandasOnUnidistDataframePartition `\n* :doc:`PandasOnUnidistDataframeVirtualPartition `\n* :doc:`PandasOnUnidistDataframePartitionManager `\n\n.. toctree::\n :hidden:\n\n dataframe\n partitioning/partition\n partitioning/axis_partition\n partitioning/partition_manager\n\nData Ingress\n''''''''''''\n\n.. image:: /img/pandas_on_unidist_data_ingress.svg\n :align: center\n\nData Egress\n'''''''''''\n\n.. image:: /img/pandas_on_unidist_data_egress.svg\n :align: center\n\n\nWhen a user calls any IO function from the ``modin.pandas.io`` module, the `API` layer queries the\n:py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` which defines a factory specific for\nthe execution, namely, the :py:class:`~modin.core.execution.dispatching.factories.factories.PandasOnUnidistFactory`. The factory, in turn,\nexposes the :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.io.PandasOnUnidistIO` class\nwhose responsibility is to perform a parallel read/write from/to a file.\n\nWhen reading data from a CSV file, for example, the :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.io.PandasOnUnidistIO` class forwards\nthe user query to the :meth:`~modin.core.io.text.CSVDispatcher._read` method of :py:class:`~modin.core.io.text.CSVDispatcher`, where the query's parameters are preprocessed\nto check if they are supported by the execution (defaulting to pandas if they are not) and computes some metadata\ncommon for all partitions to be read. Then, the file is split into row chunks, and this data is used to launch remote tasks on the Unidist workers\nvia the :meth:`~modin.core.execution.unidist.common.UnidistWrapper.deploy` method of :py:class:`~modin.core.execution.unidist.common.UnidistWrapper`.\nOn each Unidist worker, the :py:class:`~modin.core.storage_formats.pandas.parsers.PandasCSVParser` parses data.\nAfter the remote tasks are finished, additional result postprocessing is performed,\nand a new query compiler with the data read is returned.\n\nWhen writing data to a CSV file, for example, the :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.io.PandasOnUnidistIO` processes\nthe user query to execute it on Unidist workers. Then, the :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.io.PandasOnUnidistIO` asks the\n:py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.dataframe.PandasOnUnidistDataframe` to decompose the data into row-wise partitions\nthat will be written into the file in parallel in Unidist workers."
},
{
"path": "docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/partitioning/axis_partition.rst",
"content": "PandasOnUnidistDataframeVirtualPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThis class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition`,\nproviding the API to perform operations on an axis partition, using Unidist as an execution engine. The virtual partition is\na wrapper over a list of block partitions, which are stored in this class, with the capability to combine the smaller partitions into the one \"virtual\".\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframeVirtualPartition\n :members:\n\nPandasOnUnidistDataframeColumnPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframeColumnPartition\n :members:\n\nPandasOnUnidistDataframeRowPartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframeRowPartition\n :members:\n"
},
{
"path": "docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/partitioning/partition.rst",
"content": "PandasOnUnidistDataframePartition\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`,\nproviding the API to perform operations on a block partition, namely, ``pandas.DataFrame``, using Unidist as an execution engine.\n\nIn addition to wrapping a ``pandas.DataFrame``, the class also holds the following metadata:\n\n* ``length`` - length of ``pandas.DataFrame`` wrapped\n* ``width`` - width of ``pandas.DataFrame`` wrapped\n* ``ip`` - node IP address that holds ``pandas.DataFrame`` wrapped\n\nAn operation on a block partition can be performed in two modes:\n\n* asynchronously_ - via :meth:`~modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartition.apply`\n* lazily_ - via :meth:`~modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartition.add_to_apply_calls`\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartition\n :members:\n\n.. _asynchronously: https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)\n.. _lazily: https://en.wikipedia.org/wiki/Lazy_evaluation\n"
},
{
"path": "docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/partitioning/partition_manager.rst",
"content": "PandasOnUnidistDataframePartitionManager\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThis class is the specific implementation of :py:class:`~modin.core.execution.unidist.generic.partitioning.GenericUnidistDataframePartitionManager`\nusing Unidist distributed engine. This class is responsible for partition manipulation and applying a function to\nblock/row/column partitions.\n\nPublic API\n----------\n\n.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartitionManager\n :members:\n"
},
{
"path": "docs/flow/modin/core/io/index.rst",
"content": ":orphan:\n\nIO Module Description\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nDispatcher Classes Workflow Overview\n''''''''''''''''''''''''''''''''''''\n\nCalls from ``read_*`` functions of execution-specific IO classes (for example, ``PandasOnRayIO`` for\nRay engine and pandas storage format) are forwarded to the ``_read`` function of the file\nformat-specific class (for example ``CSVDispatcher`` for CSV files), where function parameters are\npreprocessed to check if they are supported (defaulting to pandas if not)\nand common metadata is computed for all partitions. The file is then split\ninto chunks (splitting mechanism described below) and the data is used to launch tasks\non the remote workers. After the remote tasks finish, additional\npostprocessing is performed on the results, and a new query compiler with the imported data will\nbe returned.\n\nData File Splitting Mechanism\n'''''''''''''''''''''''''''''\n\nModin's file splitting mechanism differs depending on the data format type:\n\n* text format type - the file is split into bytes according to user specified arguments.\n In the simplest case, when no row related parameters (such as ``nrows`` or\n ``skiprows``) are passed, data chunk limits (start and end bytes) are derived\n by dividing the file size by the number of partitions (chunks can\n slightly differ between each other because usually end byte may occurs inside a\n line and in that case the last byte of the line should be used instead of initial\n value). In other cases the same splitting mechanism is used, but chunks sizes are\n defined according to the number of lines that each partition should contain.\n\n* columnar store type - the file is split so that each chunk contains approximately the same number of columns.\n\n* SQL type - chunking is obtained by wrapping initial SQL query with a query that\n specifies initial row offset and number of rows in the chunk.\n\nAfter file splitting is complete, chunks data is passed to the parser functions\n(``PandasCSVParser.parse`` for ``read_csv`` function with pandas storage format) for\nfurther processing on each worker.\n\nSubmodules Description\n''''''''''''''''''''''\n\n``modin.core.io`` module is used mostly for storing utils and dispatcher\nclasses for reading files of different formats.\n\n* ``io.py`` - class containing basic utils and default implementation of IO functions.\n\n* ``file_dispatcher.py`` - class reading data from different kinds of files and\n handling some util functions common for all formats. Also this class contains ``read``\n function which is entry point function for all dispatchers ``_read`` functions.\n\n* text - directory for storing all text file format dispatcher classes \n \n * ``text_file_dispatcher.py`` - class for reading text formats files. This class\n holds ``partitioned_file`` function for splitting text format files into chunks,\n ``offset`` function for moving file offset at the specified amount of bytes,\n ``_read_rows`` function for moving file offset at the specified amount of rows\n and many other functions.\n \n * format/feature specific dispatchers: ``csv_dispatcher.py``, ``excel_dispatcher.py``,\n ``fwf_dispatcher.py`` and ``json_dispatcher.py``.\n\n* column_stores - directory for storing all columnar store file format dispatcher classes\n \n * ``column_store_dispatcher.py`` - class for reading columnar type files. This class\n holds ``build_query_compiler`` function that performs file splitting, deploying remote\n tasks and results postprocessing and many other functions.\n \n * format/feature specific dispatchers: ``feather_dispatcher.py``, ``hdf_dispatcher.py``\n and ``parquet_dispatcher.py``.\n\n* sql - directory for storing SQL dispatcher class\n \n * ``sql_dispatcher.py`` - class for reading SQL queries or database tables.\n\nPublic API\n''''''''''\n\n.. automodule:: modin.core.io\n :members:\n\nHandling ``skiprows`` Parameter\n'''''''''''''''''''''''''''''''\n\nHandling ``skiprows`` parameter by pandas import functions can be very tricky, especially\nfor ``read_csv`` function because of interconnection with ``header`` parameter. In this section\nthe techniques of ``skiprows`` processing by both pandas and Modin are covered.\n\nProcessing ``skiprows`` by pandas\n=================================\n\nLet's consider a simple snippet with ``pandas.read_csv`` in order to understand interconnection\nof ``header`` and ``skiprows`` parameters:\n\n.. code-block:: python\n\n import pandas\n from io import StringIO\n\n data = \"\"\"0\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n \"\"\"\n\n # `header` parameter absence is equivalent to `header=\"infer\"` or `header=0`\n # rows 1, 5, 6, 7, 8 are read with header \"0\"\n df = pandas.read_csv(StringIO(data), skiprows=[2, 3, 4])\n # rows 5, 6, 7, 8 are read with header \"1\", row 0 is skipped additionally\n df = pandas.read_csv(StringIO(data), skiprows=[2, 3, 4], header=1)\n # rows 6, 7, 8 are read with header \"5\", rows 0, 1 are skipped additionally\n df = pandas.read_csv(StringIO(data), skiprows=[2, 3, 4], header=2)\n\nIn the examples above list-like ``skiprows`` values are fixed and ``header`` is varied. In the first\nexample with no ``header`` provided, rows 2, 3, 4 are skipped and row 0 is considered as the header.\nIn the second example ``header == 1``, so the zeroth row is skipped and the next available row is\nconsidered the header. The third example illustrates when the ``header`` and ``skiprows`` parameters\nvalues are both present - in this case ``skiprows`` rows are dropped first and then the ``header`` is derived\nfrom the remaining rows (rows before header are skipped too).\n\nIn the examples above only list-like ``skiprows`` and integer ``header`` parameters are considered,\nbut the same logic is applicable for other types of the parameters.\n\nProcessing ``skiprows`` by Modin\n================================\n\nAs it can be seen, skipping rows in the pandas import functions is complicated and distributing\nthis logic across multiple workers can complicate it even more. Thus in some rare corner cases\ndefault pandas implementation is used in Modin to avoid excessive Modin code complication.\n\nModin uses two techniques for skipping rows:\n\n1) During file partitioning (setting file limits that should be read by each partition)\nexact rows can be excluded from partitioning scope, thus they won't be read at all and can be\nconsidered as skipped. This is the most effective way of skipping rows since it doesn't require\nany actual data reading and postprocessing, but in this case ``skiprows`` parameter can be an\ninteger only. When it is possible Modin always uses this approach.\n\n2) Rows for skipping can be dropped after full dataset import. This is more expensive way since\nit requires extra IO work and postprocessing afterwards, but ``skiprows`` parameter can be of any\nnon-integer type supported by ``pandas.read_csv``.\n\nIn some cases, if ``skiprows`` is uniformly distributed array (e.g. [1, 2, 3]), ``skiprows`` can be\n\"squashed\" and represented as an integer to make a fastpath by skipping these rows during file partitioning\n(using the first option). But if there is a gap between the first row for skipping\nand the last line of the header (that will be skipped too since header is read by each partition\nto ensure metadata is defined properly), then this gap should be assigned for reading first\nby assigning the first partition to read these rows by setting ``pre_reading`` parameter.\n\nLet's consider an example of skipping rows during partitioning when ``header=\"infer\"`` and\n``skiprows=[3, 4, 5]``. In this specific case fastpath can be done since ``skiprows`` is uniformly\ndistributed array, so we can \"squash\" it to an integer and set \"partitioning\" skiprows to 3. But\nif no additional action is done, these three rows will be skipped right after header line,\nthat corresponds to ``skiprows=[1, 2, 3]``. To avoid this discrepancy, we need to assign the first\npartition to read data between header line and the first row for skipping by setting special\n``pre_reading`` parameter to 2. Then, after the skipping of rows considered to be skipped during\npartitioning, the rest data will be divided between the rest of partitions, see rows assignment\nbelow:\n\n.. code-block::\n\n 0 - header line (skip during partitioning)\n 1 - pre reading (assign to read by the first partition)\n 2 - pre reading (assign to read by the first partition)\n 3 - \"partitioning\" skiprows (skip during partitioning)\n 4 - \"partitioning\" skiprows (skip during partitioning)\n 5 - \"partitioning\" skiprows (skip during partitioning)\n 6 - data to partition (divide between the rest of partitions)\n 7 - data to partition (divide between the rest of partitions)\n"
},
{
"path": "docs/flow/modin/core/storage_formats/base/query_compiler.rst",
"content": "BaseQueryCompiler\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nBrief description\n'''''''''''''''''\n:py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler` is an abstract class of query compiler, and sets a common interface\nthat every other query compiler implementation in Modin must follow. The Base class contains a basic\nimplementations for most of the interface methods, all of which\n:doc:`fallback to pandas `.\n\nSubclassing :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler`\n'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''\nIf you want to add new type of query compiler to Modin the new class needs to inherit\nfrom :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler` and implement the abstract methods:\n\n- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.from_pandas` build query compiler from pandas DataFrame.\n- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.from_arrow` build query compiler from Arrow Table.\n- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.to_pandas` get query compiler representation as pandas DataFrame.\n- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.default_to_pandas` do :doc:`fallback to pandas ` for the passed function. \n- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.finalize` finalize object constructing.\n- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.free` trigger memory cleaning.\n\n(Please refer to the code documentation to see the full documentation for these functions).\n\nThis is a minimum set of operations to ensure a new query compiler will function in the Modin architecture,\nand the rest of the API can safely default to the pandas implementation via the base class implementation. \nTo add a storage format specific implementation for some of the query compiler operations, just override \nthe corresponding method in your query compiler class.\n\nExample\n'''''''\nAs an exercise let's define a new query compiler in `Modin`, just to see how easy it is.\nUsually, the query compiler routes formed queries to the underlying :doc:`frame ` class,\nwhich submits operators to an execution engine. For the sake\nof simplicity and independence of this example, our execution engine will be the pandas itself.\n\nWe need to inherit a new class from :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler` and implement all of the abstract methods.\nIn this case, with `pandas` as an execution engine, it's trivial:\n\n.. code-block:: python\n\n from modin.core.storage_formats import BaseQueryCompiler\n\n class DefaultToPandasQueryCompiler(BaseQueryCompiler):\n def __init__(self, pandas_df):\n self._pandas_df = pandas_df\n\n @classmethod\n def from_pandas(cls, df, *args, **kwargs):\n return cls(df)\n\n @classmethod\n def from_arrow(cls, at, *args, **kwargs):\n return cls(at.to_pandas())\n\n def to_pandas(self):\n return self._pandas_df.copy()\n\n def default_to_pandas(self, pandas_op, *args, **kwargs):\n return type(self)(pandas_op(self.to_pandas(), *args, **kwargs))\n \n def finalize(self):\n pass\n\n def free(self):\n pass\n\nAll done! Now you've got a fully functional query compiler, which is ready for extensions\nand already can be used in Modin DataFrame:\n\n.. code-block:: python\n\n import pandas\n pandas_df = pandas.DataFrame({\"col1\": [1, 2, 2, 1], \"col2\": [10, 2, 3, 40]})\n # Building our query compiler from pandas object\n qc = DefaultToPandasQueryCompiler.from_pandas(pandas_df)\n\n import modin.pandas as pd\n # Building Modin DataFrame from newly created query compiler\n modin_df = pd.DataFrame(query_compiler=qc)\n\n # Got fully functional Modin DataFrame\n >>> print(modin_df.groupby(\"col1\").sum().reset_index())\n col1 col2\n 0 1 50\n 1 2 5\n\nTo be able to select this query compiler as default via ``modin.config`` you also need\nto define the combination of your query compiler and pandas engine as an execution\nby adding the corresponding factory. To find more information about factories,\nvisit :doc:`dispatching ` page.\n\nQuery Compiler API\n''''''''''''''''''\n\n.. autoclass:: modin.core.storage_formats.base.query_compiler.BaseQueryCompiler\n :members:\n"
},
{
"path": "docs/flow/modin/core/storage_formats/index.rst",
"content": ":orphan:\n\nStorage Formats\n===============\nStorage format is one of the components that form Modin's execution, it describes the type(s)\nof objects that are stored in the partitions of the selected Core Modin Dataframe implementation.\n\nThe base storage format in Modin is pandas. In that format, Modin Dataframe operates with\npartitions that hold ``pandas.DataFrame`` objects. Pandas is the most natural storage format\nsince high-level DataFrame objects mirror its API.\n\nThe storage format + execution engine (Ray, Dask, etc.) form the execution backend. \nThe Query Compiler (QC) converts high-level pandas API calls to queries that are understood \nby the execution backend.\n\n.. _query_compiler_def:\n\nQuery Compiler\n==============\n\n.. toctree::\n :hidden:\n\n base/query_compiler\n pandas/index\n\nModin supports several execution backends (storage format + execution engine). Calling any\nDataFrame API function will end up in some execution-specific method. The query compiler is\na bridge between pandas DataFrame API and the actual Core Modin Dataframe implementation for the\ncorresponding execution.\n\n.. image:: /img/simplified_query_flow.svg\n :align: right\n :width: 300px\n\nEach storage format has its own Query Compiler class that implements the most optimal\nquery routing for the selected format.\n\nQuery compilers of all storage formats implement a common API, which is used by the high-level Modin DataFrame\nto support dataframe queries. The role of the query compiler is to translate its API into\na pairing of known user-defined functions and dataframe algebra operators. Each query compiler instance contains a\n:doc:`Core Modin Dataframe ` of the selected execution implementation and queries\nit with the compiled queries to get the result. The query compiler object is immutable,\nso the result of every method is a new query compiler.\n\nThe query compilers API is defined by the :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler` class\nand may resemble the pandas API, however, they're not equal. The query compilers API\nis significantly reduced in comparison with pandas, since many corner cases or even the\nwhole methods can be handled at the API layer with the existing API.\n\nThe query compiler is the level where Modin stops distinguishing DataFrame and Series (or column) objects.\nA Series is represented by a `1xN` query compiler, where the Series name is the column label.\nIf Series is unnamed, then the label is ``MODIN_UNNAMED_SERIES_LABEL``, which is equal to ``\"__reduced__\"``. The high-level DataFrame API layer\ninterprets a one-column query compiler as Series or DataFrame depending on the operation context.\n\n.. note::\n Although we're declaring that there is no difference between DataFrame and Series at the query compiler,\n you still may find methods like ``method_ser`` and ``method_df`` which are implemented differently because they're\n emulating either Series or DataFrame logic, or you may find parameters, which indicates whether this one-column\n query compiler is representing Series or not. All of these are hacks, and we're working on getting rid of them.\n\nHigh-level module overview\n''''''''''''''''''''''''''\n\nThis module houses submodules of all of the stable storage formats:\n\n- :doc:`Base module ` contains an abstract query compiler class which defines common API.\n- :doc:`Pandas module ` contains query compiler and text parsers for pandas storage format.\n"
},
{
"path": "docs/flow/modin/core/storage_formats/pandas/index.rst",
"content": ":orphan:\n\nPandas storage format\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n.. toctree::\n :hidden:\n\n query_compiler\n parsers\n\nHigh-Level Module Overview\n''''''''''''''''''''''''''\nThis module houses submodules which are responsible for communication between\nthe query compiler level and execution implementation level for pandas storage format:\n\n- :doc:`Query compiler ` is responsible for compiling efficient queries for :doc:`PandasDataframe `.\n- :doc:`Parsers ` are responsible for parsing data on workers during IO operations.\n"
},
{
"path": "docs/flow/modin/core/storage_formats/pandas/parsers.rst",
"content": "Pandas Parsers Module Description\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\nHigh-Level Module Overview\n''''''''''''''''''''''''''\n\nThis module houses parser classes (classes that are used for data parsing on the workers)\nand util functions for handling parsing results. ``PandasParser`` is base class for parser\nclasses with pandas storage format, that contains methods common for all child classes. Other\nmodule classes implement ``parse`` function that performs parsing of specific format data\nbasing on the chunk information computed in the ``modin.core.io`` module. After\nthe chunk is parsed, the resulting ``DataFrame``-s will be split into smaller\n``DataFrame``-s according to the ``num_splits`` parameter, data type, or number of\nrows/columns in the parsed chunk. These frames, along with some additional metadata, are then returned.\n\n.. note:: \n If you are interested in the data parsing mechanism implementation details, please refer\n to the source code documentation.\n\nPublic API\n''''''''''\n\n.. automodule:: modin.core.storage_formats.pandas.parsers\n :members:\n"
},
{
"path": "docs/flow/modin/core/storage_formats/pandas/query_compiler.rst",
"content": "PandasQueryCompiler\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for compiling\na set of known predefined functions and pairing those with dataframe algebra operators in the\n:doc:`PandasDataframe `, specifically for dataframes backed by\n``pandas.DataFrame`` objects.\n\nEach :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` contains an instance of\n:py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` which it queries to get the result.\n\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` supports methods built by \nthe :doc:`algebra module `.\nIf you want to add an implementation for a query compiler method, visit the algebra module documentation\nto see whether the new operation fits one of the existing function templates and can be easily implemented\nwith them.\n\nPublic API\n''''''''''\n:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` implements common query compilers API\ndefined by the :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler`. Some functionalities\nare inherited from the base class, in the following section only overridden methods are presented.\n\n.. autoclass:: modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler\n :members:\n"
},
{
"path": "docs/flow/modin/distributed/dataframe/pandas.rst",
"content": "Pandas partitioning API\n=======================\n\nThis page contains a description of the API to extract partitions from and build Modin Dataframes.\n\nunwrap_partitions\n-----------------\n\n.. autofunction:: modin.distributed.dataframe.pandas.unwrap_partitions\n\nfrom_partitions\n---------------\n.. autofunction:: modin.distributed.dataframe.pandas.from_partitions\n\nExample\n-------\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.distributed.dataframe.pandas import unwrap_partitions, from_partitions\n import numpy as np\n data = np.random.randint(0, 100, size=(2 ** 10, 2 ** 8))\n df = pd.DataFrame(data)\n partitions = unwrap_partitions(df, axis=0, get_ip=True)\n print(partitions)\n new_df = from_partitions(partitions, axis=0)\n print(new_df)\n"
},
{
"path": "docs/flow/modin/experimental/batch.rst",
"content": "Batch Pipeline API \n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThis API exposes the ability to pipeline row-parallel batch queries on a Modin DataFrame. Currently,\nthis feature is only supported for the ``PandasOnRay`` execution.\n\nAPI\n'''\n\n.. automodule:: modin.experimental.batch.pipeline\n :members:\n\n"
},
{
"path": "docs/flow/modin/experimental/core/io/index.rst",
"content": ":orphan:\n\nExperimental IO Module Description\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe module is used mostly for storing experimental utils and\ndispatcher classes for reading/writing files of different formats.\n\nSubmodules Description\n''''''''''''''''''''''\n\n* text - directory for storing all text file format dispatcher classes\n\n * format/feature specific dispatchers: ``csv_glob_dispatcher.py``,\n ``custom_text_dispatcher.py``.\n\n* sql - directory for storing SQL dispatcher class\n\n * format/feature specific dispatchers: ``sql_dispatcher.py``\n\n* pickle - directory for storing Pickle dispatcher class\n\n * format/feature specific dispatchers: ``pickle_dispatcher.py``\n\nPublic API\n''''''''''\n\n.. automodule:: modin.experimental.core.io\n :members:\n"
},
{
"path": "docs/flow/modin/experimental/index.rst",
"content": ":orphan:\n\nExperimental Modules Overview\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nIn some cases Modin can give the user the opportunity to extend (not modify) typical pandas\nAPI or to try new functionality in order to get more flexibility. Depending on the exact\nexperimental feature user may need to install additional packages, change configurations or\nreplace the standard Modin import statement ``import modin.pandas as pd`` with modified version\n``import modin.experimental.pandas as pd``.\n\n``modin.experimental`` holds experimental functionality that is under development right now\nand provides a limited set of functionality:\n\n* :doc:`xgboost `\n* :doc:`sklearn `\n* :doc:`batch `\n\n\n.. toctree::\n :hidden:\n\n sklearn\n xgboost\n batch\n"
},
{
"path": "docs/flow/modin/experimental/pandas.rst",
"content": ":orphan:\n\nExperimental Pandas API\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n.. automodule:: modin.experimental.pandas\n :noindex:\n\nExperimental API Reference\n''''''''''''''''''''''''''\n\n.. autofunction:: read_sql\n.. autofunction:: read_csv_glob\n.. autofunction:: read_custom_text\n.. autofunction:: read_pickle_glob\n.. autofunction:: read_parquet_glob\n.. autofunction:: read_json_glob\n.. autofunction:: read_xml_glob\n.. automethod:: modin.pandas.DataFrame.modin::to_pandas\n.. automethod:: modin.pandas.DataFrame.modin::to_ray\n.. automethod:: modin.pandas.DataFrame.modin::to_pickle_glob\n.. automethod:: modin.pandas.DataFrame.modin::to_parquet_glob\n.. automethod:: modin.pandas.DataFrame.modin::to_json_glob\n.. automethod:: modin.pandas.DataFrame.modin::to_xml_glob\n"
},
{
"path": "docs/flow/modin/experimental/range_partitioning_groupby.rst",
"content": ":orphan:\n\n.. redirect to the new page\n.. raw:: html\n\n "
},
{
"path": "docs/flow/modin/experimental/reshuffling_groupby.rst",
"content": ":orphan:\n\n.. redirect to the new page\n.. raw:: html\n\n \n"
},
{
"path": "docs/flow/modin/experimental/sklearn.rst",
"content": "Scikit-learn module description\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThis module holds experimental scikit-learn-specific functionality for Modin.\n\nAPI\n'''\n.. automodule:: modin.experimental.sklearn.model_selection\n :members:\n\n"
},
{
"path": "docs/flow/modin/experimental/xgboost.rst",
"content": "Modin XGBoost module description\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\nHigh-level Module Overview\n''''''''''''''''''''''''''\n\nThis module holds classes, public interface and internal functions for distributed XGBoost in Modin.\n\nPublic classes :py:class:`~modin.experimental.xgboost.Booster`, :py:class:`~modin.experimental.xgboost.DMatrix`\nand function :py:func:`~modin.experimental.xgboost.train` provide the user with familiar XGBoost interfaces.\nThey are located in the ``modin.experimental.xgboost.xgboost`` module.\n\nThe internal module ``modin.experimental.xgboost.xgboost.xgboost_ray`` contains the implementation of Modin XGBoost\nfor the Ray execution engine. This module mainly consists of the Ray actor-class :py:class:`~modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor`,\na function to distribute Modin's partitions between actors :py:func:`~modin.experimental.xgboost.xgboost_ray._assign_row_partitions_to_actors`,\nan internal :py:func:`~modin.experimental.xgboost.xgboost_ray._train`/:py:func:`~modin.experimental.xgboost.xgboost_ray._predict`\nfunction used from the public interfaces and additional util functions for computing cluster resources, actor creations etc.\n\nPublic interfaces\n'''''''''''''''''\n\n:py:class:`~modin.experimental.xgboost.DMatrix` inherits original class ``xgboost.DMatrix`` and overrides\nits constructor, which currently supports only `data` and `label` parameters. Both of the parameters must\nbe ``modin.pandas.DataFrame``, which will be internally unwrapped to lists of delayed objects of Modin's\nrow partitions using the function :py:func:`~modin.distributed.dataframe.pandas.unwrap_partitions`.\n\n.. autoclass:: modin.experimental.xgboost.DMatrix\n :members:\n\n:py:class:`~modin.experimental.xgboost.Booster` inherits original class ``xgboost.Booster`` and\noverrides method ``predict``. The difference from original class interface for ``predict``\nmethod is changing the type of the `data` parameter to :py:class:`~modin.experimental.xgboost.DMatrix`.\n\n.. autoclass:: modin.experimental.xgboost.Booster\n :members:\n\n:py:func:`~modin.experimental.xgboost.train` function has 2 differences from the original ``train`` function - (1) the\ndata type of `dtrain` parameter is :py:class:`~modin.experimental.xgboost.DMatrix` and (2) a new parameter `num_actors`.\n\n.. autofunction:: modin.experimental.xgboost.train\n\nInternal execution flow on Ray engine\n'''''''''''''''''''''''''''''''''''''\n\nInternal functions :py:func:`~modin.experimental.xgboost.xgboost_ray._train` and\n:py:func:`~modin.experimental.xgboost.xgboost_ray._predict` work similar to xgboost.\n\n\nTraining\n********\n\n1. The data is passed to the :py:func:`~modin.experimental.xgboost.xgboost_ray._train`\n function as a :py:class:`~modin.experimental.xgboost.DMatrix` object. Lists of ``ray.ObjectRef``\n corresponding to row partitions of Modin DataFrames are extracted by iterating over the \n :py:class:`~modin.experimental.xgboost.DMatrix`. Example:\n\n .. code-block:: python\n\n # Extract lists of row partitions from dtrain (DMatrix object)\n X_row_parts, y_row_parts = dtrain\n ..\n\n2. On this step, the parameter `num_actors` is processed. The internal function :py:func:`~modin.experimental.xgboost.xgboost_ray._get_num_actors`\n examines the value provided by the user. In case the value isn't provided, the `num_actors` will be computed using condition that 1 actor should use maximum 2 CPUs.\n This condition was chosen for using maximum parallel workers with multithreaded XGBoost training (2 threads\n per worker will be used in this case).\n\n.. note:: `num_actors` parameter is made available for public function :py:func:`~modin.experimental.xgboost.train` to allow\n fine-tuning for obtaining the best performance in specific use cases.\n\n3. :py:class:`~modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor` objects are created.\n\n4. Data `dtrain` is split between actors evenly. The internal function\n :py:func:`~modin.experimental.xgboost.xgboost_ray._split_data_across_actors` runs assigning row partitions to actors\n using internal function :py:func:`~modin.experimental.xgboost.xgboost_ray._assign_row_partitions_to_actors`.\n This function creates a dictionary in the form: `{actor_rank: ([part_i0, part_i3, ..], [0, 3, ..]), ..}`.\n\n.. note:: :py:func:`~modin.experimental.xgboost.xgboost_ray._assign_row_partitions_to_actors` takes into account IP\n addresses of row partitions of `dtrain` data to minimize excess data transfer.\n\n5. For each :py:class:`~modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor` object ``set_train_data`` method is\n called remotely. This method runs loading row partitions in actor according to the dictionary with partitions\n distribution from previous step. When data is passed to the actor, the row partitions are automatically materialized\n (``ray.ObjectRef`` -> ``pandas.DataFrame``).\n\n6. ``train`` method of :py:class:`~modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor` class object is called remotely. This method\n runs XGBoost training on local data of actor, connects to ``Rabit Tracker`` for sharing training state between\n actors and returns dictionary with `booster` and `evaluation results`.\n\n7. At the final stage results from actors are returned. `booster` and `evals_result` are returned using ``ray.get``\n function from remote actor.\n\n\nPrediction\n**********\n\n1. The data is passed to :py:func:`~modin.experimental.xgboost.xgboost_ray._predict`\n function as a :py:class:`~modin.experimental.xgboost.DMatrix` object.\n\n2. :py:func:`~modin.experimental.xgboost.xgboost_ray._map_predict` function is applied remotely for each partition\n of the data to make a partial prediction.\n\n3. Result ``modin.pandas.DataFrame`` is created from ``ray.ObjectRef`` objects, obtained in the previous step.\n\n\nInternal API\n''''''''''''\n.. autoclass:: modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor\n :members:\n :private-members:\n\n.. autofunction:: modin.experimental.xgboost.xgboost_ray._assign_row_partitions_to_actors\n.. autofunction:: modin.experimental.xgboost.xgboost_ray._train\n.. autofunction:: modin.experimental.xgboost.xgboost_ray._predict\n.. autofunction:: modin.experimental.xgboost.xgboost_ray._get_num_actors\n.. autofunction:: modin.experimental.xgboost.xgboost_ray._split_data_across_actors\n.. autofunction:: modin.experimental.xgboost.xgboost_ray._map_predict\n"
},
{
"path": "docs/flow/modin/pandas/base.rst",
"content": "Base pandas Dataset API\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe class implements functionality that is common to Modin's pandas API for both ``DataFrame`` and ``Series`` classes.\n\nPublic API\n----------\n\n.. autoclass:: modin.pandas.base.BasePandasDataset\n :noindex:\n :members:\n"
},
{
"path": "docs/flow/modin/pandas/dataframe.rst",
"content": ":orphan:\n\nDataFrame Module Overview\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin's ``pandas.DataFrame`` API\n''''''''''''''''''''''''''''''''\n\nModin's ``pandas.DataFrame`` API is backed by a distributed object providing an identical\nAPI to pandas. After the user calls some ``DataFrame`` function, this call is internally\nrewritten into a representation that can be processed in parallel by the partitions. These\nresults can be e.g., reduced to single output, identical to the single threaded\npandas ``DataFrame`` method output.\n\n..\n TODO: add link to the docs with detailed description of queries compilation\n and execution ater DOCS-#2996 is merged.\n\nPublic API\n----------\n\n.. autoclass:: modin.pandas.dataframe.DataFrame\n\nUsage Guide\n'''''''''''\n\nThe most efficient way to create Modin ``DataFrame`` is to import data from external\nstorage using the highly efficient Modin IO methods (for example using ``pd.read_csv``,\nsee details for Modin IO methods in the :doc:`IO ` page),\nbut even if the data does not originate from a file, any pandas supported data type or\n``pandas.DataFrame`` can be used. Internally, the ``DataFrame`` data is divided into\npartitions, which number along an axis usually corresponds to the number of the user's hardware CPUs. If needed,\nthe number of partitions can be changed by setting ``modin.config.NPartitions``.\n\nLet's consider simple example of creation and interacting with Modin ``DataFrame``:\n\n.. code-block:: python\n\n import modin.config\n\n # This explicitly sets the number of partitions\n modin.config.NPartitions.put(4)\n\n import modin.pandas as pd\n import pandas\n\n # Create Modin DataFrame from the external file\n pd_dataframe = pd.read_csv(\"test_data.csv\")\n # Create Modin DataFrame from the python object\n # data = {f'col{x}': [f'col{x}_{y}' for y in range(100, 356)] for x in range(4)}\n # pd_dataframe = pd.DataFrame(data)\n # Create Modin DataFrame from the pandas object\n # pd_dataframe = pd.DataFrame(pandas.DataFrame(data))\n\n # Show created DataFrame\n print(pd_dataframe)\n\n # List DataFrame partitions. Note, that internal API is intended for\n # developers needs and was used here for presentation purposes\n # only.\n partitions = pd_dataframe._query_compiler._modin_frame._partitions\n print(partitions)\n\n # Show the first DataFrame partition\n print(partitions[0][0].get())\n\n Output:\n\n # created DataFrame\n\n col0 col1 col2 col3\n 0 col0_100 col1_100 col2_100 col3_100\n 1 col0_101 col1_101 col2_101 col3_101\n 2 col0_102 col1_102 col2_102 col3_102\n 3 col0_103 col1_103 col2_103 col3_103\n 4 col0_104 col1_104 col2_104 col3_104\n .. ... ... ... ...\n 251 col0_351 col1_351 col2_351 col3_351\n 252 col0_352 col1_352 col2_352 col3_352\n 253 col0_353 col1_353 col2_353 col3_353\n 254 col0_354 col1_354 col2_354 col3_354\n 255 col0_355 col1_355 col2_355 col3_355\n\n [256 rows x 4 columns]\n\n # List of DataFrame partitions\n\n [[]\n []\n []\n []]\n\n # The first DataFrame partition\n \n col0 col1 col2 col3\n 0 col0_100 col1_100 col2_100 col3_100\n 1 col0_101 col1_101 col2_101 col3_101\n 2 col0_102 col1_102 col2_102 col3_102\n 3 col0_103 col1_103 col2_103 col3_103\n 4 col0_104 col1_104 col2_104 col3_104\n .. ... ... ... ...\n 60 col0_160 col1_160 col2_160 col3_160\n 61 col0_161 col1_161 col2_161 col3_161\n 62 col0_162 col1_162 col2_162 col3_162\n 63 col0_163 col1_163 col2_163 col3_163\n 64 col0_164 col1_164 col2_164 col3_164\n\n [65 rows x 4 columns]\n\nAs we show in the example above, Modin ``DataFrame`` can be easily created, and supports any input that pandas ``DataFrame`` supports.\nAlso note that tuning of the ``DataFrame`` partitioning can be done by just setting a single config.\n"
},
{
"path": "docs/flow/modin/pandas/series.rst",
"content": ":orphan:\n\nSeries Module Overview\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin's ``pandas.Series`` API\n'''''''''''''''''''''''''''''\n\nModin's ``pandas.Series`` API is backed by a distributed object providing an identical\nAPI to pandas. After the user calls some ``Series`` function, this call is internally rewritten\ninto a representation that can be processed in parallel by the partitions. These\nresults can be e.g., reduced to single output, identical to the single threaded\npandas ``Series`` method output.\n\n..\n TODO: add link to the docs with detailed description of queries compilation\n and execution ater DOCS-#2996 is merged.\n\nPublic API\n----------\n\n.. autoclass:: modin.pandas.series.Series\n\nUsage Guide\n'''''''''''\n\nThe most efficient way to create Modin ``Series`` is to import data from external\nstorage using the highly efficient Modin IO methods (for example using ``pd.read_csv``,\nsee details for Modin IO methods in the :doc:`IO ` page),\nbut even if the data does not originate from a file, any pandas supported data type or\n``pandas.Series`` can be used. Internally, the ``Series`` data is divided into\npartitions, which number along an axis usually corresponds to the number of the user's hardware CPUs. If needed,\nthe number of partitions can be changed by setting ``modin.config.NPartitions``.\n\nLet's consider simple example of creation and interacting with Modin ``Series``:\n\n.. code-block:: python\n\n import modin.config\n\n # This explicitly sets the number of partitions\n modin.config.NPartitions.put(4)\n\n import modin.pandas as pd\n import pandas\n\n # Create Modin Series from the external file\n pd_series = pd.read_csv(\"test_data.csv\", header=None).squeeze()\n # Create Modin Series from the python object\n # pd_series = pd.Series([x for x in range(256)])\n # Create Modin Series from the pandas object\n # pd_series = pd.Series(pandas.Series([x for x in range(256)]))\n\n # Show created `Series`\n print(pd_series)\n\n # List `Series` partitions. Note, that internal API is intended for\n # developers needs and was used here for presentation purposes\n # only.\n partitions = pd_series._query_compiler._modin_frame._partitions\n print(partitions)\n\n # Show the first `Series` partition\n print(partitions[0][0].get())\n\n Output:\n\n # created `Series`\n\n 0 100\n 1 101\n 2 102\n 3 103\n 4 104\n ...\n 251 351\n 252 352\n 253 353\n 254 354\n 255 355\n Name: 0, Length: 256, dtype: int64\n\n # List of `Series` partitions\n\n [[]\n []\n []\n []]\n\n # The first `Series` partition\n \n 0\n 0 100\n 1 101\n 2 102\n 3 103\n 4 104\n .. ...\n 60 160\n 61 161\n 62 162\n 63 163\n 64 164\n\n [65 rows x 1 columns]\n\nAs we show in the example above, Modin ``Series`` can be easily created, and supports any input that pandas ``Series`` supports.\nAlso note that tuning of the ``Series`` partitioning can be done by just setting a single config.\n"
},
{
"path": "docs/flow/modin/utils.rst",
"content": ":orphan:\n\nModin Utils\n\"\"\"\"\"\"\"\"\"\"\"\n\nHere are utilities that can be useful when working with Modin.\n\nPublic API\n''''''''''\n\n.. autofunction:: modin.utils.try_cast_to_pandas\n.. autofunction:: modin.utils.execute\n"
},
{
"path": "docs/getting_started/examples.rst",
"content": "Examples and Resources\n======================\n\nHere you can find additional resources to learn about Modin. To learn more about \nadvanced usage for Modin, please refer to :doc:`Usage Guide ` section..\n\nUsage Examples\n''''''''''''''\n\nThe following notebooks demonstrate how Modin can be used for scalable data science:\n\n- Quickstart Guide to Modin [`Source `__]\n- Using Modin with the NYC Taxi Dataset [`Source `__]\n- Modin for Machine Learning with scikit-learn [`Source `__]\n\nTutorials\n'''''''''\n\nThe following tutorials cover the basic usage of Modin. `Here `__ is a one hour video tutorial that walks through these basic exercises.\n\n- Exercise 1: Introduction to Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]\n- Exercise 2: Speed Improvements with Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]\n- Exercise 3: Defaulting to pandas with Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]\n\nThe following tutorials covers more advanced features in Modin:\n\n- Exercise 4: Experimental Features in Modin (Spreadsheet, Progress Bar) [`Source PandasOnRay `__, `Source PandasOnDask `__]\n- Exercise 5: Setting up Modin in a Cluster Environment [`Source PandasOnRay `__]\n- Exercise 6: Running Modin in a Cluster Environment [`Source PandasOnRay `__]\n\nHow to get required dependencies for the tutorial notebooks and to run them please refer to the respective `README.md `__ file.\n\nTalks & Podcasts\n''''''''''''''''\n\n- `Scaling Interactive Data Science with Modin and Ray `_ (20 minute, Ray Summit 2021)\n- `Unleash The Power Of Dataframes At Any Scale With Modin `_ (40 minute, Python Podcast 2021)\n- `[Russian] Distributed Data Processing and XGBoost Training and Prediction with Modin `_ (30 minute, PyCon Russia 2021)\n- `[Russian] Efficient Data Science with Modin `_ (30 minute, ISP RAS Open 2021)\n- `Modin: Scaling the Capabilities of the Data Scientist, not the Machine `_ (1 hour, RISE Camp 2020)\n- `Modin: Pandas Scalability with Devin Petersohn `_ (1 hour, Software Engineering Daily Podcast 2020)\n- `Introduction to the DataFrame and Modin `_ (20 minute, RISECamp 2019)\n- `Scaling Interactive Pandas Workflows with Modin `_ (40 minute, PyData NYC 2018)\n\nCommunity contributions\n'''''''''''''''''''''''\n\nHere are some blogposts and articles about Modin:\n\n- `Anaconda Blog: Scale your pandas workflow with Modin by Vasilij Litvinov `_\n- `The Modin view of Scaling Pandas by Devin Petersohn `_\n- `Data Science at Scale with Modin by Areg Melik-Adamyan `_\n- `Speed up Pandas using Modin by Eric D. Brown, D.Sc. `_\n- `Explore Python Libraries: Make Your DataFrames Parallel With Modin by Zachary Bennett `_\n- `Get faster pandas with Modin, even on your laptops by Parul Pandey `_\n- `How to speedup pandas by changing one line of code by Shrivarsheni `_\n- `How To Accelerate Pandas With Just One Line Of Code by Analytics India `_\n- `An Easy Introduction to Modin: A Step-by-Step Guide to Accelerating Pandas by Intel `_\n\n\nHere are some articles contributed by the international community:\n\n- `[Chinese] 用 Modin 来提速 pandas 工作流程 by Python Chinese Community `_\n- `[German] Was ist Modin? by Dipl.-Ing. (FH) Stefan Luber `_\n- `[Russian] Ускоряем Pandas при помощи модуля modin by Разработка `_\n- `[Korean] modin 으로 pandas 더 빠르게 사용하기 by 분석뉴비 `_\n\nIf you would like your articles to be featured here, please `submit a pull request `_ to let us know!\n"
},
{
"path": "docs/getting_started/faq.rst",
"content": "Frequently Asked Questions (FAQs)\n=================================\n\nBelow, you will find answers to the most commonly asked questions about\nModin. If you still cannot find the answer you are looking for, please post your\nquestion on the #support channel on our Slack_ community or open a Github issue_.\n\nFAQs: Why choose Modin?\n-----------------------\n\nWhat’s wrong with pandas and why should I use Modin?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nWhile pandas works extremely well on small datasets, as soon as you start working with\nmedium to large datasets that are more than a few GBs, pandas can become painfully\nslow or run out of memory. This is because pandas is single-threaded. In other words,\nyou can only process your data with one core at a time. This approach does not scale to\nlarger data sets and adding more hardware does not lead to more performance gain.\n\nThe :py:class:`~modin.pandas.dataframe.DataFrame` is a highly\nscalable, parallel DataFrame. Modin transparently distributes the data and computation so\nthat you can continue using the same pandas API while being able to work with more data faster.\nModin lets you use all the CPU cores on your machine, and because it is lightweight, it\noften has less memory overhead than pandas. See :doc:` Why Modin? `\npage to learn more about how Modin is different from pandas.\n\nWhy not just improve pandas?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\npandas is a massive community and well established codebase. Many of the issues\nwe have identified and resolved with pandas are fundamental to its current\nimplementation. While we would be happy to donate parts of Modin that\nmake sense in pandas, many of these components would require significant (or\ntotal) redesign of the pandas architecture. Modin's architecture goes beyond\npandas, which is why the pandas API is just a thin layer at the user level. To learn\nmore about Modin's architecture, see the :doc:`architecture ` documentation.\n\nHow much faster can I go with Modin compared to pandas?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin is designed to scale with the amount of hardware available.\nEven in a traditionally serial task like ``read_csv``, we see large gains by efficiently\ndistributing the work across your entire machine. Because it is so light-weight,\nModin provides speed-ups of up to 4x on a laptop with 4 physical cores. This speedup scales\nefficiently to larger machines with more cores. We have several published papers_ that\ninclude performance results and comparisons against pandas.\n\nHow much more data would I be able to process with Modin?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nOften data scientists have to use different tools for operating on datasets of different sizes.\nThis is not only because processing large dataframes is slow, but also pandas does not support working\nwith dataframes that don't fit into the available memory. As a result, pandas workflows that work well\nfor prototyping on a few MBs of data do not scale to tens or hundreds of GBs (depending on the size\nof your machine). Modin supports operating on data that does not fit in memory, so that you can comfortably\nwork with hundreds of GBs without worrying about substantial slowdown or memory errors. For more information,\nsee :doc:`out-of-memory support ` for Modin.\n\nHow does Modin compare to Dask DataFrame and Koalas?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nTLDR: Modin has better coverage of the pandas API, has a flexible backend, better ordering semantics,\nand supports both row and column-parallel operations.\nCheck out :doc:`Modin vs Dask vs Koalas ` page detailing\nthe differences!\n\nHow does Modin work under the hood?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin is logically separated into different layers that represent the hierarchy of a\ntypical Database Management System. User queries which perform data transformation,\ndata ingress or data egress pass through the Modin Query Compiler which translates\nqueries from the top-level pandas API Layer that users interact with to the Modin Core\nDataframe layer.\nThe Modin Core DataFrame is our efficient DataFrame implementation that utilizes a partitioning schema\nwhich allows for distributing tasks and queries. From here, the Modin DataFrame works with engines like\nRay, Dask or Unidist to execute computation, and then return the results to the user.\n\nFor more details, take a look at our system :doc:`architecture `.\n\nFAQs: How to use Modin?\n-----------------------\n\nIf I’m only using my laptop, can I still get the benefits of Modin?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nAbsolutely! Unlike other parallel DataFrame systems, Modin is an extremely\nlight-weight, robust DataFrame. Because it is so light-weight, Modin provides\nspeed-ups of up to 4x on a laptop with 4 physical cores\nand allows you to work on data that doesn't fit in your laptop's RAM.\n\nHow do I use Jupyter or Colab notebooks with Modin?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nYou can take a look at this Google Colab installation guide_ and\nthis notebook tutorial_. Once Modin is installed, simply replace your pandas\nimport with Modin import:\n\n.. code-block:: python\n\n # import pandas as pd\n import modin.pandas as pd\n\nWhich execution engine (Ray, Dask or Unidist) should I use for Modin?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin lets you effortlessly speed up your pandas workflows with either Ray_'s, Dask_'s or Unidist_'s execution engine.\nYou don't need to know anything about either engine in order to use it with Modin. If you only have one engine\ninstalled, Modin will automatically detect which engine you have installed and use that for scheduling computation.\nIf you don't have a preference, we recommend starting with Modin's default Ray engine.\nIf you want to use a specific compute engine, you can set the environment variable ``MODIN_ENGINE``\nand Modin will do computation with that engine:\n\n.. code-block:: bash\n\n pip install \"modin[ray]\" # Install Modin dependencies and Ray to run on Ray\n export MODIN_ENGINE=ray # Modin will use Ray\n\n pip install \"modin[dask]\" # Install Modin dependencies and Dask to run on Dask\n export MODIN_ENGINE=dask # Modin will use Dask\n\n pip install \"modin[mpi]\" # Install Modin dependencies and MPI to run on MPI through unidist.\n export MODIN_ENGINE=unidist # Modin will use Unidist\n export UNIDIST_BACKEND=mpi # Unidist will use MPI backend.\n\nThis can also be done with:\n\n.. code-block:: python\n\n import modin.config as modin_cfg\n import unidist.config as unidist_cfg\n\n modin_cfg.Engine.put(\"ray\") # Modin will use Ray\n modin_cfg.Engine.put(\"dask\") # Modin will use Dask\n\n modin_cfg.Engine.put('unidist') # Modin will use Unidist\n unidist_cfg.Backend.put('mpi') # Unidist will use MPI backend\n\nWe plan to support more execution engines in future. If you have a specific request,\nplease post on the #feature-requests channel on our Slack_ community.\n\nHow do I connect Modin to a database via `read_sql`?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nTo read from a SQL database, you have two options:\n\n1) Pass a connection string, e.g. ``postgresql://reader:NWDMCE5xdipIjRrp@hh-pgsql-public.ebi.ac.uk:5432/pfmegrnargs``\n2) Pass an open database connection, e.g. for psycopg2, ``psycopg2.connect(\"dbname=pfmegrnargs user=reader password=NWDMCE5xdipIjRrp host=hh-pgsql-public.ebi.ac.uk\")``\n\nThe first option works with both Modin and pandas. If you try the second option\nin Modin, Modin will default to pandas because open database connections cannot be pickled.\nPickling is required to send connection details to remote workers.\nTo handle the unique requirements of distributed database access, Modin has a distributed\ndatabase connection called ``ModinDatabaseConnection``:\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.db_conn import ModinDatabaseConnection\n con = ModinDatabaseConnection(\n 'psycopg2',\n host='hh-pgsql-public.ebi.ac.uk',\n dbname='pfmegrnargs',\n user='reader',\n password='NWDMCE5xdipIjRrp')\n df = pd.read_sql(\"SELECT * FROM rnc_database\",\n con,\n index_col=None,\n coerce_float=True,\n params=None,\n parse_dates=None,\n chunksize=None)\n\n\nThe ``ModinDatabaseConnection`` will save any arguments you supply it and forward\nthem to the workers to make their own connections.\n\nHow can I contribute to Modin?\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n**Modin is currently under active development. Requests and contributions are welcome!**\n\nIf you are interested in contributing please check out the :doc:`Contributing Guide`\nand then refer to the :doc:`Development Documentation`,\nwhere you can find system architecture, internal implementation details, and other useful information.\nAlso check out the `Github`_ to view open issues and make contributions.\n\n.. _issue: https://github.com/modin-project/modin/issues\n.. _Slack: https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA\n.. _Github: https://github.com/modin-project/modin\n.. _Ray: https://github.com/ray-project/ray/\n.. _Dask: https://github.com/dask/dask\n.. _Unidist: https://github.com/modin-project/unidist\n.. _papers: https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf\n.. _guide: https://modin.readthedocs.io/en/latest/getting_started/installation.html#installing-on-google-colab\n.. _tutorial: https://github.com/modin-project/modin/tree/main/examples/tutorial\n"
},
{
"path": "docs/getting_started/installation.rst",
"content": "=============\nInstallation\n=============\n\n.. note:: \n | *Estimated Reading Time: 15 minutes*\n | If you already installed Modin on your machine, you can skip this section.\n\nThere are several ways to install Modin. Most users will want to install with\n``pip`` or using ``conda`` tool, but some users may want to build from the main branch\non the `GitHub repo`_. The main branch has the most recent patches, but may be less\nstable than a release installed from ``pip`` or ``conda``.\n\nInstalling with pip\n-------------------\n\nStable version\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin can be installed with ``pip`` on Linux, Windows and MacOS. \nTo install the most recent stable release run the following:\n\n.. code-block:: bash\n\n pip install -U modin # -U for upgrade in case you have an older version\n\nModin can be used with :doc:`Ray`, :doc:`Dask`,\n:doc:`Unidist` engines.\nIf you don't have Ray_, Dask_ or Unidist_ installed, you will need to install Modin with one of the targets:\n\n.. code-block:: bash\n\n pip install \"modin[ray]\" # Install Modin dependencies and Ray to run on Ray\n pip install \"modin[dask]\" # Install Modin dependencies and Dask to run on Dask\n pip install \"modin[mpi]\" # Install Modin dependencies and MPI to run on MPI through unidist\n pip install \"modin[all]\" # Install Ray and Dask\n\nTo get Modin on MPI through unidist (as of unidist 0.5.0) fully working\nit is required to have a working MPI implementation installed beforehand.\nOtherwise, installation of ``modin[mpi]`` may fail. Refer to\n`Installing with pip`_ section of the unidist documentation for more details about installation.\n\n**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: ``ray`` instead of ``ray[default]``.\nThis means that the dashboard and cluster launcher are no longer installed by default.\nIf you need those, consider installing ``ray[default]`` along with ``modin[ray]``.\n\nModin will automatically detect which engine you have installed and use that for\nscheduling computation!\n\nRelease candidates\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nBefore most major releases, we will upload a release candidate to test and check if there are any problems. If you would like to install a pre-release of Modin, run the following:\n\n.. code-block:: bash\n\n pip install --pre modin\n\nThese pre-releases are uploaded for dependencies and users to test their existing code\nto ensure that it still works. If you find something wrong, please raise an issue_ or\nemail the bug reporter: bug_reports@modin.org.\n\nInstalling specific dependency sets\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin has a number of specific dependency sets for running Modin on different execution engines and\nstorage formats or for different functionalities of Modin. Here is a list of dependency sets for Modin:\n\n.. code-block:: bash\n\n pip install \"modin[ray]\" # If you want to use the Ray execution engine\n\n.. code-block:: bash\n\n pip install \"modin[dask]\" # If you want to use the Dask execution engine\n\n.. code-block:: bash\n\n pip install \"modin[mpi]\" # If you want to use MPI through unidist execution engine\n\n\nConsortium Standard-compatible implementation based on Modin\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n.. code-block:: bash\n\n pip install \"modin[consortium-standard]\"\n\n\nInstalling on Google Colab\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin can be used with Google Colab_ via the ``pip`` command, by running the following code in a new cell:\n\n.. code-block:: bash\n\n !pip install \"modin[all]\"\n\nSince Colab preloads several of Modin's dependencies by default, we need to restart the Colab environment once Modin is installed by either clicking on the :code:`\"RESTART RUNTIME\"` button in the installation output or by run the following code:\n\n.. code-block:: python\n\n # Post-install automatically kill and restart Colab environment\n import os\n os.kill(os.getpid(), 9)\n\nOnce you have restarted the Colab environment, you can use Modin in Colab in subsequent sessions.\n\nNote that on the free version of Colab, there is a `limit on the compute resource `_. To leverage the full power of Modin, you may have to upgrade to Colab Pro to get access to more compute resources.\n\nInstalling with conda\n---------------------\n\nUsing conda-forge channel\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin releases can be installed using ``conda`` from conda-forge channel. Starting from 0.10.1\nit is possible to install modin with chosen engine(s) alongside. Current options are:\n\n+---------------------------------+---------------------------+-----------------------------+\n| **Package name in conda-forge** | **Engine(s)** | **Supported OSs** |\n+---------------------------------+---------------------------+-----------------------------+\n| modin | Dask_ | Linux, Windows, MacOS |\n+---------------------------------+---------------------------+-----------------------------+\n| modin-dask | Dask | Linux, Windows, MacOS |\n+---------------------------------+---------------------------+-----------------------------+\n| modin-ray | Ray_ | Linux, Windows |\n+---------------------------------+---------------------------+-----------------------------+\n| modin-mpi | MPI_ through unidist_ | Linux, Windows, MacOS |\n+---------------------------------+---------------------------+-----------------------------+\n| modin-all | Dask, Ray, Unidist | Linux |\n+---------------------------------+---------------------------+-----------------------------+\n\n**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: ``ray-core`` instead of ``ray-default``.\nThis means that the dashboard and cluster launcher are no longer installed by default.\nIf you need those, consider installing ``ray-default`` along with ``modin-ray``.\n\nFor installing Dask, Ray and MPI through unidist engines into conda environment following command should be used:\n\n.. code-block:: bash\n\n conda install -c conda-forge modin-ray modin-dask modin-mpi\n\nAll set of engines could be available in conda environment by specifying:\n\n.. code-block:: bash\n\n conda install -c conda-forge modin-all\n\nor explicitly:\n\n.. code-block:: bash\n\n conda install -c conda-forge modin-ray modin-dask modin-mpi\n\nRefer to `Installing with conda`_ section of the unidist documentation\nfor more details on how to install a specific MPI implementation to run on.\n\n``conda`` may be slow installing ``modin-all`` or combitations of execution engines so we currently recommend using libmamba solver for the installation process.\nTo do this install it in a base environment:\n\n.. code-block:: bash\n\n conda install -n base conda-libmamba-solver\n\nThen it can be used during installation either like\n\n.. code-block:: bash\n\n conda install -c conda-forge modin-ray modin- --experimental-solver=libmamba\n\nor starting from conda 22.11 and libmamba solver 22.12 versions\n\n.. code-block:: bash\n\n conda install -c conda-forge modin-ray --solver=libmamba\n\n\nInstalling from the GitHub main branch\n--------------------------------------\n\nIf you'd like to try Modin using the most recent updates from the main branch, you can\nalso use ``pip``.\n\n.. code-block:: bash\n\n pip install \"modin[all] @ git+https://github.com/modin-project/modin\"\n\nThis will install directly from the repo without you having to manually clone it! Please be aware\nthat these changes have not made it into a release and may not be completely stable.\n\nIf you would like to install Modin with a specific engine, you can use ``modin[ray]`` or ``modin[dask]`` or ``modin[mpi]`` instead of ``modin[all]`` in the command above.\n\nWindows\n-------\n\nAll Modin engines are available both on Windows and Linux as mentioned above.\nDefault engine on Windows is :doc:`Ray`.\nIt is also possible to use Windows Subsystem For Linux (WSL_), but this is generally \nnot recommended due to the limitations and poor performance of Ray on WSL, a roughly \n2-3x worse than native Windows. \n\nBuilding Modin from Source\n--------------------------\n\nIf you're planning on :doc:`contributing ` to Modin, you will need to ensure that you are\nbuilding Modin from the local repository that you are working off of. Occasionally,\nthere are issues in overlapping Modin installs from pypi and from source. To avoid these\nissues, we recommend uninstalling Modin before you install from source:\n\n.. code-block:: bash\n\n pip uninstall modin\n\nTo build from source, you first must clone the repo. We recommend forking the repository first\nthrough the GitHub interface, then cloning as follows:\n\n.. code-block:: bash\n\n git clone https://github.com//modin.git\n\nOnce cloned, ``cd`` into the ``modin`` directory and use ``pip`` to install:\n\n.. code-block:: bash\n\n cd modin\n pip install -e .\n pip install -e \".[all]\" # will install dependencies for all engines\n\n.. _`GitHub repo`: https://github.com/modin-project/modin/tree/main\n.. _issue: https://github.com/modin-project/modin/issues\n.. _WSL: https://docs.microsoft.com/en-us/windows/wsl/install-win10\n.. _Ray: http://ray.readthedocs.io\n.. _Dask: https://github.com/dask/dask\n.. _MPI: https://www.mpi-forum.org/\n.. _Unidist: https://github.com/modin-project/unidist\n.. _`Installing with pip`: https://unidist.readthedocs.io/en/latest/installation.html#installing-with-pip\n.. _`Installing with conda`: https://unidist.readthedocs.io/en/latest/installation.html#installing-with-conda\n.. _`Intel Distribution of Modin`: https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/distribution-of-modin.html#gs.86stqv\n.. _`Intel Distribution of Modin Getting Started`: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-distribution-of-modin-getting-started-guide.html\n.. |reg| unicode:: U+000AE .. REGISTERED SIGN\n.. _Colab: https://colab.research.google.com/\n"
},
{
"path": "docs/getting_started/quickstart.rst",
"content": "Getting Started\n===============\n\n.. note:: \n | *Estimated Reading Time: 10 minutes*\n | You can follow along this tutorial in a Jupyter notebook `here `_. \n\n.. toctree::\n :hidden:\n :maxdepth: 4\n \n 10-min Quickstart Guide \n installation\n using_modin/using_modin\n why_modin/why_modin\n examples\n faq\n troubleshooting\n\n.. meta::\n :description lang=en:\n Introduction to Modin.\n\nQuick Start Guide\n-----------------\n\nTo install the most recent stable release for Modin run the following:\n\n.. code-block:: bash\n\n pip install \"modin[all]\" \n\nFor further instructions on how to install Modin with conda or for specific platforms \nor engines, see our detailed `installation guide <../getting_started/installation.html>`_.\n\nModin acts as a drop-in replacement for pandas so you simply have to replace the import \nof pandas with the import of Modin as follows to speed up your pandas workflows:\n\n.. code-block:: bash\n\n # import pandas as pd\n import modin.pandas as pd\n\nExample: Instant Scalability with No Extra Effort\n-------------------------------------------------\n\nWhen working on large datasets, pandas becomes painfully slow or :doc:`runs out of memory`. Modin automatically scales up your \npandas workflows by parallelizing the dataframe operations, so that you can more \neffectively leverage the compute resources available.\n\nFor the purpose of demonstration, we will load in modin as ``pd`` and pandas as \n``pandas``.\n\n.. code-block:: python\n\n import modin.pandas as pd\n import pandas\n\n #############################################\n ### For the purpose of timing comparisons ###\n #############################################\n import time\n import ray\n # Look at the Ray documentation with respect to the Ray configuration suited to you most.\n ray.init()\n #############################################\n\nIn this toy example, we look at the NYC taxi dataset, which is around 200MB in size. You can download `this dataset `_ to run the example locally.\n\n.. code-block:: python\n\n # This may take a few minutes to download\n import urllib.request\n dataset_url = \"https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv\"\n urllib.request.urlretrieve(dataset_url, \"taxi.csv\") \n\nFaster Data Loading with ``read_csv``\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n.. code-block:: python\n \n start = time.time()\n\n pandas_df = pandas.read_csv(dataset_url, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)\n\n end = time.time()\n pandas_duration = end - start\n print(\"Time to read with pandas: {} seconds\".format(round(pandas_duration, 3)))\n\nBy running the same command ``read_csv`` with Modin, we generally get around 4X speedup \nfor loading in the data in parallel. \n\n.. code-block:: python\n\n start = time.time()\n\n modin_df = pd.read_csv(dataset_url, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)\n\n end = time.time()\n modin_duration = end - start\n print(\"Time to read with Modin: {} seconds\".format(round(modin_duration, 3)))\n\n print(\"Modin is {}x faster than pandas at `read_csv`!\".format(round(pandas_duration / modin_duration, 2)))\n\nFaster ``concat`` across multiple dataframes\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nOur previous ``read_csv`` example operated on a relatively small dataframe. In the \nfollowing example, we duplicate the same taxi dataset 100 times and then concatenate \nthem together, resulting in a dataset around 19GB in size.\n\n.. code-block:: python\n\n start = time.time()\n\n big_pandas_df = pandas.concat([pandas_df for _ in range(25)])\n\n end = time.time()\n pandas_duration = end - start\n print(\"Time to concat with pandas: {} seconds\".format(round(pandas_duration, 3)))\n\n.. code-block:: python\n\n start = time.time()\n\n big_modin_df = pd.concat([modin_df for _ in range(25)])\n\n end = time.time()\n modin_duration = end - start\n print(\"Time to concat with Modin: {} seconds\".format(round(modin_duration, 3)))\n\n print(\"Modin is {}x faster than pandas at `concat`!\".format(round(pandas_duration / modin_duration, 2)))\n\nModin speeds up the ``concat`` operation by more than 60X, taking less than a second to \ncreate the large dataframe, while pandas took close to a minute.\n\n\nFaster ``apply`` over a single column\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe performance benefits of Modin become apparent when we operate on large \ngigabyte-scale datasets. Let's say we want to round up values \nacross a single column via the ``apply`` operation. \n\n.. code-block:: python\n\n start = time.time()\n rounded_trip_distance_pandas = big_pandas_df[\"trip_distance\"].apply(round)\n\n end = time.time()\n pandas_duration = end - start\n print(\"Time to apply with pandas: {} seconds\".format(round(pandas_duration, 3)))\n\n.. code-block:: python\n \n start = time.time()\n\n rounded_trip_distance_modin = big_modin_df[\"trip_distance\"].apply(round)\n\n end = time.time()\n modin_duration = end - start\n print(\"Time to apply with Modin: {} seconds\".format(round(modin_duration, 3)))\n\n print(\"Modin is {}x faster than pandas at `apply` on one column!\".format(round(pandas_duration / modin_duration, 2)))\n\nModin is more than 30X faster at applying a single column of data, operating on 130+ \nmillion rows in a second.\n\nIn short, Modin provides orders of magnitude speed up over pandas for a variety of operations out of the box. \n\n.. figure:: ../img/quickstart_speedup.svg\n :align: center\n\nSummary\n-------\n\nHopefully, this tutorial demonstrated how Modin delivers significant speedup on pandas \noperations without the need for any extra effort. Throughout example, we moved from \nworking with 100MBs of data to 20GBs of data all without having to change anything or \nmanually optimize our code to achieve the level of scalable performance that Modin \nprovides.\n\nNote that in this quickstart example, we've only shown ``read_csv``, ``concat``, \n``apply``, but these are not the only pandas operations that Modin optimizes for. In \nfact, Modin covers `more than 90\\% of the pandas API `_, yielding considerable speedups for \nmany common operations.\n"
},
{
"path": "docs/getting_started/troubleshooting.rst",
"content": "Troubleshooting\n===============\n\nWe hope your experience with Modin is bug-free, but there are some quirks about Modin\nthat may require troubleshooting. If you are still having issues, please post on\nthe #support channel on our Slack_ community or open a Github issue_.\n\nFrequently encountered issues\n-----------------------------\n\nThis is a list of the most frequently encountered issues when using Modin. Some of these\nare working as intended, while others are known bugs that are being actively worked on.\n\nWarning during execution: ``defaulting to pandas``\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nPlease note, that while Modin covers a large portion of the pandas API, not all functionality is implemented. For methods that are not yet implemented, such as ``asfreq``, you may see the following:\n\n.. code-block:: text\n\n UserWarning: `DataFrame.asfreq` defaulting to pandas implementation.\n\nTo understand which functions will lead to this warning, we have compiled a list of :doc:`currently supported methods `. When you see this warning, Modin defaults to pandas by converting the Modin dataframe to pandas to perform the operation. Once the operation is complete in pandas, it is converted back to a Modin dataframe. These operations will have a high overhead due to the communication involved and will take longer than pandas. When this is happening, a warning will be given to the user to inform them that this operation will take longer than usual. You can learn more about this :doc:`here `.\n\nIf you would like to request a particular method be implemented, feel free to open an\n`issue`_. Before you open an issue please make sure that someone else has not already\nrequested that functionality.\n\nHanging on ``import modin.pandas as pd``\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThis can happen when Ray fails to start. It will keep retrying, but often it is faster\nto just restart the notebook or interpreter. Generally, this should not happen. Most\ncommonly this is encountered when starting multiple notebooks or interpreters in quick\nsuccession.\n\n**Solution**\n\nRestart your interpreter or notebook kernel.\n\n**Avoiding this Error**\n\nAvoid starting many Modin notebooks or interpreters in quick succession. Wait 2-3\nseconds before starting the next one.\n\nImporting heterogeneous data using ``read_csv``\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nSince Modin's ``read_csv`` imports data in parallel, it is possible for data across\npartitions to be heterogeneously typed (this can happen when columns contain\nheterogeneous data, i.e. values in the same column are of different types). An example\nof how this is handled is shown below.\n\n.. code-block:: python\n\n import os\n import pandas\n import modin.pandas as pd\n from modin.config import NPartitions\n\n NPartitions.put(2)\n\n test_filename = \"test.csv\"\n # data with heterogeneous values in the first column\n data = \"\"\"one,2\n 3,4\n 5,6\n 7,8\n 9.0,10\n \"\"\"\n kwargs = {\n # names of the columns to set, if `names` parameter is set,\n # header inffering from the first data row/rows will be disabled\n \"names\": [\"col1\", \"col2\"],\n\n # explicit setting of data type of column/columns with heterogeneous\n # data will force partitions to read data with correct dtype\n # \"dtype\": {\"col1\": str},\n }\n\n\n try :\n with open(test_filename, \"w\") as f:\n f.write(data)\n\n pandas_df = pandas.read_csv(test_filename, **kwargs)\n pd_df = pd.read_csv(test_filename, **kwargs)\n\n print(pandas_df)\n print(pd_df)\n finally:\n os.remove(test_filename)\n\n Output:\n\n pandas_df:\n col1 col2\n 0 one 2\n 1 3 4\n 2 5 6\n 3 7 8\n 4 9.0 10\n\n pd_df:\n col1 col2\n 0 one 2\n 1 3 4\n 2 5 6\n 3 7.0 8\n 4 9.0 10\n\n\nIn this case, ``col1`` of the `DataFrame` read by pandas contains only ``str`` data\nbecause the first value (\"one\") is inferred to have type ``str``, which forces pandas to handle the rest of the values in the column\nas strings. The first Modin partition (the first three rows) handles the data as pandas does,\nbut the second partition (the last two rows) reads the data as floats. This is because the\nsecond column contains an int and a float, and thus the column type is inferred to be float. As a\nresult, `7` is interpreted as `7.0`, which differs from the pandas output.\n\nThe above example demonstrates heterogenous data import with str, int, and float types,\nbut heterogeneous data consisting of other data/parameter combinations can also result in \ndata type mismatches with pandas.\n\n**Solution**\n\nWhen heterogeneous data is detected, a warning will be raised.\nCurrently, these discrepancies aren't properly handled\nby Modin, so to avoid this issue, you need to set the ``dtype`` parameter of ``read_csv``\nmanually to force the correct data type coercion during data import. Note that \nto avoid excessive performance degradation, the ``dtype`` value should only be set for columns that may contain heterogenous data.\nas possible (specify ``dtype`` parameter only for columns with heterogeneous data).\n\nSpecifying the ``dtype`` parameter will work well in most cases. If the file\ncontains a column that should be interpreted as the index\n(the ``index_col`` parameter is specified) there may still be type discrepancies in the index, since the ``dtype`` parameter is only responsible for data\nfields. If in the above example, ``kwargs`` was set like so:\n\n.. code-block:: python\n\n kwargs = {\n \"names\": [\"col1\", \"col2\"],\n \"dtype\": {\"col1\": str},\n \"index_col\": \"col1\",\n }\n\nThe resulting Modin DataFrame will contain incorrect values - just as if ``dtype``\nhad not been specified:\n\n.. code-block:: python\n\n col1\n one 2\n 3 4\n 5 6\n 7.0 8\n 9.0 10\n\nOne workaround is to import the data without setting the ``index_col`` parameter, and then \nset the index column using the ``DataFrame.set_index`` function as shown in\nthe example below:\n\n.. code-block:: python\n\n pd_df = pd.read_csv(filename, dtype=data_dtype, index_col=None)\n pd_df = pd_df.set_index(index_col_name)\n pd_df.index.name = None\n\n\nUsing Modin with python multiprocessing\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nWe strongly recommend against using a distributed execution engine (e.g. Ray or Dask)\nin conjunction with Python multiprocessing because that can lead to undefined behavior.\nOne such example is shown below:\n\n.. code-block:: python\n\n import modin.pandas as pd\n\n # Ray engine is used by default\n df = pandas.DataFrame([1, 2, 3])\n\n def f(arg):\n return df + arg\n\n if __name__ == '__main__':\n from multiprocessing import Pool\n\n with Pool(5) as p:\n print(p.map(f, [1]))\n\nAlthough this example may work on your machine, we do not recommend it, because\nthe Python multiprocessing library will duplicate Ray clusters, causing both\nexcessive resource usage and conflict over the available resources.\n\nPoor performance of the first operation with Modin on Ray engine\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThere might be cases when the first operation with Modin on Ray engine is much slower than the subsequent calls of the operation.\nThat happens because Ray workers may not be fully set up yet to perform computation after initialization of the engine\nwith ``ray.init(runtime_env={'env_vars': {'__MODIN_AUTOIMPORT_PANDAS__': '1'}})``, which is the default behavior of Modin on Ray engine\nif Ray has not been initialised yet. Modin intentionaly initializes Ray this way to import ``pandas`` in workers\nonce Python interpreter is started in them so that to avoid a race condition in Ray between the import thread and the thread executing the code.\n\n..\n See more details on why we started using ``ray.init(runtime_env={'env_vars': {'__MODIN_AUTOIMPORT_PANDAS__': '1'}})` in\n https://github.com/modin-project/modin/pull/4603.\n\n.. code-block:: python\n\n import time\n import pandas\n import numpy as np\n import ray\n import modin.pandas as pd\n import modin.config as cfg\n\n # Look at the Ray documentation with respect to the Ray configuration suited to you most.\n ray.init(runtime_env={'env_vars': {'__MODIN_AUTOIMPORT_PANDAS__': '1'}})\n\n pandas_df = pandas.DataFrame(\n np.random.randint(0, 100, size=(1000000, 13))\n )\n pandas_df.to_csv(\"foo.csv\", index=False)\n\n def read_csv_with_pandas():\n start_time = time.time()\n pandas_df = pandas.read_csv(\"foo.csv\", index_col=0)\n end_time = time.time()\n pandas_duration = end_time - start_time\n print(\"Time to read_csv with pandas: {} seconds\".format(round(pandas_duration, 3)))\n return pandas_df\n\n def read_csv_with_modin():\n start_time = time.time()\n modin_df = pd.read_csv(\"foo.csv\", index_col=0)\n end_time = time.time()\n modin_duration = end_time - start_time\n print(\"Time to read_csv with Modin: {} seconds\".format(round(modin_duration, 3))) \n return modin_df\n\n for i in range(5):\n pandas_df = read_csv_with_pandas()\n modin_df = read_csv_with_modin()\n\n Time to read_csv with pandas: 0.708 seconds\n Time to read_csv with Modin: 4.132 seconds\n Time to read_csv with pandas: 0.735 seconds\n Time to read_csv with Modin: 0.37 seconds\n Time to read_csv with pandas: 0.646 seconds\n Time to read_csv with Modin: 0.377 seconds\n Time to read_csv with pandas: 0.673 seconds\n Time to read_csv with Modin: 0.371 seconds\n Time to read_csv with pandas: 0.672 seconds\n Time to read_csv with Modin: 0.379 seconds\n\n**Solution**\n\nSo far there is no a solution to fix or work around the problem rather than not to pass a non-empty runtime_env to ``ray.init()``.\nHowever, this may lead to other problem regarding a race condition in Ray between the import thread and the thread executing the code.\nSo for now we just highlight the problem in hope of a future fix in Ray itself.\n\nAlso, it is worth noting that every distributed engine by its nature has a little overhead for the first operation being called,\nwhich may be important for microbenchmarks. What you likely want to do is warm up worker processes\neither by excluding the time of the first iteration from your measurements or execute a simple function in workers to fully set up them.\n\nCommon errors\n-------------\n\nError when using Dask engine: ``RuntimeError: if __name__ == '__main__':``\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe following `script.py` uses Modin with Dask as an execution engine and produces errors:\n\n.. code-block:: python\n\n # script.py\n import modin.pandas as pd\n import modin.config as cfg\n\n cfg.Engine.put(\"dask\")\n\n df = pd.DataFrame([0,1,2,3])\n print(df)\n\nA part of the produced errors by the script above would be the following:\n\n.. code-block::\n\n File \"/path/python3.9/multiprocessing/spawn.py\", line 134, in _check_not_importing_main\n raise RuntimeError('''\n RuntimeError: \n An attempt has been made to start a new process before the\n current process has finished its bootstrapping phase.\n\n This probably means that you are not using fork to start your\n child processes and you have forgotten to use the proper idiom\n in the main module:\n\n if __name__ == '__main__':\n freeze_support()\n ...\n\n The \"freeze_support()\" line can be omitted if the program\n is not going to be frozen to produce an executable.\n\nThis happens because Dask Client uses `fork `_\nto start processes.\n\n**Solution**\n\nTo avoid the problem the Dask Client creation code needs to be moved into the ``__main__`` scope of the module.\n\nThe corrected `script.py` would look like:\n\n.. code-block:: python\n\n # script.py\n import modin.pandas as pd\n import modin.config as cfg\n\n cfg.Engine.put(\"dask\")\n\n if __name__ == \"__main__\":\n df = pd.DataFrame([0, 1, 2, 3]) # Dask Client creation is hidden in the first call of Modin functionality.\n print(df)\n\nor\n\n.. code-block:: python\n\n # script.py\n from distributed import Client\n import modin.pandas as pd\n import modin.config as cfg\n\n cfg.Engine.put(\"dask\")\n\n if __name__ == \"__main__\":\n # Explicit Dask Client creation.\n # Look at the Dask Distributed documentation with respect to the Client configuration suited to you most.\n client = Client()\n df = pd.DataFrame([0, 1, 2, 3])\n print(df)\n\nSpurious error \"cannot import partially initialised pandas module\" on custom Ray cluster\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nIf you're using some pre-configured Ray cluster to run Modin, it's possible you would\nbe seeing spurious errors like\n\n.. code-block::\n\n ray.exceptions.RaySystemError: System error: partially initialized module 'pandas' has no attribute 'core' (most likely due to a circular import)\n traceback: Traceback (most recent call last):\n File \"/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/ray/serialization.py\", line 340, in deserialize_objects\n obj = self._deserialize_object(data, metadata, object_ref)\n File \"/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/ray/serialization.py\", line 237, in _deserialize_object\n return self._deserialize_msgpack_data(data, metadata_fields)\n File \"/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/ray/serialization.py\", line 192, in _deserialize_msgpack_data\n python_objects = self._deserialize_pickle5_data(pickle5_data)\n File \"/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/ray/serialization.py\", line 180, in _deserialize_pickle5_data\n obj = pickle.loads(in_band, buffers=buffers)\n File \"/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/pandas/__init__.py\", line 135, in \n from pandas import api, arrays, errors, io, plotting, testing, tseries\n File \"/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/pandas/testing.py\", line 6, in \n from pandas._testing import (\n File \"/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/pandas/_testing/__init__.py\", line 979, in \n cython_table = pd.core.common._cython_table.items()\n AttributeError: partially initialized module 'pandas' has no attribute 'core' (most likely due to a circular import)\n\n**Solution**\n\nModin contains a workaround that should automatically do ``import pandas`` upon worker process starts.\n\nIt is triggered by the presence of non-empty ``__MODIN_AUTOIMPORT_PANDAS__`` environment variable which\nModin sets up automatically on the Ray clusters it spawns, but it might be missing on pre-configured clusters.\n\nSo if you're seeing the issue like shown above, please make sure you set this environment variable on all\nworker nodes of your cluster before actually spawning the workers.\n\n.. _issue: https://github.com/modin-project/modin/issues\n.. _Slack: https://modin.org/slack.html\n"
},
{
"path": "docs/getting_started/using_modin/using_modin.rst",
"content": "Using Modin\n===========\n\nIn this section, we show how Modin can be used to accelerate your pandas workflows on a \nsingle machine up to multiple machines in a cluster setting.\n\n.. toctree::\n :maxdepth: 4\n \n using_modin_locally\n using_modin_cluster\n "
},
{
"path": "docs/getting_started/using_modin/using_modin_cluster.rst",
"content": "Using Modin in a Cluster\n========================\n\n.. note::\n | *Estimated Reading Time: 15 minutes*\n\nOften in practice we have a need to exceed the capabilities of a single machine.\nModin works and performs well in both local mode and in a cluster environment.\nThe key advantage of Modin is that your python code does not change between\nlocal development and cluster execution. Users are not required to think about\nhow many workers exist or how to distribute and partition their data;\nModin handles all of this seamlessly and transparently.\n\n.. note::\n It is possible to use a Jupyter notebook, but you will have to deploy a Jupyter server \n on the remote cluster head node and connect to it.\n\n.. image:: ../../img/modin_cluster.png\n :alt: Modin cluster\n :align: center\n\nExtra requirements for AWS authentication\n-----------------------------------------\n\nFirst of all, install the necessary dependencies in your environment:\n\n.. code-block:: bash\n\n pip install boto3\n\nThe next step is to setup your AWS credentials. One can set ``AWS_ACCESS_KEY_ID``, \n``AWS_SECRET_ACCESS_KEY`` and ``AWS_SESSION_TOKEN`` (Optional)\n(refer to `AWS CLI environment variables`_ to get more insight on this) or \njust run the following command:\n\n.. code-block:: bash\n\n aws configure\n\nStarting and connecting to the cluster\n--------------------------------------\n\nThis example starts 1 head node (m5.24xlarge) and 5 worker nodes (m5.24xlarge), 576 total CPUs.\nYou can check the `Amazon EC2 pricing`_ page.\n\nIt is possble to manually create AWS EC2 instances and configure them or just use the `Ray CLI`_ to \ncreate and initialize a Ray cluster on AWS using `Modin's Ray cluster setup config`_,\nwhich we are going to utilize in this example.\nRefer to `Ray's autoscaler options`_ page on how to modify the file.\n\nMore details on how to launch a Ray cluster can be found on `Ray's cluster docs`_.\n\nTo start up the Ray cluster, run the following command in your terminal:\n\n.. code-block:: bash\n\n ray up modin-cluster.yaml\n\nOnce the head node has completed initialization, you can optionally connect to it by running the following command.\n\n.. code-block:: bash\n\n ray attach modin-cluster.yaml\n\nTo exit the ssh session and return back into your local shell session, type:\n\n.. code-block:: bash\n\n exit\n\nExecuting in a cluster environment\n----------------------------------\n\n.. note::\n Be careful when using the `Ray client`_ to connect to a remote cluster.\n We don't recommend this connection mode, beacuse it may not work. Known bugs:\n - https://github.com/ray-project/ray/issues/38713,\n - https://github.com/modin-project/modin/issues/6641.\n\nModin lets you instantly speed up your workflows with a large data by scaling pandas\non a cluster. In this tutorial, we will use a 12.5 GB ``big_yellow.csv`` file that was\ncreated by concatenating a 200MB `NYC Taxi dataset`_ file 64 times. Preparing this\nfile was provided as part of our `Modin's Ray cluster setup config`_.\n\nIf you want to use the other dataset, you should provide it to each of\nthe cluster nodes with the same path. We recomnend doing this by customizing the\n``setup_commands`` section of the `Modin's Ray cluster setup config`_.\n\nTo run any script in a remote cluster, you need to submit it to the Ray. In this way,\nthe script file is sent to the the remote cluster head node and executed there. \n\nIn this tutorial, we provide the `exercise_5.py`_ script, which reads the data from the\nCSV file and executes such pandas operations as count, groupby and map.\nAs the result, you will see the size of the file being read and the execution time of the entire script.\n\nYou can submit this script to the existing remote cluster by running the following command.\n\n.. code-block:: bash\n\n ray submit modin-cluster.yaml exercise_5.py\n\nTo download or upload files to the cluster head node, use ``ray rsync_down`` or ``ray rsync_up``.\nIt may help if you want to use some other Python modules that should be available to\nexecute your own script or download a result file after executing the script.\n\n.. code-block:: bash\n\n # download a file from the cluster to the local machine:\n ray rsync_down modin-cluster.yaml '/path/on/cluster' '/local/path'\n # upload a file from the local machine to the cluster:\n ray rsync_up modin-cluster.yaml '/local/path' '/path/on/cluster'\n\nShutting down the cluster\n--------------------------\n\nNow that we have finished the computation, we need to shut down the cluster with `ray down` command.\n\n.. code-block:: bash\n\n ray down modin-cluster.yaml\n\n.. _`Ray's autoscaler options`: https://docs.ray.io/en/latest/cluster/vms/references/ray-cluster-configuration.html#cluster-config\n.. _`Ray's cluster docs`: https://docs.ray.io/en/latest/cluster/getting-started.html\n.. _`NYC Taxi dataset`: https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv\n.. _`Modin's Ray cluster setup config`: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/execution/pandas_on_ray/cluster/modin-cluster.yaml\n.. _`Amazon EC2 pricing`: https://aws.amazon.com/ec2/pricing/on-demand/\n.. _`exercise_5.py`: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/execution/pandas_on_ray/cluster/exercise_5.py\n.. _`Ray client`: https://docs.ray.io/en/latest/cluster/running-applications/job-submission/ray-client.html\n.. _`Ray CLI`: https://docs.ray.io/en/latest/cluster/vms/getting-started.html#running-applications-on-a-ray-cluster\n.. _`AWS CLI environment variables`: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html"
},
{
"path": "docs/getting_started/using_modin/using_modin_locally.rst",
"content": "===================\nUsing Modin Locally\n===================\n\n.. note::\n | *Estimated Reading Time: 5 minutes*\n | You can follow along this tutorial in the `Jupyter notebook`_.\n\nIn our quickstart example, we have already seen how you can achieve considerable\nspeedup from Modin, even on a single machine. Users do not need to know how many\ncores their system has, nor do they need to specify how to distribute the data. In fact,\nusers can **continue using their existing pandas code** while experiencing a\nconsiderable speedup from Modin, even on a single machine.\n\nTo use Modin on a single machine, only a modification of the import statement is needed.\nOnce you've changed your import statement, you're ready to use Modin\njust like you would pandas, since the API is identical to pandas.\n\n.. code-block:: python\n\n # import pandas as pd\n import modin.pandas as pd\n\n**That's it. You're ready to use Modin on your previous pandas workflows!**\n\nAdvanced: Configuring the resources Modin uses\n----------------------------------------------\n\nModin automatically check the number of CPUs available on your machine and sets the\nnumber of partitions to be equal to the number of CPUs. You can verify this by running\nthe following code:\n\n.. code-block:: python\n\n import modin\n print(modin.config.NPartitions.get()) #prints 16 on a laptop with 16 physical cores\n\nModin fully utilizes the resources on your machine. To read more about how this works,\nsee :doc:`Why Modin? ` page for more details.\n\nSince Modin will use all of the resources available on your machine by default, at\ntimes, it is possible that you may like to limit the amount of resources Modin uses to\nfree resources for another task or user. Here is how you would limit the number of CPUs\nModin used in your bash environment variables:\n\n.. code-block:: bash\n\n export MODIN_CPUS=4\n\n\nYou can also specify this in your python script with ``os.environ``:\n\n.. code-block:: python\n\n import os\n os.environ[\"MODIN_CPUS\"] = \"4\"\n import modin.pandas as pd\n\nIf you're using a specific engine and want more control over the environment Modin\nuses, you can start Ray or Dask in your environment and Modin will connect to it.\n\n.. code-block:: python\n\n import ray\n ray.init(num_cpus=4)\n import modin.pandas as pd\n\nSpecifying ``num_cpus`` limits the number of processors that Modin uses. You may also\nspecify more processors than you have available on your machine; however this will not\nimprove the performance (and might end up hurting the performance of the system).\n\n.. note::\n Make sure to update the ``MODIN_CPUS`` configuration and initialize your preferred\n engine before you start working with the first operation using Modin! Otherwise,\n Modin will opt for the default setting.\n\n\n.. _`Jupyter notebook`: https://github.com/modin-project/modin/tree/main/examples/quickstart.ipynb\n"
},
{
"path": "docs/getting_started/why_modin/modin_vs_dask_vs_koalas.rst",
"content": "Modin vs. Dask DataFrame vs. Koalas\n===================================\n\nLibraries such as `Dask DataFrame `_ (DaskDF for short) and `Koalas `_ aim to support the pandas API on top of distributed computing frameworks, Dask and Spark respectively. Instead, Modin aims to preserve the pandas API and behavior as is, while abstracting away the details of the distributed computing framework underneath. Thus, the aims of these libraries are fundamentally different.\n\nSpecifically, Modin enables pandas-like\n\n* row and column-parallel operations, unlike DaskDF and Koalas that only support row-parallel operations\n* indexing & ordering semantics, unlike DaskDF and Koalas that deviate from these semantics\n* eager execution, unlike DaskDF and Koalas that provide lazy execution\n\nAs a result, Modin's coverage is `more than 90% `_ of the pandas API, while DaskDF and Koalas' coverage is about 55%. \n\n.. figure:: ../../img/api_coverage_comparison.svg\n :align: center\n :alt: Percentage coverage of the pandas API after deduplication\n\nFor more technical details please see our VLDB 2022 research paper, referenced `here `_. \n\nBrief Overview of DaskDF and Koalas\n-----------------------------------\n\nDask's `DataFrame `_ (DaskDF) is effectively a meta-DataFrame, partitioning and scheduling many smaller ``pandas.DataFrame`` objects. Users construct a task graph of dataframe computation step by step and then trigger computation using the ``compute`` function.\n\nSpark's `Koalas `_ provides the pandas API on Spark, leveraging the preexisting Spark SQL optimizer to execute select pandas commands. Like DaskDF, Koalas also employs lazy computation, only triggering computation when the user requests to see the results.\n\nPartitioning and Parallelization\n--------------------------------\n\nModin, DaskDF, Koalas are all examples of parallel dataframe systems. Parallelism is achieved by partitioning a large dataframe into smaller ones that can be operated on in parallel. As a result, the partitioning scheme chosen by the system dictates the pandas functions that can or can not be supported.\n\n**DaskDF and Koalas only support row-oriented partitioning and parallelism.** This approach is analogous to relational databases. The dataframe is conceptually broken down into horizontal partitions along rows, where each partition is independently processed if possible. When DaskDF or Koalas are required to perform column-parallel operations that to be done on columns independently (e.g., dropping columns with null values via ``dropna`` on the column ``axis``), they either perform very poorly with no parallelism or do not support that operation.\n\n**Modin supports both row, column, and cell-oriented partitioning and parallelism**. That is, the dataframe can be conceptually broken down as groups of rows, groups of columns, or both groups of rows and groups of columns (effectively a block or sub-matrix). Modin will transparently reshape the partitioning as necessary for the corresponding operation, based on whether the operation is row-parallel, column-parallel, or cell-parallel (independently applied to each unit cell). This allows Modin to support more of the pandas API and do so efficiently. Due to the finer-grained control over the partitioning, Modin can support a number of operations that are very challenging to parallelize in row-oriented systems (e.g., ``transpose``, ``median``, ``quantile``). This flexibility in partitioning also gives Modin tremendous power to implement efficient straggler mitigation and improve utilization over the entire cluster.\n\nAPI Coverage\n------------\n\nOne of the key benefits of pandas is its versatility, due to the wide array of operations, with more than 600+ API operations for data cleaning, feature engineering, data transformation, data summarization, data exploration, and machine learning. However, it is not trivial to develop scalable implementations of each of these operations in a dataframe system.\n**DaskDF and Koalas only implements about** `55% `_ **of the pandas API**; they do not implement certain APIs that would deviate from the row-wise partitioning approach, or would be inefficient with the row-wise parallelization. For example, Dask does not implement ``iloc``, ``MultiIndex``, ``apply(axis=0)``, ``quantile`` (only approximate quantile is available), ``median``, and more. Given DaskDF's row-oriented architecture, ``iloc``, for example, can technically be implemented, but it would be inefficient, and column-wise operations such as ``apply(axis=0)`` would be impossible to implement. Similarly, Koalas does not implement ``apply(axis=0)`` (it only applies the function per row partition, giving a different result), ``quantile``, ``median`` (only approximate quantile/median is available), ``MultiIndex``, ``combine``, ``compare`` and more.\n\n**Modin supports all of the above pandas API functions, as well as others, with** `more than 90% `_ **coverage of the pandas API.** Modin additionally acts as a drop-in replacement for pandas, such that even if the API is not yet supported, it still works by falling back to running vanilla pandas. One of the key features of being a drop-in replacement is that not only will it work for existing code, if a user wishes to go back to running pandas directly, they are not locked in to using Modin and can switch between Modin and pandas at no cost. In other words, scripts and notebooks written in Modin can be converted to and from pandas as the user desires by simply replacing the import statement.\n\nExecution Semantics\n---------------------\n\n**DaskDF and Koalas make use of lazy evaluation, which means that the computation is delayed until users explicitly evaluate the results.** This mode of evaluation places a lot of optimization responsibility on the user, forcing them to think about when it would be useful to inspect the intermediate results or delay doing so. Specifically, DaskDF's API differs from pandas in that it requires users to explicitly call ``.compute()`` to materialize the result of the computation. Often if that computation corresponds to a long chain of operators, this call can take a very long time to execute. Overall, the need to explicitly trigger computation makes the API less convenient to work with, but gives DaskDF and Koalas the opportunity to perform holistic optimizations over the entire dataflow graph. However, to the best of our knowledge, neither DaskDF nor Koalas actually leverage holistic optimizations.\n\n**Modin employs eager evaluation, like pandas.** Eager evaluation is the default mode of operation for data scientists when working with pandas in an interactive environment, such as Jupyter Notebooks. Modin reproduces this familiar behavior by performing all computations eagerly as soon as it is issued, so that users can inspect intermediate results and quickly see the results of their computations without having to wait or explicitly trigger computation. This is especially useful during interactive data analysis, where users often iterate on their dataframe workflows or build up their dataframe queries in an incremental fashion. We also have developed techniques for `opportunistic evaluation `_ that bridges the gap between lazy and eager evaluation that will be incorporated in Modin in the future.\n\nOrdering Semantics\n------------------\n\nBy default, pandas preserves the order of the dataframe, so that users can expect a consistent, ordered view as they are operating on their dataframe. \n\n**Both DaskDF and Koalas make no guarantees about the order of rows in the DataFrame.** This is because DaskDF sorts the ``index`` for optimization purposes to speed up computations that involve the row index; and as a result, it does not support user-specified order. Likewise, Koalas `does not support ordering `_ by default because it will lead to a performance overhead when operating on distributed datasets. \n\n**DaskDF additionally does not support multi-indexing or sorting.** \nDaskDF sorts the data based on a single set of row labels for fast row lookups, and builds an indexing structure based on these row labels. Data is both logically and physically stored in the same order. As a result, DaskDF does not support a `sort` function.\n\n**Modin reproduces the intuitive behavior in pandas where the order of the DataFrame is preserved, and supports multi-indexing.** Enforcing ordering on a parallel dataframe system like Modin requires non-trivial effort that involves decoupling of the logical and physical representation of the data, enabling the order to be lazily kept up-to-date, but eagerly computed based on user needs (See Section 4.2 in `our recent paper `_). Modin abstracts away the physical representation of the data and provides an ordered view that is consistent with user's expectations.\n\nCompatibility with Computational Frameworks\n-------------------------------------------\n\n**DaskDF and Koalas are meant to be run on Dask and Spark respectively.** They are highly tuned to the corresponding frameworks, and cannot be ported to other computational frameworks.\n\n**Modin's highly modular design is architected to run on a variety of systems, and support a variety of APIs.** The goal for the extensible design is that users can take the same notebook or script and seamlessly move between different clusters and environments, with Modin being able to support the pandas API on your preexisting infrastructure. Currently, Modin support running on Dask's compute engine in addition to Ray. The modular design makes it easier for developers to different execution engines or compile to different memory formats. Modin can run on a Dask cluster in the same way that DaskDF can, but they differ in the ways described above. In addition, Modin is continually expanding to support popular data processing APIs (SQL in addition to pandas, among other DSLs for data processing) while leveraging the same underlying execution framework. Modin's flexible architecture also means that as the `pandas API continues to evolve `_, Modin can quickly move towards supporting new versions of the pandas API.\n\n.. figure:: ../../img/performance-all-supported.svg\n :align: center\n :alt: Scalability of operators supported by Modin and other systems\n :width: 95%\n\nPerformance Comparison\n----------------------\n\n**On operations supported by all systems, Modin provides substantial speedups.** Thanks to its optimized design, Modin is able to take advantage of multiple cores relative to both Koalas and DaskDF to efficiently execute pandas operations. It is notable that Koalas is often slower than pandas, due to the overhead of Spark. \n\n.. figure:: ../../img/performance-not-all-supported.svg\n :align: center\n :alt: Scalability of operators supported by Modin but not by other systems\n\n**Modin provides substantial speedups even on operators not supported by other systems.** Thanks to its flexible partitioning schemes that enable it to support the vast majority of pandas operations — be it row, column, or cell-oriented - Modin provides benefits on operations such as ``join``, ``median``, and ``infer_types``. While Koalas performs ``join`` slower than Pandas, Dask failed to support ``join`` on more than 20M rows, likely due poor support for `shuffles `_. Details of the benchmark and additional join experiments can be found in `our paper `_.\n\n.. _documentation: http://docs.dask.org/en/latest/DataFrame.html#design.\n.. _Modin's documentation: https://modin.readthedocs.io/en/latest/development/architecture.html\n"
},
{
"path": "docs/getting_started/why_modin/out_of_core.rst",
"content": "Out-of-memory data with Modin\n=============================\n\n.. note::\n | *Estimated Reading Time: 10 minutes*\n \nWhen using pandas, you might run into a memory error if you are working with large datasets that cannot fit in memory or perform certain memory-intensive operations (e.g., joins). \n\nModin solves this problem by spilling over to disk, in other words, it uses your disk as an overflow for memory so that you can work with datasets that are too large to fit in memory. By default, Modin leverages out-of-core methods to handle datasets that don't fit in memory for both Ray and Dask engines.\n\n.. note::\n Object spilling is disabled in a multi-node Ray cluster by default. To enable object spilling\n use `Ray instruction `_.\n\n\nMotivating Example: Memory error with pandas\n--------------------------------------------\n\npandas makes use of in-memory data structures to store and operate on data, which means that if you have a dataset that is too large to fit in memory, it will cause an error on pandas. As an example, let's creates a 80GB DataFrame by appending together 40 different 2GB DataFrames. \n\n.. code-block:: python\n\n import pandas\n import numpy as np\n df = pandas.concat([pandas.DataFrame(np.random.randint(0, 100, size=(2**20, 2**8))) for _ in range(40)]) # Memory Error!\n\nWhen we run this on a laptop with 32GB of RAM, pandas will run out of memory and throw an error (e.g., :code:`MemoryError` , :code:`Killed: 9`). \n\nThe `pandas documentation `_ has a great section on recommendations for scaling your analysis to these larger datasets. However, this generally involves loading in less data or rewriting your pandas code to process the data in smaller chunks. \n\nOperating on out-of-memory data with Modin\n------------------------------------------\n\nIn order to work with data that exceeds memory constraints, you can use Modin to handle these large datasets.\n\n.. code-block:: python\n\n import modin.pandas as pd\n import numpy as np\n df = pd.concat([pd.DataFrame(np.random.randint(0, 100, size=(2**20, 2**8))) for _ in range(40)]) # 40x2GB frames -- Working!\n df.info()\n\nNot only does Modin let you work with datasets that are too large to fit in memory, we can perform various operations on them without worrying about memory constraints. \n\nAdvanced: Configuring out-of-core settings\n------------------------------------------\n\n.. why would you want to disable out of core?\n\nBy default, out-of-core functionality is enabled by the compute engine selected. \nTo disable it, start your preferred compute engine with the appropriate arguments. For example:\n\n.. code-block:: python\n\n import modin.pandas as pd\n import ray\n\n ray.init(_plasma_directory=\"/tmp\") # setting to disable out of core in Ray\n df = pd.read_csv(\"some.csv\")\n\nIf you are using Dask, you have to modify local configuration files. Visit the\nDask documentation_ on object spilling for more details.\n\n\n.. _documentation: https://distributed.dask.org/en/latest/worker.html#memory-management\n"
},
{
"path": "docs/getting_started/why_modin/pandas.rst",
"content": "How does Modin differ from pandas?\n==================================\n\n.. note:: \n | *Estimated Reading Time: 10 minutes*\n\nIn the earlier tutorials, we have seen how Modin can be used to speed up pandas workflows. Here, we discuss at a high level how Modin works, in particular, how Modin's dataframe implementation differs from pandas. \n\nScalablity of implementation\n----------------------------\n\nModin exposes the pandas API through ``modin.pandas``, but it does not inherit the same pitfalls and design decisions that make it difficult to scale. \nThe pandas implementation is inherently single-threaded. This means that only one of\nyour CPU cores can be utilized at any given time. In a laptop, it would look something\nlike this with pandas:\n\n.. image:: /img/pandas_multicore.png\n :alt: pandas is single threaded!\n :align: center\n :scale: 80%\n\nHowever, Modin's implementation enables you to use all of the cores on your machine, or\nall of the cores in an entire cluster. On a laptop, it will look something like this:\n\n.. image:: /img/modin_multicore.png\n :alt: modin uses all of the cores!\n :align: center\n :scale: 80%\n\nThe additional utilization leads to improved performance, however if you want to scale\nto an entire cluster, Modin suddenly looks something like this:\n\n.. image:: /img/modin_cluster.png\n :alt: modin works on a cluster too!\n :align: center\n :scale: 30%\n\nModin is able to efficiently make use of all of the hardware available to it!\n\nMemory usage and immutability\n-----------------------------\n\nThe pandas API contains many cases of \"inplace\" updates, which are known to be\ncontroversial. This is due in part to the way pandas manages memory: the user may\nthink they are saving memory, but pandas is usually copying the data whether an\noperation was inplace or not.\n\nModin allows for inplace semantics, but the underlying data structures within Modin's\nimplementation are immutable, unlike pandas. This immutability gives Modin the ability\nto internally chain operators and better manage memory layouts, because they will not\nbe changed. This leads to improvements over pandas in memory usage in many common cases,\ndue to the ability to share common memory blocks among all dataframes.\n\nModin provides the inplace semantics by having a mutable pointer to the immutable\ninternal Modin dataframe. This pointer can change, but the underlying data cannot, so\nwhen an inplace update is triggered, Modin will treat it as if it were not inplace and\njust update the pointer to the resulting Modin dataframe.\n\nAPI vs implementation\n---------------------\n\nIt is well known that the pandas API contains many duplicate ways of performing the same\noperation. Modin instead enforces that any one behavior have one and only one\nimplementation internally. This guarantee enables Modin to focus on and optimize a\nsmaller code footprint while still guaranteeing that it covers the entire pandas API.\nModin has an internal algebra, which is roughly 15 operators, narrowed down from the\noriginal >200 that exist in pandas. The algebra is grounded in both practical and\ntheoretical work. Learn more in our `VLDB 2020 paper`_. More information about this\nalgebra can be found in the :doc:`architecture ` documentation.\n\n.. _VLDB 2020 paper: https://arxiv.org/abs/2001.00888\n"
},
{
"path": "docs/getting_started/why_modin/why_modin.rst",
"content": "Why Modin?\n==========\n\nIn this section, we explain the design and motivation behind Modin and why you should use Modin to scale up your pandas workflows. We first describe the architectural differences between pandas and Modin. Then we describe how Modin can also help resolve out-of-memory issues common to pandas. Finally, we look at the key differences between Modin and other distributed dataframe libraries. \n\n.. toctree::\n :maxdepth: 4\n \n pandas\n out_of_core\n modin_vs_dask_vs_koalas\n\nModin is built on many years of research and development at UC Berkeley. For more information on how this works underneath the hoods, check out our publications in this space:\n\n- `Flexible Rule-Based Decomposition and Metadata Independence in Modin `_ (VLDB 2021)\n- `Enhancing the Interactivity of Dataframe Queries by Leveraging Think Time `_ (IEEE Data Eng 2021)\n- `Dataframe Systems: Theory, Architecture, and Implementation `_ (PhD Dissertation 2021)\n- `Scaling Data Science does not mean Scaling Machines `_ (CIDR 2021)\n- `Towards Scalable Dataframe Systems `_ (VLDB 2020)\n"
},
{
"path": "docs/index.rst",
"content": ".. image:: img/MODIN_ver2_hrz.png\n :width: 400px\n :alt: modin logo\n :align: center\n\n====\n\n.. toctree::\n :hidden:\n\n getting_started/quickstart\n usage_guide/index\n supported_apis/index\n development/index\n ecosystem\n contact\n\n.. raw:: html\n\n
To use Modin, replace the pandas import:
\n\n.. figure:: img/Modin_Pandas_Import.gif\n :align: center\n\nScale your pandas workflow by changing a single line of code\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin uses Ray_, Dask_ or Unidist_ to provide an effortless way to speed up your pandas notebooks,\nscripts, and libraries. Unlike other distributed DataFrame libraries, Modin provides\nseamless integration and compatibility with existing pandas code. Even using the\nDataFrame constructor is identical.\n\n.. code-block:: python\n\n import modin.pandas as pd\n import numpy as np\n\n frame_data = np.random.randint(0, 100, size=(2**10, 2**8))\n df = pd.DataFrame(frame_data)\n\nIt is not necessary to know in advance the available hardware resources in order to use Modin.\nAdditionally, it is not necessary to specify how to distribute or place data.\nModin acts as a drop-in replacement for pandas, which means that you can continue using your previous\npandas notebooks, *unchanged*, while experiencing a considerable speedup thanks to Modin, even on a single\nmachine. Once you've changed your import statement, you’re ready to use Modin just like\nyou would pandas.\n\nInstallation and choosing your compute engine\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin can be installed from PyPI:\n\n.. code-block:: bash\n\n pip install modin\n\n\nIf you don't have Ray_, Dask_ or Unidist_ installed, you will need to install Modin with one\nof the targets:\n\n.. code-block:: bash\n\n pip install \"modin[ray]\" # Install Modin dependencies and Ray to run on Ray\n pip install \"modin[dask]\" # Install Modin dependencies and Dask to run on Dask\n pip install \"modin[mpi]\" # Install Modin dependencies and MPI to run on MPI through unidist\n pip install \"modin[all]\" # Install all of the above\n\nModin will automatically detect which engine you have installed and use that for\nscheduling computation!\n\nIf you want to choose a specific compute engine to run on, you can set the environment\nvariable ``MODIN_ENGINE`` and Modin will do computation with that engine:\n\n.. code-block:: bash\n\n export MODIN_ENGINE=ray # Modin will use Ray\n export MODIN_ENGINE=dask # Modin will use Dask\n export MODIN_ENGINE=unidist # Modin will use Unidist\n\nIf you want to choose the Unidist engine, you should set the additional environment \nvariable ``UNIDIST_BACKEND``, because currently Modin only supports MPI through unidist:\n\n.. code-block:: bash\n\n export UNIDIST_BACKEND=mpi # Unidist will use MPI backend\n\nThis can also be done within a notebook/interpreter before you import Modin:\n\n.. code-block:: python\n\n import os\n\n os.environ[\"MODIN_ENGINE\"] = \"ray\" # Modin will use Ray\n os.environ[\"MODIN_ENGINE\"] = \"dask\" # Modin will use Dask\n\n os.environ[\"MODIN_ENGINE\"] = \"unidist\" # Modin will use Unidist\n os.environ[\"UNIDIST_BACKEND\"] = \"mpi\" # Unidist will use MPI backend\n\n import modin.pandas as pd\n\nFaster pandas, even on your laptop\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n.. image:: img/read_csv_benchmark.png\n :height: 350px\n :width: 300px\n :alt: Plot of read_csv\n :align: right\n\nThe ``modin.pandas`` `DataFrame`_ is an extremely light-weight parallel DataFrame. Modin\ntransparently distributes the data and computation so that all you need to do is\ncontinue using the pandas API as you were before installing Modin. Unlike other parallel\nDataFrame systems, Modin is an extremely light-weight, robust DataFrame. Because it is so\nlight-weight, Modin provides speed-ups of up to 4x on a laptop with 4 physical cores.\n\nIn pandas, you are only able to use one core at a time when you are doing computation of\nany kind. With Modin, you are able to use all of the CPU cores on your machine. Even in\n``read_csv``, we see large gains by efficiently distributing the work across your entire\nmachine.\n\n.. code-block:: python\n\n import modin.pandas as pd\n\n df = pd.read_csv(\"my_dataset.csv\")\n\nModin is a DataFrame for datasets from 1MB to 1TB+\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nWe have focused heavily on bridging the solutions between DataFrames for small data\n(e.g. pandas) and large data. Often data scientists require different tools for doing\nthe same thing on different sizes of data. The DataFrame solutions that exist for 1MB do\nnot scale to 1TB+, and the overheads of the solutions for 1TB+ are too costly for\ndatasets in the 1KB range. With Modin, because of its light-weight, robust, and scalable\nnature, you get a fast DataFrame at 1MB and 1TB+.\n\n**Modin is currently under active development. Requests and contributions are welcome!**\n\nIf you are interested in learning more about Modin, please check out the :doc:`Getting Started`\nguide then refer to the :doc:`Developer Documentation` section,\nwhere you can find system architecture, internal implementation details, and other useful information.\nAlso check out the `Github`_ to view open issues and make contributions.\n\n.. _Dataframe: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html\n.. _Ray: https://github.com/ray-project/ray/\n.. _Dask: https://dask.org/\n.. _Unidist: https://github.com/modin-project/unidist/\n.. _Github: https://github.com/modin-project/modin\n"
},
{
"path": "docs/release-procedure.md",
"content": "## Versioning\n\n### Patch release\n\nModin uses semantic versioning. So when doing a patch release, please make a separate branch\noff the previous release tag, and `git cherry-pick` **only** the commits we would like to have in our\npatch release (assuming previous release was versioned `X.Y.Z`):\n\n git checkout -b release-X.Y.Z+1 X.Y.Z\n\n### Major and Minor releases\n\nA major (`xx.0.0`) or minor (`0.xx.0`) release could be done by branching from `main`:\n\n git checkout -b release-X.Y.0 main\n\n## Preparing the release\n\nBefore continuing with the release process, make sure that automated CI which runs on each commit passed successfully with the commit you deem as a \"release candidate\".\n\nModin follows the \"no push\" logic, which is _only_ circumvented for cherry-picked commits,\nas reviewing them again would not add a lot of value but would add lots of excess work.\n\nHence non-cherry-pick commits should happen in a separate branch in your own fork, and\nbe delivered to the release branch by using a PR.\n\nNote that Modin uses fully signed commits, so you have to have GPG keys set up. See [onboarding instructions](https://github.com/modin-project/modin/blob/main/contributing/contributing.md) on where to get started.\n\nTo update Modin version, follow the instructions below.\n\n### Preparing the repo for a Major or Minor Version\n\n**Note**: this should be done in your fork of Modin.\n\nFirst, update your fork of Modin's main with the main repo's main. From your main, create a new\nbranch called `release-X.Y.0` off of main. Create an empty commit in your new branch with the message\n`Release version X.Y.0`. Make sure to sign this commit with both your GPG key\nand with the conventional `git commit -s` (so `git commit -s -S`). Open a PR against modin-project/modin with just this commit.\n\n### Preparing the repo for a Patch Version\n\n**Note**: this should be done in the original Modin repository (in `upstream`) .\n\nFirst, you must create a new branch in the upstream (main modin-project/modin) repo for the new release.\nThis branch must be named `release-X.Y.Z`, and should be made off of the tag for the last release. To\ndo this, use `git checkout -b release-X.Y.Z+1 X.Y.Z` to create the branch for the new release. Once\nthis branch has been created, cherry-pick the commits that will go into this release, and push this\nbranch to `upstream`.\n\n**Note**: now you must switch to your fork of Modin.\n\nFrom your fork of Modin, fetch the upstream repo, and checkout the release branch you made above.\nFrom this release branch, create a new branch.\n\nFrom your new branch, edit the `README.md` so that the PyPi badge will\npoint to the badge for this specific version (instead of latest) and so that the docs link will point\nto the docs for this specific version (rather than latest).\n\nOnce the badges have been edited, create a commit, the same as for a major or minor version,\nwith the message `Release version X.Y.Z`, and make sure to sign it with both your GPG key, and the\ntraditional git sign-off. Create a PR using your branch against the `release-X.Y.Z` branch in the\noriginal Modin repo.\n\n### Tag commit\n\nAfter the PR has been merged, clone a clean copy of the Modin repo from the modin-project organization.\nYou now need to tag the commit that corresponds to the above PR with the appropriate tag for this release.\n\n**Note**: from now on you work on the `main` branch (in `upstream`) for a major or minor release,\nor the `release-X.Y.Z` branch (in `upstream`) for a patch release.\n\n git tag -as X.Y.Z\n\n * Use `scripts/release.py` to draft the release notes (might be as simple as `python scripts/release.py notes > draft.txt`)\n * If you're experiencing [rate limiting by GitHub](https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting) during username resolving, pass a token via `--token` option to the script\n * Fill in the placeholder for summary of the release\n * Please look into PR sections and split them if necessary into smaller but better fitting ones, as the script only categorizes by prefix (`FIX-`, `TEST-`, etc.)\n * Make sure to correctly resolve contributors whom script failed to transform to GitHub usernames if there are any!\n * Include release documentation in the annotation and make sure it is signed.\n * Push the tag to `main` or `release-X.Y.Z` branch: `git push upstream X.Y.Z`\n * If you're re-pushing a tag (beware! you shouldn't be doing that, no, _really_!), you can remove remote tag and push a local one by `git push upstream :refs/tags/X.Y.Z`\n\n\n### Build wheel:\n\n**Note**: This should be done from your clean clone of the `upstream` Modin\nrepository from the modin-project organization, where you made the release tag.\n\n```bash\n# Install/update tools\npip install --upgrade build twine\n# Build a pure Python wheel.\npython3 setup.py sdist bdist_wheel\n```\n\nYou may see the wheel in the `dist` folder: `ls -l dist`. Make sure the version is correct.\nAlso make sure there is a `tar` file that contains the source.\n\n### Upload wheels:\n\nMake sure you have an active PyPI account which has write access to Modin pypi repo, and make sure you have a pypi token set up.\n\nUse `twine` to upload wheels:\n\n```bash\ntwine upload dist/*\n```\n\nWhen asked for account, provide `__token__` (exactly as stated), when asked for password, present your token from pypi.\n\n### Check with `pip install`:\n\nRun `pip install -U \"modin[all]\"` on Linux, Mac, and Windows systems in a new environment\nto test that the wheels were uploaded correctly.\n\n## Make Github and conda-forge release\n\n### Github\n\nOnce the tag has been published, we need to make the release on GitHub. Go to the\n[Release page](https://github.com/modin-project/modin/releases), and click on `Draft a new release`.\nChoose the tag you made above from the dropdown menu, and copy paste the name of the release \nin the `Release title` box. Next, copy paste the release notes from above into the box labelled\n`Describe this release`. This will ensure that the release notes on GitHub are Markdown formatted.\n\nDouble check that everything looks good by clicking `Preview`, and then hit the green `Publish release`\nbutton!\n\n### Conda-forge\n\nConda-forge has a bot which watches for new releases of software packaged through it,\nand in case of Modin it waits either for Github releases or for tags and then makes\na new automatic PR with version increment.\n\nYou should watch for that PR and, fixing any issues if there are some, merge it\nto make new Modin release appear in `conda-forge` channel. For detailed instructions\non how to ensure the PR passes CI and is merge-able, check out [the how-to page in the modin-feedstock repo](https://github.com/conda-forge/modin-feedstock/blob/main/HOWTO.md)!\n\n## Publicize Release\nOnce the release has been finalized, make sure to post an announcement\nin the #general channel of the public Modin Slack!\n"
},
{
"path": "docs/release_notes/release_notes-0.14.0.rst",
"content": ":orphan:\n\nModin 0.14.0\n\nKey Features and Updates\n------------------------\n\n* Stability and Bugfixes\n * FIX-#4058: Allow pickling empty dataframes and series (#4095)\n * FIX-#4136: Fix exercise_3.ipynb example notebook (#4137)\n * FIX-#4105: Fix names of pandas options to avoid `OptionError` (#4109)\n * FIX-#3417: Fix read_csv with skiprows and header parameters (#3419)\n * FIX-#4142: Fix OmniSci enabling (#4146)\n * FIX-#4162: Use `skipif` instead of `skip` for compatibility with pytest 7.0 (#4163)\n * FIX-#4158: Do not print OmniSci logs to stdout by default (#4159)\n * FIX-#4177: Support read_feather from pathlike objects (#4177)\n * FIX-#4234: Upgrade pandas to 1.4.1 (#4235)\n * FIX-#3368: support unsigned integers in OmniSci backend (#4256)\n * FIX-#4057: Allow reading an empty parquet file (#4075)\n * FIX-#3884: Fix read_excel() dropping empty rows (#4161)\n * FIX-#4257: Fix Categorical() for scalar categories (#4258)\n * FIX-#4300: Fix Modin Categorical column dtype categories (#4276)\n * FIX-#4208: Fix lazy metadata update for `PandasDataFrame.from_labels` (#4209)\n * FIX-#3981, FIX-#3801, FIX-#4149: Stop broadcasting scalars to set items (#4160)\n * FIX-#4185: Fix rolling across column partitions (#4262)\n * FIX-#4303: Fix the syntax error in reading from postgres (#4304)\n * FIX-#4308: Add proper error handling in df.set_index (#4309)\n * FIX-#4056: Allow an empty parse_date list in `read_csv_glob` (#4074)\n * FIX-#4312: Fix constructing categorical frame with duplicate column names (#4313).\n * FIX-#4314: Allow passing a series of dtypes to astype (#4318)\n * FIX-#4310: Handle lists of lists of ints in read_csv_glob (#4319)\n* Performance enhancements\n * FIX-#4138, FIX-#4009: remove redundant sorting in the internal '.mask()' flow (#4140)\n * FIX-#4183: Stop shallow copies from creating global shared state. (#4184)\n* Benchmarking enhancements\n * FIX-#4221: add `wait` method for `PandasOnRayDataframeColumnPartition` class (#4231)\n* Refactor Codebase\n * REFACTOR-#3990: remove code duplication in `PandasDataframePartition` hierarchy (#3991)\n * REFACTOR-#4229: remove unused `dask_client` global variable in `modin\\pandas\\__init__.py` (#4230)\n * REFACTOR-#3997: remove code duplication for `broadcast_apply` method (#3996)\n * REFACTOR-#3994: remove code duplication for `get_indices` function (#3995)\n * REFACTOR-#4331: remove code duplication for `to_pandas`, `to_numpy` functions in `QueryCompiler` hierarchy (#4332)\n * REFACTOR-#4213: Refactor `modin/examples/tutorial/` directory (#4214)\n * REFACTOR-#4206: add assert check into `__init__` method of `PandasOnDaskDataframePartition` class (#4207)\n * REFACTOR-#3900: add flake8-no-implicit-concat plugin and refactor flake8 error codes (#3901)\n * REFACTOR-#4093: Refactor base to be smaller (#4220)\n * REFACTOR-#4047: Rename `cluster` directory to `cloud` in examples (#4212)\n * REFACTOR-#3853: interacting with Dask interface through `DaskWrapper` class (#3854)\n * REFACTOR-#4322: Move is_reduce_fn outside of groupby_agg (#4323)\n* Pandas API implementations and improvements\n * FEAT-#3603: add experimental `read_custom_text` function that can read custom line-by-line text files (#3441)\n * FEAT-#979: Enable reading from SQL server (#4279)\n* OmniSci enhancements\n *\n* XGBoost enhancements\n *\n* Developer API enhancements\n * FEAT-#4245: Define base interface for dataframe exchange protocol (#4246)\n * FEAT-#4244: Implement dataframe exchange protocol for HdkOnNative execution (#4269)\n * FEAT-#4144: Implement dataframe exchange protocol for pandas storage format (#4150)\n * FEAT-#4342: Support `from_dataframe`` for pandas storage format (#4343)\n* Update testing suite\n * TEST-#3628: Report coverage data for `test-internals` CI job (#4198)\n * TEST-#3938: Test tutorial notebooks in CI (#4145)\n * TEST-#4153: Fix condition of running lint-commit and set of CI triggers (#4156)\n * TEST-#4201: Add read_parquet, explode, tail, and various arithmetic functions to asv_bench (#4203)\n* Documentation improvements\n * DOCS-#4077: Add release notes template to docs folder (#4078)\n * DOCS-#4082: Add pdf/epub/htmlzip formats for doc builds (#4083)\n * DOCS-#4168: Fix rendering the examples on troubleshooting page (#4169)\n * DOCS-#4151: Add info in troubleshooting page related to Dask engine usage (#4152)\n * DOCS-#4172: Refresh Intel Distribution of Modin paragraph (#4175)\n * DOCS-#4173: Mention strict channel priority in conda install section (#4178)\n * DOCS-#4176: Update OmniSci usage section (#4192)\n * DOCS-#4027: Add GIF images and chart to Modin README demonstrating speedups (#4232)\n * DOCS-#3954: Add Dask example notebooks (#4139)\n * DOCS-#4272: Add bar chart comparisons to quick start guide (#4277)\n * DOCS-#3953: Add docs and notebook examples on running Modin with OmniSci (#4001)\n * DOCS-#4280: Change links in jupyter notebooks (#4281)\n * DOCS-#4290: Add changes for OmniSci notebooks (#4291)\n * DOCS-#4241: Update warnings and docs regarding defaulting to pandas (#4242)\n * DOCS-#3099: Fix `BasePandasDataSet` docstrings warnings (#4333)\n * DOCS-#4339: Reformat I/O functions docstrings (#4341)\n * DOCS-#4336: Reformat general utilities docstrings (#4338)\n* Dependencies\n * FIX-#4113, FIX-#4116, FIX-#4115: Apply new `black` formatting, fix pydocstyle check and readthedocs build (#4114)\n * TEST-#3227: Use codecov github action instead of bash form in GA workflows (#3226)\n * FIX-#4115: Unpin `pip` in readthedocs deps list (#4170)\n * TEST-#4217: Pin `Dask<2022.2.0` as a temporary fix of CI (#4218)\n\nContributors\n------------\n\n@prutskov\n@amyskov\n@paulovn\n@anmyachev\n@YarShev\n@RehanSD\n@devin-petersohn\n@dchigarev\n@Garra1980\n@mvashishtha\n@naren-ponder\n@jeffreykennethli\n@dorisjlee\n@Rubtsowa\n"
},
{
"path": "docs/release_notes/release_notes-0.15.0.rst",
"content": ":orphan:\n\nModin 0.15.0\n\nKey Features and Updates\n------------------------\n\n* Stability and Bugfixes\n * FIX-#4376: Upgrade pandas to 1.4.2 (#4377)\n * FIX-#3615: Relax some deps in development env (#4365)\n * FIX-#4370: Fix broken docstring links (#4375)\n * FIX-#4392: Align Modin XGBoost with xgb>=1.6 (#4393)\n * FIX-#4385: Get rid of `use-deprecated` option in `pip` (#4386)\n * FIX-#3527: Fix parquet partitioning issue causing negative row length partitions (#4368)\n * FIX-#4330: Override the memory limit to start ray 1.11.0 on Macs (#4335)\n * FIX-#4407: Align `insert` function with pandas in case of numpy array with several columns (#4408)\n * FIX-#4373: Fix invalid file path when trying `read_csv_glob` with `usecols` parameter (#4405)\n * FIX-#4394: Fix issue with multiindex metadata desync (#4395)\n * FIX-#4438: Fix `reindex` function that doesn't preserve initial index metadata (#4442)\n * FIX-#4425: Add parameters to groupby pct_change (#4429)\n * FIX-#4457: Fix `loc` in case when need reindex item (#4457)\n * FIX-#4414: Add missing f prefix on f-strings found at https://codereview.doctor (#4415)\n * FIX-#4461: Fix S3 CSV data path (#4462)\n * FIX-#4467: `drop_duplicates` no longer removes items based on index values (#4468)\n * FIX-#4449: Drain the call queue before waiting on result in benchmark mode (#4472)\n * FIX-#4518: Fix Modin Logging to report specific Modin warnings/errors (#4519)\n * FIX-#4481: Allow clipping with a Modin Series of bounds (#4486) \n * FIX-#4504: Support na_action in applymap (#4505)\n * FIX-#4503: Stop the memory logging thread after session exit (#4515)\n * FIX-#4531: Fix a makedirs race condition in to_parquet (#4533)\n * FIX-#4464: Refactor Ray utils and quick fix groupby.count failing on virtual partitions (#4490)\n * FIX-#4436: Fix to_pydatetime dtype for timezone None (#4437)\n * FIX-#4541: Fix merge_asof with non-unique right index (#4542)\n* Performance enhancements\n * FEAT-#4320: Add connectorx as an alternative engine for read_sql (#4346)\n * PERF-#4493: Use partition size caches more in Modin dataframe (#4495)\n* Benchmarking enhancements\n * FEAT-#4371: Add logging to Modin (#4372)\n * FEAT-#4501: Add RSS Memory Profiling to Modin Logging (#4502)\n * FEAT-#4524: Split Modin API and Memory log files (#4526)\n* Refactor Codebase\n * REFACTOR-#4284: use variable length unpacking when getting results from `deploy` function (#4285)\n * REFACTOR-#3642: Move PyArrow storage format usage from main feature to experimental ones (#4374)\n * REFACTOR-#4003: Delete the deprecated cloud mortgage example (#4406)\n * REFACTOR-#4513: Fix spelling mistakes in docs and docstrings (#4514)\n * REFACTOR-#4510: Align experimental and regular IO modules initializations (#4511)\n* Pandas API implementations and improvements\n *\n* OmniSci enhancements\n *\n* XGBoost enhancements\n *\n* Developer API enhancements\n * FEAT-#4359: Add __dataframe__ method to the protocol dataframe (#4360)\n* Update testing suite\n * TEST-#4363: Use Ray from pypi in CI (#4364)\n * FIX-#4422: get rid of case sensitivity for `warns_that_defaulting_to_pandas` (#4423)\n * TEST-#4426: Stop passing is_default kwarg to Modin and pandas (#4428)\n * FIX-#4439: Fix flake8 CI fail (#4440)\n * FIX-#4409: Fix `eval_insert` utility that doesn't actually check results of `insert` function (#4410)\n * TEST-#4482: Fix getitem and loc with series of bools (#4483).\n* Documentation improvements\n * DOCS-#4296: Fix docs warnings (#4297)\n * DOCS-#4388: Turn off fail_on_warning option for docs build (#4389)\n * DOCS-#4469: Say that commit messages can start with PERF (#4470).\n * DOCS-#4466: Recommend GitHub issues over bug_reports@modin.org (#4474). \n * DOCS-#4487: Recommend GitHub issues over feature_requests@modin.org (#4489).\n * DOCS-#4545: Add socials to README (#4555).\n* Dependencies\n * FIX-#4327: Update min pin for xgboost version (#4328)\n * FIX-#4383: Remove `pathlib` from deps (#4384)\n * FIX-#4390: Add `redis` to Modin dependencies (#4396)\n * FIX-#3689: Add black and flake8 into development environment files (#4480)\n * TEST-#4516: Add numpydoc to developer requirements (#4517)\n* New Features\n * FEAT-#4412: Add Batch Pipeline API to Modin (#4452)\n\nContributors\n------------\n@YarShev\n@Garra1980\n@prutskov\n@alexander3774\n@amyskov\n@wangxiaoying\n@jeffreykennethli\n@mvashishtha\n@anmyachev\n@dchigarev\n@devin-petersohn\n@jrsacher\n@orcahmlee\n@naren-ponder\n@RehanSD\n"
},
{
"path": "docs/release_notes/release_notes-0.16.0.rst",
"content": ":orphan:\n\nModin 0.16.0\n\nKey Features and Updates\n------------------------\n\n* Stability and Bugfixes\n * FIX-#4570: Replace ``np.bool`` -> ``np.bool_`` (#4571)\n * FIX-#4543: Fix `read_csv` in case skiprows=<0, []> (#4544)\n * FIX-#4059: Add cell-wise execution for binary ops, fix bin ops for empty dataframes (#4391)\n * FIX-#4589: Pin protobuf<4.0.0 to fix ray (#4590)\n * FIX-#4577: Set attribute of Modin dataframe to updated value (#4588)\n * FIX-#4411: Fix binary_op between datetime64 Series and pandas timedelta (#4592)\n * FIX-#4604: Fix `groupby` + `agg` in case when multicolumn can arise (#4642)\n * FIX-#4582: Inherit custom log layer (#4583)\n * FIX-#4639: Fix `storage_options` usage for `read_csv` and `read_csv_glob` (#4644)\n * FIX-#4593: Ensure Modin warns when setting columns via attributes (#4621)\n * FIX-#4584: Enable pdb debug when running cloud tests (#4585)\n * FIX-#4564: Workaround import issues in Ray: auto-import pandas on python start if env var is set (#4603)\n * FIX-#4641: Reindex pandas partitions in `df.describe()` (#4651)\n * FIX-#2064: Fix `iloc`/`loc` assignment when dataframe is empty (#4677)\n * FIX-#4634: Check for FrozenList as `by` in `df.groupby()` (#4667)\n * FIX-#4680: Fix `read_csv` that started defaulting to pandas again in case of reading from a buffer and when a buffer has a non-zero starting position (#4681)\n * FIX-#4491: Wait for all partitions in parallel in benchmark mode (#4656)\n * FIX-#4358: MultiIndex `loc` shouldn't drop levels for full-key lookups (#4608)\n * FIX-#4658: Expand exception handling for `read_*` functions from s3 storages (#4659)\n * FIX-#4672: Fix incorrect warning when setting `frame.index` or `frame.columns` (#4721)\n * FIX-#4686: Propagate metadata and drain call queue in unwrap_partitions (#4697)\n * FIX-#4652: Support categorical data in `from_dataframe` (#4737)\n * FIX-#4756: Correctly propagate `storage_options` in `read_parquet` (#4764)\n * FIX-#4657: Use `fsspec` for handling s3/http-like paths instead of `s3fs` (#4710)\n * FIX-#4676: drain sub-virtual-partition call queues (#4695)\n * FIX-#4782: Exclude certain non-parquet files in `read_parquet` (#4783)\n * FIX-#4808: Set dtypes correctly after column rename (#4809)\n * FIX-#4811: Apply dataframe -> not_dataframe functions to virtual partitions (#4812)\n * FIX-#4099: Use mangled column names but keep the original when building frames from arrow (#4767)\n * FIX-#4838: Bump up modin-spreadsheet to latest master (#4839)\n * FIX-#4840: Change modin-spreadsheet version for notebook requirements (#4841)\n * FIX-#4835: Handle Pathlike paths in `read_parquet` (#4837)\n * FIX-#4872: Stop checking the private ray mac memory limit (#4873)\n * FIX-#4914: `base_lengths` should be computed from `base_frame` instead of `self` in `copartition` (#4915)\n * FIX-#4848: Fix rebalancing partitions when NPartitions == 1 (#4874)\n * FIX-#4927: Fix `dtypes` computation in `dataframe.filter` (#4928)\n * FIX-#4907: Implement `radd` for Series and DataFrame (#4908)\n * FIZ-#4945: Fix `_take_2d_positional` that loses indexes due to filtering empty dataframes (#4951)\n * FIX-#4818, PERF-#4825: Fix where by using the new n-ary operator (#4820)\n * FIX-#3983: FIX-#4107: Materialize 'rowid' columns when selecting rows by position (#4834)\n * FIX-#4845: Fix KeyError from `__getitem_bool` for single row dataframes (#4845)\n * FIX-#4734: Handle Series.apply when return type is a DataFrame (#4830)\n * FIX-#4983: Set `frac` to `None` in _sample when `n=0` (#4984)\n * FIX-#4993: Return `_default_to_pandas` in `df.attrs` (#4995)\n * FIX-#5043: Fix `execute` function in ASV utils failed if `len(partitions) == 0` (#5044)\n * FIX-#4597: Refactor Partition handling of func, args, kwargs (#4715)\n * FIX-#4996: Evaluate BenchmarkMode at each function call (#4997)\n * FIX-#4022: Fixed empty data frame with index (#4910)\n * FIX-#4090: Fixed check if the index is trivial (#4936)\n * FIX-#4966: Fix `to_timedelta` to return Series instead of TimedeltaIndex (#5028)\n * FIX-#5042: Fix series __getitem__ with invalid strings (#5048)\n * FIX-#4691: Fix binary operations between virtual partitions (#5049) \n * FIX-#5045: Fix ray virtual_partition.wait with duplicate object refs (#5058)\n* Performance enhancements\n * PERF-#4182: Add cell-wise execution for binary ops, fix bin ops for empty dataframes (#4391)\n * PERF-#4288: Improve perf of `groupby.mean` for narrow data (#4591)\n * PERF-#4772: Remove `df.copy` call from `from_pandas` since it is not needed for Ray and Dask (#4781)\n * PERF-#4325: Improve perf of multi-column assignment in `__setitem__` when no new column names are assigning (#4455)\n * PERF-#3844: Improve perf of `drop` operation (#4694)\n * PERF-#4727: Improve perf of `concat` operation (#4728)\n * PERF-#4705: Improve perf of arithmetic operations between `Series` objects with shared `.index` (#4689)\n * PERF-#4703: Improve performance in accessing `ser.cat.categories`, `ser.cat.ordered`, and `ser.__array_priority__` (#4704)\n * PERF-#4305: Parallelize `read_parquet` over row groups (#4700)\n * PERF-#4773: Compute `lengths` and `widths` in `put` method of Dask partition like Ray do (#4780)\n * PERF-#4732: Avoid overwriting already-evaluated `PandasOnRayDataframePartition._length_cache` and `PandasOnRayDataframePartition._width_cache` (#4754)\n * PERF-#4862: Don't call `compute_sliced_len.remote` when `row_labels/col_labels == slice(None)` (#4863)\n * PERF-#4713: Stop overriding the ray MacOS object store size limit (#4792)\n * PERF-#4851: Compute `dtypes` for binary operations that can only return bool type and the right operand is not a Modin object (#4852)\n * PERF-#4842: `copy` should not trigger any previous computations (#4843)\n * PERF-#4849: Compute `dtypes` in `concat` also for ROW_WISE case when possible (#4850)\n * PERF-#4929: Compute `dtype` when using `Series.dt` accessor (#4930)\n * PERF-#4892: Compute `lengths` in `rebalance_partitions` when possible (#4893)\n * PERF-#4794: Compute caches in `_propagate_index_objs` (#4888)\n * PERF-#4860: `PandasDataframeAxisPartition.deploy_axis_func` should be serialized only once (#4861)\n * PERF-#4890: `PandasDataframeAxisPartition.drain` should be serialized only once (#4891)\n * PERF-#4870: Avoid index materialization in `__getattribute__` and `__getitem__` (4911)\n * PERF-#4886: Use lazy index and columns evaluation in `query` method (#4887)\n * PERF-#4866: `iloc` function that used in `partition.mask` should be serialized only once (#4901)\n * PERF-#4920: Avoid index and cache computations in `take_2d_labels_or_positional` unless they are needed (#4921)\n * PERF-#4999: don't call `apply` in virtual partition' `drain_call_queue` if `call_queue` is empty (#4975)\n * PERF-#4268: Implement partition-parallel __getitem__ for bool Series masks (#4753)\n * PERF-#5017: `reset_index` shouldn't trigger index materialization if possible (#5018)\n * PERF-#4963: Use partition `width/length` methods instead of `_compute_axis_labels_and_lengths` if index is already known (#4964)\n * PERF-#4940: Optimize categorical dtype check in `concatenate` (#4953)\n* Benchmarking enhancements\n * TEST-#5066: Add outer join case for `TimeConcat` benchmark (#5067)\n * TEST-#5083: Add `merge` op with categorical data (#5084)\n * FEAT-#4706: Add Modin ClassLogger to PandasDataframePartitionManager (#4707)\n * TEST-#5014: Simplify adding new ASV benchmarks (#5015)\n * TEST-#5064: Update `TimeConcat` benchmark with new parameter `ignore_index` (#5065)\n * PERF-#4944: Avoid default_to_pandas in ``Series.cat.codes``, ``Series.dt.tz``, and ``Series.dt.to_pytimedelta`` (#4833)\n * TEST-#5068: Add binary op benchmark for Series (#5069)\n* Refactor Codebase\n * REFACTOR-#4530: Standardize access to physical data in partitions (#4563)\n * REFACTOR-#4534: Replace logging meta class with class decorator (#4535)\n * REFACTOR-#4708: Delete combine dtypes (#4709)\n * REFACTOR-#4629: Add type annotations to modin/config (#4685)\n * REFACTOR-#4717: Improve PartitionMgr.get_indices() usage (#4718)\n * REFACTOR-#4730: make Indexer immutable (#4731)\n * REFACTOR-#4774: remove `_build_treereduce_func` call from `_compute_dtypes` (#4775)\n * REFACTOR-#4750: Delete BaseDataframeAxisPartition.shuffle (#4751)\n * REFACTOR-#4722: Stop suppressing undefined name lint (#4723)\n * REFACTOR-#4832: unify `split_result_of_axis_func_pandas` (#4831)\n * REFACTOR-#4796: Introduce constant for __reduced__ column name (#4799)\n * REFACTOR-#4000: Remove code duplication for `PandasOnRayDataframePartitionManager` (#4895)\n * REFACTOR-#3780: Remove code duplication for `PandasOnDaskDataframe` (#3781)\n * REFACTOR-#4530: Unify access to physical data for any partition type (#4829)\n * REFACTOR-#4978: Align `modin/core/execution/dask/common/__init__.py` with `modin/core/execution/ray/common/__init__.py` (#4979)\n * REFACTOR-#4949: Remove code duplication in `default2pandas/dataframe.py` and `default2pandas/any.py` (#4950)\n * REFACTOR-#4976: Rename `RayTask` to `RayWrapper` in accordance with Dask (#4977)\n * REFACTOR-#4885: De-duplicated take_2d_labels_or_positional methods (#4883)\n * REFACTOR-#5005: Use `finalize` method instead of list comprehension + `drain_call_queue` (#5006)\n * REFACTOR-#5001: Remove `jenkins` stuff (#5002)\n * REFACTOR-#5026: Change exception names to simplify grepping (#5027)\n * REFACTOR-#4970: Rewrite base implementations of a partition' `width/length` (#4971) \n * REFACTOR-#4942: Remove `call` method in favor of `register` due to duplication (4943)\n * REFACTOR-#4922: Helpers for take_2d_labels_or_positional (#4865)\n * REFACTOR-#5024: Make `_row_lengths` and `_column_widths` public (#5025)\n * REFACTOR-#5009: Use `RayWrapper.materialize` instead of `ray.get` (#5010)\n * REFACTOR-#4755: Rewrite Pandas version mismatch warning (#4965)\n * REFACTOR-#5012: Add mypy checks for singleton files in base modin directory (#5013)\n * REFACTOR-#5038: Remove unnecessary _method argument from resamplers (#5039)\n * REFACTOR-#5081: Remove `c323f7fe385011ed849300155de07645.db` file (#5082)\n* Pandas API implementations and improvements\n * FEAT-#4670: Implement convert_dtypes by mapping across partitions (#4671)\n* OmniSci enhancements\n * FEAT-#4913: Enabling pyhdk\n* XGBoost enhancements\n *\n* Developer API enhancements\n *\n* Update testing suite\n * TEST-#4508: Reduce test_partition_api pytest threads to deflake it (#4551)\n * TEST-#4550: Use much less data in test_partition_api (#4554)\n * TEST-#4610: Remove explicit installation of `black`/`flake8` for omnisci ci-notebooks (#4609)\n * TEST-#2564: Add caching and use mamba for conda setups in GH (#4607)\n * TEST-#4557: Delete multiindex sorts instead of xfailing (#4559)\n * TEST-#4698: Stop passing invalid storage_options param (#4699)\n * TEST-#4745: Pin flake8 to <5 to workaround installation conflict (#4752)\n * TEST-#4875: XFail tests failing due to file gone missing (#4876)\n * TEST-#4879: Use pandas `ensure_clean()` in place of `io_tests_data` (#4881)\n * TEST-#4562: Use local Ray cluster in CI to resolve flaky `test-compat-win` (#5007)\n * TEST-#5040: Rework test_series using eval_general() (#5041)\n * TEST-#5050: Add black to pre-commit hook (#5051)\n* Documentation improvements\n * DOCS-#4552: Change default sphinx language to en to fix sphinx >= 5.0.0 build (#4553)\n * DOCS-#4628: Add to_parquet partial support notes (#4648)\n * DOCS-#4668: Set light theme for readthedocs page, remove theme switcher (#4669)\n * DOCS-#4748: Apply the Triage label to new issues (#4749)\n * DOCS-#4790: Give all templates issue type and triage labels (#4791)\n * DOCS-#4521: Document how to benchmark modin (#5020)\n* Dependencies\n * FEAT-#4598: Add support for pandas 1.4.3 (#4599)\n * FEAT-#4619: Integrate mypy static type checking (#4620)\n * FEAT-#4202: Allow dask past 2022.2.0 (#4769)\n * FEAT-#4925: Upgrade pandas to 1.4.4 (#4926)\n * TEST-#4998: Add flake8 plugins to dev requirements (#5000)\n* New Features\n * FEAT-4463: Add experimental fuzzydata integration for testing against a randomized dataframe workflow (#4556)\n * FEAT-#4419: Extend virtual partitioning API to pandas on Dask (#4420)\n * FEAT-#4147: Add partial compatibility with Python 3.6 and pandas 1.1 (#4301)\n * FEAT-#4569: Add error message when `read_` function defaults to pandas (#4647)\n * FEAT-#4725: Make index and columns lazy in Modin DataFrame (#4726)\n * FEAT-#4664: Finalize compatibility support for Python 3.6 (#4800)\n * FEAT-#4746: Sync interchange protocol with recent API changes (#4763)\n * FEAT-#4733: Support fastparquet as engine for `read_parquet` (#4807)\n * FEAT-#4766: Support fsspec URLs in `read_csv` and `read_csv_glob` (#4898)\n * FEAT-#4827: Implement `infer_types` dataframe algebra operator (#4871)\n * FEAT-#4989: Switch pandas version to 1.5 (#5037)\n\nContributors\n------------\n@mvashishtha\n@NickCrews\n@prutskov\n@vnlitvinov\n@pyrito\n@suhailrehman\n@RehanSD\n@helmeleegy\n@anmyachev\n@d33bs\n@noloerino\n@devin-petersohn\n@YarShev\n@naren-ponder\n@jbrockmendel\n@ienkovich\n@Garra1980\n@Billy2551\n"
},
{
"path": "docs/release_notes/release_notes-template.rst",
"content": ":orphan:\n\nModin X.X.X\n\nKey Features and Updates\n------------------------\n\n* Stability and Bugfixes\n *\n* Performance enhancements\n *\n* Benchmarking enhancements\n *\n* Refactor Codebase\n *\n* Pandas API implementations and improvements\n *\n* HDK enhancements\n *\n* XGBoost enhancements\n *\n* Developer API enhancements\n *\n* Update testing suite\n *\n* Documentation improvements\n *\n* Dependencies\n *\n* New Features\n\nContributors\n------------\n\n"
},
{
"path": "docs/requirements-doc.txt",
"content": "# install current modin checkout to bring all required dependencies\n.[all]\n# now install some more optional dependencies\ncolorama\nclick\nflatbuffers\nfuncsigs\nmock\nopencv-python\npydata_sphinx_theme\npyyaml\nrecommonmark\nsphinx<6.0.0\nsphinx-click\nray>=2.10.0,<3\n# Override to latest version of modin-spreadsheet\ngit+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5\nsphinxcontrib_plantuml\nsphinx-issues\n"
},
{
"path": "docs/supported_apis/dataframe_supported.rst",
"content": "``pd.DataFrame`` supported APIs\n===================================\n\nThe following table lists both implemented and not implemented methods. If you have need\nof an operation that is listed as not implemented, feel free to open an issue on the\n`GitHub repository`_, or give a thumbs up to already created issues. Contributions are\nalso welcome!\n\nThe following table is structured as follows: The first column contains the method name.\nThe second column contains link to a description of corresponding pandas method.\nThe third column is a flag for whether or not there is an implementation in Modin for\nthe method in the left column. ``Y`` stands for yes, ``N`` stands for no, ``P`` stands\nfor partial (meaning some parameters may not be supported yet), and ``D`` stands for\ndefault to pandas.\n\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| DataFrame method | pandas Doc link | Implemented? (Y/N/P/D) | Notes for Current implementation |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``T`` | `T`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``abs`` | `abs`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``add`` | `add`_ | Y | **Ray** and **Dask**: Shuffles data in operations |\n| | | | between DataFrames. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``add_prefix`` | `add_prefix`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``add_suffix`` | `add_suffix`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``agg`` / ``aggregate`` | `agg`_ / `aggregate`_ | P | - Dictionary ``func`` parameter defaults to pandas |\n| | | | - Numpy operations default to pandas |\n| | | | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``align`` | `align`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``all`` | `all`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``any`` | `any`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``apply`` | `apply`_ | Y | See ``agg`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``applymap`` | `applymap`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``asfreq`` | `asfreq`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``asof`` | `asof`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``assign`` | `assign`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``astype`` | `astype`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``at`` | `at`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``at_time`` | `at_time`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``axes`` | `axes`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``between_time`` | `between_time`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``bfill`` | `bfill`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``bool`` | `bool`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``boxplot`` | `boxplot`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``clip`` | `clip`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``combine`` | `combine`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``combine_first`` | `combine_first`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``compare`` | `compare`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``copy`` | `copy`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``corr`` | `corr`_ | P | Correlation floating point precision may slightly |\n| | | | differ from pandas. For now pearson method is |\n| | | | available only. For other methods and for |\n| | | | ``numeric_only`` defaults to pandas. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``corrwith`` | `corrwith`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``count`` | `count`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``cov`` | `cov`_ | P | Covariance floating point precision may slightly |\n| | | | differ from pandas. For ``numeric_only`` |\n| | | | defaults to pandas. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``cummax`` | `cummax`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``cummin`` | `cummin`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``cumprod`` | `cumprod`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``cumsum`` | `cumsum`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``describe`` | `describe`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``diff`` | `diff`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``div`` | `div`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``divide`` | `divide`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``dot`` | `dot`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``drop`` | `drop`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``droplevel`` | `droplevel`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``drop_duplicates`` | `drop_duplicates`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``dropna`` | `dropna`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``dtypes`` | `dtypes`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``duplicated`` | `duplicated`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``empty`` | `empty`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``eq`` | `eq`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``equals`` | `equals`_ | Y | Requires shuffle, can be further optimized |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``eval`` | `eval`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``ewm`` | `ewm`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``expanding`` | `expanding`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``explode`` | `explode`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``ffill`` | `ffill`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``fillna`` | `fillna`_ | P | ``value`` parameter of type DataFrame defaults to |\n| | | | pandas. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``filter`` | `filter`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``first`` | `first`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``first_valid_index`` | `first_valid_index`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``floordiv`` | `floordiv`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``from_dict`` | `from_dict`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``from_records`` | `from_records`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``ge`` | `ge`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``get`` | `get`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``groupby`` | `groupby`_ | Y | Not yet optimized for all operations. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``gt`` | `gt`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``head`` | `head`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``hist`` | `hist`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``iat`` | `iat`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``idxmax`` | `idxmax`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``idxmin`` | `idxmin`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``iloc`` | `iloc`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``infer_objects`` | `infer_objects`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``info`` | `info`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``insert`` | `insert`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``interpolate`` | `interpolate`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``isetitem`` | `isetitem`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``isin`` | `isin`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``isna`` | `isna`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``isnull`` | `isnull`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``items`` | `items`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``iterrows`` | `iterrows`_ | P | Modin does not parallelize iteration in Python |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``itertuples`` | `itertuples`_ | P | Modin does not parallelize iteration in Python |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``join`` | `join`_ | P | When ``on`` is set to ``right`` or ``outer`` or |\n| | | | when ``validate`` is given defaults to pandas |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``keys`` | `keys`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``kurt`` | `kurt`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``kurtosis`` | `kurtosis`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``last`` | `last`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``last_valid_index`` | `last_valid_index`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``le`` | `le`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``loc`` | `loc`_ | P | We do not support: boolean array, callable. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``lt`` | `lt`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``mask`` | `mask`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``max`` | `max`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``mean`` | `mean`_ | P | Modin defaults to pandas if given the ``level`` |\n| | | | param. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``median`` | `median`_ | P | Modin defaults to pandas if given the ``level`` |\n| | | | param. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``melt`` | `melt`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``memory_usage`` | `memory_usage`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| | | | Implemented the following cases: |\n| | | | ``left_index=True`` and ``right_index=True``, |\n| | | | ``how=left`` and ``how=inner`` for all values |\n| ``merge`` | `merge`_ | P | of parameters except ``left_index=True`` and |\n| | | | ``right_index=False`` or ``left_index=False`` |\n| | | | and ``right_index=True``. |\n| | | | Defaults to pandas otherwise. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``min`` | `min`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``mod`` | `mod`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``mode`` | `mode`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``mul`` | `mul`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``multiply`` | `multiply`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``ndim`` | `ndim`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``ne`` | `ne`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``nlargest`` | `nlargest`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``notna`` | `notna`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``notnull`` | `notnull`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``nsmallest`` | `nsmallest`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``nunique`` | `nunique`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``pct_change`` | `pct_change`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``pipe`` | `pipe`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``pivot`` | `pivot`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``pivot_table`` | `pivot_table`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``plot`` | `plot`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``pop`` | `pop`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``pow`` | `pow`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``prod`` | `prod`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``product`` | `product`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``quantile`` | `quantile`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``query`` | `query`_ | P | Local variables not yet supported |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``radd`` | `radd`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rank`` | `rank`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rdiv`` | `rdiv`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``reindex`` | `reindex`_ | Y | Shuffles data |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``reindex_like`` | `reindex_like`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rename`` | `rename`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rename_axis`` | `rename_axis`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``reorder_levels`` | `reorder_levels`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``replace`` | `replace`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``resample`` | `resample`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``reset_index`` | `reset_index`_ | P | **Ray** and **Dask**: ``D`` when ``names`` or |\n| | | | ``allow_duplicates`` is non-default |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rfloordiv`` | `rfloordiv`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rmod`` | `rmod`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rmul`` | `rmul`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rolling`` | `rolling`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``round`` | `round`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rpow`` | `rpow`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rsub`` | `rsub`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``rtruediv`` | `rtruediv`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``sample`` | `sample`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``select_dtypes`` | `select_dtypes`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``sem`` | `sem`_ | P | Modin defaults to pandas if given the ``level`` |\n| | | | param. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``set_axis`` | `set_axis`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``set_index`` | `set_index`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``shape`` | `shape`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``shift`` | `shift`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``size`` | `size`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``skew`` | `skew`_ | P | Modin defaults to pandas if given the ``level`` |\n| | | | param |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``sort_index`` | `sort_index`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``sort_values`` | `sort_values`_ | Y | Shuffles data. Order of indexes that have the |\n| | | | same sort key is not guaranteed to be the same |\n| | | | across sorts |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``sparse`` | `sparse`_ | N | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``squeeze`` | `squeeze`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``stack`` | `stack`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``std`` | `std`_ | P | Modin defaults to pandas if given the ``level`` |\n| | | | param. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``style`` | `style`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``sub`` | `sub`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``subtract`` | `subtract`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``sum`` | `sum`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``swapaxes`` | `swapaxes`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``swaplevel`` | `swaplevel`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``tail`` | `tail`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``take`` | `take`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_clipboard`` | `to_clipboard`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_csv`` | `to_csv`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_dict`` | `to_dict`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_excel`` | `to_excel`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_feather`` | `to_feather`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_gbq`` | `to_gbq`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_hdf`` | `to_hdf`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_html`` | `to_html`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_json`` | `to_json`_ | D | |\n| | | | Experimental implementation: |\n| | | | DataFrame.modin.to_json_glob |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_xml`` | `to_xml`_ | D | |\n| | | | Experimental implementation: |\n| | | | DataFrame.modin.to_xml_glob |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_latex`` | `to_latex`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_orc`` | `to_orc`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_parquet`` | `to_parquet`_ | P | **Ray/Dask/Unidist**: Parallel implementation only |\n| | | | if path parameter is a string. In that case, the |\n| | | | ``path`` parameter specifies a directory where one |\n| | | | file is written per row partition of the Modin |\n| | | | dataframe. |\n| | | | Experimental implementation: |\n| | | | DataFrame.modin.to_parquet_glob |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_period`` | `to_period`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_pickle`` | `to_pickle`_ | D | Experimental implementation: |\n| | | | DataFrame.modin.to_pickle_glob |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_records`` | `to_records`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_sql`` | `to_sql`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_stata`` | `to_stata`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_string`` | `to_string`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_timestamp`` | `to_timestamp`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``to_xarray`` | `to_xarray`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``transform`` | `transform`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``transpose`` | `transpose`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``truediv`` | `truediv`_ | Y | See ``add`` |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``truncate`` | `truncate`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``tz_convert`` | `tz_convert`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``tz_localize`` | `tz_localize`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``unstack`` | `unstack`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``update`` | `update`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``values`` | `values`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``value_counts`` | `value_counts`_ | D | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``var`` | `var`_ | P | Modin defaults to pandas if given the ``level`` |\n| | | | param. |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n| ``where`` | `where`_ | Y | |\n+----------------------------+---------------------------+------------------------+----------------------------------------------------+\n\n.. _`GitHub repository`: https://github.com/modin-project/modin/issues\n.. _`T`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.T.html#pandas.DataFrame.T\n.. _`abs`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.abs.html#pandas.DataFrame.abs\n.. _`add`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.add.html#pandas.DataFrame.add\n.. _`add_prefix`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.add_prefix.html#pandas.DataFrame.add_prefix\n.. _`add_suffix`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.add_suffix.html#pandas.DataFrame.add_suffix\n.. _`agg`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.agg.html#pandas.DataFrame.agg\n.. _`aggregate`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.aggregate.html#pandas.DataFrame.aggregate\n.. _`align`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.align.html#pandas.DataFrame.align\n.. _`all`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.all.html#pandas.DataFrame.all\n.. _`any`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.any.html#pandas.DataFrame.any\n.. _`apply`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html#pandas.DataFrame.apply\n.. _`applymap`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.applymap.html#pandas.DataFrame.applymap\n.. _`asfreq`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.asfreq.html#pandas.DataFrame.asfreq\n.. _`asof`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.asof.html#pandas.DataFrame.asof\n.. _`assign`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.assign.html#pandas.DataFrame.assign\n.. _`astype`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.astype.html#pandas.DataFrame.astype\n.. _`at`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.at.html#pandas.DataFrame.at\n.. _`at_time`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.at_time.html#pandas.DataFrame.at_time\n.. _`axes`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.axes.html#pandas.DataFrame.axes\n.. _`between_time`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.between_time.html#pandas.DataFrame.between_time\n.. _`bfill`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.bfill.html#pandas.DataFrame.bfill\n.. _`bool`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.bool.html#pandas.DataFrame.bool\n.. _`boxplot`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.boxplot.html#pandas.DataFrame.boxplot\n.. _`clip`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.clip.html#pandas.DataFrame.clip\n.. _`combine`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.combine.html#pandas.DataFrame.combine\n.. _`combine_first`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.combine_first.html#pandas.DataFrame.combine_first\n.. _`compare`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.compare.html#pandas.DataFrame.compare\n.. _`compound`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.compound.html#pandas.DataFrame.compound\n.. _`copy`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.copy.html#pandas.DataFrame.copy\n.. _`corr`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corr.html#pandas.DataFrame.corr\n.. _`corrwith`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corrwith.html#pandas.DataFrame.corrwith\n.. _`count`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.count.html#pandas.DataFrame.count\n.. _`cov`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cov.html#pandas.DataFrame.cov\n.. _`cummax`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cummax.html#pandas.DataFrame.cummax\n.. _`cummin`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cummin.html#pandas.DataFrame.cummin\n.. _`cumprod`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cumprod.html#pandas.DataFrame.cumprod\n.. _`cumsum`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cumsum.html#pandas.DataFrame.cumsum\n.. _`describe`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html#pandas.DataFrame.describe\n.. _`diff`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.diff.html#pandas.DataFrame.diff\n.. _`div`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.div.html#pandas.DataFrame.div\n.. _`divide`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.divide.html#pandas.DataFrame.divide\n.. _`dot`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dot.html#pandas.DataFrame.dot\n.. _`drop`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop.html#pandas.DataFrame.drop\n.. _`droplevel`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.droplevel.html\n.. _`drop_duplicates`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop_duplicates.html#pandas.DataFrame.drop_duplicates\n.. _`dropna`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html#pandas.DataFrame.dropna\n.. _`dtypes`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dtypes.html#pandas.DataFrame.dtypes\n.. _`duplicated`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.duplicated.html#pandas.DataFrame.duplicated\n.. _`empty`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.empty.html#pandas.DataFrame.empty\n.. _`eq`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.eq.html#pandas.DataFrame.eq\n.. _`equals`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.equals.html#pandas.DataFrame.equals\n.. _`eval`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.eval.html#pandas.DataFrame.eval\n.. _`ewm`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ewm.html#pandas.DataFrame.ewm\n.. _`expanding`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.expanding.html#pandas.DataFrame.expanding\n.. _`explode`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.explode.html#pandas-dataframe-explode\n.. _`ffill`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ffill.html#pandas.DataFrame.ffill\n.. _`fillna`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html#pandas.DataFrame.fillna\n.. _`filter`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.filter.html#pandas.DataFrame.filter\n.. _`first`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.first.html#pandas.DataFrame.first\n.. _`first_valid_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.first_valid_index.html#pandas.DataFrame.first_valid_index\n.. _`floordiv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.floordiv.html#pandas.DataFrame.floordiv\n.. _`from_dict`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.from_dict.html#pandas.DataFrame.from_dict\n.. _`from_records`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.from_records.html#pandas.DataFrame.from_records\n.. _`ge`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ge.html#pandas.DataFrame.ge\n.. _`get`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get.html#pandas.DataFrame.get\n.. _`get_dtype_counts`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get_dtype_counts.html#pandas.DataFrame.get_dtype_counts\n.. _`get_ftype_counts`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get_ftype_counts.html#pandas.DataFrame.get_ftype_counts\n.. _`get_value`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get_value.html#pandas.DataFrame.get_value\n.. _`get_values`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get_values.html#pandas.DataFrame.get_values\n.. _`groupby`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html#pandas.DataFrame.groupby\n.. _`gt`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.gt.html#pandas.DataFrame.gt\n.. _`head`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.head.html#pandas.DataFrame.head\n.. _`hist`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.hist.html#pandas.DataFrame.hist\n.. _`iat`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iat.html#pandas.DataFrame.iat\n.. _`idxmax`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.idxmax.html#pandas.DataFrame.idxmax\n.. _`idxmin`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.idxmin.html#pandas.DataFrame.idxmin\n.. _`iloc`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iloc.html#pandas.DataFrame.iloc\n.. _`infer_objects`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.infer_objects.html#pandas.DataFrame.infer_objects\n.. _`info`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.info.html#pandas.DataFrame.info\n.. _`insert`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.insert.html#pandas.DataFrame.insert\n.. _`interpolate`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.interpolate.html#pandas.DataFrame.interpolate\n.. _`is_copy`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.is_copy.html#pandas.DataFrame.is_copy\n.. _`isetitem`: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isetitem.html?#pandas-dataframe-isetitem\n.. _`isin`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isin.html#pandas.DataFrame.isin\n.. _`isna`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isna.html#pandas.DataFrame.isna\n.. _`isnull`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isnull.html#pandas.DataFrame.isnull\n.. _`items`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.items.html#pandas.DataFrame.items\n.. _`iterrows`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iterrows.html#pandas.DataFrame.iterrows\n.. _`itertuples`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.itertuples.html#pandas.DataFrame.itertuples\n.. _`ix`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ix.html#pandas.DataFrame.ix\n.. _`join`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html#pandas.DataFrame.join\n.. _`keys`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.keys.html#pandas.DataFrame.keys\n.. _`kurt`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurt.html#pandas.DataFrame.kurt\n.. _`kurtosis`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurtosis.html#pandas.DataFrame.kurtosis\n.. _`last`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.last.html#pandas.DataFrame.last\n.. _`last_valid_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.last_valid_index.html#pandas.DataFrame.last_valid_index\n.. _`le`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.le.html#pandas.DataFrame.le\n.. _`loc`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html#pandas.DataFrame.loc\n.. _`lt`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.lt.html#pandas.DataFrame.lt\n.. _`mask`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mask.html#pandas.DataFrame.mask\n.. _`max`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.max.html#pandas.DataFrame.max\n.. _`mean`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mean.html#pandas.DataFrame.mean\n.. _`median`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.median.html#pandas.DataFrame.median\n.. _`melt`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.melt.html#pandas.DataFrame.melt\n.. _`memory_usage`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.memory_usage.html#pandas.DataFrame.memory_usage\n.. _`merge`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html#pandas.DataFrame.merge\n.. _`min`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.min.html#pandas.DataFrame.min\n.. _`mod`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mod.html#pandas.DataFrame.mod\n.. _`mode`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mode.html#pandas.DataFrame.mode\n.. _`mul`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mul.html#pandas.DataFrame.mul\n.. _`multiply`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.multiply.html#pandas.DataFrame.multiply\n.. _`ndim`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ndim.html#pandas.DataFrame.ndim\n.. _`ne`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ne.html#pandas.DataFrame.ne\n.. _`nlargest`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nlargest.html#pandas.DataFrame.nlargest\n.. _`notna`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.notna.html#pandas.DataFrame.notna\n.. _`notnull`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.notnull.html#pandas.DataFrame.notnull\n.. _`nsmallest`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nsmallest.html#pandas.DataFrame.nsmallest\n.. _`nunique`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nunique.html#pandas.DataFrame.nunique\n.. _`pct_change`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pct_change.html#pandas.DataFrame.pct_change\n.. _`pipe`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pipe.html#pandas.DataFrame.pipe\n.. _`pivot`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pivot.html#pandas.DataFrame.pivot\n.. _`pivot_table`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pivot_table.html#pandas.DataFrame.pivot_table\n.. _`plot`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.html#pandas.DataFrame.plot\n.. _`pop`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pop.html#pandas.DataFrame.pop\n.. _`pow`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pow.html#pandas.DataFrame.pow\n.. _`prod`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.prod.html#pandas.DataFrame.prod\n.. _`product`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.product.html#pandas.DataFrame.product\n.. _`quantile`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.quantile.html#pandas.DataFrame.quantile\n.. _`query`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html#pandas.DataFrame.query\n.. _`radd`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.radd.html#pandas.DataFrame.radd\n.. _`rank`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rank.html#pandas.DataFrame.rank\n.. _`rdiv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rdiv.html#pandas.DataFrame.rdiv\n.. _`reindex`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reindex.html#pandas.DataFrame.reindex\n.. _`reindex_like`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reindex_like.html#pandas.DataFrame.reindex_like\n.. _`rename`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rename.html#pandas.DataFrame.rename\n.. _`rename_axis`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rename_axis.html#pandas.DataFrame.rename_axis\n.. _`reorder_levels`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reorder_levels.html#pandas.DataFrame.reorder_levels\n.. _`replace`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.replace.html#pandas.DataFrame.replace\n.. _`resample`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.resample.html#pandas.DataFrame.resample\n.. _`reset_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reset_index.html#pandas.DataFrame.reset_index\n.. _`rfloordiv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rfloordiv.html#pandas.DataFrame.rfloordiv\n.. _`rmod`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rmod.html#pandas.DataFrame.rmod\n.. _`rmul`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rmul.html#pandas.DataFrame.rmul\n.. _`rolling`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rolling.html#pandas.DataFrame.rolling\n.. _`round`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.round.html#pandas.DataFrame.round\n.. _`rpow`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rpow.html#pandas.DataFrame.rpow\n.. _`rsub`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rsub.html#pandas.DataFrame.rsub\n.. _`rtruediv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rtruediv.html#pandas.DataFrame.rtruediv\n.. _`sample`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sample.html#pandas.DataFrame.sample\n.. _`select_dtypes`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.select_dtypes.html#pandas.DataFrame.select_dtypes\n.. _`sem`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sem.html#pandas.DataFrame.sem\n.. _`set_axis`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.set_axis.html#pandas.DataFrame.set_axis\n.. _`set_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.set_index.html#pandas.DataFrame.set_index\n.. _`set_value`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.set_value.html#pandas.DataFrame.set_value\n.. _`shape`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shape.html#pandas.DataFrame.shape\n.. _`shift`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shift.html#pandas.DataFrame.shift\n.. _`size`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.size.html#pandas.DataFrame.size\n.. _`skew`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.skew.html#pandas.DataFrame.skew\n.. _`sort_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sort_index.html#pandas.DataFrame.sort_index\n.. _`sort_values`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sort_values.html#pandas.DataFrame.sort_values\n.. _`sparse`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sparse.html#pandas-dataframe-sparse\n.. _`squeeze`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.squeeze.html#pandas.DataFrame.squeeze\n.. _`stack`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.stack.html#pandas.DataFrame.stack\n.. _`std`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.std.html#pandas.DataFrame.std\n.. _`style`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.style.html#pandas.DataFrame.style\n.. _`sub`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sub.html#pandas.DataFrame.sub\n.. _`subtract`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.subtract.html#pandas.DataFrame.subtract\n.. _`sum`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sum.html#pandas.DataFrame.sum\n.. _`swapaxes`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.swapaxes.html#pandas.DataFrame.swapaxes\n.. _`swaplevel`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.swaplevel.html#pandas.DataFrame.swaplevel\n.. _`tail`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tail.html#pandas.DataFrame.tail\n.. _`take`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.take.html#pandas.DataFrame.take\n.. _`to_clipboard`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_clipboard.html#pandas.DataFrame.to_clipboard\n.. _`to_csv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html#pandas.DataFrame.to_csv\n.. _`to_dict`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_dict.html#pandas.DataFrame.to_dict\n.. _`to_excel`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html#pandas.DataFrame.to_excel\n.. _`to_feather`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_feather.html#pandas.DataFrame.to_feather\n.. _`to_gbq`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_gbq.html#pandas.DataFrame.to_gbq\n.. _`to_hdf`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html#pandas.DataFrame.to_hdf\n.. _`to_html`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_html.html#pandas.DataFrame.to_html\n.. _`to_json`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html#pandas.DataFrame.to_json\n.. _`to_xml`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html#pandas.DataFrame.to_xml\n.. _`to_latex`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_latex.html#pandas.DataFrame.to_latex\n.. _`to_orc`: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_orc.html#pandas.DataFrame.to_orc\n.. _`to_parquet`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html#pandas.DataFrame.to_parquet\n.. _`to_period`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_period.html#pandas.DataFrame.to_period\n.. _`to_pickle`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_pickle.html#pandas.DataFrame.to_pickle\n.. _`to_records`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_records.html#pandas.DataFrame.to_records\n.. _`to_sql`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html#pandas.DataFrame.to_sql\n.. _`to_stata`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_stata.html#pandas.DataFrame.to_stata\n.. _`to_string`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_string.html#pandas.DataFrame.to_string\n.. _`to_timestamp`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_timestamp.html#pandas.DataFrame.to_timestamp\n.. _`to_xarray`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xarray.html#pandas.DataFrame.to_xarray\n.. _`transform`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.transform.html#pandas.DataFrame.transform\n.. _`transpose`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.transpose.html#pandas.DataFrame.transpose\n.. _`truediv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.truediv.html#pandas.DataFrame.truediv\n.. _`truncate`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.truncate.html#pandas.DataFrame.truncate\n.. _`tz_convert`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tz_convert.html#pandas.DataFrame.tz_convert\n.. _`tz_localize`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tz_localize.html#pandas.DataFrame.tz_localize\n.. _`unstack`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.unstack.html#pandas.DataFrame.unstack\n.. _`update`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.update.html#pandas.DataFrame.update\n.. _`value_counts`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.value_counts.html#pandas.DataFrame.value_counts\n.. _`values`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.values.html#pandas.DataFrame.values\n.. _`var`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.var.html#pandas.DataFrame.var\n.. _`where`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.where.html#pandas.DataFrame.where\n.. _`xs`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.xs.html#pandas.DataFrame.xs\n"
},
{
"path": "docs/supported_apis/defaulting_to_pandas.rst",
"content": "Defaulting to pandas\n====================\n\nCurrently Modin does not support distributed execution for all methods from pandas API.\nThe remaining unimplemented methods are being executed in a mode called \"default to pandas\".\nThis allows users to continue using Modin even though their workloads contain functions not yet implemented in Modin.\nHere is a diagram of how we convert to pandas and perform the operation:\n\n.. image:: /img/convert_to_pandas.png\n :align: center\n\nWe first convert to a pandas DataFrame, then perform the operation. **There is a\nperformance penalty for going from a partitioned Modin DataFrame to pandas because of\nthe communication cost and single-threaded nature of pandas.** Once the pandas operation\nhas completed, we convert the DataFrame back into a partitioned Modin DataFrame. This\nway, operations performed after something defaults to pandas will be optimized with\nModin.\n\nThe exact methods we have implemented are listed in the respective subsections:\n\n* :doc:`DataFrame `\n* :doc:`Series `\n* :doc:`utilities `\n* :doc:`I/O `\n\nWe have taken a community-driven approach to implementing new methods. We did a `study\non pandas usage`_ to learn what the most-used APIs are. Modin currently supports **93%**\nof the pandas API based on our study of pandas usage, and we are actively expanding the\nAPI.\n**To request implementation, file an issue at https://github.com/modin-project/modin/issues\nor send an email to feature_requests@modin.org.**\n\n.. _`study on pandas usage`: https://github.com/modin-project/study_kaggle_usage\n"
},
{
"path": "docs/supported_apis/index.rst",
"content": "Supported APIs\n==============\n\nFor your convenience, we have compiled a list of currently implemented APIs and methods\navailable in Modin. This documentation is updated as new methods and APIs are merged\ninto the main branch, and not necessarily correct as of the most recent release. \n\nTo view the docs for the most recent release, check that you're viewing the \n`stable version`_ of the docs.\n\nIn order to install the latest version of Modin, follow the directions found on the\n:doc:`installation page `.\n\nQuestions on implementation details\n-----------------------------------\n\nIf you have a question about the implementation details or would like more information\nabout an API or method in Modin, please contact the Modin `developer mailing list`_.\n\n.. toctree::\n :titlesonly:\n :hidden:\n\n defaulting_to_pandas\n dataframe_supported\n series_supported\n utilities_supported\n io_supported\n older_pandas_compat\n\n.. meta::\n :description lang=en:\n Compilation of implemented pandas APIs in Modin.\n\n.. _developer mailing list: https://groups.google.com/forum/#!forum/modin-dev\n.. _stable version: https://modin.readthedocs.io/en/stable/supported_apis/index.html\n"
},
{
"path": "docs/supported_apis/io_supported.rst",
"content": "``pd.read_`` and I/O APIs\r\n=================================\r\n\r\nA number of IO methods default to pandas. We have parallelized ``read_csv``,\r\n``read_parquet`` and some more (see table), though many of the remaining methods\r\ncan be relatively easily parallelized. Some of the operations default to the\r\npandas implementation, meaning it will read in serially as a single, non-distributed\r\nDataFrame and distribute it. Performance will be affected by this.\r\n\r\nThe following table is structured as follows: The first column contains the method name.\r\nThe second column is a flag for whether or not there is an implementation in Modin for\r\nthe method in the left column. ``Y`` stands for yes, ``N`` stands for no, ``P`` stands\r\nfor partial (meaning some parameters may not be supported yet), and ``D`` stands for\r\ndefault to pandas.\r\n\r\n.. note::\r\n Support for fully asynchronous reading has been added for the following functions:\r\n ``read_csv``, ``read_fwf``, ``read_table``, ``read_custom_text``.\r\n This mode is disabled by default, one can enable it using ``MODIN_ASYNC_READ_MODE=True``\r\n environment variable. Some parameter combinations are not supported and the function\r\n will be executed in synchronous mode.\r\n\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| IO method | Modin Implementation? (Y/N/P/D) | Notes for Current implementation |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_csv`_ | Y | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_fwf`_ | Y | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_table`_ | Y | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_parquet`_ | P | Parameters besides ``filters`` and ``storage_options`` |\r\n| | | passed via ``**kwargs`` are not supported. |\r\n| | | ``use_nullable_dtypes`` == True is not supported. |\r\n| | | |\r\n| | | Experimental implementation: read_parquet_glob |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_json`_ | P | Implemented for ``lines=True`` |\r\n| | | Experimental implementation: read_json_glob |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_xml` | D | Experimental implementation: read_xml_glob |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_html`_ | D | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_clipboard`_ | D | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_excel`_ | D | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_hdf`_ | D | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_feather`_ | Y | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_stata`_ | D | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_sas`_ | D | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_pickle`_ | D | Experimental implementation: |\r\n| | | read_pickle_glob |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n| `read_sql`_ | Y | |\r\n+-------------------+---------------------------------+--------------------------------------------------------+\r\n\r\n.. _`read_csv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html#pandas.read_csv\r\n.. _`read_fwf`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html#pandas.read_fwf\r\n.. _`read_table`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html#pandas.read_table\r\n.. _`read_parquet`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html#pandas.read_parquet\r\n.. _`read_json`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html#pandas.read_json\r\n.. _`read_html`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_html.html#pandas.read_html\r\n.. _`read_clipboard`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_clipboard.html#pandas.read_clipboard\r\n.. _`read_excel`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html#pandas.read_excel\r\n.. _`read_hdf`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html#pandas.read_hdf\r\n.. _`read_feather`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_feather.html#pandas.read_feather\r\n.. _`read_stata`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_stata.html#pandas.read_stata\r\n.. _`read_sas`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sas.html#pandas.read_sas\r\n.. _`read_pickle`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_pickle.html#pandas.read_pickle\r\n.. _`read_sql`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sql.html#pandas.read_sql\r\n"
},
{
"path": "docs/supported_apis/older_pandas_compat.rst",
"content": "===================================\nPandas backwards compatibility mode\n===================================\n\nModin verions 0.16 and 0.17, but no later minor versions, had limited support\nfor running with legacy pandas versions. The latest version of Modin no longer\nhas such support.\n\nMotivation for compatibility mode\n---------------------------------\n\nModin aims to keep compatibility with latest pandas release, hopefully catching up each release\nwithin a few days.\n\nHowever, due to certain restrictions like need to use Python 3.6 it forces some users to\nuse older pandas (1.1.x for Python 3.6, specifically), which normally would mean they're\nbound to be using ancient Modin as well.\n\nTo overcome this, Modin has special \"compatibility mode\" where some basic functionality\nworks, but please note that the support is \"best possible effort\" (e.g. not all older bugs\nare worth fixing).\n\nKnown issues with pandas 1.1.x\n------------------------------\n\n* ``pd.append()`` does not preserve the order of columns in older pandas while Modin does\n* ``.astype()`` produces different error type on incompatible dtypes\n* ``read_csv()`` does not support reading from ZIP file *with compression* in parallel mode\n* ``read_*`` do not support ``storage_option`` named argument\n* ``to_csv()`` does not support binary mode for output file\n* ``read_excel()`` does not support ``.xlsx`` files\n* ``read_fwf()`` has a bug with list of skiprows and non-None nrows: `pandas-dev#10261`_\n* ``.agg(int-value)`` produces TypeError in older pandas but Modin raises AssertionError\n* ``Series.reset_index(drop=True)`` does not ignore ``name`` in older pandas while Modin ignores it\n* ``.sort_index(ascending=None)`` does not raise ValueError in older pandas while Modin raises it\n\nPlease keep in mind that there are probably more issues which are not yet uncovered!\n\n.. _`pandas-dev#10261`: https://github.com/pandas-dev/pandas/issues/10261\n"
},
{
"path": "docs/supported_apis/series_supported.rst",
"content": "``pd.Series`` supported APIs\r\n============================\r\n\r\nThe following table lists both implemented and not implemented methods. If you have need\r\nof an operation that is listed as not implemented, feel free to open an issue on the\r\n`GitHub repository`_, or give a thumbs up to already created issues. Contributions are\r\nalso welcome!\r\n\r\nThe following table is structured as follows: The first column contains the method name.\r\nThe second column is a flag for whether or not there is an implementation in Modin for\r\nthe method in the left column. ``Y`` stands for yes, ``N`` stands for no, ``P`` stands\r\nfor partial (meaning some parameters may not be supported yet), and ``D`` stands for\r\ndefault to pandas. To learn more about the implementations that default to pandas, see\r\nthe related section on :doc:`Defaulting to pandas `.\r\n\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| Series method | Modin Implementation? (Y/N/P/D) | Notes for Current implementation |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``abs`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``add`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``add_prefix`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``add_suffix`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``agg`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``aggregate`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``align`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``all`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``any`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``apply`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``argmax`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``argmin`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``argsort`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``array`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``asfreq`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``asobject`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``asof`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``astype`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``at`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``at_time`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``autocorr`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``axes`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``base`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``between`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``between_time`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``bfill`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``bool`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``cat`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``clip`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``combine`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``combine_first`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``compare`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``compress`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``copy`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``corr`` | Y | Correlation floating point precision may slightly |\r\n| | | differ from pandas. For now pearson method is |\r\n| | | available only. For other methods defaults to |\r\n| | | pandas. |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``count`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``cov`` | Y | Covariance floating point precision may slightly |\r\n| | | differ from pandas. |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``cummax`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``cummin`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``cumprod`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``cumsum`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``data`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``describe`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``diff`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``div`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``divide`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``divmod`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``dot`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``drop`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``drop_duplicates`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``droplevel`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``dropna`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``dt`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``dtype`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``dtypes`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``duplicated`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``empty`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``eq`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``equals`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``ewm`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``expanding`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``explode`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``factorize`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``ffill`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``fillna`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``filter`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``first`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``first_valid_index`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``flags`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``floordiv`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``from_array`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``ftype`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``ge`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``get`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``get_dtype_counts`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``get_ftype_counts`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``get_value`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``get_values`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``groupby`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``gt`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``hasnans`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``head`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``hist`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``iat`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``idxmax`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``idxmin`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``iloc`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``imag`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``index`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``infer_objects`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``interpolate`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``is_monotonic_decreasing`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``is_monotonic_increasing`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``is_unique`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``isin`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``isna`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``isnull`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``item`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``items`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``itemsize`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``keys`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``kurt`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``kurtosis`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``last`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``last_valid_index`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``le`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``loc`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``lt`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``map`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``mask`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``max`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``mean`` | P | Modin defaults to pandas if given the ``level`` |\r\n| | | param. |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``median`` | P | Modin defaults to pandas if given the ``level`` |\r\n| | | param. |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``memory_usage`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``min`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``mod`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``mode`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``mul`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``multiply`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``name`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``nbytes`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``ndim`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``ne`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``nlargest`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``nonzero`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``notna`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``notnull`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``nsmallest`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``nunique`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``pct_change`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``pipe`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``plot`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``pop`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``pow`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``prod`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``product`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``ptp`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``put`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``quantile`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``radd`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rank`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``ravel`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rdiv`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rdivmod`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``real`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``reindex`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``reindex_like`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rename`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rename_axis`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``reorder_levels`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``repeat`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``replace`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``resample`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``reset_index`` | P | **Ray** and **Dask**: ``D`` when ``names`` or |\r\n| | | ``allow_duplicates`` is non-default |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rfloordiv`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rmod`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rmul`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rolling`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``round`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rpow`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rsub`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``rtruediv`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``sample`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``searchsorted`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``sem`` | P | Modin defaults to pandas if given the ``level`` |\r\n| | | param. |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``set_axis`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``set_value`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``shape`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``shift`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``size`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``skew`` | P | Modin defaults to pandas if given the ``level`` |\r\n| | | param. |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``sort_index`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``sort_values`` | Y | Order of indexes that have the same sort key |\r\n| | | is not guaranteed to be the same across sorts; |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``sparse`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``squeeze`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``std`` | P | Modin defaults to pandas if given the ``level`` |\r\n| | | param. |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``str`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``strides`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``sub`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``subtract`` | Y | See ``add``; |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``sum`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``swapaxes`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``swaplevel`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``tail`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``take`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_clipboard`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_csv`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_dict`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_excel`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_frame`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_hdf`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_json`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_latex`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_list`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_numpy`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_period`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_pickle`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_sql`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_string`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_timestamp`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``to_xarray`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``tolist`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``transform`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``transpose`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``truediv`` | Y | See ``add`` |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``truncate`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``tz_convert`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``tz_localize`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``unique`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``unstack`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``update`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``valid`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``value_counts`` | Y | The indices order of resulting object may differ |\r\n| | | from pandas. |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``values`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``var`` | P | Modin defaults to pandas if given the ``level`` |\r\n| | | param. |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``view`` | D | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n| ``where`` | Y | |\r\n+-----------------------------+---------------------------------+----------------------------------------------------+\r\n\r\n.. _`GitHub repository`: https://github.com/modin-project/modin/issues\r\n"
},
{
"path": "docs/supported_apis/utilities_supported.rst",
"content": "pandas Utilities Supported\r\n==========================\r\n\r\nIf you run ``import modin.pandas as pd``, the following operations are available from\r\n``pd.``, e.g. ``pd.concat``. If you do not see an operation that pandas enables and\r\nwould like to request it, feel free to `open an issue`_. Make sure you tell us your\r\nprimary use-case so we can make it happen faster!\r\n\r\nThe following table is structured as follows: The first column contains the method name.\r\nThe second column is a flag for whether or not there is an implementation in Modin for\r\nthe method in the left column. ``Y`` stands for yes, ``N`` stands for no, ``P`` stands\r\nfor partial (meaning some parameters may not be supported yet), and ``D`` stands for\r\ndefault to pandas.\r\n\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| Utility method | Modin Implementation? (Y/N/P/D) | Notes for Current implementation |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.concat`_ | Y | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.eval`_ | Y | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.unique`_ | Y | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| ``pd.value_counts`` | Y | The indices order of resulting object may differ |\r\n| | | from pandas. |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.cut`_ | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.to_numeric`_ | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.factorize`_ | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.from_dummies`_ | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.qcut`_ | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| ``pd.match`` | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.to_datetime`_ | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.get_dummies`_ | Y | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.date_range`_ | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.bdate_range`_ | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| `pd.to_timedelta`_ | D | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n| ``pd.options`` | Y | |\r\n+---------------------------+---------------------------------+----------------------------------------------------+\r\n\r\nOther objects & structures\r\n--------------------------\r\n\r\nThis list is a list of objects not currently distributed by Modin. All of these objects\r\nare compatible with the distributed components of Modin. If you are interested in\r\ncontributing a distributed version of any of these objects, feel free to open a\r\n`pull request`_.\r\n\r\n* Panel\r\n* Index\r\n* MultiIndex\r\n* CategoricalIndex\r\n* DatetimeIndex\r\n* Timedelta\r\n* Timestamp\r\n* NaT\r\n* PeriodIndex\r\n* Categorical\r\n* Interval\r\n* UInt8Dtype\r\n* UInt16Dtype\r\n* UInt32Dtype\r\n* UInt64Dtype\r\n* SparseDtype\r\n* Int8Dtype\r\n* Int16Dtype\r\n* Int32Dtype\r\n* Int64Dtype\r\n* CategoricalDtype\r\n* DatetimeTZDtype\r\n* IntervalDtype\r\n* PeriodDtype\r\n* RangeIndex\r\n* TimedeltaIndex\r\n* IntervalIndex\r\n* IndexSlice\r\n* TimeGrouper\r\n* Grouper\r\n* array\r\n* Period\r\n* DateOffset\r\n* ExcelWriter\r\n* SparseArray\r\n\r\n.. _open an issue: https://github.com/modin-project/modin/issues\r\n.. _pull request: https://github.com/modin-project/modin/pulls\r\n.. _`pd.concat`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html#pandas.concat\r\n.. _`pd.eval`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.eval.html#pandas.eval\r\n.. _`pd.unique`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.unique.html#pandas.unique\r\n.. _`pd.cut`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html#pandas.cut\r\n.. _`pd.to_numeric`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_numeric.html#pandas.to_numeric\r\n.. _`pd.factorize`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.factorize.html#pandas.factorize\r\n.. _`pd.from_dummies`: https://pandas.pydata.org/docs/reference/api/pandas.from_dummies.html#pandas-from-dummies\r\n.. _`pd.qcut`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.qcut.html#pandas.qcut\r\n.. _`pd.to_datetime`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html#pandas.to_datetime\r\n.. _`pd.get_dummies`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html#pandas.get_dummies\r\n.. _`pd.date_range`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.date_range.html#pandas.date_range\r\n.. _`pd.bdate_range`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.bdate_range.html#pandas.bdate_range\r\n.. _`pd.to_timedelta`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_timedelta.html#pandas.to_timedelta\r\n"
},
{
"path": "docs/usage_guide/advanced_usage/batch.rst",
"content": "Batch Pipline API Usage Guide\n=============================\n\nModin provides an experimental batching feature that pipelines row-parallel queries. This feature \nis currently only supported for the ``PandasOnRay`` engine. Please note that this feature is experimental\nand behavior or interfaces could be changed.\n\nUsage examples\n--------------\n\nIn examples below we build and run some pipelines. It is important to note that the queries passed to\nthe pipeline operate on Modin DataFrame partitions, which are backed by ``pandas``. When using ``pandas``-\nmodule level functions, please make sure to import and use ``pandas`` rather than ``modin.pandas``.\n\nSimple Batch Pipelining\n^^^^^^^^^^^^^^^^^^^^^^^\n\nThis example walks through a simple batch pipeline in order to familiarize the user with the API.\n\n.. code-block:: python\n\n from modin.experimental.batch import PandasQueryPipeline\n import modin.pandas as pd\n import numpy as np\n\n df = pd.DataFrame(\n np.random.randint(0, 100, (100, 100)),\n columns=[f\"col {i}\" for i in range(1, 101)],\n ) # Build the dataframe we will pipeline.\n pipeline = PandasQueryPipeline(df) # Build the pipeline.\n pipeline.add_query(lambda df: df + 1, is_output=True) # Add the first query and specify that\n # it is an output query.\n pipeline.add_query(\n lambda df: df.rename(columns={f\"col {i}\":f\"col {i-1}\" for i in range(1, 101)})\n ) # Add a second query.\n pipeline.add_query(\n lambda df: df.drop(columns=['col 99']),\n is_output=True,\n ) # Add a third query and specify that it is an output query.\n new_df = pd.DataFrame(\n np.ones((100, 100)),\n columns=[f\"col {i}\" for i in range(1, 101)],\n ) # Build a second dataframe that we will pipeline now instead.\n pipeline.update_df(new_df) # Update the dataframe that we will pipeline to be `new_df`\n # instead of `df`.\n result_dfs = pipeline.compute_batch() # Begin batch processing.\n\n # Print pipeline results\n print(f\"Result of Query 1:\\n{result_dfs[0]}\")\n print(f\"Result of Query 2:\\n{result_dfs[1]}\")\n # Output IDs can also be specified\n pipeline = PandasQueryPipeline(df) # Build the pipeline.\n pipeline.add_query(\n lambda df: df + 1,\n is_output=True,\n output_id=1,\n ) # Add the first query, specify that it is an output query, as well as specify an output id.\n pipeline.add_query(\n lambda df: df.rename(columns={f\"col {i}\":f\"col {i-1}\" for i in range(1, 101)})\n ) # Add a second query.\n pipeline.add_query(\n lambda df: df.drop(columns=['col 99']),\n is_output=True,\n output_id=2,\n ) # Add a third query, specify that it is an output query, and specify an output_id.\n result_dfs = pipeline.compute_batch() # Begin batch processing.\n\n # Print pipeline results - should be a dictionary mapping Output IDs to resulting dataframes:\n print(f\"Mapping of Output ID to dataframe:\\n{result_dfs}\")\n # Print results\n for query_id, res_df in result_dfs.items():\n print(f\"Query {query_id} resulted in\\n{res_df}\")\n\nBatch Pipelining with Postprocessing\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nA postprocessing function can also be provided when calling ``pipeline.compute_batch``. The example\nbelow runs a similar pipeline as above, but the postprocessing function writes the output dfs to \na parquet file.\n\n.. code-block:: python\n\n from modin.experimental.batch import PandasQueryPipeline\n import modin.pandas as pd\n import numpy as np\n import os\n import shutil\n\n df = pd.DataFrame(\n np.random.randint(0, 100, (100, 100)),\n columns=[f\"col {i}\" for i in range(1, 101)],\n ) # Build the dataframe we will pipeline.\n pipeline = PandasQueryPipeline(df) # Build the pipeline.\n pipeline.add_query(\n lambda df: df + 1,\n is_output=True,\n output_id=1,\n ) # Add the first query, specify that it is an output query, as well as specify an output id.\n pipeline.add_query(\n lambda df: df.rename(columns={f\"col {i}\":f\"col {i-1}\" for i in range(1, 101)})\n ) # Add a second query.\n pipeline.add_query(\n lambda df: df.drop(columns=['col 99']),\n is_output=True,\n output_id=2,\n ) # Add a third query, specify that it is an output query, and specify an output_id.\n def postprocessing_func(df, output_id, partition_id):\n filepath = f\"query_{output_id}/\"\n os.makedirs(filepath, exist_ok=True)\n filepath += f\"part-{partition_id:04d}.parquet\"\n df.to_parquet(filepath)\n return df\n result_dfs = pipeline.compute_batch(\n postprocessor=postprocessing_func,\n pass_partition_id=True,\n pass_output_id=True,\n ) # Begin computation, pass in a postprocessing function, and specify that partition ID and \n # output ID should be passed to that postprocessing function.\n\n print(os.system(\"ls query_1/\")) # Should show `NPartitions.get()` parquet files - which\n # correspond to partitions of the output of query 1.\n print(os.system(\"ls query_2/\")) # Should show `NPartitions.get()` parquet files - which\n # correspond to partitions of the output of query 2.\n\n for query_id, res_df in result_dfs.items():\n written_df = pd.read_parquet(f\"query_{query_id}/\")\n shutil.rmtree(f\"query_{query_id}/\") # Clean up\n print(f\"Written and Computed DF are \" +\n f\"{'equal' if res_df.equals(written_df) else 'not equal'} for query {query_id}\")\n\nBatch Pipelining with Fan Out\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIf the input dataframe to a query is small (consisting of only one partition), it is possible to \ninduce additional parallelism using the ``fan_out`` argument. The ``fan_out`` argument replicates\nthe input partition, applies the query to each replica, and then coalesces all of the replicas back\nto one partition using the ``reduce_fn`` that must also be specified when ``fan_out`` is ``True``.\n\nIt is possible to control the parallelism via the ``num_partitions`` parameter passed to the\nconstructor of the ``PandasQueryPipeline``. This parameter designates the desired number of partitions,\nand defaults to ``NPartitions.get()`` when not specified. During fan out, the input partition is replicated\n``num_partitions`` times. In the previous examples, ``num_partitions`` was not specified, and so defaulted\nto ``NPartitions.get()``.\n\nThe example below demonstrates the usage of ``fan_out`` and ``num_partitions``. We first demonstrate\nan example of a function that would benefit from this computation pattern:\n\n.. code-block:: python\n\n import glob\n from PIL import Image\n import torchvision.transforms as T\n import torchvision\n\n transforms = T.Compose([T.ToTensor()])\n model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)\n model.eval()\n COCO_INSTANCE_CATEGORY_NAMES = [\n '__background__', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus',\n 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'N/A', 'stop sign',\n 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',\n 'elephant', 'bear', 'zebra', 'giraffe', 'N/A', 'backpack', 'umbrella', 'N/A', 'N/A',\n 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball',\n 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket',\n 'bottle', 'N/A', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl',\n 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza',\n 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'N/A', 'dining table',\n 'N/A', 'N/A', 'toilet', 'N/A', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone',\n 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'N/A', 'book',\n 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush'\n ]\n\n def contains_cat(image, model):\n image = transforms(image)\n labels = [COCO_INSTANCE_CATEGORY_NAMES[i] for i in model([image])[0]['labels']]\n return 'cat' in labels\n\n def serial_query(df):\n \"\"\"\n This function takes as input a dataframe with a single row corresponding to a folder\n containing images to parse. Each image in the folder is passed through a neural network\n that detects whether it contains a cat, in serial, and a new column is computed for the\n dataframe that counts the number of images containing cats.\n\n Parameters\n ----------\n df : a dataframe\n The dataframe to process\n \n Returns\n -------\n The same dataframe as before, with an additional column containing the count of images \n containing cats.\n \"\"\"\n model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)\n model.eval()\n img_folder = df['images'][0]\n images = sorted(glob.glob(f\"{img_folder}/*.jpg\"))\n cats = 0\n for img in images:\n cats = cats + 1 if contains_cat(Image.open(img), model) else cats\n df['cat_count'] = cats\n return df\n \nTo download the image files to test out this code, run the following bash script, which downloads\nthe images from the fast-ai-coco S3 bucket to a folder called ``images`` in your current working\ndirectory:\n\n.. code-block:: shell\n\n aws s3 cp s3://fast-ai-coco/coco_tiny.tgz . --no-sign-request; tar -xf coco_tiny.tgz; mkdir \\\n images; mv coco_tiny/train/* images/; rm -rf coco_tiny; rm -rf coco_tiny.tgz\n\nWe can pipeline that code like so:\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.experimental.batch import PandasQueryPipeline\n from time import time\n df = pd.DataFrame([['images']], columns=['images'])\n pipeline = PandasQueryPipeline(df)\n pipeline.add_query(serial_query, is_output=True)\n serial_start = time()\n df_with_cat_count = pipeline.compute_batch()[0]\n serial_end = time()\n print(f\"Result of pipeline:\\n{df_with_cat_count}\")\n\nWe can induce `8x` parallelism into the pipeline above by combining the ``fan_out`` and ``num_partitions`` parameters like so:\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.experimental.batch import PandasQueryPipeline\n import shutil\n from time import time\n df = pd.DataFrame([['images']], columns=['images'])\n desired_num_partitions = 8\n def parallel_query(df, partition_id):\n \"\"\"\n This function takes as input a dataframe with a single row corresponding to a folder\n containing images to parse. It parses `total_images/desired_num_partitions` images every\n time it is called. A new column is computed for the dataframe that counts the number of\n images containing cats.\n\n Parameters\n ----------\n df : a dataframe\n The dataframe to process\n partition_id : int\n The partition id of the dataframe that this function runs on.\n \n Returns\n -------\n The same dataframe as before, with an additional column containing the count of images\n containing cats.\n \"\"\"\n model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)\n model.eval()\n img_folder = df['images'][0]\n images = sorted(glob.glob(f\"{img_folder}/*.jpg\"))\n total_images = len(images)\n cats = 0\n start_index = partition_id * (total_images // desired_num_partitions)\n if partition_id == desired_num_partitions - 1: # Last partition must parse to end of list\n images = images[start_index:]\n else:\n end_index = (partition_id + 1) * (total_images // desired_num_partitions)\n images = images[start_index:end_index]\n for img in images:\n cats = cats + 1 if contains_cat(Image.open(img), model) else cats\n df['cat_count'] = cats\n return df\n\n def reduce_fn(dfs):\n \"\"\"\n Coalesce the results of fanning out the `parallel_query` query.\n\n Parameters\n ----------\n dfs : a list of dataframes\n The resulting dataframes from fanning out `parallel_query`\n \n Returns\n -------\n A new dataframe whose `cat_count` column is the sum of the `cat_count` column of all\n dataframes in `dfs`\n \"\"\"\n df = dfs[0]\n cat_count = df['cat_count'][0]\n for dataframe in dfs[1:]:\n cat_count += dataframe['cat_count'][0]\n df['cat_count'] = cat_count\n return df\n pipeline = PandasQueryPipeline(df, desired_num_partitions)\n pipeline.add_query(\n parallel_query,\n fan_out=True,\n reduce_fn=reduce_fn,\n is_output=True,\n pass_partition_id=True\n )\n parallel_start = time()\n df_with_cat_count = pipeline.compute_batch()[0]\n parallel_end = time()\n print(f\"Result of pipeline:\\n{df_with_cat_count}\")\n print(f\"Total Time in Serial: {serial_end - serial_start}\")\n print(f\"Total time with induced parallelism: {parallel_end - parallel_start}\")\n shutil.rmtree(\"images/\") # Clean up\n\nBatch Pipelining with Dynamic Repartitioning\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nSimilarly, it is also possible to hint to the Pipeline API to repartition after a node completes\ncomputation. This is currently only supported if the input dataframe consists of only one partition.\nThe number of partitions after repartitioning is controlled by the ``num_partitions`` parameter\npassed to the constructor of the ``PandasQueryPipeline``.\n\nThe following example demonstrates how to use the ``repartition_after`` parameter.\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.experimental.batch import PandasQueryPipeline\n import numpy as np\n\n small_df = pd.DataFrame([[1, 2, 3]]) # Create a small dataframe\n \n def increase_dataframe_size(df):\n import pandas\n new_df = pandas.concat([df] * 1000)\n new_df = new_df.reset_index(drop=True) # Get a new range index that isn't duplicated\n return new_df\n \n desired_num_partitions = 24 # We will repartition to 24 partitions\n\n def add_partition_id_to_df(df, partition_id):\n import pandas\n new_col = pandas.Series([partition_id]*len(df), name=\"partition_id\", index=df.index)\n return pandas.concat([df, new_col], axis=1)\n \n pipeline = PandasQueryPipeline(small_df, desired_num_partitions)\n pipeline.add_query(increase_dataframe_size, repartition_after=True)\n pipeline.add_query(add_partition_id_to_df, pass_partition_id=True, is_output=True)\n result_df = pipeline.compute_batch()[0]\n print(f\"Number of partitions passed to second query: \" + \n f\"{len(np.unique(result_df['partition_id'].values))}\")\n print(f\"Result of pipeline:\\n{result_df}\")\n\n"
},
{
"path": "docs/usage_guide/advanced_usage/index.rst",
"content": "Advanced Usage\n==============\n\n.. toctree::\n :titlesonly:\n :hidden:\n\n /flow/modin/distributed/dataframe/pandas\n spreadsheets_api\n progress_bar\n modin_xgboost\n modin_logging\n modin_metrics\n batch\n modin_engines\n\n.. meta::\n :description lang=en:\n Description of Modin's advanced features.\n\nModin aims to not only optimize pandas, but also provide a comprehensive,\nintegrated toolkit for data scientists. We are actively developing data science tools\nsuch as DataFrame spreadsheet integration, DataFrame algebra, progress bars, SQL queries\non DataFrames, and more. Join us on `Slack`_ for the latest updates!\n\nModin engines\n-------------\n\nModin supports a series of execution engines such as Ray_, Dask_, `MPI through unidist`_,\neach of which might be a more beneficial choice for a specific scenario. When doing the first operation\nwith Modin it automatically initializes one of the engines to further perform distributed/parallel computation.\nIf you are familiar with a concrete execution engine, it is possible to initialize the engine on your own and\nModin will automatically attach to it. Refer to :doc:`Modin engines ` page\nfor more details.\n\nAdditional APIs\n---------------\n\nModin also supports these additional APIs on top of pandas to improve user experience.\n\n- :py:meth:`~modin.pandas.DataFrame.modin.to_pandas` -- convert a Modin DataFrame/Series to a pandas DataFrame/Series.\n- :py:meth:`~modin.pandas.DataFrame.get_backend` -- Get the ``Backend`` :doc:`configuration variable ` of this ``DataFrame``.\n- :py:meth:`~modin.pandas.DataFrame.move_to` -- Move data and execution for this ``DataFrame`` to the given ``Backend`` :doc:`configuration variable `. This method is an alias for ``DataFrame.set_backend``.\n- :py:meth:`~modin.pandas.DataFrame.set_backend` -- Move data and execution for this ``DataFrame`` to the given ``Backend`` :doc:`configuration variable `. This method is an alias for ``DatFrame.move_to``.\n- :py:func:`~modin.pandas.io.from_pandas` -- convert a pandas DataFrame to a Modin DataFrame.\n- :py:meth:`~modin.pandas.DataFrame.modin.to_ray` -- convert a Modin DataFrame/Series to a Ray Dataset.\n- :py:func:`~modin.pandas.io.from_ray` -- convert a Ray Dataset to a Modin DataFrame.\n- :py:meth:`~modin.pandas.DataFrame.modin.to_dask` -- convert a Modin DataFrame/Series to a Ray Dataset.\n- :py:func:`~modin.pandas.io.from_dask` -- convert a Modin DataFrame/Series to a Dask DataFrame/Series.\n- :py:func:`~modin.pandas.io.from_map` -- create a Modin DataFrame from map function applied to an iterable object.\n- :py:func:`~modin.pandas.io.from_arrow` -- convert an Arrow Table to a Modin DataFrame.\n- :py:func:`~modin.experimental.pandas.read_csv_glob` -- read multiple files in a directory.\n- :py:func:`~modin.experimental.pandas.read_sql` -- add optional parameters for the database connection.\n- :py:func:`~modin.experimental.pandas.read_custom_text` -- read custom text data from file.\n- :py:func:`~modin.experimental.pandas.read_pickle_glob` -- read multiple pickle files in a directory.\n- :py:func:`~modin.experimental.pandas.read_parquet_glob` -- read multiple parquet files in a directory.\n- :py:func:`~modin.experimental.pandas.read_json_glob` -- read multiple json files in a directory.\n- :py:func:`~modin.experimental.pandas.read_xml_glob` -- read multiple xml files in a directory.\n- :py:meth:`~modin.pandas.DataFrame.modin.to_pickle_glob` -- write to multiple pickle files in a directory.\n- :py:meth:`~modin.pandas.DataFrame.modin.to_parquet_glob` -- write to multiple parquet files in a directory.\n- :py:meth:`~modin.pandas.DataFrame.modin.to_json_glob` -- write to multiple json files in a directory.\n- :py:meth:`~modin.pandas.DataFrame.modin.to_xml_glob` -- write to multiple xml files in a directory.\n\nDataFrame partitioning API\n--------------------------\n\nModin DataFrame provides an API to directly access partitions: you can extract physical partitions from\na :py:class:`~modin.pandas.dataframe.DataFrame`, modify their structure by reshuffling or applying some\nfunctions, and create a DataFrame from those modified partitions. Visit\n:doc:`pandas partitioning API ` documentation to learn more.\n\nModin Spreadsheet API\n---------------------\n\nThe Spreadsheet API for Modin allows you to render the dataframe as a spreadsheet to easily explore\nyour data and perform operations on a graphical user interface. The API also includes features for recording\nthe changes made to the dataframe and exporting them as reproducible code. Built on top of Modin and SlickGrid,\nthe spreadsheet interface is able to provide interactive response times even at a scale of billions of rows.\nSee our `Modin Spreadsheet API documentation`_ for more details.\n\n.. figure:: /img/modin_spreadsheet_mini_demo.gif\n :align: center\n :width: 650px\n :height: 350px\n\nProgress Bar\n------------\n\nVisual progress bar for Dataframe operations such as groupby and fillna, as well as for file reading operations such as\nread_csv. Built using the `tqdm`_ library and Ray execution engine. See `Progress Bar documentation`_ for more details.\n\n.. figure:: /img/progress_bar_example.png\n :align: center\n\nDataframe Algebra\n-----------------\n\nA minimal set of operators that can be composed to express any dataframe query for use in query planning and optimization.\nSee our `paper`_ for more information, and full documentation is coming soon!\n\nDistributed XGBoost on Modin\n----------------------------\n\nModin provides an implementation of `distributed XGBoost`_ machine learning algorithm on Modin DataFrames. See our\n:doc:`Distributed XGBoost on Modin documentation ` for details about installation and usage, as well as\n:doc:`Modin XGBoost architecture documentation ` for information about implementation and\ninternal execution flow.\n\nLogging with Modin\n------------------\n\nModin logging offers users greater insight into their queries by logging internal Modin API calls, partition metadata,\nand system memory. Logging is disabled by default, but when it is enabled, log files are written to a local `.modin` directory\nat the same directory level as the notebook/script used to run Modin. See our :doc:`Logging with Modin documentation `\nfor usage information.\n\nBatch Pipeline API\n------------------\nModin provides an experimental batched API that pipelines row parallel queries. See our :doc:`Batch Pipline API Usage Guide `\nfor a walkthrough on how to use this feature, as well as :doc:`Batch Pipeline API documentation `\nfor more information about the API.\n\nFuzzydata Testing\n-----------------\n\nAn experimental GitHub Action on pull request has been added to Modin, which automatically runs the Modin codebase against\n`fuzzydata`, a random dataframe workflow generator. The resulting workflow that was used to test Modin codebase can be\ndownloaded as an artifact from the GitHub Actions tab for further inspection. See `fuzzydata`_ for more details.\n\n.. _`Modin Spreadsheet API documentation`: spreadsheets_api.html\n.. _`Progress Bar documentation`: progress_bar.html\n.. _`Paper`: https://arxiv.org/pdf/2001.00888.pdf\n.. _`Slack`: https://modin.org/slack.html\n.. _`tqdm`: https://github.com/tqdm/tqdm\n.. _`distributed XGBoost`: https://medium.com/intel-analytics-software/distributed-xgboost-with-modin-on-ray-fc17edef7720\n.. _`fuzzydata`: https://github.com/suhailrehman/fuzzydata\n.. _Ray: https://github.com/ray-project/ray\n.. _Dask: https://github.com/dask/distributed\n.. _`MPI through unidist`: https://github.com/modin-project/unidist\n"
},
{
"path": "docs/usage_guide/advanced_usage/modin_engines.rst",
"content": "Modin engines\n=============\n\nAs a rule, you don't have to worry about initialization of an execution engine as\nModin itself automatically initializes one when performing the first operation.\nAlso, Modin has a broad range of :doc:`configuration settings `, which\nyou can use to configure an execution engine. If there is a reason to initialize an execution engine\non your own and you are sure what to do, Modin will automatically attach to whichever engine is available.\nBelow, you can find some examples on how to initialize a specific execution engine on your own.\n\nRay\n---\n\nYou can initialize Ray engine with a specific number of CPUs (worker processes) to perform computation.\n\n.. code-block:: python\n\n import ray\n import modin.config as modin_cfg\n\n ray.init(num_cpus=)\n modin_cfg.Engine.put(\"ray\") # Modin will use Ray engine\n modin_cfg.CpuCount.put()\n\nTo get more details on all possible parameters for initialization refer to `Ray documentation`_.\n\nDask\n----\n\nYou can initialize Dask engine with a specific number of worker processes and threads per worker to perform computation.\n\n.. code-block:: python\n\n from distributed import Client\n import modin.config as modin_cfg\n\n client = Client(n_workers=, threads_per_worker=)\n modin_cfg.Engine.put(\"dask\") # # Modin will use Dask engine\n modin_cfg.CpuCount.put()\n\nTo get more details on all possible parameters for initialization refer to `Dask Distributed documentation`_.\n\nMPI through unidist\n-------------------\n\nYou can initialize MPI through unidist engine with a specific number of CPUs (worker processes) to perform computation.\n\n.. code-block:: python\n\n import unidist\n import unidist.config as unidist_cfg\n import modin.config as modin_cfg\n\n unidist_cfg.Backend.put(\"mpi\")\n unidist_cfg.CpuCount.put()\n unidist.init()\n\n modin_cfg.Engine.put(\"unidist\") # # Modin will use MPI through unidist engine\n modin_cfg.CpuCount.put()\n\nTo get more details on all possible parameters for initialization refer to `unidist documentation`_.\n\n.. _`Ray documentation`: https://docs.ray.io/en/latest\n.. _Dask Distributed documentation: https://distributed.dask.org/en/latest\n.. _`unidist documentation`: https://unidist.readthedocs.io/en/latest\n"
},
{
"path": "docs/usage_guide/advanced_usage/modin_logging.rst",
"content": "Modin Logging\n=============\n\nModin logging offers users greater insight into their queries by logging internal Modin API calls, partition metadata,\nand profiling system memory. When Modin logging is enabled (default disabled), log files are written to a local ``.modin`` directory at the same\ndirectory level as the notebook/script used to run Modin.\n\nThe logs generated by Modin Logging will be written to a ``.modin/logs/job_`` directory, uniquely named after the job uuid.\nThe logs that contain the Modin API stack traces are named ``trace.log``. The logs that contain the memory utilization metrics are\nnamed ``memory.log``. By default, if any log file exceeds 10MB (configurable with ``LogFileSize``), that file will be saved and a \nseparate log file will be created. For instance, if users have 20MB worth of Modin API logs, they can expect to find ``trace.log.1`` \nand ``trace.log.2`` in the ``.modin/logs/job_`` directory. After ``10 * LogFileSize`` MB or by default 100MB of logs, the logs will \nrollover and the original log files beginning with ``trace.log.1`` will be overwritten with the new log lines.\n\n**Developer Warning:** In some cases, running services like JupyterLab in the ``modin/modin`` directory may result in circular dependency issues.\nThis is due to a naming conflict between the ``modin/logging`` directory and the Python ``logging`` module, which may be used as a default in\nsuch environments. To resolve this, please run Jupyterlab or other similar services from directories other than ``modin/modin``.\n\nUsage examples\n--------------\n\nIn the example below, we enable logging for internal Modin API calls, partition metadata and memory profiling.\nWe can set the granularity (in seconds) at which the system memory utilization is logged using ``LogMemoryInterval``.\nWe can also set the maximum size of the logs (in MBs) using ``LogFileSize``.\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.config import LogMode, LogMemoryInterval, LogFileSize\n LogMode.enable()\n LogMemoryInterval.put(2) # Defaults to 5 seconds, new interval is 2 seconds\n LogFileSize.put(5) # Defaults to 10 MB per log file, new size is 5 MB\n\n # User code goes here\n\nDisable Modin logging like so:\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.config import LogMode\n LogMode.disable()\n\n # User code goes here\n\nIn Modin the lower-level functionality is logged in debug level, and higher level functionality in info level.\nBy default when logging is enabled in Modin, both high level and low level functionality are logged.\nThe below example script could be used to switch between logging all functions vs only logging higher level functions.\nSetting logger level to ``logging.INFO`` logs only higher level functions.\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.logging.config import get_logger\n from modin.config import LogMode\n import logging\n LogMode.enable()\n logger = get_logger()\n logger.setLevel(logging.INFO) # Replace with logger.setLevel(logging.DEBUG) for lower level logs\n df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})\n df = pd.concat([df, df])\n\n\nDebugging from user defined functions:\n\n.. warning:: \n When attempting to use Modin logging in user defined functions that execute in workers for logging lower-level operators\n as in example below, multiple log directories ``.modin/logs/job_**`` would be created for each worker executing the UDF.\n\n.. code-block:: python\n\n import modin.pandas as pd\n\n def udf(x):\n from modin.config import LogMode\n \n LogMode.enable()\n \n return x + 1\n \n modin_df = pd.DataFrame([0, 1, 2, 3])\n print(modin_df.map(udf))\n\nSo the **recommended** approach would be to use a different logger as in the below snipet\nto log from user defined functions that execute on workers.\nBelow is an an example to log from UDF. For this the logger config has to be specified inside the UDF that would execute on a remote worker.\n\n.. code-block:: python\n\n import logging\n import modin.pandas as pd\n \n def udf(x):\n logging.basicConfig(filename='modin_udf.log', level=logging.INFO)\n logging.info(\"This log message will be written to modin_udf.log \")\n\n # User code goes here\n return x + 1\n\n modin_df = pd.DataFrame([0, 1, 2, 3])\n print(modin_df.map(udf))\n"
},
{
"path": "docs/usage_guide/advanced_usage/modin_metrics.rst",
"content": "Modin Metrics\n=============\n\nModin allows for third-party systems to register a metrics handler to collect specific API statistics.\nMetrics have a name and a value, can be aggregated, discarded, or emitted without impact to the program.\n\nCPU load, memory usage, and disk usage are all typical metrics; but modin currently only emits metrics on API timings which can be used to optimize end-user interactive performance. New metrics may \nbe added in the future.\n\nIt is the responsibility of the handler to process or forward these metrics. The name of the metric will \nbe in \"dot format\" and all lowercase, similar to graphite or rrd. The value is an integer or float.\n\nExample metric names include:\n\n* 'modin.core-dataframe.pandasdataframe.copy_index_cache'\n* 'modin.core-dataframe.pandasdataframe.transpose'\n* 'modin.query-compiler.pandasquerycompiler.transpose'\n* 'modin.query-compiler.basequerycompiler.columnarize'\n* 'modin.pandas-api.series.__init__'\n* 'modin.pandas-api.dataframe._reduce_dimension'\n* 'modin.pandas-api.dataframe.sum'\n\nHandlers are functions of the form: `fn(str, int|float)` and can be registered with:\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.logging.metrics import add_metric_handler\n\n def func(name: str, value: int | float):\n print(f\"Got metric {name} value {value}\")\n\n add_metric_handler(func)\n\n.. warning:: \n A metric handler should be non-blocking, returning within 100ms, although this is not enforced. It must not throw exceptions or it will\n be deregistered. These restrictions are to help guard against the implementation of a metrics collector which would impact\n interactice performance significantly. The data from metrics should generally be offloaded to another system for processing\n and not involve any blocking network calls.\n\nMetrics are enabled by default. Modin metrics can be disabled like so:\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.config import MetricsMode\n MetricsMode.disable()\n"
},
{
"path": "docs/usage_guide/advanced_usage/modin_xgboost.rst",
"content": "Distributed XGBoost on Modin\n============================\n\nModin provides an implementation of `distributed XGBoost`_ machine learning\nalgorithm on Modin DataFrames. Please note that this feature is experimental and behavior or\ninterfaces could be changed.\n\nInstall XGBoost on Modin\n------------------------\n\nModin comes with all the dependencies except ``xgboost`` package by default.\nCurrently, distributed XGBoost on Modin is only supported on the Ray execution engine, therefore, see\nthe :doc:`installation page ` for more information on installing Modin with the Ray engine.\nTo install ``xgboost`` package you can use ``pip``:\n\n.. code-block:: bash\n\n pip install xgboost\n\n\nXGBoost Train and Predict\n-------------------------\n\nDistributed XGBoost functionality is placed in ``modin.experimental.xgboost`` module.\n``modin.experimental.xgboost`` provides a drop-in replacement API for ``train`` and ``Booster.predict`` xgboost functions.\n\n.. automodule:: modin.experimental.xgboost\n :noindex:\n :members: train\n\n.. autoclass:: modin.experimental.xgboost.Booster\n :noindex:\n :members: predict\n\n\nModinDMatrix\n------------\n\nData is passed to ``modin.experimental.xgboost`` functions via a Modin ``DMatrix`` object.\n\n.. automodule:: modin.experimental.xgboost\n :noindex:\n :members: DMatrix\n\nCurrently, the Modin ``DMatrix`` supports ``modin.pandas.DataFrame`` only as an input.\n\n\nA Single Node / Cluster setup\n-----------------------------\n\nThe XGBoost part of Modin uses a Ray resources by similar way as all Modin functions.\n\nTo start the Ray runtime on a single node:\n\n.. code-block:: python\n\n import ray\n # Look at the Ray documentation with respect to the Ray configuration suited to you most.\n ray.init()\n\nIf you already had the Ray cluster you can connect to it by next way:\n\n.. code-block:: python\n\n import ray\n ray.init(address='auto')\n\nA detailed information about initializing the Ray runtime you can find in `starting ray`_ page.\n\n\nUsage example\n-------------\n\nIn example below we train XGBoost model using `the Iris Dataset`_ and get prediction on the same data.\nAll processing will be in a `single node` mode.\n\n.. code-block:: python\n\n from sklearn import datasets\n\n import ray\n # Look at the Ray documentation with respect to the Ray configuration suited to you most.\n ray.init() # Start the Ray runtime for single-node\n\n import modin.pandas as pd\n import modin.experimental.xgboost as xgb\n\n # Load iris dataset from sklearn\n iris = datasets.load_iris()\n\n # Create Modin DataFrames\n X = pd.DataFrame(iris.data)\n y = pd.DataFrame(iris.target)\n\n # Create DMatrix\n dtrain = xgb.DMatrix(X, y)\n dtest = xgb.DMatrix(X, y)\n\n # Set training parameters\n xgb_params = {\n \"eta\": 0.3,\n \"max_depth\": 3,\n \"objective\": \"multi:softprob\",\n \"num_class\": 3,\n \"eval_metric\": \"mlogloss\",\n }\n steps = 20\n\n # Create dict for evaluation results\n evals_result = dict()\n\n # Run training\n model = xgb.train(\n xgb_params,\n dtrain,\n steps,\n evals=[(dtrain, \"train\")],\n evals_result=evals_result\n )\n\n # Print evaluation results\n print(f'Evals results:\\n{evals_result}')\n\n # Predict results\n prediction = model.predict(dtest)\n\n # Print prediction results\n print(f'Prediction results:\\n{prediction}')\n\n\n\n.. _Dataframe: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html\n.. _`starting ray`: https://docs.ray.io/en/master/starting-ray.html\n.. _`the Iris Dataset`: https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html\n.. _`distributed XGBoost`: https://medium.com/intel-analytics-software/distributed-xgboost-with-modin-on-ray-fc17edef7720\n"
},
{
"path": "docs/usage_guide/advanced_usage/progress_bar.rst",
"content": "Progress Bar\n============\n\nThe progress bar allows users to see the estimated progress and completion time of each line they run, \nin environments such as a shell or Jupyter notebook.\n\n.. figure:: /img/progress_bar.gif\n :align: center\n\nQuickstart\n\"\"\"\"\"\"\"\"\"\"\n\nThe progress bar uses the `tqdm` library to visualize displays:\n\n.. code-block:: bash\n\n pip install tqdm\n\n\nImport the progress bar into your notebook by running the following:\n\n\n.. code-block:: python\n\n from modin.config import ProgressBar\n ProgressBar.enable()\n"
},
{
"path": "docs/usage_guide/advanced_usage/spreadsheets_api.rst",
"content": "Modin Spreadsheets API\n======================\n\nGetting started\n---------------\nInstall Modin-spreadsheet using pip:\n\n.. code-block:: bash\n\n pip install \"modin[spreadsheet]\"\n\n\nThe following code snippet creates a spreadsheet using the FiveThirtyEight dataset on labor force information by college majors (licensed under CC BY 4.0):\n\n.. code-block:: python\n\n import modin.pandas as pd\n import modin.experimental.spreadsheet as mss\n df = pd.read_csv('https://raw.githubusercontent.com/fivethirtyeight/data/master/college-majors/all-ages.csv')\n spreadsheet = mss.from_dataframe(df)\n spreadsheet\n\n\n.. figure:: /img/modin_spreadsheets_installation.png\n :align: center\n\nBasic Manipulations through User Interface\n------------------------------------------\n\nThe Spreadsheet API allows users to manipulate the DataFrame with simple graphical controls for sorting, filtering, and editing. \n\nHere are the instructions for each operation:\n * **Sort**: Click on the column header of the column to sort on.\n * **Filter**: Click on the filter button on the column header and apply the desired filter to the column. The filter dropdown changes depending on the type of the column. Multiple filters are automatically combined.\n * **Edit Cell**: Double click on a cell and enter the new value.\n * **Add Rows**: Click on the “Add Row” button in the toolbar to duplicate the last row in the DataFrame. The duplicated values provide a convenient default and can be edited as necessary.\n * **Remove Rows**: Select row(s) and click the “Remove Row” button. Select a single row by clicking on it. Multiple rows can be selected with Cmd+Click (Windows: Ctrl+Click) on the desired rows or with Shift+Click to specify a range of rows. \n\nSome of these operations can also be done through the spreadsheet’s programmatic interface. Sorts and filters can be reset using the toolbar buttons. Edits and adding/removing rows can only be undone manually.\n\nVirtual Rendering\n-----------------\n\nThe spreadsheet will only render data based on the user’s viewport. This allows for quick rendering \neven on very large DataFrames because only a handful of rows are loaded at any given time. As a result, scrolling and viewing your data is smooth and responsive.\n\nTransformation History and Exporting Code\n-----------------------------------------\n\nAll operations on the spreadsheet are recorded and are easily exported as code for sharing or reproducibility. \nThis history is automatically displayed in the history cell, which is generated below the spreadsheet whenever the spreadsheet widget is displayed. \nThe history cell is displayed on default, but this can be turned off. Modin Spreadsheet API provides a few methods for interacting with the history:\n\n * `SpreadsheetWidget.get_history()`: Retrieves the transformation history in the form of reproducible code. \n * `SpreadsheetWidget.filter_relevant_history(persist=True)`: Returns the transformation history that contains only code relevant to the final state of the spreadsheet. The `persist` parameter determines whether the internal state and the displayed history is also filtered.\n * `SpreadsheetWidget.reset_history()`: Clears the history of transformation.\n\nCustomizable Interface\n----------------------\n\nThe spreadsheet widget provides a number of options that allows the user to change the appearance and the interactivity of the spreadsheet. Options include:\n\n * Row height/Column width\n * Preventing edits, sorts, or filters on the whole spreadsheet or on a per-column basis\n * Hiding the toolbar and history cell\n * Float precision\n * Highlighting of cells and rows\n * Viewport size\n\nConverting Spreadsheets To and From Dataframes\n----------------------------------------------\n\n.. automodule:: modin.experimental.spreadsheet.general\n :noindex:\n :members: from_dataframe\n\n \n.. automodule:: modin.experimental.spreadsheet.general\n :noindex:\n :members: to_dataframe\n\n\nFurther API Documentation\n-------------------------\n\n.. automodule:: modin_spreadsheet.grid\n :noindex:\n :members: SpreadsheetWidget"
},
{
"path": "docs/usage_guide/benchmarking.rst",
"content": "Benchmarking Modin\n==================\n\nSummary\n-------\nTo benchmark a single Modin function, often turning on the\n:doc:`configuration variable ` variable\n:code:`BenchmarkMode` will suffice.\n\nThere is no simple way to benchmark more complex Modin workflows, though\nbenchmark mode or calling ``modin.utils.execute`` on Modin objects may be useful.\nThe :doc:`Modin logs ` may help you\nidentify bottlenecks in your code, and they may also help profile the execution\nof each Modin function.\n\nModin's execution and benchmark mode\n------------------------------------\n\nMost of Modin's execution happens asynchronously, i.e. in separate processes that run\nindependently of the main program flow. Some execution is also lazy, meaning that it\ndoesn't start immediately once the user calls a Modin function. While Modin provides\nthe same API as pandas, lazy and asynchronous execution can often make it hard to\ntell how much time each Modin function call takes, as well as to compare Modin's\nperformance to pandas and other similar libraries.\n\n.. note::\n All examples in this doc use the system specified at the bottom of this page.\n\nConsider the following ipython script:\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.config import MinRowPartitionSize\n import time\n import ray\n\n # Look at the Ray documentation with respect to the Ray configuration suited to you most.\n ray.init()\n df = pd.DataFrame(list(range(MinRowPartitionSize.get() * 2)))\n %time result = df.map(lambda x: time.sleep(0.1) or x)\n %time print(result)\n\n\nModin takes just 2.68 milliseconds for the ``map``, and 3.78 seconds to print\nthe result. However, if we run this script in pandas by replacing\n:code:`import modin.pandas as pd` with :code:`import pandas as pd`, the ``map``\ntakes 6.63 seconds, and printing the result takes just 5.53 milliseconds.\n\nBoth pandas and Modin start executing the ``map`` as soon as the interpreter\nevalutes it. While pandas blocks until the ``map`` has finished, Modin just kicks\noff asynchronous functions in remote ray processes. Printing the function result\nis fairly fast in pandas and Modin, but before Modin can print the data, it has to\nwait until all the remote functions complete.\n\nTo time how long Modin takes for a single operation, you should typically use\nbenchmark mode. Benchmark mode will wait for all asynchronous remote execution\nto complete. You can turn on benchmark mode on at any point as follows:\n\n.. code-block:: python\n\n from modin.config import BenchmarkMode\n BenchmarkMode.put(True)\n\nRerunning the script above with benchmark mode on, the Modin ``map`` takes\n3.59 seconds, and the ``print`` takes 183 milliseconds. These timings better\nreflect where Modin is spending its execution time.\n\nA caveat about benchmark mode\n-----------------------------\n\nWhile benchmark code is often good for measuring the performance of a single\nModin function call, it can underestimate Modin's performance in cases where\nModin's asynchronous execution improves Modin's performance. Consider the\nfollowing script with benchmark mode on:\n\n.. code-block:: python\n\n import numpy as np\n import time\n import ray\n from io import BytesIO\n\n import modin.pandas as pd\n from modin.config import BenchmarkMode, MinRowPartitionSize\n\n BenchmarkMode.put(True)\n\n start = time.time()\n df = pd.DataFrame(list(range(MinRowPartitionSize.get())), columns=['A'])\n result1 = df.map(lambda x: time.sleep(0.2) or x + 1)\n result2 = df.map(lambda x: time.sleep(0.2) or x + 2)\n result1.to_parquet(BytesIO())\n result2.to_parquet(BytesIO())\n end = time.time()\n print(f'map and write to parquet took {end - start} seconds.')\n\n.. code-block::python\n\nThe script does two slow ``map`` on a dataframe and then writes each result\nto a buffer. The whole script takes 13 seconds with benchmark mode on, but\njust 7 seconds with benchmark mode off. Because Modin can run the ``map``\nasynchronously, it can start writing the first result to its buffer while\nit's still computing the second result. With benchmark mode on, Modin has to\nexecute every function synchronously instead.\n\nHow to benchmark complex workflows\n----------------------------------\n\nTypically, to benchmark Modin's overall performance on your workflow, you\nshould start by looking at end-to-end performance with benchmark mode off.\nIt's common for Modin worfklows to end with writing results to one or more\nfiles, or with printing some Modin objects to an interactive console. Such\nend points are natural ways to make sure that all of the Modin execution that\nyou require is complete.\n\nTo measure more fine-grained performance, it can be helpful to turn\nbenchmark mode on, but beware that doing so may reduce your script's overall\nperformance and thus may not reflect where Modin is normally spending execution\ntime, as pointed out above.\n\nTurning on :doc:`Modin logging ` and\nusing the Modin logs can also help you profile your workflow. The Modin logs\ncan also give a detailed break down of the performance of each Modin function\nat each Modin :doc:`layer `. Log mode is more\nuseful when used in conjuction with benchmark mode.\n\nSometimes, if you don't have a natural end-point to your workflow, you can\njust call ``modin.utils.execute`` on the workflow's final Modin objects.\nThat will typically block on any asynchronous computation:\n\n.. code-block:: python\n\n import time\n import ray\n from io import BytesIO\n\n import modin.pandas as pd\n from modin.config import MinRowPartitionSize, NPartitions\n import modin.utils\n\n MinRowPartitionSize.put(32)\n NPartitions.put(16)\n\n def slow_add_one(x):\n if x == 5000:\n time.sleep(10)\n return x + 1\n\n # Look at the Ray documentation with respect to the Ray configuration suited to you most.\n ray.init()\n df1 = pd.DataFrame(list(range(10_000)), columns=['A'])\n result = df1.map(slow_add_one)\n # %time modin.utils.execute(result)\n %time result.to_parquet(BytesIO())\n.. code-block::python\n\nWriting the result to a buffer takes 9.84 seconds. However, if you uncomment\nthe :code:`%time modin.utils.execute(result)` before the :code:`to_parquet`\ncall, the :code:`to_parquet` takes just 23.8 milliseconds!\n\n.. note::\n If you see any Modin documentation touting Modin's speed without using\n benchmark mode or otherwise guaranteeing that Modin is finishing all asynchronous\n and deferred computation, you should file an issue on the Modin GitHub. It's\n not fair to compare the speed of an async Modin function call to an equivalent\n synchronous call using another library.\n\nAppendix: System Information\n----------------------------\nThe example scripts here were run on the following system:\n\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS Monterey 12.4\n- **Modin version**: d6d503ac7c3028d871c34d9e99e925ddb0746df6\n- **Ray version**: 2.0.0\n- **Python version**: 3.10.4\n- **Machine**: MacBook Pro (16-inch, 2019)\n- **Processor**: 2.3 GHz 8-core Intel Core i9 processor\n- **Memory**: 16 GB 2667 MHz DDR4\n"
},
{
"path": "docs/usage_guide/examples/index.rst",
"content": "Modin Usage Examples\n====================\n\nThis section shows Modin usage examples in different scenarios like Modin on a local/remote cluster,\nthe use of Modin spreadsheet.\n\nTutorials\n'''''''''\n\nThe following tutorials cover the basic usage of Modin. `Here `_ is a one hour video tutorial that walks through these basic exercises.\n\n- Exercise 1: Introduction to Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]\n- Exercise 2: Speed Improvements with Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]\n- Exercise 3: Defaulting to pandas with Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]\n\nThe following tutorials covers more advanced features in Modin:\n\n- Exercise 4: Experimental Features in Modin (Spreadsheet, Progress Bar) [`Source PandasOnRay `__, `Source PandasOnDask `__]\n- Exercise 5: Setting up Modin in a Cluster Environment [`Source PandasOnRay `__]\n\nHow to get required dependencies for the tutorial notebooks and to run them please refer to the respective `README.md `__ file.\n\n\nData Science Benchmarks\n'''''''''''''''''''''''\n\n- Using Modin with the NYC Taxi Dataset [`Source `__]\n- Using Modin with the Census Dataset (coming soon...)\n- Using Modin with the Plasticc Dataset (coming soon...)\n\nModin Spreadsheets\n''''''''''''''''''\n\n- Using Modin along with the Spreadsheets API [`Source `__]\n\nModin with scikit-learn\n'''''''''''''''''''''''\n\n- Modin for Machine Learning with scikit-learn [`Source `__]\n"
},
{
"path": "docs/usage_guide/index.rst",
"content": "Usage Guide\n===========\n\nThis guide describes both basic and advanced Modin usage, including usage examples, \ndetails regarding Modin configuration settings, as well as tips and tricks on \nhow you can further optimize the performance of your workload with Modin.\n\n.. toctree::\n :maxdepth: 4\n\n /flow/modin/config\n examples/index\n advanced_usage/index\n optimization_notes/index\n benchmarking\n integrations\n\n.. meta::\n :description lang=en:\n Usage-specific documentation."
},
{
"path": "docs/usage_guide/integrations.rst",
"content": "Third Party Library Integrations\n================================\n\nModin is a drop-in replacement for Pandas, so we want it to interoperate with third-party libraries just as Pandas does. \nTo see where Modin performs well and where it needs to improve, we've selected a number of important machine learning + visualization + statistics libraries, \nand then looked at examples (from their documentation, if possible) about how they work with Pandas. Then we ran those same workflows with Modin, and \ntracked what worked, and what failed.\n\nIn the table below, you'll see, for each third-party library we tested, the number of successful test calls / total test calls, and a qualitative description of how both Pandas and Modin integrate with that library.\n\nIn the deeper dive, you can view the Jupyter notebook we have used to test API calls and the corresponding Github issues filed. If you come across other issues/ examples \nin your own workflows we encourage you to file an `issue `_ or contribute a `PR `_!\n\n\n.. note::\n These interoperability metrics are preliminary and not all APIs for each library have been tested. Feel free to add more!\n\n\nModin Interoperability by Library\n'''''''''''''''''''''''''''''''''\n.. list-table::\n :widths: 5 5 20\n :header-rows: 1\n\n * - Library\n - API successes / calls\n - Interoperability\n \n * - seaborn\n - 73% (11/15)\n - **Pandas**: Accepts Pandas DataFrames as inputs for producing plot |br|\n **Modin**: Mostly accepts Modin DataFrames as inputs for producing plots, but fails completely in some cases (pairplot, lmplot), and in others (catplot, objects.Plot) only works for some parameter combinations\n\n * - plotly\n - 78% (7 / 9)\n - **Pandas**: Accepts Pandas DataFrames as inputs for producing plots, including specifying X and Y parameters as df columns |br|\n **Modin**: Mostly accepts Modin DataFrames as inputs for producing plots (the exception is choropleth), but fails when specifying X and Y parameters as df columns\n \n * - matplotlib\n - 100% (5 / 5)\n - **Pandas**: Accepts Pandas DataFrames as inputs for producing plots like scatter, barh, etc. |br|\n **Modin**: Accepts Modin DataFrames as inputs for producing plots like scatter, barh, etc.\n \n * - altair\n - 0% (0 / 1)\n - **Pandas**: Accepts Pandas DataFrames as inputs for producing charts through Chart |br|\n **Modin**: Does not accept Modin DataFrames as inputs for producing charts through Chart\n\n * - bokeh\n - 0% (0 / 1)\n - **Pandas**: Loads Pandas DataFrames through ColumnDataSource |br|\n **Modin**: Does not load Modin DataFrames through ColumnDataSource\n \n * - sklearn\n - 100% (6 / 6)\n - **Pandas**: Many functions take Pandas DataFrames as inputs |br|\n **Modin**: Many functions take Modin DataFrames as inputs\n \n * - Hugging Face (Transformers, Datasets)\n - 100% (2 / 2) \n - **Pandas**: Loads Pandas DataFrames into Datasets, and processes Pandas DataFrame rows as inputs using Transformers.InputExample (deprecated) |br|\n **Modin**: Loads Modin DataFrames into Datasets (though slowly), and processes Modin DataFrame rows as inputs through Transformers.InputExample (deprecated)\n \n * - Tensorflow\n - 75% (3 / 4)\n - **Pandas**: Converts Pandas dataframes to tensors |br|\n **Modin**: Converts Modin DataFrames to tensors, but specialized APIs like Keras might not work yet\n \n * - NLTK\n - 100% (1 / 1)\n - **Pandas**: Performs transformations like tokenization on Pandas DataFrames |br|\n **Modin**: Performs transformations like tokenization on Modin DataFrames\n \n * - XGBoost\n - 100% (1 / 1)\n - **Pandas**: Loads Pandas DataFrames through the DMatrix function |br|\n **Modin**: Loads Modin DataFrames through the DMatrix function\n \n * - statsmodels\n - 50% (1 / 2)\n - **Pandas**: Can accept Pandas DataFrames when fitting models |br|\n **Modin**: Sometimes accepts Modin DataFrames when fitting models (e.g., formula.api.ols), but does not in others (e.g., api.OLS)\n \n.. |br| raw:: html\n\n \n\nA Deeper Dive\n''''''''''''''\n\n**seaborn**\n-----------\n\n`Jupyter Notebook `__\n\nGithub Issues\n * https://github.com/modin-project/modin/issues/5435 \n * https://github.com/modin-project/modin/issues/5433\n\n**plotly**\n----------\n\n`Jupyter Notebook `__\n\nGithub Issues\n * https://github.com/modin-project/modin/issues/5447 \n * https://github.com/modin-project/modin/issues/5445\n\n**matplotlib**\n--------------\n\n`Jupyter Notebook `__\n\n\n**altair**\n----------\n\n`Jupyter Notebook `__\n\nGithub Issues\n * https://github.com/modin-project/modin/issues/5438\n\n**bokeh**\n---------\n\n`Jupyter Notebook `__\n\nGithub Issues\n * https://github.com/modin-project/modin/issues/5437\n\n**sklearn**\n-----------\n\n`Jupyter Notebook `__\n\n**Hugging Face**\n----------------\n\n`Jupyter Notebook `__\n\n**Tensorflow**\n--------------\n\n`Jupyter Notebook `__\n\nGithub Issues\n * https://github.com/modin-project/modin/issues/5439\n\n**NLTK**\n---------\n\n`Jupyter Notebook `__\n\n**XGBoost**\n-----------\n\n`Jupyter Notebook `__\n\n**statsmodels**\n---------------\n\n`Jupyter Notebook `__\n\nGithub Issues\n * https://github.com/modin-project/modin/issues/5440\n\nAppendix: System Information\n'''''''''''''''''''''''''''''\nThe example scripts here were run on the following system:\n\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS Big Sur 11.5.2\n- **Modin version**: 0.18.0+3.g4114183f\n- **Ray version**: 2.0.1\n- **Python version**: 3.9.7.final.0\n- **Machine**: MacBook Pro (16-inch, 2019)\n- **Processor**: 2.3 GHz 8-core Intel Core i9 processor\n- **Memory**: 16 GB 2667 MHz DDR4\n"
},
{
"path": "docs/usage_guide/optimization_notes/index.rst",
"content": "Optimization Notes\n==================\n\nModin has chosen default values for a lot of the configurations here that provide excellent performance in most\ncases. This page is for those who love to optimize their code and those who are curious about existing optimizations\nwithin Modin. Here you can find more information about Modin's optimizations both for a pipeline of operations as\nwell as for specific operations. If you want to go ahead and tune the Modin behavior on your own, refer to\n:doc:`Modin Configuration Settings ` page for the full set of configurations available in Modin.\n\nRange-partitioning in Modin\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin utilizes a range-partitioning approach for specific operations, significantly enhancing\nparallelism and reducing memory consumption in certain scenarios. Range-partitioning is typically\nengaged for operations that has key columns (to group on, to merge on, etc).\n\nYou can enable `range-partitioning`_ by specifying ``cfg.RangePartitioning`` :doc:`configuration variable: `\n\n.. code-block:: python\n\n import modin.pandas as pd\n import modin.config as cfg\n\n cfg.RangePartitioning.put(True) # past this point methods that support range-partitioning\n # will use it\n\n pd.DataFrame(...).groupby(...).mean() # use range-partitioning for groupby.mean()\n\n cfg.Range-partitioning.put(False)\n\n pd.DataFrame(...).groupby(...).mean() # use MapReduce implementation for groupby.mean()\n\nBuilding range-partitioning assumes data reshuffling, which may result into breaking the original\norder of rows, for some operation, it will mean that the result will be different from Pandas.\n\nRange-partitioning is not a silver bullet, meaning that enabling it is not always beneficial. Below you find\na link to the list of operations that have support for range-partitioning and practical advices on when one should\nenable it: :doc:`operations that support range-partitioning `.\n\nDynamic-partitioning in Modin\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nRay engine experiences slowdowns when running a large number of small remote tasks at the same time. Ray Core recommends to `avoid tiny task`_.\nWhen modin DataFrame has a large number of partitions, some functions produce a large number of remote tasks, which can cause slowdowns. \nTo solve this problem, Modin suggests using dynamic partitioning. This approach reduces the number of remote tasks \nby combining multiple partitions into a single virtual partition and perform a common remote task on them.\n\nDynamic partitioning is typically used for operations that are fully or partially executed on all partitions separately.\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.config import context\n\n df = pd.DataFrame(...)\n\n with context(DynamicPartitioning=True):\n df.abs()\n\nDynamic partitioning is also not always useful, and this approach is usually used for medium-sized DataFrames with a large number of columns.\nIf the number of columns is small, the number of partitions will be close to the number of CPUs, and Ray will not have this problem.\nIf the DataFrame has too many rows, this is also not a good case for using Dynamic-partitioning, since each task is no longer tiny and performing \nthe combined tasks carries more overhead than assigning them separately.\n\nUnfortunately, the use of Dynamic-partitioning depends on various factors such as data size, number of CPUs, operations performed, \nand it is up to the user to determine whether Dynamic-partitioning will give a boost in his case or not.\n\n..\n TODO: Define heuristics to automatically enable dynamic partitioning without performance penalty.\n `Issue #7370 `_\n\nUnderstanding Modin's partitioning mechanism\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin's partitioning is crucial for performance; so we recommend expert users to understand Modin's\npartitioning mechanism and how to tune it in order to achieve better performance.\n\nHow Modin partitions a dataframe\n--------------------------------\n\nModin uses a partitioning scheme that partitions a dataframe along both axes, resulting in a matrix\nof partitions. The row and column chunk sizes are computed independently based\non the length of the appropriate axis and Modin's special :doc:`configuration variables `\n(``NPartitions``, ``MinRowPartitionSize`` and ``MinColumnPartitionSize``):\n\n- ``NPartitions`` is the maximum number of splits along an axis; by default, it equals to the number of cores\n on your local machine or cluster of nodes.\n- ``MinRowPartitionSize`` is the minimum number of rows to do a split. For instance, if ``MinRowPartitionSize``\n is 32, the row axis will not be split unless the amount of rows is greater than 32. If it is is greater, for example, 34,\n then the row axis is sliced into two partitions: containing 32 and 2 rows accordingly.\n- ``MinColumnPartitionSize`` is the minimum number of columns to do a split. For instance, if ``MinColumnPartitionSize``\n is 32, the column axis will not be split unless the amount of columns is greater than 32. If it is is greater, for example, 34,\n then the column axis is sliced into two partitions: containing 32 and 2 columns accordingly.\n\nBeware that ``NPartitions`` specifies a limit for the number of partitions `along a single axis`, which means, that\nthe actual limit for the entire dataframe itself is the square of ``NPartitions``.\n\n.. figure:: /img/partitioning_mechanism/partitioning_examples.svg\n :align: center\n\nFull-axis functions\n-------------------\n\nSome of the aggregation functions require knowledge about the entire axis, for example at ``.apply(foo, axis=0)``\nthe passed function ``foo`` expects to receive data for the whole column at once.\n\nWhen a full-axis function is applied, the partitions along this axis are collected at a single worker\nthat processes the function. After the function is done, the partitioning of the data is back to normal.\n\n.. figure:: /img/partitioning_mechanism/full_axis_function.svg\n :align: center\n\nNote that the amount of remote calls is equal to the number of partitions, which means that since the number\nof partitions is decreased for full-axis functions it also decreases the potential for parallelism.\n\nAlso note, that reduce functions such as ``.sum()``, ``.mean()``, ``.max()``, etc, are not considered\nto be full-axis, so they do not suffer from the decreasing level of parallelism.\n\nHow to tune partitioning\n------------------------\n\nConfigure Modin's default partitioning scheme\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAs you can see from the examples above, the more the dataframe's shape is closer to a square, the closer the number of\npartitions to the square of ``NPartitions``. In the case of ``NPartitions`` equals to the number of workers,\nthat means that a single worker is going to process multiple partitions at once, which slows down overall performance.\n\nIf your workflow mainly operates with wide dataframes and non-full-axis functions, it makes sense to reduce the\n``NPartitions`` value so a single worker would process a single partition.\n\n.. figure:: /img/partitioning_mechanism/repartition_square_frames.svg\n :align: center\n\nCopy-pastable example, showing how tuning ``NPartitions`` value for wide frames may improve performance on your machine:\n\n.. code-block:: python\n\n from multiprocessing import cpu_count\n from modin.distributed.dataframe.pandas import unwrap_partitions\n import modin.config as cfg\n import modin.pandas as pd\n import numpy as np\n import timeit\n\n # Generating data for a square-like dataframe\n data = np.random.randint(0, 100, size=(5000, 5000))\n\n # Explicitly setting `NPartitions` to its default value\n cfg.NPartitions.put(cpu_count())\n\n # Each worker processes `cpu_count()` amount of partitions\n df = pd.DataFrame(data)\n print(f\"NPartitions: {cfg.NPartitions.get()}\")\n # Getting raw partitions to count them\n partitions_shape = np.array(unwrap_partitions(df)).shape\n print(\n f\"The frame has {partitions_shape[0]}x{partitions_shape[1]}={np.prod(partitions_shape)} partitions \"\n f\"when the CPU has only {cpu_count()} cores.\"\n )\n print(f\"10 times of .abs(): {timeit.timeit(lambda: df.abs(), number=10)}s.\")\n # Possible output:\n # NPartitions: 112\n # The frame has 112x112=12544 partitions when the CPU has only 112 cores.\n # 10 times of .abs(): 23.64s.\n\n # Taking a square root of the the current `cpu_count` to make more even partitioning\n cfg.NPartitions.put(int(cpu_count() ** 0.5))\n\n # Each worker processes a single partition\n df = pd.DataFrame(data)\n print(f\"NPartitions: {cfg.NPartitions.get()}\")\n # Getting raw partitions to count them\n partitions_shape = np.array(unwrap_partitions(df)).shape\n print(\n f\"The frame has {partitions_shape[0]}x{partitions_shape[1]}={np.prod(partitions_shape)} \"\n f\"when the CPU has {cpu_count()} cores.\"\n )\n print(f\"10 times of .abs(): {timeit.timeit(lambda: df.abs(), number=10)}s.\")\n # Possible output:\n # NPartitions: 10\n # The frame has 10x10=100 partitions when the CPU has 112 cores.\n # 10 times of .abs(): 0.25s.\n\nManually trigger repartitioning\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nIf you're getting unexpectedly poor performance, although you configured ``MODIN_NPARTITIONS``\ncorrectly, then this might be caused by unbalanced partitioning that occurred during the\nworkflow's execution.\n\nModin's idealogy is to handle partitioning internally and not let users worry about the possible\nconsequences of applying a lot of \"bad\" operations that may affect DataFrame's partitioning.\nWe're constantly making efforts to find and fix cases where partitioning may cause a headache\nfor users.\n\nHowever, if you feel that you're dealing with unbalanced partitioning you may try to call an\ninternal :py:meth:`modin.pandas.dataframe.DataFrame._repartition` method on your :py:class:`~modin.pandas.dataframe.DataFrame` in order to manually\ntrigger partitions rebalancing and see whether it improves performance for your case.\n\n.. automethod:: modin.pandas.dataframe.DataFrame._repartition\n\nAn actual use-case for this method may be the following:\n\n.. code-block:: python\n\n import modin.pandas as pd\n import timeit\n\n df = pd.DataFrame({\"col0\": [1, 2, 3, 4]})\n\n # Appending a lot of columns may result into unbalanced partitioning\n for i in range(1, 128):\n df[f\"col{i}\"] = pd.Series([1, 2, 3, 4])\n\n print(\n \"DataFrame with unbalanced partitioning:\",\n timeit.timeit(lambda: df.sum(), number=10)\n ) # 1.44s\n\n df = df._repartition()\n print(\n \"DataFrame after '._repartition()':\",\n timeit.timeit(lambda: df.sum(), number=10)\n ) # 0.21s.\n\nAvoid iterating over Modin DataFrame\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nUse ``df.apply()`` or other aggregation methods when possible instead of iterating over a dataframe.\nFor-loops don't scale and forces the distributed data to be collected back at the driver.\n\nCopy-pastable example, showing how replacing a for-loop to the equivalent ``.apply()`` may improve performance:\n\n.. code-block:: python\n\n import modin.pandas as pd\n import numpy as np\n from timeit import default_timer as timer\n\n data = np.random.randint(1, 100, (2 ** 10, 2 ** 2))\n\n md_df = pd.DataFrame(data)\n\n result = []\n t1 = timer()\n # Iterating over a dataframe forces to collect distributed data to the driver and doesn't scale\n for idx, row in md_df.iterrows():\n result.append((row[1] + row[2]) / row[3])\n print(f\"Filling a list by iterating a Modin frame: {timer() - t1:.2f}s.\")\n # Possible output: 36.15s.\n\n t1 = timer()\n # Using `.apply()` perfectly scales to all axis-partitions\n result = md_df.apply(lambda row: (row[1] + row[2]) / row[3], axis=1).to_numpy().tolist()\n print(f\"Filling a list by using '.apply()' and converting the result to a list: {timer() - t1:.2f}s.\")\n # Possible output: 0.22s.\n\nUse Modin's Dataframe Algebra API to implement custom parallel functions\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nModin provides a set of low-level parallel-implemented operators which can be used to build most of the\naggregation functions. These operators are present in the :doc:`algebra module `.\nModin DataFrame allows users to use their own aggregations built with this module. Visit the\n:doc:`DataFrame's algebra ` page of the documentation for the steps to do it.\n\nAvoid mixing pandas and Modin DataFrames\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nAlthough Modin is considered to be a drop-in replacement for pandas, Modin and pandas are not intended to be used together\nin a single flow. Passing a pandas DataFrame as an argument for a Modin's DataFrame method may either slowdown\nthe function (because it has to process non-distributed object) or raise an error. You would also get an undefined\nbehavior if you pass a Modin DataFrame as an input to pandas methods, since pandas identifies Modin's objects as a simple iterable,\nand so can't leverage its benefits as a distributed dataframe.\n\nCopy-pastable example, showing how mixing pandas and Modin DataFrames in a single flow may bottleneck performance:\n\n.. code-block:: python\n\n import modin.pandas as pd\n import numpy as np\n import timeit\n import pandas\n\n data = np.random.randint(0, 100, (2 ** 20, 2 ** 2))\n\n md_df, md_df_copy = pd.DataFrame(data), pd.DataFrame(data)\n pd_df, pd_df_copy = pandas.DataFrame(data), pandas.DataFrame(data)\n\n print(\"concat modin frame + pandas frame:\")\n # Concatenating modin frame + pandas frame using modin '.concat()'\n # This case is bad because Modin have to process non-distributed pandas object\n time = timeit.timeit(lambda: pd.concat([md_df, pd_df]), number=10)\n print(f\"\\t{time}s.\\n\")\n # Possible output: 0.44s.\n\n print(\"concat modin frame + modin frame:\")\n # Concatenating modin frame + modin frame using modin '.concat()'\n # This is an ideal case, Modin is being used as intended\n time = timeit.timeit(lambda: pd.concat([md_df, md_df_copy]), number=10)\n print(f\"\\t{time}s.\\n\")\n # Possible output: 0.05s.\n\n print(\"concat pandas frame + pandas frame:\")\n # Concatenating pandas frame + pandas frame using pandas '.concat()'\n time = timeit.timeit(lambda: pandas.concat([pd_df, pd_df_copy]), number=10)\n print(f\"\\t{time}s.\\n\")\n # Possible output: 0.31s.\n\n print(\"concat pandas frame + modin frame:\")\n # Concatenating pandas frame + modin frame using pandas '.concat()'\n time = timeit.timeit(lambda: pandas.concat([pd_df, md_df]), number=10)\n print(f\"\\t{time}s.\\n\")\n # Possible output: TypeError\n\n\nUsing pandas to execute queries with Modin's ``\"Pandas\"`` backend\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nBy default, Modin distributes the data in a dataframe (or series) and attempts\nto process data for different partitions in parallel.\n\nHowever, for certain scenarios, such as handling small datasets, Modin's\nparallel execution may introduce unnecessary overhead. In such cases, it's more\nefficient to use serial execution with a single, unpartitioned pandas dataframe.\nYou can enable this kind of local pandas execution by setting Modin's\n``Backend``\n:doc:`configuration variable ` to ``\"Pandas\"``.\n\nDataFrames created while Modin's global backend is set to ``\"Pandas\"``\nwill continue to use native execution even if you switch the global backend\nlater. Modin supports interoperability between distributed Modin DataFrames\nand those using the pandas backend.\n\nHere is an example of using the pandas backend.\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.config import Backend\n\n # This dataframe will use Modin's default, distributed execution.\n original_backend = Backend.get()\n assert original_backend != \"Pandas\"\n distributed_df_1 = pd.DataFrame([0])\n\n # Set backend to \"Pandas\" for local pandas execution.\n Backend.put(\"Pandas\")\n modin_on_pandas_df = pd.DataFrame([1])\n assert modin_on_pandas_df.get_backend() == \"Pandas\"\n\n # Revert to default settings for distributed execution\n Backend.put(original_backend)\n distributed_df_2 = pd.DataFrame([2])\n assert distributed_df_2.get_backend() == original_backend\n\nYou can also use the pandas backend for some dataframes while using different\nbackends for other dataframes. You can switch the backend of an individual\ndataframe or series with ``set_backend()`` or its synonym ``move_to()``.\nHere's an example of switching the backend for an individual dataframe.\n\n.. code-block:: python\n\n import modin.pandas as pd\n\n # This dataframe will use Modin's default, distributed execution.\n original_backend = Backend.get()\n assert original_backend != \"Pandas\"\n distributed_df_1 = pd.DataFrame([0])\n\n pandas_df_1 = distributed_df_1.move_to(\"Pandas\")\n assert pandas_df_1.get_backend() == \"Pandas\"\n pandas_df_1 = pandas_df_1.sort_values(0)\n assert pandas_df_1.get_backend() == \"Pandas\"\n\n new_df = pandas_df_1.move_to(original_backend)\n assert new_df.get_backend() == original_backend\n\n new_df.set_backend(\"Pandas\", inplace=True)\n assert new_df.get_backend() == \"Pandas\"\n\n\nAutomatic backend switching\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\n*This feature is under active development, and the API is subject to change.*\n\nModin's backends may define heuristics for whether to automatically move data to another backend\nfor more efficient computation of certain operations. Modin does not currently define these heuristics\nfor any of its default backends, but any backends that wish to do so should implement the query\ncompiler methods discussed in\n:ref:`the architecture document`.\n\nAfter implementing the relevant query compiler methods, the following APIs can be used to control\nwhen automatic switching occurs:\n\n.. code-block:: python\n\n import modin.pandas as pd\n from modin.core.storage_formats.pandas.query_compiler_caster import (\n register_function_for_post_op_switch,\n register_function_for_pre_op_switch,\n )\n from modin.config import AutoSwitchBackend\n\n # Enable automatic switching BEFORE computation for DataFrame.apply\n # when the DataFrame's backend is Pandas\n register_function_for_pre_op_switch(\n class_name=\"DataFrame\",\n method=\"apply\",\n backend=\"Pandas\",\n )\n\n # Enable automatic switching AFTER computation for Series.max\n # when the Series's backend is Pandas\n register_function_for_post_op_switch(\n class_name=\"Series\",\n method=\"max\",\n backend=\"Pandas\",\n )\n\n # Enable automatic switching globally (use .disable() to turn off)\n AutoSwitchBackend.enable()\n\n df = pd.DataFrame([[1, 2, 3]])\n # \"pin\" a single DataFrame/Series, preventing it from\n # automatically switching backends\n df.pin_backend(inplace=True)\n # \"unpin\" it to re-enable automatic switching\n df.unpin_backend(inplace=True)\n\n\nOperation-specific optimizations\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nmerge\n-----\n\n``merge`` operation in Modin uses the broadcast join algorithm: combining a right Modin DataFrame into a pandas DataFrame and\nbroadcasting it to the row partitions of the left Modin DataFrame. In order to minimize interprocess communication cost when doing\nan inner join you may want to swap left and right DataFrames.\n\n.. code-block:: python\n\n import modin.pandas as pd\n import numpy as np\n\n left_data = np.random.randint(0, 100, size=(2**8, 2**8))\n right_data = np.random.randint(0, 100, size=(2**12, 2**12))\n\n left_df = pd.DataFrame(left_data)\n right_df = pd.DataFrame(right_data)\n %timeit left_df.merge(right_df, how=\"inner\", on=10)\n 3.59 s 107 ms per loop (mean std. dev. of 7 runs, 1 loop each)\n\n %timeit right_df.merge(left_df, how=\"inner\", on=10)\n 1.22 s 40.1 ms per loop (mean std. dev. of 7 runs, 1 loop each)\n\nNote that result columns order may differ for first and second ``merge``.\n\n.. _range-partitioning: https://www.techopedia.com/definition/31994/range-partitioning\n.. _`avoid tiny task`: https://docs.ray.io/en/latest/ray-core/tips-for-first-time.html#tip-2-avoid-tiny-tasks\n"
},
{
"path": "docs/usage_guide/optimization_notes/range_partitioning_ops.rst",
"content": ":orphan:\n\nOperations that support range-partitioning in Modin\n###################################################\n\nThe following operations change their behavior once ``cfg.RangePartitioning`` variable is set to ``True``.\nGo through the list find out when it could be beneficial to engage range-partitioning for a certain method.\n\nGroupBy\n=======\n\n.. note::\n When grouping on multiple columns using range-partitioning implementation, the result\n may not be sorted even if ``groupby(sort=True, ...)`` was passed: https://github.com/modin-project/modin/issues/6875.\n\nRange-partitioning groupby implementation is automatically engaged for ``groupby.apply()``, ``groupby.transform()``,\n``groupby.rolling()``. For groupby aggregations from `this list`_, MapReduce implementation is used by default.\nMapReduce tends to show better performance for groupby with low-cardinality. If the cardinality of your columns\nto group is expected to be high, it's recommended to engage range-partitioning implementation.\n\n\nMerge\n=====\n\n.. note::\n Range-partitioning approach is implemented only for \"left\" and \"inner\" merge and only\n when merging on a single column using `on` argument.\n\nRange-partitioning merge replaces broadcast merge. It is recommended to use range-partitioning implementation\nif the right dataframe in merge is as big as the left dataframe. In this case, range-partitioning\nimplementation works faster and consumes less RAM.\n\nUnder the spoiler you can find performance comparison of range-partitioning and broadcast merge in different\nscenarios:\n\n.. raw:: html\n\n \n Performance measurements for merge\n\nThe performance was measured on `h2o join queries`_ using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores),\nwith the number of cores allocated for Modin limited by 44 (``MODIN_CPUS=44``).\n\nMeasurements for small 500mb data:\n\n.. image:: /img/range_partitioning_measurements/merge_h2o_500mb.jpg\n :align: center\n\nMeasurements for medium 5gb data:\n\n.. image:: /img/range_partitioning_measurements/merge_h2o_5gb.png\n :align: center\n\n.. raw:: html\n\n \n\n\n``.unique()`` and ``.drop_duplicates()``\n========================================\n\n.. note::\n When range-partitioning is enabled, both ``.unique()`` and ``.drop_duplicates()`` will\n yield results that are sorted along rows. If range-partitioning is disabled,\n the original order will be maintained.\n\nRange-partitioning implementation of ``.unique()`` / ``.drop_duplicates()`` works best when the input data size is big (more than\n5_000_000 rows) and when the output size is also expected to be big (no more than 80% values are duplicates).\n\nUnder the spoiler you can find performance comparisons in different scenarios:\n\n.. raw:: html\n\n \n Performance measurements for ``.unique()``\n\nThe performance was measured on randomly generated data using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores).\nThe `duplicate rate` shows the procentage of duplicated rows in the dataset. You can learn more about this micro-benchmark\nby reading its source code:\n\n.. raw:: html\n\n \n Micro-benchmark's source code\n\n.. code-block:: python\n\n import modin.pandas as pd\n import numpy as np\n import modin.config as cfg\n\n from modin.utils import execute\n from timeit import default_timer as timer\n import pandas\n\n cfg.CpuCount.put(16)\n\n def get_data(nrows, dtype):\n if dtype == int:\n return np.arange(nrows)\n elif dtype == float:\n return np.arange(nrows).astype(float)\n elif dtype == str:\n return np.array([f\"value{i}\" for i in range(nrows)])\n else:\n raise NotImplementedError(dtype)\n\n pd.DataFrame(np.arange(cfg.NPartitions.get() * cfg.MinRowPartitionSize.get())).to_numpy()\n\n nrows = [1_000_000, 5_000_000, 10_000_000, 25_000_000, 50_000_000, 100_000_000]\n duplicate_rate = [0, 0.1, 0.5, 0.95]\n dtypes = [int, str]\n use_range_part = [True, False]\n\n columns = pandas.MultiIndex.from_product([dtypes, duplicate_rate, use_range_part], names=[\"dtype\", \"duplicate rate\", \"use range-part\"])\n result = pandas.DataFrame(index=nrows, columns=columns)\n\n i = 0\n total_its = len(nrows) * len(duplicate_rate) * len(dtypes) * len(use_range_part)\n\n for dt in dtypes:\n for nrow in nrows:\n data = get_data(nrow, dt)\n np.random.shuffle(data)\n for dpr in duplicate_rate:\n data_c = data.copy()\n dupl_val = data_c[0]\n\n num_duplicates = int(dpr * nrow)\n dupl_indices = np.random.choice(np.arange(nrow), num_duplicates, replace=False)\n data_c[dupl_indices] = dupl_val\n\n for impl in use_range_part:\n print(f\"{round((i / total_its) * 100, 2)}%\")\n i += 1\n cfg.RangePartitioning.put(impl)\n\n sr = pd.Series(data_c)\n execute(sr)\n\n t1 = timer()\n # returns a list, so no need for materialization\n sr.unique()\n tm = timer() - t1\n print(nrow, dpr, dt, impl, tm)\n result.loc[nrow, (dt, dpr, impl)] = tm\n result.to_excel(\"unique.xlsx\")\n\n.. raw:: html\n\n \n\nMeasurements with 16 cores being allocated for Modin (``MODIN_CPUS=16``):\n\n.. image:: /img/range_partitioning_measurements/unique_16cpus.jpg\n :align: center\n\nMeasurements with 44 cores being allocated for Modin (``MODIN_CPUS=44``):\n\n.. image:: /img/range_partitioning_measurements/unique_44cpus.jpg\n :align: center\n\n.. raw:: html\n\n \n\n\n.. raw:: html\n\n \n Performance measurements for ``.drop_duplicates()``\n\nThe performance was measured on randomly generated data using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores).\nThe `duplicate rate` shows the procentage of duplicated rows in the dataset. The `subset size` shows the number of\ncolumns being specified as a ``subset`` parameter for ``df.drop_duplicates()``. You can learn more about this\nmicro-benchmark by reading its source code:\n\n.. raw:: html\n\n \n Micro-benchmark's source code\n\n.. code-block:: python\n\n import modin.pandas as pd\n import numpy as np\n import modin.config as cfg\n\n from modin.utils import execute\n from timeit import default_timer as timer\n import pandas\n\n cfg.CpuCount.put(16)\n\n pd.DataFrame(np.arange(cfg.NPartitions.get() * cfg.MinRowPartitionSize.get())).to_numpy()\n\n nrows = [1_000_000, 5_000_000, 10_000_000, 25_000_000]\n duplicate_rate = [0, 0.1, 0.5, 0.95]\n subset = [[\"col0\"], [\"col1\", \"col2\", \"col3\", \"col4\"], None]\n ncols = 15\n use_range_part = [True, False]\n\n columns = pandas.MultiIndex.from_product(\n [\n [len(sbs) if sbs is not None else ncols for sbs in subset],\n duplicate_rate,\n use_range_part\n ],\n names=[\"subset size\", \"duplicate rate\", \"use range-part\"]\n )\n result = pandas.DataFrame(index=nrows, columns=columns)\n\n i = 0\n total_its = len(nrows) * len(duplicate_rate) * len(subset) * len(use_range_part)\n\n for sbs in subset:\n for nrow in nrows:\n data = {f\"col{i}\": np.arange(nrow) for i in range(ncols)}\n pandas_df = pandas.DataFrame(data)\n\n for dpr in duplicate_rate:\n pandas_df_c = pandas_df.copy()\n dupl_val = pandas_df_c.iloc[0]\n\n num_duplicates = int(dpr * nrow)\n dupl_indices = np.random.choice(np.arange(nrow), num_duplicates, replace=False)\n pandas_df_c.iloc[dupl_indices] = dupl_val\n\n for impl in use_range_part:\n print(f\"{round((i / total_its) * 100, 2)}%\")\n i += 1\n cfg.RangePartitioning.put(impl)\n\n md_df = pd.DataFrame(pandas_df_c)\n execute(md_df)\n\n t1 = timer()\n res = md_df.drop_duplicates(subset=sbs)\n execute(res)\n tm = timer() - t1\n\n sbs_s = len(sbs) if sbs is not None else ncols\n print(\"len()\", res.shape, nrow, dpr, sbs_s, impl, tm)\n result.loc[nrow, (sbs_s, dpr, impl)] = tm\n result.to_excel(\"drop_dupl.xlsx\")\n\n.. raw:: html\n\n \n\nMeasurements with 16 cores being allocated for Modin (``MODIN_CPUS=16``):\n\n.. image:: /img/range_partitioning_measurements/drop_duplicates_16cpus.jpg\n :align: center\n\nMeasurements with 44 cores being allocated for Modin (``MODIN_CPUS=44``):\n\n.. image:: /img/range_partitioning_measurements/drop_duplicates_44cpus.jpg\n :align: center\n\n.. raw:: html\n\n \n\n\n'.nunique()'\n============\n\n.. note::\n\n Range-partitioning approach is implemented only for ``pd.Series.nunique()`` and 1-column dataframes.\n For multi-column dataframes ``.nunique()`` can only use full-axis reduce implementation.\n\nRange-partitioning implementation of '.nunique()'' works best when the input data size is big (more than\n5_000_000 rows) and when the output size is also expected to be big (no more than 80% values are duplicates).\n\nUnder the spoiler you can find performance comparisons in different scenarios:\n\n.. raw:: html\n\n \n Performance measurements for ``.nunique()``\n\nThe performance was measured on randomly generated data using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores).\nThe `duplicate rate` shows the procentage of duplicated rows in the dataset. You can learn more about this micro-benchmark\nby reading its source code:\n\n.. raw:: html\n\n \n Micro-benchmark's source code\n\n.. code-block:: python\n\n import modin.pandas as pd\n import numpy as np\n import modin.config as cfg\n\n from modin.utils import execute\n from timeit import default_timer as timer\n import pandas\n\n cfg.CpuCount.put(16)\n\n def get_data(nrows, dtype):\n if dtype == int:\n return np.arange(nrows)\n elif dtype == float:\n return np.arange(nrows).astype(float)\n elif dtype == str:\n return np.array([f\"value{i}\" for i in range(nrows)])\n else:\n raise NotImplementedError(dtype)\n\n pd.DataFrame(np.arange(cfg.NPartitions.get() * cfg.MinRowPartitionSize.get())).to_numpy()\n\n nrows = [1_000_000, 5_000_000, 10_000_000, 25_000_000, 50_000_000, 100_000_000]\n duplicate_rate = [0, 0.1, 0.5, 0.95]\n dtypes = [int, str]\n use_range_part = [True, False]\n\n columns = pandas.MultiIndex.from_product([dtypes, duplicate_rate, use_range_part], names=[\"dtype\", \"duplicate rate\", \"use range-part\"])\n result = pandas.DataFrame(index=nrows, columns=columns)\n\n i = 0\n total_its = len(nrows) * len(duplicate_rate) * len(dtypes) * len(use_range_part)\n\n for dt in dtypes:\n for nrow in nrows:\n data = get_data(nrow, dt)\n np.random.shuffle(data)\n for dpr in duplicate_rate:\n data_c = data.copy()\n dupl_val = data_c[0]\n\n num_duplicates = int(dpr * nrow)\n dupl_indices = np.random.choice(np.arange(nrow), num_duplicates, replace=False)\n data_c[dupl_indices] = dupl_val\n\n for impl in use_range_part:\n print(f\"{round((i / total_its) * 100, 2)}%\")\n i += 1\n cfg.RangePartitioning.put(impl)\n\n sr = pd.Series(data_c)\n execute(sr)\n\n t1 = timer()\n # returns a scalar, so no need for materialization\n res = sr.nunique()\n tm = timer() - t1\n print(nrow, dpr, dt, impl, tm)\n result.loc[nrow, (dt, dpr, impl)] = tm\n result.to_excel(\"nunique.xlsx\")\n\n.. raw:: html\n\n \n\nMeasurements with 16 cores being allocated for Modin (``MODIN_CPUS=16``):\n\n.. image:: /img/range_partitioning_measurements/nunique_16cpus.jpg\n :align: center\n\n\n.. raw:: html\n\n \n\nResample\n========\n\n.. note::\n\n Range-partitioning approach doesn't support transform-like functions (like `.interpolate()`, `.ffill()`, `.bfill()`, ...)\n\nIt is recommended to use range-partitioning for resampling if you're dealing with a dataframe that has more than\n5_000_000 rows and the expected output is also expected to be big (more than 500_000 rows).\n\nUnder the spoiler you can find performance comparisons in different scenarios:\n\n.. raw:: html\n\n \n Performance measurements for ``.resample()``\n\nThe script below measures performance of ``df.resample(rule).sum()`` using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores).\nYou can learn more about this micro-benchmark by reading its source code:\n\n.. raw:: html\n\n \n Micro-benchmark's source code\n\n.. code-block:: python\n\n import pandas\n import numpy as np\n import modin.pandas as pd\n import modin.config as cfg\n\n from timeit import default_timer as timer\n\n from modin.utils import execute\n\n cfg.CpuCount.put(16)\n\n nrows = [1_000_000, 5_000_000, 10_000_000]\n ncols = [5, 33]\n rules = [\n \"500ms\", # doubles nrows\n \"30s\", # decreases nrows in 30 times\n \"5min\", # decreases nrows in 300\n ]\n use_rparts = [True, False]\n\n cols = pandas.MultiIndex.from_product([rules, ncols, use_rparts], names=[\"rule\", \"ncols\", \"USE RANGE PART\"])\n rres = pandas.DataFrame(index=nrows, columns=cols)\n\n total_nits = len(nrows) * len(ncols) * len(rules) * len(use_rparts)\n i = 0\n\n for nrow in nrows:\n for ncol in ncols:\n index = pandas.date_range(\"31/12/2000\", periods=nrow, freq=\"s\")\n data = {f\"col{i}\": np.arange(nrow) for i in range(ncol)}\n pd_df = pandas.DataFrame(data, index=index)\n for rule in rules:\n for rparts in use_rparts:\n print(f\"{round((i / total_nits) * 100, 2)}%\")\n i += 1\n cfg.RangePartitioning.put(rparts)\n\n df = pd.DataFrame(data, index=index)\n execute(df)\n\n t1 = timer()\n res = df.resample(rule).sum()\n execute(res)\n ts = timer() - t1\n print(nrow, ncol, rule, rparts, ts)\n\n rres.loc[nrow, (rule, ncol, rparts)] = ts\n rres.to_excel(\"resample.xlsx\")\n\n.. raw:: html\n\n \n\nMeasurements with 16 cores being allocated for Modin (``MODIN_CPUS=16``):\n\n.. image:: /img/range_partitioning_measurements/resample_16cpus.jpg\n :align: center\n\n\n.. raw:: html\n\n \n\npivot_table\n===========\n\nRange-partitioning implementation is automatically applied for ``df.pivot_table``\nwhenever possible, users can't control this.\n\nsort_values\n===========\n\nRange-partitioning implementation is automatically applied for ``df.sort_values``\nwhenever possible, users can't control this.\n\n\n.. _h2o join queries: https://h2oai.github.io/db-benchmark/\n.. _this list: https://github.com/modin-project/modin/blob/7b233e4a920d5f03dce7a82847847b92ae7ad617/modin/core/storage_formats/pandas/groupby.py#L236-L247\n"
},
{
"path": "environment-dev.yml",
"content": "name: modin\nchannels:\n - conda-forge\ndependencies:\n - pip\n\n # required dependencies\n - pandas>=2.2,<2.4\n - numpy>=1.22.4\n - fsspec>=2022.11.0\n - packaging>=21.0\n - psutil>=5.8.0\n\n # optional dependencies\n # NOTE Keep the ray and dask dependencies in sync with the Linux and Windows\n # Unidist environment dependencies.\n - ray-core>=2.10.0,<3\n - pyarrow>=10.0.1\n # workaround for https://github.com/conda/conda/issues/11744\n - grpcio!=1.45.*\n - grpcio!=1.46.*\n - dask>=2.22.0\n - distributed>=2.22.0\n - xarray>=2022.12.0\n - jinja2>=3.1.2\n - scipy>=1.10.0\n - s3fs>=2022.11.0\n - lxml>=4.9.2\n - openpyxl>=3.1.0\n - xlrd>=2.0.1\n - matplotlib>=3.6.3\n - sqlalchemy>=2.0.0\n - pandas-gbq>=0.19.0\n - pytables>=3.8.0\n # pymssql==2.2.8 broken: https://github.com/modin-project/modin/issues/6429\n - pymssql>=2.1.5,!=2.2.8\n - psycopg2>=2.9.6\n - fastparquet>=2022.12.0\n - tqdm>=4.60.0\n - numexpr>=2.8.4\n\n # dependencies for making release\n - pygithub>=v1.58.0\n - pygit2>=1.9.2\n\n # test dependencies\n - coverage>=7.1.0\n - moto>=4.1.0\n - pytest>=7.3.2\n - pytest-benchmark>=4.0.0\n - pytest-cov>=4.0.0\n - pytest-xdist>=3.2.0\n - typing_extensions\n\n # code linters\n - black>=24.1.0\n - flake8>=6.0.0\n - flake8-no-implicit-concat>=0.3.4\n - flake8-print>=5.0.0\n - mypy>=1.0.0\n - pandas-stubs>=2.0.0\n - isort>=5.12\n\n - pip:\n - dataframe-api-compat>=0.2.7\n - asv==0.5.1\n # no conda package for windows so we install it with pip\n - connectorx>=0.2.6a4\n - fuzzydata>=0.0.11\n # Fixes breaking ipywidgets changes, but didn't release yet.\n - git+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5\n # The `numpydoc` version should match the version installed in the `lint-pydocstyle` job of the CI.\n - numpydoc==1.6.0\n - polars\n"
},
{
"path": "examples/data/boston_housing.csv",
"content": ",CRIM,ZN,INDUS,CHAS,NOX,RM,AGE,DIS,RAD,TAX,PTRATIO,B,LSTAT,PRICE\n0,0.00632,18.0,2.31,0.0,0.538,6.575,65.2,4.09,1.0,296.0,15.3,396.9,4.98,24.0\n1,0.02731,0.0,7.07,0.0,0.469,6.421,78.9,4.9671,2.0,242.0,17.8,396.9,9.14,21.6\n2,0.02729,0.0,7.07,0.0,0.469,7.185,61.1,4.9671,2.0,242.0,17.8,392.83,4.03,34.7\n3,0.03237,0.0,2.18,0.0,0.458,6.998,45.8,6.0622,3.0,222.0,18.7,394.63,2.94,33.4\n4,0.06905,0.0,2.18,0.0,0.458,7.147,54.2,6.0622,3.0,222.0,18.7,396.9,5.33,36.2\n5,0.02985,0.0,2.18,0.0,0.458,6.43,58.7,6.0622,3.0,222.0,18.7,394.12,5.21,28.7\n6,0.08829,12.5,7.87,0.0,0.524,6.012,66.6,5.5605,5.0,311.0,15.2,395.6,12.43,22.9\n7,0.14455,12.5,7.87,0.0,0.524,6.172,96.1,5.9505,5.0,311.0,15.2,396.9,19.15,27.1\n8,0.21124,12.5,7.87,0.0,0.524,5.631,100.0,6.0821,5.0,311.0,15.2,386.63,29.93,16.5\n9,0.17004,12.5,7.87,0.0,0.524,6.004,85.9,6.5921,5.0,311.0,15.2,386.71,17.1,18.9\n10,0.22489,12.5,7.87,0.0,0.524,6.377,94.3,6.3467,5.0,311.0,15.2,392.52,20.45,15.0\n11,0.11747,12.5,7.87,0.0,0.524,6.009,82.9,6.2267,5.0,311.0,15.2,396.9,13.27,18.9\n12,0.09378,12.5,7.87,0.0,0.524,5.889,39.0,5.4509,5.0,311.0,15.2,390.5,15.71,21.7\n13,0.62976,0.0,8.14,0.0,0.538,5.949,61.8,4.7075,4.0,307.0,21.0,396.9,8.26,20.4\n14,0.63796,0.0,8.14,0.0,0.538,6.096,84.5,4.4619,4.0,307.0,21.0,380.02,10.26,18.2\n15,0.62739,0.0,8.14,0.0,0.538,5.834,56.5,4.4986,4.0,307.0,21.0,395.62,8.47,19.9\n16,1.05393,0.0,8.14,0.0,0.538,5.935,29.3,4.4986,4.0,307.0,21.0,386.85,6.58,23.1\n17,0.7842,0.0,8.14,0.0,0.538,5.99,81.7,4.2579,4.0,307.0,21.0,386.75,14.67,17.5\n18,0.80271,0.0,8.14,0.0,0.538,5.456,36.6,3.7965,4.0,307.0,21.0,288.99,11.69,20.2\n19,0.7258,0.0,8.14,0.0,0.538,5.727,69.5,3.7965,4.0,307.0,21.0,390.95,11.28,18.2\n20,1.25179,0.0,8.14,0.0,0.538,5.57,98.1,3.7979,4.0,307.0,21.0,376.57,21.02,13.6\n21,0.85204,0.0,8.14,0.0,0.538,5.965,89.2,4.0123,4.0,307.0,21.0,392.53,13.83,19.6\n22,1.23247,0.0,8.14,0.0,0.538,6.142,91.7,3.9769,4.0,307.0,21.0,396.9,18.72,15.2\n23,0.98843,0.0,8.14,0.0,0.538,5.813,100.0,4.0952,4.0,307.0,21.0,394.54,19.88,14.5\n24,0.75026,0.0,8.14,0.0,0.538,5.924,94.1,4.3996,4.0,307.0,21.0,394.33,16.3,15.6\n25,0.84054,0.0,8.14,0.0,0.538,5.599,85.7,4.4546,4.0,307.0,21.0,303.42,16.51,13.9\n26,0.67191,0.0,8.14,0.0,0.538,5.813,90.3,4.682,4.0,307.0,21.0,376.88,14.81,16.6\n27,0.95577,0.0,8.14,0.0,0.538,6.047,88.8,4.4534,4.0,307.0,21.0,306.38,17.28,14.8\n28,0.77299,0.0,8.14,0.0,0.538,6.495,94.4,4.4547,4.0,307.0,21.0,387.94,12.8,18.4\n29,1.00245,0.0,8.14,0.0,0.538,6.674,87.3,4.239,4.0,307.0,21.0,380.23,11.98,21.0\n30,1.13081,0.0,8.14,0.0,0.538,5.713,94.1,4.233,4.0,307.0,21.0,360.17,22.6,12.7\n31,1.35472,0.0,8.14,0.0,0.538,6.072,100.0,4.175,4.0,307.0,21.0,376.73,13.04,14.5\n32,1.38799,0.0,8.14,0.0,0.538,5.95,82.0,3.99,4.0,307.0,21.0,232.6,27.71,13.2\n33,1.15172,0.0,8.14,0.0,0.538,5.701,95.0,3.7872,4.0,307.0,21.0,358.77,18.35,13.1\n34,1.61282,0.0,8.14,0.0,0.538,6.096,96.9,3.7598,4.0,307.0,21.0,248.31,20.34,13.5\n35,0.06417,0.0,5.96,0.0,0.499,5.933,68.2,3.3603,5.0,279.0,19.2,396.9,9.68,18.9\n36,0.09744,0.0,5.96,0.0,0.499,5.841,61.4,3.3779,5.0,279.0,19.2,377.56,11.41,20.0\n37,0.08014,0.0,5.96,0.0,0.499,5.85,41.5,3.9342,5.0,279.0,19.2,396.9,8.77,21.0\n38,0.17505,0.0,5.96,0.0,0.499,5.966,30.2,3.8473,5.0,279.0,19.2,393.43,10.13,24.7\n39,0.02763,75.0,2.95,0.0,0.428,6.595,21.8,5.4011,3.0,252.0,18.3,395.63,4.32,30.8\n40,0.03359,75.0,2.95,0.0,0.428,7.024,15.8,5.4011,3.0,252.0,18.3,395.62,1.98,34.9\n41,0.12744,0.0,6.91,0.0,0.448,6.77,2.9,5.7209,3.0,233.0,17.9,385.41,4.84,26.6\n42,0.1415,0.0,6.91,0.0,0.448,6.169,6.6,5.7209,3.0,233.0,17.9,383.37,5.81,25.3\n43,0.15936,0.0,6.91,0.0,0.448,6.211,6.5,5.7209,3.0,233.0,17.9,394.46,7.44,24.7\n44,0.12269,0.0,6.91,0.0,0.448,6.069,40.0,5.7209,3.0,233.0,17.9,389.39,9.55,21.2\n45,0.17142,0.0,6.91,0.0,0.448,5.682,33.8,5.1004,3.0,233.0,17.9,396.9,10.21,19.3\n46,0.18836,0.0,6.91,0.0,0.448,5.786,33.3,5.1004,3.0,233.0,17.9,396.9,14.15,20.0\n47,0.22927,0.0,6.91,0.0,0.448,6.03,85.5,5.6894,3.0,233.0,17.9,392.74,18.8,16.6\n48,0.25387,0.0,6.91,0.0,0.448,5.399,95.3,5.87,3.0,233.0,17.9,396.9,30.81,14.4\n49,0.21977,0.0,6.91,0.0,0.448,5.602,62.0,6.0877,3.0,233.0,17.9,396.9,16.2,19.4\n50,0.08873,21.0,5.64,0.0,0.439,5.963,45.7,6.8147,4.0,243.0,16.8,395.56,13.45,19.7\n51,0.04337,21.0,5.64,0.0,0.439,6.115,63.0,6.8147,4.0,243.0,16.8,393.97,9.43,20.5\n52,0.0536,21.0,5.64,0.0,0.439,6.511,21.1,6.8147,4.0,243.0,16.8,396.9,5.28,25.0\n53,0.04981,21.0,5.64,0.0,0.439,5.998,21.4,6.8147,4.0,243.0,16.8,396.9,8.43,23.4\n54,0.0136,75.0,4.0,0.0,0.41,5.888,47.6,7.3197,3.0,469.0,21.1,396.9,14.8,18.9\n55,0.01311,90.0,1.22,0.0,0.403,7.249,21.9,8.6966,5.0,226.0,17.9,395.93,4.81,35.4\n56,0.02055,85.0,0.74,0.0,0.41,6.383,35.7,9.1876,2.0,313.0,17.3,396.9,5.77,24.7\n57,0.01432,100.0,1.32,0.0,0.411,6.816,40.5,8.3248,5.0,256.0,15.1,392.9,3.95,31.6\n58,0.15445,25.0,5.13,0.0,0.453,6.145,29.2,7.8148,8.0,284.0,19.7,390.68,6.86,23.3\n59,0.10328,25.0,5.13,0.0,0.453,5.927,47.2,6.932,8.0,284.0,19.7,396.9,9.22,19.6\n60,0.14932,25.0,5.13,0.0,0.453,5.741,66.2,7.2254,8.0,284.0,19.7,395.11,13.15,18.7\n61,0.17171,25.0,5.13,0.0,0.453,5.966,93.4,6.8185,8.0,284.0,19.7,378.08,14.44,16.0\n62,0.11027,25.0,5.13,0.0,0.453,6.456,67.8,7.2255,8.0,284.0,19.7,396.9,6.73,22.2\n63,0.1265,25.0,5.13,0.0,0.453,6.762,43.4,7.9809,8.0,284.0,19.7,395.58,9.5,25.0\n64,0.01951,17.5,1.38,0.0,0.4161,7.104,59.5,9.2229,3.0,216.0,18.6,393.24,8.05,33.0\n65,0.03584,80.0,3.37,0.0,0.398,6.29,17.8,6.6115,4.0,337.0,16.1,396.9,4.67,23.5\n66,0.04379,80.0,3.37,0.0,0.398,5.787,31.1,6.6115,4.0,337.0,16.1,396.9,10.24,19.4\n67,0.05789,12.5,6.07,0.0,0.409,5.878,21.4,6.498,4.0,345.0,18.9,396.21,8.1,22.0\n68,0.13554,12.5,6.07,0.0,0.409,5.594,36.8,6.498,4.0,345.0,18.9,396.9,13.09,17.4\n69,0.12816,12.5,6.07,0.0,0.409,5.885,33.0,6.498,4.0,345.0,18.9,396.9,8.79,20.9\n70,0.08826,0.0,10.81,0.0,0.413,6.417,6.6,5.2873,4.0,305.0,19.2,383.73,6.72,24.2\n71,0.15876,0.0,10.81,0.0,0.413,5.961,17.5,5.2873,4.0,305.0,19.2,376.94,9.88,21.7\n72,0.09164,0.0,10.81,0.0,0.413,6.065,7.8,5.2873,4.0,305.0,19.2,390.91,5.52,22.8\n73,0.19539,0.0,10.81,0.0,0.413,6.245,6.2,5.2873,4.0,305.0,19.2,377.17,7.54,23.4\n74,0.07896,0.0,12.83,0.0,0.437,6.273,6.0,4.2515,5.0,398.0,18.7,394.92,6.78,24.1\n75,0.09512,0.0,12.83,0.0,0.437,6.286,45.0,4.5026,5.0,398.0,18.7,383.23,8.94,21.4\n76,0.10153,0.0,12.83,0.0,0.437,6.279,74.5,4.0522,5.0,398.0,18.7,373.66,11.97,20.0\n77,0.08707,0.0,12.83,0.0,0.437,6.14,45.8,4.0905,5.0,398.0,18.7,386.96,10.27,20.8\n78,0.05646,0.0,12.83,0.0,0.437,6.232,53.7,5.0141,5.0,398.0,18.7,386.4,12.34,21.2\n79,0.08387,0.0,12.83,0.0,0.437,5.874,36.6,4.5026,5.0,398.0,18.7,396.06,9.1,20.3\n80,0.04113,25.0,4.86,0.0,0.426,6.727,33.5,5.4007,4.0,281.0,19.0,396.9,5.29,28.0\n81,0.04462,25.0,4.86,0.0,0.426,6.619,70.4,5.4007,4.0,281.0,19.0,395.63,7.22,23.9\n82,0.03659,25.0,4.86,0.0,0.426,6.302,32.2,5.4007,4.0,281.0,19.0,396.9,6.72,24.8\n83,0.03551,25.0,4.86,0.0,0.426,6.167,46.7,5.4007,4.0,281.0,19.0,390.64,7.51,22.9\n84,0.05059,0.0,4.49,0.0,0.449,6.389,48.0,4.7794,3.0,247.0,18.5,396.9,9.62,23.9\n85,0.05735,0.0,4.49,0.0,0.449,6.63,56.1,4.4377,3.0,247.0,18.5,392.3,6.53,26.6\n86,0.05188,0.0,4.49,0.0,0.449,6.015,45.1,4.4272,3.0,247.0,18.5,395.99,12.86,22.5\n87,0.07151,0.0,4.49,0.0,0.449,6.121,56.8,3.7476,3.0,247.0,18.5,395.15,8.44,22.2\n88,0.0566,0.0,3.41,0.0,0.489,7.007,86.3,3.4217,2.0,270.0,17.8,396.9,5.5,23.6\n89,0.05302,0.0,3.41,0.0,0.489,7.079,63.1,3.4145,2.0,270.0,17.8,396.06,5.7,28.7\n90,0.04684,0.0,3.41,0.0,0.489,6.417,66.1,3.0923,2.0,270.0,17.8,392.18,8.81,22.6\n91,0.03932,0.0,3.41,0.0,0.489,6.405,73.9,3.0921,2.0,270.0,17.8,393.55,8.2,22.0\n92,0.04203,28.0,15.04,0.0,0.464,6.442,53.6,3.6659,4.0,270.0,18.2,395.01,8.16,22.9\n93,0.02875,28.0,15.04,0.0,0.464,6.211,28.9,3.6659,4.0,270.0,18.2,396.33,6.21,25.0\n94,0.04294,28.0,15.04,0.0,0.464,6.249,77.3,3.615,4.0,270.0,18.2,396.9,10.59,20.6\n95,0.12204,0.0,2.89,0.0,0.445,6.625,57.8,3.4952,2.0,276.0,18.0,357.98,6.65,28.4\n96,0.11504,0.0,2.89,0.0,0.445,6.163,69.6,3.4952,2.0,276.0,18.0,391.83,11.34,21.4\n97,0.12083,0.0,2.89,0.0,0.445,8.069,76.0,3.4952,2.0,276.0,18.0,396.9,4.21,38.7\n98,0.08187,0.0,2.89,0.0,0.445,7.82,36.9,3.4952,2.0,276.0,18.0,393.53,3.57,43.8\n99,0.0686,0.0,2.89,0.0,0.445,7.416,62.5,3.4952,2.0,276.0,18.0,396.9,6.19,33.2\n100,0.14866,0.0,8.56,0.0,0.52,6.727,79.9,2.7778,5.0,384.0,20.9,394.76,9.42,27.5\n101,0.11432,0.0,8.56,0.0,0.52,6.781,71.3,2.8561,5.0,384.0,20.9,395.58,7.67,26.5\n102,0.22876,0.0,8.56,0.0,0.52,6.405,85.4,2.7147,5.0,384.0,20.9,70.8,10.63,18.6\n103,0.21161,0.0,8.56,0.0,0.52,6.137,87.4,2.7147,5.0,384.0,20.9,394.47,13.44,19.3\n104,0.1396,0.0,8.56,0.0,0.52,6.167,90.0,2.421,5.0,384.0,20.9,392.69,12.33,20.1\n105,0.13262,0.0,8.56,0.0,0.52,5.851,96.7,2.1069,5.0,384.0,20.9,394.05,16.47,19.5\n106,0.1712,0.0,8.56,0.0,0.52,5.836,91.9,2.211,5.0,384.0,20.9,395.67,18.66,19.5\n107,0.13117,0.0,8.56,0.0,0.52,6.127,85.2,2.1224,5.0,384.0,20.9,387.69,14.09,20.4\n108,0.12802,0.0,8.56,0.0,0.52,6.474,97.1,2.4329,5.0,384.0,20.9,395.24,12.27,19.8\n109,0.26363,0.0,8.56,0.0,0.52,6.229,91.2,2.5451,5.0,384.0,20.9,391.23,15.55,19.4\n110,0.10793,0.0,8.56,0.0,0.52,6.195,54.4,2.7778,5.0,384.0,20.9,393.49,13.0,21.7\n111,0.10084,0.0,10.01,0.0,0.547,6.715,81.6,2.6775,6.0,432.0,17.8,395.59,10.16,22.8\n112,0.12329,0.0,10.01,0.0,0.547,5.913,92.9,2.3534,6.0,432.0,17.8,394.95,16.21,18.8\n113,0.22212,0.0,10.01,0.0,0.547,6.092,95.4,2.548,6.0,432.0,17.8,396.9,17.09,18.7\n114,0.14231,0.0,10.01,0.0,0.547,6.254,84.2,2.2565,6.0,432.0,17.8,388.74,10.45,18.5\n115,0.17134,0.0,10.01,0.0,0.547,5.928,88.2,2.4631,6.0,432.0,17.8,344.91,15.76,18.3\n116,0.13158,0.0,10.01,0.0,0.547,6.176,72.5,2.7301,6.0,432.0,17.8,393.3,12.04,21.2\n117,0.15098,0.0,10.01,0.0,0.547,6.021,82.6,2.7474,6.0,432.0,17.8,394.51,10.3,19.2\n118,0.13058,0.0,10.01,0.0,0.547,5.872,73.1,2.4775,6.0,432.0,17.8,338.63,15.37,20.4\n119,0.14476,0.0,10.01,0.0,0.547,5.731,65.2,2.7592,6.0,432.0,17.8,391.5,13.61,19.3\n120,0.06899,0.0,25.65,0.0,0.581,5.87,69.7,2.2577,2.0,188.0,19.1,389.15,14.37,22.0\n121,0.07165,0.0,25.65,0.0,0.581,6.004,84.1,2.1974,2.0,188.0,19.1,377.67,14.27,20.3\n122,0.09299,0.0,25.65,0.0,0.581,5.961,92.9,2.0869,2.0,188.0,19.1,378.09,17.93,20.5\n123,0.15038,0.0,25.65,0.0,0.581,5.856,97.0,1.9444,2.0,188.0,19.1,370.31,25.41,17.3\n124,0.09849,0.0,25.65,0.0,0.581,5.879,95.8,2.0063,2.0,188.0,19.1,379.38,17.58,18.8\n125,0.16902,0.0,25.65,0.0,0.581,5.986,88.4,1.9929,2.0,188.0,19.1,385.02,14.81,21.4\n126,0.38735,0.0,25.65,0.0,0.581,5.613,95.6,1.7572,2.0,188.0,19.1,359.29,27.26,15.7\n127,0.25915,0.0,21.89,0.0,0.624,5.693,96.0,1.7883,4.0,437.0,21.2,392.11,17.19,16.2\n128,0.32543,0.0,21.89,0.0,0.624,6.431,98.8,1.8125,4.0,437.0,21.2,396.9,15.39,18.0\n129,0.88125,0.0,21.89,0.0,0.624,5.637,94.7,1.9799,4.0,437.0,21.2,396.9,18.34,14.3\n130,0.34006,0.0,21.89,0.0,0.624,6.458,98.9,2.1185,4.0,437.0,21.2,395.04,12.6,19.2\n131,1.19294,0.0,21.89,0.0,0.624,6.326,97.7,2.271,4.0,437.0,21.2,396.9,12.26,19.6\n132,0.59005,0.0,21.89,0.0,0.624,6.372,97.9,2.3274,4.0,437.0,21.2,385.76,11.12,23.0\n133,0.32982,0.0,21.89,0.0,0.624,5.822,95.4,2.4699,4.0,437.0,21.2,388.69,15.03,18.4\n134,0.97617,0.0,21.89,0.0,0.624,5.757,98.4,2.346,4.0,437.0,21.2,262.76,17.31,15.6\n135,0.55778,0.0,21.89,0.0,0.624,6.335,98.2,2.1107,4.0,437.0,21.2,394.67,16.96,18.1\n136,0.32264,0.0,21.89,0.0,0.624,5.942,93.5,1.9669,4.0,437.0,21.2,378.25,16.9,17.4\n137,0.35233,0.0,21.89,0.0,0.624,6.454,98.4,1.8498,4.0,437.0,21.2,394.08,14.59,17.1\n138,0.2498,0.0,21.89,0.0,0.624,5.857,98.2,1.6686,4.0,437.0,21.2,392.04,21.32,13.3\n139,0.54452,0.0,21.89,0.0,0.624,6.151,97.9,1.6687,4.0,437.0,21.2,396.9,18.46,17.8\n140,0.2909,0.0,21.89,0.0,0.624,6.174,93.6,1.6119,4.0,437.0,21.2,388.08,24.16,14.0\n141,1.62864,0.0,21.89,0.0,0.624,5.019,100.0,1.4394,4.0,437.0,21.2,396.9,34.41,14.4\n142,3.32105,0.0,19.58,1.0,0.871,5.403,100.0,1.3216,5.0,403.0,14.7,396.9,26.82,13.4\n143,4.0974,0.0,19.58,0.0,0.871,5.468,100.0,1.4118,5.0,403.0,14.7,396.9,26.42,15.6\n144,2.77974,0.0,19.58,0.0,0.871,4.903,97.8,1.3459,5.0,403.0,14.7,396.9,29.29,11.8\n145,2.37934,0.0,19.58,0.0,0.871,6.13,100.0,1.4191,5.0,403.0,14.7,172.91,27.8,13.8\n146,2.15505,0.0,19.58,0.0,0.871,5.628,100.0,1.5166,5.0,403.0,14.7,169.27,16.65,15.6\n147,2.36862,0.0,19.58,0.0,0.871,4.926,95.7,1.4608,5.0,403.0,14.7,391.71,29.53,14.6\n148,2.33099,0.0,19.58,0.0,0.871,5.186,93.8,1.5296,5.0,403.0,14.7,356.99,28.32,17.8\n149,2.73397,0.0,19.58,0.0,0.871,5.597,94.9,1.5257,5.0,403.0,14.7,351.85,21.45,15.4\n150,1.6566,0.0,19.58,0.0,0.871,6.122,97.3,1.618,5.0,403.0,14.7,372.8,14.1,21.5\n151,1.49632,0.0,19.58,0.0,0.871,5.404,100.0,1.5916,5.0,403.0,14.7,341.6,13.28,19.6\n152,1.12658,0.0,19.58,1.0,0.871,5.012,88.0,1.6102,5.0,403.0,14.7,343.28,12.12,15.3\n153,2.14918,0.0,19.58,0.0,0.871,5.709,98.5,1.6232,5.0,403.0,14.7,261.95,15.79,19.4\n154,1.41385,0.0,19.58,1.0,0.871,6.129,96.0,1.7494,5.0,403.0,14.7,321.02,15.12,17.0\n155,3.53501,0.0,19.58,1.0,0.871,6.152,82.6,1.7455,5.0,403.0,14.7,88.01,15.02,15.6\n156,2.44668,0.0,19.58,0.0,0.871,5.272,94.0,1.7364,5.0,403.0,14.7,88.63,16.14,13.1\n157,1.22358,0.0,19.58,0.0,0.605,6.943,97.4,1.8773,5.0,403.0,14.7,363.43,4.59,41.3\n158,1.34284,0.0,19.58,0.0,0.605,6.066,100.0,1.7573,5.0,403.0,14.7,353.89,6.43,24.3\n159,1.42502,0.0,19.58,0.0,0.871,6.51,100.0,1.7659,5.0,403.0,14.7,364.31,7.39,23.3\n160,1.27346,0.0,19.58,1.0,0.605,6.25,92.6,1.7984,5.0,403.0,14.7,338.92,5.5,27.0\n161,1.46336,0.0,19.58,0.0,0.605,7.489,90.8,1.9709,5.0,403.0,14.7,374.43,1.73,50.0\n162,1.83377,0.0,19.58,1.0,0.605,7.802,98.2,2.0407,5.0,403.0,14.7,389.61,1.92,50.0\n163,1.51902,0.0,19.58,1.0,0.605,8.375,93.9,2.162,5.0,403.0,14.7,388.45,3.32,50.0\n164,2.24236,0.0,19.58,0.0,0.605,5.854,91.8,2.422,5.0,403.0,14.7,395.11,11.64,22.7\n165,2.924,0.0,19.58,0.0,0.605,6.101,93.0,2.2834,5.0,403.0,14.7,240.16,9.81,25.0\n166,2.01019,0.0,19.58,0.0,0.605,7.929,96.2,2.0459,5.0,403.0,14.7,369.3,3.7,50.0\n167,1.80028,0.0,19.58,0.0,0.605,5.877,79.2,2.4259,5.0,403.0,14.7,227.61,12.14,23.8\n168,2.3004,0.0,19.58,0.0,0.605,6.319,96.1,2.1,5.0,403.0,14.7,297.09,11.1,23.8\n169,2.44953,0.0,19.58,0.0,0.605,6.402,95.2,2.2625,5.0,403.0,14.7,330.04,11.32,22.3\n170,1.20742,0.0,19.58,0.0,0.605,5.875,94.6,2.4259,5.0,403.0,14.7,292.29,14.43,17.4\n171,2.3139,0.0,19.58,0.0,0.605,5.88,97.3,2.3887,5.0,403.0,14.7,348.13,12.03,19.1\n172,0.13914,0.0,4.05,0.0,0.51,5.572,88.5,2.5961,5.0,296.0,16.6,396.9,14.69,23.1\n173,0.09178,0.0,4.05,0.0,0.51,6.416,84.1,2.6463,5.0,296.0,16.6,395.5,9.04,23.6\n174,0.08447,0.0,4.05,0.0,0.51,5.859,68.7,2.7019,5.0,296.0,16.6,393.23,9.64,22.6\n175,0.06664,0.0,4.05,0.0,0.51,6.546,33.1,3.1323,5.0,296.0,16.6,390.96,5.33,29.4\n176,0.07022,0.0,4.05,0.0,0.51,6.02,47.2,3.5549,5.0,296.0,16.6,393.23,10.11,23.2\n177,0.05425,0.0,4.05,0.0,0.51,6.315,73.4,3.3175,5.0,296.0,16.6,395.6,6.29,24.6\n178,0.06642,0.0,4.05,0.0,0.51,6.86,74.4,2.9153,5.0,296.0,16.6,391.27,6.92,29.9\n179,0.0578,0.0,2.46,0.0,0.488,6.98,58.4,2.829,3.0,193.0,17.8,396.9,5.04,37.2\n180,0.06588,0.0,2.46,0.0,0.488,7.765,83.3,2.741,3.0,193.0,17.8,395.56,7.56,39.8\n181,0.06888,0.0,2.46,0.0,0.488,6.144,62.2,2.5979,3.0,193.0,17.8,396.9,9.45,36.2\n182,0.09103,0.0,2.46,0.0,0.488,7.155,92.2,2.7006,3.0,193.0,17.8,394.12,4.82,37.9\n183,0.10008,0.0,2.46,0.0,0.488,6.563,95.6,2.847,3.0,193.0,17.8,396.9,5.68,32.5\n184,0.08308,0.0,2.46,0.0,0.488,5.604,89.8,2.9879,3.0,193.0,17.8,391.0,13.98,26.4\n185,0.06047,0.0,2.46,0.0,0.488,6.153,68.8,3.2797,3.0,193.0,17.8,387.11,13.15,29.6\n186,0.05602,0.0,2.46,0.0,0.488,7.831,53.6,3.1992,3.0,193.0,17.8,392.63,4.45,50.0\n187,0.07875,45.0,3.44,0.0,0.437,6.782,41.1,3.7886,5.0,398.0,15.2,393.87,6.68,32.0\n188,0.12579,45.0,3.44,0.0,0.437,6.556,29.1,4.5667,5.0,398.0,15.2,382.84,4.56,29.8\n189,0.0837,45.0,3.44,0.0,0.437,7.185,38.9,4.5667,5.0,398.0,15.2,396.9,5.39,34.9\n190,0.09068,45.0,3.44,0.0,0.437,6.951,21.5,6.4798,5.0,398.0,15.2,377.68,5.1,37.0\n191,0.06911,45.0,3.44,0.0,0.437,6.739,30.8,6.4798,5.0,398.0,15.2,389.71,4.69,30.5\n192,0.08664,45.0,3.44,0.0,0.437,7.178,26.3,6.4798,5.0,398.0,15.2,390.49,2.87,36.4\n193,0.02187,60.0,2.93,0.0,0.401,6.8,9.9,6.2196,1.0,265.0,15.6,393.37,5.03,31.1\n194,0.01439,60.0,2.93,0.0,0.401,6.604,18.8,6.2196,1.0,265.0,15.6,376.7,4.38,29.1\n195,0.01381,80.0,0.46,0.0,0.422,7.875,32.0,5.6484,4.0,255.0,14.4,394.23,2.97,50.0\n196,0.04011,80.0,1.52,0.0,0.404,7.287,34.1,7.309,2.0,329.0,12.6,396.9,4.08,33.3\n197,0.04666,80.0,1.52,0.0,0.404,7.107,36.6,7.309,2.0,329.0,12.6,354.31,8.61,30.3\n198,0.03768,80.0,1.52,0.0,0.404,7.274,38.3,7.309,2.0,329.0,12.6,392.2,6.62,34.6\n199,0.0315,95.0,1.47,0.0,0.403,6.975,15.3,7.6534,3.0,402.0,17.0,396.9,4.56,34.9\n200,0.01778,95.0,1.47,0.0,0.403,7.135,13.9,7.6534,3.0,402.0,17.0,384.3,4.45,32.9\n201,0.03445,82.5,2.03,0.0,0.415,6.162,38.4,6.27,2.0,348.0,14.7,393.77,7.43,24.1\n202,0.02177,82.5,2.03,0.0,0.415,7.61,15.7,6.27,2.0,348.0,14.7,395.38,3.11,42.3\n203,0.0351,95.0,2.68,0.0,0.4161,7.853,33.2,5.118,4.0,224.0,14.7,392.78,3.81,48.5\n204,0.02009,95.0,2.68,0.0,0.4161,8.034,31.9,5.118,4.0,224.0,14.7,390.55,2.88,50.0\n205,0.13642,0.0,10.59,0.0,0.489,5.891,22.3,3.9454,4.0,277.0,18.6,396.9,10.87,22.6\n206,0.22969,0.0,10.59,0.0,0.489,6.326,52.5,4.3549,4.0,277.0,18.6,394.87,10.97,24.4\n207,0.25199,0.0,10.59,0.0,0.489,5.783,72.7,4.3549,4.0,277.0,18.6,389.43,18.06,22.5\n208,0.13587,0.0,10.59,1.0,0.489,6.064,59.1,4.2392,4.0,277.0,18.6,381.32,14.66,24.4\n209,0.43571,0.0,10.59,1.0,0.489,5.344,100.0,3.875,4.0,277.0,18.6,396.9,23.09,20.0\n210,0.17446,0.0,10.59,1.0,0.489,5.96,92.1,3.8771,4.0,277.0,18.6,393.25,17.27,21.7\n211,0.37578,0.0,10.59,1.0,0.489,5.404,88.6,3.665,4.0,277.0,18.6,395.24,23.98,19.3\n212,0.21719,0.0,10.59,1.0,0.489,5.807,53.8,3.6526,4.0,277.0,18.6,390.94,16.03,22.4\n213,0.14052,0.0,10.59,0.0,0.489,6.375,32.3,3.9454,4.0,277.0,18.6,385.81,9.38,28.1\n214,0.28955,0.0,10.59,0.0,0.489,5.412,9.8,3.5875,4.0,277.0,18.6,348.93,29.55,23.7\n215,0.19802,0.0,10.59,0.0,0.489,6.182,42.4,3.9454,4.0,277.0,18.6,393.63,9.47,25.0\n216,0.0456,0.0,13.89,1.0,0.55,5.888,56.0,3.1121,5.0,276.0,16.4,392.8,13.51,23.3\n217,0.07013,0.0,13.89,0.0,0.55,6.642,85.1,3.4211,5.0,276.0,16.4,392.78,9.69,28.7\n218,0.11069,0.0,13.89,1.0,0.55,5.951,93.8,2.8893,5.0,276.0,16.4,396.9,17.92,21.5\n219,0.11425,0.0,13.89,1.0,0.55,6.373,92.4,3.3633,5.0,276.0,16.4,393.74,10.5,23.0\n220,0.35809,0.0,6.2,1.0,0.507,6.951,88.5,2.8617,8.0,307.0,17.4,391.7,9.71,26.7\n221,0.40771,0.0,6.2,1.0,0.507,6.164,91.3,3.048,8.0,307.0,17.4,395.24,21.46,21.7\n222,0.62356,0.0,6.2,1.0,0.507,6.879,77.7,3.2721,8.0,307.0,17.4,390.39,9.93,27.5\n223,0.6147,0.0,6.2,0.0,0.507,6.618,80.8,3.2721,8.0,307.0,17.4,396.9,7.6,30.1\n224,0.31533,0.0,6.2,0.0,0.504,8.266,78.3,2.8944,8.0,307.0,17.4,385.05,4.14,44.8\n225,0.52693,0.0,6.2,0.0,0.504,8.725,83.0,2.8944,8.0,307.0,17.4,382.0,4.63,50.0\n226,0.38214,0.0,6.2,0.0,0.504,8.04,86.5,3.2157,8.0,307.0,17.4,387.38,3.13,37.6\n227,0.41238,0.0,6.2,0.0,0.504,7.163,79.9,3.2157,8.0,307.0,17.4,372.08,6.36,31.6\n228,0.29819,0.0,6.2,0.0,0.504,7.686,17.0,3.3751,8.0,307.0,17.4,377.51,3.92,46.7\n229,0.44178,0.0,6.2,0.0,0.504,6.552,21.4,3.3751,8.0,307.0,17.4,380.34,3.76,31.5\n230,0.537,0.0,6.2,0.0,0.504,5.981,68.1,3.6715,8.0,307.0,17.4,378.35,11.65,24.3\n231,0.46296,0.0,6.2,0.0,0.504,7.412,76.9,3.6715,8.0,307.0,17.4,376.14,5.25,31.7\n232,0.57529,0.0,6.2,0.0,0.507,8.337,73.3,3.8384,8.0,307.0,17.4,385.91,2.47,41.7\n233,0.33147,0.0,6.2,0.0,0.507,8.247,70.4,3.6519,8.0,307.0,17.4,378.95,3.95,48.3\n234,0.44791,0.0,6.2,1.0,0.507,6.726,66.5,3.6519,8.0,307.0,17.4,360.2,8.05,29.0\n235,0.33045,0.0,6.2,0.0,0.507,6.086,61.5,3.6519,8.0,307.0,17.4,376.75,10.88,24.0\n236,0.52058,0.0,6.2,1.0,0.507,6.631,76.5,4.148,8.0,307.0,17.4,388.45,9.54,25.1\n237,0.51183,0.0,6.2,0.0,0.507,7.358,71.6,4.148,8.0,307.0,17.4,390.07,4.73,31.5\n238,0.08244,30.0,4.93,0.0,0.428,6.481,18.5,6.1899,6.0,300.0,16.6,379.41,6.36,23.7\n239,0.09252,30.0,4.93,0.0,0.428,6.606,42.2,6.1899,6.0,300.0,16.6,383.78,7.37,23.3\n240,0.11329,30.0,4.93,0.0,0.428,6.897,54.3,6.3361,6.0,300.0,16.6,391.25,11.38,22.0\n241,0.10612,30.0,4.93,0.0,0.428,6.095,65.1,6.3361,6.0,300.0,16.6,394.62,12.4,20.1\n242,0.1029,30.0,4.93,0.0,0.428,6.358,52.9,7.0355,6.0,300.0,16.6,372.75,11.22,22.2\n243,0.12757,30.0,4.93,0.0,0.428,6.393,7.8,7.0355,6.0,300.0,16.6,374.71,5.19,23.7\n244,0.20608,22.0,5.86,0.0,0.431,5.593,76.5,7.9549,7.0,330.0,19.1,372.49,12.5,17.6\n245,0.19133,22.0,5.86,0.0,0.431,5.605,70.2,7.9549,7.0,330.0,19.1,389.13,18.46,18.5\n246,0.33983,22.0,5.86,0.0,0.431,6.108,34.9,8.0555,7.0,330.0,19.1,390.18,9.16,24.3\n247,0.19657,22.0,5.86,0.0,0.431,6.226,79.2,8.0555,7.0,330.0,19.1,376.14,10.15,20.5\n248,0.16439,22.0,5.86,0.0,0.431,6.433,49.1,7.8265,7.0,330.0,19.1,374.71,9.52,24.5\n249,0.19073,22.0,5.86,0.0,0.431,6.718,17.5,7.8265,7.0,330.0,19.1,393.74,6.56,26.2\n250,0.1403,22.0,5.86,0.0,0.431,6.487,13.0,7.3967,7.0,330.0,19.1,396.28,5.9,24.4\n251,0.21409,22.0,5.86,0.0,0.431,6.438,8.9,7.3967,7.0,330.0,19.1,377.07,3.59,24.8\n252,0.08221,22.0,5.86,0.0,0.431,6.957,6.8,8.9067,7.0,330.0,19.1,386.09,3.53,29.6\n253,0.36894,22.0,5.86,0.0,0.431,8.259,8.4,8.9067,7.0,330.0,19.1,396.9,3.54,42.8\n254,0.04819,80.0,3.64,0.0,0.392,6.108,32.0,9.2203,1.0,315.0,16.4,392.89,6.57,21.9\n255,0.03548,80.0,3.64,0.0,0.392,5.876,19.1,9.2203,1.0,315.0,16.4,395.18,9.25,20.9\n256,0.01538,90.0,3.75,0.0,0.394,7.454,34.2,6.3361,3.0,244.0,15.9,386.34,3.11,44.0\n257,0.61154,20.0,3.97,0.0,0.647,8.704,86.9,1.801,5.0,264.0,13.0,389.7,5.12,50.0\n258,0.66351,20.0,3.97,0.0,0.647,7.333,100.0,1.8946,5.0,264.0,13.0,383.29,7.79,36.0\n259,0.65665,20.0,3.97,0.0,0.647,6.842,100.0,2.0107,5.0,264.0,13.0,391.93,6.9,30.1\n260,0.54011,20.0,3.97,0.0,0.647,7.203,81.8,2.1121,5.0,264.0,13.0,392.8,9.59,33.8\n261,0.53412,20.0,3.97,0.0,0.647,7.52,89.4,2.1398,5.0,264.0,13.0,388.37,7.26,43.1\n262,0.52014,20.0,3.97,0.0,0.647,8.398,91.5,2.2885,5.0,264.0,13.0,386.86,5.91,48.8\n263,0.82526,20.0,3.97,0.0,0.647,7.327,94.5,2.0788,5.0,264.0,13.0,393.42,11.25,31.0\n264,0.55007,20.0,3.97,0.0,0.647,7.206,91.6,1.9301,5.0,264.0,13.0,387.89,8.1,36.5\n265,0.76162,20.0,3.97,0.0,0.647,5.56,62.8,1.9865,5.0,264.0,13.0,392.4,10.45,22.8\n266,0.7857,20.0,3.97,0.0,0.647,7.014,84.6,2.1329,5.0,264.0,13.0,384.07,14.79,30.7\n267,0.57834,20.0,3.97,0.0,0.575,8.297,67.0,2.4216,5.0,264.0,13.0,384.54,7.44,50.0\n268,0.5405,20.0,3.97,0.0,0.575,7.47,52.6,2.872,5.0,264.0,13.0,390.3,3.16,43.5\n269,0.09065,20.0,6.96,1.0,0.464,5.92,61.5,3.9175,3.0,223.0,18.6,391.34,13.65,20.7\n270,0.29916,20.0,6.96,0.0,0.464,5.856,42.1,4.429,3.0,223.0,18.6,388.65,13.0,21.1\n271,0.16211,20.0,6.96,0.0,0.464,6.24,16.3,4.429,3.0,223.0,18.6,396.9,6.59,25.2\n272,0.1146,20.0,6.96,0.0,0.464,6.538,58.7,3.9175,3.0,223.0,18.6,394.96,7.73,24.4\n273,0.22188,20.0,6.96,1.0,0.464,7.691,51.8,4.3665,3.0,223.0,18.6,390.77,6.58,35.2\n274,0.05644,40.0,6.41,1.0,0.447,6.758,32.9,4.0776,4.0,254.0,17.6,396.9,3.53,32.4\n275,0.09604,40.0,6.41,0.0,0.447,6.854,42.8,4.2673,4.0,254.0,17.6,396.9,2.98,32.0\n276,0.10469,40.0,6.41,1.0,0.447,7.267,49.0,4.7872,4.0,254.0,17.6,389.25,6.05,33.2\n277,0.06127,40.0,6.41,1.0,0.447,6.826,27.6,4.8628,4.0,254.0,17.6,393.45,4.16,33.1\n278,0.07978,40.0,6.41,0.0,0.447,6.482,32.1,4.1403,4.0,254.0,17.6,396.9,7.19,29.1\n279,0.21038,20.0,3.33,0.0,0.4429,6.812,32.2,4.1007,5.0,216.0,14.9,396.9,4.85,35.1\n280,0.03578,20.0,3.33,0.0,0.4429,7.82,64.5,4.6947,5.0,216.0,14.9,387.31,3.76,45.4\n281,0.03705,20.0,3.33,0.0,0.4429,6.968,37.2,5.2447,5.0,216.0,14.9,392.23,4.59,35.4\n282,0.06129,20.0,3.33,1.0,0.4429,7.645,49.7,5.2119,5.0,216.0,14.9,377.07,3.01,46.0\n283,0.01501,90.0,1.21,1.0,0.401,7.923,24.8,5.885,1.0,198.0,13.6,395.52,3.16,50.0\n284,0.00906,90.0,2.97,0.0,0.4,7.088,20.8,7.3073,1.0,285.0,15.3,394.72,7.85,32.2\n285,0.01096,55.0,2.25,0.0,0.389,6.453,31.9,7.3073,1.0,300.0,15.3,394.72,8.23,22.0\n286,0.01965,80.0,1.76,0.0,0.385,6.23,31.5,9.0892,1.0,241.0,18.2,341.6,12.93,20.1\n287,0.03871,52.5,5.32,0.0,0.405,6.209,31.3,7.3172,6.0,293.0,16.6,396.9,7.14,23.2\n288,0.0459,52.5,5.32,0.0,0.405,6.315,45.6,7.3172,6.0,293.0,16.6,396.9,7.6,22.3\n289,0.04297,52.5,5.32,0.0,0.405,6.565,22.9,7.3172,6.0,293.0,16.6,371.72,9.51,24.8\n290,0.03502,80.0,4.95,0.0,0.411,6.861,27.9,5.1167,4.0,245.0,19.2,396.9,3.33,28.5\n291,0.07886,80.0,4.95,0.0,0.411,7.148,27.7,5.1167,4.0,245.0,19.2,396.9,3.56,37.3\n292,0.03615,80.0,4.95,0.0,0.411,6.63,23.4,5.1167,4.0,245.0,19.2,396.9,4.7,27.9\n293,0.08265,0.0,13.92,0.0,0.437,6.127,18.4,5.5027,4.0,289.0,16.0,396.9,8.58,23.9\n294,0.08199,0.0,13.92,0.0,0.437,6.009,42.3,5.5027,4.0,289.0,16.0,396.9,10.4,21.7\n295,0.12932,0.0,13.92,0.0,0.437,6.678,31.1,5.9604,4.0,289.0,16.0,396.9,6.27,28.6\n296,0.05372,0.0,13.92,0.0,0.437,6.549,51.0,5.9604,4.0,289.0,16.0,392.85,7.39,27.1\n297,0.14103,0.0,13.92,0.0,0.437,5.79,58.0,6.32,4.0,289.0,16.0,396.9,15.84,20.3\n298,0.06466,70.0,2.24,0.0,0.4,6.345,20.1,7.8278,5.0,358.0,14.8,368.24,4.97,22.5\n299,0.05561,70.0,2.24,0.0,0.4,7.041,10.0,7.8278,5.0,358.0,14.8,371.58,4.74,29.0\n300,0.04417,70.0,2.24,0.0,0.4,6.871,47.4,7.8278,5.0,358.0,14.8,390.86,6.07,24.8\n301,0.03537,34.0,6.09,0.0,0.433,6.59,40.4,5.4917,7.0,329.0,16.1,395.75,9.5,22.0\n302,0.09266,34.0,6.09,0.0,0.433,6.495,18.4,5.4917,7.0,329.0,16.1,383.61,8.67,26.4\n303,0.1,34.0,6.09,0.0,0.433,6.982,17.7,5.4917,7.0,329.0,16.1,390.43,4.86,33.1\n304,0.05515,33.0,2.18,0.0,0.472,7.236,41.1,4.022,7.0,222.0,18.4,393.68,6.93,36.1\n305,0.05479,33.0,2.18,0.0,0.472,6.616,58.1,3.37,7.0,222.0,18.4,393.36,8.93,28.4\n306,0.07503,33.0,2.18,0.0,0.472,7.42,71.9,3.0992,7.0,222.0,18.4,396.9,6.47,33.4\n307,0.04932,33.0,2.18,0.0,0.472,6.849,70.3,3.1827,7.0,222.0,18.4,396.9,7.53,28.2\n308,0.49298,0.0,9.9,0.0,0.544,6.635,82.5,3.3175,4.0,304.0,18.4,396.9,4.54,22.8\n309,0.3494,0.0,9.9,0.0,0.544,5.972,76.7,3.1025,4.0,304.0,18.4,396.24,9.97,20.3\n310,2.63548,0.0,9.9,0.0,0.544,4.973,37.8,2.5194,4.0,304.0,18.4,350.45,12.64,16.1\n311,0.79041,0.0,9.9,0.0,0.544,6.122,52.8,2.6403,4.0,304.0,18.4,396.9,5.98,22.1\n312,0.26169,0.0,9.9,0.0,0.544,6.023,90.4,2.834,4.0,304.0,18.4,396.3,11.72,19.4\n313,0.26938,0.0,9.9,0.0,0.544,6.266,82.8,3.2628,4.0,304.0,18.4,393.39,7.9,21.6\n314,0.3692,0.0,9.9,0.0,0.544,6.567,87.3,3.6023,4.0,304.0,18.4,395.69,9.28,23.8\n315,0.25356,0.0,9.9,0.0,0.544,5.705,77.7,3.945,4.0,304.0,18.4,396.42,11.5,16.2\n316,0.31827,0.0,9.9,0.0,0.544,5.914,83.2,3.9986,4.0,304.0,18.4,390.7,18.33,17.8\n317,0.24522,0.0,9.9,0.0,0.544,5.782,71.7,4.0317,4.0,304.0,18.4,396.9,15.94,19.8\n318,0.40202,0.0,9.9,0.0,0.544,6.382,67.2,3.5325,4.0,304.0,18.4,395.21,10.36,23.1\n319,0.47547,0.0,9.9,0.0,0.544,6.113,58.8,4.0019,4.0,304.0,18.4,396.23,12.73,21.0\n320,0.1676,0.0,7.38,0.0,0.493,6.426,52.3,4.5404,5.0,287.0,19.6,396.9,7.2,23.8\n321,0.18159,0.0,7.38,0.0,0.493,6.376,54.3,4.5404,5.0,287.0,19.6,396.9,6.87,23.1\n322,0.35114,0.0,7.38,0.0,0.493,6.041,49.9,4.7211,5.0,287.0,19.6,396.9,7.7,20.4\n323,0.28392,0.0,7.38,0.0,0.493,5.708,74.3,4.7211,5.0,287.0,19.6,391.13,11.74,18.5\n324,0.34109,0.0,7.38,0.0,0.493,6.415,40.1,4.7211,5.0,287.0,19.6,396.9,6.12,25.0\n325,0.19186,0.0,7.38,0.0,0.493,6.431,14.7,5.4159,5.0,287.0,19.6,393.68,5.08,24.6\n326,0.30347,0.0,7.38,0.0,0.493,6.312,28.9,5.4159,5.0,287.0,19.6,396.9,6.15,23.0\n327,0.24103,0.0,7.38,0.0,0.493,6.083,43.7,5.4159,5.0,287.0,19.6,396.9,12.79,22.2\n328,0.06617,0.0,3.24,0.0,0.46,5.868,25.8,5.2146,4.0,430.0,16.9,382.44,9.97,19.3\n329,0.06724,0.0,3.24,0.0,0.46,6.333,17.2,5.2146,4.0,430.0,16.9,375.21,7.34,22.6\n330,0.04544,0.0,3.24,0.0,0.46,6.144,32.2,5.8736,4.0,430.0,16.9,368.57,9.09,19.8\n331,0.05023,35.0,6.06,0.0,0.4379,5.706,28.4,6.6407,1.0,304.0,16.9,394.02,12.43,17.1\n332,0.03466,35.0,6.06,0.0,0.4379,6.031,23.3,6.6407,1.0,304.0,16.9,362.25,7.83,19.4\n333,0.05083,0.0,5.19,0.0,0.515,6.316,38.1,6.4584,5.0,224.0,20.2,389.71,5.68,22.2\n334,0.03738,0.0,5.19,0.0,0.515,6.31,38.5,6.4584,5.0,224.0,20.2,389.4,6.75,20.7\n335,0.03961,0.0,5.19,0.0,0.515,6.037,34.5,5.9853,5.0,224.0,20.2,396.9,8.01,21.1\n336,0.03427,0.0,5.19,0.0,0.515,5.869,46.3,5.2311,5.0,224.0,20.2,396.9,9.8,19.5\n337,0.03041,0.0,5.19,0.0,0.515,5.895,59.6,5.615,5.0,224.0,20.2,394.81,10.56,18.5\n338,0.03306,0.0,5.19,0.0,0.515,6.059,37.3,4.8122,5.0,224.0,20.2,396.14,8.51,20.6\n339,0.05497,0.0,5.19,0.0,0.515,5.985,45.4,4.8122,5.0,224.0,20.2,396.9,9.74,19.0\n340,0.06151,0.0,5.19,0.0,0.515,5.968,58.5,4.8122,5.0,224.0,20.2,396.9,9.29,18.7\n341,0.01301,35.0,1.52,0.0,0.442,7.241,49.3,7.0379,1.0,284.0,15.5,394.74,5.49,32.7\n342,0.02498,0.0,1.89,0.0,0.518,6.54,59.7,6.2669,1.0,422.0,15.9,389.96,8.65,16.5\n343,0.02543,55.0,3.78,0.0,0.484,6.696,56.4,5.7321,5.0,370.0,17.6,396.9,7.18,23.9\n344,0.03049,55.0,3.78,0.0,0.484,6.874,28.1,6.4654,5.0,370.0,17.6,387.97,4.61,31.2\n345,0.03113,0.0,4.39,0.0,0.442,6.014,48.5,8.0136,3.0,352.0,18.8,385.64,10.53,17.5\n346,0.06162,0.0,4.39,0.0,0.442,5.898,52.3,8.0136,3.0,352.0,18.8,364.61,12.67,17.2\n347,0.0187,85.0,4.15,0.0,0.429,6.516,27.7,8.5353,4.0,351.0,17.9,392.43,6.36,23.1\n348,0.01501,80.0,2.01,0.0,0.435,6.635,29.7,8.344,4.0,280.0,17.0,390.94,5.99,24.5\n349,0.02899,40.0,1.25,0.0,0.429,6.939,34.5,8.7921,1.0,335.0,19.7,389.85,5.89,26.6\n350,0.06211,40.0,1.25,0.0,0.429,6.49,44.4,8.7921,1.0,335.0,19.7,396.9,5.98,22.9\n351,0.0795,60.0,1.69,0.0,0.411,6.579,35.9,10.7103,4.0,411.0,18.3,370.78,5.49,24.1\n352,0.07244,60.0,1.69,0.0,0.411,5.884,18.5,10.7103,4.0,411.0,18.3,392.33,7.79,18.6\n353,0.01709,90.0,2.02,0.0,0.41,6.728,36.1,12.1265,5.0,187.0,17.0,384.46,4.5,30.1\n354,0.04301,80.0,1.91,0.0,0.413,5.663,21.9,10.5857,4.0,334.0,22.0,382.8,8.05,18.2\n355,0.10659,80.0,1.91,0.0,0.413,5.936,19.5,10.5857,4.0,334.0,22.0,376.04,5.57,20.6\n356,8.98296,0.0,18.1,1.0,0.77,6.212,97.4,2.1222,24.0,666.0,20.2,377.73,17.6,17.8\n357,3.8497,0.0,18.1,1.0,0.77,6.395,91.0,2.5052,24.0,666.0,20.2,391.34,13.27,21.7\n358,5.20177,0.0,18.1,1.0,0.77,6.127,83.4,2.7227,24.0,666.0,20.2,395.43,11.48,22.7\n359,4.26131,0.0,18.1,0.0,0.77,6.112,81.3,2.5091,24.0,666.0,20.2,390.74,12.67,22.6\n360,4.54192,0.0,18.1,0.0,0.77,6.398,88.0,2.5182,24.0,666.0,20.2,374.56,7.79,25.0\n361,3.83684,0.0,18.1,0.0,0.77,6.251,91.1,2.2955,24.0,666.0,20.2,350.65,14.19,19.9\n362,3.67822,0.0,18.1,0.0,0.77,5.362,96.2,2.1036,24.0,666.0,20.2,380.79,10.19,20.8\n363,4.22239,0.0,18.1,1.0,0.77,5.803,89.0,1.9047,24.0,666.0,20.2,353.04,14.64,16.8\n364,3.47428,0.0,18.1,1.0,0.718,8.78,82.9,1.9047,24.0,666.0,20.2,354.55,5.29,21.9\n365,4.55587,0.0,18.1,0.0,0.718,3.561,87.9,1.6132,24.0,666.0,20.2,354.7,7.12,27.5\n366,3.69695,0.0,18.1,0.0,0.718,4.963,91.4,1.7523,24.0,666.0,20.2,316.03,14.0,21.9\n367,13.5222,0.0,18.1,0.0,0.631,3.863,100.0,1.5106,24.0,666.0,20.2,131.42,13.33,23.1\n368,4.89822,0.0,18.1,0.0,0.631,4.97,100.0,1.3325,24.0,666.0,20.2,375.52,3.26,50.0\n369,5.66998,0.0,18.1,1.0,0.631,6.683,96.8,1.3567,24.0,666.0,20.2,375.33,3.73,50.0\n370,6.53876,0.0,18.1,1.0,0.631,7.016,97.5,1.2024,24.0,666.0,20.2,392.05,2.96,50.0\n371,9.2323,0.0,18.1,0.0,0.631,6.216,100.0,1.1691,24.0,666.0,20.2,366.15,9.53,50.0\n372,8.26725,0.0,18.1,1.0,0.668,5.875,89.6,1.1296,24.0,666.0,20.2,347.88,8.88,50.0\n373,11.1081,0.0,18.1,0.0,0.668,4.906,100.0,1.1742,24.0,666.0,20.2,396.9,34.77,13.8\n374,18.4982,0.0,18.1,0.0,0.668,4.138,100.0,1.137,24.0,666.0,20.2,396.9,37.97,13.8\n375,19.6091,0.0,18.1,0.0,0.671,7.313,97.9,1.3163,24.0,666.0,20.2,396.9,13.44,15.0\n376,15.288,0.0,18.1,0.0,0.671,6.649,93.3,1.3449,24.0,666.0,20.2,363.02,23.24,13.9\n377,9.82349,0.0,18.1,0.0,0.671,6.794,98.8,1.358,24.0,666.0,20.2,396.9,21.24,13.3\n378,23.6482,0.0,18.1,0.0,0.671,6.38,96.2,1.3861,24.0,666.0,20.2,396.9,23.69,13.1\n379,17.8667,0.0,18.1,0.0,0.671,6.223,100.0,1.3861,24.0,666.0,20.2,393.74,21.78,10.2\n380,88.9762,0.0,18.1,0.0,0.671,6.968,91.9,1.4165,24.0,666.0,20.2,396.9,17.21,10.4\n381,15.8744,0.0,18.1,0.0,0.671,6.545,99.1,1.5192,24.0,666.0,20.2,396.9,21.08,10.9\n382,9.18702,0.0,18.1,0.0,0.7,5.536,100.0,1.5804,24.0,666.0,20.2,396.9,23.6,11.3\n383,7.99248,0.0,18.1,0.0,0.7,5.52,100.0,1.5331,24.0,666.0,20.2,396.9,24.56,12.3\n384,20.0849,0.0,18.1,0.0,0.7,4.368,91.2,1.4395,24.0,666.0,20.2,285.83,30.63,8.8\n385,16.8118,0.0,18.1,0.0,0.7,5.277,98.1,1.4261,24.0,666.0,20.2,396.9,30.81,7.2\n386,24.3938,0.0,18.1,0.0,0.7,4.652,100.0,1.4672,24.0,666.0,20.2,396.9,28.28,10.5\n387,22.5971,0.0,18.1,0.0,0.7,5.0,89.5,1.5184,24.0,666.0,20.2,396.9,31.99,7.4\n388,14.3337,0.0,18.1,0.0,0.7,4.88,100.0,1.5895,24.0,666.0,20.2,372.92,30.62,10.2\n389,8.15174,0.0,18.1,0.0,0.7,5.39,98.9,1.7281,24.0,666.0,20.2,396.9,20.85,11.5\n390,6.96215,0.0,18.1,0.0,0.7,5.713,97.0,1.9265,24.0,666.0,20.2,394.43,17.11,15.1\n391,5.29305,0.0,18.1,0.0,0.7,6.051,82.5,2.1678,24.0,666.0,20.2,378.38,18.76,23.2\n392,11.5779,0.0,18.1,0.0,0.7,5.036,97.0,1.77,24.0,666.0,20.2,396.9,25.68,9.7\n393,8.64476,0.0,18.1,0.0,0.693,6.193,92.6,1.7912,24.0,666.0,20.2,396.9,15.17,13.8\n394,13.3598,0.0,18.1,0.0,0.693,5.887,94.7,1.7821,24.0,666.0,20.2,396.9,16.35,12.7\n395,8.71675,0.0,18.1,0.0,0.693,6.471,98.8,1.7257,24.0,666.0,20.2,391.98,17.12,13.1\n396,5.87205,0.0,18.1,0.0,0.693,6.405,96.0,1.6768,24.0,666.0,20.2,396.9,19.37,12.5\n397,7.67202,0.0,18.1,0.0,0.693,5.747,98.9,1.6334,24.0,666.0,20.2,393.1,19.92,8.5\n398,38.3518,0.0,18.1,0.0,0.693,5.453,100.0,1.4896,24.0,666.0,20.2,396.9,30.59,5.0\n399,9.91655,0.0,18.1,0.0,0.693,5.852,77.8,1.5004,24.0,666.0,20.2,338.16,29.97,6.3\n400,25.0461,0.0,18.1,0.0,0.693,5.987,100.0,1.5888,24.0,666.0,20.2,396.9,26.77,5.6\n401,14.2362,0.0,18.1,0.0,0.693,6.343,100.0,1.5741,24.0,666.0,20.2,396.9,20.32,7.2\n402,9.59571,0.0,18.1,0.0,0.693,6.404,100.0,1.639,24.0,666.0,20.2,376.11,20.31,12.1\n403,24.8017,0.0,18.1,0.0,0.693,5.349,96.0,1.7028,24.0,666.0,20.2,396.9,19.77,8.3\n404,41.5292,0.0,18.1,0.0,0.693,5.531,85.4,1.6074,24.0,666.0,20.2,329.46,27.38,8.5\n405,67.9208,0.0,18.1,0.0,0.693,5.683,100.0,1.4254,24.0,666.0,20.2,384.97,22.98,5.0\n406,20.7162,0.0,18.1,0.0,0.659,4.138,100.0,1.1781,24.0,666.0,20.2,370.22,23.34,11.9\n407,11.9511,0.0,18.1,0.0,0.659,5.608,100.0,1.2852,24.0,666.0,20.2,332.09,12.13,27.9\n408,7.40389,0.0,18.1,0.0,0.597,5.617,97.9,1.4547,24.0,666.0,20.2,314.64,26.4,17.2\n409,14.4383,0.0,18.1,0.0,0.597,6.852,100.0,1.4655,24.0,666.0,20.2,179.36,19.78,27.5\n410,51.1358,0.0,18.1,0.0,0.597,5.757,100.0,1.413,24.0,666.0,20.2,2.6,10.11,15.0\n411,14.0507,0.0,18.1,0.0,0.597,6.657,100.0,1.5275,24.0,666.0,20.2,35.05,21.22,17.2\n412,18.811,0.0,18.1,0.0,0.597,4.628,100.0,1.5539,24.0,666.0,20.2,28.79,34.37,17.9\n413,28.6558,0.0,18.1,0.0,0.597,5.155,100.0,1.5894,24.0,666.0,20.2,210.97,20.08,16.3\n414,45.7461,0.0,18.1,0.0,0.693,4.519,100.0,1.6582,24.0,666.0,20.2,88.27,36.98,7.0\n415,18.0846,0.0,18.1,0.0,0.679,6.434,100.0,1.8347,24.0,666.0,20.2,27.25,29.05,7.2\n416,10.8342,0.0,18.1,0.0,0.679,6.782,90.8,1.8195,24.0,666.0,20.2,21.57,25.79,7.5\n417,25.9406,0.0,18.1,0.0,0.679,5.304,89.1,1.6475,24.0,666.0,20.2,127.36,26.64,10.4\n418,73.5341,0.0,18.1,0.0,0.679,5.957,100.0,1.8026,24.0,666.0,20.2,16.45,20.62,8.8\n419,11.8123,0.0,18.1,0.0,0.718,6.824,76.5,1.794,24.0,666.0,20.2,48.45,22.74,8.4\n420,11.0874,0.0,18.1,0.0,0.718,6.411,100.0,1.8589,24.0,666.0,20.2,318.75,15.02,16.7\n421,7.02259,0.0,18.1,0.0,0.718,6.006,95.3,1.8746,24.0,666.0,20.2,319.98,15.7,14.2\n422,12.0482,0.0,18.1,0.0,0.614,5.648,87.6,1.9512,24.0,666.0,20.2,291.55,14.1,20.8\n423,7.05042,0.0,18.1,0.0,0.614,6.103,85.1,2.0218,24.0,666.0,20.2,2.52,23.29,13.4\n424,8.79212,0.0,18.1,0.0,0.584,5.565,70.6,2.0635,24.0,666.0,20.2,3.65,17.16,11.7\n425,15.8603,0.0,18.1,0.0,0.679,5.896,95.4,1.9096,24.0,666.0,20.2,7.68,24.39,8.3\n426,12.2472,0.0,18.1,0.0,0.584,5.837,59.7,1.9976,24.0,666.0,20.2,24.65,15.69,10.2\n427,37.6619,0.0,18.1,0.0,0.679,6.202,78.7,1.8629,24.0,666.0,20.2,18.82,14.52,10.9\n428,7.36711,0.0,18.1,0.0,0.679,6.193,78.1,1.9356,24.0,666.0,20.2,96.73,21.52,11.0\n429,9.33889,0.0,18.1,0.0,0.679,6.38,95.6,1.9682,24.0,666.0,20.2,60.72,24.08,9.5\n430,8.49213,0.0,18.1,0.0,0.584,6.348,86.1,2.0527,24.0,666.0,20.2,83.45,17.64,14.5\n431,10.0623,0.0,18.1,0.0,0.584,6.833,94.3,2.0882,24.0,666.0,20.2,81.33,19.69,14.1\n432,6.44405,0.0,18.1,0.0,0.584,6.425,74.8,2.2004,24.0,666.0,20.2,97.95,12.03,16.1\n433,5.58107,0.0,18.1,0.0,0.713,6.436,87.9,2.3158,24.0,666.0,20.2,100.19,16.22,14.3\n434,13.9134,0.0,18.1,0.0,0.713,6.208,95.0,2.2222,24.0,666.0,20.2,100.63,15.17,11.7\n435,11.1604,0.0,18.1,0.0,0.74,6.629,94.6,2.1247,24.0,666.0,20.2,109.85,23.27,13.4\n436,14.4208,0.0,18.1,0.0,0.74,6.461,93.3,2.0026,24.0,666.0,20.2,27.49,18.05,9.6\n437,15.1772,0.0,18.1,0.0,0.74,6.152,100.0,1.9142,24.0,666.0,20.2,9.32,26.45,8.7\n438,13.6781,0.0,18.1,0.0,0.74,5.935,87.9,1.8206,24.0,666.0,20.2,68.95,34.02,8.4\n439,9.39063,0.0,18.1,0.0,0.74,5.627,93.9,1.8172,24.0,666.0,20.2,396.9,22.88,12.8\n440,22.0511,0.0,18.1,0.0,0.74,5.818,92.4,1.8662,24.0,666.0,20.2,391.45,22.11,10.5\n441,9.72418,0.0,18.1,0.0,0.74,6.406,97.2,2.0651,24.0,666.0,20.2,385.96,19.52,17.1\n442,5.66637,0.0,18.1,0.0,0.74,6.219,100.0,2.0048,24.0,666.0,20.2,395.69,16.59,18.4\n443,9.96654,0.0,18.1,0.0,0.74,6.485,100.0,1.9784,24.0,666.0,20.2,386.73,18.85,15.4\n444,12.8023,0.0,18.1,0.0,0.74,5.854,96.6,1.8956,24.0,666.0,20.2,240.52,23.79,10.8\n445,0.6718,0.0,18.1,0.0,0.74,6.459,94.8,1.9879,24.0,666.0,20.2,43.06,23.98,11.8\n446,6.28807,0.0,18.1,0.0,0.74,6.341,96.4,2.072,24.0,666.0,20.2,318.01,17.79,14.9\n447,9.92485,0.0,18.1,0.0,0.74,6.251,96.6,2.198,24.0,666.0,20.2,388.52,16.44,12.6\n448,9.32909,0.0,18.1,0.0,0.713,6.185,98.7,2.2616,24.0,666.0,20.2,396.9,18.13,14.1\n449,7.52601,0.0,18.1,0.0,0.713,6.417,98.3,2.185,24.0,666.0,20.2,304.21,19.31,13.0\n450,6.71772,0.0,18.1,0.0,0.713,6.749,92.6,2.3236,24.0,666.0,20.2,0.32,17.44,13.4\n451,5.44114,0.0,18.1,0.0,0.713,6.655,98.2,2.3552,24.0,666.0,20.2,355.29,17.73,15.2\n452,5.09017,0.0,18.1,0.0,0.713,6.297,91.8,2.3682,24.0,666.0,20.2,385.09,17.27,16.1\n453,8.24809,0.0,18.1,0.0,0.713,7.393,99.3,2.4527,24.0,666.0,20.2,375.87,16.74,17.8\n454,9.51363,0.0,18.1,0.0,0.713,6.728,94.1,2.4961,24.0,666.0,20.2,6.68,18.71,14.9\n455,4.75237,0.0,18.1,0.0,0.713,6.525,86.5,2.4358,24.0,666.0,20.2,50.92,18.13,14.1\n456,4.66883,0.0,18.1,0.0,0.713,5.976,87.9,2.5806,24.0,666.0,20.2,10.48,19.01,12.7\n457,8.20058,0.0,18.1,0.0,0.713,5.936,80.3,2.7792,24.0,666.0,20.2,3.5,16.94,13.5\n458,7.75223,0.0,18.1,0.0,0.713,6.301,83.7,2.7831,24.0,666.0,20.2,272.21,16.23,14.9\n459,6.80117,0.0,18.1,0.0,0.713,6.081,84.4,2.7175,24.0,666.0,20.2,396.9,14.7,20.0\n460,4.81213,0.0,18.1,0.0,0.713,6.701,90.0,2.5975,24.0,666.0,20.2,255.23,16.42,16.4\n461,3.69311,0.0,18.1,0.0,0.713,6.376,88.4,2.5671,24.0,666.0,20.2,391.43,14.65,17.7\n462,6.65492,0.0,18.1,0.0,0.713,6.317,83.0,2.7344,24.0,666.0,20.2,396.9,13.99,19.5\n463,5.82115,0.0,18.1,0.0,0.713,6.513,89.9,2.8016,24.0,666.0,20.2,393.82,10.29,20.2\n464,7.83932,0.0,18.1,0.0,0.655,6.209,65.4,2.9634,24.0,666.0,20.2,396.9,13.22,21.4\n465,3.1636,0.0,18.1,0.0,0.655,5.759,48.2,3.0665,24.0,666.0,20.2,334.4,14.13,19.9\n466,3.77498,0.0,18.1,0.0,0.655,5.952,84.7,2.8715,24.0,666.0,20.2,22.01,17.15,19.0\n467,4.42228,0.0,18.1,0.0,0.584,6.003,94.5,2.5403,24.0,666.0,20.2,331.29,21.32,19.1\n468,15.5757,0.0,18.1,0.0,0.58,5.926,71.0,2.9084,24.0,666.0,20.2,368.74,18.13,19.1\n469,13.0751,0.0,18.1,0.0,0.58,5.713,56.7,2.8237,24.0,666.0,20.2,396.9,14.76,20.1\n470,4.34879,0.0,18.1,0.0,0.58,6.167,84.0,3.0334,24.0,666.0,20.2,396.9,16.29,19.9\n471,4.03841,0.0,18.1,0.0,0.532,6.229,90.7,3.0993,24.0,666.0,20.2,395.33,12.87,19.6\n472,3.56868,0.0,18.1,0.0,0.58,6.437,75.0,2.8965,24.0,666.0,20.2,393.37,14.36,23.2\n473,4.64689,0.0,18.1,0.0,0.614,6.98,67.6,2.5329,24.0,666.0,20.2,374.68,11.66,29.8\n474,8.05579,0.0,18.1,0.0,0.584,5.427,95.4,2.4298,24.0,666.0,20.2,352.58,18.14,13.8\n475,6.39312,0.0,18.1,0.0,0.584,6.162,97.4,2.206,24.0,666.0,20.2,302.76,24.1,13.3\n476,4.87141,0.0,18.1,0.0,0.614,6.484,93.6,2.3053,24.0,666.0,20.2,396.21,18.68,16.7\n477,15.0234,0.0,18.1,0.0,0.614,5.304,97.3,2.1007,24.0,666.0,20.2,349.48,24.91,12.0\n478,10.233,0.0,18.1,0.0,0.614,6.185,96.7,2.1705,24.0,666.0,20.2,379.7,18.03,14.6\n479,14.3337,0.0,18.1,0.0,0.614,6.229,88.0,1.9512,24.0,666.0,20.2,383.32,13.11,21.4\n480,5.82401,0.0,18.1,0.0,0.532,6.242,64.7,3.4242,24.0,666.0,20.2,396.9,10.74,23.0\n481,5.70818,0.0,18.1,0.0,0.532,6.75,74.9,3.3317,24.0,666.0,20.2,393.07,7.74,23.7\n482,5.73116,0.0,18.1,0.0,0.532,7.061,77.0,3.4106,24.0,666.0,20.2,395.28,7.01,25.0\n483,2.81838,0.0,18.1,0.0,0.532,5.762,40.3,4.0983,24.0,666.0,20.2,392.92,10.42,21.8\n484,2.37857,0.0,18.1,0.0,0.583,5.871,41.9,3.724,24.0,666.0,20.2,370.73,13.34,20.6\n485,3.67367,0.0,18.1,0.0,0.583,6.312,51.9,3.9917,24.0,666.0,20.2,388.62,10.58,21.2\n486,5.69175,0.0,18.1,0.0,0.583,6.114,79.8,3.5459,24.0,666.0,20.2,392.68,14.98,19.1\n487,4.83567,0.0,18.1,0.0,0.583,5.905,53.2,3.1523,24.0,666.0,20.2,388.22,11.45,20.6\n488,0.15086,0.0,27.74,0.0,0.609,5.454,92.7,1.8209,4.0,711.0,20.1,395.09,18.06,15.2\n489,0.18337,0.0,27.74,0.0,0.609,5.414,98.3,1.7554,4.0,711.0,20.1,344.05,23.97,7.0\n490,0.20746,0.0,27.74,0.0,0.609,5.093,98.0,1.8226,4.0,711.0,20.1,318.43,29.68,8.1\n491,0.10574,0.0,27.74,0.0,0.609,5.983,98.8,1.8681,4.0,711.0,20.1,390.11,18.07,13.6\n492,0.11132,0.0,27.74,0.0,0.609,5.983,83.5,2.1099,4.0,711.0,20.1,396.9,13.35,20.1\n493,0.17331,0.0,9.69,0.0,0.585,5.707,54.0,2.3817,6.0,391.0,19.2,396.9,12.01,21.8\n494,0.27957,0.0,9.69,0.0,0.585,5.926,42.6,2.3817,6.0,391.0,19.2,396.9,13.59,24.5\n495,0.17899,0.0,9.69,0.0,0.585,5.67,28.8,2.7986,6.0,391.0,19.2,393.29,17.6,23.1\n496,0.2896,0.0,9.69,0.0,0.585,5.39,72.9,2.7986,6.0,391.0,19.2,396.9,21.14,19.7\n497,0.26838,0.0,9.69,0.0,0.585,5.794,70.6,2.8927,6.0,391.0,19.2,396.9,14.1,18.3\n498,0.23912,0.0,9.69,0.0,0.585,6.019,65.3,2.4091,6.0,391.0,19.2,396.9,12.92,21.2\n499,0.17783,0.0,9.69,0.0,0.585,5.569,73.5,2.3999,6.0,391.0,19.2,395.77,15.1,17.5\n500,0.22438,0.0,9.69,0.0,0.585,6.027,79.7,2.4982,6.0,391.0,19.2,396.9,14.33,16.8\n501,0.06263,0.0,11.93,0.0,0.573,6.593,69.1,2.4786,1.0,273.0,21.0,391.99,9.67,22.4\n502,0.04527,0.0,11.93,0.0,0.573,6.12,76.7,2.2875,1.0,273.0,21.0,396.9,9.08,20.6\n503,0.06076,0.0,11.93,0.0,0.573,6.976,91.0,2.1675,1.0,273.0,21.0,396.9,5.64,23.9\n504,0.10959,0.0,11.93,0.0,0.573,6.794,89.3,2.3889,1.0,273.0,21.0,393.45,6.48,22.0\n505,0.04741,0.0,11.93,0.0,0.573,6.03,80.8,2.505,1.0,273.0,21.0,396.9,7.88,11.9\n"
},
{
"path": "examples/data/census_1k.csv",
"content": "\"YEAR\",\"DATANUM\",\"SERIAL\",\"CBSERIAL\",\"HHWT\",\"CPI99\",\"GQ\",\"QGQ\",\"PERNUM\",\"PERWT\",\"SEX\",\"AGE\",\"EDUC\",\"EDUCD\",\"INCTOT\",\"SEX_HEAD\",\"SEX_MOM\",\"SEX_POP\",\"SEX_SP\",\"SEX_MOM2\",\"SEX_POP2\",\"AGE_HEAD\",\"AGE_MOM\",\"AGE_POP\",\"AGE_SP\",\"AGE_MOM2\",\"AGE_POP2\",\"EDUC_HEAD\",\"EDUC_MOM\",\"EDUC_POP\",\"EDUC_SP\",\"EDUC_MOM2\",\"EDUC_POP2\",\"EDUCD_HEAD\",\"EDUCD_MOM\",\"EDUCD_POP\",\"EDUCD_SP\",\"EDUCD_MOM2\",\"EDUCD_POP2\",\"INCTOT_HEAD\",\"INCTOT_MOM\",\"INCTOT_POP\",\"INCTOT_SP\",\"INCTOT_MOM2\",\"INCTOT_POP2\"\n1970,2,1,,100,4.54,1,0,1,100,1,39,6,60,12450,1,,,2,,,39,,,36,,,6,,,3,,,60,,,30,,,12450,,,3450,,\n1970,2,1,,100,4.54,1,0,2,100,2,36,3,30,3450,1,,,1,,,39,,,39,,,6,,,6,,,60,,,60,,,12450,,,12450,,\n1970,2,2,,100,4.54,1,0,1,100,1,56,7,70,9050,1,,,2,,,56,,,54,,,7,,,6,,,70,,,60,,,9050,,,0,,\n1970,2,2,,100,4.54,1,0,2,100,2,54,6,60,0,1,,,1,,,56,,,56,,,7,,,7,,,70,,,70,,,9050,,,9050,,\n1970,2,4,,100,4.54,1,0,1,100,1,82,1,17,7450,1,,,2,,,82,,,74,,,1,,,2,,,17,,,23,,,7450,,,650,,\n1970,2,4,,100,4.54,1,0,2,100,2,74,2,23,650,1,,,1,,,82,,,82,,,1,,,1,,,17,,,17,,,7450,,,7450,,\n1970,2,5,,100,4.54,1,0,1,100,1,66,10,100,6950,1,,,2,,,66,,,62,,,10,,,6,,,100,,,60,,,6950,,,250,,\n1970,2,5,,100,4.54,1,0,2,100,2,62,6,60,250,1,,,1,,,66,,,66,,,10,,,10,,,100,,,100,,,6950,,,6950,,\n1970,2,6,,100,4.54,1,0,1,100,2,70,4,40,1250,2,,,,,,70,,,,,,4,,,,,,40,,,,,,1250,,,,,\n1970,2,7,,100,4.54,1,0,1,100,1,25,6,60,11150,1,,,2,,,25,,,22,,,6,,,6,,,60,,,60,,,11150,,,4050,,\n1970,2,7,,100,4.54,1,0,2,100,2,22,6,60,4050,1,,,1,,,25,,,25,,,6,,,6,,,60,,,60,,,11150,,,11150,,\n1970,2,7,,100,4.54,1,0,3,100,1,1,0,1,9999999,1,2,1,,,,25,22,25,,,,6,6,6,,,,60,60,60,,,,11150,4050,11150,,,\n1970,2,8,,100,4.54,3,0,1,100,2,98,2,26,550,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n1970,2,9,,100,4.54,1,0,1,100,1,25,10,100,6150,1,,,2,,,25,,,27,,,10,,,9,,,100,,,90,,,6150,,,1050,,\n1970,2,9,,100,4.54,1,0,2,100,2,27,9,90,1050,1,,,1,,,25,,,25,,,10,,,10,,,100,,,100,,,6150,,,6150,,\n1970,2,10,,100,4.54,1,0,1,100,1,41,11,111,8050,1,2,,,,,41,78,,,,,11,6,,,,,111,60,,,,,8050,0,,,,\n1970,2,10,,100,4.54,1,0,2,100,2,78,6,60,0,1,,,,,,41,,,,,,11,,,,,,111,,,,,,8050,,,,,\n1970,2,10,,100,4.54,1,0,3,100,2,38,6,60,7150,1,2,,,,,41,78,,,,,11,6,,,,,111,60,,,,,8050,0,,,,\n1970,2,11,,100,4.54,1,0,1,100,1,20,6,60,2050,1,,,,,,20,,,,,,6,,,,,,60,,,,,,2050,,,,,\n1970,2,13,,100,4.54,1,0,1,100,1,37,6,65,16850,1,,,2,,,37,,,30,,,6,,,8,,,65,,,80,,,16850,,,350,,\n1970,2,13,,100,4.54,1,0,2,100,2,30,8,80,350,1,,,1,,,37,,,37,,,6,,,6,,,65,,,65,,,16850,,,16850,,\n1970,2,13,,100,4.54,1,0,3,100,1,5,0,2,9999999,1,2,1,,,,37,30,37,,,,6,8,6,,,,65,80,65,,,,16850,350,16850,,,\n1970,2,13,,100,4.54,1,0,4,100,2,1,0,1,9999999,1,2,1,,,,37,30,37,,,,6,8,6,,,,65,80,65,,,,16850,350,16850,,,\n1970,2,14,,100,4.54,1,0,1,100,1,49,2,23,8850,1,,,2,,,49,,,35,,,2,,,3,,,23,,,30,,,8850,,,4850,,\n1970,2,14,,100,4.54,1,0,2,100,2,35,3,30,4850,1,,,1,,,49,,,49,,,2,,,2,,,23,,,23,,,8850,,,8850,,\n1970,2,14,,100,4.54,1,0,3,100,2,17,3,30,250,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,\n1970,2,14,,100,4.54,1,0,4,100,2,14,2,25,0,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,\n1970,2,14,,100,4.54,1,0,5,100,1,10,1,15,9999999,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,\n1970,2,14,,100,4.54,1,0,6,100,2,8,1,14,9999999,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,\n1970,2,14,,100,4.54,1,0,7,100,1,0,0,1,9999999,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,\n1970,2,15,,100,4.54,1,0,1,100,2,62,7,70,7750,2,,,,,,62,,,,,,7,,,,,,70,,,,,,7750,,,,,\n1970,2,15,,100,4.54,1,0,2,100,1,35,11,111,5350,2,2,,,,,62,62,,,,,7,7,,,,,70,70,,,,,7750,7750,,,,\n1970,2,16,,100,4.54,1,0,1,100,1,57,4,40,11250,1,,,2,,,57,,,54,,,4,,,2,,,40,,,26,,,11250,,,150,,\n1970,2,16,,100,4.54,1,0,2,100,2,54,2,26,150,1,2,,1,,,57,86,,57,,,4,2,,4,,,40,26,,40,,,11250,1250,,11250,,\n1970,2,16,,100,4.54,1,0,3,100,2,86,2,26,1250,1,,,,,,57,,,,,,4,,,,,,40,,,,,,11250,,,,,\n1970,2,17,,100,4.54,1,0,1,100,1,54,6,60,6050,1,,,,,,54,,,,,,6,,,,,,60,,,,,,6050,,,,,\n1970,2,17,,100,4.54,1,0,2,100,2,64,2,26,0,1,,,,,,54,,,,,,6,,,,,,60,,,,,,6050,,,,,\n1970,2,18,,100,4.54,1,0,1,100,1,52,7,70,12050,1,,,2,,,52,,,44,,,7,,,6,,,70,,,60,,,12050,,,650,,\n1970,2,18,,100,4.54,1,0,2,100,2,44,6,60,650,1,,,1,,,52,,,52,,,7,,,7,,,70,,,70,,,12050,,,12050,,\n1970,2,18,,100,4.54,1,0,3,100,2,16,4,40,950,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,\n1970,2,18,,100,4.54,1,0,4,100,2,15,3,30,350,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,\n1970,2,18,,100,4.54,1,0,5,100,1,14,2,25,350,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,\n1970,2,18,,100,4.54,1,0,6,100,1,12,2,22,9999999,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,\n1970,2,18,,100,4.54,1,0,7,100,1,6,1,12,9999999,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,\n1970,2,19,,100,4.54,1,0,1,100,1,77,2,26,250,1,,,2,,,77,,,79,,,2,,,2,,,26,,,26,,,250,,,0,,\n1970,2,19,,100,4.54,1,0,2,100,2,79,2,26,0,1,,,1,,,77,,,77,,,2,,,2,,,26,,,26,,,250,,,250,,\n1970,2,20,,100,4.54,1,0,1,100,1,36,6,60,11450,1,,,2,,,36,,,32,,,6,,,6,,,60,,,60,,,11450,,,5550,,\n1970,2,20,,100,4.54,1,0,2,100,2,32,6,60,5550,1,,,1,,,36,,,36,,,6,,,6,,,60,,,60,,,11450,,,11450,,\n1970,2,20,,100,4.54,1,0,3,100,2,9,1,16,9999999,1,2,1,,,,36,32,36,,,,6,6,6,,,,60,60,60,,,,11450,5550,11450,,,\n1970,2,21,,100,4.54,1,0,1,100,1,21,4,40,2450,1,,,2,,,21,,,20,,,4,,,4,,,40,,,40,,,2450,,,4550,,\n1970,2,21,,100,4.54,1,0,2,100,2,20,4,40,4550,1,,,1,,,21,,,21,,,4,,,4,,,40,,,40,,,2450,,,2450,,\n1970,2,21,,100,4.54,1,0,3,100,1,5,1,12,9999999,1,2,1,,,,21,20,21,,,,4,4,4,,,,40,40,40,,,,2450,4550,2450,,,\n1970,2,21,,100,4.54,1,0,4,100,1,4,1,11,9999999,1,2,1,,,,21,20,21,,,,4,4,4,,,,40,40,40,,,,2450,4550,2450,,,\n1970,2,22,,100,4.54,1,0,1,100,1,23,2,26,5050,1,,,,,,23,,,,,,2,,,,,,26,,,,,,5050,,,,,\n1970,2,22,,100,4.54,1,0,2,100,2,23,3,30,1850,1,,,,,,23,,,,,,2,,,,,,26,,,,,,5050,,,,,\n1970,2,23,,100,4.54,1,0,1,100,1,63,6,60,5050,1,,,,,,63,,,,,,6,,,,,,60,,,,,,5050,,,,,\n1970,2,24,,100,4.54,1,0,1,100,2,68,3,30,2150,2,,,,,,68,,,,,,3,,,,,,30,,,,,,2150,,,,,\n1970,2,25,,100,4.54,1,0,1,100,1,65,2,22,4850,1,,,2,,,65,,,61,,,2,,,4,,,22,,,40,,,4850,,,4350,,\n1970,2,25,,100,4.54,1,0,2,100,2,61,4,40,4350,1,,,1,,,65,,,65,,,2,,,2,,,22,,,22,,,4850,,,4850,,\n1970,2,26,,100,4.54,1,0,1,100,1,61,8,80,2150,1,,,2,,,61,,,66,,,8,,,6,,,80,,,60,,,2150,,,5650,,\n1970,2,26,,100,4.54,1,0,2,100,2,66,6,60,5650,1,,,1,,,61,,,61,,,8,,,8,,,80,,,80,,,2150,,,2150,,\n1970,2,27,,100,4.54,1,0,1,100,2,77,1,14,4050,2,,,,,,77,,,,,,1,,,,,,14,,,,,,4050,,,,,\n1970,2,27,,100,4.54,1,0,2,100,1,75,1,14,2050,2,,,,,,77,,,,,,1,,,,,,14,,,,,,4050,,,,,\n1970,2,28,,100,4.54,1,0,1,100,1,32,8,80,5050,1,,,,,,32,,,,,,8,,,,,,80,,,,,,5050,,,,,\n1970,2,29,,100,4.54,1,0,1,100,1,59,5,50,15050,1,,,2,,,59,,,55,,,5,,,6,,,50,,,60,,,15050,,,0,,\n1970,2,29,,100,4.54,1,0,2,100,2,55,6,60,0,1,,,1,,,59,,,59,,,5,,,5,,,50,,,50,,,15050,,,15050,,\n1970,2,30,,100,4.54,1,0,1,100,2,47,6,60,0,2,,,,,,47,,,,,,6,,,,,,60,,,,,,0,,,,,\n1970,2,31,,100,4.54,1,0,1,100,1,43,8,80,7050,1,,,2,,,43,,,41,,,8,,,6,,,80,,,60,,,7050,,,2050,,\n1970,2,31,,100,4.54,1,0,2,100,2,41,6,60,2050,1,,,1,,,43,,,43,,,8,,,8,,,80,,,80,,,7050,,,7050,,\n1970,2,31,,100,4.54,1,0,3,100,2,18,6,65,4050,1,2,1,,,,43,41,43,,,,8,6,8,,,,80,60,80,,,,7050,2050,7050,,,\n1970,2,31,,100,4.54,1,0,4,100,2,15,2,26,0,1,2,1,,,,43,41,43,,,,8,6,8,,,,80,60,80,,,,7050,2050,7050,,,\n1970,2,31,,100,4.54,1,0,5,100,2,10,1,17,9999999,1,2,1,,,,43,41,43,,,,8,6,8,,,,80,60,80,,,,7050,2050,7050,,,\n1970,2,32,,100,4.54,1,0,1,100,1,40,10,100,13350,1,,,2,,,40,,,36,,,10,,,10,,,100,,,100,,,13350,,,0,,\n1970,2,32,,100,4.54,1,0,2,100,2,36,10,100,0,1,,,1,,,40,,,40,,,10,,,10,,,100,,,100,,,13350,,,13350,,\n1970,2,32,,100,4.54,1,0,3,100,1,14,2,25,0,1,2,1,,,,40,36,40,,,,10,10,10,,,,100,100,100,,,,13350,0,13350,,,\n1970,2,32,,100,4.54,1,0,4,100,1,10,1,16,9999999,1,2,1,,,,40,36,40,,,,10,10,10,,,,100,100,100,,,,13350,0,13350,,,\n1970,2,32,,100,4.54,1,0,5,100,2,4,1,11,9999999,1,2,1,,,,40,36,40,,,,10,10,10,,,,100,100,100,,,,13350,0,13350,,,\n1970,2,33,,100,4.54,1,0,1,100,1,40,11,111,25050,1,,,2,,,40,,,32,,,11,,,10,,,111,,,100,,,25050,,,0,,\n1970,2,33,,100,4.54,1,0,2,100,2,32,10,100,0,1,,,1,,,40,,,40,,,11,,,11,,,111,,,111,,,25050,,,25050,,\n1970,2,33,,100,4.54,1,0,3,100,1,5,0,2,9999999,1,2,1,,,,40,32,40,,,,11,10,11,,,,111,100,111,,,,25050,0,25050,,,\n1970,2,33,,100,4.54,1,0,4,100,1,3,0,2,9999999,1,2,1,,,,40,32,40,,,,11,10,11,,,,111,100,111,,,,25050,0,25050,,,\n1970,2,34,,100,4.54,1,0,1,100,1,31,11,111,19350,1,,,2,,,31,,,31,,,11,,,10,,,111,,,100,,,19350,,,0,,\n1970,2,34,,100,4.54,1,0,2,100,2,31,10,100,0,1,,,1,,,31,,,31,,,11,,,11,,,111,,,111,,,19350,,,19350,,\n1970,2,34,,100,4.54,1,0,3,100,2,3,0,2,9999999,1,2,1,,,,31,31,31,,,,11,10,11,,,,111,100,111,,,,19350,0,19350,,,\n1970,2,35,,100,4.54,1,0,1,100,1,64,11,111,17150,1,,,,,,64,,,,,,11,,,,,,111,,,,,,17150,,,,,\n1970,2,36,,100,4.54,1,0,1,100,1,55,3,30,9050,1,,,2,,,55,,,51,,,3,,,6,,,30,,,60,,,9050,,,2950,,\n1970,2,36,,100,4.54,1,0,2,100,2,51,6,60,2950,1,,,1,,,55,,,55,,,3,,,3,,,30,,,30,,,9050,,,9050,,\n1970,2,37,,100,4.54,1,0,1,100,1,43,11,111,50000,1,,,2,,,43,,,40,,,11,,,10,,,111,,,100,,,50000,,,1150,,\n1970,2,37,,100,4.54,1,0,2,100,2,40,10,100,1150,1,,,1,,,43,,,43,,,11,,,11,,,111,,,111,,,50000,,,50000,,\n1970,2,37,,100,4.54,1,0,3,100,1,16,4,40,250,1,2,1,,,,43,40,43,,,,11,10,11,,,,111,100,111,,,,50000,1150,50000,,,\n1970,2,37,,100,4.54,1,0,4,100,2,15,2,26,50,1,2,1,,,,43,40,43,,,,11,10,11,,,,111,100,111,,,,50000,1150,50000,,,\n1970,2,37,,100,4.54,1,0,5,100,1,12,2,22,9999999,1,2,1,,,,43,40,43,,,,11,10,11,,,,111,100,111,,,,50000,1150,50000,,,\n1970,2,38,,100,4.54,1,0,1,100,1,32,10,100,22150,1,,,2,,,32,,,31,,,10,,,7,,,100,,,70,,,22150,,,0,,\n1970,2,38,,100,4.54,1,0,2,100,2,31,7,70,0,1,,,1,,,32,,,32,,,10,,,10,,,100,,,100,,,22150,,,22150,,\n1970,2,38,,100,4.54,1,0,3,100,2,5,0,2,9999999,1,2,1,,,,32,31,32,,,,10,7,10,,,,100,70,100,,,,22150,0,22150,,,\n1970,2,38,,100,4.54,1,0,4,100,1,4,0,2,9999999,1,2,1,,,,32,31,32,,,,10,7,10,,,,100,70,100,,,,22150,0,22150,,,\n1970,2,38,,100,4.54,1,0,5,100,2,2,0,1,9999999,1,2,1,,,,32,31,32,,,,10,7,10,,,,100,70,100,,,,22150,0,22150,,,\n1970,2,39,,100,4.54,1,0,1,100,1,54,8,80,17850,1,,,2,,,54,,,47,,,8,,,6,,,80,,,60,,,17850,,,0,,\n1970,2,39,,100,4.54,1,0,2,100,2,47,6,60,0,1,,,1,,,54,,,54,,,8,,,8,,,80,,,80,,,17850,,,17850,,\n1970,2,39,,100,4.54,1,0,3,100,1,19,6,60,2750,1,2,1,,,,54,47,54,,,,8,6,8,,,,80,60,80,,,,17850,0,17850,,,\n1970,2,39,,100,4.54,1,0,4,100,1,12,2,23,9999999,1,2,1,,,,54,47,54,,,,8,6,8,,,,80,60,80,,,,17850,0,17850,,,\n1970,2,40,,100,4.54,1,0,1,100,1,46,6,60,17150,1,,,2,,,46,,,46,,,6,,,6,,,60,,,60,,,17150,,,5050,,\n1970,2,40,,100,4.54,1,0,2,100,2,46,6,60,5050,1,,,1,,,46,,,46,,,6,,,6,,,60,,,60,,,17150,,,17150,,\n1970,2,41,,100,4.54,1,0,1,100,1,70,6,60,11450,1,,,,,,70,,,,,,6,,,,,,60,,,,,,11450,,,,,\n1970,2,41,,100,4.54,1,0,2,100,2,28,5,50,2150,1,,,,,,70,,,,,,6,,,,,,60,,,,,,11450,,,,,\n1970,2,41,,100,4.54,1,0,3,100,1,33,8,80,9850,1,,1,,,,70,,70,,,,6,,6,,,,60,,60,,,,11450,,11450,,,\n1970,2,41,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,,1,,,,70,,33,,,,6,,8,,,,60,,80,,,,11450,,9850,,,\n1970,2,41,,100,4.54,1,0,5,100,2,5,0,2,9999999,1,,1,,,,70,,33,,,,6,,8,,,,60,,80,,,,11450,,9850,,,\n1970,2,42,,100,4.54,1,0,1,100,1,53,3,30,10050,1,,,2,,,53,,,51,,,3,,,6,,,30,,,60,,,10050,,,0,,\n1970,2,42,,100,4.54,1,0,2,100,2,51,6,60,0,1,,,1,,,53,,,53,,,3,,,3,,,30,,,30,,,10050,,,10050,,\n1970,2,42,,100,4.54,1,0,3,100,2,14,2,26,0,1,2,1,,,,53,51,53,,,,3,6,3,,,,30,60,30,,,,10050,0,10050,,,\n1970,2,43,,100,4.54,1,0,1,100,1,41,11,111,32050,1,,,2,,,41,,,40,,,11,,,11,,,111,,,111,,,32050,,,250,,\n1970,2,43,,100,4.54,1,0,2,100,2,40,11,111,250,1,,,1,,,41,,,41,,,11,,,11,,,111,,,111,,,32050,,,32050,,\n1970,2,43,,100,4.54,1,0,3,100,2,10,1,17,9999999,1,2,1,,,,41,40,41,,,,11,11,11,,,,111,111,111,,,,32050,250,32050,,,\n1970,2,43,,100,4.54,1,0,4,100,2,6,1,12,9999999,1,2,1,,,,41,40,41,,,,11,11,11,,,,111,111,111,,,,32050,250,32050,,,\n1970,2,44,,100,4.54,1,0,1,100,1,47,2,26,14050,1,,,2,,,47,,,44,,,2,,,2,,,26,,,26,,,14050,,,0,,\n1970,2,44,,100,4.54,1,0,2,100,2,44,2,26,0,1,,,1,,,47,,,47,,,2,,,2,,,26,,,26,,,14050,,,14050,,\n1970,2,44,,100,4.54,1,0,3,100,2,21,8,80,2050,1,2,1,,,,47,44,47,,,,2,2,2,,,,26,26,26,,,,14050,0,14050,,,\n1970,2,44,,100,4.54,1,0,4,100,1,18,5,50,1750,1,2,1,,,,47,44,47,,,,2,2,2,,,,26,26,26,,,,14050,0,14050,,,\n1970,2,44,,100,4.54,1,0,5,100,1,4,0,2,9999999,1,2,1,,,,47,44,47,,,,2,2,2,,,,26,26,26,,,,14050,0,14050,,,\n1970,2,45,,100,4.54,1,0,1,100,1,60,8,80,16550,1,,,2,,,60,,,50,,,8,,,6,,,80,,,60,,,16550,,,2950,,\n1970,2,45,,100,4.54,1,0,2,100,2,50,6,60,2950,1,,,1,,,60,,,60,,,8,,,8,,,80,,,80,,,16550,,,16550,,\n1970,2,46,,100,4.54,1,0,1,100,1,47,10,100,16250,1,2,,,,,47,83,,,,,10,2,,,,,100,22,,,,,16250,1250,,,,\n1970,2,46,,100,4.54,1,0,2,100,2,83,2,22,1250,1,,,,,,47,,,,,,10,,,,,,100,,,,,,16250,,,,,\n1970,2,46,,100,4.54,1,0,3,100,2,83,2,22,1450,1,,,,,,47,,,,,,10,,,,,,100,,,,,,16250,,,,,\n1970,2,48,,100,4.54,1,0,1,100,1,49,6,60,8850,1,2,,2,,,49,89,,48,,,6,2,,6,,,60,26,,60,,,8850,450,,6050,,\n1970,2,48,,100,4.54,1,0,2,100,2,48,6,60,6050,1,,,1,,,49,,,49,,,6,,,6,,,60,,,60,,,8850,,,8850,,\n1970,2,48,,100,4.54,1,0,3,100,2,89,2,26,450,1,,,,,,49,,,,,,6,,,,,,60,,,,,,8850,,,,,\n1970,2,49,,100,4.54,1,0,1,100,1,30,10,100,12050,1,,,2,,,30,,,30,,,10,,,10,,,100,,,100,,,12050,,,5650,,\n1970,2,49,,100,4.54,1,0,2,100,2,30,10,100,5650,1,,,1,,,30,,,30,,,10,,,10,,,100,,,100,,,12050,,,12050,,\n1970,2,49,,100,4.54,1,0,3,100,1,3,0,2,9999999,1,2,1,,,,30,30,30,,,,10,10,10,,,,100,100,100,,,,12050,5650,12050,,,\n1970,2,49,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,2,1,,,,30,30,30,,,,10,10,10,,,,100,100,100,,,,12050,5650,12050,,,\n1970,2,50,,100,4.54,1,0,1,100,1,54,8,80,10150,1,,,,,,54,,,,,,8,,,,,,80,,,,,,10150,,,,,\n1970,2,51,,100,4.54,1,0,1,100,2,64,10,100,8650,2,,,,,,64,,,,,,10,,,,,,100,,,,,,8650,,,,,\n1970,2,52,,100,4.54,1,0,1,100,2,37,7,70,11350,2,,,,,,37,,,,,,7,,,,,,70,,,,,,11350,,,,,\n1970,2,52,,100,4.54,1,0,2,100,2,6,1,12,9999999,2,2,,,,,37,37,,,,,7,7,,,,,70,70,,,,,11350,11350,,,,\n1970,2,53,,100,4.54,1,0,1,100,2,34,6,60,8050,2,,,,,,34,,,,,,6,,,,,,60,,,,,,8050,,,,,\n1970,2,53,,100,4.54,1,0,2,100,1,11,2,22,9999999,2,2,,,,,34,34,,,,,6,6,,,,,60,60,,,,,8050,8050,,,,\n1970,2,53,,100,4.54,1,0,3,100,2,9,1,15,9999999,2,2,,,,,34,34,,,,,6,6,,,,,60,60,,,,,8050,8050,,,,\n1970,2,54,,100,4.54,1,0,1,100,1,48,0,2,0,1,,,2,,,48,,,51,,,0,,,2,,,2,,,26,,,0,,,0,,\n1970,2,54,,100,4.54,1,0,2,100,2,51,2,26,0,1,,,1,,,48,,,48,,,0,,,0,,,2,,,2,,,0,,,0,,\n1970,2,55,,100,4.54,1,0,1,100,2,27,2,25,150,2,,,,,,27,,,,,,2,,,,,,25,,,,,,150,,,,,\n1970,2,55,,100,4.54,1,0,2,100,2,12,2,23,9999999,2,2,,,,,27,27,,,,,2,2,,,,,25,25,,,,,150,150,,,,\n1970,2,55,,100,4.54,1,0,3,100,2,7,1,14,9999999,2,2,,,,,27,27,,,,,2,2,,,,,25,25,,,,,150,150,,,,\n1970,2,55,,100,4.54,1,0,4,100,2,6,1,12,9999999,2,2,,,,,27,27,,,,,2,2,,,,,25,25,,,,,150,150,,,,\n1970,2,55,,100,4.54,1,0,5,100,2,5,1,11,9999999,2,2,,,,,27,27,,,,,2,2,,,,,25,25,,,,,150,150,,,,\n1970,2,56,,100,4.54,1,0,1,100,1,58,2,22,8050,1,,,2,,,58,,,52,,,2,,,6,,,22,,,60,,,8050,,,0,,\n1970,2,56,,100,4.54,1,0,2,100,2,52,6,60,0,1,,,1,,,58,,,58,,,2,,,2,,,22,,,22,,,8050,,,8050,,\n1970,2,56,,100,4.54,1,0,3,100,2,23,9,90,0,1,2,1,,,,58,52,58,,,,2,6,2,,,,22,60,22,,,,8050,0,8050,,,\n1970,2,57,,100,4.54,1,0,1,100,1,32,2,26,7050,1,,,2,,,32,,,32,,,2,,,5,,,26,,,50,,,7050,,,5050,,\n1970,2,57,,100,4.54,1,0,2,100,2,32,5,50,5050,1,,,1,,,32,,,32,,,2,,,2,,,26,,,26,,,7050,,,7050,,\n1970,2,57,,100,4.54,1,0,3,100,2,5,0,2,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,\n1970,2,57,,100,4.54,1,0,4,100,2,3,0,2,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,\n1970,2,57,,100,4.54,1,0,5,100,2,13,2,23,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,\n1970,2,57,,100,4.54,1,0,6,100,1,10,1,17,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,\n1970,2,57,,100,4.54,1,0,7,100,2,9,1,16,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,\n1970,2,57,,100,4.54,1,0,8,100,2,8,1,15,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,\n1970,2,57,,100,4.54,1,0,9,100,1,6,1,12,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,\n1970,2,58,,100,4.54,1,0,1,100,1,24,11,110,1350,1,,,2,,,24,,,25,,,11,,,11,,,110,,,110,,,1350,,,8150,,\n1970,2,58,,100,4.54,1,0,2,100,2,25,11,110,8150,1,,,1,,,24,,,24,,,11,,,11,,,110,,,110,,,1350,,,1350,,\n1970,2,58,,100,4.54,1,0,3,100,1,1,0,1,9999999,1,2,1,,,,24,25,24,,,,11,11,11,,,,110,110,110,,,,1350,8150,1350,,,\n1970,2,59,,100,4.54,1,0,1,100,1,34,5,50,14150,1,,,2,,,34,,,31,,,5,,,6,,,50,,,60,,,14150,,,5050,,\n1970,2,59,,100,4.54,1,0,2,100,2,31,6,60,5050,1,,,1,,,34,,,34,,,5,,,5,,,50,,,50,,,14150,,,14150,,\n1970,2,59,,100,4.54,1,0,3,100,2,12,2,22,9999999,1,2,1,,,,34,31,34,,,,5,6,5,,,,50,60,50,,,,14150,5050,14150,,,\n1970,2,59,,100,4.54,1,0,4,100,1,11,2,22,9999999,1,2,1,,,,34,31,34,,,,5,6,5,,,,50,60,50,,,,14150,5050,14150,,,\n1970,2,59,,100,4.54,1,0,5,100,2,10,1,17,9999999,1,2,1,,,,34,31,34,,,,5,6,5,,,,50,60,50,,,,14150,5050,14150,,,\n1970,2,59,,100,4.54,1,0,6,100,1,6,1,12,9999999,1,2,1,,,,34,31,34,,,,5,6,5,,,,50,60,50,,,,14150,5050,14150,,,\n1970,2,60,,100,4.54,1,0,1,100,1,35,10,100,14050,1,,,,,,35,,,,,,10,,,,,,100,,,,,,14050,,,,,\n1970,2,62,,100,4.54,1,0,1,100,1,53,6,60,10650,1,,,2,,,53,,,49,,,6,,,6,,,60,,,60,,,10650,,,6650,,\n1970,2,62,,100,4.54,1,0,2,100,2,49,6,60,6650,1,,,1,,,53,,,53,,,6,,,6,,,60,,,60,,,10650,,,10650,,\n1970,2,63,,100,4.54,1,0,1,100,1,78,2,26,1250,1,,,,,,78,,,,,,2,,,,,,26,,,,,,1250,,,,,\n1970,2,63,,100,4.54,1,0,2,100,2,38,7,70,7050,1,,1,,,,78,,78,,,,2,,2,,,,26,,26,,,,1250,,1250,,,\n1970,2,64,,100,4.54,1,0,1,100,1,37,6,60,7050,1,,,2,,,37,,,36,,,6,,,6,,,60,,,60,,,7050,,,0,,\n1970,2,64,,100,4.54,1,0,2,100,2,36,6,60,0,1,,,1,,,37,,,37,,,6,,,6,,,60,,,60,,,7050,,,7050,,\n1970,2,64,,100,4.54,1,0,3,100,1,13,2,25,9999999,1,2,1,,,,37,36,37,,,,6,6,6,,,,60,60,60,,,,7050,0,7050,,,\n1970,2,64,,100,4.54,1,0,4,100,1,12,2,22,9999999,1,2,1,,,,37,36,37,,,,6,6,6,,,,60,60,60,,,,7050,0,7050,,,\n1970,2,64,,100,4.54,1,0,5,100,1,5,0,2,9999999,1,2,1,,,,37,36,37,,,,6,6,6,,,,60,60,60,,,,7050,0,7050,,,\n1970,2,65,,100,4.54,1,0,1,100,2,33,8,80,8050,2,,,,,,33,,,,,,8,,,,,,80,,,,,,8050,,,,,\n1970,2,66,,100,4.54,1,0,1,100,1,25,4,40,8050,1,,,2,,,25,,,23,,,4,,,6,,,40,,,60,,,8050,,,6050,,\n1970,2,66,,100,4.54,1,0,2,100,2,23,6,60,6050,1,,,1,,,25,,,25,,,4,,,4,,,40,,,40,,,8050,,,8050,,\n1970,2,66,,100,4.54,1,0,3,100,2,4,1,11,9999999,1,2,1,,,,25,23,25,,,,4,6,4,,,,40,60,40,,,,8050,6050,8050,,,\n1970,2,66,,100,4.54,1,0,4,100,1,2,0,1,9999999,1,2,1,,,,25,23,25,,,,4,6,4,,,,40,60,40,,,,8050,6050,8050,,,\n1970,2,68,,100,4.54,1,0,1,100,1,40,5,50,3850,1,,,2,,,40,,,34,,,5,,,10,,,50,,,100,,,3850,,,850,,\n1970,2,68,,100,4.54,1,0,2,100,2,34,10,100,850,1,,,1,,,40,,,40,,,5,,,5,,,50,,,50,,,3850,,,3850,,\n1970,2,68,,100,4.54,1,0,3,100,1,1,0,1,9999999,1,2,1,,,,40,34,40,,,,5,10,5,,,,50,100,50,,,,3850,850,3850,,,\n1970,2,68,,100,4.54,1,0,4,100,2,0,0,1,9999999,1,2,1,,,,40,34,40,,,,5,10,5,,,,50,100,50,,,,3850,850,3850,,,\n1970,2,68,,100,4.54,1,0,5,100,2,5,1,11,9999999,1,2,1,,,,40,34,40,,,,5,10,5,,,,50,100,50,,,,3850,850,3850,,,\n1970,2,69,,100,4.54,1,0,1,100,2,35,4,40,3850,2,,,,,,35,,,,,,4,,,,,,40,,,,,,3850,,,,,\n1970,2,69,,100,4.54,1,0,2,100,2,8,1,15,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,\n1970,2,69,,100,4.54,1,0,3,100,2,7,1,15,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,\n1970,2,69,,100,4.54,1,0,4,100,2,6,1,14,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,\n1970,2,69,,100,4.54,1,0,5,100,2,2,0,1,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,\n1970,2,69,,100,4.54,1,0,6,100,1,13,2,23,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,\n1970,2,70,,100,4.54,1,0,1,100,1,29,2,26,1750,1,,,,,,29,,,,,,2,,,,,,26,,,,,,1750,,,,,\n1970,2,71,,100,4.54,1,0,1,100,1,35,7,70,12050,1,,,,,,35,,,,,,7,,,,,,70,,,,,,12050,,,,,\n1970,2,71,,100,4.54,1,0,2,100,2,31,10,100,5450,1,,,,,,35,,,,,,7,,,,,,70,,,,,,12050,,,,,\n1970,2,72,,100,4.54,1,0,1,100,2,52,4,40,1550,2,,,,,,52,,,,,,4,,,,,,40,,,,,,1550,,,,,\n1970,2,72,,100,4.54,1,0,2,100,2,26,5,50,2450,2,2,,,,,52,52,,,,,4,4,,,,,40,40,,,,,1550,1550,,,,\n1970,2,72,,100,4.54,1,0,3,100,2,6,1,14,9999999,2,2,,,,,52,52,,,,,4,4,,,,,40,40,,,,,1550,1550,,,,\n1970,2,72,,100,4.54,1,0,4,100,2,2,0,1,9999999,2,2,,,,,52,52,,,,,4,4,,,,,40,40,,,,,1550,1550,,,,\n1970,2,72,,100,4.54,1,0,5,100,1,1,0,1,9999999,2,2,,,,,52,52,,,,,4,4,,,,,40,40,,,,,1550,1550,,,,\n1970,2,74,,100,4.54,1,0,1,100,2,33,6,60,10050,2,,,,,,33,,,,,,6,,,,,,60,,,,,,10050,,,,,\n1970,2,74,,100,4.54,1,0,2,100,1,13,2,22,9999999,2,2,,,,,33,33,,,,,6,6,,,,,60,60,,,,,10050,10050,,,,\n1970,2,74,,100,4.54,1,0,3,100,1,11,1,17,9999999,2,2,,,,,33,33,,,,,6,6,,,,,60,60,,,,,10050,10050,,,,\n1970,2,75,,100,4.54,1,0,1,100,2,26,9,90,10550,2,,,,,,26,,,,,,9,,,,,,90,,,,,,10550,,,,,\n1970,2,76,,100,4.54,1,0,1,100,1,35,11,111,17250,1,,,,,,35,,,,,,11,,,,,,111,,,,,,17250,,,,,\n1970,2,77,,100,4.54,1,0,1,100,1,49,6,60,11050,1,,,2,,,49,,,49,,,6,,,6,,,60,,,60,,,11050,,,0,,\n1970,2,77,,100,4.54,1,0,2,100,2,49,6,60,0,1,,,1,,,49,,,49,,,6,,,6,,,60,,,60,,,11050,,,11050,,\n1970,2,77,,100,4.54,1,0,3,100,2,16,4,40,3550,1,2,1,,,,49,49,49,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,\n1970,2,77,,100,4.54,1,0,4,100,1,12,2,23,9999999,1,2,1,,,,49,49,49,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,\n1970,2,77,,100,4.54,1,0,5,100,2,20,6,60,6050,1,2,1,,,,49,49,49,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,\n1970,2,78,,100,4.54,1,0,1,100,1,72,2,26,2050,1,,,2,,,72,,,65,,,2,,,2,,,26,,,26,,,2050,,,750,,\n1970,2,78,,100,4.54,1,0,2,100,2,65,2,26,750,1,,,1,,,72,,,72,,,2,,,2,,,26,,,26,,,2050,,,2050,,\n1970,2,79,,100,4.54,1,0,1,100,1,37,6,60,7550,1,,,2,,,37,,,35,,,6,,,9,,,60,,,90,,,7550,,,5550,,\n1970,2,79,,100,4.54,1,0,2,100,2,35,9,90,5550,1,,,1,,,37,,,37,,,6,,,6,,,60,,,60,,,7550,,,7550,,\n1970,2,80,,100,4.54,1,0,1,100,1,44,6,60,6050,1,,,2,,,44,,,45,,,6,,,6,,,60,,,60,,,6050,,,0,,\n1970,2,80,,100,4.54,1,0,2,100,2,45,6,60,0,1,,,1,,,44,,,44,,,6,,,6,,,60,,,60,,,6050,,,6050,,\n1970,2,80,,100,4.54,1,0,3,100,1,25,6,60,5050,1,2,1,,,,44,45,44,,,,6,6,6,,,,60,60,60,,,,6050,0,6050,,,\n1970,2,81,,100,4.54,1,0,1,100,1,35,2,22,4050,1,,,2,,,35,,,31,,,2,,,2,,,22,,,23,,,4050,,,0,,\n1970,2,81,,100,4.54,1,0,2,100,2,31,2,23,0,1,,,1,,,35,,,35,,,2,,,2,,,22,,,22,,,4050,,,4050,,\n1970,2,81,,100,4.54,1,0,3,100,2,13,2,23,9999999,1,2,1,,,,35,31,35,,,,2,2,2,,,,22,23,22,,,,4050,0,4050,,,\n1970,2,81,,100,4.54,1,0,4,100,2,4,0,2,9999999,1,2,1,,,,35,31,35,,,,2,2,2,,,,22,23,22,,,,4050,0,4050,,,\n1970,2,81,,100,4.54,1,0,5,100,2,1,0,1,9999999,1,2,1,,,,35,31,35,,,,2,2,2,,,,22,23,22,,,,4050,0,4050,,,\n1970,2,82,,100,4.54,1,0,1,100,1,45,6,60,10050,1,,,2,,,45,,,46,,,6,,,7,,,60,,,70,,,10050,,,4050,,\n1970,2,82,,100,4.54,1,0,2,100,2,46,7,70,4050,1,,,1,,,45,,,45,,,6,,,6,,,60,,,60,,,10050,,,10050,,\n1970,2,82,,100,4.54,1,0,3,100,1,19,6,60,2050,1,2,1,,,,45,46,45,,,,6,7,6,,,,60,70,60,,,,10050,4050,10050,,,\n1970,2,82,,100,4.54,1,0,4,100,2,13,2,23,9999999,1,2,1,,,,45,46,45,,,,6,7,6,,,,60,70,60,,,,10050,4050,10050,,,\n1970,2,83,,100,4.54,1,0,1,100,1,58,2,26,9650,1,,,2,,,58,,,59,,,2,,,4,,,26,,,40,,,9650,,,0,,\n1970,2,83,,100,4.54,1,0,2,100,2,59,4,40,0,1,,,1,,,58,,,58,,,2,,,2,,,26,,,26,,,9650,,,9650,,\n1970,2,83,,100,4.54,1,0,3,100,1,23,10,100,2050,1,2,1,,,,58,59,58,,,,2,4,2,,,,26,40,26,,,,9650,0,9650,,,\n1970,2,84,,100,4.54,1,0,1,100,1,68,2,26,5050,1,,,,,,68,,,,,,2,,,,,,26,,,,,,5050,,,,,\n1970,2,85,,100,4.54,1,0,1,100,1,40,2,22,8050,1,,,2,,,40,,,35,,,2,,,1,,,22,,,17,,,8050,,,0,,\n1970,2,85,,100,4.54,1,0,2,100,2,35,1,17,0,1,,,1,,,40,,,40,,,2,,,2,,,22,,,22,,,8050,,,8050,,\n1970,2,85,,100,4.54,1,0,3,100,2,1,0,1,9999999,1,2,1,,,,40,35,40,,,,2,1,2,,,,22,17,22,,,,8050,0,8050,,,\n1970,2,85,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,2,1,,,,40,35,40,,,,2,1,2,,,,22,17,22,,,,8050,0,8050,,,\n1970,2,86,,100,4.54,1,0,1,100,1,56,2,23,7950,1,,,,,,56,,,,,,2,,,,,,23,,,,,,7950,,,,,\n1970,2,86,,100,4.54,1,0,2,100,2,82,2,26,1150,1,,1,,,,56,,56,,,,2,,2,,,,23,,23,,,,7950,,7950,,,\n1970,2,87,,100,4.54,1,0,1,100,1,28,6,60,12150,1,,,2,,,28,,,28,,,6,,,6,,,60,,,60,,,12150,,,0,,\n1970,2,87,,100,4.54,1,0,2,100,2,28,6,60,0,1,,,1,,,28,,,28,,,6,,,6,,,60,,,60,,,12150,,,12150,,\n1970,2,87,,100,4.54,1,0,3,100,1,1,0,1,9999999,1,2,1,,,,28,28,28,,,,6,6,6,,,,60,60,60,,,,12150,0,12150,,,\n1970,2,88,,100,4.54,1,0,1,100,2,70,2,26,3550,2,,,,,,70,,,,,,2,,,,,,26,,,,,,3550,,,,,\n1970,2,89,,100,4.54,1,0,1,100,1,54,2,26,7150,1,,,2,,,54,,,55,,,2,,,4,,,26,,,40,,,7150,,,0,,\n1970,2,89,,100,4.54,1,0,2,100,2,55,4,40,0,1,,,1,,,54,,,54,,,2,,,2,,,26,,,26,,,7150,,,7150,,\n1970,2,90,,100,4.54,1,0,1,100,2,42,6,60,0,2,,,,,,42,,,,,,6,,,,,,60,,,,,,0,,,,,\n1970,2,90,,100,4.54,1,0,2,100,1,14,2,26,0,2,2,,,,,42,42,,,,,6,6,,,,,60,60,,,,,0,0,,,,\n1970,2,90,,100,4.54,1,0,3,100,1,13,2,23,9999999,2,2,,,,,42,42,,,,,6,6,,,,,60,60,,,,,0,0,,,,\n1970,2,90,,100,4.54,1,0,4,100,1,11,2,22,9999999,2,2,,,,,42,42,,,,,6,6,,,,,60,60,,,,,0,0,,,,\n1970,2,90,,100,4.54,1,0,5,100,2,8,1,14,9999999,2,2,,,,,42,42,,,,,6,6,,,,,60,60,,,,,0,0,,,,\n1970,2,91,,100,4.54,1,0,1,100,2,36,6,60,3250,2,,,,,,36,,,,,,6,,,,,,60,,,,,,3250,,,,,\n1970,2,92,,100,4.54,1,0,1,100,1,42,4,40,6250,1,,,2,,,42,,,33,,,4,,,4,,,40,,,40,,,6250,,,750,,\n1970,2,92,,100,4.54,1,0,2,100,2,33,4,40,750,1,,,1,,,42,,,42,,,4,,,4,,,40,,,40,,,6250,,,6250,,\n1970,2,92,,100,4.54,1,0,3,100,2,13,2,25,9999999,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,\n1970,2,92,,100,4.54,1,0,4,100,1,12,1,17,9999999,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,\n1970,2,92,,100,4.54,1,0,5,100,1,10,1,15,9999999,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,\n1970,2,92,,100,4.54,1,0,6,100,2,6,1,12,9999999,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,\n1970,2,92,,100,4.54,1,0,7,100,2,95,2,23,1750,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,\n1970,2,93,,100,4.54,1,0,1,100,1,35,10,100,12050,1,,,2,,,35,,,31,,,10,,,6,,,100,,,60,,,12050,,,3250,,\n1970,2,93,,100,4.54,1,0,2,100,2,31,6,60,3250,1,,,1,,,35,,,35,,,10,,,10,,,100,,,100,,,12050,,,12050,,\n1970,2,93,,100,4.54,1,0,3,100,1,14,2,26,0,1,2,1,,,,35,31,35,,,,10,6,10,,,,100,60,100,,,,12050,3250,12050,,,\n1970,2,93,,100,4.54,1,0,4,100,2,12,2,23,9999999,1,2,1,,,,35,31,35,,,,10,6,10,,,,100,60,100,,,,12050,3250,12050,,,\n1970,2,93,,100,4.54,1,0,5,100,2,10,1,16,9999999,1,2,1,,,,35,31,35,,,,10,6,10,,,,100,60,100,,,,12050,3250,12050,,,\n1970,2,94,,100,4.54,1,0,1,100,2,72,9,90,6350,2,,,,,,72,,,,,,9,,,,,,90,,,,,,6350,,,,,\n1970,2,94,,100,4.54,1,0,2,100,2,32,11,111,8650,2,2,,,,,72,72,,,,,9,9,,,,,90,90,,,,,6350,6350,,,,\n1970,2,94,,100,4.54,1,0,3,100,2,42,6,60,7050,2,2,,,,,72,72,,,,,9,9,,,,,90,90,,,,,6350,6350,,,,\n1970,2,95,,100,4.54,1,0,1,100,1,50,5,50,16150,1,,,2,,,50,,,48,,,5,,,6,,,50,,,60,,,16150,,,50,,\n1970,2,95,,100,4.54,1,0,2,100,2,48,6,60,50,1,,,1,,,50,,,50,,,5,,,5,,,50,,,50,,,16150,,,16150,,\n1970,2,95,,100,4.54,1,0,3,100,2,15,3,30,0,1,2,1,,,,50,48,50,,,,5,6,5,,,,50,60,50,,,,16150,50,16150,,,\n1970,2,95,,100,4.54,1,0,4,100,1,13,2,23,9999999,1,2,1,,,,50,48,50,,,,5,6,5,,,,50,60,50,,,,16150,50,16150,,,\n1970,2,96,,100,4.54,1,0,1,100,1,21,4,40,12050,1,,,2,,,21,,,19,,,4,,,6,,,40,,,60,,,12050,,,12050,,\n1970,2,96,,100,4.54,1,0,2,100,2,19,6,60,12050,1,,,1,,,21,,,21,,,4,,,4,,,40,,,40,,,12050,,,12050,,\n1970,2,97,,100,4.54,1,0,1,100,1,66,4,40,7150,1,,,2,,,66,,,64,,,4,,,2,,,40,,,23,,,7150,,,550,,\n1970,2,97,,100,4.54,1,0,2,100,2,64,2,23,550,1,,,1,,,66,,,66,,,4,,,4,,,40,,,40,,,7150,,,7150,,\n1970,2,98,,100,4.54,1,0,1,100,1,56,6,60,11050,1,,,2,,,56,,,53,,,6,,,6,,,60,,,60,,,11050,,,0,,\n1970,2,98,,100,4.54,1,0,2,100,2,53,6,60,0,1,,,1,,,56,,,56,,,6,,,6,,,60,,,60,,,11050,,,11050,,\n1970,2,98,,100,4.54,1,0,3,100,1,29,7,70,5050,1,2,1,,,,56,53,56,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,\n1970,2,98,,100,4.54,1,0,4,100,1,18,5,50,0,1,2,1,,,,56,53,56,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,\n1970,2,99,,100,4.54,1,0,1,100,1,51,8,80,12050,1,,,2,,,51,,,55,,,8,,,4,,,80,,,40,,,12050,,,0,,\n1970,2,99,,100,4.54,1,0,2,100,2,55,4,40,0,1,,,1,,,51,,,51,,,8,,,8,,,80,,,80,,,12050,,,12050,,\n1970,2,99,,100,4.54,1,0,3,100,2,11,2,22,9999999,1,2,1,,,,51,55,51,,,,8,4,8,,,,80,40,80,,,,12050,0,12050,,,\n1970,2,100,,100,4.54,1,0,1,100,2,56,4,40,6250,2,,,,,,56,,,,,,4,,,,,,40,,,,,,6250,,,,,\n1970,2,101,,100,4.54,1,0,1,100,1,42,9,90,21850,1,,,2,,,42,,,39,,,9,,,6,,,90,,,60,,,21850,,,650,,\n1970,2,101,,100,4.54,1,0,2,100,2,39,6,60,650,1,,,1,,,42,,,42,,,9,,,9,,,90,,,90,,,21850,,,21850,,\n1970,2,101,,100,4.54,1,0,3,100,1,18,5,50,650,1,2,1,,,,42,39,42,,,,9,6,9,,,,90,60,90,,,,21850,650,21850,,,\n1970,2,101,,100,4.54,1,0,4,100,2,11,2,22,9999999,1,2,1,,,,42,39,42,,,,9,6,9,,,,90,60,90,,,,21850,650,21850,,,\n1970,2,102,,100,4.54,1,0,1,100,1,49,6,60,19150,1,,,2,,,49,,,46,,,6,,,6,,,60,,,60,,,19150,,,0,,\n1970,2,102,,100,4.54,1,0,2,100,2,46,6,60,0,1,,,1,,,49,,,49,,,6,,,6,,,60,,,60,,,19150,,,19150,,\n1970,2,102,,100,4.54,1,0,3,100,2,9,1,16,9999999,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,\n1970,2,102,,100,4.54,1,0,4,100,1,6,1,12,9999999,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,\n1970,2,102,,100,4.54,1,0,5,100,1,17,5,50,2050,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,\n1970,2,102,,100,4.54,1,0,6,100,1,16,3,30,1450,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,\n1970,2,102,,100,4.54,1,0,7,100,1,16,3,30,1450,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,\n1970,2,102,,100,4.54,1,0,8,100,2,11,2,22,9999999,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,\n1970,2,103,,100,4.54,1,0,1,100,1,59,7,70,8850,1,,,,,,59,,,,,,7,,,,,,70,,,,,,8850,,,,,\n1970,2,103,,100,4.54,1,0,2,100,2,69,6,60,450,1,,,,,,59,,,,,,7,,,,,,70,,,,,,8850,,,,,\n1970,2,104,,100,4.54,1,0,1,100,1,59,5,50,6750,1,,,2,,,59,,,59,,,5,,,2,,,50,,,26,,,6750,,,4850,,\n1970,2,104,,100,4.54,1,0,2,100,2,59,2,26,4850,1,,,1,,,59,,,59,,,5,,,5,,,50,,,50,,,6750,,,6750,,\n1970,2,104,,100,4.54,1,0,3,100,1,20,6,60,1650,1,2,1,,,,59,59,59,,,,5,2,5,,,,50,26,50,,,,6750,4850,6750,,,\n1970,2,105,,100,4.54,1,0,1,100,2,55,7,70,9450,2,,,,,,55,,,,,,7,,,,,,70,,,,,,9450,,,,,\n1970,2,106,,100,4.54,1,0,1,100,1,63,2,23,7950,1,,,2,,,63,,,57,,,2,,,4,,,23,,,40,,,7950,,,4650,,\n1970,2,106,,100,4.54,1,0,2,100,2,57,4,40,4650,1,,,1,,,63,,,63,,,2,,,2,,,23,,,23,,,7950,,,7950,,\n1970,2,106,,100,4.54,1,0,3,100,1,21,8,80,1450,1,2,1,,,,63,57,63,,,,2,4,2,,,,23,40,23,,,,7950,4650,7950,,,\n1970,2,107,,100,4.54,1,0,1,100,1,40,3,30,18650,1,,,2,,,40,,,37,,,3,,,6,,,30,,,60,,,18650,,,0,,\n1970,2,107,,100,4.54,1,0,2,100,2,37,6,60,0,1,,,1,,,40,,,40,,,3,,,3,,,30,,,30,,,18650,,,18650,,\n1970,2,107,,100,4.54,1,0,3,100,1,14,2,26,0,1,2,1,,,,40,37,40,,,,3,6,3,,,,30,60,30,,,,18650,0,18650,,,\n1970,2,107,,100,4.54,1,0,4,100,1,8,1,15,9999999,1,2,1,,,,40,37,40,,,,3,6,3,,,,30,60,30,,,,18650,0,18650,,,\n1970,2,108,,100,4.54,1,0,1,100,1,36,6,65,4050,1,,,2,,,36,,,36,,,6,,,2,,,65,,,26,,,4050,,,1550,,\n1970,2,108,,100,4.54,1,0,2,100,2,36,2,26,1550,1,,,1,,,36,,,36,,,6,,,6,,,65,,,65,,,4050,,,4050,,\n1970,2,108,,100,4.54,1,0,3,100,2,17,5,50,950,1,2,1,,,,36,36,36,,,,6,2,6,,,,65,26,65,,,,4050,1550,4050,,,\n1970,2,108,,100,4.54,1,0,4,100,2,14,2,26,0,1,2,1,,,,36,36,36,,,,6,2,6,,,,65,26,65,,,,4050,1550,4050,,,\n1970,2,108,,100,4.54,1,0,5,100,1,10,1,17,9999999,1,2,1,,,,36,36,36,,,,6,2,6,,,,65,26,65,,,,4050,1550,4050,,,\n1970,2,108,,100,4.54,1,0,6,100,2,8,1,15,9999999,1,2,1,,,,36,36,36,,,,6,2,6,,,,65,26,65,,,,4050,1550,4050,,,\n1970,2,109,,100,4.54,1,0,1,100,1,37,10,100,1550,1,,,,,,37,,,,,,10,,,,,,100,,,,,,1550,,,,,\n1970,2,109,,100,4.54,1,0,2,100,1,48,10,100,11050,1,,,,,,37,,,,,,10,,,,,,100,,,,,,1550,,,,,\n1970,2,110,,100,4.54,1,0,1,100,2,78,6,60,3950,2,,,,,,78,,,,,,6,,,,,,60,,,,,,3950,,,,,\n1970,2,111,,100,4.54,1,0,1,100,1,32,4,40,6050,1,,,,,,32,,,,,,4,,,,,,40,,,,,,6050,,,,,\n1970,2,112,,100,4.54,1,0,1,100,2,63,2,25,250,2,,,,,,63,,,,,,2,,,,,,25,,,,,,250,,,,,\n1970,2,113,,100,4.54,1,0,1,100,1,42,5,50,10050,1,,,2,,,42,,,34,,,5,,,6,,,50,,,60,,,10050,,,550,,\n1970,2,113,,100,4.54,1,0,2,100,2,34,6,60,550,1,,,1,,,42,,,42,,,5,,,5,,,50,,,50,,,10050,,,10050,,\n1970,2,113,,100,4.54,1,0,3,100,1,13,2,23,9999999,1,2,1,,,,42,34,42,,,,5,6,5,,,,50,60,50,,,,10050,550,10050,,,\n1970,2,113,,100,4.54,1,0,4,100,1,12,2,22,9999999,1,2,1,,,,42,34,42,,,,5,6,5,,,,50,60,50,,,,10050,550,10050,,,\n1970,2,113,,100,4.54,1,0,5,100,2,11,1,17,9999999,1,2,1,,,,42,34,42,,,,5,6,5,,,,50,60,50,,,,10050,550,10050,,,\n1970,2,114,,100,4.54,1,0,1,100,1,82,2,26,0,1,,,2,,,82,,,75,,,2,,,2,,,26,,,26,,,0,,,1450,,\n1970,2,114,,100,4.54,1,0,2,100,2,75,2,26,1450,1,,,1,,,82,,,82,,,2,,,2,,,26,,,26,,,0,,,0,,\n1970,2,114,,100,4.54,1,0,3,100,1,47,0,2,0,1,2,1,,,,82,75,82,,,,2,2,2,,,,26,26,26,,,,0,1450,0,,,\n1970,2,115,,100,4.54,1,0,1,100,2,70,2,23,1250,2,,,,,,70,,,,,,2,,,,,,23,,,,,,1250,,,,,\n1970,2,116,,100,4.54,1,0,1,100,1,65,3,30,3450,1,,,2,,,65,,,65,,,3,,,6,,,30,,,60,,,3450,,,350,,\n1970,2,116,,100,4.54,1,0,2,100,2,65,6,60,350,1,,,1,,,65,,,65,,,3,,,3,,,30,,,30,,,3450,,,3450,,\n1970,2,116,,100,4.54,1,0,3,100,1,42,7,70,6750,1,2,1,,,,65,65,65,,,,3,6,3,,,,30,60,30,,,,3450,350,3450,,,\n1970,2,117,,100,4.54,1,0,1,100,1,34,6,60,14050,1,,,2,,,34,,,31,,,6,,,7,,,60,,,70,,,14050,,,0,,\n1970,2,117,,100,4.54,1,0,2,100,2,31,7,70,0,1,,,1,,,34,,,34,,,6,,,6,,,60,,,60,,,14050,,,14050,,\n1970,2,117,,100,4.54,1,0,3,100,1,8,1,15,9999999,1,2,1,,,,34,31,34,,,,6,7,6,,,,60,70,60,,,,14050,0,14050,,,\n1970,2,117,,100,4.54,1,0,4,100,2,7,1,14,9999999,1,2,1,,,,34,31,34,,,,6,7,6,,,,60,70,60,,,,14050,0,14050,,,\n1970,2,117,,100,4.54,1,0,5,100,2,5,1,11,9999999,1,2,1,,,,34,31,34,,,,6,7,6,,,,60,70,60,,,,14050,0,14050,,,\n1970,2,117,,100,4.54,1,0,6,100,2,1,0,1,9999999,1,2,1,,,,34,31,34,,,,6,7,6,,,,60,70,60,,,,14050,0,14050,,,\n1970,2,118,,100,4.54,1,0,1,100,1,23,6,60,6050,1,,,2,,,23,,,20,,,6,,,7,,,60,,,70,,,6050,,,6050,,\n1970,2,118,,100,4.54,1,0,2,100,2,20,7,70,6050,1,,,1,,,23,,,23,,,6,,,6,,,60,,,60,,,6050,,,6050,,\n1970,2,119,,100,4.54,1,0,1,100,1,24,11,110,8650,1,,,2,,,24,,,25,,,11,,,6,,,110,,,60,,,8650,,,4050,,\n1970,2,119,,100,4.54,1,0,2,100,2,25,6,60,4050,1,,,1,,,24,,,24,,,11,,,11,,,110,,,110,,,8650,,,8650,,\n1970,2,119,,100,4.54,1,0,3,100,2,0,0,1,9999999,1,2,1,,,,24,25,24,,,,11,6,11,,,,110,60,110,,,,8650,4050,8650,,,\n1970,2,120,,100,4.54,1,0,1,100,1,35,2,26,10150,1,,,2,,,35,,,29,,,2,,,6,,,26,,,60,,,10150,,,2550,,\n1970,2,120,,100,4.54,1,0,2,100,2,29,6,60,2550,1,,,1,,,35,,,35,,,2,,,2,,,26,,,26,,,10150,,,10150,,\n1970,2,120,,100,4.54,1,0,3,100,2,2,0,1,9999999,1,2,1,,,,35,29,35,,,,2,6,2,,,,26,60,26,,,,10150,2550,10150,,,\n1970,2,120,,100,4.54,1,0,4,100,1,4,0,2,9999999,1,2,1,,,,35,29,35,,,,2,6,2,,,,26,60,26,,,,10150,2550,10150,,,\n1970,2,121,,100,4.54,1,0,1,100,1,45,4,40,12550,1,,,,,,45,,,,,,4,,,,,,40,,,,,,12550,,,,,\n1970,2,122,,100,4.54,1,0,1,100,1,64,6,60,5550,1,,,2,,,64,,,62,,,6,,,2,,,60,,,26,,,5550,,,0,,\n1970,2,122,,100,4.54,1,0,2,100,2,62,2,26,0,1,,,1,,,64,,,64,,,6,,,6,,,60,,,60,,,5550,,,5550,,\n1970,2,123,,100,4.54,1,0,1,100,1,26,1,12,1950,1,,,2,,,26,,,27,,,1,,,8,,,12,,,80,,,1950,,,550,,\n1970,2,123,,100,4.54,1,0,2,100,2,27,8,80,550,1,,,1,,,26,,,26,,,1,,,1,,,12,,,12,,,1950,,,1950,,\n1970,2,123,,100,4.54,1,0,3,100,1,4,0,2,9999999,1,2,1,,,,26,27,26,,,,1,8,1,,,,12,80,12,,,,1950,550,1950,,,\n1970,2,123,,100,4.54,1,0,4,100,1,4,0,2,9999999,1,2,1,,,,26,27,26,,,,1,8,1,,,,12,80,12,,,,1950,550,1950,,,\n1970,2,123,,100,4.54,1,0,5,100,2,2,0,1,9999999,1,2,1,,,,26,27,26,,,,1,8,1,,,,12,80,12,,,,1950,550,1950,,,\n1970,2,124,,100,4.54,1,0,1,100,1,44,2,23,10750,1,,,2,,,44,,,48,,,2,,,2,,,23,,,26,,,10750,,,0,,\n1970,2,124,,100,4.54,1,0,2,100,2,48,2,26,0,1,,,1,,,44,,,44,,,2,,,2,,,23,,,23,,,10750,,,10750,,\n1970,2,124,,100,4.54,1,0,3,100,2,10,1,17,9999999,1,2,1,,,,44,48,44,,,,2,2,2,,,,23,26,23,,,,10750,0,10750,,,\n1970,2,125,,100,4.54,1,0,1,100,1,55,2,26,11450,1,,,2,,,55,,,50,,,2,,,7,,,26,,,70,,,11450,,,7050,,\n1970,2,125,,100,4.54,1,0,2,100,2,50,7,70,7050,1,,,1,,,55,,,55,,,2,,,2,,,26,,,26,,,11450,,,11450,,\n1970,2,126,,100,4.54,1,0,1,100,1,53,2,26,11150,1,,,2,,,53,,,49,,,2,,,2,,,26,,,26,,,11150,,,0,,\n1970,2,126,,100,4.54,1,0,2,100,2,49,2,26,0,1,,,1,,,53,,,53,,,2,,,2,,,26,,,26,,,11150,,,11150,,\n1970,2,126,,100,4.54,1,0,3,100,1,21,5,50,750,1,2,1,,,,53,49,53,,,,2,2,2,,,,26,26,26,,,,11150,0,11150,,,\n1970,2,126,,100,4.54,1,0,4,100,2,15,3,30,0,1,2,1,,,,53,49,53,,,,2,2,2,,,,26,26,26,,,,11150,0,11150,,,\n1970,2,127,,100,4.54,1,0,1,100,1,40,8,80,11550,1,,,2,,,40,,,39,,,8,,,6,,,80,,,65,,,11550,,,2950,,\n1970,2,127,,100,4.54,1,0,2,100,2,39,6,65,2950,1,,,1,,,40,,,40,,,8,,,8,,,80,,,80,,,11550,,,11550,,\n1970,2,127,,100,4.54,1,0,3,100,2,17,5,50,250,1,2,1,,,,40,39,40,,,,8,6,8,,,,80,65,80,,,,11550,2950,11550,,,\n1970,2,127,,100,4.54,1,0,4,100,1,14,2,26,350,1,2,1,,,,40,39,40,,,,8,6,8,,,,80,65,80,,,,11550,2950,11550,,,\n1970,2,127,,100,4.54,1,0,5,100,2,8,1,15,9999999,1,2,1,,,,40,39,40,,,,8,6,8,,,,80,65,80,,,,11550,2950,11550,,,\n1970,2,128,,100,4.54,1,0,1,100,1,48,3,30,9050,1,,,2,,,48,,,44,,,3,,,3,,,30,,,30,,,9050,,,0,,\n1970,2,128,,100,4.54,1,0,2,100,2,44,3,30,0,1,,,1,,,48,,,48,,,3,,,3,,,30,,,30,,,9050,,,9050,,\n1970,2,128,,100,4.54,1,0,3,100,1,21,6,60,6850,1,2,1,,,,48,44,48,,,,3,3,3,,,,30,30,30,,,,9050,0,9050,,,\n1970,2,128,,100,4.54,1,0,4,100,2,16,3,30,0,1,2,1,,,,48,44,48,,,,3,3,3,,,,30,30,30,,,,9050,0,9050,,,\n1970,2,128,,100,4.54,1,0,5,100,2,15,3,30,0,1,2,1,,,,48,44,48,,,,3,3,3,,,,30,30,30,,,,9050,0,9050,,,\n1970,2,128,,100,4.54,1,0,6,100,2,11,2,22,9999999,1,2,1,,,,48,44,48,,,,3,3,3,,,,30,30,30,,,,9050,0,9050,,,\n1970,2,130,,100,4.54,1,0,1,100,1,61,2,22,10250,1,,,2,,,61,,,58,,,2,,,2,,,22,,,25,,,10250,,,3050,,\n1970,2,130,,100,4.54,1,0,2,100,2,58,2,25,3050,1,,,1,,,61,,,61,,,2,,,2,,,22,,,22,,,10250,,,10250,,\n1970,2,131,,100,4.54,1,0,1,100,1,59,4,40,6550,1,,,,,,59,,,,,,4,,,,,,40,,,,,,6550,,,,,\n1970,2,132,,100,4.54,1,0,1,100,1,44,2,26,8850,1,,,2,,,44,,,39,,,2,,,1,,,26,,,17,,,8850,,,0,,\n1970,2,132,,100,4.54,1,0,2,100,2,39,1,17,0,1,,,1,,,44,,,44,,,2,,,2,,,26,,,26,,,8850,,,8850,,\n1970,2,132,,100,4.54,1,0,3,100,1,15,4,40,0,1,2,1,,,,44,39,44,,,,2,1,2,,,,26,17,26,,,,8850,0,8850,,,\n1970,2,132,,100,4.54,1,0,4,100,2,10,1,17,9999999,1,2,1,,,,44,39,44,,,,2,1,2,,,,26,17,26,,,,8850,0,8850,,,\n1970,2,134,,100,4.54,1,0,1,100,1,22,1,16,4850,1,,,,,,22,,,,,,1,,,,,,16,,,,,,4850,,,,,\n1970,2,134,,100,4.54,1,0,2,100,1,20,2,23,4850,1,,,,,,22,,,,,,1,,,,,,16,,,,,,4850,,,,,\n1970,2,135,,100,4.54,1,0,1,100,1,72,2,25,3350,1,,,2,,,72,,,75,,,2,,,2,,,25,,,26,,,3350,,,0,,\n1970,2,135,,100,4.54,1,0,2,100,2,75,2,26,0,1,,,1,,,72,,,72,,,2,,,2,,,25,,,25,,,3350,,,3350,,\n1970,2,136,,100,4.54,3,0,1,100,1,14,2,23,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n1970,2,137,,100,4.54,1,0,1,100,1,26,6,60,650,1,,,2,,,26,,,23,,,6,,,6,,,60,,,60,,,650,,,0,,\n1970,2,137,,100,4.54,1,0,2,100,2,23,6,60,0,1,,,1,,,26,,,26,,,6,,,6,,,60,,,60,,,650,,,650,,\n1970,2,137,,100,4.54,1,0,3,100,1,3,0,2,9999999,1,2,1,,,,26,23,26,,,,6,6,6,,,,60,60,60,,,,650,0,650,,,\n1970,2,137,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,26,23,26,,,,6,6,6,,,,60,60,60,,,,650,0,650,,,\n1970,2,137,,100,4.54,1,0,5,100,1,0,0,1,9999999,1,2,1,,,,26,23,26,,,,6,6,6,,,,60,60,60,,,,650,0,650,,,\n1970,2,138,,100,4.54,1,0,1,100,1,32,6,60,-850,1,,,,,,32,,,,,,6,,,,,,60,,,,,,-850,,,,,\n1970,2,138,,100,4.54,1,0,2,100,2,38,6,60,2650,1,,,,,,32,,,,,,6,,,,,,60,,,,,,-850,,,,,\n1970,2,140,,100,4.54,1,0,1,100,1,57,1,15,3050,1,,,2,,,57,,,52,,,1,,,2,,,15,,,25,,,3050,,,0,,\n1970,2,140,,100,4.54,1,0,2,100,2,52,2,25,0,1,,,1,,,57,,,57,,,1,,,1,,,15,,,15,,,3050,,,3050,,\n1970,2,140,,100,4.54,1,0,3,100,1,30,6,60,4050,1,2,1,,,,57,52,57,,,,1,2,1,,,,15,25,15,,,,3050,0,3050,,,\n1970,2,141,,100,4.54,1,0,1,100,2,75,2,23,5550,2,,,,,,75,,,,,,2,,,,,,23,,,,,,5550,,,,,\n1970,2,142,,100,4.54,1,0,1,100,1,76,2,22,750,1,,,,,,76,,,,,,2,,,,,,22,,,,,,750,,,,,\n1970,2,142,,100,4.54,1,0,2,100,2,72,2,26,250,1,,,,,,76,,,,,,2,,,,,,22,,,,,,750,,,,,\n1970,2,143,,100,4.54,1,0,1,100,1,58,6,60,7050,1,,,2,,,58,,,57,,,6,,,6,,,60,,,60,,,7050,,,6550,,\n1970,2,143,,100,4.54,1,0,2,100,2,57,6,60,6550,1,,,1,,,58,,,58,,,6,,,6,,,60,,,60,,,7050,,,7050,,\n1970,2,144,,100,4.54,1,0,1,100,1,30,11,110,15050,1,,,2,,,30,,,29,,,11,,,9,,,110,,,90,,,15050,,,0,,\n1970,2,144,,100,4.54,1,0,2,100,2,29,9,90,0,1,,,1,,,30,,,30,,,11,,,11,,,110,,,110,,,15050,,,15050,,\n1970,2,144,,100,4.54,1,0,3,100,2,3,0,2,9999999,1,2,1,,,,30,29,30,,,,11,9,11,,,,110,90,110,,,,15050,0,15050,,,\n1970,2,145,,100,4.54,1,0,1,100,1,30,2,26,5150,1,,,2,,,30,,,29,,,2,,,4,,,26,,,40,,,5150,,,3850,,\n1970,2,145,,100,4.54,1,0,2,100,2,29,4,40,3850,1,,,1,,,30,,,30,,,2,,,2,,,26,,,26,,,5150,,,5150,,\n1970,2,145,,100,4.54,1,0,3,100,2,12,2,22,9999999,1,2,1,,,,30,29,30,,,,2,4,2,,,,26,40,26,,,,5150,3850,5150,,,\n1970,2,145,,100,4.54,1,0,4,100,1,10,1,17,9999999,1,2,1,,,,30,29,30,,,,2,4,2,,,,26,40,26,,,,5150,3850,5150,,,\n1970,2,145,,100,4.54,1,0,5,100,2,8,1,14,9999999,1,2,1,,,,30,29,30,,,,2,4,2,,,,26,40,26,,,,5150,3850,5150,,,\n1970,2,145,,100,4.54,1,0,6,100,2,3,0,2,9999999,1,2,1,,,,30,29,30,,,,2,4,2,,,,26,40,26,,,,5150,3850,5150,,,\n1970,2,146,,100,4.54,1,0,1,100,2,70,6,60,1950,2,,,,,,70,,,,,,6,,,,,,60,,,,,,1950,,,,,\n1970,2,146,,100,4.54,1,0,2,100,1,37,2,26,7550,2,2,,,,,70,70,,,,,6,6,,,,,60,60,,,,,1950,1950,,,,\n1970,2,147,,100,4.54,1,0,1,100,1,34,9,90,13050,1,,,2,,,34,,,34,,,9,,,11,,,90,,,110,,,13050,,,0,,\n1970,2,147,,100,4.54,1,0,2,100,2,34,11,110,0,1,,,1,,,34,,,34,,,9,,,9,,,90,,,90,,,13050,,,13050,,\n1970,2,147,,100,4.54,1,0,3,100,2,9,1,16,9999999,1,2,1,,,,34,34,34,,,,9,11,9,,,,90,110,90,,,,13050,0,13050,,,\n1970,2,147,,100,4.54,1,0,4,100,1,7,1,14,9999999,1,2,1,,,,34,34,34,,,,9,11,9,,,,90,110,90,,,,13050,0,13050,,,\n1970,2,148,,100,4.54,1,0,1,100,2,79,6,60,1150,2,,,,,,79,,,,,,6,,,,,,60,,,,,,1150,,,,,\n1970,2,149,,100,4.54,1,0,1,100,1,70,2,26,11050,1,,,2,,,70,,,68,,,2,,,2,,,26,,,26,,,11050,,,0,,\n1970,2,149,,100,4.54,1,0,2,100,2,68,2,26,0,1,,,1,,,70,,,70,,,2,,,2,,,26,,,26,,,11050,,,11050,,\n1970,2,150,,100,4.54,1,0,1,100,1,35,6,60,15050,1,,,2,,,35,,,31,,,6,,,6,,,60,,,60,,,15050,,,1550,,\n1970,2,150,,100,4.54,1,0,2,100,2,31,6,60,1550,1,,,1,,,35,,,35,,,6,,,6,,,60,,,60,,,15050,,,15050,,\n1970,2,150,,100,4.54,1,0,3,100,1,8,1,14,9999999,1,2,1,,,,35,31,35,,,,6,6,6,,,,60,60,60,,,,15050,1550,15050,,,\n1970,2,150,,100,4.54,1,0,4,100,2,5,1,11,9999999,1,2,1,,,,35,31,35,,,,6,6,6,,,,60,60,60,,,,15050,1550,15050,,,\n1970,2,152,,100,4.54,1,0,1,100,1,58,6,60,12650,1,,,2,,,58,,,55,,,6,,,2,,,60,,,26,,,12650,,,0,,\n1970,2,152,,100,4.54,1,0,2,100,2,55,2,26,0,1,,,1,,,58,,,58,,,6,,,6,,,60,,,60,,,12650,,,12650,,\n1970,2,153,,100,4.54,1,0,1,100,2,49,4,40,4550,2,,,,,,49,,,,,,4,,,,,,40,,,,,,4550,,,,,\n1970,2,153,,100,4.54,1,0,2,100,1,39,5,50,6050,2,2,,,,,49,49,,,,,4,4,,,,,40,40,,,,,4550,4550,,,,\n1970,2,154,,100,4.54,3,0,1,100,1,73,2,26,4250,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n1970,2,155,,100,4.54,1,0,1,100,1,88,4,40,1550,1,,,2,,,88,,,84,,,4,,,2,,,40,,,26,,,1550,,,850,,\n1970,2,155,,100,4.54,1,0,2,100,2,84,2,26,850,1,,,1,,,88,,,88,,,4,,,4,,,40,,,40,,,1550,,,1550,,\n1970,2,156,,100,4.54,1,0,1,100,1,25,5,50,10050,1,,,2,,,25,,,24,,,5,,,6,,,50,,,60,,,10050,,,750,,\n1970,2,156,,100,4.54,1,0,2,100,2,24,6,60,750,1,,,1,,,25,,,25,,,5,,,5,,,50,,,50,,,10050,,,10050,,\n1970,2,156,,100,4.54,1,0,3,100,2,6,1,12,9999999,1,2,1,,,,25,24,25,,,,5,6,5,,,,50,60,50,,,,10050,750,10050,,,\n1970,2,156,,100,4.54,1,0,4,100,1,3,0,2,9999999,1,2,1,,,,25,24,25,,,,5,6,5,,,,50,60,50,,,,10050,750,10050,,,\n1970,2,157,,100,4.54,1,0,1,100,2,56,0,2,3450,2,,,,,,56,,,,,,0,,,,,,2,,,,,,3450,,,,,\n1970,2,158,,100,4.54,3,0,1,100,2,79,6,60,2250,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n1970,2,159,,100,4.54,1,0,1,100,1,25,8,80,1850,1,,,2,,,25,,,36,,,8,,,6,,,80,,,60,,,1850,,,3250,,\n1970,2,159,,100,4.54,1,0,2,100,2,36,6,60,3250,1,,,1,,,25,,,25,,,8,,,8,,,80,,,80,,,1850,,,1850,,\n1970,2,160,,100,4.54,1,0,1,100,2,28,6,60,3550,2,,,,,,28,,,,,,6,,,,,,60,,,,,,3550,,,,,\n1970,2,160,,100,4.54,1,0,2,100,1,7,1,12,9999999,2,2,,,,,28,28,,,,,6,6,,,,,60,60,,,,,3550,3550,,,,\n1970,2,161,,100,4.54,1,0,1,100,2,21,8,80,1350,2,,,,,,21,,,,,,8,,,,,,80,,,,,,1350,,,,,\n1970,2,161,,100,4.54,1,0,2,100,2,23,10,100,2750,2,,,,,,21,,,,,,8,,,,,,80,,,,,,1350,,,,,\n1970,2,162,,100,4.54,1,0,1,100,1,22,10,100,2050,1,,,2,,,22,,,22,,,10,,,10,,,100,,,100,,,2050,,,1250,,\n1970,2,162,,100,4.54,1,0,2,100,2,22,10,100,1250,1,,,1,,,22,,,22,,,10,,,10,,,100,,,100,,,2050,,,2050,,\n1970,2,163,,100,4.54,1,0,1,100,2,60,6,60,4350,2,,,,,,60,,,,,,6,,,,,,60,,,,,,4350,,,,,\n1970,2,164,,100,4.54,1,0,1,100,1,20,8,80,2250,1,,,,,,20,,,,,,8,,,,,,80,,,,,,2250,,,,,\n1970,2,165,,100,4.54,1,0,1,100,1,78,2,23,3250,1,,,,,,78,,,,,,2,,,,,,23,,,,,,3250,,,,,\n1970,2,166,,100,4.54,1,0,1,100,1,24,10,100,9050,1,,,2,,,24,,,24,,,10,,,8,,,100,,,80,,,9050,,,3250,,\n1970,2,166,,100,4.54,1,0,2,100,2,24,8,80,3250,1,,,1,,,24,,,24,,,10,,,10,,,100,,,100,,,9050,,,9050,,\n1970,2,166,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,24,24,24,,,,10,8,10,,,,100,80,100,,,,9050,3250,9050,,,\n1970,2,167,,100,4.54,1,0,1,100,1,34,10,100,17050,1,,,2,,,34,,,33,,,10,,,10,,,100,,,100,,,17050,,,0,,\n1970,2,167,,100,4.54,1,0,2,100,2,33,10,100,0,1,,,1,,,34,,,34,,,10,,,10,,,100,,,100,,,17050,,,17050,,\n1970,2,167,,100,4.54,1,0,3,100,1,9,1,16,9999999,1,2,1,,,,34,33,34,,,,10,10,10,,,,100,100,100,,,,17050,0,17050,,,\n1970,2,167,,100,4.54,1,0,4,100,2,7,1,12,9999999,1,2,1,,,,34,33,34,,,,10,10,10,,,,100,100,100,,,,17050,0,17050,,,\n1970,2,168,,100,4.54,1,0,1,100,1,27,6,60,16650,1,,,2,,,27,,,26,,,6,,,6,,,60,,,60,,,16650,,,0,,\n1970,2,168,,100,4.54,1,0,2,100,2,26,6,60,0,1,,,1,,,27,,,27,,,6,,,6,,,60,,,60,,,16650,,,16650,,\n1970,2,168,,100,4.54,1,0,3,100,2,5,1,11,9999999,1,2,1,,,,27,26,27,,,,6,6,6,,,,60,60,60,,,,16650,0,16650,,,\n1970,2,168,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,27,26,27,,,,6,6,6,,,,60,60,60,,,,16650,0,16650,,,\n1970,2,168,,100,4.54,1,0,5,100,2,1,0,1,9999999,1,2,1,,,,27,26,27,,,,6,6,6,,,,60,60,60,,,,16650,0,16650,,,\n1970,2,169,,100,4.54,1,0,1,100,1,28,6,65,11250,1,,,2,,,28,,,23,,,6,,,7,,,65,,,70,,,11250,,,6550,,\n1970,2,169,,100,4.54,1,0,2,100,2,23,7,70,6550,1,,,1,,,28,,,28,,,6,,,6,,,65,,,65,,,11250,,,11250,,\n1970,2,170,,100,4.54,1,0,1,100,1,45,6,60,12950,1,,,2,,,45,,,46,,,6,,,4,,,60,,,40,,,12950,,,0,,\n1970,2,170,,100,4.54,1,0,2,100,2,46,4,40,0,1,,,1,,,45,,,45,,,6,,,6,,,60,,,60,,,12950,,,12950,,\n1970,2,170,,100,4.54,1,0,3,100,1,18,6,65,1650,1,2,1,,,,45,46,45,,,,6,4,6,,,,60,40,60,,,,12950,0,12950,,,\n1970,2,170,,100,4.54,1,0,4,100,2,23,6,60,5550,1,2,1,,,,45,46,45,,,,6,4,6,,,,60,40,60,,,,12950,0,12950,,,\n1970,2,170,,100,4.54,1,0,5,100,2,0,0,1,9999999,1,2,,,,,45,23,,,,,6,6,,,,,60,60,,,,,12950,5550,,,,\n1970,2,171,,100,4.54,1,0,1,100,1,26,6,60,11450,1,,,2,,,26,,,27,,,6,,,6,,,60,,,60,,,11450,,,0,,\n1970,2,171,,100,4.54,1,0,2,100,2,27,6,60,0,1,,,1,,,26,,,26,,,6,,,6,,,60,,,60,,,11450,,,11450,,\n1970,2,171,,100,4.54,1,0,3,100,1,4,0,2,9999999,1,2,1,,,,26,27,26,,,,6,6,6,,,,60,60,60,,,,11450,0,11450,,,\n1970,2,171,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,26,27,26,,,,6,6,6,,,,60,60,60,,,,11450,0,11450,,,\n1970,2,171,,100,4.54,1,0,5,100,2,0,0,1,9999999,1,2,1,,,,26,27,26,,,,6,6,6,,,,60,60,60,,,,11450,0,11450,,,\n1970,2,172,,100,4.54,1,0,1,100,1,33,6,60,1450,1,,,2,,,33,,,37,,,6,,,2,,,60,,,26,,,1450,,,3450,,\n1970,2,172,,100,4.54,1,0,2,100,2,37,2,26,3450,1,,,1,,,33,,,33,,,6,,,6,,,60,,,60,,,1450,,,1450,,\n1970,2,172,,100,4.54,1,0,3,100,2,6,1,12,9999999,1,2,1,,,,33,37,33,,,,6,2,6,,,,60,26,60,,,,1450,3450,1450,,,\n1970,2,172,,100,4.54,1,0,4,100,1,5,1,11,9999999,1,2,1,,,,33,37,33,,,,6,2,6,,,,60,26,60,,,,1450,3450,1450,,,\n1970,2,174,,100,4.54,1,0,1,100,1,60,4,40,13850,1,,,2,,,60,,,48,,,4,,,6,,,40,,,60,,,13850,,,0,,\n1970,2,174,,100,4.54,1,0,2,100,2,48,6,60,0,1,,,1,,,60,,,60,,,4,,,4,,,40,,,40,,,13850,,,13850,,\n1970,2,175,,100,4.54,1,0,1,100,1,37,1,15,3550,1,,,,,,37,,,,,,1,,,,,,15,,,,,,3550,,,,,\n1970,2,175,,100,4.54,1,0,2,100,1,29,1,17,1550,1,,,,,,37,,,,,,1,,,,,,15,,,,,,3550,,,,,\n1970,2,175,,100,4.54,1,0,3,100,1,28,1,17,2050,1,,,,,,37,,,,,,1,,,,,,15,,,,,,3550,,,,,\n1970,2,175,,100,4.54,1,0,4,100,1,29,2,23,1350,1,,,,,,37,,,,,,1,,,,,,15,,,,,,3550,,,,,\n1970,2,176,,100,4.54,4,0,1,100,1,34,11,111,11550,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n1970,2,177,,100,4.54,1,0,1,100,2,75,8,80,50000,2,,,,,,75,,,,,,8,,,,,,80,,,,,,50000,,,,,\n1970,2,177,,100,4.54,1,0,2,100,2,44,6,60,0,2,2,,,,,75,75,,,,,8,8,,,,,80,80,,,,,50000,50000,,,,\n1970,2,178,,100,4.54,1,0,1,100,2,28,8,80,6050,2,,,,,,28,,,,,,8,,,,,,80,,,,,,6050,,,,,\n1970,2,178,,100,4.54,1,0,2,100,1,5,1,11,9999999,2,2,,,,,28,28,,,,,8,8,,,,,80,80,,,,,6050,6050,,,,\n1970,2,179,,100,4.54,1,0,1,100,1,35,11,111,19150,1,,,2,,,35,,,30,,,11,,,11,,,111,,,110,,,19150,,,4050,,\n1970,2,179,,100,4.54,1,0,2,100,2,30,11,110,4050,1,,,1,,,35,,,35,,,11,,,11,,,111,,,111,,,19150,,,19150,,\n1970,2,179,,100,4.54,1,0,3,100,1,4,1,11,9999999,1,2,1,,,,35,30,35,,,,11,11,11,,,,111,110,111,,,,19150,4050,19150,,,\n1970,2,180,,100,4.54,1,0,1,100,1,28,10,100,2350,1,,,2,,,28,,,26,,,10,,,11,,,100,,,110,,,2350,,,7250,,\n1970,2,180,,100,4.54,1,0,2,100,2,26,11,110,7250,1,,,1,,,28,,,28,,,10,,,10,,,100,,,100,,,2350,,,2350,,\n1970,2,182,,100,4.54,1,0,1,100,1,73,2,25,1050,1,,,,,,73,,,,,,2,,,,,,25,,,,,,1050,,,,,\n1970,2,183,,100,4.54,1,0,1,100,1,27,6,60,15050,1,,,2,,,27,,,27,,,6,,,6,,,60,,,60,,,15050,,,0,,\n1970,2,183,,100,4.54,1,0,2,100,2,27,6,60,0,1,,,1,,,27,,,27,,,6,,,6,,,60,,,60,,,15050,,,15050,,\n1970,2,183,,100,4.54,1,0,3,100,2,4,0,2,9999999,1,2,1,,,,27,27,27,,,,6,6,6,,,,60,60,60,,,,15050,0,15050,,,\n1970,2,183,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,2,1,,,,27,27,27,,,,6,6,6,,,,60,60,60,,,,15050,0,15050,,,\n1970,2,184,,100,4.54,1,0,1,100,2,49,2,25,2550,2,,,,,,49,,,,,,2,,,,,,25,,,,,,2550,,,,,\n1970,2,184,,100,4.54,1,0,2,100,2,48,2,25,0,2,,,,,,49,,,,,,2,,,,,,25,,,,,,2550,,,,,\n1970,2,185,,100,4.54,1,0,1,100,1,29,6,60,8250,1,,,2,,,29,,,22,,,6,,,6,,,60,,,60,,,8250,,,5050,,\n1970,2,185,,100,4.54,1,0,2,100,2,22,6,60,5050,1,,,1,,,29,,,29,,,6,,,6,,,60,,,60,,,8250,,,8250,,\n1970,2,187,,100,4.54,1,0,1,100,1,51,8,80,10050,1,,,2,,,51,,,44,,,8,,,6,,,80,,,60,,,10050,,,1450,,\n1970,2,187,,100,4.54,1,0,2,100,2,44,6,60,1450,1,,,1,,,51,,,51,,,8,,,8,,,80,,,80,,,10050,,,10050,,\n1970,2,187,,100,4.54,1,0,3,100,2,16,4,40,0,1,2,1,,,,51,44,51,,,,8,6,8,,,,80,60,80,,,,10050,1450,10050,,,\n1970,2,187,,100,4.54,1,0,4,100,2,10,1,17,9999999,1,2,1,,,,51,44,51,,,,8,6,8,,,,80,60,80,,,,10050,1450,10050,,,\n1970,2,187,,100,4.54,1,0,5,100,2,9,1,15,9999999,1,2,1,,,,51,44,51,,,,8,6,8,,,,80,60,80,,,,10050,1450,10050,,,\n1970,2,187,,100,4.54,1,0,6,100,2,19,6,60,0,1,2,1,,,,51,44,51,,,,8,6,8,,,,80,60,80,,,,10050,1450,10050,,,\n1970,2,187,,100,4.54,1,0,7,100,2,0,0,1,9999999,1,2,,,,,51,19,,,,,8,6,,,,,80,60,,,,,10050,0,,,,\n1970,2,188,,100,4.54,1,0,1,100,1,55,6,60,11250,1,,,2,,,55,,,50,,,6,,,2,,,60,,,26,,,11250,,,650,,\n1970,2,188,,100,4.54,1,0,2,100,2,50,2,26,650,1,,,1,,,55,,,55,,,6,,,6,,,60,,,60,,,11250,,,11250,,\n1970,2,188,,100,4.54,1,0,3,100,2,10,2,22,9999999,1,2,1,,,,55,50,55,,,,6,2,6,,,,60,26,60,,,,11250,650,11250,,,\n1970,2,189,,100,4.54,1,0,1,100,1,51,6,60,8050,1,,,2,,,51,,,51,,,6,,,4,,,60,,,40,,,8050,,,0,,\n1970,2,189,,100,4.54,1,0,2,100,2,51,4,40,0,1,,,1,,,51,,,51,,,6,,,6,,,60,,,60,,,8050,,,8050,,\n1970,2,189,,100,4.54,1,0,3,100,2,22,8,80,0,1,2,1,,,,51,51,51,,,,6,4,6,,,,60,40,60,,,,8050,0,8050,,,\n1970,2,189,,100,4.54,1,0,4,100,1,17,5,50,1250,1,2,1,,,,51,51,51,,,,6,4,6,,,,60,40,60,,,,8050,0,8050,,,\n1970,2,190,,100,4.54,1,0,1,100,1,43,11,111,21150,1,,,2,,,43,,,39,,,11,,,6,,,111,,,60,,,21150,,,0,,\n1970,2,190,,100,4.54,1,0,2,100,2,39,6,60,0,1,,,1,,,43,,,43,,,11,,,11,,,111,,,111,,,21150,,,21150,,\n1970,2,190,,100,4.54,1,0,3,100,1,15,2,26,0,1,2,1,,,,43,39,43,,,,11,6,11,,,,111,60,111,,,,21150,0,21150,,,\n1970,2,190,,100,4.54,1,0,4,100,1,12,2,22,9999999,1,2,1,,,,43,39,43,,,,11,6,11,,,,111,60,111,,,,21150,0,21150,,,\n1970,2,190,,100,4.54,1,0,5,100,1,9,1,16,9999999,1,2,1,,,,43,39,43,,,,11,6,11,,,,111,60,111,,,,21150,0,21150,,,\n1970,2,191,,100,4.54,1,0,1,100,1,54,3,30,7550,1,,,2,,,54,,,50,,,3,,,2,,,30,,,26,,,7550,,,5050,,\n1970,2,191,,100,4.54,1,0,2,100,2,50,2,26,5050,1,,,1,,,54,,,54,,,3,,,3,,,30,,,30,,,7550,,,7550,,\n1970,2,192,,100,4.54,1,0,1,100,2,62,2,26,3550,2,,,,,,62,,,,,,2,,,,,,26,,,,,,3550,,,,,\n1970,2,193,,100,4.54,1,0,1,100,1,26,4,40,11050,1,,,2,,,26,,,21,,,4,,,6,,,40,,,60,,,11050,,,0,,\n1970,2,193,,100,4.54,1,0,2,100,2,21,6,60,0,1,,,1,,,26,,,26,,,4,,,4,,,40,,,40,,,11050,,,11050,,\n1970,2,193,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,26,21,26,,,,4,6,4,,,,40,60,40,,,,11050,0,11050,,,\n1970,2,194,,100,4.54,1,0,1,100,1,28,1,16,2550,1,,,2,,,28,,,23,,,1,,,0,,,16,,,2,,,2550,,,1550,,\n1970,2,194,,100,4.54,1,0,2,100,2,23,0,2,1550,1,,,1,,,28,,,28,,,1,,,1,,,16,,,16,,,2550,,,2550,,\n1970,2,194,,100,4.54,1,0,3,100,1,4,0,2,9999999,1,2,1,,,,28,23,28,,,,1,0,1,,,,16,2,16,,,,2550,1550,2550,,,\n1970,2,194,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,28,23,28,,,,1,0,1,,,,16,2,16,,,,2550,1550,2550,,,\n1970,2,195,,100,4.54,1,0,1,100,1,48,6,60,9450,1,,,2,,,48,,,45,,,6,,,4,,,60,,,40,,,9450,,,3650,,\n1970,2,195,,100,4.54,1,0,2,100,2,45,4,40,3650,1,,,1,,,48,,,48,,,6,,,6,,,60,,,60,,,9450,,,9450,,\n1970,2,196,,100,4.54,1,0,1,100,1,43,3,30,18050,1,,,2,,,43,,,37,,,3,,,6,,,30,,,60,,,18050,,,2550,,\n1970,2,196,,100,4.54,1,0,2,100,2,37,6,60,2550,1,,,1,,,43,,,43,,,3,,,3,,,30,,,30,,,18050,,,18050,,\n1970,2,196,,100,4.54,1,0,3,100,2,12,2,22,9999999,1,2,1,,,,43,37,43,,,,3,6,3,,,,30,60,30,,,,18050,2550,18050,,,\n1970,2,196,,100,4.54,1,0,4,100,2,8,1,14,9999999,1,2,1,,,,43,37,43,,,,3,6,3,,,,30,60,30,,,,18050,2550,18050,,,\n1970,2,196,,100,4.54,1,0,5,100,2,14,2,26,0,1,2,1,,,,43,37,43,,,,3,6,3,,,,30,60,30,,,,18050,2550,18050,,,\n1970,2,197,,100,4.54,1,0,1,100,1,56,2,23,7650,1,,,,,,56,,,,,,2,,,,,,23,,,,,,7650,,,,,\n1970,2,198,,100,4.54,1,0,1,100,1,62,2,22,12050,1,,,2,,,62,,,59,,,2,,,7,,,22,,,70,,,12050,,,2650,,\n1970,2,198,,100,4.54,1,0,2,100,2,59,7,70,2650,1,,,1,,,62,,,62,,,2,,,2,,,22,,,22,,,12050,,,12050,,\n1970,2,198,,100,4.54,1,0,3,100,2,22,7,70,4350,1,2,1,,,,62,59,62,,,,2,7,2,,,,22,70,22,,,,12050,2650,12050,,,\n1970,2,198,,100,4.54,1,0,4,100,2,20,1,14,1050,1,2,1,,,,62,59,62,,,,2,7,2,,,,22,70,22,,,,12050,2650,12050,,,\n1970,2,199,,100,4.54,1,0,1,100,1,50,2,26,12050,1,,,2,,,50,,,51,,,2,,,2,,,26,,,25,,,12050,,,0,,\n1970,2,199,,100,4.54,1,0,2,100,2,51,2,25,0,1,,,1,,,50,,,50,,,2,,,2,,,26,,,26,,,12050,,,12050,,\n1970,2,199,,100,4.54,1,0,3,100,2,18,5,50,750,1,2,1,,,,50,51,50,,,,2,2,2,,,,26,25,26,,,,12050,0,12050,,,\n1970,2,200,,100,4.54,1,0,1,100,1,63,4,40,12350,1,,,2,,,63,,,59,,,4,,,6,,,40,,,60,,,12350,,,0,,\n1970,2,200,,100,4.54,1,0,2,100,2,59,6,60,0,1,,,1,,,63,,,63,,,4,,,4,,,40,,,40,,,12350,,,12350,,\n1970,2,201,,100,4.54,1,0,1,100,1,67,4,40,6350,1,,,2,,,67,,,61,,,4,,,2,,,40,,,26,,,6350,,,0,,\n1970,2,201,,100,4.54,1,0,2,100,2,61,2,26,0,1,,,1,,,67,,,67,,,4,,,4,,,40,,,40,,,6350,,,6350,,\n1970,2,202,,100,4.54,1,0,1,100,1,51,6,60,10050,1,,,2,,,51,,,45,,,6,,,6,,,60,,,60,,,10050,,,0,,\n1970,2,202,,100,4.54,1,0,2,100,2,45,6,60,0,1,,,1,,,51,,,51,,,6,,,6,,,60,,,60,,,10050,,,10050,,\n1970,2,202,,100,4.54,1,0,3,100,2,17,4,40,150,1,2,1,,,,51,45,51,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,\n1970,2,203,,100,4.54,1,0,1,100,1,42,6,60,13750,1,,,2,,,42,,,43,,,6,,,3,,,60,,,30,,,13750,,,2750,,\n1970,2,203,,100,4.54,1,0,2,100,2,43,3,30,2750,1,,,1,,,42,,,42,,,6,,,6,,,60,,,60,,,13750,,,13750,,\n1970,2,203,,100,4.54,1,0,3,100,1,19,5,50,1550,1,2,1,,,,42,43,42,,,,6,3,6,,,,60,30,60,,,,13750,2750,13750,,,\n1970,2,203,,100,4.54,1,0,4,100,1,15,3,30,50,1,2,1,,,,42,43,42,,,,6,3,6,,,,60,30,60,,,,13750,2750,13750,,,\n1970,2,203,,100,4.54,1,0,5,100,2,11,1,17,9999999,1,2,1,,,,42,43,42,,,,6,3,6,,,,60,30,60,,,,13750,2750,13750,,,\n1970,2,204,,100,4.54,1,0,1,100,1,46,2,23,5250,1,,,2,,,46,,,42,,,2,,,1,,,23,,,17,,,5250,,,0,,\n1970,2,204,,100,4.54,1,0,2,100,2,42,1,17,0,1,,,1,,,46,,,46,,,2,,,2,,,23,,,23,,,5250,,,5250,,\n1970,2,205,,100,4.54,1,0,1,100,2,47,2,26,3550,2,,,,,,47,,,,,,2,,,,,,26,,,,,,3550,,,,,\n1970,2,205,,100,4.54,1,0,2,100,1,8,1,14,9999999,2,2,,,,,47,28,,,,,2,5,,,,,26,50,,,,,3550,6050,,,,\n1970,2,205,,100,4.54,1,0,3,100,2,28,5,50,6050,2,2,,,,,47,47,,,,,2,2,,,,,26,26,,,,,3550,3550,,,,\n1970,2,205,,100,4.54,1,0,4,100,1,1,0,1,9999999,2,2,,,,,47,28,,,,,2,5,,,,,26,50,,,,,3550,6050,,,,\n1970,2,206,,100,4.54,1,0,1,100,1,29,2,23,7250,1,,,2,,,29,,,24,,,2,,,2,,,23,,,26,,,7250,,,0,,\n1970,2,206,,100,4.54,1,0,2,100,2,24,2,26,0,1,,,1,,,29,,,29,,,2,,,2,,,23,,,23,,,7250,,,7250,,\n1970,2,206,,100,4.54,1,0,3,100,1,2,0,1,9999999,1,2,1,,,,29,24,29,,,,2,2,2,,,,23,26,23,,,,7250,0,7250,,,\n1970,2,206,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,2,1,,,,29,24,29,,,,2,2,2,,,,23,26,23,,,,7250,0,7250,,,\n1970,2,207,,100,4.54,1,0,1,100,2,49,6,60,7450,2,,,,,,49,,,,,,6,,,,,,60,,,,,,7450,,,,,\n1970,2,208,,100,4.54,1,0,1,100,1,35,4,40,7250,1,,,2,,,35,,,29,,,4,,,2,,,40,,,22,,,7250,,,0,,\n1970,2,208,,100,4.54,1,0,2,100,2,29,2,22,0,1,,,1,,,35,,,35,,,4,,,4,,,40,,,40,,,7250,,,7250,,\n1970,2,208,,100,4.54,1,0,3,100,2,7,1,16,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,\n1970,2,208,,100,4.54,1,0,4,100,1,8,1,14,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,\n1970,2,208,,100,4.54,1,0,5,100,1,5,0,2,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,\n1970,2,208,,100,4.54,1,0,6,100,1,3,0,2,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,\n1970,2,208,,100,4.54,1,0,7,100,1,0,0,1,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,\n1970,2,209,,100,4.54,1,0,1,100,1,29,4,40,7050,1,,,2,,,29,,,40,,,4,,,0,,,40,,,2,,,7050,,,0,,\n1970,2,209,,100,4.54,1,0,2,100,2,40,0,2,0,1,,,1,,,29,,,29,,,4,,,4,,,40,,,40,,,7050,,,7050,,\n1970,2,209,,100,4.54,1,0,3,100,1,7,1,14,9999999,1,2,1,,,,29,40,29,,,,4,0,4,,,,40,2,40,,,,7050,0,7050,,,\n1970,2,210,,100,4.54,1,0,1,100,1,77,1,15,1750,1,,,2,,,77,,,64,,,1,,,2,,,15,,,26,,,1750,,,1050,,\n1970,2,210,,100,4.54,1,0,2,100,2,64,2,26,1050,1,,,1,,,77,,,77,,,1,,,1,,,15,,,15,,,1750,,,1750,,\n1970,2,210,,100,4.54,1,0,3,100,1,26,6,65,5350,1,,,,,,77,,,,,,1,,,,,,15,,,,,,1750,,,,,\n1970,2,211,,100,4.54,1,0,1,100,1,92,0,2,2250,1,,,2,,,92,,,76,,,0,,,1,,,2,,,16,,,2250,,,1150,,\n1970,2,211,,100,4.54,1,0,2,100,2,76,1,16,1150,1,,,1,,,92,,,92,,,0,,,0,,,2,,,2,,,2250,,,2250,,\n1970,2,212,,100,4.54,1,0,1,100,1,41,6,60,7350,1,,,2,,,41,,,38,,,6,,,4,,,60,,,40,,,7350,,,0,,\n1970,2,212,,100,4.54,1,0,2,100,2,38,4,40,0,1,,,1,,,41,,,41,,,6,,,6,,,60,,,60,,,7350,,,7350,,\n1970,2,212,,100,4.54,1,0,3,100,1,14,2,26,650,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,\n1970,2,212,,100,4.54,1,0,4,100,2,18,6,60,0,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,\n1970,2,212,,100,4.54,1,0,5,100,1,17,4,40,0,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,\n1970,2,212,,100,4.54,1,0,6,100,2,12,2,23,9999999,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,\n1970,2,212,,100,4.54,1,0,7,100,1,10,1,16,9999999,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,\n1970,2,212,,100,4.54,1,0,8,100,2,15,2,25,2150,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,\n1970,2,213,,100,4.54,1,0,1,100,2,55,2,22,1650,2,,,,,,55,,,,,,2,,,,,,22,,,,,,1650,,,,,\n1970,2,213,,100,4.54,1,0,2,100,1,23,6,60,0,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,\n1970,2,213,,100,4.54,1,0,3,100,1,21,6,65,0,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,\n1970,2,213,,100,4.54,1,0,4,100,1,16,4,40,0,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,\n1970,2,213,,100,4.54,1,0,5,100,1,13,2,26,9999999,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,\n1970,2,213,,100,4.54,1,0,6,100,1,12,2,23,9999999,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,\n1970,2,214,,100,4.54,1,0,1,100,1,37,2,25,9550,1,,,2,,,37,,,47,,,2,,,6,,,25,,,60,,,9550,,,10650,,\n1970,2,214,,100,4.54,1,0,2,100,2,47,6,60,10650,1,,,1,,,37,,,37,,,2,,,2,,,25,,,25,,,9550,,,9550,,\n1970,2,214,,100,4.54,1,0,3,100,1,19,6,65,1950,1,2,1,,,,37,47,37,,,,2,6,2,,,,25,60,25,,,,9550,10650,9550,,,\n1970,2,215,,100,4.54,1,0,1,100,1,52,6,60,15350,1,,,2,,,52,,,51,,,6,,,6,,,60,,,60,,,15350,,,0,,\n1970,2,215,,100,4.54,1,0,2,100,2,51,6,60,0,1,,,1,,,52,,,52,,,6,,,6,,,60,,,60,,,15350,,,15350,,\n1970,2,215,,100,4.54,1,0,3,100,1,26,6,60,5750,1,2,1,,,,52,51,52,,,,6,6,6,,,,60,60,60,,,,15350,0,15350,,,\n1970,2,216,,100,4.54,1,0,1,100,1,50,10,100,6450,1,,,2,,,50,,,38,,,10,,,6,,,100,,,60,,,6450,,,2450,,\n1970,2,216,,100,4.54,1,0,2,100,2,38,6,60,2450,1,,,1,,,50,,,50,,,10,,,10,,,100,,,100,,,6450,,,6450,,\n1970,2,216,,100,4.54,1,0,3,100,1,17,4,40,650,1,2,1,,,,50,38,50,,,,10,6,10,,,,100,60,100,,,,6450,2450,6450,,,\n1970,2,216,,100,4.54,1,0,4,100,1,16,4,40,950,1,2,1,,,,50,38,50,,,,10,6,10,,,,100,60,100,,,,6450,2450,6450,,,\n1970,2,216,,100,4.54,1,0,5,100,1,14,2,26,150,1,2,1,,,,50,38,50,,,,10,6,10,,,,100,60,100,,,,6450,2450,6450,,,\n1970,2,217,,100,4.54,1,0,1,100,1,39,6,60,11050,1,,,2,,,39,,,41,,,6,,,6,,,60,,,60,,,11050,,,1250,,\n1970,2,217,,100,4.54,1,0,2,100,2,41,6,60,1250,1,,,1,,,39,,,39,,,6,,,6,,,60,,,60,,,11050,,,11050,,\n1970,2,217,,100,4.54,1,0,3,100,1,17,4,40,0,1,2,1,,,,39,41,39,,,,6,6,6,,,,60,60,60,,,,11050,1250,11050,,,\n1970,2,218,,100,4.54,1,0,1,100,1,29,6,60,17750,1,,,2,,,29,,,29,,,6,,,6,,,60,,,60,,,17750,,,0,,\n1970,2,218,,100,4.54,1,0,2,100,2,29,6,60,0,1,,,1,,,29,,,29,,,6,,,6,,,60,,,60,,,17750,,,17750,,\n1970,2,218,,100,4.54,1,0,3,100,1,6,1,12,9999999,1,2,1,,,,29,29,29,,,,6,6,6,,,,60,60,60,,,,17750,0,17750,,,\n1970,2,218,,100,4.54,1,0,4,100,2,3,0,2,9999999,1,2,1,,,,29,29,29,,,,6,6,6,,,,60,60,60,,,,17750,0,17750,,,\n1970,2,219,,100,4.54,1,0,1,100,1,30,9,90,10050,1,,,2,,,30,,,22,,,9,,,6,,,90,,,60,,,10050,,,6050,,\n1970,2,219,,100,4.54,1,0,2,100,2,22,6,60,6050,1,,,1,,,30,,,30,,,9,,,9,,,90,,,90,,,10050,,,10050,,\n1970,2,220,,100,4.54,1,0,1,100,1,34,10,100,12050,1,,,2,,,34,,,30,,,10,,,10,,,100,,,100,,,12050,,,0,,\n1970,2,220,,100,4.54,1,0,2,100,2,30,10,100,0,1,,,1,,,34,,,34,,,10,,,10,,,100,,,100,,,12050,,,12050,,\n1970,2,221,,100,4.54,1,0,1,100,1,28,10,100,35050,1,,,,,,28,,,,,,10,,,,,,100,,,,,,35050,,,,,\n1970,2,222,,100,4.54,1,0,1,100,1,31,6,60,9850,1,,,2,,,31,,,26,,,6,,,7,,,60,,,70,,,9850,,,0,,\n1970,2,222,,100,4.54,1,0,2,100,2,26,7,70,0,1,,,1,,,31,,,31,,,6,,,6,,,60,,,60,,,9850,,,9850,,\n1970,2,222,,100,4.54,1,0,3,100,1,2,0,1,9999999,1,2,1,,,,31,26,31,,,,6,7,6,,,,60,70,60,,,,9850,0,9850,,,\n1970,2,222,,100,4.54,1,0,4,100,2,0,0,1,9999999,1,2,1,,,,31,26,31,,,,6,7,6,,,,60,70,60,,,,9850,0,9850,,,\n1970,2,223,,100,4.54,1,0,1,100,1,34,6,60,13250,1,,,2,,,34,,,30,,,6,,,6,,,60,,,60,,,13250,,,0,,\n1970,2,223,,100,4.54,1,0,2,100,2,30,6,60,0,1,,,1,,,34,,,34,,,6,,,6,,,60,,,60,,,13250,,,13250,,\n1970,2,223,,100,4.54,1,0,3,100,1,7,1,14,9999999,1,2,1,,,,34,30,34,,,,6,6,6,,,,60,60,60,,,,13250,0,13250,,,\n1970,2,223,,100,4.54,1,0,4,100,1,5,0,2,9999999,1,2,1,,,,34,30,34,,,,6,6,6,,,,60,60,60,,,,13250,0,13250,,,\n1970,2,223,,100,4.54,1,0,5,100,1,2,0,1,9999999,1,2,1,,,,34,30,34,,,,6,6,6,,,,60,60,60,,,,13250,0,13250,,,\n1970,2,224,,100,4.54,1,0,1,100,1,52,8,80,15250,1,,,2,,,52,,,56,,,8,,,7,,,80,,,70,,,15250,,,0,,\n1970,2,224,,100,4.54,1,0,2,100,2,56,7,70,0,1,,,1,,,52,,,52,,,8,,,8,,,80,,,80,,,15250,,,15250,,\n1970,2,224,,100,4.54,1,0,3,100,2,19,6,60,3950,1,2,1,,,,52,56,52,,,,8,7,8,,,,80,70,80,,,,15250,0,15250,,,\n1970,2,225,,100,4.54,1,0,1,100,1,34,7,70,14050,1,,,2,,,34,,,32,,,7,,,8,,,70,,,80,,,14050,,,0,,\n1970,2,225,,100,4.54,1,0,2,100,2,32,8,80,0,1,,,1,,,34,,,34,,,7,,,7,,,70,,,70,,,14050,,,14050,,\n1970,2,225,,100,4.54,1,0,3,100,2,8,1,15,9999999,1,2,1,,,,34,32,34,,,,7,8,7,,,,70,80,70,,,,14050,0,14050,,,\n1970,2,225,,100,4.54,1,0,4,100,1,2,0,1,9999999,1,2,1,,,,34,32,34,,,,7,8,7,,,,70,80,70,,,,14050,0,14050,,,\n1970,2,226,,100,4.54,1,0,1,100,1,39,7,70,15650,1,,,2,,,39,,,38,,,7,,,3,,,70,,,30,,,15650,,,0,,\n1970,2,226,,100,4.54,1,0,2,100,2,38,3,30,0,1,,,1,,,39,,,39,,,7,,,7,,,70,,,70,,,15650,,,15650,,\n1970,2,226,,100,4.54,1,0,3,100,2,11,1,17,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,\n1970,2,226,,100,4.54,1,0,4,100,2,9,1,15,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,\n1970,2,226,,100,4.54,1,0,5,100,2,5,0,2,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,\n1970,2,226,,100,4.54,1,0,6,100,1,2,0,1,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,\n1970,2,226,,100,4.54,1,0,7,100,1,1,0,1,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,\n1970,2,227,,100,4.54,1,0,1,100,2,45,6,60,8050,2,,,,,,45,,,,,,6,,,,,,60,,,,,,8050,,,,,\n1970,2,227,,100,4.54,1,0,2,100,2,17,5,50,1350,2,2,,,,,45,45,,,,,6,6,,,,,60,60,,,,,8050,8050,,,,\n1970,2,227,,100,4.54,1,0,3,100,2,20,8,80,2050,2,2,,,,,45,45,,,,,6,6,,,,,60,60,,,,,8050,8050,,,,\n1970,2,228,,100,4.54,1,0,1,100,1,60,8,80,18150,1,,,2,,,60,,,56,,,8,,,6,,,80,,,60,,,18150,,,250,,\n1970,2,228,,100,4.54,1,0,2,100,2,56,6,60,250,1,,,1,,,60,,,60,,,8,,,8,,,80,,,80,,,18150,,,18150,,\n1970,2,228,,100,4.54,1,0,3,100,2,62,2,26,0,1,,,,,,60,,,,,,8,,,,,,80,,,,,,18150,,,,,\n1970,2,230,,100,4.54,1,0,1,100,2,63,7,70,4250,2,,,,,,63,,,,,,7,,,,,,70,,,,,,4250,,,,,\n1970,2,230,,100,4.54,1,0,2,100,2,57,6,60,4350,2,,,,,,63,,,,,,7,,,,,,70,,,,,,4250,,,,,\n1970,2,231,,100,4.54,1,0,1,100,2,65,5,50,2250,2,,,,,,65,,,,,,5,,,,,,50,,,,,,2250,,,,,\n1970,2,232,,100,4.54,1,0,1,100,1,75,2,26,23850,1,,,2,,,75,,,58,,,2,,,6,,,26,,,60,,,23850,,,1850,,\n1970,2,232,,100,4.54,1,0,2,100,2,58,6,60,1850,1,,,1,,,75,,,75,,,2,,,2,,,26,,,26,,,23850,,,23850,,\n1970,2,233,,100,4.54,4,4,1,100,1,42,2,22,150,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n1970,2,234,,100,4.54,1,0,1,100,1,50,1,17,3850,1,,,,,,50,,,,,,1,,,,,,17,,,,,,3850,,,,,\n1970,2,235,,100,4.54,1,0,1,100,2,22,5,50,1050,2,,,,,,22,,,,,,5,,,,,,50,,,,,,1050,,,,,\n1970,2,235,,100,4.54,1,0,2,100,1,3,0,2,9999999,2,2,,,,,22,22,,,,,5,5,,,,,50,50,,,,,1050,1050,,,,\n1970,2,235,,100,4.54,1,0,3,100,2,20,6,60,3450,2,,,,,,22,,,,,,5,,,,,,50,,,,,,1050,,,,,\n1970,2,235,,100,4.54,1,0,4,100,1,1,0,1,9999999,2,2,,,,,22,20,,,,,5,6,,,,,50,60,,,,,1050,3450,,,,\n1970,2,236,,100,4.54,1,0,1,100,2,79,2,26,2150,2,,,,,,79,,,,,,2,,,,,,26,,,,,,2150,,,,,\n1970,2,237,,100,4.54,1,0,1,100,2,45,3,30,2150,2,,,,,,45,,,,,,3,,,,,,30,,,,,,2150,,,,,\n1970,2,237,,100,4.54,1,0,2,100,2,12,2,22,9999999,2,,,,,,45,,,,,,3,,,,,,30,,,,,,2150,,,,,\n1970,2,238,,100,4.54,1,0,1,100,2,40,6,60,2750,2,,,,,,40,,,,,,6,,,,,,60,,,,,,2750,,,,,\n1970,2,238,,100,4.54,1,0,2,100,1,18,6,60,650,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,\n1970,2,238,,100,4.54,1,0,3,100,1,17,3,30,350,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,\n1970,2,238,,100,4.54,1,0,4,100,1,13,2,25,9999999,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,\n1970,2,238,,100,4.54,1,0,5,100,1,11,1,17,9999999,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,\n1970,2,238,,100,4.54,1,0,6,100,2,10,1,17,9999999,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,\n1970,2,238,,100,4.54,1,0,7,100,2,9,1,15,9999999,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,\n1970,2,240,,100,4.54,1,0,1,100,1,49,4,40,350,1,,,2,,,49,,,54,,,4,,,5,,,40,,,50,,,350,,,750,,\n1970,2,240,,100,4.54,1,0,2,100,2,54,5,50,750,1,,,1,,,49,,,49,,,4,,,4,,,40,,,40,,,350,,,350,,\n1970,2,240,,100,4.54,1,0,3,100,2,2,0,1,9999999,1,2,1,,,,49,54,49,,,,4,5,4,,,,40,50,40,,,,350,750,350,,,\n1970,2,241,,100,4.54,1,0,1,100,1,65,2,26,5550,1,,,2,,,65,,,49,,,2,,,4,,,26,,,40,,,5550,,,5350,,\n1970,2,241,,100,4.54,1,0,2,100,2,49,4,40,5350,1,,,1,,,65,,,65,,,2,,,2,,,26,,,26,,,5550,,,5550,,\n1970,2,241,,100,4.54,1,0,3,100,1,21,6,65,7550,1,2,1,,,,65,49,65,,,,2,4,2,,,,26,40,26,,,,5550,5350,5550,,,\n1970,2,241,,100,4.54,1,0,4,100,2,19,5,50,550,1,2,1,,,,65,49,65,,,,2,4,2,,,,26,40,26,,,,5550,5350,5550,,,\n1970,2,241,,100,4.54,1,0,5,100,2,74,6,60,650,1,,,,,,65,,,,,,2,,,,,,26,,,,,,5550,,,,,\n1970,2,242,,100,4.54,1,0,1,100,1,72,7,70,1250,1,,,2,,,72,,,62,,,7,,,2,,,70,,,23,,,1250,,,350,,\n1970,2,242,,100,4.54,1,0,2,100,2,62,2,23,350,1,,,1,,,72,,,72,,,7,,,7,,,70,,,70,,,1250,,,1250,,\n1970,2,243,,100,4.54,1,0,1,100,1,28,6,60,12050,1,,,2,,,28,,,28,,,6,,,6,,,60,,,60,,,12050,,,1550,,\n1970,2,243,,100,4.54,1,0,2,100,2,28,6,60,1550,1,,,1,,,28,,,28,,,6,,,6,,,60,,,60,,,12050,,,12050,,\n1970,2,243,,100,4.54,1,0,3,100,1,6,1,11,9999999,1,2,1,,,,28,28,28,,,,6,6,6,,,,60,60,60,,,,12050,1550,12050,,,\n1970,2,243,,100,4.54,1,0,4,100,2,4,1,11,9999999,1,2,1,,,,28,28,28,,,,6,6,6,,,,60,60,60,,,,12050,1550,12050,,,\n1970,2,243,,100,4.54,1,0,5,100,2,0,0,1,9999999,1,2,1,,,,28,28,28,,,,6,6,6,,,,60,60,60,,,,12050,1550,12050,,,\n1970,2,244,,100,4.54,1,0,1,100,1,28,6,60,8050,1,,,,,,28,,,,,,6,,,,,,60,,,,,,8050,,,,,\n1970,2,244,,100,4.54,1,0,2,100,2,25,6,60,6550,1,,,,,,28,,,,,,6,,,,,,60,,,,,,8050,,,,,\n1970,2,245,,100,4.54,1,0,1,100,1,44,4,40,10050,1,,,2,,,44,,,43,,,4,,,4,,,40,,,40,,,10050,,,1450,,\n1970,2,245,,100,4.54,1,0,2,100,2,43,4,40,1450,1,,,1,,,44,,,44,,,4,,,4,,,40,,,40,,,10050,,,10050,,\n1970,2,245,,100,4.54,1,0,3,100,2,18,6,65,2750,1,2,1,,,,44,43,44,,,,4,4,4,,,,40,40,40,,,,10050,1450,10050,,,\n1970,2,245,,100,4.54,1,0,4,100,1,11,1,17,9999999,1,2,1,,,,44,43,44,,,,4,4,4,,,,40,40,40,,,,10050,1450,10050,,,\n1970,2,246,,100,4.54,1,0,1,100,1,27,10,100,5050,1,,,,,,27,,,,,,10,,,,,,100,,,,,,5050,,,,,\n1970,2,246,,100,4.54,1,0,2,100,1,23,9,90,7350,1,,,,,,27,,,,,,10,,,,,,100,,,,,,5050,,,,,\n1970,2,247,,100,4.54,1,0,1,100,1,59,4,40,350,1,,,,,,59,,,,,,4,,,,,,40,,,,,,350,,,,,\n1970,2,247,,100,4.54,1,0,2,100,2,52,4,40,7350,1,,,,,,59,,,,,,4,,,,,,40,,,,,,350,,,,,\n1970,2,248,,100,4.54,1,0,1,100,1,57,4,40,10750,1,,,2,,,57,,,38,,,4,,,6,,,40,,,60,,,10750,,,0,,\n1970,2,248,,100,4.54,1,0,2,100,2,38,6,60,0,1,,,1,,,57,,,57,,,4,,,4,,,40,,,40,,,10750,,,10750,,\n1970,2,248,,100,4.54,1,0,3,100,2,16,2,23,0,1,2,1,,,,57,38,57,,,,4,6,4,,,,40,60,40,,,,10750,0,10750,,,\n1970,2,248,,100,4.54,1,0,4,100,2,4,0,2,9999999,1,2,1,,,,57,38,57,,,,4,6,4,,,,40,60,40,,,,10750,0,10750,,,\n1970,2,248,,100,4.54,1,0,5,100,2,8,1,14,9999999,1,2,1,,,,57,38,57,,,,4,6,4,,,,40,60,40,,,,10750,0,10750,,,\n1970,2,250,,100,4.54,1,0,1,100,1,59,6,60,7050,1,,,,,,59,,,,,,6,,,,,,60,,,,,,7050,,,,,\n1970,2,251,,100,4.54,1,0,1,100,1,41,6,60,10050,1,,,2,,,41,,,33,,,6,,,6,,,60,,,60,,,10050,,,0,,\n1970,2,251,,100,4.54,1,0,2,100,2,33,6,60,0,1,,,1,,,41,,,41,,,6,,,6,,,60,,,60,,,10050,,,10050,,\n1970,2,251,,100,4.54,1,0,3,100,1,10,2,22,9999999,1,2,1,,,,41,33,41,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,\n1970,2,251,,100,4.54,1,0,4,100,1,6,1,11,9999999,1,2,1,,,,41,33,41,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,\n1970,2,252,,100,4.54,1,0,1,100,1,51,2,26,10050,1,,,2,,,51,,,46,,,2,,,2,,,26,,,26,,,10050,,,0,,\n1970,2,252,,100,4.54,1,0,2,100,2,46,2,26,0,1,,,1,,,51,,,51,,,2,,,2,,,26,,,26,,,10050,,,10050,,\n1970,2,252,,100,4.54,1,0,3,100,1,15,2,26,0,1,2,1,,,,51,46,51,,,,2,2,2,,,,26,26,26,,,,10050,0,10050,,,\n1970,2,253,,100,4.54,1,0,1,100,2,47,6,60,8050,2,,,,,,47,,,,,,6,,,,,,60,,,,,,8050,,,,,\n1970,2,254,,100,4.54,1,0,1,100,1,33,8,80,6050,1,,,2,,,33,,,41,,,8,,,6,,,80,,,60,,,6050,,,5050,,\n1970,2,254,,100,4.54,1,0,2,100,2,41,6,60,5050,1,,,1,,,33,,,33,,,8,,,8,,,80,,,80,,,6050,,,6050,,\n1970,2,254,,100,4.54,1,0,3,100,1,15,3,30,150,1,2,1,,,,33,41,33,,,,8,6,8,,,,80,60,80,,,,6050,5050,6050,,,\n1970,2,254,,100,4.54,1,0,4,100,1,14,2,26,0,1,2,1,,,,33,41,33,,,,8,6,8,,,,80,60,80,,,,6050,5050,6050,,,\n1970,2,254,,100,4.54,1,0,5,100,2,12,2,23,9999999,1,2,1,,,,33,41,33,,,,8,6,8,,,,80,60,80,,,,6050,5050,6050,,,\n1970,2,254,,100,4.54,1,0,6,100,2,47,8,80,6050,1,,,,,,33,,,,,,8,,,,,,80,,,,,,6050,,,,,\n1970,2,255,,100,4.54,1,0,1,100,1,42,6,60,20050,1,,,2,,,42,,,36,,,6,,,10,,,60,,,100,,,20050,,,13150,,\n1970,2,255,,100,4.54,1,0,2,100,2,36,10,100,13150,1,,,1,,,42,,,42,,,6,,,6,,,60,,,60,,,20050,,,20050,,\n1970,2,256,,100,4.54,1,0,1,100,1,62,11,111,16950,1,,,2,,,62,,,57,,,11,,,10,,,111,,,100,,,16950,,,50,,\n1970,2,256,,100,4.54,1,0,2,100,2,57,10,100,50,1,,,1,,,62,,,62,,,11,,,11,,,111,,,111,,,16950,,,16950,,\n1970,2,256,,100,4.54,1,0,3,100,2,19,7,70,850,1,2,1,,,,62,57,62,,,,11,10,11,,,,111,100,111,,,,16950,50,16950,,,\n1970,2,257,,100,4.54,1,0,1,100,1,38,11,111,23050,1,,,2,,,38,,,37,,,11,,,9,,,111,,,90,,,23050,,,0,,\n1970,2,257,,100,4.54,1,0,2,100,2,37,9,90,0,1,,,1,,,38,,,38,,,11,,,11,,,111,,,111,,,23050,,,23050,,\n1970,2,257,,100,4.54,1,0,3,100,2,14,2,26,0,1,2,1,,,,38,37,38,,,,11,9,11,,,,111,90,111,,,,23050,0,23050,,,\n1970,2,257,,100,4.54,1,0,4,100,2,11,2,22,9999999,1,2,1,,,,38,37,38,,,,11,9,11,,,,111,90,111,,,,23050,0,23050,,,\n1970,2,257,,100,4.54,1,0,5,100,1,9,1,16,9999999,1,2,1,,,,38,37,38,,,,11,9,11,,,,111,90,111,,,,23050,0,23050,,,\n1970,2,258,,100,4.54,1,0,1,100,1,38,6,65,3050,1,,,,,,38,,,,,,6,,,,,,65,,,,,,3050,,,,,\n1970,2,259,,100,4.54,1,0,1,100,2,69,3,30,4150,2,,,,,,69,,,,,,3,,,,,,30,,,,,,4150,,,,,\n1970,2,259,,100,4.54,1,0,2,100,1,26,2,23,1650,2,,,,,,69,,,,,,3,,,,,,30,,,,,,4150,,,,,\n1970,2,260,,100,4.54,3,0,1,100,2,89,6,60,850,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n1970,2,261,,100,4.54,1,0,1,100,1,36,6,60,11450,1,,,2,,,36,,,35,,,6,,,6,,,60,,,60,,,11450,,,250,,\n1970,2,261,,100,4.54,1,0,2,100,2,35,6,60,250,1,,,1,,,36,,,36,,,6,,,6,,,60,,,60,,,11450,,,11450,,\n1970,2,261,,100,4.54,1,0,3,100,2,8,1,14,9999999,1,2,1,,,,36,35,36,,,,6,6,6,,,,60,60,60,,,,11450,250,11450,,,\n1970,2,261,,100,4.54,1,0,4,100,2,5,0,2,9999999,1,2,1,,,,36,35,36,,,,6,6,6,,,,60,60,60,,,,11450,250,11450,,,\n1970,2,262,,100,4.54,1,0,1,100,1,59,6,60,18250,1,,,2,,,59,,,51,,,6,,,6,,,60,,,60,,,18250,,,0,,\n1970,2,262,,100,4.54,1,0,2,100,2,51,6,60,0,1,,,1,,,59,,,59,,,6,,,6,,,60,,,60,,,18250,,,18250,,\n1970,2,263,,100,4.54,1,0,1,100,1,48,6,60,10050,1,,,2,,,48,,,47,,,6,,,3,,,60,,,30,,,10050,,,0,,\n1970,2,263,,100,4.54,1,0,2,100,2,47,3,30,0,1,,,1,,,48,,,48,,,6,,,6,,,60,,,60,,,10050,,,10050,,\n1970,2,263,,100,4.54,1,0,3,100,1,20,6,60,0,1,2,1,,,,48,47,48,,,,6,3,6,,,,60,30,60,,,,10050,0,10050,,,\n1970,2,263,,100,4.54,1,0,4,100,1,14,2,23,0,1,2,1,,,,48,47,48,,,,6,3,6,,,,60,30,60,,,,10050,0,10050,,,\n1970,2,263,,100,4.54,1,0,5,100,2,11,1,16,9999999,1,2,1,,,,48,47,48,,,,6,3,6,,,,60,30,60,,,,10050,0,10050,,,\n1970,2,264,,100,4.54,1,0,1,100,1,34,11,110,10850,1,,,2,,,34,,,34,,,11,,,11,,,110,,,110,,,10850,,,150,,\n1970,2,264,,100,4.54,1,0,2,100,2,34,11,110,150,1,,,1,,,34,,,34,,,11,,,11,,,110,,,110,,,10850,,,10850,,\n1970,2,264,,100,4.54,1,0,3,100,2,5,1,11,9999999,1,2,1,,,,34,34,34,,,,11,11,11,,,,110,110,110,,,,10850,150,10850,,,\n1970,2,264,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,34,34,34,,,,11,11,11,,,,110,110,110,,,,10850,150,10850,,,\n1970,2,265,,100,4.54,1,0,1,100,1,67,8,80,5450,1,,,2,,,67,,,62,,,8,,,7,,,80,,,70,,,5450,,,11550,,\n1970,2,265,,100,4.54,1,0,2,100,2,62,7,70,11550,1,,,1,,,67,,,67,,,8,,,8,,,80,,,80,,,5450,,,5450,,\n1970,2,265,,100,4.54,1,0,3,100,1,21,8,80,0,1,2,1,,,,67,62,67,,,,8,7,8,,,,80,70,80,,,,5450,11550,5450,,,\n1970,2,266,,100,4.54,1,0,1,100,1,77,4,40,15750,1,,,2,,,77,,,74,,,4,,,6,,,40,,,60,,,15750,,,2250,,\n1970,2,266,,100,4.54,1,0,2,100,2,74,6,60,2250,1,,,1,,,77,,,77,,,4,,,4,,,40,,,40,,,15750,,,15750,,\n1970,2,267,,100,4.54,1,0,1,100,1,53,6,60,12050,1,,,2,,,53,,,52,,,6,,,6,,,60,,,60,,,12050,,,3550,,\n1970,2,267,,100,4.54,1,0,2,100,2,52,6,60,3550,1,,,1,,,53,,,53,,,6,,,6,,,60,,,60,,,12050,,,12050,,\n1970,2,267,,100,4.54,1,0,3,100,2,20,8,80,2550,1,2,1,,,,53,52,53,,,,6,6,6,,,,60,60,60,,,,12050,3550,12050,,,\n1970,2,268,,100,4.54,1,0,1,100,1,39,11,111,1650,1,,,2,,,39,,,37,,,11,,,10,,,111,,,100,,,1650,,,3350,,\n1970,2,268,,100,4.54,1,0,2,100,2,37,10,100,3350,1,,,1,,,39,,,39,,,11,,,11,,,111,,,111,,,1650,,,1650,,\n1970,2,268,,100,4.54,1,0,3,100,1,6,1,12,9999999,1,2,1,,,,39,37,39,,,,11,10,11,,,,111,100,111,,,,1650,3350,1650,,,\n1970,2,269,,100,4.54,1,0,1,100,2,50,2,26,2850,2,,,,,,50,,,,,,2,,,,,,26,,,,,,2850,,,,,\n1970,2,269,,100,4.54,1,0,2,100,2,17,3,30,2750,2,2,,,,,50,50,,,,,2,2,,,,,26,26,,,,,2850,2850,,,,\n1970,2,269,,100,4.54,1,0,3,100,1,26,2,25,0,2,,,,,,50,,,,,,2,,,,,,26,,,,,,2850,,,,,\n1970,2,269,,100,4.54,1,0,4,100,1,26,1,17,13250,2,,,,,,50,,,,,,2,,,,,,26,,,,,,2850,,,,,\n1970,2,269,,100,4.54,1,0,5,100,1,26,1,15,13250,2,,,,,,50,,,,,,2,,,,,,26,,,,,,2850,,,,,\n1970,2,270,,100,4.54,1,0,1,100,1,44,7,70,7150,1,,,2,,,44,,,48,,,7,,,8,,,70,,,80,,,7150,,,2650,,\n1970,2,270,,100,4.54,1,0,2,100,2,48,8,80,2650,1,,,1,,,44,,,44,,,7,,,7,,,70,,,70,,,7150,,,7150,,\n1970,2,270,,100,4.54,1,0,3,100,2,12,2,22,9999999,1,2,1,,,,44,48,44,,,,7,8,7,,,,70,80,70,,,,7150,2650,7150,,,\n1970,2,270,,100,4.54,1,0,4,100,2,10,1,17,9999999,1,2,1,,,,44,48,44,,,,7,8,7,,,,70,80,70,,,,7150,2650,7150,,,\n1970,2,271,,100,4.54,1,0,1,100,1,34,11,111,5550,1,,,2,,,34,,,29,,,11,,,11,,,111,,,111,,,5550,,,8250,,\n1970,2,271,,100,4.54,1,0,2,100,2,29,11,111,8250,1,,,1,,,34,,,34,,,11,,,11,,,111,,,111,,,5550,,,5550,,\n1970,2,271,,100,4.54,1,0,3,100,2,2,0,1,9999999,1,2,1,,,,34,29,34,,,,11,11,11,,,,111,111,111,,,,5550,8250,5550,,,\n1970,2,272,,100,4.54,1,0,1,100,1,43,11,110,10850,1,,,2,,,43,,,34,,,11,,,11,,,110,,,111,,,10850,,,11550,,\n1970,2,272,,100,4.54,1,0,2,100,2,34,11,111,11550,1,,,1,,,43,,,43,,,11,,,11,,,110,,,110,,,10850,,,10850,,\n1970,2,272,,100,4.54,1,0,3,100,2,9,1,17,9999999,1,2,1,,,,43,34,43,,,,11,11,11,,,,110,111,110,,,,10850,11550,10850,,,\n1970,2,273,,100,4.54,1,0,1,100,2,28,2,26,4150,2,2,,,,,28,59,,,,,2,2,,,,,26,23,,,,,4150,0,,,,\n1970,2,273,,100,4.54,1,0,2,100,1,7,1,14,9999999,2,2,,,,,28,28,,,,,2,2,,,,,26,26,,,,,4150,4150,,,,\n1970,2,273,,100,4.54,1,0,3,100,2,59,2,23,0,2,,,,,,28,,,,,,2,,,,,,26,,,,,,4150,,,,,\n1970,2,274,,100,4.54,1,0,1,100,1,38,2,26,5050,1,,,2,,,38,,,35,,,2,,,6,,,26,,,60,,,5050,,,4050,,\n1970,2,274,,100,4.54,1,0,2,100,2,35,6,60,4050,1,,,1,,,38,,,38,,,2,,,2,,,26,,,26,,,5050,,,5050,,\n1970,2,274,,100,4.54,1,0,3,100,2,15,3,30,350,1,2,1,,,,38,35,38,,,,2,6,2,,,,26,60,26,,,,5050,4050,5050,,,\n1970,2,274,,100,4.54,1,0,4,100,2,14,2,25,0,1,2,1,,,,38,35,38,,,,2,6,2,,,,26,60,26,,,,5050,4050,5050,,,\n1970,2,274,,100,4.54,1,0,5,100,1,9,1,15,9999999,1,2,1,,,,38,35,38,,,,2,6,2,,,,26,60,26,,,,5050,4050,5050,,,\n1970,2,274,,100,4.54,1,0,6,100,1,8,1,14,9999999,1,2,1,,,,38,35,38,,,,2,6,2,,,,26,60,26,,,,5050,4050,5050,,,\n1970,2,275,,100,4.54,1,0,1,100,1,72,1,14,0,1,,,,,,72,,,,,,1,,,,,,14,,,,,,0,,,,,\n1970,2,276,,100,4.54,1,0,1,100,2,27,3,30,3250,2,,,,,,27,,,,,,3,,,,,,30,,,,,,3250,,,,,\n1970,2,276,,100,4.54,1,0,2,100,2,9,1,16,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,\n1970,2,276,,100,4.54,1,0,3,100,1,8,1,15,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,\n1970,2,276,,100,4.54,1,0,4,100,1,7,1,14,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,\n1970,2,276,,100,4.54,1,0,5,100,1,6,1,11,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,\n1970,2,276,,100,4.54,1,0,6,100,1,4,0,2,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,\n1970,2,276,,100,4.54,1,0,7,100,2,3,0,2,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,\n1970,2,277,,100,4.54,1,0,1,100,1,74,2,26,4450,1,,,2,,,74,,,58,,,2,,,2,,,26,,,26,,,4450,,,4450,,\n1970,2,277,,100,4.54,1,0,2,100,2,58,2,26,4450,1,,,1,,,74,,,74,,,2,,,2,,,26,,,26,,,4450,,,4450,,\n1970,2,278,,100,4.54,1,0,1,100,1,34,3,30,9350,1,,,2,,,34,,,24,,,3,,,6,,,30,,,60,,,9350,,,0,,\n1970,2,278,,100,4.54,1,0,2,100,2,24,6,60,0,1,,,1,,,34,,,34,,,3,,,3,,,30,,,30,,,9350,,,9350,,\n1970,2,279,,100,4.54,1,0,1,100,2,64,6,60,7550,2,,,,,,64,,,,,,6,,,,,,60,,,,,,7550,,,,,\n1970,2,280,,100,4.54,1,0,1,100,1,48,6,60,7550,1,,,,,,48,,,,,,6,,,,,,60,,,,,,7550,,,,,\n1970,2,281,,100,4.54,1,0,1,100,1,47,11,110,9250,1,,,2,,,47,,,40,,,11,,,6,,,110,,,60,,,9250,,,4450,,\n1970,2,281,,100,4.54,1,0,2,100,2,40,6,60,4450,1,,,1,,,47,,,47,,,11,,,11,,,110,,,110,,,9250,,,9250,,\n1970,2,281,,100,4.54,1,0,3,100,1,16,4,40,0,1,2,1,,,,47,40,47,,,,11,6,11,,,,110,60,110,,,,9250,4450,9250,,,\n1970,2,281,,100,4.54,1,0,4,100,1,12,2,22,9999999,1,2,1,,,,47,40,47,,,,11,6,11,,,,110,60,110,,,,9250,4450,9250,,,\n1970,2,282,,100,4.54,1,0,1,100,1,36,2,26,5250,1,,,,,,36,,,,,,2,,,,,,26,,,,,,5250,,,,,\n1970,2,282,,100,4.54,1,0,2,100,1,39,2,26,5050,1,,,,,,36,,,,,,2,,,,,,26,,,,,,5250,,,,,\n1970,2,282,,100,4.54,1,0,3,100,2,77,2,23,650,1,,,,,,36,,,,,,2,,,,,,26,,,,,,5250,,,,,\n1970,2,282,,100,4.54,1,0,4,100,1,47,4,40,0,1,,,,,,36,,,,,,2,,,,,,26,,,,,,5250,,,,,\n1970,2,282,,100,4.54,1,0,5,100,2,14,2,26,2450,1,,1,,,,36,,47,,,,2,,4,,,,26,,40,,,,5250,,0,,,\n1970,2,282,,100,4.54,1,0,6,100,2,10,1,16,9999999,1,,1,,,,36,,47,,,,2,,4,,,,26,,40,,,,5250,,0,,,\n1970,2,282,,100,4.54,1,0,7,100,1,8,1,14,9999999,1,,1,,,,36,,47,,,,2,,4,,,,26,,40,,,,5250,,0,,,\n1970,2,283,,100,4.54,1,0,1,100,2,23,8,80,7850,2,,,,,,23,,,,,,8,,,,,,80,,,,,,7850,,,,,\n1970,2,283,,100,4.54,1,0,2,100,2,18,4,40,0,2,,,,,,23,,,,,,8,,,,,,80,,,,,,7850,,,,,\n1970,2,284,,100,4.54,1,0,1,100,2,31,1,16,6250,2,,,,,,31,,,,,,1,,,,,,16,,,,,,6250,,,,,\n1970,2,284,,100,4.54,1,0,2,100,1,5,0,2,9999999,2,2,,,,,31,31,,,,,1,1,,,,,16,16,,,,,6250,6250,,,,\n1970,2,284,,100,4.54,1,0,3,100,2,4,0,2,9999999,2,2,,,,,31,31,,,,,1,1,,,,,16,16,,,,,6250,6250,,,,\n1970,2,285,,100,4.54,1,0,1,100,1,49,2,26,9050,1,,,2,,,49,,,50,,,2,,,2,,,26,,,26,,,9050,,,1550,,\n1970,2,285,,100,4.54,1,0,2,100,2,50,2,26,1550,1,,,1,,,49,,,49,,,2,,,2,,,26,,,26,,,9050,,,9050,,\n1970,2,286,,100,4.54,1,0,1,100,2,61,2,26,8450,2,,,,,,61,,,,,,2,,,,,,26,,,,,,8450,,,,,\n1970,2,287,,100,4.54,1,0,1,100,2,14,2,26,0,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,\n1970,2,287,,100,4.54,1,0,2,100,1,10,1,17,9999999,2,2,,,,,14,14,,,,,2,2,,,,,26,26,,,,,0,0,,,,\n1970,2,287,,100,4.54,1,0,3,100,1,12,1,17,9999999,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,\n1970,2,287,,100,4.54,1,0,4,100,1,9,1,16,9999999,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,\n1970,2,287,,100,4.54,1,0,5,100,1,5,1,12,9999999,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,\n1970,2,287,,100,4.54,1,0,6,100,1,48,2,25,6450,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,\n1970,2,287,,100,4.54,1,0,7,100,2,2,0,1,9999999,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,\n1970,2,288,,100,4.54,1,0,1,100,1,58,4,40,8050,1,,,2,,,58,,,53,,,4,,,6,,,40,,,60,,,8050,,,7050,,\n1970,2,288,,100,4.54,1,0,2,100,2,53,6,60,7050,1,,,1,,,58,,,58,,,4,,,4,,,40,,,40,,,8050,,,8050,,\n1970,2,289,,100,4.54,1,0,1,100,1,67,3,30,14750,1,,,,,,67,,,,,,3,,,,,,30,,,,,,14750,,,,,\n1970,2,290,,100,4.54,1,0,1,100,1,73,2,25,1950,1,,,2,,,73,,,68,,,2,,,2,,,25,,,25,,,1950,,,650,,\n1970,2,290,,100,4.54,1,0,2,100,2,68,2,25,650,1,2,,1,,,73,89,,73,,,2,2,,2,,,25,25,,25,,,1950,850,,1950,,\n1970,2,290,,100,4.54,1,0,3,100,2,89,2,25,850,1,,,,,,73,,,,,,2,,,,,,25,,,,,,1950,,,,,\n1970,2,291,,100,4.54,1,0,1,100,1,68,2,26,3450,1,,,2,,,68,,,65,,,2,,,4,,,26,,,40,,,3450,,,1750,,\n1970,2,291,,100,4.54,1,0,2,100,2,65,4,40,1750,1,,,1,,,68,,,68,,,2,,,2,,,26,,,26,,,3450,,,3450,,\n1970,2,292,,100,4.54,1,0,1,100,1,29,6,60,12050,1,,,2,,,29,,,28,,,6,,,6,,,60,,,60,,,12050,,,4050,,\n1970,2,292,,100,4.54,1,0,2,100,2,28,6,60,4050,1,,,1,,,29,,,29,,,6,,,6,,,60,,,60,,,12050,,,12050,,\n1970,2,292,,100,4.54,1,0,3,100,2,7,1,12,9999999,1,2,1,,,,29,28,29,,,,6,6,6,,,,60,60,60,,,,12050,4050,12050,,,\n1970,2,292,,100,4.54,1,0,4,100,1,6,1,11,9999999,1,2,1,,,,29,28,29,,,,6,6,6,,,,60,60,60,,,,12050,4050,12050,,,\n1970,2,292,,100,4.54,1,0,5,100,1,3,0,2,9999999,1,2,1,,,,29,28,29,,,,6,6,6,,,,60,60,60,,,,12050,4050,12050,,,\n1970,2,293,,100,4.54,1,0,1,100,2,63,6,60,9950,2,,,,,,63,,,,,,6,,,,,,60,,,,,,9950,,,,,\n1970,2,294,,100,4.54,1,0,1,100,1,42,6,60,30150,1,,,2,,,42,,,39,,,6,,,7,,,60,,,70,,,30150,,,1050,,\n1970,2,294,,100,4.54,1,0,2,100,2,39,7,70,1050,1,,,1,,,42,,,42,,,6,,,6,,,60,,,60,,,30150,,,30150,,\n1970,2,294,,100,4.54,1,0,3,100,2,16,4,40,450,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,\n1970,2,294,,100,4.54,1,0,4,100,1,15,3,30,450,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,\n1970,2,294,,100,4.54,1,0,5,100,1,14,2,26,0,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,\n1970,2,294,,100,4.54,1,0,6,100,1,11,2,22,9999999,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,\n1970,2,294,,100,4.54,1,0,7,100,2,9,1,16,9999999,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,\n1970,2,295,,100,4.54,1,0,1,100,1,70,5,50,450,1,,,2,,,70,,,70,,,5,,,5,,,50,,,50,,,450,,,150,,\n1970,2,295,,100,4.54,1,0,2,100,2,70,5,50,150,1,,,1,,,70,,,70,,,5,,,5,,,50,,,50,,,450,,,450,,\n1970,2,296,,100,4.54,1,0,1,100,1,47,5,50,7050,1,,,2,,,47,,,44,,,5,,,5,,,50,,,50,,,7050,,,4550,,\n1970,2,296,,100,4.54,1,0,2,100,2,44,5,50,4550,1,2,,1,,,47,71,,47,,,5,4,,5,,,50,40,,50,,,7050,650,,7050,,\n1970,2,296,,100,4.54,1,0,3,100,2,71,4,40,650,1,,,,,,47,,,,,,5,,,,,,50,,,,,,7050,,,,,\n1970,2,296,,100,4.54,1,0,4,100,1,17,5,50,550,1,2,1,,,,47,44,47,,,,5,5,5,,,,50,50,50,,,,7050,4550,7050,,,\n1970,2,297,,100,4.54,1,0,1,100,1,33,10,100,17650,1,,,2,,,33,,,33,,,10,,,10,,,100,,,100,,,17650,,,0,,\n1970,2,297,,100,4.54,1,0,2,100,2,33,10,100,0,1,,,1,,,33,,,33,,,10,,,10,,,100,,,100,,,17650,,,17650,,\n1970,2,297,,100,4.54,1,0,3,100,1,8,1,15,9999999,1,2,1,,,,33,33,33,,,,10,10,10,,,,100,100,100,,,,17650,0,17650,,,\n1970,2,297,,100,4.54,1,0,4,100,2,1,0,1,9999999,1,2,1,,,,33,33,33,,,,10,10,10,,,,100,100,100,,,,17650,0,17650,,,\n1970,2,297,,100,4.54,1,0,5,100,2,4,1,11,9999999,1,2,1,,,,33,33,33,,,,10,10,10,,,,100,100,100,,,,17650,0,17650,,,\n1970,2,298,,100,4.54,1,0,1,100,1,49,2,26,14050,1,,,2,,,49,,,46,,,2,,,6,,,26,,,60,,,14050,,,7450,,\n1970,2,298,,100,4.54,1,0,2,100,2,46,6,60,7450,1,,,1,,,49,,,49,,,2,,,2,,,26,,,26,,,14050,,,14050,,\n1970,2,298,,100,4.54,1,0,3,100,2,13,2,23,9999999,1,2,1,,,,49,46,49,,,,2,6,2,,,,26,60,26,,,,14050,7450,14050,,,\n1970,2,299,,100,4.54,1,0,1,100,1,47,6,60,15750,1,,,2,,,47,,,46,,,6,,,6,,,60,,,60,,,15750,,,4850,,\n1970,2,299,,100,4.54,1,0,2,100,2,46,6,60,4850,1,,,1,,,47,,,47,,,6,,,6,,,60,,,60,,,15750,,,15750,,\n1970,2,299,,100,4.54,1,0,3,100,2,18,6,65,2550,1,2,1,,,,47,46,47,,,,6,6,6,,,,60,60,60,,,,15750,4850,15750,,,\n1970,2,300,,100,4.54,1,0,1,100,1,64,5,50,2450,1,,,2,,,64,,,63,,,5,,,2,,,50,,,22,,,2450,,,0,,\n1970,2,300,,100,4.54,1,0,2,100,2,63,2,22,0,1,,,1,,,64,,,64,,,5,,,5,,,50,,,50,,,2450,,,2450,,\n1970,2,301,,100,4.54,1,0,1,100,1,20,7,70,8050,1,,,2,,,20,,,21,,,7,,,7,,,70,,,70,,,8050,,,3050,,\n1970,2,301,,100,4.54,1,0,2,100,2,21,7,70,3050,1,,,1,,,20,,,20,,,7,,,7,,,70,,,70,,,8050,,,8050,,\n1970,2,301,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,20,21,20,,,,7,7,7,,,,70,70,70,,,,8050,3050,8050,,,\n1970,2,302,,100,4.54,1,0,1,100,2,23,2,26,4450,2,,,,,,23,,,,,,2,,,,,,26,,,,,,4450,,,,,\n1970,2,303,,100,4.54,1,0,1,100,1,45,6,60,11750,1,,,2,,,45,,,41,,,6,,,6,,,60,,,60,,,11750,,,0,,\n1970,2,303,,100,4.54,1,0,2,100,2,41,6,60,0,1,2,1,1,,,45,67,71,45,,,6,2,2,6,,,60,26,23,60,,,11750,1450,3350,11750,,\n1970,2,303,,100,4.54,1,0,3,100,2,21,9,90,550,1,2,1,,,,45,41,45,,,,6,6,6,,,,60,60,60,,,,11750,0,11750,,,\n1970,2,303,,100,4.54,1,0,4,100,2,17,5,50,550,1,2,1,,,,45,41,45,,,,6,6,6,,,,60,60,60,,,,11750,0,11750,,,\n1970,2,303,,100,4.54,1,0,5,100,1,15,3,30,0,1,2,1,,,,45,41,45,,,,6,6,6,,,,60,60,60,,,,11750,0,11750,,,\n1970,2,303,,100,4.54,1,0,6,100,1,71,2,23,3350,1,,,2,,,45,,,67,,,6,,,2,,,60,,,26,,,11750,,,1450,,\n1970,2,303,,100,4.54,1,0,7,100,2,67,2,26,1450,1,,,1,,,45,,,71,,,6,,,2,,,60,,,23,,,11750,,,3350,,\n1970,2,304,,100,4.54,1,0,1,100,1,75,2,26,3650,1,,,2,,,75,,,75,,,2,,,2,,,26,,,26,,,3650,,,850,,\n1970,2,304,,100,4.54,1,0,2,100,2,75,2,26,850,1,,,1,,,75,,,75,,,2,,,2,,,26,,,26,,,3650,,,3650,,\n1970,2,305,,100,4.54,1,0,1,100,1,38,6,60,10050,1,,,2,,,38,,,38,,,6,,,6,,,60,,,60,,,10050,,,0,,\n1970,2,305,,100,4.54,1,0,2,100,2,38,6,60,0,1,,,1,,,38,,,38,,,6,,,6,,,60,,,60,,,10050,,,10050,,\n1970,2,305,,100,4.54,1,0,3,100,1,11,2,22,9999999,1,2,1,,,,38,38,38,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,\n1970,2,305,,100,4.54,1,0,4,100,1,9,1,16,9999999,1,2,1,,,,38,38,38,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,\n1970,2,306,,100,4.54,1,0,1,100,1,47,3,30,11050,1,,,2,,,47,,,40,,,3,,,6,,,30,,,60,,,11050,,,0,,\n1970,2,306,,100,4.54,1,0,2,100,2,40,6,60,0,1,,,1,,,47,,,47,,,3,,,3,,,30,,,30,,,11050,,,11050,,\n1970,2,306,,100,4.54,1,0,3,100,1,16,3,30,0,1,2,1,,,,47,40,47,,,,3,6,3,,,,30,60,30,,,,11050,0,11050,,,\n1970,2,306,,100,4.54,1,0,4,100,2,9,1,16,9999999,1,2,1,,,,47,40,47,,,,3,6,3,,,,30,60,30,,,,11050,0,11050,,,\n1970,2,306,,100,4.54,1,0,5,100,2,7,1,14,9999999,1,2,1,,,,47,40,47,,,,3,6,3,,,,30,60,30,,,,11050,0,11050,,,\n1970,2,307,,100,4.54,1,0,1,100,1,35,6,60,150,1,2,,,,,35,73,,,,,6,2,,,,,60,26,,,,,150,1250,,,,\n1970,2,307,,100,4.54,1,0,2,100,2,73,2,26,1250,1,,,,,,35,,,,,,6,,,,,,60,,,,,,150,,,,,\n1970,2,308,,100,4.54,1,0,1,100,1,77,2,22,4050,1,,,,,,77,,,,,,2,,,,,,22,,,,,,4050,,,,,\n1970,2,308,,100,4.54,1,0,2,100,1,50,4,40,650,1,,1,,,,77,,77,,,,2,,2,,,,22,,22,,,,4050,,4050,,,\n1970,2,308,,100,4.54,1,0,3,100,1,41,2,26,6550,1,,1,,,,77,,77,,,,2,,2,,,,22,,22,,,,4050,,4050,,,\n1970,2,309,,100,4.54,1,0,1,100,1,61,5,50,3550,1,,,,,,61,,,,,,5,,,,,,50,,,,,,3550,,,,,\n1970,2,309,,100,4.54,1,0,2,100,1,17,5,50,550,1,,1,,,,61,,61,,,,5,,5,,,,50,,50,,,,3550,,3550,,,\n1970,2,310,,100,4.54,1,0,1,100,1,56,6,60,7050,1,,,2,,,56,,,61,,,6,,,6,,,60,,,60,,,7050,,,0,,\n1970,2,310,,100,4.54,1,0,2,100,2,61,6,60,0,1,,,1,,,56,,,56,,,6,,,6,,,60,,,60,,,7050,,,7050,,\n1970,2,311,,100,4.54,1,0,1,100,1,43,11,110,11850,1,,,2,,,43,,,42,,,11,,,10,,,110,,,100,,,11850,,,250,,\n1970,2,311,,100,4.54,1,0,2,100,2,42,10,100,250,1,,,1,,,43,,,43,,,11,,,11,,,110,,,110,,,11850,,,11850,,\n1970,2,312,,100,4.54,1,0,1,100,2,26,6,60,4050,2,,,,,,26,,,,,,6,,,,,,60,,,,,,4050,,,,,\n1970,2,312,,100,4.54,1,0,2,100,1,9,1,15,9999999,2,2,,,,,26,26,,,,,6,6,,,,,60,60,,,,,4050,4050,,,,\n1970,2,312,,100,4.54,1,0,3,100,1,7,1,12,9999999,2,2,,,,,26,26,,,,,6,6,,,,,60,60,,,,,4050,4050,,,,\n1970,2,313,,100,4.54,1,0,1,100,1,22,10,100,0,1,,,,,,22,,,,,,10,,,,,,100,,,,,,0,,,,,\n1970,2,314,,100,4.54,1,0,1,100,1,27,6,60,7050,1,,,,,,27,,,,,,6,,,,,,60,,,,,,7050,,,,,\n1970,2,315,,100,4.54,3,0,1,100,2,61,8,80,150,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n1970,2,316,,100,4.54,1,0,1,100,1,62,4,40,12050,1,,,2,,,62,,,61,,,4,,,4,,,40,,,40,,,12050,,,1350,,\n1970,2,316,,100,4.54,1,0,2,100,2,61,4,40,1350,1,,,1,,,62,,,62,,,4,,,4,,,40,,,40,,,12050,,,12050,,\n1970,2,317,,100,4.54,1,0,1,100,2,45,8,80,5050,2,,,,,,45,,,,,,8,,,,,,80,,,,,,5050,,,,,\n1970,2,318,,100,4.54,1,0,1,100,2,23,8,80,3050,2,,,,,,23,,,,,,8,,,,,,80,,,,,,3050,,,,,\n1970,2,318,,100,4.54,1,0,2,100,1,2,0,1,9999999,2,2,,,,,23,23,,,,,8,8,,,,,80,80,,,,,3050,3050,,,,\n1970,2,319,,100,4.54,1,0,1,100,1,43,10,100,22050,1,,,2,,,43,,,37,,,10,,,10,,,100,,,100,,,22050,,,150,,\n1970,2,319,,100,4.54,1,0,2,100,2,37,10,100,150,1,,,1,,,43,,,43,,,10,,,10,,,100,,,100,,,22050,,,22050,,\n1970,2,319,,100,4.54,1,0,3,100,2,14,2,26,0,1,2,1,,,,43,37,43,,,,10,10,10,,,,100,100,100,,,,22050,150,22050,,,\n1970,2,319,,100,4.54,1,0,4,100,1,11,1,17,9999999,1,2,1,,,,43,37,43,,,,10,10,10,,,,100,100,100,,,,22050,150,22050,,,\n1970,2,320,,100,4.54,1,0,1,100,2,79,2,26,750,2,,,,,,79,,,,,,2,,,,,,26,,,,,,750,,,,,\n1970,2,321,,100,4.54,1,0,1,100,1,24,6,60,5050,1,,,2,,,24,,,21,,,6,,,6,,,60,,,60,,,5050,,,4050,,\n1970,2,321,,100,4.54,1,0,2,100,2,21,6,60,4050,1,,,1,,,24,,,24,,,6,,,6,,,60,,,60,,,5050,,,5050,,\n1970,2,321,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,24,21,24,,,,6,6,6,,,,60,60,60,,,,5050,4050,5050,,,\n1970,2,322,,100,4.54,1,0,1,100,2,60,6,60,6550,2,,,,,,60,,,,,,6,,,,,,60,,,,,,6550,,,,,\n1970,2,322,,100,4.54,1,0,2,100,1,24,6,60,1350,2,2,,,,,60,60,,,,,6,6,,,,,60,60,,,,,6550,6550,,,,\n1970,2,323,,100,4.54,1,0,1,100,1,77,7,70,10050,1,,,2,,,77,,,71,,,7,,,3,,,70,,,30,,,10050,,,0,,\n1970,2,323,,100,4.54,1,0,2,100,2,71,3,30,0,1,,,1,,,77,,,77,,,7,,,7,,,70,,,70,,,10050,,,10050,,\n1970,2,324,,100,4.54,1,0,1,100,1,66,3,30,8050,1,,,2,,,66,,,60,,,3,,,5,,,30,,,50,,,8050,,,5050,,\n1970,2,324,,100,4.54,1,0,2,100,2,60,5,50,5050,1,,,1,,,66,,,66,,,3,,,3,,,30,,,30,,,8050,,,8050,,\n1970,2,325,,100,4.54,1,0,1,100,1,40,2,25,7050,1,,,2,,,40,,,34,,,2,,,4,,,25,,,40,,,7050,,,6050,,\n1970,2,325,,100,4.54,1,0,2,100,2,34,4,40,6050,1,,,1,,,40,,,40,,,2,,,2,,,25,,,25,,,7050,,,7050,,\n1970,2,325,,100,4.54,1,0,3,100,2,16,3,30,3850,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,\n1970,2,325,,100,4.54,1,0,4,100,1,15,2,25,0,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,\n1970,2,325,,100,4.54,1,0,5,100,2,13,2,25,9999999,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,\n1970,2,325,,100,4.54,1,0,6,100,2,13,2,22,9999999,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,\n1970,2,325,,100,4.54,1,0,7,100,2,0,0,1,9999999,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,\n1970,2,326,,100,4.54,1,0,1,100,1,37,6,60,4850,1,,,2,,,37,,,40,,,6,,,5,,,60,,,50,,,4850,,,0,,\n1970,2,326,,100,4.54,1,0,2,100,2,40,5,50,0,1,,,1,,,37,,,37,,,6,,,6,,,60,,,60,,,4850,,,4850,,\n1970,2,326,,100,4.54,1,0,3,100,2,19,6,60,2650,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,\n1970,2,326,,100,4.54,1,0,4,100,2,17,5,50,0,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,\n1970,2,326,,100,4.54,1,0,5,100,2,15,2,26,0,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,\n1970,2,326,,100,4.54,1,0,6,100,1,14,2,25,0,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,\n1970,2,326,,100,4.54,1,0,7,100,2,13,2,23,9999999,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,\n1970,2,326,,100,4.54,1,0,8,100,2,12,2,22,9999999,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,\n1970,2,326,,100,4.54,1,0,9,100,1,1,0,1,9999999,1,2,,,,,37,19,,,,,6,6,,,,,60,60,,,,,4850,2650,,,,\n1970,2,327,,100,4.54,1,0,1,100,2,53,2,23,550,2,,,,,,53,,,,,,2,,,,,,23,,,,,,550,,,,,\n1970,2,329,,100,4.54,1,0,1,100,1,33,1,17,7050,1,,,2,,,33,,,27,,,1,,,2,,,17,,,23,,,7050,,,2050,,\n1970,2,329,,100,4.54,1,0,2,100,2,27,2,23,2050,1,,,1,,,33,,,33,,,1,,,1,,,17,,,17,,,7050,,,7050,,\n1970,2,329,,100,4.54,1,0,3,100,2,0,0,1,9999999,1,2,1,,,,33,27,33,,,,1,2,1,,,,17,23,17,,,,7050,2050,7050,,,\n1970,2,330,,100,4.54,1,0,1,100,1,29,3,30,9550,1,,,2,,,29,,,24,,,3,,,3,,,30,,,30,,,9550,,,0,,\n1970,2,330,,100,4.54,1,0,2,100,2,24,3,30,0,1,,,1,,,29,,,29,,,3,,,3,,,30,,,30,,,9550,,,9550,,\n1970,2,330,,100,4.54,1,0,3,100,2,3,0,2,9999999,1,2,1,,,,29,24,29,,,,3,3,3,,,,30,30,30,,,,9550,0,9550,,,\n1970,2,330,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,29,24,29,,,,3,3,3,,,,30,30,30,,,,9550,0,9550,,,\n1970,2,330,,100,4.54,1,0,5,100,1,0,0,1,9999999,1,2,1,,,,29,24,29,,,,3,3,3,,,,30,30,30,,,,9550,0,9550,,,\n1970,2,330,,100,4.54,1,0,6,100,1,0,0,1,9999999,1,2,1,,,,29,24,29,,,,3,3,3,,,,30,30,30,,,,9550,0,9550,,,\n1970,2,331,,100,4.54,1,0,1,100,2,45,1,17,0,2,,,,,,45,,,,,,1,,,,,,17,,,,,,0,,,,,\n1970,2,331,,100,4.54,1,0,2,100,1,45,1,16,6150,2,,,,,,45,,,,,,1,,,,,,17,,,,,,0,,,,,\n1970,2,331,,100,4.54,1,0,3,100,2,20,2,22,4150,2,2,,,,,45,45,,,,,1,1,,,,,17,17,,,,,0,0,,,,\n1970,2,332,,100,4.54,1,0,1,100,1,39,6,60,6550,1,,,,,,39,,,,,,6,,,,,,60,,,,,,6550,,,,,\n1970,2,333,,100,4.54,1,0,1,100,1,19,6,65,1750,1,,,,,,19,,,,,,6,,,,,,65,,,,,,1750,,,,,\n1970,2,333,,100,4.54,1,0,2,100,1,18,7,70,2050,1,,,,,,19,,,,,,6,,,,,,65,,,,,,1750,,,,,\n1970,2,333,,100,4.54,1,0,3,100,1,21,9,90,1950,1,,,,,,19,,,,,,6,,,,,,65,,,,,,1750,,,,,\n1970,2,334,,100,4.54,1,0,1,100,1,40,3,30,9450,1,,,2,,,40,,,42,,,3,,,2,,,30,,,26,,,9450,,,0,,\n1970,2,334,,100,4.54,1,0,2,100,2,42,2,26,0,1,,,1,,,40,,,40,,,3,,,3,,,30,,,30,,,9450,,,9450,,\n1970,2,336,,100,4.54,1,0,1,100,2,29,5,50,3750,2,,,,,,29,,,,,,5,,,,,,50,,,,,,3750,,,,,\n1970,2,336,,100,4.54,1,0,2,100,2,5,0,2,9999999,2,2,,,,,29,29,,,,,5,5,,,,,50,50,,,,,3750,3750,,,,\n1970,2,336,,100,4.54,1,0,3,100,2,4,0,2,9999999,2,2,,,,,29,29,,,,,5,5,,,,,50,50,,,,,3750,3750,,,,\n1970,2,337,,100,4.54,1,0,1,100,1,48,11,110,15050,1,,,2,,,48,,,49,,,11,,,2,,,110,,,23,,,15050,,,0,,\n1970,2,337,,100,4.54,1,0,2,100,2,49,2,23,0,1,,,1,,,48,,,48,,,11,,,11,,,110,,,110,,,15050,,,15050,,\n1970,2,337,,100,4.54,1,0,3,100,1,18,6,60,1750,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,\n1970,2,337,,100,4.54,1,0,4,100,1,17,5,50,1050,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,\n1970,2,337,,100,4.54,1,0,5,100,1,14,2,25,50,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,\n1970,2,337,,100,4.54,1,0,6,100,2,11,2,22,9999999,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,\n1970,2,337,,100,4.54,1,0,7,100,1,10,1,17,9999999,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,\n1970,2,338,,100,4.54,1,0,1,100,1,57,9,90,10050,1,,,2,,,57,,,49,,,9,,,6,,,90,,,60,,,10050,,,250,,\n1970,2,338,,100,4.54,1,0,2,100,2,49,6,60,250,1,,,1,,,57,,,57,,,9,,,9,,,90,,,90,,,10050,,,10050,,\n1970,2,338,,100,4.54,1,0,3,100,1,16,4,40,1450,1,2,1,,,,57,49,57,,,,9,6,9,,,,90,60,90,,,,10050,250,10050,,,\n1970,2,338,,100,4.54,1,0,4,100,2,14,2,26,0,1,2,1,,,,57,49,57,,,,9,6,9,,,,90,60,90,,,,10050,250,10050,,,\n1970,2,338,,100,4.54,1,0,5,100,1,11,2,22,9999999,1,2,1,,,,57,49,57,,,,9,6,9,,,,90,60,90,,,,10050,250,10050,,,\n1970,2,338,,100,4.54,1,0,6,100,1,6,1,12,9999999,1,2,1,,,,57,49,57,,,,9,6,9,,,,90,60,90,,,,10050,250,10050,,,\n1970,2,339,,100,4.54,1,0,1,100,1,30,6,60,12050,1,,,2,,,30,,,28,,,6,,,6,,,60,,,60,,,12050,,,0,,\n1970,2,339,,100,4.54,1,0,2,100,2,28,6,60,0,1,,,1,,,30,,,30,,,6,,,6,,,60,,,60,,,12050,,,12050,,\n1970,2,339,,100,4.54,1,0,3,100,1,7,1,14,9999999,1,2,1,,,,30,28,30,,,,6,6,6,,,,60,60,60,,,,12050,0,12050,,,\n1970,2,339,,100,4.54,1,0,4,100,1,5,1,11,9999999,1,2,1,,,,30,28,30,,,,6,6,6,,,,60,60,60,,,,12050,0,12050,,,\n1970,2,339,,100,4.54,1,0,5,100,2,0,0,1,9999999,1,2,1,,,,30,28,30,,,,6,6,6,,,,60,60,60,,,,12050,0,12050,,,\n1970,2,340,,100,4.54,1,0,1,100,1,57,3,30,5250,1,,,2,,,57,,,62,,,3,,,2,,,30,,,26,,,5250,,,0,,\n1970,2,340,,100,4.54,1,0,2,100,2,62,2,26,0,1,,,1,,,57,,,57,,,3,,,3,,,30,,,30,,,5250,,,5250,,\n1970,2,341,,100,4.54,1,0,1,100,1,73,1,16,1750,1,,,2,,,73,,,67,,,1,,,1,,,16,,,16,,,1750,,,750,,\n1970,2,341,,100,4.54,1,0,2,100,2,67,1,16,750,1,,,1,,,73,,,73,,,1,,,1,,,16,,,16,,,1750,,,1750,,\n1970,2,342,,100,4.54,1,0,1,100,1,29,6,60,7050,1,,,2,,,29,,,21,,,6,,,5,,,60,,,50,,,7050,,,3350,,\n1970,2,342,,100,4.54,1,0,2,100,2,21,5,50,3350,1,,,1,,,29,,,29,,,6,,,6,,,60,,,60,,,7050,,,7050,,\n1970,2,342,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,29,21,29,,,,6,5,6,,,,60,50,60,,,,7050,3350,7050,,,\n1970,2,343,,100,4.54,1,0,1,100,1,36,11,111,24250,1,,,2,,,36,,,30,,,11,,,10,,,111,,,100,,,24250,,,7050,,\n1970,2,343,,100,4.54,1,0,2,100,2,30,10,100,7050,1,,,1,,,36,,,36,,,11,,,11,,,111,,,111,,,24250,,,24250,,\n1970,2,343,,100,4.54,1,0,3,100,2,7,1,14,9999999,1,2,1,,,,36,30,36,,,,11,10,11,,,,111,100,111,,,,24250,7050,24250,,,\n1970,2,343,,100,4.54,1,0,4,100,2,3,0,2,9999999,1,2,1,,,,36,30,36,,,,11,10,11,,,,111,100,111,,,,24250,7050,24250,,,\n1970,2,344,,100,4.54,1,0,1,100,1,24,11,111,250,1,,,,,,24,,,,,,11,,,,,,111,,,,,,250,,,,,\n1970,2,344,,100,4.54,1,0,2,100,1,6,1,12,9999999,1,,1,,,,24,,24,,,,11,,11,,,,111,,111,,,,250,,250,,,\n1970,2,344,,100,4.54,1,0,3,100,1,8,1,15,9999999,1,,1,,,,24,,24,,,,11,,11,,,,111,,111,,,,250,,250,,,\n1970,2,345,,100,4.54,1,0,1,100,1,58,8,80,11850,1,,,2,,,58,,,57,,,8,,,6,,,80,,,60,,,11850,,,6450,,\n1970,2,345,,100,4.54,1,0,2,100,2,57,6,60,6450,1,,,1,,,58,,,58,,,8,,,8,,,80,,,80,,,11850,,,11850,,\n1970,2,345,,100,4.54,1,0,3,100,2,28,7,70,6150,1,2,1,,,,58,57,58,,,,8,6,8,,,,80,60,80,,,,11850,6450,11850,,,\n1970,2,346,,100,4.54,1,0,1,100,1,39,6,60,12050,1,,,2,,,39,,,33,,,6,,,6,,,60,,,60,,,12050,,,3050,,\n1970,2,346,,100,4.54,1,0,2,100,2,33,6,60,3050,1,,,1,,,39,,,39,,,6,,,6,,,60,,,60,,,12050,,,12050,,\n1970,2,346,,100,4.54,1,0,3,100,1,16,3,30,0,1,2,1,,,,39,33,39,,,,6,6,6,,,,60,60,60,,,,12050,3050,12050,,,\n1970,2,346,,100,4.54,1,0,4,100,1,14,2,25,0,1,2,1,,,,39,33,39,,,,6,6,6,,,,60,60,60,,,,12050,3050,12050,,,\n1970,2,346,,100,4.54,1,0,5,100,1,4,1,11,9999999,1,2,1,,,,39,33,39,,,,6,6,6,,,,60,60,60,,,,12050,3050,12050,,,\n1970,2,347,,100,4.54,1,0,1,100,1,38,11,111,46550,1,,,2,,,38,,,33,,,11,,,6,,,111,,,60,,,46550,,,0,,\n1970,2,347,,100,4.54,1,0,2,100,2,33,6,60,0,1,,,1,,,38,,,38,,,11,,,11,,,111,,,111,,,46550,,,46550,,\n1970,2,347,,100,4.54,1,0,3,100,1,10,1,16,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,\n1970,2,347,,100,4.54,1,0,4,100,1,9,1,15,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,\n1970,2,347,,100,4.54,1,0,5,100,2,7,1,14,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,\n1970,2,347,,100,4.54,1,0,6,100,1,5,1,11,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,\n1970,2,347,,100,4.54,1,0,7,100,2,4,0,2,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,\n1970,2,347,,100,4.54,1,0,8,100,1,1,0,1,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,\n1970,2,348,,100,4.54,1,0,1,100,2,57,8,80,9050,2,,,,,,57,,,,,,8,,,,,,80,,,,,,9050,,,,,\n"
},
{
"path": "examples/data/nyc-taxi_1k.csv",
"content": "1460000001,2,2017-12-15 00:00:28,2017-12-15 00:15:43,N,1,,,,,2,1.50,11,0.5,0.5,1.25,0,,0.3,13.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000002,2,2017-12-15 00:33:12,2017-12-15 00:51:04,N,1,,,,,3,2.53,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000003,2,2017-12-15 00:56:59,2017-12-15 00:59:51,N,1,,,,,3,0.06,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000004,2,2017-12-15 00:09:19,2017-12-15 00:18:54,N,1,,,,,5,1.47,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000005,2,2017-12-15 00:21:02,2017-12-15 00:25:30,N,1,,,,,5,0.23,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000006,2,2017-12-15 00:35:38,2017-12-15 01:26:19,N,1,,,,,5,18.48,58,0.5,0.5,9.76,5.76,,0.3,76.77,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000007,1,2017-12-15 00:48:29,2017-12-15 01:01:42,N,1,,,,,1,2.60,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000008,2,2017-12-15 00:09:01,2017-12-15 00:11:57,N,1,,,,,1,0.85,4.5,0.5,0.5,0,0,,0.3,5.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000009,2,2017-12-15 00:32:00,2017-12-15 00:35:42,N,1,,,,,1,0.48,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000010,2,2017-12-15 00:42:13,2017-12-15 00:54:48,N,1,,,,,1,2.09,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000011,2,2017-12-15 00:05:46,2017-12-15 00:09:49,N,1,,,,,3,0.60,4.5,0.5,0.5,0.87,0,,0.3,6.67,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000012,2,2017-12-15 00:10:48,2017-12-15 00:20:24,N,1,,,,,3,0.97,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000013,2,2017-12-15 00:26:39,2017-12-15 00:39:18,N,1,,,,,3,1.82,9.5,0.5,0.5,3,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000014,2,2017-12-15 00:40:49,2017-12-15 01:16:48,N,1,,,,,3,8.99,31,0.5,0.5,6.46,0,,0.3,38.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000015,2,2017-12-15 00:24:04,2017-12-15 00:34:56,N,1,,,,,1,5.04,16,0.5,0.5,5.19,0,,0.3,22.49,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000016,1,2017-12-15 00:06:12,2017-12-15 00:35:33,N,1,,,,,1,5.30,21.5,0.5,0.5,1.5,0,,0.3,24.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000017,1,2017-12-15 00:38:38,2017-12-15 00:54:18,N,1,,,,,1,0.90,10.5,0.5,0.5,1,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000018,1,2017-12-15 00:33:57,2017-12-15 00:47:58,N,1,,,,,1,2.10,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000019,1,2017-12-15 00:00:23,2017-12-15 00:13:38,N,1,,,,,1,1.40,9.5,0.5,0.5,1.5,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000020,1,2017-12-15 00:21:30,2017-12-15 00:31:23,N,1,,,,,1,1.50,8,0.5,0.5,2.3,0,,0.3,11.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000021,1,2017-12-15 00:47:17,2017-12-15 01:02:26,N,1,,,,,1,2.80,12,0.5,0.5,1,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000022,1,2017-12-15 00:22:56,2017-12-15 00:29:31,N,1,,,,,1,1.50,7,0.5,0.5,3.5,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000023,1,2017-12-15 00:44:16,2017-12-15 00:50:13,N,1,,,,,1,0.90,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000024,2,2017-12-15 00:21:58,2017-12-15 00:44:00,N,1,,,,,1,13.41,37,0.5,0.5,0,0,,0.3,38.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000025,2,2017-12-15 00:18:22,2017-12-15 00:33:48,N,1,,,,,1,1.71,11,0.5,0.5,0,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000026,2,2017-12-15 00:39:06,2017-12-15 00:52:25,N,1,,,,,1,1.01,9,0.5,0.5,1.54,0,,0.3,11.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000027,2,2017-12-15 01:00:32,2017-12-15 01:20:29,N,1,,,,,1,3.69,16,0.5,0.5,2.6,0,,0.3,19.9,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000028,1,2017-12-15 00:09:13,2017-12-15 00:25:51,N,1,,,,,1,1.90,12,0.5,0.5,1,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000029,1,2017-12-15 00:27:17,2017-12-15 00:35:39,N,1,,,,,1,1.00,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000030,1,2017-12-15 00:37:01,2017-12-15 00:41:47,N,1,,,,,1,0.60,5,0.5,0.5,1.25,0,,0.3,7.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000031,1,2017-12-15 00:44:51,2017-12-15 00:52:31,N,1,,,,,1,1.80,8,0.5,0.5,1,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000032,1,2017-12-15 00:58:04,2017-12-15 01:42:27,N,1,,,,,1,12.10,41,0.5,0.5,2,0,,0.3,44.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000033,1,2017-12-15 00:04:01,2017-12-15 00:20:26,N,1,,,,,1,1.70,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000034,1,2017-12-15 00:21:36,2017-12-15 00:23:09,N,1,,,,,1,0.20,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000035,1,2017-12-15 00:24:05,2017-12-15 00:38:33,N,1,,,,,1,3.10,12.5,0.5,0.5,0,0,,0.3,13.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000036,1,2017-12-15 00:49:14,2017-12-15 01:05:23,N,1,,,,,1,3.90,15,0.5,0.5,3.25,0,,0.3,19.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000037,1,2017-12-15 00:19:03,2017-12-15 00:44:02,N,1,,,,,1,3.50,17,0.5,0.5,2.75,0,,0.3,21.05,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000038,1,2017-12-15 00:47:45,2017-12-15 01:01:16,N,1,,,,,1,4.90,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000039,2,2017-12-15 00:23:06,2017-12-15 00:24:06,N,1,,,,,1,0.36,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000040,2,2017-12-15 00:27:28,2017-12-15 00:30:57,N,1,,,,,1,0.99,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000041,2,2017-12-15 00:18:20,2017-12-15 00:41:29,N,1,,,,,1,3.74,18,0.5,0.5,3.86,0,,0.3,23.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000042,1,2017-12-15 00:08:31,2017-12-15 00:11:07,N,1,,,,,1,0.50,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000043,1,2017-12-15 00:13:03,2017-12-15 00:33:47,N,1,,,,,1,7.20,22.5,0.5,0.5,0,5.76,,0.3,29.56,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000044,2,2017-12-15 00:30:10,2017-12-15 00:45:02,N,1,,,,,1,3.02,12.5,0.5,0.5,0,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000045,2,2017-12-15 00:49:42,2017-12-15 01:13:51,N,1,,,,,1,5.27,20,0.5,0.5,2,0,,0.3,23.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000046,2,2017-12-15 00:05:58,2017-12-15 00:25:43,N,1,,,,,1,4.15,17,0.5,0.5,4.58,0,,0.3,22.88,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000047,2,2017-12-15 00:37:28,2017-12-15 00:42:26,N,1,,,,,1,1.00,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000048,1,2017-12-15 00:15:17,2017-12-15 00:37:01,N,1,,,,,1,4.00,16.5,0.5,0.5,3.55,0,,0.3,21.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000049,1,2017-12-15 00:42:53,2017-12-15 00:59:39,N,1,,,,,1,3.30,14.5,0.5,0.5,3.15,0,,0.3,18.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000050,2,2017-12-15 00:33:52,2017-12-15 00:40:49,N,1,,,,,1,0.40,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000051,2,2017-12-15 00:46:35,2017-12-15 00:56:49,N,1,,,,,1,1.26,8.5,0.5,0.5,2,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000052,1,2017-12-15 00:06:04,2017-12-15 00:31:04,N,1,,,,,1,4.50,20,0.5,0.5,4.26,0,,0.3,25.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000053,2,2017-12-15 00:30:11,2017-12-15 00:34:09,N,1,,,,,3,1.14,5.5,0.5,0.5,2.04,0,,0.3,8.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000054,2,2017-12-15 00:45:10,2017-12-15 01:11:33,N,1,,,,,5,10.50,31.5,0.5,0.5,0,5.76,,0.3,38.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000055,2,2017-12-15 00:31:27,2017-12-15 00:53:23,N,1,,,,,2,4.40,18,0.5,0.5,4.82,0,,0.3,24.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000056,2,2017-12-15 00:23:15,2017-12-15 00:33:59,N,1,,,,,1,1.26,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000057,1,2017-12-15 00:18:49,2017-12-15 00:25:53,N,1,,,,,2,1.10,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000058,1,2017-12-15 00:28:53,2017-12-15 00:35:26,N,1,,,,,1,0.80,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000059,1,2017-12-15 00:36:04,2017-12-15 00:58:54,N,1,,,,,1,4.00,17,0.5,0.5,3.65,0,,0.3,21.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000060,1,2017-12-15 00:52:09,2017-12-15 00:59:47,N,1,,,,,1,1.00,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000061,2,2017-12-15 00:07:57,2017-12-15 00:25:09,N,1,,,,,1,2.79,13,0.5,0.5,0,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000062,2,2017-12-15 00:26:09,2017-12-15 00:29:52,N,1,,,,,1,0.68,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000063,2,2017-12-15 00:31:59,2017-12-15 00:40:18,N,1,,,,,1,1.31,7.5,0.5,0.5,1.25,0,,0.3,10.05,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000064,2,2017-12-15 00:49:06,2017-12-15 01:07:13,N,1,,,,,1,4.21,15.5,0.5,0.5,3.36,0,,0.3,20.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000065,2,2017-12-15 00:56:03,2017-12-15 01:37:49,N,1,,,,,1,22.40,62.5,0.5,0.5,17.39,5.76,,0.3,86.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000066,1,2017-12-15 00:01:19,2017-12-15 00:12:58,N,1,,,,,1,1.80,10,0.5,0.5,1,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000067,1,2017-12-15 00:18:24,2017-12-15 01:04:16,N,5,,,,,1,5.00,0,0,0,0,10.5,,0.3,10.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000068,2,2017-12-15 00:08:27,2017-12-15 00:38:37,N,1,,,,,1,4.88,21.5,0.5,0.5,2.28,0,,0.3,25.08,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000069,2,2017-12-15 00:42:27,2017-12-15 00:53:50,N,1,,,,,1,5.28,17,0.5,0.5,4.81,5.76,,0.3,28.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000070,2,2017-12-15 00:05:56,2017-12-15 00:22:42,N,1,,,,,2,2.29,11.5,0.5,0.5,2.56,0,,0.3,15.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000071,2,2017-12-15 00:27:09,2017-12-15 00:31:47,N,1,,,,,2,0.77,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000072,2,2017-12-15 00:42:03,2017-12-15 01:00:47,N,1,,,,,2,3.28,14.5,0.5,0.5,3.95,0,,0.3,19.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000073,1,2017-12-15 00:45:21,2017-12-15 00:50:56,N,1,,,,,1,1.50,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000074,2,2017-12-15 00:08:09,2017-12-15 00:21:11,N,1,,,,,5,1.97,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000075,2,2017-12-15 00:21:48,2017-12-15 00:57:13,N,1,,,,,5,5.40,24,0.5,0.5,0,0,,0.3,25.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000076,1,2017-12-15 00:09:11,2017-12-15 00:35:18,N,1,,,,,4,5.20,21.5,0.5,0.5,0,0,,0.3,22.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000077,1,2017-12-15 00:42:08,2017-12-15 00:42:09,N,1,,,,,1,5.30,2.5,0.5,0.5,0,0,,0.3,3.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000078,1,2017-12-15 00:45:28,2017-12-15 01:05:27,N,1,,,,,1,6.90,22.5,0.5,0.5,3,5.76,,0.3,32.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000079,2,2017-12-15 00:03:48,2017-12-15 00:38:38,N,1,,,,,6,7.86,29,0.5,0.5,9.09,0,,0.3,39.39,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000080,1,2017-12-15 00:25:04,2017-12-15 00:28:18,N,1,,,,,2,0.60,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000081,1,2017-12-15 00:34:23,2017-12-15 00:59:40,N,1,,,,,1,3.60,17.5,0.5,0.5,3,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000082,2,2017-12-15 00:53:37,2017-12-15 01:11:06,N,1,,,,,1,2.77,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000083,2,2017-12-15 00:09:00,2017-12-15 00:42:14,N,1,,,,,1,4.94,24.5,0.5,0.5,5.16,0,,0.3,30.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000084,1,2017-12-15 00:00:49,2017-12-15 00:07:51,N,1,,,,,1,0.50,6,0.5,0.5,1,0,,0.3,8.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000085,1,2017-12-15 00:22:47,2017-12-15 00:30:21,N,1,,,,,4,0.60,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000086,1,2017-12-15 00:32:04,2017-12-15 00:39:38,N,1,,,,,1,0.90,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000087,1,2017-12-15 00:50:23,2017-12-15 01:05:29,N,1,,,,,1,3.00,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000088,2,2017-12-15 00:08:01,2017-12-15 00:25:35,N,1,,,,,1,2.80,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000089,2,2017-12-15 00:27:27,2017-12-15 00:55:33,N,1,,,,,1,4.05,19.5,0.5,0.5,5.2,0,,0.3,26,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000090,2,2017-12-15 00:11:22,2017-12-15 00:37:10,N,1,,,,,2,5.04,19.5,0.5,0.5,0,0,,0.3,20.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000091,2,2017-12-15 01:00:50,2017-12-15 01:29:18,N,1,,,,,2,7.11,24.5,0.5,0.5,0,0,,0.3,25.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000092,1,2017-12-15 00:26:01,2017-12-15 00:48:49,N,1,,,,,1,14.60,40,0.5,0.5,8.25,0,,0.3,49.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000093,1,2017-12-15 00:10:53,2017-12-15 00:48:57,N,1,,,,,1,14.30,43.5,0.5,0.5,8.95,0,,0.3,53.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000094,2,2017-12-15 00:31:37,2017-12-15 00:37:18,N,1,,,,,1,0.48,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000095,2,2017-12-15 00:41:42,2017-12-15 00:51:54,N,1,,,,,1,1.66,9,0.5,0.5,1,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000096,2,2017-12-15 00:53:10,2017-12-15 01:12:40,N,1,,,,,1,3.08,15.5,0.5,0.5,1,0,,0.3,17.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000097,2,2017-12-15 00:17:32,2017-12-15 00:30:33,N,1,,,,,5,1.14,9,0.5,0.5,1,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000098,2,2017-12-15 00:57:51,2017-12-15 01:11:23,N,1,,,,,5,2.01,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000099,1,2017-12-15 00:43:52,2017-12-15 00:52:43,N,1,,,,,4,1.60,8,0.5,0.5,0,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000100,1,2017-12-15 00:02:26,2017-12-15 00:13:59,N,1,,,,,1,1.10,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000101,1,2017-12-15 00:14:19,2017-12-15 00:56:54,N,5,,,,,1,18.50,100,0,0,15,0,,0.3,115.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000102,1,2017-12-15 00:40:45,2017-12-15 00:54:30,N,1,,,,,1,2.10,12,0.5,0.5,2.2,0,,0.3,15.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000103,2,2017-12-15 00:37:36,2017-12-15 00:53:05,N,1,,,,,1,3.60,14,0.5,0.5,2,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000104,2,2017-12-15 00:53:59,2017-12-15 00:55:30,N,1,,,,,1,0.22,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000105,1,2017-12-15 00:36:23,2017-12-15 00:51:06,N,1,,,,,1,2.30,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000106,2,2017-12-15 00:19:27,2017-12-15 00:31:53,N,1,,,,,6,1.49,9.5,0.5,0.5,2.16,0,,0.3,12.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000107,2,2017-12-15 00:39:17,2017-12-15 00:44:01,N,1,,,,,6,0.79,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000108,1,2017-12-15 00:09:38,2017-12-15 00:47:23,N,1,,,,,2,17.90,51,0.5,0.5,1,17.28,,0.3,70.58,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000109,1,2017-12-15 00:49:21,2017-12-15 01:11:02,N,1,,,,,1,14.00,38,0.5,0.5,1,0,,0.3,40.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000110,2,2017-12-15 00:32:24,2017-12-15 00:38:56,N,1,,,,,1,0.80,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000111,2,2017-12-15 00:40:45,2017-12-15 00:55:44,N,1,,,,,1,1.54,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000112,1,2017-12-15 00:34:39,2017-12-15 00:41:03,N,1,,,,,1,1.00,6.5,0.5,0.5,1.95,0,,0.3,9.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000113,1,2017-12-15 00:43:04,2017-12-15 00:58:19,N,1,,,,,1,2.80,12.5,0.5,0.5,2.75,0,,0.3,16.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000114,2,2017-12-15 00:32:50,2017-12-15 00:45:30,N,1,,,,,1,1.56,9.5,0.5,0.5,2.16,0,,0.3,12.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000115,2,2017-12-15 00:50:04,2017-12-15 00:59:07,N,1,,,,,1,1.70,8.5,0.5,0.5,2.45,0,,0.3,12.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000116,1,2017-12-15 00:31:07,2017-12-15 00:35:35,N,1,,,,,1,0.60,5,0.5,0.5,1,0,,0.3,7.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000117,1,2017-12-15 00:44:08,2017-12-15 01:12:08,N,1,,,,,1,7.30,26.5,0.5,0.5,0,0,,0.3,27.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000118,2,2017-12-15 00:40:24,2017-12-15 00:45:14,N,1,,,,,1,1.47,6.5,0.5,0.5,0,0,,0.3,7.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000119,2,2017-12-15 00:46:19,2017-12-15 01:02:19,N,1,,,,,2,1.87,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000120,2,2017-12-15 00:02:07,2017-12-15 00:04:51,N,1,,,,,1,0.36,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000121,2,2017-12-15 00:22:58,2017-12-15 00:34:06,N,1,,,,,2,0.73,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000122,2,2017-12-15 00:41:31,2017-12-15 00:48:10,N,1,,,,,1,0.82,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000123,2,2017-12-15 00:52:19,2017-12-15 01:01:01,N,1,,,,,1,1.54,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000124,1,2017-12-15 00:31:59,2017-12-15 00:44:42,N,1,,,,,0,2.20,10.5,0.5,0.5,2,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000125,2,2017-12-15 00:38:43,2017-12-15 00:44:54,N,1,,,,,1,1.44,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000126,2,2017-12-15 00:51:10,2017-12-15 01:05:47,N,1,,,,,5,2.75,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000127,2,2017-12-15 00:09:57,2017-12-15 00:25:53,N,1,,,,,1,2.61,12.5,0.5,0.5,2.76,0,,0.3,16.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000128,2,2017-12-15 00:27:34,2017-12-15 00:44:22,N,1,,,,,1,7.11,22.5,0.5,0.5,3.8,0,,0.3,27.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000129,2,2017-12-15 00:29:13,2017-12-15 00:53:56,N,1,,,,,2,9.73,30,0.5,0.5,0,0,,0.3,31.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000130,2,2017-12-15 00:28:32,2017-12-15 00:33:57,N,1,,,,,6,0.59,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000131,2,2017-12-15 00:37:51,2017-12-15 01:55:38,N,1,,,,,6,15.34,58,0.5,0.5,0,0,,0.3,59.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000132,2,2017-12-15 00:04:24,2017-12-15 00:13:56,N,1,,,,,3,1.46,8,0.5,0.5,2,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000133,2,2017-12-15 00:38:48,2017-12-15 01:01:48,N,1,,,,,2,2.72,15.5,0.5,0.5,4.2,0,,0.3,21,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000134,1,2017-12-15 00:33:15,2017-12-15 00:47:55,N,1,,,,,1,9.80,27,0.5,0.5,8,0,,0.3,36.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000135,2,2017-12-15 00:41:39,2017-12-15 01:14:47,N,1,,,,,1,8.90,30.5,0.5,0.5,0,0,,0.3,31.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000136,1,2017-12-15 00:50:41,2017-12-15 00:59:29,N,1,,,,,1,1.30,8,0.5,0.5,1.5,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000137,2,2017-12-15 00:49:15,2017-12-15 00:58:07,N,1,,,,,4,1.87,9,0.5,0.5,2.06,0,,0.3,12.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000138,2,2017-12-14 23:51:20,2017-12-15 00:50:09,N,1,,,,,1,10.08,43.5,0.5,0.5,8.96,0,,0.3,53.76,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000139,2,2017-12-15 00:05:31,2017-12-15 00:19:55,N,1,,,,,5,2.97,12,0.5,0.5,2.66,0,,0.3,15.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000140,2,2017-12-15 00:29:06,2017-12-15 01:09:26,N,1,,,,,3,13.26,41.5,0.5,0.5,0,0,,0.3,42.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000141,2,2017-12-15 00:26:34,2017-12-15 00:51:30,N,1,,,,,1,5.36,21,0.5,0.5,2,0,,0.3,24.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000142,2,2017-12-15 00:04:47,2017-12-15 00:23:09,N,1,,,,,1,2.68,13.5,0.5,0.5,4.44,0,,0.3,19.24,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000143,2,2017-12-15 00:31:16,2017-12-15 00:44:45,N,1,,,,,1,2.40,11,0.5,0.5,1,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000144,2,2017-12-15 00:49:37,2017-12-15 01:21:04,N,1,,,,,1,6.83,25.5,0.5,0.5,5.36,0,,0.3,32.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000145,2,2017-12-15 00:10:01,2017-12-15 00:16:49,N,1,,,,,1,0.91,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000146,2,2017-12-15 00:19:03,2017-12-15 00:30:11,N,1,,,,,1,1.81,9.5,0.5,0.5,2,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000147,2,2017-12-15 00:31:07,2017-12-15 00:34:53,N,1,,,,,1,0.92,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000148,2,2017-12-15 00:43:48,2017-12-15 01:04:53,N,1,,,,,1,4.63,18,0.5,0.5,3.86,0,,0.3,23.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000149,1,2017-12-15 00:22:11,2017-12-15 00:32:11,N,1,,,,,1,2.20,10,0.5,0.5,2.25,0,,0.3,13.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000150,1,2017-12-15 00:48:02,2017-12-15 01:01:45,N,1,,,,,1,1.30,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000151,2,2017-12-15 00:05:43,2017-12-15 00:12:57,N,1,,,,,1,2.05,9,0.5,0.5,2.06,0,,0.3,14.31,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000152,2,2017-12-15 00:32:48,2017-12-15 00:55:01,N,1,,,,,1,9.23,28,0.5,0.5,6,5.76,,0.3,41.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000153,2,2017-12-15 01:00:10,2017-12-15 01:16:04,N,1,,,,,1,4.37,15.5,0.5,0.5,3.36,0,,0.3,20.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000154,2,2017-12-15 00:08:13,2017-12-15 00:28:37,N,1,,,,,1,3.22,15,0.5,0.5,0,0,,0.3,16.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000155,2,2017-12-15 00:31:00,2017-12-15 00:41:23,N,1,,,,,1,1.60,8.5,0.5,0.5,0,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000156,2,2017-12-15 00:42:50,2017-12-15 00:53:34,N,1,,,,,1,1.37,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000157,2,2017-12-15 00:55:45,2017-12-15 00:59:31,N,1,,,,,1,1.08,5.5,0.5,0.5,1.7,0,,0.3,8.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000158,2,2017-12-15 00:09:20,2017-12-15 00:23:24,N,1,,,,,1,2.62,11.5,0.5,0.5,3.2,0,,0.3,16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000159,2,2017-12-15 00:24:46,2017-12-15 00:36:33,N,1,,,,,1,1.35,9.5,0.5,0.5,0,0,,0.3,10.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000160,2,2017-12-15 00:40:16,2017-12-15 00:57:48,N,1,,,,,1,3.03,13.5,0.5,0.5,1,0,,0.3,15.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000161,1,2017-12-15 00:00:35,2017-12-15 00:12:21,N,1,,,,,1,2.00,9.5,0.5,0.5,3.2,0,,0.3,14,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000162,1,2017-12-15 00:13:48,2017-12-15 00:20:03,N,1,,,,,1,1.00,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000163,1,2017-12-15 00:20:47,2017-12-15 00:31:56,N,1,,,,,1,1.70,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000164,1,2017-12-15 00:33:29,2017-12-15 01:02:24,N,1,,,,,1,9.00,29,0.5,0.5,7.55,0,,0.3,37.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000165,1,2017-12-15 00:09:42,2017-12-15 00:18:12,N,1,,,,,2,2.40,9,0.5,0.5,0,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000166,1,2017-12-15 00:57:43,2017-12-15 01:16:05,N,1,,,,,1,3.30,13.5,0.5,0.5,2.95,0,,0.3,17.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000167,2,2017-12-15 00:09:55,2017-12-15 00:19:30,N,1,,,,,1,1.45,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000168,2,2017-12-15 00:20:11,2017-12-15 00:33:01,N,1,,,,,1,2.18,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000169,2,2017-12-15 00:35:09,2017-12-15 01:09:59,N,1,,,,,1,17.48,51,0.5,0.5,0,5.76,,0.3,58.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000170,2,2017-12-15 00:26:34,2017-12-15 00:41:45,N,1,,,,,1,1.81,11,0.5,0.5,3.69,0,,0.3,15.99,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000171,2,2017-12-15 00:46:38,2017-12-15 01:08:50,N,1,,,,,1,3.31,16,0.5,0.5,3.46,0,,0.3,20.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000172,2,2017-12-15 00:19:37,2017-12-15 00:35:33,N,1,,,,,2,1.61,10.5,0.5,0.5,5,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000173,2,2017-12-15 00:38:17,2017-12-15 00:45:48,N,1,,,,,2,1.01,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000174,2,2017-12-15 00:58:49,2017-12-15 01:35:08,N,1,,,,,2,5.57,26.5,0.5,0.5,5.56,0,,0.3,33.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000175,1,2017-12-15 00:17:10,2017-12-15 00:52:01,N,1,,,,,1,4.00,23,0.5,0.5,6,5.76,,0.3,36.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000176,2,2017-12-15 00:51:45,2017-12-15 01:16:36,N,2,,,,,2,16.55,52,0,0.5,10.56,0,,0.3,63.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000177,2,2017-12-15 00:53:53,2017-12-15 00:59:08,N,1,,,,,3,1.09,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000178,2,2017-12-15 00:17:56,2017-12-15 00:41:33,N,1,,,,,1,5.12,19.5,0.5,0.5,4.16,0,,0.3,24.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000179,2,2017-12-15 00:45:41,2017-12-15 00:58:18,N,1,,,,,1,2.83,11.5,0.5,0.5,3.2,0,,0.3,16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000180,2,2017-12-15 00:01:12,2017-12-15 00:11:38,N,1,,,,,1,1.65,8.5,0.5,0.5,2.45,0,,0.3,12.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000181,2,2017-12-15 00:12:58,2017-12-15 00:19:54,N,1,,,,,1,0.89,6.5,0.5,0.5,1.95,0,,0.3,9.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000182,2,2017-12-15 00:24:50,2017-12-15 00:26:01,N,5,,,,,1,0.00,100,0,0.5,8.2,0,,0.3,109,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000183,1,2017-12-15 00:32:04,2017-12-15 01:01:36,N,2,,,,,3,20.80,52,0,0.5,0,5.76,,0.3,58.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000184,2,2017-12-15 00:18:06,2017-12-15 01:07:36,N,1,,,,,1,25.24,71,0.5,0.5,15.61,5.76,,0.3,93.67,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000185,2,2017-12-15 00:10:06,2017-12-15 00:20:55,N,1,,,,,1,1.84,9.5,0.5,0.5,2.5,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000186,2,2017-12-15 00:31:27,2017-12-15 00:45:44,N,1,,,,,1,1.66,10.5,0.5,0.5,1.2,0,,0.3,13,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000187,2,2017-12-15 00:53:13,2017-12-15 01:13:27,N,1,,,,,1,2.40,14,0.5,0.5,0,0,,0.3,15.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000188,2,2017-12-15 00:03:43,2017-12-15 00:11:39,N,1,,,,,2,1.58,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000189,2,2017-12-15 00:18:02,2017-12-15 00:43:48,N,1,,,,,2,4.61,19,0.5,0.5,0,0,,0.3,20.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000190,2,2017-12-15 00:48:20,2017-12-15 01:09:19,N,1,,,,,2,2.67,15,0.5,0.5,1,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000191,1,2017-12-15 00:05:45,2017-12-15 00:26:31,N,1,,,,,2,3.50,17.5,0.5,0.5,2,0,,0.3,20.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000192,1,2017-12-15 00:35:30,2017-12-15 00:47:19,N,1,,,,,1,2.50,10.5,0.5,0.5,2,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000193,1,2017-12-15 00:55:01,2017-12-15 01:06:13,N,1,,,,,1,0.90,8.5,0.5,0.5,2.45,0,,0.3,12.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000194,2,2017-12-15 00:23:31,2017-12-15 00:30:12,N,1,,,,,1,0.98,6.5,0.5,0.5,1,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000195,2,2017-12-15 00:33:17,2017-12-15 00:55:09,N,1,,,,,1,3.20,15.5,0.5,0.5,3.36,0,,0.3,20.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000196,2,2017-12-15 00:56:39,2017-12-15 01:14:39,N,1,,,,,1,4.65,18,0.5,0.5,3.86,0,,0.3,23.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000197,2,2017-12-15 00:12:57,2017-12-15 00:18:57,N,1,,,,,1,1.03,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000198,2,2017-12-15 00:20:41,2017-12-15 00:46:44,N,1,,,,,1,5.33,21,0.5,0.5,1,0,,0.3,23.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000199,2,2017-12-14 23:56:15,2017-12-15 00:20:06,N,1,,,,,1,10.45,30.5,0.5,0.5,7.51,5.76,,0.3,45.07,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000200,2,2017-12-15 00:30:30,2017-12-15 01:22:30,N,1,,,,,1,13.51,48,0.5,0.5,0,0,,0.3,49.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000201,2,2017-12-15 00:03:15,2017-12-15 00:21:04,N,1,,,,,1,2.38,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000202,2,2017-12-15 00:23:15,2017-12-15 00:27:40,N,1,,,,,1,0.47,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000203,2,2017-12-15 00:31:32,2017-12-15 00:49:33,N,1,,,,,1,2.91,14,0.5,0.5,1,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000204,2,2017-12-15 00:08:42,2017-12-15 00:12:21,N,1,,,,,1,0.39,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000205,2,2017-12-15 00:13:31,2017-12-15 00:24:20,N,1,,,,,1,2.49,10.5,0.5,0.5,1.2,0,,0.3,13,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000206,2,2017-12-15 00:29:19,2017-12-15 00:58:28,N,1,,,,,1,3.72,20,0.5,0.5,4.26,0,,0.3,25.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000207,2,2017-12-15 00:28:14,2017-12-15 00:34:25,N,1,,,,,1,1.55,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000208,2,2017-12-15 00:36:22,2017-12-15 00:37:55,N,1,,,,,1,0.45,3.5,0.5,0.5,0.96,0,,0.3,5.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000209,2,2017-12-15 00:39:37,2017-12-15 01:35:03,N,1,,,,,1,13.96,47,0.5,0.5,5.08,5.76,,0.3,59.14,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000210,2,2017-12-15 00:25:04,2017-12-15 00:33:25,N,1,,,,,1,1.64,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000211,2,2017-12-15 00:34:48,2017-12-15 00:43:08,N,1,,,,,1,1.60,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000212,2,2017-12-15 01:01:52,2017-12-15 01:36:51,N,1,,,,,1,9.76,35,0.5,0.5,0,0,,0.3,36.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000213,1,2017-12-15 00:46:44,2017-12-15 00:49:28,N,1,,,,,1,0.50,4,0.5,0.5,0,0,,0.3,5.3,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000214,2,2017-12-15 00:06:12,2017-12-15 00:31:47,N,1,,,,,1,4.02,18,0.5,0.5,1.5,0,,0.3,20.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000215,2,2017-12-15 00:44:58,2017-12-15 01:05:54,N,1,,,,,1,4.67,18,0.5,0.5,0,0,,0.3,19.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000216,2,2017-12-15 00:17:30,2017-12-15 00:24:38,N,1,,,,,1,1.71,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000217,2,2017-12-15 00:34:51,2017-12-15 00:47:29,N,1,,,,,1,1.75,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000218,2,2017-12-15 00:50:18,2017-12-15 01:10:23,N,1,,,,,1,3.51,16,0.5,0.5,2.5,0,,0.3,19.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000219,1,2017-12-15 00:21:43,2017-12-15 00:30:20,N,1,,,,,1,1.20,7.5,0.5,0.5,1.75,0,,0.3,10.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000220,1,2017-12-15 00:45:42,2017-12-15 01:20:41,N,1,,,,,1,6.70,27,0.5,0.5,6.8,5.76,,0.3,40.86,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000221,2,2017-12-15 00:16:52,2017-12-15 00:27:51,N,1,,,,,2,1.33,8.5,0.5,0.5,1,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000222,2,2017-12-15 00:31:34,2017-12-15 00:45:50,N,1,,,,,2,6.73,20.5,0.5,0.5,4.36,0,,0.3,26.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000223,2,2017-12-15 00:02:41,2017-12-15 00:30:26,N,1,,,,,1,6.01,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000224,2,2017-12-15 00:47:01,2017-12-15 00:55:49,N,1,,,,,1,1.98,8.5,0.5,0.5,2,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000225,2,2017-12-15 00:58:15,2017-12-15 01:14:41,N,1,,,,,1,3.40,13.5,0.5,0.5,4.44,0,,0.3,19.24,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000226,2,2017-12-15 00:13:35,2017-12-15 00:27:29,N,1,,,,,1,2.59,11.5,0.5,0.5,3.84,0,,0.3,16.64,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000227,2,2017-12-15 00:38:30,2017-12-15 00:43:51,N,1,,,,,1,1.15,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000228,2,2017-12-15 00:45:24,2017-12-15 00:51:18,N,1,,,,,1,1.00,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000229,2,2017-12-15 00:55:04,2017-12-15 01:08:57,N,1,,,,,1,3.47,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000230,2,2017-12-15 00:04:17,2017-12-15 00:10:33,N,1,,,,,1,1.23,6.5,0.5,0.5,1.95,0,,0.3,9.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000231,2,2017-12-15 00:13:03,2017-12-15 00:26:55,N,1,,,,,1,2.78,12,0.5,0.5,3.32,0,,0.3,16.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000232,2,2017-12-15 00:30:01,2017-12-15 00:37:15,N,1,,,,,1,0.89,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000233,2,2017-12-15 00:38:43,2017-12-15 00:47:26,N,1,,,,,1,1.40,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000234,2,2017-12-15 00:49:45,2017-12-15 00:53:07,N,1,,,,,1,0.59,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000235,2,2017-12-15 00:56:01,2017-12-15 01:07:10,N,1,,,,,1,2.46,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000236,2,2017-12-15 00:33:40,2017-12-15 00:39:42,N,1,,,,,2,1.31,6.5,0.5,0.5,1,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000237,2,2017-12-15 00:55:08,2017-12-15 01:29:46,N,1,,,,,2,11.64,37,0.5,0.5,8.81,5.76,,0.3,52.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000238,2,2017-12-15 00:38:07,2017-12-15 00:42:29,N,1,,,,,6,0.95,5.5,0.5,0.5,1.7,0,,0.3,8.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000239,1,2017-12-15 00:02:32,2017-12-15 00:14:54,N,1,,,,,2,2.40,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000240,1,2017-12-15 00:47:13,2017-12-15 01:23:55,N,1,,,,,2,9.40,32.5,0.5,0.5,0,0,,0.3,33.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000241,2,2017-12-15 00:07:00,2017-12-15 00:22:39,N,1,,,,,2,2.66,12,0.5,0.5,3.99,0,,0.3,17.29,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000242,2,2017-12-15 00:24:20,2017-12-15 00:39:27,N,1,,,,,2,3.08,12.5,0.5,0.5,1,0,,0.3,14.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000243,2,2017-12-15 00:42:39,2017-12-15 00:57:57,N,1,,,,,2,2.33,11.5,0.5,0.5,2.56,0,,0.3,15.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000244,1,2017-12-15 00:00:39,2017-12-15 00:03:05,N,1,,,,,1,0.40,4,0.5,0.5,0,0,,0.3,5.3,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000245,2,2017-12-15 00:10:32,2017-12-15 00:43:16,N,1,,,,,5,8.04,28,0.5,0.5,3.7,0,,0.3,33,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000246,2,2017-12-15 00:35:42,2017-12-15 00:51:25,N,1,,,,,1,8.76,25,0.5,0.5,5,5.76,,0.3,37.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000247,2,2017-12-15 00:52:46,2017-12-15 01:04:05,N,1,,,,,3,1.78,9.5,0.5,0.5,1,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000248,2,2017-12-15 00:05:05,2017-12-15 00:11:34,N,1,,,,,1,1.08,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000249,2,2017-12-15 00:12:28,2017-12-15 00:37:12,N,1,,,,,1,3.21,16.5,0.5,0.5,3.56,0,,0.3,21.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000250,2,2017-12-15 00:44:06,2017-12-15 01:04:05,N,1,,,,,1,5.06,17,0.5,0.5,0,0,,0.3,18.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000251,1,2017-12-15 00:05:32,2017-12-15 00:09:10,N,1,,,,,1,0.60,4.5,0.5,0.5,1.15,0,,0.3,6.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000252,1,2017-12-15 00:27:29,2017-12-15 00:39:05,N,1,,,,,1,2.20,10,0.5,0.5,2.25,0,,0.3,13.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000253,1,2017-12-15 00:07:54,2017-12-15 00:10:50,N,1,,,,,2,0.60,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000254,1,2017-12-15 00:12:02,2017-12-15 00:17:01,N,1,,,,,2,1.00,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000255,1,2017-12-15 00:24:58,2017-12-15 00:37:38,N,1,,,,,1,1.60,9.5,0.5,0.5,1.7,0,,0.3,12.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000256,1,2017-12-15 00:41:49,2017-12-15 00:53:48,N,1,,,,,1,3.90,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000257,2,2017-12-14 23:58:09,2017-12-15 00:02:07,N,1,,,,,1,0.49,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000258,2,2017-12-15 00:05:04,2017-12-15 00:18:50,N,1,,,,,1,2.02,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000259,2,2017-12-15 00:21:45,2017-12-15 00:55:32,N,1,,,,,1,5.18,23.5,0.5,0.5,4.96,0,,0.3,29.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000260,2,2017-12-15 00:04:25,2017-12-15 00:11:39,N,1,,,,,1,1.13,7,0.5,0.5,2.08,0,,0.3,10.38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000261,2,2017-12-15 00:20:57,2017-12-15 00:31:42,N,1,,,,,1,1.08,8,0.5,0.5,1,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000262,2,2017-12-15 00:39:26,2017-12-15 00:50:31,N,1,,,,,1,2.09,9.5,0.5,0.5,0,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000263,2,2017-12-15 00:53:53,2017-12-15 01:04:58,N,1,,,,,1,1.76,9.5,0.5,0.5,1,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000264,2,2017-12-15 00:08:30,2017-12-15 00:20:52,N,1,,,,,1,1.75,9.5,0.5,0.5,0,0,,0.3,10.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000265,2,2017-12-15 00:23:40,2017-12-15 00:41:49,N,1,,,,,1,2.18,12.5,0.5,0.5,2.76,0,,0.3,16.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000266,2,2017-12-15 00:44:35,2017-12-15 00:49:32,N,1,,,,,1,0.79,5.5,0.5,0.5,2.04,0,,0.3,8.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000267,2,2017-12-15 00:52:36,2017-12-15 01:02:00,N,1,,,,,1,0.73,7.5,0.5,0.5,1.76,0,,0.3,10.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000268,2,2017-12-15 00:31:52,2017-12-15 00:47:40,N,1,,,,,1,1.80,11.5,0.5,0.5,1.08,0,,0.3,13.88,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000269,2,2017-12-15 00:53:43,2017-12-15 01:05:05,N,1,,,,,1,2.54,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000270,2,2017-12-15 00:06:34,2017-12-15 00:13:18,N,1,,,,,2,1.21,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000271,2,2017-12-15 00:18:09,2017-12-15 00:25:11,N,1,,,,,2,1.29,7,0.5,0.5,2.08,0,,0.3,10.38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000272,2,2017-12-15 00:30:15,2017-12-15 00:38:42,N,1,,,,,2,1.38,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000273,2,2017-12-15 00:40:42,2017-12-15 00:44:57,N,1,,,,,2,1.32,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000274,1,2017-12-15 00:47:31,2017-12-15 00:49:29,N,1,,,,,3,0.20,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000275,1,2017-12-15 00:52:16,2017-12-15 01:28:36,N,1,,,,,1,7.30,29.5,0.5,0.5,0,0,,0.3,30.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000276,1,2017-12-15 00:01:57,2017-12-15 00:18:51,N,1,,,,,1,11.40,31.5,0.5,0.5,8.2,0,,0.3,41,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000277,1,2017-12-15 00:39:16,2017-12-15 01:04:21,N,1,,,,,1,10.10,30.5,0.5,0.5,7.5,5.76,,0.3,45.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000278,1,2017-12-15 00:18:06,2017-12-15 00:26:33,N,1,,,,,1,1.70,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000279,1,2017-12-15 00:28:42,2017-12-15 01:10:48,N,1,,,,,1,8.70,34.5,0.5,0.5,7.15,0,,0.3,42.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000280,2,2017-12-15 00:20:44,2017-12-15 00:47:16,N,1,,,,,1,3.10,17.5,0.5,0.5,3.76,0,,0.3,22.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000281,2,2017-12-15 00:48:38,2017-12-15 00:58:46,N,1,,,,,1,1.87,9,0.5,0.5,2.06,0,,0.3,12.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000282,2,2017-12-15 00:04:31,2017-12-15 00:25:09,N,1,,,,,1,2.28,14.5,0.5,0.5,3.16,0,,0.3,18.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000283,2,2017-12-15 00:26:46,2017-12-15 00:33:08,N,1,,,,,1,0.56,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000284,2,2017-12-15 00:37:12,2017-12-15 01:04:43,N,1,,,,,1,6.54,24,0.5,0.5,0,0,,0.3,25.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000285,2,2017-12-15 00:32:16,2017-12-15 01:27:04,N,1,,,,,1,14.64,50,0.5,0.5,10.26,0,,0.3,61.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000286,2,2017-12-15 00:59:27,2017-12-15 01:20:51,N,1,,,,,1,5.95,20,0.5,0.5,0,0,,0.3,21.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000287,2,2017-12-15 00:48:18,2017-12-15 01:14:34,N,1,,,,,1,10.73,32.5,0.5,0.5,0,5.76,,0.3,39.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000288,1,2017-12-15 00:07:31,2017-12-15 00:19:09,N,1,,,,,2,2.20,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000289,1,2017-12-15 00:24:03,2017-12-15 00:42:25,N,1,,,,,1,2.30,13,0.5,0.5,3.55,0,,0.3,17.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000290,1,2017-12-15 00:44:12,2017-12-15 00:55:18,N,1,,,,,1,1.80,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000291,1,2017-12-15 00:56:48,2017-12-15 01:13:09,N,1,,,,,2,2.60,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000292,1,2017-12-15 00:19:32,2017-12-15 00:25:08,N,1,,,,,1,1.20,6,0.5,0.5,1,0,,0.3,8.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000293,1,2017-12-15 00:27:00,2017-12-15 00:51:04,N,1,,,,,1,4.60,19,0.5,0.5,4.05,0,,0.3,24.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000294,1,2017-12-15 00:20:38,2017-12-15 00:25:15,N,1,,,,,1,1.20,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000295,1,2017-12-15 00:41:06,2017-12-15 00:52:18,N,1,,,,,1,1.80,9.5,0.5,0.5,2.7,0,,0.3,13.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000296,1,2017-12-15 00:55:55,2017-12-15 01:12:31,N,1,,,,,1,3.30,14,0.5,0.5,1,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000297,2,2017-12-15 00:05:55,2017-12-15 00:14:30,N,1,,,,,1,1.63,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000298,2,2017-12-15 00:27:26,2017-12-15 01:03:16,N,1,,,,,1,6.38,26,0.5,0.5,5.46,0,,0.3,32.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000299,1,2017-12-15 00:08:21,2017-12-15 00:28:24,N,1,,,,,1,3.50,15,0.5,0.5,0,0,,0.3,16.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000300,1,2017-12-15 00:32:08,2017-12-15 00:43:23,N,1,,,,,2,2.80,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000301,1,2017-12-15 00:49:47,2017-12-15 00:53:20,N,1,,,,,2,0.60,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000302,1,2017-12-15 00:30:10,2017-12-15 01:07:08,N,1,,,,,1,9.40,33.5,0.5,0.5,5,5.76,,0.3,45.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000303,1,2017-12-15 00:04:37,2017-12-15 00:11:07,N,1,,,,,1,1.00,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000304,1,2017-12-15 00:11:47,2017-12-15 00:44:18,N,1,,,,,1,5.70,24.5,0.5,0.5,0,0,,0.3,25.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000305,1,2017-12-15 00:58:49,2017-12-15 01:17:57,N,1,,,,,1,5.20,18.5,0.5,0.5,3.95,0,,0.3,23.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000306,1,2017-12-15 00:02:04,2017-12-15 00:06:33,N,1,,,,,1,0.80,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000307,1,2017-12-15 00:19:59,2017-12-15 00:38:04,Y,1,,,,,1,2.80,13,0.5,0.5,3,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000308,2,2017-12-15 00:04:13,2017-12-15 00:44:18,N,1,,,,,6,7.07,30,0.5,0.5,6.26,0,,0.3,37.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000309,2,2017-12-15 00:54:44,2017-12-15 01:03:50,N,1,,,,,5,1.55,8,0.5,0.5,2.32,0,,0.3,11.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000310,1,2017-12-15 00:43:05,2017-12-15 00:52:01,N,1,,,,,1,2.10,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000311,1,2017-12-15 00:05:03,2017-12-15 00:18:37,N,1,,,,,1,2.60,11,0.5,0.5,2.45,0,,0.3,14.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000312,1,2017-12-15 00:20:35,2017-12-15 00:38:49,N,1,,,,,1,2.50,13.5,0.5,0.5,4.4,0,,0.3,19.2,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000313,1,2017-12-15 00:39:52,2017-12-15 00:44:38,N,1,,,,,1,1.10,5.5,0.5,0.5,1.35,0,,0.3,8.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000314,1,2017-12-15 00:49:40,2017-12-15 01:00:41,N,1,,,,,1,1.70,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000315,1,2017-12-15 00:47:28,2017-12-15 00:52:07,N,1,,,,,1,1.20,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000316,2,2017-12-15 00:06:56,2017-12-15 00:44:48,N,1,,,,,5,3.86,23.5,0.5,0.5,4.96,0,,0.3,29.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000317,2,2017-12-15 00:55:16,2017-12-15 01:06:36,N,1,,,,,5,1.94,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000318,1,2017-12-15 00:30:37,2017-12-15 00:30:47,N,2,,,,,1,0.00,52,0,0.5,8,5.76,,0.3,66.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000319,1,2017-12-15 00:34:47,2017-12-15 01:55:20,N,1,,,,,1,20.40,75.5,0.5,0.5,5,5.76,,0.3,87.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000320,2,2017-12-14 23:51:49,2017-12-15 00:09:43,N,1,,,,,1,1.19,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000321,2,2017-12-15 00:19:42,2017-12-15 00:27:12,N,1,,,,,1,0.78,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000322,2,2017-12-15 00:28:34,2017-12-15 00:40:23,N,1,,,,,1,2.75,11.5,0.5,0.5,1.5,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000323,2,2017-12-15 00:35:15,2017-12-15 00:55:59,N,1,,,,,1,7.24,23.5,0.5,0.5,0,5.76,,0.3,30.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000324,2,2017-12-15 00:26:02,2017-12-15 00:29:23,N,1,,,,,1,0.35,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000325,2,2017-12-15 00:31:42,2017-12-15 00:36:19,N,1,,,,,1,1.07,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000326,2,2017-12-15 00:36:55,2017-12-15 00:43:24,N,1,,,,,1,1.38,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000327,2,2017-12-15 00:45:40,2017-12-15 01:24:42,N,1,,,,,1,6.01,27,0.5,0.5,4.24,0,,0.3,32.54,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000328,2,2017-12-15 00:11:05,2017-12-15 00:19:49,N,1,,,,,2,1.16,7.5,0.5,0.5,1,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000329,2,2017-12-15 00:20:45,2017-12-15 00:54:08,N,1,,,,,2,5.22,23.5,0.5,0.5,2.4,0,,0.3,27.2,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000330,2,2017-12-14 23:59:30,2017-12-15 00:27:27,N,1,,,,,6,4.86,21,0.5,0.5,5.58,0,,0.3,27.88,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000331,2,2017-12-15 00:32:34,2017-12-15 00:43:36,N,1,,,,,6,2.73,10.5,0.5,0.5,0.59,0,,0.3,12.39,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000332,2,2017-12-15 00:55:30,2017-12-15 01:14:02,N,1,,,,,6,3.22,14.5,0.5,0.5,1.5,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000333,2,2017-12-15 00:48:56,2017-12-15 00:57:15,N,1,,,,,6,1.23,7.5,0.5,0.5,1.76,0,,0.3,10.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000334,1,2017-12-15 00:17:24,2017-12-15 00:28:44,N,1,,,,,1,2.10,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000335,1,2017-12-15 00:29:20,2017-12-15 00:40:38,N,1,,,,,1,1.70,9,0.5,0.5,2.5,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000336,1,2017-12-15 00:43:13,2017-12-15 01:04:32,N,1,,,,,1,1.90,14.5,0.5,0.5,3.15,0,,0.3,18.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000337,2,2017-12-15 00:18:55,2017-12-15 00:35:52,N,1,,,,,5,2.25,12.5,0.5,0.5,2,0,,0.3,15.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000338,2,2017-12-15 00:37:39,2017-12-15 00:52:21,N,1,,,,,5,1.76,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000339,2,2017-12-15 00:02:22,2017-12-15 00:18:02,N,1,,,,,2,1.79,11.5,0.5,0.5,1.5,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000340,2,2017-12-15 00:19:03,2017-12-15 00:55:49,N,1,,,,,2,8.96,33.5,0.5,0.5,6.96,0,,0.3,41.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000341,2,2017-12-15 00:38:47,2017-12-15 01:02:25,N,1,,,,,1,5.67,19.5,0.5,0.5,4.16,0,,0.3,24.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000342,1,2017-12-15 00:54:07,2017-12-15 00:56:25,N,1,,,,,1,0.30,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000343,1,2017-12-15 00:59:33,2017-12-15 01:15:36,N,1,,,,,1,4.10,16,0.5,0.5,3.45,0,,0.3,20.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000344,1,2017-12-15 00:16:35,2017-12-15 00:38:22,N,1,,,,,1,3.70,16.5,0.5,0.5,3.55,0,,0.3,21.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000345,1,2017-12-15 00:41:35,2017-12-15 00:46:07,N,1,,,,,1,1.10,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000346,1,2017-12-15 00:28:16,2017-12-15 00:49:43,N,1,,,,,1,2.10,14.5,0.5,0.5,1,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000347,1,2017-12-15 00:50:32,2017-12-15 01:02:25,N,1,,,,,1,2.00,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000348,2,2017-12-15 00:06:30,2017-12-15 00:32:46,N,2,,,,,2,18.00,52,0,0.5,8,0,,0.3,60.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000349,2,2017-12-15 00:35:57,2017-12-15 00:49:57,N,1,,,,,2,3.00,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000350,2,2017-12-15 00:51:07,2017-12-15 01:03:37,N,1,,,,,2,1.78,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000351,2,2017-12-14 23:58:18,2017-12-15 00:17:19,N,1,,,,,1,2.89,13.5,0.5,0.5,0,0,,0.3,14.8,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000352,2,2017-12-15 00:25:33,2017-12-15 00:37:17,N,1,,,,,1,1.84,9.5,0.5,0.5,2.7,0,,0.3,13.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000353,2,2017-12-15 00:38:50,2017-12-15 01:12:55,N,5,,,,,1,20.52,95,0,0.5,21.26,10.5,,0.3,127.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000354,1,2017-12-15 00:10:39,2017-12-15 00:23:43,N,1,,,,,0,1.80,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000355,1,2017-12-15 00:26:21,2017-12-15 00:46:41,N,1,,,,,0,3.90,15.5,0.5,0.5,0,0,,0.3,16.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000356,1,2017-12-15 00:51:23,2017-12-15 01:23:51,N,1,,,,,0,4.60,21,0.5,0.5,3.2,0,,0.3,25.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000357,1,2017-12-15 00:01:01,2017-12-15 00:09:09,N,1,,,,,1,1.50,7.5,0.5,0.5,2.2,0,,0.3,11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000358,1,2017-12-15 00:11:16,2017-12-15 00:28:12,N,1,,,,,1,4.20,16.5,0.5,0.5,3.55,0,,0.3,21.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000359,1,2017-12-15 00:41:38,2017-12-15 00:49:41,N,1,,,,,1,1.00,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000360,1,2017-12-15 00:53:12,2017-12-15 01:03:55,N,1,,,,,1,2.30,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000361,2,2017-12-15 00:17:03,2017-12-15 00:23:50,N,1,,,,,6,0.78,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000362,2,2017-12-15 00:26:35,2017-12-15 00:30:26,N,1,,,,,6,0.70,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000363,2,2017-12-15 00:33:08,2017-12-15 00:49:10,N,1,,,,,6,3.51,14.5,0.5,0.5,0,0,,0.3,15.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000364,2,2017-12-15 00:54:31,2017-12-15 01:18:59,N,1,,,,,6,6.01,21.5,0.5,0.5,0,0,,0.3,22.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000365,1,2017-12-15 00:06:33,2017-12-15 00:44:01,N,1,,,,,1,9.40,34,0.5,0.5,7.05,0,,0.3,42.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000366,1,2017-12-15 00:24:39,2017-12-15 00:49:07,N,1,,,,,1,2.60,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000367,1,2017-12-15 00:53:08,2017-12-15 01:07:56,N,1,,,,,1,3.30,13,0.5,0.5,3.55,0,,0.3,17.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000368,1,2017-12-15 00:02:28,2017-12-15 00:13:55,N,1,,,,,1,1.80,9.5,0.5,0.5,2,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000369,1,2017-12-15 00:33:04,2017-12-15 00:40:02,N,1,,,,,1,2.10,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000370,1,2017-12-15 00:03:29,2017-12-15 00:08:56,N,1,,,,,1,1.00,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000371,1,2017-12-15 00:10:46,2017-12-15 00:20:15,N,1,,,,,1,1.20,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000372,1,2017-12-15 00:22:56,2017-12-15 00:37:19,N,1,,,,,1,4.00,15,0.5,0.5,0,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000373,1,2017-12-15 00:42:52,2017-12-15 00:51:19,N,1,,,,,1,5.00,15.5,0.5,0.5,3.35,0,,0.3,20.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000374,1,2017-12-15 00:53:47,2017-12-15 01:23:10,N,1,,,,,1,4.70,21,0.5,0.5,0,0,,0.3,22.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000375,2,2017-12-15 00:36:27,2017-12-15 01:12:34,N,1,,,,,1,10.62,35.5,0.5,0.5,4.2,0,,0.3,41,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000376,1,2017-12-15 00:28:33,2017-12-15 00:45:42,N,1,,,,,1,3.10,14,0.5,0.5,3.8,0,,0.3,19.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000377,2,2017-12-15 00:15:28,2017-12-15 00:50:41,N,1,,,,,2,6.56,28,0.5,0.5,0,0,,0.3,29.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000378,2,2017-12-15 00:56:38,2017-12-15 01:09:17,N,1,,,,,2,6.86,20.5,0.5,0.5,1.1,0,,0.3,22.9,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000379,1,2017-12-15 00:13:36,2017-12-15 01:14:17,N,1,,,,,1,13.00,47.5,0.5,0.5,9.75,0,,0.3,58.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000380,1,2017-12-15 00:03:32,2017-12-15 00:28:38,N,1,,,,,1,4.60,19.5,0.5,0.5,1,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000381,1,2017-12-15 00:46:55,2017-12-15 01:10:49,N,1,,,,,1,5.00,20.5,0.5,0.5,0,5.76,,0.3,27.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000382,1,2017-12-15 00:09:42,2017-12-15 00:20:45,N,1,,,,,1,2.50,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000383,1,2017-12-15 00:23:14,2017-12-15 00:27:02,N,1,,,,,1,0.60,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000384,1,2017-12-15 00:31:40,2017-12-15 00:36:49,N,1,,,,,1,1.00,6,0.5,0.5,2,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000385,1,2017-12-15 00:52:15,2017-12-15 01:13:40,N,1,,,,,1,4.60,17.5,0.5,0.5,3.75,0,,0.3,22.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000386,2,2017-12-15 00:13:40,2017-12-15 00:33:20,N,1,,,,,1,1.93,13.5,0.5,0.5,1.48,0,,0.3,16.28,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000387,2,2017-12-15 00:52:47,2017-12-15 01:19:23,N,1,,,,,1,5.91,23.5,0.5,0.5,7.44,0,,0.3,32.24,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000388,1,2017-12-15 00:38:36,2017-12-15 00:56:37,N,1,,,,,1,5.90,20,0.5,0.5,4.25,0,,0.3,25.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000389,2,2017-12-15 00:15:53,2017-12-15 00:48:54,N,1,,,,,1,3.61,22,0.5,0.5,1,0,,0.3,24.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000390,2,2017-12-15 00:20:20,2017-12-15 00:40:11,N,1,,,,,1,2.02,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000391,2,2017-12-15 00:41:22,2017-12-15 00:57:21,N,1,,,,,1,3.29,14,0.5,0.5,3.06,0,,0.3,18.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000392,2,2017-12-15 00:59:16,2017-12-15 01:09:53,N,1,,,,,1,1.08,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000393,1,2017-12-15 00:44:17,2017-12-15 01:13:28,N,2,,,,,1,17.20,52,0,0.5,10.56,0,,0.3,63.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000394,2,2017-12-15 00:06:55,2017-12-15 00:17:28,N,1,,,,,2,1.58,9,0.5,0.5,2.06,0,,0.3,12.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000395,2,2017-12-15 00:22:35,2017-12-15 00:34:52,N,1,,,,,2,1.84,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000396,2,2017-12-15 00:44:25,2017-12-15 00:54:41,N,1,,,,,2,0.92,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000397,2,2017-12-15 00:10:17,2017-12-15 00:44:44,N,1,,,,,2,4.78,23,0.5,0.5,4.86,0,,0.3,29.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000398,1,2017-12-15 00:07:26,2017-12-15 00:11:49,N,1,,,,,1,0.80,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000399,1,2017-12-15 00:16:45,2017-12-15 00:46:00,N,1,,,,,1,5.10,21.5,0.5,0.5,4.55,0,,0.3,27.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000400,1,2017-12-15 00:56:23,2017-12-15 00:59:44,N,1,,,,,1,0.80,4.5,0.5,0.5,1.45,0,,0.3,7.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000401,1,2017-12-15 00:02:45,2017-12-15 00:17:46,N,1,,,,,1,2.60,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000402,1,2017-12-15 00:33:58,2017-12-15 01:10:08,N,1,,,,,1,13.80,42.5,0.5,0.5,12,0,,0.3,55.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000403,1,2017-12-15 00:08:20,2017-12-15 00:19:54,N,1,,,,,1,2.20,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000404,1,2017-12-15 00:22:02,2017-12-15 00:24:32,N,1,,,,,1,0.40,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000405,1,2017-12-15 00:49:45,2017-12-15 01:07:55,N,1,,,,,1,4.30,16.5,0.5,0.5,3.55,0,,0.3,21.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000406,1,2017-12-15 00:19:42,2017-12-15 00:35:10,N,1,,,,,1,7.20,21.5,0.5,0.5,4,0,,0.3,26.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000407,1,2017-12-15 00:36:17,2017-12-15 00:39:28,N,1,,,,,1,0.60,4.5,0.5,0.5,1.2,0,,0.3,7,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000408,1,2017-12-15 00:44:47,2017-12-15 00:47:53,N,1,,,,,1,0.50,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000409,1,2017-12-15 00:06:26,2017-12-15 01:03:38,N,1,,,,,1,12.30,48,0.5,0.5,13.75,5.76,,0.3,68.81,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000410,1,2017-12-15 00:03:14,2017-12-15 00:10:39,N,1,,,,,1,1.30,7,0.5,0.5,1.5,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000411,1,2017-12-15 00:12:39,2017-12-15 00:37:19,N,1,,,,,1,4.20,18,0.5,0.5,2.5,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000412,1,2017-12-15 00:42:49,2017-12-15 00:46:16,N,1,,,,,1,0.80,5,0.5,0.5,1.55,0,,0.3,7.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000413,2,2017-12-14 19:10:23,2017-12-14 19:18:28,N,1,,,,,1,1.02,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000414,2,2017-12-14 19:23:38,2017-12-15 19:06:32,N,5,,,,,3,4.24,60,0,0.5,0,10.5,,0.3,71.3,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000415,2,2017-12-15 00:55:49,2017-12-15 01:11:02,N,1,,,,,2,7.83,23.5,0.5,0.5,0,0,,0.3,24.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000416,2,2017-12-15 00:05:09,2017-12-15 00:20:48,N,1,,,,,2,1.36,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000417,2,2017-12-15 00:29:27,2017-12-15 00:52:13,N,1,,,,,2,3.98,17.5,0.5,0.5,3.76,0,,0.3,22.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000418,2,2017-12-15 00:02:35,2017-12-15 00:21:33,N,1,,,,,1,2.34,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000419,2,2017-12-15 00:26:40,2017-12-15 01:10:02,N,1,,,,,1,9.00,34.5,0.5,0.5,2.2,0,,0.3,38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000420,2,2017-12-15 00:41:13,2017-12-15 00:47:16,N,1,,,,,5,0.95,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000421,2,2017-12-15 00:48:07,2017-12-15 00:51:55,N,1,,,,,5,0.82,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000422,2,2017-12-15 00:54:41,2017-12-15 01:36:33,N,1,,,,,5,8.13,31.5,0.5,0.5,6.56,0,,0.3,39.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000423,1,2017-12-15 00:10:16,2017-12-15 00:30:12,N,1,,,,,3,4.20,17,0.5,0.5,0,0,,0.3,18.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000424,1,2017-12-15 00:09:09,2017-12-15 00:36:56,N,1,,,,,1,4.20,20,0.5,0.5,0.01,0,,0.3,21.31,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000425,1,2017-12-15 00:54:57,2017-12-15 01:17:59,N,1,,,,,1,5.40,19.5,0.5,0.5,4.15,0,,0.3,24.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000426,2,2017-12-15 00:13:43,2017-12-15 00:42:45,N,1,,,,,1,6.63,25.5,0.5,0.5,9.77,5.76,,0.3,42.33,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000427,2,2017-12-15 00:09:34,2017-12-15 00:21:47,N,1,,,,,1,1.91,10,0.5,0.5,0,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000428,2,2017-12-15 00:26:37,2017-12-15 00:54:33,N,1,,,,,1,6.54,25,0.5,0.5,0,5.76,,0.3,32.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000429,1,2017-12-15 00:09:48,2017-12-15 00:19:19,N,1,,,,,1,2.60,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000430,1,2017-12-15 00:32:26,2017-12-15 00:36:48,N,1,,,,,1,0.50,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000431,1,2017-12-15 00:42:32,2017-12-15 01:03:48,N,1,,,,,1,3.90,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000432,2,2017-12-14 23:59:37,2017-12-15 00:04:06,N,1,,,,,1,0.61,5,0.5,0.5,1,0,,0.3,7.3,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000433,2,2017-12-15 00:06:30,2017-12-15 00:32:11,N,1,,,,,1,4.48,18.5,0.5,0.5,2,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000434,2,2017-12-15 00:40:44,2017-12-15 01:00:28,N,1,,,,,1,4.72,16.5,0.5,0.5,4.71,5.76,,0.3,28.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000435,2,2017-12-15 00:10:53,2017-12-15 00:17:52,N,1,,,,,1,1.12,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000436,2,2017-12-15 00:35:36,2017-12-15 00:59:12,N,1,,,,,1,10.99,32.5,0.5,0.5,6.76,0,,0.3,40.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000437,2,2017-12-15 00:43:04,2017-12-15 01:16:19,N,2,,,,,1,17.90,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000438,1,2017-12-15 00:02:42,2017-12-15 00:10:12,N,1,,,,,1,1.30,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000439,1,2017-12-15 00:11:22,2017-12-15 00:40:14,N,1,,,,,1,7.30,25.5,0.5,0.5,6.7,0,,0.3,33.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000440,1,2017-12-15 00:20:08,2017-12-15 00:40:28,N,1,,,,,1,2.40,14,0.5,0.5,3.05,0,,0.3,18.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000441,1,2017-12-15 00:43:20,2017-12-15 01:08:32,N,1,,,,,2,1.30,15.5,0.5,0.5,5,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000442,1,2017-12-15 00:11:32,2017-12-15 00:47:05,N,1,,,,,1,5.10,26.5,0.5,0.5,8.3,0,,0.3,36.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000443,2,2017-12-15 00:09:29,2017-12-15 00:29:03,N,1,,,,,1,5.70,20.5,0.5,0.5,4.36,0,,0.3,26.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000444,2,2017-12-15 00:30:13,2017-12-15 00:37:34,N,1,,,,,1,1.67,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000445,2,2017-12-14 23:58:56,2017-12-15 00:30:50,N,2,,,,,1,20.04,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000446,2,2017-12-15 00:36:57,2017-12-15 00:45:41,N,1,,,,,1,1.46,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000447,1,2017-12-15 00:04:54,2017-12-15 00:37:27,N,1,,,,,1,4.60,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000448,2,2017-12-15 00:48:43,2017-12-15 00:53:39,N,1,,,,,1,1.30,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000449,2,2017-12-15 00:57:01,2017-12-15 01:01:12,N,1,,,,,1,0.68,5,0.5,0.5,0.63,0,,0.3,6.93,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000450,1,2017-12-15 00:05:18,2017-12-15 00:49:34,N,1,,,,,1,15.40,51,0.5,0.5,5,0,,0.3,57.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000451,2,2017-12-15 00:45:45,2017-12-15 01:07:45,N,1,,,,,1,9.52,28,0.5,0.5,0,5.76,,0.3,35.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000452,2,2017-12-15 00:08:35,2017-12-15 00:56:01,N,1,,,,,5,7.01,31,0.5,0.5,9.69,0,,0.3,41.99,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000453,1,2017-12-15 00:10:24,2017-12-15 00:31:26,N,1,,,,,1,3.20,16,0.5,0.5,0,0,,0.3,17.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000454,2,2017-12-15 00:04:37,2017-12-15 00:53:48,N,1,,,,,1,11.07,40.5,0.5,0.5,5,5.76,,0.3,52.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000455,1,2017-12-15 00:05:10,2017-12-15 00:20:23,N,1,,,,,1,3.10,13,0.5,0.5,1.43,0,,0.3,15.73,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000456,1,2017-12-15 00:21:02,2017-12-15 00:36:30,N,1,,,,,2,2.30,11.5,0.5,0.5,1.92,0,,0.3,14.72,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000457,1,2017-12-15 00:38:38,2017-12-15 01:10:20,N,1,,,,,1,4.90,22.5,0.5,0.5,5.95,0,,0.3,29.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000458,1,2017-12-15 00:03:57,2017-12-15 00:07:19,N,1,,,,,2,1.10,5.5,0.5,0.5,1,0,,0.3,7.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000459,1,2017-12-15 00:26:00,2017-12-15 00:34:10,N,1,,,,,1,1.30,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000460,1,2017-12-15 00:36:16,2017-12-15 00:49:08,N,1,,,,,1,2.60,11,0.5,0.5,2.45,0,,0.3,14.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000461,1,2017-12-15 00:53:25,2017-12-15 01:00:12,N,1,,,,,1,1.90,8,0.5,0.5,2.3,0,,0.3,11.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000462,2,2017-12-15 00:15:12,2017-12-15 00:26:54,N,1,,,,,1,1.17,8.5,0.5,0.5,2.94,0,,0.3,12.74,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000463,2,2017-12-15 00:47:47,2017-12-15 00:59:33,N,1,,,,,1,1.40,9,0.5,0.5,2.06,0,,0.3,12.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000464,2,2017-12-15 00:08:58,2017-12-15 00:43:22,N,1,,,,,5,6.10,25.5,0.5,0.5,8.04,0,,0.3,34.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000465,2,2017-12-15 00:48:26,2017-12-15 01:01:21,N,1,,,,,5,2.41,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000466,1,2017-12-15 00:42:54,2017-12-15 00:46:52,N,1,,,,,1,0.70,4.5,0.5,0.5,1.15,0,,0.3,6.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000467,1,2017-12-15 00:01:07,2017-12-15 00:05:50,N,1,,,,,1,0.90,5,0.5,0.5,0,0,,0.3,6.3,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000468,1,2017-12-15 00:09:05,2017-12-15 00:26:40,N,1,,,,,1,3.60,16,0.5,0.5,1,0,,0.3,18.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000469,1,2017-12-15 00:59:32,2017-12-15 01:16:09,N,1,,,,,1,7.70,23,0.5,0.5,7.5,5.76,,0.3,37.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000470,1,2017-12-15 00:15:12,2017-12-15 00:36:25,N,1,,,,,1,1.40,13.5,0.5,0.5,2.95,0,,0.3,17.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000471,1,2017-12-15 00:39:37,2017-12-15 01:19:53,N,1,,,,,1,8.00,31,0.5,0.5,8.05,0,,0.3,40.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000472,1,2017-12-15 00:04:23,2017-12-15 00:37:18,N,1,,,,,1,6.20,25,0.5,0.5,5.25,0,,0.3,31.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000473,1,2017-12-15 00:53:13,2017-12-15 00:59:08,N,1,,,,,1,1.20,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000474,1,2017-12-15 00:14:58,2017-12-15 00:43:09,N,1,,,,,1,4.20,19.5,0.5,0.5,4.16,0,,0.3,24.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000475,1,2017-12-15 00:54:26,2017-12-15 01:01:22,N,1,,,,,1,1.10,6.5,0.5,0.5,1,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000476,2,2017-12-15 00:03:41,2017-12-15 00:33:29,N,1,,,,,1,6.35,25,0.5,0.5,5.26,0,,0.3,31.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000477,2,2017-12-15 00:55:24,2017-12-15 01:29:13,N,1,,,,,1,8.48,30,0.5,0.5,5,0,,0.3,36.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000478,2,2017-12-15 00:13:25,2017-12-15 00:43:14,N,2,,,,,2,17.12,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000479,1,2017-12-15 00:10:35,2017-12-15 00:21:11,N,1,,,,,2,1.10,8.5,0.5,0.5,1.95,0,,0.3,11.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000480,1,2017-12-15 00:22:05,2017-12-15 00:25:54,N,1,,,,,1,0.60,4.5,0.5,0.5,1.15,0,,0.3,6.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000481,1,2017-12-15 00:27:36,2017-12-15 00:40:36,N,1,,,,,1,1.00,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000482,1,2017-12-15 00:41:44,2017-12-15 00:47:23,N,1,,,,,1,1.20,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000483,1,2017-12-15 00:49:04,2017-12-15 01:03:47,N,1,,,,,1,2.10,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000484,1,2017-12-15 00:05:27,2017-12-15 00:29:12,N,1,,,,,0,10.50,31.5,0.5,0.5,9,5.76,,0.3,47.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000485,1,2017-12-15 00:35:54,2017-12-15 00:40:31,N,1,,,,,2,0.70,5.5,0.5,0.5,1.35,0,,0.3,8.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000486,1,2017-12-15 00:43:11,2017-12-15 00:53:22,N,1,,,,,2,1.00,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000487,2,2017-12-15 00:31:19,2017-12-15 00:49:31,N,1,,,,,1,6.31,20.5,0.5,0.5,4.36,0,,0.3,26.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000488,2,2017-12-15 00:19:14,2017-12-15 00:29:43,N,1,,,,,1,1.05,8,0.5,0.5,1.5,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000489,2,2017-12-15 00:32:35,2017-12-15 00:44:57,N,1,,,,,1,2.42,10.5,0.5,0.5,1,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000490,2,2017-12-15 00:48:40,2017-12-15 00:57:57,N,1,,,,,1,1.22,8,0.5,0.5,2.32,0,,0.3,11.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000491,2,2017-12-15 00:38:41,2017-12-15 01:21:39,N,1,,,,,2,17.74,52,0.5,0.5,11.81,5.76,,0.3,70.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000492,2,2017-12-15 00:08:26,2017-12-15 00:19:44,N,1,,,,,1,1.66,8.5,0.5,0.5,2.5,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000493,2,2017-12-15 00:22:53,2017-12-15 00:36:57,N,1,,,,,1,1.74,10,0.5,0.5,1,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000494,2,2017-12-15 01:00:01,2017-12-15 01:05:22,N,1,,,,,1,0.96,5.5,0.5,0.5,1.7,0,,0.3,8.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000495,2,2017-12-15 00:02:53,2017-12-15 00:09:38,N,1,,,,,1,0.65,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000496,2,2017-12-15 00:11:31,2017-12-15 00:12:25,N,5,,,,,1,0.00,15,0,0.5,40,0,,0.3,55.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000497,2,2017-12-15 00:43:55,2017-12-15 00:56:49,N,1,,,,,1,1.94,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000498,1,2017-12-15 00:25:49,2017-12-15 00:35:05,N,1,,,,,1,1.10,7.5,0.5,0.5,1.75,0,,0.3,10.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000499,1,2017-12-15 00:52:06,2017-12-15 00:57:28,N,1,,,,,2,0.60,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000500,1,2017-12-15 00:34:18,2017-12-15 01:21:09,N,1,,,,,1,6.60,30,0.5,0.5,9.35,0,,0.3,40.65,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000501,2,2017-12-15 00:07:37,2017-12-15 00:59:00,N,1,,,,,1,10.17,42,0.5,0.5,0,0,,0.3,43.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000502,1,2017-12-15 00:07:42,2017-12-15 00:30:07,N,1,,,,,2,2.90,16,0.5,0.5,0,0,,0.3,17.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000503,1,2017-12-15 00:30:45,2017-12-15 00:38:40,N,1,,,,,1,1.30,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000504,1,2017-12-15 00:42:23,2017-12-15 00:55:29,N,1,,,,,2,1.80,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000505,1,2017-12-15 00:57:04,2017-12-15 00:57:04,N,1,,,,,1,0.00,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000506,1,2017-12-15 00:59:48,2017-12-15 01:21:26,N,1,,,,,1,4.50,17.5,0.5,0.5,3.75,0,,0.3,22.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000507,2,2017-12-15 00:20:44,2017-12-15 00:30:11,N,1,,,,,2,0.78,7.5,0.5,0.5,0,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000508,2,2017-12-15 00:37:52,2017-12-15 00:52:16,N,1,,,,,2,1.16,10.5,0.5,0.5,1.5,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000509,2,2017-12-15 00:58:26,2017-12-15 01:04:02,N,1,,,,,2,1.00,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000510,1,2017-12-15 00:05:09,2017-12-15 00:25:20,N,1,,,,,1,2.60,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000511,1,2017-12-15 00:29:09,2017-12-15 00:36:48,N,1,,,,,1,1.40,7.5,0.5,0.5,2.2,0,,0.3,11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000512,1,2017-12-15 00:38:25,2017-12-15 00:51:42,N,1,,,,,1,1.50,10,0.5,0.5,3.35,0,,0.3,14.65,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000513,1,2017-12-15 00:23:44,2017-12-15 00:49:22,N,1,,,,,1,4.40,20.5,0.5,0.5,4.35,0,,0.3,26.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000514,1,2017-12-15 00:09:29,2017-12-15 00:14:30,N,1,,,,,1,0.80,5.5,0.5,0.5,1.7,0,,0.3,8.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000515,1,2017-12-15 00:16:07,2017-12-15 00:35:34,N,1,,,,,1,4.60,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000516,1,2017-12-15 00:58:07,2017-12-15 01:05:15,N,1,,,,,2,1.40,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000517,2,2017-12-15 00:25:15,2017-12-15 00:34:38,N,1,,,,,1,1.21,8,0.5,0.5,2.79,0,,0.3,12.09,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000518,2,2017-12-15 00:40:54,2017-12-15 00:52:56,N,1,,,,,1,0.86,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000519,2,2017-12-15 00:54:22,2017-12-15 01:00:26,N,1,,,,,1,0.89,5.5,0.5,0.5,1,0,,0.3,7.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000520,1,2017-12-15 00:07:03,2017-12-15 00:17:51,N,1,,,,,1,1.00,8,0.5,0.5,1,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000521,1,2017-12-15 00:27:43,2017-12-15 01:02:24,N,1,,,,,1,6.80,27,0.5,0.5,0,0,,0.3,28.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000522,2,2017-12-15 00:14:12,2017-12-15 00:57:30,N,4,,,,,1,33.89,158,0.5,0.5,10,0,,0.3,169.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000523,2,2017-12-15 00:01:06,2017-12-15 00:16:46,N,1,,,,,1,3.00,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000524,2,2017-12-15 00:30:25,2017-12-15 00:41:04,N,1,,,,,1,3.33,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000525,2,2017-12-15 00:42:14,2017-12-15 00:52:16,N,1,,,,,1,4.32,14,0.5,0.5,3.06,0,,0.3,18.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000526,1,2017-12-15 00:44:17,2017-12-15 00:53:15,N,1,,,,,1,2.10,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000527,1,2017-12-15 00:58:36,2017-12-15 01:17:56,N,1,,,,,2,5.00,17.5,0.5,0.5,3.75,0,,0.3,22.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000528,2,2017-12-15 00:14:35,2017-12-15 00:18:59,N,1,,,,,1,0.79,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000529,2,2017-12-15 00:26:09,2017-12-15 01:04:29,N,1,,,,,1,11.54,37.5,0.5,0.5,7.76,0,,0.3,48.51,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000530,2,2017-12-15 00:14:54,2017-12-15 00:31:45,N,1,,,,,1,2.37,12.5,0.5,0.5,2.07,0,,0.3,15.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000531,2,2017-12-15 00:33:52,2017-12-15 00:40:05,N,1,,,,,2,1.43,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000532,2,2017-12-15 00:49:02,2017-12-15 00:52:27,N,1,,,,,2,1.01,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000533,2,2017-12-15 00:21:05,2017-12-15 00:32:58,N,1,,,,,2,1.94,10,0.5,0.5,2.82,0,,0.3,14.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000534,2,2017-12-15 00:36:08,2017-12-15 00:44:27,N,1,,,,,2,1.33,7.5,0.5,0.5,1.2,0,,0.3,10,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000535,2,2017-12-15 00:49:05,2017-12-15 01:22:21,N,1,,,,,2,6.61,25.5,0.5,0.5,5.36,0,,0.3,32.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000536,2,2017-12-15 00:43:23,2017-12-15 01:16:38,N,1,,,,,1,7.28,27,0.5,0.5,5,0,,0.3,33.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000537,1,2017-12-15 00:08:26,2017-12-15 00:11:22,N,1,,,,,2,0.40,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000538,1,2017-12-15 00:13:29,2017-12-15 00:46:41,N,1,,,,,1,6.30,25,0.5,0.5,5.25,0,,0.3,31.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000539,2,2017-12-15 00:11:39,2017-12-15 00:11:42,N,5,,,,,1,0.00,125,0,0,40,0,,0.3,165.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000540,2,2017-12-15 00:56:03,2017-12-15 01:00:44,N,1,,,,,1,0.97,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000541,1,2017-12-15 00:42:52,2017-12-15 01:02:34,N,1,,,,,1,13.00,36,0.5,0.5,6,0,,0.3,43.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000542,1,2017-12-15 00:03:04,2017-12-15 00:52:32,N,1,,,,,1,6.60,32,0.5,0.5,6.65,0,,0.3,39.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000543,1,2017-12-15 00:13:41,2017-12-15 00:28:38,N,1,,,,,1,4.20,14.5,0.5,0.5,3.5,0,,0.3,19.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000544,1,2017-12-15 00:33:27,2017-12-15 00:40:04,N,1,,,,,1,1.70,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000545,1,2017-12-15 00:47:29,2017-12-15 00:58:09,N,1,,,,,1,2.20,10,0.5,0.5,2.8,0,,0.3,14.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000546,1,2017-12-15 00:05:19,2017-12-15 00:38:17,N,1,,,,,1,8.20,30,0.5,0.5,6.25,0,,0.3,37.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000547,2,2017-12-15 00:53:06,2017-12-15 01:14:50,N,1,,,,,4,12.29,35,0.5,0.5,0,0,,0.3,36.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000548,1,2017-12-15 00:02:50,2017-12-15 00:07:18,N,1,,,,,1,0.60,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000549,1,2017-12-15 00:12:02,2017-12-15 00:14:27,N,1,,,,,1,0.40,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000550,1,2017-12-15 00:15:16,2017-12-15 00:21:59,N,1,,,,,1,0.90,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000551,1,2017-12-15 00:24:38,2017-12-15 00:46:00,N,1,,,,,1,4.00,17,0.5,0.5,1,0,,0.3,19.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000552,1,2017-12-15 00:53:46,2017-12-15 00:59:26,N,1,,,,,1,1.20,6.5,0.5,0.5,2,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000553,1,2017-12-15 00:00:57,2017-12-15 00:15:13,N,1,,,,,2,2.60,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000554,1,2017-12-15 00:19:11,2017-12-15 01:02:12,N,1,,,,,2,6.70,30,0.5,0.5,7.8,0,,0.3,39.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000555,2,2017-12-15 00:35:27,2017-12-15 00:54:46,N,1,,,,,1,2.59,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000556,2,2017-12-15 00:06:54,2017-12-15 00:37:31,N,1,,,,,1,9.21,31,0.5,0.5,8.08,0,,0.3,40.38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000557,2,2017-12-15 00:38:32,2017-12-15 00:53:06,N,1,,,,,1,4.34,14.5,0.5,0.5,3.95,0,,0.3,19.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000558,2,2017-12-15 00:56:49,2017-12-15 01:03:38,N,1,,,,,1,1.81,7.5,0.5,0.5,0,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000559,2,2017-12-15 00:03:50,2017-12-15 00:13:29,N,1,,,,,1,2.34,9.5,0.5,0.5,2.16,0,,0.3,12.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000560,2,2017-12-15 00:28:29,2017-12-15 00:35:00,N,1,,,,,1,1.32,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000561,2,2017-12-15 00:50:11,2017-12-15 01:05:29,N,1,,,,,1,3.22,13.5,0.5,0.5,4.44,0,,0.3,19.24,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000562,1,2017-12-15 00:02:47,2017-12-15 00:41:54,N,1,,,,,1,7.80,31.5,0.5,0.5,6.55,0,,0.3,39.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000563,2,2017-12-15 00:05:55,2017-12-15 00:55:30,N,1,,,,,1,14.85,47,0.5,0.5,0,0,,0.3,48.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000564,2,2017-12-15 00:57:55,2017-12-15 00:58:01,N,5,,,,,1,0.00,10,0,0.5,0,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000565,2,2017-12-15 00:13:16,2017-12-15 00:22:13,N,1,,,,,2,1.57,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000566,2,2017-12-15 00:22:47,2017-12-15 00:24:31,N,1,,,,,1,0.43,3.5,0.5,0.5,0.72,0,,0.3,5.52,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000567,2,2017-12-15 00:32:29,2017-12-15 00:35:46,N,1,,,,,2,0.72,4.5,0.5,0.5,1.45,0,,0.3,7.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000568,2,2017-12-15 00:36:36,2017-12-15 00:52:02,N,1,,,,,2,2.10,11.5,0.5,0.5,2.56,0,,0.3,15.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000569,2,2017-12-15 00:54:25,2017-12-15 00:58:58,N,1,,,,,2,0.00,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000570,1,2017-12-15 00:10:19,2017-12-15 00:42:43,N,1,,,,,1,5.40,22.5,0.5,0.5,0,0,,0.3,23.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000571,2,2017-12-15 00:22:17,2017-12-15 00:54:37,N,1,,,,,1,5.00,24,0.5,0.5,4,0,,0.3,29.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000572,2,2017-12-14 23:58:52,2017-12-15 00:49:35,N,1,,,,,1,11.20,41,0.5,0.5,7.21,5.76,,0.3,55.27,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000573,1,2017-12-15 00:36:07,2017-12-15 01:03:18,Y,1,,,,,1,5.00,20,0.5,0.5,4.25,0,,0.3,25.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000574,1,2017-12-15 00:15:45,2017-12-15 00:50:45,N,1,,,,,1,9.20,32,0.5,0.5,0,5.76,,0.3,39.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000575,1,2017-12-15 00:22:12,2017-12-15 00:38:30,N,1,,,,,1,2.20,12.5,0.5,0.5,0,0,,0.3,13.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000576,1,2017-12-15 00:41:46,2017-12-15 01:01:28,Y,1,,,,,1,3.70,15.5,0.5,0.5,3.35,0,,0.3,20.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000577,1,2017-12-15 00:25:24,2017-12-15 01:02:48,N,1,,,,,1,6.00,27.5,0.5,0.5,5.75,0,,0.3,34.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000578,2,2017-12-15 00:07:55,2017-12-15 00:12:34,N,1,,,,,1,0.96,5.5,0.5,0.5,0,0,,0.3,6.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000579,2,2017-12-15 00:13:36,2017-12-15 00:45:06,N,1,,,,,1,4.72,22,0.5,0.5,0,0,,0.3,23.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000580,2,2017-12-15 00:56:28,2017-12-15 01:00:43,N,1,,,,,1,0.79,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000581,2,2017-12-15 00:08:58,2017-12-15 00:16:04,N,1,,,,,1,0.73,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000582,2,2017-12-15 00:17:18,2017-12-15 00:50:54,N,1,,,,,1,3.66,22.5,0.5,0.5,4.76,0,,0.3,28.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000583,2,2017-12-15 00:49:08,2017-12-15 01:09:20,N,1,,,,,1,4.16,17,0.5,0.5,3.66,0,,0.3,21.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000584,2,2017-12-15 00:50:52,2017-12-15 01:03:54,N,1,,,,,1,2.09,10,0.5,0.5,2.82,0,,0.3,14.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000585,2,2017-12-15 00:31:15,2017-12-15 00:35:58,N,1,,,,,1,1.11,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000586,2,2017-12-15 00:51:58,2017-12-15 01:15:57,N,1,,,,,1,2.79,16,0.5,0.5,0,5.76,,0.3,23.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000587,2,2017-12-15 00:03:58,2017-12-15 00:14:13,N,1,,,,,1,1.68,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000588,2,2017-12-15 00:16:35,2017-12-15 01:25:44,N,1,,,,,1,15.30,59,0.5,0.5,12.06,0,,0.3,72.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000589,2,2017-12-15 00:06:16,2017-12-15 00:20:06,N,1,,,,,1,2.16,11,0.5,0.5,0,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000590,2,2017-12-15 00:23:57,2017-12-15 00:34:07,N,1,,,,,1,1.73,8.5,0.5,0.5,2.45,0,,0.3,12.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000591,2,2017-12-15 00:35:00,2017-12-15 00:41:32,N,1,,,,,1,1.44,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000592,2,2017-12-15 00:42:57,2017-12-15 01:17:17,N,1,,,,,1,8.22,28.5,0.5,0.5,7.45,0,,0.3,37.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000593,2,2017-12-15 00:11:18,2017-12-15 00:41:56,N,1,,,,,1,4.03,20.5,0.5,0.5,3.5,0,,0.3,25.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000594,2,2017-12-15 00:50:15,2017-12-15 01:05:16,N,1,,,,,1,2.87,12.5,0.5,0.5,0,0,,0.3,13.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000595,2,2017-12-15 00:44:35,2017-12-15 01:36:39,N,1,,,,,1,13.58,47,0.5,0.5,9,0,,0.3,57.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000596,2,2017-12-15 00:10:26,2017-12-15 01:14:11,N,2,,,,,5,22.01,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000597,1,2017-12-15 00:20:22,2017-12-15 00:59:40,N,2,,,,,2,18.00,52,0,0.5,0,0,,0.3,52.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000598,1,2017-12-15 00:11:10,2017-12-15 00:22:38,N,1,,,,,2,2.00,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000599,1,2017-12-15 00:25:37,2017-12-15 00:41:15,N,1,,,,,3,2.60,12.5,0.5,0.5,0,0,,0.3,13.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000600,2,2017-12-15 00:09:00,2017-12-15 00:36:33,N,1,,,,,2,2.91,18.5,0.5,0.5,0,0,,0.3,19.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000601,2,2017-12-15 00:53:19,2017-12-15 01:19:39,N,1,,,,,2,7.12,24,0.5,0.5,6.32,0,,0.3,31.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000602,2,2017-12-15 00:29:14,2017-12-15 00:38:05,N,1,,,,,1,1.50,8,0.5,0.5,2.32,0,,0.3,11.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000603,2,2017-12-15 00:39:24,2017-12-15 00:57:43,N,1,,,,,1,3.18,14,0.5,0.5,1,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000604,2,2017-12-15 00:22:31,2017-12-15 00:29:31,N,1,,,,,3,1.44,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000605,2,2017-12-15 00:48:10,2017-12-15 01:20:50,N,1,,,,,3,6.85,25.5,0.5,0.5,0,0,,0.3,26.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000606,1,2017-12-15 00:17:11,2017-12-15 00:24:41,N,1,,,,,1,1.30,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000607,1,2017-12-15 00:26:31,2017-12-15 00:36:40,N,1,,,,,1,1.40,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000608,1,2017-12-15 00:43:07,2017-12-15 01:24:10,N,1,,,,,1,10.60,37.5,0.5,0.5,2,0,,0.3,40.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000609,2,2017-12-15 00:22:39,2017-12-15 00:43:55,N,2,,,,,1,8.03,52,0,0.5,2,5.76,,0.3,60.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000610,1,2017-12-15 00:02:34,2017-12-15 00:11:48,N,1,,,,,1,4.90,15.5,0.5,0.5,3,0,,0.3,19.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000611,1,2017-12-15 00:29:52,2017-12-15 00:32:26,N,1,,,,,1,0.30,3.5,0.5,0.5,1.4,0,,0.3,6.2,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000612,1,2017-12-15 00:37:00,2017-12-15 00:40:47,N,1,,,,,1,0.70,5,0.5,0.5,1,0,,0.3,7.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000613,1,2017-12-15 00:23:17,2017-12-15 00:56:40,N,1,,,,,1,3.90,21.5,0.5,0.5,4.55,0,,0.3,27.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000614,2,2017-12-15 00:45:58,2017-12-15 01:16:25,N,1,,,,,1,15.59,44.5,0.5,0.5,0,5.76,,0.3,51.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000615,2,2017-12-15 00:10:34,2017-12-15 00:11:56,N,1,,,,,1,0.08,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000616,2,2017-12-15 00:17:20,2017-12-15 00:36:29,N,1,,,,,1,2.99,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000617,2,2017-12-15 00:10:12,2017-12-15 00:30:28,N,1,,,,,1,2.17,14,0.5,0.5,3.06,0,,0.3,18.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000618,2,2017-12-15 00:35:21,2017-12-15 00:55:02,N,1,,,,,1,6.77,22.5,0.5,0.5,4.76,0,,0.3,28.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000619,1,2017-12-15 00:46:25,2017-12-15 01:09:39,N,1,,,,,1,5.70,20.5,0.5,0.5,0,0,,0.3,21.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000620,1,2017-12-15 00:04:11,2017-12-15 00:18:22,N,1,,,,,1,3.10,12,0.5,0.5,2,0,,0.3,15.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000621,1,2017-12-15 00:31:24,2017-12-15 00:42:28,N,1,,,,,1,0.60,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000622,1,2017-12-15 00:44:08,2017-12-15 00:54:43,N,1,,,,,1,1.60,9,0.5,0.5,3.05,0,,0.3,13.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000623,2,2017-12-15 00:19:42,2017-12-15 00:47:45,N,1,,,,,1,13.76,39,0.5,0.5,0,5.76,,0.3,46.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000624,1,2017-12-15 00:52:27,2017-12-15 01:08:45,N,1,,,,,1,7.70,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000625,1,2017-12-15 00:19:22,2017-12-15 00:34:50,N,1,,,,,1,3.10,14,0.5,0.5,2,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000626,1,2017-12-15 00:44:21,2017-12-15 01:23:25,N,1,,,,,1,10.70,35,0.5,0.5,7,0,,0.3,43.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000627,1,2017-12-15 00:50:13,2017-12-15 01:15:53,N,1,,,,,1,16.10,44,0.5,0.5,11.3,0,,0.3,56.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000628,2,2017-12-14 23:58:56,2017-12-15 00:03:51,N,1,,,,,3,1.18,6,0.5,0.5,1.02,0,,0.3,8.32,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000629,2,2017-12-15 00:05:10,2017-12-15 00:41:52,N,1,,,,,4,9.78,33.5,0.5,0.5,2,5.76,,0.3,42.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000630,1,2017-12-15 00:02:37,2017-12-15 00:09:45,N,1,,,,,1,1.30,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000631,1,2017-12-15 00:16:03,2017-12-15 00:22:28,N,1,,,,,1,0.50,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000632,1,2017-12-15 00:24:13,2017-12-15 00:37:26,N,1,,,,,2,1.90,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000633,1,2017-12-15 00:48:50,2017-12-15 00:52:32,N,1,,,,,1,0.50,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000634,2,2017-12-15 00:10:46,2017-12-15 00:39:54,N,1,,,,,2,4.28,21,0.5,0.5,4.46,0,,0.3,26.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000635,1,2017-12-15 00:03:20,2017-12-15 00:09:54,N,1,,,,,5,1.00,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000636,1,2017-12-15 00:11:44,2017-12-15 00:13:57,N,1,,,,,2,0.50,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000637,1,2017-12-15 00:14:52,2017-12-15 00:31:37,N,1,,,,,1,3.60,14.5,0.5,0.5,3.15,0,,0.3,18.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000638,1,2017-12-15 00:28:49,2017-12-15 00:35:36,N,1,,,,,1,1.30,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000639,1,2017-12-15 00:46:13,2017-12-15 01:11:12,N,1,,,,,1,4.90,19,0.5,0.5,0,0,,0.3,20.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000640,1,2017-12-15 00:52:19,2017-12-15 01:28:24,N,1,,,,,1,6.60,29,0.5,0.5,6.05,0,,0.3,36.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000641,1,2017-12-15 00:09:02,2017-12-15 00:15:32,N,1,,,,,2,1.30,7,0.5,0.5,0,0,,0.3,8.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000642,1,2017-12-15 00:32:12,2017-12-15 00:38:02,N,1,,,,,1,1.30,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000643,1,2017-12-15 00:57:40,2017-12-15 01:05:08,N,1,,,,,1,1.40,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000644,2,2017-12-15 00:07:59,2017-12-15 00:37:06,N,1,,,,,1,3.73,19,0.5,0.5,1,0,,0.3,21.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000645,2,2017-12-15 00:49:50,2017-12-15 00:56:39,N,1,,,,,1,1.95,8.5,0.5,0.5,0,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000646,2,2017-12-15 00:01:20,2017-12-15 00:25:50,N,1,,,,,5,5.33,20.5,0.5,0.5,4.36,0,,0.3,26.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000647,2,2017-12-15 00:28:14,2017-12-15 00:29:15,N,1,,,,,5,0.03,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000648,2,2017-12-15 00:39:06,2017-12-15 00:53:21,N,1,,,,,5,1.21,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000649,2,2017-12-15 00:54:22,2017-12-15 01:20:16,N,1,,,,,5,4.22,18.5,0.5,0.5,4.95,0,,0.3,24.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000650,2,2017-12-15 00:21:40,2017-12-15 00:42:49,N,1,,,,,2,3.51,15.5,0.5,0.5,0,0,,0.3,16.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000651,2,2017-12-15 01:00:21,2017-12-15 01:20:15,N,1,,,,,2,8.83,26.5,0.5,0.5,0,5.76,,0.3,33.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000652,1,2017-12-15 00:51:00,2017-12-15 01:03:07,N,1,,,,,1,2.10,10,0.5,0.5,2.3,0,,0.3,13.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000653,1,2017-12-15 00:00:05,2017-12-15 00:14:23,N,1,,,,,1,2.60,12,0.5,0.5,3.95,0,,0.3,17.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000654,1,2017-12-15 00:26:10,2017-12-15 00:30:41,N,1,,,,,1,0.70,5,0.5,0.5,0.95,0,,0.3,7.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000655,1,2017-12-15 00:32:06,2017-12-15 00:42:31,N,1,,,,,1,1.70,9,0.5,0.5,0,0,,0.3,10.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000656,1,2017-12-15 00:43:47,2017-12-15 01:04:09,N,1,,,,,1,3.90,15.5,0.5,0.5,0,0,,0.3,16.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000657,2,2017-12-15 00:00:11,2017-12-15 00:29:57,N,1,,,,,3,5.36,24,0.5,0.5,7.59,0,,0.3,32.89,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000658,2,2017-12-15 00:55:15,2017-12-15 01:12:55,N,1,,,,,4,2.32,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000659,2,2017-12-15 00:05:39,2017-12-15 00:32:38,N,1,,,,,1,3.69,19,0.5,0.5,6.09,0,,0.3,26.39,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000660,2,2017-12-15 00:36:55,2017-12-15 01:15:17,N,1,,,,,1,10.12,36,0.5,0.5,7.46,0,,0.3,44.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000661,2,2017-12-15 00:11:13,2017-12-15 00:26:46,N,1,,,,,3,4.16,15.5,0.5,0.5,3.36,0,,0.3,20.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000662,2,2017-12-15 00:29:43,2017-12-15 01:09:14,N,1,,,,,3,4.33,25.5,0.5,0.5,3,0,,0.3,29.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000663,2,2017-12-15 00:04:03,2017-12-15 00:18:22,N,1,,,,,2,1.95,11,0.5,0.5,0,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000664,2,2017-12-15 00:23:19,2017-12-15 00:25:03,N,1,,,,,2,0.55,3.5,0.5,0.5,1.2,0,,0.3,6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000665,2,2017-12-15 00:28:30,2017-12-15 00:48:26,N,1,,,,,2,3.86,16.5,0.5,0.5,2,0,,0.3,19.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000666,2,2017-12-15 00:54:59,2017-12-16 00:48:26,N,1,,,,,2,6.51,27,0.5,0.5,0,0,,0.3,28.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000667,1,2017-12-15 00:00:19,2017-12-15 00:07:29,N,1,,,,,1,1.30,6.5,0.5,0.5,0.5,0,,0.3,8.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000668,1,2017-12-15 00:14:16,2017-12-15 00:24:38,N,1,,,,,1,1.70,8.5,0.5,0.5,1.95,0,,0.3,11.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000669,1,2017-12-15 00:28:03,2017-12-15 00:48:26,N,1,,,,,1,3.70,15.5,0.5,0.5,1.5,0,,0.3,18.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000670,2,2017-12-15 00:04:25,2017-12-15 00:10:06,N,1,,,,,1,0.92,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000671,2,2017-12-15 00:11:22,2017-12-15 00:36:07,N,1,,,,,1,4.65,20,0.5,0.5,5.32,0,,0.3,26.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000672,2,2017-12-15 00:49:56,2017-12-15 00:56:19,N,1,,,,,1,1.85,7.5,0.5,0.5,1.76,0,,0.3,10.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000673,1,2017-12-15 00:54:32,2017-12-15 01:06:37,N,1,,,,,2,4.40,15,0.5,0.5,0,0,,0.3,16.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000674,2,2017-12-15 00:10:56,2017-12-15 00:49:42,N,2,,,,,1,19.22,52,0,0.5,0,5.76,,0.3,58.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000675,2,2017-12-15 00:53:14,2017-12-15 01:10:10,N,1,,,,,1,2.44,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000676,1,2017-12-15 00:24:30,2017-12-15 00:27:49,N,1,,,,,1,0.80,5,0.5,0.5,1,0,,0.3,7.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000677,1,2017-12-15 00:35:28,2017-12-15 00:43:21,N,1,,,,,1,1.20,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000678,1,2017-12-15 00:45:35,2017-12-15 00:55:48,N,1,,,,,1,1.60,8.5,0.5,0.5,2.9,0,,0.3,12.7,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000679,2,2017-12-15 00:07:29,2017-12-15 00:38:34,N,1,,,,,1,13.41,39.5,0.5,0.5,9.31,5.76,,0.3,55.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000680,2,2017-12-15 00:48:16,2017-12-15 01:03:17,N,1,,,,,1,3.08,13.5,0.5,0.5,0,0,,0.3,14.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000681,1,2017-12-15 00:22:27,2017-12-15 00:38:33,N,1,,,,,1,1.20,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000682,1,2017-12-15 00:41:29,2017-12-15 00:46:11,N,1,,,,,1,0.80,5.5,0.5,0.5,12.3,0,,0.3,19.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000683,1,2017-12-15 00:57:43,2017-12-15 01:22:11,N,1,,,,,1,6.00,21.5,0.5,0.5,0,0,,0.3,22.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000684,2,2017-12-15 00:36:07,2017-12-15 01:08:34,N,1,,,,,1,5.41,22.5,0.5,0.5,0,0,,0.3,23.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000685,2,2017-12-15 00:32:43,2017-12-15 00:38:26,N,1,,,,,1,1.00,6.5,0.5,0.5,1,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000686,2,2017-12-15 00:57:05,2017-12-15 01:05:25,N,1,,,,,1,1.67,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000687,2,2017-12-15 00:19:35,2017-12-15 00:31:13,N,1,,,,,1,2.63,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000688,1,2017-12-15 00:04:18,2017-12-15 00:12:26,N,1,,,,,1,0.90,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000689,1,2017-12-15 00:14:22,2017-12-15 00:59:18,N,1,,,,,1,6.60,29.5,0.5,0.5,6.15,0,,0.3,36.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000690,1,2017-12-15 00:00:52,2017-12-15 00:49:27,N,1,,,,,1,8.70,40,0.5,0.5,0,5.76,,0.3,47.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000691,1,2017-12-15 00:17:13,2017-12-15 00:37:07,N,1,,,,,1,3.10,14.5,0.5,0.5,3.15,0,,0.3,18.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000692,1,2017-12-15 00:38:56,2017-12-15 00:49:41,N,1,,,,,1,1.10,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000693,1,2017-12-15 00:52:01,2017-12-15 01:17:31,N,1,,,,,1,4.70,19.5,0.5,0.5,4,0,,0.3,24.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000694,2,2017-12-15 00:03:29,2017-12-15 00:03:31,N,5,,,,,1,0.06,75,0,0,15.06,0,,0.3,90.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000873,1,2017-12-15 00:48:42,2017-12-15 00:51:50,Y,1,,,,,1,0.40,4,0.5,0.5,0.79,0,,0.3,6.09,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000695,2,2017-12-15 00:22:23,2017-12-15 00:29:43,N,1,,,,,2,1.23,6.5,0.5,0.5,2.34,0,,0.3,10.14,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000696,2,2017-12-15 00:44:53,2017-12-15 00:48:01,N,1,,,,,2,0.38,4,0.5,0.5,1,0,,0.3,6.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000697,1,2017-12-15 00:19:26,2017-12-15 00:28:01,N,1,,,,,1,1.40,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000698,1,2017-12-15 00:39:46,2017-12-15 01:14:14,N,1,,,,,1,7.80,28,0.5,0.5,3,0,,0.3,32.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000699,1,2017-12-15 00:16:44,2017-12-15 00:45:52,N,2,,,,,3,17.30,52,0,0.5,0,5.76,,0.3,58.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000700,2,2017-12-15 00:02:11,2017-12-15 00:29:09,N,1,,,,,1,6.58,24,0.5,0.5,5.06,0,,0.3,30.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000701,2,2017-12-15 00:30:14,2017-12-15 00:46:04,N,1,,,,,1,2.80,12.5,0.5,0.5,0,0,,0.3,13.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000702,1,2017-12-15 00:10:46,2017-12-15 00:40:01,N,1,,,,,1,9.90,31,0.5,0.5,7.6,5.76,,0.3,45.66,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000703,1,2017-12-15 00:48:41,2017-12-15 01:06:29,N,1,,,,,1,3.50,15,0.5,0.5,4.05,0,,0.3,20.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000704,2,2017-12-15 00:22:48,2017-12-15 00:42:03,N,1,,,,,1,3.05,15,0.5,0.5,2.44,0,,0.3,18.74,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000705,2,2017-12-15 00:46:30,2017-12-15 01:13:28,N,1,,,,,1,3.78,19,0.5,0.5,5.08,0,,0.3,25.38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000706,2,2017-12-15 00:01:57,2017-12-15 00:25:18,N,1,,,,,1,4.73,19.5,0.5,0.5,6.24,0,,0.3,27.04,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000707,2,2017-12-15 00:01:06,2017-12-15 00:05:42,N,1,,,,,2,1.25,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000708,2,2017-12-15 00:28:01,2017-12-15 00:35:18,N,1,,,,,2,1.25,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000709,2,2017-12-15 00:36:39,2017-12-15 00:50:41,N,1,,,,,2,3.16,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000710,2,2017-12-15 00:02:19,2017-12-15 00:22:36,N,1,,,,,1,3.68,15.5,0.5,0.5,3.36,0,,0.3,22.11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000711,2,2017-12-15 00:23:44,2017-12-15 00:33:57,N,1,,,,,1,1.77,9,0.5,0.5,2.58,0,,0.3,12.88,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000712,2,2017-12-15 00:37:09,2017-12-15 00:56:53,N,1,,,,,1,2.92,15,0.5,0.5,2,0,,0.3,18.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000713,2,2017-12-15 00:12:13,2017-12-15 00:29:34,N,1,,,,,1,12.02,33,0.5,0.5,8.58,0,,0.3,42.88,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000714,2,2017-12-15 00:11:36,2017-12-15 00:40:50,N,1,,,,,3,5.63,23,0.5,0.5,4.86,0,,0.3,29.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000715,2,2017-12-15 00:43:02,2017-12-15 00:49:08,N,1,,,,,3,1.68,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000716,1,2017-12-15 00:05:41,2017-12-15 00:32:29,N,1,,,,,2,4.20,20,0.5,0.5,5.3,0,,0.3,26.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000717,1,2017-12-15 00:44:27,2017-12-15 01:04:20,N,1,,,,,1,4.90,17,0.5,0.5,5.45,0,,0.3,23.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000718,1,2017-12-15 00:30:44,2017-12-15 00:40:04,N,1,,,,,1,1.50,8.5,0.5,0.5,1.5,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000719,1,2017-12-15 00:41:48,2017-12-15 00:50:54,N,1,,,,,1,1.50,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000720,1,2017-12-15 00:59:42,2017-12-15 01:12:12,N,1,,,,,1,4.20,13.5,0.5,0.5,2,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000721,2,2017-12-15 00:08:17,2017-12-15 00:23:49,N,1,,,,,2,6.81,21.5,0.5,0.5,3.5,0,,0.3,26.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000722,2,2017-12-15 00:54:07,2017-12-15 01:12:23,N,1,,,,,2,8.71,25,0.5,0.5,0,5.76,,0.3,32.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000723,1,2017-12-15 00:10:46,2017-12-15 00:23:46,N,1,,,,,1,1.80,10.5,0.5,0.5,1,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000724,1,2017-12-15 00:25:04,2017-12-15 00:40:45,N,1,,,,,1,3.10,13.5,0.5,0.5,3.7,0,,0.3,18.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000725,1,2017-12-15 00:46:29,2017-12-15 01:22:33,N,1,,,,,1,5.80,26.5,0.5,0.5,5.55,0,,0.3,33.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000726,2,2017-12-15 00:07:51,2017-12-15 00:14:32,N,1,,,,,1,0.81,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000727,2,2017-12-15 00:17:00,2017-12-15 00:37:28,N,1,,,,,1,1.28,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000728,2,2017-12-15 00:19:24,2017-12-15 00:34:41,N,1,,,,,1,4.52,15.5,0.5,0.5,0,0,,0.3,16.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000729,2,2017-12-15 00:01:16,2017-12-15 00:17:43,N,1,,,,,1,1.05,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000730,2,2017-12-15 00:18:40,2017-12-15 00:31:35,N,1,,,,,1,1.22,9.5,0.5,0.5,0,0,,0.3,10.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000731,1,2017-12-15 00:31:52,2017-12-15 00:38:45,N,1,,,,,1,1.30,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000732,1,2017-12-15 00:18:16,2017-12-15 00:43:11,N,1,,,,,1,7.00,25,0.5,0.5,7,0,,0.3,33.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000733,2,2017-12-15 00:05:53,2017-12-15 00:35:02,N,1,,,,,1,6.85,26,0.5,0.5,0,0,,0.3,27.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000734,2,2017-12-15 00:53:47,2017-12-15 01:24:10,N,1,,,,,1,8.55,29.5,0.5,0.5,3,5.76,,0.3,39.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000735,1,2017-12-15 00:18:57,2017-12-15 00:36:41,N,1,,,,,1,2.40,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000736,1,2017-12-15 00:39:32,2017-12-15 00:51:27,N,1,,,,,1,2.20,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000737,1,2017-12-15 00:09:35,2017-12-15 00:10:30,N,4,,,,,0,0.00,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000738,1,2017-12-15 00:10:38,2017-12-15 00:50:50,N,2,,,,,2,18.30,52,0,0.5,10,0,,0.3,62.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000739,1,2017-12-15 00:59:40,2017-12-15 01:12:42,N,1,,,,,1,1.90,10,0.5,0.5,2.25,0,,0.3,13.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000740,2,2017-12-15 00:10:27,2017-12-15 00:32:38,N,1,,,,,1,15.36,41.5,0.5,0.5,8.56,0,,0.3,51.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000741,2,2017-12-15 00:43:57,2017-12-15 01:14:33,N,1,,,,,1,7.44,28,0.5,0.5,8.76,5.76,,0.3,43.82,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000742,1,2017-12-15 00:23:14,2017-12-15 00:40:55,N,1,,,,,1,3.10,14,0.5,0.5,1.5,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000743,1,2017-12-15 00:45:14,2017-12-15 01:06:11,N,1,,,,,1,4.60,19.5,0.5,0.5,4.15,0,,0.3,24.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000744,1,2017-12-15 00:05:13,2017-12-15 00:31:01,N,1,,,,,1,4.90,20.5,0.5,0.5,4.35,0,,0.3,26.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000745,1,2017-12-15 00:55:03,2017-12-15 01:00:44,N,1,,,,,1,0.90,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000746,1,2017-12-15 00:08:52,2017-12-15 00:44:05,N,1,,,,,1,3.80,23,0.5,0.5,6.05,0,,0.3,30.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000747,2,2017-12-15 00:33:05,2017-12-15 00:48:27,N,1,,,,,1,2.99,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000748,2,2017-12-15 00:15:29,2017-12-15 00:54:51,N,1,,,,,1,4.36,26,0.5,0.5,5.46,0,,0.3,32.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000749,1,2017-12-15 00:01:36,2017-12-15 00:04:58,N,1,,,,,1,0.80,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000750,1,2017-12-15 00:15:24,2017-12-15 00:21:09,N,1,,,,,1,1.80,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000751,1,2017-12-15 00:40:23,2017-12-15 00:46:29,N,1,,,,,1,1.00,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000752,1,2017-12-15 00:49:17,2017-12-15 01:00:42,N,1,,,,,1,1.30,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000753,2,2017-12-15 00:11:14,2017-12-15 00:23:43,N,1,,,,,3,2.00,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000754,2,2017-12-15 00:31:31,2017-12-15 00:45:43,N,1,,,,,2,2.13,11.5,0.5,0.5,1,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000755,2,2017-12-15 00:40:55,2017-12-15 01:00:47,N,1,,,,,1,2.88,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000756,2,2017-12-15 00:08:22,2017-12-15 00:10:21,N,1,,,,,1,0.68,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000757,2,2017-12-15 00:11:45,2017-12-15 00:32:57,N,1,,,,,1,4.32,18,0.5,0.5,3.86,0,,0.3,23.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000758,2,2017-12-15 00:45:14,2017-12-15 01:04:07,N,1,,,,,1,4.99,18,0.5,0.5,5.79,0,,0.3,25.09,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000759,1,2017-12-15 00:31:29,2017-12-15 00:31:58,N,5,,,,,1,0.00,18,0,0,3.65,0,,0.3,21.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000760,1,2017-12-15 00:34:33,2017-12-15 00:39:54,N,1,,,,,1,1.60,7,0.5,0.5,2.05,0,,0.3,10.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000761,1,2017-12-15 00:42:43,2017-12-15 00:51:05,N,1,,,,,1,1.50,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000762,1,2017-12-15 00:17:16,2017-12-15 00:26:19,N,1,,,,,1,2.30,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000763,1,2017-12-15 00:35:40,2017-12-15 00:49:25,N,1,,,,,1,3.20,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000764,2,2017-12-15 00:23:04,2017-12-15 00:42:31,N,1,,,,,1,8.98,26.5,0.5,0.5,2,0,,0.3,29.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000765,1,2017-12-15 00:46:24,2017-12-15 01:01:21,N,1,,,,,1,2.80,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000766,2,2017-12-14 23:57:15,2017-12-15 00:15:42,N,1,,,,,1,5.55,20,0.5,0.5,4.26,0,,0.3,25.56,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000767,2,2017-12-15 00:23:19,2017-12-15 00:34:54,N,1,,,,,1,1.26,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000768,2,2017-12-15 00:41:52,2017-12-15 00:48:13,N,1,,,,,1,0.82,6,0.5,0.5,1.82,0,,0.3,9.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000769,2,2017-12-15 00:49:53,2017-12-15 01:02:43,N,1,,,,,1,2.31,11,0.5,0.5,3.69,0,,0.3,15.99,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000770,2,2017-12-15 00:29:46,2017-12-15 01:15:02,N,1,,,,,5,9.27,37.5,0.5,0.5,8.91,5.76,,0.3,53.47,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000771,2,2017-12-15 00:03:21,2017-12-15 00:16:01,N,1,,,,,5,2.26,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000772,2,2017-12-15 00:16:49,2017-12-15 00:24:40,N,1,,,,,5,1.39,7.5,0.5,0.5,1,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000773,2,2017-12-15 00:29:56,2017-12-15 00:42:19,N,1,,,,,5,1.61,9.5,0.5,0.5,2.16,0,,0.3,12.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000774,2,2017-12-15 00:43:51,2017-12-15 00:54:08,N,1,,,,,5,1.04,7,0.5,0.5,0.7,0,,0.3,9,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000775,2,2017-12-15 00:55:15,2017-12-15 00:59:45,N,1,,,,,5,0.38,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000776,2,2017-12-15 00:15:43,2017-12-15 00:43:10,N,1,,,,,3,3.34,19,0.5,0.5,4.06,0,,0.3,24.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000777,2,2017-12-15 00:54:12,2017-12-15 01:05:13,N,1,,,,,3,2.16,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000778,2,2017-12-15 00:22:21,2017-12-15 00:48:52,N,1,,,,,1,9.82,30.5,0.5,0.5,0,0,,0.3,31.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000779,2,2017-12-15 00:53:35,2017-12-15 00:55:32,N,1,,,,,1,0.36,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000780,2,2017-12-15 00:33:11,2017-12-15 00:39:57,N,1,,,,,1,1.35,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000781,2,2017-12-15 00:40:32,2017-12-15 00:59:34,N,1,,,,,1,4.29,16,0.5,0.5,0,0,,0.3,17.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000782,2,2017-12-15 00:21:31,2017-12-15 00:25:39,N,1,,,,,1,0.27,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000783,2,2017-12-15 00:22:28,2017-12-15 00:57:26,N,1,,,,,1,5.58,26,0.5,0.5,1,0,,0.3,28.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000784,1,2017-12-15 00:44:31,2017-12-15 00:48:39,N,1,,,,,3,0.60,4.5,0.5,0.5,1.7,0,,0.3,7.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000785,1,2017-12-15 00:54:03,2017-12-15 01:08:58,N,1,,,,,1,2.60,12,0.5,0.5,2.65,0,,0.3,15.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000786,1,2017-12-15 00:17:08,2017-12-15 00:38:33,N,1,,,,,1,9.00,27.5,0.5,0.5,6,5.76,,0.3,40.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000787,1,2017-12-15 00:40:42,2017-12-15 00:58:49,N,1,,,,,1,3.70,15,0.5,0.5,0.02,0,,0.3,16.32,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000788,1,2017-12-15 00:07:49,2017-12-15 00:32:05,N,1,,,,,1,3.90,17.5,0.5,0.5,1.88,0,,0.3,20.68,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000789,1,2017-12-15 00:36:09,2017-12-15 00:37:14,N,1,,,,,1,0.30,3,0.5,0.5,0.85,0,,0.3,5.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000790,1,2017-12-15 00:41:57,2017-12-15 01:13:05,N,1,,,,,1,8.30,29.5,0.5,0.5,7.7,0,,0.3,38.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000791,2,2017-12-15 00:33:18,2017-12-15 00:39:28,N,1,,,,,2,1.87,8,0.5,0.5,1,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000792,2,2017-12-15 00:41:31,2017-12-15 01:14:47,N,1,,,,,2,9.43,32.5,0.5,0.5,6.76,0,,0.3,40.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000793,2,2017-12-15 00:51:28,2017-12-15 01:18:40,N,1,,,,,3,15.96,45,0.5,0.5,0,0,,0.3,46.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000794,2,2017-12-15 00:24:56,2017-12-15 00:57:40,N,1,,,,,1,5.40,23.5,0.5,0.5,0,5.76,,0.3,30.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000795,2,2017-12-15 00:50:36,2017-12-15 01:11:29,N,1,,,,,1,14.03,38,0.5,0.5,7.86,0,,0.3,47.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000796,2,2017-12-15 00:16:48,2017-12-15 00:25:42,N,1,,,,,1,3.31,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000797,2,2017-12-15 00:28:58,2017-12-15 00:45:19,N,1,,,,,1,2.56,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000798,2,2017-12-15 00:47:35,2017-12-15 01:25:59,N,1,,,,,1,5.91,26,0.5,0.5,2,5.76,,0.3,35.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000799,1,2017-12-15 00:04:08,2017-12-15 00:15:31,N,1,,,,,1,1.90,9,0.5,0.5,2.55,0,,0.3,12.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000800,1,2017-12-15 00:17:18,2017-12-15 00:31:04,N,1,,,,,1,2.50,11.5,0.5,0.5,1.5,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000801,2,2017-12-15 00:17:03,2017-12-15 00:54:29,N,1,,,,,2,6.26,25.5,0.5,0.5,0,0,,0.3,26.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000802,1,2017-12-15 00:18:30,2017-12-15 00:32:04,N,1,,,,,2,1.40,9.5,0.5,0.5,1,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000803,1,2017-12-15 00:40:55,2017-12-15 00:52:32,N,1,,,,,2,1.70,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000804,1,2017-12-15 00:58:53,2017-12-15 01:14:54,N,1,,,,,1,3.00,13,0.5,0.5,3.55,0,,0.3,17.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000805,1,2017-12-15 00:21:00,2017-12-15 00:39:48,N,1,,,,,1,4.40,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000806,1,2017-12-15 00:41:50,2017-12-15 00:54:39,N,1,,,,,1,5.00,16.5,0.5,0.5,0,0,,0.3,17.8,4,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000807,1,2017-12-15 00:16:11,2017-12-15 00:23:24,N,1,,,,,1,1.10,6.5,0.5,0.5,1.4,0,,0.3,9.2,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000808,1,2017-12-15 00:36:58,2017-12-15 01:23:38,N,1,,,,,3,4.50,29.5,0.5,0.5,6.15,0,,0.3,36.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000809,1,2017-12-15 00:02:18,2017-12-15 00:27:13,N,1,,,,,1,5.20,20,0.5,0.5,4.25,0,,0.3,25.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000810,1,2017-12-15 00:45:21,2017-12-15 00:53:02,N,1,,,,,1,1.40,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000811,1,2017-12-15 00:54:15,2017-12-15 01:08:34,N,1,,,,,1,2.90,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000812,1,2017-12-15 00:05:06,2017-12-15 00:10:53,N,1,,,,,1,1.60,7,0.5,0.5,3,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000813,1,2017-12-15 00:13:19,2017-12-15 00:26:11,N,1,,,,,1,2.40,11,0.5,0.5,2.45,0,,0.3,14.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000814,1,2017-12-15 00:56:51,2017-12-15 01:03:21,N,1,,,,,1,0.50,6,0.5,0.5,0,0,,0.3,7.3,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000815,1,2017-12-15 00:05:46,2017-12-15 00:43:24,N,1,,,,,1,3.20,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000816,1,2017-12-15 00:49:45,2017-12-15 01:01:15,N,1,,,,,1,1.80,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000817,1,2017-12-15 00:06:47,2017-12-15 00:19:20,N,1,,,,,1,2.60,11,0.5,0.5,2.45,0,,0.3,14.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000818,1,2017-12-15 00:21:08,2017-12-15 00:39:13,N,1,,,,,1,4.20,16,0.5,0.5,2,0,,0.3,19.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000819,1,2017-12-15 00:41:09,2017-12-15 00:46:56,N,1,,,,,1,0.70,5.5,0.5,0.5,1.35,0,,0.3,8.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000820,1,2017-12-15 00:49:48,2017-12-15 00:56:43,N,1,,,,,4,0.80,6,0.5,0.5,1.8,0,,0.3,9.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000821,1,2017-12-15 00:57:42,2017-12-15 01:16:36,N,1,,,,,1,3.00,14,0.5,0.5,2.5,0,,0.3,17.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000822,1,2017-12-15 00:04:26,2017-12-15 00:19:40,N,1,,,,,2,1.40,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000823,1,2017-12-15 00:26:21,2017-12-15 00:34:11,N,1,,,,,1,1.10,7,0.5,0.5,2,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000824,1,2017-12-15 00:39:20,2017-12-15 00:56:50,N,1,,,,,1,5.00,17,0.5,0.5,3.65,0,,0.3,21.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000825,2,2017-12-15 00:12:02,2017-12-15 00:26:03,N,1,,,,,1,2.27,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000826,2,2017-12-15 00:32:43,2017-12-15 00:43:57,N,1,,,,,1,3.13,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000827,2,2017-12-15 00:45:55,2017-12-15 00:54:31,N,1,,,,,1,1.89,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000828,2,2017-12-15 00:04:28,2017-12-15 00:48:05,N,1,,,,,1,7.08,31,0.5,0.5,6.46,0,,0.3,38.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000829,1,2017-12-15 00:01:24,2017-12-15 00:38:04,N,1,,,,,2,6.60,27.5,0.5,0.5,5.75,0,,0.3,34.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000830,1,2017-12-15 00:17:14,2017-12-15 00:26:11,N,1,,,,,1,2.00,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000831,1,2017-12-15 00:30:59,2017-12-15 00:47:45,N,1,,,,,1,3.00,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000832,1,2017-12-15 00:51:48,2017-12-15 01:05:41,N,1,,,,,1,6.70,20.5,0.5,0.5,4.35,0,,0.3,26.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000833,2,2017-12-15 00:04:55,2017-12-15 00:17:37,N,1,,,,,1,2.77,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000834,2,2017-12-15 00:27:10,2017-12-15 00:40:45,N,1,,,,,1,4.32,15,0.5,0.5,3.26,0,,0.3,19.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000835,2,2017-12-15 00:47:01,2017-12-15 01:00:18,N,1,,,,,1,1.12,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000836,2,2017-12-15 00:34:54,2017-12-15 00:51:54,N,1,,,,,1,1.76,11.5,0.5,0.5,2.56,0,,0.3,15.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000837,1,2017-12-15 00:41:18,2017-12-15 00:54:16,N,1,,,,,1,2.70,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000838,1,2017-12-15 00:16:37,2017-12-15 00:25:14,N,1,,,,,1,1.00,7.5,0.5,0.5,2.2,0,,0.3,11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000839,1,2017-12-15 00:29:29,2017-12-15 00:35:48,N,1,,,,,1,0.80,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000840,1,2017-12-15 00:44:11,2017-12-15 00:54:00,N,1,,,,,1,1.30,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000841,2,2017-12-15 00:03:15,2017-12-15 00:12:39,N,1,,,,,1,1.44,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000842,2,2017-12-15 00:13:34,2017-12-15 00:25:05,N,1,,,,,1,0.70,8,0.5,0.5,0,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000843,2,2017-12-15 00:29:16,2017-12-15 00:32:52,N,1,,,,,1,0.63,4.5,0.5,0.5,1.74,0,,0.3,7.54,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000844,2,2017-12-15 00:44:44,2017-12-15 00:55:18,N,1,,,,,1,2.22,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000845,2,2017-12-15 00:56:56,2017-12-15 01:08:12,N,1,,,,,1,2.25,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000846,1,2017-12-15 00:05:27,2017-12-15 00:40:39,N,1,,,,,1,4.10,23,0.5,0.5,3,0,,0.3,27.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000847,1,2017-12-15 00:50:51,2017-12-15 00:55:54,N,1,,,,,1,1.00,5.5,0.5,0.5,1.35,0,,0.3,8.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000848,1,2017-12-15 00:59:52,2017-12-15 01:11:28,N,1,,,,,2,1.30,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000849,2,2017-12-15 00:43:01,2017-12-15 01:04:17,N,1,,,,,1,4.68,18,0.5,0.5,2.5,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000850,1,2017-12-15 00:52:23,2017-12-15 01:16:27,N,1,,,,,1,4.00,17.5,0.5,0.5,1.2,0,,0.3,20,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000851,1,2017-12-15 00:50:17,2017-12-15 01:18:00,N,1,,,,,1,7.70,26,0.5,0.5,5.45,0,,0.3,32.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000852,2,2017-12-15 00:29:48,2017-12-15 00:47:32,N,1,,,,,5,4.13,15.5,0.5,0.5,4.2,0,,0.3,21,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000853,1,2017-12-15 00:20:05,2017-12-15 00:28:06,N,1,,,,,1,0.60,6.5,0.5,0.5,2,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000854,1,2017-12-15 00:30:22,2017-12-15 01:10:08,N,1,,,,,1,5.80,27,0.5,0.5,7.05,0,,0.3,35.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000855,2,2017-12-15 00:30:08,2017-12-15 00:36:48,N,1,,,,,1,0.72,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000856,2,2017-12-15 00:38:28,2017-12-15 00:57:44,N,1,,,,,1,1.65,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000857,1,2017-12-15 00:11:26,2017-12-15 00:33:14,N,1,,,,,1,4.30,18.5,0.5,0.5,2,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000858,1,2017-12-15 00:53:08,2017-12-15 01:24:53,N,1,,,,,2,8.90,31,0.5,0.5,5,5.76,,0.3,43.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000859,1,2017-12-15 00:53:46,2017-12-15 01:07:14,N,1,,,,,2,1.90,10.5,0.5,0.5,1.2,0,,0.3,13,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000860,2,2017-12-15 00:31:50,2017-12-15 00:37:17,N,1,,,,,1,1.77,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000861,1,2017-12-15 00:08:40,2017-12-15 00:58:36,N,1,,,,,1,19.60,59.5,0.5,0.5,13,0,,0.3,73.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000862,2,2017-12-15 00:02:27,2017-12-15 00:06:04,N,1,,,,,5,0.60,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000863,2,2017-12-15 00:17:42,2017-12-15 01:05:29,N,1,,,,,5,6.86,30,0.5,0.5,0,0,,0.3,31.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000864,2,2017-12-15 00:22:02,2017-12-15 00:40:51,N,1,,,,,1,3.48,15.5,0.5,0.5,0,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000865,2,2017-12-15 00:42:18,2017-12-15 00:49:35,N,1,,,,,1,1.42,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000866,2,2017-12-15 00:02:19,2017-12-15 00:24:09,N,1,,,,,1,10.38,31,0.5,0.5,6.46,0,,0.3,38.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000867,2,2017-12-15 00:21:42,2017-12-15 01:25:06,N,1,,,,,1,11.80,45.5,0.5,0.5,0,0,,0.3,46.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000868,2,2017-12-15 00:03:13,2017-12-15 00:11:15,N,1,,,,,1,1.05,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000869,2,2017-12-15 00:13:04,2017-12-15 00:43:07,N,1,,,,,1,5.54,22,0.5,0.5,0,0,,0.3,23.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000870,2,2017-12-15 00:03:01,2017-12-15 00:17:43,N,1,,,,,1,1.14,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000871,2,2017-12-15 00:19:26,2017-12-15 00:36:55,N,1,,,,,1,3.51,15,0.5,0.5,20,0,,0.3,36.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000872,1,2017-12-15 00:42:05,2017-12-15 00:46:23,Y,1,,,,,2,0.70,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000874,1,2017-12-15 00:32:08,2017-12-15 00:38:53,N,1,,,,,1,0.60,6,0.5,0.5,0,0,,0.3,7.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000875,1,2017-12-15 00:52:04,2017-12-15 01:04:53,N,1,,,,,1,2.00,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000876,2,2017-12-15 00:39:40,2017-12-15 01:22:10,N,2,,,,,6,21.28,52,0,0.5,6,5.76,,0.3,64.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000877,1,2017-12-15 00:04:14,2017-12-15 00:13:52,N,1,,,,,2,2.10,9.5,0.5,0.5,1.2,0,,0.3,12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000878,1,2017-12-15 00:44:57,2017-12-15 00:57:59,N,1,,,,,2,1.40,9.5,0.5,0.5,2,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000879,2,2017-12-15 00:12:01,2017-12-15 00:18:24,N,1,,,,,1,1.32,7,0.5,0.5,0.1,0,,0.3,8.4,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000880,2,2017-12-15 00:22:58,2017-12-15 00:40:06,N,1,,,,,1,4.69,16.5,0.5,0.5,3.56,0,,0.3,21.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000881,2,2017-12-15 00:47:43,2017-12-15 00:54:11,N,1,,,,,1,1.30,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000882,2,2017-12-15 00:16:46,2017-12-15 00:24:04,N,1,,,,,1,1.86,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000883,2,2017-12-15 00:29:13,2017-12-15 00:43:55,N,1,,,,,1,2.95,12.5,0.5,0.5,2.2,0,,0.3,16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000884,2,2017-12-15 00:48:04,2017-12-15 00:49:49,N,1,,,,,1,0.34,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000885,2,2017-12-15 00:51:53,2017-12-15 00:55:13,N,1,,,,,1,0.86,5,0.5,0.5,1.89,0,,0.3,8.19,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000886,2,2017-12-15 00:21:27,2017-12-15 01:14:34,N,1,,,,,1,10.62,42.5,0.5,0.5,10.95,0,,0.3,54.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000887,1,2017-12-15 00:04:08,2017-12-15 00:36:58,N,1,,,,,2,5.10,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000888,1,2017-12-15 00:40:51,2017-12-15 00:57:22,N,1,,,,,2,3.10,14,0.5,0.5,3.8,0,,0.3,19.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000889,1,2017-12-15 00:08:02,2017-12-15 00:29:01,N,1,,,,,1,2.50,14.5,0.5,0.5,1,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000890,1,2017-12-15 00:48:54,2017-12-15 01:33:00,N,1,,,,,1,15.60,48.5,0.5,0.5,5.2,0,,0.3,55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000891,2,2017-12-14 23:59:19,2017-12-15 00:23:03,N,1,,,,,1,3.84,17.5,0.5,0.5,1,0,,0.3,19.8,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000892,2,2017-12-15 00:42:48,2017-12-15 01:03:15,N,1,,,,,1,2.45,14.5,0.5,0.5,1,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000893,2,2017-12-15 00:16:33,2017-12-15 00:26:08,N,1,,,,,1,1.67,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000894,2,2017-12-15 00:38:41,2017-12-15 00:56:22,N,1,,,,,1,2.95,13.5,0.5,0.5,0,0,,0.3,14.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000895,2,2017-12-15 00:57:11,2017-12-15 01:14:48,N,1,,,,,1,0.82,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000896,2,2017-12-15 00:02:45,2017-12-15 00:36:35,N,2,,,,,1,18.85,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000897,2,2017-12-15 00:43:23,2017-12-15 01:00:33,N,1,,,,,1,2.82,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000898,2,2017-12-15 00:08:07,2017-12-15 00:41:12,N,1,,,,,1,5.36,24,0.5,0.5,6.32,0,,0.3,31.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000899,2,2017-12-15 00:48:01,2017-12-15 00:54:37,N,1,,,,,1,0.87,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000900,2,2017-12-15 01:00:05,2017-12-15 01:04:23,N,1,,,,,1,1.04,5.5,0.5,0.5,2.04,0,,0.3,8.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000901,1,2017-12-15 00:43:08,2017-12-15 00:50:53,N,1,,,,,1,2.10,8.5,0.5,0.5,1.95,0,,0.3,11.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000902,1,2017-12-15 00:51:34,2017-12-15 00:54:56,N,1,,,,,1,1.10,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000903,2,2017-12-15 00:53:48,2017-12-15 01:04:17,N,1,,,,,1,1.66,9,0.5,0.5,0,0,,0.3,10.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000904,1,2017-12-15 00:22:51,2017-12-15 00:50:12,N,2,,,,,1,18.40,52,0,0.5,10.55,0,,0.3,63.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000905,1,2017-12-15 00:02:29,2017-12-15 00:27:44,N,1,,,,,1,4.20,18,0.5,0.5,0,0,,0.3,19.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000906,1,2017-12-15 00:31:33,2017-12-15 00:50:28,N,1,,,,,1,3.40,15,0.5,0.5,4.05,0,,0.3,20.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000907,1,2017-12-15 00:53:36,2017-12-15 00:57:11,N,1,,,,,1,0.90,5,0.5,0.5,1.25,0,,0.3,7.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000908,2,2017-12-15 00:01:47,2017-12-15 00:35:22,N,1,,,,,1,6.78,26.5,0.5,0.5,2,0,,0.3,29.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000909,2,2017-12-15 00:55:50,2017-12-15 01:21:26,N,1,,,,,1,4.70,19,0.5,0.5,0,0,,0.3,20.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000910,1,2017-12-15 00:15:27,2017-12-15 00:20:38,N,1,,,,,1,1.40,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000911,1,2017-12-15 00:34:52,2017-12-15 01:01:21,N,1,,,,,1,5.10,21,0.5,0.5,3.35,0,,0.3,25.65,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000912,1,2017-12-15 00:15:57,2017-12-15 00:22:46,N,1,,,,,1,1.80,7.5,0.5,0.5,1.75,0,,0.3,10.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000913,1,2017-12-15 00:31:27,2017-12-15 01:11:20,N,1,,,,,1,5.30,26.5,0.5,0.5,1,0,,0.3,28.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000914,1,2017-12-15 00:14:15,2017-12-15 00:29:52,N,1,,,,,1,3.00,13,0.5,0.5,1,0,,0.3,15.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000915,1,2017-12-15 00:48:36,2017-12-15 01:24:10,N,1,,,,,1,4.90,26,0.5,0.5,8.15,0,,0.3,35.45,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000916,1,2017-12-15 00:22:04,2017-12-15 00:32:41,Y,1,,,,,2,1.90,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000917,1,2017-12-15 00:33:58,2017-12-15 00:47:41,N,1,,,,,1,3.30,12.5,0.5,0.5,0,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000918,1,2017-12-15 00:54:47,2017-12-15 00:55:21,N,5,,,,,1,0.00,14,0,0,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000919,1,2017-12-15 00:35:43,2017-12-15 01:08:42,N,1,,,,,1,11.70,37.5,0.5,0.5,9.7,0,,0.3,48.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000920,2,2017-12-15 00:13:56,2017-12-15 00:41:08,N,1,,,,,1,3.73,19,0.5,0.5,3,0,,0.3,23.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000921,2,2017-12-15 00:50:32,2017-12-15 01:06:42,N,1,,,,,1,2.81,12,0.5,0.5,2.66,0,,0.3,15.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000922,1,2017-12-15 00:39:58,2017-12-15 01:01:50,N,1,,,,,1,4.50,18,0.5,0.5,3.85,0,,0.3,23.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000923,1,2017-12-15 00:08:23,2017-12-15 01:07:29,N,1,,,,,1,7.70,38,0.5,0.5,0,0,,0.3,39.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000924,1,2017-12-15 00:00:04,2017-12-15 00:14:22,N,1,,,,,2,2.30,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000925,1,2017-12-15 00:15:32,2017-12-15 00:39:24,N,1,,,,,2,3.10,16.5,0.5,0.5,2,0,,0.3,19.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000926,1,2017-12-15 00:58:47,2017-12-15 01:01:23,N,1,,,,,1,0.60,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000927,2,2017-12-15 00:19:03,2017-12-15 00:26:33,N,1,,,,,1,1.07,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000928,2,2017-12-15 00:27:47,2017-12-15 01:05:07,N,1,,,,,1,5.63,26,0.5,0.5,5.46,0,,0.3,32.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000929,1,2017-12-15 00:02:07,2017-12-15 00:28:16,N,1,,,,,0,6.70,23.5,0.5,0.5,0,0,,0.3,24.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000930,1,2017-12-15 00:33:18,2017-12-15 01:03:18,N,1,,,,,1,4.80,21.5,0.5,0.5,5.7,5.76,,0.3,34.26,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000931,1,2017-12-15 00:38:52,2017-12-15 00:38:57,N,2,,,,,1,4.60,52,0,0.5,0,0,,0.3,52.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000932,1,2017-12-15 00:43:53,2017-12-15 01:18:44,N,1,,,,,1,6.40,26.5,0,0.5,0,0,,0.3,27.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000933,1,2017-12-15 00:04:13,2017-12-15 00:51:06,N,1,,,,,2,9.20,36,0.5,0.5,7.45,0,,0.3,44.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000934,2,2017-12-15 00:09:49,2017-12-15 00:23:32,N,1,,,,,1,2.51,11.5,0.5,0.5,0,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000935,2,2017-12-15 00:28:33,2017-12-15 01:04:59,N,1,,,,,1,6.32,25.5,0.5,0.5,5.36,0,,0.3,32.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000936,2,2017-12-15 00:06:03,2017-12-15 00:22:31,N,1,,,,,5,3.26,14,0.5,0.5,3.06,0,,0.3,18.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000937,2,2017-12-15 00:37:08,2017-12-15 01:03:40,N,1,,,,,5,2.83,17.5,0.5,0.5,0,0,,0.3,18.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000938,2,2017-12-15 00:57:16,2017-12-15 01:16:42,N,1,,,,,1,2.93,14,0.5,0.5,1,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000939,2,2017-12-15 00:10:53,2017-12-15 00:22:45,N,1,,,,,2,1.44,9,0.5,0.5,2.5,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000940,2,2017-12-15 00:50:00,2017-12-15 01:26:28,N,1,,,,,2,10.03,35.5,0.5,0.5,0,0,,0.3,36.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000941,2,2017-12-15 00:07:21,2017-12-15 00:20:07,N,1,,,,,1,1.85,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000942,2,2017-12-15 00:29:10,2017-12-15 00:36:06,N,1,,,,,1,1.68,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000943,2,2017-12-15 00:47:04,2017-12-15 01:20:23,N,1,,,,,1,15.51,45,0.5,0.5,9.26,0,,0.3,55.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000944,1,2017-12-15 00:19:02,2017-12-15 00:29:05,N,1,,,,,1,1.90,9.5,0.5,0.5,1,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000945,1,2017-12-15 00:31:52,2017-12-15 00:44:24,N,1,,,,,1,2.20,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000946,1,2017-12-15 00:46:42,2017-12-15 01:05:12,N,1,,,,,1,6.20,21,0.5,0.5,3.35,0,,0.3,25.65,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000947,2,2017-12-15 00:11:42,2017-12-15 00:43:20,N,1,,,,,1,6.04,25.5,0.5,0.5,5.36,0,,0.3,34.11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000948,1,2017-12-15 00:21:34,2017-12-15 00:41:16,N,1,,,,,1,2.30,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000949,1,2017-12-15 00:42:11,2017-12-15 00:55:13,N,1,,,,,2,3.50,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000950,2,2017-12-15 00:00:16,2017-12-15 00:17:07,N,1,,,,,2,4.05,15.5,0.5,0.5,0,0,,0.3,16.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000951,2,2017-12-15 00:47:25,2017-12-15 01:08:08,N,1,,,,,2,4.73,18,0.5,0.5,0,0,,0.3,19.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000952,2,2017-12-14 23:58:52,2017-12-15 23:48:39,N,1,,,,,2,0.27,3.5,0.5,0.5,0,0,,0.3,4.8,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,\n1460000953,2,2017-12-15 00:02:43,2017-12-15 00:31:33,N,1,,,,,1,3.81,19.5,0.5,0.5,3,0,,0.3,23.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000954,2,2017-12-15 00:08:07,2017-12-15 00:22:27,N,1,,,,,1,1.58,10.5,0.5,0.5,1.5,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000955,2,2017-12-15 00:23:59,2017-12-15 00:43:51,N,1,,,,,1,3.09,14.5,0.5,0.5,4.74,0,,0.3,20.54,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000956,2,2017-12-15 00:53:01,2017-12-15 00:53:05,N,5,,,,,1,0.00,5,0,0.5,0,0,,0.3,5.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000957,2,2017-12-15 00:55:09,2017-12-15 01:07:32,N,1,,,,,1,2.52,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000958,2,2017-12-15 00:22:57,2017-12-15 00:36:58,N,1,,,,,1,2.75,12,0.5,0.5,0.66,0,,0.3,13.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000959,2,2017-12-15 00:03:25,2017-12-15 00:13:02,N,1,,,,,1,0.99,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000960,2,2017-12-15 00:17:06,2017-12-15 00:31:31,N,1,,,,,1,1.79,10.5,0.5,0.5,0,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000961,2,2017-12-15 00:37:42,2017-12-15 01:37:47,N,5,,,,,1,16.84,80,0,0.5,0,0,,0.3,80.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000962,1,2017-12-15 00:24:46,2017-12-15 00:34:31,N,1,,,,,1,1.60,8.5,0.5,0.5,1.95,0,,0.3,11.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000963,1,2017-12-15 00:47:28,2017-12-15 01:07:04,N,1,,,,,1,4.20,16.5,0.5,0.5,5,0,,0.3,22.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000964,2,2017-12-15 00:01:11,2017-12-15 00:09:14,N,1,,,,,1,1.57,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000965,2,2017-12-15 00:30:12,2017-12-15 01:01:05,N,1,,,,,1,4.68,22.5,0.5,0.5,4.76,0,,0.3,28.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000966,1,2017-12-15 00:24:00,2017-12-15 00:38:13,N,1,,,,,1,3.20,12.5,0.5,0.5,2.75,0,,0.3,16.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000967,2,2017-12-15 00:49:53,2017-12-15 01:17:25,N,1,,,,,3,6.37,22.5,0.5,0.5,4.76,0,,0.3,28.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000968,2,2017-12-15 00:05:30,2017-12-15 00:37:30,N,2,,,,,4,18.09,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000969,2,2017-12-15 00:47:11,2017-12-15 01:02:37,N,1,,,,,4,3.35,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000970,2,2017-12-15 00:32:31,2017-12-15 01:15:13,N,1,,,,,6,10.56,38.5,0.5,0.5,4,5.76,,0.3,49.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000971,1,2017-12-15 00:55:25,2017-12-15 01:00:03,N,1,,,,,1,0.50,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000972,1,2017-12-15 00:00:54,2017-12-15 00:04:07,N,1,,,,,1,0.90,5,0.5,0.5,1.55,0,,0.3,7.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000973,1,2017-12-15 00:11:51,2017-12-15 00:24:10,N,1,,,,,1,1.70,10,0.5,0.5,2,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000974,1,2017-12-15 00:33:26,2017-12-15 01:19:20,N,1,,,,,1,8.50,34.5,0.5,0.5,7.15,0,,0.3,42.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000975,1,2017-12-15 00:11:27,2017-12-15 00:54:09,N,1,,,,,2,7.90,31.5,0.5,0.5,6.55,0,,0.3,39.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000976,2,2017-12-15 00:10:38,2017-12-15 00:50:52,N,1,,,,,1,6.88,29.5,0.5,0.5,7.7,0,,0.3,38.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000977,2,2017-12-15 00:04:00,2017-12-15 00:21:02,N,1,,,,,6,1.51,12,0.5,0.5,3.99,0,,0.3,17.29,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000978,2,2017-12-15 00:21:43,2017-12-15 00:33:56,N,1,,,,,6,0.71,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000979,2,2017-12-15 00:36:45,2017-12-15 00:48:13,N,1,,,,,6,1.00,8.5,0.5,0.5,2.94,0,,0.3,12.74,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000980,2,2017-12-15 00:50:35,2017-12-15 01:14:51,N,1,,,,,6,4.34,18,0.5,0.5,4.82,0,,0.3,24.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000981,2,2017-12-15 00:35:10,2017-12-15 00:40:56,N,1,,,,,1,1.21,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000982,2,2017-12-15 00:53:17,2017-12-15 01:09:14,N,1,,,,,1,3.38,13.5,0.5,0.5,3.7,0,,0.3,18.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000983,2,2017-12-15 00:48:43,2017-12-15 00:58:44,N,1,,,,,1,1.34,8.5,0.5,0.5,1,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000984,2,2017-12-15 00:37:51,2017-12-15 00:38:05,N,5,,,,,1,0.00,60,0,0,0,0,,0.3,60.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000985,2,2017-12-15 00:45:54,2017-12-15 00:52:31,N,1,,,,,1,1.92,7.5,0.5,0.5,2,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000986,2,2017-12-15 00:07:43,2017-12-15 00:28:15,N,1,,,,,2,10.02,29.5,0.5,0.5,0,0,,0.3,30.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000987,2,2017-12-15 00:32:40,2017-12-15 00:41:01,N,1,,,,,2,1.29,7.5,0.5,0.5,1,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000988,2,2017-12-15 00:41:52,2017-12-15 00:48:28,N,1,,,,,2,1.00,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000989,2,2017-12-15 00:52:43,2017-12-15 01:37:15,N,5,,,,,2,4.32,70,0,0.5,0,0,,0.3,70.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000990,1,2017-12-15 00:03:36,2017-12-15 00:25:14,N,1,,,,,2,3.90,17.5,0.5,0.5,3.75,0,,0.3,22.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000991,1,2017-12-15 00:39:01,2017-12-15 01:05:04,N,1,,,,,2,4.70,19.5,0.5,0.5,3,0,,0.3,23.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000992,2,2017-12-15 00:04:21,2017-12-15 00:09:29,N,1,,,,,1,0.69,5,0.5,0.5,1.89,0,,0.3,8.19,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000993,2,2017-12-15 00:19:51,2017-12-15 00:24:23,N,1,,,,,1,0.79,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000994,2,2017-12-15 00:25:57,2017-12-15 00:55:44,N,1,,,,,1,3.62,19.5,0.5,0.5,4.16,0,,0.3,24.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000995,1,2017-12-15 00:00:00,2017-12-15 00:15:00,N,1,,,,,1,2.20,12,0.5,0.5,2.65,0,,0.3,15.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000996,1,2017-12-15 00:16:51,2017-12-15 00:49:40,N,1,,,,,2,8.50,29,0.5,0.5,7.2,5.76,,0.3,43.26,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000997,2,2017-12-15 00:42:37,2017-12-15 01:24:17,N,1,,,,,1,8.46,31.5,0.5,0.5,8.2,0,,0.3,41,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000998,2,2017-12-15 00:06:02,2017-12-15 00:18:45,N,1,,,,,1,2.28,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460000999,2,2017-12-15 00:21:04,2017-12-15 00:31:14,N,1,,,,,1,1.01,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n1460001000,2,2017-12-15 00:33:21,2017-12-15 00:42:19,N,1,,,,,1,1.87,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,\n"
},
{
"path": "examples/data/plasticc_test_set_1k.csv",
"content": "object_id,mjd,passband,flux,flux_err,detected\n13,59798.3205,2,-1.299735,1.357315,0\n13,59798.3281,1,-2.095392,1.148654,0\n13,59798.3357,3,-0.923794,1.763655,0\n13,59798.3466,4,-4.009815,2.602911,0\n13,59798.3576,5,-3.403503,5.367328,0\n13,59801.3553,2,-1.778855,2.448943,0\n13,59801.3629,1,2.491993,3.540421,0\n13,59801.3705,3,1.644129,2.284999,0\n13,59801.3815,4,-0.158192,2.515900,0\n13,59801.3924,5,-6.457387,5.381231,0\n13,59818.2740,0,1.962846,1.795587,0\n13,59819.2541,0,-1.697929,2.433431,0\n13,59820.2522,0,-1.698675,1.898612,0\n13,59821.2478,0,-0.776626,2.435191,0\n13,59822.2433,0,-3.826187,2.853957,0\n13,59823.2659,0,-0.837001,2.690573,0\n13,59826.3105,2,0.529480,0.920972,0\n13,59826.3181,1,-0.702092,0.923219,0\n13,59826.3258,3,-0.797231,1.508073,0\n13,59826.3367,4,-3.898993,2.156529,0\n13,59826.3477,5,-2.883462,5.535779,0\n13,59842.2456,2,-0.838558,0.849664,0\n13,59842.2532,1,-0.038403,0.813772,0\n13,59842.2608,3,2.863780,1.322607,0\n13,59842.2718,4,0.088068,2.166394,0\n13,59842.2827,5,-1.672899,6.068114,0\n13,59851.1792,0,-0.737309,2.574759,0\n13,59854.1485,2,0.040905,1.224270,0\n13,59854.1563,1,-0.265884,1.336087,0\n13,59854.1640,3,-0.369843,1.901700,0\n13,59854.1750,4,2.415236,2.771065,0\n13,59854.1860,5,-4.745292,5.788470,0\n13,59857.1408,2,4.814427,2.132096,0\n13,59857.1485,1,-2.618018,2.915296,0\n13,59857.1563,3,-2.852325,2.092946,0\n13,59857.1673,4,-1.584260,2.676820,0\n13,59857.1782,5,-12.249563,6.628258,0\n13,59867.1112,2,-1.164210,1.083173,0\n13,59867.1189,1,-1.176300,0.974351,0\n13,59867.1267,3,0.451480,1.538879,0\n13,59867.1377,4,0.359918,2.357359,0\n13,59867.1487,5,-2.699421,5.830664,0\n13,59870.1049,2,1.381781,0.971909,0\n13,59870.1126,1,0.387448,0.822428,0\n13,59870.1204,3,0.817313,1.373219,0\n13,59870.1314,4,0.151035,2.283455,0\n13,59870.1424,5,2.986818,5.280643,0\n13,59873.0971,2,-1.240265,1.040900,0\n13,59873.1049,1,0.931574,1.042814,0\n13,59873.1126,3,2.134935,1.442823,0\n13,59873.1236,4,-0.530458,2.068875,0\n13,59873.1346,5,-0.314489,4.587787,0\n13,59874.1461,0,-2.490195,1.611522,0\n13,59875.0995,0,-1.276708,2.019888,0\n13,59876.0980,0,3.425512,2.443817,0\n13,59877.0976,0,2.518677,1.776579,0\n13,59878.0964,0,1.872855,2.020787,0\n13,59879.0895,0,0.522355,1.864210,0\n13,59880.1017,0,-0.761489,1.371363,0\n13,59884.1760,2,0.408623,0.768234,0\n13,59884.1836,1,0.610256,0.887502,0\n13,59884.1913,3,-0.134252,1.138571,0\n13,59884.2022,4,-0.009705,1.609182,0\n13,59884.2132,5,6.633543,4.214399,0\n13,59887.2856,2,1.784338,1.826830,0\n13,59887.2933,1,1.591402,2.206727,0\n13,59887.3009,3,0.109085,2.350564,0\n13,59887.3118,4,-2.874674,3.145338,0\n13,59887.3228,5,-2.586321,7.947824,0\n13,59896.1307,2,1.084506,0.822793,0\n13,59896.1384,1,-0.561922,0.703150,0\n13,59896.1460,3,0.284255,1.393400,0\n13,59896.1569,4,0.466466,2.192684,0\n13,59896.1679,5,0.433727,5.948182,0\n13,59899.1519,2,0.102152,1.365073,0\n13,59899.1595,1,1.624047,1.388812,0\n13,59899.1672,3,1.820581,2.378815,0\n13,59899.1781,4,-3.816161,3.709328,0\n13,59899.1891,5,-8.805643,7.905914,0\n13,59902.1384,2,0.196845,1.186146,0\n13,59902.1460,1,0.993784,1.079431,0\n13,59902.1537,3,1.891725,1.943785,0\n13,59902.1646,4,-1.455820,2.762865,0\n13,59902.1755,5,-0.622898,7.024046,0\n13,59904.1053,0,-4.067552,2.281505,0\n13,59905.0555,0,0.633017,2.504281,0\n13,59906.0562,0,1.127558,2.030226,0\n13,59907.0567,0,0.456890,1.749502,0\n13,59908.0681,0,-1.925056,2.683702,0\n13,59909.0582,0,-0.678187,1.834134,0\n13,59910.0503,0,1.194135,1.869489,0\n13,59914.0526,2,-1.644144,1.438881,0\n13,59914.0602,1,-1.866351,1.956770,0\n13,59914.0678,3,-0.710913,1.686324,0\n13,59914.0788,4,5.825469,2.269697,0\n13,59914.0897,5,3.513655,5.356808,0\n13,59924.1060,2,-0.028742,1.131780,0\n13,59924.1136,1,-1.108478,1.574127,0\n13,59924.1212,3,-1.323133,1.728779,0\n13,59924.1322,4,-5.513097,2.527147,0\n13,59924.1431,5,4.266651,5.633382,0\n13,59927.1074,2,-0.400232,1.190879,0\n13,59927.1151,1,1.286833,1.135341,0\n13,59927.1227,3,2.155155,1.593354,0\n13,59927.1336,4,2.415218,2.299777,0\n13,59927.1446,5,3.778454,5.891465,0\n13,59930.1236,2,1.373614,0.942084,0\n13,59930.1312,1,-1.140287,0.828822,0\n13,59930.1388,3,0.832613,1.283629,0\n13,59930.1498,4,2.345280,2.033006,0\n13,59930.1607,5,-4.211092,5.330056,0\n13,59933.1249,2,0.395000,0.691634,0\n13,59933.1325,1,-0.423847,0.693980,0\n13,59933.1401,3,-1.108444,1.268825,0\n13,59933.1511,4,-0.328867,1.913908,0\n13,59933.1620,5,7.551563,5.429599,0\n13,59934.0638,0,-1.113758,1.141001,0\n13,59935.0646,0,-0.818122,1.122888,0\n13,59936.0642,0,2.141036,1.531045,0\n13,59937.0650,0,-1.277371,1.639949,0\n13,59938.0647,0,4.045178,2.015162,0\n13,59939.0650,0,-0.050647,2.194245,0\n13,60165.3032,2,-0.244917,1.195023,0\n13,60165.3109,1,-0.051452,1.347307,0\n13,60165.3186,3,-1.446614,1.518360,0\n13,60165.3295,4,-1.239533,2.022870,0\n13,60165.3405,5,-2.233357,4.782032,0\n13,60168.2892,2,-0.272922,0.834000,0\n13,60168.2970,1,-0.551885,0.787909,0\n13,60168.3047,3,0.215221,1.300493,0\n13,60168.3157,4,-0.354714,2.065893,0\n13,60168.3267,5,9.800998,4.821743,0\n13,60176.2820,0,-2.711282,2.363240,0\n13,60177.2726,0,2.907485,2.777568,0\n13,60181.4088,2,-0.778592,0.924176,0\n13,60181.4164,1,0.776635,1.262891,0\n13,60181.4232,3,5.557160,3.239328,0\n13,60183.2660,2,0.008992,1.915637,0\n13,60183.2736,1,3.495092,2.936934,0\n13,60183.2812,3,-4.313775,2.548015,0\n13,60183.2922,4,4.681122,2.996080,0\n13,60183.3031,5,-0.644280,6.178194,0\n13,60195.2812,2,-0.992008,0.933265,0\n13,60195.2888,1,-0.199896,1.020470,0\n13,60195.2964,3,1.175473,1.626211,0\n13,60195.3073,4,-0.928975,2.339218,0\n13,60195.3183,5,3.537798,6.276374,0\n13,60198.2690,0,0.879443,2.172960,0\n13,60199.2186,0,1.393119,2.161035,0\n13,60200.2139,0,0.226138,2.589170,0\n13,60201.2072,0,0.024500,2.490959,0\n13,60202.2089,0,-1.152337,2.483400,0\n13,60209.1811,2,-0.534034,1.565458,0\n13,60209.1888,1,0.822128,1.410793,0\n13,60209.1965,3,-0.684217,1.831758,0\n13,60209.2075,4,-3.066642,2.909104,0\n13,60209.2184,5,0.602669,6.696602,0\n13,60212.1675,2,1.117115,1.936030,0\n13,60212.1753,1,1.782217,2.752773,0\n13,60212.1830,3,0.321750,2.295181,0\n13,60212.1941,4,-1.936185,2.590936,0\n13,60212.2050,5,11.339810,5.666715,0\n13,60223.2416,2,0.123572,1.182341,0\n13,60223.2493,1,0.568967,1.241492,0\n13,60223.2569,3,0.040453,1.648758,0\n13,60223.2678,4,0.459117,2.472937,0\n13,60223.2788,5,-7.159738,5.794432,0\n13,60226.3337,2,0.034159,0.944593,0\n13,60226.3413,1,1.465278,0.946326,0\n13,60226.3489,3,0.247305,1.793391,0\n13,60226.3599,4,4.326116,3.071592,0\n13,60226.3708,5,1.558216,8.109918,0\n13,60238.3197,2,-0.738155,1.053325,0\n13,60238.3273,1,-0.990783,1.152318,0\n13,60238.3349,3,1.786266,2.171606,0\n13,60238.3459,4,3.042687,3.829919,0\n13,60238.3568,5,8.879921,11.257108,0\n13,60241.0870,2,-0.147198,1.999162,0\n13,60241.0948,1,-1.383542,3.066069,0\n13,60241.1025,3,-0.347256,2.444988,0\n13,60241.1136,4,2.242954,2.917921,0\n13,60241.1245,5,4.122897,5.941785,0\n13,60250.1708,2,2.348668,2.103043,0\n13,60250.1957,1,1.762817,2.811031,0\n13,60250.2034,3,-4.162498,2.927869,0\n13,60250.2143,4,3.563345,3.983845,0\n13,60250.2253,5,5.870506,7.937796,0\n13,60261.1296,0,2.993462,2.083503,0\n13,60262.0550,0,1.952810,2.311998,0\n13,60263.0556,0,1.620923,1.977922,0\n13,60264.0559,0,3.218953,2.187246,0\n13,60265.0780,0,-4.046216,2.407065,0\n13,60268.0449,2,1.923153,1.404630,0\n13,60268.0525,1,-0.380431,1.849085,0\n13,60268.0601,3,1.653331,2.130050,0\n13,60268.0711,4,-2.881836,2.734059,0\n13,60268.0820,5,7.530805,6.747447,0\n13,60278.0993,2,-1.574151,1.643888,0\n13,60278.1069,1,-4.148108,2.273913,0\n13,60278.1145,3,-1.136458,2.169661,0\n13,60278.1255,4,-4.712543,3.059077,0\n13,60278.1364,5,-4.843711,7.638536,0\n13,60281.1023,2,-0.579702,0.798074,0\n13,60281.1099,1,0.559643,0.795803,0\n13,60281.1175,3,0.985490,1.369862,0\n13,60281.1285,4,-0.125175,1.990803,0\n13,60281.1394,5,-6.211294,5.176151,0\n13,60284.1027,2,0.920098,0.837512,0\n13,60284.1104,1,-0.531105,0.761165,0\n13,60284.1180,3,-1.038009,1.160178,0\n13,60284.1289,4,-0.543047,1.702779,0\n13,60284.1399,5,-5.946274,4.435276,0\n13,60287.1047,2,-0.388991,1.250426,0\n13,60287.1123,1,0.622866,1.083712,0\n13,60287.1200,3,-1.365885,1.796073,0\n13,60287.1309,4,-1.566869,2.963949,0\n13,60287.1418,5,-12.680235,7.013420,0\n13,60290.0761,0,-0.996401,1.912932,0\n13,60291.0689,0,-1.639395,1.661116,0\n13,60292.0699,0,-0.087568,1.855572,0\n13,60293.0699,0,3.384405,1.578842,0\n13,60294.0708,0,0.056788,1.397227,0\n13,60532.3019,2,24.529644,1.046373,1\n13,60532.3097,1,23.404964,0.992504,1\n13,60532.3173,3,36.069386,1.568371,1\n13,60532.3282,4,42.765503,2.305008,1\n13,60532.3392,5,36.567162,5.439748,1\n13,60535.2802,2,23.662449,1.422315,1\n13,60535.2879,1,20.202259,1.361849,1\n13,60535.2957,3,39.966290,2.106815,1\n13,60535.3068,4,39.323189,3.034410,1\n13,60535.3177,5,25.412567,7.170496,1\n13,60538.2826,2,19.899044,1.856537,1\n13,60538.2903,1,14.983138,2.683123,1\n13,60538.2980,3,32.445835,1.936947,1\n13,60538.3089,4,37.566696,2.327296,1\n13,60538.3199,5,36.802090,5.025538,1\n13,60554.2651,0,2.890699,2.258298,0\n13,60555.2411,0,1.421973,2.038275,0\n13,60556.2370,0,5.816270,2.553418,0\n13,60557.2322,0,2.976124,1.849058,0\n13,60558.2332,0,3.465247,2.503530,0\n13,60559.2274,0,2.022449,2.213130,0\n13,60560.2268,0,3.503388,3.087183,0\n13,60567.3291,2,14.211590,0.945795,1\n13,60567.3368,1,8.278180,1.052801,1\n13,60567.3444,3,24.237070,1.263879,1\n13,60567.3553,4,31.986134,1.675546,1\n13,60567.3663,5,26.146296,4.267163,1\n13,60580.1736,2,10.443220,1.455966,1\n13,60580.1813,1,4.004329,1.395011,0\n13,60580.1889,3,21.601105,2.202044,1\n13,60580.1999,4,33.927616,3.210988,1\n13,60580.2108,5,31.062853,7.647927,0\n13,60582.1681,0,1.094513,2.405975,0\n13,60583.1640,0,-1.259893,2.979695,0\n13,60584.1591,0,1.617939,2.344479,0\n13,60585.1601,0,1.637716,1.874226,0\n13,60586.1564,0,0.117864,1.815223,0\n13,60587.1540,0,0.302183,2.121386,0\n13,60588.1461,0,3.094076,1.792624,0\n13,60593.1209,2,10.800429,1.153115,1\n13,60593.1287,1,4.406773,1.308495,0\n13,60593.1365,3,16.092447,1.545535,1\n13,60593.1476,4,24.343342,2.181189,1\n13,60593.1585,5,19.054943,5.111032,1\n13,60596.1351,2,12.383826,2.473228,0\n13,60596.1427,1,0.372005,3.515397,0\n13,60596.1504,3,16.204580,2.651616,1\n13,60596.1613,4,26.244993,3.235833,1\n13,60596.1723,5,20.389833,7.186456,0\n13,60605.0908,2,7.542615,0.977457,1\n13,60605.0986,1,3.585346,0.917699,0\n13,60605.1063,3,15.296724,1.463083,1\n13,60605.1174,4,21.654158,2.154012,1\n13,60605.1283,5,12.261124,5.138537,1\n13,60608.0836,2,7.212741,0.886919,1\n13,60608.0913,1,3.103603,0.823795,0\n13,60608.0991,3,15.016788,1.333418,1\n13,60608.1101,4,22.194675,1.964862,1\n13,60608.1211,5,16.844475,4.664538,0\n13,60611.0756,2,5.219585,0.772730,1\n13,60611.0833,1,2.315308,0.703447,0\n13,60611.0911,3,13.987851,1.177967,1\n13,60611.1021,4,23.238731,1.752712,1\n13,60611.1130,5,13.764248,4.170574,0\n13,60612.0813,0,-0.500034,1.440229,0\n13,60613.0818,0,1.002482,1.524789,0\n13,60614.0803,0,-1.957333,1.736466,0\n13,60615.0761,0,-1.864921,1.981273,0\n13,60616.0769,0,-0.823935,2.007182,0\n13,60617.0737,0,4.029193,1.584846,0\n13,60620.1444,0,1.135807,1.713108,0\n13,60621.2673,2,5.227184,0.942700,1\n13,60621.2749,1,0.979372,0.945305,0\n13,60621.2825,3,10.753699,1.650229,1\n13,60621.2934,4,15.025331,2.743479,1\n13,60621.3044,5,20.601212,7.516092,0\n13,60624.1290,2,4.608896,1.191301,0\n13,60624.1366,1,3.302126,1.565634,0\n13,60624.1442,3,13.692765,1.502649,1\n13,60624.1551,4,22.563152,2.025444,1\n13,60624.1661,5,17.466129,4.765408,0\n13,60633.0541,2,5.740388,1.090800,1\n13,60633.0617,1,-0.248447,1.055601,0\n13,60633.0693,3,11.721886,1.678442,1\n13,60633.0803,4,21.894281,2.489034,1\n13,60633.0912,5,8.198497,5.988672,0\n13,60636.0482,2,4.445689,1.029320,0\n13,60636.0558,1,0.510712,0.992636,0\n13,60636.0635,3,8.838101,1.589973,0\n13,60636.0744,4,18.146425,2.370814,1\n13,60636.0854,5,20.537880,5.701680,0\n13,60640.0972,2,2.708286,0.871447,0\n13,60640.1049,1,0.387127,0.847005,0\n13,60640.1125,3,11.844381,1.395034,1\n13,60640.1234,4,16.336960,2.129130,1\n13,60640.1344,5,22.784824,5.287734,0\n13,60642.0643,0,-0.045857,2.075920,0\n13,60643.0521,0,-0.339986,1.668015,0\n13,60644.0621,0,-0.061904,1.665856,0\n13,60645.0625,0,-1.683454,1.768579,0\n13,60646.0636,0,-1.090810,1.782456,0\n13,60647.0635,0,-0.917836,1.968023,0\n13,60648.0642,0,-0.471162,1.443392,0\n13,60652.1289,2,2.063019,0.939241,0\n13,60652.1365,1,0.914091,1.117558,0\n13,60652.1441,3,8.505517,1.381162,1\n13,60652.1550,4,20.247869,2.050198,1\n13,60652.1660,5,4.584575,5.200393,0\n14,59798.3205,2,14.465278,1.364599,1\n14,59798.3281,1,13.748290,1.165200,1\n14,59798.3357,3,8.555202,1.766522,1\n14,59798.3466,4,7.253281,2.603756,0\n14,59798.3576,5,1.134257,5.365180,0\n14,59801.3553,2,9.855556,2.449258,0\n14,59801.3629,1,10.256726,3.538008,0\n14,59801.3705,3,10.469539,2.286622,0\n14,59801.3815,4,4.839510,2.516017,0\n14,59801.3924,5,6.016302,5.381228,0\n14,59818.2740,0,-1.523922,1.781903,0\n14,59819.2541,0,6.043163,2.455132,0\n14,59820.2522,0,-3.440181,1.895149,0\n14,59821.2478,0,-1.079028,2.430641,0\n14,59822.2433,0,-0.827454,2.848530,0\n14,59823.2659,0,5.701578,2.706588,0\n14,59826.3105,2,2.735408,0.921596,0\n14,59826.3181,1,2.361817,0.925133,0\n14,59826.3258,3,4.882393,1.509568,1\n14,59826.3367,4,5.898125,2.157517,0\n14,59826.3477,5,7.172944,5.536250,0\n14,59842.2456,2,1.210632,0.849698,0\n14,59842.2532,1,0.480856,0.813325,0\n14,59842.2608,3,3.207553,1.321817,0\n14,59842.2718,4,4.451898,2.166812,0\n14,59842.2827,5,4.518466,6.066945,0\n14,59851.1792,0,4.186630,2.586060,0\n14,59854.1485,2,0.507158,1.223279,0\n14,59854.1563,1,1.509088,1.335596,0\n14,59854.1640,3,3.079645,1.901563,0\n14,59854.1750,4,2.476080,2.769468,0\n14,59854.1860,5,5.571058,5.787987,0\n14,59857.1408,2,2.902585,2.129320,0\n14,59857.1485,1,0.076045,2.911045,0\n14,59857.1563,3,5.179635,2.093322,0\n14,59857.1673,4,4.421298,2.676636,0\n14,59857.1782,5,4.276851,6.626348,0\n14,59867.1112,2,-0.034593,1.082066,0\n14,59867.1189,1,-1.146521,0.972931,0\n14,59867.1267,3,4.685015,1.540005,0\n14,59867.1377,4,3.080778,2.356951,0\n14,59867.1487,5,-3.318216,5.827800,0\n14,59870.1049,2,0.554243,0.970361,0\n14,59870.1126,1,0.375991,0.821212,0\n14,59870.1204,3,4.517774,1.374431,0\n14,59870.1314,4,1.906460,2.282754,0\n14,59870.1424,5,-0.145795,5.276641,0\n14,59873.0971,2,2.108685,1.041195,0\n14,59873.1049,1,0.095400,1.040191,0\n14,59873.1126,3,3.909108,1.442765,0\n14,59873.1236,4,9.230726,2.071694,0\n14,59873.1346,5,5.649127,4.588614,0\n14,59874.1461,0,-3.214277,1.608558,0\n14,59875.0995,0,2.425352,2.028404,0\n14,59876.0980,0,3.646141,2.440229,0\n14,59877.0976,0,-1.473760,1.759196,0\n14,59878.0964,0,0.576515,2.010597,0\n14,59879.0895,0,0.572945,1.861002,0\n14,59880.1017,0,-0.224757,1.368790,0\n14,59884.1760,2,-0.132164,0.767148,0\n14,59884.1836,1,-0.991678,0.885456,0\n14,59884.1913,3,2.358108,1.139322,0\n14,59884.2022,4,3.557226,1.610064,0\n14,59884.2132,5,1.323540,4.209188,0\n14,59887.2856,2,-2.655194,1.824434,0\n14,59887.2933,1,-3.118680,2.202677,0\n14,59887.3009,3,4.665102,2.350384,0\n14,59887.3118,4,1.135563,3.143775,0\n14,59887.3228,5,4.491710,7.945457,0\n14,59896.1307,2,-0.963295,0.821164,0\n14,59896.1384,1,-0.071138,0.702119,0\n14,59896.1460,3,2.277665,1.393450,0\n14,59896.1569,4,7.248254,2.194077,0\n14,59896.1679,5,4.658720,5.946997,0\n14,59899.1519,2,-1.774250,1.363683,0\n14,59899.1595,1,-1.174699,1.385324,0\n14,59899.1672,3,-0.953666,2.376407,0\n14,59899.1781,4,8.315199,3.709089,0\n14,59899.1891,5,-5.444651,7.901953,0\n14,59902.1384,2,0.053353,1.184878,0\n14,59902.1460,1,-0.247240,1.076702,0\n14,59902.1537,3,3.547660,1.943007,0\n14,59902.1646,4,3.456085,2.762274,0\n14,59902.1755,5,-2.950504,7.020600,0\n14,59904.1053,0,4.177117,2.292928,0\n14,59905.0555,0,-1.216755,2.497270,0\n14,59906.0562,0,2.410480,2.032158,0\n14,59907.0567,0,-1.391141,1.743943,0\n14,59908.0681,0,3.657772,2.691214,0\n14,59909.0582,0,0.521167,1.833233,0\n14,59910.0503,0,-0.100852,1.860389,0\n14,59914.0526,2,1.198847,1.437903,0\n14,59914.0602,1,-4.574095,1.954059,0\n14,59914.0678,3,0.367522,1.685196,0\n14,59914.0788,4,2.276051,2.267016,0\n14,59914.0897,5,10.662569,5.357585,0\n14,59924.1060,2,0.898902,1.131097,0\n14,59924.1136,1,-0.533751,1.571826,0\n14,59924.1212,3,0.148105,1.727512,0\n14,59924.1322,4,4.691455,2.527090,0\n14,59924.1431,5,-2.120271,5.628747,0\n14,59927.1074,2,-0.466719,1.189681,0\n14,59927.1151,1,-0.370304,1.132207,0\n14,59927.1227,3,2.069784,1.592095,0\n14,59927.1336,4,2.988137,2.298649,0\n14,59927.1446,5,2.182713,5.887877,0\n14,59930.1236,2,0.240297,0.940370,0\n14,59930.1312,1,-0.920183,0.827606,0\n14,59930.1388,3,1.243185,1.282906,0\n14,59930.1498,4,3.796952,2.032441,0\n14,59930.1607,5,4.695318,5.329674,0\n14,59933.1249,2,0.006509,0.690589,0\n14,59933.1325,1,1.157228,0.695202,0\n14,59933.1401,3,2.223172,1.269250,0\n14,59933.1511,4,2.050447,1.913679,0\n14,59933.1620,5,6.551545,5.426427,0\n14,59934.0638,0,-1.038826,1.138865,0\n14,59935.0646,0,-0.855022,1.120789,0\n14,59936.0642,0,4.664287,1.541755,0\n14,59937.0650,0,0.068496,1.637206,0\n14,59938.0647,0,-0.037584,1.994424,0\n14,59939.0650,0,0.482500,2.191997,0\n14,60165.3032,2,-0.497132,1.193823,0\n14,60165.3109,1,2.598282,1.347751,0\n14,60165.3186,3,-2.635130,1.517240,0\n14,60165.3295,4,5.500406,2.024001,0\n14,60165.3405,5,-1.459476,4.779670,0\n14,60168.2892,2,-0.132310,0.833159,0\n14,60168.2970,1,1.305878,0.789044,0\n14,60168.3047,3,1.487449,1.300279,0\n14,60168.3157,4,2.190609,2.065560,0\n14,60168.3267,5,-2.313916,4.814189,0\n14,60176.2820,0,-2.258721,2.358804,0\n14,60177.2726,0,-0.240173,2.761198,0\n14,60181.4088,2,-0.118050,0.923237,0\n14,60181.4164,1,-1.757872,1.260326,0\n14,60181.4232,3,-0.566288,3.235162,0\n14,60183.2660,2,-0.732747,1.913709,0\n14,60183.2736,1,0.012732,2.931251,0\n14,60183.2812,3,1.402672,2.546405,0\n14,60183.2922,4,-1.725346,2.993038,0\n14,60183.3031,5,0.494266,6.175346,0\n14,60195.2812,2,3.029382,0.934420,0\n14,60195.2888,1,-0.670324,1.018990,0\n14,60195.2964,3,-0.662677,1.624419,0\n14,60195.3073,4,-3.872562,2.337922,0\n14,60195.3183,5,2.712977,6.272959,0\n14,60198.2690,0,2.956521,2.177715,0\n14,60199.2186,0,1.331341,2.156719,0\n14,60200.2139,0,1.351745,2.588871,0\n14,60201.2072,0,2.853797,2.498285,0\n14,60202.2089,0,1.906982,2.486693,0\n14,60209.1811,2,-1.602512,1.563901,0\n14,60209.1888,1,-0.671917,1.407978,0\n14,60209.1965,3,0.374204,1.830516,0\n14,60209.2075,4,2.568392,2.908100,0\n14,60209.2184,5,6.678216,6.695596,0\n14,60212.1675,2,3.776216,1.935000,0\n14,60212.1753,1,0.021062,2.747988,0\n14,60212.1830,3,-1.807372,2.293340,0\n14,60212.1941,4,2.568171,2.590221,0\n14,60212.2050,5,10.840719,5.663691,0\n14,60223.2416,2,-0.194212,1.181089,0\n14,60223.2493,1,-0.036469,1.239146,0\n14,60223.2569,3,1.530641,1.648199,0\n14,60223.2678,4,1.801575,2.471934,0\n14,60223.2788,5,4.744702,5.793488,0\n14,60226.3337,2,-0.744695,0.943627,0\n14,60226.3413,1,-0.106145,0.942899,0\n14,60226.3489,3,2.442077,1.793019,0\n14,60226.3599,4,1.509347,3.068994,0\n14,60226.3708,5,-6.627614,8.105484,0\n14,60238.3197,2,-0.863180,1.052267,0\n14,60238.3273,1,1.557186,1.152551,0\n14,60238.3349,3,-4.336288,2.169387,0\n14,60238.3459,4,4.612921,3.828061,0\n14,60238.3568,5,-2.021824,11.249375,0\n14,60241.0870,2,0.738195,1.997376,0\n14,60241.0948,1,-1.971425,3.061613,0\n14,60241.1025,3,-2.025038,2.443142,0\n14,60241.1136,4,-0.312927,2.915593,0\n14,60241.1245,5,3.056638,5.938406,0\n14,60250.1708,2,-2.482518,2.100286,0\n14,60250.1957,1,-0.029834,2.806254,0\n14,60250.2034,3,4.678575,2.926827,0\n14,60250.2143,4,-3.449543,3.980825,0\n14,60250.2253,5,-3.656875,7.932066,0\n14,60261.1296,0,-3.352771,2.067494,0\n14,60262.0550,0,-3.894481,2.299585,0\n14,60263.0556,0,0.061202,1.966700,0\n14,60264.0559,0,-0.441779,2.168793,0\n14,60265.0780,0,-0.070926,2.402439,0\n14,60268.0449,2,-2.115148,1.402396,0\n14,60268.0525,1,-1.156284,1.846402,0\n14,60268.0601,3,-0.248761,2.127820,0\n14,60268.0711,4,1.514031,2.732877,0\n14,60268.0820,5,-11.142164,6.741534,0\n14,60278.0993,2,0.058940,1.642234,0\n14,60278.1069,1,1.004120,2.270966,0\n14,60278.1145,3,-0.427660,2.168000,0\n14,60278.1255,4,0.148053,3.057248,0\n14,60278.1364,5,14.839427,7.639682,0\n14,60281.1023,2,0.190632,0.797410,0\n14,60281.1099,1,-0.334674,0.793767,0\n14,60281.1175,3,-1.839893,1.368291,0\n14,60281.1285,4,-1.943368,1.989664,0\n14,60281.1394,5,3.423336,5.175146,0\n14,60284.1027,2,0.880585,0.836640,0\n14,60284.1104,1,-1.071148,0.760074,0\n14,60284.1180,3,-0.493422,1.159284,0\n14,60284.1289,4,-1.769790,1.701805,0\n14,60284.1399,5,10.491048,4.439059,0\n14,60287.1047,2,-2.440768,1.249228,0\n14,60287.1123,1,-0.587990,1.081420,0\n14,60287.1200,3,-0.658860,1.794693,0\n14,60287.1309,4,6.071679,2.963972,0\n14,60287.1418,5,-0.912052,7.009711,0\n14,60290.0761,0,0.791311,1.912837,0\n14,60291.0689,0,-0.916465,1.657989,0\n14,60292.0699,0,-0.940145,1.852110,0\n14,60293.0699,0,-0.065888,1.558097,0\n14,60294.0708,0,-0.362213,1.394291,0\n14,60532.3019,2,-0.128229,1.022716,0\n14,60532.3097,1,-0.763027,0.950918,0\n14,60532.3173,3,-0.986068,1.539248,0\n14,60532.3282,4,-0.845234,2.279716,0\n14,60532.3392,5,8.809470,5.421995,0\n14,60535.2802,2,-0.482045,1.404915,0\n14,60535.2879,1,1.358840,1.336113,0\n14,60535.2957,3,-0.531306,2.081220,0\n14,60535.3068,4,-3.251427,3.016502,0\n14,60535.3177,5,-1.253254,7.157213,0\n14,60538.2826,2,1.802223,1.846149,0\n14,60538.2903,1,-0.026881,2.671491,0\n14,60538.2980,3,1.533429,1.916902,0\n14,60538.3089,4,-0.856327,2.306201,0\n14,60538.3199,5,-1.663799,5.002286,0\n14,60554.2651,0,-1.160009,2.241721,0\n14,60555.2411,0,-2.432893,2.027424,0\n14,60556.2370,0,2.398724,2.534231,0\n14,60557.2322,0,-1.568709,1.829169,0\n14,60558.2332,0,-4.116055,2.484439,0\n14,60559.2274,0,0.190265,2.200325,0\n14,60560.2268,0,1.347325,3.073875,0\n14,60567.3291,2,0.910389,0.934006,0\n14,60567.3368,1,-1.444133,1.041686,0\n14,60567.3444,3,-1.500671,1.243064,0\n14,60567.3553,4,2.684830,1.653721,0\n14,60567.3663,5,0.345307,4.248466,0\n14,60580.1736,2,0.807077,1.449346,0\n14,60580.1813,1,2.870951,1.391824,0\n14,60580.1889,3,-0.262873,2.190222,0\n14,60580.1999,4,-6.388680,3.196808,0\n14,60580.2108,5,9.181289,7.635880,0\n14,60582.1681,0,3.604234,2.412230,0\n14,60583.1640,0,-2.491946,2.974151,0\n14,60584.1591,0,0.669646,2.335869,0\n14,60585.1601,0,0.574023,1.865054,0\n14,60586.1564,0,-0.297618,1.811183,0\n14,60587.1540,0,-1.218754,2.115987,0\n14,60588.1461,0,-1.992268,1.771815,0\n14,60593.1209,2,-0.511945,1.144706,0\n14,60593.1287,1,-0.914986,1.302187,0\n14,60593.1365,3,-1.629705,1.534286,0\n14,60593.1476,4,-3.500163,2.167956,0\n14,60593.1585,5,-7.225812,5.098632,0\n14,60596.1351,2,1.942903,2.467501,0\n14,60596.1427,1,-3.828583,3.510221,0\n14,60596.1504,3,-0.379595,2.643848,0\n14,60596.1613,4,-4.746652,3.225241,0\n14,60596.1723,5,-0.740908,7.175334,0\n14,60605.0908,2,-0.240784,0.970734,0\n14,60605.0986,1,1.228715,0.912676,0\n14,60605.1063,3,-0.173487,1.451932,0\n14,60605.1174,4,-3.157050,2.142289,0\n14,60605.1283,5,-4.353776,5.129833,0\n14,60608.0836,2,-1.290518,0.880033,0\n14,60608.0913,1,0.897031,0.818780,0\n14,60608.0991,3,-1.789510,1.321652,0\n14,60608.1101,4,-0.283683,1.951819,0\n14,60608.1211,5,-1.071998,4.652730,0\n14,60611.0756,2,0.428420,0.767492,0\n14,60611.0833,1,-0.175538,0.697855,0\n14,60611.0911,3,-1.351069,1.165860,0\n14,60611.1021,4,3.187355,1.739082,0\n14,60611.1130,5,-2.026892,4.159950,0\n14,60612.0813,0,1.714202,1.449054,0\n14,60613.0818,0,-1.208979,1.515573,0\n14,60614.0803,0,2.366692,1.746634,0\n14,60615.0761,0,0.562962,1.980418,0\n14,60616.0769,0,-0.108613,2.003422,0\n14,60617.0737,0,1.100774,1.564082,0\n14,60620.1444,0,-3.990349,1.704574,0\n14,60621.2673,2,-1.006580,0.937842,0\n14,60621.2749,1,-0.619488,0.942508,0\n14,60621.2825,3,1.860522,1.643903,0\n14,60621.2934,4,1.823893,2.737007,0\n14,60621.3044,5,-2.549210,7.504488,0\n14,60624.1290,2,-1.234065,1.187694,0\n14,60624.1366,1,0.769079,1.561521,0\n14,60624.1442,3,-1.651142,1.493277,0\n14,60624.1551,4,-1.015730,2.012733,0\n14,60624.1661,5,-1.555369,4.753550,0\n14,60633.0541,2,0.136783,1.086145,0\n14,60633.0617,1,-0.152341,1.054063,0\n14,60633.0693,3,-0.510173,1.670818,0\n14,60633.0803,4,0.315384,2.478524,0\n14,60633.0912,5,-6.652580,5.982395,0\n14,60636.0482,2,-0.451914,1.025376,0\n14,60636.0558,1,0.286247,0.990897,0\n14,60636.0635,3,0.255769,1.584027,0\n14,60636.0744,4,-0.340420,2.361790,0\n14,60636.0854,5,9.500138,5.693579,0\n14,60640.0972,2,0.332550,0.868813,0\n14,60640.1049,1,-0.473409,0.845206,0\n14,60640.1125,3,0.729877,1.386701,0\n14,60640.1234,4,-1.934210,2.120461,0\n14,60640.1344,5,-4.023204,5.273717,0\n14,60642.0643,0,6.371965,2.098372,0\n14,60643.0521,0,0.069668,1.665262,0\n14,60644.0621,0,-0.031334,1.662740,0\n14,60645.0625,0,0.388892,1.767081,0\n14,60646.0636,0,-0.740859,1.779115,0\n14,60647.0635,0,3.793745,1.980666,0\n14,60648.0642,0,-2.506170,1.440771,0\n14,60652.1289,2,2.536718,0.938627,0\n14,60652.1365,1,-0.345612,1.114977,0\n14,60652.1441,3,-0.218325,1.374793,0\n14,60652.1550,4,3.384219,2.040273,0\n14,60652.1660,5,5.932649,5.198509,0\n17,59750.4229,2,0.384775,1.502702,0\n17,59750.4306,1,2.970657,2.100801,0\n17,59750.4383,3,-3.890317,2.298941,0\n17,59750.4450,4,-7.424517,10.310197,0\n17,59752.4070,2,1.180267,1.063318,0\n17,59752.4147,1,-0.427451,1.103617,0\n17,59752.4224,3,0.804619,1.459435,0\n17,59752.4334,4,0.082026,2.345975,0\n17,59752.4435,5,-12.127212,9.469489,0\n17,59767.2968,2,0.300545,0.879810,0\n17,59767.3045,1,-0.789039,0.757829,0\n17,59767.3122,3,-2.516992,1.335309,0\n17,59767.3233,4,-0.305087,2.137903,0\n17,59767.3343,5,-2.361575,4.981940,0\n17,59770.2179,2,2.652607,2.031536,0\n17,59770.2256,1,0.104747,2.751110,0\n17,59770.2334,3,-1.251170,2.138434,0\n17,59770.2445,4,1.627103,2.665143,0\n17,59770.2557,5,3.234555,6.275570,0\n17,59779.3188,2,-1.560282,2.314641,0\n17,59779.3265,1,1.138921,2.800010,0\n17,59779.3342,3,1.399184,2.246128,0\n17,59779.3452,4,-4.086601,2.966733,0\n17,59779.3562,5,3.600588,7.075419,0\n17,59782.1897,2,-0.205182,1.441666,0\n17,59782.1974,1,-0.008989,1.378359,0\n17,59782.2051,3,-3.370768,2.336834,0\n17,59782.2162,4,-1.506593,3.089562,0\n17,59782.2274,5,-1.340868,7.552328,0\n17,59797.2861,2,0.574704,1.059202,0\n17,59797.2938,1,-1.925344,1.095736,0\n17,59797.3015,3,-1.303792,1.916743,0\n17,59797.3126,4,-0.918545,2.786953,0\n17,59797.3237,5,-7.784748,6.963860,0\n17,59800.3168,2,0.594011,2.277387,0\n17,59800.3244,1,6.002035,3.310603,0\n17,59800.3320,3,0.828272,2.636445,0\n17,59800.3429,4,-2.143990,3.045394,0\n17,59800.3539,5,2.182031,6.526192,0\n17,59807.1738,2,0.345367,1.600768,0\n17,59807.1815,1,1.497795,2.061084,0\n17,59807.1892,3,-1.738689,1.721755,0\n17,59807.2003,4,-2.008608,2.187723,0\n17,59807.2114,5,-4.591490,5.073585,0\n17,59810.1045,2,-0.190427,1.024519,0\n17,59810.1122,1,1.409750,0.987204,0\n17,59810.1200,3,2.097727,1.766024,0\n17,59810.1311,4,4.661858,2.723413,0\n17,59810.1422,5,1.852001,6.541757,0\n17,59813.1044,2,-0.609008,0.979608,0\n17,59813.1122,1,-1.283370,0.947661,0\n17,59813.1199,3,0.443231,1.563090,0\n17,59813.1310,4,0.463331,2.578759,0\n17,59813.1422,5,-4.206830,5.891186,0\n17,59819.1532,0,-0.111748,1.900487,0\n17,59820.1047,0,-1.127300,1.845100,0\n17,59821.1026,0,-2.983045,2.653126,0\n17,59822.1105,0,-3.786463,3.135153,0\n17,59823.1505,0,-6.251946,2.924953,0\n17,59835.0600,2,-0.105554,1.651957,0\n17,59835.0678,1,-3.813735,2.461054,0\n17,59835.0755,3,-5.183056,2.248735,0\n17,59835.0866,4,-3.032220,2.768835,0\n17,59835.0978,5,-1.849971,5.832443,0\n17,59839.0306,2,1.615393,1.178528,0\n17,59839.0384,1,-0.088986,1.136387,0\n17,59839.0461,3,-3.522223,1.587615,0\n17,59839.0573,4,-2.323542,2.134739,0\n17,59839.0684,5,2.161493,5.221899,0\n17,59842.0207,2,1.318505,1.033401,0\n17,59842.0285,1,0.922532,1.039005,0\n17,59842.0362,3,1.569138,1.599077,0\n17,59842.0473,4,3.807324,2.315629,0\n17,59842.0585,5,-1.946610,5.253097,0\n17,59851.1114,0,0.466809,1.833261,0\n17,59854.0796,2,-0.432977,1.106505,0\n17,59854.0873,1,-0.349620,1.186553,0\n17,59854.0950,3,-1.028248,1.475277,0\n17,59854.1061,4,0.624902,2.189718,0\n17,59854.1172,5,-6.577788,5.223208,0\n17,59857.0453,2,1.048279,1.770880,0\n17,59857.0531,1,4.293002,2.442868,0\n17,59857.0608,3,0.670081,1.885174,0\n17,59857.0719,4,-1.393756,2.439591,0\n17,59857.0830,5,7.867962,5.441516,0\n17,59864.0162,2,1.446581,0.850881,0\n17,59864.0239,1,1.881013,0.881634,0\n17,59864.0316,3,1.176724,1.403741,0\n17,59864.0428,4,1.061918,2.333516,0\n17,59864.0539,5,7.476207,5.917978,0\n17,59867.0178,2,0.380713,1.309081,0\n17,59867.0255,1,-3.510653,1.302897,0\n17,59867.0332,3,-2.578208,2.183999,0\n17,59867.0443,4,1.046922,3.227464,0\n17,59867.0554,5,-14.202744,9.457583,0\n17,59870.0194,2,-1.554325,0.927003,0\n17,59870.0272,1,-0.831158,0.920545,0\n17,59870.0349,3,-1.187623,1.500118,0\n17,59870.0459,4,2.125045,2.528746,0\n17,59870.0571,5,2.077787,5.911781,0\n17,59873.0212,2,-0.457801,0.738598,0\n17,59873.0289,1,2.796051,0.760300,0\n17,59873.0366,3,0.138800,1.232345,0\n17,59873.0477,4,0.273784,1.886916,0\n17,59873.0588,5,-5.356641,4.801971,0\n17,59874.0599,0,-2.398834,1.584590,0\n17,59875.0311,0,-0.540180,1.315395,0\n17,59876.0231,0,0.411913,2.120848,0\n17,59877.0238,0,-0.891288,1.592397,0\n17,59878.0246,0,0.252658,1.625712,0\n17,59879.0248,0,0.494818,1.668736,0\n17,59880.0258,0,1.609941,1.357399,0\n17,59884.0823,2,1.431002,1.091537,0\n17,59884.0900,1,-0.669697,1.493729,0\n17,59884.0976,3,0.632236,1.522321,0\n17,59884.1085,4,2.388964,2.008811,0\n17,59884.1195,5,-3.279281,4.883777,0\n17,59887.0298,2,-1.951323,1.872626,0\n17,59887.0375,1,-0.140049,2.947258,0\n17,59887.0451,3,-0.987463,2.277290,0\n17,59887.0562,4,-1.669648,2.256846,0\n17,59887.0673,5,4.776433,5.354571,0\n17,60118.4163,0,1.237189,1.560750,0\n17,60124.2541,2,-0.027002,2.279922,0\n17,60124.2618,1,-1.582832,2.932222,0\n17,60124.2695,3,-2.243939,2.501337,0\n17,60124.2807,4,-2.010008,2.976800,0\n17,60124.2918,5,-3.812106,6.686803,0\n17,60140.2290,0,2.889142,1.885088,0\n17,60141.2225,0,-2.675623,3.532677,0\n17,60142.2202,0,3.701951,3.092139,0\n17,60143.2212,0,0.107055,2.421978,0\n17,60144.2186,0,2.121859,1.934957,0\n17,60145.2123,0,3.519005,2.431906,0\n17,60153.2274,2,1.331808,1.381732,0\n17,60153.2351,1,1.678943,1.884794,0\n17,60153.2428,3,-0.669065,1.646313,0\n17,60153.2539,4,-2.336847,2.069594,0\n17,60153.2650,5,-3.040281,5.269622,0\n17,60162.1477,2,0.274822,1.883733,0\n17,60162.1554,1,-1.090832,2.591505,0\n17,60162.1631,3,-4.857473,2.765956,0\n17,60162.1742,4,-10.275881,3.141578,0\n17,60162.1853,5,-9.252888,7.983027,0\n17,60165.1369,2,-0.468279,0.846974,0\n17,60165.1446,1,-0.430437,0.708165,0\n17,60165.1524,3,-0.581242,1.364931,0\n17,60165.1635,4,3.187243,2.182051,0\n17,60165.1746,5,6.814413,5.636336,0\n17,60168.1260,2,1.795461,0.881347,0\n17,60168.1337,1,-0.350579,0.854263,0\n17,60168.1414,3,-2.100255,1.659393,0\n17,60168.1525,4,-1.526452,2.377539,0\n17,60168.1637,5,2.514493,5.409767,0\n17,60176.1332,0,-1.022222,2.365174,0\n17,60177.1370,0,-3.289955,2.790762,0\n17,60181.3147,2,-1.503265,1.258997,0\n17,60181.3223,1,1.620242,1.237350,0\n17,60181.3299,3,2.305993,2.212692,0\n17,60181.3409,4,-0.697285,4.016526,0\n17,60181.3518,5,3.620666,8.753998,0\n17,60184.3625,2,-0.132555,2.135256,0\n17,60184.3701,1,-0.614091,2.560603,0\n17,60184.3777,3,0.653611,2.486900,0\n17,60184.3887,4,1.677764,2.891815,0\n17,60184.3996,5,7.855544,6.898280,0\n17,60194.1575,2,1.309505,1.067185,0\n17,60194.1652,1,-0.309205,1.053532,0\n17,60194.1729,3,-0.334617,1.786434,0\n17,60194.1839,4,0.655740,3.139240,0\n17,60194.1926,5,5.044827,11.278636,0\n17,60197.1181,2,0.552177,1.097758,0\n17,60197.1258,1,0.541816,0.909236,0\n17,60197.1335,3,0.639729,1.336890,0\n17,60197.1446,4,-0.978055,2.037800,0\n17,60197.1557,5,4.619338,4.987423,0\n17,60198.1077,0,2.149141,2.452106,0\n17,60199.0914,0,5.486719,3.143868,0\n17,60200.0650,0,1.228997,2.524841,0\n17,60201.0680,0,6.056491,2.897650,0\n17,60202.0552,0,3.771713,2.094126,0\n17,60206.1107,0,1.964452,1.559410,0\n17,60207.1469,0,1.315419,2.819893,0\n17,60208.0229,2,-0.600505,1.423023,0\n17,60208.0307,1,0.017527,1.343573,0\n17,60208.0384,3,-1.550505,1.831250,0\n17,60208.0495,4,3.882280,2.726225,0\n17,60208.0606,5,9.559813,7.414577,0\n17,60211.0124,2,1.860704,2.250480,0\n17,60211.0202,1,-2.090354,3.034887,0\n17,60211.0279,3,0.410192,2.599820,0\n17,60211.0390,4,-1.062310,3.468099,0\n17,60211.0502,5,8.321412,9.169344,0\n17,60221.0153,2,0.448097,0.997295,0\n17,60221.0230,1,0.082163,0.883182,0\n17,60221.0308,3,1.515744,1.365631,0\n17,60221.0419,4,-0.955922,2.171834,0\n17,60221.0530,5,6.189006,5.659595,0\n17,60224.0140,2,-0.977404,1.146213,0\n17,60224.0217,1,-1.527085,1.044619,0\n17,60224.0294,3,-2.026714,1.896492,0\n17,60224.0405,4,-2.270058,2.563563,0\n17,60224.0516,5,1.532559,5.857242,0\n17,60227.0151,2,0.269555,1.054280,0\n17,60227.0228,1,1.115057,0.999069,0\n17,60227.0305,3,1.552577,1.598891,0\n17,60227.0416,4,0.899977,2.226238,0\n17,60227.0527,5,-13.314797,5.586868,0\n17,60228.0187,0,1.561344,2.716171,0\n17,60229.0162,0,2.115411,2.732387,0\n17,60234.0265,0,-0.632177,2.254489,0\n17,60237.2206,2,0.991648,0.921157,0\n17,60237.2283,1,1.011999,0.870146,0\n17,60237.2359,3,1.932343,1.516595,0\n17,60237.2468,4,0.683186,2.506894,0\n17,60237.2578,5,-2.942844,7.042232,0\n17,60240.0223,2,0.455927,1.997859,0\n17,60240.0300,1,2.247968,2.647892,0\n17,60240.0377,3,6.028008,2.647428,0\n17,60240.0488,4,2.535219,3.012484,0\n17,60240.0598,5,8.448805,5.811503,0\n17,60249.0338,2,0.460854,0.866548,0\n17,60249.0415,1,0.452554,0.772836,0\n17,60249.0492,3,-1.191506,1.286252,0\n17,60249.0602,4,2.830095,2.116775,0\n17,60249.0712,5,-4.378712,5.177276,0\n17,60260.0423,0,-7.464523,2.644202,0\n17,60261.0361,0,0.930443,1.914282,0\n17,60262.0367,0,4.964463,1.854471,0\n17,60263.0373,0,0.783741,2.196650,0\n17,60264.0465,0,-3.080768,1.916776,0\n17,60490.2647,2,-0.977129,1.482041,0\n17,60490.2725,1,-2.611819,1.794772,0\n17,60490.2802,3,1.078630,2.071858,0\n17,60490.2913,4,-0.492853,2.955702,0\n17,60490.3024,5,3.792550,7.104066,0\n17,60493.2372,2,0.021979,1.054925,0\n17,60493.2450,1,-1.376170,0.980580,0\n17,60493.2527,3,1.015628,1.668951,0\n17,60493.2639,4,-0.002133,2.546342,0\n17,60493.2750,5,-11.551120,6.173368,0\n17,60499.2467,0,0.438197,2.600874,0\n17,60500.2437,0,-1.686091,2.074696,0\n17,60501.2385,0,0.020948,2.299457,0\n17,60502.2355,0,1.068030,2.629417,0\n17,60508.2638,2,-2.790141,2.436066,0\n17,60508.2715,1,5.700086,3.490426,0\n17,60508.2792,3,0.113332,2.677060,0\n17,60508.2903,4,-1.245703,3.280095,0\n17,60508.3014,5,16.609587,7.427779,0\n17,60524.2390,0,-0.490797,2.080026,0\n17,60525.1736,0,2.666769,2.941305,0\n17,60532.3489,2,-0.207433,0.864004,0\n17,60532.3565,1,-0.134437,0.845488,0\n17,60532.3641,3,-0.515472,1.462743,0\n17,60532.3751,4,0.897895,2.323136,0\n17,60532.3860,5,1.356515,5.973135,0\n17,60535.1253,2,3.357187,1.895133,0\n17,60535.1330,1,1.788090,2.447693,0\n17,60535.1408,3,-0.746670,2.393641,0\n17,60535.1519,4,1.761451,3.298109,0\n17,60535.1630,5,-3.134825,7.769337,0\n17,60538.2351,2,7.960939,1.803240,0\n17,60538.2428,1,6.926830,2.602727,0\n17,60538.2505,3,3.353295,1.972350,0\n17,60538.2615,4,5.710919,2.318584,0\n17,60538.2725,5,0.716567,5.113697,0\n17,60546.3406,2,12.973317,1.903937,1\n17,60546.3482,1,11.756248,2.837219,1\n17,60546.3558,3,10.891992,2.308349,0\n17,60546.3668,4,10.706098,3.126808,0\n17,60546.3777,5,1.998101,7.646282,0\n17,60549.0879,2,8.457420,1.114208,1\n17,60549.0956,1,10.614448,1.048073,1\n17,60549.1034,3,9.131392,1.747332,1\n17,60549.1145,4,12.109291,2.651627,0\n17,60549.1256,5,-0.454903,6.443316,0\n17,60554.0964,0,3.385612,2.492159,0\n17,60555.0951,0,-0.653454,2.008215,0\n17,60556.0879,0,4.359657,2.500103,0\n17,60557.0831,0,0.208935,1.804377,0\n17,60558.1093,0,2.002906,2.298758,0\n17,60559.1097,0,-1.713623,1.974990,0\n17,60560.1065,0,-4.030066,2.814673,0\n17,60567.2821,2,5.086202,1.698282,0\n17,60567.2897,1,4.862064,2.316888,0\n17,60567.2973,3,2.821583,1.933260,0\n17,60567.3083,4,8.017381,2.382588,0\n17,60567.3192,5,5.673804,5.440264,0\n17,60574.1118,2,4.409955,1.550580,0\n17,60574.1195,1,2.795052,2.196265,0\n17,60574.1272,3,4.197219,1.987329,0\n17,60574.1383,4,10.014592,2.528254,0\n17,60574.1493,5,12.793771,5.745250,0\n17,60577.0186,2,4.357023,1.470498,0\n17,60577.0263,1,1.377891,1.400257,0\n17,60577.0340,3,3.097299,2.332945,0\n17,60577.0451,4,5.029422,3.520210,0\n17,60577.0563,5,9.703295,8.530192,0\n17,60580.0095,2,0.080061,1.449560,0\n17,60580.0173,1,0.636153,1.387539,0\n17,60580.0250,3,1.814262,2.309473,0\n17,60580.0361,4,2.268791,3.480996,0\n17,60580.0472,5,3.766253,8.433430,0\n17,60582.0840,0,2.160855,2.057876,0\n17,60583.0169,0,-2.377121,2.935992,0\n17,60584.0117,0,-7.030778,2.272231,0\n17,60585.0117,0,-2.432227,1.843772,0\n17,60586.0123,0,0.942960,1.782520,0\n17,60587.0127,0,-0.601708,2.076788,0\n17,60588.0131,0,-2.069814,1.647215,0\n17,60593.0636,2,0.346279,0.950896,0\n17,60593.0713,1,1.045972,1.126506,0\n17,60593.0790,3,1.954885,1.349871,0\n17,60593.0901,4,4.170722,1.979371,0\n17,60593.1012,5,5.737381,4.821476,0\n17,60596.0304,2,1.884329,2.578470,0\n17,60596.0381,1,-2.693286,3.627853,0\n17,60596.0458,3,0.705422,2.812787,0\n17,60596.0569,4,1.202975,3.396812,0\n17,60596.0680,5,16.761280,7.451248,0\n17,60603.0208,2,0.386061,0.711484,0\n17,60603.0286,1,-0.007518,0.663680,0\n17,60603.0363,3,0.688574,1.169908,0\n17,60603.0473,4,4.215033,1.826377,0\n17,60603.0584,5,4.616831,4.548802,0\n17,60606.0225,2,-0.884437,1.024675,0\n17,60606.0303,1,-0.808401,0.984848,0\n17,60606.0379,3,1.421415,1.656912,0\n17,60606.0490,4,-0.165550,2.551519,0\n17,60606.0601,5,-0.249700,6.335219,0\n17,60609.0247,2,0.342698,0.744517,0\n17,60609.0323,1,0.365452,0.700571,0\n17,60609.0400,3,3.474475,1.220512,0\n17,60609.0510,4,5.966081,1.906627,0\n17,60609.0621,5,-4.503197,4.756433,0\n17,60612.0266,0,-1.850411,1.158724,0\n17,60613.0269,0,-1.095945,1.252873,0\n17,60614.0276,0,1.658578,1.438634,0\n17,60615.0375,0,-1.868202,1.603315,0\n"
},
{
"path": "examples/data/plasticc_test_set_metadata_1k.csv",
"content": "object_id,ra,decl,gal_l,gal_b,ddf,hostgal_specz,hostgal_photoz,hostgal_photoz_err,distmod,mwebv\n13,34.453125,-5.229529,169.987075,-59.956185,1,0.3048,0.3193,0.0542,41.1123,0.019\n14,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.6323,0.0179,42.8774,0.018\n17,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.8297,0.0605,43.6000,0.016\n23,34.804688,-5.829153,171.307861,-60.174401,1,nan,0.6533,0.1479,42.9640,0.023\n34,351.321442,-64.198746,317.458993,-50.429931,1,0.4557,0.4617,0.0122,42.0540,0.023\n35,35.332031,-5.979157,172.286722,-59.931743,1,nan,0.8388,0.0375,43.6290,0.022\n43,0.574468,-45.981140,327.041068,-68.778764,1,nan,0.6669,0.0546,43.0186,0.006\n50,0.574468,-45.981140,327.041068,-68.778764,1,nan,1.4663,0.0529,45.1281,0.006\n60,346.562500,-63.448284,320.824720,-49.866957,1,nan,0.9462,0.0116,43.9519,0.021\n69,349.160583,-64.760857,318.219706,-49.458924,1,nan,1.0432,0.1092,44.2138,0.020\n88,349.160583,-64.760857,318.219706,-49.458924,1,0.1608,0.1650,0.0053,39.4929,0.020\n96,151.171875,2.537361,237.288526,43.169764,1,0.3277,0.3680,0.0340,41.4711,0.024\n106,1.666667,-44.399834,327.519190,-70.529554,1,nan,0.8532,0.0602,43.6747,0.009\n114,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.7996,0.2747,43.5011,0.020\n115,151.347656,4.181528,235.568369,44.259942,1,nan,0.8979,0.0515,43.8114,0.016\n116,150.468750,1.641510,237.714575,42.075234,1,nan,1.1244,0.0363,44.4151,0.017\n130,34.277344,-5.679190,170.314930,-60.410322,1,0.3395,0.3368,0.0728,41.2464,0.020\n142,1.694561,-45.191612,326.278557,-69.858253,1,nan,1.2710,0.0796,44.7444,0.011\n147,150.820312,1.641510,237.994507,42.358984,1,nan,0.2904,0.1155,40.8738,0.020\n151,151.171875,1.342993,238.602520,42.464379,1,nan,0.5090,0.0122,42.3075,0.026\n168,349.429535,-62.508568,320.039643,-51.393745,1,nan,0.0000,0.0000,nan,0.020\n171,52.910156,-27.953188,223.774083,-54.639214,1,nan,0.8623,0.0583,43.7031,0.007\n173,150.996094,4.181528,235.291975,43.970869,1,nan,0.4490,0.0219,41.9820,0.015\n176,52.910156,-27.953188,223.774083,-54.639214,1,0.3775,0.3642,0.0064,41.4450,0.007\n184,352.711273,-63.823658,316.922299,-51.059403,1,nan,0.9112,0.0513,43.8508,0.024\n186,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.9302,0.0118,43.9062,0.011\n195,152.050781,2.985506,237.495952,44.143927,1,nan,0.4658,0.0250,42.0768,0.019\n198,1.694561,-45.191612,326.278557,-69.858253,1,0.4060,0.3959,0.0146,41.6579,0.011\n204,349.046051,-61.943836,320.796530,-51.753706,1,0.5584,0.4997,0.0312,42.2594,0.017\n211,53.613281,-27.953188,223.929533,-54.024772,1,0.5469,0.5644,0.0113,42.5781,0.007\n216,150.820312,1.641510,237.994507,42.358984,1,nan,0.4056,0.0489,41.7202,0.020\n236,2.457983,-45.389202,324.632685,-69.945696,1,0.3436,0.2885,0.0162,40.8574,0.011\n240,151.171875,2.537361,237.288526,43.169764,1,nan,1.0936,0.0318,44.3405,0.024\n260,150.820312,3.732834,235.666318,43.572109,1,nan,0.7554,0.0425,43.3496,0.016\n268,149.589844,3.583322,234.885369,42.474696,1,nan,0.6234,0.0184,42.8401,0.024\n272,149.414062,3.433834,234.919132,42.245550,1,nan,0.7059,0.0220,43.1693,0.027\n277,348.595886,-63.072620,320.023289,-50.713060,1,nan,0.8751,0.0187,43.7426,0.021\n289,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.7442,0.0682,43.3099,0.006\n306,148.886719,2.686724,235.347248,41.389003,1,0.7180,0.7265,0.0182,43.2458,0.028\n316,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.5615,0.1166,42.5647,0.017\n337,150.117188,2.836105,236.124718,42.483719,1,nan,1.4098,0.0499,45.0228,0.016\n349,34.453125,-5.229529,169.987075,-59.956185,1,nan,0.7679,0.0318,43.3934,0.019\n357,349.966217,-62.696659,319.542989,-51.376556,1,nan,0.8937,0.0213,43.7988,0.021\n366,53.613281,-26.944359,222.237403,-53.863858,1,nan,1.3577,0.2274,44.9217,0.009\n384,359.816315,-44.003082,331.451340,-70.123054,1,nan,0.8134,0.0374,43.5469,0.013\n402,349.891296,-64.573555,317.972107,-49.786192,1,nan,0.5684,0.0804,42.5965,0.023\n406,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.8989,0.0967,43.8145,0.018\n409,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.9110,0.0551,43.8503,0.017\n413,349.429535,-62.508568,320.039643,-51.393745,1,0.6430,0.6411,0.0083,42.9139,0.020\n443,150.996094,2.388015,237.313912,42.939977,1,0.3682,0.3649,0.0113,41.4497,0.021\n451,349.615387,-63.636005,318.927246,-50.506542,1,nan,0.8853,0.0298,43.7734,0.018\n455,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.4671,0.0117,42.0842,0.011\n466,34.277344,-5.079716,169.526841,-59.956640,1,0.4986,0.5527,0.0171,42.5229,0.019\n467,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.6573,0.0463,42.9800,0.009\n478,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.7283,0.0517,43.2524,0.008\n483,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.7835,0.0575,43.4469,0.016\n489,0.574468,-45.981140,327.041068,-68.778764,1,nan,0.5798,0.0174,42.6490,0.006\n524,34.277344,-5.079716,169.526841,-59.956640,1,0.4067,0.3530,0.0854,41.3660,0.019\n561,349.891296,-64.573555,317.972107,-49.786192,1,nan,0.9267,0.0135,43.8959,0.023\n565,152.050781,3.284369,237.157374,44.318466,1,nan,0.8253,0.0201,43.5857,0.019\n568,53.789062,-27.784405,223.685697,-53.845803,1,nan,1.0649,0.2186,44.2692,0.009\n583,149.414062,2.238686,236.239766,41.565558,1,0.7070,0.7129,0.0137,43.1952,0.017\n607,349.160583,-64.760857,318.219706,-49.458924,1,0.2424,0.4948,0.3294,42.2339,0.020\n611,149.589844,3.583322,234.885369,42.474696,1,0.5068,0.5222,0.0459,42.3744,0.024\n613,150.820312,1.641510,237.994507,42.358984,1,0.3014,0.6138,0.3121,42.7991,0.020\n622,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.9127,0.3417,43.8552,0.009\n639,52.207031,-28.630989,224.800211,-55.343637,1,nan,0.6900,0.0255,43.1086,0.009\n662,33.750000,-4.630479,168.146242,-59.949072,1,0.4181,0.4889,0.0244,42.2029,0.019\n670,51.855469,-28.630989,224.733260,-55.649872,1,nan,1.3168,0.1915,44.8395,0.009\n672,350.230255,-61.943836,320.053946,-52.070537,1,nan,0.9496,0.0789,43.9616,0.017\n674,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.6032,0.0466,42.7531,0.018\n680,32.871094,-4.780192,166.959493,-60.615132,1,nan,1.4883,0.2383,45.1681,0.017\n683,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.8370,0.0432,43.6235,0.011\n686,358.665253,-45.783966,330.353593,-68.203652,1,nan,1.3769,0.2502,44.9594,0.009\n687,150.468750,3.732834,235.392208,43.283244,1,nan,0.3617,0.2373,41.4273,0.020\n694,34.453125,-5.229529,169.987075,-59.956185,1,0.4544,0.4391,0.0361,41.9241,0.019\n699,0.589520,-47.161343,325.385896,-67.769893,1,0.5659,0.5592,0.0064,42.5537,0.009\n721,358.665253,-45.783966,330.353593,-68.203652,1,0.4664,0.4355,0.3217,41.9033,0.009\n725,359.446716,-44.201530,331.730015,-69.805709,1,nan,0.7186,0.0156,43.2165,0.010\n729,52.207031,-26.610098,221.298836,-55.042928,1,nan,0.7994,0.0124,43.5005,0.014\n731,346.276581,-64.011238,320.448031,-49.344136,1,0.5315,0.5418,0.0087,42.4710,0.019\n734,34.804688,-5.829153,171.307861,-60.174401,1,nan,0.9281,0.0119,43.9001,0.023\n747,54.667969,-27.615883,223.610785,-53.050840,1,nan,1.3670,0.0656,44.9399,0.009\n759,151.347656,3.583322,236.252362,43.918627,1,nan,1.5633,0.2334,45.2997,0.015\n779,347.861847,-61.943836,321.519104,-51.424048,1,0.7469,0.7381,0.0118,43.2877,0.017\n793,359.058563,-45.191612,330.695783,-68.844915,1,nan,0.6826,0.0160,43.0802,0.011\n810,35.332031,-5.979157,172.286722,-59.931743,1,nan,0.9306,0.0189,43.9073,0.022\n830,1.694561,-45.191612,326.278557,-69.858253,1,nan,0.6893,0.0081,43.1060,0.011\n833,150.292969,2.686724,236.427488,42.541447,1,nan,0.4998,0.0202,42.2604,0.016\n834,359.058563,-45.191612,330.695783,-68.844915,1,0.5767,0.5866,0.0128,42.6794,0.011\n843,53.789062,-27.784405,223.685697,-53.845803,1,nan,0.8523,0.0367,43.6717,0.009\n868,35.332031,-5.979157,172.286722,-59.931743,1,0.3881,0.3855,1.3203,41.5900,0.022\n883,53.261719,-27.615883,223.280041,-54.281374,1,nan,0.8622,0.0603,43.7027,0.006\n886,33.574219,-4.780192,168.064587,-60.175886,1,0.4615,0.4476,0.7778,41.9738,0.019\n887,358.648071,-46.375080,329.462659,-67.716008,1,nan,1.2642,0.2378,44.7300,0.009\n888,359.814819,-44.399834,330.775011,-69.801007,1,nan,1.2159,0.1488,44.6254,0.009\n905,0.189873,-45.586655,328.254458,-68.969298,1,nan,0.6630,0.0275,43.0028,0.007\n916,150.292969,2.686724,236.427488,42.541447,1,0.4052,0.4393,0.0217,41.9253,0.016\n917,1.666667,-44.399834,327.519190,-70.529554,1,nan,1.3559,0.2676,44.9180,0.009\n943,2.457983,-45.389202,324.632685,-69.945696,1,0.8582,0.8663,0.0291,43.7153,0.011\n946,359.446716,-44.201530,331.730015,-69.805709,1,nan,1.5205,0.0959,45.2254,0.010\n960,351.382965,-64.011238,317.574052,-50.604657,1,nan,0.6480,0.0129,42.9424,0.023\n962,150.820312,1.641510,237.994507,42.358984,1,nan,0.6779,0.0117,43.0616,0.020\n965,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.6636,0.0081,43.0054,0.011\n968,52.910156,-27.953188,223.774083,-54.639214,1,nan,0.2472,0.0116,40.4743,0.007\n978,34.101562,-5.829153,170.247753,-60.638325,1,0.4938,0.4954,0.0349,42.2373,0.019\n979,148.710938,2.836105,235.050801,41.328739,1,nan,1.1407,0.1072,44.4539,0.031\n983,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.3106,0.0065,41.0429,0.014\n1017,351.299988,-62.320400,319.038597,-52.026867,1,nan,1.4323,0.0481,45.0652,0.018\n1018,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.9262,0.0111,43.8945,0.018\n1020,149.238281,3.882372,234.283829,42.351155,1,nan,1.0264,0.2278,44.1701,0.033\n1030,352.711273,-63.823658,316.922299,-51.059403,1,nan,1.2671,0.0235,44.7362,0.024\n1039,150.644531,3.583322,235.698235,43.342784,1,nan,0.7124,0.0358,43.1936,0.018\n1049,1.723404,-45.981140,325.117958,-69.180825,1,nan,1.2628,0.1822,44.7271,0.010\n1059,53.085938,-27.784405,223.525509,-54.460748,1,nan,1.0223,0.0247,44.1595,0.007\n1063,53.789062,-27.784405,223.685697,-53.845803,1,nan,0.0000,0.0000,nan,0.009\n1065,53.613281,-26.944359,222.237403,-53.863858,1,0.4795,0.4443,0.0390,41.9546,0.009\n1067,2.071130,-45.191612,325.606223,-69.989264,1,0.7417,0.8350,0.0544,43.6171,0.011\n1084,0.965665,-46.375080,325.845907,-68.579427,1,0.2126,0.5647,0.4351,42.5797,0.007\n1087,352.132874,-63.636005,317.424173,-51.095855,1,nan,1.1039,0.1661,44.3657,0.021\n1088,34.277344,-5.679190,170.314930,-60.410322,1,0.7550,0.7142,0.0477,43.2001,0.020\n1100,34.101562,-5.829153,170.247753,-60.638325,1,0.9332,0.8409,0.0631,43.6357,0.019\n1106,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.9258,0.0219,43.8935,0.022\n1108,51.855469,-26.276812,220.627031,-55.293792,1,0.6459,0.7378,0.0318,43.2867,0.014\n1111,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.3981,0.7822,41.6724,0.007\n1114,33.398438,-4.331149,167.226341,-59.936551,1,0.5401,0.5514,0.0091,42.5169,0.018\n1115,150.820312,3.732834,235.666318,43.572109,1,nan,1.1354,0.0545,44.4412,0.016\n1123,151.171875,2.238686,237.619933,42.994783,1,0.5819,0.5516,0.3972,42.5177,0.024\n1127,0.965665,-46.375080,325.845907,-68.579427,1,nan,1.0761,0.1787,44.2973,0.007\n1128,149.414062,1.940072,236.565366,41.393323,1,nan,0.3618,0.0387,41.4282,0.018\n1138,359.816315,-44.003082,331.451340,-70.123054,1,nan,1.4331,0.2745,45.0668,0.013\n1151,33.574219,-4.780192,168.064587,-60.175886,1,nan,0.9335,0.2471,43.9155,0.019\n1168,347.861847,-61.943836,321.519104,-51.424048,1,nan,1.5435,0.1436,45.2656,0.017\n1174,0.949367,-45.586655,326.991548,-69.251686,1,nan,1.0215,0.1042,44.1572,0.013\n1193,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.7156,0.0493,43.2055,0.017\n1216,53.964844,-28.630989,225.142950,-53.813613,1,nan,0.9028,0.0148,43.8260,0.009\n1245,53.085938,-28.122234,224.100909,-54.509752,1,0.4937,0.5333,0.0166,42.4294,0.007\n1254,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.5079,0.3209,45.2031,0.023\n1265,54.667969,-27.615883,223.610785,-53.050840,1,0.5310,0.6159,0.0352,42.8082,0.009\n1266,149.414062,1.940072,236.565366,41.393323,1,nan,0.6497,0.0183,42.9492,0.018\n1271,33.750000,-4.630479,168.146242,-59.949072,1,nan,1.0088,0.0104,44.1239,0.019\n1274,0.189873,-45.586655,328.254458,-68.969298,1,nan,1.4221,0.2779,45.0460,0.007\n1288,33.222656,-4.780192,167.515653,-60.396584,1,nan,0.9169,0.1056,43.8676,0.018\n1289,346.276581,-64.011238,320.448031,-49.344136,1,nan,1.1128,0.0552,44.3872,0.019\n1304,347.846710,-64.760857,318.929827,-49.143596,1,0.3102,0.3081,0.0119,41.0226,0.019\n1321,151.523438,3.134927,236.900695,43.803170,1,nan,1.2033,0.1759,44.5975,0.019\n1347,349.429535,-62.508568,320.039643,-51.393745,1,nan,0.7305,0.0458,43.2603,0.020\n1354,53.085938,-28.122234,224.100909,-54.509752,1,nan,0.8265,0.0527,43.5896,0.007\n1365,1.708861,-45.586655,325.688716,-69.520253,1,0.6978,0.7233,0.0170,43.2338,0.011\n1380,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.9291,0.0148,43.9029,0.020\n1388,52.910156,-26.276812,220.926149,-54.363918,1,nan,0.8363,0.0411,43.6210,0.008\n1393,152.050781,2.985506,237.495952,44.143927,1,0.4202,2.8977,0.8121,46.9354,0.019\n1415,51.328125,-27.784405,223.130589,-55.999499,1,0.7371,0.7551,0.0216,43.3483,0.013\n1421,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.6717,0.0415,43.0374,0.009\n1439,34.101562,-5.829153,170.247753,-60.638325,1,nan,1.0442,0.1545,44.2165,0.019\n1450,53.964844,-28.630989,225.142950,-53.813613,1,nan,0.8016,0.0528,43.5077,0.009\n1463,1.363636,-46.768478,324.669342,-68.371416,1,0.2333,0.5070,0.1130,42.2975,0.008\n1467,151.523438,3.134927,236.900695,43.803170,1,nan,0.8282,0.1183,43.5952,0.019\n1468,346.562500,-63.448284,320.824720,-49.866957,1,nan,0.8677,0.0290,43.7199,0.021\n1469,149.414062,2.238686,236.239766,41.565558,1,nan,1.0850,0.0150,44.3193,0.017\n1487,150.468750,3.732834,235.392208,43.283244,1,0.6295,0.5829,0.0311,42.6627,0.020\n1492,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.7396,0.0157,43.2934,0.021\n1498,152.050781,3.284369,237.157374,44.318466,1,nan,0.8187,0.0238,43.5644,0.019\n1500,149.414062,2.238686,236.239766,41.565558,1,nan,1.0392,0.0139,44.2035,0.017\n1522,352.132874,-63.636005,317.424173,-51.095855,1,nan,1.2524,0.0277,44.7049,0.021\n1523,150.468750,1.641510,237.714575,42.075234,1,0.2669,0.2407,0.0254,40.4086,0.017\n1536,33.574219,-5.079716,168.448505,-60.407218,1,nan,0.5789,0.1445,42.6449,0.016\n1545,346.562500,-63.448284,320.824720,-49.866957,1,0.6731,0.7236,0.0459,43.2351,0.021\n1567,152.050781,2.985506,237.495952,44.143927,1,0.5581,0.5870,0.0381,42.6815,0.019\n1570,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.7967,0.0516,43.4914,0.017\n1578,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.5831,0.1709,42.6637,0.009\n1589,51.679688,-27.447618,222.618229,-55.642263,1,0.3366,0.3400,0.0077,41.2708,0.010\n1593,150.996094,4.181528,235.291975,43.970869,1,nan,1.2366,0.1482,44.6708,0.015\n1597,351.299988,-62.320400,319.038597,-52.026867,1,0.3161,0.3182,0.0159,41.1033,0.018\n1599,52.207031,-26.610098,221.298836,-55.042928,1,nan,1.0088,0.2519,44.1237,0.014\n1600,34.101562,-5.829153,170.247753,-60.638325,1,0.5596,0.5490,0.0474,42.5054,0.019\n1601,53.085938,-28.122234,224.100909,-54.509752,1,0.7155,0.7376,0.0244,43.2861,0.007\n1619,2.097458,-45.783966,324.737840,-69.478613,1,nan,1.1271,0.0568,44.4216,0.011\n1630,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.7373,0.0283,43.2851,0.014\n1636,0.965665,-46.375080,325.845907,-68.579427,1,nan,0.9029,0.0862,43.8264,0.007\n1660,359.446716,-44.201530,331.730015,-69.805709,1,nan,0.8022,0.0194,43.5100,0.010\n1667,33.574219,-5.079716,168.448505,-60.407218,1,nan,1.0792,0.0660,44.3050,0.016\n1676,0.965665,-46.375080,325.845907,-68.579427,1,nan,0.9666,0.2691,44.0090,0.007\n1678,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.7378,0.0146,43.2866,0.011\n1687,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.9107,0.0140,43.8494,0.006\n1691,152.050781,2.985506,237.495952,44.143927,1,0.7346,0.7037,0.0529,43.1609,0.019\n1695,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.9741,0.2405,44.0299,0.020\n1702,150.996094,2.388015,237.313912,42.939977,1,nan,0.7806,0.0916,43.4371,0.021\n1720,1.694561,-45.191612,326.278557,-69.858253,1,nan,0.6746,0.0276,43.0488,0.011\n1729,51.328125,-27.447618,222.535046,-55.950727,1,0.6110,0.5932,0.0122,42.7092,0.013\n1730,150.117188,3.732834,235.120533,42.993809,1,nan,0.6999,0.0471,43.1464,0.020\n1754,148.710938,2.836105,235.050801,41.328739,1,nan,0.9109,0.0431,43.8500,0.031\n1763,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.8860,0.0855,43.7757,0.017\n1770,0.190678,-45.783966,327.956322,-68.803772,1,0.2562,0.2577,0.0118,40.5774,0.005\n1780,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.3954,0.0285,41.6551,0.011\n1781,351.382965,-64.011238,317.574052,-50.604657,1,0.4976,0.4790,0.0223,42.1493,0.023\n1830,51.328125,-27.784405,223.130589,-55.999499,1,nan,0.7349,0.0181,43.2763,0.013\n1833,32.871094,-4.780192,166.959493,-60.615132,1,0.3202,0.3262,0.0567,41.1659,0.017\n1841,0.589520,-47.161343,325.385896,-67.769893,1,nan,0.9916,0.0165,44.0776,0.009\n1871,33.574219,-4.780192,168.064587,-60.175886,1,nan,0.5952,0.0260,42.7180,0.019\n1894,150.468750,1.641510,237.714575,42.075234,1,nan,1.2779,0.0607,44.7590,0.017\n1914,359.816315,-44.003082,331.451340,-70.123054,1,0.6648,0.6345,0.0370,42.8867,0.013\n1933,150.117188,2.836105,236.124718,42.483719,1,nan,0.4788,0.2350,42.1486,0.016\n1934,351.734680,-62.884678,318.284128,-51.651217,1,0.6909,0.7567,0.0533,43.3540,0.019\n1941,52.207031,-26.610098,221.298836,-55.042928,1,nan,0.9255,0.0135,43.8926,0.014\n1948,359.814819,-44.399834,330.775011,-69.801007,1,nan,1.2610,0.0630,44.7233,0.009\n1949,33.398438,-3.732834,166.492280,-59.466614,1,0.5070,0.5068,0.0084,42.2963,0.022\n1971,149.414062,1.940072,236.565366,41.393323,1,nan,1.1763,0.1964,44.5365,0.018\n1978,54.667969,-27.615883,223.610785,-53.050840,1,nan,0.8494,0.0510,43.6626,0.009\n1999,349.160583,-64.760857,318.219706,-49.458924,1,0.6580,0.5147,0.5022,42.3368,0.020\n2000,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.6559,0.0240,42.9743,0.017\n2017,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.3325,0.1725,41.2145,0.010\n2021,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.6515,0.0269,42.9566,0.014\n2046,150.644531,3.583322,235.698235,43.342784,1,0.7425,0.7898,0.0207,43.4684,0.018\n2054,152.050781,2.985506,237.495952,44.143927,1,nan,0.7912,0.0208,43.4730,0.019\n2074,359.446716,-44.201530,331.730015,-69.805709,1,0.5175,0.4921,0.0570,42.2198,0.010\n2093,150.996094,4.181528,235.291975,43.970869,1,nan,0.3252,0.0452,41.1578,0.015\n2097,151.347656,4.181528,235.568369,44.259942,1,nan,0.6327,0.0348,42.8789,0.016\n2106,1.694561,-45.191612,326.278557,-69.858253,1,nan,1.1768,0.0524,44.5376,0.011\n2109,358.665253,-45.783966,330.353593,-68.203652,1,0.4692,0.4797,0.0133,42.1532,0.009\n2133,349.891296,-64.573555,317.972107,-49.786192,1,nan,0.6612,0.0157,42.9957,0.023\n2145,51.328125,-27.447618,222.535046,-55.950727,1,nan,1.2076,0.0605,44.6070,0.013\n2147,0.589520,-47.161343,325.385896,-67.769893,1,nan,1.5213,0.2618,45.2269,0.009\n2160,34.277344,-5.079716,169.526841,-59.956640,1,nan,0.5716,0.0242,42.6114,0.019\n2168,359.446716,-44.201530,331.730015,-69.805709,1,0.2214,0.2308,0.2438,40.3055,0.010\n2172,0.189873,-45.586655,328.254458,-68.969298,1,0.6266,0.6681,0.0264,43.0232,0.007\n2182,151.523438,3.134927,236.900695,43.803170,1,nan,0.8239,0.0320,43.5812,0.019\n2183,148.710938,2.836105,235.050801,41.328739,1,0.2383,0.2618,0.0166,40.6165,0.031\n2187,34.101562,-5.829153,170.247753,-60.638325,1,0.3599,0.3656,0.5424,41.4546,0.019\n2193,347.846710,-64.760857,318.929827,-49.143596,1,nan,1.4268,0.0800,45.0550,0.019\n2195,150.468750,3.732834,235.392208,43.283244,1,nan,0.4598,0.0125,42.0435,0.020\n2198,351.382965,-64.011238,317.574052,-50.604657,1,nan,0.9277,0.2414,43.8988,0.023\n2206,52.910156,-25.944481,220.366350,-54.301439,1,nan,1.5014,0.0616,45.1915,0.010\n2208,352.398651,-62.696659,318.017427,-51.967966,1,0.7188,0.7126,0.0121,43.1942,0.020\n2223,358.648071,-46.375080,329.462659,-67.716008,1,nan,1.1672,0.0321,44.5157,0.009\n2228,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.6837,0.0267,43.0844,0.014\n2243,150.820312,3.134927,236.341348,43.230123,1,nan,1.3813,0.1932,44.9680,0.016\n2246,348.529419,-61.755440,321.293980,-51.763351,1,0.6467,0.7143,0.0413,43.2008,0.016\n2252,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.5851,0.0120,42.6728,0.008\n2265,359.058563,-45.191612,330.695783,-68.844915,1,0.2786,0.2719,0.0249,40.7105,0.011\n2270,151.171875,2.238686,237.619933,42.994783,1,nan,0.0000,0.0000,nan,0.024\n2276,346.655182,-63.260487,320.952196,-50.040935,1,nan,0.6827,0.0726,43.0806,0.019\n2281,346.655182,-63.260487,320.952196,-50.040935,1,0.3442,0.5476,0.1871,42.4986,0.019\n2287,351.953644,-62.132156,318.777388,-52.347124,1,nan,0.5387,0.0152,42.4558,0.019\n2292,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.9430,0.0086,43.9427,0.014\n2293,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.9148,0.0376,43.8614,0.010\n2296,1.694561,-45.191612,326.278557,-69.858253,1,nan,0.8033,0.0310,43.5136,0.011\n2299,352.132874,-63.636005,317.424173,-51.095855,1,nan,0.8109,0.0190,43.5386,0.021\n2304,1.708861,-45.586655,325.688716,-69.520253,1,nan,1.3102,0.0895,44.8261,0.011\n2333,349.966217,-62.696659,319.542989,-51.376556,1,0.4284,0.4611,0.0224,42.0509,0.021\n2348,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.5904,0.0092,42.6966,0.018\n2351,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.7537,0.1688,43.3435,0.018\n2354,51.679688,-27.447618,222.618229,-55.642263,1,0.3829,0.3935,0.0180,41.6425,0.010\n2370,35.683594,-5.379379,171.992947,-59.253501,1,nan,0.6099,0.0096,42.7820,0.020\n2372,34.453125,-5.229529,169.987075,-59.956185,1,0.3353,0.4757,0.6757,42.1315,0.019\n2388,54.667969,-27.615883,223.610785,-53.050840,1,nan,1.0298,0.0768,44.1791,0.009\n2395,359.814819,-44.399834,330.775011,-69.801007,1,0.6315,0.6359,0.0276,42.8924,0.009\n2407,0.190678,-45.783966,327.956322,-68.803772,1,0.5859,0.5249,0.0829,42.3879,0.005\n2410,52.207031,-26.610098,221.298836,-55.042928,1,0.3848,0.4258,0.0241,41.8449,0.014\n2451,359.814819,-44.399834,330.775011,-69.801007,1,nan,0.7416,0.0343,43.3003,0.009\n2476,149.414062,1.940072,236.565366,41.393323,1,nan,0.4730,0.1009,42.1166,0.018\n2538,346.130127,-63.072620,321.423103,-50.042305,1,0.6846,0.6643,0.0319,43.0081,0.020\n2550,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.3640,0.1891,41.4439,0.020\n2603,0.965665,-46.375080,325.845907,-68.579427,1,nan,1.2175,0.4670,44.6290,0.007\n2615,52.910156,-26.276812,220.926149,-54.363918,1,nan,1.2323,0.0534,44.6613,0.008\n2660,151.523438,3.134927,236.900695,43.803170,1,1.0412,1.3254,0.1765,44.8570,0.019\n2661,346.130127,-63.072620,321.423103,-50.042305,1,0.5160,0.5185,0.0319,42.3560,0.020\n2683,0.965665,-46.375080,325.845907,-68.579427,1,0.6609,0.6757,0.0091,43.0530,0.007\n2687,348.529419,-61.755440,321.293980,-51.763351,1,0.6669,0.6531,0.0406,42.9628,0.016\n2693,358.665253,-45.783966,330.353593,-68.203652,1,nan,1.5860,0.2426,45.3384,0.009\n2701,349.891296,-64.573555,317.972107,-49.786192,1,0.4057,0.3811,0.0322,41.5609,0.023\n2702,35.683594,-5.379379,171.992947,-59.253501,1,0.5081,0.5489,0.0165,42.5049,0.020\n2707,33.398438,-4.331149,167.226341,-59.936551,1,0.1061,0.1359,0.0193,39.0322,0.018\n2744,348.595886,-63.072620,320.023289,-50.713060,1,nan,0.5735,0.0245,42.6203,0.021\n2753,51.328125,-27.784405,223.130589,-55.999499,1,0.4302,0.3267,0.0389,41.1697,0.013\n2759,148.886719,2.686724,235.347248,41.389003,1,nan,0.5974,0.0069,42.7274,0.028\n2760,149.414062,3.433834,234.919132,42.245550,1,nan,0.3415,0.2635,41.2821,0.027\n2766,150.468750,3.732834,235.392208,43.283244,1,nan,1.0362,0.0118,44.1956,0.020\n2786,349.615387,-63.636005,318.927246,-50.506542,1,nan,0.3010,0.0504,40.9634,0.018\n2790,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.0000,0.0000,nan,0.018\n2794,359.415588,-46.768478,327.729895,-67.686097,1,0.6983,0.7077,0.0266,43.1760,0.009\n2814,150.117188,2.836105,236.124718,42.483719,1,nan,0.7215,0.0782,43.2272,0.016\n2818,52.207031,-28.291550,224.208534,-55.300157,1,nan,0.8251,0.0216,43.5851,0.007\n2856,53.964844,-28.630989,225.142950,-53.813613,1,nan,0.6348,0.0515,42.8877,0.009\n2858,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.1145,0.0322,44.3914,0.023\n2860,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.4398,0.2606,41.9285,0.016\n2871,346.655182,-63.260487,320.952196,-50.040935,1,nan,0.4647,0.0229,42.0709,0.019\n2882,151.171875,1.342993,238.602520,42.464379,1,nan,2.7824,0.2944,46.8290,0.026\n2886,148.710938,2.836105,235.050801,41.328739,1,nan,0.3475,0.0073,41.3259,0.031\n2909,359.446716,-44.201530,331.730015,-69.805709,1,nan,0.3229,0.3149,41.1406,0.010\n2932,352.711273,-63.823658,316.922299,-51.059403,1,nan,0.1911,0.0104,39.8466,0.024\n2933,348.586945,-64.573555,318.693903,-49.477869,1,0.6336,0.6157,0.0140,42.8070,0.018\n2943,0.589520,-47.161343,325.385896,-67.769893,1,0.7816,0.7511,0.0242,43.3344,0.009\n2945,52.910156,-25.944481,220.366350,-54.301439,1,nan,1.1553,0.0720,44.4880,0.010\n2949,52.910156,-25.944481,220.366350,-54.301439,1,nan,1.2125,0.0257,44.6178,0.010\n2958,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.5145,0.0383,42.3356,0.009\n2961,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.3771,0.0426,41.5336,0.019\n2962,149.414062,3.433834,234.919132,42.245550,1,nan,0.7453,0.0637,43.3138,0.027\n2970,53.085938,-27.111860,222.384291,-54.355086,1,0.4506,0.5123,0.0647,42.3247,0.007\n2975,53.085938,-28.122234,224.100909,-54.509752,1,nan,1.4981,0.2615,45.1857,0.007\n3004,349.966217,-62.696659,319.542989,-51.376556,1,0.4053,0.4978,0.0481,42.2498,0.021\n3008,358.312500,-44.993881,332.185785,-68.685906,1,nan,0.1344,0.7139,39.0065,0.009\n3012,150.117188,2.238686,236.784618,42.139082,1,nan,1.4853,0.0818,45.1627,0.016\n3022,52.207031,-28.630989,224.800211,-55.343637,1,0.4266,0.3630,0.0443,41.4365,0.009\n3025,359.415588,-46.768478,327.729895,-67.686097,1,0.6636,0.6827,0.0129,43.0806,0.009\n3028,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.6075,0.2450,42.7719,0.017\n3029,347.617462,-62.508568,321.121462,-50.904708,1,nan,0.7168,0.0344,43.2100,0.019\n3037,150.820312,3.732834,235.666318,43.572109,1,0.3807,0.3856,0.0770,41.5907,0.016\n3052,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.9395,0.0475,43.9327,0.019\n3073,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.5457,0.7707,42.4896,0.014\n3077,0.189873,-45.586655,328.254458,-68.969298,1,0.2853,0.3011,0.0497,40.9649,0.007\n3133,0.574468,-45.981140,327.041068,-68.778764,1,nan,1.6167,0.0739,45.3897,0.006\n3163,34.277344,-5.079716,169.526841,-59.956640,1,0.6388,0.6322,0.0085,42.8770,0.019\n3170,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.4754,0.0150,42.1298,0.018\n3171,33.925781,-5.979157,170.179895,-60.866303,1,0.5550,0.5525,0.0096,42.5220,0.022\n3175,347.617462,-62.508568,321.121462,-50.904708,1,nan,1.0217,0.0243,44.1579,0.019\n3176,150.820312,3.134927,236.341348,43.230123,1,nan,0.7794,0.0377,43.4329,0.016\n3190,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.7705,0.0220,43.4023,0.022\n3193,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.8700,0.0495,43.7269,0.021\n3196,358.648071,-46.375080,329.462659,-67.716008,1,0.3112,0.2901,0.0115,40.8716,0.009\n3206,53.085938,-27.111860,222.384291,-54.355086,1,0.6472,0.6860,0.0225,43.0931,0.007\n3209,1.363636,-46.768478,324.669342,-68.371416,1,0.4369,0.4994,1.2314,42.2578,0.008\n3210,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.7133,0.0739,43.1968,0.008\n3214,348.908447,-63.823658,319.169886,-50.176186,1,0.2600,0.2876,0.3195,40.8500,0.018\n3217,151.347656,4.181528,235.568369,44.259942,1,0.1826,0.4846,0.3727,42.1795,0.016\n3247,1.708861,-45.586655,325.688716,-69.520253,1,nan,1.3152,0.1148,44.8363,0.011\n3252,349.615387,-63.636005,318.927246,-50.506542,1,nan,0.8013,0.0111,43.5070,0.018\n3278,53.613281,-27.953188,223.929533,-54.024772,1,nan,1.5343,0.1417,45.2497,0.007\n3284,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.4121,1.2419,41.7612,0.011\n3286,33.574219,-4.780192,168.064587,-60.175886,1,nan,0.7985,0.0224,43.4974,0.019\n3288,51.855469,-26.276812,220.627031,-55.293792,1,nan,1.2928,0.0725,44.7902,0.014\n3325,53.613281,-28.630989,225.073365,-54.119461,1,0.4825,0.5158,0.0204,42.3423,0.006\n3336,34.453125,-5.229529,169.987075,-59.956185,1,nan,0.7542,0.0326,43.3453,0.019\n3347,52.910156,-27.953188,223.774083,-54.639214,1,nan,0.5940,0.0536,42.7127,0.007\n3351,359.811707,-45.191612,329.485675,-69.150905,1,0.5478,0.5828,0.0191,42.6626,0.010\n3362,347.617462,-62.508568,321.121462,-50.904708,1,nan,0.6700,0.1306,43.0306,0.019\n3373,352.398651,-62.696659,318.017427,-51.967966,1,nan,0.4474,0.2471,41.9728,0.020\n3395,53.613281,-27.953188,223.929533,-54.024772,1,0.3963,0.3657,0.8633,41.4551,0.007\n3398,34.804688,-5.829153,171.307861,-60.174401,1,nan,1.0070,0.0407,44.1188,0.023\n3410,349.615387,-63.636005,318.927246,-50.506542,1,nan,1.1667,0.0446,44.5144,0.018\n3418,0.190678,-45.783966,327.956322,-68.803772,1,nan,1.0109,0.0153,44.1294,0.005\n3441,0.189873,-45.586655,328.254458,-68.969298,1,nan,0.4974,0.0111,42.2477,0.007\n3457,151.347656,4.181528,235.568369,44.259942,1,nan,0.7204,0.0318,43.2232,0.016\n3475,149.414062,3.433834,234.919132,42.245550,1,nan,0.7214,0.0251,43.2270,0.027\n3498,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.9131,0.0795,43.8565,0.019\n3499,150.996094,2.985506,236.647967,43.287350,1,0.9971,1.0190,0.0171,44.1508,0.020\n3500,52.207031,-28.630989,224.800211,-55.343637,1,0.2306,0.4849,0.6744,42.1816,0.009\n3504,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.5829,0.0306,42.6630,0.019\n3509,53.261719,-27.615883,223.280041,-54.281374,1,0.5035,0.5169,0.2812,42.3478,0.006\n3522,52.910156,-27.279613,222.625192,-54.536648,1,0.8636,0.9140,0.0250,43.8590,0.007\n3529,348.586945,-64.573555,318.693903,-49.477869,1,nan,0.9039,0.0247,43.8293,0.018\n3535,359.814819,-44.399834,330.775011,-69.801007,1,nan,1.4756,0.1618,45.1450,0.009\n3538,150.996094,2.985506,236.647967,43.287350,1,0.4615,0.4664,0.0103,42.0806,0.020\n3547,2.097458,-45.783966,324.737840,-69.478613,1,nan,0.9467,0.0338,43.9533,0.011\n3551,358.636353,-46.768478,328.890146,-67.388837,1,0.2573,0.2393,0.0145,40.3948,0.008\n3566,151.171875,2.238686,237.619933,42.994783,1,0.6331,0.5448,0.6502,42.4852,0.024\n3581,51.855469,-26.276812,220.627031,-55.293792,1,0.4844,0.4938,0.0183,42.2287,0.014\n3585,150.820312,3.732834,235.666318,43.572109,1,nan,0.5231,0.1034,42.3791,0.016\n3597,2.457983,-45.389202,324.632685,-69.945696,1,0.8409,0.8742,0.0306,43.7396,0.011\n3620,35.859375,-4.630479,171.270769,-58.580806,1,nan,0.7190,0.0186,43.2181,0.022\n3629,51.855469,-28.630989,224.733260,-55.649872,1,nan,1.3951,0.1453,44.9947,0.009\n3641,149.414062,3.433834,234.919132,42.245550,1,nan,0.5965,0.0131,42.7237,0.027\n3645,346.562500,-63.448284,320.824720,-49.866957,1,0.5351,0.5274,0.0205,42.4003,0.021\n3652,149.238281,3.882372,234.283829,42.351155,1,nan,0.8499,0.0477,43.6644,0.033\n3657,34.277344,-5.679190,170.314930,-60.410322,1,0.3050,2.7750,0.9745,46.8221,0.020\n3661,347.812500,-63.448284,320.128971,-50.202348,1,nan,1.3297,0.0487,44.8658,0.021\n3666,150.117188,2.836105,236.124718,42.483719,1,nan,1.1820,0.1059,44.5495,0.016\n3672,348.529419,-61.755440,321.293980,-51.763351,1,0.6020,0.5785,0.0108,42.6431,0.016\n3681,150.996094,2.985506,236.647967,43.287350,1,0.6459,0.6429,0.0113,42.9215,0.020\n3702,151.171875,2.238686,237.619933,42.994783,1,nan,0.8082,0.1270,43.5299,0.024\n3706,51.328125,-27.784405,223.130589,-55.999499,1,0.3284,0.3150,0.1745,41.0780,0.013\n3709,151.171875,2.238686,237.619933,42.994783,1,nan,0.8753,0.0139,43.7432,0.024\n3710,1.666667,-44.399834,327.519190,-70.529554,1,nan,0.8356,0.0335,43.6190,0.009\n3735,0.574468,-45.981140,327.041068,-68.778764,1,nan,0.7982,0.1565,43.4967,0.006\n3737,150.292969,2.686724,236.427488,42.541447,1,nan,0.7567,0.0404,43.3543,0.016\n3748,52.910156,-25.944481,220.366350,-54.301439,1,nan,1.0812,0.0621,44.3099,0.010\n3749,359.415588,-46.768478,327.729895,-67.686097,1,0.7033,0.7840,0.0371,43.4487,0.009\n3751,352.132874,-63.636005,317.424173,-51.095855,1,0.6527,0.6351,0.0190,42.8890,0.021\n3753,346.562500,-63.448284,320.824720,-49.866957,1,0.1191,0.0761,0.0195,37.6870,0.021\n3763,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.3349,0.0091,41.2320,0.017\n3764,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.8530,0.0218,43.6740,0.021\n3777,33.574219,-5.079716,168.448505,-60.407218,1,nan,0.5510,0.0652,42.5152,0.016\n3779,348.595886,-63.072620,320.023289,-50.713060,1,nan,0.9939,0.2200,44.0838,0.021\n3785,359.415588,-46.768478,327.729895,-67.686097,1,0.5052,0.5168,0.0229,42.3475,0.009\n3794,52.910156,-26.276812,220.926149,-54.363918,1,0.6236,0.6316,0.0331,42.8743,0.008\n3803,151.171875,1.342993,238.602520,42.464379,1,nan,0.7038,0.0688,43.1613,0.026\n3821,52.910156,-27.953188,223.774083,-54.639214,1,nan,1.4631,0.2823,45.1222,0.007\n3822,358.648071,-46.375080,329.462659,-67.716008,1,0.3455,2.9896,1.2696,47.0170,0.009\n3824,1.753247,-46.768478,324.030235,-68.498041,1,0.8063,0.8352,0.0339,43.6178,0.014\n3841,2.071130,-45.191612,325.606223,-69.989264,1,nan,0.5971,0.0990,42.7263,0.011\n3844,2.457983,-45.389202,324.632685,-69.945696,1,0.4789,0.3748,0.4652,41.5181,0.011\n3855,349.891296,-64.573555,317.972107,-49.786192,1,0.4507,0.3704,0.5053,41.4880,0.023\n3857,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.7254,0.0296,43.2418,0.011\n3875,52.207031,-28.630989,224.800211,-55.343637,1,0.6825,0.6468,0.0279,42.9374,0.009\n3878,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.9110,0.2538,43.8503,0.008\n3880,351.382965,-64.011238,317.574052,-50.604657,1,nan,1.5002,0.2529,45.1894,0.023\n3889,53.964844,-28.630989,225.142950,-53.813613,1,0.1582,0.2015,1.1876,39.9743,0.009\n3925,150.996094,4.181528,235.291975,43.970869,1,nan,0.2421,1.1561,40.4235,0.015\n3928,150.996094,2.985506,236.647967,43.287350,1,nan,0.7806,0.0538,43.4371,0.020\n3929,149.589844,3.583322,234.885369,42.474696,1,nan,1.8897,0.1427,45.8062,0.024\n3936,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.0853,0.0192,44.3200,0.023\n3950,352.132874,-63.636005,317.424173,-51.095855,1,0.7004,0.6924,0.0235,43.1180,0.021\n3970,1.753247,-46.768478,324.030235,-68.498041,1,nan,1.4637,0.1090,45.1234,0.014\n3972,33.574219,-5.079716,168.448505,-60.407218,1,nan,0.7985,0.0333,43.4976,0.016\n4001,150.996094,4.181528,235.291975,43.970869,1,nan,1.5533,0.1631,45.2826,0.015\n4007,150.820312,3.732834,235.666318,43.572109,1,nan,0.7855,0.0241,43.4536,0.016\n4016,346.130127,-63.072620,321.423103,-50.042305,1,nan,1.2676,0.0738,44.7372,0.020\n4023,0.589520,-47.161343,325.385896,-67.769893,1,nan,0.9574,0.1398,43.9835,0.009\n4025,33.222656,-4.780192,167.515653,-60.396584,1,nan,0.0000,0.0000,nan,0.018\n4038,35.859375,-4.630479,171.270769,-58.580806,1,nan,1.1349,0.0866,44.4401,0.022\n4044,151.171875,2.238686,237.619933,42.994783,1,nan,0.6693,0.0222,43.0278,0.024\n4054,34.453125,-5.229529,169.987075,-59.956185,1,0.5653,0.5478,0.0237,42.4998,0.019\n4062,358.312500,-44.993881,332.185785,-68.685906,1,nan,1.1896,0.0988,44.5666,0.009\n4063,348.586945,-64.573555,318.693903,-49.477869,1,nan,0.6406,0.1744,42.9119,0.018\n4065,35.332031,-5.979157,172.286722,-59.931743,1,nan,1.3023,0.0984,44.8098,0.022\n4077,150.468750,1.641510,237.714575,42.075234,1,nan,1.1662,0.1634,44.5133,0.017\n4103,349.160583,-64.760857,318.219706,-49.458924,1,0.3113,0.3331,0.0201,41.2185,0.020\n4109,52.910156,-26.276812,220.926149,-54.363918,1,nan,0.0000,0.0000,nan,0.008\n4183,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.8906,0.0209,43.7897,0.009\n4197,54.667969,-27.615883,223.610785,-53.050840,1,nan,0.7358,0.0471,43.2794,0.009\n4201,34.453125,-5.229529,169.987075,-59.956185,1,0.2193,0.2338,0.1899,40.3374,0.019\n4202,33.398438,-3.732834,166.492280,-59.466614,1,0.6916,0.7529,0.0348,43.3408,0.022\n4216,53.964844,-28.630989,225.142950,-53.813613,1,0.4807,0.4515,0.0246,41.9964,0.009\n4224,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.6240,0.0370,42.8426,0.022\n4225,346.562500,-63.448284,320.824720,-49.866957,1,0.2704,0.3546,0.2924,41.3770,0.021\n4249,51.679688,-27.447618,222.618229,-55.642263,1,nan,1.3003,0.1720,44.8058,0.010\n4262,346.130127,-63.072620,321.423103,-50.042305,1,nan,0.7108,0.0343,43.1877,0.020\n4267,33.222656,-4.780192,167.515653,-60.396584,1,0.4186,0.3797,0.0298,41.5512,0.018\n4272,53.613281,-26.944359,222.237403,-53.863858,1,0.2892,0.2718,0.0268,40.7089,0.009\n4274,351.321442,-64.198746,317.458993,-50.429931,1,0.6479,0.6553,0.0091,42.9721,0.023\n4278,149.414062,2.238686,236.239766,41.565558,1,nan,0.8546,0.0815,43.6790,0.017\n4282,53.085938,-27.784405,223.525509,-54.460748,1,nan,0.4929,0.0100,42.2237,0.007\n4283,52.910156,-27.279613,222.625192,-54.536648,1,0.3708,0.3947,0.0245,41.6505,0.007\n4293,1.753247,-46.768478,324.030235,-68.498041,1,nan,1.6305,0.1495,45.4125,0.014\n4294,150.117188,2.836105,236.124718,42.483719,1,nan,0.8510,0.1009,43.6678,0.016\n4300,51.855469,-26.276812,220.627031,-55.293792,1,0.6127,0.6028,0.0088,42.7511,0.014\n4303,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.9797,0.0098,44.0451,0.018\n4304,34.277344,-5.679190,170.314930,-60.410322,1,nan,0.2127,0.0101,40.1063,0.020\n4306,151.171875,2.537361,237.288526,43.169764,1,nan,1.4930,0.1628,45.1765,0.024\n4313,149.414062,3.433834,234.919132,42.245550,1,nan,1.1395,0.1626,44.4510,0.027\n4322,151.347656,4.181528,235.568369,44.259942,1,0.7816,0.7648,0.0184,43.3824,0.016\n4325,34.980469,-6.279288,172.180075,-60.389399,1,nan,0.6802,0.0118,43.0708,0.023\n4330,53.613281,-28.630989,225.073365,-54.119461,1,nan,1.1409,0.1366,44.4543,0.006\n4337,34.804688,-5.829153,171.307861,-60.174401,1,nan,0.6082,0.0100,42.7747,0.023\n4347,35.683594,-5.379379,171.992947,-59.253501,1,nan,1.5051,0.0376,45.1982,0.020\n4369,53.964844,-28.630989,225.142950,-53.813613,1,nan,1.1974,0.0630,44.5843,0.009\n4372,351.259003,-64.386185,317.344860,-50.255113,1,0.8588,0.8990,0.0160,43.8146,0.020\n4376,33.222656,-4.780192,167.515653,-60.396584,1,nan,0.9810,0.0139,44.0488,0.018\n4390,53.437500,-29.142223,225.908120,-54.336118,1,nan,1.0152,0.3139,44.1407,0.008\n4424,151.699219,3.583322,236.533224,44.205648,1,nan,1.0317,0.0256,44.1840,0.016\n4467,53.261719,-27.615883,223.280041,-54.281374,1,0.3489,0.3023,0.1233,40.9750,0.006\n4469,150.820312,1.641510,237.994507,42.358984,1,nan,1.0352,0.2064,44.1930,0.020\n4475,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.5952,0.0666,42.7177,0.014\n4489,35.332031,-5.979157,172.286722,-59.931743,1,nan,0.3478,0.0152,41.3280,0.022\n4490,1.723404,-45.981140,325.117958,-69.180825,1,nan,1.1304,0.0573,44.4295,0.010\n4492,358.636353,-46.768478,328.890146,-67.388837,1,0.2686,0.6256,0.1162,42.8492,0.008\n4494,2.097458,-45.783966,324.737840,-69.478613,1,0.4329,0.4492,0.0147,41.9831,0.011\n4507,35.683594,-5.379379,171.992947,-59.253501,1,nan,1.1883,0.0370,44.5636,0.020\n4508,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.7777,0.0513,43.4272,0.021\n4525,149.589844,3.583322,234.885369,42.474696,1,nan,0.7945,0.0729,43.4840,0.024\n4528,359.811707,-45.191612,329.485675,-69.150905,1,0.6181,0.5632,0.0527,42.5723,0.010\n4533,150.996094,2.985506,236.647967,43.287350,1,nan,0.9019,0.2483,43.8233,0.020\n4551,358.312500,-44.993881,332.185785,-68.685906,1,0.2666,0.6085,0.0540,42.7761,0.009\n4559,0.929752,-44.597992,328.531426,-70.083244,1,0.3229,0.3027,0.6578,40.9778,0.011\n4561,351.953644,-62.132156,318.777388,-52.347124,1,nan,0.7697,0.0339,43.3995,0.019\n4576,150.292969,2.686724,236.427488,42.541447,1,0.5613,0.5636,0.0277,42.5742,0.016\n4586,2.071130,-45.191612,325.606223,-69.989264,1,0.3810,0.3822,0.0308,41.5678,0.011\n4589,150.820312,1.641510,237.994507,42.358984,1,nan,1.2748,0.1135,44.7525,0.020\n4592,52.207031,-26.610098,221.298836,-55.042928,1,0.7073,0.7753,0.0429,43.4190,0.014\n4597,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.6120,0.0070,42.7913,0.018\n4611,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.8291,0.0669,43.5981,0.007\n4625,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.7723,0.0357,43.4086,0.021\n4644,35.859375,-4.630479,171.270769,-58.580806,1,0.4138,0.4027,0.0331,41.7017,0.022\n4645,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.6546,0.1405,42.9690,0.009\n4653,150.292969,2.686724,236.427488,42.541447,1,nan,0.9818,0.0169,44.0509,0.016\n4673,352.711273,-63.823658,316.922299,-51.059403,1,nan,0.5303,0.0066,42.4145,0.024\n4677,53.085938,-27.784405,223.525509,-54.460748,1,0.3093,0.1395,0.2071,39.0948,0.007\n4695,349.966217,-62.696659,319.542989,-51.376556,1,0.6195,0.6168,0.0156,42.8120,0.021\n4707,359.814819,-44.399834,330.775011,-69.801007,1,nan,0.8457,0.0478,43.6511,0.009\n4712,0.190678,-45.783966,327.956322,-68.803772,1,0.1206,0.5167,0.0880,42.3468,0.005\n4713,358.665253,-45.783966,330.353593,-68.203652,1,0.8042,0.7677,0.0483,43.3928,0.009\n4719,0.965665,-46.375080,325.845907,-68.579427,1,0.3619,0.3727,0.0303,41.5035,0.007\n4720,359.805206,-46.768478,327.135979,-67.829903,1,0.5530,0.5247,0.1218,42.3871,0.011\n4724,352.398651,-62.696659,318.017427,-51.967966,1,0.5390,0.4826,0.1275,42.1692,0.020\n4738,35.332031,-5.979157,172.286722,-59.931743,1,nan,1.0768,0.0572,44.2988,0.022\n4739,51.855469,-26.276812,220.627031,-55.293792,1,nan,0.7661,0.0148,43.3871,0.014\n4750,0.965665,-46.375080,325.845907,-68.579427,1,0.4822,0.2376,0.3459,40.3769,0.007\n4759,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.7922,0.2472,43.4764,0.010\n4760,350.230255,-61.943836,320.053946,-52.070537,1,nan,1.1776,0.0799,44.5394,0.017\n4762,51.679688,-27.447618,222.618229,-55.642263,1,nan,1.0371,0.2322,44.1980,0.010\n4768,349.285706,-62.884678,319.786163,-51.046461,1,nan,1.0012,0.0457,44.1033,0.018\n4776,150.117188,2.238686,236.784618,42.139082,1,0.4602,0.4465,0.0429,41.9677,0.016\n4812,151.171875,1.342993,238.602520,42.464379,1,nan,0.8278,0.0161,43.5940,0.026\n4822,149.589844,3.583322,234.885369,42.474696,1,nan,0.8752,0.0381,43.7427,0.024\n4824,346.655182,-63.260487,320.952196,-50.040935,1,0.4116,0.3961,1.0457,41.6592,0.019\n4825,150.468750,3.732834,235.392208,43.283244,1,nan,0.8466,0.0165,43.6540,0.020\n4830,1.666667,-44.399834,327.519190,-70.529554,1,nan,0.8267,0.0561,43.5904,0.009\n4833,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.6975,0.0469,43.1373,0.011\n4834,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.6570,0.0487,42.9789,0.008\n4840,151.171875,2.537361,237.288526,43.169764,1,nan,0.9905,0.0337,44.0747,0.024\n4844,34.277344,-5.679190,170.314930,-60.410322,1,nan,1.0144,0.1816,44.1385,0.020\n4853,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.8895,0.1178,43.7862,0.017\n4854,0.949367,-45.586655,326.991548,-69.251686,1,nan,0.7034,0.0178,43.1597,0.013\n4864,151.171875,1.342993,238.602520,42.464379,1,0.1623,0.1825,0.0221,39.7354,0.026\n4866,351.953644,-62.132156,318.777388,-52.347124,1,nan,0.6113,0.0972,42.7883,0.019\n4903,151.171875,2.238686,237.619933,42.994783,1,nan,0.4125,0.0223,41.7635,0.024\n4910,1.723404,-45.981140,325.117958,-69.180825,1,nan,0.6558,0.0453,42.9739,0.010\n4934,52.031250,-26.443335,220.963669,-55.168557,1,nan,1.2051,0.0888,44.6015,0.014\n4937,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.5072,0.0257,42.2985,0.010\n4948,151.171875,2.537361,237.288526,43.169764,1,nan,1.0614,0.0814,44.2604,0.024\n4986,32.871094,-4.780192,166.959493,-60.615132,1,0.5554,0.4904,0.4394,42.2108,0.017\n4990,149.414062,1.940072,236.565366,41.393323,1,nan,0.6203,0.0205,42.8268,0.018\n4994,150.468750,1.641510,237.714575,42.075234,1,nan,0.8252,0.0451,43.5856,0.017\n5024,352.711273,-63.823658,316.922299,-51.059403,1,0.3343,0.5197,0.1503,42.3618,0.024\n5034,346.276581,-64.011238,320.448031,-49.344136,1,nan,0.2533,0.8554,40.5348,0.019\n5057,150.468750,3.732834,235.392208,43.283244,1,0.8187,0.7582,0.0593,43.3595,0.020\n5061,150.292969,2.686724,236.427488,42.541447,1,0.4812,0.4838,0.0111,42.1754,0.016\n5084,52.207031,-26.610098,221.298836,-55.042928,1,nan,1.0849,0.0363,44.3190,0.014\n5088,349.160583,-64.760857,318.219706,-49.458924,1,0.7677,0.7828,0.0261,43.4447,0.020\n5094,2.071130,-45.191612,325.606223,-69.989264,1,nan,0.3655,0.0077,41.4542,0.011\n5099,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.3400,0.2409,41.2708,0.014\n5135,33.925781,-5.979157,170.179895,-60.866303,1,0.7377,0.7425,0.0053,43.3035,0.022\n5145,150.468750,1.641510,237.714575,42.075234,1,nan,1.1364,0.0385,44.4437,0.017\n5153,149.414062,2.238686,236.239766,41.565558,1,0.7765,0.7695,0.0155,43.3989,0.017\n5158,359.415588,-46.768478,327.729895,-67.686097,1,0.4004,0.4266,0.0302,41.8498,0.009\n5162,52.558594,-27.279613,222.538937,-54.845107,1,nan,0.5872,0.0094,42.6823,0.008\n5166,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.7222,0.0331,43.2299,0.008\n5176,359.814819,-44.399834,330.775011,-69.801007,1,nan,0.8147,0.0386,43.5513,0.009\n5204,33.574219,-4.780192,168.064587,-60.175886,1,0.4879,0.5068,0.0211,42.2965,0.019\n5210,33.574219,-4.780192,168.064587,-60.175886,1,0.4971,0.4954,0.0328,42.2371,0.019\n5217,53.261719,-27.615883,223.280041,-54.281374,1,nan,1.6018,0.0726,45.3649,0.006\n5234,51.328125,-27.784405,223.130589,-55.999499,1,0.7616,0.8177,0.0589,43.5609,0.013\n5236,35.332031,-5.979157,172.286722,-59.931743,1,0.3651,0.4588,0.1365,42.0379,0.022\n5249,348.586945,-64.573555,318.693903,-49.477869,1,nan,0.9021,0.0339,43.8239,0.018\n5264,347.617462,-62.508568,321.121462,-50.904708,1,nan,0.6893,0.0103,43.1059,0.019\n5278,53.789062,-27.784405,223.685697,-53.845803,1,0.4394,0.3893,0.1926,41.6151,0.009\n5280,351.299988,-62.320400,319.038597,-52.026867,1,nan,1.2309,0.2137,44.6583,0.018\n5283,347.013428,-62.508568,321.472056,-50.735330,1,nan,0.6296,0.1513,42.8661,0.018\n5286,347.846710,-64.760857,318.929827,-49.143596,1,0.5384,0.4991,0.1472,42.2564,0.019\n5307,2.457983,-45.389202,324.632685,-69.945696,1,0.4932,0.4967,0.0234,42.2437,0.011\n5313,349.891296,-64.573555,317.972107,-49.786192,1,0.3924,0.4004,0.3110,41.6868,0.023\n5315,359.805206,-46.768478,327.135979,-67.829903,1,nan,0.7107,0.0354,43.1872,0.011\n5317,33.574219,-5.379379,168.838090,-60.637536,1,0.2383,0.2830,0.1436,40.8096,0.017\n5319,148.710938,2.836105,235.050801,41.328739,1,nan,0.5440,0.0154,42.4816,0.031\n5322,0.929752,-44.597992,328.531426,-70.083244,1,nan,1.0052,0.0257,44.1143,0.011\n5335,348.908447,-63.823658,319.169886,-50.176186,1,0.2108,0.2283,0.1361,40.2787,0.018\n5338,152.050781,3.284369,237.157374,44.318466,1,nan,0.8439,0.0429,43.6452,0.019\n5354,34.980469,-6.279288,172.180075,-60.389399,1,0.2692,0.3663,0.1994,41.4598,0.023\n5365,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.9173,0.0221,43.8686,0.018\n5370,358.636353,-46.768478,328.890146,-67.388837,1,0.7437,0.6297,0.0526,42.8666,0.008\n5372,348.908447,-63.823658,319.169886,-50.176186,1,0.5219,0.5300,0.0576,42.4134,0.018\n5386,34.453125,-5.229529,169.987075,-59.956185,1,nan,0.6316,0.0102,42.8746,0.019\n5403,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.9827,0.0233,44.0533,0.009\n5412,151.347656,4.181528,235.568369,44.259942,1,0.3435,0.3761,0.2378,41.5268,0.016\n5417,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.7730,0.0296,43.4109,0.017\n5419,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.5571,0.0159,42.5440,0.008\n5422,0.190678,-45.783966,327.956322,-68.803772,1,nan,1.1195,0.1674,44.4033,0.005\n5433,53.085938,-28.122234,224.100909,-54.509752,1,nan,1.1700,0.1749,44.5221,0.007\n5453,150.117188,3.732834,235.120533,42.993809,1,nan,0.5035,0.3843,42.2794,0.020\n5473,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.5390,0.0204,42.4575,0.008\n5478,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.9349,0.0204,43.9195,0.018\n5520,152.050781,2.985506,237.495952,44.143927,1,0.2992,0.2421,0.0411,40.4233,0.019\n5521,51.328125,-27.784405,223.130589,-55.999499,1,nan,0.9909,0.0426,44.0758,0.013\n5535,53.261719,-27.615883,223.280041,-54.281374,1,nan,0.9054,0.0151,43.8337,0.006\n5543,33.398438,-3.732834,166.492280,-59.466614,1,nan,0.8296,0.1240,43.5996,0.022\n5550,151.171875,1.342993,238.602520,42.464379,1,nan,0.5898,0.0156,42.6937,0.026\n5562,351.734680,-62.884678,318.284128,-51.651217,1,0.3430,0.4407,0.1538,41.9338,0.019\n5564,351.321442,-64.198746,317.458993,-50.429931,1,nan,0.6240,0.0197,42.8426,0.023\n5565,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.7586,0.0376,43.3607,0.011\n5566,347.013428,-62.508568,321.472056,-50.735330,1,nan,0.9684,0.0328,44.0142,0.018\n5569,33.574219,-5.079716,168.448505,-60.407218,1,0.2696,0.2658,0.6279,40.6536,0.016\n5570,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.6613,0.0352,42.9962,0.017\n5576,0.589520,-47.161343,325.385896,-67.769893,1,nan,1.4255,0.1860,45.0525,0.009\n5591,350.230255,-61.943836,320.053946,-52.070537,1,nan,0.7811,0.0143,43.4387,0.017\n5623,1.753247,-46.768478,324.030235,-68.498041,1,0.3915,0.4107,1.0343,41.7522,0.014\n5635,359.415588,-46.768478,327.729895,-67.686097,1,nan,1.0468,0.1246,44.2231,0.009\n5655,1.753247,-46.768478,324.030235,-68.498041,1,nan,1.4730,0.2564,45.1403,0.014\n5656,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.9835,0.0168,44.0555,0.011\n5661,35.683594,-5.379379,171.992947,-59.253501,1,nan,0.2668,0.2983,40.6628,0.020\n5696,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.8087,0.0795,43.5316,0.018\n5704,350.230255,-61.943836,320.053946,-52.070537,1,nan,0.5054,0.0636,42.2892,0.017\n5723,351.321442,-64.198746,317.458993,-50.429931,1,nan,0.8582,0.1911,43.6902,0.023\n5729,53.085938,-27.784405,223.525509,-54.460748,1,nan,1.3930,0.0637,44.9906,0.007\n5758,52.910156,-25.944481,220.366350,-54.301439,1,nan,0.9711,0.0522,44.0216,0.010\n5761,150.468750,1.641510,237.714575,42.075234,1,nan,0.9866,0.2871,44.0641,0.017\n5763,349.891296,-64.573555,317.972107,-49.786192,1,nan,1.0138,0.2387,44.1370,0.023\n5794,351.299988,-62.320400,319.038597,-52.026867,1,nan,1.3563,0.0554,44.9189,0.018\n5798,51.328125,-27.784405,223.130589,-55.999499,1,nan,0.8854,0.0513,43.7737,0.013\n5801,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.8968,0.0208,43.8080,0.006\n5817,34.277344,-5.079716,169.526841,-59.956640,1,0.5298,0.5217,0.0200,42.3721,0.019\n5834,352.398651,-62.696659,318.017427,-51.967966,1,nan,0.8078,0.0389,43.5284,0.020\n5844,2.097458,-45.783966,324.737840,-69.478613,1,nan,1.1469,0.0339,44.4683,0.011\n5852,0.190678,-45.783966,327.956322,-68.803772,1,nan,0.8192,0.0304,43.5659,0.005\n5864,51.855469,-28.630989,224.733260,-55.649872,1,nan,1.3871,0.1040,44.9791,0.009\n5877,151.347656,4.181528,235.568369,44.259942,1,0.3096,0.3166,0.7338,41.0910,0.016\n5881,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.5213,0.0134,42.3702,0.018\n5895,348.908447,-63.823658,319.169886,-50.176186,1,nan,0.8744,0.0299,43.7403,0.018\n5911,33.750000,-4.630479,168.146242,-59.949072,1,0.2117,0.5363,0.4038,42.4441,0.019\n5922,1.694561,-45.191612,326.278557,-69.858253,1,0.1845,0.2240,0.3763,40.2324,0.011\n5924,51.855469,-28.630989,224.733260,-55.649872,1,0.7987,0.8068,0.0181,43.5253,0.009\n5930,149.589844,3.583322,234.885369,42.474696,1,nan,1.3364,0.0830,44.8793,0.024\n5937,2.457983,-45.389202,324.632685,-69.945696,1,nan,1.2012,0.0366,44.5926,0.011\n5954,347.617462,-62.508568,321.121462,-50.904708,1,nan,1.1290,0.0618,44.4261,0.019\n5956,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.8904,0.0380,43.7889,0.009\n5957,150.468750,3.732834,235.392208,43.283244,1,nan,0.7308,0.0756,43.2613,0.020\n5978,346.655182,-63.260487,320.952196,-50.040935,1,nan,0.5152,0.0154,42.3393,0.019\n5982,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.4710,0.0617,42.1060,0.022\n5995,359.058563,-45.191612,330.695783,-68.844915,1,nan,1.5541,0.1950,45.2840,0.011\n6000,0.574468,-45.981140,327.041068,-68.778764,1,nan,1.2101,0.0361,44.6126,0.006\n6001,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.8961,0.0542,43.8060,0.008\n6003,349.891296,-64.573555,317.972107,-49.786192,1,0.6069,0.5395,0.0894,42.4600,0.023\n6017,151.347656,4.181528,235.568369,44.259942,1,nan,0.5050,0.5080,42.2869,0.016\n6021,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.9940,0.0435,44.0841,0.008\n6022,352.711273,-63.823658,316.922299,-51.059403,1,nan,0.2460,0.2146,40.4626,0.024\n6023,348.595886,-63.072620,320.023289,-50.713060,1,nan,0.4699,0.0341,42.0998,0.021\n6028,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.7000,0.0060,43.1467,0.009\n6030,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.7964,0.0420,43.4906,0.008\n6076,0.190678,-45.783966,327.956322,-68.803772,1,nan,1.2728,0.0758,44.7483,0.005\n6087,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.8998,0.0507,43.8170,0.010\n6091,359.058563,-45.191612,330.695783,-68.844915,1,0.1812,0.1971,0.0181,39.9205,0.011\n6092,348.595886,-63.072620,320.023289,-50.713060,1,nan,1.2574,0.2139,44.7155,0.021\n6096,351.953644,-62.132156,318.777388,-52.347124,1,nan,1.1012,0.0271,44.3592,0.019\n6106,150.117188,2.836105,236.124718,42.483719,1,nan,0.5103,0.0193,42.3145,0.016\n6119,1.708861,-45.586655,325.688716,-69.520253,1,0.3975,0.4588,0.5979,42.0377,0.011\n6120,346.655182,-63.260487,320.952196,-50.040935,1,nan,1.1674,0.4165,44.5161,0.019\n6126,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.7902,0.0393,43.4696,0.007\n6135,348.586945,-64.573555,318.693903,-49.477869,1,0.8477,0.8451,0.1229,43.6492,0.018\n6151,150.292969,2.686724,236.427488,42.541447,1,nan,0.8598,0.0520,43.6952,0.016\n6162,2.071130,-45.191612,325.606223,-69.989264,1,nan,1.3977,0.0957,44.9997,0.011\n6164,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.5785,0.3260,45.3258,0.023\n6173,150.117188,2.836105,236.124718,42.483719,1,0.3881,0.3821,0.0142,41.5673,0.016\n6179,348.586945,-64.573555,318.693903,-49.477869,1,0.5991,0.5030,0.0291,42.2767,0.018\n6187,349.891296,-64.573555,317.972107,-49.786192,1,nan,1.1064,0.0831,44.3719,0.023\n6191,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.2328,0.0166,40.3267,0.006\n6197,151.523438,3.134927,236.900695,43.803170,1,nan,1.0306,0.0174,44.1812,0.019\n6205,33.574219,-6.579593,170.455585,-61.548219,1,0.5161,0.5873,0.0270,42.6825,0.021\n6211,0.949367,-45.586655,326.991548,-69.251686,1,0.5940,0.5967,0.0095,42.7247,0.013\n6217,35.683594,-5.379379,171.992947,-59.253501,1,nan,0.6309,0.0313,42.8717,0.020\n6223,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.2937,0.0805,40.9020,0.017\n6287,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.6517,0.0263,42.9572,0.009\n6289,52.207031,-26.610098,221.298836,-55.042928,1,0.2186,0.3975,0.2967,41.6682,0.014\n6293,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.8837,0.0179,43.7686,0.022\n6297,53.964844,-28.630989,225.142950,-53.813613,1,0.5674,0.6449,0.0377,42.9295,0.009\n6308,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.9701,0.0203,44.0187,0.021\n6313,349.966217,-62.696659,319.542989,-51.376556,1,nan,0.3825,0.0215,41.5700,0.021\n6343,33.398438,-4.331149,167.226341,-59.936551,1,nan,1.0126,0.0334,44.1339,0.018\n6345,0.190678,-45.783966,327.956322,-68.803772,1,nan,1.0255,0.0178,44.1678,0.005\n6347,149.414062,3.433834,234.919132,42.245550,1,nan,0.2243,0.0261,40.2360,0.027\n6350,33.574219,-5.379379,168.838090,-60.637536,1,nan,1.2142,0.1253,44.6216,0.017\n6352,150.996094,4.181528,235.291975,43.970869,1,nan,1.4079,0.0591,45.0191,0.015\n6354,34.804688,-5.829153,171.307861,-60.174401,1,0.3701,0.4966,0.3999,42.2433,0.023\n6368,351.299988,-62.320400,319.038597,-52.026867,1,nan,0.9252,0.0390,43.8918,0.018\n6369,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.5996,0.0136,42.7374,0.017\n6372,359.805206,-46.768478,327.135979,-67.829903,1,0.7964,0.7850,0.0422,43.4520,0.011\n6376,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.7511,0.0307,43.3345,0.018\n6378,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.6002,0.0209,42.7397,0.016\n6390,0.589520,-47.161343,325.385896,-67.769893,1,0.5421,0.4998,0.2593,42.2602,0.009\n6391,34.101562,-5.829153,170.247753,-60.638325,1,nan,1.1253,0.1662,44.4173,0.019\n6402,51.679688,-27.447618,222.618229,-55.642263,1,0.4580,0.4038,0.0453,41.7087,0.010\n6405,34.804688,-5.829153,171.307861,-60.174401,1,nan,1.3943,0.2658,44.9931,0.023\n6436,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.4164,0.3756,41.7878,0.006\n6437,33.925781,-5.979157,170.179895,-60.866303,1,0.4279,0.4344,0.0322,41.8968,0.022\n6450,0.190678,-45.783966,327.956322,-68.803772,1,nan,0.7339,0.0231,43.2726,0.005\n6458,352.711273,-63.823658,316.922299,-51.059403,1,1.1032,1.0971,0.1861,44.3491,0.024\n6460,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.3954,0.0255,41.6549,0.010\n6470,358.636353,-46.768478,328.890146,-67.388837,1,nan,1.0362,0.0170,44.1958,0.008\n6474,51.855469,-27.953188,223.543603,-55.561470,1,nan,0.7091,0.1441,43.1812,0.008\n6498,348.586945,-64.573555,318.693903,-49.477869,1,nan,1.1153,0.1670,44.3933,0.018\n6514,2.071130,-45.191612,325.606223,-69.989264,1,0.3119,0.3565,0.4554,41.3906,0.011\n6515,34.804688,-5.829153,171.307861,-60.174401,1,nan,1.1225,0.0927,44.4106,0.023\n6537,152.050781,2.985506,237.495952,44.143927,1,nan,1.2730,0.1522,44.7487,0.019\n6546,53.085938,-28.122234,224.100909,-54.509752,1,0.2509,0.3826,0.2720,41.5706,0.007\n6548,148.710938,2.836105,235.050801,41.328739,1,nan,1.1286,0.1579,44.4252,0.031\n6555,151.699219,3.583322,236.533224,44.205648,1,0.2053,0.2236,0.0177,40.2274,0.016\n6560,53.613281,-27.953188,223.929533,-54.024772,1,nan,0.7824,0.0662,43.4432,0.007\n6567,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.7820,0.0137,43.4419,0.017\n6568,51.679688,-27.447618,222.618229,-55.642263,1,nan,1.0385,0.0242,44.2017,0.010\n6575,33.398438,-3.732834,166.492280,-59.466614,1,0.5197,0.5580,0.0155,42.5483,0.022\n6598,0.589520,-47.161343,325.385896,-67.769893,1,nan,0.7918,0.0238,43.4750,0.009\n6606,151.523438,3.134927,236.900695,43.803170,1,nan,0.5715,0.0474,42.6108,0.019\n6620,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.9861,0.0153,44.0628,0.020\n6623,33.750000,-4.630479,168.146242,-59.949072,1,nan,0.8838,0.0439,43.7691,0.019\n6633,349.891296,-64.573555,317.972107,-49.786192,1,0.5312,0.5073,0.1220,42.2989,0.023\n6654,150.468750,3.732834,235.392208,43.283244,1,0.8058,0.8088,0.0111,43.5318,0.020\n6655,0.929752,-44.597992,328.531426,-70.083244,1,0.6631,0.6768,0.0337,43.0574,0.011\n6663,52.910156,-26.276812,220.926149,-54.363918,1,nan,0.4449,0.3362,41.9584,0.008\n6673,150.820312,3.732834,235.666318,43.572109,1,0.5028,0.5238,0.0312,42.3823,0.016\n6682,352.711273,-63.823658,316.922299,-51.059403,1,0.2960,0.5328,0.7393,42.4271,0.024\n6692,1.694561,-45.191612,326.278557,-69.858253,1,nan,0.4504,0.0611,41.9897,0.011\n6704,348.586945,-64.573555,318.693903,-49.477869,1,0.3131,0.2939,0.0390,40.9044,0.018\n6723,349.160583,-64.760857,318.219706,-49.458924,1,0.2401,0.2742,0.1134,40.7312,0.020\n6729,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.9192,0.0320,43.8743,0.011\n6742,349.615387,-63.636005,318.927246,-50.506542,1,nan,1.7240,0.2208,45.5614,0.018\n6756,359.816315,-44.003082,331.451340,-70.123054,1,nan,1.0236,0.0293,44.1629,0.013\n6814,152.050781,3.284369,237.157374,44.318466,1,0.6160,0.6226,0.6151,42.8365,0.019\n6821,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.3834,0.0765,41.5757,0.011\n6827,358.636353,-46.768478,328.890146,-67.388837,1,nan,1.1052,0.1737,44.3688,0.008\n6833,150.996094,2.388015,237.313912,42.939977,1,nan,1.4382,0.0724,45.0762,0.021\n6837,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.8994,0.0420,43.8158,0.017\n6847,33.574219,-5.079716,168.448505,-60.407218,1,nan,0.6653,0.0303,43.0120,0.016\n6855,346.500000,-62.320400,321.951129,-50.736054,1,nan,0.9313,0.0387,43.9092,0.020\n6856,1.363636,-46.768478,324.669342,-68.371416,1,0.3068,0.2935,0.0090,40.9003,0.008\n6876,150.644531,3.583322,235.698235,43.342784,1,0.3246,0.3216,0.4661,41.1300,0.018\n6878,32.871094,-4.780192,166.959493,-60.615132,1,0.5919,0.5848,0.1811,42.6713,0.017\n6884,52.910156,-27.279613,222.625192,-54.536648,1,0.2612,0.2825,1.1744,40.8053,0.007\n6897,359.814819,-44.399834,330.775011,-69.801007,1,nan,0.6130,0.0239,42.7954,0.009\n6907,151.171875,1.342993,238.602520,42.464379,1,0.4791,0.5900,0.3591,42.6946,0.026\n6911,0.929752,-44.597992,328.531426,-70.083244,1,0.3125,0.3006,0.0066,40.9604,0.011\n6919,346.655182,-63.260487,320.952196,-50.040935,1,nan,1.6301,0.2474,45.4117,0.019\n6927,151.171875,2.537361,237.288526,43.169764,1,nan,0.5389,0.2528,42.4568,0.024\n6930,52.207031,-26.610098,221.298836,-55.042928,1,0.6478,0.5687,0.2240,42.5980,0.014\n6932,52.910156,-27.953188,223.774083,-54.639214,1,nan,1.3042,0.0324,44.8137,0.007\n6937,0.965665,-46.375080,325.845907,-68.579427,1,nan,0.8820,0.0370,43.7636,0.007\n6949,346.130127,-63.072620,321.423103,-50.042305,1,nan,0.8422,0.0195,43.6399,0.020\n6951,0.589520,-47.161343,325.385896,-67.769893,1,0.5372,0.5224,0.0077,42.3758,0.009\n6952,150.117188,2.238686,236.784618,42.139082,1,nan,1.7307,0.2954,45.5719,0.016\n6964,359.816315,-44.003082,331.451340,-70.123054,1,nan,1.1443,0.2764,44.4622,0.013\n6982,358.312500,-44.993881,332.185785,-68.685906,1,0.3227,0.3189,0.0263,41.1089,0.009\n6988,150.996094,2.985506,236.647967,43.287350,1,nan,0.8042,0.0261,43.5165,0.020\n7013,34.277344,-5.679190,170.314930,-60.410322,1,nan,0.5236,0.0416,42.3817,0.020\n7017,349.429535,-62.508568,320.039643,-51.393745,1,nan,0.3210,0.0095,41.1251,0.020\n7021,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.9959,0.0260,44.0892,0.009\n7032,51.328125,-27.784405,223.130589,-55.999499,1,nan,0.8584,0.0214,43.6910,0.013\n7055,34.277344,-5.679190,170.314930,-60.410322,1,nan,0.9347,0.0439,43.9190,0.020\n7060,52.207031,-28.291550,224.208534,-55.300157,1,nan,0.6635,0.0179,43.0048,0.007\n7065,52.910156,-26.276812,220.926149,-54.363918,1,nan,1.5901,0.1813,45.3453,0.008\n7129,51.328125,-27.447618,222.535046,-55.950727,1,nan,1.1651,0.0362,44.5107,0.013\n7140,150.820312,3.134927,236.341348,43.230123,1,nan,0.9363,0.0122,43.9237,0.016\n7161,1.363636,-46.768478,324.669342,-68.371416,1,0.3784,0.3433,0.1675,41.2947,0.008\n7162,52.910156,-27.953188,223.774083,-54.639214,1,0.3593,0.4691,0.9910,42.0956,0.007\n7163,53.085938,-27.784405,223.525509,-54.460748,1,nan,1.1234,0.1962,44.4129,0.007\n7172,151.171875,1.342993,238.602520,42.464379,1,nan,0.6339,0.0129,42.8842,0.026\n7175,347.617462,-62.508568,321.121462,-50.904708,1,nan,1.0926,0.1310,44.3382,0.019\n7224,53.964844,-28.630989,225.142950,-53.813613,1,nan,1.2376,0.0743,44.6730,0.009\n7226,52.207031,-26.610098,221.298836,-55.042928,1,0.3847,0.3273,0.8615,41.1744,0.014\n7233,35.683594,-5.379379,171.992947,-59.253501,1,nan,1.2494,0.2608,44.6985,0.020\n7241,150.820312,3.732834,235.666318,43.572109,1,nan,0.7200,0.0438,43.2218,0.016\n7246,151.347656,4.181528,235.568369,44.259942,1,nan,0.6039,0.0147,42.7560,0.016\n7265,151.171875,2.238686,237.619933,42.994783,1,nan,0.7073,0.0647,43.1743,0.024\n7275,1.708861,-45.586655,325.688716,-69.520253,1,0.4268,0.4094,0.0089,41.7441,0.011\n7282,51.855469,-28.630989,224.733260,-55.649872,1,nan,1.4386,0.0998,45.0769,0.009\n7292,33.574219,-5.379379,168.838090,-60.637536,1,nan,0.7780,0.0880,43.4282,0.017\n7297,0.574468,-45.981140,327.041068,-68.778764,1,0.5290,0.5535,0.4055,42.5268,0.006\n7326,351.299988,-62.320400,319.038597,-52.026867,1,0.2394,0.2364,0.0161,40.3644,0.018\n7344,33.398438,-4.331149,167.226341,-59.936551,1,0.5794,0.5951,0.0111,42.7176,0.018\n7378,347.846710,-64.760857,318.929827,-49.143596,1,nan,1.0318,0.1438,44.1843,0.019\n7381,52.031250,-26.443335,220.963669,-55.168557,1,0.3373,0.5151,0.2782,42.3389,0.014\n7385,52.910156,-26.276812,220.926149,-54.363918,1,0.7536,0.7669,0.0410,43.3898,0.008\n7389,359.446716,-44.201530,331.730015,-69.805709,1,0.6180,0.6621,0.1010,42.9994,0.010\n7447,0.190678,-45.783966,327.956322,-68.803772,1,nan,0.6276,0.0506,42.8577,0.005\n7451,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.7431,0.0243,43.3058,0.019\n7464,346.500000,-62.320400,321.951129,-50.736054,1,0.5846,0.6343,0.0145,42.8857,0.020\n7479,359.058563,-45.191612,330.695783,-68.844915,1,0.3899,0.3252,0.7307,41.1585,0.011\n7496,0.190678,-45.783966,327.956322,-68.803772,1,nan,0.6583,0.0121,42.9840,0.005\n7508,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.1198,0.0531,44.4042,0.023\n7515,150.820312,3.134927,236.341348,43.230123,1,nan,1.0715,0.0116,44.2857,0.016\n7535,33.222656,-4.780192,167.515653,-60.396584,1,nan,1.4866,0.1394,45.1649,0.018\n7556,53.085938,-27.784405,223.525509,-54.460748,1,0.4003,0.4870,0.4078,42.1925,0.007\n7590,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.7702,0.0277,43.4013,0.010\n7593,53.085938,-27.784405,223.525509,-54.460748,1,nan,1.1395,0.0880,44.4511,0.007\n7596,358.312500,-44.993881,332.185785,-68.685906,1,nan,1.0214,0.2333,44.1572,0.009\n7597,152.050781,2.985506,237.495952,44.143927,1,0.7873,0.8649,0.0311,43.7112,0.019\n7599,346.130127,-63.072620,321.423103,-50.042305,1,nan,0.6460,0.0576,42.9341,0.020\n7616,151.347656,3.583322,236.252362,43.918627,1,nan,0.6020,0.0567,42.7478,0.015\n7651,2.097458,-45.783966,324.737840,-69.478613,1,0.4739,0.4618,0.0214,42.0550,0.011\n7655,150.117188,2.836105,236.124718,42.483719,1,nan,0.7487,0.0442,43.3259,0.016\n7657,352.398651,-62.696659,318.017427,-51.967966,1,nan,1.1347,0.0316,44.4397,0.020\n7667,53.613281,-27.953188,223.929533,-54.024772,1,nan,0.6882,0.0180,43.1017,0.007\n7693,349.615387,-63.636005,318.927246,-50.506542,1,0.5467,0.5108,0.1636,42.3168,0.018\n7705,347.013428,-62.508568,321.472056,-50.735330,1,0.6672,0.6780,0.0682,43.0621,0.018\n7723,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.7023,0.0297,43.1555,0.018\n7764,349.429535,-62.508568,320.039643,-51.393745,1,nan,0.5974,0.0081,42.7277,0.020\n7771,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.5341,0.0966,42.4333,0.017\n7775,349.615387,-63.636005,318.927246,-50.506542,1,0.3416,0.3512,0.0280,41.3526,0.018\n7788,346.130127,-63.072620,321.423103,-50.042305,1,nan,1.3320,0.0596,44.8704,0.020\n7793,33.925781,-5.979157,170.179895,-60.866303,1,nan,1.1112,0.0841,44.3835,0.022\n7806,152.050781,2.985506,237.495952,44.143927,1,0.1438,1.9587,0.6082,45.9018,0.019\n7809,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.6914,0.0120,43.1142,0.009\n7820,148.710938,2.836105,235.050801,41.328739,1,0.3550,0.2846,0.0753,40.8234,0.031\n7830,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.6696,0.0570,43.0291,0.011\n7858,1.363636,-46.768478,324.669342,-68.371416,1,0.4673,0.4927,0.0100,42.2231,0.008\n7867,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.6117,0.0088,42.7899,0.010\n7873,150.468750,1.641510,237.714575,42.075234,1,0.5708,0.6019,0.0233,42.7474,0.017\n7920,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.8342,0.0228,43.6144,0.009\n7928,151.171875,1.342993,238.602520,42.464379,1,nan,0.8590,0.0235,43.6927,0.026\n7931,33.574219,-6.579593,170.455585,-61.548219,1,nan,0.5225,0.4053,42.3762,0.021\n7940,52.910156,-25.944481,220.366350,-54.301439,1,0.6969,0.7104,0.0284,43.1859,0.010\n7942,0.190678,-45.783966,327.956322,-68.803772,1,0.2814,0.3115,0.5654,41.0497,0.005\n7957,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.9375,0.0504,43.9271,0.011\n7977,33.398438,-3.732834,166.492280,-59.466614,1,nan,1.0780,0.0578,44.3020,0.022\n7980,358.312500,-44.993881,332.185785,-68.685906,1,nan,1.2456,0.1373,44.6902,0.009\n7982,346.276581,-64.011238,320.448031,-49.344136,1,0.4520,0.4716,0.0369,42.1091,0.019\n7983,0.949367,-45.586655,326.991548,-69.251686,1,nan,1.0778,0.0319,44.3015,0.013\n7984,34.453125,-5.229529,169.987075,-59.956185,1,nan,1.4448,0.2401,45.0886,0.019\n7990,1.723404,-45.981140,325.117958,-69.180825,1,nan,0.8379,0.0368,43.6263,0.010\n8019,33.574219,-6.579593,170.455585,-61.548219,1,0.2036,0.2032,0.0104,39.9944,0.021\n8034,52.207031,-26.610098,221.298836,-55.042928,1,nan,1.5348,0.1021,45.2506,0.014\n8036,51.328125,-27.447618,222.535046,-55.950727,1,0.3381,0.5336,0.3006,42.4311,0.013\n8037,0.189873,-45.586655,328.254458,-68.969298,1,nan,0.8038,0.0994,43.5152,0.007\n8054,52.910156,-25.944481,220.366350,-54.301439,1,nan,0.8669,0.0250,43.7173,0.010\n8057,359.058563,-45.191612,330.695783,-68.844915,1,nan,0.7677,0.0225,43.3927,0.011\n8058,52.031250,-26.443335,220.963669,-55.168557,1,nan,1.0909,0.0215,44.3339,0.014\n8063,150.468750,3.732834,235.392208,43.283244,1,nan,1.0023,0.2042,44.1065,0.020\n8094,149.414062,2.238686,236.239766,41.565558,1,nan,0.5647,0.0343,42.5793,0.017\n8102,2.071130,-45.191612,325.606223,-69.989264,1,nan,1.1741,0.0373,44.5314,0.011\n8129,359.814819,-44.399834,330.775011,-69.801007,1,0.4569,0.4721,0.2974,42.1119,0.009\n8135,0.190678,-45.783966,327.956322,-68.803772,1,0.2615,0.2935,0.0129,40.9008,0.005\n8153,350.230255,-61.943836,320.053946,-52.070537,1,0.2924,0.3076,0.0101,41.0184,0.017\n8165,33.750000,-4.630479,168.146242,-59.949072,1,nan,1.0355,0.1353,44.1940,0.019\n8166,150.292969,2.686724,236.427488,42.541447,1,nan,1.2423,0.1331,44.6832,0.016\n8179,52.207031,-28.630989,224.800211,-55.343637,1,nan,0.6201,0.0376,42.8258,0.009\n8181,0.965665,-46.375080,325.845907,-68.579427,1,0.6159,0.6147,0.0327,42.8026,0.007\n8182,348.586945,-64.573555,318.693903,-49.477869,1,nan,1.0126,0.0872,44.1337,0.018\n8196,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.6223,0.0130,42.8354,0.007\n8197,33.750000,-4.630479,168.146242,-59.949072,1,nan,0.8678,0.0526,43.7202,0.019\n8201,53.261719,-27.615883,223.280041,-54.281374,1,nan,1.0299,0.0450,44.1792,0.006\n8242,150.468750,1.641510,237.714575,42.075234,1,nan,0.9643,0.0352,44.0026,0.017\n8250,51.855469,-27.953188,223.543603,-55.561470,1,nan,1.3103,0.1902,44.8263,0.008\n8252,359.446716,-44.201530,331.730015,-69.805709,1,nan,0.8891,0.1482,43.7851,0.010\n8263,32.695312,-4.929937,166.868469,-60.841230,1,0.7023,0.7653,0.0398,43.3844,0.018\n8271,53.964844,-28.630989,225.142950,-53.813613,1,nan,1.2974,0.0994,44.7998,0.009\n8283,349.160583,-64.760857,318.219706,-49.458924,1,nan,1.1567,0.0332,44.4912,0.020\n8284,35.859375,-4.630479,171.270769,-58.580806,1,0.4045,0.4502,0.0352,41.9889,0.022\n8315,152.050781,3.284369,237.157374,44.318466,1,0.7654,0.7712,0.0372,43.4049,0.019\n8322,150.117188,2.836105,236.124718,42.483719,1,0.0718,0.0803,0.1025,37.8083,0.016\n8329,52.031250,-26.443335,220.963669,-55.168557,1,nan,1.5343,0.1326,45.2496,0.014\n8331,33.574219,-5.079716,168.448505,-60.407218,1,0.6347,0.6283,0.0176,42.8605,0.016\n8332,35.859375,-4.630479,171.270769,-58.580806,1,0.5166,0.5551,0.0652,42.5347,0.022\n8345,359.058563,-45.191612,330.695783,-68.844915,1,0.5651,0.5479,0.0133,42.5004,0.011\n8361,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.9078,0.0244,43.8409,0.009\n8386,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.9630,0.0958,43.9990,0.010\n8388,33.750000,-4.630479,168.146242,-59.949072,1,nan,1.0142,0.0893,44.1381,0.019\n8399,1.363636,-46.768478,324.669342,-68.371416,1,0.6489,0.6460,0.0193,42.9339,0.008\n8401,152.050781,2.985506,237.495952,44.143927,1,0.5695,0.5583,0.0070,42.5495,0.019\n8404,52.910156,-25.944481,220.366350,-54.301439,1,nan,0.9683,0.0275,44.0136,0.010\n8413,349.966217,-62.696659,319.542989,-51.376556,1,nan,0.9296,0.2134,43.9042,0.021\n8420,359.811707,-45.191612,329.485675,-69.150905,1,nan,1.2904,0.0367,44.7852,0.010\n8430,51.855469,-26.276812,220.627031,-55.293792,1,nan,1.3057,0.0606,44.8168,0.014\n8441,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.6746,0.0423,43.0487,0.018\n8442,351.953644,-62.132156,318.777388,-52.347124,1,nan,0.0000,0.0000,nan,0.019\n8455,151.347656,3.583322,236.252362,43.918627,1,nan,1.0052,0.0667,44.1141,0.015\n8463,150.644531,3.583322,235.698235,43.342784,1,nan,0.9599,0.0304,43.9903,0.018\n8469,348.586945,-64.573555,318.693903,-49.477869,1,nan,0.7521,0.0183,43.3379,0.018\n8487,53.613281,-26.944359,222.237403,-53.863858,1,nan,0.6150,0.0095,42.8042,0.009\n8492,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.9894,0.1246,44.0715,0.006\n8512,52.558594,-27.279613,222.538937,-54.845107,1,0.8384,0.8562,0.0216,43.6841,0.008\n8513,33.574219,-5.079716,168.448505,-60.407218,1,0.3943,0.4243,0.0320,41.8362,0.016\n8521,33.750000,-4.630479,168.146242,-59.949072,1,0.4536,0.4521,0.0212,41.9996,0.019\n8534,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.8932,0.0144,43.7973,0.018\n8548,151.347656,3.583322,236.252362,43.918627,1,0.4318,0.5090,0.0311,42.3075,0.015\n8549,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.7646,0.0436,43.3818,0.008\n8553,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.4066,1.0187,41.7265,0.021\n8565,359.058563,-45.191612,330.695783,-68.844915,1,0.6011,0.5779,0.1187,42.6400,0.011\n8592,53.613281,-27.953188,223.929533,-54.024772,1,nan,0.8871,0.0176,43.7788,0.007\n8619,53.789062,-27.784405,223.685697,-53.845803,1,nan,1.4192,0.1271,45.0406,0.009\n8625,351.734680,-62.884678,318.284128,-51.651217,1,0.3067,0.3063,0.0081,41.0077,0.019\n8644,52.207031,-28.630989,224.800211,-55.343637,1,0.6866,0.7308,0.0554,43.2613,0.009\n8646,1.694561,-45.191612,326.278557,-69.858253,1,nan,1.3300,0.1125,44.8664,0.011\n8661,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.9744,0.0308,44.0307,0.017\n8664,52.558594,-27.279613,222.538937,-54.845107,1,nan,0.5444,0.0180,42.4836,0.008\n8665,151.171875,2.537361,237.288526,43.169764,1,nan,1.2262,0.0861,44.6480,0.024\n8672,35.683594,-5.379379,171.992947,-59.253501,1,nan,1.1123,0.0767,44.3861,0.020\n8674,53.789062,-27.784405,223.685697,-53.845803,1,0.9353,0.7665,0.0891,43.3883,0.009\n8682,349.046051,-61.943836,320.796530,-51.753706,1,nan,1.0543,0.0550,44.2423,0.017\n8683,52.910156,-27.953188,223.774083,-54.639214,1,0.6644,0.6624,0.0227,43.0003,0.007\n8684,51.855469,-27.953188,223.543603,-55.561470,1,nan,0.8518,0.0200,43.6702,0.008\n8691,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.6565,0.0183,42.9769,0.019\n8697,351.734680,-62.884678,318.284128,-51.651217,1,nan,1.4889,0.0787,45.1691,0.019\n8701,347.812500,-63.448284,320.128971,-50.202348,1,0.2506,0.2395,0.0127,40.3969,0.021\n8702,1.753247,-46.768478,324.030235,-68.498041,1,nan,1.2922,0.0134,44.7890,0.014\n8705,150.820312,3.732834,235.666318,43.572109,1,nan,0.9669,0.0557,44.0099,0.016\n8717,52.207031,-28.291550,224.208534,-55.300157,1,nan,1.4656,0.3245,45.1269,0.007\n8724,2.071130,-45.191612,325.606223,-69.989264,1,0.2852,0.2994,0.0072,40.9502,0.011\n8730,348.586945,-64.573555,318.693903,-49.477869,1,0.4718,2.9626,1.3143,46.9933,0.018\n8734,348.586945,-64.573555,318.693903,-49.477869,1,0.4341,0.4524,0.0446,42.0014,0.018\n8738,52.910156,-26.276812,220.926149,-54.363918,1,nan,1.0394,0.1965,44.2041,0.008\n8739,53.789062,-27.784405,223.685697,-53.845803,1,nan,1.1723,0.2076,44.5273,0.009\n8767,150.820312,3.134927,236.341348,43.230123,1,nan,0.6681,0.0088,43.0230,0.016\n8803,2.097458,-45.783966,324.737840,-69.478613,1,nan,1.0540,0.2342,44.2414,0.011\n8805,0.965665,-46.375080,325.845907,-68.579427,1,nan,1.1091,0.1745,44.3785,0.007\n8806,352.398651,-62.696659,318.017427,-51.967966,1,nan,1.3680,0.2494,44.9420,0.020\n8812,151.347656,4.181528,235.568369,44.259942,1,nan,1.2164,0.1126,44.6264,0.016\n8816,149.414062,3.433834,234.919132,42.245550,1,nan,0.8846,0.0380,43.7713,0.027\n8821,35.859375,-4.630479,171.270769,-58.580806,1,nan,0.7988,0.0369,43.4987,0.022\n8837,351.299988,-62.320400,319.038597,-52.026867,1,nan,0.5320,0.0097,42.4234,0.018\n8850,346.562500,-63.448284,320.824720,-49.866957,1,nan,1.0271,0.0826,44.1720,0.021\n8857,150.117188,2.238686,236.784618,42.139082,1,nan,1.0907,0.1270,44.3334,0.016\n8861,351.321442,-64.198746,317.458993,-50.429931,1,nan,0.3476,1.1210,41.3264,0.023\n8871,32.695312,-4.929937,166.868469,-60.841230,1,nan,1.8695,0.2279,45.7776,0.018\n8883,359.446716,-44.201530,331.730015,-69.805709,1,nan,1.3015,0.2137,44.8082,0.010\n8902,151.171875,2.238686,237.619933,42.994783,1,nan,0.4347,0.0093,41.8984,0.024\n8904,347.812500,-63.448284,320.128971,-50.202348,1,nan,1.1116,0.0337,44.3843,0.021\n8917,34.101562,-5.829153,170.247753,-60.638325,1,0.5976,0.5969,0.0088,42.7253,0.019\n8919,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.9152,0.0404,43.8626,0.010\n8921,150.644531,3.583322,235.698235,43.342784,1,0.5296,0.5248,0.0115,42.3874,0.018\n8933,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.0000,0.0000,nan,0.017\n8947,33.574219,-5.379379,168.838090,-60.637536,1,0.3757,2.9135,0.9540,46.9496,0.017\n8962,2.071130,-45.191612,325.606223,-69.989264,1,nan,1.2366,0.1848,44.6708,0.011\n8978,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.5523,0.1275,42.5212,0.016\n8983,53.085938,-28.122234,224.100909,-54.509752,1,0.4014,0.3910,0.0331,41.6261,0.007\n8987,0.574468,-45.981140,327.041068,-68.778764,1,nan,1.0629,0.0133,44.2641,0.006\n9033,0.190678,-45.783966,327.956322,-68.803772,1,0.3132,0.2593,0.0293,40.5923,0.005\n9041,347.617462,-62.508568,321.121462,-50.904708,1,nan,1.0424,0.0183,44.2118,0.019\n9051,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.9828,0.0093,44.0536,0.009\n9053,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.5661,0.0926,42.5860,0.011\n9080,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.8653,0.0487,43.7124,0.022\n9084,52.910156,-26.276812,220.926149,-54.363918,1,nan,0.8458,0.0301,43.6512,0.008\n9099,352.398651,-62.696659,318.017427,-51.967966,1,nan,0.5841,0.0141,42.6684,0.020\n9107,150.468750,1.641510,237.714575,42.075234,1,0.4687,0.5719,0.2556,42.6127,0.017\n9110,0.929752,-44.597992,328.531426,-70.083244,1,0.2531,0.2978,0.0321,40.9371,0.011\n9115,53.261719,-27.615883,223.280041,-54.281374,1,nan,0.5434,0.6581,42.4785,0.006\n9124,34.277344,-5.679190,170.314930,-60.410322,1,nan,1.5382,0.2744,45.2564,0.020\n9138,352.132874,-63.636005,317.424173,-51.095855,1,0.2703,0.2829,0.0150,40.8088,0.021\n9165,349.429535,-62.508568,320.039643,-51.393745,1,nan,1.1058,0.0670,44.3703,0.020\n9167,346.130127,-63.072620,321.423103,-50.042305,1,nan,1.1829,0.0376,44.5516,0.020\n9170,346.500000,-62.320400,321.951129,-50.736054,1,nan,0.5880,0.0130,42.6856,0.020\n9187,35.859375,-4.630479,171.270769,-58.580806,1,nan,0.3500,0.0275,41.3440,0.022\n9197,2.071130,-45.191612,325.606223,-69.989264,1,nan,1.3572,0.2236,44.9207,0.011\n9209,150.644531,3.583322,235.698235,43.342784,1,nan,0.5872,0.0397,42.6820,0.018\n9210,359.415588,-46.768478,327.729895,-67.686097,1,nan,1.2507,0.0994,44.7013,0.009\n9216,359.811707,-45.191612,329.485675,-69.150905,1,0.9399,1.5923,0.1936,45.3491,0.010\n9220,52.207031,-28.630989,224.800211,-55.343637,1,nan,0.6692,0.0244,43.0276,0.009\n9229,52.910156,-27.953188,223.774083,-54.639214,1,0.3075,0.3138,0.0076,41.0683,0.007\n9237,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.5799,0.2003,42.6494,0.011\n9277,149.414062,1.940072,236.565366,41.393323,1,nan,0.7257,0.0452,43.2426,0.018\n9293,151.171875,2.238686,237.619933,42.994783,1,0.5639,0.5209,0.5082,42.3682,0.024\n9302,351.321442,-64.198746,317.458993,-50.429931,1,nan,1.3310,0.0582,44.8683,0.023\n9303,34.453125,-5.229529,169.987075,-59.956185,1,nan,1.0260,0.0163,44.1691,0.019\n9316,33.574219,-5.379379,168.838090,-60.637536,1,nan,0.6163,0.0159,42.8095,0.017\n9322,33.574219,-5.079716,168.448505,-60.407218,1,nan,1.3730,0.1469,44.9518,0.016\n9346,351.321442,-64.198746,317.458993,-50.429931,1,nan,1.3099,0.0995,44.8254,0.023\n9362,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.2692,0.0170,40.6858,0.008\n9365,34.277344,-5.679190,170.314930,-60.410322,1,0.2935,0.2933,0.0217,40.8992,0.020\n9374,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.7189,0.0162,43.2179,0.007\n9380,346.655182,-63.260487,320.952196,-50.040935,1,nan,0.7849,0.0428,43.4518,0.019\n9408,346.500000,-62.320400,321.951129,-50.736054,1,nan,0.9664,0.0147,44.0085,0.020\n9423,351.382965,-64.011238,317.574052,-50.604657,1,nan,0.5558,0.0329,42.5379,0.023\n9444,349.046051,-61.943836,320.796530,-51.753706,1,nan,1.1578,0.0579,44.4939,0.017\n9451,351.382965,-64.011238,317.574052,-50.604657,1,nan,0.9855,0.0096,44.0612,0.023\n9461,358.665253,-45.783966,330.353593,-68.203652,1,nan,1.1498,0.0929,44.4752,0.009\n9468,51.855469,-26.276812,220.627031,-55.293792,1,nan,0.7969,0.0460,43.4923,0.014\n9469,32.871094,-4.780192,166.959493,-60.615132,1,0.8180,0.7700,0.0220,43.4006,0.017\n9482,53.085938,-28.122234,224.100909,-54.509752,1,nan,0.7314,0.0368,43.2634,0.007\n9497,51.328125,-27.784405,223.130589,-55.999499,1,0.4664,0.4234,0.0109,41.8306,0.013\n9507,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.7375,0.0178,43.2857,0.010\n9550,346.130127,-63.072620,321.423103,-50.042305,1,0.6995,0.7070,0.0097,43.1732,0.020\n9554,51.679688,-27.447618,222.618229,-55.642263,1,0.6646,0.6800,0.0238,43.0701,0.010\n9562,349.160583,-64.760857,318.219706,-49.458924,1,nan,0.3891,0.0546,41.6136,0.020\n9566,52.910156,-25.944481,220.366350,-54.301439,1,nan,0.8552,0.0311,43.6808,0.010\n9588,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.8782,0.0321,43.7520,0.009\n9597,53.964844,-28.630989,225.142950,-53.813613,1,nan,0.8452,0.2344,43.6493,0.009\n9603,349.046051,-61.943836,320.796530,-51.753706,1,nan,1.6743,0.3261,45.4832,0.017\n9626,150.644531,3.583322,235.698235,43.342784,1,nan,1.0665,0.0241,44.2730,0.018\n9652,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.6624,0.0296,43.0006,0.006\n9662,53.613281,-26.944359,222.237403,-53.863858,1,nan,1.4334,0.0635,45.0673,0.009\n9676,1.753247,-46.768478,324.030235,-68.498041,1,0.6764,0.7221,0.0336,43.2295,0.014\n9678,151.523438,3.134927,236.900695,43.803170,1,nan,1.1508,0.7055,44.4777,0.019\n9699,2.097458,-45.783966,324.737840,-69.478613,1,nan,0.6400,0.1023,42.9095,0.011\n9705,352.711273,-63.823658,316.922299,-51.059403,1,0.5977,0.5685,0.0384,42.5972,0.024\n9720,351.734680,-62.884678,318.284128,-51.651217,1,0.6071,0.5353,0.0256,42.4395,0.019\n9725,350.230255,-61.943836,320.053946,-52.070537,1,nan,1.2775,0.0723,44.7581,0.017\n9726,358.312500,-44.993881,332.185785,-68.685906,1,nan,0.7580,0.0243,43.3588,0.009\n9761,52.207031,-28.291550,224.208534,-55.300157,1,nan,1.4229,0.1718,45.0476,0.007\n9772,53.613281,-26.944359,222.237403,-53.863858,1,nan,0.5526,0.0238,42.5224,0.009\n9776,351.953644,-62.132156,318.777388,-52.347124,1,0.4229,0.4114,0.0259,41.7568,0.019\n9782,149.589844,3.583322,234.885369,42.474696,1,nan,1.1776,0.1586,44.5394,0.024\n9799,346.276581,-64.011238,320.448031,-49.344136,1,nan,0.9878,0.3118,44.0673,0.019\n9806,0.965665,-46.375080,325.845907,-68.579427,1,0.4619,0.5359,0.0762,42.4424,0.007\n9809,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.6301,0.0342,42.8682,0.021\n9839,53.613281,-27.953188,223.929533,-54.024772,1,nan,1.5628,0.1334,45.2989,0.007\n9864,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.7718,0.0734,43.4067,0.011\n9865,150.117188,3.732834,235.120533,42.993809,1,0.4651,2.3854,0.8502,46.4240,0.020\n9868,54.667969,-27.615883,223.610785,-53.050840,1,nan,0.7611,0.0247,43.3696,0.009\n9884,51.328125,-27.447618,222.535046,-55.950727,1,0.6206,0.6001,0.0161,42.7395,0.013\n9893,33.750000,-4.630479,168.146242,-59.949072,1,0.4883,0.5053,0.0956,42.2888,0.019\n9916,349.891296,-64.573555,317.972107,-49.786192,1,nan,0.6122,0.0358,42.7920,0.023\n9924,349.285706,-62.884678,319.786163,-51.046461,1,0.7986,0.8711,0.0680,43.7303,0.018\n9950,51.328125,-27.447618,222.535046,-55.950727,1,nan,0.5462,0.0962,42.4921,0.013\n9951,346.276581,-64.011238,320.448031,-49.344136,1,0.8417,0.7427,0.0547,43.3044,0.019\n9961,53.789062,-27.784405,223.685697,-53.845803,1,nan,1.0492,0.2680,44.2292,0.009\n9967,150.292969,2.686724,236.427488,42.541447,1,nan,1.4395,0.2623,45.0786,0.016\n9989,346.130127,-63.072620,321.423103,-50.042305,1,nan,0.9517,0.2190,43.9673,0.020\n9998,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.5127,0.0376,42.3265,0.011\n10021,52.207031,-28.630989,224.800211,-55.343637,1,nan,1.4337,0.0978,45.0679,0.009\n10056,52.558594,-27.279613,222.538937,-54.845107,1,0.4511,0.3564,0.0517,41.3900,0.008\n10071,0.949367,-45.586655,326.991548,-69.251686,1,0.3219,0.4590,0.0597,42.0387,0.013\n10102,359.058563,-45.191612,330.695783,-68.844915,1,nan,0.7450,0.0248,43.3128,0.011\n10103,359.805206,-46.768478,327.135979,-67.829903,1,nan,0.8130,0.0375,43.5455,0.011\n10110,54.667969,-27.615883,223.610785,-53.050840,1,0.7638,0.8007,0.0367,43.5050,0.009\n10119,351.953644,-62.132156,318.777388,-52.347124,1,0.6209,0.6434,0.0311,42.9234,0.019\n10133,352.398651,-62.696659,318.017427,-51.967966,1,nan,0.9030,0.0216,43.8266,0.020\n10139,350.230255,-61.943836,320.053946,-52.070537,1,nan,1.3210,0.0562,44.8481,0.017\n10146,346.500000,-62.320400,321.951129,-50.736054,1,nan,0.5522,0.0066,42.5205,0.020\n10150,359.814819,-44.399834,330.775011,-69.801007,1,0.4395,2.7379,0.4044,46.7868,0.009\n10155,0.589520,-47.161343,325.385896,-67.769893,1,nan,1.0100,0.2301,44.1270,0.009\n10180,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.5829,0.0112,42.6631,0.010\n10213,359.058563,-45.191612,330.695783,-68.844915,1,nan,0.5634,0.0192,42.5732,0.011\n10218,52.558594,-27.279613,222.538937,-54.845107,1,0.3131,0.3414,0.0364,41.2809,0.008\n10228,151.171875,1.342993,238.602520,42.464379,1,nan,0.7524,0.0449,43.3390,0.026\n10244,53.789062,-27.784405,223.685697,-53.845803,1,nan,0.7687,0.0207,43.3962,0.009\n"
},
{
"path": "examples/data/plasticc_training_set_1k.csv",
"content": "object_id,mjd,passband,flux,flux_err,detected\n615,59750.4229,2,-544.810303,3.622952,1\n615,59750.4306,1,-816.434326,5.553370,1\n615,59750.4383,3,-471.385529,3.801213,1\n615,59750.4450,4,-388.984985,11.395031,1\n615,59752.4070,2,-681.858887,4.041204,1\n615,59752.4147,1,-1061.457031,6.472994,1\n615,59752.4224,3,-524.954590,3.552751,1\n615,59752.4334,4,-393.480225,3.599346,1\n615,59752.4435,5,-355.886780,10.421921,1\n615,59767.2968,2,-548.013550,3.462291,1\n615,59767.3045,1,-815.188599,5.293019,1\n615,59767.3122,3,-475.516052,3.340643,1\n615,59767.3233,4,-405.663818,3.496113,1\n615,59767.3343,5,-421.199066,6.377517,1\n615,59770.2179,2,-554.903198,3.927843,1\n615,59770.2256,1,-820.042786,5.875329,1\n615,59770.2334,3,-477.004730,3.736262,1\n615,59770.2445,4,-400.270386,3.834955,1\n615,59770.2557,5,-415.286896,7.435979,1\n615,59779.3188,2,-630.523682,4.333287,1\n615,59779.3265,1,-921.002502,6.306800,1\n615,59779.3342,3,-518.533997,3.915225,1\n615,59779.3452,4,-422.184509,4.089213,1\n615,59779.3562,5,-422.815094,8.124096,1\n615,59782.1897,2,-280.039520,2.819228,1\n615,59782.1974,1,-449.095612,4.028310,1\n615,59782.2051,3,-316.704865,3.491153,1\n615,59782.2162,4,-332.885437,4.021619,1\n615,59782.2274,5,-365.075775,8.514805,1\n615,59797.2861,2,391.399231,3.098059,1\n615,59797.2938,1,35.511822,3.163646,1\n615,59797.3015,3,330.623901,3.388776,1\n615,59797.3126,4,360.397858,3.980607,1\n615,59797.3237,5,369.439667,8.207490,1\n615,59800.3168,2,168.739899,3.128495,1\n615,59800.3244,1,129.541901,4.358776,1\n615,59800.3320,3,30.120724,3.396606,1\n615,59800.3429,4,-60.942333,3.704243,1\n615,59800.3539,5,-128.920334,7.495701,1\n615,59807.1738,2,-256.660980,2.781354,1\n615,59807.1815,1,-420.796417,4.037735,1\n615,59807.1892,3,-298.936859,3.041390,1\n615,59807.2003,4,-311.977783,3.318007,1\n615,59807.2114,5,-344.536072,6.367201,1\n615,59810.1045,2,-342.819763,2.808321,1\n615,59810.1122,1,-527.020325,4.204173,1\n615,59810.1200,3,-363.282532,3.237536,1\n615,59810.1311,4,-348.628662,3.774855,1\n615,59810.1422,5,-391.271271,7.657067,1\n615,59813.1044,2,-678.045715,4.032819,1\n615,59813.1122,1,-1100.440063,6.709106,1\n615,59813.1199,3,-506.687408,3.547398,1\n615,59813.1310,4,-304.049713,3.590496,1\n615,59813.1422,5,-187.285919,6.984019,1\n615,59819.1532,0,6.878784,3.633152,0\n615,59820.1047,0,39.364853,3.775619,1\n615,59821.1026,0,-10.422381,4.172683,0\n615,59822.1105,0,-65.485130,4.362876,1\n615,59823.1505,0,-113.349159,4.069051,1\n615,59835.0600,2,-54.949490,2.575779,1\n615,59835.0678,1,-178.149399,3.809858,1\n615,59835.0755,3,-140.818436,3.149077,1\n615,59835.0866,4,-200.294128,3.578005,1\n615,59835.0978,5,-263.578430,6.954262,1\n615,59839.0306,2,-639.035950,3.928531,1\n615,59839.0384,1,-953.883728,6.035410,1\n615,59839.0461,3,-518.293274,3.595869,1\n615,59839.0573,4,-418.723907,3.536483,1\n615,59839.0684,5,-418.799927,6.580595,1\n615,59842.0207,2,-502.215332,3.348443,1\n615,59842.0285,1,-1003.971497,6.256784,1\n615,59842.0362,3,-233.167755,2.872840,1\n615,59842.0473,4,111.507675,3.229112,1\n615,59842.0585,5,206.425323,6.615822,1\n615,59851.1114,0,-68.502457,3.338555,1\n615,59854.0796,2,459.452667,3.336711,1\n615,59854.0873,1,217.894211,3.332742,1\n615,59854.0950,3,361.023438,3.237847,1\n615,59854.1061,4,374.446442,3.622074,1\n615,59854.1172,5,370.346283,6.789766,1\n615,59857.0453,2,599.812195,4.121032,1\n615,59857.0531,1,646.523193,5.291624,1\n615,59857.0608,3,354.961365,3.407785,1\n615,59857.0719,4,293.879608,3.581862,1\n615,59857.0830,5,232.535995,6.761845,1\n615,59864.0162,2,-637.105347,3.818432,1\n615,59864.0239,1,-942.167908,5.916004,1\n615,59864.0316,3,-524.586548,3.538931,1\n615,59864.0428,4,-414.447723,3.635253,1\n615,59864.0539,5,-408.089233,7.119730,1\n615,59867.0178,2,-332.763123,2.872951,1\n615,59867.0255,1,-910.677734,5.852032,1\n615,59867.0332,3,-62.065010,3.073413,1\n615,59867.0443,4,202.288223,4.026751,1\n615,59867.0554,5,270.584869,10.321785,1\n615,59870.0194,2,604.344543,3.901750,1\n615,59870.0272,1,659.486694,4.991051,1\n615,59870.0349,3,373.986511,3.294667,1\n615,59870.0459,4,322.604034,3.716555,1\n615,59870.0571,5,263.481476,7.193131,1\n615,59873.0212,2,4.656033,2.130510,0\n615,59873.0289,1,-98.796974,3.034533,1\n615,59873.0366,3,-93.732880,2.499724,1\n615,59873.0477,4,-165.793457,2.906058,1\n615,59873.0588,5,-233.501724,6.087882,1\n615,59874.0599,0,-97.353195,3.133990,1\n615,59875.0311,0,-97.523880,2.963075,1\n615,59876.0231,0,-108.672577,3.449714,1\n615,59877.0238,0,-116.913223,3.097836,1\n615,59878.0246,0,-102.768921,3.135772,1\n615,59879.0248,0,-52.407089,3.261559,1\n615,59880.0258,0,55.567715,3.355268,1\n615,59884.0823,2,-274.711029,2.572093,1\n615,59884.0900,1,-437.425110,3.831595,1\n615,59884.0976,3,-310.010925,2.957125,1\n615,59884.1085,4,-317.630920,3.213168,1\n615,59884.1195,5,-351.278198,6.231324,1\n615,59887.0298,2,-491.146423,3.592675,1\n615,59887.0375,1,-743.267212,5.624708,1\n615,59887.0451,3,-449.714752,3.728483,1\n615,59887.0562,4,-393.971649,3.532816,1\n615,59887.0673,5,-406.549103,6.648589,1\n615,60118.4163,0,-107.080536,3.102513,1\n615,60124.2541,2,-588.397949,4.176047,1\n615,60124.2618,1,-878.043396,6.210247,1\n615,60124.2695,3,-495.472015,4.011444,1\n615,60124.2807,4,-417.145325,4.094360,1\n615,60124.2918,5,-413.673431,7.793959,1\n615,60140.2290,0,-88.981155,3.468430,1\n615,60141.2225,0,-50.179337,4.734193,1\n615,60142.2202,0,50.008640,4.636651,1\n615,60143.2212,0,110.753555,4.380840,1\n615,60144.2186,0,120.867218,4.103332,1\n615,60145.2123,0,111.464226,4.367030,1\n615,60153.2274,2,-322.420471,2.833071,1\n615,60153.2351,1,-917.875488,5.951387,1\n615,60153.2428,3,-52.056461,2.698249,1\n615,60153.2539,4,205.180893,3.169676,1\n615,60153.2650,5,269.709167,6.682271,1\n615,60162.1477,2,31.499735,2.753767,1\n615,60162.1554,1,-62.120552,3.818642,1\n615,60162.1631,3,-72.958771,3.515574,1\n615,60162.1742,4,-151.126511,3.838288,1\n615,60162.1853,5,-216.914032,8.832489,1\n615,60165.1369,2,-568.408875,3.553168,1\n615,60165.1446,1,-836.233154,5.405759,1\n615,60165.1524,3,-483.071381,3.387615,1\n615,60165.1635,4,-409.470642,3.541994,1\n615,60165.1746,5,-412.820221,6.907444,1\n615,60168.1260,2,-628.321350,3.804775,1\n615,60168.1337,1,-1077.347900,6.591075,1\n615,60168.1414,3,-421.859406,3.336656,1\n615,60168.1525,4,-93.729095,3.211201,1\n615,60168.1637,5,31.207939,6.615005,0\n615,60176.1332,0,-49.905262,3.874426,1\n615,60177.1370,0,-87.160583,4.078375,1\n615,60181.3147,2,-180.729568,2.498579,1\n615,60181.3223,1,-339.875153,3.626661,1\n615,60181.3299,3,-249.205673,3.280824,1\n615,60181.3409,4,-275.762329,4.694962,1\n615,60181.3518,5,-330.891327,9.580047,1\n615,60184.3625,2,-555.853943,3.979171,1\n615,60184.3701,1,-1028.441528,6.719577,1\n615,60184.3777,3,-306.200500,3.557627,1\n615,60184.3887,4,49.555847,3.627351,1\n615,60184.3996,5,154.876785,7.988054,1\n615,60194.1575,2,469.654999,3.386857,1\n615,60194.1652,1,276.757751,3.594162,1\n615,60194.1729,3,374.669556,3.437137,1\n615,60194.1839,4,374.948822,4.267094,1\n615,60194.1926,5,363.130493,12.845472,1\n615,60197.1181,2,607.786804,3.960346,1\n615,60197.1258,1,650.984314,4.970811,1\n615,60197.1335,3,365.408752,3.197298,1\n615,60197.1446,4,305.330750,3.360043,1\n615,60197.1557,5,256.966217,6.443069,1\n615,60198.1077,0,100.129280,4.266314,1\n615,60199.0914,0,86.776741,4.679742,1\n615,60200.0650,0,82.078186,4.342434,1\n615,60201.0680,0,41.947815,4.467065,1\n615,60202.0552,0,9.061676,3.831397,0\n615,60206.1107,0,-83.072884,3.130236,1\n615,60207.1469,0,108.483109,4.458607,1\n615,60208.0229,2,-672.681335,4.138056,1\n615,60208.0307,1,-1094.027588,6.717340,1\n615,60208.0384,3,-503.870422,3.665424,1\n615,60208.0495,4,-284.747498,3.666287,1\n615,60208.0606,5,-176.409851,8.308295,1\n615,60211.0124,2,208.281052,3.229784,1\n615,60211.0202,1,-370.189575,4.525907,1\n615,60211.0279,3,269.200806,3.688238,1\n615,60211.0390,4,326.272308,4.424663,1\n615,60211.0502,5,358.320099,10.152412,1\n615,60221.0153,2,-648.682312,3.906177,1\n615,60221.0230,1,-1086.777710,6.620100,1\n615,60221.0308,3,-455.588196,3.295532,1\n615,60221.0419,4,-145.305023,3.087424,1\n615,60221.0530,5,2.939076,6.798505,0\n615,60224.0140,2,491.748383,3.509416,1\n615,60224.0217,1,346.335083,3.835582,1\n615,60224.0294,3,384.185303,3.529593,1\n615,60224.0405,4,381.953735,3.885009,1\n615,60224.0516,5,378.118225,7.311360,1\n615,60227.0151,2,341.057709,2.940147,1\n615,60227.0228,1,356.632690,3.856145,1\n615,60227.0305,3,153.004929,2.847803,1\n615,60227.0416,4,52.912033,3.109148,1\n615,60227.0527,5,-19.384567,6.713308,0\n615,60228.0187,0,6.768485,4.174600,0\n615,60229.0162,0,-35.149330,4.086384,1\n615,60234.0265,0,-52.922794,3.681808,1\n615,60237.2206,2,-676.669189,4.009161,1\n615,60237.2283,1,-1098.651489,6.689435,1\n615,60237.2359,3,-511.148254,3.546333,1\n615,60237.2468,4,-347.090027,3.624199,1\n615,60237.2578,5,-240.316895,8.035271,1\n615,60240.0223,2,85.162651,2.829378,1\n615,60240.0300,1,14.526012,3.783879,0\n615,60240.0377,3,-24.350578,3.397041,1\n615,60240.0488,4,-111.062698,3.698180,1\n615,60240.0598,5,-180.234787,6.894514,1\n615,60249.0338,2,611.984558,3.908728,1\n615,60249.0415,1,660.626343,4.961018,1\n615,60249.0492,3,386.311920,3.240422,1\n615,60249.0602,4,325.401184,3.454910,1\n615,60249.0712,5,280.721069,6.623785,1\n615,60260.0423,0,108.020546,4.337497,1\n615,60261.0361,0,125.182808,3.909554,1\n615,60262.0367,0,107.649780,3.796611,1\n615,60263.0373,0,61.068066,3.877589,1\n615,60264.0465,0,-9.100937,3.518127,0\n615,60490.2647,2,-408.570984,3.169784,1\n615,60490.2725,1,-624.518799,4.704853,1\n615,60490.2802,3,-405.614258,3.513195,1\n615,60490.2913,4,-371.286377,3.985296,1\n615,60490.3024,5,-395.406128,8.139952,1\n615,60493.2372,2,-680.489441,4.065931,1\n615,60493.2450,1,-1031.102905,6.378702,1\n615,60493.2527,3,-530.644592,3.672556,1\n615,60493.2639,4,-406.733521,3.772714,1\n615,60493.2750,5,-358.876160,7.310321,1\n615,60499.2467,0,106.447296,4.481476,1\n615,60500.2437,0,67.234062,4.020935,1\n615,60501.2385,0,24.868933,4.027500,1\n615,60502.2355,0,-15.392517,4.142292,0\n615,60508.2638,2,365.607056,3.696270,1\n615,60508.2715,1,-32.986282,4.440859,0\n615,60508.2792,3,319.249847,3.828632,1\n615,60508.2903,4,360.507599,4.336362,1\n615,60508.3014,5,370.305267,8.601955,1\n615,60524.2390,0,89.070496,3.901179,1\n615,60525.1736,0,118.935989,4.737393,1\n615,60532.3489,2,510.690094,3.489832,1\n615,60532.3565,1,566.281433,4.607503,1\n615,60532.3641,3,271.663910,3.007311,1\n615,60532.3751,4,204.409866,3.353202,1\n615,60532.3860,5,128.521851,7.134325,1\n615,60535.1253,2,-664.729675,4.282414,1\n615,60535.1330,1,-1084.891113,6.952323,1\n615,60535.1408,3,-488.010925,3.928481,1\n615,60535.1519,4,-222.254257,4.034600,1\n615,60535.1630,5,-85.524307,8.625449,1\n615,60538.2351,2,113.021248,2.712380,1\n615,60538.2428,1,51.060081,3.762334,1\n615,60538.2505,3,-4.268328,2.895656,0\n615,60538.2615,4,-96.020035,3.141703,1\n615,60538.2725,5,-175.912643,6.308159,1\n615,60546.3406,2,178.624359,2.905459,1\n615,60546.3482,1,142.089966,4.065646,1\n615,60546.3558,3,41.418739,3.163731,1\n615,60546.3668,4,-52.460590,3.784039,1\n615,60546.3777,5,-112.286079,8.527776,1\n615,60549.0879,2,-629.010254,3.867215,1\n615,60549.0956,1,-1076.652100,6.604701,1\n615,60549.1034,3,-435.558533,3.417534,1\n615,60549.1145,4,-111.499573,3.424588,1\n615,60549.1256,5,30.267401,7.478198,0\n615,60554.0964,0,82.168922,4.318140,1\n615,60555.0951,0,49.886921,3.917516,1\n615,60556.0879,0,9.075453,4.103900,0\n615,60557.0831,0,-30.764908,3.555157,1\n615,60558.1093,0,-101.419899,3.653430,1\n615,60559.1097,0,-110.688477,3.426444,1\n615,60560.1065,0,-114.774445,4.013463,1\n615,60567.2821,2,-447.681580,3.368270,1\n615,60567.2897,1,-972.201111,6.347886,1\n615,60567.2973,3,-176.163651,2.960412,1\n615,60567.3083,4,140.860611,3.302721,1\n615,60567.3192,5,228.033112,6.797573,1\n615,60574.1118,2,-143.843872,2.547544,1\n615,60574.1195,1,-812.792908,5.570116,1\n615,60574.1272,3,86.606873,2.966459,1\n615,60574.1383,4,257.570221,3.575394,1\n615,60574.1493,5,302.167328,7.100554,1\n615,60577.0186,2,-425.988464,3.239578,1\n615,60577.0263,1,-963.216980,6.134610,1\n615,60577.0340,3,-148.178238,3.247297,1\n615,60577.0451,4,161.872543,4.228243,1\n615,60577.0563,5,238.576889,9.461221,1\n615,60580.0095,2,586.178345,3.996895,1\n615,60580.0173,1,655.284058,5.148244,1\n615,60580.0250,3,445.737061,3.952905,1\n615,60580.0361,4,361.595764,4.508256,1\n615,60580.0472,5,328.836731,9.460338,1\n615,60582.0840,0,-51.614189,3.517908,1\n615,60583.0169,0,20.364273,4.460314,0\n615,60584.0117,0,-24.682575,3.866380,1\n615,60585.0117,0,-63.546600,3.497667,1\n615,60586.0123,0,-101.819290,3.383004,1\n615,60587.0127,0,-110.978699,3.555624,1\n615,60588.0131,0,-113.588432,3.241369,1\n615,60593.0636,2,226.696259,2.514855,1\n615,60593.0713,1,205.029755,3.258004,1\n615,60593.0790,3,73.384720,2.584785,1\n615,60593.0901,4,-19.212976,2.899512,0\n615,60593.1012,5,-83.394951,6.073453,1\n615,60596.0304,2,-224.917938,3.388916,1\n615,60596.0381,1,-388.231476,4.931039,1\n615,60596.0458,3,-274.108429,3.720238,1\n615,60596.0569,4,-292.558990,4.188871,1\n615,60596.0680,5,-354.074280,8.392479,1\n615,60603.0208,2,404.391388,3.043772,1\n615,60603.0286,1,70.494507,3.060846,1\n615,60603.0363,3,338.994537,3.051842,1\n615,60603.0473,4,362.888550,3.381572,1\n615,60603.0584,5,378.188141,6.295821,1\n615,60606.0225,2,422.610779,3.198191,1\n615,60606.0303,1,457.502197,4.173640,1\n615,60606.0379,3,205.937546,2.957614,1\n615,60606.0490,4,123.048210,3.402847,1\n615,60606.0601,5,33.726837,7.368811,0\n615,60609.0247,2,-355.611389,2.720825,1\n615,60609.0323,1,-537.169312,4.090708,1\n615,60609.0400,3,-372.485565,2.985755,1\n615,60609.0510,4,-350.518677,3.225662,1\n615,60609.0621,5,-371.873230,6.150734,1\n615,60612.0266,0,-110.649872,2.844200,1\n615,60613.0269,0,-89.973892,2.937887,1\n615,60614.0276,0,-10.015225,3.212408,0\n615,60615.0375,0,99.438087,3.662484,1\n615,60616.0290,0,120.849113,3.776495,1\n615,60617.0295,0,121.411896,3.569777,1\n615,60621.1734,2,56.559818,2.259825,1\n615,60621.1810,1,-607.040771,4.452463,1\n615,60621.1886,3,208.770279,2.959783,1\n615,60621.1996,4,297.624725,3.718585,1\n615,60621.2105,5,332.919006,8.157172,1\n615,60624.1760,2,552.150269,3.917989,1\n615,60624.1836,1,607.047668,5.140991,1\n615,60624.1913,3,296.946533,3.475000,1\n615,60624.2022,4,235.489929,3.926538,1\n615,60624.2132,5,157.080200,8.453112,1\n713,59825.2600,2,9.110147,1.013889,0\n713,59825.2676,1,7.615042,1.160329,1\n713,59825.2752,3,6.673631,1.932316,1\n713,59825.2862,4,5.214194,3.018003,0\n713,59825.2971,5,12.060948,7.163382,0\n713,59839.2161,2,4.953065,1.196956,0\n713,59839.2236,1,3.131028,1.351706,0\n713,59839.2313,3,6.108739,1.789895,0\n713,59839.2422,4,8.283792,2.527953,0\n713,59839.2532,5,9.686500,5.882469,0\n713,59842.1987,2,6.472355,1.110572,0\n713,59842.2064,1,5.914848,1.134476,1\n713,59842.2140,3,5.311658,1.683777,0\n713,59842.2250,4,4.680908,2.349234,0\n713,59842.2359,5,6.921503,5.573885,0\n713,59851.2006,0,7.267655,2.866838,0\n713,59854.2089,2,3.945918,0.922779,0\n713,59854.2165,1,2.956484,0.953529,0\n713,59854.2242,3,4.768611,1.422675,0\n713,59854.2351,4,4.065430,1.943197,0\n713,59854.2461,5,1.290383,3.980583,0\n713,59857.1879,2,3.937931,2.088610,0\n713,59857.1956,1,-2.223347,2.861396,0\n713,59857.2032,3,-0.476698,2.280299,0\n713,59857.2141,4,8.054095,2.712542,0\n713,59857.2251,5,-9.332252,5.174713,0\n713,59867.1600,2,1.380378,1.105223,0\n713,59867.1676,1,1.569406,1.369900,0\n713,59867.1753,3,1.433712,1.730335,0\n713,59867.1862,4,5.299760,2.530279,0\n713,59867.1971,5,2.355590,5.574841,0\n713,59870.1521,2,3.404463,0.889940,0\n713,59870.1597,1,2.961649,0.946383,0\n713,59870.1673,3,2.594970,1.322884,0\n713,59870.1782,4,2.805032,1.753258,0\n713,59870.1892,5,3.101222,4.041600,0\n713,59873.1442,2,3.869869,0.763644,0\n713,59873.1519,1,4.354049,0.790372,1\n713,59873.1595,3,2.949366,1.101570,0\n713,59873.1704,4,3.117238,1.623630,0\n713,59873.1814,5,4.010789,3.832515,0\n713,59874.1612,0,4.171277,2.180456,0\n713,59875.1175,0,2.671449,2.536783,0\n713,59876.1160,0,-0.354117,2.566333,0\n713,59877.1178,0,2.332870,2.217193,0\n713,59878.1127,0,3.797837,2.270967,0\n713,59879.1104,0,4.900619,2.339577,0\n713,59880.1181,0,1.331082,1.865762,0\n713,59884.1292,2,3.091794,0.995209,0\n713,59884.1368,1,1.712878,1.129837,0\n713,59884.1444,3,2.552051,1.229161,0\n713,59884.1554,4,2.099711,1.737169,0\n713,59884.1663,5,5.788035,3.869600,0\n713,59887.0951,2,5.011691,1.668699,0\n713,59887.1027,1,3.719429,2.314904,0\n713,59887.1103,3,10.036420,1.852814,0\n713,59887.1213,4,5.964674,2.277389,0\n713,59887.1322,5,8.822542,4.767565,0\n713,59896.0839,2,5.064992,0.999215,0\n713,59896.0915,1,5.780192,1.083338,0\n713,59896.0992,3,3.587355,1.642954,1\n713,59896.1101,4,6.185760,2.277758,0\n713,59896.1211,5,-4.762829,5.042903,0\n713,59899.0854,2,4.823127,1.244829,0\n713,59899.0930,1,6.899071,1.246326,0\n713,59899.1007,3,3.249064,2.133093,0\n713,59899.1116,4,7.382133,3.388385,0\n713,59899.1226,5,-4.677240,9.115748,0\n713,59902.0445,2,9.166100,1.426165,0\n713,59902.0522,1,8.076466,1.395627,0\n713,59902.0598,3,11.330316,2.051576,1\n713,59902.0707,4,9.245844,2.876306,0\n713,59902.0817,5,0.942024,7.248375,0\n713,59904.0584,0,3.223553,2.679078,0\n713,59905.0468,0,14.509829,3.098125,0\n713,59906.0474,0,5.995616,2.589032,0\n713,59907.0480,0,5.440472,2.469325,0\n713,59908.0487,0,5.961231,3.348282,0\n713,59909.0494,0,10.137896,2.151001,0\n713,59910.0590,0,8.248549,2.160179,0\n713,59913.2446,2,5.475236,0.822163,1\n713,59913.2522,1,6.833441,0.969664,0\n713,59913.2599,3,6.275328,1.430679,0\n713,59913.2708,4,4.298039,2.311868,0\n713,59913.2818,5,3.143612,5.875287,0\n713,59916.0544,2,10.529041,1.787002,0\n713,59916.0621,1,9.129021,2.415574,0\n713,59916.0697,3,5.509865,2.141148,0\n713,59916.0806,4,9.827934,2.274502,0\n713,59916.0915,5,2.627945,4.551546,0\n713,59924.0589,2,5.190053,0.786980,1\n713,59924.0665,1,6.531730,0.851491,1\n713,59924.0742,3,9.141804,1.210878,1\n713,59924.0851,4,9.810373,1.739901,0\n713,59924.0961,5,2.349317,4.040898,0\n713,59927.0604,2,5.366942,0.863455,0\n713,59927.0680,1,4.619713,0.947374,0\n713,59927.0756,3,6.296741,1.472587,0\n713,59927.0866,4,2.465199,2.073566,0\n713,59927.0975,5,-0.702472,6.396966,0\n713,59930.0619,2,2.780317,0.759708,1\n713,59930.0695,1,4.959312,0.809846,0\n713,59930.0771,3,4.033259,1.196190,0\n713,59930.0881,4,4.485665,1.901773,0\n713,59930.1063,5,6.218721,4.791905,0\n713,59933.0632,2,1.830853,0.639458,0\n713,59933.0709,1,1.716145,0.707228,0\n713,59933.0785,3,4.893567,0.968482,0\n713,59933.0944,4,3.197614,1.429430,0\n713,59933.1150,5,3.335699,3.523145,0\n713,59935.0739,0,0.554208,1.573855,0\n713,59936.0735,0,2.584441,1.804314,0\n713,59937.0743,0,6.470248,1.848658,0\n713,59938.0754,0,0.724684,2.076312,0\n713,59939.0808,0,2.375108,2.243821,0\n713,59942.0746,2,-0.148046,0.800387,0\n713,59942.0889,1,0.648101,0.878962,0\n713,59942.0965,3,0.328905,1.080046,0\n713,59942.1074,4,-4.550706,1.669870,0\n713,59942.1184,5,2.364145,4.108390,0\n713,59945.0770,2,-3.002108,1.474453,0\n713,59945.0846,1,-1.725136,2.059556,0\n713,59945.0922,3,-1.422123,1.846779,0\n713,59945.1032,4,-6.208874,2.036851,0\n713,59945.1141,5,-2.945050,4.744831,0\n713,60192.2930,2,7.250862,1.972519,0\n713,60192.3006,1,7.834616,2.653803,0\n713,60192.3082,3,6.543319,2.216304,0\n713,60192.3192,4,7.960829,2.695718,0\n713,60192.3301,5,3.404367,5.894906,0\n713,60195.2343,2,4.326025,1.101823,0\n713,60195.2419,1,3.302556,1.100982,1\n713,60195.2496,3,5.533146,1.655807,0\n713,60195.2605,4,5.854890,2.367182,0\n713,60195.2715,5,3.825871,5.774144,0\n713,60198.2332,0,2.208139,3.192551,0\n713,60199.2358,0,8.620851,2.547614,0\n713,60200.2314,0,3.770694,2.643626,0\n713,60201.2281,0,6.634655,2.975509,0\n713,60202.2255,0,9.813441,2.750465,0\n713,60209.2281,2,3.984369,1.161990,0\n713,60209.2357,1,1.381281,1.139709,0\n713,60209.2433,3,1.715379,1.724909,0\n713,60209.2543,4,0.545876,2.622813,0\n713,60209.2652,5,6.504875,6.293293,0\n713,60212.2147,2,2.873843,1.790648,0\n713,60212.2223,1,1.546698,2.668681,0\n713,60212.2300,3,5.084908,2.077699,0\n713,60212.2409,4,3.087726,2.654123,0\n713,60212.2519,5,-2.272981,5.844298,0\n713,60223.1948,2,0.405613,1.246678,0\n713,60223.2024,1,1.120193,1.277229,0\n713,60223.2100,3,1.011539,1.974625,0\n713,60223.2210,4,3.507817,2.882992,0\n713,60223.2319,5,14.770886,6.656366,0\n713,60226.2721,2,1.071414,0.746168,0\n713,60226.2797,1,1.648819,0.776689,0\n713,60226.2931,3,1.727918,1.133994,0\n713,60226.3129,4,-0.916487,1.736045,0\n713,60226.3238,5,3.996732,4.304620,0\n713,60236.1862,0,4.067199,2.328237,0\n713,60238.2696,2,-0.087907,0.758784,0\n713,60238.2803,1,-0.829578,0.881391,0\n713,60238.2879,3,-0.576265,1.265385,0\n713,60238.2988,4,0.329135,1.983817,0\n713,60238.3098,5,-4.923808,4.832184,0\n713,60241.1342,2,2.333379,1.636026,0\n713,60241.1418,1,-0.293893,2.097461,0\n713,60241.1495,3,3.571144,1.816384,0\n713,60241.1604,4,1.146531,2.471305,0\n713,60241.1713,5,-7.436915,5.122927,0\n713,60260.0773,0,1.232121,3.040076,0\n713,60261.0632,0,2.412768,2.796987,0\n713,60262.0637,0,-1.678317,2.631186,0\n713,60263.0643,0,-0.390618,2.617705,0\n713,60264.0716,0,1.131548,2.522264,0\n713,60265.0867,0,-2.189290,2.832789,0\n713,60267.0443,2,-2.785007,1.335972,0\n713,60267.0519,1,-1.797494,1.524965,0\n713,60267.0595,3,-4.881196,1.920069,0\n713,60267.0705,4,3.231234,2.734247,0\n713,60267.0814,5,-12.699218,5.767424,0\n713,60270.1082,2,0.393975,2.344530,0\n713,60270.1158,1,2.370688,3.545214,0\n713,60270.1234,3,2.489378,2.970400,0\n713,60270.1344,4,-7.822262,3.554679,0\n713,60270.1453,5,-10.877887,6.829591,0\n713,60278.0525,2,-0.607012,0.969379,0\n713,60278.0601,1,-0.027766,1.022582,0\n713,60278.0677,3,2.024312,1.532588,0\n713,60278.0786,4,-2.256550,2.154194,0\n713,60278.0896,5,-1.112494,5.843420,0\n713,60281.0552,2,-1.695972,1.034572,0\n713,60281.0629,1,-1.833499,1.044974,0\n713,60281.0705,3,-0.919016,1.470630,0\n713,60281.0814,4,-1.391540,2.003621,0\n713,60281.0924,5,-3.945375,4.607381,0\n713,60284.0557,2,-2.572076,0.783297,0\n713,60284.0633,1,-2.387862,0.801296,0\n713,60284.0709,3,-5.832908,1.240340,0\n713,60284.0819,4,-1.649157,1.857165,0\n713,60284.0928,5,8.627832,4.482957,0\n713,60287.0577,2,-0.764727,0.906658,0\n713,60287.0653,1,-1.477176,0.966182,0\n713,60287.0729,3,-0.536819,1.463827,0\n713,60287.0839,4,-0.667864,2.361719,0\n713,60287.0948,5,-5.286497,6.810267,0\n713,60290.0641,0,-1.816348,2.038470,0\n713,60291.0599,0,-5.132619,1.989085,0\n713,60292.0607,0,-5.080487,1.908693,0\n713,60293.0607,0,-5.075594,1.939040,0\n713,60294.0616,0,-0.566193,1.833754,0\n713,60295.0621,0,-3.857503,2.114682,0\n713,60297.1169,2,-4.830737,0.921245,0\n713,60297.1245,1,-4.334117,1.068175,0\n713,60297.1321,3,-4.767125,1.234546,0\n713,60297.1431,4,-4.473659,1.723665,0\n713,60297.1540,5,-10.414721,4.258311,0\n713,60300.0641,2,-5.492156,1.607434,0\n713,60300.0717,1,-3.599649,2.103016,0\n713,60300.0793,3,-3.476922,1.725975,0\n713,60300.1031,4,-2.745461,2.269754,0\n713,60300.1160,5,7.738044,4.946638,0\n713,60554.2916,0,-9.100129,2.393532,0\n713,60555.2620,0,-8.218450,2.308315,1\n713,60556.2548,0,-5.576579,2.770439,0\n713,60557.2501,0,-9.173389,2.218352,0\n713,60558.2534,0,-13.083604,2.663738,0\n713,60559.2490,0,-9.237353,2.428750,0\n713,60560.2424,0,-10.050170,3.275514,0\n713,60567.2231,2,-8.265152,1.515329,0\n713,60567.2308,1,-8.954789,2.105672,1\n713,60567.2384,3,-8.418892,1.630414,0\n713,60567.2493,4,-12.286801,1.978125,1\n713,60567.2603,5,-11.054881,4.445991,0\n713,60578.2746,2,-4.951467,1.229683,0\n713,60578.2822,1,-7.403615,1.305094,0\n713,60578.2898,3,-5.050255,1.839125,1\n713,60578.3008,4,-7.385537,2.667687,0\n713,60578.3117,5,-6.356452,6.387929,0\n713,60581.1779,2,-5.760825,1.288651,1\n713,60581.1855,1,-7.428378,1.275975,0\n713,60581.1931,3,-6.902376,1.927237,0\n713,60581.2041,4,-9.594004,2.818656,0\n713,60581.2150,5,-14.211164,6.624023,0\n713,60582.2087,0,-11.829331,2.358846,0\n713,60583.1842,0,-9.363182,3.042286,0\n713,60584.1807,0,-9.220502,2.544668,0\n713,60585.1757,0,-3.587870,2.280919,0\n713,60586.1765,0,-9.129416,2.146863,0\n713,60587.1702,0,-5.876253,2.481174,0\n713,60588.1666,0,-9.116284,2.157747,0\n713,60593.1682,2,-9.569608,0.985850,1\n713,60593.1758,1,-8.809836,1.078624,1\n713,60593.1834,3,-9.553467,1.347112,1\n713,60593.1944,4,-9.193518,1.914358,1\n713,60593.2053,5,-8.280509,4.493694,0\n713,60596.1820,2,-8.760753,2.037911,0\n713,60596.1896,1,-4.396494,2.863201,0\n713,60596.1972,3,-11.907238,2.252078,1\n713,60596.2081,4,-4.786119,2.784098,0\n713,60596.2191,5,-5.489277,6.255779,0\n713,60605.1380,2,-9.696579,0.873996,1\n713,60605.1456,1,-11.159884,0.883977,1\n713,60605.1532,3,-12.394593,1.305202,1\n713,60605.1642,4,-9.511388,1.906236,0\n713,60605.1751,5,-6.906372,4.587698,0\n713,60608.1308,2,-9.163915,0.800012,1\n713,60608.1384,1,-11.715749,0.823976,1\n713,60608.1460,3,-11.449253,1.202452,0\n713,60608.1569,4,-12.221146,1.749559,1\n713,60608.1679,5,-2.633516,4.132709,0\n713,60611.1227,2,-10.067919,0.717739,1\n713,60611.1303,1,-9.289042,0.761477,1\n713,60611.1380,3,-10.801243,1.080986,1\n713,60611.1489,4,-11.623042,1.560488,1\n713,60611.1599,5,-7.861447,3.710802,0\n713,60612.1183,0,-11.605895,1.778605,1\n713,60613.1019,0,-11.340659,1.930082,1\n713,60614.0960,0,-10.934606,2.143276,1\n713,60615.0917,0,-14.735178,2.326417,0\n713,60616.0927,0,-12.353376,2.357691,1\n713,60617.0896,0,-6.599936,2.023456,0\n713,60620.1350,0,-6.110061,2.056073,0\n713,60621.1263,2,-5.537477,0.829998,1\n713,60621.1339,1,-7.972793,0.877838,1\n713,60621.1416,3,-7.565215,1.233034,1\n713,60621.1525,4,-6.638791,1.780862,0\n713,60621.1635,5,-9.333499,4.242186,0\n713,60624.0821,2,-4.490414,1.314625,0\n713,60624.0897,1,-5.545699,1.695655,0\n713,60624.0974,3,-7.286825,1.600662,0\n713,60624.1083,4,-6.478677,2.086655,0\n713,60624.1193,5,-7.099849,4.810002,0\n713,60627.2801,2,-6.219934,2.530638,0\n713,60627.2877,1,-5.039655,3.228468,0\n713,60627.2954,3,-0.950650,2.782719,0\n713,60627.3063,4,-4.143107,3.379841,0\n713,60627.3173,5,0.243241,7.639313,0\n713,60632.0400,2,-5.855491,1.492071,0\n713,60632.0476,1,-6.847743,1.453141,0\n713,60632.0552,3,-9.374930,2.217679,0\n713,60632.0662,4,-10.557325,3.237312,0\n713,60632.0771,5,-2.921649,7.857955,0\n713,60635.0469,2,-6.480945,1.091159,1\n713,60635.0545,1,-6.966879,1.087843,1\n713,60635.0621,3,-6.185159,1.629475,1\n713,60635.0731,4,-5.490345,2.377301,0\n713,60635.0840,5,-2.292507,5.646507,0\n713,60640.0504,2,-7.312206,0.974580,1\n713,60640.0580,1,-7.250492,0.991461,0\n713,60640.0656,3,-10.161006,1.479101,1\n713,60640.0766,4,-6.631466,2.145102,0\n713,60640.0875,5,-10.591419,5.138685,0\n713,60643.0609,0,-9.289350,1.992813,1\n713,60644.0533,0,-8.482151,2.118450,0\n713,60645.0537,0,-2.605739,2.197297,1\n713,60646.0548,0,-8.104684,2.135281,0\n713,60647.0546,0,-7.506279,2.275638,0\n713,60648.0553,0,-10.602926,1.838902,1\n713,60649.0561,0,-12.232555,1.708795,0\n713,60651.1265,2,-9.331477,0.865811,1\n713,60651.1451,1,-10.061421,0.932510,1\n713,60651.1527,3,-9.335849,1.315029,1\n713,60651.1637,4,-6.167844,1.952829,0\n713,60651.1746,5,-10.171921,4.815349,0\n713,60654.0597,2,-9.607999,1.647062,1\n713,60654.0673,1,-6.258916,2.222855,0\n713,60654.0749,3,-9.524345,1.901351,0\n713,60654.0859,4,-9.513783,2.389906,0\n713,60654.0968,5,-9.744430,5.404162,0\n713,60662.1451,2,-5.698765,1.334831,0\n713,60662.1527,1,-4.317381,2.017339,0\n713,60662.1603,3,-6.093997,1.675434,0\n713,60662.1713,4,-5.760686,2.093239,0\n713,60662.1822,5,-5.400730,4.721159,0\n713,60665.0637,2,-6.826318,0.676851,1\n713,60665.0713,1,-6.498077,0.740772,1\n713,60665.0789,3,-5.938825,1.048616,1\n713,60665.1017,4,-6.440791,1.543503,1\n713,60665.1156,5,-13.727009,3.770338,0\n713,60668.0647,2,-6.938087,0.920544,1\n713,60668.0723,1,-8.995543,0.954973,1\n713,60668.0893,3,-10.263328,1.437371,1\n713,60668.1055,4,-5.455149,2.164149,0\n713,60668.1165,5,-9.138229,5.354884,0\n713,60671.0655,0,-10.165054,1.726118,1\n713,60672.0693,0,-10.828177,1.470152,1\n713,60673.0745,0,-12.148479,2.243120,0\n713,60674.0798,0,-8.669188,2.216094,0\n730,59798.3205,2,1.177371,1.364300,0\n730,59798.3281,1,2.320849,1.159247,0\n730,59798.3357,3,2.939447,1.771328,0\n730,59798.3466,4,2.128097,2.610659,0\n730,59798.3576,5,-12.809639,5.380097,0\n730,59801.3553,2,0.111235,2.460576,0\n730,59801.3629,1,-3.393080,3.564052,0\n730,59801.3705,3,-1.899219,2.292693,0\n730,59801.3815,4,2.284906,2.523534,0\n730,59801.3924,5,5.203419,5.395980,0\n730,59818.2740,0,-2.342200,1.801066,0\n730,59819.2541,0,3.380978,2.469600,0\n730,59820.2522,0,-2.230815,1.915426,0\n730,59821.2478,0,1.159034,2.461736,0\n730,59822.2433,0,5.942166,2.901580,0\n730,59823.2659,0,-0.180970,2.714361,0\n730,59826.3105,2,0.521923,0.925337,0\n730,59826.3181,1,-1.421768,0.929596,0\n730,59826.3258,3,0.972355,1.513987,0\n730,59826.3367,4,-0.570261,2.162375,0\n730,59826.3477,5,-2.301237,5.548611,0\n730,59842.2456,2,0.156290,0.853800,0\n730,59842.2532,1,-0.567360,0.819375,0\n730,59842.2608,3,-0.251899,1.325633,0\n730,59842.2718,4,2.019500,2.173066,0\n730,59842.2827,5,10.142254,6.086383,0\n730,59851.1792,0,-1.472170,2.597541,0\n730,59854.1485,2,0.368931,1.230250,0\n730,59854.1563,1,0.664051,1.345911,0\n730,59854.1640,3,3.201455,1.909905,0\n730,59854.1750,4,3.012713,2.778862,0\n730,59854.1860,5,3.750187,5.803461,0\n730,59857.1408,2,1.076537,2.141015,0\n730,59857.1485,1,5.693109,2.937809,0\n730,59857.1563,3,-2.640246,2.100464,0\n730,59857.1673,4,0.402461,2.684283,0\n730,59857.1782,5,-4.509360,6.643411,0\n730,59867.1112,2,-0.449365,1.088300,0\n730,59867.1189,1,0.282022,0.981426,0\n730,59867.1267,3,0.026595,1.544194,0\n730,59867.1377,4,0.956947,2.364042,0\n730,59867.1487,5,2.724518,5.845339,0\n730,59870.1049,2,1.070328,0.976301,0\n730,59870.1126,1,0.511964,0.828288,0\n730,59870.1204,3,-0.505236,1.377689,0\n730,59870.1314,4,0.251195,2.289763,0\n730,59870.1424,5,4.119082,5.293428,0\n730,59873.0971,2,-0.211154,1.045822,0\n730,59873.1049,1,-1.287062,1.048773,0\n730,59873.1126,3,-1.557674,1.446841,0\n730,59873.1236,4,-0.739414,2.074561,0\n730,59873.1346,5,5.151175,4.601235,0\n730,59874.1461,0,0.412505,1.627923,0\n730,59875.0995,0,-2.200486,2.037783,0\n730,59876.0980,0,-2.931559,2.450620,0\n730,59877.0976,0,2.024089,1.789397,0\n730,59878.0964,0,-1.250103,2.029308,0\n730,59879.0895,0,-0.671039,1.877854,0\n730,59880.1017,0,0.189355,1.384724,0\n730,59884.1760,2,-0.108323,0.771566,0\n730,59884.1836,1,-1.113737,0.892852,0\n730,59884.1913,3,-0.427802,1.142666,0\n730,59884.2022,4,1.402694,1.614300,0\n730,59884.2132,5,-1.060647,4.220271,0\n730,59887.2856,2,1.474370,1.835391,0\n730,59887.2933,1,-3.203188,2.221069,0\n730,59887.3009,3,-5.435799,2.359130,0\n730,59887.3118,4,0.192088,3.154000,0\n730,59887.3228,5,-1.082339,7.966248,0\n730,59896.1307,2,1.145174,0.826742,0\n730,59896.1384,1,-0.032153,0.707979,0\n730,59896.1460,3,-0.357363,1.398256,0\n730,59896.1569,4,-2.788487,2.198583,0\n730,59896.1679,5,2.883538,5.962979,0\n730,59899.1519,2,-1.456884,1.371527,0\n730,59899.1595,1,-0.707794,1.396877,0\n730,59899.1672,3,5.298447,2.388603,0\n730,59899.1781,4,6.412822,3.720956,0\n730,59899.1891,5,1.091714,7.924479,0\n730,59902.1384,2,-0.887660,1.191683,0\n730,59902.1460,1,0.168580,1.085883,0\n730,59902.1537,3,-1.369444,1.950019,0\n730,59902.1646,4,0.303218,2.770533,0\n730,59902.1755,5,11.777126,7.044582,0\n730,59904.1053,0,-0.620050,2.301550,0\n730,59905.0555,0,0.922903,2.527480,0\n730,59906.0562,0,-1.153271,2.043133,0\n730,59907.0567,0,0.449173,1.764913,0\n730,59908.0681,0,0.837362,2.710272,0\n730,59909.0582,0,-0.985495,1.850359,0\n730,59910.0503,0,-0.355463,1.880359,0\n730,59914.0526,2,-0.833646,1.445693,0\n730,59914.0602,1,0.370377,1.970406,0\n730,59914.0678,3,-1.719942,1.692403,0\n730,59914.0788,4,1.004354,2.274112,0\n730,59914.0897,5,0.138586,5.367689,0\n730,59924.1060,2,-0.787230,1.137160,0\n730,59924.1136,1,-1.572903,1.584968,0\n730,59924.1212,3,0.555294,1.735223,0\n730,59924.1322,4,-2.475216,2.533980,0\n730,59924.1431,5,-0.816748,5.644574,0\n730,59927.1074,2,-0.474538,1.196533,0\n730,59927.1151,1,0.973025,1.142775,0\n730,59927.1227,3,-2.039601,1.598035,0\n730,59927.1336,4,-1.036243,2.305239,0\n730,59927.1446,5,-3.438392,5.903537,0\n730,59930.1236,2,-0.114812,0.945627,0\n730,59930.1312,1,0.475511,0.835235,0\n730,59930.1388,3,0.226621,1.287869,0\n730,59930.1498,4,-3.755495,2.037717,0\n730,59930.1607,5,2.542647,5.343603,0\n730,59933.1249,2,0.603719,0.695106,0\n730,59933.1325,1,-0.226574,0.698751,0\n730,59933.1401,3,0.106692,1.273440,0\n730,59933.1511,4,0.993756,1.919590,0\n730,59933.1620,5,5.318815,5.441072,0\n730,59934.0638,0,1.190260,1.159169,0\n730,59935.0646,0,-0.320948,1.132809,0\n730,59936.0642,0,-1.230814,1.533033,0\n730,59937.0650,0,-0.751357,1.654440,0\n730,59938.0647,0,-3.109122,2.015928,0\n730,59939.0650,0,1.571790,2.219707,0\n730,60165.3032,2,-0.502432,1.200698,0\n730,60165.3109,1,-2.832010,1.356671,0\n730,60165.3186,3,1.843434,1.524752,0\n730,60165.3295,4,3.196369,2.029726,0\n730,60165.3405,5,-8.889149,4.793297,0\n730,60168.2892,2,-0.176546,0.837958,0\n730,60168.2970,1,0.201754,0.793672,0\n730,60168.3047,3,0.369397,1.305260,0\n730,60168.3157,4,-2.235131,2.071596,0\n730,60168.3267,5,-5.047883,4.827778,0\n730,60176.2820,0,1.753881,2.391554,0\n730,60177.2726,0,-1.260694,2.790846,0\n730,60181.4088,2,1.103341,0.929294,0\n730,60181.4164,1,1.391831,1.272189,0\n730,60181.4232,3,-4.766650,3.249354,0\n730,60183.2660,2,-2.848838,1.924783,0\n730,60183.2736,1,-3.133007,2.955767,0\n730,60183.2812,3,2.159384,2.557742,0\n730,60183.2922,4,-5.836310,3.003132,0\n730,60183.3031,5,-8.716421,6.192660,0\n730,60195.2812,2,-0.162802,0.937681,0\n730,60195.2888,1,0.864197,1.028533,0\n730,60195.2964,3,-2.294667,1.631539,0\n730,60195.3073,4,-2.693345,2.345676,0\n730,60195.3183,5,-0.570636,6.289552,0\n730,60198.2690,0,1.026459,2.192766,0\n730,60199.2186,0,0.527036,2.176080,0\n730,60200.2139,0,-1.516695,2.611164,0\n730,60201.2072,0,0.150572,2.513518,0\n730,60202.2089,0,-1.982165,2.505382,0\n730,60209.1811,2,-0.462435,1.572888,0\n730,60209.1888,1,2.153122,1.421764,0\n730,60209.1965,3,0.765511,1.838664,0\n730,60209.2075,4,-1.591831,2.917069,0\n730,60209.2184,5,0.939232,6.712256,0\n730,60212.1675,2,-2.669531,1.944909,0\n730,60212.1753,1,3.706729,2.772595,0\n730,60212.1830,3,-1.340184,2.303333,0\n730,60212.1941,4,-3.525083,2.598093,0\n730,60212.2050,5,7.831807,5.678200,0\n730,60223.2416,2,1.328195,1.188578,0\n730,60223.2493,1,-0.298775,1.249490,0\n730,60223.2569,3,2.987647,1.656098,0\n730,60223.2678,4,4.241424,2.481011,0\n730,60223.2788,5,9.333996,5.811805,0\n730,60226.3337,2,-0.679090,0.949060,0\n730,60226.3413,1,1.282680,0.952563,0\n730,60226.3489,3,0.970081,1.800158,0\n730,60226.3599,4,1.366870,3.079182,0\n730,60226.3708,5,-4.552550,8.128254,0\n730,60238.3197,2,-0.599053,1.058323,0\n730,60238.3273,1,0.734427,1.161131,0\n730,60238.3349,3,2.553997,2.179708,0\n730,60238.3459,4,1.377842,3.840058,0\n730,60238.3568,5,-19.159811,11.281384,0\n730,60241.0870,2,0.823192,2.008905,0\n730,60241.0948,1,3.386674,3.088520,0\n730,60241.1025,3,0.043122,2.453789,0\n730,60241.1136,4,-1.052531,2.925313,0\n730,60241.1245,5,-8.036972,5.953956,0\n730,60250.1708,2,-0.962673,2.112349,0\n730,60250.1957,1,0.580816,2.829899,0\n730,60250.2034,3,6.845217,2.940232,0\n730,60250.2143,4,0.204509,3.994097,0\n730,60250.2253,5,1.290714,7.954757,0\n730,60261.1296,0,1.237353,2.094631,0\n730,60262.0550,0,3.469973,2.338792,0\n730,60263.0556,0,2.352035,1.998888,0\n730,60264.0559,0,-2.396658,2.192123,0\n730,60265.0780,0,3.070599,2.439756,0\n730,60268.0449,2,-0.465229,1.410433,0\n730,60268.0525,1,0.174091,1.861911,0\n730,60268.0601,3,0.538344,2.137292,0\n730,60268.0711,4,-3.556071,2.741589,0\n730,60268.0820,5,-3.639747,6.760314,0\n730,60278.0993,2,2.521567,1.652593,0\n730,60278.1069,1,-2.468382,2.289480,0\n730,60278.1145,3,-1.407348,2.177464,0\n730,60278.1255,4,3.475310,3.068326,0\n730,60278.1364,5,8.474236,7.658961,0\n730,60281.1023,2,-1.139811,0.801878,0\n730,60281.1099,1,-1.247972,0.800422,0\n730,60281.1175,3,-1.347594,1.374244,0\n730,60281.1285,4,-0.890039,1.996277,0\n730,60281.1394,5,2.285095,5.189152,0\n730,60284.1027,2,-0.679968,0.840813,0\n730,60284.1104,1,-0.530991,0.766401,0\n730,60284.1180,3,-1.148911,1.164351,0\n730,60284.1289,4,0.493227,1.707686,0\n730,60284.1399,5,-4.683412,4.445528,0\n730,60287.1047,2,0.288175,1.256500,0\n730,60287.1123,1,-2.067724,1.090506,0\n730,60287.1200,3,-0.846692,1.802521,0\n730,60287.1309,4,-5.051833,2.972183,0\n730,60287.1418,5,-13.252449,7.029711,0\n730,60290.0761,0,-3.000368,1.929932,0\n730,60291.0689,0,1.081815,1.681175,0\n730,60292.0699,0,2.211185,1.882060,0\n730,60293.0699,0,0.182480,1.575780,0\n730,60294.0708,0,1.605139,1.418435,0\n730,60532.3019,2,20.994711,1.047298,1\n730,60532.3097,1,1.504146,0.960956,0\n730,60532.3173,3,31.523088,1.569497,1\n730,60532.3282,4,41.159981,2.310168,1\n730,60532.3392,5,46.795868,5.458707,1\n730,60535.2802,2,20.880348,1.426747,0\n730,60535.2879,1,2.271271,1.348233,0\n730,60535.2957,3,30.361010,2.107024,1\n730,60535.3068,4,40.715591,3.043571,1\n730,60535.3177,5,47.310059,7.197146,1\n730,60538.2826,2,19.450977,1.865142,1\n730,60538.2903,1,3.462672,2.695356,0\n730,60538.2980,3,33.572102,1.944897,1\n730,60538.3089,4,38.518837,2.334413,1\n730,60538.3199,5,40.146099,5.039364,1\n730,60554.2651,0,0.190944,2.266587,0\n730,60555.2411,0,0.098122,2.049620,0\n730,60556.2370,0,-0.253067,2.551228,0\n730,60557.2322,0,-2.200897,1.848830,0\n730,60558.2332,0,-3.459960,2.511074,0\n730,60559.2274,0,0.328893,2.224590,0\n730,60560.2268,0,2.453341,3.110694,0\n730,60567.3291,2,15.044784,0.951184,1\n730,60567.3368,1,-0.142653,1.050350,0\n730,60567.3444,3,18.416132,1.262663,1\n730,60567.3553,4,28.234451,1.676854,1\n730,60567.3663,5,31.623583,4.281011,1\n730,60580.1736,2,12.164557,1.463993,1\n730,60580.1813,1,2.065962,1.402610,0\n730,60580.1889,3,10.053763,2.203885,1\n730,60580.1999,4,19.975168,3.213686,1\n730,60580.2108,5,24.093925,7.662856,0\n730,60582.1681,0,-0.473370,2.422541,0\n730,60583.1640,0,-3.070249,3.006098,0\n730,60584.1591,0,0.970706,2.362254,0\n730,60585.1601,0,-0.533032,1.881978,0\n730,60586.1564,0,-0.049936,1.830623,0\n730,60587.1540,0,-2.202578,2.138732,0\n730,60588.1461,0,1.361049,1.798501,0\n730,60593.1209,2,6.307311,1.155241,1\n730,60593.1287,1,0.462838,1.313489,0\n730,60593.1365,3,8.789671,1.545997,1\n730,60593.1476,4,10.031554,2.179338,0\n730,60593.1585,5,10.850924,5.118365,0\n730,60596.1351,2,8.231540,2.483539,0\n730,60596.1427,1,1.623348,3.539990,0\n730,60596.1504,3,9.673650,2.658536,0\n730,60596.1613,4,8.778720,3.238315,0\n730,60596.1723,5,10.870938,7.199404,0\n730,60605.0908,2,5.607800,0.980471,1\n730,60605.0986,1,-0.587054,0.918425,0\n730,60605.1063,3,6.155015,1.461859,1\n730,60605.1174,4,7.274523,2.152366,0\n730,60605.1283,5,2.950838,5.145659,0\n730,60608.0836,2,5.938226,0.889949,0\n730,60608.0913,1,0.898013,0.825604,0\n730,60608.0991,3,5.509429,1.330889,0\n730,60608.1101,4,9.166319,1.962560,0\n730,60608.1211,5,3.346682,4.667600,0\n730,60611.0756,2,2.112415,0.773398,1\n730,60611.0833,1,0.247475,0.704158,0\n730,60611.0911,3,1.898379,1.172223,0\n730,60611.1021,4,4.244992,1.745410,0\n730,60611.1130,5,6.172510,4.175368,0\n730,60612.0813,0,1.228119,1.461220,0\n730,60613.0818,0,1.540095,1.541647,0\n730,60614.0803,0,1.231758,1.758784,0\n730,60615.0761,0,-0.502854,1.998764,0\n730,60616.0769,0,-2.247711,2.024976,0\n730,60617.0737,0,-1.035569,1.574103,0\n730,60620.1444,0,-1.018565,1.722706,0\n730,60621.2673,2,2.330264,0.944892,0\n730,60621.2749,1,-0.153496,0.950369,0\n730,60621.2825,3,3.589653,1.651967,0\n730,60621.2934,4,1.950011,2.746167,0\n730,60621.3044,5,6.258384,7.527862,0\n"
},
{
"path": "examples/data/plasticc_training_set_metadata_1k.csv",
"content": "object_id,ra,decl,gal_l,gal_b,ddf,hostgal_specz,hostgal_photoz,hostgal_photoz_err,distmod,mwebv,target\n615,349.046051,-61.943836,320.796530,-51.753706,1,0.0000,0.0000,0.0000,nan,0.017,92\n713,53.085938,-27.784405,223.525509,-54.460748,1,1.8181,1.6267,0.2552,45.4063,0.007,88\n730,33.574219,-6.579593,170.455585,-61.548219,1,0.2320,0.2262,0.0157,40.2561,0.021,42\n745,0.189873,-45.586655,328.254458,-68.969298,1,0.3037,0.2813,1.1523,40.7951,0.007,90\n1124,352.711273,-63.823658,316.922299,-51.059403,1,0.1934,0.2415,0.0176,40.4166,0.024,90\n1227,35.683594,-5.379379,171.992947,-59.253501,1,0.0000,0.0000,0.0000,nan,0.020,65\n1598,347.846710,-64.760857,318.929827,-49.143596,1,0.1352,0.1820,0.0304,39.7279,0.019,90\n1632,348.595886,-63.072620,320.023289,-50.713060,1,0.6857,0.7014,0.0100,43.1524,0.021,42\n1920,149.414062,3.433834,234.919132,42.245550,1,0.3088,0.3229,0.3360,41.1401,0.027,90\n1926,149.414062,1.940072,236.565366,41.393323,1,0.0000,0.0000,0.0000,nan,0.018,65\n2072,0.965665,-46.375080,325.845907,-68.579427,1,0.1516,0.1900,0.0104,39.8317,0.007,90\n2103,346.500000,-62.320400,321.951129,-50.736054,1,0.1695,0.5409,0.2283,42.4667,0.020,42\n2300,359.446716,-44.201530,331.730015,-69.805709,1,0.2360,2.7474,0.5335,46.7959,0.010,42\n2330,359.805206,-46.768478,327.135979,-67.829903,1,0.4541,0.5736,0.2827,42.6207,0.011,90\n2624,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,65\n2677,53.964844,-28.630989,225.142950,-53.813613,1,0.0000,0.0000,0.0000,nan,0.009,16\n2922,352.398651,-62.696659,318.017427,-51.967966,1,0.1539,0.1469,0.0094,39.2171,0.020,67\n3041,346.130127,-63.072620,321.423103,-50.042305,1,0.1069,0.1274,0.0198,38.8800,0.020,67\n3285,150.820312,1.641510,237.994507,42.358984,1,0.1610,0.1818,0.0079,39.7258,0.020,42\n3423,349.615387,-63.636005,318.927246,-50.506542,1,1.9876,1.1213,0.1591,44.4078,0.018,95\n3489,150.117188,2.836105,236.124718,42.483719,1,1.1330,1.4377,0.2168,45.0753,0.016,88\n3910,0.589520,-47.161343,325.385896,-67.769893,1,0.1969,2.6766,0.5926,46.7274,0.009,62\n4088,0.965665,-46.375080,325.845907,-68.579427,1,0.4833,0.4644,0.0321,42.0691,0.007,88\n4132,359.811707,-45.191612,329.485675,-69.150905,1,0.0561,0.0556,0.0301,36.9750,0.010,42\n4171,2.097458,-45.783966,324.737840,-69.478613,1,0.0000,0.0000,0.0000,nan,0.011,16\n4173,152.050781,3.284369,237.157374,44.318466,1,0.5149,0.5512,0.0221,42.5158,0.019,15\n4220,358.648071,-46.375080,329.462659,-67.716008,1,0.1197,0.1322,0.3351,38.9679,0.009,42\n4389,151.699219,3.583322,236.533224,44.205648,1,0.2333,0.2205,0.9667,40.1939,0.016,90\n4595,349.615387,-63.636005,318.927246,-50.506542,1,0.5919,0.5995,0.0127,42.7370,0.018,90\n4819,35.332031,-5.979157,172.286722,-59.931743,1,0.3053,0.2870,0.0076,40.8445,0.022,90\n5527,347.861847,-61.943836,321.519104,-51.424048,1,0.1315,0.2487,0.8604,40.4896,0.017,42\n6180,33.222656,-4.780192,167.515653,-60.396584,1,0.3201,0.2685,0.5211,40.6793,0.018,90\n6266,0.929752,-44.597992,328.531426,-70.083244,1,0.0000,0.0000,0.0000,nan,0.011,65\n6762,348.595886,-63.072620,320.023289,-50.713060,1,0.3863,0.3983,0.0132,41.6735,0.021,90\n6947,34.277344,-5.679190,170.314930,-60.410322,1,0.5680,0.5667,0.0181,42.5888,0.020,90\n7033,52.207031,-28.291550,224.208534,-55.300157,1,0.0826,0.0850,0.0073,37.9414,0.007,42\n7164,347.861847,-61.943836,321.519104,-51.424048,1,0.4299,0.4245,0.0288,41.8371,0.017,90\n7315,2.071130,-45.191612,325.606223,-69.989264,1,0.1330,0.1337,0.0171,38.9942,0.011,88\n7409,352.398651,-62.696659,318.017427,-51.967966,1,3.4451,0.5176,1.2609,42.3516,0.020,88\n7566,359.446716,-44.201530,331.730015,-69.805709,1,0.0000,0.0000,0.0000,nan,0.010,16\n7698,347.013428,-62.508568,321.472056,-50.735330,1,0.2628,0.1876,0.0216,39.8011,0.018,90\n7703,53.085938,-28.122234,224.100909,-54.509752,1,0.0830,0.0820,0.2257,37.8568,0.007,62\n7756,149.414062,2.238686,236.239766,41.565558,1,0.0000,0.0000,0.0000,nan,0.017,16\n8328,1.694561,-45.191612,326.278557,-69.858253,1,0.3779,0.4808,0.2970,42.1592,0.011,90\n8688,32.695312,-4.929937,166.868469,-60.841230,1,0.0000,0.0000,0.0000,nan,0.018,65\n8745,349.966217,-62.696659,319.542989,-51.376556,1,0.6276,0.6136,0.0129,42.7983,0.021,90\n8784,34.101562,-5.829153,170.247753,-60.638325,1,0.0000,0.0000,0.0000,nan,0.019,16\n9006,34.277344,-5.079716,169.526841,-59.956640,1,0.0000,0.0000,0.0000,nan,0.019,65\n9172,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,65\n9184,0.949367,-45.586655,326.991548,-69.251686,1,1.4031,1.2719,0.4971,44.7463,0.013,88\n9203,51.855469,-27.953188,223.543603,-55.561470,1,0.2138,0.1111,0.0626,38.5591,0.008,90\n9543,352.132874,-63.636005,317.424173,-51.095855,1,0.0000,0.0000,0.0000,nan,0.021,65\n9936,32.871094,-4.780192,166.959493,-60.615132,1,0.1633,0.0719,0.0389,37.5580,0.017,42\n9985,150.820312,3.732834,235.666318,43.572109,1,0.0000,0.0000,0.0000,nan,0.016,65\n10321,358.312500,-44.993881,332.185785,-68.685906,1,1.0833,1.1162,0.1020,44.3954,0.009,95\n10337,54.667969,-27.615883,223.610785,-53.050840,1,0.6830,0.6725,0.0089,43.0404,0.009,90\n10349,34.980469,-6.279288,172.180075,-60.389399,1,0.0000,0.0000,0.0000,nan,0.023,65\n10478,52.910156,-27.953188,223.774083,-54.639214,1,0.5552,0.2233,0.2002,40.2248,0.007,90\n10586,358.636353,-46.768478,328.890146,-67.388837,1,0.6052,0.6017,0.0153,42.7467,0.008,88\n10757,52.910156,-26.276812,220.926149,-54.363918,1,0.1699,0.1711,0.0185,39.5801,0.008,52\n10796,52.910156,-25.944481,220.366350,-54.301439,1,0.0000,0.0000,0.0000,nan,0.010,65\n10798,351.299988,-62.320400,319.038597,-52.026867,1,0.1778,0.1872,0.0121,39.7959,0.018,42\n11165,150.996094,2.985506,236.647967,43.287350,1,0.0000,0.0000,0.0000,nan,0.020,16\n11359,349.966217,-62.696659,319.542989,-51.376556,1,0.1529,0.1415,0.0072,39.1281,0.021,42\n11507,53.085938,-28.122234,224.100909,-54.509752,1,0.3312,0.5095,0.0718,42.3102,0.007,90\n11770,346.130127,-63.072620,321.423103,-50.042305,1,0.1415,0.2171,0.4350,40.1560,0.020,62\n11773,150.644531,3.583322,235.698235,43.342784,1,0.2207,0.5279,0.1679,42.4027,0.018,52\n11931,149.589844,3.583322,234.885369,42.474696,1,0.0000,0.0000,0.0000,nan,0.024,65\n11978,358.648071,-46.375080,329.462659,-67.716008,1,0.4920,0.4605,0.0179,42.0472,0.009,90\n12695,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,92\n12872,347.861847,-61.943836,321.519104,-51.424048,1,0.0000,0.0000,0.0000,nan,0.017,65\n13079,151.699219,3.583322,236.533224,44.205648,1,0.2019,2.4470,1.0434,46.4913,0.016,90\n13138,346.655182,-63.260487,320.952196,-50.040935,1,0.0756,0.5192,0.2158,42.3596,0.019,52\n13194,53.789062,-27.784405,223.685697,-53.845803,1,0.5195,0.5624,0.2843,42.5685,0.009,90\n13459,150.117188,2.836105,236.124718,42.483719,1,0.3495,0.3449,0.6556,41.3068,0.016,90\n13482,33.750000,-4.630479,168.146242,-59.949072,1,0.2929,0.3115,0.0205,41.0501,0.019,90\n13504,1.363636,-46.768478,324.669342,-68.371416,1,0.4469,0.3816,0.0766,41.5643,0.008,90\n14080,150.996094,4.181528,235.291975,43.970869,1,0.0000,0.0000,0.0000,nan,0.015,65\n14156,53.085938,-27.111860,222.384291,-54.355086,1,0.0000,0.0000,0.0000,nan,0.007,65\n14279,54.667969,-27.615883,223.610785,-53.050840,1,0.3434,0.5728,0.4518,42.6167,0.009,52\n14398,2.071130,-45.191612,325.606223,-69.989264,1,0.2812,0.2634,1.0581,40.6310,0.011,90\n14539,150.644531,3.583322,235.698235,43.342784,1,0.2882,0.2359,0.0434,40.3590,0.018,88\n14553,359.805206,-46.768478,327.135979,-67.829903,1,1.1897,1.1667,0.1717,44.5143,0.011,95\n14601,32.695312,-4.929937,166.868469,-60.841230,1,0.3837,0.3653,0.2005,41.4527,0.018,90\n14674,33.750000,-4.630479,168.146242,-59.949072,1,0.2012,0.0567,0.4176,37.0171,0.019,90\n14983,349.615387,-63.636005,318.927246,-50.506542,1,0.3391,0.3238,0.0255,41.1476,0.018,90\n15002,349.046051,-61.943836,320.796530,-51.753706,1,0.3409,0.3512,0.0531,41.3530,0.017,90\n15251,32.871094,-4.780192,166.959493,-60.615132,1,0.4653,2.3270,0.6097,46.3585,0.017,90\n15475,351.382965,-64.011238,317.574052,-50.604657,1,0.0000,0.0000,0.0000,nan,0.023,65\n15626,346.130127,-63.072620,321.423103,-50.042305,1,0.0000,0.0000,0.0000,nan,0.020,16\n15674,0.965665,-46.375080,325.845907,-68.579427,1,0.2927,0.2727,0.3286,40.7172,0.007,90\n15700,359.415588,-46.768478,327.729895,-67.686097,1,0.0000,0.0000,0.0000,nan,0.009,16\n15718,51.855469,-27.953188,223.543603,-55.561470,1,0.1193,2.3179,0.7672,46.3482,0.008,52\n15845,53.789062,-27.784405,223.685697,-53.845803,1,0.3174,0.3471,0.8216,41.3232,0.009,90\n15968,149.414062,2.238686,236.239766,41.565558,1,0.3509,0.4729,0.4544,42.1164,0.017,90\n16339,51.328125,-27.447618,222.535046,-55.950727,1,0.0000,0.0000,0.0000,nan,0.013,16\n16349,150.820312,3.134927,236.341348,43.230123,1,0.0000,0.0000,0.0000,nan,0.016,16\n16463,151.699219,3.583322,236.533224,44.205648,1,0.2023,0.1805,0.0254,39.7082,0.016,90\n16496,359.415588,-46.768478,327.729895,-67.686097,1,0.3391,0.3895,0.2635,41.6162,0.009,52\n16802,53.437500,-29.142223,225.908120,-54.336118,1,0.3145,0.3319,0.0234,41.2094,0.008,90\n16983,150.117188,3.732834,235.120533,42.993809,1,0.2899,0.2762,0.1879,40.7495,0.020,90\n17094,52.207031,-28.291550,224.208534,-55.300157,1,0.0000,0.0000,0.0000,nan,0.007,16\n17172,53.437500,-29.142223,225.908120,-54.336118,1,0.0000,0.0000,0.0000,nan,0.008,16\n17285,148.710938,2.836105,235.050801,41.328739,1,0.3073,0.3057,0.0484,41.0025,0.031,90\n17366,349.285706,-62.884678,319.786163,-51.046461,1,0.2387,0.2024,0.0247,39.9853,0.018,90\n17370,0.949367,-45.586655,326.991548,-69.251686,1,0.3138,0.3391,0.4176,41.2636,0.013,62\n17515,52.207031,-28.630989,224.800211,-55.343637,1,0.3577,0.3487,0.0073,41.3345,0.009,90\n18029,359.415588,-46.768478,327.729895,-67.686097,1,0.3525,0.3609,0.0112,41.4219,0.009,90\n18507,352.711273,-63.823658,316.922299,-51.059403,1,0.3755,0.3457,0.0230,41.3125,0.024,88\n18556,51.855469,-26.276812,220.627031,-55.293792,1,0.0000,0.0000,0.0000,nan,0.014,6\n18645,358.636353,-46.768478,328.890146,-67.388837,1,0.1640,2.3025,1.1022,46.3306,0.008,62\n18706,34.277344,-5.679190,170.314930,-60.410322,1,0.1706,0.1766,0.0158,39.6556,0.020,62\n18937,348.595886,-63.072620,320.023289,-50.713060,1,0.2142,0.2222,0.0102,40.2123,0.021,90\n18952,151.699219,3.583322,236.533224,44.205648,1,0.2800,0.2658,1.1944,40.6541,0.016,90\n19154,351.382965,-64.011238,317.574052,-50.604657,1,0.2354,2.4138,0.5022,46.4553,0.023,67\n19213,1.753247,-46.768478,324.030235,-68.498041,1,0.1254,0.1484,0.0086,39.2403,0.014,62\n19866,359.814819,-44.399834,330.775011,-69.801007,1,0.2608,0.2877,0.0235,40.8505,0.009,90\n20567,351.259003,-64.386185,317.344860,-50.255113,1,0.1549,0.1481,0.2206,39.2350,0.020,62\n20934,348.908447,-63.823658,319.169886,-50.176186,1,0.0999,2.5704,1.2137,46.6209,0.018,42\n21335,33.574219,-5.379379,168.838090,-60.637536,1,0.1542,0.2082,0.4220,40.0542,0.017,90\n22184,358.312500,-44.993881,332.185785,-68.685906,1,0.3508,0.3850,0.6064,41.5869,0.009,90\n22574,150.996094,2.985506,236.647967,43.287350,1,0.0000,0.0000,0.0000,nan,0.020,16\n22901,151.171875,1.342993,238.602520,42.464379,1,0.2581,0.2502,0.0061,40.5039,0.026,90\n23116,53.261719,-27.615883,223.280041,-54.281374,1,0.8237,0.7520,0.0300,43.3376,0.006,15\n23127,149.414062,3.433834,234.919132,42.245550,1,0.3221,0.4025,0.7933,41.7004,0.027,52\n23299,33.222656,-4.780192,167.515653,-60.396584,1,0.5869,0.5400,0.0151,42.4624,0.018,88\n23373,150.117188,3.732834,235.120533,42.993809,1,0.5442,0.5636,0.2043,42.5744,0.020,88\n23396,359.811707,-45.191612,329.485675,-69.150905,1,0.5667,0.6192,0.1193,42.8220,0.010,90\n23409,348.595886,-63.072620,320.023289,-50.713060,1,0.1407,0.1392,0.0136,39.0882,0.021,52\n23539,34.277344,-5.079716,169.526841,-59.956640,1,0.4550,0.2524,0.3112,40.5254,0.019,95\n23795,51.855469,-26.276812,220.627031,-55.293792,1,0.0000,0.0000,0.0000,nan,0.014,65\n23822,2.457983,-45.389202,324.632685,-69.945696,1,0.2411,0.2420,0.9270,40.4218,0.011,52\n23848,33.925781,-5.979157,170.179895,-60.866303,1,0.3316,0.3185,1.0181,41.1057,0.022,90\n23857,151.699219,3.583322,236.533224,44.205648,1,0.2988,0.4769,0.0894,42.1379,0.016,90\n23931,32.695312,-4.929937,166.868469,-60.841230,1,0.6282,0.6337,0.0073,42.8832,0.018,88\n24193,152.050781,2.985506,237.495952,44.143927,1,2.0958,1.3937,0.2518,44.9919,0.019,88\n24236,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,65\n24592,349.966217,-62.696659,319.542989,-51.376556,1,0.2901,0.2846,0.0249,40.8234,0.021,90\n24849,53.085938,-27.111860,222.384291,-54.355086,1,0.0000,0.0000,0.0000,nan,0.007,16\n24903,52.031250,-26.443335,220.963669,-55.168557,1,0.0000,0.0000,0.0000,nan,0.014,65\n24947,150.117188,2.238686,236.784618,42.139082,1,0.4723,0.4521,0.0193,41.9998,0.016,90\n24989,34.804688,-5.829153,171.307861,-60.174401,1,0.4468,0.4763,0.0117,42.1349,0.023,90\n25003,359.814819,-44.399834,330.775011,-69.801007,1,0.3137,0.2996,0.0218,40.9523,0.009,90\n25039,346.562500,-63.448284,320.824720,-49.866957,1,0.3161,0.2675,1.1577,40.6696,0.021,90\n25474,151.523438,3.134927,236.900695,43.803170,1,0.5236,0.5626,0.0155,42.5697,0.019,90\n25529,358.312500,-44.993881,332.185785,-68.685906,1,0.2835,0.5789,0.2180,42.6448,0.009,90\n25577,348.529419,-61.755440,321.293980,-51.763351,1,0.4028,0.3918,0.0170,41.6314,0.016,90\n25783,150.820312,3.134927,236.341348,43.230123,1,0.1040,0.1439,0.0116,39.1669,0.016,42\n25920,150.644531,3.583322,235.698235,43.342784,1,0.0000,0.0000,0.0000,nan,0.018,16\n25925,35.332031,-5.979157,172.286722,-59.931743,1,1.7327,1.7075,0.1320,45.5358,0.022,88\n26161,359.415588,-46.768478,327.729895,-67.686097,1,0.0000,0.0000,0.0000,nan,0.009,92\n26338,151.171875,2.537361,237.288526,43.169764,1,0.1892,0.2250,0.0141,40.2436,0.024,62\n26352,1.708861,-45.586655,325.688716,-69.520253,1,0.0000,0.0000,0.0000,nan,0.011,65\n26401,151.699219,3.583322,236.533224,44.205648,1,0.0000,0.0000,0.0000,nan,0.016,16\n26531,351.259003,-64.386185,317.344860,-50.255113,1,2.5314,2.4324,0.2792,46.4755,0.020,88\n26660,347.846710,-64.760857,318.929827,-49.143596,1,0.0000,0.0000,0.0000,nan,0.019,65\n26783,150.820312,1.641510,237.994507,42.358984,1,0.0000,0.0000,0.0000,nan,0.020,92\n27124,351.299988,-62.320400,319.038597,-52.026867,1,0.0000,0.0000,0.0000,nan,0.018,16\n27339,51.855469,-26.276812,220.627031,-55.293792,1,0.1432,0.1625,0.0226,39.4561,0.014,90\n27941,149.414062,1.940072,236.565366,41.393323,1,0.3632,0.3746,0.0319,41.5166,0.018,90\n28220,1.694561,-45.191612,326.278557,-69.858253,1,0.2985,0.3605,0.3149,41.4193,0.011,90\n28301,0.189873,-45.586655,328.254458,-68.969298,1,0.3606,0.2852,1.3620,40.8288,0.007,90\n28391,351.953644,-62.132156,318.777388,-52.347124,1,0.0000,0.0000,0.0000,nan,0.019,92\n28636,51.855469,-28.630989,224.733260,-55.649872,1,0.1743,0.4412,0.3366,41.9364,0.009,67\n28843,151.171875,2.537361,237.288526,43.169764,1,0.3664,0.3611,0.0225,41.4234,0.024,90\n28915,53.789062,-27.784405,223.685697,-53.845803,1,0.0000,0.0000,0.0000,nan,0.009,16\n29088,52.558594,-27.279613,222.538937,-54.845107,1,0.3037,0.3244,0.0203,41.1521,0.008,90\n29252,51.855469,-28.630989,224.733260,-55.649872,1,0.1439,0.1421,0.0233,39.1376,0.009,42\n29416,1.694561,-45.191612,326.278557,-69.858253,1,0.2168,0.1921,0.0349,39.8588,0.011,90\n29420,2.097458,-45.783966,324.737840,-69.478613,1,0.5849,0.5559,0.0102,42.5385,0.011,90\n29576,346.655182,-63.260487,320.952196,-50.040935,1,0.2362,2.5224,1.0484,46.5713,0.019,90\n29668,151.699219,3.583322,236.533224,44.205648,1,0.1461,0.1584,0.0175,39.3960,0.016,42\n29670,1.694561,-45.191612,326.278557,-69.858253,1,0.1135,0.1208,0.0198,38.7544,0.011,62\n30066,351.259003,-64.386185,317.344860,-50.255113,1,0.0000,0.0000,0.0000,nan,0.020,65\n30172,33.574219,-5.379379,168.838090,-60.637536,1,0.5444,0.5455,0.0094,42.4889,0.017,90\n30191,150.117188,2.238686,236.784618,42.139082,1,1.5405,1.3073,0.1521,44.8201,0.016,88\n30505,151.171875,2.238686,237.619933,42.994783,1,0.0000,0.0000,0.0000,nan,0.024,16\n30545,2.071130,-45.191612,325.606223,-69.989264,1,0.2160,0.2221,0.0338,40.2113,0.011,90\n30576,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,65\n30673,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,16\n30895,349.429535,-62.508568,320.039643,-51.393745,1,0.0000,0.0000,0.0000,nan,0.020,16\n31033,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,65\n31100,1.694561,-45.191612,326.278557,-69.858253,1,0.3614,0.3471,1.2589,41.3231,0.011,90\n31310,149.238281,3.882372,234.283829,42.351155,1,0.0000,0.0000,0.0000,nan,0.033,92\n31569,346.655182,-63.260487,320.952196,-50.040935,1,1.0133,1.0031,0.0118,44.1084,0.019,95\n31605,150.292969,2.686724,236.427488,42.541447,1,0.0000,0.0000,0.0000,nan,0.016,65\n31824,352.398651,-62.696659,318.017427,-51.967966,1,0.1838,0.0844,0.3311,37.9246,0.020,42\n32238,34.101562,-5.829153,170.247753,-60.638325,1,0.0000,0.0000,0.0000,nan,0.019,16\n32309,34.804688,-5.829153,171.307861,-60.174401,1,0.2251,0.2258,0.9011,40.2521,0.023,42\n32375,53.964844,-28.630989,225.142950,-53.813613,1,0.0000,0.0000,0.0000,nan,0.009,65\n32695,358.636353,-46.768478,328.890146,-67.388837,1,0.7689,0.7806,0.0113,43.4371,0.008,90\n33088,351.259003,-64.386185,317.344860,-50.255113,1,0.3437,0.3495,0.0200,41.3400,0.020,90\n33179,51.855469,-27.953188,223.543603,-55.561470,1,0.4407,0.4765,0.4079,42.1357,0.008,90\n33191,151.171875,2.238686,237.619933,42.994783,1,0.4030,0.4039,0.0174,41.7094,0.024,42\n33409,33.222656,-4.780192,167.515653,-60.396584,1,0.0000,0.0000,0.0000,nan,0.018,65\n33419,150.820312,3.732834,235.666318,43.572109,1,0.7462,0.7461,0.0356,43.3165,0.016,90\n33422,33.574219,-6.579593,170.455585,-61.548219,1,1.1111,1.1054,0.0101,44.3693,0.021,88\n34012,35.683594,-5.379379,171.992947,-59.253501,1,0.0853,0.0793,0.0210,37.7805,0.020,52\n34166,0.189873,-45.586655,328.254458,-68.969298,1,0.0873,0.0909,0.0246,38.0959,0.007,42\n34243,34.101562,-5.829153,170.247753,-60.638325,1,0.1416,0.5642,0.3631,42.5771,0.019,88\n34299,346.276581,-64.011238,320.448031,-49.344136,1,0.1901,0.2016,0.0086,39.9759,0.019,62\n34437,152.050781,2.985506,237.495952,44.143927,1,0.2657,0.2629,1.0604,40.6263,0.019,67\n35197,51.679688,-27.447618,222.618229,-55.642263,1,0.2509,0.2366,0.1269,40.3670,0.010,42\n35315,150.468750,3.732834,235.392208,43.283244,1,1.8476,1.5239,0.2256,45.2314,0.020,95\n35555,359.805206,-46.768478,327.135979,-67.829903,1,0.0000,0.0000,0.0000,nan,0.011,65\n35743,34.277344,-5.679190,170.314930,-60.410322,1,0.0781,0.0752,0.0197,37.6598,0.020,42\n35772,150.117188,2.836105,236.124718,42.483719,1,0.2385,0.2588,0.0217,40.5879,0.016,90\n35855,0.929752,-44.597992,328.531426,-70.083244,1,0.3815,0.4401,0.5094,41.9300,0.011,90\n36085,352.398651,-62.696659,318.017427,-51.967966,1,0.1689,0.1759,0.5357,39.6465,0.020,42\n36153,150.468750,1.641510,237.714575,42.075234,1,0.2547,0.2589,0.0256,40.5887,0.017,52\n36337,52.558594,-27.279613,222.538937,-54.845107,1,0.0000,0.0000,0.0000,nan,0.008,65\n36362,53.085938,-27.784405,223.525509,-54.460748,1,0.0000,0.0000,0.0000,nan,0.007,65\n36671,149.589844,3.583322,234.885369,42.474696,1,0.0000,0.0000,0.0000,nan,0.024,65\n36783,349.966217,-62.696659,319.542989,-51.376556,1,0.1287,0.1431,0.0129,39.1539,0.021,90\n37149,359.816315,-44.003082,331.451340,-70.123054,1,0.9435,0.9017,0.0524,43.8228,0.013,90\n37168,53.613281,-27.953188,223.929533,-54.024772,1,0.0000,0.0000,0.0000,nan,0.007,65\n37661,32.871094,-4.780192,166.959493,-60.615132,1,0.1226,0.0973,0.0168,38.2528,0.017,52\n37776,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,16\n37865,151.171875,2.238686,237.619933,42.994783,1,0.2263,0.2221,0.0172,40.2112,0.024,90\n37872,150.820312,3.134927,236.341348,43.230123,1,0.2517,0.2448,0.0217,40.4506,0.016,67\n38174,1.694561,-45.191612,326.278557,-69.858253,1,1.6152,1.7388,0.1564,45.5843,0.011,88\n38205,33.750000,-4.630479,168.146242,-59.949072,1,0.2945,0.2311,1.2272,40.3089,0.019,42\n38244,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,65\n38690,33.222656,-4.780192,167.515653,-60.396584,1,0.1801,0.1274,0.0307,38.8795,0.018,90\n38730,53.261719,-27.615883,223.280041,-54.281374,1,0.0000,0.0000,0.0000,nan,0.006,16\n38754,33.574219,-6.579593,170.455585,-61.548219,1,0.2646,0.2656,0.0093,40.6515,0.021,90\n38899,1.666667,-44.399834,327.519190,-70.529554,1,0.4828,0.4754,0.0332,42.1297,0.009,90\n39223,150.996094,2.388015,237.313912,42.939977,1,0.0000,0.0000,0.0000,nan,0.021,65\n39305,346.562500,-63.448284,320.824720,-49.866957,1,0.4045,0.2986,0.1602,40.9435,0.021,90\n39398,51.679688,-27.447618,222.618229,-55.642263,1,0.3347,0.5555,0.5802,42.5365,0.010,90\n39597,53.085938,-28.122234,224.100909,-54.509752,1,0.1280,0.1327,0.0064,38.9763,0.007,62\n39626,149.414062,2.238686,236.239766,41.565558,1,0.5197,0.4293,0.7344,41.8662,0.017,90\n39846,351.382965,-64.011238,317.574052,-50.604657,1,0.1886,0.2780,0.6915,40.7654,0.023,62\n40290,35.859375,-4.630479,171.270769,-58.580806,1,0.3153,0.5118,0.8469,42.3221,0.022,42\n41515,358.648071,-46.375080,329.462659,-67.716008,1,0.5720,0.5797,0.0188,42.6484,0.009,90\n41738,150.117188,3.732834,235.120533,42.993809,1,0.1206,0.1277,0.0222,38.8865,0.020,42\n42118,0.574468,-45.981140,327.041068,-68.778764,1,0.1801,0.1977,0.0131,39.9287,0.006,62\n42224,51.328125,-27.784405,223.130589,-55.999499,1,0.1119,0.0888,0.1482,38.0396,0.013,42\n42288,359.415588,-46.768478,327.729895,-67.686097,1,0.3487,0.3864,0.0241,41.5962,0.009,90\n42333,346.562500,-63.448284,320.824720,-49.866957,1,0.1921,0.2046,0.0101,40.0116,0.021,67\n42469,2.071130,-45.191612,325.606223,-69.989264,1,1.5989,1.4913,0.1216,45.1735,0.011,95\n42689,346.562500,-63.448284,320.824720,-49.866957,1,0.0000,0.0000,0.0000,nan,0.021,65\n42776,152.050781,3.284369,237.157374,44.318466,1,0.0000,0.0000,0.0000,nan,0.019,16\n42852,351.321442,-64.198746,317.458993,-50.429931,1,0.6771,0.6680,0.0223,43.0226,0.023,88\n43028,51.679688,-27.447618,222.618229,-55.642263,1,0.1366,0.1364,0.0092,39.0408,0.010,42\n43151,34.980469,-6.279288,172.180075,-60.389399,1,0.1096,0.1352,0.0222,39.0199,0.023,52\n43211,34.980469,-6.279288,172.180075,-60.389399,1,0.3321,0.3275,0.0164,41.1762,0.023,90\n43337,51.328125,-27.447618,222.535046,-55.950727,1,0.1775,0.2488,0.0180,40.4902,0.013,90\n43413,348.595886,-63.072620,320.023289,-50.713060,1,0.0000,0.0000,0.0000,nan,0.021,16\n43509,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,92\n43812,150.820312,3.134927,236.341348,43.230123,1,0.2783,0.2821,0.0219,40.8022,0.016,90\n43962,347.861847,-61.943836,321.519104,-51.424048,1,0.6595,0.6813,0.0340,43.0750,0.017,90\n44102,152.050781,3.284369,237.157374,44.318466,1,0.2450,0.2640,0.2125,40.6366,0.019,42\n44217,51.855469,-27.953188,223.543603,-55.561470,1,0.4288,0.1772,0.6584,39.6645,0.008,90\n44309,34.980469,-6.279288,172.180075,-60.389399,1,0.3131,0.3059,1.2366,41.0045,0.023,90\n44480,53.964844,-28.630989,225.142950,-53.813613,1,0.0000,0.0000,0.0000,nan,0.009,16\n44836,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,65\n45060,346.276581,-64.011238,320.448031,-49.344136,1,0.3613,0.3300,0.1387,41.1948,0.019,62\n45115,2.097458,-45.783966,324.737840,-69.478613,1,0.0000,0.0000,0.0000,nan,0.011,65\n45127,35.859375,-4.630479,171.270769,-58.580806,1,0.1378,0.1359,0.0125,39.0331,0.022,90\n45203,150.820312,3.134927,236.341348,43.230123,1,0.0000,0.0000,0.0000,nan,0.016,16\n45319,348.595886,-63.072620,320.023289,-50.713060,1,0.1270,0.0737,0.0272,37.6138,0.021,42\n45349,32.695312,-4.929937,166.868469,-60.841230,1,0.2821,2.6404,0.9837,46.6916,0.018,67\n45549,52.207031,-28.291550,224.208534,-55.300157,1,0.6733,0.7639,0.0513,43.3795,0.007,42\n46210,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,65\n46567,149.414062,3.433834,234.919132,42.245550,1,0.0000,0.0000,0.0000,nan,0.027,16\n46804,352.398651,-62.696659,318.017427,-51.967966,1,0.2933,2.3248,0.6338,46.3561,0.020,62\n46958,150.820312,3.732834,235.666318,43.572109,1,0.0000,0.0000,0.0000,nan,0.016,65\n47148,53.085938,-27.784405,223.525509,-54.460748,1,0.3153,0.3234,0.0167,41.1439,0.007,90\n47725,351.259003,-64.386185,317.344860,-50.255113,1,0.3615,0.2950,0.7526,40.9133,0.020,90\n48187,52.207031,-26.610098,221.298836,-55.042928,1,0.5466,0.5379,0.1063,42.4521,0.014,90\n48260,52.207031,-28.291550,224.208534,-55.300157,1,0.1677,0.1746,0.4651,39.6283,0.007,42\n48426,351.734680,-62.884678,318.284128,-51.651217,1,0.0000,0.0000,0.0000,nan,0.019,16\n48473,51.855469,-27.953188,223.543603,-55.561470,1,0.0809,0.0825,0.0165,37.8722,0.008,42\n48575,32.695312,-4.929937,166.868469,-60.841230,1,0.7796,0.7698,0.0437,43.4000,0.018,90\n48687,347.812500,-63.448284,320.128971,-50.202348,1,0.7916,0.8371,0.0551,43.6237,0.021,88\n48725,53.613281,-28.630989,225.073365,-54.119461,1,0.3658,0.3599,0.1804,41.4145,0.006,67\n48749,348.529419,-61.755440,321.293980,-51.763351,1,1.6645,1.5782,0.0657,45.3252,0.016,88\n48817,348.586945,-64.573555,318.693903,-49.477869,1,0.0962,0.1446,0.6309,39.1788,0.018,90\n48981,358.648071,-46.375080,329.462659,-67.716008,1,0.0000,0.0000,0.0000,nan,0.009,16\n49219,33.398438,-3.732834,166.492280,-59.466614,1,0.0000,0.0000,0.0000,nan,0.022,65\n49389,349.285706,-62.884678,319.786163,-51.046461,1,0.0000,0.0000,0.0000,nan,0.018,92\n49529,53.613281,-27.953188,223.929533,-54.024772,1,0.1974,0.2117,0.0123,40.0947,0.007,62\n49783,51.328125,-27.784405,223.130589,-55.999499,1,0.0000,0.0000,0.0000,nan,0.013,65\n49937,151.347656,3.583322,236.252362,43.918627,1,0.0947,0.1091,0.0218,38.5175,0.015,52\n50277,349.046051,-61.943836,320.796530,-51.753706,1,0.3009,0.2478,1.3214,40.4807,0.017,67\n50395,349.891296,-64.573555,317.972107,-49.786192,1,0.1552,0.1574,0.3022,39.3801,0.023,90\n51178,52.558594,-27.279613,222.538937,-54.845107,1,0.9668,0.9534,0.0188,43.9721,0.008,88\n51279,1.708861,-45.586655,325.688716,-69.520253,1,0.1760,0.1824,0.3979,39.7343,0.011,90\n51318,34.277344,-5.679190,170.314930,-60.410322,1,0.2987,0.3038,0.0273,40.9868,0.020,62\n51490,0.574468,-45.981140,327.041068,-68.778764,1,0.3707,0.4717,0.3810,42.1099,0.006,90\n51987,352.711273,-63.823658,316.922299,-51.059403,1,0.0000,0.0000,0.0000,nan,0.024,92\n52150,52.207031,-26.610098,221.298836,-55.042928,1,0.2746,0.3069,0.0656,41.0122,0.014,90\n52320,52.910156,-25.944481,220.366350,-54.301439,1,0.1656,2.0324,0.6458,46.0000,0.010,88\n52370,352.711273,-63.823658,316.922299,-51.059403,1,0.0000,0.0000,0.0000,nan,0.024,16\n52425,52.910156,-26.276812,220.926149,-54.363918,1,0.1587,0.1182,0.6266,38.7042,0.008,42\n52740,53.613281,-27.953188,223.929533,-54.024772,1,0.3653,0.4721,0.8955,42.1118,0.007,90\n52854,149.414062,1.940072,236.565366,41.393323,1,0.4383,0.4343,0.0335,41.8957,0.018,90\n53025,358.636353,-46.768478,328.890146,-67.388837,1,0.0000,0.0000,0.0000,nan,0.008,65\n53249,349.285706,-62.884678,319.786163,-51.046461,1,0.3941,0.4211,0.4203,41.8166,0.018,90\n53354,34.980469,-6.279288,172.180075,-60.389399,1,0.2231,0.2134,0.0125,40.1138,0.023,62\n53525,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,16\n53574,0.574468,-45.981140,327.041068,-68.778764,1,0.4176,0.4427,0.0420,41.9454,0.006,90\n53782,53.261719,-27.615883,223.280041,-54.281374,1,0.3798,0.3714,0.0189,41.4948,0.006,90\n53938,53.437500,-29.142223,225.908120,-54.336118,1,0.0000,0.0000,0.0000,nan,0.008,16\n54416,347.812500,-63.448284,320.128971,-50.202348,1,0.3708,0.3734,0.0162,41.5087,0.021,42\n54883,347.617462,-62.508568,321.121462,-50.904708,1,0.7844,0.8578,0.0392,43.6891,0.019,88\n54915,148.886719,2.686724,235.347248,41.389003,1,0.0000,0.0000,0.0000,nan,0.028,65\n55002,52.207031,-28.291550,224.208534,-55.300157,1,0.4345,0.4175,0.0286,41.7946,0.007,90\n55018,350.230255,-61.943836,320.053946,-52.070537,1,0.0000,0.0000,0.0000,nan,0.017,65\n55033,151.171875,1.342993,238.602520,42.464379,1,0.4920,0.5345,0.0204,42.4355,0.026,42\n55060,51.855469,-28.630989,224.733260,-55.649872,1,0.0234,0.0824,0.0202,37.8680,0.009,42\n55141,2.097458,-45.783966,324.737840,-69.478613,1,0.0000,0.0000,0.0000,nan,0.011,65\n55155,34.101562,-5.829153,170.247753,-60.638325,1,0.0000,0.0000,0.0000,nan,0.019,65\n55354,350.230255,-61.943836,320.053946,-52.070537,1,0.1958,2.3870,0.2904,46.4258,0.017,90\n55419,150.820312,3.732834,235.666318,43.572109,1,0.0967,0.1092,0.0137,38.5190,0.016,62\n55946,347.617462,-62.508568,321.121462,-50.904708,1,0.0000,0.0000,0.0000,nan,0.019,65\n56053,150.292969,2.686724,236.427488,42.541447,1,0.8507,0.8283,0.0406,43.5954,0.016,90\n56245,1.708861,-45.586655,325.688716,-69.520253,1,0.0000,0.0000,0.0000,nan,0.011,16\n56334,358.312500,-44.993881,332.185785,-68.685906,1,0.1955,0.1948,0.0100,39.8925,0.009,62\n56349,1.694561,-45.191612,326.278557,-69.858253,1,0.1646,0.2000,0.1961,39.9565,0.011,52\n56461,347.812500,-63.448284,320.128971,-50.202348,1,0.1760,2.4036,1.0097,46.4440,0.021,90\n56769,33.574219,-5.079716,168.448505,-60.407218,1,0.0000,0.0000,0.0000,nan,0.016,65\n56821,52.207031,-26.610098,221.298836,-55.042928,1,0.0775,0.0884,0.0181,38.0309,0.014,62\n56893,150.117188,2.836105,236.124718,42.483719,1,0.0000,0.0000,0.0000,nan,0.016,65\n56987,33.574219,-6.579593,170.455585,-61.548219,1,0.2910,0.3152,0.0132,41.0794,0.021,62\n57205,359.816315,-44.003082,331.451340,-70.123054,1,0.5891,0.6057,0.0306,42.7639,0.013,90\n57237,52.207031,-26.610098,221.298836,-55.042928,1,0.2544,0.3807,0.6581,41.5580,0.014,42\n57263,351.299988,-62.320400,319.038597,-52.026867,1,0.0000,0.0000,0.0000,nan,0.018,65\n57561,152.050781,3.284369,237.157374,44.318466,1,0.0000,0.0000,0.0000,nan,0.019,65\n57666,1.666667,-44.399834,327.519190,-70.529554,1,0.4399,0.4648,0.0183,42.0716,0.009,90\n57784,348.529419,-61.755440,321.293980,-51.763351,1,0.0000,0.0000,0.0000,nan,0.016,65\n58174,348.529419,-61.755440,321.293980,-51.763351,1,1.1032,1.1018,0.1226,44.3607,0.016,42\n58265,0.190678,-45.783966,327.956322,-68.803772,1,0.0650,0.1027,1.1126,38.3778,0.005,67\n58323,349.615387,-63.636005,318.927246,-50.506542,1,1.0155,1.0187,0.0471,44.1500,0.018,88\n59068,34.277344,-5.079716,169.526841,-59.956640,1,0.5716,0.5429,0.1404,42.4762,0.019,42\n59128,348.529419,-61.755440,321.293980,-51.763351,1,0.6794,0.4288,0.2152,41.8629,0.016,90\n59163,150.644531,3.583322,235.698235,43.342784,1,0.0000,0.0000,0.0000,nan,0.018,65\n59427,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,65\n59463,52.031250,-26.443335,220.963669,-55.168557,1,0.1439,0.1438,0.0169,39.1654,0.014,42\n59580,1.753247,-46.768478,324.030235,-68.498041,1,0.4178,0.5020,0.7485,42.2716,0.014,42\n59644,34.980469,-6.279288,172.180075,-60.389399,1,0.4238,0.3347,0.1863,41.2308,0.023,90\n59732,347.812500,-63.448284,320.128971,-50.202348,1,0.2489,2.7125,0.9243,46.7623,0.021,90\n60023,359.814819,-44.399834,330.775011,-69.801007,1,0.6857,0.6858,0.0077,43.0925,0.009,90\n60098,53.261719,-27.615883,223.280041,-54.281374,1,0.2529,0.2478,0.0697,40.4804,0.006,90\n60340,148.710938,2.836105,235.050801,41.328739,1,0.3521,0.3472,0.0096,41.3237,0.031,67\n60350,52.558594,-27.279613,222.538937,-54.845107,1,0.5295,0.5982,0.1284,42.7310,0.008,62\n60376,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,16\n60407,346.130127,-63.072620,321.423103,-50.042305,1,0.0000,0.0000,0.0000,nan,0.020,65\n60554,347.013428,-62.508568,321.472056,-50.735330,1,0.2356,0.2869,0.0805,40.8438,0.018,67\n60742,346.562500,-63.448284,320.824720,-49.866957,1,0.0000,0.0000,0.0000,nan,0.021,65\n60976,358.648071,-46.375080,329.462659,-67.716008,1,0.0615,0.0821,0.0078,37.8606,0.009,52\n61101,151.171875,2.537361,237.288526,43.169764,1,0.0000,0.0000,0.0000,nan,0.024,16\n61165,53.789062,-27.784405,223.685697,-53.845803,1,0.1089,0.1122,0.0140,38.5832,0.009,62\n61407,53.613281,-27.953188,223.929533,-54.024772,1,0.0000,0.0000,0.0000,nan,0.007,92\n61763,52.207031,-28.291550,224.208534,-55.300157,1,0.5749,0.5237,0.0263,42.3819,0.007,90\n62078,0.965665,-46.375080,325.845907,-68.579427,1,0.0000,0.0000,0.0000,nan,0.007,65\n62187,33.398438,-3.732834,166.492280,-59.466614,1,0.1391,0.1301,1.1896,38.9300,0.022,64\n62230,53.085938,-27.784405,223.525509,-54.460748,1,0.0000,0.0000,0.0000,nan,0.007,16\n62253,51.328125,-27.447618,222.535046,-55.950727,1,0.7671,0.7610,0.0367,43.3693,0.013,90\n62254,53.085938,-27.111860,222.384291,-54.355086,1,0.2738,0.2895,0.0180,40.8666,0.007,90\n62384,351.321442,-64.198746,317.458993,-50.429931,1,0.0000,0.0000,0.0000,nan,0.023,65\n62541,347.013428,-62.508568,321.472056,-50.735330,1,0.5490,0.3280,0.8012,41.1797,0.018,90\n62908,150.820312,3.134927,236.341348,43.230123,1,0.2372,0.2111,0.0189,40.0878,0.016,62\n63561,359.814819,-44.399834,330.775011,-69.801007,1,0.7386,0.7247,0.0129,43.2390,0.009,90\n63718,0.965665,-46.375080,325.845907,-68.579427,1,0.2891,0.4200,0.0613,41.8100,0.007,90\n63860,351.382965,-64.011238,317.574052,-50.604657,1,0.1617,0.1988,0.0128,39.9412,0.023,42\n64248,2.097458,-45.783966,324.737840,-69.478613,1,0.1653,0.1710,0.0196,39.5787,0.011,52\n64485,348.586945,-64.573555,318.693903,-49.477869,1,0.0000,0.0000,0.0000,nan,0.018,16\n64854,1.723404,-45.981140,325.117958,-69.180825,1,0.1163,0.0705,0.0098,37.5117,0.010,62\n64888,52.031250,-26.443335,220.963669,-55.168557,1,0.3802,0.4086,0.7097,41.7390,0.014,90\n64896,347.846710,-64.760857,318.929827,-49.143596,1,0.0991,0.1224,1.1874,38.7854,0.019,42\n64911,150.820312,1.641510,237.994507,42.358984,1,0.2375,2.2621,1.0009,46.2837,0.020,90\n65745,53.085938,-28.122234,224.100909,-54.509752,1,0.0680,0.0552,1.1581,36.9555,0.007,90\n65749,33.398438,-4.331149,167.226341,-59.936551,1,0.0000,0.0000,0.0000,nan,0.018,65\n65877,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,65\n66126,348.908447,-63.823658,319.169886,-50.176186,1,0.0601,0.0718,0.0163,37.5531,0.018,42\n66325,53.261719,-27.615883,223.280041,-54.281374,1,0.0000,0.0000,0.0000,nan,0.006,65\n66536,150.996094,2.388015,237.313912,42.939977,1,0.3144,0.3150,0.2975,41.0779,0.021,90\n66548,347.617462,-62.508568,321.121462,-50.904708,1,0.3713,0.4282,1.3857,41.8598,0.019,90\n66852,2.071130,-45.191612,325.606223,-69.989264,1,0.5802,0.5679,0.0103,42.5945,0.011,42\n66904,349.285706,-62.884678,319.786163,-51.046461,1,0.0000,0.0000,0.0000,nan,0.018,16\n66967,150.996094,2.388015,237.313912,42.939977,1,0.4580,0.4551,0.0163,42.0168,0.021,88\n66999,33.222656,-4.780192,167.515653,-60.396584,1,0.3635,0.4147,0.0332,41.7770,0.018,90\n67245,150.292969,2.686724,236.427488,42.541447,1,0.0482,0.0598,0.0147,37.1379,0.016,42\n67480,1.753247,-46.768478,324.030235,-68.498041,1,0.0000,0.0000,0.0000,nan,0.014,65\n67514,34.101562,-5.829153,170.247753,-60.638325,1,0.2756,0.5256,0.1081,42.3913,0.019,90\n67686,359.415588,-46.768478,327.729895,-67.686097,1,0.1785,0.2040,0.0109,40.0042,0.009,90\n67730,52.910156,-25.944481,220.366350,-54.301439,1,0.3927,0.3926,0.0321,41.6368,0.010,90\n67898,347.812500,-63.448284,320.128971,-50.202348,1,0.3824,0.3802,0.0081,41.5546,0.021,42\n67981,151.523438,3.134927,236.900695,43.803170,1,0.0000,0.0000,0.0000,nan,0.019,65\n68003,150.820312,3.732834,235.666318,43.572109,1,0.4201,0.4099,0.0162,41.7474,0.016,90\n68276,33.398438,-3.732834,166.492280,-59.466614,1,0.0251,0.0342,0.0167,35.8818,0.022,42\n68298,349.429535,-62.508568,320.039643,-51.393745,1,0.6264,0.6544,0.1479,42.9683,0.020,90\n68667,348.595886,-63.072620,320.023289,-50.713060,1,0.0000,0.0000,0.0000,nan,0.021,16\n68835,348.908447,-63.823658,319.169886,-50.176186,1,0.0177,0.0504,0.0318,36.7530,0.018,42\n68886,2.457983,-45.389202,324.632685,-69.945696,1,0.3991,0.4463,0.3629,41.9666,0.011,52\n69271,148.710938,2.836105,235.050801,41.328739,1,0.4269,0.4491,0.8217,41.9826,0.031,90\n69490,51.855469,-26.276812,220.627031,-55.293792,1,0.2919,0.4825,0.1693,42.1687,0.014,42\n69767,1.694561,-45.191612,326.278557,-69.858253,1,0.1603,2.4521,1.2066,46.4968,0.011,62\n70046,351.321442,-64.198746,317.458993,-50.429931,1,0.3441,0.3129,0.0280,41.0610,0.023,90\n70135,0.949367,-45.586655,326.991548,-69.251686,1,0.4500,0.4728,0.0167,42.1160,0.013,42\n70171,53.789062,-27.784405,223.685697,-53.845803,1,0.4708,0.5059,0.0358,42.2916,0.009,90\n70272,34.277344,-5.079716,169.526841,-59.956640,1,0.3088,0.3314,0.0357,41.2058,0.019,88\n70276,151.171875,1.342993,238.602520,42.464379,1,0.8102,0.7948,0.0774,43.4853,0.026,88\n70430,150.820312,1.641510,237.994507,42.358984,1,0.0451,0.0858,0.0255,37.9624,0.020,42\n70571,149.414062,1.940072,236.565366,41.393323,1,0.3902,0.4137,1.1078,41.7706,0.018,90\n70816,349.429535,-62.508568,320.039643,-51.393745,1,0.0000,0.0000,0.0000,nan,0.020,65\n70898,151.171875,2.238686,237.619933,42.994783,1,0.5346,0.5478,0.0157,42.4996,0.024,90\n70977,51.328125,-27.784405,223.130589,-55.999499,1,0.3428,0.3404,0.0099,41.2737,0.013,90\n71068,34.453125,-5.229529,169.987075,-59.956185,1,0.2726,0.2679,1.0038,40.6730,0.019,88\n71080,53.964844,-28.630989,225.142950,-53.813613,1,0.0000,0.0000,0.0000,nan,0.009,92\n71084,52.558594,-27.279613,222.538937,-54.845107,1,0.1522,0.1472,0.0208,39.2216,0.008,42\n71126,53.964844,-28.630989,225.142950,-53.813613,1,0.3021,0.5146,0.5741,42.3363,0.009,62\n71438,52.558594,-27.279613,222.538937,-54.845107,1,0.0000,0.0000,0.0000,nan,0.008,65\n71676,53.437500,-29.142223,225.908120,-54.336118,1,0.4396,0.4103,0.0209,41.7496,0.008,88\n71890,348.595886,-63.072620,320.023289,-50.713060,1,0.7036,0.7024,0.0078,43.1558,0.021,88\n71954,1.666667,-44.399834,327.519190,-70.529554,1,0.2483,0.2571,0.6448,40.5711,0.009,90\n72053,53.613281,-27.953188,223.929533,-54.024772,1,0.2832,0.2616,0.0190,40.6141,0.007,88\n72256,358.636353,-46.768478,328.890146,-67.388837,1,0.0000,0.0000,0.0000,nan,0.008,65\n72337,34.277344,-5.079716,169.526841,-59.956640,1,0.2449,0.2107,0.1165,40.0824,0.019,90\n72385,150.117188,2.836105,236.124718,42.483719,1,0.3029,0.2983,0.3068,40.9409,0.016,67\n72426,51.679688,-27.447618,222.618229,-55.642263,1,0.5166,0.3846,0.3149,41.5839,0.010,90\n72428,33.574219,-5.379379,168.838090,-60.637536,1,0.2214,2.4663,0.4396,46.5120,0.017,90\n72489,53.613281,-26.944359,222.237403,-53.863858,1,0.8358,0.8312,0.0197,43.6047,0.009,90\n72525,34.101562,-5.829153,170.247753,-60.638325,1,0.2407,0.2580,0.0145,40.5802,0.019,90\n72735,151.699219,3.583322,236.533224,44.205648,1,0.2648,0.2313,0.0265,40.3104,0.016,67\n73031,34.277344,-5.079716,169.526841,-59.956640,1,0.3151,0.5833,0.2442,42.6648,0.019,52\n73236,33.398438,-3.732834,166.492280,-59.466614,1,0.1398,0.5280,0.4528,42.4036,0.022,90\n73339,351.299988,-62.320400,319.038597,-52.026867,1,0.5183,0.2010,0.3771,39.9686,0.018,90\n73433,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,65\n73509,34.453125,-5.229529,169.987075,-59.956185,1,0.2953,0.2879,0.0259,40.8522,0.019,42\n73610,34.277344,-5.079716,169.526841,-59.956640,1,0.8134,1.5079,0.3508,45.2031,0.019,95\n74093,351.259003,-64.386185,317.344860,-50.255113,1,0.3940,0.4643,1.0625,42.0686,0.020,90\n75116,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,16\n75223,351.382965,-64.011238,317.574052,-50.604657,1,0.5652,0.5648,0.0157,42.5800,0.023,90\n75562,346.562500,-63.448284,320.824720,-49.866957,1,0.0000,0.0000,0.0000,nan,0.021,65\n75598,1.723404,-45.981140,325.117958,-69.180825,1,0.1652,0.1477,0.0151,39.2287,0.010,42\n75646,348.529419,-61.755440,321.293980,-51.763351,1,0.2103,0.2089,0.0083,40.0620,0.016,67\n75754,151.347656,3.583322,236.252362,43.918627,1,0.0000,0.0000,0.0000,nan,0.015,16\n75792,148.886719,2.686724,235.347248,41.389003,1,0.3699,0.3602,0.0451,41.4171,0.028,90\n75886,358.636353,-46.768478,328.890146,-67.388837,1,0.3985,0.3735,0.0375,41.5091,0.008,90\n75987,54.667969,-27.615883,223.610785,-53.050840,1,0.9014,0.8506,0.0414,43.6664,0.009,88\n76242,152.050781,2.985506,237.495952,44.143927,1,0.4916,0.5572,0.3767,42.5443,0.019,90\n76304,348.595886,-63.072620,320.023289,-50.713060,1,2.4303,2.6811,1.0262,46.7317,0.021,95\n76305,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,16\n76639,346.500000,-62.320400,321.951129,-50.736054,1,0.5322,0.5077,0.0148,42.3012,0.020,90\n77010,34.453125,-5.229529,169.987075,-59.956185,1,0.0000,0.0000,0.0000,nan,0.019,65\n77041,346.276581,-64.011238,320.448031,-49.344136,1,0.3793,0.4070,0.6742,41.7290,0.019,90\n77157,51.328125,-27.784405,223.130589,-55.999499,1,0.0000,0.0000,0.0000,nan,0.013,6\n77192,151.347656,4.181528,235.568369,44.259942,1,0.3227,0.2764,0.0329,40.7513,0.016,62\n77222,151.171875,1.342993,238.602520,42.464379,1,0.4342,0.3228,0.3114,41.1398,0.026,90\n77292,34.277344,-5.079716,169.526841,-59.956640,1,0.1250,0.1890,0.2309,39.8194,0.019,62\n77306,148.710938,2.836105,235.050801,41.328739,1,0.7167,0.6876,0.0165,43.0996,0.031,90\n77340,346.500000,-62.320400,321.951129,-50.736054,1,0.8207,0.8217,0.0401,43.5741,0.020,88\n77391,346.130127,-63.072620,321.423103,-50.042305,1,1.3214,1.4667,0.1459,45.1288,0.020,95\n77518,53.437500,-29.142223,225.908120,-54.336118,1,0.3531,0.3522,0.0125,41.3602,0.008,67\n77623,0.190678,-45.783966,327.956322,-68.803772,1,0.2765,0.2994,0.0135,40.9507,0.005,42\n77825,349.046051,-61.943836,320.796530,-51.753706,1,0.1071,0.1065,0.0160,38.4603,0.017,42\n77906,359.811707,-45.191612,329.485675,-69.150905,1,0.1265,0.0738,0.0297,37.6154,0.010,90\n77952,358.665253,-45.783966,330.353593,-68.203652,1,0.1258,0.0858,0.0192,37.9614,0.009,90\n78095,151.699219,3.583322,236.533224,44.205648,1,0.0000,0.0000,0.0000,nan,0.016,65\n78233,148.710938,2.836105,235.050801,41.328739,1,0.2391,0.2060,0.0706,40.0283,0.031,90\n78677,53.437500,-29.142223,225.908120,-54.336118,1,0.0000,0.0000,0.0000,nan,0.008,65\n78702,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,92\n78705,350.230255,-61.943836,320.053946,-52.070537,1,0.0000,0.0000,0.0000,nan,0.017,92\n78727,51.679688,-27.447618,222.618229,-55.642263,1,0.1592,0.1479,0.0203,39.2318,0.010,90\n78974,152.050781,3.284369,237.157374,44.318466,1,0.6592,0.6572,0.0085,42.9797,0.019,90\n79002,2.097458,-45.783966,324.737840,-69.478613,1,0.4446,0.2938,0.8441,40.9035,0.011,90\n79155,1.666667,-44.399834,327.519190,-70.529554,1,0.4097,0.4154,0.0160,41.7815,0.009,42\n79235,52.031250,-26.443335,220.963669,-55.168557,1,0.1639,0.1669,0.0564,39.5206,0.014,42\n79428,33.398438,-3.732834,166.492280,-59.466614,1,0.0000,0.0000,0.0000,nan,0.022,65\n79515,53.964844,-28.630989,225.142950,-53.813613,1,0.2207,0.2188,1.1526,40.1744,0.009,90\n79743,32.695312,-4.929937,166.868469,-60.841230,1,0.6266,0.6062,0.0094,42.7661,0.018,90\n79819,150.820312,1.641510,237.994507,42.358984,1,0.1742,0.1725,0.0134,39.5994,0.020,42\n79921,352.132874,-63.636005,317.424173,-51.095855,1,0.1638,0.1709,0.0219,39.5766,0.021,42\n80155,53.085938,-28.122234,224.100909,-54.509752,1,0.3628,0.3633,0.0418,41.4386,0.007,90\n80205,33.925781,-5.979157,170.179895,-60.866303,1,0.8134,0.7508,0.1176,43.3333,0.022,95\n80780,152.050781,3.284369,237.157374,44.318466,1,0.1684,0.1902,0.0581,39.8341,0.019,42\n80832,150.117188,2.238686,236.784618,42.139082,1,0.3950,0.3665,0.0146,41.4614,0.016,90\n80852,151.347656,3.583322,236.252362,43.918627,1,0.2569,0.2659,0.0082,40.6545,0.015,62\n80903,52.910156,-26.276812,220.926149,-54.363918,1,0.4140,0.4279,0.0171,41.8576,0.008,90\n81000,150.996094,2.985506,236.647967,43.287350,1,0.2989,0.3106,0.0100,41.0422,0.020,42\n81252,33.574219,-4.780192,168.064587,-60.175886,1,0.1494,0.1737,0.6985,39.6157,0.019,62\n81464,149.414062,3.433834,234.919132,42.245550,1,0.2854,0.2818,0.3207,40.7987,0.027,42\n81665,53.789062,-27.784405,223.685697,-53.845803,1,0.0000,0.0000,0.0000,nan,0.009,65\n82302,346.276581,-64.011238,320.448031,-49.344136,1,0.0000,0.0000,0.0000,nan,0.019,65\n82401,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,16\n82409,348.595886,-63.072620,320.023289,-50.713060,1,0.1147,2.6274,1.1973,46.6786,0.021,42\n82702,35.683594,-5.379379,171.992947,-59.253501,1,0.3194,0.2986,0.0204,40.9437,0.020,90\n82740,349.615387,-63.636005,318.927246,-50.506542,1,1.0263,1.0228,0.2611,44.1609,0.018,88\n83348,349.429535,-62.508568,320.039643,-51.393745,1,0.0000,0.0000,0.0000,nan,0.020,6\n83410,51.855469,-26.276812,220.627031,-55.293792,1,0.5138,0.5071,0.0093,42.2980,0.014,90\n83462,150.820312,3.134927,236.341348,43.230123,1,0.3424,0.3239,0.0179,41.1483,0.016,90\n83634,349.285706,-62.884678,319.786163,-51.046461,1,2.1107,1.3813,0.4158,44.9680,0.018,95\n83821,359.415588,-46.768478,327.729895,-67.686097,1,0.0000,0.0000,0.0000,nan,0.009,65\n83872,149.589844,3.583322,234.885369,42.474696,1,0.2160,0.2054,0.2819,40.0211,0.024,90\n83954,346.276581,-64.011238,320.448031,-49.344136,1,0.4390,0.4535,0.0428,42.0079,0.019,90\n83961,1.753247,-46.768478,324.030235,-68.498041,1,0.2177,0.2211,0.0151,40.2004,0.014,90\n84306,151.171875,2.537361,237.288526,43.169764,1,0.0000,0.0000,0.0000,nan,0.024,16\n84716,151.523438,3.134927,236.900695,43.803170,1,0.4303,0.2925,1.0152,40.8923,0.019,90\n84758,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,65\n85125,53.085938,-28.122234,224.100909,-54.509752,1,0.0000,0.0000,0.0000,nan,0.007,65\n85470,33.398438,-4.331149,167.226341,-59.936551,1,0.0000,0.0000,0.0000,nan,0.018,65\n85490,348.908447,-63.823658,319.169886,-50.176186,1,0.2537,0.5506,0.2880,42.5130,0.018,90\n85789,53.437500,-29.142223,225.908120,-54.336118,1,0.2990,0.2893,0.0367,40.8644,0.008,62\n86456,33.574219,-5.379379,168.838090,-60.637536,1,0.2127,0.2191,0.0141,40.1785,0.017,42\n86487,0.574468,-45.981140,327.041068,-68.778764,1,0.3850,0.3847,0.0333,41.5849,0.006,90\n86759,348.529419,-61.755440,321.293980,-51.763351,1,0.0000,0.0000,0.0000,nan,0.016,65\n86834,149.589844,3.583322,234.885369,42.474696,1,0.3160,0.4622,0.4742,42.0568,0.024,90\n87180,150.820312,1.641510,237.994507,42.358984,1,0.1782,0.1820,0.0103,39.7288,0.020,62\n87467,150.820312,3.134927,236.341348,43.230123,1,0.0000,0.0000,0.0000,nan,0.016,65\n87498,152.050781,2.985506,237.495952,44.143927,1,0.0000,0.0000,0.0000,nan,0.019,16\n87608,358.665253,-45.783966,330.353593,-68.203652,1,0.1894,0.2101,0.0179,40.0763,0.009,90\n87685,347.861847,-61.943836,321.519104,-51.424048,1,0.3743,0.3612,0.0219,41.4237,0.017,90\n87703,51.855469,-26.276812,220.627031,-55.293792,1,0.3246,0.5019,0.3014,42.2711,0.014,90\n88073,347.013428,-62.508568,321.472056,-50.735330,1,0.0000,0.0000,0.0000,nan,0.018,92\n88180,149.414062,1.940072,236.565366,41.393323,1,0.2996,2.6936,1.3051,46.7439,0.018,67\n88195,53.964844,-28.630989,225.142950,-53.813613,1,0.3403,0.3448,0.0304,41.3063,0.009,90\n88511,53.613281,-28.630989,225.073365,-54.119461,1,0.0000,0.0000,0.0000,nan,0.006,65\n88587,352.711273,-63.823658,316.922299,-51.059403,1,0.7839,0.7789,0.0120,43.4314,0.024,90\n88600,351.321442,-64.198746,317.458993,-50.429931,1,0.3303,0.3150,0.3295,41.0778,0.023,90\n88627,32.871094,-4.780192,166.959493,-60.615132,1,0.3934,0.4406,0.0433,41.9329,0.017,67\n88980,351.382965,-64.011238,317.574052,-50.604657,1,0.0000,0.0000,0.0000,nan,0.023,16\n89157,348.908447,-63.823658,319.169886,-50.176186,1,0.0000,0.0000,0.0000,nan,0.018,92\n89298,359.811707,-45.191612,329.485675,-69.150905,1,0.0000,0.0000,0.0000,nan,0.010,16\n89387,346.130127,-63.072620,321.423103,-50.042305,1,0.6210,0.4739,0.0648,42.1217,0.020,90\n89455,0.189873,-45.586655,328.254458,-68.969298,1,0.2603,0.2485,0.0142,40.4873,0.007,90\n89709,52.910156,-25.944481,220.366350,-54.301439,1,0.0000,0.0000,0.0000,nan,0.010,92\n89999,149.238281,3.882372,234.283829,42.351155,1,0.2640,0.3194,0.0576,41.1125,0.033,90\n90399,51.328125,-27.784405,223.130589,-55.999499,1,0.0000,0.0000,0.0000,nan,0.013,65\n90534,152.050781,3.284369,237.157374,44.318466,1,0.1759,0.1913,0.0131,39.8481,0.019,62\n90645,51.855469,-28.630989,224.733260,-55.649872,1,0.4496,0.4486,0.0173,41.9797,0.009,90\n90814,348.595886,-63.072620,320.023289,-50.713060,1,0.2252,0.2208,0.0172,40.1971,0.021,62\n90892,152.050781,3.284369,237.157374,44.318466,1,0.0322,0.0365,0.0161,36.0320,0.019,52\n91219,150.820312,3.134927,236.341348,43.230123,1,0.2921,0.5086,0.1231,42.3058,0.016,90\n91291,352.711273,-63.823658,316.922299,-51.059403,1,0.1826,0.1746,0.0132,39.6288,0.024,90\n91335,151.699219,3.583322,236.533224,44.205648,1,1.0655,1.4889,0.2658,45.1692,0.016,88\n91337,53.613281,-27.953188,223.929533,-54.024772,1,0.0000,0.0000,0.0000,nan,0.007,65\n91460,53.613281,-28.630989,225.073365,-54.119461,1,0.2782,0.5294,0.2771,42.4103,0.006,90\n91610,346.130127,-63.072620,321.423103,-50.042305,1,0.2326,0.2302,0.0070,40.2992,0.020,42\n91644,349.891296,-64.573555,317.972107,-49.786192,1,0.1893,0.1839,0.1322,39.7535,0.023,90\n91917,152.050781,2.985506,237.495952,44.143927,1,0.2448,0.2740,0.0117,40.7289,0.019,90\n91988,150.468750,1.641510,237.714575,42.075234,1,0.0000,0.0000,0.0000,nan,0.017,65\n92334,350.230255,-61.943836,320.053946,-52.070537,1,0.0000,0.0000,0.0000,nan,0.017,65\n92354,51.328125,-27.784405,223.130589,-55.999499,1,0.5449,0.5807,0.0374,42.6530,0.013,88\n92566,1.753247,-46.768478,324.030235,-68.498041,1,0.0000,0.0000,0.0000,nan,0.014,16\n92577,351.734680,-62.884678,318.284128,-51.651217,1,0.4426,0.3854,0.6831,41.5893,0.019,90\n92904,347.812500,-63.448284,320.128971,-50.202348,1,0.2536,0.2814,0.0325,40.7958,0.021,62\n92929,348.908447,-63.823658,319.169886,-50.176186,1,0.0000,0.0000,0.0000,nan,0.018,65\n93333,151.171875,1.342993,238.602520,42.464379,1,0.4029,0.3217,0.4659,41.1310,0.026,90\n93362,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,92\n93509,51.855469,-26.276812,220.627031,-55.293792,1,0.0587,0.0644,0.0151,37.3066,0.014,42\n93663,53.964844,-28.630989,225.142950,-53.813613,1,0.4196,0.4099,0.0367,41.7471,0.009,90\n94004,52.910156,-27.953188,223.774083,-54.639214,1,0.2354,0.2257,0.0162,40.2510,0.007,62\n94107,34.980469,-6.279288,172.180075,-60.389399,1,0.0000,0.0000,0.0000,nan,0.023,16\n94229,52.207031,-28.291550,224.208534,-55.300157,1,0.3853,0.4477,0.0268,41.9745,0.007,90\n94613,2.457983,-45.389202,324.632685,-69.945696,1,0.0739,0.0616,0.0188,37.2044,0.011,62\n94704,349.966217,-62.696659,319.542989,-51.376556,1,0.2722,0.2658,0.0080,40.6537,0.021,90\n95127,351.299988,-62.320400,319.038597,-52.026867,1,1.8136,1.6691,0.2276,45.4751,0.018,95\n95147,149.238281,3.882372,234.283829,42.351155,1,0.0000,0.0000,0.0000,nan,0.033,65\n95369,53.613281,-28.630989,225.073365,-54.119461,1,0.1313,0.5834,0.7684,42.6652,0.006,62\n95455,351.259003,-64.386185,317.344860,-50.255113,1,0.4942,0.5590,0.1899,42.5528,0.020,90\n95483,150.820312,3.732834,235.666318,43.572109,1,0.8512,0.8164,0.0436,43.5568,0.016,88\n95508,149.414062,2.238686,236.239766,41.565558,1,0.6107,0.5480,0.0306,42.5009,0.017,88\n95566,33.574219,-5.079716,168.448505,-60.407218,1,0.3880,0.4580,0.0992,42.0331,0.016,90\n95580,34.101562,-5.829153,170.247753,-60.638325,1,0.4200,0.4311,0.0132,41.8770,0.019,90\n95690,351.734680,-62.884678,318.284128,-51.651217,1,0.3442,0.3507,0.0052,41.3494,0.019,42\n95741,35.332031,-5.979157,172.286722,-59.931743,1,0.5064,0.5368,0.0209,42.4469,0.022,52\n95864,53.085938,-27.111860,222.384291,-54.355086,1,0.0000,0.0000,0.0000,nan,0.007,92\n96284,152.050781,3.284369,237.157374,44.318466,1,0.1593,2.4014,0.4125,46.4417,0.019,42\n97053,150.117188,3.732834,235.120533,42.993809,1,0.0000,0.0000,0.0000,nan,0.020,65\n97406,347.846710,-64.760857,318.929827,-49.143596,1,0.1163,0.0897,0.8592,38.0649,0.019,15\n97687,346.655182,-63.260487,320.952196,-50.040935,1,0.4603,0.4626,0.0271,42.0594,0.019,90\n97850,351.259003,-64.386185,317.344860,-50.255113,1,0.3090,0.3005,0.0119,40.9599,0.020,90\n97920,150.820312,3.134927,236.341348,43.230123,1,0.3588,0.3524,0.0238,41.3611,0.016,67\n97957,1.723404,-45.981140,325.117958,-69.180825,1,0.2947,0.2906,0.0093,40.8754,0.010,90\n98533,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,65\n98570,1.708861,-45.586655,325.688716,-69.520253,1,0.6197,0.5296,0.0328,42.4113,0.011,42\n98749,33.750000,-4.630479,168.146242,-59.949072,1,0.1473,0.0973,0.6573,38.2516,0.019,67\n99013,350.230255,-61.943836,320.053946,-52.070537,1,0.5255,0.5303,0.0220,42.4150,0.017,90\n99050,52.207031,-26.610098,221.298836,-55.042928,1,0.2972,0.3036,0.1492,40.9851,0.014,42\n99261,53.613281,-27.953188,223.929533,-54.024772,1,0.1411,0.0857,0.0282,37.9579,0.007,90\n99280,359.811707,-45.191612,329.485675,-69.150905,1,0.2037,0.2150,0.0079,40.1324,0.010,62\n99293,347.846710,-64.760857,318.929827,-49.143596,1,0.3106,0.3644,0.0251,41.4467,0.019,90\n99294,348.529419,-61.755440,321.293980,-51.763351,1,0.5552,0.5204,0.0185,42.3656,0.016,90\n99452,352.711273,-63.823658,316.922299,-51.059403,1,0.8420,0.8479,0.0465,43.6581,0.024,88\n99642,347.846710,-64.760857,318.929827,-49.143596,1,0.2320,0.2369,1.1397,40.3699,0.019,90\n99862,52.207031,-28.291550,224.208534,-55.300157,1,0.5810,0.5733,0.0141,42.6194,0.007,42\n99932,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,65\n100057,346.130127,-63.072620,321.423103,-50.042305,1,0.8320,0.8006,0.0197,43.5045,0.020,90\n100097,348.529419,-61.755440,321.293980,-51.763351,1,0.1566,0.1745,0.0186,39.6271,0.016,42\n100133,346.655182,-63.260487,320.952196,-50.040935,1,0.2309,0.2215,0.0190,40.2046,0.019,42\n100331,52.207031,-26.610098,221.298836,-55.042928,1,0.1684,0.1999,0.0173,39.9556,0.014,42\n101050,32.695312,-4.929937,166.868469,-60.841230,1,0.2222,0.2529,0.0906,40.5303,0.018,90\n101298,34.277344,-5.079716,169.526841,-59.956640,1,0.1848,0.1983,0.0069,39.9355,0.019,90\n101374,51.855469,-26.276812,220.627031,-55.293792,1,0.0000,0.0000,0.0000,nan,0.014,16\n101489,359.816315,-44.003082,331.451340,-70.123054,1,0.0721,0.0695,0.0025,37.4803,0.013,42\n101508,358.648071,-46.375080,329.462659,-67.716008,1,0.0656,0.0249,0.0130,35.1822,0.009,90\n101890,346.276581,-64.011238,320.448031,-49.344136,1,0.0000,0.0000,0.0000,nan,0.019,65\n102036,53.789062,-27.784405,223.685697,-53.845803,1,0.2331,0.2269,0.0197,40.2635,0.009,42\n102330,148.886719,2.686724,235.347248,41.389003,1,0.0000,0.0000,0.0000,nan,0.028,16\n102343,51.328125,-27.447618,222.535046,-55.950727,1,0.1081,0.1422,0.0164,39.1394,0.013,67\n102363,349.160583,-64.760857,318.219706,-49.458924,1,0.8046,0.7835,0.0167,43.4470,0.020,90\n102745,349.615387,-63.636005,318.927246,-50.506542,1,0.2460,0.2333,1.0359,40.3315,0.018,90\n102823,347.861847,-61.943836,321.519104,-51.424048,1,0.0000,0.0000,0.0000,nan,0.017,65\n102864,0.574468,-45.981140,327.041068,-68.778764,1,0.2123,0.2254,0.0097,40.2473,0.006,42\n103023,359.811707,-45.191612,329.485675,-69.150905,1,0.3221,0.2994,0.0125,40.9500,0.010,90\n103026,348.529419,-61.755440,321.293980,-51.763351,1,0.1949,0.2006,0.0174,39.9637,0.016,42\n103100,34.101562,-5.829153,170.247753,-60.638325,1,0.5057,0.4350,0.0240,41.9001,0.019,90\n103145,349.160583,-64.760857,318.219706,-49.458924,1,0.6603,0.6380,0.0433,42.9010,0.020,90\n103162,350.230255,-61.943836,320.053946,-52.070537,1,0.1101,0.1207,0.0096,38.7539,0.017,90\n103171,1.753247,-46.768478,324.030235,-68.498041,1,0.0000,0.0000,0.0000,nan,0.014,92\n103350,359.446716,-44.201530,331.730015,-69.805709,1,0.6777,0.6935,0.0507,43.1220,0.010,90\n103354,150.117188,2.836105,236.124718,42.483719,1,2.7124,2.7655,0.0801,46.8131,0.016,88\n103572,52.207031,-28.291550,224.208534,-55.300157,1,0.0000,0.0000,0.0000,nan,0.007,65\n103927,150.820312,1.641510,237.994507,42.358984,1,0.2130,0.2282,0.0218,40.2779,0.020,90\n103948,33.574219,-4.780192,168.064587,-60.175886,1,0.2257,0.2319,0.1182,40.3169,0.019,52\n103967,0.190678,-45.783966,327.956322,-68.803772,1,0.5094,0.4789,0.0608,42.1493,0.005,42\n104212,53.789062,-27.784405,223.685697,-53.845803,1,0.3895,0.3850,1.0136,41.5869,0.009,90\n104397,349.966217,-62.696659,319.542989,-51.376556,1,0.3220,0.3038,0.0187,40.9870,0.021,90\n104476,51.855469,-28.630989,224.733260,-55.649872,1,1.4820,1.4819,0.1602,45.1565,0.009,95\n104498,149.238281,3.882372,234.283829,42.351155,1,0.1312,0.1100,0.0094,38.5371,0.033,67\n104523,152.050781,3.284369,237.157374,44.318466,1,0.0000,0.0000,0.0000,nan,0.019,16\n104526,349.615387,-63.636005,318.927246,-50.506542,1,0.2353,0.2389,0.0148,40.3902,0.018,42\n104701,352.132874,-63.636005,317.424173,-51.095855,1,0.0000,0.0000,0.0000,nan,0.021,65\n105744,348.908447,-63.823658,319.169886,-50.176186,1,0.3087,0.3318,1.0382,41.2090,0.018,90\n106177,358.312500,-44.993881,332.185785,-68.685906,1,0.6593,0.5956,0.1033,42.7194,0.009,90\n106429,152.050781,2.985506,237.495952,44.143927,1,0.0000,0.0000,0.0000,nan,0.019,16\n106434,33.574219,-4.780192,168.064587,-60.175886,1,0.2861,0.2795,0.0098,40.7789,0.019,62\n106594,0.589520,-47.161343,325.385896,-67.769893,1,0.2403,0.2370,1.0591,40.3710,0.009,88\n106730,347.013428,-62.508568,321.472056,-50.735330,1,0.1576,2.2437,0.3333,46.2621,0.018,52\n106743,0.574468,-45.981140,327.041068,-68.778764,1,0.2154,0.2539,0.0150,40.5405,0.006,90\n106818,348.595886,-63.072620,320.023289,-50.713060,1,0.0000,0.0000,0.0000,nan,0.021,92\n106937,53.085938,-28.122234,224.100909,-54.509752,1,0.3656,0.3882,0.8841,41.6079,0.007,52\n107193,359.058563,-45.191612,330.695783,-68.844915,1,0.2109,0.1761,0.0082,39.6492,0.011,90\n107439,150.468750,3.732834,235.392208,43.283244,1,0.9187,1.5247,0.3061,45.2328,0.020,88\n107451,347.861847,-61.943836,321.519104,-51.424048,1,0.0000,0.0000,0.0000,nan,0.017,65\n107568,33.750000,-4.630479,168.146242,-59.949072,1,0.1042,0.0802,0.0143,37.8074,0.019,67\n107615,150.117188,2.836105,236.124718,42.483719,1,0.1323,2.3872,0.9055,46.4260,0.016,42\n107712,53.261719,-27.615883,223.280041,-54.281374,1,0.1282,0.1450,0.0286,39.1863,0.006,90\n107901,359.058563,-45.191612,330.695783,-68.844915,1,0.0000,0.0000,0.0000,nan,0.011,65\n108021,150.996094,4.181528,235.291975,43.970869,1,0.2815,0.3756,0.8142,41.5232,0.015,62\n108141,53.789062,-27.784405,223.685697,-53.845803,1,0.2179,0.2239,0.0151,40.2317,0.009,90\n108229,351.321442,-64.198746,317.458993,-50.429931,1,0.1858,0.2022,1.1539,39.9826,0.023,90\n108358,349.891296,-64.573555,317.972107,-49.786192,1,0.0769,0.2433,1.0869,40.4350,0.023,52\n108487,359.816315,-44.003082,331.451340,-70.123054,1,0.1787,0.2401,0.6645,40.4026,0.013,42\n108554,33.222656,-4.780192,167.515653,-60.396584,1,0.4478,0.4542,0.0258,42.0120,0.018,90\n108693,0.574468,-45.981140,327.041068,-68.778764,1,0.4868,0.4183,0.0364,41.7991,0.006,90\n108739,53.085938,-28.122234,224.100909,-54.509752,1,0.4453,0.4388,0.0272,41.9226,0.007,90\n108888,358.648071,-46.375080,329.462659,-67.716008,1,0.6247,0.5951,0.0599,42.7172,0.009,90\n109036,1.753247,-46.768478,324.030235,-68.498041,1,0.1924,0.1862,0.2684,39.7838,0.014,90\n109057,348.595886,-63.072620,320.023289,-50.713060,1,0.4499,0.3307,1.0499,41.2005,0.021,90\n109294,359.814819,-44.399834,330.775011,-69.801007,1,0.3014,0.3643,0.8070,41.4461,0.009,90\n109516,1.753247,-46.768478,324.030235,-68.498041,1,0.3828,0.3941,0.0095,41.6466,0.014,95\n109654,347.013428,-62.508568,321.472056,-50.735330,1,0.1395,0.1231,0.0131,38.8003,0.018,90\n109860,0.929752,-44.597992,328.531426,-70.083244,1,0.2708,0.2639,0.0184,40.6357,0.011,42\n109903,150.996094,2.388015,237.313912,42.939977,1,0.3892,0.3579,0.0137,41.4006,0.021,62\n109937,149.414062,3.433834,234.919132,42.245550,1,0.6620,0.6423,0.0188,42.9190,0.027,88\n110241,34.453125,-5.229529,169.987075,-59.956185,1,0.1072,0.0924,0.4007,38.1327,0.019,42\n110257,148.886719,2.686724,235.347248,41.389003,1,0.9318,0.8495,0.0379,43.6631,0.028,88\n110270,54.667969,-27.615883,223.610785,-53.050840,1,0.8259,0.8541,0.0446,43.6776,0.009,90\n110304,53.261719,-27.615883,223.280041,-54.281374,1,0.2475,0.5457,0.2676,42.4899,0.006,62\n110387,151.347656,4.181528,235.568369,44.259942,1,0.4318,0.4218,0.6713,41.8205,0.016,90\n110551,51.328125,-27.447618,222.535046,-55.950727,1,0.4861,0.3027,0.3931,40.9783,0.013,88\n110768,351.734680,-62.884678,318.284128,-51.651217,1,2.9378,2.8626,1.1139,46.9035,0.019,88\n110958,349.615387,-63.636005,318.927246,-50.506542,1,0.4630,0.4742,0.5031,42.1236,0.018,90\n111281,0.589520,-47.161343,325.385896,-67.769893,1,1.4967,1.6014,0.0895,45.3643,0.009,42\n111283,150.468750,1.641510,237.714575,42.075234,1,0.6565,0.6363,0.0252,42.8941,0.017,90\n111448,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,65\n111650,351.734680,-62.884678,318.284128,-51.651217,1,0.0000,0.0000,0.0000,nan,0.019,65\n111795,34.453125,-5.229529,169.987075,-59.956185,1,0.4070,0.3077,1.0584,41.0187,0.019,90\n111799,52.558594,-27.279613,222.538937,-54.845107,1,0.2813,0.5741,0.4749,42.6227,0.008,42\n112151,349.966217,-62.696659,319.542989,-51.376556,1,0.5341,0.5419,0.0204,42.4716,0.021,90\n112462,33.574219,-4.780192,168.064587,-60.175886,1,0.4045,0.4356,0.0299,41.9040,0.019,90\n112629,346.276581,-64.011238,320.448031,-49.344136,1,0.0000,0.0000,0.0000,nan,0.019,16\n112717,35.683594,-5.379379,171.992947,-59.253501,1,0.2867,0.3138,0.8216,41.0680,0.020,67\n112764,347.812500,-63.448284,320.128971,-50.202348,1,0.3208,0.2995,0.0255,40.9517,0.021,90\n112782,32.871094,-4.780192,166.959493,-60.615132,1,0.4211,0.4038,0.0224,41.7088,0.017,90\n112886,351.953644,-62.132156,318.777388,-52.347124,1,0.0000,0.0000,0.0000,nan,0.019,65\n113028,151.699219,3.583322,236.533224,44.205648,1,0.0000,0.0000,0.0000,nan,0.016,65\n113206,52.207031,-28.291550,224.208534,-55.300157,1,0.2446,0.2208,0.0078,40.1969,0.007,42\n113335,358.636353,-46.768478,328.890146,-67.388837,1,0.0000,0.0000,0.0000,nan,0.008,65\n113625,149.589844,3.583322,234.885369,42.474696,1,0.2615,0.2381,0.0160,40.3824,0.024,90\n113669,351.299988,-62.320400,319.038597,-52.026867,1,0.2914,0.2905,0.0061,40.8748,0.018,15\n113982,359.805206,-46.768478,327.135979,-67.829903,1,0.7438,0.9216,0.1482,43.8813,0.011,90\n114191,33.574219,-6.579593,170.455585,-61.548219,1,0.0000,0.0000,0.0000,nan,0.021,65\n114341,151.699219,3.583322,236.533224,44.205648,1,0.1709,2.3232,0.5051,46.3542,0.016,42\n114626,2.071130,-45.191612,325.606223,-69.989264,1,0.5702,0.5408,0.0178,42.4659,0.011,90\n114670,150.996094,2.388015,237.313912,42.939977,1,0.2930,0.2607,0.0670,40.6057,0.021,90\n114715,0.965665,-46.375080,325.845907,-68.579427,1,0.2141,0.2020,0.0362,39.9807,0.007,67\n114808,52.207031,-26.610098,221.298836,-55.042928,1,0.3037,0.2944,0.0104,40.9082,0.014,90\n115053,33.925781,-5.979157,170.179895,-60.866303,1,0.1649,0.1424,0.2329,39.1427,0.022,42\n115079,1.694561,-45.191612,326.278557,-69.858253,1,0.2207,0.2325,0.0072,40.3232,0.011,90\n115157,2.457983,-45.389202,324.632685,-69.945696,1,0.6382,0.5628,0.0442,42.5706,0.011,90\n115336,351.734680,-62.884678,318.284128,-51.651217,1,1.7312,1.7123,0.0766,45.5432,0.019,95\n115638,358.665253,-45.783966,330.353593,-68.203652,1,0.6365,0.6239,0.0079,42.8420,0.009,90\n115670,151.347656,3.583322,236.252362,43.918627,1,0.0000,0.0000,0.0000,nan,0.015,16\n115792,352.711273,-63.823658,316.922299,-51.059403,1,0.2389,0.2859,0.0505,40.8350,0.024,90\n115859,148.886719,2.686724,235.347248,41.389003,1,1.4248,1.2871,0.1757,44.7782,0.028,88\n115937,51.679688,-27.447618,222.618229,-55.642263,1,0.0000,0.0000,0.0000,nan,0.010,65\n116132,150.292969,2.686724,236.427488,42.541447,1,0.0878,0.0965,0.0207,38.2336,0.016,62\n116212,150.644531,3.583322,235.698235,43.342784,1,0.2344,0.2757,1.1756,40.7443,0.018,90\n116570,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,16\n116720,34.980469,-6.279288,172.180075,-60.389399,1,0.4081,0.3212,0.0367,41.1269,0.023,90\n116818,359.805206,-46.768478,327.135979,-67.829903,1,0.6528,0.7091,0.0211,43.1811,0.011,88\n117016,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,92\n117104,346.130127,-63.072620,321.423103,-50.042305,1,0.0000,0.0000,0.0000,nan,0.020,92\n117184,352.132874,-63.636005,317.424173,-51.095855,1,0.0000,0.0000,0.0000,nan,0.021,16\n117393,52.910156,-25.944481,220.366350,-54.301439,1,0.8259,0.8295,0.0204,43.5993,0.010,90\n117461,53.261719,-27.615883,223.280041,-54.281374,1,0.0000,0.0000,0.0000,nan,0.006,92\n117513,151.347656,3.583322,236.252362,43.918627,1,0.8515,0.8845,0.0148,43.7712,0.015,90\n117774,346.276581,-64.011238,320.448031,-49.344136,1,0.5589,0.5608,0.1532,42.5611,0.019,90\n118211,151.699219,3.583322,236.533224,44.205648,1,0.2420,0.2139,1.1778,40.1199,0.016,90\n118422,149.589844,3.583322,234.885369,42.474696,1,0.0000,0.0000,0.0000,nan,0.024,65\n118455,347.861847,-61.943836,321.519104,-51.424048,1,0.1346,0.1396,0.3139,39.0951,0.017,90\n118770,53.085938,-28.122234,224.100909,-54.509752,1,0.2642,0.6375,0.0267,42.8991,0.007,90\n118868,350.230255,-61.943836,320.053946,-52.070537,1,0.0000,0.0000,0.0000,nan,0.017,65\n118979,2.457983,-45.389202,324.632685,-69.945696,1,0.2374,0.5280,0.1420,42.4036,0.011,67\n119215,52.031250,-26.443335,220.963669,-55.168557,1,0.2490,0.2486,0.0069,40.4887,0.014,90\n119383,0.949367,-45.586655,326.991548,-69.251686,1,0.3501,0.2602,0.6723,40.6012,0.013,42\n119494,150.117188,2.836105,236.124718,42.483719,1,0.2113,0.2256,0.0219,40.2502,0.016,90\n119647,151.523438,3.134927,236.900695,43.803170,1,0.1844,0.1571,0.0210,39.3762,0.019,62\n119811,348.586945,-64.573555,318.693903,-49.477869,1,0.7287,0.7966,0.0308,43.4912,0.018,90\n119882,358.312500,-44.993881,332.185785,-68.685906,1,0.0000,0.0000,0.0000,nan,0.009,65\n120356,32.695312,-4.929937,166.868469,-60.841230,1,0.0582,0.0543,0.0185,36.9203,0.018,42\n120927,152.050781,3.284369,237.157374,44.318466,1,0.4807,0.4897,0.0334,42.2071,0.019,90\n121107,35.332031,-5.979157,172.286722,-59.931743,1,0.5440,0.2209,0.2405,40.1982,0.022,90\n121182,53.085938,-28.122234,224.100909,-54.509752,1,0.0000,0.0000,0.0000,nan,0.007,65\n121224,359.811707,-45.191612,329.485675,-69.150905,1,0.1244,2.3005,0.4304,46.3283,0.010,42\n121266,151.347656,3.583322,236.252362,43.918627,1,0.4062,0.5002,0.0481,42.2623,0.015,42\n121301,52.207031,-28.630989,224.800211,-55.343637,1,0.1459,0.1449,1.1250,39.1845,0.009,62\n121440,53.613281,-28.630989,225.073365,-54.119461,1,0.2230,0.2219,0.1530,40.2089,0.006,42\n121447,0.189873,-45.586655,328.254458,-68.969298,1,0.2571,0.2465,0.0106,40.4674,0.007,90\n121704,359.805206,-46.768478,327.135979,-67.829903,1,1.5357,1.5245,0.1053,45.2324,0.011,88\n121705,352.132874,-63.636005,317.424173,-51.095855,1,0.1125,0.1219,0.0187,38.7767,0.021,42\n121783,35.332031,-5.979157,172.286722,-59.931743,1,0.1324,0.0956,0.0132,38.2109,0.022,90\n121803,348.586945,-64.573555,318.693903,-49.477869,1,0.2563,0.3703,1.1769,41.4870,0.018,90\n121883,150.117188,2.836105,236.124718,42.483719,1,0.0000,0.0000,0.0000,nan,0.016,65\n122235,53.085938,-28.122234,224.100909,-54.509752,1,0.0000,0.0000,0.0000,nan,0.007,65\n122275,54.667969,-27.615883,223.610785,-53.050840,1,0.3172,0.5059,0.1684,42.2916,0.009,90\n122716,33.398438,-3.732834,166.492280,-59.466614,1,0.0000,0.0000,0.0000,nan,0.022,65\n122965,53.085938,-28.122234,224.100909,-54.509752,1,0.7135,0.7416,0.0359,43.3006,0.007,90\n123035,150.468750,3.732834,235.392208,43.283244,1,0.2740,0.3135,0.5471,41.0660,0.020,52\n123151,52.207031,-28.291550,224.208534,-55.300157,1,0.2548,0.2657,0.0184,40.6528,0.007,62\n123211,150.996094,2.985506,236.647967,43.287350,1,1.6177,1.6654,0.0205,45.4691,0.020,88\n123244,1.363636,-46.768478,324.669342,-68.371416,1,0.4582,0.3682,0.3963,41.4728,0.008,42\n123437,348.595886,-63.072620,320.023289,-50.713060,1,0.1987,0.2137,0.0164,40.1178,0.021,52\n123493,34.804688,-5.829153,171.307861,-60.174401,1,1.0395,0.9736,0.2348,44.0284,0.023,95\n123743,152.050781,2.985506,237.495952,44.143927,1,0.2016,0.2129,0.0188,40.1084,0.019,90\n123926,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,16\n123927,150.468750,3.732834,235.392208,43.283244,1,0.0000,0.0000,0.0000,nan,0.020,16\n124006,2.457983,-45.389202,324.632685,-69.945696,1,0.3574,0.3496,0.7517,41.3410,0.011,90\n124183,33.574219,-6.579593,170.455585,-61.548219,1,0.0000,0.0000,0.0000,nan,0.021,65\n124188,348.908447,-63.823658,319.169886,-50.176186,1,0.0000,0.0000,0.0000,nan,0.018,65\n124361,51.855469,-27.953188,223.543603,-55.561470,1,0.3256,0.3420,0.8779,41.2856,0.008,88\n124394,52.207031,-26.610098,221.298836,-55.042928,1,0.9198,1.4761,0.2825,45.1459,0.014,90\n124679,346.276581,-64.011238,320.448031,-49.344136,1,0.3602,0.3409,0.0345,41.2771,0.019,52\n124762,347.861847,-61.943836,321.519104,-51.424048,1,0.1442,2.8043,0.6276,46.8496,0.017,42\n125095,352.711273,-63.823658,316.922299,-51.059403,1,0.0000,0.0000,0.0000,nan,0.024,65\n125242,349.891296,-64.573555,317.972107,-49.786192,1,0.2721,0.5662,0.2621,42.5866,0.023,90\n125258,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,65\n125426,349.429535,-62.508568,320.039643,-51.393745,1,1.1327,1.1093,0.0315,44.3789,0.020,88\n125470,34.277344,-5.079716,169.526841,-59.956640,1,0.5731,0.5848,0.0092,42.6713,0.019,90\n125518,359.805206,-46.768478,327.135979,-67.829903,1,0.0000,0.0000,0.0000,nan,0.011,65\n125743,347.013428,-62.508568,321.472056,-50.735330,1,0.9950,1.0046,0.0133,44.1126,0.018,42\n125762,1.753247,-46.768478,324.030235,-68.498041,1,0.0000,0.0000,0.0000,nan,0.014,65\n126061,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,65\n126084,149.238281,3.882372,234.283829,42.351155,1,0.1250,0.1223,0.4578,38.7846,0.033,90\n126970,150.996094,2.985506,236.647967,43.287350,1,0.0000,0.0000,0.0000,nan,0.020,16\n127056,34.453125,-5.229529,169.987075,-59.956185,1,0.1629,0.1385,0.0373,39.0772,0.019,42\n127488,348.908447,-63.823658,319.169886,-50.176186,1,0.0000,0.0000,0.0000,nan,0.018,65\n127773,53.085938,-27.111860,222.384291,-54.355086,1,0.0000,0.0000,0.0000,nan,0.007,65\n127942,348.595886,-63.072620,320.023289,-50.713060,1,0.3899,0.5026,0.2876,42.2748,0.021,90\n127996,346.562500,-63.448284,320.824720,-49.866957,1,0.6993,0.6759,0.0121,43.0539,0.021,95\n128339,0.965665,-46.375080,325.845907,-68.579427,1,0.2192,2.1719,1.1420,46.1760,0.007,67\n128405,352.398651,-62.696659,318.017427,-51.967966,1,0.0000,0.0000,0.0000,nan,0.020,65\n128488,150.468750,1.641510,237.714575,42.075234,1,0.1333,0.1194,0.0257,38.7277,0.017,42\n128518,34.101562,-5.829153,170.247753,-60.638325,1,0.5644,0.5449,0.0420,42.4859,0.019,90\n128564,150.117188,2.238686,236.784618,42.139082,1,0.4541,2.8846,1.1987,46.9235,0.016,90\n128737,52.558594,-27.279613,222.538937,-54.845107,1,0.4204,0.4270,0.0598,41.8523,0.008,90\n128746,152.050781,2.985506,237.495952,44.143927,1,0.3408,0.3671,0.5585,41.4654,0.019,52\n128967,150.644531,3.583322,235.698235,43.342784,1,0.3145,0.3149,0.0153,41.0770,0.018,90\n129179,346.130127,-63.072620,321.423103,-50.042305,1,0.1422,0.1600,0.0245,39.4198,0.020,52\n129490,34.453125,-5.229529,169.987075,-59.956185,1,0.3803,0.4821,0.2384,42.1664,0.019,42\n129503,150.292969,2.686724,236.427488,42.541447,1,0.1939,0.1882,0.0140,39.8092,0.016,52\n129637,33.925781,-5.979157,170.179895,-60.866303,1,0.0000,0.0000,0.0000,nan,0.022,92\n129648,348.529419,-61.755440,321.293980,-51.763351,1,0.3613,2.9518,0.4926,46.9838,0.016,62\n129861,351.321442,-64.198746,317.458993,-50.429931,1,0.5180,0.5215,0.0294,42.3710,0.023,90\n130220,346.276581,-64.011238,320.448031,-49.344136,1,0.5002,0.4635,0.0365,42.0643,0.019,42\n130404,150.820312,3.732834,235.666318,43.572109,1,0.2531,0.2629,0.0072,40.6262,0.016,42\n130502,35.683594,-5.379379,171.992947,-59.253501,1,0.2141,0.1909,0.0935,39.8439,0.020,90\n130625,1.708861,-45.586655,325.688716,-69.520253,1,0.0000,0.0000,0.0000,nan,0.011,16\n130750,359.816315,-44.003082,331.451340,-70.123054,1,0.2536,0.2917,0.7396,40.8850,0.013,90\n131075,52.910156,-25.944481,220.366350,-54.301439,1,0.1709,0.1776,0.0174,39.6689,0.010,42\n131181,34.101562,-5.829153,170.247753,-60.638325,1,0.0000,0.0000,0.0000,nan,0.019,16\n131305,351.382965,-64.011238,317.574052,-50.604657,1,0.2846,0.3113,0.0407,41.0484,0.023,90\n131368,150.996094,2.388015,237.313912,42.939977,1,0.0000,0.0000,0.0000,nan,0.021,65\n131488,0.965665,-46.375080,325.845907,-68.579427,1,0.2691,0.2621,0.0183,40.6187,0.007,90\n131492,150.820312,3.732834,235.666318,43.572109,1,0.0000,0.0000,0.0000,nan,0.016,16\n131629,33.574219,-5.079716,168.448505,-60.407218,1,1.1119,1.0817,0.0329,44.3113,0.016,95\n131814,151.699219,3.583322,236.533224,44.205648,1,0.2823,0.5387,0.2038,42.4557,0.016,90\n131815,32.695312,-4.929937,166.868469,-60.841230,1,0.3521,0.3520,0.0254,41.3584,0.018,90\n132021,359.058563,-45.191612,330.695783,-68.844915,1,0.2525,0.6394,0.1855,42.9069,0.011,90\n132278,359.415588,-46.768478,327.729895,-67.686097,1,0.0000,0.0000,0.0000,nan,0.009,92\n133074,347.617462,-62.508568,321.121462,-50.904708,1,0.2610,0.2672,0.8507,40.6666,0.019,90\n133191,346.276581,-64.011238,320.448031,-49.344136,1,0.4648,0.4423,0.0247,41.9430,0.019,90\n133234,151.523438,3.134927,236.900695,43.803170,1,0.2288,0.2866,0.0891,40.8416,0.019,15\n133354,358.665253,-45.783966,330.353593,-68.203652,1,0.4990,0.5076,0.0086,42.3007,0.009,90\n133513,34.980469,-6.279288,172.180075,-60.389399,1,0.4337,0.4221,0.0715,41.8224,0.023,90\n133773,149.414062,3.433834,234.919132,42.245550,1,0.0000,0.0000,0.0000,nan,0.027,53\n134380,150.996094,2.985506,236.647967,43.287350,1,0.3943,0.3670,0.0425,41.4645,0.020,90\n134824,351.734680,-62.884678,318.284128,-51.651217,1,0.1196,0.1143,0.0103,38.6270,0.019,42\n135054,350.230255,-61.943836,320.053946,-52.070537,1,0.3893,0.4001,0.3373,41.6849,0.017,90\n135067,148.886719,2.686724,235.347248,41.389003,1,0.5285,0.5059,0.6484,42.2917,0.028,90\n135097,151.171875,1.342993,238.602520,42.464379,1,0.2066,0.3032,0.0238,40.9819,0.026,42\n135357,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,65\n135588,151.523438,3.134927,236.900695,43.803170,1,0.0000,0.0000,0.0000,nan,0.019,16\n135790,150.117188,2.238686,236.784618,42.139082,1,0.4136,0.3284,0.7703,41.1829,0.016,90\n135813,0.589520,-47.161343,325.385896,-67.769893,1,0.2577,0.2445,0.0139,40.4473,0.009,52\n136110,53.261719,-27.615883,223.280041,-54.281374,1,0.3785,0.4183,0.7565,41.7995,0.006,90\n136352,51.328125,-27.784405,223.130589,-55.999499,1,0.0000,0.0000,0.0000,nan,0.013,65\n136407,33.574219,-4.780192,168.064587,-60.175886,1,0.1110,0.0954,0.0100,38.2070,0.019,42\n136704,150.820312,3.732834,235.666318,43.572109,1,0.2725,1.2302,1.1813,44.6568,0.016,62\n136931,52.558594,-27.279613,222.538937,-54.845107,1,0.3304,0.3783,0.1057,41.5421,0.008,52\n136949,351.321442,-64.198746,317.458993,-50.429931,1,0.0000,0.0000,0.0000,nan,0.023,65\n137510,346.562500,-63.448284,320.824720,-49.866957,1,0.0000,0.0000,0.0000,nan,0.021,65\n137645,51.855469,-26.276812,220.627031,-55.293792,1,0.1590,0.1754,0.0130,39.6390,0.014,42\n138010,52.207031,-28.630989,224.800211,-55.343637,1,0.2411,0.2709,0.0169,40.7009,0.009,42\n138068,150.292969,2.686724,236.427488,42.541447,1,0.2270,2.7496,0.8355,46.7979,0.016,42\n138263,51.328125,-27.784405,223.130589,-55.999499,1,0.4085,0.4242,0.0398,41.8353,0.013,90\n138415,349.429535,-62.508568,320.039643,-51.393745,1,0.2761,0.2500,0.0203,40.5026,0.020,42\n138553,347.812500,-63.448284,320.128971,-50.202348,1,0.3010,0.2608,0.2763,40.6065,0.021,90\n138947,349.160583,-64.760857,318.219706,-49.458924,1,0.5965,0.5575,0.2970,42.5456,0.020,42\n139016,151.523438,3.134927,236.900695,43.803170,1,0.0000,0.0000,0.0000,nan,0.019,16\n139329,151.699219,3.583322,236.533224,44.205648,1,0.5029,0.5454,0.2843,42.4885,0.016,90\n139362,149.238281,3.882372,234.283829,42.351155,1,0.0809,0.0779,0.0097,37.7408,0.033,64\n139405,1.666667,-44.399834,327.519190,-70.529554,1,0.3701,0.4458,0.8403,41.9636,0.009,90\n139637,359.816315,-44.003082,331.451340,-70.123054,1,0.4142,0.4028,0.3993,41.7023,0.013,90\n140096,2.457983,-45.389202,324.632685,-69.945696,1,0.2211,0.2342,0.0540,40.3413,0.011,90\n140472,52.558594,-27.279613,222.538937,-54.845107,1,0.0000,0.0000,0.0000,nan,0.008,65\n140948,1.666667,-44.399834,327.519190,-70.529554,1,0.2048,0.1926,0.0117,39.8645,0.009,62\n141212,53.613281,-28.630989,225.073365,-54.119461,1,0.1256,2.3091,0.5229,46.3382,0.006,90\n141302,150.292969,2.686724,236.427488,42.541447,1,0.4316,0.4620,0.3119,42.0560,0.016,42\n141334,149.238281,3.882372,234.283829,42.351155,1,0.0000,0.0000,0.0000,nan,0.033,65\n141686,53.085938,-28.122234,224.100909,-54.509752,1,0.3431,0.3179,0.2942,41.1008,0.007,67\n141937,151.523438,3.134927,236.900695,43.803170,1,0.4487,0.4658,0.0138,42.0770,0.019,90\n142099,52.207031,-26.610098,221.298836,-55.042928,1,0.0000,0.0000,0.0000,nan,0.014,65\n142254,52.910156,-25.944481,220.366350,-54.301439,1,0.4858,0.5210,0.3055,42.3685,0.010,90\n142368,347.013428,-62.508568,321.472056,-50.735330,1,0.2586,0.0878,0.7050,38.0148,0.018,90\n142866,150.996094,2.388015,237.313912,42.939977,1,0.6327,0.6476,0.0215,42.9408,0.021,42\n142867,349.966217,-62.696659,319.542989,-51.376556,1,0.1548,0.1591,0.0118,39.4056,0.021,42\n142885,52.207031,-28.291550,224.208534,-55.300157,1,0.4057,0.3852,0.0218,41.5882,0.007,90\n143066,0.189873,-45.586655,328.254458,-68.969298,1,0.1937,0.5270,0.1240,42.3983,0.007,42\n143275,2.071130,-45.191612,325.606223,-69.989264,1,0.1114,0.1030,0.0117,38.3830,0.011,42\n143651,34.101562,-5.829153,170.247753,-60.638325,1,0.3590,0.3440,0.0396,41.3003,0.019,90\n143865,33.925781,-5.979157,170.179895,-60.866303,1,0.0000,0.0000,0.0000,nan,0.022,16\n144204,53.085938,-27.111860,222.384291,-54.355086,1,0.4825,0.5378,0.3853,42.4513,0.007,90\n144244,33.222656,-4.780192,167.515653,-60.396584,1,0.2120,0.6014,0.1268,42.7453,0.018,90\n145107,34.980469,-6.279288,172.180075,-60.389399,1,0.0000,0.0000,0.0000,nan,0.023,16\n145160,0.189873,-45.586655,328.254458,-68.969298,1,0.0000,0.0000,0.0000,nan,0.007,65\n145257,52.207031,-28.291550,224.208534,-55.300157,1,0.2570,0.2708,0.2312,40.7001,0.007,90\n145675,51.855469,-27.953188,223.543603,-55.561470,1,0.8655,0.8841,0.0293,43.7698,0.008,90\n145859,34.980469,-6.279288,172.180075,-60.389399,1,0.2213,0.2411,0.0230,40.4126,0.023,90\n145926,152.050781,3.284369,237.157374,44.318466,1,0.2061,0.2269,0.0092,40.2640,0.019,42\n145990,33.222656,-4.780192,167.515653,-60.396584,1,0.4648,0.4366,0.0319,41.9096,0.018,90\n146187,33.398438,-3.732834,166.492280,-59.466614,1,0.0000,0.0000,0.0000,nan,0.022,92\n146410,359.805206,-46.768478,327.135979,-67.829903,1,0.2555,0.4043,0.6020,41.7117,0.011,90\n146429,150.117188,2.238686,236.784618,42.139082,1,0.1260,0.1220,0.0168,38.7782,0.016,62\n147214,51.855469,-27.953188,223.543603,-55.561470,1,0.1677,0.1843,0.0127,39.7584,0.008,90\n147571,0.190678,-45.783966,327.956322,-68.803772,1,0.1885,0.2575,0.1606,40.5748,0.005,90\n147642,52.910156,-26.276812,220.926149,-54.363918,1,0.2453,0.2263,0.0090,40.2576,0.008,62\n147752,151.523438,3.134927,236.900695,43.803170,1,0.2042,0.1823,0.0158,39.7324,0.019,90\n147816,34.277344,-5.079716,169.526841,-59.956640,1,0.0000,0.0000,0.0000,nan,0.019,92\n148204,151.171875,2.238686,237.619933,42.994783,1,0.0000,0.0000,0.0000,nan,0.024,65\n148466,151.171875,2.238686,237.619933,42.994783,1,0.5915,0.5684,0.0143,42.5965,0.024,90\n148535,34.277344,-5.679190,170.314930,-60.410322,1,0.1966,0.1835,0.0107,39.7482,0.020,42\n148543,349.046051,-61.943836,320.796530,-51.753706,1,0.4717,0.5098,0.0652,42.3120,0.017,90\n148976,53.964844,-28.630989,225.142950,-53.813613,1,0.6993,0.6715,0.0353,43.0365,0.009,90\n148996,51.855469,-26.276812,220.627031,-55.293792,1,0.0258,0.0954,0.0390,38.2066,0.014,15\n149129,0.589520,-47.161343,325.385896,-67.769893,1,0.1440,2.5349,0.7043,46.5843,0.009,90\n149130,151.171875,2.537361,237.288526,43.169764,1,0.1925,0.1847,0.0130,39.7633,0.024,90\n149478,2.071130,-45.191612,325.606223,-69.989264,1,0.1744,0.5308,0.1019,42.4174,0.011,42\n149673,33.398438,-4.331149,167.226341,-59.936551,1,0.3411,0.3499,0.0077,41.3436,0.018,90\n150266,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,16\n150344,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,65\n150561,53.261719,-27.615883,223.280041,-54.281374,1,0.2265,0.2307,0.0085,40.3046,0.006,42\n150765,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,65\n150818,53.613281,-26.944359,222.237403,-53.863858,1,0.1401,0.1231,0.0150,38.8002,0.009,90\n150880,351.382965,-64.011238,317.574052,-50.604657,1,0.1168,0.0932,0.0179,38.1533,0.023,42\n151356,51.679688,-27.447618,222.618229,-55.642263,1,0.0000,0.0000,0.0000,nan,0.010,16\n151427,0.574468,-45.981140,327.041068,-68.778764,1,0.0370,0.0313,0.0117,35.6883,0.006,42\n151458,53.261719,-27.615883,223.280041,-54.281374,1,0.0000,0.0000,0.0000,nan,0.006,65\n151462,33.574219,-4.780192,168.064587,-60.175886,1,0.2901,0.3886,0.4161,41.6102,0.019,42\n151498,359.446716,-44.201530,331.730015,-69.805709,1,0.4319,0.4487,0.5711,41.9804,0.010,90\n151694,54.667969,-27.615883,223.610785,-53.050840,1,0.0000,0.0000,0.0000,nan,0.009,92\n151704,150.292969,2.686724,236.427488,42.541447,1,0.1558,0.1295,0.6463,38.9183,0.016,52\n151973,150.996094,4.181528,235.291975,43.970869,1,0.0000,0.0000,0.0000,nan,0.015,92\n152079,33.574219,-5.379379,168.838090,-60.637536,1,0.3503,0.3429,0.0254,41.2924,0.017,90\n152083,150.468750,3.732834,235.392208,43.283244,1,0.5001,0.5041,0.0165,42.2827,0.020,90\n152300,33.574219,-6.579593,170.455585,-61.548219,1,0.7306,0.6981,0.0380,43.1398,0.021,42\n152425,150.820312,3.732834,235.666318,43.572109,1,0.0000,0.0000,0.0000,nan,0.016,65\n152453,52.910156,-27.953188,223.774083,-54.639214,1,0.2741,0.5064,0.2047,42.2944,0.007,90\n152567,150.996094,4.181528,235.291975,43.970869,1,0.0000,0.0000,0.0000,nan,0.015,16\n152618,33.574219,-5.079716,168.448505,-60.407218,1,0.0000,0.0000,0.0000,nan,0.016,16\n152640,33.574219,-4.780192,168.064587,-60.175886,1,0.0000,0.0000,0.0000,nan,0.019,65\n152682,51.679688,-27.447618,222.618229,-55.642263,1,0.0000,0.0000,0.0000,nan,0.010,16\n152756,150.996094,4.181528,235.291975,43.970869,1,0.1326,0.1837,0.4894,39.7513,0.015,62\n152787,351.299988,-62.320400,319.038597,-52.026867,1,0.0000,0.0000,0.0000,nan,0.018,65\n152812,32.695312,-4.929937,166.868469,-60.841230,1,2.2378,2.2813,0.0673,46.3061,0.018,88\n153089,35.332031,-5.979157,172.286722,-59.931743,1,0.0000,0.0000,0.0000,nan,0.022,16\n153539,54.667969,-27.615883,223.610785,-53.050840,1,0.3371,0.3157,0.0155,41.0832,0.009,88\n153880,51.679688,-27.447618,222.618229,-55.642263,1,0.3501,0.3152,0.0731,41.0793,0.010,90\n154053,34.101562,-5.829153,170.247753,-60.638325,1,0.1526,0.1479,0.0167,39.2327,0.019,42\n154402,151.171875,2.238686,237.619933,42.994783,1,0.0294,0.0619,0.0172,37.2157,0.024,90\n154631,358.648071,-46.375080,329.462659,-67.716008,1,0.9576,0.8148,0.0688,43.5515,0.009,90\n154648,359.811707,-45.191612,329.485675,-69.150905,1,1.4174,1.4408,0.0548,45.0811,0.010,88\n154762,35.859375,-4.630479,171.270769,-58.580806,1,0.3187,0.2788,1.0862,40.7724,0.022,90\n154986,150.996094,2.985506,236.647967,43.287350,1,0.0000,0.0000,0.0000,nan,0.020,16\n155110,148.886719,2.686724,235.347248,41.389003,1,0.0000,0.0000,0.0000,nan,0.028,65\n155380,150.996094,2.388015,237.313912,42.939977,1,0.2147,0.2288,0.0127,40.2844,0.021,88\n155468,150.996094,4.181528,235.291975,43.970869,1,0.0000,0.0000,0.0000,nan,0.015,65\n155541,151.347656,4.181528,235.568369,44.259942,1,0.5115,0.4788,0.0312,42.1482,0.016,90\n155613,148.710938,2.836105,235.050801,41.328739,1,0.3585,0.3694,0.0090,41.4811,0.031,90\n155778,53.085938,-27.111860,222.384291,-54.355086,1,0.2234,0.5912,0.2372,42.6999,0.007,42\n156386,151.523438,3.134927,236.900695,43.803170,1,1.4308,1.2425,0.1528,44.6834,0.019,88\n156537,352.132874,-63.636005,317.424173,-51.095855,1,2.0260,2.3090,0.0769,46.3380,0.021,88\n156739,351.321442,-64.198746,317.458993,-50.429931,1,0.3405,0.4842,0.8467,42.1775,0.023,90\n157120,51.328125,-27.447618,222.535046,-55.950727,1,0.2126,0.2539,0.5364,40.5402,0.013,42\n157299,34.453125,-5.229529,169.987075,-59.956185,1,0.5504,0.6351,0.0335,42.8890,0.019,42\n157477,359.805206,-46.768478,327.135979,-67.829903,1,0.9586,0.7920,0.0736,43.4756,0.011,95\n157746,149.414062,1.940072,236.565366,41.393323,1,0.2167,0.5603,0.2926,42.5588,0.018,52\n158042,1.666667,-44.399834,327.519190,-70.529554,1,0.1965,0.2093,0.0089,40.0667,0.009,90\n158241,347.013428,-62.508568,321.472056,-50.735330,1,0.0424,0.0344,0.0149,35.8982,0.018,62\n158507,351.299988,-62.320400,319.038597,-52.026867,1,0.6109,0.7017,0.0936,43.1535,0.018,90\n158515,52.910156,-26.276812,220.926149,-54.363918,1,0.3265,0.3196,0.0097,41.1146,0.008,42\n158573,33.574219,-4.780192,168.064587,-60.175886,1,0.2736,0.5337,0.1790,42.4315,0.019,90\n158697,35.332031,-5.979157,172.286722,-59.931743,1,0.3830,0.4049,0.0394,41.7159,0.022,90\n158731,347.846710,-64.760857,318.929827,-49.143596,1,0.0000,0.0000,0.0000,nan,0.019,65\n158813,150.820312,3.134927,236.341348,43.230123,1,0.1966,0.1962,0.0134,39.9093,0.016,62\n158904,2.097458,-45.783966,324.737840,-69.478613,1,1.8591,1.6874,0.1207,45.5041,0.011,88\n159277,359.805206,-46.768478,327.135979,-67.829903,1,0.6452,0.5890,0.0356,42.6903,0.011,90\n159316,151.699219,3.583322,236.533224,44.205648,1,0.0000,0.0000,0.0000,nan,0.016,65\n159491,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,16\n159665,33.574219,-4.780192,168.064587,-60.175886,1,0.3253,0.3336,0.0122,41.2221,0.019,90\n159925,150.820312,3.134927,236.341348,43.230123,1,1.6095,1.6763,0.0691,45.4866,0.016,88\n160048,51.855469,-28.630989,224.733260,-55.649872,1,0.3638,0.4409,0.3427,41.9350,0.009,90\n160426,351.321442,-64.198746,317.458993,-50.429931,1,0.3511,0.3353,0.0637,41.2356,0.023,90\n160527,2.097458,-45.783966,324.737840,-69.478613,1,0.3162,0.2266,0.6723,40.2609,0.011,90\n160737,52.207031,-28.291550,224.208534,-55.300157,1,0.3593,0.5374,0.2042,42.4496,0.007,88\n160921,349.160583,-64.760857,318.219706,-49.458924,1,0.0000,0.0000,0.0000,nan,0.020,65\n161135,52.207031,-28.291550,224.208534,-55.300157,1,0.4293,0.4326,0.0209,41.8860,0.007,90\n161411,150.468750,3.732834,235.392208,43.283244,1,0.0000,0.0000,0.0000,nan,0.020,65\n161432,149.414062,2.238686,236.239766,41.565558,1,0.0000,0.0000,0.0000,nan,0.017,16\n161521,346.130127,-63.072620,321.423103,-50.042305,1,0.1147,0.1640,0.9394,39.4785,0.020,90\n161591,150.996094,4.181528,235.291975,43.970869,1,0.0761,0.1416,0.0294,39.1295,0.015,62\n161877,150.468750,1.641510,237.714575,42.075234,1,0.0000,0.0000,0.0000,nan,0.017,65\n161988,51.679688,-27.447618,222.618229,-55.642263,1,0.5641,0.5875,0.0180,42.6834,0.010,90\n162093,51.855469,-28.630989,224.733260,-55.649872,1,0.1580,0.1822,0.0192,39.7311,0.009,62\n162139,150.820312,3.134927,236.341348,43.230123,1,0.1036,0.0926,0.0114,38.1367,0.016,42\n162152,348.529419,-61.755440,321.293980,-51.763351,1,0.0000,0.0000,0.0000,nan,0.016,16\n162531,347.861847,-61.943836,321.519104,-51.424048,1,0.5290,0.4535,0.0438,42.0076,0.017,90\n162538,150.996094,4.181528,235.291975,43.970869,1,0.2795,0.5154,0.2561,42.3402,0.015,90\n162994,53.085938,-28.122234,224.100909,-54.509752,1,0.5827,0.5520,0.0825,42.5199,0.007,90\n163208,33.398438,-4.331149,167.226341,-59.936551,1,0.1546,0.1227,0.0022,38.7928,0.018,42\n163680,0.574468,-45.981140,327.041068,-68.778764,1,0.3407,0.3516,0.0201,41.3557,0.006,90\n163894,52.207031,-28.630989,224.800211,-55.343637,1,0.3027,0.3162,0.0110,41.0874,0.009,88\n164582,32.871094,-4.780192,166.959493,-60.615132,1,0.6341,0.6230,0.0125,42.8383,0.017,90\n164805,51.679688,-27.447618,222.618229,-55.642263,1,0.0000,0.0000,0.0000,nan,0.010,92\n165406,351.321442,-64.198746,317.458993,-50.429931,1,0.0000,0.0000,0.0000,nan,0.023,65\n165494,349.615387,-63.636005,318.927246,-50.506542,1,0.2616,0.2665,0.0118,40.6604,0.018,90\n165507,151.171875,1.342993,238.602520,42.464379,1,0.2323,0.2251,0.0187,40.2445,0.026,67\n165821,149.414062,2.238686,236.239766,41.565558,1,0.2458,1.0072,0.4491,44.1195,0.017,42\n165985,51.679688,-27.447618,222.618229,-55.642263,1,0.4551,0.4933,0.2305,42.2259,0.010,88\n166103,0.589520,-47.161343,325.385896,-67.769893,1,0.2871,0.2079,0.4627,40.0507,0.009,90\n166165,150.117188,2.836105,236.124718,42.483719,1,0.0000,0.0000,0.0000,nan,0.016,16\n166186,51.679688,-27.447618,222.618229,-55.642263,1,0.5156,0.5329,0.0107,42.4278,0.010,42\n166195,149.238281,3.882372,234.283829,42.351155,1,0.0000,0.0000,0.0000,nan,0.033,16\n166330,149.414062,1.940072,236.565366,41.393323,1,0.0000,0.0000,0.0000,nan,0.018,65\n166697,53.613281,-28.630989,225.073365,-54.119461,1,1.1664,1.1160,0.0368,44.3951,0.006,95\n166727,34.453125,-5.229529,169.987075,-59.956185,1,0.2659,0.2946,0.0220,40.9098,0.019,62\n166956,359.811707,-45.191612,329.485675,-69.150905,1,0.0000,0.0000,0.0000,nan,0.010,65\n167123,149.414062,2.238686,236.239766,41.565558,1,0.0000,0.0000,0.0000,nan,0.017,65\n167220,359.446716,-44.201530,331.730015,-69.805709,1,0.7136,0.7235,0.0416,43.2348,0.010,90\n167260,2.071130,-45.191612,325.606223,-69.989264,1,0.3449,0.3554,0.0193,41.3826,0.011,62\n167310,349.615387,-63.636005,318.927246,-50.506542,1,0.3079,0.3246,0.0181,41.1538,0.018,42\n167417,349.046051,-61.943836,320.796530,-51.753706,1,0.5774,0.5558,0.0410,42.5377,0.017,90\n167436,350.230255,-61.943836,320.053946,-52.070537,1,0.1918,0.2263,0.0138,40.2576,0.017,90\n167488,348.586945,-64.573555,318.693903,-49.477869,1,0.3928,0.3939,0.0171,41.6449,0.018,90\n167910,348.908447,-63.823658,319.169886,-50.176186,1,0.4761,0.4646,0.0434,42.0701,0.018,90\n168146,51.855469,-26.276812,220.627031,-55.293792,1,0.0000,0.0000,0.0000,nan,0.014,65\n168465,149.414062,1.940072,236.565366,41.393323,1,2.1492,2.4337,0.1988,46.4769,0.018,95\n168659,53.613281,-27.953188,223.929533,-54.024772,1,0.2663,0.3135,0.2502,41.0656,0.007,67\n168952,358.312500,-44.993881,332.185785,-68.685906,1,0.0000,0.0000,0.0000,nan,0.009,65\n168957,53.085938,-27.784405,223.525509,-54.460748,1,0.0000,0.0000,0.0000,nan,0.007,65\n168967,347.812500,-63.448284,320.128971,-50.202348,1,0.1845,0.2142,0.1608,40.1234,0.021,90\n168989,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,16\n169133,347.013428,-62.508568,321.472056,-50.735330,1,0.4656,0.4799,0.2303,42.1546,0.018,90\n169203,347.861847,-61.943836,321.519104,-51.424048,1,0.1833,0.1791,0.1849,39.6900,0.017,90\n169282,149.414062,3.433834,234.919132,42.245550,1,0.3181,0.3458,0.3165,41.3133,0.027,90\n"
},
{
"path": "examples/docker/modin-ray/Dockerfile",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n# Build image from this dockerfile like this:\n# docker build -t modin-ray:latest .\n\nFROM ubuntu:20.04\n\n# Proxy settings\nENV http_proxy=${http_proxy}\nENV https_proxy=${https_proxy}\nENV no_proxy=${no_proxy}\n\nRUN apt-get update --yes \\\n && apt-get install wget --yes \\\n && rm -rf /var/lib/apt/lists/*\n\nENV USER modin\nENV UID 1000\nENV HOME /home/$USER\n\nRUN adduser --disabled-password \\\n --gecos \"Non-root user\" \\\n --uid $UID \\\n --home $HOME \\\n $USER\n\n# Conda settings\nENV CONDA_DIR=${HOME}/miniconda\nENV CONDA_ENV_NAME=modin-ray\nENV PATH=\"${CONDA_DIR}/bin:${PATH}\"\n\nRUN wget -nv https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /tmp/miniconda3.sh \\\n && bash /tmp/miniconda3.sh -b -p \"${CONDA_DIR}\" -f -u \\\n && \"${CONDA_DIR}/bin/conda\" init bash \\\n && rm -f /tmp/miniconda3.sh\n\nRUN conda update -n base -c defaults conda -y \\\n && conda create -n ${CONDA_ENV_NAME} --yes -c conda-forge --strict-channel-priority \\\n modin-ray \\\n ray-dashboard \\\n scikit-learn \\\n scikit-learn-intelex \\\n xgboost \\\n && conda clean --all --yes\n\n# Activate ${CONDA_ENV_NAME} for interactive shells\nRUN echo \"source ${CONDA_DIR}/bin/activate ${CONDA_ENV_NAME}\" >> \"${HOME}/.bashrc\"\n# Activate ${CONDA_ENV_NAME} for non-interactive shells\n# The following line comments out line that prevents ~/.bashrc execution in\n# non-interactive mode.\nRUN sed -e 's,\\(^[[:space:]]\\+[*]) return;;$\\),# \\1,' -i \"${HOME}/.bashrc\"\nENV BASH_ENV=\"${HOME}/.bashrc\"\n\n# Set up benchmark scripts\nCOPY nyc-taxi.py \"${HOME}\"\nCOPY census.py \"${HOME}\"\nCOPY plasticc.py \"${HOME}\"\nRUN mkdir /dataset\nWORKDIR ${HOME}\n\n# Clean up proxy settings to publish on Docker Hub\nENV http_proxy=\nENV https_proxy=\nENV no_proxy=\n\n# Set entrypoint with arguments expansion\nENTRYPOINT [\"/bin/bash\", \"-c\", \"exec $0 $*\"]\n"
},
{
"path": "examples/docker/modin-ray/build-docker-image.sh",
"content": "#!/bin/bash -e\n\n# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\ncd \"`dirname \\\"$0\\\"`\"\n\ndocker build -t modin-ray .\n\necho -e '\\nNYC TAXI BENCHMARK\nUser is responsible for preparing the dataset.\nIt Can be generated by following the instructions on the link:\nhttps://github.com/toddwschneider/nyc-taxi-data#instructions\nTo run the benchmark execute:\n\\tdocker run --rm -v /path/to/dataset:/dataset modin-ray python nyc-taxi.py \n\nCENSUS BENCHMARK\nUser is responsible for preparing the dataset.\nIt can be downloaded from the following link:\nhttps://rapidsai-data.s3.us-east-2.amazonaws.com/datasets/ipums_education2income_1970-2010.csv.gz\nTo run the benchmark execute:\n\\tdocker run --rm -v /path/to/dataset:/dataset modin-ray python census.py \n\nPLASTICC BENCHMARK\nUser is responsible for preparing the datasets.\nThe datasets must include four files: training set, test set,\ntraining set metadata and test set metadata.\nTo run the benchmark execute:\n\\tdocker run --rm -v /path/to/dataset:/dataset modin-ray python plasticc.py \\n'\n"
},
{
"path": "examples/docker/modin-ray/census.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport sys\nimport time\n\nimport sklearnex\nfrom sklearn import config_context\n\nimport modin.pandas as pd\n\nsklearnex.patch_sklearn()\nimport numpy as np\nimport sklearn.linear_model as lm\nfrom sklearn.model_selection import train_test_split\n\n\ndef read(filename):\n columns_names = [\n \"YEAR0\",\n \"DATANUM\",\n \"SERIAL\",\n \"CBSERIAL\",\n \"HHWT\",\n \"CPI99\",\n \"GQ\",\n \"QGQ\",\n \"PERNUM\",\n \"PERWT\",\n \"SEX\",\n \"AGE\",\n \"EDUC\",\n \"EDUCD\",\n \"INCTOT\",\n \"SEX_HEAD\",\n \"SEX_MOM\",\n \"SEX_POP\",\n \"SEX_SP\",\n \"SEX_MOM2\",\n \"SEX_POP2\",\n \"AGE_HEAD\",\n \"AGE_MOM\",\n \"AGE_POP\",\n \"AGE_SP\",\n \"AGE_MOM2\",\n \"AGE_POP2\",\n \"EDUC_HEAD\",\n \"EDUC_MOM\",\n \"EDUC_POP\",\n \"EDUC_SP\",\n \"EDUC_MOM2\",\n \"EDUC_POP2\",\n \"EDUCD_HEAD\",\n \"EDUCD_MOM\",\n \"EDUCD_POP\",\n \"EDUCD_SP\",\n \"EDUCD_MOM2\",\n \"EDUCD_POP2\",\n \"INCTOT_HEAD\",\n \"INCTOT_MOM\",\n \"INCTOT_POP\",\n \"INCTOT_SP\",\n \"INCTOT_MOM2\",\n \"INCTOT_POP2\",\n ]\n columns_types = [\n \"int64\",\n \"int64\",\n \"int64\",\n \"float64\",\n \"int64\",\n \"float64\",\n \"int64\",\n \"float64\",\n \"int64\",\n \"int64\",\n \"int64\",\n \"int64\",\n \"int64\",\n \"int64\",\n \"int64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n \"float64\",\n ]\n dtypes = {columns_names[i]: columns_types[i] for i in range(len(columns_names))}\n\n df = pd.read_csv(\n filename,\n names=columns_names,\n dtype=dtypes,\n skiprows=1,\n )\n\n return df\n\n\ndef etl(df):\n keep_cols = [\n \"YEAR0\",\n \"DATANUM\",\n \"SERIAL\",\n \"CBSERIAL\",\n \"HHWT\",\n \"CPI99\",\n \"GQ\",\n \"PERNUM\",\n \"SEX\",\n \"AGE\",\n \"INCTOT\",\n \"EDUC\",\n \"EDUCD\",\n \"EDUC_HEAD\",\n \"EDUC_POP\",\n \"EDUC_MOM\",\n \"EDUCD_MOM2\",\n \"EDUCD_POP2\",\n \"INCTOT_MOM\",\n \"INCTOT_POP\",\n \"INCTOT_MOM2\",\n \"INCTOT_POP2\",\n \"INCTOT_HEAD\",\n \"SEX_HEAD\",\n ]\n df = df[keep_cols]\n\n df = df[df[\"INCTOT\"] != 9999999]\n df = df[df[\"EDUC\"] != -1]\n df = df[df[\"EDUCD\"] != -1]\n\n df[\"INCTOT\"] = df[\"INCTOT\"] * df[\"CPI99\"]\n\n for column in keep_cols:\n df[column] = df[column].fillna(-1)\n\n df[column] = df[column].astype(\"float64\")\n\n y = df[\"EDUC\"]\n X = df.drop(columns=[\"EDUC\", \"CPI99\"])\n\n return (df, X, y)\n\n\ndef mse(y_test, y_pred):\n return ((y_test - y_pred) ** 2).mean()\n\n\ndef cod(y_test, y_pred):\n y_bar = y_test.mean()\n total = ((y_test - y_bar) ** 2).sum()\n residuals = ((y_test - y_pred) ** 2).sum()\n return 1 - (residuals / total)\n\n\ndef ml(X, y, random_state, n_runs, test_size):\n clf = lm.Ridge()\n\n X = np.ascontiguousarray(X, dtype=np.float64)\n y = np.ascontiguousarray(y, dtype=np.float64)\n\n mse_values, cod_values = [], []\n ml_scores = {}\n\n print(\"ML runs: \", n_runs)\n for i in range(n_runs):\n (X_train, X_test, y_train, y_test) = train_test_split(\n X, y, test_size=test_size, random_state=random_state\n )\n random_state += 777\n\n with config_context(assume_finite=True):\n model = clf.fit(X_train, y_train)\n\n y_pred = model.predict(X_test)\n\n mse_values.append(mse(y_test, y_pred))\n cod_values.append(cod(y_test, y_pred))\n\n ml_scores[\"mse_mean\"] = sum(mse_values) / len(mse_values)\n ml_scores[\"cod_mean\"] = sum(cod_values) / len(cod_values)\n ml_scores[\"mse_dev\"] = pow(\n sum([(mse_value - ml_scores[\"mse_mean\"]) ** 2 for mse_value in mse_values])\n / (len(mse_values) - 1),\n 0.5,\n )\n ml_scores[\"cod_dev\"] = pow(\n sum([(cod_value - ml_scores[\"cod_mean\"]) ** 2 for cod_value in cod_values])\n / (len(cod_values) - 1),\n 0.5,\n )\n\n return ml_scores\n\n\ndef measure(name, func, *args, **kw):\n t0 = time.time()\n res = func(*args, **kw)\n t1 = time.time()\n print(f\"{name}: {t1 - t0} sec\")\n return res\n\n\ndef main():\n if len(sys.argv) != 2:\n print(\n f\"USAGE: docker run --rm -v /path/to/dataset:/dataset python census.py \"\n )\n return\n # ML specific\n N_RUNS = 50\n TEST_SIZE = 0.1\n RANDOM_STATE = 777\n\n df = measure(\"Reading\", read, sys.argv[1])\n _, X, y = measure(\"ETL\", etl, df)\n measure(\n \"ML\", ml, X, y, random_state=RANDOM_STATE, n_runs=N_RUNS, test_size=TEST_SIZE\n )\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "examples/docker/modin-ray/nyc-taxi.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport sys\nimport time\n\nimport modin.pandas as pd\n\n\ndef read(filename):\n columns_names = [\n \"trip_id\",\n \"vendor_id\",\n \"pickup_datetime\",\n \"dropoff_datetime\",\n \"store_and_fwd_flag\",\n \"rate_code_id\",\n \"pickup_longitude\",\n \"pickup_latitude\",\n \"dropoff_longitude\",\n \"dropoff_latitude\",\n \"passenger_count\",\n \"trip_distance\",\n \"fare_amount\",\n \"extra\",\n \"mta_tax\",\n \"tip_amount\",\n \"tolls_amount\",\n \"ehail_fee\",\n \"improvement_surcharge\",\n \"total_amount\",\n \"payment_type\",\n \"trip_type\",\n \"pickup\",\n \"dropoff\",\n \"cab_type\",\n \"precipitation\",\n \"snow_depth\",\n \"snowfall\",\n \"max_temperature\",\n \"min_temperature\",\n \"average_wind_speed\",\n \"pickup_nyct2010_gid\",\n \"pickup_ctlabel\",\n \"pickup_borocode\",\n \"pickup_boroname\",\n \"pickup_ct2010\",\n \"pickup_boroct2010\",\n \"pickup_cdeligibil\",\n \"pickup_ntacode\",\n \"pickup_ntaname\",\n \"pickup_puma\",\n \"dropoff_nyct2010_gid\",\n \"dropoff_ctlabel\",\n \"dropoff_borocode\",\n \"dropoff_boroname\",\n \"dropoff_ct2010\",\n \"dropoff_boroct2010\",\n \"dropoff_cdeligibil\",\n \"dropoff_ntacode\",\n \"dropoff_ntaname\",\n \"dropoff_puma\",\n ]\n parse_dates = [\"pickup_datetime\", \"dropoff_datetime\"]\n return pd.read_csv(\n filename, names=columns_names, header=None, parse_dates=parse_dates\n )\n\n\ndef q1(df):\n return df.groupby(\"cab_type\")[\"cab_type\"].count()\n\n\ndef q2(df):\n return df.groupby(\"passenger_count\", as_index=False).mean()[\n [\"passenger_count\", \"total_amount\"]\n ]\n\n\ndef q3(df):\n transformed = pd.DataFrame(\n {\n \"pickup_datetime\": df[\"pickup_datetime\"].dt.year,\n \"passenger_count\": df[\"passenger_count\"],\n }\n )\n return transformed.groupby(\n [\"pickup_datetime\", \"passenger_count\"], as_index=False\n ).size()\n\n\ndef q4(df):\n transformed = pd.DataFrame(\n {\n \"passenger_count\": df[\"passenger_count\"],\n \"pickup_datetime\": df[\"pickup_datetime\"].dt.year,\n \"trip_distance\": df[\"trip_distance\"].astype(\"int64\"),\n }\n )\n return (\n transformed.groupby(\n [\"passenger_count\", \"pickup_datetime\", \"trip_distance\"], as_index=False\n )\n .size()\n .sort_values(by=[\"pickup_datetime\", \"size\"], ascending=[True, False])\n )\n\n\ndef measure(name, func, *args, **kw):\n t0 = time.time()\n res = func(*args, **kw)\n t1 = time.time()\n print(f\"{name}: {t1 - t0} sec\")\n return res\n\n\ndef main():\n if len(sys.argv) != 2:\n print(\n f\"USAGE: docker run --rm -v /path/to/dataset:/dataset python nyc-taxi.py \"\n )\n return\n df = measure(\"Reading\", read, sys.argv[1])\n measure(\"Q1\", q1, df)\n measure(\"Q2\", q2, df)\n measure(\"Q3\", q3, df)\n measure(\"Q4\", q4, df)\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "examples/docker/modin-ray/plasticc.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport sys\nimport time\nfrom functools import partial\n\nimport numpy as np\nimport sklearnex\nimport xgboost as xgb\n\nimport modin.pandas as pd\n\nsklearnex.patch_sklearn()\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder\n\n\n################ helper functions ###############################\ndef create_dtypes():\n dtypes = dict(\n [\n (\"object_id\", \"int32\"),\n (\"mjd\", \"float32\"),\n (\"passband\", \"int32\"),\n (\"flux\", \"float32\"),\n (\"flux_err\", \"float32\"),\n (\"detected\", \"int32\"),\n ]\n )\n\n # load metadata\n columns_names = [\n \"object_id\",\n \"ra\",\n \"decl\",\n \"gal_l\",\n \"gal_b\",\n \"ddf\",\n \"hostgal_specz\",\n \"hostgal_photoz\",\n \"hostgal_photoz_err\",\n \"distmod\",\n \"mwebv\",\n \"target\",\n ]\n meta_dtypes = [\"int32\"] + [\"float32\"] * 4 + [\"int32\"] + [\"float32\"] * 5 + [\"int32\"]\n meta_dtypes = dict(\n [(columns_names[i], meta_dtypes[i]) for i in range(len(meta_dtypes))]\n )\n return dtypes, meta_dtypes\n\n\ndef ravel_column_names(cols):\n d0 = cols.get_level_values(0)\n d1 = cols.get_level_values(1)\n return [\"%s_%s\" % (i, j) for i, j in zip(d0, d1)]\n\n\ndef measure(name, func, *args, **kw):\n t0 = time.time()\n res = func(*args, **kw)\n t1 = time.time()\n print(f\"{name}: {t1 - t0} sec\")\n return res\n\n\ndef all_etl(train, train_meta, test, test_meta):\n train_final = etl(train, train_meta)\n test_final = etl(test, test_meta)\n return (train_final, test_final)\n\n\ndef split_step(train_final, test_final):\n X = train_final.drop([\"object_id\", \"target\"], axis=1).values\n Xt = test_final.drop([\"object_id\"], axis=1).values\n\n y = train_final[\"target\"]\n assert X.shape[1] == Xt.shape[1]\n classes = sorted(y.unique())\n\n class_weights = {c: 1 for c in classes}\n class_weights.update({c: 2 for c in [64, 15]})\n\n lbl = LabelEncoder()\n y = lbl.fit_transform(y)\n\n X_train, X_test, y_train, y_test = train_test_split(\n X, y, test_size=0.1, stratify=y, random_state=126\n )\n\n return X_train, y_train, X_test, y_test, Xt, classes, class_weights\n\n\ndef multi_weighted_logloss(y_true, y_preds, classes, class_weights):\n \"\"\"\n refactor from\n @author olivier https://www.kaggle.com/ogrellier\n multi logloss for PLAsTiCC challenge\n \"\"\"\n y_p = y_preds.reshape(y_true.shape[0], len(classes), order=\"F\")\n y_ohe = pd.get_dummies(y_true)\n y_p = np.clip(a=y_p, a_min=1e-15, a_max=1 - 1e-15)\n y_p_log = np.log(y_p)\n y_log_ones = np.sum(y_ohe.values * y_p_log, axis=0)\n nb_pos = y_ohe.sum(axis=0).values.astype(float)\n class_arr = np.array([class_weights[k] for k in sorted(class_weights.keys())])\n y_w = y_log_ones * class_arr / nb_pos\n\n loss = -np.sum(y_w) / np.sum(class_arr)\n return loss\n\n\ndef xgb_multi_weighted_logloss(y_predicted, y_true, classes, class_weights):\n loss = multi_weighted_logloss(\n y_true.get_label(), y_predicted, classes, class_weights\n )\n return \"wloss\", loss\n\n\n################ helper functions ###############################\n\n\ndef read(\n training_set_filename,\n test_set_filename,\n training_set_metadata_filename,\n test_set_metadata_filename,\n dtypes,\n meta_dtypes,\n):\n train = pd.read_csv(training_set_filename, dtype=dtypes)\n test = pd.read_csv(\n test_set_filename,\n names=list(dtypes.keys()),\n dtype=dtypes,\n header=0,\n )\n\n train_meta = pd.read_csv(training_set_metadata_filename, dtype=meta_dtypes)\n target = meta_dtypes.pop(\"target\")\n test_meta = pd.read_csv(test_set_metadata_filename, dtype=meta_dtypes)\n meta_dtypes[\"target\"] = target\n\n dfs = (train, train_meta, test, test_meta)\n return dfs\n\n\ndef etl(df, df_meta):\n # workaround for Modin_on_ray. Eventually this should be fixed\n df[\"flux_ratio_sq\"] = (df[\"flux\"] / df[\"flux_err\"]) * (\n df[\"flux\"] / df[\"flux_err\"]\n ) # np.power(df[\"flux\"] / df[\"flux_err\"], 2.0)\n df[\"flux_by_flux_ratio_sq\"] = df[\"flux\"] * df[\"flux_ratio_sq\"]\n\n aggs = {\n \"passband\": [\"mean\"],\n \"flux\": [\"min\", \"max\", \"mean\", \"skew\"],\n \"flux_err\": [\"min\", \"max\", \"mean\"],\n \"detected\": [\"mean\"],\n \"mjd\": [\"max\", \"min\"],\n \"flux_ratio_sq\": [\"sum\"],\n \"flux_by_flux_ratio_sq\": [\"sum\"],\n }\n agg_df = df.groupby(\"object_id\", sort=False).agg(aggs)\n\n agg_df.columns = ravel_column_names(agg_df.columns)\n\n agg_df[\"flux_diff\"] = agg_df[\"flux_max\"] - agg_df[\"flux_min\"]\n agg_df[\"flux_dif2\"] = agg_df[\"flux_diff\"] / agg_df[\"flux_mean\"]\n agg_df[\"flux_w_mean\"] = (\n agg_df[\"flux_by_flux_ratio_sq_sum\"] / agg_df[\"flux_ratio_sq_sum\"]\n )\n agg_df[\"flux_dif3\"] = agg_df[\"flux_diff\"] / agg_df[\"flux_w_mean\"]\n agg_df[\"mjd_diff\"] = agg_df[\"mjd_max\"] - agg_df[\"mjd_min\"]\n\n agg_df = agg_df.drop([\"mjd_max\", \"mjd_min\"], axis=1)\n\n agg_df = agg_df.reset_index()\n\n df_meta = df_meta.drop([\"ra\", \"decl\", \"gal_l\", \"gal_b\"], axis=1)\n\n df_meta = df_meta.merge(agg_df, on=\"object_id\", how=\"left\")\n\n return df_meta\n\n\ndef ml(train_final, test_final):\n X_train, y_train, X_test, y_test, Xt, classes, class_weights = split_step(\n train_final, test_final\n )\n\n cpu_params = {\n \"objective\": \"multi:softprob\",\n \"eval_metric\": \"merror\",\n \"tree_method\": \"hist\",\n \"nthread\": 16,\n \"num_class\": 14,\n \"max_depth\": 7,\n \"verbosity\": 1,\n \"subsample\": 0.7,\n \"colsample_bytree\": 0.7,\n }\n\n func_loss = partial(\n xgb_multi_weighted_logloss, classes=classes, class_weights=class_weights\n )\n\n dtrain = xgb.DMatrix(data=X_train, label=y_train)\n dvalid = xgb.DMatrix(data=X_test, label=y_test)\n dtest = xgb.DMatrix(data=Xt)\n\n watchlist = [(dvalid, \"eval\"), (dtrain, \"train\")]\n\n clf = xgb.train(\n cpu_params,\n dtrain=dtrain,\n num_boost_round=60,\n evals=watchlist,\n feval=func_loss,\n early_stopping_rounds=10,\n verbose_eval=None,\n )\n\n yp = clf.predict(dvalid)\n cpu_loss = multi_weighted_logloss(y_test, yp, classes, class_weights)\n ysub = clf.predict(dtest) # noqa: F841 (unused variable)\n\n return cpu_loss\n\n\ndef main():\n if len(sys.argv) != 5:\n print(\n f\"USAGE: docker run --rm -v /path/to/dataset:/dataset python plasticc.py \"\n )\n return\n\n dtypes, meta_dtypes = create_dtypes()\n\n train, train_meta, test, test_meta = measure(\n \"Reading\",\n read,\n sys.argv[1],\n sys.argv[2],\n sys.argv[3],\n sys.argv[4],\n dtypes,\n meta_dtypes,\n )\n train_final, test_final = measure(\n \"ETL\", all_etl, train, train_meta, test, test_meta\n )\n cpu_loss = measure(\"ML\", ml, train_final, test_final)\n\n print(\"validation cpu_loss:\", cpu_loss)\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "examples/jupyter/Modin_Taxi.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"cc4bd9e9\",\n \"metadata\": {\n \"slideshow\": {\n \"slide_type\": \"skip\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# To run this notebook as done in the README GIFs, you must first locally download the 2015 NYC Taxi Trip Data.\\n\",\n \"import urllib.request\\n\",\n \"url_path = \\\"https://modin-datasets.intel.com/green-taxi/green_tripdata_2015-01.csv\\\"\\n\",\n \"urllib.request.urlretrieve(url_path, \\\"taxi.csv\\\")\\n\",\n \"\\n\",\n \"from modin.config import Engine\\n\",\n \"Engine.put(\\\"dask\\\")\\n\",\n \"from dask.distributed import Client\\n\",\n \"client = Client(n_workers=12)\\n\",\n \"\\n\",\n \"from modin.config import BenchmarkMode\\n\",\n \"BenchmarkMode.put(True)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"id\": \"97b245e5\",\n \"metadata\": {\n \"slideshow\": {\n \"slide_type\": \"slide\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"id\": \"b65b121c\",\n \"metadata\": {\n \"slideshow\": {\n \"slide_type\": \"slide\"\n }\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"CPU times: user 1.57 s, sys: 683 ms, total: 2.26 s\\n\",\n \"Wall time: 14.2 s\\n\"\n ]\n }\n ],\n \"source\": [\n \"%time df = pd.read_csv(\\\"taxi.csv\\\", parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"id\": \"c48193b2\",\n \"metadata\": {\n \"slideshow\": {\n \"slide_type\": \"slide\"\n }\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"CPU times: user 138 ms, sys: 27.3 ms, total: 166 ms\\n\",\n \"Wall time: 404 ms\\n\"\n ]\n }\n ],\n \"source\": [\n \"%time isnull = df.isnull()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"id\": \"1d32ed7c\",\n \"metadata\": {\n \"slideshow\": {\n \"slide_type\": \"slide\"\n }\n },\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"CPU times: user 175 ms, sys: 28.4 ms, total: 203 ms\\n\",\n \"Wall time: 663 ms\\n\"\n ]\n }\n ],\n \"source\": [\n \"%time rounded_trip_distance = df[[\\\"pickup_longitude\\\"]].applymap(round)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"3ef271dc\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.11\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
},
{
"path": "examples/jupyter/Pandas_Taxi.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"id\": \"5d674ce8\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# To run this notebook as done in the README GIFs, you must first locally download the 2015 NYC Taxi Trip Data.\\n\",\n \"import urllib.request\\n\",\n \"url_path = \\\"https://modin-datasets.intel.com/green-taxi/green_tripdata_2015-01.csv\\\"\\n\",\n \"urllib.request.urlretrieve(url_path, \\\"taxi.csv\\\")\\n\",\n \"\\n\",\n \"import warnings\\n\",\n \"warnings.filterwarnings(\\\"ignore\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"id\": \"27f7321c\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pandas as pd\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"id\": \"8de98215\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"CPU times: user 30.7 s, sys: 4.25 s, total: 35 s\\n\",\n \"Wall time: 35.3 s\\n\"\n ]\n }\n ],\n \"source\": [\n \"%time df = pd.read_csv(\\\"taxi.csv\\\", parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"id\": \"14422c3f\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"CPU times: user 1.37 s, sys: 300 ms, total: 1.67 s\\n\",\n \"Wall time: 1.67 s\\n\"\n ]\n }\n ],\n \"source\": [\n \"%time isnull = df.isnull()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"id\": \"f8f87974\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"CPU times: user 3.07 s, sys: 305 ms, total: 3.37 s\\n\",\n \"Wall time: 3.37 s\\n\"\n ]\n }\n ],\n \"source\": [\n \"%time rounded_trip_distance = df[[\\\"pickup_longitude\\\"]].applymap(round)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"2c7d62bf\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.11\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
},
{
"path": "examples/jupyter/integrations/NLTK.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Demonstrating NLTK Modin Interoperability\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## All the examples in this section are taken / adapted from https://www.kirenz.com/post/2021-12-11-text-mining-and-sentiment-analysis-with-nltk-and-pandas-in-python/text-mining-and-sentiment-analysis-with-nltk-and-pandas-in-python/\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"import nltk\\n\",\n \"from nltk.tokenize import RegexpTokenizer\\n\",\n \"from nltk.corpus import stopwords\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Import some Tweets from Barack Obama \\n\",\n \"modin_df = pd.read_csv(\\\"https://raw.githubusercontent.com/kirenz/twitter-tweepy/main/tweets-obama.csv\\\")\\n\",\n \"modin_df.head(3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df['text'] = modin_df['text'].astype(str).str.lower()\\n\",\n \"modin_df.head(3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"regexp = RegexpTokenizer('\\\\w+')\\n\",\n \"\\n\",\n \"modin_df['text_token']=modin_df['text'].apply(regexp.tokenize)\\n\",\n \"modin_df.head(3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"nltk.download('stopwords')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Make a list of english stopwords\\n\",\n \"stopwords = nltk.corpus.stopwords.words(\\\"english\\\")\\n\",\n \"\\n\",\n \"# Extend the list with your own custom stopwords\\n\",\n \"my_stopwords = ['https']\\n\",\n \"stopwords.extend(my_stopwords)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Remove stopwords\\n\",\n \"modin_df['text_token'] = modin_df['text_token'].apply(lambda x: [item for item in x if item not in stopwords])\\n\",\n \"modin_df.head(3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df['text_string'] = modin_df['text_token'].apply(lambda x: ' '.join([item for item in x if len(item)>2]))\\n\",\n \"modin_df[['text', 'text_token', 'text_string']].head()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"nltk.download('punkt')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"all_words = ' '.join([word for word in modin_df['text_string']])\\n\",\n \"tokenized_words = nltk.tokenize.word_tokenize(all_words)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from nltk.probability import FreqDist\\n\",\n \"\\n\",\n \"fdist = FreqDist(tokenized_words)\\n\",\n \"fdist\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df['text_string_fdist'] = modin_df['text_token'].apply(lambda x: ' '.join([item for item in x if fdist[item] >= 1 ]))\\n\",\n \"modin_df[['text', 'text_token', 'text_string', 'text_string_fdist']].head()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"#lemmatization\\n\",\n \"nltk.download('wordnet')\\n\",\n \"nltk.download('omw-1.4')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from nltk.stem import WordNetLemmatizer\\n\",\n \"\\n\",\n \"wordnet_lem = WordNetLemmatizer()\\n\",\n \"\\n\",\n \"modin_df['text_string_lem'] = modin_df['text_string_fdist'].apply(wordnet_lem.lemmatize)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# check if the columns are equal\\n\",\n \"modin_df['is_equal']= (modin_df['text_string_fdist']==modin_df['text_string_lem'])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# show level count\\n\",\n \"modin_df.is_equal.value_counts()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"all_words_lem = ' '.join([word for word in modin_df['text_string_lem']])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%matplotlib inline\\n\",\n \"import matplotlib.pyplot as plt\\n\",\n \"from wordcloud import WordCloud\\n\",\n \"\\n\",\n \"wordcloud = WordCloud(width=600, \\n\",\n \" height=400, \\n\",\n \" random_state=2, \\n\",\n \" max_font_size=100).generate(all_words_lem)\\n\",\n \"\\n\",\n \"plt.figure(figsize=(10, 7))\\n\",\n \"plt.imshow(wordcloud, interpolation='bilinear')\\n\",\n \"plt.axis('off');\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Replicating NLTK workflow with pandas\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Import some Tweets from Barack Obama as pandas df\\n\",\n \"pandas_df = pandas.read_csv(\\\"https://raw.githubusercontent.com/kirenz/twitter-tweepy/main/tweets-obama.csv\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df['text'] = pandas_df['text'].astype(str).str.lower()\\n\",\n \"pandas_df.head(3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"regexp = RegexpTokenizer('\\\\w+')\\n\",\n \"\\n\",\n \"pandas_df['text_token']=pandas_df['text'].apply(regexp.tokenize)\\n\",\n \"pandas_df.head(3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Remove stopwords\\n\",\n \"pandas_df['text_token'] = pandas_df['text_token'].apply(lambda x: [item for item in x if item not in stopwords])\\n\",\n \"pandas_df.head(3)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df['text_string'] = pandas_df['text_token'].apply(lambda x: ' '.join([item for item in x if len(item)>2]))\\n\",\n \"pandas_df[['text', 'text_token', 'text_string']].head()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"all_words = ' '.join([word for word in pandas_df['text_string']])\\n\",\n \"tokenized_words = nltk.tokenize.word_tokenize(all_words)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from nltk.probability import FreqDist\\n\",\n \"\\n\",\n \"fdist = FreqDist(tokenized_words)\\n\",\n \"fdist\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df['text_string_fdist'] = pandas_df['text_token'].apply(lambda x: ' '.join([item for item in x if fdist[item] >= 1 ]))\\n\",\n \"pandas_df[['text', 'text_token', 'text_string', 'text_string_fdist']].head()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from nltk.stem import WordNetLemmatizer\\n\",\n \"\\n\",\n \"wordnet_lem = WordNetLemmatizer()\\n\",\n \"\\n\",\n \"pandas_df['text_string_lem'] = pandas_df['text_string_fdist'].apply(wordnet_lem.lemmatize)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# check if the columns are equal\\n\",\n \"pandas_df['is_equal']= (pandas_df['text_string_fdist']==pandas_df['text_string_lem'])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# show level count\\n\",\n \"pandas_df.is_equal.value_counts()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"all_words_lem = ' '.join([word for word in pandas_df['text_string_lem']])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%matplotlib inline\\n\",\n \"import matplotlib.pyplot as plt\\n\",\n \"from wordcloud import WordCloud\\n\",\n \"\\n\",\n \"wordcloud = WordCloud(width=600, \\n\",\n \" height=400, \\n\",\n \" random_state=2, \\n\",\n \" max_font_size=100).generate(all_words_lem)\\n\",\n \"\\n\",\n \"plt.figure(figsize=(10, 7))\\n\",\n \"plt.imshow(wordcloud, interpolation='bilinear')\\n\",\n \"plt.axis('off');\"\n ]\n }\n ],\n \"metadata\": {\n \"interpreter\": {\n \"hash\": \"9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f\"\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3.9.7 ('base')\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.7\"\n },\n \"orig_nbformat\": 4\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/jupyter/integrations/altair.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Demonstrating Altair Modin Interoperability\\n\",\n \"### Currently Altair is not interoperable with Modin. Each visualization is created with a Modin and then pandas dataframe for comparison.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import altair as alt\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from vega_datasets import data\\n\",\n \"pandas_cars = data.cars()\\n\",\n \"modin_cars = pd.DataFrame(data.cars())\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"alt.Chart(modin_cars).mark_point().encode(\\n\",\n \" x='Horsepower',\\n\",\n \" y='Miles_per_Gallon',\\n\",\n \" color='Origin',\\n\",\n \").interactive()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"alt.Chart(pandas_cars).mark_point().encode(\\n\",\n \" x='Horsepower',\\n\",\n \" y='Miles_per_Gallon',\\n\",\n \" color='Origin',\\n\",\n \").interactive()\"\n ]\n }\n ],\n \"metadata\": {\n \"interpreter\": {\n \"hash\": \"9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f\"\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3.9.7 ('base')\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.18\"\n },\n \"orig_nbformat\": 4\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/jupyter/integrations/bokeh.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Demonstrating Bokeh Modin Interoperability\\n\",\n \"### Currently Boken is not interoperable with Modin. Each visualization is created with a Modin and then pandas dataframe for comparison.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"from bokeh.plotting import figure\\n\",\n \"from bokeh.models import ColumnDataSource\\n\",\n \"from bokeh.io import show\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"UserWarning: `from_dict` is not currently supported by PandasOnRay, defaulting to pandas implementation.\\n\",\n \"Please refer to https://modin.readthedocs.io/en/stable/supported_apis/defaulting_to_pandas.html for explanation.\\n\",\n \"2023-04-06 12:14:58,510\\tINFO worker.py:1544 -- Started a local Ray instance. View the dashboard at \\u001b[1m\\u001b[32m127.0.0.1:8265 \\u001b[39m\\u001b[22m\\n\",\n \"UserWarning: When using a pre-initialized Ray cluster, please ensure that the runtime env sets environment variable __MODIN_AUTOIMPORT_PANDAS__ to 1\\n\"\n ]\n },\n {\n \"ename\": \"ValueError\",\n \"evalue\": \"expected a dict or pandas.DataFrame, got x_values y_values\\n0 1 6\\n1 2 7\\n2 3 2\\n3 4 3\\n4 5 6\",\n \"output_type\": \"error\",\n \"traceback\": [\n \"\\u001b[0;31m---------------------------------------------------------------------------\\u001b[0m\",\n \"\\u001b[0;31mValueError\\u001b[0m Traceback (most recent call last)\",\n \"\\u001b[0;32m/var/folders/qj/jybppsbd2jl75s8y2q8s2xx80000gn/T/ipykernel_5953/1336630338.py\\u001b[0m in \\u001b[0;36m\\u001b[0;34m\\u001b[0m\\n\\u001b[1;32m 3\\u001b[0m \\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m 4\\u001b[0m \\u001b[0;31m# create a ColumnDataSource by passing the dict\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m----> 5\\u001b[0;31m \\u001b[0msource\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0mColumnDataSource\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0mmodin_data\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[1;32m 6\\u001b[0m \\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m 7\\u001b[0m \\u001b[0mp\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0mfigure\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\",\n \"\\u001b[0;32m~/opt/anaconda3/lib/python3.9/site-packages/bokeh/models/sources.py\\u001b[0m in \\u001b[0;36m__init__\\u001b[0;34m(self, *args, **kwargs)\\u001b[0m\\n\\u001b[1;32m 229\\u001b[0m \\u001b[0mraw_data\\u001b[0m \\u001b[0;34m=\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0m_data_from_groupby\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0mraw_data\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m 230\\u001b[0m \\u001b[0;32melse\\u001b[0m\\u001b[0;34m:\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0;32m--> 231\\u001b[0;31m \\u001b[0;32mraise\\u001b[0m \\u001b[0mValueError\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0;34mf\\\"expected a dict or pandas.DataFrame, got {raw_data}\\\"\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[0m\\u001b[1;32m 232\\u001b[0m \\u001b[0msuper\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0m__init__\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0;34m**\\u001b[0m\\u001b[0mkwargs\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\\u001b[1;32m 233\\u001b[0m \\u001b[0mself\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mdata\\u001b[0m\\u001b[0;34m.\\u001b[0m\\u001b[0mupdate\\u001b[0m\\u001b[0;34m(\\u001b[0m\\u001b[0mraw_data\\u001b[0m\\u001b[0;34m)\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0;34m\\u001b[0m\\u001b[0m\\n\",\n \"\\u001b[0;31mValueError\\u001b[0m: expected a dict or pandas.DataFrame, got x_values y_values\\n0 1 6\\n1 2 7\\n2 3 2\\n3 4 3\\n4 5 6\"\n ]\n }\n ],\n \"source\": [\n \"# Create a visualization with Modin df \\n\",\n \"modin_data = pd.DataFrame.from_dict({'x_values': [1, 2, 3, 4, 5], 'y_values': [6, 7, 2, 3, 6]})\\n\",\n \"\\n\",\n \"# create a ColumnDataSource by passing the dict\\n\",\n \"source = ColumnDataSource(modin_data)\\n\",\n \"\\n\",\n \"p = figure()\\n\",\n \"p.circle(x='x_values', y='y_values', source=source)\\n\",\n \"show(p)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df \\n\",\n \"pandas_data = pandas.DataFrame.from_dict({'x_values': [1, 2, 3, 4, 5], 'y_values': [6, 7, 2, 3, 6]})\\n\",\n \"\\n\",\n \"# create a ColumnDataSource by passing the dict\\n\",\n \"source = ColumnDataSource(pandas_data)\\n\",\n \"\\n\",\n \"p = figure()\\n\",\n \"p.circle(x='x_values', y='y_values', source=source)\\n\",\n \"show(p)\"\n ]\n }\n ],\n \"metadata\": {\n \"interpreter\": {\n \"hash\": \"9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f\"\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3.9.7 ('base')\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.7\"\n },\n \"orig_nbformat\": 4\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/jupyter/integrations/huggingface.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Demonstrating Hugging Face Modin Interoperability\\n\",\n \"## All the examples in this section are taken/ adapted from https://www.kaggle.com/code/satyampd/imdb-sentiment-analysis-using-bert-w-huggingface/notebook\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import numpy as np # linear algebra\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import tensorflow as tf\\n\",\n \"import sklearn\\n\",\n \"from tqdm import tqdm\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import urllib.request\\n\",\n \"url_path = \\\"https://modin-datasets.intel.com/testing/IMDB_Dataset.csv\\\"\\n\",\n \"urllib.request.urlretrieve(url_path, \\\"imdb.csv\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%%time\\n\",\n \"modin_df = pd.read_csv(\\\"imdb.csv\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df.head()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"type(modin_df)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df.sample()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from transformers import BertTokenizer, TFBertForSequenceClassification\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Loading the BERT Classifier and Tokenizer along with Input module\\n\",\n \"from transformers import InputExample, InputFeatures\\n\",\n \"\\n\",\n \"model = TFBertForSequenceClassification.from_pretrained(\\\"bert-base-uncased\\\")\\n\",\n \"tokenizer = BertTokenizer.from_pretrained(\\\"bert-base-uncased\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"model.summary()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# changing positive and negative into numeric values\\n\",\n \"\\n\",\n \"def cat2num(value):\\n\",\n \" if value=='positive': \\n\",\n \" return 1\\n\",\n \" else: \\n\",\n \" return 0\\n\",\n \" \\n\",\n \"modin_df['sentiment'] = modin_df['sentiment'].apply(cat2num)\\n\",\n \"train = modin_df[:45000]\\n\",\n \"test = modin_df[45000:]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# But first see BERT tokenizer exmaples and other required stuff!\\n\",\n \"\\n\",\n \"example='In this Kaggle notebook, I will do sentiment analysis using BERT with Huggingface'\\n\",\n \"tokens=tokenizer.tokenize(example)\\n\",\n \"token_ids = tokenizer.convert_tokens_to_ids(tokens)\\n\",\n \"print(tokens)\\n\",\n \"print(token_ids)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"type(train)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def convert_data_to_examples(train, test, review, sentiment): \\n\",\n \" train_InputExamples = train.apply(lambda x: InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this case\\n\",\n \" text_a = x[review], \\n\",\n \" label = x[sentiment]), axis = 1)\\n\",\n \"\\n\",\n \" validation_InputExamples = test.apply(lambda x: InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this case\\n\",\n \" text_a = x[review], \\n\",\n \" label = x[sentiment]), axis = 1,)\\n\",\n \" \\n\",\n \" return train_InputExamples, validation_InputExamples\\n\",\n \"\\n\",\n \"train_InputExamples, validation_InputExamples = convert_data_to_examples(train, test, 'review', 'sentiment')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def convert_examples_to_tf_dataset(examples, tokenizer, max_length=128):\\n\",\n \" features = [] # -> will hold InputFeatures to be converted later\\n\",\n \"\\n\",\n \" for e in tqdm(examples):\\n\",\n \" input_dict = tokenizer.encode_plus(\\n\",\n \" e.text_a,\\n\",\n \" add_special_tokens=True, # Add 'CLS' and 'SEP'\\n\",\n \" max_length=max_length, # truncates if len(s) > max_length\\n\",\n \" return_token_type_ids=True,\\n\",\n \" return_attention_mask=True,\\n\",\n \" pad_to_max_length=True, # pads to the right by default # CHECK THIS for pad_to_max_length\\n\",\n \" truncation=True\\n\",\n \" )\\n\",\n \"\\n\",\n \" input_ids, token_type_ids, attention_mask = (input_dict[\\\"input_ids\\\"],input_dict[\\\"token_type_ids\\\"], input_dict['attention_mask'])\\n\",\n \" features.append(InputFeatures( input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, label=e.label) )\\n\",\n \"\\n\",\n \" def gen():\\n\",\n \" for f in features:\\n\",\n \" yield (\\n\",\n \" {\\n\",\n \" \\\"input_ids\\\": f.input_ids,\\n\",\n \" \\\"attention_mask\\\": f.attention_mask,\\n\",\n \" \\\"token_type_ids\\\": f.token_type_ids,\\n\",\n \" },\\n\",\n \" f.label,\\n\",\n \" )\\n\",\n \"\\n\",\n \" return tf.data.Dataset.from_generator(\\n\",\n \" gen,\\n\",\n \" ({\\\"input_ids\\\": tf.int32, \\\"attention_mask\\\": tf.int32, \\\"token_type_ids\\\": tf.int32}, tf.int64),\\n\",\n \" (\\n\",\n \" {\\n\",\n \" \\\"input_ids\\\": tf.TensorShape([None]),\\n\",\n \" \\\"attention_mask\\\": tf.TensorShape([None]),\\n\",\n \" \\\"token_type_ids\\\": tf.TensorShape([None]),\\n\",\n \" },\\n\",\n \" tf.TensorShape([]),\\n\",\n \" ),\\n\",\n \" )\\n\",\n \"\\n\",\n \"\\n\",\n \"DATA_COLUMN = 'review'\\n\",\n \"LABEL_COLUMN = 'sentiment'\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"train_InputExamples\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"train_data = convert_examples_to_tf_dataset(list(train_InputExamples), tokenizer)\\n\",\n \"train_data = train_data.shuffle(100).batch(32).repeat(2)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"validation_data = convert_examples_to_tf_dataset(list(validation_InputExamples), tokenizer)\\n\",\n \"validation_data = validation_data.batch(32)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0), \\n\",\n \" loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), \\n\",\n \" metrics=[tf.keras.metrics.SparseCategoricalAccuracy('accuracy')])\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"model.fit(train_data, epochs=2, validation_data=validation_data)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pred_sentences = ['worst movie of my life, will never watch movies from this series', \\n\",\n \" 'Wow, blew my mind, what a movie by Marvel, animation and story is amazing']\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"tf_batch = tokenizer(pred_sentences, max_length=128, padding=True, truncation=True, return_tensors='tf') # we are tokenizing before sending into our trained model\\n\",\n \"tf_outputs = model(tf_batch) \\n\",\n \"tf_predictions = tf.nn.softmax(tf_outputs[0], axis=-1) # axis=-1, this means that the index that will be returned by argmax will be taken from the *last* axis.\\n\",\n \"labels = ['Negative','Positive']\\n\",\n \"label = tf.argmax(tf_predictions, axis=1)\\n\",\n \"label = label.numpy()\\n\",\n \"for i in range(len(pred_sentences)):\\n\",\n \" print(pred_sentences[i], \\\": \\\", labels[label[i]])\"\n ]\n }\n ],\n \"metadata\": {\n \"interpreter\": {\n \"hash\": \"9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f\"\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3.9.7 ('base')\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.7\"\n },\n \"orig_nbformat\": 4\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/jupyter/integrations/matplotlib.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Demonstrating Matplotlib Modin Interoperability\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"import numpy as np\\n\",\n \"import matplotlib.pyplot as plt\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"# Example modified from https://matplotlib.org/3.1.1/gallery/lines_bars_and_markers/xcorr_acorr_demo.html#sphx-glr-gallery-lines-bars-and-markers-xcorr-acorr-demo-py\\n\",\n \"\\n\",\n \"# Fixing random state for reproducibility\\n\",\n \"np.random.seed(19680801)\\n\",\n \"\\n\",\n \"x = pd.DataFrame(np.random.randn(100, 1),columns=[\\\"Col_1\\\"])\\n\",\n \"y = pd.DataFrame(np.random.randn(100, 1),columns=[\\\"Col_1\\\"])\\n\",\n \"\\n\",\n \"fig, [ax1, ax2] = plt.subplots(2, 1, sharex=True)\\n\",\n \"ax1.xcorr(x[\\\"Col_1\\\"], y[\\\"Col_1\\\"], usevlines=True, maxlags=50, normed=True, lw=2)\\n\",\n \"ax1.grid(True)\\n\",\n \"\\n\",\n \"ax2.acorr(x[\\\"Col_1\\\"], usevlines=True, normed=True, maxlags=50, lw=2)\\n\",\n \"ax2.grid(True)\\n\",\n \"\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"# Example modified from https://matplotlib.org/3.1.1/gallery/lines_bars_and_markers/xcorr_acorr_demo.html#sphx-glr-gallery-lines-bars-and-markers-xcorr-acorr-demo-py\\n\",\n \"\\n\",\n \"# Fixing random state for reproducibility\\n\",\n \"np.random.seed(19680801)\\n\",\n \"\\n\",\n \"x = pandas.DataFrame(np.random.randn(100, 1),columns=[\\\"Col_1\\\"])\\n\",\n \"y = pandas.DataFrame(np.random.randn(100, 1),columns=[\\\"Col_1\\\"])\\n\",\n \"\\n\",\n \"fig, [ax1, ax2] = plt.subplots(2, 1, sharex=True)\\n\",\n \"ax1.xcorr(x[\\\"Col_1\\\"], y[\\\"Col_1\\\"], usevlines=True, maxlags=50, normed=True, lw=2)\\n\",\n \"ax1.grid(True)\\n\",\n \"\\n\",\n \"ax2.acorr(x[\\\"Col_1\\\"], usevlines=True, normed=True, maxlags=50, lw=2)\\n\",\n \"ax2.grid(True)\\n\",\n \"\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\\n\",\n \"\\n\",\n \"names = ['group_a', 'group_b', 'group_c']\\n\",\n \"values = [1, 10, 100]\\n\",\n \"\\n\",\n \"modin_df = pd.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\\n\",\n \"\\n\",\n \"plt.figure(figsize=(9, 3))\\n\",\n \"\\n\",\n \"plt.subplot(131)\\n\",\n \"plt.bar(modin_df['names'], modin_df['values'])\\n\",\n \"plt.subplot(132)\\n\",\n \"#plt.scatter(df['names'], df['values'])\\n\",\n \"#plt.subplot(133)\\n\",\n \"plt.plot(modin_df['names'], modin_df['values'])\\n\",\n \"plt.suptitle('Categorical Plotting')\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\\n\",\n \"\\n\",\n \"names = ['group_a', 'group_b', 'group_c']\\n\",\n \"values = [1, 10, 100]\\n\",\n \"\\n\",\n \"pandas_df = pandas.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\\n\",\n \"\\n\",\n \"plt.figure(figsize=(9, 3))\\n\",\n \"\\n\",\n \"plt.subplot(131)\\n\",\n \"plt.bar(pandas_df['names'], pandas_df['values'])\\n\",\n \"plt.subplot(132)\\n\",\n \"\\n\",\n \"plt.plot(pandas_df['names'], pandas_df['values'])\\n\",\n \"plt.suptitle('Categorical Plotting')\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\\n\",\n \"\\n\",\n \"names = ['group_a', 'group_b', 'group_c']\\n\",\n \"values = [1, 10, 100]\\n\",\n \"\\n\",\n \"modin_df = pd.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\\n\",\n \"\\n\",\n \"plt.figure(figsize=(9, 3))\\n\",\n \"\\n\",\n \"plt.subplot(131)\\n\",\n \"plt.barh(modin_df['names'], modin_df['values'])\\n\",\n \"plt.subplot(132)\\n\",\n \"#plt.scatter(df['names'], df['values'])\\n\",\n \"#plt.subplot(133)\\n\",\n \"plt.plot(modin_df['names'], modin_df['values'])\\n\",\n \"plt.suptitle('Categorical Plotting')\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\\n\",\n \"\\n\",\n \"names = ['group_a', 'group_b', 'group_c']\\n\",\n \"values = [1, 10, 100]\\n\",\n \"\\n\",\n \"pandas_df = pandas.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\\n\",\n \"\\n\",\n \"plt.figure(figsize=(9, 3))\\n\",\n \"\\n\",\n \"plt.subplot(131)\\n\",\n \"plt.barh(pandas_df['names'], pandas_df['values'])\\n\",\n \"plt.subplot(132)\\n\",\n \"plt.plot(pandas_df['names'], pandas_df['values'])\\n\",\n \"plt.suptitle('Categorical Plotting')\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"plt.figure(figsize=(9, 3))\\n\",\n \"\\n\",\n \"plt.subplot(131)\\n\",\n \"plt.hlines(pandas_df['values'], 1, 3)\\n\",\n \"plt.suptitle('Categorical Plotting')\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\\n\",\n \"\\n\",\n \"names = ['group_a', 'group_b', 'group_c']\\n\",\n \"values = [1, 10, 100]\\n\",\n \"\\n\",\n \"modin_df = pd.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\\n\",\n \"\\n\",\n \"plt.figure(figsize=(9, 3))\\n\",\n \"\\n\",\n \"plt.subplot(131)\\n\",\n \"plt.bar(modin_df['names'], modin_df['values'])\\n\",\n \"plt.subplot(132)\\n\",\n \"#plt.scatter(df['names'], df['values'])\\n\",\n \"#plt.subplot(133)\\n\",\n \"plt.plot(modin_df['names'], modin_df['values'])\\n\",\n \"plt.suptitle('Categorical Plotting')\\n\",\n \"plt.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\\n\",\n \"\\n\",\n \"names = ['group_a', 'group_b', 'group_c']\\n\",\n \"values = [1, 10, 100]\\n\",\n \"\\n\",\n \"pandas_df = pandas.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\\n\",\n \"\\n\",\n \"plt.figure(figsize=(9, 3))\\n\",\n \"\\n\",\n \"plt.subplot(131)\\n\",\n \"plt.bar(pandas_df['names'], pandas_df['values'])\\n\",\n \"plt.subplot(132)\\n\",\n \"#plt.scatter(df['names'], df['values'])\\n\",\n \"#plt.subplot(133)\\n\",\n \"plt.plot(pandas_df['names'], pandas_df['values'])\\n\",\n \"plt.suptitle('Categorical Plotting')\\n\",\n \"plt.show()\"\n ]\n }\n ],\n \"metadata\": {\n \"interpreter\": {\n \"hash\": \"aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49\"\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3.9.10 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.7\"\n },\n \"orig_nbformat\": 4\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/jupyter/integrations/plotly.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Demonstrating Plotly Modin Interoperability\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Currently Plotly is not completely interoperable with Modin. Each visualization is created with a Modin and then pandas dataframe for comparison.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"import numpy as np\\n\",\n \"import plotly.express as px\\n\",\n \"import plotly.io as pio\\n\",\n \"pio.renderers.default = \\\"notebook\\\"\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df = pd.DataFrame(dict(a=[1,3,2,4], b=[3,2,1,0]))\\n\",\n \"pandas_df = pandas.DataFrame(dict(a=[1,3,2,4], b=[3,2,1,0]))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"fig2 = px.bar(modin_df)\\n\",\n \"fig2.show()\\n\",\n \"# py.iplot(fig2 , filename='jupyter-basic_bar')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"fig2 = px.bar(pandas_df)\\n\",\n \"fig2.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"fig = px.line(modin_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"fig = px.line(pandas_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"fig = px.area(modin_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"fig = px.area(pandas_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"fig = px.area(modin_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"fig = px.area(pandas_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"fig = px.violin(modin_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"fig = px.violin(pandas_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"fig = px.box(modin_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"fig = px.box(pandas_df)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"fig = px.histogram(modin_df, opacity=0.5, orientation='h', nbins=5)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"fig = px.histogram(pandas_df, opacity=0.5, orientation='h', nbins=5)\\n\",\n \"fig.show()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with Modin df\\n\",\n \"# Example from https://plotly.com/python/mapbox-county-choropleth/#choropleth-map-using-plotlyexpress-and-carto-base-map-no-token-needed\\n\",\n \"from urllib.request import urlopen\\n\",\n \"import json\\n\",\n \"with urlopen('https://raw.githubusercontent.com/plotly/datasets/master/geojson-counties-fips.json') as response:\\n\",\n \" counties = json.load(response)\\n\",\n \"import modin.pandas as pd\\n\",\n \"modin_df = pd.read_csv(\\\"https://raw.githubusercontent.com/plotly/datasets/master/fips-unemp-16.csv\\\",\\n\",\n \" dtype={\\\"fips\\\": str})\\n\",\n \"fig = px.choropleth(modin_df, geojson=counties, locations='fips', color='unemp',\\n\",\n \" color_continuous_scale=\\\"Viridis\\\",\\n\",\n \" range_color=(0, 12),\\n\",\n \" scope=\\\"usa\\\",\\n\",\n \" labels={'unemp':'unemployment rate'}\\n\",\n \" )\\n\",\n \"fig.update_layout(margin={\\\"r\\\":0,\\\"t\\\":0,\\\"l\\\":0,\\\"b\\\":0})\\n\",\n \"fig.show()\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Create a visualization with pandas df\\n\",\n \"# Example from https://plotly.com/python/mapbox-county-choropleth/#choropleth-map-using-plotlyexpress-and-carto-base-map-no-token-needed\\n\",\n \"from urllib.request import urlopen\\n\",\n \"import json\\n\",\n \"with urlopen('https://raw.githubusercontent.com/plotly/datasets/master/geojson-counties-fips.json') as response:\\n\",\n \" counties = json.load(response)\\n\",\n \"import pandas\\n\",\n \"pandas_df = pandas.read_csv(\\\"https://raw.githubusercontent.com/plotly/datasets/master/fips-unemp-16.csv\\\",\\n\",\n \" dtype={\\\"fips\\\": str})\\n\",\n \"\\n\",\n \"fig = px.choropleth(pandas_df, geojson=counties, locations='fips', color='unemp',\\n\",\n \" color_continuous_scale=\\\"Viridis\\\",\\n\",\n \" range_color=(0, 12),\\n\",\n \" scope=\\\"usa\\\",\\n\",\n \" labels={'unemp':'unemployment rate'}\\n\",\n \" )\\n\",\n \"fig.update_layout(margin={\\\"r\\\":0,\\\"t\\\":0,\\\"l\\\":0,\\\"b\\\":0})\\n\",\n \"fig.show()\\n\"\n ]\n }\n ],\n \"metadata\": {\n \"interpreter\": {\n \"hash\": \"9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f\"\n },\n \"kernelspec\": {\n \"display_name\": \"Python 3.9.7 ('base')\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.7\"\n },\n \"orig_nbformat\": 4\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/jupyter/integrations/seaborn.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Demonstrating Seaborn Modin Interoperability\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### All the examples in this section are taken / adapted from https://seaborn.pydata.org/tutorial/introduction.html. Each visualization is created with a Modin and then pandas dataframe for comparison.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 26,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import seaborn as sns\\n\",\n \"import modin.pandas as pd\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 27,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"UserWarning: Distributing object. This may take some time.\\n\"\n ]\n },\n {\n \"data\": {\n \"text/plain\": [\n \"modin.pandas.dataframe.DataFrame\"\n ]\n },\n \"execution_count\": 27,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"# Apply the default theme\\n\",\n \"sns.set_theme()\\n\",\n \"\\n\",\n \"# Load an example dataset\\n\",\n \"pandas_tips = sns.load_dataset(\\\"tips\\\")\\n\",\n \"modin_tips = pd.DataFrame(pandas_tips)\\n\",\n \"\\n\",\n \"type(modin_tips)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 28,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/html\": [\n \"
\\n\",\n \"\\n\",\n \"
\\n\",\n \" \\n\",\n \"
\\n\",\n \"
\\n\",\n \"
total_bill
\\n\",\n \"
tip
\\n\",\n \"
sex
\\n\",\n \"
smoker
\\n\",\n \"
day
\\n\",\n \"
time
\\n\",\n \"
size
\\n\",\n \"
\\n\",\n \" \\n\",\n \" \\n\",\n \"
\\n\",\n \"
0
\\n\",\n \"
16.99
\\n\",\n \"
1.01
\\n\",\n \"
Female
\\n\",\n \"
No
\\n\",\n \"
Sun
\\n\",\n \"
Dinner
\\n\",\n \"
2
\\n\",\n \"
\\n\",\n \"
\\n\",\n \"
1
\\n\",\n \"
10.34
\\n\",\n \"
1.66
\\n\",\n \"
Male
\\n\",\n \"
No
\\n\",\n \"
Sun
\\n\",\n \"
Dinner
\\n\",\n \"
3
\\n\",\n \"
\\n\",\n \"
\\n\",\n \"
2
\\n\",\n \"
21.01
\\n\",\n \"
3.50
\\n\",\n \"
Male
\\n\",\n \"
No
\\n\",\n \"
Sun
\\n\",\n \"
Dinner
\\n\",\n \"
3
\\n\",\n \"
\\n\",\n \"
\\n\",\n \"
3
\\n\",\n \"
23.68
\\n\",\n \"
3.31
\\n\",\n \"
Male
\\n\",\n \"
No
\\n\",\n \"
Sun
\\n\",\n \"
Dinner
\\n\",\n \"
2
\\n\",\n \"
\\n\",\n \"
\\n\",\n \"
4
\\n\",\n \"
24.59
\\n\",\n \"
3.61
\\n\",\n \"
Female
\\n\",\n \"
No
\\n\",\n \"
Sun
\\n\",\n \"
Dinner
\\n\",\n \"
4
\\n\",\n \"
\\n\",\n \" \\n\",\n \"
\\n\",\n \"
\"\n ],\n \"text/plain\": [\n \" total_bill tip sex smoker day time size\\n\",\n \"0 16.99 1.01 Female No Sun Dinner 2\\n\",\n \"1 10.34 1.66 Male No Sun Dinner 3\\n\",\n \"2 21.01 3.50 Male No Sun Dinner 3\\n\",\n \"3 23.68 3.31 Male No Sun Dinner 2\\n\",\n \"4 24.59 3.61 Female No Sun Dinner 4\"\n ]\n },\n \"execution_count\": 28,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"modin_tips.head()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 29,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"\"\n ]\n },\n \"execution_count\": 29,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n },\n {\n \"data\": {\n \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAwEAAAFcCAYAAACQkLIVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAACZvElEQVR4nOzdZ4BU1dnA8f+902dntvdCWXrvKEXBggXEAhp7iRo1+qoxMUaNpvhGo4mvRGOKJiZGYxcQsKNYqErvvSxsYXubXu59PywsLLsLy/bdeX5fYO7MnHvO7O6Z+9xzznMUXdd1hBBCCCGEEBFD7egKCCGEEEIIIdqXBAFCCCGEEEJEGAkChBBCCCGEiDASBAghhBBCCBFhJAgQQgghhBAiwkgQIIQQQgghRISRIEB0GbfeeitlZWUA/OhHP2LPnj0dWp+HH36YV155pc3Pc+ONN/Lpp5+2+XmEEN1TZ+w7zzrrLC677DIuu+wypk+fzq9+9SuKi4sBKCws5JprrunQOgoRCYwdXQEhmmr58uW1///HP/7RgTURQoiuozP2nbfccgu33XYbALqu89JLL3H77bczb948UlJSePvttzu4hkJ0fxIEiC7hkUceAeDmm2/m5Zdf5vrrr+f555/H4/Hw3HPPkZaWxv79+7HZbNxxxx28/vrr7N+/nwsuuIBHH30UgCVLlvC3v/2NYDCI1WrlF7/4BaNGjapznj179vCzn/2s3vlvuukmZs+e3aS65ubmMnPmTNavX1/v8bx581i8eDGqqpKTk4PVauWZZ56hT58+FBcX8+tf/5p9+/ahqirXXHMNN910EwBffvklr7zyCiUlJUyYMIHf/e53qKoM5AkhTq4r9J2KonDXXXcxf/58li9fTnZ2dm2f+ec//5m8vDyKi4vJy8sjJSWFP/7xjyQnJ3PuuedyxRVXsHLlSgoKCrjsssv4yU9+ctI6//nPf2bDhg0UFRUxYMAAnn322Vb4lIXoonQhuoj+/fvrpaWluq7r+jnnnKNv2rRJX7VqlT5o0CB969atuq7r+m233aZfffXVut/v10tLS/UhQ4bohw8f1vfv369fcsklellZma7rur5r1y590qRJutvtbnZ9fvGLX+j//Oc/6x0/dOiQPnLkyAYfz507Vx8zZoxeUFCg67quP/HEE/pDDz2k67qu33PPPfozzzyj67quV1VV6TNmzNAPHDig33DDDfqPf/xjPRQK6R6PR580aZK+evXqZtdbCBFZukrfee+99+r/+Mc/6vSZL7zwgn7eeefp1dXVuq7r+p133qk///zztW15+umndV3X9cOHD+vDhg3TDx48eNI6v/DCC/qFF16oB4PBZtdfiO5CRgJEl5eZmcngwYMB6NGjB06nE7PZTHx8PFFRUVRWVrJ69WqKioq45ZZbat+nKAoHDx5k4MCBtcdaYyTgVIYMGUJqaioAgwcPZvHixQCsWLGCn//85wA4nU4+/PDD2vdMnz4dg8GAzWajV69elJaWtkpdhBCRq7P1nYqiYLPZ6h0fP348DocDqOkzKysra58777zzAEhJSSEhIYHKyko2btzYaJ0BRo4cidEolz9CyF+B6PLMZnOdxw117pqmMWHCBP70pz/VHisoKCA5ObnO6/r27cuCBQtaVB9FUdB1vfZxMBis87zVam3wtUajEUVRap87dOgQcXFxtc81Vr4QQjRHZ+o7dV1n69at3HDDDfWea6zPBLBYLPWeO1mdFy9ejN1ub3Y9hehOZFKx6DIMBgOhUKhZ750wYQLLly9n7969AHzzzTdceuml+Hy+1qwiANHR0QSDwdoMHB999FGT6zh37lwAqqurufnmmzlw4ECr108IEVk6e98ZDof5y1/+QlxcHOPGjWtxee3Z3wvRlclIgOgyLrroIm688Ub+/Oc/n/Z7+/btyxNPPMFPf/pTdF3HaDTyt7/9jaioqBbVac6cObz44ou1j8855xyee+45fv7zn/OjH/2I+Ph4LrrooiaV9atf/Yrf/OY3zJw5E13XufPOOxk6dGiL6ieEEJ2x73z11VdZuHAhiqIQDocZNmwYL7/8covKbOs6C9HdKLrMKxBCCCGEECKiyHQgIYQQQgghIowEAUIIIYQQQkQYCQKEEEIIIYSIMBIECCGEEEIIEWEkCBBCCCGEECLCdLkUoaWlLjSt+yY0iouzU17u6ehqtLtIbHckthmk3a0hKcnZrPd19/4TIvP3KxLbDJHZ7khsM7R+u5vbh3Y3MhLQyRiNho6uQoeIxHZHYptB2i3aViR+zpHYZojMdkdimyFy293WJAgQQgghhBAiwkgQIIQQQgghRISRIEAIIYQQQogII0GAEEIIIYQQEUaCACGEEEIIISKMBAFCCCGEEEJEGAkChBBCCCGEiDBtGgS4XC4uueQScnNzAVixYgUzZ87kggsuYM6cOW15aiGEEEII0Y0oCpg1d+1jix55G6e1pjYLAjZu3Mi1117LgQMHAPD5fDz66KP89a9/5eOPP2bLli188803bXV6IYQQQgjRTSgKmMr2UP3Rc1j8pVjceVR+8Hss7vyOrlqX1WZBwLvvvsuvf/1rkpOTAdi0aRM9e/YkKysLo9HIzJkz+fTTT9vq9EIIIYQQoptQtRDBgl0ECvZSsfBZyuY9Q7D4EFp5PorS0bXrmoxtVfCTTz5Z53FRURFJSUm1j5OTkyksLGyr0wshhBBCiG4irBgxDjqfKFc57g2LAYg55ybCmaPQ9Q6uXBfVZkHAiTRNQzkuVNN1vc7jpkpIcLRmtTqlpCRnR1ehQ0RiuyOxzSDt7iiR0H9Cx3/OHSES2wyR2e5IbDPUtNtfsJfKXd/XHvNu+ZrkAWMxJ2R0YM26rnYLAlJTUykuLq59XFxcXDtV6HSUlrrQtO4b8iUlOSkuru7oarS7SGx3JLYZpN2tVVZzdPf+EyLz9ysS2wyR2e5IbDPUtLu8pJzgxq/QPJXETrudcHUZ1avm4d6zkSo9+rRGAyI1kDpRuwUBI0aMYP/+/eTk5JCZmcmHH37I7Nmz2+v0QgghhBCiiwrpRswjZpDUZwyB2N4Y9TBJWQMJxfSU6UDN1G5BgMVi4emnn+bee+/F7/czZcoULrroovY6vRBCCCGE6MKCRgfB2L4AaIqBUFz/Dq5R19bmQcCSJUtq/z9hwgQWLlzY1qcUQgghhBBCnITsGCyEEEIIIUSEkSBACCGEEEKICCNBgBBCCCGEEBFGggAhhBBCCCEijAQBQgghhBBCRBgJAoQQQgghhIgwEgQIIYQQQggRYSQIEEIIIYQQIsJIECCEEEIIIUSEkSBACCGEEEKICCNBgBBCCCGEEBFGggAhhBBCCCEijAQBQgghhBBCRBgJAoQQQgghhIgwEgQIIYQQQggRYSQIEEIIIYQQIsJIECCEEEIIIUSEkSBACCGEEEKICCNBgBBCCCGEEBFGggAhhBBCCNGuFKWjayCMHV0BIYQQQggRGTyBMPsLqjlwuIqEaCv9s2KJd5g7uloRSYIAIYQQQgjR5jz+MC8v2sqmPSW1x6KsRn55y3hSY60dWLPIJNOBhBBCCCFEm9uTX1knAABw+0J8uHw/egfVKZJJECCEEEIIIdqUoijsOlTR4HOrtxfiDYTbt0JCggAhhBBCCNG2dF0nKdbW4HNJsTZMRrkkbW/yiQshhBBCiDY3uFc8FpOh3vEfnNcPkyrpgtqbBAFCCCGEEKLNJcdYeOyH4xjeLxGApFgr9101gsE9Yzu2YhFKsgMJIYQQQog2p+uQEW/nvlnD8QTCmI0qZoOMAHQUCQKEEEIIIUS7URVwWOpPCxLtS6YDCSGEEEIIEWEkCBBCCCGEECLCSBAghBBCCCFEhJEgQAghhBBCiAgjQYAQQgghhBARRoIAIYQQQgghIowEAUIIIYQQQkQYCQKEEEIIIYSIMBIECCGEEEIIEWEkCBBCCCGEECLCSBAghBBCCCFEhJEgQAghhBBCiAgjQYAQQgghhBARRoIAIYQQQgghIowEAUIIIYQQQkQYCQKEEEIIIYSIMBIECCGEEEIIEWEkCBBCCCGEECLCSBAghBBCCCFEhJEgQAghhBBCiAjTIUHAggULmDFjBjNmzOCZZ57piCoIIYQQQggRsdo9CPB6vTz55JO8/vrrLFiwgDVr1rBixYr2roYQQgghhBARq92DgHA4jKZpeL1eQqEQoVAIi8XS3tUQQgghhBAiYhnb+4QOh4P777+fiy++GJvNxrhx4xg9enR7V0MIIYQQQoiIpei6rrfnCXfs2MHDDz/MK6+8gtPp5MEHH2T48OHcfvvt7VkNIYQQQgghIla7jwQsW7aMCRMmkJCQAMCsWbN48803mxwElJa60LR2jVvaVVKSk+Li6o6uRruLxHZHYptB2t1aZTVHd+8/ITJ/vyKxzRCZ7Y7ENkPrt7u5fWh30+5rAgYOHMiKFSvweDzous6SJUsYNmxYe1dDCCGEEEKIiNXuIwGTJ09m27ZtzJo1C5PJxLBhw7jjjjvauxpCCCGEEEJErHYPAgDuuOMOufAXQgghhBCig8iOwUIIIYQQQkQYCQKEEEIIIYSIMBIECCGEEEII0QkcTaPfHjpkTYAQQgghhGhYSNPJL/OQX+zGYjbQOy2aWLupo6sluhkJAoQQQgghOomQprN4zSHeW7Kn9pjNYuSXN4+T/PadiNvt5pFHHiEnJwdVVRkyZAgzZszgT3/6E2lpaezfvx+bzcYdd9zB66+/zv79+7ngggt49NFHAXjnnXd4/fXXUVWVxMREHn/8cXr37l3nHE899RQ7d+7kr3/9KyaTiWeffZbVq1cTDocZPHgwjz32GA6Hg3PPPZfhw4ezc+dOfvrTnzJt2rQmtUGmAwkhhBBCdBL5ZZ46AQCA1x/i3x9tw+0NdlCtxIkWL16M2+1mwYIFvP/++wDk5uayefNm7rjjDhYsWIDD4eDll1/mpZdeYt68ebz55psUFhaycuVK/vnPf/Laa6+xcOFCLrnkEu655x50vWYzR13XeeKJJ8jPz+cf//gHUVFRvPzyyxgMBubNm8fChQtJTk7m2Wefra1Pv379+OSTT5ocAICMBAghhBBCdBr5Je4Gj+/Nq6Sw3IPTJPdvO4MxY8YwZ84cbrzxRiZOnMjNN99MWVkZmZmZDB48GIAePXrgdDoxm83Ex8cTFRVFZWUlS5cuZfr06cTHxwMwa9YsnnzySXJzcwF49dVXKS0t5YMPPsBsNgPw9ddfU11dzYoVKwAIBoMkJCTU1mfs2LGn3QYJAoQQQgghOgmLydDgcaNBwWKUAKCzyMrKYvHixXz33XesWrWKH/7whzzxxBO1F+1HGY31L7U1Tat3TNd1QqEQAOPGjWP06NE88sgjvPPOO5hMJjRN49FHH2XKlClAzXQkv99f+3673X7abZDfJiGEEEKITqJ3WjQ2S/0Lx4sn9iIt0dEBNRINefPNN3nkkUeYPHkyP//5z5k8eTLbtm1r0nvPOussPv74Y8rKygCYO3cusbGx9OzZE4ChQ4dyww034HQ6efHFFwGYPHkyb7zxBoFAAE3TePzxx3nuueda1AYJAoQQQgghOolYu4lf3jKOvpmxQM0IwMyzejNtbBaqqnRs5UStyy+/nHA4zPTp05k1axbV1dUMGDCgSe+dNGkSt9xyCzfffDMzZszggw8+4KWXXkJVj12WK4rCU089xZtvvsm6deu4++67ycjI4IorrmD69Onous7DDz/cojYo+tFVCF1EaakLTetSVT4tSUlOiourO7oa7S4S2x2JbQZpd2uV1Rzdvf+EyPz9isQ2Q/dvd0jTqXAHMBlVYuwm0Lt/mxvT2u2WLEs1ZE2AEEIIIUQnY1QVEp2WmgfdO3YXHUSmAwkhhBBCCBFhJAgQQgghhBAiwkgQIIQQQgghRISRIEAIIYQQQogII0GAEEIIIYQQEUaCACGEEEIIISKMBAFCCCGEEEJ0Erm5uQwYMIDly5fXOX7uueeSm5vbaueRIEAIIYQQQohOxGQy8fjjj+NyudrsHLJZmBBCCCGEEKfh67WHeO2T7ZSUe0mMs3HTxYOYOiar1cpPTk5m4sSJPPPMM/zv//5vnef+/ve/s3DhQgwGA5MmTeLnP/85BoPhtM8hIwFCCCGEEEI00ddrD/HiexspLveiA8XlXl58byNfrz3Uqud5+OGHWbZsWZ1pQd9++y1Llixh7ty5zJ8/n5ycHN5+++1mlS9BgBBCCCGEEE302ifb8QfDdY75g2Fe+2R7q57H4XDwv//7v3WmBa1atYoZM2Zgs9kwGo3Mnj2blStXNqt8CQKEEEIIIYRoopJy72kdb4nJkyfXTgsC0DSt3mtCoVCzypYgQAghhBBCiCZKjLOd1vGWOjotqKioiDPPPJOPPvoIn89HKBRi7ty5nHnmmc0qV4IAIYQQQgghmuimiwdhMdVdiGsxGbjp4kFtcr6j04KCwSBTp05l6tSpzJ49mxkzZpCens4NN9zQrHIlO5AQQgghhBBNdDQLUFtlB8rMzGTJkiV1jk2ePJmdO3cCcPfdd3P33Xe3+DwSBAjRzRkMCgDhsN7BNRFCCCFAA0IhDbNJhS761TR1TFarpgTtCBIECNFNufwh9uZVsXp7IaqqMG5QCtlp0URZTj+XsBBCCNFSYV1nV24lC5buo6jMy5lDUzl3TCaJTktHVy0iSRAgRDdU5Q3ywvsb2ZdXVXts2cZ8BvWK567Lh+K0yp++EEKI9rXtQAVz3llf+/jTVTl8t+Uwv7p1PDF2UwfWLDLJwmAhuhlFUVi2qaBOAHDU9gNlrNtVjKJ0QMWEEEJErEBY47+f7ah3vNzlZ09eZQfUSEgQIEQ34/aH+HD5/kafX7h0H75Q/TzDQgghRFvxBzWKKxrJr1/Z+vn1xalJECBENxMK6/gC4Uafr/YEZJGwEEKIdmUzG+ibGdvgcxmJjvatjAAkCBCi27FbDPTJiGn0+aF9ErGa5U9fCCFE+zGqCtdd0B+DWnc+6sCecfROc3ZQrTqfJ554gvvuu6/OsWXLlnHeeefhcrla9VxyJSBEN2NUFa48t2+jz18yqTcqsihACHGMMeTGGD42JcMarmrXtUOyTiky9Ep28Nvbz2TW1D5MGp7G3bOGc/cVw4iySLKKo372s5+xZcsWvvzySwA8Hg+/+c1veOqpp3A4WnfERD51IbqhvunR3HfVCP790TaqPUEAYh0Wbps5mN4pMuwqhDjGFHLjW/0+itGCZezlqK4iyhbNIW7GfQRie6O34exBRQFT0VYwmgkl9MdUeQDNVUYobSS63KzoltLjbWRM6IWqdt39a/RwkMPvPg1AyuwHKZz7LACpP3gYxdCyLEdRUVH87ne/49FHH2XChAm88MILnHvuudhsNq699lp8Ph9xcXH89re/JSsri3//+9/Mnz8fVVUZPnw4TzzxRJPPJUGAEN2QQVEY1TeBfndO5HC5B1WBlDg7drPsESCEqMvgL8OzbSmEQ2ieCvwHNqH5PXg2fYll8o2ElLbL4W4NlFK86HkA4s7/IaVLXkMP+km+6Wm8luQ2O6/oWLquE2586Vqnd/jdp/Ed3AbAwRfuQA+Hao+nXft4i8ufOHEikydP5pFHHmHfvn28+eabXH/99fz9738nPT2dpUuX8vjjj/PKK6/w0ksvsXTpUgwGA7/85S8pLCwkJSWlSeeRIECIbkrXIcpioE+qzLUUQjTO7+hBwuUPUjr/Wbw7VwFg6zsW64QfEGzDAADAa0og/tIHKP3gWco+fQlQiJ95Pz5LUpueV4jWoIcC6KEAAIrR3KplP/zww0ydOpW//OUvFBQUcOjQIX784x/XPu9yuTAYDIwaNYorr7yS8847jx/+8IdNDgBAggAhhBAioum6jmK2o6gq+pHswWpULLraPps3KRY7iqKiEwZFQbU5auYJdc2ZIiICpMx+sGYE4EgAAKAYjKTM/nmrncPhcBAdHU1GRgYul4vMzEwWLFgAQDgcpqSkBIC//vWvbNiwgW+//Zbbb7+dZ599lvHjxzfpHLIwWAghhIhgVnceZfOeRg8FMCX3BtWIe+MXBNcvwqD72/bcgVJK5z2DroVxnnk5KCql8/6A1VvYpucVoiUK5z5bOwXoKD0conDuH9vkfNnZ2VRWVrJmzRoA5s6dy4MPPkhZWRnTp0+nf//+3H///UyaNImdO3c2uVwZCRBCCCEimGZ2YErri2q0EDX1ZvTyXMoWPIc5axAh1dymd+QD5lhiz7kJxWRFyxhBQvoAwuWHCVgT2u6kQrQSxWhGMRjrBQStzWw28/zzz/Pkk0/i9/txOBw888wzxMfHc/XVV3PllVdis9no3bs3s2fPbnK5iq635br/1lda6kLTulSVT0tSkpPi4uqOrka7i8R2R2KbQdrdWmU1R3fvPyEyf79ao83mQAW6ohI0RaMoCtZACT5zPLre9hl6FMKAWpsNSCWMxqmTGMjPOnK0drub24ce1ZbZgdqTjAQIIYQQES5gjq39v67reE0J7TYnXz/hgr8pAYAQHUkxmOpkAWqNjEAdQdYECCGEEEIIEWEkCBBCCCGEECLCSBAghBBCCCFEhJEgQAghhBBCiAgjQYAQQgghhBARRoIAIYQQQkQERQGjUUVV2z71qRCdXYcEAUuWLGHWrFlcfPHF/O53v+uIKgghhBAigpS6Any2JpenXl/L3G/3UVDuRZFYQLSizZs3c99993V0NZqs3fcJOHToEL/+9a957733SEhI4Oabb+abb75hypQp7V0VIYQQQkSAMleAp15dTbnLD8COnHI+W5XD47eOJyPe3sG1E93FsGHDeOGFFzq6Gk3W7kHA4sWLmT59OqmpqQDMmTMHi8XS3tUQQgghRCdX6Q1iNKg4LEZ0vXm7lykKbNpbUhsAHBUIaXz+/UF+ePHAdtsYTXQfbrebRx55hJycHFRVZciQIcyYMYMnn3ySDz/8kNtuu42SkhIAPB4Phw4d4tNPPyU9PZ1nn32W1atXEw6HGTx4MI899hgOh6Pd29Du04FycnIIh8PcddddXHbZZbz55pvExMS0dzWEEEII0UlVeIK88vF2fvrCUh76y3K+3VRAMKw1qyxVVdl2oLzB57buKyMQal65IrItXrwYt9vNggULeP/99wHIzc2tff6VV15hwYIFvPfee6SkpPDTn/6UXr168fLLL2MwGJg3bx4LFy4kOTmZZ599tkPa0O4jAeFwmDVr1vD6669jt9v58Y9/zPz585k1a1aT3p+Q0P6RUntLSnJ2dBU6RCS2OxLbDNLujhIJ/Sd0/OfcEbpTmwOhMP/4aC0rNxcA4PWH+PdH20hOsDN5REad1za13QN6xrFme2H94z3iSE50YDIaWl7xdtKdftano7O1e8yYMcyZM4cbb7yRiRMncvPNN1NWVlbnNZqm8eCDD5Kdnc0dd9wBwNdff011dTUrVqwAIBgMkpCQ0O71hw4IAhITE5kwYQLx8fEAnH/++WzatKnJQUBpqQtN677jdklJToqLqzu6Gu2uPdtdXOVnT14FBoNK/8xYYu2mdjnvieRnHVlas93N/TLs7v0nRObvV3drc7knWBsAHO+bdbkMzoolfGRE4HTaPaJPAvOtRjy+UO0xVVW48MweVJR7Wqfi7aC7/aybqrXb3RoBRVZWFosXL+a7775j1apV/PCHP+SJJ56o85onn3wSr9fLnDlzao9pmsajjz5auxbW7Xbj99edqtZe2j0IOOecc/jFL35BVVUVUVFRLF26lPPOO6+9qyEiVEm1n9+88h1ef80XQXKcncduGYvD0u5/CkIIIRpgVBUsJgP+YLjO8VinpdnrAhKdFn71w/F8uzGfDbuK6ZMRw/nje5CVaJf1AKJZ3nzzTdauXcuzzz7LWWedRWlpKdu2bat9/uWXX2b9+vW8/vrrGAzHRpomT57MG2+8wYQJEzAajTz++OPY7fYOyZbZ7lc+I0aM4Pbbb+e6664jGAwyadIkZs+e3d7VEBFqf0FVbQAAUFTuIbfIzcAsWZcihBCdQbTNyJXn9OWNz3fWHjOoChOHprVoJCs5xspVU7KZdXY2BkWpCSgkABDNdPnll/P9998zffp0bDYbaWlpDBgwgE8//ZTCwkKee+45evfuzQ033ICm1Yxe3Xfffdx9990888wzXHHFFYTDYQYNGsTDDz/cIW3okNufV155JVdeeWVHnFpEOJOx/lp4k0n2zBNCiM5C1+Gs4Wkkxtr4dkMeCTFWzhqZQWYr3LXX9ZqMKM0dURDiKLvdzp/+9Kd6x6dNmwbAjh07Gn3vr3/967aq1mmRORAiovRJj6FPRgx78yoBmDA0lczEqA6ulRBCiOOZjSojsuMZ3S8RXddrRgDkul2IViVBgIgoTquRn10zkrwSNwaDSnqCHbNBRgKEEAJAOTpNhpr8+h19wzzczLSgQohTkyBARByryUCftOiOroYQQnQq5mAleuEu9IzhGDzF6D43gYQBHV2tRgVDYUqq/fj8IRKirdjMXSfNpxCdgQQBQgghhECpzKP0478QNXQq3j1rMMam4LzkAQJq59tfwuUP8dbcjXyx+hC6DrEOC3ddMYz+mdEybUiIJpJ5EEIIIYQglDiA6AmzcG/5Gj3oJ/biuztlAKAo8NW6PBZ/f6h2ulKFy88f3lhLYbmvYysnRBfS5CCgsrISl8vVlnURQgghRAcxugpwrf8c1WJHDwfxbluKQet8F9XVvhAfLd9f77im6ewrqOyAGgnRNZ0yCNi3bx+zZ89mwoQJnHHGGdxwww3k5+e3R92EEEII0U4UdExJWSRc9ztip90GWrhTTq3RNJ1gIwuGAyFZSCy6B5fLxSWXXEJubm6bneOUQcAjjzzCVVddxcaNG1m/fj0XXnghv/zlL9usQkIIIYRofz5HFo6L7sdnikfvPRHDiBmEDdaOrlY9TpuJCUPTGnyuV4qznWsjROvbuHEj1157LQcOHGjT85wyCPB6vVxzzTWYTCbMZjM33ngjJSUlbVopIYQQQrS/gFJz0a9hIKx2vgAAQAGumNKHlHh7nePXXtBf9n0R7SoUClFSUkIoFGrVct99911+/etfk5yc3KrlnuiU2YGys7NZt24do0ePBmDXrl1kZma2aaWEEEIIIRqT4DDz9D2T2bG/FI8/RFqCnaQYKwZF6eiqiQixceNG7r//fgKBAGazmeeff54RI0a0StlPPvlkq5RzKqcMAvLz87nxxhsZMGAARqORbdu2kZSUxMyZMwFYtGhRm1dSCCGEEOJ4ibE2BmTGdHQ1RAQKhULcf//9tQlzAoEA999/P19++SUGQ9fZr+KUQcCDDz7YHvUQQgghhBCi06uoqCAQCNQ5FggEKC8vJzExsYNqdfoaDQL27t1Lnz59iIpqeH7dkCFD2qxSQgghhBBCdEaxsbGYzeY6gYDZbCYuLq4Da3X6Gg0C/vCHP/DSSy9x1VVXkZaWhq4fyxPm9XpZuXJlu1RQCCGEEKIzUBTQO2HaVNG+jEYjzz//fL01AV1pKhCcJAh45plnqKiooE+fPrz++uvouo6iKASDQW644Yb2rKMQopWoqoKiKIQbybEthBCivgpPkN25FRSVe8lMdtAnPRqH5ZQzqkU3NmLECL788kvKy8uJi4trkwBgyZIlrV7m8Rr9Df7Zz37G8uXLURSFCRMm1B43GAxceOGFbVopIUTr0tE5WOxh1dbDFJV5OGNIKoN7xuGwypeYEKJ1WL2HCVpiCatWDJoPk78Cny211cpXFHD7wxRVeDEaVGxRllYr+2RKqv089Z/VVLiOTf3okxHDvVcNJ9pqapc6iM7JYDB0qTUAJ2r0CuCVV14BajYL+/3vf99uFRJCtL5duVX84Y21tcPY63cVM7h3PPdcMQybuWsNX4ruRVF0LMFKfMZYjJoXJRwgaJKML12NpTqHkrlP4xg5DevIi/Bt+oTKjV+SMOsX+J09W+EMOlsPVvKPBVuoPHIx3ivNyR2XDSM1tg33M1Dg63V5dQIAgL15lWw/UM4ZA9s2j7sQbemUm4VJACBE1+YPabz68bZ681i37S/jQJGrYyolxBHm0l2Uvv0rrK5DBNYuwPPNq5jD7o6uljgNigK6z4Ue9FP9/ULK3/kN1d8vQg/60X0uWiN1/8ESD//35rraAADgQEE1T/1nNVXeYMtP0IhASGPV1sMNPrd+VzEGwykvo4TotOS3V4huzuULUVjmbfC50sqGjwvRbqISMUTFUvzm47jXf4p94ERCBltH10qcBl2HYPJg4mf+BIBQZREA8TPvJ5g8uMULaVVVYcXmggbLcXmD7C+obtkJTsJkUOvtTHxUepIDTZNVwqLrkiBAiCYqdwdYv7eUFdsK2ZlXiTcY7ugqNYndYiDW0fDc2Zh2mlN7SgqUuQLklrip9oVQTnHr8PinFYVWudMoOkbYZMec2rfmgcGIIS4NTb6amkeBck/N31GFJ9iufxdq2E/w8O46x4IFe1DD/haXrSgK+/KqGn2+3NXyczR6buCys7LrHTeoCmMGJAMSBJwOX0hjZ14VK7YVsn5vKWXuwKnfJNqMrAoU4lQU2H6okj+9vZ5g6FhWnYwkBw9cPZJ4h7ndq2RQwoR1Q73/N8RuNnD9RQP4y/ub6hxPT4yid5qzTevZFC5/iI9X5fD5dwfRNB2bxcjV5/fjzMEpmBsYajcHK9EObUTpOYaQMQpT6U5AJZjQT1L3dUFqyV7cm5cQe/6tuDd+QcUnfyNm9qP4lYb3qBEN8wTCLFmXx8Kl+wiFNcxGlSum9OHskenYTHX7B0WpuRhToMG/sdNhUMLoihElZy3V3y8C1UD0GZdRteoDqr9fSFxcCmrvyS26Y67rOkOyE9iTW9Hg8zbzsUsZTdc5XOHjcKmHWIeZzKSoFrexT7qTB68bzZuf7yS/xE3/HrGcMyaLtxbvYNygVEb1SyLaJpdTp1LmDvCndzaQe9w0VJNR5SdXj2JQjxiJpzqA/NYKcQqFFT6ee3Md4RO+xPKKXfxj0VYevGYUhna842YKuQhu+gTLoLMA8G9fimn4xQSNjgZfr+swIjuBn18/mg+XH6CsysdZI9KZMDStw1Pc6cDcb/bxzbrc2mNef4hXP9qO1Wxk/ICkOq83aT7cy/6Lb/dq7MP2Y+s7ltKFc1BQSLz2N/jsGe3cAtFSWmJfkq77LQFHFjEZQyAckADgNGmazpJ1ucz7em/tsUBI450vd6OoKheNy6y9CA/rOut3l/LmZztRVYVbpg9icM841Gb0YRZfMf6tS7CMnI4am4IlcxCOsTMIpw4lIaUPrnUfo6YNJtTCKTOapjN2YDIfr9hf50YMQEq8HU3XUFUFTddYsbWIfy3aWvv8xOFp3DCtP1ZT8xMgGBSFwT1i+fWt4zlY7OLDZQd4+YPN6Dps21/Oyh4F3H/ViHrBljgmrOu8smhrnQAAIBjS+L+31vHknRNIiWnDBd6iQRIECHEKO3LK6wUAR+3MKSe/1ENWYsNzRtuCGvLg2b0az47lNQcMZuIGT4VGggAAo6owKCuWgdeOJBzWMRnUOhsAdpSiCl+dAOB4b3+xi2HZ8XW+WEMGK1EjL8S/fyOezV/h2fwVAPZRFxKydK2dGkWNkGol5KjJHuO3JHRwbbqmvGIXC5fub/C5+d/s4czBKbV3qg8Vu/nr3GOjgs+9vZ7/vWMCGfHNWIfhq6R6w2J8B7cSLM0jbur16Am9CGugpQzGMa0XAUPrBHQZCTZuu3QIi787yN68SlQFRg9MoVdaNAo1gUJJdYBXP9pW530rNhUweXg6AzNbnnEqENR44Z0NVHvqLkTedbCCg0UuBmRIVqvG5Jd52X6gvMHnNE1ne04ZKcPT27lWndeLL77IJ598AsCUKVN46KGH2uQ8MvFSiJNQVYX8kpNnKqn2tO+cRr81mfiL7iTsqiDsqiD+orvwW5NO/UZA0WsCgs4QAABUneSzq6j24wvUXXeh6xBK6EvM2dfUHlNtTuxjZxIytF8gJkRnUuUOEGpkA0B/IIzbd+yitbC8fjKA4gpPs84bjO9LzMQrCRYfBEXB1GsUQUPNzQhdV1otAKgpENLi7cQ4zFx6VjaXTM6mqNzDt+tyGdgzHoDyan+D045KK32tUgW3L1gvADiqvKrt1iV0B65TfE/mF7tRmzMc1Q2tWLGCZcuWMX/+fD744AO2bt3K4sWL2+RcEgQIcRKappOZ1PgddoDoqPZdE2DxFVH20YsYnIkYnImUffwiFl9Ru9ahtcQ4zI0uXoyPttaZ6ws1c5mNJTup+ObN2mOatxrPqnmYJK2kiFCxDjNmY8Nf5zaLkajj5qunnpDpRlEgJa55AbSpdDeVy9/DnNYHgOpv38AUaru0w1mJUcya0pdwWGP3oXLOHZPFL24cQ6y9ZsOueKcVQwMXkomxrZNtymEzNdrfx0fLVJaTcdpP/j2Zkdz1Mi2Vl5fzt7/9jXvuuYe///3vlJc3PNJxupKSknj44Ycxm82YTCb69OlDfn5+q5R9IpkOJMQpDOgRi9GgEArX76CGZMeT1pxh9BbQjFFEDZ2KuX/NTt6B3SvRTF1zDnVStJXzxmbxxepD9Z67dlp/rKa6FzbGsBfv5q8gHCJq1IXY+oyh9INn8exYjn3E+QTtXfNzEKIl0pMcXD6lD+9+ubvec1ee25cYm7l29C8z0c79PxjJm4t3YlJVrr9oIKlxzbyAtcfhHHcJ5mEXQHkumquMsKFtL4bT421cNbUPqqoQFxdFcfGx9KAJTjO3XzqElxdsqU0SMHV0Jj2TW6dfiLIYuHn6IP783sY6x4dkx5OVfPKbRZEuPd7GsD4JbN5bWu85o0FhUM+uNZ2zvLycq6++murqaoLBIOvXr2f+/Pm8/fbbxMW1rC39+vWr/f+BAwf45JNPeOutt1pa5QYpemeZF9BEpaWuLhctno6kJGedTi1SdPZ2786v4rm31+M/bnpKr7Ro7r1yOHHNHAloSZsNikZYV+v9vys4sd1uf5gv1h7io+UHCIU1HDYT110wgDEDkjA1cFfPFKpCz98KGSMIG+2YyvagqCqB2OxOnR2oNX/Hk5Kal9Wpu/ef0Pn7kraQlOTkYH4FSzcdZv43e/AHwtgsRq48ty8Th6RiaWCUIKjpKNRMD2yJo/2PogC6ht6OEwwa+lnr1CRzKCr3EOMwkx5vx9SKG3ppus6+w9UsWZtLeZWfs0amMyw7Aae1fe6pduXf73J3gBfnbmJ//rF0rxazgQeuGUX/9OiTvre1293cPvSov/3tb7z22msEg8emh5lMJm666SZ+/OMft7R6AOzevZs777yTe++9lyuuuKJVyjyRjAQI0QT9M6J5+scTOXi4Gk8gTEK0laykqAa/XNvD8Rf9XSkAaEiUxcBlk3oxZWQGXn+IaLsZh9XQ6AV90BiN0nNizZ1NvWZesq4j6eVERLOZDFw0LpMzB6fg9gVrpq7YTI2u/2kowG6Oo/1PzWk6vi9SgNRYK6mxbTMioSoKfdOiGXDZUADCjazFaAtGjl1wKgqoWpCwYmq387dUXJSZh64bTW6xm5IqHzazgZ4pTmKjTJ36Bk5DtmzZUicAAAgGg2zZsqVVyl+7di333Xcfjz76KDNmzGiVMhsiQYAQTaDrEGMzMax3fEdXpXvSIdZuqp3be6ovhOMvbLral4cQbUXTdKJtxtpMQG090K/pUFzpI7/UjcVkoEeKo8PTDreX9rz4BzAFK/Gvno9//HQUUwqm0l0EDm3BOOwiQmrX2WHbYlTpk+akz3F71HTFPnzYsGGsX7+eQODYgmez2cywYcNaXHZBQQH33HMPc+bMYcKECS0u72Qi469VCCGEEK1G03WWby3k1Y+21V7ExTksPHTjmIjL964oCsGwhtGgtMmIpKrohHYtxb3la7z71hN7zo2UffoSejhIQmo2SuqILnkh3ZVdffXVzJ8/n6qqKoLBIGazGafTydVXX93isl955RX8fj9PP/107bFrrrmGa6+9tsVln0iCACGEEEKclsPlPv79Yd2c/OUuP699up2f/WAkamNpv7qZSm+Q5ZsKWLoxn8xkBzMn96ZHYusmKNB0BXP/yVgObcd/cCtlH70IQNSI89ES+0oA0AHi4uJ4++23efvtt9myZQvDhg3j6quvbvGiYIDHHnuMxx57rBVqeWoSBAghhBDitBwqajgV6Pb95ZS5AiQ6Le1co/YX1nVe/3QH63YWA1BY5mHj7mKe+NGZpLZSWtKjguZYHGOm4z94bDdk+7Bz8LXmXgzitMTFxbXaIuCO0vGreIQQQgjRpZiMDd/pV1UFQytm4+nMiip8tQHAUaGwzq5DFa16HkWp2ZOhbOGfah4batZOlc77AxZP2+SPF5EhMv5ShRBCCNFqeqZGN7hB2TljMomzd52MNS3R2MLrcGun4dU1dG8VejiEc9Q0km7+A5YeQ9ADXgjU3wFaiKaS6UBCCCGEOC2JTjMP3zSWv83bTHGFF0WBScPTmTmxV0dXrVYwrJFX6sHtDZIUZyMp2kprrlRIjrUxsGccO3KO7RSrKtA/K7YVzwI6KqHMUSRd+2tsKVmUeww4zvsRqr8Kv7OHpEcWzSZBgBBCCCFOi65Dr2QHT9x+BsWVPiwmA4nRlla9yG6JSm+QfyzayrZ9ZUDNxfm1Fwxgyoj0Fm+QdpRRVfjRpUP57Psclm7IJz0ximum9Scj3t4q5R9P11V8zl44o5zgqSZgigVTrAQAokUkCBBCCCFEs1iMKpkJrX/R2xKKAl+vy6sNAKBmT4M3PttJr7Ro+qS2bLfY48VFmbj23H5cNjkbs1HF0FmiICGaQNYECCGEEKLb8ATCfPpdToPP7cwpp7Wzl+q6js0kAYBoPc8//zzTp09nxowZ/Pvf/26z88hIgBBCCCFaRFFAVVU0TevwvPUKCoZGpvyorTQVSAgAn89HSUkJiYmJWK2ts0ne999/z6pVq1i4cCGhUIjp06czZcoUsrOzW6X840kQIIQQQohmK6n2s2F3CXvzKhjQI57hfROIjzJ3WH1sZgMzJ/XmnS9313tuUK/4Dg9SRNcXCoV44YUXmDt3bm3wO3v2bO677z6MxpZdWo8fP57XXnsNo9FIYWEh4XAYu71tptxJECCEEEJEAFVV0HW9VS+Ci6v8PPGv73D7QgB8t7WQOIeFR28ZR4KjYwIBXdeZNCyNvBI3yzbW5NG3mAzcMmMwWYmda/2C6JpeeOEF5s2bh9/vrz02b948FEXhgQceaHH5JpOJF154gX/9619cdNFFpKSktLjMhsiaACGEEKKNmANlWMp2oipgqdyPxXu43esQ0nS251byn093sGR9PpWeYKuUaw6UUVFUQFjTeeTSNPql1Vxgl7v8rN1Z1Opz70+Hw2rklosG8NSdE/jlzWN5+scTOXNQMmpHVkp0Cz6fj7lz5+Lz+eodf//99+sdb6777ruPlStXUlBQwLvvvtsqZZ5IggAh2pE3qLE7v4otOeXsPlgu2d2E6MZUFcL711Iy9xnY9iml857Gs/ZDTErrXIQ31a7cSv7437V8tS6P1z/dwT8XbSPUwg2tzJoH11f/Jm3XPH4zI5q075/nx+M1HLaajcI27Czu8J2DVUUhNc5Gn7RoYuymRjf3EuJ0lJSUoKoN/26rqkpJSUmLyt+7dy/bt28HwGazccEFF7Bz584WldkYmQ4kRDtQFNhX6OLP722kotpfe+ycMZlcflY2Dov8KQrR3WgamPtNxpKzmcqlb2OMTsI+4Sr8evvtqGswqHyzIa/Osa37Symt9pMS0/yFjEGDHee4SymZ9wy23E1oSX3YWObA5a25AOrbIxattXfOFaITSExMRNO0Bp/TdZ3ExMQWlZ+bm8sLL7zAW2+9BcCXX37J7NmzW1RmY2QkQHQIgx7E5slDQcccrMBfeKCjq9SmCit9PPPamtoAAGo221myJpdPvztIp9lhRwjRahQFlOoCAnk7QTUSqi5FK9qP0o5jgJqmk5XsqHPMYjZgM7fsxoOuA0YTypHpNbrRgjtQc2FkMqpMGJIqQYDolqxWK7Nnz66XDaix46drypQpTJ06lcsvv5zZs2czatQoZsyY0aIyGyO3H0WHMJbtpWjeH4i76C5c276lsrqM2FmP4lcdp35zF6MosHlvKYFQw3cOPl2VwzmjMklwdlw2DSFEW9DR3BWYU/sSM+02XKsXEa4sQk0NEVbaZzRA13UmDk1j9fZCDhW6MBlV7rp8WIunx5h1L1VL38KUmo111AyqP/kzZ/WfTPnYLKaOziA93iZZeES3dd9996EoCu+//36d7ED33ntvq5R/7733tlpZJ6PoXWySXGmpq1vfXUhKclJcXN3R1WhzprAb38q38WxbCopK6nW/wuPo1S2/NAwGlefnbmTdjuJGX/PLm8fSJy26HWvVcSLld/xErdnupKTm7Xja3ftP6Hy/Xwo6prCbgMGBMeRGN5gIK60b8Delzf6QRkmljyibibgoU6v0tWZfCagqQUs8Vk8BbnMcqtFKew10dLafdXuIxDZD67e7uX3oidpin4D2JCMBokMoWpCw68iW7rqO5qmCKJ3uOC9G13XS4qOAxoMAm6X95ggLIdqPjkLAUDPCGTJGneR1OsVVfkorfaTE2Ulwmpt9oe4LahwqcWEyqKQn2DEbVCxGlYyEmuw9rXWzJWBNPFp5vLa0mvnF3TvGFKIOq9VKZmZmR1ej2SQIEB1CqS4iULCHxKsexbNjBZXfLcI5oy9+5WgOZx1FUTrlyIAOhDUds1Ft0l1VTdMZPSCZj1YcaPD5Qb3jSI3rencQhBCtQ1Fgy/4K5ryzHl2vmVP/6M3j6JnUeNBwIkv5bjBH4TLEoh3cxLaDFhasKeesEelcf0F/zB2cqUcI0fl0aK/wzDPP8PDDD3dkFcRpMFGT1k5RwKgHTvraoKbjD/ix+Q6jKDoKOjbvYVTCAATi+5F0/ZME4vtjHXcVCRfeRnDHt5hCLhRFx3R4M6aSHbWLzjqDkKaz41Alz769gV/+YxUfLDtAmevkn8NRPZKjuOGiAfWOx0db+OH0wZK7WohmMOjHUm0e7Z86+zmNR96j6iFUvWaDLW9Q45+Lttbe9AiGNN7/aneTb6oHq0qp/Pq/lM1/Bn3dXMJf/52ze2iYjSpLN+aTX+I5ZRlW72GMmhcAi68IU7DytNrlC4bZmVfJ9zuLySl2EW7kBomR9v+ZdUaBsMbew9V8v7OYfYerCYQbXjMmRFvqsJGAlStXMn/+fKZOndpRVRCnwRwow/vdXKLGz0L3VuDbvwHTiOmEVFu91+aXeZj/1S6u7V1EYO3bxF76U1Rdo2jRn4i74EfoPc9ER8FrTgQdFFWlbPEb+HI2Yy/Lw9prJGUf/wVUlaRrn8BnT++AFte39UA5z7+7ofbxB9/uZcXmAn5581ic1pP/KRkUhSnD0+mfGcfOQ+VUuQP0y4qjZ7IDp00G5IQ4XUbNT3j7YszJ2Shx6XhWvod93OVA68z1bfSc2xZjTslGiU3Hs6rmnH5rUpPeb3Hn4t28BNv42YQObqy5ydFjHGFNwesP1XltlSuApoOhCfcHTNEJxM24l6LXHsa/5Uu04Zfw8nfh2mQEgVD4pO+3unMpef8p7EPOJmrYuZQt+D+MCRnYp/6QoPHUn6c3GOZfH21n7Y6i2mPXTuvP+WMy60zwNIVcBNYtwDpoMhgteNZ9hG38bAKm2FM3shsJhjXe/3ovX6w+VHvsojN7csVZ2Zia8gMXopV0yNVHRUUFc+bM4a677mLHjh0dUQVxmpSQD1/OFvyHtqH53JgzBmAO+cFcNwjwBTX+9M4G3L4QroFJxBtNVCx4tqYMsxVDXBqhE+b9Bw024s66ksP5u/BsXYpn61IAnOMuJ2RrWb7d1uILabz+af3f1aJyD/vyqxiRHX/KMgyqQmaincxEO4qikJjoiMgFXkK0BkULECzJpWrlfAzOeDSvC/uoC9vhnIeoWnX8OS9q2nsV0L1VuLd8g//QVkLlhTjHz8SQFcJhsTJral/e+WJX7etnTe3bpAAAIOzz4N2xArRQTT+7+1suH/kj/ljgJi3BTmaSA1WtmV7ZUC4QzRSFKTEL97pPca/7FICoUdPQDE2bpnig0FUnAAB4+4tdjOibSPJxexEoYT/+w3txb1uGYjSjGM3Yx/ghwpZE5ZV66gQAUJMl7syhqfRIbPoUMCFaqkOCgF/96lc88MADFBQUnPZ7ExK6XwrJE7XWqvXWNQBl2i0UL3gegMTzbsSSllXvVfvyKimprNky+5XvAjw4/BJYNxeAuLOvwZk9hGil/iw0PWEwsZOvovyr/wJgiIohdtyFmKIT2qpBp+VQYTVlVQ1vBV7lCTT7Z9Y5f9ZtT9rdMbpX/+nEPuUacnd+R7iymIRpt+LsNRhoy8/ZiX3qteTuOnLOC27D2WsQ0U2czqfFjiIw4jyq13+OanUQO346ppiaUYRLz+5DdmYMRWUespKdDOmTgLWJufzD7ioIeki44HZsvYdSuPBFeiXbufeqEQzqlUCFy8+n3x/CZFSYNDyDAT3j6rXLMu0W8l99BABjfDoxI87B6Iht0vm/21k/6YGugycQPuFn4cR8wW0UvPZL9ICXlB88QlRW3yadozEd/TfVHJtzKho87vKGmtSertjm1hCp7W5L7R4EvPfee6SlpTFhwgTmzZt32u/v7inuOmv6L6s7l5KP/oY5rS+hymIKP3ie2MsexG+q+2ViNSj0THVS7fZx97gwrJqPYqxJh1f25WsojiQCyYM5PguQouhYS7ZR/vWbtcfC7kpKl/wX6xlXEzR2/IWLQdfJTHKQW+yq91xCtLVZP7PO+rNua9Lu1imrObpT/2kOVVH9yYsoZivmlN6Ufvkf1PgsnH2Ht9nv17Fz2mrO+cWrqHGZ+GJ6n/K9iqJgzFtL9frPsfYeiS9nM6VfvoF10g0E1Zq75f1SnfRLrfnZVld6aWorkpKiMY68lJDBTKVuJOaSn+FXbIwAiiq9PPbyKkJH5px/ujKH/73jTOKjjqUptXiLKF84BwDV6iBUlk/pt+9hHn0ZIUP9KZ8nSoiuP2JgNChE20x1fhYWfwnl8/8PQ3QiqiWKogXPk3jV4/jsaU1s6Ynt7pp9SVwje8LEOcynbE9XbXNLddYUoW3tmWeeoby8nKeffrpNym/3IODjjz+muLiYyy67jMrKSjweD0899RSPPvpoe1dFnIawOYboCbMw9p2AEnQTPryHUAMX5yaDwj2zh/PlmoMYDSWoZiuxl/8cFZ3SD56lofxxxrAX9/YVoGs4z5yFpccQSuf/Ad/edUSNnt4pggCTQeGm6QN5+rU1HH8NNbBnHL1Su0ZnIkR3EjLYiRoxDUNcKro9AeuuZWj2E+9wt8U5z8cQl3ba59R1HTU+i9ipN6D2P4uo4j2AQkBtncxgIYO99v9+5diF+8HC6toAAMDrD5Fb7CI+6rgpjAqg68RMvQFL9lgqPv87NX110wLGHklRXDKpNx8u3w+AqircecUw4h11L3ZDZifOsTMwZg1DV42E9q0mbI6M/VGOlx5r4/oLB/DGZzuBmqliN140iORYyRLXlbhcLj777DNycnLo2bMnF154IQ5H612vtMfa2Q7dLGzevHl8//33pxXhdKc7WQ3pzFG+qlB7AawoOrp+kiFwBcKahiNUidcUh6KAJVCG3xzX4PviLAHcOdsIpwxCUy2YK/aBwYTfUX/KUUc6VOLm+22FFJS5GT8olUE94065KLgxnfln3Zak3a1TVnN0t/5TQUc/Mqp4tH9q69+vhs55Ok71HgUN/bjEfQZFI6yfPJHfydq8/VAFf3xjXZ1jj90yjuwTbl5YAqWETNGEFRPmQDmawVInsGiwrke6cl2vSZtcWOGlwhUgOc5GgtOM0sC+L8e3vzmf3/G6cl+io1NY4aOsyk9CjJXkaCtNmVXWldvcEp1tJGDNmjU88MAD6LqOz+fDarWiKApz5sxh7NixLa5fRUUFd9xxB9OnT2fHjh3dZyRAdF3Hd9YnDQAAdDAoKt4j04V0HXym+EZvLBmjEwimjaxJkaeDPya7dSrdyrISo+g5tQ+KohCWlG5CdCj9uIvM9optWnrOk71HJYx6aC3+6EyqjIkkmP2Et3yGeej5zc6g0zvVydTRmXy9LheASyb1IrOB/Qf85mPrrwLmk49uhHWdQ8Vuth8oR9d1BvaMp0dyFOnxdtLjTx44HN/+bhSPnjYFhdRYG6mxp55uJToXl8vFAw88gNfrrT3m89WsGXzggQf45JNPWjwi0JK1s6ejQ4OAWbNmMWvWrI6sguhEOuPGYA2puZPaRSorhOhwvmAYtz9MjN2EUW38BoqlOpedVUb+9vEBCst3MLhHNDcOjCPxu7lYJ99EsBlpdKwmA9dP68eFZ2TVZCVzWjlJFU5Nge+2FfHPhVvrHL5p+iCmjkiFU90gEqKL++yzzxrMsgU10/4+++wzZs+e3ezyW7p29nTISIAQQgjRJnS2Hazknwu3UuHy0zPVyY8uHUp6fMN3f6ttqby8Mp/C8pq7itsOVvFVfCI3nzsCfwvyaBoUhZSY1rnjXO4K8OpH2+sd/++nOxjaO55Ep6VVztNZKGhYqg4SticRNEZhDlag+irxR/fsMjeuROvKycmpvfN/Ip/Px8GDB1tUfnuunZUgQAghhGgD+WU+/u+tdbUXizmHq3nmv2t48s4JOCz1v35dQZX8Em+dY9vy/fjMiQ3Mru8YxZW+OguNj9I0neIKb7cKAhR0jPkbKP7wz9iHTcUx7lIql7xCIHcn8bMeIhDXr6OrKDpAz549sVqtDQYCVquVHj16tKj8f//737X/P7p2tq2S55x8tZEQQgghmuVQUXW9u8XVniCHSz0Nvj7e6GNUn9g6x87ub8NatqvB17cXDQiENdyBMKFw47e/o2zdbNcvRUG1x6AYTXg2f0XRf36OP2cLisWOYpZNvSLVhRdeWLPbdwMUReHCC9t208LWJEGAEEII0QYsZkMjxxsehDf7K7mmTxkzxiTSM9XJjedmMTq0nkDeTgx6oC2r2gidPQXV/PGt9fzy5VV8uHw/+cVuRvRLqvfKkf0SSTvFouCuRtchENeH+EsfqDkQDgGQcMVD+KPSO7BmoiM5HA7mzJmDzWbDaq1J62q1WrHZbMyZM6dV04TOmjWrzTIDgUwHEkIIIdpE77Ro4qMtlFX5a4+NHpBEaiNrAnyOTDL7hbluiB2/NRmT5kUtDRKKzyasNLzBVFs6WOzh96+trh3N+Oy7g4wdlEL/HrEkxdlYvjEfXde5eEIvpozMwNSiFcedkylQgWvtR3WOuTd9gXXclQSNMhoQqcaOHcsnn3zCZ599xsGDB+nRo0er7xPQHiQIEEIIIdpAjM3EIzeNY+3OIvYcqmD0gCSG9E7AoCi4/GEsJrXehXPYmUEII+g6umrEnzi4fsFK7f5e9Z9SQNWC+HUjvqBGlClMTkmASpefuGgr6fH2k2YoOlaOwvrdxfXOsXZHIRlJUazZXsjkEenEx1iZPr5Ht0yZrCoQ2rMKf84WVHs0cefeRNmnL+PZ/BW2XsNR0kdF1OJgI0FCRxaoG7QgmsF46nTh3ZjD4WhRFqDOQIKAbk1H1cNoihEDIcJt8OP2BsPsL6ii3BUgK9lBVmJUp1nAJoQQHS3BYebCsZlcfORCubDSxz8WbqW4wsuFZ/ZEUSAh2krfjGhivLn4dq7EPGomatCNZ91H2MZdUbtHgKLAwRIPC5buw+0JctnZ2QzIOLbjrqKAuXwvnv0b+cw1iPE9jBgPLmdzYATzV5cDcNbIdK45tx92k8b+Ej95RS6yU+0kxDowG06YIXzcBa5BVZh2Rk8sJgOxTgtnj8zg8+9yGDsopdF0iV2dpoO1z3iiynIxJ2TgWr+Y2EmzCbur0FMGRlQAYAq7CWz4EGu/MwlHpxPetQRTfCbBlMERHQh0dRIEdFOKomMq3EbYU46l11gCmz/H0nskfkfLVq3XPQl8vDKHj1YcqD30yE1j6ZceedvACyFEY3QdwmENtz/M06+vodIV4LoLB/D6J9trLyTvubQvI4o/wbtzFTZ3JYHDewlXFmHLHoVyZCPF0uoAT/77ewKhmrvuf/jvWn5163iSk2v6XJPmpWrVPAIHt3JWnxIMS7cQ9FQx+ZwRfLheJRjSWLohn5F9Exjm/R6UDAweIzHb11HW6xzSsrJq66PrOiP7J7Jw2T50Ha6Y2pcvVh+korpmapPNYmT2uf3ITo/uVrtQn8hnjscx4SqK/vtLdJ8LS88hmMZcTkDvZougT8HgLcW17jPcm7/GPmgS7o1fYErMJOayX+A3tGz3XdFxZGFwN6WGgwQP76Xi839SueBpqlbOI1x+GEVpeWd9dFV8hSvAJysP1HnuoxUHULvhvFAhhGip/FI3la4AvdKi2bqvrM6d5L8u2oMy7geYMwfh3bmScGUR0WddSzhlSO3rCso8tQHAUXnFrtr/BxQb/rE3QFwmyt4VaJ4qvKOv4YWlPoLHve+rdflolSUkrvk7Q/b/F33vKnzVlbj94Tpl90iK4uEbxzJuUDKHy9y1AQCA1x9iw64ieiQ1PgfaoPux+IsBUPUQtkAxjSRV6bRMYTdVS99CD3ixZA2iasV8KNrV5drRUn5HFvEz/gc94MW98QsMzgRiZvxEAoAuToKAbiqsmrEMOx9jdBLBwgPY+o9D6TG6RcN2bl+I73cWM2/pPnbmVoKqYDxh+Nhpj6y7I0KIrqW02s/WnAoOV3jbfd9vo6Gm//UHwlhPyBxkNKiYtADhqpLaY6GKAtTwsaxA8Q3k4I+PrrvI2KoEUHxVtY9N7mLSY+v2y3abkcCg89EDPqg8THDcdbyywlPvBo6CQr/0aH506RD251Vxop055fiD4XrHAQyan/DmTymf+xRW32HUg99T/MZjmCv2Nfj6zipksGPrfyYJl/0Mx0X3ET35KpTo5IiaCgSg6kFCFYW1jzW/BwLuDqyRaA0SBHRTBj2Ab8MnhKqKsWQMwLtrNXrOmmaPBOjA3G/38ff5m1m0bD/P/HctBaVubpo+qPY1FrOBi87s2a2HhoUQXVdOkZtH/r6C/3trHY/+fSUb95W26x3d9IQo+mTEUFDqJjsjBttxG4Y9cs1g/KveJlxVTPTkq7H0GoFn89coRTtr65gSa+XHs4ZhsxgxqApXTO1DdtqxO7Ehn4vw8v+ge6swT7oeQ2pfjDu/5KrhKibjsa/7c0akYlj2T1SLHeIyMH7/X359eTI2U8OXBCZVYUS/xHrHB2cnYG3kPbpqxOCIJ+wqp+TNX1H+2csoJiuYrM345DqOjkIobQSBpMEEFBvKkIvwW+qnSO3uzK58qpa9gzE+nbgL70AP+qn4+EWsWnVHV61buvHGG5kxYwaXXXYZl112GRs3bmyT88iagG4qrJgx9xhCXEImao+RWLd/hSEhi1AzRwIqPEG+WZ9b59jCb/fz8+tH0TPVSaUrQFqCnXiHOeLukAghOj8deHfJ7jqbXf1r0TaeuXtSoxe/rc1iVLln9nBWbT3M2h1F3DZzCAZVITHGSlqcDVvSDdgHTEDLHI2j30TseZvRjluAqioK4wckM7hXHOGwTozdVPtcSbWfJ/61jjumziI9u5znvjNyz8W3EFO9j39t1AiGNCwmA9dc0J/+6U4M3omQ0p9qzYotZxmW6FiCx/fdCri8IVRVIcpi4NzRmazZVkhRRc2OxlE2E1ef1w+1kShKw4C5zwRsBzbi3bMGgPjpd+O3d738+vpx6S40PTLvnQYcmcRf/GPU5GwClkQSLnWgOOLxqZE5HSgUCrFw4ULeeustiouLSUpK4tprr+XSSy/FaGzZpbWu6xw4cICvvvqqxWWdigQB3VggYSBKgkYIFXXwBfhb0HmZDSp2ixG3L1R7LDXRjlFRyIi3k3FkkxgJAIQQnZGug8cfrHMsEAyjtXOnFWs3cfH4Hlx8Rg8UqB05VRRw6QlYeiWhhXXCphjU7LPqjazquk7Ukc3Gjq/6pr0luLwh/vRpEdEOCxXVXn75xh6eu/8sfpAR5uKzg0RHmYmLOhI4DDwPTVeJAgwJVxA87vvBH9L4cm0uH3y7F4vZyG0zBzMiO4HHbx3PoSIXYU0nKymK6JPsEKzqIcIHvq8JAAxGCIco//TvxF3+EH57Wmt9nKKdhBUjes8zCR75fQymjui2WaFOJRQKcd9997Fp0yZ8Ph8ALpeL5557ji+++IIXXnihRRfv+/bVTJm79dZbqaio4Ac/+AE33HBDq9T9RJEZ0kYQ/ciPuKV3L+JNPu65cjhGg8IV4xLonxnNRWc0PPUnpOnkl3nYk1+FO9DwfFEhhGhPqgKzp/atc2z2Of1wWBre1bfJ5arKaSdD0HUdXdNr+89qX4gPlh/gl/9YyWuf7aTcXbMO4HSmVhYfuUOv6dQu4A2GNFzuIHFRZnokRRF73MjB0e+EkKaTU+JjT0EVniP99b6CKt7/ag+hsI7bG+TP722ksMJLlNnAwMwYhvSIPWkAAKDoYcIVhaj2GJKu+19ip92G5nND0Nv0D0p0Ksf/PkZqAACwcOHCOgHAUT6fj02bNrFo0aIWlV9VVcWECRP4y1/+wquvvsrbb7/N8uXLW1RmY2QkQJySxVdM5cd/Zvg5N/HS/4zE/80rXD71HLQYS72FdZqus3hNLu8t2Q1AcqyNh24YQ7yj/Xe7FEKI4w3qEcuvbh1PXombhGgr2WnOFo1eFlb6WL6pALcvyNkjMuiZfPo7yCoKLFmXy4Jva+7+LVmTS4XLz92XD0Vt4q4rIVc5o/on8+36fC4eFc+8VUUA9Ex1Eh9dfzFx7fs0nYUrDvDhsv0AZCRF8bNrR1NSWffiRtehvNpPamzDOx03JKxaMAy/mIRBk/FZklB7J5OYPgC/NZl2X5EtRCt666236gUAR/l8Pt58802uuOKKZpc/atQoRo0aVfv4yiuv5JtvvmHSpEnNLrMxEgSIU/NUECzLp2z+HzDGJhMsySUYl4Y5fTAhte6XQkmVvzYAACiq8LJhTzHnjsxo71oLIUQdBkWhV7KDXsmNp7VsqkpvkN+9uhq3t2aK0Tfrcvntj86snRrZVIGwzopNBXWOrdtRjMcfxmFp/Cs6pOkUVXhJMLop2PAdg/qM5I9XJaGvnUv81Ev5fLufH88aXm9H4uMVVfhqAwCAvGI323PKyUysG8yYjSopcafXLoCwaiVsqVkIrGHAZ2n/AMAcrCRstBFWzFjDlfiNMTJtVbRIcXFxi54/lTVr1hAMBpkwYQJQM+rSVmsDZDqQOCV/fD8SLrkXPRQgWJKLpddwrONn1wsAAPzB+lvHuzzB2r0FhBCiO8gvcdcGAFAzDSfncE2mFIuvEEvlflQFLOW7MAfKGi3HZFAY3rdu5p3+PWKxmRv/0g+GdT5Yto9vN+bzyaZqfvt9InM+LeRwRQCjM55xg9N49KaxJJ9kFABoML2nxxeiR4qDn107iuyMaEb0S+SXt4wnwdn80VxNC1NZWcnGfaWUl5cTDLfPNFFzsBLXFy+jbfsCa3UOpW89jqUqp13OLbqvpKSTZ4c61fOnUl1dzR/+8Af8fj8ul4v58+czbdq0FpXZGBkJEKdkDlbgWv957eNA7g6oOIQS36/eHZWUOBuDe8Wz7UDNl56qKozsnxTR8weFEN2Pw1b/othpN2NSQnjWfYx3x0qiJ86iZOm7xEz+Aergi2hwir8O0yf0pNIdYM32Qvr3iOW2S4ZgOMl9k8PlXvblV2E1G9mwq+auY2EZbN6v8MRNV5Iek9Sku91pCTayM6LZd2QPAKNBYVDPOAyKwpCecQy6cSyqUlPH5nbhgWCQ0L7v0PN2oyRPRt37BZ6sURh7jyHK0rx9Zazew2gGC+DEoAcxewvx29PRTrivGTZFYe0zmsqvX4flYE7rh2apyWajoGMOVeM3RmMMe4CaPQGEOJVrr72W5557rsEpQVarleuuu65F5Z9zzjls3LiRyy+/HE3TuO666+pMD2pNEgSIU1Iq8/Ef2krUiPOx9h5F6cI5uNd/hn1qJsETRgPMBoU7LhvKrtwKXN4AfTNiyUy0yxxQIUSXp6hKbUaftHgb10wbwLtf7kLTdM4akU6f9GiCuoGoM2YTyN1B5bdvY+kxBOPAKQSO9IEuf4i9eVVUeQKkJUTRM9lBXJSZ+2b0JDQthbAtAXuglLAGIbXhO/m+QIgBPeJZ8O3eOsfDms7m3YVkx+l4LSmnbI/VaODeK0ew61AlHn+QfpmxpMcd69NVaHHffbDITbRPw7bnW/ocWo0W8OFOHcnuncWcPSL9tIMLi6eAsvnPYIhPxz7zHrSdyyn69i3ip9+NnjmmNhkGQBgj1rR+x97bYwhhkwNFAdPhLVR8+xbxl/4E94bPAQXzuFkNjnALcbxLL72UL774ot7iYKvVyvDhw5k5c2aLz/GTn/yEn/zkJy0u51QkCBCnFEwcQNI1vyIUlUrAYCfp6sfQrHEEGukso21Gxh6/sYwEAEKILkxHZ29+NYuWH8BuNTBjYm+yEu1MG5PBmAFJgE5slBmDoqCgEy4+QKiyGFQjgfzdKJX5KHF9qPKGmPPOBg4UHNt99+rz+jF9XCrB9QsJHN5H7Dk3UbroeZxjZxDMnopBVTlxWn96YhT7C+ru4BtlMzF6QDKpGfEErXFN7ndjbCbG9a+/EVhrUFWFZZsLyYxOZEJ8BlpZLmpqf74rcbJsxwHOGJKKxahiCVbgM8YCNXfoLaHK2seNCRzaRt6/foHmc0EjC6it4SrKFs3BnNYPa59RVC17l6SeQ/HF9kV1JqIHvBT95yFAIX7G/xBS228jM4OiET6SoUlFQ1GofXzi86JzMRqNvPDCCyxatIg333yzdp+A66+/nksuuaTNc/u3pq5TU9HuKj1B9uVX4QuG6ZGcSrrBhgL4nL2aXaaCjqoFajJHaD40gwW9kQ3MFEWh2hdE0+puilP/dTV1VVUFp9UkU4+EEK0qt8TLc++sZ8LQdJx2E+98sYtbZgzGZlY5UFBFhTtAdno0PeONOPCgqeAYdwlRQyYRLjqA7qsGXWdvfhUHC6uZPqk3FpNKOKzTK9FMiUcndfBZhN3lFL35K9ToRNzx/Vi4dB+KonDGkFSSYqzYTDXpTB0WI+MGpbJ1Xymb95ZyzphMLGYj3289zJa9JRRP7MWEgQnYbVaMmr/REYX20CfVxijXt2jl+SiDp6Ft+4JzU3dRmNYXg0HFVLqT0o9eJP6ynxGM6YUhfyNlS/5D/OUPErIlElItddrgt6eRMOthiv776JEAAOIuuI1w1hj0Ey6afYZo4i/9KZo5irA5muTMQQQdNZuVha2xWLIG4tmxEtVqx5DUk2ATszG1lMVXRGDXSkxDzidssGI8vIlQWT6WHsPxO7KweAsJ7FmFafA0go1MUTIRIIgZBQ1VCxLuwJ9xJDIajVxxxRUtygLUGUgQIBpU6vLz1H/WUH4k37SiwAPXjGJYr7hmzw1V0DEe3kgwfzfWURfjWbsIa8/hBJIH13ttWNf5fnsRr32ynVBI44opfThvTCYWY91O3h/SWLwmlwXf7sVoVLn54kGMH5jU6C6WQghxug6XebjynH58uHw/FdV+RvZPosLl58udRVQdyemfrpZh08NUbfoKa88hmKMT8G5dij9/L9FnX4uOQkW1j5mTs1m6IY+yKh+3T00ka+d8VOeVKEoVChA18EzcRXkcLguSmeJk0+4SfvPP74h1WLh79nCy0xys3l7Eqx9v58IzejK0TyI7c8pZt/PYju5vfLaTfQcTuGtmPwKr52HtM4ZA4qB2/9w0TSczNY5DhSOIn9iP1zcZuWVSH/ICUYwYlEpUqJKyT/+O5q2mdN7TRE+YTdm3b4GuUb30LaKGnY01fTCeNQuw9h5FIGkwBj2IL2cT6MeSULi3r8CZPoSAOa5eHXyOrNr/e6N7AzU3mMjbiGfHKmLPvZnq7xZS9cU/cUz/CQHadjTAormo/OgFgqW52N3lmNP6ogU8VC57D9XyEfEz7qV88SuEq0uItUShDjwf7YR8GxZPPq4V7+GcchPhor0ES/MwDL2QcDuOZIjuQcaaRD2KAmt3FNcGAFCzKOyVRVtbuPmXjh7wU73mI8refAz3+s/Rgz6UBsatC8q8/GPBFvyBMGFN5/2v9nDgSOaN4+0/XM28r/cQ1nT8gTAvL9hCQfmxzWg0HQ6VuPlyQz6L1+axv9BF6DQ24BFCiKQ4G/O/3lO7CdeGXcXsya1kxaZ8kuPsrNpcgMmgo3kq8eXuoGr1R5R98W/CrgrMKb3QLNEAZCU78QdClFXVzCO2mRS0inzCHz9NyQd/IuypRut7FoqrhFTPLipdAb7fVghAhcvPn95Zz+EKHy99UNM3Lly6D13XWbezqF6dV24vZf+mTbg3LUEP+hvsZ9tDz2QH1dZU/vh1iG05lTy5JESOL5oR2fH4jDHEXfYgqs2BHvBR+c0boGuYknriHDed0o/+WvNdseEL9FAABR1TZQ5V374JKMSfdxOGqFgCh7bhX/8RRqVp30+6rkPaEJKufhy931Tir3iI6Kk3t3kAABAwOIg574coRjOezV9T8fk/CZbmYes9Es3voWTeM4SrSzCn9cXQa3S9AABA0cL483ZS9u5vKPvoL2h+N4rewAuFOAUZCRD1qKrKnrzKescrXQG8/hB2U/N22NRRMWQNx5Tci2DRAcwZ/VEyhtZZyHXU8QHIUaVVPiCmzrGjX6Z13lvlJyPejqIorN9Twl/e31jn+ZsuHsSUEWntNPArhOjqTAYVty9U59ieQxX07xFHtSeApsOrK108frYRx/BzqF77CYrJgiUtG0NKX3ymWAB6pzlZvOZQbRmvLq/gN+dfgXHZyzXnGTeb+948zAPT/odd5Qb2l9Tthz2+EBXVgTrHTqzX8ar9Oqk9BqOkDULvoB5PVRXOGJjMwB5xVLj8RNmMJDgttWsWgo4MoideScWXr9a+J+bs69ATemBKzCJYcghL1pDaNoSiM3COm4kpqQcxw89GTRuEa+mbWEdehF9v+ndT0OgkGOMEHXz2tFZudeN0HUJxvYgacT6utR8D4MvfTcLFd+Pdv6H2dTFnXYvPHN9gGQFnBtHjL6Ny6VugGogaeSFeyWwkmkFGAkQ94bDG6P7189z2THUSbW9+rmiDEsa/8ROCRTlEDTmbQN4uQlu/QFXq36FKT7BjNNT99cxKdtZ7XWZS3U1/TEaVtCMb3VS4A/xz4ZZ67/nvp9spqaofZAghREMSYyxknNDXnDksjfPHZpEQbUVVFX56bjROc5jqtZ9g6z0CUHBvX4lv5wosnprNwFSo07f+/MJ4TCv/DakDMMQk4/78r9x6dgK/W1jI3OUFDOtTd8FuaoKd1HgbpuOmRaqKgqGBDcFUVSE5MRp/zhZCO7+t7WctngIs/hIURcPqysGoeeu9t7Xpuk60zUiPpCgSHMcCAAUdQ8FGKpa8Vuf1pYvmQP5WQpXFRA05G/+hrYS2f42qaIRUG4YRlxDOGoNiMOK3pxF14f/gt7TN4ubWphKGvStqAwAUFcegSZQseK7O60oXPofFdaiBEsBUupvKpW9j6zcexWSh4tO/YQ672rrqohuSkQDRoKG94xk3KIXV22uGoqOjzNxx2dCT7j55KmHdgKXfGZhSekPGcCy9RqDGpePXFSpcfg5XeHHazUSZDSRGW3jsh+NZuGwfPn+ISydn16QaPUFmYhS/uHEsC77di91m4rLJ2SQ6zeg6lFT58DcwfUnToajcS9IpNtIRQggAi0Hl/h+M4PPvD7Int5ILz+jB0F5xWIwqyXE2EmNt7CovJqVXInFTr0Pze3COuQjN66Jy3WdYhpxXW9aQXnGcOTSVVVsO881uH5ePvxpTz5FEmzUCh/eSm2umb2YMZwxNQ1F0brx4EEs35NE7PZqLz+xJgsPMY7eMZ8GyfXi8QYb3SSDGYea1j7fXqfO15/YirV9PFMs9qAlZ+HUFi+6h8rO/omsaztEXUbz4XyRc+hOUtBEdsouuOVRFxXFTgGoyI81B87pwb19B4qwHCcf1xtJzGIaELHxHFv6GVUud7EcBuk5fbgh58R7cDEDstNsxJmYSLthVM3UsrS+xF95F9bK38e5ZQ7g8H8WZWS95hu5MIW7arSg9xxLlLkb3VBA0RDV0OiFOStG7WCqV0lIXWjee052U5KS4uP7c944Q0nQOl3vxB8KkxtuJsjRvGtCJFEVB1/Xaf4ur/Tz/znrySzxER5l54JpR9Ew60qEpoKCcMuOPoiig6MevFeNwuZdHX1rZ4Ot/9cPx9EpxNPhce+lMP+v2JO1unbKao7v3n9C2v1+KqtRcLDfQHykqqIqKEvahBL2EzLGoeghjqBr/Cekuj+9b0xPttVl/1CM3WVz+EIeKXOw4UEZ8jJURfROJsZvRj/vZKSqg1/SNDqeNDbsK2bC7BHSdEf2SyE5zYlAUVFWp8zO3evIpefs36KEA9iFTsJx5dYdukmXxFuJe8S5Rk6/Db0nA4s7DvXoh9knXEjTHoet6vTYc1VX7ElOoGrUyj2BCP3TFgLnqIErAheZMJWCOxxSsQq0qIBjfB02pe6/2aJuP/0yOfpd2Z639s25uH3qivLy82hShGRkZrVImwJIlS3jxxRfxer1MmjSJxx57rNXKPp6MBHRnioJywoXx6TCqCpkJrf/lcLSz0nUdFPhw2X7yS2p2bKxyB/jb/E389rYzsBjUmp0qm7CgTdf1enmxk2KtTByWxorNBXWOD+gZS3qi3DURQpwe/SQBlK5BGA0wg8kMuk4YA+EG8t0f7VuDYR1/SEOH2k3IAOwmAwMyYhiUFVt77MRz1/TrNcdsViP9MmLonxkLul7ngvn4/yuKRri8AD0UBCBweA/2kLtDgwC/LQXbeXfip2aqqT8qA9vU2whgrg22ulvgGjQ6IWFgzQMd/M4edZ83RUNC9EnLOP4z6e4BQGe0bds2nnrqKfbv34/JZCIYDNK7d28effRRBg+un/HwdBw6dIhf//rXvPfeeyQkJHDzzTfzzTffMGXKlFaq/TESBHRDIU1n64FyPly+n5R4O5dM6k1qbNtlPWjsLk1TaDrsy6+76U1xuZdAUKsJAlrAoChcfV4/UhPsfLTiAOGwzrTxPZg2PguzQZYFCyEapygK3mAYg6Jgau3+QoE9+VW8+tEO8ktcjB6QxNXn9yfJWXdaS5P6VQXW7Szi/SW70TSdy8/Opk9qw3c5jSEv3l2riBp5PvahUyib/yx6dQmKNalDpgMdFcR80sdCdCbbtm3jjjvuqN0t2O+vWWO4Y8cO7rjjDl5++eUWBQKLFy9m+vTppKamAjBnzhwslraZ8iZBQDe0t6Ca59/dUPP/vEo27y3hyTsn4LC07o+7whNg455Siso9jB6QTO8Ux2nn5zcocMEZPfj3h9tqj501IgOntXXq6rQauXRiL84dnYkONVOa5KaJEOIkAmGNlVsLef+rPURHmfjhjCH0TXe2Wt9RUObl6dfX1l7kr9tZTG6xi9/eeka9vVBOpbDCx2//uaq2rO37S3nijgmkx9Xf0T1oiMI2+UZ01YjPYCf+6t/gN8Z0aAAgRFfz1FNP1QYAJ/L5fPz+97/n9ddfb3b5OTk5mEwm7rrrLgoKCpg6dSo/+clPml3eyUh2oG5GUWDXwfI6x6o9QQrLWzcDRCCs8dd5m/nPx9v5ZGUOT766mv2Fp5+dQNdh/MBk/ueqEYzqn8T1Fw5k1tTsVq2rpunYzQaizBIACCFObW9+Ff/5eDtub5CCEg9/+O8aSk9IzdkSOYer693lLyrzklfiPu2y8kvcdaf/6JBX3HhfHDRF107/8RljOyx1qBBdUV5eHvv37z/pa/bt20deXl6zzxEOh1m5ciVPPfUU77zzDps2bWL+/PnNLu9kJAjoZnSdeqnsDKpCrKN1h5KKK3zsya2bw3r19sLahW2nw2JUufDMXvzkqhGcPzqdaKuptaophBCnLf+Ei/FQWKe0svVupJyY/vioPbmV5JZ6TqusOGf9vj0+WnaOFaItFBcXYzKd/BrFZDJRXFzc7HMkJiYyYcIE4uPjsVqtnH/++WzatKnZ5Z2MBAHd0MAeMVxwRg9UBZx2E/dfPYoER+vOsbRajPVyUyfF2lq0QCkc1mRYWgjRbCGtZvfc73YWc7DY3aSkAg058UaK0aAS42i9C+vs9GjsJ0x5HNAjjt25FTzz37W4TrIB2ImyEqO4beYQLCYDJqPK9RcMoEdSx2Y+E6K7SkpKIhgMnvQ1wWCQpKT6ey011TnnnMOyZcuoqqoiHA6zdOlShgwZ0uzyTkbWBHRDdrORa87ty8Vn9sRkUImyGFr94jrBYeLWmUN4ZdFWNE2nZ6qT0f07dnFZR1MUsATL8BnjMehBjMFq/I3s+CiEaF2KovD9jkL+uXBr7bGfXjuaoT1jT7usvunRXHfhAD5ZcQCn3cyU0Zks35THrLNaZ6pivMPM47eMZ/Hqg+zLr2Jgzzh04LNVOQAUlHnol37y7DBHGVSFy6b0YUSfBHQg2mZs1X5Y03WqfSHsFmOL9okRojvIyMigd+/e7Nixo9HXZGdntyhd6IgRI7j99tu57rrrCAaDTJo0idmzZze7vJORIKCb8IU0yqp8OO3m2i+BGFvNkFXbXJgrTBiURL/MSXj9QZJjbae9oK27sbjyKHn/SeIuvptweQEVm74k9rKHJBAQoh14AiHe/HxnnWPvfrGLgbeOw3iaCQs0XeebdbkM65uI2xvkjc92oCpw0Rk9sZtbZ7+UlFgrYwcmU1ThZdnGfFzeY3cXDQaFjfvKCGs6WckOEqPNKCeZu68oCo4jIwut2d+XuQL8++NtbN1XRq+0aO66YijJMtWoW1MUMGh+QkrNNDMTAcnWdIJHH320Tnag41mtVh555JEWn+PKK6/kyiuvbHE5pyJBQDdQUu1nztsbKCh1Y7MY+ck1o+iX1nCKuLCms+9wNTmHq0mJt9EvIwarqfEvNVXRsFTloKhGQtZYjJ5iNJMdnzUVUEh0msEpHQRA2BKNtdcIyj54FoDos64hZJJheSHag6Io9ebam4wqiqKgoGNxHSRkT0bRNAy+UvzOrNqdWF3+EAVlHnIOV2NQFcZmR/PoxXHMWVzGpL52pvdP5L2NIaq8QVZuPUyc00L/zNjaC++m1Q9UVSUc1mofa8DhUnedAGDsoBQ+XZnDmh1FAKgK3HvVSEb0iW9SYoOQpuMNhLGZDRhbcudegUXL9rN1XxkABwqqeGXRNh66bjSSYbl7UhQwl+/Du/UbLGdchRqoxr3qfeyTrycgN7NqDR48mJdffpnf//737Nu3r3afgOzsbB555JEW7xPQniQI6OIUFT777iAFpTUL2bz+EH+du5Gn7pyIzVT/zvzmA2W88O7G2sdXTOnDpRN7Nnr3yOIppOitJ7APGA+Kgmf3GhKm3Yol04FflQvc42kGK6akLLw7a3YpNiZkElQkQBKiPdhMKrfMGFTbv6kKXHNBfwyA1VdI0Vu/JWrE+eihAJ5ty0i+8Sm8lmTCus6mvaW8v2QPFS4/qqow+soEQl88xz2jr0TN3YDhYBk/mflL7vnrCo5cwzNuUAp3XDoYQxNGGaq8QdbtKmbXoQrGDUphSM84jEaVd77YxcRh6QRDGkXlHjKTHWQmO/nL+8f6aE2Hv8zdyNM/nkT8SdZ26cDuvCre+HwHhwpd9Ex1cv2FA+mT5jjpKEJjAiGNzftK6xzbk1uBLxButd3jRediCPvxbPkKz7alhDwVhIoOEnaXYx8yBSU1PqKn+55o8ODBvP766222Y3B7kSCgy1M4UFB3s61KVwBfIFwvCNCBBd/uq3Ns0bJ9nDMqo9E7Wn5bCnHTbqV88SsAxE6ajZrQE58EAPWYPYcpWvYu0ROvIliaS/nHfyHx+ifxmeQOihBtTddhRO94nvmfyeQXVZMSH1W7SaLfmkzcBT+i/LOXAIif8T/4LDUL98qq/RRX+Khw1Wz4o2k6/13r50ejpuNf9z4Asdf8hv+uKa8NAKAmG9qsKX1IOcVGjLoO73y5m5VbDgOwasth7rpiGOMHJpGdHsPCpfuwWYzEOi3kF7soq/LXKyMU1skvcZ80CDhQ6OLp19fUPs45XM3vX1vNr249g55Jp79DusWocvaoDD74Zm/tsTOHpGKXAKDbCqkWrBN+QNhTiX9/TSAae+7NhFKGSADQiIyMjC558X9UZE/i7gZ0TeeCM+puOT5haCqx9voprBQF4k6Yz2m3mjCcZGdeQ9hLIG9X7eNgWQG6qwRVb3r2ikjht6eRdN0TqEOmYZt0HYlX/VLWAwjRjhRFYXDvBEZkJ9TZJd0Q8uLPP9aP+fN3Y9Bq5vOajAbUE7rAoekWwoV7ah+HS3LISKibitOgKpiasA6q0husDQCO+uy7HEBhxsReZCQ58PpDuL1BbrlkCKu2FDRYjvkk0zbDms7n3x+sd1zX4dsNeRiaMX9H1+HcURlceW4/0hOjmD6xF1ed2092FejmDH4XoeJDtY/9uTsxhE4vba3oOmQkoBsY0SeB+68eyfdbC+mXFcPo/kk0OEKtw+wpfdhxoAxfIIyqwI8uG4rdrDYa5RsC1XgPbCT2rB+g2qKpXPkBlqxBGMJeNGPD6w4ilaYY8TlqArKw0UzQ2Ho7jAohms8QdOHft574S+5HDwepWvYOtmHnE7LaiI0y0TstmhH9kti4uxhVVRiRYUDbl0/SNb/Gd2Aznh0rOWPKMOY5LZRX19ypv+GigcQ7Taf8G7eYVGIdltqRBoDs9BgUBRKdFh6/eSyFlTUjERt2F9MnI4YdOXU3fIx3Wsg4yd38UEijoJGNxnKLXIBCczojh9XIjDN6cOG4LAxqs4oQXYiJANUr3yPsLif2nJvx5+3Au2sV9gHjUdJHy2hAN6ToLUns3gFKS131dlrsTpKSnBQXV5/0NRo6Fa4gFpMBh/VY+k+DQUXTTp5rX1Gg3BOkqNxDnMN6yqwTANZQBaAQNEdj8ZcRxEjQFHN6DTuFprS7u4nENoO0u7XKao7u3n9C45+zNVSB3xiDgo45VIXPGFv7nK7rFFcHKKn0YjKq9Ex2EK3XvMaoB1CDHgLm2JoFxKUeHDYTKbFW1CZmHdpzuJo/vb0ejy9EWkIUD1wzksQjm3y5/SH++OY6Dh7Zcf2KqX3ZfaicHQfKuXhiL2xmA3arkd5p0WQmRjXYWyclOfnvJ9t454vd9Z676eKBnDMyvVtewEViX9LWbbYEytDLDhFKHYoh6EEp3kU4dQhhtWOzQrV2u5vbh3Y3MhLQxVT7Qrz9xS5WbjlMlM3Ej68YxuAesQC1WSdORtch1mYi1tb0i3ivMYZ9h118uHwzFpOB6RN70TOprVKPCiFE6zt60a+j1AkAoGYaUXK0heToY1N+fNS8JqSYwVwzF99hMTY5f//x+qU5+f1dE6n2BkhwWuukU95fUF0bAADM/3oPA3vFce9VI3hl0Vaq3IHa5x64ZhTDe8c12PeOH5TCN+vyOFx2bOpGRpKDkf0ie/8WcXr85niUtJpFwJrJiZIxRn5/ujEJArqYTXtLaueXur1Bnn93A8/cPanBNQCtJb/My1P/WV3bEazbWcxTd02ovZMlhBDdgaIoLdr1vE5ZKuSXesk5XI3ZZCA7PZr0OHu915VV1881nl/sZvO+0joBAMC/PtzaaOa3uCgzj9w4ltxSF8GQTpTNRFqcrdX2NRCR4/g/AQkAujcJAroQVVXIOXxsOExVICPRRrUnQKz96I+y9Zdt5Ra563QEobBGfom7UwUBqqqgKEqTRkOEEOJE5kA5evE+9IwRGCprFtkGYrNPOb3SH9IwqirHp+QvrvaxN6+KfyzYUvv+mCgzj94yjqQT+s34BjbfinNaKCqrvxiz0hXA4w9hMzWcJchsUtmTW8nCb/dhMqrcPH0Q4wYkNXnakhAiskh2oC5E03RG9qtJa6cqcN+FSfxs6GF6xqmYCjdjKtyMorR+2B7XwMV+bCcJABQF8su9/HfxLn732hqWbz2M2y+Zi4QQTacoCnrRHso++jOB79+mdN4zuFYvwqjVv0t/VCCk8e2mAn758ir+750N5B25aHcHwny1Lp/3l+ypE0BUugN8t/UwygkX5L1To+mTUXeKUX6Jm9EDkuuds09GDDEnGfXdV1DN/K/3EtZ0fIEwL32whcPl3qZ8BEKICCQjAV1M/8xo7v/BSJZuzKO/00fwy/n4inbgz9uJffBkLEn9CKm2Vj1nrxQHs6b2ZcG3e1FVhesuGEB6fP1h7Y5QWOHjf//1Pf5gGIDdhyqYMiqDGy8YQEs2yxRCRA5d19GzRhE1/DzcG75AtTqIPucWvErjiyF3HKrg3x9tB6C00sfTr63hqbsm1i4uLquqH0DsOliBYbJCKHQsOoiyGPifK0ewaU8pyzblkxpvZ+roTBJjrew8mMbKzTUpQ+Ojrdw2c8hJdwFu6Jzl1f5O018LIToXCQK6GKOqMiI7nlH9EjCgESi7ANf6z1GtDqImXImvlQMAALNR5ZIJPThreBqKohBjN3aaeYI7D1bUBgBHfbM+j4sn9CS5gWF2IYRoiKHiIJ7ty1CtUWg+F/69qzH0m0JYqX/nXVUVdh6sqHPM7QtRWuXDaTeTX+yid3o0+/PrbuQ4fnAKoVD9KYsxNhNnD09l6sh04FiSh1unD2TGxF74A2FS4+0NrgU4XmZy3TSiZqNKWsLpbxQmhIgMEgR0UbqmoxRtw7XhC0yJmQRLcvGs+xjT6MvbJpWXTu0wdGcJAAC8jUz9aeiLVgghGqXrWDIHEX3OLfh2r0IPh1AaSYyvaTp9M+pmWLOYDSSavCSYPEwclkaFy095tZ+KI/sKjOyfxIi+iSc7fb01TQZFIT2u6Td2shId/OLGsSz4di8Om4mZZ2WT4DR3qj77lBSo9oYwGBSizMZWW6gthKhPgoAuStdBiYrDMepCrGNmouVtBSCsNr6tfHMEwzrBsEaUxdApv0gGHEmPerzs9GiSYlt/REQI0X0F4voQdd6deFUbhoHnAhBWGu9PB/aM5cpz+rBw6X4SYyzcfkEWykf/i3/wZMaNvYpyT5hhfRKpdPkxG1VS4u0EQxpBTcfUSnMVPYEwhuPKUhUYkBHNL64bDYqOrrXspo2igKqF0FQjqh5CUWrWQvg1A1aTodXTUFR5Q3y86gBfrD6E1Wzg6vP6ccbgFMyN7GofKM1DJQqNmgxI1mAZflMcuuxrLESTdEgQ8OKLL/LJJ58AMGXKFB566KGOqEbrU6DCFUCnJl3b6TLrXvSQFT3gxR8MYrcY0U02dL2mQ9OAclcAk0Ehxm7Cb0/HNG42pT4IxwwjyezCEPJgNULQ48ZliCHW4MFrjEdRwBT24FdtlLuCGAwKBlXFFwgT5zBhUBQUNMxlu9EdSWgGC5TsY68vga15AUb3spOUnFgnv3VnkJXk4KfXjOI/H2+ntMrHuEEp/OC8fq32JSuE6CQUqHAH0TSdOIf5pJd5iqJjDHsJqnaMmhdNNddeKJ7sBKUBE/6AnzinhUp3AE33owDJdo3ygJFgWCfFoaOHdaJMJmac2ZOZY+Jxbf2W0OJXMCRmYBk5A7+m4LQacaLj8YXYlVvBlv1lhMM663cVMWtKHwZlxaIe108Zwj50gwkNA2bdS0Bp/EZGSXWAT1YdYOmGPKJsJmZN6Utqgh2zSSUjIQpjE3f3VRQwaTXnMug16UiPBj6KAqbSnYQK92POGIhWeRh/4QFM6QMorAjzVX4054/vcVrTLlU9hKKFCBusmDQvIYO19vsNBT5edYDPv6vJzOTxhfj3R9uJdlgY0Tv+WBlomF25KEYThz/8M/Hn3oAWnQGeckrm/4G4C+4glDZCAgEhmqDdg4AVK1awbNky5s+fj6Io3H777SxevJhp06a1d1VaVTCss2R9Hu8v2Y2u68w8K5uLx/fA3MSLZouvmOov/4l6xqUYq8sJ7F1POGMAhtgUSB+O2x/mva/38c26XMxGlR9eMpjxA1PYfNDNX+Zu5Ipx8Zx5+F3iz5hO+foviBpwJk6bneq963GOmU44FMa1ehGHsi/jqbkHMKgqF0/sxeY9JaQnRnHNef1INFRRsuh5TAmZGGJT8W5bysDLH8QXYyJhw7uYxs2CpL5t/EmeHlWBob3i+N2dZxIIakRZjZLySohuRtN0lm09zH8/3UlY05g2vgeXTe6NzVT/wl7XNYwFm/Ht/g7HxKvxrl2EMSUbQ+ZQDL5K/I7Mmqk3mk5RpY9gWCM13s6OnDJe+mALg3rFE+s089XaXAyqwvUX9MOgBXn9yxyCIY3bz09nAhswJmRhzhhI1eKXcPYbQ4XRRLD4EHrJfpSUIei6wvbcKp59Y23t3Xir2cBlZ/fh/95az8+vH8OgrJopRcawh8D6RZgSs7BmDKDq85dxTrkRvyOzXvtc/hB/fGMtxRU1WX8qXQH+/dE2LjqzJ99vK2T6hF6cOzq9SUGAuXwf1d/NJ+a82/Dt+R7CQQyDpxFWzFj9JRR/8H/ooQDR42YQLMvHu3c9ht2rSTvnh+z9rpTvtxfxm9vOaNI+NaoeQsn5nmDJIaJGXYxr+VtYB0wkmDIUUKj2hvhi9aF671uy5hCj+yYQDusoChiLtlE8/1kcI87DOWwKRfOfw953DCgKut9DxZf/IuGa3+Izxp36AxAiwrV7EJCUlMTDDz+M+cgOjH369CE/P7+9q9HqckvcvPPFrtrHC77dx+Ce8fTLaOLukoqCFvBQsugF9HAIa9Yg9LJDVFkScSoqu3LL+WZdLlAzHPvygi30SHXywnsb8AfCfLqxkiFnXUR0VQlhVxkVK+aiB3zY+4+neu0nePeuw5wxiF2HqtD1mlz/i5bu44qpfZn/9R6G901k/IAkEi5/kJJ3noD8XURPuZ6qr16ll89NGAhqYFU615qAoywGFUsjQ8ZCiK6toNzLq0cy8QB8/t1BhmUnMKRnQxd6CorBgGfnKnz71qP5vcSfewPBzYspX/cpSTc8RZUxgQXL9vPxigMADOmdQGqiHa8/RK/0aOZ9tQeAUFinyqsx/+v9taVv3F/FuN4GXIv/gWqxg2oASzQFEx4k9dBiAq4K1GQNf1jhtY+31ekvfYEwBwurSU2wM/frPTxywxgMCuiKimo0UvH5y0fKNNbcim9AzuHq2gDgeF+tzeW88T14+4tdjBmQdNJUorVUlUDBbkreeBTN7yH67Os4uteMz5xA3MU/pmzRC1St/qjmkzWaiRk/k0/2m9hfWFOH/QVVjOqTcMpT6YqKajDgXvcJ3q3foAX92AadXXs+g6pgMRnqrfNy2s1HXqPXTIONSceUko1r45c1LzAYsaT2pvK7D1FtTuKv+IUEAEI0UbsHAf369av9/4EDB/jkk0946623mvz+hARHW1SrxbYeqqh3zOULkZTkBCBYWYwpJolgZTFGZzyKWvcOlq47CJ9xBeUfvgBAVL+xlO9cy2qjk0tHWalw+U94Pbi8QfyBmsw4gZCGplpxbVmKY8hZVK5aAIBj9AUUvfMk6BrGkdNZ+GZRnXKOLkQrrvASZw9T/v0KQAFVxbPre+z9x1O1+iP0QdPICcQyKdHZ4s+qMUc/q0gSiW0GaXdH6az956nsPm6TxKOqvaFGP09Hr8G4MwfiO7gVU0I6YVc51d9/SNLMe4lK68H+fWW1AQDA1v2l9D8SUJyYVODExbqr97q49fzJsOEzNL8H+5mz+Tw/hv9+k8P5w8ZRuBp+NtCCIaRRWFb/Yr2wzENijI39+ZWYrUbio22Ak8DI86he8zGa30PsWT/AkdmHaGP9C3nv7pIG2+wPhjEc2fHYajU16XdNi+mLf8jZVK//HNViJ2bIREzxx6behGzDqU7pSbCw5rOyZY/Cu38DYx1lrMnIYFeeG4+/8Z/DiUKWUVTHpREqL8DaeziOXoMw2I+997oLBvDKoq21jxUFLp7Yi/j44zMcOVEnXUHRvP8DwJo1iEDhATSfi9izrsKRkU10I5updRcd3Y90lEhtd1vqsIXBu3fv5s477+Shhx6iV69eTX5faakLTet8t6KTY2yYjCrBI18gBlUhNcFGcXE11nAl5XOfImbK9VQue5v46f+D15Ze5/1WTx7lH/8VS8+haNVllC9/n5jxMzkrsAtPdV96ptb95XfYTKTE2emR4uRgYTWzz0ggZutrxJ51FSWL/oKt9wgCpbmUfvRX4s+/BdeWb/F++jw/vegn/H5hzciL9bjt5Af0iMNdUUWgNJ/4S+5FsUdTufgVVHs0ev8pKNsXM6jPEEpKEttkJCApyUlxcf0v+u4sEtsM0u7WKqs5Omv/eSpJ0VZsFmPtXWJFgYykqAY/z8QEOxWrP8V3cBuOMRfhWr+YkKsca9Zg/KX5hIrLqaz213vfURazAVWBox+TAkRZjbh9Nee+7qwUtK//hmqxYes7FvequYw65z7eNap8sbkSgPwiF8mxVsYOTGbNjro3Xgb0iOOb9bmMG5xK2B+iuLgai+aicsEcVEsUtr6jqVj6LsaUPvjjB9RvX0zDc/CT42xUuv1cdnYfLAZO+bumKArGvDVUr/+cqJHT8GxbStHHL+G44G4Cig1T2I1v1Ts1AYCigq7j2fUdjnEzCccN5NDKmmAkKcbWpN9rI0ECq94mVFmEY8zFuNZ+QsXaz2DQhbWf9bgBSThsI/lyzSGio8ycPy6L9DhrnfIt5bspWfA8isGEGhWD78BmHEPPxpI5kIql74EtFr3XmU1YA9I1Sf/ZeuWJDgoC1q5dy3333cejjz7KjBkzOqIKrS45xsLjPxzP4u8PEgxrXHBGD9LibKCD3xhL9OSrKV3wHI6R5xO01d8JMmSNJ/7iOzHEpRPyenBXlhOyR2GNTiSkKfRMiuKRm8by2Xc5xEdbOW9sFtFWI/deNYKlG/NZd6iSsefeg+o0E3/h7YRjs7CiYaguRE8dgDNjGHrJPtyWNMYNCmO3mRjeJ5FVWwp46PoxZKc4CKgKjgvuJmiIAkUhdvrd5Be72OwcwDkXD8WQ1Itw17t+EEJ0cbF2E4/dMo4v1hzC5QlywRk9yEpoeOGsohowZY8lISEDY1w6Jmc8poQsAhWFVK9aQFy/CWQkxhIfbaGsqiYYMBpUhvdJIBAIcbjExU+vG8O3G3IxGQxkpTr45TUD+XJLBRWuAP16JhE74BqwxRC2J0LqAD7apxA4cgMoKdZKrMOMCvzg/H4cLvWQW+wCYFifRAIhDUVRuOiMHrV19qsOos++FsXqJGxLxNJjCFp0WoPty0iwc/64rDrz52s2cRyI3WqgR7KjSesBdF1HScwmfsa96Bn/396dx0lVn/ke/5xau3pf6I1eAWn2HRVwAST2NTar0YhmMIbMJLnXV5wk9w4xhmtmMiZRbl5xifEmNyZmcV7jK8YtMmKcIRCNC8giKCog0DQNTdP0vtZ2fvePhoJm7dbqharv+y+quuqc8zTVT53n/H7n+U3BN/YqsG0Cju7vLSvURfB4NZbbS9ZN3ybQ2kjLy4/RVlfD68dH0ekPMWdSPiW5vRtdCuHGN2kBCaOvIJw9hqyCsVgZw/GfdqweV/c6ONNHZwHWWaMwAJY3GVdSBmlzlhFuqaN97xa8hWMI292j4q2bXiSjYAJ+d3qvjksknllmgJvw1tTUsGzZMh566CFmz57d5/cP9StZzhPz0k9PXt5QM00vPIC3YAydezaTdctquhKHn/Xe+jY/2/bUkZeZyIQRmTg5e/690+nAGNPjd2BOdM041tBBXmZid+eMyHxSQ/jEmbt1Yqj45Dag+8vjQj31T8Zj23a/3gsQj1c34jFmUNzR2tYnMdTz58U4HBaWde6Tw5NO/p6tE/cvJYQa8LsycJgw7mAzXZ7u+ev1bQHe23ecrkCYyaOGUZDlw7KsSE7szn0msj/L6t539+/PRPJhVyjMX7Ye5tXNVZTkpXD79WPITT91tT4Qsqmqa8cfDNPY2oXTYVFWlEFWcs8pK9Zp91udzNXn4w/ZVNa2sqeqieREN5MvyyYr2d2rk/8zXWhf3kADlr8Vf2oJFgZHw34q23xsPRSgrCid0ryUc96cfeH9nWhx/SnuL0sIt2DjJLDtRRInzqPLl4cr2ELXOy/gm34j/oSzL7TFCuXP6G1PBqEIuP/++3n22WcpLj51FWT58uXcdtttvXr/pfglZlk2Ce01BJNycbXXEkrIJHTGyr7tgTD3/b+3aTwx9/8fFk9g9vjcXmwc/vbeUX699gMAEhNcfPfOy8m/xPrkx2Nii8eYQXFHa1ufxKWYP/vqXL9np90JlpOw5cFjtxN0Jp12wv3pmx1YFnQFbdxOBxfqTnyhfVkWBG2D02Fh9fJ4Tm5voP+movE7i4asNDf1zcHIY5cVImRie/kj5c/obU8GYTrQ6tWrWb169UDvdlAZ46AzsQAMhBILzvp5IGRz+Hg7c2cUcriujXc+qOXI8faLXg2C7hGA3637KPK4oyvEX7ZUs6K8LOa/7EVEzsXpdERGC5x2J6HtL+FISMZXdiWNLz9G2twv4E/vbnccjZNZY+jVGirn21cwbLNt73FefH0/2ek+Pr9gNAWZiZ94e/1tKBQAAA5PAnCqCIj1AkAk2vQXM8g6A2F+9R8fsG13HQDjSjOZP72QOZPze7VcejhsEzpjaLy5PXBiOpDBckAoDC7HxQsKEZFLWXNHkG1vV+IPhMnPSqQwKwmnw4UzJYvmjb+nddMLWG4vuIfWSOmewy384oX3ATha38G+w8388KtzSPXpK1pE+o8aqw+yg7VtkQIA4MPKBqaNyen1dJ6MFC/XzTy1oIxlQfkVxYTDNo3tAZ7+y8fc98QmXt5cRdsZ/ZdFRGJFyDZ8cLCRjVur+eWL7/PYH3fy7v56QpYbz4ipWO4ETChA4rg5hJOGzpxxy4KPq5t6PNfRFeJ489ntRUVEokmXGQaZPxg+67lQ+OznztTSGaLqWCsel5NFV49g8qhsmtv9lOSmUJSdhG0M//bq7kiB8cz6vWDgxiuLNSIgIjGnpTPIurcqqT7WRrKvu53o48/u5Gd3zyCw/v9iudwkTy+nddOfyCoch5U/NarTWtzBZhwtRwhlleFqPwrGxp9cdNH3GQPFuT3nJ7ucDjJSvNE7OBGRc1ARMMhK8lJITfLQ0h4AICXRTUnehVcZbmgL8MPfvkPDiV7XM8fm8PcLx+M5bU5qmz/MgcPNFA1L4NDxLsYUJvP2rhquv7wQ13lWohQRuVQ1tfmpPtZGWrKbVfM8BC0Pf620cB/aRur8FYQtF6GkXIYNL8POKI5qAWBZFtTu5vjLj5N61S00bF2HO7uI5Bv+kYB17r7+pxtbnM6yuaN46W/7SUvy8pWlE8lIiu0Fr0Rk8KkIGGTpiW7+952Xs/tQEwaYMjqbZPeFZ2ntrW6KFAAAWz46xo1zSinNOdWv2ed18N3PeLA7mtncXsBV7g8wwyfhdlhD5qYuEZFoCNk2gZDN4mtGMj7PRcpH/w61e1laOJnOv24hcen/JJg7CWMgnD0x6vs3xmAKppAwYS4tbzyDIyEJz9yVBB0JvWrb6XM7WTSnhOtmFOJyWnidmqkrIv1PRcAAuVBLtawUL3NOtAM9sw3W6R2CTv7bPseGzuwE5DUh0v01tG95nrnDSrGPV5I2vAiDoXsNzMExVFrLiUhsMMB/bT3MH9bvBeBVj5MH7/g7rJf+BSq3kDhzIeHcCRfNO582N9ktxwjs2woOF7a/k/rdO3CVXU1a0sVHAk4GkuSJvVVue9PlTkQGh4qAftbUEeS9/fU0tvqZMCKTEXnJOHpxEt7eFWLXwUYO17UxtSyb9s4gew41UZqXyojhaST53LR3drdGG1eayfBhST3eH3Z4SJx8Pf4P/0boeCW+sstxlswgaAanAGjpDPH+gXrqmjqZNHIYpblJODQtSUQ+pca2AH/8y97I44lFPhwfv4Yd7MKdXYhduw9XUxXhtFK8/uMQ6CCUXoLn+EeQkErAm0VlbQs7DraTluRl0qgs0hPdfToGy4JgezMmOQt71p0MsxppPHIEn78WLy78SWe3ho51YWM4cLSN9/fXk5fpY8KILFISdMohMpToLzLKDBAM2SR4nHQEwjz6zLtU1nRf2X/xtX38w9JJTCzJIPkCydAAz762j43bDuPzunA6Hbzw132Rn980bxTf//srOVDTgsftZEReKgln9Kh2mgBd764j1FKHp6CMzj3vkDByOlbJLMwAFwJB2/DLP+1i14F6AF58bT/3fvFyLsvXYh0i0pPDYbBti+4ux2EM5786HrINbf4Qpw+EdgQMeJNh8b+wsQp2flzP1YfczEgMYG9bS8fut0m/ZjnHNj5F6rRyavOu4p9/vzsyClCcm8w9K2aelVMvxBiodpWwv2A5M1qO0vDaL3BdcQtdr/6UkC+V1MX/hN8aWm1J+9vHh1t48KmtkcfTx2bz35dMxKmLPyJDhoqAKKpr8fPMho/5uLqJz8wsYvLoYZECALq/KPZVN+HAcMWY87eoa2oP8tfthwGYdNkw3n6/psfP//T6fuZPL2T6ZcPOu42w5cFTPIGMrEIcxVMJfbgBZ1YxoUEYCTje3BUpAE56670jlBWM1YJmIhLh9dcT2PMGngkLoOkIdnsDdtFM7HMUAu3+EL97ZTcNrV3MnpTPW+9158mPDnfgXzyPZ/96kLfeP9r9XFUTQdtB+RXL8B/6iMb1v8GTfxkGw+bd9T2mAVXVtnG0oaPHPVa9kZ+ZxNu73PyxxsPnRl1FaPMzWO4E0pauoivOCgCn08GG7dU9ntv2UR318/3kpPVyepSI9DsVAVESCNs8+swODte1AfDHDR/j9bhI8rlo7zzVnz/B42JfdROzxuWe9wTY43KQ6HXR3hWiuc1PZmoCR+s7Ij9PT/bivtDa9CePKWssVpZNCAeO8eX4zcDfbBYMGyzL4pqpw3nrvaORhc2yMxI1T1REeupooOXtF/AcfJ9g7QE8xeNJGj4B23n2CfnHh1t458NaADJTElg27zIyUryMK83E6bQiBcBJ//XOIeaPKCDUchyAYH01SWOuJDN4dl70efr+1ehxObh57igcbTV0vLwbGzDBLkLH9uMozjpnIROrjDHkZfacoupxOfC64+d3IHIpUAuCKGluD0YKgJPe+bCWlQvHc3L0s/DElaVJo7IveAU8OcHJV5ZOwuW02H2wkcvH55J0YvqQx+XgK0sn9WgHeiHmxH+xPQgFwOGGDn74+y3c+/M32XOwkeXXl5Ge7CU/K4krxuXqBmER6SGYNZq0a24lcGQPxhjS5t9J8BwFAEDbiXuioDvXPr/xY4alJTC2NBOfy0FeVmKP15cVpcHxSnwjp5J354O4M4eDZTFt/HBKc7tfa1lw+/VjyE77ZD36XU6LhGALDoeTnC+uIXnGjfgPfYgj7L/4m2OIbRuunpxPdnr3VX+Hw+LLiyeQntS3ey1EpH9Z5hK7HFtf3zYkp5D4wzb3/XITdU2nVnm87foxXD9zOFV1HVTVttLQ3MXw7GQmj8zEc54WcJHuQBbUtwZobQ+Qk+EjbBuON3eRkewh/RLoH93SGeTeX7xFR9epURCnw+K7X7qc3HQfvjOuCJ3ZFSkexGPMoLijta1PYqjmz5O8zfup/+OPcKZlE2o8im/MLHxXfYGgM+ms1x5u6OT//NtWFlzevUK61+NkzsQ8LivOpK6ulUP1HTz89HYaW/2U5qfyP26aTG4yOMJdBF2p+AJ1hNypBC0v4UAn1c02Po+T7DQv1qfooGZZBm+olS5nKi67E8sOE3T1bWpRXw3Vv6nOQJjapk6SfW6GpXiIdme6oRp3f4rHmCH6cX/SHBprNB0oSrxOB99cPo1fr/2Ayppm5s8oYtaEXDAWxcOSKMlJ7r7y3duay0BWsoes5FMn/CkJ/ftFEk1Hjnf0KAAAwrahvrmL0uxLJw4RGUDeZJImziVh5hJMfRXhlmPYjnNflS/I8vH1W6aw5qmtBEPd0wxDIZuS4ekAFGUlcv9XZtHWGSI92YPbYREGwq7unNrpyY5sy+nxUZJ95h4+GWMsupzdCz6GHL64Hm/3eZx9vrdCRAaOioAoyktP4NtfmI4/GMbncUauefhDNofq2uj0hyjOSSGtj+3nLkXu80xX8ro0J1REzs2fkIN71m34jRMrdzxWzhjs83xNOSyLN9+riRQAcyblY4DXtlczPCuJo/VtJHhdFGUn9+oeKhGReKMiIMqcFiSetuBLyDY8vX5vpNtPSqKb1V+6guyUTzbn9FJRmJ3E6OJ09lY1RZ7LSfdRkqchOBE5v5Dpzp/GWJiLfEWdnNo0a2Ie9S1dvHmiQ1BWWgKzJ+Wz9m8HmD+9kNuuH41LrSlFRHqI44HKgVHb2BkpAABaO4K8v7/+Au+IDV6Xg7uWTeKOz45l4sgsbv3MaFb93QwtFiMiUWHbhutmFOFyWuRlJbH7YGPkZ/XNXbhO3He1YVs1tY1dg3WYMkQ57S4cdI8ieUx83bgtcpLOyPqZfY57AMLhoXtjXjSl+tzMnzqcz8woJBw2agkqIlFVnJ3Ev35lNofr2i/4OuUeOZ3L7iT07lrcOaV4ckfSsv7XpFz7BfyJwwf70EQGlIqAfpaX4WPm2By2fHQMgASPk0kjswb5qAaOMd0364mIRJsxhty0BFITPYwuTGNvdTMAqUmeSA+Gy8flkpveu8W6LMvgCPsJOxJw2n5shwcT5Y42MgQYgx0O0fDy4zh8SZHnROKNioB+5nY6uPPGscybXkinP0RJXgrZqV7lGxGRKPG5HXz95ikcONqCwSI73UfN8Tb+1+3TKc1Lwe3szYm8wV37AV0Hd5I4czFdO9bhzh9NKG+yCoEYE3ImkjS1nPZ3/xO7s420a28nmJQ/2IclMuBUBAyARI+L8cXpkccqAEREois5wcWk0sxIP/H8EwtV9ZaFwQS7aN/+Kl17NhFubyLjhq8BNsTRar/xwBtupfE/HsWRkEhCyWSaX/t3huWOwJ8xerAPTWRAqQgQEZF+4TAhnLafoDMJd7idsMOLbQ29rx3LAne4EwomkjxlAaHGIziS0nEWTyHUDwVAbXMX1cfaSE30UJybjLeXK8BLdARcyaTOuQUrKQPbl4m3dBJ2cu5gH5bIgBt62VhERIYEb8cRcHq6hy/tYJ9unLSwcVS9Q2flTlKuvpXWvz1NQukUTMmVmDMa09U2d/He/nqy03yMK0k/74rq/cGywF2/h9Y3/kD61TeDHcSVkQ8NR/C/uw7ntCWETfQKgUPHO/jXJzcROtEgomJOKTddO+JTrVIsfWOMRTB3YmRU3iqZo5vHJS6pCBARkbM4LAgd3Uvb1nUAJM/4LI5Rw7F7vei5A0diGp17N9O1byvGDpM4/lpCZxQArV0hfvjbd2jtCALwj5+fypSRmVGN5YLHacDyJBJqrKHu+Z9gQgEyPvMlvCOnYyWm449iAeBwWGzcXh0pAABefquSudMKGZbiucA7JdpOP+dXASDxSmOQIiJyFtuAp3AC4dYGwq0NeAon9LoAiMgqwT2sGBMK4B5WDFklZ72k0x+KFAAAxxo7P+WR952dlIOv7EpMKIDDm4iraBLBnIn4kwqivi/HGYuWWXSPRoiIDDQVASIichaHBf4D20iZ/TlSZn8O/4HtOPpwsuq0bAI71hE8XkXqVTcTPF5FYMc6nFbPlsEZyR7+25XdxUGSz82kUQPbQtmygCM7ad/5F1JmLQHLouU/f4nb7oj6vmzbMG96Ae7T7gFYfM1IsjQKICKDQNOBRETkLLYB1+hrMCdv5DUh+rLOYdg48I6bi7d0MsGsMobll0FiBn7T89qT2+ngpmtHMG9aAYkJrgFfVdwYsHJGM2zZPxHKGUvmyMvBhPFbvVtboK8KsxL5wVdnc+hYG6lJHoqyk0CzUURkEKgIEBGRcwo5TjsRttx9fr8/IRsSssEGf2bZeV/ndjrI7WNLz2gKuNMgOw0MhJML+3VfxsCwFC/DUrz9uh8RkYvRdCARERERkTijIkBEREREJM6oCBARERERiTMqAkRERERE4oyKABERERGROKMiQEREREQkzqgIEBERERGJMyoCRERERETizCW3WJijL+vWX6LiIcZzice44zFmUNzxuv+BEi9xni4eY4b4jDseY4b4jbs/WcYYLVguIiIiIhJHNB1IRERERCTOqAgQEREREYkzKgJEREREROKMigARERERkTijIkBEREREJM6oCBARERERiTMqAkRERERE4oyKABERERGROKMiQEREREQkzqgIGGRtbW0sXLiQ6upqAN58800WLVpEeXk5Dz300CAfXf947LHHqKiooKKigjVr1gDxEfcjjzzCjTfeSEVFBU8++SQQH3EDPPjgg9xzzz1AfMS8YsUKKioqWLJkCUuWLGHHjh1xEfdgiLccqvyp/BnrMSt/DiAjg+bdd981CxcuNBMmTDCHDh0ynZ2dZu7cuaaqqsoEg0GzcuVKs3HjxsE+zKh64403zK233mr8fr8JBALmjjvuMC+99FLMx71p0yazfPlyEwwGTWdnp5k/f7758MMPYz5uY4x58803zZVXXmm+/e1vx8Vn3LZtc/XVV5tgMBh5Lh7iHgzxlkOVP5U/Yz1m5c+BpZGAQfSHP/yB733ve+Tk5ACwc+dOSkpKKCoqwuVysWjRIl555ZVBPsroys7O5p577sHj8eB2uxk1ahSVlZUxH/cVV1zB7373O1wuF/X19YTDYVpaWmI+7qamJh566CG+9rWvAfHxGd+/fz8AK1euZPHixTz11FNxEfdgiLccqvyp/BnrMSt/DiwVAYPoBz/4ATNnzow8PnbsGNnZ2ZHHOTk51NbWDsah9ZvRo0czdepUACorK1m3bh2WZcV83ABut5tHH32UiooKZs+eHRf/3/fddx/f/OY3SU1NBeLjM97S0sLs2bP52c9+xm9+8xuefvppjhw5EvNxD4Z4y6HKn8qfsR6z8ufAUhEwhNi2jWVZkcfGmB6PY8nevXtZuXIlq1atoqioKG7ivvvuu3nrrbeoqamhsrIypuN+5plnyM/PZ/bs2ZHn4uEzPm3aNNasWUNKSgqZmZncfPPNPProozEf91AQD58vUP5U/uwWazGD8udAcw32AcgpeXl51NXVRR7X1dVFhrljydatW7n77ru59957qaioYPPmzTEf9759+wgEAowbNw6fz0d5eTmvvPIKTqcz8ppYi/vll1+mrq6OJUuW0NzcTEdHB4cPH47pmAG2bNlCMBiMfHkbYygoKIj5z/hQEA85VPlT+fOkWIsZlD8HmkYChpApU6Zw4MABDh48SDgcZu3atVx77bWDfVhRVVNTw1133cWPf/xjKioqgPiIu7q6mtWrVxMIBAgEAqxfv57ly5fHdNxPPvkka9eu5cUXX+Tuu+/muuuu44knnojpmAFaW1tZs2YNfr+ftrY2nn/+eb71rW/FfNxDQaznEuVP5c9YjhmUPweaRgKGEK/XywMPPMDXv/51/H4/c+fO5YYbbhjsw4qqX/3qV/j9fh544IHIc8uXL4/5uOfOncvOnTtZunQpTqeT8vJyKioqyMzMjOm4zxQPn/H58+ezY8cOli5dim3b3H777UybNi3m4x4KYv3zpfyp/Bnr/9fKnwPLMsaYwT4IEREREREZOJoOJCIiIiISZ1QEiIiIiIjEGRUBIiIiIiJxRkWAiIiIiEicUREgIiIiIhJnVASIiIiIiMQZFQES01auXElDQ8Onfs2mTZtYuHDhRfc3ZsyYc25r/fr13H///QCsWLGCV155herqaqZNm3bRbYqIDAblT5HYpsXCJKa98cYbUXnNp7VgwQIWLFjQ7/sREYkW5U+R2KaRAIlZ3/nOdwD44he/yObNm1mxYgWLFi1i8eLFvPDCC2e9pqamhg0bNrB8+XJuuukm5s2bx8MPP9zn/T788MMsW7aMJUuWsGHDBgCee+45vvrVr0YlLhGR/qb8KRL7NBIgMetHP/oRzz33HL/97W/5/Oc/z6pVqygvL6e2tpZbbrmFkpKSHq/JyMhg1apVPPDAA5SWllJbW8v8+fO54447+rTfwsJCvv/977Nnzx5WrFjBunXr+ilCEZH+ofwpEvtUBEjM27dvH36/n/LycgByc3MpLy/n9ddf7zGn1LIsfv7zn7Nx40bWrl3Lvn37MMbQ2dnZp/3ddtttAJSVlTFq1Ci2b98evWBERAaQ8qdI7NJ0IIl5lmVhWVaP54wxhEKhHs91dHSwbNkydu3axfjx41m1ahUulwtjTJ/253Cc+rOybRuXS7W2iFyalD9FYpeKAIlpTqeTgoICXC4Xr776KgC1tbX8+c9/Zs6cOZHXhEIhDh48SFtbG9/4xje47rrr2LRpE4FAANu2+7TP559/HoBdu3ZRVVXFlClTohuUiMgAUP4UiW0qsSWm3XDDDdx55508/vjj3H///fz0pz8lHA5z1113MWvWrMhrVqxYwSOPPMK8efP47Gc/i8fjoaysjMsuu4yDBw/i8Xh6vc9Dhw6xdOlSLMviJz/5Cenp6f0UnYhI/1H+FIltlunrWJ2IiIiIiFzSNBIg0gdPPPEEL7300jl/9uUvf5nFixcP8BGJiFwalD9FhhaNBIiIiIiIxBndGCwiIiIiEmdUBIiIiIiIxBkVASIiIiIicUZFgIiIiIhInFERICIiIiISZ/4/+yMhMUUXWKIAAAAASUVORK5CYII=\",\n \"text/plain\": [\n \"
\"\n ],\n \"text/plain\": [\n \" VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count \\\\\\n\",\n \"0 1.0 2021-01-01 00:30:10 2021-01-01 00:36:12 1.0 \\n\",\n \"1 1.0 2021-01-01 00:51:20 2021-01-01 00:52:19 1.0 \\n\",\n \"2 1.0 2021-01-01 00:43:30 2021-01-01 01:11:06 1.0 \\n\",\n \"3 1.0 2021-01-01 00:15:48 2021-01-01 00:31:01 0.0 \\n\",\n \"4 2.0 2021-01-01 00:31:49 2021-01-01 00:48:21 1.0 \\n\",\n \"... ... ... ... ... \\n\",\n \"1369760 NaN 2021-01-25 08:32:04 2021-01-25 08:49:32 NaN \\n\",\n \"1369761 NaN 2021-01-25 08:34:00 2021-01-25 09:04:00 NaN \\n\",\n \"1369762 NaN 2021-01-25 08:37:00 2021-01-25 08:53:00 NaN \\n\",\n \"1369763 NaN 2021-01-25 08:28:00 2021-01-25 08:50:00 NaN \\n\",\n \"1369764 NaN 2021-01-25 08:38:00 2021-01-25 08:50:00 NaN \\n\",\n \"\\n\",\n \" trip_distance RatecodeID store_and_fwd_flag PULocationID \\\\\\n\",\n \"0 2.10 1.0 N 142 \\n\",\n \"1 0.20 1.0 N 238 \\n\",\n \"2 14.70 1.0 N 132 \\n\",\n \"3 10.60 1.0 N 138 \\n\",\n \"4 4.94 1.0 N 68 \\n\",\n \"... ... ... ... ... \\n\",\n \"1369760 8.80 NaN NaN 135 \\n\",\n \"1369761 5.86 NaN NaN 42 \\n\",\n \"1369762 4.45 NaN NaN 14 \\n\",\n \"1369763 10.04 NaN NaN 175 \\n\",\n \"1369764 4.93 NaN NaN 248 \\n\",\n \"\\n\",\n \" DOLocationID payment_type fare_amount extra mta_tax tip_amount \\\\\\n\",\n \"0 43 2.0 8.00 3.00 0.5 0.00 \\n\",\n \"1 151 2.0 3.00 0.50 0.5 0.00 \\n\",\n \"2 165 1.0 42.00 0.50 0.5 8.65 \\n\",\n \"3 132 1.0 29.00 0.50 0.5 6.05 \\n\",\n \"4 33 1.0 16.50 0.50 0.5 4.06 \\n\",\n \"... ... ... ... ... ... ... \\n\",\n \"1369760 82 NaN 21.84 2.75 0.5 0.00 \\n\",\n \"1369761 161 NaN 26.67 2.75 0.5 0.00 \\n\",\n \"1369762 106 NaN 25.29 2.75 0.5 0.00 \\n\",\n \"1369763 216 NaN 28.24 2.75 0.5 0.00 \\n\",\n \"1369764 168 NaN 20.76 2.75 0.5 0.00 \\n\",\n \"\\n\",\n \" tolls_amount improvement_surcharge total_amount \\\\\\n\",\n \"0 0.0 0.3 11.80 \\n\",\n \"1 0.0 0.3 4.30 \\n\",\n \"2 0.0 0.3 51.95 \\n\",\n \"3 0.0 0.3 36.35 \\n\",\n \"4 0.0 0.3 24.36 \\n\",\n \"... ... ... ... \\n\",\n \"1369760 0.0 0.3 25.39 \\n\",\n \"1369761 0.0 0.3 30.22 \\n\",\n \"1369762 0.0 0.3 28.84 \\n\",\n \"1369763 0.0 0.3 31.79 \\n\",\n \"1369764 0.0 0.3 24.31 \\n\",\n \"\\n\",\n \" congestion_surcharge \\n\",\n \"0 2.5 \\n\",\n \"1 0.0 \\n\",\n \"2 0.0 \\n\",\n \"3 0.0 \\n\",\n \"4 2.5 \\n\",\n \"... ... \\n\",\n \"1369760 0.0 \\n\",\n \"1369761 0.0 \\n\",\n \"1369762 0.0 \\n\",\n \"1369763 0.0 \\n\",\n \"1369764 0.0 \\n\",\n \"\\n\",\n \"[1369765 rows x 18 columns]\"\n ]\n },\n \"execution_count\": 7,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"modin_df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Faster Append with Modin's ``concat``\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Our previous ``read_csv`` example operated on a relatively small dataframe. In the following example, we duplicate the same taxi dataset 100 times and then concatenate them together.\\n\",\n \"\\n\",\n \"Please note that this quickstart notebook is assumed to be run on a machine that has enough memory in order to be able to perform the operations both with pandas and Modin in a single pipeline (which at least doubles the amount of required memory). If your machine doesn't have enough resources to execute every cell of the notebook and you see an OOM issue, you most likely need to reduce ``N_copies`` in the cell below.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Time to concat with pandas: 34.144 seconds\\n\"\n ]\n }\n ],\n \"source\": [\n \"N_copies= 100\\n\",\n \"start = time.time()\\n\",\n \"\\n\",\n \"big_pandas_df = pandas.concat([pandas_df for _ in range(N_copies)])\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to concat with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 9,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Time to concat with Modin: 0.564 seconds\\n\"\n ]\n },\n {\n \"data\": {\n \"text/markdown\": [\n \"### Modin is 60.57x faster than pandas at `concat`!\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n }\n ],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"big_modin_df = pd.concat([modin_df for _ in range(N_copies)])\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to concat with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `concat`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"The result dataset is around 19GB in size.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 10,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"\\u001b[2m\\u001b[36m(apply_list_of_funcs pid=73415)\\u001b[0m \\n\",\n \"\\u001b[2m\\u001b[36m(apply_list_of_funcs pid=73416)\\u001b[0m \\n\",\n \"\\n\",\n \"Int64Index: 136976500 entries, 0 to 1369764\\n\",\n \"Data columns (total 18 columns):\\n\",\n \" # Column Non-Null Count Dtype \\n\",\n \"--- --------------------- ------------------ ----- \\n\",\n \" 0 VendorID 127141300 non-null float64\\n\",\n \" 1 tpep_pickup_datetime 136976500 non-null datetime64[ns]\\n\",\n \" 2 tpep_dropoff_datetime 136976500 non-null datetime64[ns]\\n\",\n \" 3 passenger_count 127141300 non-null float64\\n\",\n \" 4 trip_distance 136976500 non-null float64\\n\",\n \" 5 RatecodeID 127141300 non-null float64\\n\",\n \" 6 store_and_fwd_flag 127141300 non-null object\\n\",\n \" 7 PULocationID 136976500 non-null int64\\n\",\n \" 8 DOLocationID 136976500 non-null int64\\n\",\n \" 9 payment_type 127141300 non-null float64\\n\",\n \" 10 fare_amount 136976500 non-null float64\\n\",\n \" 11 extra 136976500 non-null float64\\n\",\n \" 12 mta_tax 136976500 non-null float64\\n\",\n \" 13 tip_amount 136976500 non-null float64\\n\",\n \" 14 tolls_amount 136976500 non-null float64\\n\",\n \" 15 improvement_surcharge 136976500 non-null float64\\n\",\n \" 16 total_amount 136976500 non-null float64\\n\",\n \" 17 congestion_surcharge 136976500 non-null float64\\n\",\n \"dtypes: float64(13), datetime64[ns](2), int64(2), object(1)\\n\",\n \"memory usage: 19.4 GB\\n\"\n ]\n },\n {\n \"name\": \"stderr\",\n \"output_type\": \"stream\",\n \"text\": [\n \"UserWarning: Distributing object. This may take some time.\\n\"\n ]\n }\n ],\n \"source\": [\n \"big_modin_df.info()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Faster ``apply`` over a single column\\n\",\n \"\\n\",\n \"The performance benefits of Modin becomes aparent when we operate on large gigabyte-scale datasets. For example, let's say that we want to round up the number across a single column via the ``apply`` operation. \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 11,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Time to apply with pandas: 43.969 seconds\\n\"\n ]\n }\n ],\n \"source\": [\n \"start = time.time()\\n\",\n \"rounded_trip_distance_pandas = big_pandas_df[\\\"trip_distance\\\"].apply(round)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to apply with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 12,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Time to apply with Modin: 1.225 seconds\\n\"\n ]\n },\n {\n \"data\": {\n \"text/markdown\": [\n \"### Modin is 35.88x faster than pandas at `apply` on one column!\"\n ],\n \"text/plain\": [\n \"\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n }\n ],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"rounded_trip_distance_modin = big_modin_df[\\\"trip_distance\\\"].apply(round)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to apply with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `apply` on one column!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Summary\\n\",\n \"\\n\",\n \"Hopefully, this tutorial demonstrated how Modin delivers significant speedup on pandas operations without the need for any extra effort. Throughout example, we moved from working with 100MBs of data to 20GBs of data all without having to change anything or manually optimize our code to achieve the level of scalable performance that Modin provides.\\n\",\n \"\\n\",\n \"Note that in this quickstart example, we've only shown ``read_csv``, ``concat``, ``apply``, but these are not the only pandas operations that Modin optimizes for. In fact, Modin covers [more than 90% of the pandas API](https://github.com/modin-project/modin/blob/main/README.md#pandas-api-coverage), yielding considerable speedups for many common operations.\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.9\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/spreadsheet/requirements.txt",
"content": "ray==1.1.0\ngit+https://github.com/modin-project/modin\ngit+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5"
},
{
"path": "examples/spreadsheet/tutorial.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## modin.spreadsheet\\n\",\n \"`modin.spreadsheet` is a Jupyter notebook widget that allows users to interact with Modin DataFrames in a spreadsheet-like fashion while taking advantage of the underlying capabilities of Modin. The widget makes it quick and easy to explore, sort, filter, edit data and export reproducible code. \\n\",\n \"\\n\",\n \"This tutorial will showcase how to use `modin.spreadsheet`. Before starting, please install the required packages using `pip install -r requirements.txt` in the current directory. Then just run the cells; no editing required!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Please install the required packages using `pip install -r requirements.txt` in the current directory\\n\",\n \"# For all ways to install Modin see official documentation at:\\n\",\n \"# https://modin.readthedocs.io/en/latest/installation.html\\n\",\n \"import modin.pandas as pd\\n\",\n \"import modin.spreadsheet as mss\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Create a Modin DataFrame\\n\",\n \"The following cells creates a DataFrame using a NYC taxi dataset.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"columns_names = [\\n\",\n \" \\\"trip_id\\\", \\\"vendor_id\\\", \\\"pickup_datetime\\\", \\\"dropoff_datetime\\\", \\\"store_and_fwd_flag\\\",\\n\",\n \" \\\"rate_code_id\\\", \\\"pickup_longitude\\\", \\\"pickup_latitude\\\", \\\"dropoff_longitude\\\", \\\"dropoff_latitude\\\",\\n\",\n \" \\\"passenger_count\\\", \\\"trip_distance\\\", \\\"fare_amount\\\", \\\"extra\\\", \\\"mta_tax\\\", \\\"tip_amount\\\",\\n\",\n \" \\\"tolls_amount\\\", \\\"ehail_fee\\\", \\\"improvement_surcharge\\\", \\\"total_amount\\\", \\\"payment_type\\\",\\n\",\n \" \\\"trip_type\\\", \\\"pickup\\\", \\\"dropoff\\\", \\\"cab_type\\\", \\\"precipitation\\\", \\\"snow_depth\\\", \\\"snowfall\\\",\\n\",\n \" \\\"max_temperature\\\", \\\"min_temperature\\\", \\\"average_wind_speed\\\", \\\"pickup_nyct2010_gid\\\",\\n\",\n \" \\\"pickup_ctlabel\\\", \\\"pickup_borocode\\\", \\\"pickup_boroname\\\", \\\"pickup_ct2010\\\",\\n\",\n \" \\\"pickup_boroct2010\\\", \\\"pickup_cdeligibil\\\", \\\"pickup_ntacode\\\", \\\"pickup_ntaname\\\", \\\"pickup_puma\\\",\\n\",\n \" \\\"dropoff_nyct2010_gid\\\", \\\"dropoff_ctlabel\\\", \\\"dropoff_borocode\\\", \\\"dropoff_boroname\\\",\\n\",\n \" \\\"dropoff_ct2010\\\", \\\"dropoff_boroct2010\\\", \\\"dropoff_cdeligibil\\\", \\\"dropoff_ntacode\\\",\\n\",\n \" \\\"dropoff_ntaname\\\", \\\"dropoff_puma\\\",\\n\",\n \" ]\\n\",\n \"parse_dates=[\\\"pickup_datetime\\\", \\\"dropoff_datetime\\\"]\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df = pd.read_csv('s3://modin-datasets/trips_data.csv', names=columns_names,\\n\",\n \" header=None, parse_dates=parse_dates)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Generate a spreadsheet widget with the DataFrame\\n\",\n \"`mss.from_dataframe` takes in a DataFrame, optional configuration options, and returns a `SpreadsheetWidget`, which contains all the logic for displaying the spreadsheet view of the DataFrame. The object returned will not be rendered unless displayed.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"spreadsheet = mss.from_dataframe(df)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Displaying the Spreadsheet\\n\",\n \"The widget is displayed when the widget is returned by an input cell or passed to the `display` function e.g. `display(spreadsheet)`. When displayed, the SpreadsheetWidget will generate a transformation history cell that contains a record of the transformations applied to the DataFrame unless the cell already exists or the feature is disabled.\\n\",\n \"\\n\",\n \"### Basic Usage\\n\",\n \"`from_dataframe` creates a copy of the input DataFrame, so changes do not alter the original DataFrame.\\n\",\n \"\\n\",\n \"**Filter** - Each column can be filtered according to its datatype using the filter button to the right of the column header. Any number of columns can be filtered simultaneously.\\\\\\n\",\n \"**Sort** - Each column can be sorted by clicking on the column header. Assumptions on the order of the data should only be made according to the latest sort i.e. the 2nd last sort may not be in order even if grouped by the duplicates in the last sorted column.\\\\\\n\",\n \"**Cell Edit** - Double click on a cell to edit its value.\\\\\\n\",\n \"**Add Row**(toolbar) - Click on the `Add Row` button in the toolbar to duplicate the last row in the DataFrame.\\\\\\n\",\n \"**Remove Row**(toolbar) - Select row(s) on the spreadsheet and click the `Remove Row` button in the toolbar to remove them.\\\\\\n\",\n \"**Reset Filters**(toolbar) - Click on the `Reset Filters` button in the toolbar to remove all filters on the data.\\\\\\n\",\n \"**Reset Sort**(toolbar) - Click on the `Reset Sort` button in the toolbar to remove any sorting on the data.\\n\",\n \"\\n\",\n \"### Transformation History and Reproducible Code\\n\",\n \"The widget records the history of transformations, such as filtering, that occur on the spreadsheet. These transformations are updated in the `spreadsheet transformation history` cell as they happen and can be easily copied for reproducibility. The history can be cleared using the `Clear History` button in the toolbar.\\n\",\n \"\\n\",\n \"**Try making some changes to the spreadsheet!**\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": true\n },\n \"outputs\": [],\n \"source\": [\n \"spreadsheet\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Exporting Changes\\n\",\n \"`to_dataframe` takes in a `SpreadsheetWidget` and returns a copy of the DataFrame reflecting the current state of the UI on the widget. Specifically, any filters, edits, or sorts will be applied on the returned Dataframe.\\n\",\n \"\\n\",\n \"**Export a DataFrame after making some changes on the spreadsheet UI**\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"changed_df = mss.to_dataframe(spreadsheet)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"changed_df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## SpreadsheetWidget API\\n\",\n \"The API on `SpreadsheetWidget` allows users to replicate some of the functionality on the GUI, but also provides other functionality such as applying the transformation history on another DataFrame or getting the DataFrame that matches the spreadsheet state like `to_dataframe`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Duplicates the `Reset Filters` button\\n\",\n \"spreadsheet.reset_filters()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Duplicates the `Reset Sort` button\\n\",\n \"spreadsheet.reset_sort()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"qgrid6f69f373-ae0e-423e-8e26-429f52e1669d\": true\n },\n \"outputs\": [],\n \"source\": [\n \"# Duplicates the `Clear History` button\\n\",\n \"spreadsheet.clear_history()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Gets the modified DataFrame that matches the changes to the spreadsheet\\n\",\n \"# This is the same functionality as `mss.to_dataframe`\\n\",\n \"spreadsheet.get_changed_df()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Retrieving and Applying Transformation History \\n\",\n \"The transformation history can be retrieved as a list of code snippets using the `get_history` API. The `apply_history` API will apply the transformations on the input DataFrame and return the resultant DataFrame.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"scrolled\": false\n },\n \"outputs\": [],\n \"source\": [\n \"spreadsheet.get_history()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"another_df = df.copy()\\n\",\n \"spreadsheet.apply_history(another_df)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Additional Example\\n\",\n \"Here is another example of how to use `from_dataframe` with configuration options.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"mss.from_dataframe(df, show_toolbar=False, grid_options={'forceFitColumns': False, 'editable': False, 'highlightSelectedCell': True})\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.5\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n"
},
{
"path": "examples/tutorial/README.md",
"content": "# Modin tutorial Jupyter Notebooks\n\nTutorial for how to use different features of Modin.\n"
},
{
"path": "examples/tutorial/jupyter/README.md",
"content": "# Jupyter notebook examples to run with Modin\n\nCurrently we provide tutorial notebooks for the following execution backends:\n\n- [PandasOnRay](https://modin.readthedocs.io/en/latest/development/using_pandas_on_ray.html)\n- [PandasOnDask](https://modin.readthedocs.io/en/latest/development/using_pandas_on_dask.html)\n- [PandasOnMPI through unidist](https://modin.readthedocs.io/en/latest/development/using_pandas_on_mpi.html)\n\n## Creating a development environment\n\nTo get required dependencies for `PandasOnRay`, `PandasOnDask` and `PandasOnUnidist` Jupyter Notebooks\nyou should create a development environment with `pip`\nusing `requirements.txt` file located in the respective directory:\n\n```bash\npip install -r execution/pandas_on_ray/requirements.txt\n```\n\nto install dependencies needed to run notebooks with Modin on `PandasOnRay` execution or\n\n```bash\npip install -r execution/pandas_on_dask/requirements.txt\n```\n\nto install dependencies needed to run notebooks with Modin on `PandasOnDask` execution or\n\n```bash\npip install -r execution/pandas_on_unidist/requirements.txt\n```\n\nto install dependencies needed to run notebooks with Modin on `PandasOnUnidist` execution.\n\n**Note:** Sometimes pip is installing every version of a package. If you encounter that issue,\nplease install every package listed in `requirements.txt` file individually with `pip install `.\n\n## Run Jupyter Notebooks\n\nA Jupyter Notebook server can be run from the current directory as follows:\n\n```bash\njupyter notebook\n```\n\nNavigate to a concrete notebook (for example, to the `execution/pandas_on_ray/local/exercise_1.ipynb`).\n\n**Note:** Since there are some specifics regarding the run of jupyter notebooks with the `Unidist` engine,\nrefer to [PandasOnUnidist](https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/execution/pandas_on_unidist/README.md) document\nto get more information on the matter."
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_dask/Dockerfile",
"content": "FROM continuumio/miniconda3\n\nRUN conda install -c conda-forge psutil setproctitle\nRUN pip install -r requirements-dev.txt\n\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 5: Setting up cluster environment\\n\",\n \"\\n\",\n \"**GOAL**: Learn how to set up a Dask cluster for Modin, connect Modin to a Dask cluster and run pandas queries on a cluster.\\n\",\n \"\\n\",\n \"**NOTE**: This exercise has extra requirements. Read instructions carefully before attempting. \\n\",\n \"\\n\",\n \"**This exercise instructs users on how to start a 500+ core Dask cluster, and it is not shut down until the end of exercise.**\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Often in practice we have a need to exceed the capabilities of a single machine. Modin works and performs well \\n\",\n \"in both local mode and in a cluster environment. The key advantage of Modin is that your python code does not \\n\",\n \"change between local development and cluster execution. Users are not required to think about how many workers \\n\",\n \"exist or how to distribute and partition their data; Modin handles all of this seamlessly and transparently.\\n\",\n \"\\n\",\n \"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Extra requirements for AWS authentication\\n\",\n \"\\n\",\n \"First of all, install the necessary dependencies in your environment:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!pip install dask_cloudprovider[aws]\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"The next step is to setup your AWS credentials, namely, set ``AWS_ACCESS_KEY_ID``, ``AWS_SECRET_ACCESS_KEY``\\n\",\n \"and ``AWS_SESSION_TOKEN`` (Optional) (refer to [AWS CLI environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) to get more insight on this):\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import os\\n\",\n \"\\n\",\n \"os.environ[\\\"AWS_ACCESS_KEY_ID\\\"] = \\\"\\\"\\n\",\n \"os.environ[\\\"AWS_SECRET_ACCESS_KEY\\\"] = \\\"\\\"\\n\",\n \"os.environ[\\\"AWS_SESSION_TOKEN\\\"] = \\\"\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Starting and connecting to the cluster\\n\",\n \"\\n\",\n \"This example starts 1 scheduler node (m5.24xlarge) and 6 worker nodes (m5.24xlarge), 576 total CPUs. Keep in mind the scheduler node manages cluster operation but doesn't perform any execution.\\n\",\n \"\\n\",\n \"You can check the [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/on-demand/) page.\\n\",\n \"\\n\",\n \"Dask cluster can be deployed in different ways (refer to [Dask documentaion](https://docs.dask.org/en/latest/deploying.html) to get more information about it), but in this tutorial we will use the ``EC2Cluster`` from [dask_cloudprovider](https://cloudprovider.dask.org/en/latest/) to create and initialize a Dask cluster on Amazon Web Service (AWS).\\n\",\n \"\\n\",\n \"**Note**: EC2Cluster uses a docker container to run the scheduler and each of the workers. Probably you need to use another docker image depending on your python version and requirements. You can find more docker-images on [daskdev](https://hub.docker.com/u/daskdev) page.\\n\",\n \"\\n\",\n \"In the next cell you can see how the EC2Cluster is being created. Set your ``key_name`` and modify AWS settings as required before running it.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from dask_cloudprovider.aws import EC2Cluster\\n\",\n \"\\n\",\n \"n_workers = 6\\n\",\n \"cluster = EC2Cluster(\\n\",\n \" # AWS parameters\\n\",\n \" key_name = \\\"\\\", # set your keyname\\n\",\n \" region = \\\"us-west-2\\\",\\n\",\n \" availability_zone = [\\\"us-west-2a\\\"],\\n\",\n \" ami = \\\"ami-0387d929287ab193e\\\",\\n\",\n \" instance_type = \\\"m5.24xlarge\\\",\\n\",\n \" vpc = \\\"vpc-002bd14c63f227832\\\",\\n\",\n \" subnet_id = \\\"subnet-09860dafd79720938\\\",\\n\",\n \" filesystem_size = 200, # in GB\\n\",\n \"\\n\",\n \" # DASK parameters\\n\",\n \" n_workers = n_workers,\\n\",\n \" docker_image = \\\"daskdev/dask:latest\\\",\\n\",\n \" debug = True,\\n\",\n \" security=False,\\n\",\n \")\\n\",\n \"\\n\",\n \"scheduler_adress = cluster.scheduler_address\\n\",\n \"print(f\\\"Scheduler IP address of Dask cluster: {scheduler_adress}\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"After creating the cluster you need to connect to it. To do this you should put the ``EC2Cluster`` instance or the scheduler IP address in ``distributed.Client``.\\n\",\n \"\\n\",\n \"When you connect to the cluster, the workers may not be initialized yet, so you need to wait for them using ``client.wait_for_workers``.\\n\",\n \"\\n\",\n \"Then you can call ``client.ncores()`` and check which workers are available and how many threads are used for each of them.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from distributed import Client\\n\",\n \"\\n\",\n \"client = Client(cluster)\\n\",\n \"# Or use an IP address connection if the cluster instance is unavailable:\\n\",\n \"# client = Client(f\\\"{scheduler_adress}:8687\\\")\\n\",\n \"\\n\",\n \"client.wait_for_workers(n_workers)\\n\",\n \"client.ncores()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"After successful initialization of the cluster, you need to configure it.\\n\",\n \"\\n\",\n \"You can use plugins to install any requirements into workers:\\n\",\n \"* [InstallPlugin](https://distributed.dask.org/en/stable/plugins.html#distributed.diagnostics.plugin.InstallPlugin)\\n\",\n \"* [PipInstall](https://distributed.dask.org/en/stable/plugins.html#distributed.diagnostics.plugin.PipInstall)\\n\",\n \"* [CondaInstall](https://distributed.dask.org/en/stable/plugins.html#distributed.diagnostics.plugin.CondaInstall).\\n\",\n \"\\n\",\n \"You have to install Modin package on each worker using ``PipInstall`` plugin.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from dask.distributed import PipInstall\\n\",\n \"\\n\",\n \"client.register_plugin(PipInstall(packages=[\\\"modin\\\"]))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"If you need an additional workers configuration, you can create your own [WorkerPlugin](https://distributed.dask.org/en/stable/plugins.html#worker-plugins) or function that will be executed on each worker upon calling ``client.run()``.\\n\",\n \"\\n\",\n \"**NOTE**: Dask cluster does not check if this plugin or function has been called before. Therefore, you need to take this into account when using them.\\n\",\n \"\\n\",\n \"In this tutorial a CSV file will be read, so you need to download it to each of the workers and local machine with the same global path.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from dask.distributed import Worker\\n\",\n \"import os\\n\",\n \"import urllib\\n\",\n \"\\n\",\n \"def dataset_upload(file_url, file_path):\\n\",\n \" try:\\n\",\n \" dir_name = os.path.dirname(file_path)\\n\",\n \" if not os.path.exists(dir_name):\\n\",\n \" os.makedirs(dir_name)\\n\",\n \" if os.path.exists(file_path):\\n\",\n \" return \\\"File has already existed.\\\"\\n\",\n \" else:\\n\",\n \" urllib.request.urlretrieve(file_url, file_path)\\n\",\n \" return \\\"OK\\\"\\n\",\n \" except Exception as ex:\\n\",\n \" return str(ex)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Set the directory where it should be downloaded (the local directory will be used by default):\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"directory_path = \\\"./\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Then you need to run `dataset_upload` function on all workers. As the result, you will get a dictionary, where the result of the function execution will be for each workers:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"file_path = os.path.join(os.path.abspath(directory_path), \\\"taxi.csv\\\")\\n\",\n \"client.run(dataset_upload, \\\"https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv\\\", file_path)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You have to also execute this function on the local machine:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"dataset_upload(\\\"https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv\\\", file_path)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Congratulations! The cluster is now fully configured and we can start running Pandas queries.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Executing in a cluster environment\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Same as local mode Modin on cluster uses Ray as an execution engine by default so no additional action is required to start to use it. Alternatively, if you need to use another engine, it should be specified either by setting the Modin config or by setting Modin environment variable before the first operation with Modin as it is shown below. Also, note that the full list of Modin configs and corresponding environment variables can be found in the [Modin Configuration Settings](https://modin.readthedocs.io/en/stable/flow/modin/config.html#modin-configs-list) section of the Modin documentation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Modin engine can be specified either by config\\n\",\n \"import modin.config as cfg\\n\",\n \"cfg.Engine.put(\\\"dask\\\")\\n\",\n \"\\n\",\n \"# or by setting the environment variable\\n\",\n \"# import os\\n\",\n \"# os.environ[\\\"MODIN_ENGINE\\\"] = \\\"dask\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now you can use Modin on the Dask cluster.\\n\",\n \"\\n\",\n \"Let's read the downloaded CSV file and execute such pandas operations as count, groupby and map:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import time\\n\",\n \"\\n\",\n \"t0 = time.perf_counter()\\n\",\n \"\\n\",\n \"df = pd.read_csv(file_path, quoting=3)\\n\",\n \"df_count = df.count()\\n\",\n \"df_groupby_count = df.groupby(\\\"passenger_count\\\").count()\\n\",\n \"df_map = df.map(str)\\n\",\n \"\\n\",\n \"t1 = time.perf_counter()\\n\",\n \"print(f\\\"Full script time is {(t1 - t0):.3f}\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Shutting down the cluster\\n\",\n \"\\n\",\n \"Now that we have finished computation, we can shut down the cluster:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"cluster.close()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### This ends the cluster exercise\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.10.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_dask/local/exercise_1.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 1: How to use Modin\\n\",\n \"\\n\",\n \"**GOAL**: Learn how to import Modin to accelerate and scale pandas workflows.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Modin is a drop-in replacement for pandas that distributes the computation \\n\",\n \"across all of the cores in your machine or in a cluster.\\n\",\n \"In practical terms, this means that you can continue using the same pandas scripts\\n\",\n \"as before and expect the behavior and results to be the same. The only thing that needs\\n\",\n \"to change is the import statement. Normally, you would change:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"import pandas as pd\\n\",\n \"```\\n\",\n \"\\n\",\n \"to:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"import modin.pandas as pd\\n\",\n \"```\\n\",\n \"\\n\",\n \"Changing this line of code will allow you to use all of the cores in your machine to do computation on your data. One of the major performance bottlenecks of pandas is that it only uses a single core for any given computation. Modin exposes an API that is identical to pandas, allowing you to continue interacting with your data as you would with pandas. There are no additional commands required to use Modin locally. Partitioning, scheduling, data transfer, and other related concerns are all handled by Modin under the hood.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"
\\n\",\n \"
pandas on a multicore laptop\\n\",\n \" \\n\",\n \" Modin on a multicore laptop\\n\",\n \" \\n\",\n \"\\n\",\n \"
\\n\",\n \"\\n\",\n \"
\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for exercise: setting Modin engine\\n\",\n \"\\n\",\n \"Modin uses Ray as an execution engine by default so no additional action is required to start to use it. Alternatively, if you need to use another engine, it should be specified either by setting the Modin config or by setting Modin environment variable before the first operation with Modin as it is shown below. Also, note that the full list of Modin configs and corresponding environment variables can be found in the [Modin Configuration Settings](https://modin.readthedocs.io/en/stable/flow/modin/config.html#modin-configs-list) section of the Modin documentation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Modin engine can be specified either by config\\n\",\n \"import modin.config as cfg\\n\",\n \"cfg.Engine.put(\\\"dask\\\")\\n\",\n \"\\n\",\n \"# or by setting the environment variable\\n\",\n \"# import os\\n\",\n \"# os.environ[\\\"MODIN_ENGINE\\\"] = \\\"dask\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for exercise: Dataframe constructor\\n\",\n \"\\n\",\n \"Often when playing around in pandas, it is useful to create a DataFrame with the constructor. That is where we will start.\\n\",\n \"\\n\",\n \"```python\\n\",\n \"import numpy as np\\n\",\n \"import pandas as pd\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\\n\",\n \"df = pd.DataFrame(frame_data)\\n\",\n \"```\\n\",\n \"\\n\",\n \"When creating a dataframe from a non-distributed object, it will take extra time to partition the data. When this is happening, you will see this message:\\n\",\n \"\\n\",\n \"```\\n\",\n \"UserWarning: Distributing object. This may take some time.\\n\",\n \"```\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Note: Do not change this code!\\n\",\n \"import numpy as np\\n\",\n \"import pandas\\n\",\n \"import sys\\n\",\n \"import modin\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas.__version__\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin.__version__\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Implement your answer here. You are also free to play with the size\\n\",\n \"# and shape of the DataFrame, but beware of exceeding your memory!\\n\",\n \"\\n\",\n \"import pandas as pd\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\\n\",\n \"df = pd.DataFrame(frame_data)\\n\",\n \"\\n\",\n \"# ***** Do not change the code below! It verifies that \\n\",\n \"# ***** the exercise has been done correctly. *****\\n\",\n \"\\n\",\n \"try:\\n\",\n \" assert df is not None\\n\",\n \" assert frame_data is not None\\n\",\n \" assert isinstance(frame_data, np.ndarray)\\n\",\n \"except:\\n\",\n \" raise AssertionError(\\\"Don't change too much of the original code!\\\")\\n\",\n \"assert \\\"modin.pandas\\\" in sys.modules, \\\"Not quite correct. Remember the single line of code change (See above)\\\"\\n\",\n \"\\n\",\n \"import modin.pandas\\n\",\n \"assert pd == modin.pandas, \\\"Remember the single line of code change (See above)\\\"\\n\",\n \"assert hasattr(df, \\\"_query_compiler\\\"), \\\"Make sure that `df` is a modin.pandas DataFrame.\\\"\\n\",\n \"\\n\",\n \"print(\\\"Success! You only need to change one line of code!\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now that we have created a toy example for playing around with the DataFrame, let's print it out in different ways.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for Exercise: Data Interaction and Printing\\n\",\n \"\\n\",\n \"When interacting with data, it is very imporant to look at different parts of the data (e.g. `df.head()`). Here we will show that you can print the modin.pandas DataFrame in the same ways you would pandas.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Print the first 10 lines.\\n\",\n \"df.head(10)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Print the DataFrame.\\n\",\n \"df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Free cell for custom interaction (Play around here!)\\n\",\n \"df.add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df.count()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**Please move on to [Exercise 2](./exercise_2.ipynb) when you are ready**\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_dask/local/exercise_2.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 2: Speed improvements\\n\",\n \"\\n\",\n \"**GOAL**: Learn about common functionality that Modin speeds up by using all of your machine's cores.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for Exercise: `read_csv` speedups\\n\",\n \"\\n\",\n \"The most commonly used data ingestion method used in pandas is CSV files (link to pandas survey). This concept is designed to give an idea of the kinds of speedups possible, even on a non-distributed filesystem. Modin also supports other file formats for parallel and distributed reads, which can be found in the documentation. We will import both Modin and pandas so that the speedups are evident.\\n\",\n \"\\n\",\n \"**Note: Rerunning the `read_csv` cells many times may result in degraded performance, depending on the memory of the machine**\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"import time\\n\",\n \"from IPython.display import Markdown, display\\n\",\n \"\\n\",\n \"def printmd(string):\\n\",\n \" display(Markdown(string))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Dataset: 2015 NYC taxi trip data\\n\",\n \"\\n\",\n \"\\n\",\n \"We will be using a version of this data already in S3, originally posted in this blog post: https://matthewrocklin.com/blog/work/2017/01/12/dask-dataframes\\n\",\n \"\\n\",\n \"**Size: ~1.8GB**\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"path = \\\"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Modin execution engine setting:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.config as cfg\\n\",\n \"cfg.Engine.put(\\\"dask\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## `pandas.read_csv`\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"pandas_df = pandas.read_csv(path, parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to read with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Expect pandas to take >3 minutes on EC2, longer locally\\n\",\n \"\\n\",\n \"This is a good time to chat with your neighbor\\n\",\n \"Dicussion topics\\n\",\n \"- Do you work with a large amount of data daily?\\n\",\n \"- How big is your data?\\n\",\n \"- What’s the common use case of your data?\\n\",\n \"- Do you use any big data analytics tools?\\n\",\n \"- Do you use any interactive analytics tool?\\n\",\n \"- What’s are some drawbacks of your current interative analytic tools today?\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## `modin.pandas.read_csv`\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_df = pd.read_csv(path, parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to read with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `read_csv`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Reduces\\n\",\n \"\\n\",\n \"In pandas, a reduce would be something along the lines of a `sum` or `count`. It computes some summary statistics about the rows or columns. We will be using `count`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"pandas_count = pandas_df.count()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"\\n\",\n \"print(\\\"Time to count with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_count = modin_df.count()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to count with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `count`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_count\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_count\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Map operations\\n\",\n \"\\n\",\n \"In pandas, map operations are operations that do a single pass over the data and do not change its shape. Operations like `isnull` and `applymap` are included in this. We will be using `isnull`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"pandas_isnull = pandas_df.isnull()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"\\n\",\n \"print(\\\"Time to isnull with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_isnull = modin_df.isnull()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to isnull with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `isnull`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_isnull\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_isnull\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Apply over a single column\\n\",\n \"\\n\",\n \"Sometimes we want to compute some summary statistics on a single column from our dataset.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"rounded_trip_distance_pandas = pandas_df[\\\"trip_distance\\\"].apply(round)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to groupby with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"rounded_trip_distance_modin = modin_df[\\\"trip_distance\\\"].apply(round)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to add a column with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `apply` on one column!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"rounded_trip_distance_pandas\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"rounded_trip_distance_modin\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Add a column\\n\",\n \"\\n\",\n \"It is common to need to add a new column to an existing dataframe, here we show that this is significantly faster in Modin due to metadata management and an efficient zero copy implementation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"pandas_df[\\\"rounded_trip_distance\\\"] = rounded_trip_distance_pandas\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to groupby with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_df[\\\"rounded_trip_distance\\\"] = rounded_trip_distance_modin\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to add a column with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas add a column!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**Please move on to [Exercise 3](./exercise_3.ipynb) when you are ready**\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_dask/local/exercise_3.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 3: Not Implemented\\n\",\n \"\\n\",\n \"**GOAL**: Learn what happens when a function is not yet supported in Modin as well as how to extend Modin's functionality using the DataFrame Algebra.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"When functionality has not yet been implemented, we default to pandas\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"We convert a Modin dataframe to pandas to do the operation, then convert it back once it is finished. These operations will have a high overhead due to the communication involved and will take longer than pandas.\\n\",\n \"\\n\",\n \"When this is happening, a warning will be given to the user to inform them that this operation will take longer than usual. For example, `DataFrame.mask` is not yet implemented. In this case, when a user tries to use it, they will see this warning:\\n\",\n \"\\n\",\n \"```\\n\",\n \"UserWarning: `DataFrame.mask` defaulting to pandas implementation.\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Default to pandas\\n\",\n \"\\n\",\n \"In this section of the exercise we will see first-hand how the runtime is affected by operations that are not implemented.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"import numpy as np\\n\",\n \"import time\\n\",\n \"import modin.config as cfg\\n\",\n \"cfg.Engine.put(\\\"dask\\\")\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**18, 2**8))\\n\",\n \"df = pd.DataFrame(frame_data).add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df = pandas.DataFrame(frame_data).add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_start = time.time()\\n\",\n \"\\n\",\n \"print(df.mask(df < 50))\\n\",\n \"\\n\",\n \"modin_end = time.time()\\n\",\n \"print(\\\"Modin mask took {} seconds.\\\".format(round(modin_end - modin_start, 4)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_start = time.time()\\n\",\n \"\\n\",\n \"print(pandas_df.mask(pandas_df < 50))\\n\",\n \"\\n\",\n \"pandas_end = time.time()\\n\",\n \"print(\\\"pandas mask took {} seconds.\\\".format(round(pandas_end - pandas_start, 4)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Register custom functions\\n\",\n \"\\n\",\n \"Modin's user-facing API is pandas, but it is possible that we do not yet support your favorite or most-needed functionalities. Your user-defined function may also be able to be executed more efficiently if you pre-define the type of function it is (e.g. map, reduce, etc.) using the DataFrame Algebra. To solve either case, it is possible to register a custom function to be applied to your data.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Registering a custom function for all query compilers\\n\",\n \"\\n\",\n \"To register a custom function for a query compiler, we first need to import it:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\\n\",\n \"```\\n\",\n \"\\n\",\n \"The `PandasQueryCompiler` is responsible for defining and compiling the queries that can be operated on by Modin, and is specific to the pandas storage format. Any queries defined here must also both be compatible with and result in a `pandas.DataFrame`. Many functionalities are very simply implemented, as you can see in the current code: [Link](https://github.com/modin-project/modin/blob/7a8158873e77cb5f1a5a3b89be4ddac89f576269/modin/core/storage_formats/pandas/query_compiler.py#L216).\\n\",\n \"\\n\",\n \"If we want to register a new function, we need to understand what kind of function it is. In our example, we will try to implement a `kurtosis` on the unary negation of the values in the dataframe, which is a map (unargy negation of each cell) followed by a reduce. So we next want to import the function type so we can use it in our definition:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"from modin.core.dataframe.algebra import TreeReduce\\n\",\n \"```\\n\",\n \"\\n\",\n \"Then we can just use the `TreeReduce.register` `classmethod` and assign it to the `PandasQueryCompiler`:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"PandasQueryCompiler.neg_kurtosis = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value, pandas.DataFrame.kurtosis)\\n\",\n \"```\\n\",\n \"\\n\",\n \"We include `**kwargs` to the `lambda` function since the query compiler will pass all keyword arguments to both the map and reduce functions.\\n\",\n \"\\n\",\n \"Finally, we want a handle to it from the `DataFrame`, so we need to create a way to do that:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"def neg_kurtosis_func(self, **kwargs):\\n\",\n \" # The constructor allows you to pass in a query compiler as a keyword argument\\n\",\n \" return self.__constructor__(query_compiler=self._query_compiler.neg_kurtosis(**kwargs))\\n\",\n \"\\n\",\n \"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\\n\",\n \"```\\n\",\n \"\\n\",\n \"And then you can use it like you usually would:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"df.neg_kurtosis_custom()\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\\n\",\n \"from modin.core.dataframe.algebra import TreeReduce\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"PandasQueryCompiler.neg_kurtosis_custom = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value,\\n\",\n \" pandas.DataFrame.kurtosis)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from pandas._libs import lib\\n\",\n \"# The function signature came from the pandas documentation:\\n\",\n \"# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurtosis.html\\n\",\n \"def neg_kurtosis_func(self, axis=lib.no_default, skipna=True, level=None, numeric_only=None, **kwargs):\\n\",\n \" # We need to specify the axis for the query compiler\\n\",\n \" if axis in [None, lib.no_default]:\\n\",\n \" axis = 0\\n\",\n \" # The constructor allows you to pass in a query compiler as a keyword argument\\n\",\n \" # Reduce dimension is used for reduces\\n\",\n \" # We also pass all keyword arguments here to ensure correctness\\n\",\n \" return self._reduce_dimension(\\n\",\n \" self._query_compiler.neg_kurtosis_custom(\\n\",\n \" axis=axis, skipna=skipna, level=level, numeric_only=numeric_only, **kwargs\\n\",\n \" )\\n\",\n \" )\\n\",\n \"\\n\",\n \"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Speed improvements\\n\",\n \"If we were to try and replicate this functionality using the pandas API, we would need to call `df.applymap` with our unary negation function, and subsequently `df.kurtosis` on the result of the first call. Let's see how this compares with our new, custom function!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"print(pandas_df.applymap(lambda cell_value: ~cell_value).kurtosis())\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"pandas unary negation kurtosis took {} seconds.\\\".format(pandas_duration))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"print(df.applymap(lambda x: ~x).kurtosis())\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Modin unary negation kurtosis took {} seconds.\\\".format(modin_duration))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"custom_start = time.time()\\n\",\n \"\\n\",\n \"print(df.neg_kurtosis_custom())\\n\",\n \"\\n\",\n \"custom_end = time.time()\\n\",\n \"modin_custom_duration = custom_end - custom_start\\n\",\n \"print(\\\"Modin neg_kurtosis_custom took {} seconds.\\\".format(modin_custom_duration))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from IPython.display import Markdown, display\\n\",\n \"\\n\",\n \"display(Markdown(\\\"### As expected, Modin is {}x faster than pandas when chaining the functions; however we see that our custom function is even faster than that - beating pandas by {}x, and Modin (when chaining the functions) by {}x!\\\".format(round(pandas_duration / modin_duration, 2), round(pandas_duration / modin_custom_duration, 2), round(modin_duration / modin_custom_duration, 2))))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Congratulations! You have just implemented new DataFrame functionality!\\n\",\n \"\\n\",\n \"## Consider opening a pull request: https://github.com/modin-project/modin/pulls\\n\",\n \"\\n\",\n \"For a complete list of what is implemented, see the [Supported APIs](https://modin.readthedocs.io/en/latest/supported_apis/index.html) section.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Test your knowledge: Add a custom function for another tree reduce: finding `DataFrame.mad` after squaring all of the values\\n\",\n \"\\n\",\n \"See the pandas documentation for the correct signature: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mad.html\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_mad_custom_start = time.time()\\n\",\n \"\\n\",\n \"# Implement your function here! Put the result of your custom squared `mad` in the variable `modin_mad_custom`\\n\",\n \"# Hint: Look at the kurtosis walkthrough above\\n\",\n \"\\n\",\n \"modin_mad_custom = ...\\n\",\n \"print(modin_mad_custom)\\n\",\n \"\\n\",\n \"modin_mad_custom_end = time.time()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Evaluation code, do not change!\\n\",\n \"modin_mad_start = time.time()\\n\",\n \"modin_mad = df.applymap(lambda x: x**2).mad()\\n\",\n \"print(modin_mad)\\n\",\n \"modin_mad_end = time.time()\\n\",\n \"\\n\",\n \"assert modin_mad_end - modin_mad_start > modin_mad_custom_end - modin_mad_custom_start, \\\\\\n\",\n \" \\\"Your implementation was too slow, or you used the chaining functions approach. Try again\\\"\\n\",\n \"assert modin_mad._to_pandas().equals(modin_mad_custom._to_pandas()), \\\"Your result did not match the result of chaining the functions, try again\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Now that you are able to create custom functions, you know enough to contribute to Modin!\\n\",\n \"\\n\",\n \"**Please move on to [Exercise 4](./exercise_4.ipynb) when you are ready**\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_dask/local/exercise_4.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"99f41d2d\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"fdda1c9c\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 4: Experimental Features\\n\",\n \"\\n\",\n \"**GOAL**: Explore some of the experimental features being added to Modin.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e7bf87a5\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for exercise: Spreadsheet\\n\",\n \"\\n\",\n \"For those who have worked with Excel, the Spreadsheet API will definitely feel familiar! The Spreadsheet API is a Jupyter notebook widget that allows us to interact with Modin DataFrames in a spreadsheet-like fashion while taking advantage of the underlying capabilities of Modin. The widget makes it quick and easy to explore, sort, filter, and edit data as well as export the changes as reproducible code.\\n\",\n \"\\n\",\n \"Let's look back at a subset of the 2015 NYC Taxi Data from Exercise 2, and see how the Spreadsheet API can make it easy to play with the data!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"5d5c4a3e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!jupyter nbextension enable --py --sys-prefix modin_spreadsheet\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"dc8d5903\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import modin.experimental.spreadsheet as mss\\n\",\n \"from modin.config import Engine\\n\",\n \"Engine.put(\\\"dask\\\")\\n\",\n \"\\n\",\n \"s3_path = \\\"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\\\"\\n\",\n \"modin_df = pd.read_csv(s3_path, parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3, nrows=1000)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"145e7bbe\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"spreadsheet = mss.from_dataframe(modin_df)\\n\",\n \"spreadsheet\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"3c18b7f2\",\n \"metadata\": {},\n \"source\": [\n \"### Thank you for participating!\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_dask/requirements.txt",
"content": "fsspec>=2022.11.0\njupyterlab\nipywidgets\nmodin[dask]\nmodin[spreadsheet]\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_dask/test/test_notebooks.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport os\nimport sys\n\nimport nbformat\n\nMODIN_DIR = os.path.abspath(\n os.path.join(os.path.dirname(__file__), *[\"..\" for _ in range(6)])\n)\nsys.path.insert(0, MODIN_DIR)\nfrom examples.tutorial.jupyter.execution.test.utils import ( # noqa: E402\n _execute_notebook,\n _replace_str,\n download_taxi_dataset,\n test_dataset_path,\n)\n\nlocal_notebooks_dir = \"examples/tutorial/jupyter/execution/pandas_on_dask/local\"\n\n\n# in this notebook user should replace 'import pandas as pd' with\n# 'import modin.pandas as pd' to make notebook work\ndef test_exercise_1():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_1_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_1.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n _replace_str(nb, \"import pandas as pd\", \"import modin.pandas as pd\")\n\n nbformat.write(nb, modified_notebook_path)\n _execute_notebook(modified_notebook_path)\n\n\n# this notebook works \"as is\" but for testing purposes we can use smaller dataset\ndef test_exercise_2():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_2_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_2.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n new_cell = f'path = \"{test_dataset_path}\"\\n' + download_taxi_dataset\n\n _replace_str(\n nb,\n 'path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"',\n new_cell,\n )\n\n nbformat.write(nb, modified_notebook_path)\n _execute_notebook(modified_notebook_path)\n\n\n# in this notebook user should add custom mad implementation\n# to make notebook work\ndef test_exercise_3():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_3_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_3.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n user_mad_implementation = \"\"\"PandasQueryCompiler.sq_mad_custom = TreeReduce.register(lambda cell_value, **kwargs: cell_value ** 2,\n pandas.DataFrame.mad)\n\ndef sq_mad_func(self, axis=None, skipna=True, level=None, **kwargs):\n if axis is None:\n axis = 0\n\n return self._reduce_dimension(\n self._query_compiler.sq_mad_custom(\n axis=axis, skipna=skipna, level=level, **kwargs\n )\n )\n\npd.DataFrame.sq_mad_custom = sq_mad_func\n\nmodin_mad_custom = df.sq_mad_custom()\n \"\"\"\n\n _replace_str(nb, \"modin_mad_custom = ...\", user_mad_implementation)\n\n nbformat.write(nb, modified_notebook_path)\n # need to update example, `.mad` doesn't exist\n # _execute_notebook(modified_notebook_path)\n\n\n# this notebook works \"as is\" but for testing purposes we can use smaller dataset\ndef test_exercise_4():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_4_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_4.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n s3_path_cell = f's3_path = \"{test_dataset_path}\"\\n' + download_taxi_dataset\n _replace_str(\n nb,\n 's3_path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"',\n s3_path_cell,\n )\n\n nbformat.write(nb, modified_notebook_path)\n _execute_notebook(modified_notebook_path)\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/Dockerfile",
"content": "FROM continuumio/miniconda3\n\nRUN conda install -c conda-forge psutil setproctitle\nRUN pip install -r requirements-dev.txt\n\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/cluster/README.md",
"content": "\n\n
\n
Scale your pandas workflows on a Ray cluster\n
\n\n**NOTE**: Before starting the exercise, please read the full instructions in the \n[Modin documenation](https://modin.readthedocs.io/en/latest/getting_started/using_modin/using_modin_cluster.html).\n\nThe basic steps to run the script on a remote Ray cluster are:\n\nStep 1. Install the necessary dependencies\n\n```bash\npip install boto3\n```\n\nStep 2. Setup your AWS credentials.\n\n```bash\naws configure\n```\n\nStep 3. Modify configuration file and start up the Ray cluster.\n\n```bash\nray up modin-cluster.yaml\n```\n\nStep 4. Submit your script to the remote cluster.\n\n```bash\nray submit modin-cluster.yaml exercise_5.py\n```\n\nStep 5. Shut down the Ray remote cluster.\n\n```bash\nray down \n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/cluster/exercise_5.py",
"content": "import time\n\nimport ray\n\nimport modin.pandas as pd\n\nray.init(address=\"auto\")\ncpu_count = ray.cluster_resources()[\"CPU\"]\nassert cpu_count == 576, f\"Expected 576 CPUs, but found {cpu_count}\"\n\nfile_path = \"big_yellow.csv\"\n\nt0 = time.perf_counter()\n\ndf = pd.read_csv(file_path, quoting=3)\ndf_count = df.count()\ndf_groupby_count = df.groupby(\"passenger_count\").count()\ndf_map = df.map(str)\n\nt1 = time.perf_counter()\nprint(f\"Full script time is {(t1 - t0):.3f}\") # noqa: T201\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/cluster/modin-cluster.yaml",
"content": "# An unique identifier for the head node and workers of this cluster.\ncluster_name: modin_init\n\n# The maximum number of workers nodes to launch in addition to the head\n# node.\nmax_workers: 5\n\n# The autoscaler will scale up the cluster faster with higher upscaling speed.\n# E.g., if the task requires adding more nodes then autoscaler will gradually\n# scale up the cluster in chunks of upscaling_speed*currently_running_nodes.\n# This number should be > 0.\nupscaling_speed: 1.0\n\n# This executes all commands on all nodes in the docker container,\n# and opens all the necessary ports to support the Ray cluster.\n# Empty string means disabled.\ndocker:\n # image: \"rayproject/ray-ml:latest-gpu\" # You can change this to latest-cpu if you don't need GPU support and want a faster startup\n image: rayproject/ray:latest-cpu # use this one if you don't need ML dependencies, it's faster to pull\n container_name: \"ray_container\"\n # If true, pulls latest version of image. Otherwise, `docker run` will only pull the image\n # if no cached version is present.\n pull_before_run: True\n run_options: # Extra options to pass into \"docker run\"\n - --ulimit nofile=65536:65536\n\n # Example of running a GPU head with CPU workers\n # head_image: \"rayproject/ray-ml:latest-gpu\"\n # Allow Ray to automatically detect GPUs\n\n # worker_image: \"rayproject/ray-ml:latest-cpu\"\n # worker_run_options: []\n\n# If a node is idle for this many minutes, it will be removed.\nidle_timeout_minutes: 5\n\n# Cloud-provider specific configuration.\nprovider:\n type: aws\n region: us-west-2\n # Availability zone(s), comma-separated, that nodes may be launched in.\n # Nodes will be launched in the first listed availability zone and will\n # be tried in the subsequent availability zones if launching fails.\n availability_zone: us-west-2a,us-west-2b\n # Whether to allow node reuse. If set to False, nodes will be terminated\n # instead of stopped.\n cache_stopped_nodes: False # If not present, the default is True.\n\n# How Ray will authenticate with newly launched nodes.\nauth:\n ssh_user: ubuntu\n# By default Ray creates a new private keypair, but you can also use your own.\n# If you do so, make sure to also set \"KeyName\" in the head and worker node\n# configurations below.\n# ssh_private_key: /path/to/your/key.pem\n\n# Tell the autoscaler the allowed node types and the resources they provide.\n# The key is the name of the node type, which is just for debugging purposes.\n# The node config specifies the launch config and physical instance type.\navailable_node_types:\n ray.head.default:\n # The node type's CPU and GPU resources are auto-detected based on AWS instance type.\n # If desired, you can override the autodetected CPU and GPU resources advertised to the autoscaler.\n # You can also set custom resources.\n # For example, to mark a node type as having 1 CPU, 1 GPU, and 5 units of a resource called \"custom\", set\n # resources: {\"CPU\": 1, \"GPU\": 1, \"custom\": 5}\n resources: {}\n # Provider-specific config for this node type, e.g. instance type. By default\n # Ray will auto-configure unspecified fields such as SubnetId and KeyName.\n # For more documentation on available fields, see:\n # http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances\n node_config:\n InstanceType: m5.24xlarge\n # Default AMI for us-west-2.\n # Check https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/aws/config.py\n # for default images for other zones.\n ImageId: ami-0387d929287ab193e\n # You can provision additional disk space with a conf as follows\n BlockDeviceMappings:\n - DeviceName: /dev/sda1\n Ebs:\n VolumeSize: 500\n VolumeType: gp3\n # Additional options in the boto docs.\n ray.worker.default:\n # The minimum number of worker nodes of this type to launch.\n # This number should be >= 0.\n min_workers: 5\n # The maximum number of worker nodes of this type to launch.\n # This takes precedence over min_workers.\n max_workers: 5\n # The node type's CPU and GPU resources are auto-detected based on AWS instance type.\n # If desired, you can override the autodetected CPU and GPU resources advertised to the autoscaler.\n # You can also set custom resources.\n # For example, to mark a node type as having 1 CPU, 1 GPU, and 5 units of a resource called \"custom\", set\n # resources: {\"CPU\": 1, \"GPU\": 1, \"custom\": 5}\n resources: {}\n # Provider-specific config for this node type, e.g. instance type. By default\n # Ray will auto-configure unspecified fields such as SubnetId and KeyName.\n # For more documentation on available fields, see:\n # http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances\n node_config:\n InstanceType: m5.24xlarge\n # Default AMI for us-west-2.\n # Check https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/aws/config.py\n # for default images for other zones.\n ImageId: ami-0387d929287ab193e\n # You can provision additional disk space with a conf as follows\n BlockDeviceMappings:\n - DeviceName: /dev/sda1\n Ebs:\n VolumeSize: 500\n VolumeType: gp3\n # Run workers on spot by default. Comment this out to use on-demand.\n # NOTE: If relying on spot instances, it is best to specify multiple different instance\n # types to avoid interruption when one instance type is experiencing heightened demand.\n # Demand information can be found at https://aws.amazon.com/ec2/spot/instance-advisor/\n # InstanceMarketOptions:\n # MarketType: spot\n # Additional options can be found in the boto docs, e.g.\n # SpotOptions:\n # MaxPrice: MAX_HOURLY_PRICE\n # Additional options in the boto docs.\n\n# Specify the node type of the head node (as configured above).\nhead_node_type: ray.head.default\n\n# Files or directories to copy to the head and worker nodes. The format is a\n# dictionary from REMOTE_PATH: LOCAL_PATH, e.g.\nfile_mounts: {\n# \"/path1/on/remote/machine\": \"/path1/on/local/machine\",\n# \"/path2/on/remote/machine\": \"/path2/on/local/machine\",\n}\n\n# Files or directories to copy from the head node to the worker nodes. The format is a\n# list of paths. The same path on the head node will be copied to the worker node.\n# This behavior is a subset of the file_mounts behavior. In the vast majority of cases\n# you should just use file_mounts. Only use this if you know what you're doing!\ncluster_synced_files: []\n\n# Whether changes to directories in file_mounts or cluster_synced_files in the head node\n# should sync to the worker node continuously\nfile_mounts_sync_continuously: False\n\n# Patterns for files to exclude when running rsync up or rsync down\nrsync_exclude:\n - \"**/.git\"\n - \"**/.git/**\"\n\n# Pattern files to use for filtering out files when running rsync up or rsync down. The file is searched for\n# in the source directory and recursively through all subdirectories. For example, if .gitignore is provided\n# as a value, the behavior will match git's behavior for finding and using .gitignore files.\nrsync_filter:\n - \".gitignore\"\n\n# List of commands that will be run before `setup_commands`. If docker is\n# enabled, these commands will run outside the container and before docker\n# is setup.\ninitialization_commands: []\n\n# List of shell commands to run to set up nodes.\nsetup_commands:\n # Note: if you're developing Ray, you probably want to create a Docker image that\n # has your Ray repo pre-cloned. Then, you can replace the pip installs\n # below with a git checkout (and possibly a recompile).\n # To run the nightly version of ray (as opposed to the latest), either use a rayproject docker image\n # that has the \"nightly\" (e.g. \"rayproject/ray-ml:nightly-gpu\") or uncomment the following line:\n # - pip install -U \"ray[default] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl\"\n - conda create -n \"modin\" -c conda-forge modin \"ray-default\">=2.10.0,<3 -y\n - conda activate modin && pip install -U fsspec>=2022.11.0 boto3\n - echo \"conda activate modin\" >> ~/.bashrc\n - wget https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv\n - printf \"VendorID,tpep_pickup_datetime,tpep_dropoff_datetime,passenger_count,trip_distance,pickup_longitude,pickup_latitude,RateCodeID,store_and_fwd_flag,dropoff_longitude,dropoff_latitude,payment_type,fare_amount,extra,mta_tax,tip_amount,tolls_amount,improvement_surcharge,total_amount,congestion_surcharge,airport_fee\\n\" > big_yellow.csv\n - tail -n +2 yellow_tripdata_2015-01.csv{,}{,}{,}{,}{,}{,} >> big_yellow.csv\n - echo 'export MODIN_RAY_CLUSTER=True' >> ~/.bashrc\n \n# Custom commands that will be run on the head node after common setup.\nhead_setup_commands: \n - echo 'export MODIN_REDIS_ADDRESS=\"localhost:6379\"' >> ~/.bashrc\n\n# Custom commands that will be run on worker nodes after common setup.\nworker_setup_commands: []\n\n# Command to start ray on the head node. You don't need to change this.\nhead_start_ray_commands:\n - ray stop\n - echo 'export MEMORY_STORE_SIZE=$(awk \"/MemFree/ { printf \\\"%d \\\\n\\\", \\$2*1024}\" /proc/meminfo)' >> ~/.bashrc\n - echo 'export TMPDIR=\"$(dirname $(mktemp tmp.XXXXXXXXXX -ut))\"' >> ~/.bashrc\n - ulimit -n 65536; ray start --head --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml --object-store-memory=$MEMORY_STORE_SIZE --plasma-directory=$TMPDIR --dashboard-host=0.0.0.0\n\n# Command to start ray on worker nodes. You don't need to change this.\nworker_start_ray_commands:\n - ray stop\n - echo 'export MEMORY_STORE_SIZE=$(awk \"/MemFree/ { printf \\\"%d \\\\n\\\", \\$2*1024}\" /proc/meminfo)' >> ~/.bashrc\n - echo 'export TMPDIR=\"$(dirname $(mktemp tmp.XXXXXXXXXX -ut))\"' >> ~/.bashrc\n - ulimit -n 65536; ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076 --object-store-memory=$MEMORY_STORE_SIZE --plasma-directory=$TMPDIR\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/local/exercise_1.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 1: How to use Modin\\n\",\n \"\\n\",\n \"**GOAL**: Learn how to import Modin to accelerate and scale pandas workflows.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Modin is a drop-in replacement for pandas that distributes the computation \\n\",\n \"across all of the cores in your machine or in a cluster.\\n\",\n \"In practical terms, this means that you can continue using the same pandas scripts\\n\",\n \"as before and expect the behavior and results to be the same. The only thing that needs\\n\",\n \"to change is the import statement. Normally, you would change:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"import pandas as pd\\n\",\n \"```\\n\",\n \"\\n\",\n \"to:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"import modin.pandas as pd\\n\",\n \"```\\n\",\n \"\\n\",\n \"Changing this line of code will allow you to use all of the cores in your machine to do computation on your data. One of the major performance bottlenecks of pandas is that it only uses a single core for any given computation. Modin exposes an API that is identical to pandas, allowing you to continue interacting with your data as you would with pandas. There are no additional commands required to use Modin locally. Partitioning, scheduling, data transfer, and other related concerns are all handled by Modin under the hood.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"
\\n\",\n \"
pandas on a multicore laptop\\n\",\n \" \\n\",\n \" Modin on a multicore laptop\\n\",\n \" \\n\",\n \"\\n\",\n \"
\\n\",\n \"\\n\",\n \"
\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for exercise: Dataframe constructor\\n\",\n \"\\n\",\n \"Often when playing around in pandas, it is useful to create a DataFrame with the constructor. That is where we will start.\\n\",\n \"\\n\",\n \"```python\\n\",\n \"import numpy as np\\n\",\n \"import pandas as pd\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\\n\",\n \"df = pd.DataFrame(frame_data)\\n\",\n \"```\\n\",\n \"\\n\",\n \"When creating a dataframe from a non-distributed object, it will take extra time to partition the data. When this is happening, you will see this message:\\n\",\n \"\\n\",\n \"```\\n\",\n \"UserWarning: Distributing object. This may take some time.\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Note: Do not change this code!\\n\",\n \"import numpy as np\\n\",\n \"import pandas\\n\",\n \"import sys\\n\",\n \"import modin\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas.__version__\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin.__version__\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Implement your answer here. You are also free to play with the size\\n\",\n \"# and shape of the DataFrame, but beware of exceeding your memory!\\n\",\n \"\\n\",\n \"import pandas as pd\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\\n\",\n \"df = pd.DataFrame(frame_data)\\n\",\n \"\\n\",\n \"# ***** Do not change the code below! It verifies that \\n\",\n \"# ***** the exercise has been done correctly. *****\\n\",\n \"\\n\",\n \"try:\\n\",\n \" assert df is not None\\n\",\n \" assert frame_data is not None\\n\",\n \" assert isinstance(frame_data, np.ndarray)\\n\",\n \"except:\\n\",\n \" raise AssertionError(\\\"Don't change too much of the original code!\\\")\\n\",\n \"assert \\\"modin.pandas\\\" in sys.modules, \\\"Not quite correct. Remember the single line of code change (See above)\\\"\\n\",\n \"\\n\",\n \"import modin.pandas\\n\",\n \"assert pd == modin.pandas, \\\"Remember the single line of code change (See above)\\\"\\n\",\n \"assert hasattr(df, \\\"_query_compiler\\\"), \\\"Make sure that `df` is a modin.pandas DataFrame.\\\"\\n\",\n \"\\n\",\n \"print(\\\"Success! You only need to change one line of code!\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now that we have created a toy example for playing around with the DataFrame, let's print it out in different ways.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for Exercise: Data Interaction and Printing\\n\",\n \"\\n\",\n \"When interacting with data, it is very imporant to look at different parts of the data (e.g. `df.head()`). Here we will show that you can print the modin.pandas DataFrame in the same ways you would pandas.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Print the first 10 lines.\\n\",\n \"df.head(10)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Print the DataFrame.\\n\",\n \"df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Free cell for custom interaction (Play around here!)\\n\",\n \"df.add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df.count()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**Please move on to [Exercise 2](./exercise_2.ipynb) when you are ready**\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.6.8\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/local/exercise_2.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 2: Speed improvements\\n\",\n \"\\n\",\n \"**GOAL**: Learn about common functionality that Modin speeds up by using all of your machine's cores.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for Exercise: `read_csv` speedups\\n\",\n \"\\n\",\n \"The most commonly used data ingestion method used in pandas is CSV files (link to pandas survey). This concept is designed to give an idea of the kinds of speedups possible, even on a non-distributed filesystem. Modin also supports other file formats for parallel and distributed reads, which can be found in the documentation.\\n\",\n \"\\n\",\n \"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"We will import both Modin and pandas so that the speedups are evident.\\n\",\n \"\\n\",\n \"**Note: Rerunning the `read_csv` cells many times may result in degraded performance, depending on the memory of the machine**\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"import time\\n\",\n \"from IPython.display import Markdown, display\\n\",\n \"\\n\",\n \"def printmd(string):\\n\",\n \" display(Markdown(string))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Dataset: 2015 NYC taxi trip data\\n\",\n \"\\n\",\n \"We will be using a version of this data already in S3, originally posted in this blog post: https://matthewrocklin.com/blog/work/2017/01/12/dask-dataframes\\n\",\n \"\\n\",\n \"**Size: ~1.8GB**\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"path = \\\"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**Optional:** Note that the dataset takes a while to download. To speed things up a bit, if you prefer to download this file once locally, you can run the following code in the notebook:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# [Optional] Download data locally. This may take a few minutes to download.\\n\",\n \"# import urllib.request\\n\",\n \"# url_path = \\\"https://dask-data.s3.amazonaws.com/nyc-taxi/2015/yellow_tripdata_2015-01.csv\\\"\\n\",\n \"# urllib.request.urlretrieve(url_path, \\\"taxi.csv\\\")\\n\",\n \"# path = \\\"taxi.csv\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## `pandas.read_csv`\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"pandas_df = pandas.read_csv(path, parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to read with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Expect pandas to take >3 minutes on EC2, longer locally\\n\",\n \"\\n\",\n \"This is a good time to chat with your neighbor\\n\",\n \"Dicussion topics\\n\",\n \"- Do you work with a large amount of data daily?\\n\",\n \"- How big is your data?\\n\",\n \"- What’s the common use case of your data?\\n\",\n \"- Do you use any big data analytics tools?\\n\",\n \"- Do you use any interactive analytics tool?\\n\",\n \"- What’s are some drawbacks of your current interative analytic tools today?\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## `modin.pandas.read_csv`\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_df = pd.read_csv(path, parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to read with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `read_csv`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Reduces\\n\",\n \"\\n\",\n \"In pandas, a reduce would be something along the lines of a `sum` or `count`. It computes some summary statistics about the rows or columns. We will be using `count`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"pandas_count = pandas_df.count()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"\\n\",\n \"print(\\\"Time to count with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_count = modin_df.count()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to count with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `count`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_count\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_count\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Map operations\\n\",\n \"\\n\",\n \"In pandas, map operations are operations that do a single pass over the data and do not change its shape. Operations like `isnull` and `applymap` are included in this. We will be using `isnull`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"pandas_isnull = pandas_df.isnull()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"\\n\",\n \"print(\\\"Time to isnull with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_isnull = modin_df.isnull()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to isnull with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `isnull`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_isnull\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_isnull\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Apply over a single column\\n\",\n \"\\n\",\n \"Sometimes we want to compute some summary statistics on a single column from our dataset.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"rounded_trip_distance_pandas = pandas_df[\\\"trip_distance\\\"].apply(round)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to groupby with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"rounded_trip_distance_modin = modin_df[\\\"trip_distance\\\"].apply(round)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to add a column with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `apply` on one column!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"rounded_trip_distance_pandas\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"rounded_trip_distance_modin\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Add a column\\n\",\n \"\\n\",\n \"It is common to need to add a new column to an existing dataframe, here we show that this is significantly faster in Modin due to metadata management and an efficient zero copy implementation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"pandas_df[\\\"rounded_trip_distance\\\"] = rounded_trip_distance_pandas\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to groupby with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_df[\\\"rounded_trip_distance\\\"] = rounded_trip_distance_modin\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to add a column with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas add a column!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**Please move on to [Exercise 3](./exercise_3.ipynb) when you are ready**\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/local/exercise_3.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 3: Not Implemented\\n\",\n \"\\n\",\n \"**GOAL**: Learn what happens when a function is not yet supported in Modin as well as how to extend Modin's functionality using the DataFrame Algebra.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"When functionality has not yet been implemented, we default to pandas\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"We convert a Modin dataframe to pandas to do the operation, then convert it back once it is finished. These operations will have a high overhead due to the communication involved and will take longer than pandas.\\n\",\n \"\\n\",\n \"When this is happening, a warning will be given to the user to inform them that this operation will take longer than usual. For example, `DataFrame.mask` is not yet implemented. In this case, when a user tries to use it, they will see this warning:\\n\",\n \"\\n\",\n \"```\\n\",\n \"UserWarning: `DataFrame.mask` defaulting to pandas implementation.\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Default to pandas\\n\",\n \"\\n\",\n \"In this section of the exercise we will see first-hand how the runtime is affected by operations that are not implemented.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"import numpy as np\\n\",\n \"import time\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**18, 2**8))\\n\",\n \"df = pd.DataFrame(frame_data).add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df = pandas.DataFrame(frame_data).add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_start = time.time()\\n\",\n \"\\n\",\n \"print(df.mask(df < 50))\\n\",\n \"\\n\",\n \"modin_end = time.time()\\n\",\n \"print(\\\"Modin mask took {} seconds.\\\".format(round(modin_end - modin_start, 4)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_start = time.time()\\n\",\n \"\\n\",\n \"print(pandas_df.mask(pandas_df < 50))\\n\",\n \"\\n\",\n \"pandas_end = time.time()\\n\",\n \"print(\\\"pandas mask took {} seconds.\\\".format(round(pandas_end - pandas_start, 4)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Register custom functions\\n\",\n \"\\n\",\n \"Modin's user-facing API is pandas, but it is possible that we do not yet support your favorite or most-needed functionalities. Your user-defined function may also be able to be executed more efficiently if you pre-define the type of function it is (e.g. map, reduce, etc.) using the DataFrame Algebra. To solve either case, it is possible to register a custom function to be applied to your data.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Registering a custom function for all query compilers\\n\",\n \"\\n\",\n \"To register a custom function for a query compiler, we first need to import it:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\\n\",\n \"```\\n\",\n \"\\n\",\n \"The `PandasQueryCompiler` is responsible for defining and compiling the queries that can be operated on by Modin, and is specific to the pandas storage format. Any queries defined here must also both be compatible with and result in a `pandas.DataFrame`. Many functionalities are very simply implemented, as you can see in the current code: [Link](https://github.com/modin-project/modin/blob/7a8158873e77cb5f1a5a3b89be4ddac89f576269/modin/core/storage_formats/pandas/query_compiler.py#L216).\\n\",\n \"\\n\",\n \"If we want to register a new function, we need to understand what kind of function it is. In our example, we will try to implement a `kurtosis` on the unary negation of the values in the dataframe, which is a map (unargy negation of each cell) followed by a reduce. So we next want to import the function type so we can use it in our definition:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"from modin.core.dataframe.algebra import TreeReduce\\n\",\n \"```\\n\",\n \"\\n\",\n \"Then we can just use the `TreeReduce.register` `classmethod` and assign it to the `PandasQueryCompiler`:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"PandasQueryCompiler.neg_kurtosis = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value, pandas.DataFrame.kurtosis)\\n\",\n \"```\\n\",\n \"\\n\",\n \"We include `**kwargs` to the `lambda` function since the query compiler will pass all keyword arguments to both the map and reduce functions.\\n\",\n \"\\n\",\n \"Finally, we want a handle to it from the `DataFrame`, so we need to create a way to do that:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"def neg_kurtosis_func(self, **kwargs):\\n\",\n \" # The constructor allows you to pass in a query compiler as a keyword argument\\n\",\n \" return self.__constructor__(query_compiler=self._query_compiler.neg_kurtosis(**kwargs))\\n\",\n \"\\n\",\n \"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\\n\",\n \"```\\n\",\n \"\\n\",\n \"And then you can use it like you usually would:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"df.neg_kurtosis_custom()\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\\n\",\n \"from modin.core.dataframe.algebra import TreeReduce\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"PandasQueryCompiler.neg_kurtosis_custom = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value,\\n\",\n \" pandas.DataFrame.kurtosis)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from pandas._libs import lib\\n\",\n \"# The function signature came from the pandas documentation:\\n\",\n \"# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurtosis.html\\n\",\n \"def neg_kurtosis_func(self, axis=lib.no_default, skipna=True, level=None, numeric_only=None, **kwargs):\\n\",\n \" # We need to specify the axis for the query compiler\\n\",\n \" if axis in [None, lib.no_default]:\\n\",\n \" axis = 0\\n\",\n \" # The constructor allows you to pass in a query compiler as a keyword argument\\n\",\n \" # Reduce dimension is used for reduces\\n\",\n \" # We also pass all keyword arguments here to ensure correctness\\n\",\n \" return self._reduce_dimension(\\n\",\n \" self._query_compiler.neg_kurtosis_custom(\\n\",\n \" axis=axis, skipna=skipna, level=level, numeric_only=numeric_only, **kwargs\\n\",\n \" )\\n\",\n \" )\\n\",\n \"\\n\",\n \"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Speed improvements\\n\",\n \"If we were to try and replicate this functionality using the pandas API, we would need to call `df.applymap` with our unary negation function, and subsequently `df.kurtosis` on the result of the first call. Let's see how this compares with our new, custom function!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"print(pandas_df.applymap(lambda cell_value: ~cell_value).kurtosis())\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"pandas unary negation kurtosis took {} seconds.\\\".format(pandas_duration))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"print(df.applymap(lambda x: ~x).kurtosis())\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Modin unary negation kurtosis took {} seconds.\\\".format(modin_duration))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"custom_start = time.time()\\n\",\n \"\\n\",\n \"print(df.neg_kurtosis_custom())\\n\",\n \"\\n\",\n \"custom_end = time.time()\\n\",\n \"modin_custom_duration = custom_end - custom_start\\n\",\n \"print(\\\"Modin neg_kurtosis_custom took {} seconds.\\\".format(modin_custom_duration))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from IPython.display import Markdown, display\\n\",\n \"\\n\",\n \"display(Markdown(\\\"### As expected, Modin is {}x faster than pandas when chaining the functions; however we see that our custom function is even faster than that - beating pandas by {}x, and Modin (when chaining the functions) by {}x!\\\".format(round(pandas_duration / modin_duration, 2), round(pandas_duration / modin_custom_duration, 2), round(modin_duration / modin_custom_duration, 2))))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Congratulations! You have just implemented new DataFrame functionality!\\n\",\n \"\\n\",\n \"## Consider opening a pull request: https://github.com/modin-project/modin/pulls\\n\",\n \"\\n\",\n \"For a complete list of what is implemented, see the [Supported APIs](https://modin.readthedocs.io/en/latest/supported_apis/index.html) section.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Test your knowledge: Add a custom function for another tree reduce: finding `DataFrame.mad` after squaring all of the values\\n\",\n \"\\n\",\n \"See the pandas documentation for the correct signature: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mad.html\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_mad_custom_start = time.time()\\n\",\n \"\\n\",\n \"# Implement your function here! Put the result of your custom squared `mad` in the variable `modin_mad_custom`\\n\",\n \"# Hint: Look at the kurtosis walkthrough above\\n\",\n \"\\n\",\n \"modin_mad_custom = ...\\n\",\n \"print(modin_mad_custom)\\n\",\n \"\\n\",\n \"modin_mad_custom_end = time.time()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Evaluation code, do not change!\\n\",\n \"modin_mad_start = time.time()\\n\",\n \"modin_mad = df.applymap(lambda x: x**2).mad()\\n\",\n \"print(modin_mad)\\n\",\n \"modin_mad_end = time.time()\\n\",\n \"\\n\",\n \"assert modin_mad_end - modin_mad_start > modin_mad_custom_end - modin_mad_custom_start, \\\\\\n\",\n \" \\\"Your implementation was too slow, or you used the chaining functions approach. Try again\\\"\\n\",\n \"assert modin_mad._to_pandas().equals(modin_mad_custom._to_pandas()), \\\"Your result did not match the result of chaining the functions, try again\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Now that you are able to create custom functions, you know enough to contribute to Modin!\\n\",\n \"\\n\",\n \"**Please move on to [Exercise 4](./exercise_4.ipynb) when you are ready**\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/local/exercise_4.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"99f41d2d\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"fdda1c9c\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 4: Experimental Features\\n\",\n \"\\n\",\n \"**GOAL**: Explore some of the experimental features being added to Modin.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"9b487c51\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for exercise: Progress Bar\\n\",\n \"\\n\",\n \"\\n\",\n \"Sometimes when running long functions on DataFrames, it can be hard to tell how much progress has been made, as well as how much longer the function will run. A progress bar allows users to see the estimated progress and completion time of each line they run, in environments such as a shell or Jupyter notebook.\\n\",\n \"\\n\",\n \"To enable Modin's Progress Bar, add the following lines of code after importing `modin.pandas`:\\n\",\n \"```python\\n\",\n \"from modin.config import ProgressBar\\n\",\n \"ProgressBar.enable()\\n\",\n \"```\\n\",\n \"\\n\",\n \"In this exercise, we'll see how the progress bar can improve our experience running dataframe queries!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"f95d4874\",\n \"metadata\": {\n \"scrolled\": false\n },\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import numpy as np\\n\",\n \"from modin.config import ProgressBar\\n\",\n \"ProgressBar.enable()\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**18, 2**8))\\n\",\n \"df = pd.DataFrame(frame_data).add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"6905bc6b\",\n \"metadata\": {},\n \"source\": [\n \"On longer functions, its nice to be able to see an estimation of how much longer things will take!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"236ec8e2\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"df = df.applymap(lambda x: ~x)\\n\",\n \"df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e7bf87a5\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for exercise: Spreadsheet\\n\",\n \"\\n\",\n \"For those who have worked with Excel, the Spreadsheet API will definitely feel familiar! The Spreadsheet API is a Jupyter notebook widget that allows us to interact with Modin DataFrames in a spreadsheet-like fashion while taking advantage of the underlying capabilities of Modin. The widget makes it quick and easy to explore, sort, filter, and edit data as well as export the changes as reproducible code.\\n\",\n \"\\n\",\n \"Let's look back at a subset of the 2015 NYC Taxi Data from Exercise 2, and see how the Spreadsheet API can make it easy to play with the data!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"5d5c4a3e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!jupyter nbextension enable --py --sys-prefix modin_spreadsheet\\n\",\n \"ProgressBar.disable()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"dc8d5903\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.experimental.spreadsheet as mss\\n\",\n \"\\n\",\n \"s3_path = \\\"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\\\"\\n\",\n \"modin_df = pd.read_csv(s3_path, parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3, nrows=1000)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"145e7bbe\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"spreadsheet = mss.from_dataframe(modin_df)\\n\",\n \"spreadsheet\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"3c18b7f2\",\n \"metadata\": {},\n \"source\": [\n \"### Thank you for participating!\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/requirements.txt",
"content": "fsspec>=2022.11.0\njupyterlab\nipywidgets\ntqdm>=4.60.0\nmodin[ray]\nmodin[spreadsheet]\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_ray/test/test_notebooks.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport os\nimport sys\n\nimport nbformat\n\nMODIN_DIR = os.path.abspath(\n os.path.join(os.path.dirname(__file__), *[\"..\" for _ in range(6)])\n)\nsys.path.insert(0, MODIN_DIR)\nfrom examples.tutorial.jupyter.execution.test.utils import ( # noqa: E402\n _execute_notebook,\n _find_code_cell_idx,\n _replace_str,\n download_taxi_dataset,\n test_dataset_path,\n)\n\nlocal_notebooks_dir = \"examples/tutorial/jupyter/execution/pandas_on_ray/local\"\n\n\n# in this notebook user should replace 'import pandas as pd' with\n# 'import modin.pandas as pd' to make notebook work\ndef test_exercise_1():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_1_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_1.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n _replace_str(nb, \"import pandas as pd\", \"import modin.pandas as pd\")\n\n nbformat.write(nb, modified_notebook_path)\n _execute_notebook(modified_notebook_path)\n\n\n# this notebook works \"as is\" but for testing purposes we can use smaller dataset\ndef test_exercise_2():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_2_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_2.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n _replace_str(\n nb,\n 'path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"',\n '# path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"',\n )\n\n new_optional_cell = f'path = \"{test_dataset_path}\"\\n' + download_taxi_dataset\n\n optional_cell_idx = _find_code_cell_idx(nb, \"[Optional] Download data locally.\")\n nb[\"cells\"][optional_cell_idx][\"source\"] = new_optional_cell\n\n nbformat.write(nb, modified_notebook_path)\n _execute_notebook(modified_notebook_path)\n\n\n# in this notebook user should add custom mad implementation\n# to make notebook work\ndef test_exercise_3():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_3_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_3.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n user_mad_implementation = \"\"\"PandasQueryCompiler.sq_mad_custom = TreeReduce.register(lambda cell_value, **kwargs: cell_value ** 2,\n pandas.DataFrame.mad)\n\ndef sq_mad_func(self, axis=None, skipna=True, level=None, **kwargs):\n if axis is None:\n axis = 0\n\n return self._reduce_dimension(\n self._query_compiler.sq_mad_custom(\n axis=axis, skipna=skipna, level=level, **kwargs\n )\n )\n\npd.DataFrame.sq_mad_custom = sq_mad_func\n\nmodin_mad_custom = df.sq_mad_custom()\n \"\"\"\n\n _replace_str(nb, \"modin_mad_custom = ...\", user_mad_implementation)\n\n nbformat.write(nb, modified_notebook_path)\n # need to update example, `.mad` doesn't exist\n # _execute_notebook(modified_notebook_path)\n\n\n# this notebook works \"as is\" but for testing purposes we can use smaller dataset\ndef test_exercise_4():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_4_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_4.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n s3_path_cell = f's3_path = \"{test_dataset_path}\"\\n' + download_taxi_dataset\n _replace_str(\n nb,\n 's3_path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"',\n s3_path_cell,\n )\n\n nbformat.write(nb, modified_notebook_path)\n _execute_notebook(modified_notebook_path)\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_unidist/Dockerfile",
"content": "FROM continuumio/miniconda3\n\nRUN conda env create -f jupyter_unidist_env.yml\nRUN conda install -c conda-forge psutil setproctitle\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_unidist/README.md",
"content": "# Jupyter notebook examples to run with PandasOnUnidist\n\nCurrently, Modin supports `PandasOnUnidist` execution only with MPI backend of [unidist](https://github.com/modin-project/unidist).\nThere are some specifics on how to run a jupyter notebook with MPI, namely, you should use `mpiexec` command.\n\n```bash\nmpiexec -n 1 jupyter notebook\n```\n\n**Important**\n\nMPI is not reliable yet to work in interactive environment such as jupyter notebooks. Thus, some things may not work.\nFor example, if you are experiencing the error `The kernel appears to have died. It will restart automatically.`,\nyou may want to modify `kernel.json` file or create a new one in order to fix the problem.\n\nFor simplicity, you can just run `setup_kernel.py` script located in this directory. This will install a new MPI enabled kernel,\nwhich you can then select using the dropdown menu in your browser. Otherwise, you can follow the steps below:\n\n1. First, what you should do is locate `kernel.json` file with `jupyter kernelspec list` command. It should generally be like this.\n\n```bash\njupyter kernelspec list\n\nAvailable kernels:\n python3 $PREFIX/share/jupyter/kernels/python3\n```\n\n`kernel.json` file should be located in `python3` folder.\n\n2. Second, you should make a copy of the `python3` folder, say to `python3mpi` folder.\n\n```bash\ncp -r $PREFIX/share/jupyter/kernels/python3 $PREFIX/share/jupyter/kernels/python3mpi\n```\n\n3. Third, modify `kernel.json` file in `python3mpi` folder to add `mpiexec -n 1` command\n(like \"mpiexec\", \"-n\", \"1\") to the beginning of the launched command (`argv`).\n\n4. Fourth, change `display_name` in `kernel.json` file to something like `Python 3 (ipykernel) with MPI`.\nThat way you can specifically select the Python kernel with MPI-enabled using the dropdown menu in your browser.\n\n## Run Jupyter Notebooks with PandasOnUnidist\n\nAfter the `setup_kernel.py` script is run or the steps above are done, you can run a jupyter notebook with `PandasOnUnidist` in a normal way.\n\n```bash\njupyter notebook\n```\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_unidist/jupyter_unidist_env.yml",
"content": "name: jupyter_modin_on_unidist\nchannels:\n - conda-forge\ndependencies:\n - pip\n - fsspec>=2022.11.0\n - jupyterlab\n - ipywidgets\n - modin-mpi\n - pip:\n - modin[spreadsheet]\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_unidist/local/exercise_1.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 1: How to use Modin\\n\",\n \"\\n\",\n \"**GOAL**: Learn how to import Modin to accelerate and scale pandas workflows.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Modin is a drop-in replacement for pandas that distributes the computation \\n\",\n \"across all of the cores in your machine or in a cluster.\\n\",\n \"In practical terms, this means that you can continue using the same pandas scripts\\n\",\n \"as before and expect the behavior and results to be the same. The only thing that needs\\n\",\n \"to change is the import statement. Normally, you would change:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"import pandas as pd\\n\",\n \"```\\n\",\n \"\\n\",\n \"to:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"import modin.pandas as pd\\n\",\n \"```\\n\",\n \"\\n\",\n \"Changing this line of code will allow you to use all of the cores in your machine to do computation on your data. One of the major performance bottlenecks of pandas is that it only uses a single core for any given computation. Modin exposes an API that is identical to pandas, allowing you to continue interacting with your data as you would with pandas. There are no additional commands required to use Modin locally. Partitioning, scheduling, data transfer, and other related concerns are all handled by Modin under the hood.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"
\\n\",\n \"
pandas on a multicore laptop\\n\",\n \" \\n\",\n \" Modin on a multicore laptop\\n\",\n \" \\n\",\n \"\\n\",\n \"
\\n\",\n \"\\n\",\n \"
\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for exercise: setting Modin engine\\n\",\n \"\\n\",\n \"Modin uses Ray as an execution engine by default so no additional action is required to start to use it. Alternatively, if you need to use another engine, it should be specified either by setting the Modin config or by setting Modin environment variable before the first operation with Modin as it is shown below. Also, note that the full list of Modin configs and corresponding environment variables can be found in the [Modin Configuration Settings](https://modin.readthedocs.io/en/stable/flow/modin/config.html#modin-configs-list) section of the Modin documentation.\\n\",\n \"\\n\",\n \"One of the execution engines that Modin uses is Unidist. Currently, Modin only supports MPI through unidist, so it should be specified either by setting the Unidist config or by setting Unidist environment variable. The full list of Unidist configs and corresponding environment variables can be found in the [Unidist Configuration Settings](https://unidist.readthedocs.io/en/latest/flow/unidist/config.html#unidist-configuration-settings-list) section of the Unidist documentation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"python\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# Modin engine and Unidist backend can be specified either by config\\n\",\n \"import modin.config as modin_cfg\\n\",\n \"import unidist.config as unidist_cfg\\n\",\n \"modin_cfg.Engine.put(\\\"unidist\\\")\\n\",\n \"unidist_cfg.Backend.put(\\\"mpi\\\")\\n\",\n \"\\n\",\n \"# or by setting the environment variable\\n\",\n \"# import os\\n\",\n \"# os.environ[\\\"MODIN_ENGINE\\\"] = \\\"unidist\\\"\\n\",\n \"# os.environ[\\\"UNIDIST_BACKEND\\\"] = \\\"mpi\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for exercise: Dataframe constructor\\n\",\n \"\\n\",\n \"Often when playing around in pandas, it is useful to create a DataFrame with the constructor. That is where we will start.\\n\",\n \"\\n\",\n \"```python\\n\",\n \"import numpy as np\\n\",\n \"import pandas as pd\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\\n\",\n \"df = pd.DataFrame(frame_data)\\n\",\n \"```\\n\",\n \"\\n\",\n \"When creating a dataframe from a non-distributed object, it will take extra time to partition the data. When this is happening, you will see this message:\\n\",\n \"\\n\",\n \"```\\n\",\n \"UserWarning: Distributing object. This may take some time.\\n\",\n \"```\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"python\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# Note: Do not change this code!\\n\",\n \"import numpy as np\\n\",\n \"import pandas\\n\",\n \"import sys\\n\",\n \"import modin\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"python\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"pandas.__version__\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"python\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"modin.__version__\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"python\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# Implement your answer here. You are also free to play with the size\\n\",\n \"# and shape of the DataFrame, but beware of exceeding your memory!\\n\",\n \"\\n\",\n \"# import pandas as pd\\n\",\n \"import pandas as pd\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**5, 2**5))\\n\",\n \"df = pd.DataFrame(frame_data)\\n\",\n \"\\n\",\n \"# ***** Do not change the code below! It verifies that \\n\",\n \"# ***** the exercise has been done correctly. *****\\n\",\n \"\\n\",\n \"try:\\n\",\n \" assert df is not None\\n\",\n \" assert frame_data is not None\\n\",\n \" assert isinstance(frame_data, np.ndarray)\\n\",\n \"except:\\n\",\n \" raise AssertionError(\\\"Don't change too much of the original code!\\\")\\n\",\n \"assert \\\"modin.pandas\\\" in sys.modules, \\\"Not quite correct. Remember the single line of code change (See above)\\\"\\n\",\n \"\\n\",\n \"import modin.pandas\\n\",\n \"assert pd == modin.pandas, \\\"Remember the single line of code change (See above)\\\"\\n\",\n \"assert hasattr(df, \\\"_query_compiler\\\"), \\\"Make sure that `df` is a modin.pandas DataFrame.\\\"\\n\",\n \"\\n\",\n \"print(\\\"Success! You only need to change one line of code!\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now that we have created a toy example for playing around with the DataFrame, let's print it out in different ways.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for Exercise: Data Interaction and Printing\\n\",\n \"\\n\",\n \"When interacting with data, it is very imporant to look at different parts of the data (e.g. `df.head()`). Here we will show that you can print the modin.pandas DataFrame in the same ways you would pandas.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"python\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# Print the first 10 lines.\\n\",\n \"df.head(10)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"python\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# Print the DataFrame.\\n\",\n \"df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"python\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"# Free cell for custom interaction (Play around here!)\\n\",\n \"df.add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"vscode\": {\n \"languageId\": \"python\"\n }\n },\n \"outputs\": [],\n \"source\": [\n \"df.count()\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**Please move on to [Exercise 2](./exercise_2.ipynb) when you are ready**\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel) with MPI\",\n \"language\": \"python\",\n \"name\": \"python3mpi\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_unidist/local/exercise_2.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 2: Speed improvements\\n\",\n \"\\n\",\n \"**GOAL**: Learn about common functionality that Modin speeds up by using all of your machine's cores.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for Exercise: `read_csv` speedups\\n\",\n \"\\n\",\n \"The most commonly used data ingestion method used in pandas is CSV files (link to pandas survey). This concept is designed to give an idea of the kinds of speedups possible, even on a non-distributed filesystem. Modin also supports other file formats for parallel and distributed reads, which can be found in the documentation. We will import both Modin and pandas so that the speedups are evident.\\n\",\n \"\\n\",\n \"**Note: Rerunning the `read_csv` cells many times may result in degraded performance, depending on the memory of the machine**\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"import time\\n\",\n \"from IPython.display import Markdown, display\\n\",\n \"\\n\",\n \"def printmd(string):\\n\",\n \" display(Markdown(string))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Dataset: 2015 NYC taxi trip data\\n\",\n \"\\n\",\n \"\\n\",\n \"We will be using a version of this data already in S3, originally posted in this blog post: https://matthewrocklin.com/blog/work/2017/01/12/dask-dataframes\\n\",\n \"\\n\",\n \"**Size: ~1.8GB**\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"path = \\\"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Modin execution engine setting:\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.config as modin_cfg\\n\",\n \"import unidist.config as unidist_cfg\\n\",\n \"modin_cfg.Engine.put(\\\"unidist\\\")\\n\",\n \"unidist_cfg.Backend.put(\\\"mpi\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## `pandas.read_csv`\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"pandas_df = pandas.read_csv(path, parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to read with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Expect pandas to take >3 minutes on EC2, longer locally\\n\",\n \"\\n\",\n \"This is a good time to chat with your neighbor\\n\",\n \"Dicussion topics\\n\",\n \"- Do you work with a large amount of data daily?\\n\",\n \"- How big is your data?\\n\",\n \"- What’s the common use case of your data?\\n\",\n \"- Do you use any big data analytics tools?\\n\",\n \"- Do you use any interactive analytics tool?\\n\",\n \"- What’s are some drawbacks of your current interative analytic tools today?\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## `modin.pandas.read_csv`\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_df = pd.read_csv(path, parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to read with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `read_csv`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Reduces\\n\",\n \"\\n\",\n \"In pandas, a reduce would be something along the lines of a `sum` or `count`. It computes some summary statistics about the rows or columns. We will be using `count`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"pandas_count = pandas_df.count()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"\\n\",\n \"print(\\\"Time to count with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_count = modin_df.count()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to count with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `count`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_count\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_count\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Map operations\\n\",\n \"\\n\",\n \"In pandas, map operations are operations that do a single pass over the data and do not change its shape. Operations like `isnull` and `applymap` are included in this. We will be using `isnull`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"pandas_isnull = pandas_df.isnull()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"\\n\",\n \"print(\\\"Time to isnull with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_isnull = modin_df.isnull()\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to isnull with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `isnull`!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_isnull\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_isnull\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Apply over a single column\\n\",\n \"\\n\",\n \"Sometimes we want to compute some summary statistics on a single column from our dataset.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"rounded_trip_distance_pandas = pandas_df[\\\"trip_distance\\\"].apply(round)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to groupby with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"rounded_trip_distance_modin = modin_df[\\\"trip_distance\\\"].apply(round)\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to add a column with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas at `apply` on one column!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"rounded_trip_distance_pandas\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"rounded_trip_distance_modin\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Add a column\\n\",\n \"\\n\",\n \"It is common to need to add a new column to an existing dataframe, here we show that this is significantly faster in Modin due to metadata management and an efficient zero copy implementation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"pandas_df[\\\"rounded_trip_distance\\\"] = rounded_trip_distance_pandas\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"Time to groupby with pandas: {} seconds\\\".format(round(pandas_duration, 3)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"modin_df[\\\"rounded_trip_distance\\\"] = rounded_trip_distance_modin\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Time to add a column with Modin: {} seconds\\\".format(round(modin_duration, 3)))\\n\",\n \"\\n\",\n \"printmd(\\\"### Modin is {}x faster than pandas add a column!\\\".format(round(pandas_duration / modin_duration, 2)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Are they equal?\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_df\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**Please move on to [Exercise 3](./exercise_3.ipynb) when you are ready**\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel) with MPI\",\n \"language\": \"python\",\n \"name\": \"python3mpi\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_unidist/local/exercise_3.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 3: Not Implemented\\n\",\n \"\\n\",\n \"**GOAL**: Learn what happens when a function is not yet supported in Modin as well as how to extend Modin's functionality using the DataFrame Algebra.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"When functionality has not yet been implemented, we default to pandas\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"We convert a Modin dataframe to pandas to do the operation, then convert it back once it is finished. These operations will have a high overhead due to the communication involved and will take longer than pandas.\\n\",\n \"\\n\",\n \"When this is happening, a warning will be given to the user to inform them that this operation will take longer than usual. For example, `DataFrame.mask` is not yet implemented. In this case, when a user tries to use it, they will see this warning:\\n\",\n \"\\n\",\n \"```\\n\",\n \"UserWarning: `DataFrame.mask` defaulting to pandas implementation.\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Default to pandas\\n\",\n \"\\n\",\n \"In this section of the exercise we will see first-hand how the runtime is affected by operations that are not implemented.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import pandas\\n\",\n \"import numpy as np\\n\",\n \"import time\\n\",\n \"import modin.config as modin_cfg\\n\",\n \"import unidist.config as unidist_cfg\\n\",\n \"modin_cfg.Engine.put(\\\"unidist\\\")\\n\",\n \"unidist_cfg.Backend.put(\\\"mpi\\\")\\n\",\n \"\\n\",\n \"frame_data = np.random.randint(0, 100, size=(2**18, 2**8))\\n\",\n \"df = pd.DataFrame(frame_data).add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_df = pandas.DataFrame(frame_data).add_prefix(\\\"col\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_start = time.time()\\n\",\n \"\\n\",\n \"print(df.mask(df < 50))\\n\",\n \"\\n\",\n \"modin_end = time.time()\\n\",\n \"print(\\\"Modin mask took {} seconds.\\\".format(round(modin_end - modin_start, 4)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pandas_start = time.time()\\n\",\n \"\\n\",\n \"print(pandas_df.mask(pandas_df < 50))\\n\",\n \"\\n\",\n \"pandas_end = time.time()\\n\",\n \"print(\\\"pandas mask took {} seconds.\\\".format(round(pandas_end - pandas_start, 4)))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Concept for exercise: Register custom functions\\n\",\n \"\\n\",\n \"Modin's user-facing API is pandas, but it is possible that we do not yet support your favorite or most-needed functionalities. Your user-defined function may also be able to be executed more efficiently if you pre-define the type of function it is (e.g. map, reduce, etc.) using the DataFrame Algebra. To solve either case, it is possible to register a custom function to be applied to your data.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Registering a custom function for all query compilers\\n\",\n \"\\n\",\n \"To register a custom function for a query compiler, we first need to import it:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\\n\",\n \"```\\n\",\n \"\\n\",\n \"The `PandasQueryCompiler` is responsible for defining and compiling the queries that can be operated on by Modin, and is specific to the pandas storage format. Any queries defined here must also both be compatible with and result in a `pandas.DataFrame`. Many functionalities are very simply implemented, as you can see in the current code: [Link](https://github.com/modin-project/modin/blob/7a8158873e77cb5f1a5a3b89be4ddac89f576269/modin/core/storage_formats/pandas/query_compiler.py#L216).\\n\",\n \"\\n\",\n \"If we want to register a new function, we need to understand what kind of function it is. In our example, we will try to implement a `kurtosis` on the unary negation of the values in the dataframe, which is a map (unargy negation of each cell) followed by a reduce. So we next want to import the function type so we can use it in our definition:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"from modin.core.dataframe.algebra import TreeReduce\\n\",\n \"```\\n\",\n \"\\n\",\n \"Then we can just use the `TreeReduce.register` `classmethod` and assign it to the `PandasQueryCompiler`:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"PandasQueryCompiler.neg_kurtosis = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value, pandas.DataFrame.kurtosis)\\n\",\n \"```\\n\",\n \"\\n\",\n \"We include `**kwargs` to the `lambda` function since the query compiler will pass all keyword arguments to both the map and reduce functions.\\n\",\n \"\\n\",\n \"Finally, we want a handle to it from the `DataFrame`, so we need to create a way to do that:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"def neg_kurtosis_func(self, **kwargs):\\n\",\n \" # The constructor allows you to pass in a query compiler as a keyword argument\\n\",\n \" return self.__constructor__(query_compiler=self._query_compiler.neg_kurtosis(**kwargs))\\n\",\n \"\\n\",\n \"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\\n\",\n \"```\\n\",\n \"\\n\",\n \"And then you can use it like you usually would:\\n\",\n \"\\n\",\n \"```python\\n\",\n \"df.neg_kurtosis_custom()\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\\n\",\n \"from modin.core.dataframe.algebra import TreeReduce\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"PandasQueryCompiler.neg_kurtosis_custom = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value,\\n\",\n \" pandas.DataFrame.kurtosis)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from pandas._libs import lib\\n\",\n \"# The function signature came from the pandas documentation:\\n\",\n \"# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurtosis.html\\n\",\n \"def neg_kurtosis_func(self, axis=lib.no_default, skipna=True, level=None, numeric_only=None, **kwargs):\\n\",\n \" # We need to specify the axis for the query compiler\\n\",\n \" if axis in [None, lib.no_default]:\\n\",\n \" axis = 0\\n\",\n \" # The constructor allows you to pass in a query compiler as a keyword argument\\n\",\n \" # Reduce dimension is used for reduces\\n\",\n \" # We also pass all keyword arguments here to ensure correctness\\n\",\n \" return self._reduce_dimension(\\n\",\n \" self._query_compiler.neg_kurtosis_custom(\\n\",\n \" axis=axis, skipna=skipna, level=level, numeric_only=numeric_only, **kwargs\\n\",\n \" )\\n\",\n \" )\\n\",\n \"\\n\",\n \"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Speed improvements\\n\",\n \"If we were to try and replicate this functionality using the pandas API, we would need to call `df.applymap` with our unary negation function, and subsequently `df.kurtosis` on the result of the first call. Let's see how this compares with our new, custom function!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"print(pandas_df.applymap(lambda cell_value: ~cell_value).kurtosis())\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"pandas_duration = end - start\\n\",\n \"print(\\\"pandas unary negation kurtosis took {} seconds.\\\".format(pandas_duration))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"start = time.time()\\n\",\n \"\\n\",\n \"print(df.applymap(lambda x: ~x).kurtosis())\\n\",\n \"\\n\",\n \"end = time.time()\\n\",\n \"modin_duration = end - start\\n\",\n \"print(\\\"Modin unary negation kurtosis took {} seconds.\\\".format(modin_duration))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"custom_start = time.time()\\n\",\n \"\\n\",\n \"print(df.neg_kurtosis_custom())\\n\",\n \"\\n\",\n \"custom_end = time.time()\\n\",\n \"modin_custom_duration = custom_end - custom_start\\n\",\n \"print(\\\"Modin neg_kurtosis_custom took {} seconds.\\\".format(modin_custom_duration))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from IPython.display import Markdown, display\\n\",\n \"\\n\",\n \"display(Markdown(\\\"### As expected, Modin is {}x faster than pandas when chaining the functions; however we see that our custom function is even faster than that - beating pandas by {}x, and Modin (when chaining the functions) by {}x!\\\".format(round(pandas_duration / modin_duration, 2), round(pandas_duration / modin_custom_duration, 2), round(modin_duration / modin_custom_duration, 2))))\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Congratulations! You have just implemented new DataFrame functionality!\\n\",\n \"\\n\",\n \"## Consider opening a pull request: https://github.com/modin-project/modin/pulls\\n\",\n \"\\n\",\n \"For a complete list of what is implemented, see the [Supported APIs](https://modin.readthedocs.io/en/latest/supported_apis/index.html) section.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Test your knowledge: Add a custom function for another tree reduce: finding `DataFrame.mad` after squaring all of the values\\n\",\n \"\\n\",\n \"See the pandas documentation for the correct signature: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mad.html\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"modin_mad_custom_start = time.time()\\n\",\n \"\\n\",\n \"# Implement your function here! Put the result of your custom squared `mad` in the variable `modin_mad_custom`\\n\",\n \"# Hint: Look at the kurtosis walkthrough above\\n\",\n \"\\n\",\n \"modin_mad_custom = ...\\n\",\n \"print(modin_mad_custom)\\n\",\n \"\\n\",\n \"modin_mad_custom_end = time.time()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Evaluation code, do not change!\\n\",\n \"modin_mad_start = time.time()\\n\",\n \"modin_mad = df.applymap(lambda x: x**2).mad()\\n\",\n \"print(modin_mad)\\n\",\n \"modin_mad_end = time.time()\\n\",\n \"\\n\",\n \"assert modin_mad_end - modin_mad_start > modin_mad_custom_end - modin_mad_custom_start, \\\\\\n\",\n \" \\\"Your implementation was too slow, or you used the chaining functions approach. Try again\\\"\\n\",\n \"assert modin_mad._to_pandas().equals(modin_mad_custom._to_pandas()), \\\"Your result did not match the result of chaining the functions, try again\\\"\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Now that you are able to create custom functions, you know enough to contribute to Modin!\\n\",\n \"\\n\",\n \"**Please move on to [Exercise 4](./exercise_4.ipynb) when you are ready**\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel) with MPI\",\n \"language\": \"python\",\n \"name\": \"python3mpi\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_unidist/local/exercise_4.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"99f41d2d\",\n \"metadata\": {},\n \"source\": [\n \"\\n\",\n \"\\n\",\n \"
Scale your pandas workflows by changing one line of code
\\n\",\n \"\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"fdda1c9c\",\n \"metadata\": {},\n \"source\": [\n \"# Exercise 4: Experimental Features\\n\",\n \"\\n\",\n \"**GOAL**: Explore some of the experimental features being added to Modin.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"e7bf87a5\",\n \"metadata\": {},\n \"source\": [\n \"### Concept for exercise: Spreadsheet\\n\",\n \"\\n\",\n \"For those who have worked with Excel, the Spreadsheet API will definitely feel familiar! The Spreadsheet API is a Jupyter notebook widget that allows us to interact with Modin DataFrames in a spreadsheet-like fashion while taking advantage of the underlying capabilities of Modin. The widget makes it quick and easy to explore, sort, filter, and edit data as well as export the changes as reproducible code.\\n\",\n \"\\n\",\n \"Let's look back at a subset of the 2015 NYC Taxi Data from Exercise 2, and see how the Spreadsheet API can make it easy to play with the data!\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"5d5c4a3e\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"!jupyter nbextension enable --py --sys-prefix modin_spreadsheet\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"dc8d5903\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import modin.pandas as pd\\n\",\n \"import modin.experimental.spreadsheet as mss\\n\",\n \"import modin.config as modin_cfg\\n\",\n \"import unidist.config as unidist_cfg\\n\",\n \"modin_cfg.Engine.put(\\\"unidist\\\")\\n\",\n \"unidist_cfg.Backend.put(\\\"mpi\\\")\\n\",\n \"\\n\",\n \"s3_path = \\\"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\\\"\\n\",\n \"modin_df = pd.read_csv(s3_path, parse_dates=[\\\"tpep_pickup_datetime\\\", \\\"tpep_dropoff_datetime\\\"], quoting=3, nrows=1000)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"id\": \"145e7bbe\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"spreadsheet = mss.from_dataframe(modin_df)\\n\",\n \"spreadsheet\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"3c18b7f2\",\n \"metadata\": {},\n \"source\": [\n \"### Thank you for participating!\"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel) with MPI\",\n \"language\": \"python\",\n \"name\": \"python3mpi\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.12\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_unidist/setup_kernel.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport sys\n\nfrom ipykernel import kernelspec\n\ndefault_make_ipkernel_cmd = kernelspec.make_ipkernel_cmd\n\n\ndef custom_make_ipkernel_cmd(*args, **kwargs):\n \"\"\"\n Build modified Popen command list for launching an IPython kernel with MPI.\n\n Parameters\n ----------\n *args : iterable\n Additional positional arguments to be passed in `default_make_ipkernel_cmd`.\n **kwargs : dict\n Additional keyword arguments to be passed in `default_make_ipkernel_cmd`.\n\n Returns\n -------\n array\n A Popen command list.\n\n Notes\n -----\n The parameters of the function should be kept in sync with the ones of the original function.\n \"\"\"\n mpi_arguments = [\"mpiexec\", \"-n\", \"1\"]\n arguments = default_make_ipkernel_cmd(*args, **kwargs)\n return mpi_arguments + arguments\n\n\nkernelspec.make_ipkernel_cmd = custom_make_ipkernel_cmd\n\nif __name__ == \"__main__\":\n kernel_name = \"python3mpi\"\n display_name = \"Python 3 (ipykernel) with MPI\"\n dest = kernelspec.install(\n kernel_name=kernel_name, display_name=display_name, prefix=sys.prefix\n )\n print(f\"Installed kernelspec {kernel_name} in {dest}\") # noqa: T201\n"
},
{
"path": "examples/tutorial/jupyter/execution/pandas_on_unidist/test/test_notebooks.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport os\nimport sys\n\nimport nbformat\n\nMODIN_DIR = os.path.abspath(\n os.path.join(os.path.dirname(__file__), *[\"..\" for _ in range(6)])\n)\nsys.path.insert(0, MODIN_DIR)\nfrom examples.tutorial.jupyter.execution.test.utils import ( # noqa: E402\n _execute_notebook,\n _replace_str,\n download_taxi_dataset,\n set_kernel,\n test_dataset_path,\n)\n\n# the kernel name \"python3mpi\" must match the one\n# that is set up in `examples/tutorial/jupyter/execution/pandas_on_unidist/setup_kernel.py`\n# for `Unidist` engine\nset_kernel(kernel_name=\"python3mpi\")\n\nlocal_notebooks_dir = \"examples/tutorial/jupyter/execution/pandas_on_unidist/local\"\n\n\n# in this notebook user should replace 'import pandas as pd' with\n# 'import modin.pandas as pd' to make notebook work\ndef test_exercise_1():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_1_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_1.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n _replace_str(nb, \"import pandas as pd\", \"import modin.pandas as pd\")\n\n nbformat.write(nb, modified_notebook_path)\n _execute_notebook(modified_notebook_path)\n\n\n# this notebook works \"as is\" but for testing purposes we can use smaller dataset\ndef test_exercise_2():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_2_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_2.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n new_cell = f'path = \"{test_dataset_path}\"\\n' + download_taxi_dataset\n\n _replace_str(\n nb,\n 'path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"',\n new_cell,\n )\n\n nbformat.write(nb, modified_notebook_path)\n _execute_notebook(modified_notebook_path)\n\n\n# in this notebook user should add custom mad implementation\n# to make notebook work\ndef test_exercise_3():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_3_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_3.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n user_mad_implementation = \"\"\"PandasQueryCompiler.sq_mad_custom = TreeReduce.register(lambda cell_value, **kwargs: cell_value ** 2,\n pandas.DataFrame.mad)\n\ndef sq_mad_func(self, axis=None, skipna=True, level=None, **kwargs):\n if axis is None:\n axis = 0\n\n return self._reduce_dimension(\n self._query_compiler.sq_mad_custom(\n axis=axis, skipna=skipna, level=level, **kwargs\n )\n )\n\npd.DataFrame.sq_mad_custom = sq_mad_func\n\nmodin_mad_custom = df.sq_mad_custom()\n \"\"\"\n\n _replace_str(nb, \"modin_mad_custom = ...\", user_mad_implementation)\n\n nbformat.write(nb, modified_notebook_path)\n # need to update example, `.mad` doesn't exist\n # _execute_notebook(modified_notebook_path)\n\n\n# this notebook works \"as is\" but for testing purposes we can use smaller dataset\ndef test_exercise_4():\n modified_notebook_path = os.path.join(local_notebooks_dir, \"exercise_4_test.ipynb\")\n nb = nbformat.read(\n os.path.join(local_notebooks_dir, \"exercise_4.ipynb\"),\n as_version=nbformat.NO_CONVERT,\n )\n\n s3_path_cell = f's3_path = \"{test_dataset_path}\"\\n' + download_taxi_dataset\n _replace_str(\n nb,\n 's3_path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"',\n s3_path_cell,\n )\n\n nbformat.write(nb, modified_notebook_path)\n _execute_notebook(modified_notebook_path)\n"
},
{
"path": "examples/tutorial/jupyter/execution/test/utils.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport nbformat\nfrom nbconvert.preprocessors import ExecutePreprocessor\n\ntest_dataset_path = \"taxi.csv\"\ndownload_taxi_dataset = f\"\"\"import os\nimport urllib.request\nif not os.path.exists(\"{test_dataset_path}\"):\n url_path = \"https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv\"\n urllib.request.urlretrieve(url_path, \"{test_dataset_path}\")\n \"\"\"\n\n\n# Default kernel name for ``ExecutePreprocessor`` to be created\n_default_kernel_name = \"python3\"\n\n\ndef set_kernel(kernel_name):\n \"\"\"\n Set custom kernel for ``ExecutePreprocessor`` to be created.\n\n Parameters\n ----------\n kernel_name : str\n Kernel name.\n \"\"\"\n global _default_kernel_name\n _default_kernel_name = kernel_name\n\n\ndef make_execute_preprocessor():\n \"\"\"\n Make ``ExecutePreprocessor`` with the `_default_kernel_name`.\n\n Returns\n -------\n nbconvert.preprocessors.ExecutePreprocessor\n Execute processor entity.\n\n Notes\n -----\n Note that `_default_kernel_name` can be changed for the concrete executions\n (e.g., ``PandasOnUnidist`` with MPI backend).\n \"\"\"\n return ExecutePreprocessor(timeout=600, kernel_name=_default_kernel_name)\n\n\ndef _execute_notebook(notebook):\n \"\"\"\n Execute a jupyter notebook.\n\n Parameters\n ----------\n notebook : file-like or str\n File-like object or path to the notebook to execute.\n \"\"\"\n nb = nbformat.read(notebook, as_version=nbformat.NO_CONVERT)\n ep = make_execute_preprocessor()\n ep.preprocess(nb)\n\n\ndef _find_code_cell_idx(nb, identifier):\n \"\"\"\n Find code cell index by provided ``identifier``.\n\n Parameters\n ----------\n nb : dict\n Dictionary representation of the notebook to look for.\n identifier : str\n Unique string which target code cell should contain.\n\n Returns\n -------\n int\n Code cell index by provided ``identifier``.\n\n Notes\n -----\n Assertion will be raised if ``identifier`` is found in\n several code cells or isn't found at all.\n \"\"\"\n import_cell_idx = [\n idx\n for idx, cell in enumerate(nb[\"cells\"])\n if cell[\"cell_type\"] == \"code\" and identifier in cell[\"source\"]\n ]\n assert len(import_cell_idx) == 1\n return import_cell_idx[0]\n\n\ndef _replace_str(nb, original_str, str_to_replace):\n \"\"\"\n Replace ``original_str`` with ``str_to_replace`` in the provided notebook.\n\n Parameters\n ----------\n nb : dict\n Dictionary representation of the notebook which requires replacement.\n original_str : str\n Original string which should be replaced.\n str_to_replace : str\n String to replace original string.\n\n Notes\n -----\n Assertion will be raised if ``original_str`` is found in\n several code cells or isn't found at all.\n \"\"\"\n import_cell_idx = _find_code_cell_idx(nb, original_str)\n nb[\"cells\"][import_cell_idx][\"source\"] = nb[\"cells\"][import_cell_idx][\n \"source\"\n ].replace(original_str, str_to_replace)\n"
},
{
"path": "modin/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport warnings\nfrom typing import Any, Optional, Tuple, Type, Union\n\nfrom . import _version\n\n\ndef custom_formatwarning(\n message: Union[Warning, str],\n category: Type[Warning],\n filename: str,\n lineno: int,\n line: Optional[str] = None,\n) -> str:\n # ignore everything except the message\n return \"{}: {}\\n\".format(category.__name__, message)\n\n\nwarnings.formatwarning = custom_formatwarning\n# Filter numpy version warnings because they are not relevant\nwarnings.filterwarnings(\"ignore\", message=\"numpy.dtype size changed\")\nwarnings.filterwarnings(\"ignore\", message=\"Large object of size\")\n\n\ndef set_execution(engine: Any = None, storage_format: Any = None) -> Tuple[Any, Any]:\n \"\"\"\n Method to set the _pair_ of execution engine and storage format format simultaneously.\n This is needed because there might be cases where switching one by one would be\n impossible, as not all pairs of values are meaningful.\n\n The method returns pair of old values, so it is easy to return back.\n \"\"\"\n from .config import Backend, Engine, Execution, StorageFormat\n\n old_engine, old_storage_format = None, None\n # defer callbacks until both entities are set\n if engine is not None:\n old_engine = Engine._put_nocallback(engine)\n if storage_format is not None:\n old_storage_format = StorageFormat._put_nocallback(storage_format)\n # execute callbacks if something was changed\n if old_engine is not None:\n Engine._check_callbacks(old_engine)\n if old_storage_format is not None:\n StorageFormat._check_callbacks(old_storage_format)\n old_backend = Backend.get()\n Backend._put_nocallback(\n Backend.get_backend_for_execution(\n Execution(engine=Engine.get(), storage_format=StorageFormat.get())\n )\n )\n Backend._check_callbacks(old_backend)\n\n return old_engine, old_storage_format\n\n\n__version__ = _version.get_versions()[\"version\"]\n"
},
{
"path": "modin/__main__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Command-line interface piece, called when user issues \"python -m modin --foo\".\"\"\"\n\nimport argparse\n\n\ndef main() -> None:\n parser = argparse.ArgumentParser(\n \"python -m modin\",\n description=\"Drop-in pandas replacement; refer to https://modin.readthedocs.io/ for details.\",\n )\n parser.add_argument(\n \"--versions\",\n action=\"store_true\",\n default=False,\n help=\"Show versions of all known components\",\n )\n\n args = parser.parse_args()\n if args.versions:\n from modin.utils import show_versions\n\n show_versions()\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "modin/_version.py",
"content": "# This file helps to compute a version number in source trees obtained from\n# git-archive tarball (such as those provided by githubs download-from-tag\n# feature). Distribution tarballs (built by setup.py sdist) and build\n# directories (produced by setup.py build) will contain a much shorter file\n# that just contains the computed version number.\n\n# This file is released into the public domain.\n# Generated by versioneer-0.29\n# https://github.com/python-versioneer/python-versioneer\n\n\"\"\"Git implementation of _version.py.\"\"\"\n\nimport errno\nimport functools\nimport os\nimport re\nimport subprocess\nimport sys\nfrom typing import Any, Callable, Dict, List, Optional, Tuple\n\n\ndef get_keywords() -> Dict[str, str]:\n \"\"\"Get the keywords needed to look up the version information.\"\"\"\n # these strings will be replaced by git during git-archive.\n # setup.py/versioneer.py will grep for the variable names, so they must\n # each be defined on a line of their own. _version.py will just call\n # get_keywords().\n git_refnames = \"$Format:%d$\"\n git_full = \"$Format:%H$\"\n git_date = \"$Format:%ci$\"\n keywords = {\"refnames\": git_refnames, \"full\": git_full, \"date\": git_date}\n return keywords\n\n\nclass VersioneerConfig:\n \"\"\"Container for Versioneer configuration parameters.\"\"\"\n\n VCS: str\n style: str\n tag_prefix: str\n parentdir_prefix: str\n versionfile_source: str\n verbose: bool\n\n\ndef get_config() -> VersioneerConfig:\n \"\"\"Create, populate and return the VersioneerConfig() object.\"\"\"\n # these strings are filled in when 'setup.py versioneer' creates\n # _version.py\n cfg = VersioneerConfig()\n cfg.VCS = \"git\"\n cfg.style = \"pep440\"\n cfg.tag_prefix = \"\"\n cfg.parentdir_prefix = \"modin-\"\n cfg.versionfile_source = \"modin/_version.py\"\n cfg.verbose = False\n return cfg\n\n\nclass NotThisMethod(Exception):\n \"\"\"Exception raised if a method is not valid for the current scenario.\"\"\"\n\n\nLONG_VERSION_PY: Dict[str, str] = {}\nHANDLERS: Dict[str, Dict[str, Callable]] = {}\n\n\ndef register_vcs_handler(vcs: str, method: str) -> Callable: # decorator\n \"\"\"Create decorator to mark a method as the handler of a VCS.\"\"\"\n\n def decorate(f: Callable) -> Callable:\n \"\"\"Store f in HANDLERS[vcs][method].\"\"\"\n if vcs not in HANDLERS:\n HANDLERS[vcs] = {}\n HANDLERS[vcs][method] = f\n return f\n\n return decorate\n\n\ndef run_command(\n commands: List[str],\n args: List[str],\n cwd: Optional[str] = None,\n verbose: bool = False,\n hide_stderr: bool = False,\n env: Optional[Dict[str, str]] = None,\n) -> Tuple[Optional[str], Optional[int]]:\n \"\"\"Call the given command(s).\"\"\"\n assert isinstance(commands, list)\n process = None\n\n popen_kwargs: Dict[str, Any] = {}\n if sys.platform == \"win32\":\n # This hides the console window if pythonw.exe is used\n startupinfo = subprocess.STARTUPINFO()\n startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW\n popen_kwargs[\"startupinfo\"] = startupinfo\n\n for command in commands:\n try:\n dispcmd = str([command] + args)\n # remember shell=False, so use git.cmd on windows, not just git\n process = subprocess.Popen(\n [command] + args,\n cwd=cwd,\n env=env,\n stdout=subprocess.PIPE,\n stderr=(subprocess.PIPE if hide_stderr else None),\n **popen_kwargs,\n )\n break\n except OSError as e:\n if e.errno == errno.ENOENT:\n continue\n if verbose:\n print(\"unable to run %s\" % dispcmd)\n print(e)\n return None, None\n else:\n if verbose:\n print(\"unable to find command, tried %s\" % (commands,))\n return None, None\n stdout = process.communicate()[0].strip().decode()\n if process.returncode != 0:\n if verbose:\n print(\"unable to run %s (error)\" % dispcmd)\n print(\"stdout was %s\" % stdout)\n return None, process.returncode\n return stdout, process.returncode\n\n\ndef versions_from_parentdir(\n parentdir_prefix: str,\n root: str,\n verbose: bool,\n) -> Dict[str, Any]:\n \"\"\"Try to determine the version from the parent directory name.\n\n Source tarballs conventionally unpack into a directory that includes both\n the project name and a version string. We will also support searching up\n two directory levels for an appropriately named parent directory\n \"\"\"\n rootdirs = []\n\n for _ in range(3):\n dirname = os.path.basename(root)\n if dirname.startswith(parentdir_prefix):\n return {\n \"version\": dirname[len(parentdir_prefix) :],\n \"full-revisionid\": None,\n \"dirty\": False,\n \"error\": None,\n \"date\": None,\n }\n rootdirs.append(root)\n root = os.path.dirname(root) # up a level\n\n if verbose:\n print(\n \"Tried directories %s but none started with prefix %s\"\n % (str(rootdirs), parentdir_prefix)\n )\n raise NotThisMethod(\"rootdir doesn't start with parentdir_prefix\")\n\n\n@register_vcs_handler(\"git\", \"get_keywords\")\ndef git_get_keywords(versionfile_abs: str) -> Dict[str, str]:\n \"\"\"Extract version information from the given file.\"\"\"\n # the code embedded in _version.py can just fetch the value of these\n # keywords. When used from setup.py, we don't want to import _version.py,\n # so we do it with a regexp instead. This function is not used from\n # _version.py.\n keywords: Dict[str, str] = {}\n try:\n with open(versionfile_abs, \"r\") as fobj:\n for line in fobj:\n if line.strip().startswith(\"git_refnames =\"):\n mo = re.search(r'=\\s*\"(.*)\"', line)\n if mo:\n keywords[\"refnames\"] = mo.group(1)\n if line.strip().startswith(\"git_full =\"):\n mo = re.search(r'=\\s*\"(.*)\"', line)\n if mo:\n keywords[\"full\"] = mo.group(1)\n if line.strip().startswith(\"git_date =\"):\n mo = re.search(r'=\\s*\"(.*)\"', line)\n if mo:\n keywords[\"date\"] = mo.group(1)\n except OSError:\n pass\n return keywords\n\n\n@register_vcs_handler(\"git\", \"keywords\")\ndef git_versions_from_keywords(\n keywords: Dict[str, str],\n tag_prefix: str,\n verbose: bool,\n) -> Dict[str, Any]:\n \"\"\"Get version information from git keywords.\"\"\"\n if \"refnames\" not in keywords:\n raise NotThisMethod(\"Short version file found\")\n date = keywords.get(\"date\")\n if date is not None:\n # Use only the last line. Previous lines may contain GPG signature\n # information.\n date = date.splitlines()[-1]\n\n # git-2.2.0 added \"%cI\", which expands to an ISO-8601 -compliant\n # datestamp. However we prefer \"%ci\" (which expands to an \"ISO-8601\n # -like\" string, which we must then edit to make compliant), because\n # it's been around since git-1.5.3, and it's too difficult to\n # discover which version we're using, or to work around using an\n # older one.\n date = date.strip().replace(\" \", \"T\", 1).replace(\" \", \"\", 1)\n refnames = keywords[\"refnames\"].strip()\n if refnames.startswith(\"$Format\"):\n if verbose:\n print(\"keywords are unexpanded, not using\")\n raise NotThisMethod(\"unexpanded keywords, not a git-archive tarball\")\n refs = {r.strip() for r in refnames.strip(\"()\").split(\",\")}\n # starting in git-1.8.3, tags are listed as \"tag: foo-1.0\" instead of\n # just \"foo-1.0\". If we see a \"tag: \" prefix, prefer those.\n TAG = \"tag: \"\n tags = {r[len(TAG) :] for r in refs if r.startswith(TAG)}\n if not tags:\n # Either we're using git < 1.8.3, or there really are no tags. We use\n # a heuristic: assume all version tags have a digit. The old git %d\n # expansion behaves like git log --decorate=short and strips out the\n # refs/heads/ and refs/tags/ prefixes that would let us distinguish\n # between branches and tags. By ignoring refnames without digits, we\n # filter out many common branch names like \"release\" and\n # \"stabilization\", as well as \"HEAD\" and \"master\".\n tags = {r for r in refs if re.search(r\"\\d\", r)}\n if verbose:\n print(\"discarding '%s', no digits\" % \",\".join(refs - tags))\n if verbose:\n print(\"likely tags: %s\" % \",\".join(sorted(tags)))\n for ref in sorted(tags):\n # sorting will prefer e.g. \"2.0\" over \"2.0rc1\"\n if ref.startswith(tag_prefix):\n r = ref[len(tag_prefix) :]\n # Filter out refs that exactly match prefix or that don't start\n # with a number once the prefix is stripped (mostly a concern\n # when prefix is '')\n if not re.match(r\"\\d\", r):\n continue\n if verbose:\n print(\"picking %s\" % r)\n return {\n \"version\": r,\n \"full-revisionid\": keywords[\"full\"].strip(),\n \"dirty\": False,\n \"error\": None,\n \"date\": date,\n }\n # no suitable tags, so version is \"0+unknown\", but full hex is still there\n if verbose:\n print(\"no suitable tags, using unknown + full revision id\")\n return {\n \"version\": \"0+unknown\",\n \"full-revisionid\": keywords[\"full\"].strip(),\n \"dirty\": False,\n \"error\": \"no suitable tags\",\n \"date\": None,\n }\n\n\n@register_vcs_handler(\"git\", \"pieces_from_vcs\")\ndef git_pieces_from_vcs(\n tag_prefix: str, root: str, verbose: bool, runner: Callable = run_command\n) -> Dict[str, Any]:\n \"\"\"Get version from 'git describe' in the root of the source tree.\n\n This only gets called if the git-archive 'subst' keywords were *not*\n expanded, and _version.py hasn't already been rewritten with a short\n version string, meaning we're inside a checked out source tree.\n \"\"\"\n GITS = [\"git\"]\n if sys.platform == \"win32\":\n GITS = [\"git.cmd\", \"git.exe\"]\n\n # GIT_DIR can interfere with correct operation of Versioneer.\n # It may be intended to be passed to the Versioneer-versioned project,\n # but that should not change where we get our version from.\n env = os.environ.copy()\n env.pop(\"GIT_DIR\", None)\n runner = functools.partial(runner, env=env)\n\n _, rc = runner(GITS, [\"rev-parse\", \"--git-dir\"], cwd=root, hide_stderr=not verbose)\n if rc != 0:\n if verbose:\n print(\"Directory %s not under git control\" % root)\n raise NotThisMethod(\"'git rev-parse --git-dir' returned error\")\n\n # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty]\n # if there isn't one, this yields HEX[-dirty] (no NUM)\n describe_out, rc = runner(\n GITS,\n [\n \"describe\",\n \"--tags\",\n \"--dirty\",\n \"--always\",\n \"--long\",\n \"--match\",\n f\"{tag_prefix}[[:digit:]]*\",\n ],\n cwd=root,\n )\n # --long was added in git-1.5.5\n if describe_out is None:\n raise NotThisMethod(\"'git describe' failed\")\n describe_out = describe_out.strip()\n full_out, rc = runner(GITS, [\"rev-parse\", \"HEAD\"], cwd=root)\n if full_out is None:\n raise NotThisMethod(\"'git rev-parse' failed\")\n full_out = full_out.strip()\n\n pieces: Dict[str, Any] = {}\n pieces[\"long\"] = full_out\n pieces[\"short\"] = full_out[:7] # maybe improved later\n pieces[\"error\"] = None\n\n branch_name, rc = runner(GITS, [\"rev-parse\", \"--abbrev-ref\", \"HEAD\"], cwd=root)\n # --abbrev-ref was added in git-1.6.3\n if rc != 0 or branch_name is None:\n raise NotThisMethod(\"'git rev-parse --abbrev-ref' returned error\")\n branch_name = branch_name.strip()\n\n if branch_name == \"HEAD\":\n # If we aren't exactly on a branch, pick a branch which represents\n # the current commit. If all else fails, we are on a branchless\n # commit.\n branches, rc = runner(GITS, [\"branch\", \"--contains\"], cwd=root)\n # --contains was added in git-1.5.4\n if rc != 0 or branches is None:\n raise NotThisMethod(\"'git branch --contains' returned error\")\n branches = branches.split(\"\\n\")\n\n # Remove the first line if we're running detached\n if \"(\" in branches[0]:\n branches.pop(0)\n\n # Strip off the leading \"* \" from the list of branches.\n branches = [branch[2:] for branch in branches]\n if \"master\" in branches:\n branch_name = \"master\"\n elif not branches:\n branch_name = None\n else:\n # Pick the first branch that is returned. Good or bad.\n branch_name = branches[0]\n\n pieces[\"branch\"] = branch_name\n\n # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty]\n # TAG might have hyphens.\n git_describe = describe_out\n\n # look for -dirty suffix\n dirty = git_describe.endswith(\"-dirty\")\n pieces[\"dirty\"] = dirty\n if dirty:\n git_describe = git_describe[: git_describe.rindex(\"-dirty\")]\n\n # now we have TAG-NUM-gHEX or HEX\n\n if \"-\" in git_describe:\n # TAG-NUM-gHEX\n mo = re.search(r\"^(.+)-(\\d+)-g([0-9a-f]+)$\", git_describe)\n if not mo:\n # unparsable. Maybe git-describe is misbehaving?\n pieces[\"error\"] = \"unable to parse git-describe output: '%s'\" % describe_out\n return pieces\n\n # tag\n full_tag = mo.group(1)\n if not full_tag.startswith(tag_prefix):\n if verbose:\n fmt = \"tag '%s' doesn't start with prefix '%s'\"\n print(fmt % (full_tag, tag_prefix))\n pieces[\"error\"] = \"tag '%s' doesn't start with prefix '%s'\" % (\n full_tag,\n tag_prefix,\n )\n return pieces\n pieces[\"closest-tag\"] = full_tag[len(tag_prefix) :]\n\n # distance: number of commits since tag\n pieces[\"distance\"] = int(mo.group(2))\n\n # commit: short hex revision ID\n pieces[\"short\"] = mo.group(3)\n\n else:\n # HEX: no tags\n pieces[\"closest-tag\"] = None\n out, rc = runner(GITS, [\"rev-list\", \"HEAD\", \"--left-right\"], cwd=root)\n pieces[\"distance\"] = len(out.split()) # total number of commits\n\n # commit date: see ISO-8601 comment in git_versions_from_keywords()\n date = runner(GITS, [\"show\", \"-s\", \"--format=%ci\", \"HEAD\"], cwd=root)[0].strip()\n # Use only the last line. Previous lines may contain GPG signature\n # information.\n date = date.splitlines()[-1]\n pieces[\"date\"] = date.strip().replace(\" \", \"T\", 1).replace(\" \", \"\", 1)\n\n return pieces\n\n\ndef plus_or_dot(pieces: Dict[str, Any]) -> str:\n \"\"\"Return a + if we don't already have one, else return a .\"\"\"\n if \"+\" in pieces.get(\"closest-tag\", \"\"):\n return \".\"\n return \"+\"\n\n\ndef render_pep440(pieces: Dict[str, Any]) -> str:\n \"\"\"Build up version string, with post-release \"local version identifier\".\n\n Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you\n get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty\n\n Exceptions:\n 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty]\n \"\"\"\n if pieces[\"closest-tag\"]:\n rendered = pieces[\"closest-tag\"]\n if pieces[\"distance\"] or pieces[\"dirty\"]:\n rendered += plus_or_dot(pieces)\n rendered += \"%d.g%s\" % (pieces[\"distance\"], pieces[\"short\"])\n if pieces[\"dirty\"]:\n rendered += \".dirty\"\n else:\n # exception #1\n rendered = \"0+untagged.%d.g%s\" % (pieces[\"distance\"], pieces[\"short\"])\n if pieces[\"dirty\"]:\n rendered += \".dirty\"\n return rendered\n\n\ndef render_pep440_branch(pieces: Dict[str, Any]) -> str:\n \"\"\"TAG[[.dev0]+DISTANCE.gHEX[.dirty]] .\n\n The \".dev0\" means not master branch. Note that .dev0 sorts backwards\n (a feature branch will appear \"older\" than the master branch).\n\n Exceptions:\n 1: no tags. 0[.dev0]+untagged.DISTANCE.gHEX[.dirty]\n \"\"\"\n if pieces[\"closest-tag\"]:\n rendered = pieces[\"closest-tag\"]\n if pieces[\"distance\"] or pieces[\"dirty\"]:\n if pieces[\"branch\"] != \"master\":\n rendered += \".dev0\"\n rendered += plus_or_dot(pieces)\n rendered += \"%d.g%s\" % (pieces[\"distance\"], pieces[\"short\"])\n if pieces[\"dirty\"]:\n rendered += \".dirty\"\n else:\n # exception #1\n rendered = \"0\"\n if pieces[\"branch\"] != \"master\":\n rendered += \".dev0\"\n rendered += \"+untagged.%d.g%s\" % (pieces[\"distance\"], pieces[\"short\"])\n if pieces[\"dirty\"]:\n rendered += \".dirty\"\n return rendered\n\n\ndef pep440_split_post(ver: str) -> Tuple[str, Optional[int]]:\n \"\"\"Split pep440 version string at the post-release segment.\n\n Returns the release segments before the post-release and the\n post-release version number (or -1 if no post-release segment is present).\n \"\"\"\n vc = str.split(ver, \".post\")\n return vc[0], int(vc[1] or 0) if len(vc) == 2 else None\n\n\ndef render_pep440_pre(pieces: Dict[str, Any]) -> str:\n \"\"\"TAG[.postN.devDISTANCE] -- No -dirty.\n\n Exceptions:\n 1: no tags. 0.post0.devDISTANCE\n \"\"\"\n if pieces[\"closest-tag\"]:\n if pieces[\"distance\"]:\n # update the post release segment\n tag_version, post_version = pep440_split_post(pieces[\"closest-tag\"])\n rendered = tag_version\n if post_version is not None:\n rendered += \".post%d.dev%d\" % (post_version + 1, pieces[\"distance\"])\n else:\n rendered += \".post0.dev%d\" % (pieces[\"distance\"])\n else:\n # no commits, use the tag as the version\n rendered = pieces[\"closest-tag\"]\n else:\n # exception #1\n rendered = \"0.post0.dev%d\" % pieces[\"distance\"]\n return rendered\n\n\ndef render_pep440_post(pieces: Dict[str, Any]) -> str:\n \"\"\"TAG[.postDISTANCE[.dev0]+gHEX] .\n\n The \".dev0\" means dirty. Note that .dev0 sorts backwards\n (a dirty tree will appear \"older\" than the corresponding clean one),\n but you shouldn't be releasing software with -dirty anyways.\n\n Exceptions:\n 1: no tags. 0.postDISTANCE[.dev0]\n \"\"\"\n if pieces[\"closest-tag\"]:\n rendered = pieces[\"closest-tag\"]\n if pieces[\"distance\"] or pieces[\"dirty\"]:\n rendered += \".post%d\" % pieces[\"distance\"]\n if pieces[\"dirty\"]:\n rendered += \".dev0\"\n rendered += plus_or_dot(pieces)\n rendered += \"g%s\" % pieces[\"short\"]\n else:\n # exception #1\n rendered = \"0.post%d\" % pieces[\"distance\"]\n if pieces[\"dirty\"]:\n rendered += \".dev0\"\n rendered += \"+g%s\" % pieces[\"short\"]\n return rendered\n\n\ndef render_pep440_post_branch(pieces: Dict[str, Any]) -> str:\n \"\"\"TAG[.postDISTANCE[.dev0]+gHEX[.dirty]] .\n\n The \".dev0\" means not master branch.\n\n Exceptions:\n 1: no tags. 0.postDISTANCE[.dev0]+gHEX[.dirty]\n \"\"\"\n if pieces[\"closest-tag\"]:\n rendered = pieces[\"closest-tag\"]\n if pieces[\"distance\"] or pieces[\"dirty\"]:\n rendered += \".post%d\" % pieces[\"distance\"]\n if pieces[\"branch\"] != \"master\":\n rendered += \".dev0\"\n rendered += plus_or_dot(pieces)\n rendered += \"g%s\" % pieces[\"short\"]\n if pieces[\"dirty\"]:\n rendered += \".dirty\"\n else:\n # exception #1\n rendered = \"0.post%d\" % pieces[\"distance\"]\n if pieces[\"branch\"] != \"master\":\n rendered += \".dev0\"\n rendered += \"+g%s\" % pieces[\"short\"]\n if pieces[\"dirty\"]:\n rendered += \".dirty\"\n return rendered\n\n\ndef render_pep440_old(pieces: Dict[str, Any]) -> str:\n \"\"\"TAG[.postDISTANCE[.dev0]] .\n\n The \".dev0\" means dirty.\n\n Exceptions:\n 1: no tags. 0.postDISTANCE[.dev0]\n \"\"\"\n if pieces[\"closest-tag\"]:\n rendered = pieces[\"closest-tag\"]\n if pieces[\"distance\"] or pieces[\"dirty\"]:\n rendered += \".post%d\" % pieces[\"distance\"]\n if pieces[\"dirty\"]:\n rendered += \".dev0\"\n else:\n # exception #1\n rendered = \"0.post%d\" % pieces[\"distance\"]\n if pieces[\"dirty\"]:\n rendered += \".dev0\"\n return rendered\n\n\ndef render_git_describe(pieces: Dict[str, Any]) -> str:\n \"\"\"TAG[-DISTANCE-gHEX][-dirty].\n\n Like 'git describe --tags --dirty --always'.\n\n Exceptions:\n 1: no tags. HEX[-dirty] (note: no 'g' prefix)\n \"\"\"\n if pieces[\"closest-tag\"]:\n rendered = pieces[\"closest-tag\"]\n if pieces[\"distance\"]:\n rendered += \"-%d-g%s\" % (pieces[\"distance\"], pieces[\"short\"])\n else:\n # exception #1\n rendered = pieces[\"short\"]\n if pieces[\"dirty\"]:\n rendered += \"-dirty\"\n return rendered\n\n\ndef render_git_describe_long(pieces: Dict[str, Any]) -> str:\n \"\"\"TAG-DISTANCE-gHEX[-dirty].\n\n Like 'git describe --tags --dirty --always -long'.\n The distance/hash is unconditional.\n\n Exceptions:\n 1: no tags. HEX[-dirty] (note: no 'g' prefix)\n \"\"\"\n if pieces[\"closest-tag\"]:\n rendered = pieces[\"closest-tag\"]\n rendered += \"-%d-g%s\" % (pieces[\"distance\"], pieces[\"short\"])\n else:\n # exception #1\n rendered = pieces[\"short\"]\n if pieces[\"dirty\"]:\n rendered += \"-dirty\"\n return rendered\n\n\ndef render(pieces: Dict[str, Any], style: str) -> Dict[str, Any]:\n \"\"\"Render the given version pieces into the requested style.\"\"\"\n if pieces[\"error\"]:\n return {\n \"version\": \"unknown\",\n \"full-revisionid\": pieces.get(\"long\"),\n \"dirty\": None,\n \"error\": pieces[\"error\"],\n \"date\": None,\n }\n\n if not style or style == \"default\":\n style = \"pep440\" # the default\n\n if style == \"pep440\":\n rendered = render_pep440(pieces)\n elif style == \"pep440-branch\":\n rendered = render_pep440_branch(pieces)\n elif style == \"pep440-pre\":\n rendered = render_pep440_pre(pieces)\n elif style == \"pep440-post\":\n rendered = render_pep440_post(pieces)\n elif style == \"pep440-post-branch\":\n rendered = render_pep440_post_branch(pieces)\n elif style == \"pep440-old\":\n rendered = render_pep440_old(pieces)\n elif style == \"git-describe\":\n rendered = render_git_describe(pieces)\n elif style == \"git-describe-long\":\n rendered = render_git_describe_long(pieces)\n else:\n raise ValueError(\"unknown style '%s'\" % style)\n\n return {\n \"version\": rendered,\n \"full-revisionid\": pieces[\"long\"],\n \"dirty\": pieces[\"dirty\"],\n \"error\": None,\n \"date\": pieces.get(\"date\"),\n }\n\n\ndef get_versions() -> Dict[str, Any]:\n \"\"\"Get version information or return default if unable to do so.\"\"\"\n # I am in _version.py, which lives at ROOT/VERSIONFILE_SOURCE. If we have\n # __file__, we can work backwards from there to the root. Some\n # py2exe/bbfreeze/non-CPython implementations don't do __file__, in which\n # case we can only use expanded keywords.\n\n cfg = get_config()\n verbose = cfg.verbose\n\n try:\n return git_versions_from_keywords(get_keywords(), cfg.tag_prefix, verbose)\n except NotThisMethod:\n pass\n\n try:\n root = os.path.realpath(__file__)\n # versionfile_source is the relative path from the top of the source\n # tree (where the .git directory might live) to this file. Invert\n # this to find the root from __file__.\n for _ in cfg.versionfile_source.split(\"/\"):\n root = os.path.dirname(root)\n except NameError:\n return {\n \"version\": \"0+unknown\",\n \"full-revisionid\": None,\n \"dirty\": None,\n \"error\": \"unable to find root of source tree\",\n \"date\": None,\n }\n\n try:\n pieces = git_pieces_from_vcs(cfg.tag_prefix, root, verbose)\n return render(pieces, cfg.style)\n except NotThisMethod:\n pass\n\n try:\n if cfg.parentdir_prefix:\n return versions_from_parentdir(cfg.parentdir_prefix, root, verbose)\n except NotThisMethod:\n pass\n\n return {\n \"version\": \"0+unknown\",\n \"full-revisionid\": None,\n \"dirty\": None,\n \"error\": \"unable to compute version\",\n \"date\": None,\n }\n"
},
{
"path": "modin/config/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses config entities which can be used for Modin behavior tuning.\"\"\"\n\nfrom modin.config.envvars import (\n AsvDataSizeConfig,\n AsvImplementation,\n AsyncReadMode,\n AutoSwitchBackend,\n Backend,\n BackendJoinConsiderAllBackends,\n BackendMergeCastInPlace,\n BenchmarkMode,\n CIAWSAccessKeyID,\n CIAWSSecretAccessKey,\n CpuCount,\n DaskThreadsPerWorker,\n DocModule,\n DynamicPartitioning,\n Engine,\n EnvironmentVariable,\n Execution,\n GithubCI,\n GpuCount,\n IsDebug,\n IsExperimental,\n IsRayCluster,\n LazyExecution,\n LogFileSize,\n LogMemoryInterval,\n LogMode,\n Memory,\n MetricsMode,\n MinColumnPartitionSize,\n MinPartitionSize,\n MinRowPartitionSize,\n ModinNumpy,\n NativePandasDeepCopy,\n NativePandasMaxRows,\n NativePandasTransferThreshold,\n NPartitions,\n PersistentPickle,\n ProgressBar,\n RangePartitioning,\n RayInitCustomResources,\n RayRedisAddress,\n RayRedisPassword,\n RayTaskCustomResources,\n ReadSqlEngine,\n ShowBackendSwitchProgress,\n StorageFormat,\n TestDatasetSize,\n TestReadFromPostgres,\n TestReadFromSqlServer,\n TrackFileLeaks,\n)\nfrom modin.config.pubsub import Parameter, ValueSource, context\n\n__all__ = [\n \"EnvironmentVariable\",\n \"Parameter\",\n \"ValueSource\",\n \"context\",\n # General settings\n \"IsDebug\",\n \"Engine\",\n \"StorageFormat\",\n \"CpuCount\",\n \"GpuCount\",\n \"Memory\",\n \"Backend\",\n \"BackendJoinConsiderAllBackends\",\n \"BackendMergeCastInPlace\",\n \"Execution\",\n \"AutoSwitchBackend\",\n \"ShowBackendSwitchProgress\",\n # Ray specific\n \"IsRayCluster\",\n \"RayRedisAddress\",\n \"RayRedisPassword\",\n \"RayInitCustomResources\",\n \"RayTaskCustomResources\",\n \"LazyExecution\",\n # Dask specific\n \"DaskThreadsPerWorker\",\n # Native Pandas Specific\n \"NativePandasMaxRows\",\n \"NativePandasTransferThreshold\",\n \"NativePandasDeepCopy\",\n # Partitioning\n \"NPartitions\",\n \"MinPartitionSize\",\n \"MinRowPartitionSize\",\n \"MinColumnPartitionSize\",\n # ASV specific\n \"TestDatasetSize\",\n \"AsvImplementation\",\n \"AsvDataSizeConfig\",\n # Specific features\n \"ProgressBar\",\n \"BenchmarkMode\",\n \"PersistentPickle\",\n \"ModinNumpy\",\n \"RangePartitioning\",\n \"AsyncReadMode\",\n \"ReadSqlEngine\",\n \"IsExperimental\",\n \"DynamicPartitioning\",\n # For tests\n \"TrackFileLeaks\",\n \"TestReadFromSqlServer\",\n \"TestReadFromPostgres\",\n \"GithubCI\",\n \"CIAWSSecretAccessKey\",\n \"CIAWSAccessKeyID\",\n # Logging\n \"LogMode\",\n \"LogMemoryInterval\",\n \"LogFileSize\",\n \"MetricsMode\",\n # Plugin settings\n \"DocModule\",\n]\n"
},
{
"path": "modin/config/__main__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nContent of this file should be executed if module `modin.config` is called.\n\nIf module is called (using `python -m modin.config`) configs help will be printed.\nUsing `-export_path` option configs description can be exported to the external CSV file\nprovided with this flag.\n\"\"\"\n\nimport argparse\nfrom textwrap import dedent\n\nimport pandas\n\nimport modin.config as cfg\n\n\ndef print_config_help() -> None:\n \"\"\"Print configs help messages.\"\"\"\n for objname in sorted(cfg.__all__):\n obj = getattr(cfg, objname)\n if (\n isinstance(obj, type)\n and issubclass(obj, cfg.Parameter)\n and not obj.is_abstract\n ):\n print(f\"{obj.get_help()}\\n\\tCurrent value: {obj.get()}\") # noqa: T201\n\n\ndef export_config_help(filename: str) -> None:\n \"\"\"\n Export all configs help messages to the CSV file.\n\n Parameters\n ----------\n filename : str\n Name of the file to export configs data.\n \"\"\"\n configs_data = []\n default_values = dict(\n RayRedisPassword=\"random string\",\n CpuCount=\"multiprocessing.cpu_count()\",\n NPartitions=\"equals to MODIN_CPUS env\",\n )\n for objname in sorted(cfg.__all__):\n obj = getattr(cfg, objname)\n if (\n isinstance(obj, type)\n and issubclass(obj, cfg.Parameter)\n and not obj.is_abstract\n ):\n data = {\n \"Config Name\": obj.__name__,\n \"Env. Variable Name\": getattr(\n obj, \"varname\", \"not backed by environment\"\n ),\n \"Default Value\": default_values.get(obj.__name__, obj._get_default()),\n # `Notes` `-` underlining can't be correctly parsed inside csv table by sphinx\n \"Description\": dedent(obj.__doc__ or \"\").replace(\n \"Notes\\n-----\", \"Notes:\\n\"\n ),\n \"Options\": obj.choices,\n }\n configs_data.append(data)\n\n pandas.DataFrame(\n configs_data,\n columns=[\n \"Config Name\",\n \"Env. Variable Name\",\n \"Default Value\",\n \"Description\",\n \"Options\",\n ],\n ).to_csv(filename, index=False)\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\n \"--export-path\",\n dest=\"export_path\",\n type=str,\n required=False,\n default=None,\n help=\"File path to export configs data.\",\n )\n export_path = parser.parse_args().export_path\n if export_path:\n export_config_help(export_path)\n else:\n print_config_help()\n"
},
{
"path": "modin/config/envvars.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses Modin configs originated from environment variables.\"\"\"\n\nimport os\nimport secrets\nimport sys\nimport warnings\nfrom collections import namedtuple\nfrom textwrap import dedent\nfrom typing import Any, NoReturn, Optional\n\nfrom packaging import version\nfrom pandas.util._decorators import doc # type: ignore[attr-defined]\n\nfrom modin import set_execution\nfrom modin.config.pubsub import (\n _TYPE_PARAMS,\n _UNSET,\n DeprecationDescriptor,\n ExactStr,\n Parameter,\n ValueSource,\n)\n\n\nclass EnvironmentVariable(Parameter, type=str, abstract=True):\n \"\"\"Base class for environment variables-based configuration.\"\"\"\n\n varname: Optional[str] = None\n\n @classmethod\n def _get_value_from_config(cls) -> Any:\n \"\"\"\n Read the value from environment variable.\n\n Returns\n -------\n Any\n Config raw value if it's set, otherwise `_UNSET`.\n \"\"\"\n if cls.varname is None:\n raise TypeError(\"varname should not be None\")\n if cls.varname not in os.environ:\n return _UNSET\n raw = os.environ[cls.varname]\n if not _TYPE_PARAMS[cls.type].verify(raw):\n # TODO: use and test a better error message, like \"Invalid value\n # for {cls.varname}: {raw}\"\n raise ValueError(f\"Unsupported raw value: {raw}\")\n return _TYPE_PARAMS[cls.type].decode(raw)\n\n @classmethod\n def get_help(cls) -> str:\n \"\"\"\n Generate user-presentable help for the config.\n\n Returns\n -------\n str\n \"\"\"\n help = f\"{cls.varname}: {dedent(cls.__doc__ or 'Unknown').strip()}\\n\\tProvide {_TYPE_PARAMS[cls.type].help}\"\n if cls.choices:\n help += f\" (valid examples are: {', '.join(str(c) for c in cls.choices)})\"\n return help\n\n\nclass EnvWithSibilings(\n EnvironmentVariable,\n # 'type' is a mandatory parameter for '__init_subclasses__', so we have to pass something here,\n # this doesn't force child classes to have 'str' type though, they actually can be any type\n type=str,\n):\n \"\"\"Ensure values synchronization between sibling parameters.\"\"\"\n\n _update_sibling = True\n\n @classmethod\n def _sibling(cls) -> type[\"EnvWithSibilings\"]:\n \"\"\"Return a sibling parameter.\"\"\"\n raise NotImplementedError()\n\n @classmethod\n def get(cls) -> Any:\n \"\"\"\n Get parameter's value and ensure that it's equal to the sibling's value.\n\n Returns\n -------\n Any\n \"\"\"\n sibling = cls._sibling()\n\n if sibling._value is _UNSET and cls._value is _UNSET:\n super().get()\n with warnings.catch_warnings():\n # filter warnings that can potentially come from the potentially deprecated sibling\n warnings.filterwarnings(\"ignore\", category=FutureWarning)\n super(EnvWithSibilings, sibling).get()\n\n if (\n cls._value_source\n == sibling._value_source\n == ValueSource.GOT_FROM_CFG_SOURCE\n ):\n raise ValueError(\n f\"Configuration is ambiguous. You cannot set '{cls.varname}' and '{sibling.varname}' at the same time.\"\n )\n\n # further we assume that there are only two valid sources for the variables: 'GOT_FROM_CFG' and 'DEFAULT',\n # as otherwise we wouldn't ended-up in this branch at all, because all other ways of setting a value\n # changes the '._value' attribute from '_UNSET' to something meaningful\n from modin.error_message import ErrorMessage\n\n if cls._value_source == ValueSource.GOT_FROM_CFG_SOURCE:\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=sibling._value_source != ValueSource.DEFAULT\n )\n sibling._value = cls._value\n sibling._value_source = ValueSource.GOT_FROM_CFG_SOURCE\n elif sibling._value_source == ValueSource.GOT_FROM_CFG_SOURCE:\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=cls._value_source != ValueSource.DEFAULT\n )\n cls._value = sibling._value\n cls._value_source = ValueSource.GOT_FROM_CFG_SOURCE\n else:\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=cls._value_source != ValueSource.DEFAULT\n or sibling._value_source != ValueSource.DEFAULT\n )\n # propagating 'cls' default value to the sibling\n sibling._value = cls._value\n return super().get()\n\n @classmethod\n def put(cls, value: Any) -> None:\n \"\"\"\n Set a new value to this parameter as well as to its sibling.\n\n Parameters\n ----------\n value : Any\n \"\"\"\n super().put(value)\n # avoid getting into an infinite recursion\n if cls._update_sibling:\n cls._update_sibling = False\n try:\n with warnings.catch_warnings():\n # filter potential future warnings of the sibling\n warnings.filterwarnings(\"ignore\", category=FutureWarning)\n cls._sibling().put(value)\n finally:\n cls._update_sibling = True\n\n\nclass EnvironmentVariableDisallowingExecutionAndBackendBothSet(\n EnvironmentVariable,\n type=EnvironmentVariable.type,\n abstract=True,\n):\n \"\"\"Subclass to disallow getting this variable from the environment when both execution and backend are set in the environment.\"\"\"\n\n @classmethod\n @doc(EnvironmentVariable._get_value_from_config.__doc__)\n def _get_value_from_config(cls) -> str:\n if Backend.varname in os.environ and (\n Engine.varname in os.environ or StorageFormat.varname in os.environ\n ):\n # Handling this case is tricky, in part because the combination of\n # Backend and Engine/StorageFormat may be invalid. For now just\n # disallow it.\n raise ValueError(\"Can't specify both execution and backend in environment\")\n return super()._get_value_from_config()\n\n\nclass IsDebug(EnvironmentVariable, type=bool):\n \"\"\"Force Modin engine to be \"Python\" unless specified by $MODIN_ENGINE.\"\"\"\n\n varname = \"MODIN_DEBUG\"\n\n\nclass Engine(\n EnvironmentVariableDisallowingExecutionAndBackendBothSet,\n type=str,\n):\n \"\"\"Distribution engine to run queries by.\"\"\"\n\n varname = \"MODIN_ENGINE\"\n choices = (\"Ray\", \"Dask\", \"Python\", \"Unidist\", \"Native\")\n\n NOINIT_ENGINES = {\n \"Python\",\n \"Native\",\n } # engines that don't require initialization, useful for unit tests\n\n has_custom_engine = False\n\n @classmethod\n def _get_default(cls) -> str:\n \"\"\"\n Get default value of the config.\n\n Returns\n -------\n str\n \"\"\"\n from modin.utils import MIN_DASK_VERSION, MIN_RAY_VERSION, MIN_UNIDIST_VERSION\n\n # If there's a custom engine, we don't need to check for any engine\n # dependencies. Return the default \"Python\" engine.\n if IsDebug.get() or cls.has_custom_engine:\n return \"Python\"\n try:\n import ray\n\n except ImportError:\n pass\n else:\n if version.parse(ray.__version__) < MIN_RAY_VERSION:\n raise ImportError(\n 'Please `pip install \"modin[ray]\"` to install compatible Ray '\n + \"version \"\n + f\"(>={MIN_RAY_VERSION}).\"\n )\n return \"Ray\"\n try:\n import dask\n import distributed\n\n except ImportError:\n pass\n else:\n if (\n version.parse(dask.__version__) < MIN_DASK_VERSION\n or version.parse(distributed.__version__) < MIN_DASK_VERSION\n ):\n raise ImportError(\n f'Please `pip install \"modin[dask]\"` to install compatible Dask version (>={MIN_DASK_VERSION}).'\n )\n return \"Dask\"\n try:\n import unidist\n\n except ImportError:\n pass\n else:\n if version.parse(unidist.__version__) < MIN_UNIDIST_VERSION:\n raise ImportError(\n 'Please `pip install \"unidist[mpi]\"` to install compatible unidist on MPI '\n + \"version \"\n + f\"(>={MIN_UNIDIST_VERSION}).\"\n )\n return \"Unidist\"\n raise ImportError(\n \"Please refer to installation documentation page to install an engine\"\n )\n\n @classmethod\n @doc(Parameter.add_option.__doc__)\n def add_option(cls, choice: Any) -> Any:\n choice = super().add_option(choice)\n cls.NOINIT_ENGINES.add(choice)\n cls.has_custom_engine = True\n return choice\n\n @classmethod\n def put(cls, value: str) -> None:\n \"\"\"\n Set the engine value.\n\n Parameters\n ----------\n value : str\n Engine value to set.\n \"\"\"\n value = cls.normalize(value)\n # Backend.put() will set Engine.\n Backend.put(\n Backend.get_backend_for_execution(\n Execution(engine=value, storage_format=StorageFormat.get())\n )\n )\n\n @classmethod\n def get(cls) -> str:\n \"\"\"\n Get the engine value.\n\n Returns\n -------\n str\n Engine value.\n \"\"\"\n # We have to override get() because Engine may need to get its value\n # from the OS's environment variables for Backend or Engine.\n\n cls._warn_if_deprecated()\n\n # First, check if we've already set the engine value.\n if cls._value is not _UNSET:\n return cls._value\n\n engine_config_value = cls._get_value_from_config()\n backend_config_value = Backend._get_value_from_config()\n\n # If Engine is in the OS's configuration, use the configured Engine value.\n # Otherwise, use the Backend config value if that exists. If it doesn't,\n # fall back to the default Engine value.\n cls._value = (\n engine_config_value\n if engine_config_value is not _UNSET\n else (\n Backend.get_execution_for_backend(backend_config_value).engine\n if backend_config_value is not _UNSET\n else cls._get_default()\n )\n )\n\n return cls._value\n\n\nclass StorageFormat(EnvironmentVariableDisallowingExecutionAndBackendBothSet, type=str):\n \"\"\"Engine to run on a single node of distribution.\"\"\"\n\n @classmethod\n def put(cls, value: str) -> None:\n \"\"\"\n Set the storage format value.\n\n Parameters\n ----------\n value : str\n Storage format value to set.\n \"\"\"\n value = cls.normalize(value)\n # Backend.put() will set StorageFormat.\n Backend.put(\n Backend.get_backend_for_execution(\n Execution(engine=Engine.get(), storage_format=value)\n )\n )\n\n @classmethod\n def get(cls) -> str:\n \"\"\"\n Get the storage format value.\n\n Returns\n -------\n str\n Storage format value.\n \"\"\"\n # We have to override get() because StorageFormat may need to get its\n # value from the OS's environment variables for Backend or StorageFormat.\n\n cls._warn_if_deprecated()\n\n # First, check if we've already set the engine value.\n if cls._value is not _UNSET:\n return cls._value\n\n storage_format_config_value = cls._get_value_from_config()\n backend_config_value = Backend._get_value_from_config()\n\n # If StorageFormat is in the OS's configuration, use the configured\n # StorageFormat value. Otherwise, use the Backend config value if that\n # exists. If it doesn't, fall back to the default StorageFormat value.\n cls._value = (\n storage_format_config_value\n if storage_format_config_value is not _UNSET\n else (\n Backend.get_execution_for_backend(backend_config_value).storage_format\n if backend_config_value is not _UNSET\n else cls._get_default()\n )\n )\n\n return cls._value\n\n varname = \"MODIN_STORAGE_FORMAT\"\n default = \"Pandas\"\n choices = (\"Pandas\", \"Native\")\n\n\nExecution = namedtuple(\"Execution\", [\"storage_format\", \"engine\"])\n\n\nclass Backend(EnvironmentVariableDisallowingExecutionAndBackendBothSet, type=str):\n \"\"\"\n An alias for execution, i.e. the combination of StorageFormat and Engine.\n\n Setting backend may change StorageFormat and/or Engine to the corresponding\n respective values, and setting Engine or StorageFormat may change Backend.\n\n Modin's built-in backends include:\n - \"Ray\" <-> (StorageFormat=\"Pandas\", Engine=\"Ray\")\n - \"Dask\" <-> (StorageFormat=\"Pandas\", Engine=\"Dask\")\n - \"Python_Test\" <-> (StorageFormat=\"Pandas\", Engine=\"Python\")\n - This execution mode is meant for testing only.\n - \"Unidist\" <-> (StorageFormat=\"Pandas\", Engine=\"Unidist\")\n - \"Pandas\" <-> (StorageFormat=\"Native\", Engine=\"Native\")\n \"\"\"\n\n _BACKEND_TO_EXECUTION: dict[str, Execution] = {}\n _EXECUTION_TO_BACKEND: dict[Execution, str] = {}\n varname: str = \"MODIN_BACKEND\"\n choices: tuple[str, ...] = (\"Ray\", \"Dask\", \"Python_Test\", \"Unidist\", \"Pandas\")\n\n @classmethod\n def put(cls, value: str) -> None:\n \"\"\"\n Set the backend value.\n\n Parameters\n ----------\n value : str\n Backend value to set.\n \"\"\"\n execution = cls.get_execution_for_backend(value)\n set_execution(execution.engine, execution.storage_format)\n\n @classmethod\n def _get_default(cls) -> str:\n \"\"\"\n Get the default backend value.\n\n Returns\n -------\n str\n Default backend value.\n \"\"\"\n return cls._EXECUTION_TO_BACKEND[\n Execution(StorageFormat._get_default(), Engine._get_default())\n ]\n\n @classmethod\n def register_backend(cls: type[\"Backend\"], name: str, execution: Execution) -> None:\n \"\"\"\n Register a new backend.\n\n Parameters\n ----------\n name : str\n Backend name.\n execution : Execution\n Execution that corresponds to the backend.\n \"\"\"\n name = cls.normalize(name)\n super().add_option(name)\n if name in cls._BACKEND_TO_EXECUTION:\n raise ValueError(\n f\"Backend '{name}' is already registered with the execution {cls._BACKEND_TO_EXECUTION[name]}.\"\n )\n if execution in cls._EXECUTION_TO_BACKEND:\n raise ValueError(\n f\"{execution} is already registered with the backend {cls._EXECUTION_TO_BACKEND[execution]}.\"\n )\n cls._BACKEND_TO_EXECUTION[name] = execution\n cls._EXECUTION_TO_BACKEND[execution] = name\n\n @classmethod\n def add_option(cls, choice: str) -> NoReturn:\n \"\"\"\n Raise an exception for trying to add an option to Backend directly.\n\n Parameters\n ----------\n choice : str\n Choice to add. Unused.\n\n Raises\n ------\n ValueError\n Always.\n \"\"\"\n raise ValueError(\n \"Cannot add an option to Backend directly. Use Backend.register_backend instead.\"\n )\n\n @classmethod\n def set_active_backends(cls, new_choices: tuple) -> None:\n \"\"\"\n Set the active backends available for manual and automatic switching.\n\n Other backends may have been registered, and those backends remain registered, but the\n set of engines that can be used is dynamically modified.\n\n Parameters\n ----------\n new_choices : tuple\n Choices to add.\n\n Raises\n ------\n ValueError\n Raises a ValueError when the set of new_choices are not already registered\n \"\"\"\n registered_backends = cls._BACKEND_TO_EXECUTION\n for i in new_choices:\n if i not in registered_backends:\n raise ValueError(\n f\"Active backend choices {new_choices} are not all registered.\"\n )\n cls.choices = new_choices\n\n @classmethod\n def activate(cls, backend: str) -> None:\n \"\"\"\n Activate a backend that was previously registered.\n\n This is a no-op if the backend is already active.\n\n Parameters\n ----------\n backend : str\n Backend to activate.\n\n Raises\n ------\n ValueError\n Raises a ValueError if backend was not previously registered.\n \"\"\"\n if backend not in cls._BACKEND_TO_EXECUTION:\n raise ValueError(f\"Unknown backend '{backend}' is not registered.\")\n cls.choices = (*cls.choices, backend)\n\n @classmethod\n def get_active_backends(cls) -> tuple[str, ...]:\n \"\"\"\n Get the active backends available for manual and automatic switching.\n\n Returns\n -------\n tuple[str, ...]\n returns the active set of backends for switching\n \"\"\"\n return cls.choices\n\n @classmethod\n def get_backend_for_execution(cls, execution: Execution) -> str:\n \"\"\"\n Get the backend for the execution.\n\n Parameters\n ----------\n execution : Execution\n Execution to get the backend for.\n\n Returns\n -------\n str\n Backend for the execution.\n \"\"\"\n if execution not in cls._EXECUTION_TO_BACKEND:\n raise ValueError(\n f\"{execution} has no known backend. Please register a \"\n + \"backend for it with Backend.register_backend()\"\n )\n return cls._EXECUTION_TO_BACKEND[execution]\n\n @classmethod\n def get_execution_for_backend(cls, backend: str) -> Execution:\n \"\"\"\n Get the execution for the given backend.\n\n Parameters\n ----------\n backend : str\n Backend to get the execution for.\n\n Returns\n -------\n execution : Execution\n The execution for the given backend\n \"\"\"\n if not isinstance(backend, str):\n raise TypeError(\n \"Backend value should be a string, but instead it is \"\n + f\"{repr(backend)} of type {type(backend)}.\"\n )\n normalized_value = cls.normalize(backend)\n if normalized_value not in cls.choices:\n if normalized_value in cls._BACKEND_TO_EXECUTION:\n raise ValueError(\n f\"Backend '{backend}' is not currently active. Activate it first with Backend.activate('{backend})'.\"\n )\n backend_choice_string = \", \".join(f\"'{choice}'\" for choice in cls.choices)\n raise ValueError(\n f\"Unknown backend '{backend}'. Available backends are: \"\n + backend_choice_string\n )\n if normalized_value not in cls._BACKEND_TO_EXECUTION:\n raise ValueError(\n f\"Backend '{backend}' has no known execution. Please \"\n + \"register an execution for it with Backend.register_backend().\"\n )\n return cls._BACKEND_TO_EXECUTION[normalized_value]\n\n @classmethod\n def get(cls) -> str:\n \"\"\"\n Get the backend.\n\n Returns\n -------\n str\n Backend.\n \"\"\"\n # We have to override get() because Backend may need to get its value\n # from the OS's environment variables for Backend or Engine.\n\n cls._warn_if_deprecated()\n\n # First, check if we've already set the Backend value.\n if cls._value is not _UNSET:\n return cls._value\n\n backend_config_value = Backend._get_value_from_config()\n\n # If Backend is in the OS's configuration, use the configured Backend\n # value. Otherwise, we need to figure out the Backend value based on\n # the Engine and StorageFormat values.\n cls._value = (\n backend_config_value\n if backend_config_value is not _UNSET\n else cls.get_backend_for_execution(\n Execution(storage_format=StorageFormat.get(), engine=Engine.get())\n )\n )\n\n return cls._value\n\n\nBackend.register_backend(\"Ray\", Execution(\"Pandas\", \"Ray\"))\nBackend.register_backend(\"Dask\", Execution(\"Pandas\", \"Dask\"))\nBackend.register_backend(\"Python_Test\", Execution(\"Pandas\", \"Python\"))\nBackend.register_backend(\"Unidist\", Execution(\"Pandas\", \"Unidist\"))\nBackend.register_backend(\"Pandas\", Execution(\"Native\", \"Native\"))\n\n\nclass AutoSwitchBackend(EnvironmentVariable, type=bool):\n \"\"\"\n Whether automatic backend switching is allowed.\n\n When this flag is set, a Modin backend can attempt to automatically choose an appropriate backend\n for different operations based on features of the input data. When disabled, backends should\n avoid implicit backend switching outside of explicit operations like `to_pandas` and `to_ray`.\n \"\"\"\n\n varname = \"MODIN_AUTO_SWITCH_BACKENDS\"\n default = False\n\n @classmethod\n def enable(cls) -> None:\n \"\"\"Enable automatic backend switching.\"\"\"\n cls.put(True)\n\n @classmethod\n def disable(cls) -> None:\n \"\"\"Disable automatic backend switching.\"\"\"\n cls.put(False)\n\n\nclass ShowBackendSwitchProgress(EnvironmentVariable, type=bool):\n \"\"\"\n Whether to show progress when switching between backends.\n\n When enabled, progress messages are displayed during backend switches to inform users\n about data transfer operations. When disabled, backend switches occur silently.\n \"\"\"\n\n varname = \"MODIN_BACKEND_SWITCH_PROGRESS\"\n default = True\n\n @classmethod\n def enable(cls) -> None:\n \"\"\"Enable backend switch progress display.\"\"\"\n cls.put(True)\n\n @classmethod\n def disable(cls) -> None:\n \"\"\"Disable backend switch progress display.\"\"\"\n cls.put(False)\n\n\nclass IsExperimental(EnvironmentVariable, type=bool):\n \"\"\"Whether to Turn on experimental features.\"\"\"\n\n varname = \"MODIN_EXPERIMENTAL\"\n\n\nclass IsRayCluster(EnvironmentVariable, type=bool):\n \"\"\"Whether Modin is running on pre-initialized Ray cluster.\"\"\"\n\n varname = \"MODIN_RAY_CLUSTER\"\n\n\nclass RayRedisAddress(EnvironmentVariable, type=ExactStr):\n \"\"\"Redis address to connect to when running in Ray cluster.\"\"\"\n\n varname = \"MODIN_REDIS_ADDRESS\"\n\n\nclass RayRedisPassword(EnvironmentVariable, type=ExactStr):\n \"\"\"What password to use for connecting to Redis.\"\"\"\n\n varname = \"MODIN_REDIS_PASSWORD\"\n default = secrets.token_hex(32)\n\n\nclass RayInitCustomResources(EnvironmentVariable, type=dict):\n \"\"\"\n Ray node's custom resources to initialize with.\n\n Visit Ray documentation for more details:\n https://docs.ray.io/en/latest/ray-core/scheduling/resources.html#custom-resources\n\n Notes\n -----\n Relying on Modin to initialize Ray, you should set this config\n for the proper initialization with custom resources.\n \"\"\"\n\n varname = \"MODIN_RAY_INIT_CUSTOM_RESOURCES\"\n default = None\n\n\nclass RayTaskCustomResources(EnvironmentVariable, type=dict):\n \"\"\"\n Ray node's custom resources to request them in tasks or actors.\n\n Visit Ray documentation for more details:\n https://docs.ray.io/en/latest/ray-core/scheduling/resources.html#custom-resources\n\n Notes\n -----\n You can use this config to limit the parallelism for the entire workflow\n by setting the config at the very beginning.\n >>> import modin.config as cfg\n >>> cfg.RayTaskCustomResources.put({\"special_hardware\": 0.001})\n This way each single remote task or actor will require 0.001 of \"special_hardware\" to run.\n You can also use this config to limit the parallelism for a certain operation\n by setting the config with context.\n >>> with context(RayTaskCustomResources={\"special_hardware\": 0.001}):\n ... df.\n This way each single remote task or actor will require 0.001 of \"special_hardware\" to run\n within the context only.\n \"\"\"\n\n varname = \"MODIN_RAY_TASK_CUSTOM_RESOURCES\"\n default = None\n\n\nclass CpuCount(EnvironmentVariable, type=int):\n \"\"\"How many CPU cores to use during initialization of the Modin engine.\"\"\"\n\n varname = \"MODIN_CPUS\"\n\n @classmethod\n def _put(cls, value: int) -> None:\n \"\"\"\n Put specific value if CpuCount wasn't set by a user yet.\n\n Parameters\n ----------\n value : int\n Config value to set.\n\n Notes\n -----\n This method is used to set CpuCount from cluster resources internally\n and should not be called by a user.\n \"\"\"\n if cls.get_value_source() == ValueSource.DEFAULT:\n cls.put(value)\n\n @classmethod\n def _get_default(cls) -> int:\n \"\"\"\n Get default value of the config.\n\n Returns\n -------\n int\n \"\"\"\n import multiprocessing\n\n return multiprocessing.cpu_count()\n\n @classmethod\n def get(cls) -> int:\n \"\"\"\n Get ``CpuCount`` with extra checks.\n\n Returns\n -------\n int\n \"\"\"\n cpu_count = super().get()\n if cpu_count <= 0:\n raise ValueError(f\"`CpuCount` should be > 0; current value: {cpu_count}\")\n return cpu_count\n\n\nclass GpuCount(EnvironmentVariable, type=int):\n \"\"\"How may GPU devices to utilize across the whole distribution.\"\"\"\n\n varname = \"MODIN_GPUS\"\n\n\nclass Memory(EnvironmentVariable, type=int):\n \"\"\"\n How much memory (in bytes) give to an execution engine.\n\n Notes\n -----\n * In Ray case: the amount of memory to start the Plasma object store with.\n * In Dask case: the amount of memory that is given to each worker depending on CPUs used.\n \"\"\"\n\n varname = \"MODIN_MEMORY\"\n\n\nclass NPartitions(EnvironmentVariable, type=int):\n \"\"\"How many partitions to use for a Modin DataFrame (along each axis).\"\"\"\n\n varname = \"MODIN_NPARTITIONS\"\n\n @classmethod\n def _put(cls, value: int) -> None:\n \"\"\"\n Put specific value if NPartitions wasn't set by a user yet.\n\n Parameters\n ----------\n value : int\n Config value to set.\n\n Notes\n -----\n This method is used to set NPartitions from cluster resources internally\n and should not be called by a user.\n \"\"\"\n if cls.get_value_source() == ValueSource.DEFAULT:\n cls.put(value)\n\n @classmethod\n def _get_default(cls) -> int:\n \"\"\"\n Get default value of the config.\n\n Returns\n -------\n int\n \"\"\"\n return CpuCount.get()\n\n @classmethod\n def get(cls) -> int:\n \"\"\"\n Get ``NPartitions`` with extra checks.\n\n Returns\n -------\n int\n \"\"\"\n nparts = super().get()\n if nparts <= 0:\n raise ValueError(f\"`NPartitions` should be > 0; current value: {nparts}\")\n return nparts\n\n\nclass TestDatasetSize(EnvironmentVariable, type=str):\n \"\"\"Dataset size for running some tests.\"\"\"\n\n varname = \"MODIN_TEST_DATASET_SIZE\"\n choices = (\"Small\", \"Normal\", \"Big\")\n\n\nclass TrackFileLeaks(EnvironmentVariable, type=bool):\n \"\"\"Whether to track for open file handles leakage during testing.\"\"\"\n\n varname = \"MODIN_TEST_TRACK_FILE_LEAKS\"\n # Turn off tracking on Windows by default because\n # psutil's open_files() can be extremely slow on Windows (up to adding a few hours).\n # see https://github.com/giampaolo/psutil/pull/597\n default = sys.platform != \"win32\"\n\n\nclass AsvImplementation(EnvironmentVariable, type=ExactStr):\n \"\"\"Allows to select a library that we will use for testing performance.\"\"\"\n\n varname = \"MODIN_ASV_USE_IMPL\"\n choices = (\"modin\", \"pandas\")\n\n default = \"modin\"\n\n\nclass AsvDataSizeConfig(EnvironmentVariable, type=ExactStr):\n \"\"\"Allows to override default size of data (shapes).\"\"\"\n\n varname = \"MODIN_ASV_DATASIZE_CONFIG\"\n default = None\n\n\nclass ProgressBar(EnvironmentVariable, type=bool):\n \"\"\"Whether or not to show the progress bar.\"\"\"\n\n varname = \"MODIN_PROGRESS_BAR\"\n default = False\n\n @classmethod\n def enable(cls) -> None:\n \"\"\"Enable ``ProgressBar`` feature.\"\"\"\n cls.put(True)\n\n @classmethod\n def disable(cls) -> None:\n \"\"\"Disable ``ProgressBar`` feature.\"\"\"\n cls.put(False)\n\n @classmethod\n def put(cls, value: bool) -> None:\n \"\"\"\n Set ``ProgressBar`` value only if synchronous benchmarking is disabled.\n\n Parameters\n ----------\n value : bool\n Config value to set.\n \"\"\"\n if value and BenchmarkMode.get():\n raise ValueError(\"ProgressBar isn't compatible with BenchmarkMode\")\n super().put(value)\n\n\nclass BenchmarkMode(EnvironmentVariable, type=bool):\n \"\"\"Whether or not to perform computations synchronously.\"\"\"\n\n varname = \"MODIN_BENCHMARK_MODE\"\n default = False\n\n @classmethod\n def put(cls, value: bool) -> None:\n \"\"\"\n Set ``BenchmarkMode`` value only if progress bar feature is disabled.\n\n Parameters\n ----------\n value : bool\n Config value to set.\n \"\"\"\n if value and ProgressBar.get():\n raise ValueError(\"BenchmarkMode isn't compatible with ProgressBar\")\n super().put(value)\n\n\nclass LogMode(EnvironmentVariable, type=ExactStr):\n \"\"\"Set ``LogMode`` value if users want to opt-in.\"\"\"\n\n varname = \"MODIN_LOG_MODE\"\n choices = (\"enable\", \"disable\")\n default = \"disable\"\n\n @classmethod\n def enable(cls) -> None:\n \"\"\"Enable all logging levels.\"\"\"\n cls.put(\"enable\")\n\n @classmethod\n def disable(cls) -> None:\n \"\"\"Disable logging feature.\"\"\"\n cls.put(\"disable\")\n\n\nclass LogMemoryInterval(EnvironmentVariable, type=int):\n \"\"\"Interval (in seconds) to profile memory utilization for logging.\"\"\"\n\n varname = \"MODIN_LOG_MEMORY_INTERVAL\"\n default = 5\n\n @classmethod\n def put(cls, value: int) -> None:\n \"\"\"\n Set ``LogMemoryInterval`` with extra checks.\n\n Parameters\n ----------\n value : int\n Config value to set.\n \"\"\"\n if value <= 0:\n raise ValueError(f\"Log memory Interval should be > 0, passed value {value}\")\n super().put(value)\n\n @classmethod\n def get(cls) -> int:\n \"\"\"\n Get ``LogMemoryInterval`` with extra checks.\n\n Returns\n -------\n int\n \"\"\"\n log_memory_interval = super().get()\n if log_memory_interval <= 0:\n raise ValueError(\n f\"`LogMemoryInterval` should be > 0; current value: {log_memory_interval}\"\n )\n return log_memory_interval\n\n\nclass LogFileSize(EnvironmentVariable, type=int):\n \"\"\"Max size of logs (in MBs) to store per Modin job.\"\"\"\n\n varname = \"MODIN_LOG_FILE_SIZE\"\n default = 10\n\n @classmethod\n def put(cls, value: int) -> None:\n \"\"\"\n Set ``LogFileSize`` with extra checks.\n\n Parameters\n ----------\n value : int\n Config value to set.\n \"\"\"\n if value <= 0:\n raise ValueError(f\"Log file size should be > 0 MB, passed value {value}\")\n super().put(value)\n\n @classmethod\n def get(cls) -> int:\n \"\"\"\n Get ``LogFileSize`` with extra checks.\n\n Returns\n -------\n int\n \"\"\"\n log_file_size = super().get()\n if log_file_size <= 0:\n raise ValueError(\n f\"`LogFileSize` should be > 0; current value: {log_file_size}\"\n )\n return log_file_size\n\n\nclass MetricsMode(EnvironmentVariable, type=ExactStr):\n \"\"\"\n Set ``MetricsMode`` value to disable/enable metrics collection.\n\n Metric handlers are registered through `add_metric_handler` and can\n be used to record graphite-style timings or values. It is the\n responsibility of the handler to define how those emitted metrics\n are handled.\n \"\"\"\n\n varname = \"MODIN_METRICS_MODE\"\n choices = (\"enable\", \"disable\")\n default = \"enable\"\n\n @classmethod\n def enable(cls) -> None:\n \"\"\"Enable all metric collection.\"\"\"\n cls.put(\"enable\")\n\n @classmethod\n def disable(cls) -> None:\n \"\"\"Disable all metric collection.\"\"\"\n cls.put(\"disable\")\n\n\nclass PersistentPickle(EnvironmentVariable, type=bool):\n \"\"\"Whether serialization should be persistent.\"\"\"\n\n varname = \"MODIN_PERSISTENT_PICKLE\"\n # When set to off, it allows faster serialization which is only\n # valid in current run (i.e. useless for saving to disk).\n # When set to on, Modin objects could be saved to disk and loaded\n # but serialization/deserialization could take more time.\n default = False\n\n\nclass MinPartitionSize(EnvironmentVariable, type=int):\n \"\"\"\n Minimum number of rows/columns in a single pandas partition split.\n\n Once a partition for a pandas dataframe has more than this many elements,\n Modin adds another partition.\n \"\"\"\n\n varname = \"MODIN_MIN_PARTITION_SIZE\"\n default = 32\n\n @classmethod\n def put(cls, value: int) -> None:\n \"\"\"\n Set ``MinPartitionSize`` with extra checks.\n\n Parameters\n ----------\n value : int\n Config value to set.\n \"\"\"\n if value <= 0:\n raise ValueError(f\"Min partition size should be > 0, passed value {value}\")\n super().put(value)\n\n @classmethod\n def get(cls) -> int:\n \"\"\"\n Get ``MinPartitionSize`` with extra checks.\n\n Returns\n -------\n int\n \"\"\"\n from modin.error_message import ErrorMessage\n\n ErrorMessage.single_warning(\n \"`MinPartitionSize` is deprecated and will be removed in a future version. \"\n + \"This config has no longer effect, \"\n + \"use `MinRowPartitionSize` and `MinColumnPartitionSize` instead.\",\n FutureWarning,\n )\n min_partition_size = super().get()\n if min_partition_size <= 0:\n raise ValueError(\n f\"`MinPartitionSize` should be > 0; current value: {min_partition_size}\"\n )\n return min_partition_size\n\n\nclass MinRowPartitionSize(EnvironmentVariable, type=int):\n \"\"\"\n Minimum number of rows in a single pandas partition split.\n\n Once a partition for a pandas dataframe has more than this many elements,\n Modin adds another partition.\n \"\"\"\n\n varname = \"MODIN_MIN_ROW_PARTITION_SIZE\"\n default = 32\n\n @classmethod\n def put(cls, value: int) -> None:\n \"\"\"\n Set ``MinRowPartitionSize`` with extra checks.\n\n Parameters\n ----------\n value : int\n Config value to set.\n \"\"\"\n if value <= 0:\n raise ValueError(\n f\"Min row partition size should be > 0, passed value {value}\"\n )\n super().put(value)\n\n @classmethod\n def get(cls) -> int:\n \"\"\"\n Get ``MinRowPartitionSize`` with extra checks.\n\n Returns\n -------\n int\n \"\"\"\n min_row_partition_size = super().get()\n if min_row_partition_size <= 0:\n raise ValueError(\n f\"`MinRowPartitionSize` should be > 0; current value: {min_row_partition_size}\"\n )\n return min_row_partition_size\n\n\nclass MinColumnPartitionSize(EnvironmentVariable, type=int):\n \"\"\"\n Minimum number of columns in a single pandas partition split.\n\n Once a partition for a pandas dataframe has more than this many elements,\n Modin adds another partition.\n \"\"\"\n\n varname = \"MODIN_MIN_COLUMN_PARTITION_SIZE\"\n default = 32\n\n @classmethod\n def put(cls, value: int) -> None:\n \"\"\"\n Set ``MinColumnPartitionSize`` with extra checks.\n\n Parameters\n ----------\n value : int\n Config value to set.\n \"\"\"\n if value <= 0:\n raise ValueError(\n f\"Min column partition size should be > 0, passed value {value}\"\n )\n super().put(value)\n\n @classmethod\n def get(cls) -> int:\n \"\"\"\n Get ``MinColumnPartitionSize`` with extra checks.\n\n Returns\n -------\n int\n \"\"\"\n min_column_partition_size = super().get()\n if min_column_partition_size <= 0:\n raise ValueError(\n f\"`MinColumnPartitionSize` should be > 0; current value: {min_column_partition_size}\"\n )\n return min_column_partition_size\n\n\nclass TestReadFromSqlServer(EnvironmentVariable, type=bool):\n \"\"\"Set to true to test reading from SQL server.\"\"\"\n\n varname = \"MODIN_TEST_READ_FROM_SQL_SERVER\"\n default = False\n\n\nclass TestReadFromPostgres(EnvironmentVariable, type=bool):\n \"\"\"Set to true to test reading from Postgres.\"\"\"\n\n varname = \"MODIN_TEST_READ_FROM_POSTGRES\"\n default = False\n\n\nclass GithubCI(EnvironmentVariable, type=bool):\n \"\"\"Set to true when running Modin in GitHub CI.\"\"\"\n\n varname = \"MODIN_GITHUB_CI\"\n default = False\n\n\nclass ModinNumpy(EnvironmentVariable, type=bool):\n \"\"\"Set to true to use Modin's implementation of NumPy API.\"\"\"\n\n varname = \"MODIN_NUMPY\"\n default = False\n\n\nclass RangePartitioning(EnvironmentVariable, type=bool):\n \"\"\"\n Set to true to use Modin's range-partitioning implementation where possible.\n\n Please refer to documentation for cases where enabling this options would be beneficial:\n https://modin.readthedocs.io/en/stable/flow/modin/experimental/range_partitioning_groupby.html\n \"\"\"\n\n varname = \"MODIN_RANGE_PARTITIONING\"\n default = False\n\n\nclass CIAWSSecretAccessKey(EnvironmentVariable, type=str):\n \"\"\"Set to AWS_SECRET_ACCESS_KEY when running mock S3 tests for Modin in GitHub CI.\"\"\"\n\n varname = \"AWS_SECRET_ACCESS_KEY\"\n default = \"foobar_secret\"\n\n\nclass CIAWSAccessKeyID(EnvironmentVariable, type=str):\n \"\"\"Set to AWS_ACCESS_KEY_ID when running mock S3 tests for Modin in GitHub CI.\"\"\"\n\n varname = \"AWS_ACCESS_KEY_ID\"\n default = \"foobar_key\"\n\n\nclass AsyncReadMode(EnvironmentVariable, type=bool):\n \"\"\"\n It does not wait for the end of reading information from the source.\n\n It basically means, that the reading function only launches tasks for the dataframe\n to be read/created, but not ensures that the construction is finalized by the time\n the reading function returns a dataframe.\n\n This option was brought to improve performance of reading/construction\n of Modin DataFrames, however it may also:\n\n 1. Increase the peak memory consumption. Since the garbage collection of the\n temporary objects created during the reading is now also lazy and will only\n be performed when the reading/construction is actually finished.\n\n 2. Can break situations when the source is manually deleted after the reading\n function returns a result, for example, when reading inside of a context-block\n that deletes the file on ``__exit__()``.\n \"\"\"\n\n varname = \"MODIN_ASYNC_READ_MODE\"\n default = False\n\n\nclass ReadSqlEngine(EnvironmentVariable, type=str):\n \"\"\"Engine to run `read_sql`.\"\"\"\n\n varname = \"MODIN_READ_SQL_ENGINE\"\n default = \"Pandas\"\n choices = (\"Pandas\", \"Connectorx\")\n\n\nclass LazyExecution(EnvironmentVariable, type=str):\n \"\"\"\n Lazy execution mode.\n\n Supported values:\n `Auto` - the execution mode is chosen by the engine for each operation (default value).\n `On` - the lazy execution is performed wherever it's possible.\n `Off` - the lazy execution is disabled.\n \"\"\"\n\n varname = \"MODIN_LAZY_EXECUTION\"\n choices = (\"Auto\", \"On\", \"Off\")\n default = \"Auto\"\n\n\nclass DocModule(EnvironmentVariable, type=ExactStr):\n \"\"\"\n The module to use that will be used for docstrings.\n\n The value set here must be a valid, importable module. It should have\n a `DataFrame`, `Series`, and/or several APIs directly (e.g. `read_csv`).\n \"\"\"\n\n varname = \"MODIN_DOC_MODULE\"\n default = \"pandas\"\n\n\nclass DaskThreadsPerWorker(EnvironmentVariable, type=int):\n \"\"\"Number of threads per Dask worker.\"\"\"\n\n varname = \"MODIN_DASK_THREADS_PER_WORKER\"\n default = 1\n\n\nclass NativePandasMaxRows(EnvironmentVariable, type=int):\n \"\"\"Maximum number of rows which can be processed using local, native, pandas.\"\"\"\n\n varname = \"MODIN_NATIVE_MAX_ROWS\"\n default = 10_000_000\n\n\nclass NativePandasTransferThreshold(EnvironmentVariable, type=int):\n \"\"\"\n Targeted max number of dataframe rows which should be transferred between engines.\n\n This is often the same value as MODIN_NATIVE_MAX_ROWS but it can be independently\n set to change how transfer costs are considered.\n \"\"\"\n\n varname = \"MODIN_NATIVE_MAX_XFER_ROWS\"\n default = 10_000_000\n\n\nclass NativePandasDeepCopy(EnvironmentVariable, type=bool):\n \"\"\"\n Whether to perform deep copies when transferring data with the native pandas backend.\n\n Copies occur when constructing a Modin frame from a native pandas object with\n `pd.DataFrame(pandas.DataFrame([]))`, or when creating a native pandas frame from a Modin one\n via `df.modin.to_pandas()`.\n\n Leaving this flag disabled produces significant performance improvements by reducing the number\n of copy operations performed. However, it may create unexpected results if the user mutates\n the Modin frame or native pandas frame in-place.\n\n >>> import pandas # doctest: +SKIP\n >>> import modin.pandas as pd # doctest: +SKIP\n >>> from modin.config import Backend # doctest: + SKIP\n >>> Backend.put(\"Pandas\") # doctest: +SKIP\n >>> pandas.set_option(\"mode.copy_on_write\", False) # doctest: +SKIP\n >>> native_df = pandas.DataFrame([0]) # doctest: +SKIP\n >>> modin_df = pd.DataFrame(native_df) # doctest: +SKIP\n >>> native_df.loc[0, 0] = -1 # doctest: +SKIP\n >>> modin_df # doctest: +SKIP\n 0\n 0 -1\n \"\"\"\n\n varname = \"MODIN_NATIVE_DEEP_COPY\"\n default = False\n\n @classmethod\n def enable(cls) -> None:\n \"\"\"Enable deep copy on frames with the native pandas backend.\"\"\"\n cls.put(True)\n\n @classmethod\n def disable(cls) -> None:\n \"\"\"Disable deep copy on frames with the native pandas backend.\"\"\"\n cls.put(False)\n\n\nclass BackendMergeCastInPlace(EnvironmentVariable, type=bool):\n \"\"\"\n Whether to cast a DataFrame in-place when performing a merge when using hybrid mode.\n\n This flag modifies the behavior of a cast performed on operations involving more\n than one type of query compiler. If enabled the actual cast will be performed in-place\n and the input DataFrame will have a new backend. If disabled the original DataFrame\n will remain on the same underlying engine.\n \"\"\"\n\n varname = \"MODIN_BACKEND_MERGE_CAST_IN_PLACE\"\n default = True\n\n @classmethod\n def enable(cls) -> None:\n \"\"\"Enable casting in place when performing a merge operation betwen two different compilers.\"\"\"\n cls.put(True)\n\n @classmethod\n def disable(cls) -> None:\n \"\"\"Disable casting in place when performing a merge operation betwen two different compilers.\"\"\"\n cls.put(False)\n\n\nclass BackendJoinConsiderAllBackends(EnvironmentVariable, type=bool):\n \"\"\"\n Whether to consider all active backends when performing a pre-operation switch for join operations.\n\n Only used when AutoSwitchBackend is active.\n By default, only backends already present in the arguments of a join operation are considered when\n switching backends. Enabling this flag will allow join operations that are registered\n as pre-op switches to consider backends other than those directly present in the arguments.\n \"\"\"\n\n varname = \"MODIN_BACKEND_JOIN_CONSIDER_ALL_BACKENDS\"\n default = True\n\n @classmethod\n def enable(cls) -> None:\n \"\"\"Enable casting in place when performing a merge operation betwen two different compilers.\"\"\"\n cls.put(True)\n\n @classmethod\n def disable(cls) -> None:\n \"\"\"Disable casting in place when performing a merge operation betwen two different compilers.\"\"\"\n cls.put(False)\n\n\nclass DynamicPartitioning(EnvironmentVariable, type=bool):\n \"\"\"\n Set to true to use Modin's dynamic-partitioning implementation where possible.\n\n Please refer to documentation for cases where enabling this options would be beneficial:\n https://modin.readthedocs.io/en/stable/usage_guide/optimization_notes/index.html#dynamic-partitioning-in-modin\n \"\"\"\n\n varname = \"MODIN_DYNAMIC_PARTITIONING\"\n default = False\n\n\ndef _check_vars() -> None:\n \"\"\"\n Check validity of environment variables.\n\n Look out for any environment variables that start with \"MODIN_\" prefix\n that are unknown - they might be a typo, so warn a user.\n \"\"\"\n valid_names = {\n obj.varname\n for obj in globals().values()\n if isinstance(obj, type)\n and issubclass(obj, EnvironmentVariable)\n and not obj.is_abstract\n }\n found_names = {name for name in os.environ if name.startswith(\"MODIN_\")}\n unknown = found_names - valid_names\n deprecated: dict[str, DeprecationDescriptor] = {\n obj.varname: obj._deprecation_descriptor\n for obj in globals().values()\n if isinstance(obj, type)\n and issubclass(obj, EnvironmentVariable)\n and not obj.is_abstract\n and obj.varname is not None\n and obj._deprecation_descriptor is not None\n }\n found_deprecated = found_names & deprecated.keys()\n if unknown:\n warnings.warn(\n f\"Found unknown environment variable{'s' if len(unknown) > 1 else ''},\"\n + f\" please check {'their' if len(unknown) > 1 else 'its'} spelling: \"\n + \", \".join(sorted(unknown))\n )\n for depr_var in found_deprecated:\n warnings.warn(\n deprecated[depr_var].deprecation_message(use_envvar_names=True),\n FutureWarning,\n )\n\n\n_check_vars()\n"
},
{
"path": "modin/config/pubsub.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses ``Parameter`` class - base class for all configs.\"\"\"\n\nimport contextlib\nimport warnings\nfrom collections import defaultdict\nfrom enum import IntEnum\nfrom typing import (\n TYPE_CHECKING,\n Any,\n Callable,\n DefaultDict,\n Iterator,\n NamedTuple,\n Optional,\n Tuple,\n cast,\n)\n\nif TYPE_CHECKING:\n from modin.config.envvars import EnvironmentVariable\n\n\nclass DeprecationDescriptor:\n \"\"\"\n Describe deprecated parameter.\n\n Parameters\n ----------\n parameter : type[Parameter]\n Deprecated parameter.\n new_parameter : type[Parameter], optional\n If there's a replacement parameter for the deprecated one, specify it here.\n when_removed : str, optional\n If known, the exact release when the deprecated parameter is planned to be removed.\n \"\"\"\n\n _parameter: type[\"Parameter\"]\n _new_parameter: Optional[type[\"Parameter\"]]\n _when_removed: str\n\n def __init__(\n self,\n parameter: type[\"Parameter\"],\n new_parameter: Optional[type[\"Parameter\"]] = None,\n when_removed: Optional[str] = None,\n ):\n self._parameter = parameter\n self._new_parameter = new_parameter\n self._when_removed = \"a future\" if when_removed is None else when_removed\n\n def deprecation_message(self, use_envvar_names: bool = False) -> str:\n \"\"\"\n Generate a message to be used in a warning raised when using the deprecated parameter.\n\n Parameters\n ----------\n use_envvar_names : bool, default: False\n Whether to use environment variable names in the warning. If ``True``, both\n ``self._parameter`` and ``self._new_parameter`` have to be a type of ``EnvironmentVariable``.\n\n Returns\n -------\n str\n \"\"\"\n name = (\n cast(\"EnvironmentVariable\", self._parameter).varname\n if use_envvar_names\n else self._parameter.__name__\n )\n msg = f\"'{name}' is deprecated and will be removed in {self._when_removed} version.\"\n if self._new_parameter is not None:\n new_name = (\n cast(\"EnvironmentVariable\", self._new_parameter).varname\n if use_envvar_names\n else self._new_parameter.__name__\n )\n msg += f\" Use '{new_name}' instead.\"\n return msg\n\n\nclass TypeDescriptor(NamedTuple):\n \"\"\"\n Class for config data manipulating of exact type.\n\n Parameters\n ----------\n decode : callable\n Callable to decode config value from the raw data.\n normalize : callable\n Callable to bring different config value variations to\n the single form.\n verify : callable\n Callable to check that config value satisfies given config\n type requirements.\n help : str\n Class description string.\n \"\"\"\n\n decode: Callable[[str], object]\n normalize: Callable[[object], object]\n verify: Callable[[object], bool]\n help: str\n\n\nclass ExactStr(str):\n \"\"\"Class to be used in type params where no transformations are needed.\"\"\"\n\n\n_TYPE_PARAMS = {\n str: TypeDescriptor(\n decode=lambda value: value.strip().title(),\n normalize=lambda value: str(value).strip().title(),\n verify=lambda value: True,\n help=\"a case-insensitive string\",\n ),\n ExactStr: TypeDescriptor(\n decode=lambda value: value,\n normalize=lambda value: value,\n verify=lambda value: True,\n help=\"a string\",\n ),\n bool: TypeDescriptor(\n decode=lambda value: value.strip().lower() in {\"true\", \"yes\", \"1\"},\n normalize=bool,\n verify=lambda value: isinstance(value, bool)\n or (\n isinstance(value, str)\n and value.strip().lower() in {\"true\", \"yes\", \"1\", \"false\", \"no\", \"0\"}\n ),\n help=\"a boolean flag (any of 'true', 'yes' or '1' in case insensitive manner is considered positive)\",\n ),\n int: TypeDescriptor(\n decode=lambda value: int(value.strip()),\n normalize=int, # type: ignore\n verify=lambda value: isinstance(value, int)\n or (isinstance(value, str) and value.strip().isdigit()),\n help=\"an integer value\",\n ),\n dict: TypeDescriptor(\n decode=lambda value: {\n key: int(val) if val.isdigit() else val\n for key_value in value.split(\",\")\n for key, val in [[v.strip() for v in key_value.split(\"=\", maxsplit=1)]]\n },\n normalize=lambda value: (\n value\n if isinstance(value, dict)\n else {\n key: int(val) if val.isdigit() else val\n for key_value in str(value).split(\",\")\n for key, val in [[v.strip() for v in key_value.split(\"=\", maxsplit=1)]]\n }\n ),\n verify=lambda value: isinstance(value, dict)\n or (\n isinstance(value, str)\n and all(\n key_value.find(\"=\") not in (-1, len(key_value) - 1)\n for key_value in value.split(\",\")\n )\n ),\n help=\"a sequence of KEY=VALUE values separated by comma (Example: 'KEY1=VALUE1,KEY2=VALUE2,KEY3=VALUE3')\",\n ),\n}\n\n# special marker to distinguish unset value from None value\n# as someone may want to use None as a real value for a parameter\n_UNSET = object()\n\n\nclass ValueSource(IntEnum): # noqa: PR01\n \"\"\"Class that describes the method of getting the value for a parameter.\"\"\"\n\n # got from default, i.e. neither user nor configuration source had the value\n DEFAULT = 0\n # set by user\n SET_BY_USER = 1\n # got from parameter configuration source, like environment variable\n GOT_FROM_CFG_SOURCE = 2\n\n\nclass Parameter(object):\n \"\"\"\n Base class describing interface for configuration entities.\n\n Attributes\n ----------\n choices : Optional[Sequence[str]]\n Array with possible options of ``Parameter`` values.\n type : str\n String that denotes ``Parameter`` type.\n default : Optional[Any]\n ``Parameter`` default value.\n is_abstract : bool, default: True\n Whether or not ``Parameter`` is abstract.\n _value_source : Optional[ValueSource]\n Source of the ``Parameter`` value, should be set by\n ``ValueSource``.\n _deprecation_descriptor : Optional[DeprecationDescriptor]\n Indicate whether this parameter is deprecated.\n \"\"\"\n\n choices: Optional[Tuple[str, ...]] = None\n type = str\n default: Optional[Any] = None\n is_abstract = True\n _value_source: Optional[ValueSource] = None\n _value: Any = _UNSET\n _subs: list = []\n _once: DefaultDict[Any, list] = defaultdict(list)\n _deprecation_descriptor: Optional[DeprecationDescriptor] = None\n\n @classmethod\n def _warn_if_deprecated(cls) -> None:\n \"\"\"Warn that the variable is deprecated if it has a deprecation descriptor.\"\"\"\n if cls._deprecation_descriptor is not None:\n warnings.warn(\n cls._deprecation_descriptor.deprecation_message(), FutureWarning\n )\n\n @classmethod\n def _get_value_from_config(cls) -> Any:\n \"\"\"\n Read the value from config storage.\n\n Returns\n -------\n Any\n Config raw value if it's set, otherwise `_UNSET`.\n\n Notes\n -----\n Config storage can be config file or environment variable or whatever.\n Method should be implemented in the child class.\n \"\"\"\n raise NotImplementedError()\n\n @classmethod\n def get_help(cls) -> str:\n \"\"\"\n Generate user-presentable help for the option.\n\n Returns\n -------\n str\n\n Notes\n -----\n Method should be implemented in the child class.\n \"\"\"\n raise NotImplementedError()\n\n def __init_subclass__(cls, type: Any, abstract: bool = False, **kw: dict):\n \"\"\"\n Initialize subclass.\n\n Parameters\n ----------\n type : Any\n Type of the config.\n abstract : bool, default: False\n Whether config is abstract.\n **kw : dict\n Optional arguments for config initialization.\n \"\"\"\n assert type in _TYPE_PARAMS, f\"Unsupported variable type: {type}\"\n cls.type = type\n cls.is_abstract = abstract\n cls._value = _UNSET\n cls._subs = []\n cls._once = defaultdict(list)\n super().__init_subclass__(**kw)\n\n @classmethod\n def subscribe(cls, callback: Callable) -> None:\n \"\"\"\n Add `callback` to the `_subs` list and then execute it.\n\n Parameters\n ----------\n callback : callable\n Callable to execute.\n \"\"\"\n cls._subs.append(callback)\n callback(cls)\n\n @classmethod\n def _get_default(cls) -> Any:\n \"\"\"\n Get default value of the config.\n\n Returns\n -------\n Any\n \"\"\"\n return cls.default\n\n @classmethod\n def get_value_source(cls) -> ValueSource:\n \"\"\"\n Get value source of the config.\n\n Returns\n -------\n ValueSource\n \"\"\"\n if cls._value_source is None:\n # dummy call to .get() to initialize the value\n cls.get()\n assert (\n cls._value_source is not None\n ), \"_value_source must be initialized by now in get()\"\n return cls._value_source\n\n @classmethod\n def get(cls) -> Any:\n \"\"\"\n Get config value.\n\n Returns\n -------\n Any\n Decoded and verified config value.\n \"\"\"\n cls._warn_if_deprecated()\n if cls._value is _UNSET:\n # get the value from env\n config_value = cls._get_value_from_config()\n if config_value is _UNSET:\n cls._value = cls._get_default()\n cls._value_source = ValueSource.DEFAULT\n else:\n cls._value = config_value\n cls._value_source = ValueSource.GOT_FROM_CFG_SOURCE\n return cls._value\n\n @classmethod\n def put(cls, value: Any) -> None:\n \"\"\"\n Set config value.\n\n Parameters\n ----------\n value : Any\n Config value to set.\n \"\"\"\n cls._warn_if_deprecated()\n cls._check_callbacks(cls._put_nocallback(value))\n cls._value_source = ValueSource.SET_BY_USER\n\n @classmethod\n def normalize(cls, value: Any) -> Any:\n \"\"\"\n Normalize config value.\n\n Parameters\n ----------\n value : Any\n Config value to normalize.\n\n Returns\n -------\n Any\n Normalized config value.\n \"\"\"\n return _TYPE_PARAMS[cls.type].normalize(value)\n\n @classmethod\n def once(cls, onvalue: Any, callback: Callable) -> None:\n \"\"\"\n Execute `callback` if config value matches `onvalue` value.\n\n Otherwise accumulate callbacks associated with the given `onvalue`\n in the `_once` container.\n\n Parameters\n ----------\n onvalue : Any\n Config value to set.\n callback : callable\n Callable that should be executed if config value matches `onvalue`.\n \"\"\"\n onvalue = cls.normalize(onvalue)\n if onvalue == cls.get():\n callback(cls)\n else:\n cls._once[onvalue].append(callback)\n\n @classmethod\n def _put_nocallback(cls, value: Any) -> Any:\n \"\"\"\n Set config value without executing callbacks.\n\n Parameters\n ----------\n value : Any\n Config value to set.\n\n Returns\n -------\n Any\n Replaced (old) config value.\n \"\"\"\n if not _TYPE_PARAMS[cls.type].verify(value):\n raise ValueError(f\"Unsupported value: {value}\")\n value = cls.normalize(value)\n oldvalue, cls._value = cls.get(), value\n return oldvalue\n\n @classmethod\n def _check_callbacks(cls, oldvalue: Any) -> None:\n \"\"\"\n Execute all needed callbacks if config value was changed.\n\n Parameters\n ----------\n oldvalue : Any\n Previous (old) config value.\n \"\"\"\n if oldvalue == cls.get():\n return\n for callback in cls._subs:\n callback(cls)\n for callback in cls._once.pop(cls.get(), ()):\n callback(cls)\n\n @classmethod\n def add_option(cls, choice: Any) -> Any:\n \"\"\"\n Add a new choice for the parameter.\n\n Parameters\n ----------\n choice : Any\n New choice to add to the available choices.\n\n Returns\n -------\n Any\n Added choice normalized according to the parameter type.\n \"\"\"\n if cls.choices is not None:\n if not _TYPE_PARAMS[cls.type].verify(choice):\n raise ValueError(f\"Unsupported choice value: {choice}\")\n choice = cls.normalize(choice)\n if choice not in cls.choices:\n cls.choices += (choice,)\n return choice\n raise TypeError(\"Cannot add a choice to a parameter where choices is None\")\n\n\n@contextlib.contextmanager\ndef context(**config: dict[str, Any]) -> Iterator[None]:\n \"\"\"\n Set a value(s) for the specified config(s) from ``modin.config`` in the scope of the context.\n\n Parameters\n ----------\n **config : dict[str, Any]\n Keyword describing a name of a config variable from ``modin.config`` as a key\n and a new value as a value.\n\n Examples\n --------\n >>> RangePartitioning.get()\n False\n >>> with context(RangePartitioning=True):\n ... print(RangePartitioning.get()) # True\n True\n False\n >>> RangePartitioning.get()\n False\n >>> with context(RangePartitioning=True, AsyncReadMode=True):\n ... print(RangePartitioning.get()) # True\n ... print(AsyncReadMode.get()) # True\n True\n True\n >>> RangePartitioning.get()\n False\n >>> AsyncReadMode.get()\n False\n \"\"\"\n import modin.config as cfg\n\n old_values = {}\n for name, val in config.items():\n var = getattr(cfg, name)\n old_values[var] = var.get()\n var.put(val)\n try:\n yield\n finally:\n for var, val in old_values.items():\n var.put(val)\n\n\n__all__ = [\"Parameter\", \"context\"]\n"
},
{
"path": "modin/conftest.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n# We turn off mypy type checks in this file because it's not imported anywhere\n# type: ignore\n\nimport copy\nimport logging\nimport os\nimport platform\nimport shutil\nimport subprocess\nimport sys\nimport time\nfrom collections import defaultdict\nfrom contextlib import contextmanager\nfrom typing import Iterable, Optional\n\nimport boto3\nimport numpy as np\nimport pandas\nimport pytest\nimport requests\nimport s3fs\nfrom pandas.util._decorators import doc\n\nfrom modin.config import Backend, Execution\n\nassert (\n \"modin.utils\" not in sys.modules\n), \"Do not import modin.utils before patching, or tests could fail\"\n# every import under this assert has to be postfixed with 'noqa: E402'\n# as flake8 complains about that... but we _have_ to make sure we\n# monkey-patch at the right spot, otherwise testing doc URLs might\n# not catch all of them\nimport modin.utils # noqa: E402\n\n_generated_doc_urls = set()\n\n\ndef _saving_make_api_url(token, _make_api_url=modin.utils._make_api_url):\n url = _make_api_url(token)\n _generated_doc_urls.add(url)\n return url\n\n\nmodin.utils._make_api_url = _saving_make_api_url\n\nimport uuid # noqa: E402\n\nimport modin # noqa: E402\nimport modin.config # noqa: E402\nimport modin.pandas as pd # noqa: E402\nimport modin.tests.config # noqa: E402\nfrom modin.config import ( # noqa: E402\n AsyncReadMode,\n BenchmarkMode,\n GithubCI,\n IsExperimental,\n MinRowPartitionSize,\n NPartitions,\n)\nfrom modin.core.execution.dispatching.factories import factories # noqa: E402\nfrom modin.core.execution.python.implementations.pandas_on_python.io import ( # noqa: E402\n PandasOnPythonIO,\n)\nfrom modin.core.storage_formats import ( # noqa: E402\n BaseQueryCompiler,\n PandasQueryCompiler,\n)\nfrom modin.core.storage_formats.pandas.query_compiler_caster import ( # noqa: E402\n _CLASS_AND_BACKEND_TO_POST_OP_SWITCH_METHODS,\n _CLASS_AND_BACKEND_TO_PRE_OP_SWITCH_METHODS,\n _GENERAL_EXTENSIONS,\n)\nfrom modin.tests.pandas.utils import ( # noqa: E402\n NROWS,\n _make_csv_file,\n get_unique_filename,\n make_default_file,\n)\n\n\ndef pytest_addoption(parser):\n parser.addoption(\n \"--execution\",\n action=\"store\",\n default=None,\n help=\"specifies execution to run tests on\",\n )\n\n\ndef set_experimental_env(mode):\n IsExperimental.put(mode == \"experimental\")\n\n\n@pytest.fixture(scope=\"session\", autouse=True)\ndef enforce_config():\n \"\"\"\n A fixture that ensures that all checks for MODIN_* variables\n are done using modin.config to prevent leakage\n \"\"\"\n orig_env = os.environ\n modin_start = os.path.dirname(modin.__file__)\n modin_exclude = [\n os.path.dirname(modin.config.__file__),\n os.path.dirname(modin.tests.config.__file__),\n ]\n\n class PatchedEnv:\n @staticmethod\n def __check_var(name):\n if name.upper().startswith(\"MODIN_\"):\n frame = sys._getframe()\n try:\n # get the path to module where caller of caller is defined;\n # caller of this function is inside PatchedEnv, and we're\n # interested in whomever called a method on PatchedEnv\n caller_file = frame.f_back.f_back.f_code.co_filename\n finally:\n del frame\n pkg_name = os.path.dirname(caller_file)\n if pkg_name.startswith(modin_start):\n assert any(\n pkg_name.startswith(excl) for excl in modin_exclude\n ), \"Do not access MODIN_ environment variable bypassing modin.config\"\n\n def __getitem__(self, name):\n self.__check_var(name)\n return orig_env[name]\n\n def __setitem__(self, name, value):\n self.__check_var(name)\n orig_env[name] = value\n\n def __delitem__(self, name):\n self.__check_var(name)\n del orig_env[name]\n\n def pop(self, name, default=object()):\n self.__check_var(name)\n return orig_env.pop(name, default)\n\n def get(self, name, default=None):\n self.__check_var(name)\n return orig_env.get(name, default)\n\n def __contains__(self, name):\n self.__check_var(name)\n return name in orig_env\n\n def __getattr__(self, name):\n return getattr(orig_env, name)\n\n def __iter__(self):\n return iter(orig_env)\n\n os.environ = PatchedEnv()\n yield\n os.environ = orig_env\n\n\nBASE_EXECUTION_NAME = \"BaseOnPython\"\n\n\nclass TestQC(BaseQueryCompiler):\n def __init__(self, modin_frame):\n self._modin_frame = modin_frame\n\n storage_format = property(\n lambda self: \"Base\", doc=BaseQueryCompiler.storage_format.__doc__\n )\n engine = property(lambda self: \"Python\", doc=BaseQueryCompiler.engine.__doc__)\n\n def finalize(self):\n self._modin_frame.finalize()\n\n def execute(self):\n self.finalize()\n self._modin_frame.wait_computations()\n\n @classmethod\n def from_pandas(cls, df, data_cls):\n return cls(data_cls.from_pandas(df))\n\n @classmethod\n def from_arrow(cls, at, data_cls):\n return cls(data_cls.from_arrow(at))\n\n def free(self):\n pass\n\n def to_interchange_dataframe(\n self, nan_as_null: bool = False, allow_copy: bool = True\n ):\n raise NotImplementedError(\n \"The selected execution does not implement the DataFrame exchange protocol.\"\n )\n\n @classmethod\n def from_interchange_dataframe(cls, df, data_cls):\n raise NotImplementedError(\n \"The selected execution does not implement the DataFrame exchange protocol.\"\n )\n\n to_pandas = PandasQueryCompiler.to_pandas\n default_to_pandas = PandasQueryCompiler.default_to_pandas\n\n\nclass BaseOnPythonIO(PandasOnPythonIO):\n query_compiler_cls = TestQC\n\n\nclass BaseOnPythonFactory(factories.BaseFactory):\n @classmethod\n def prepare(cls):\n cls.io_cls = BaseOnPythonIO\n\n\ndef set_base_execution(name=BASE_EXECUTION_NAME):\n setattr(factories, f\"{name}Factory\", BaseOnPythonFactory)\n Backend.register_backend(\n \"BaseOnPython\",\n Execution(\n engine=\"Python\",\n storage_format=\"Base\",\n ),\n )\n modin.set_execution(engine=\"python\", storage_format=name.split(\"On\")[0])\n\n\n@pytest.fixture(scope=\"function\")\ndef get_unique_base_execution():\n \"\"\"Setup unique execution for a single function and yield its QueryCompiler that's suitable for inplace modifications.\"\"\"\n # It's better to use decimal IDs rather than hex ones due to factory names formatting\n execution_id = int(uuid.uuid4().hex, 16)\n format_name = f\"Base{execution_id}\"\n engine_name = \"Python\"\n execution_name = f\"{format_name}On{engine_name}\"\n\n # Dynamically building all the required classes to form a new execution\n base_qc = type(\n format_name, (TestQC,), {\"get_backend\": (lambda self: execution_name)}\n )\n base_io = type(\n f\"{execution_name}IO\", (BaseOnPythonIO,), {\"query_compiler_cls\": base_qc}\n )\n base_factory = type(\n f\"{execution_name}Factory\",\n (BaseOnPythonFactory,),\n {\"prepare\": classmethod(lambda cls: setattr(cls, \"io_cls\", base_io))},\n )\n\n # Setting up the new execution\n setattr(factories, f\"{execution_name}Factory\", base_factory)\n Backend.register_backend(\n execution_name, Execution(engine=engine_name, storage_format=format_name)\n )\n old_engine, old_format = modin.set_execution(\n engine=engine_name, storage_format=format_name\n )\n yield base_qc\n\n # Teardown the new execution\n modin.set_execution(engine=old_engine, storage_format=old_format)\n try:\n delattr(factories, f\"{execution_name}Factory\")\n except AttributeError:\n pass\n\n\ndef pytest_configure(config):\n execution = config.option.execution\n\n if execution is None:\n return\n\n if execution == BASE_EXECUTION_NAME:\n set_base_execution(BASE_EXECUTION_NAME)\n config.addinivalue_line(\n \"filterwarnings\", \"default:.*defaulting to pandas.*:UserWarning\"\n )\n else:\n partition, engine = execution.split(\"On\")\n modin.set_execution(engine=engine, storage_format=partition)\n\n\ndef pytest_runtest_call(item):\n custom_markers = [\"xfail\", \"skip\"]\n\n # dynamicly adding custom markers to tests\n for custom_marker in custom_markers:\n for marker in item.iter_markers(name=f\"{custom_marker}_executions\"):\n executions = marker.args[0]\n if not isinstance(executions, list):\n executions = [executions]\n\n current_execution = modin.utils.get_current_execution()\n reason = marker.kwargs.pop(\"reason\", \"\")\n\n item.add_marker(\n getattr(pytest.mark, custom_marker)(\n condition=current_execution in executions,\n reason=f\"Execution {current_execution} does not pass this test. {reason}\",\n **marker.kwargs,\n )\n )\n\n\n_doc_pytest_fixture = \"\"\"\nPytest fixture factory that makes temp {file_type} files for testing.\n\nYields:\n Function that generates {file_type} files\n\"\"\"\n\n\n@pytest.fixture(scope=\"class\")\ndef TestReadCSVFixture(tmp_path_factory):\n tmp_path = tmp_path_factory.mktemp(\"TestReadCSVFixture\")\n\n creator = _make_csv_file(data_dir=tmp_path)\n # each xdist worker spawned in separate process with separate namespace and dataset\n pytest.csvs_names = {}\n # test_read_csv_col_handling, test_read_csv_parsing\n pytest.csvs_names[\"test_read_csv_regular\"] = creator()\n # test_read_csv_parsing\n pytest.csvs_names[\"test_read_csv_yes_no\"] = creator(\n additional_col_values=[\"Yes\", \"true\", \"No\", \"false\"],\n )\n # test_read_csv_col_handling\n pytest.csvs_names[\"test_read_csv_blank_lines\"] = creator(\n add_blank_lines=True,\n )\n # test_read_csv_nans_handling\n pytest.csvs_names[\"test_read_csv_nans\"] = creator(\n add_blank_lines=True,\n additional_col_values=[\"\", \"N/A\", \"NA\", \"NULL\", \"custom_nan\", \"73\"],\n )\n # test_read_csv_error_handling\n pytest.csvs_names[\"test_read_csv_bad_lines\"] = creator(\n add_bad_lines=True,\n )\n yield\n\n\n@pytest.fixture\n@doc(_doc_pytest_fixture, file_type=\"csv\")\ndef make_csv_file(tmp_path):\n yield _make_csv_file(data_dir=tmp_path)\n\n\ndef create_fixture(file_type):\n @doc(_doc_pytest_fixture, file_type=file_type)\n def fixture(tmp_path):\n yield make_default_file(file_type=file_type, data_dir=tmp_path)\n\n return fixture\n\n\nfor file_type in (\"json\", \"html\", \"excel\", \"feather\", \"stata\", \"hdf\", \"pickle\", \"fwf\"):\n fixture = create_fixture(file_type)\n fixture.__name__ = f\"make_{file_type}_file\"\n globals()[fixture.__name__] = pytest.fixture(fixture)\n\n\n@pytest.fixture\ndef make_parquet_file():\n \"\"\"Pytest fixture factory that makes a parquet file/dir for testing.\n\n Yields:\n Function that generates a parquet file/dir\n \"\"\"\n filenames = []\n\n def _make_parquet_file(\n filename,\n nrows=NROWS,\n ncols=2,\n force=True,\n range_index_start=0,\n range_index_step=1,\n range_index_name=None,\n partitioned_columns=[],\n row_group_size: Optional[int] = None,\n ):\n \"\"\"Helper function to generate parquet files/directories.\n\n Args:\n filename: The name of test file, that should be created.\n nrows: Number of rows for the dataframe.\n ncols: Number of cols for the dataframe.\n force: Create a new file/directory even if one already exists.\n partitioned_columns: Create a partitioned directory using pandas.\n row_group_size: Maximum size of each row group.\n \"\"\"\n if force or not os.path.exists(filename):\n df = pandas.DataFrame(\n {f\"col{x + 1}\": np.arange(nrows) for x in range(ncols)}\n )\n index = pandas.RangeIndex(\n start=range_index_start,\n stop=range_index_start + (nrows * range_index_step),\n step=range_index_step,\n name=range_index_name,\n )\n if (\n range_index_start == 0\n and range_index_step == 1\n and range_index_name is None\n ):\n assert df.index.equals(index)\n else:\n df.index = index\n if len(partitioned_columns) > 0:\n df.to_parquet(\n filename,\n partition_cols=partitioned_columns,\n row_group_size=row_group_size,\n )\n else:\n df.to_parquet(filename, row_group_size=row_group_size)\n filenames.append(filename)\n\n # Return function that generates parquet files\n yield _make_parquet_file\n\n # Delete parquet file that was created\n for path in filenames:\n if os.path.exists(path):\n if os.path.isdir(path):\n shutil.rmtree(path)\n else:\n os.remove(path)\n\n\n@pytest.fixture\ndef make_sql_connection():\n \"\"\"Sets up sql connections and takes them down after the caller is done.\n\n Yields:\n Factory that generates sql connection objects\n \"\"\"\n\n def _sql_connection(filename, table=\"\"):\n # Remove file if exists\n if os.path.exists(filename):\n os.remove(filename)\n # Create connection and, if needed, table\n conn = \"sqlite:///{}\".format(filename)\n if table:\n df = pandas.DataFrame(\n {\n \"col1\": [0, 1, 2, 3, 4, 5, 6],\n \"col2\": [7, 8, 9, 10, 11, 12, 13],\n \"col3\": [14, 15, 16, 17, 18, 19, 20],\n \"col4\": [21, 22, 23, 24, 25, 26, 27],\n \"col5\": [0, 0, 0, 0, 0, 0, 0],\n }\n )\n df.to_sql(table, conn)\n return conn\n\n yield _sql_connection\n\n\n@pytest.fixture(scope=\"class\")\ndef TestReadGlobCSVFixture(tmp_path_factory):\n tmp_path = tmp_path_factory.mktemp(\"TestReadGlobCSVFixture\")\n\n base_name = get_unique_filename(extension=\"\")\n pytest.glob_path = str(tmp_path / \"{}_*.csv\".format(base_name))\n pytest.files = [str(tmp_path / \"{}_{}.csv\".format(base_name, i)) for i in range(11)]\n for fname in pytest.files:\n # Glob does not guarantee ordering so we have to remove the randomness in the generated csvs.\n _make_csv_file(data_dir=tmp_path)(fname, row_size=11, remove_randomness=True)\n\n yield\n\n\n@pytest.fixture\ndef get_generated_doc_urls():\n return lambda: _generated_doc_urls\n\n\n@pytest.fixture\ndef set_num_partitions(request):\n old_num_partitions = NPartitions.get()\n NPartitions.put(request.param)\n yield\n NPartitions.put(old_num_partitions)\n\n\n@pytest.fixture()\ndef set_benchmark_mode(request):\n old_benchmark_mode = BenchmarkMode.get()\n BenchmarkMode.put(request.param)\n yield\n BenchmarkMode.put(old_benchmark_mode)\n\n\n@pytest.fixture\ndef set_async_read_mode(request):\n old_async_read_mode = AsyncReadMode.get()\n AsyncReadMode.put(request.param)\n yield\n AsyncReadMode.put(old_async_read_mode)\n\n\n@pytest.fixture\ndef set_min_row_partition_size(request):\n old_min_row_partition_size = MinRowPartitionSize.get()\n MinRowPartitionSize.put(request.param)\n yield\n MinRowPartitionSize.put(old_min_row_partition_size)\n\n\nray_client_server = None\n\n\n@pytest.fixture\ndef s3_storage_options(worker_id):\n # # copied from pandas conftest.py:\n # https://github.com/pandas-dev/pandas/blob/32f789fbc5d5a72d9d1ac14935635289eeac9009/pandas/tests/io/conftest.py#L45\n # worker_id is a pytest fixture\n if GithubCI.get():\n url = \"http://localhost:5000/\"\n else:\n # If we hit this else-case, this test is being run locally. In that case, we want\n # each worker to point to a different port for its mock S3 service. The easiest way\n # to do that is to use the `worker_id`, which is unique, to determine what port to point\n # to. We arbitrarily assign `5` as a worker id to the master worker, since we need a number\n # for each worker, and we never run tests with more than `pytest -n 4`.\n worker_id = \"5\" if worker_id == \"master\" else worker_id.lstrip(\"gw\")\n url = f\"http://127.0.0.1:555{worker_id}/\"\n return {\"client_kwargs\": {\"endpoint_url\": url}}\n\n\n@pytest.fixture(scope=\"session\")\ndef monkeysession():\n with pytest.MonkeyPatch.context() as mp:\n yield mp\n\n\n@pytest.fixture(scope=\"session\")\ndef s3_base(worker_id, monkeysession):\n \"\"\"\n Fixture for mocking S3 interaction.\n\n Sets up moto server in separate process locally.\n\n Yields\n ------\n str\n URL for motoserver/moto CI service.\n \"\"\"\n # copied from pandas conftest.py\n # still need access keys for https://github.com/getmoto/moto/issues/1924\n monkeysession.setenv(\"AWS_ACCESS_KEY_ID\", \"foobar_key\")\n monkeysession.setenv(\"AWS_SECRET_ACCESS_KEY\", \"foobar_secret\")\n monkeysession.setenv(\"AWS_REGION\", \"us-west-2\")\n if GithubCI.get():\n if sys.platform in (\"darwin\", \"win32\", \"cygwin\") or (\n platform.machine() in (\"arm64\", \"aarch64\")\n or platform.machine().startswith(\"armv\")\n ):\n # pandas comments say:\n # DO NOT RUN on Windows/macOS/ARM, only Ubuntu\n # - subprocess in CI can cause timeouts\n # - GitHub Actions do not support\n # container services for the above OSs\n pytest.skip(\n \"S3 tests do not have a corresponding service in Windows, macOS \"\n + \"or ARM platforms\"\n )\n else:\n # assume CI has started moto in docker container:\n # https://docs.getmoto.org/en/latest/docs/server_mode.html#run-using-docker\n # It would be nice to start moto on another thread as in the\n # instructions here:\n # https://docs.getmoto.org/en/latest/docs/server_mode.html#start-within-python\n # but that gives 403 forbidden error when we try to create the bucket\n yield \"http://localhost:5000\"\n else:\n # Launching moto in server mode, i.e., as a separate process\n # with an S3 endpoint on localhost\n\n # If we hit this else-case, this test is being run locally. In that case, we want\n # each worker to point to a different port for its mock S3 service. The easiest way\n # to do that is to use the `worker_id`, which is unique, to determine what port to point\n # to.\n endpoint_port = (\n 5500 if worker_id == \"master\" else (5550 + int(worker_id.lstrip(\"gw\")))\n )\n endpoint_uri = f\"http://127.0.0.1:{endpoint_port}/\"\n\n # pipe to null to avoid logging in terminal\n # TODO any way to throw the error from here? e.g. i had an annoying problem\n # where I didn't have flask-cors and moto just failed .if there's an error\n # in the popen command and we throw an error within the body of the context\n # manager, the test just hangs forever.\n with subprocess.Popen(\n [\"moto_server\", \"s3\", \"-p\", str(endpoint_port)],\n stdout=subprocess.DEVNULL,\n stderr=subprocess.PIPE,\n ) as proc:\n for _ in range(50):\n try:\n # OK to go once server is accepting connections\n if requests.get(endpoint_uri).ok:\n break\n except Exception:\n # try again while we still have retries\n time.sleep(0.1)\n else:\n proc.terminate()\n _, errs = proc.communicate()\n raise RuntimeError(\n \"Could not connect to moto server after 50 tries. \"\n + f\"See stderr for extra info: {errs}\"\n )\n yield endpoint_uri\n\n proc.terminate()\n\n\n@pytest.fixture\ndef s3_resource(s3_base):\n \"\"\"\n Set up S3 bucket with contents. The primary bucket name is \"modin-test\".\n\n When running locally, this function should be safe even if there are multiple pytest\n workers running in parallel because each worker gets its own endpoint. When running\n in CI, we use a single endpoint for all workers, so we can't have multiple pytest\n workers running in parallel.\n \"\"\"\n bucket = \"modin-test\"\n conn = boto3.resource(\"s3\", endpoint_url=s3_base)\n cli = boto3.client(\"s3\", endpoint_url=s3_base)\n\n # https://github.com/getmoto/moto/issues/3292\n # without location, I get\n # botocore.exceptions.ClientError: An error occurred\n # (IllegalLocationConstraintException) when calling the CreateBucket operation:\n # The unspecified location constraint is incompatible for the region specific\n # endpoint this request was sent to.\n # even if I delete os.environ['AWS_REGION'] but somehow pandas can get away with\n # this.\n try:\n cli.create_bucket(\n Bucket=bucket, CreateBucketConfiguration={\"LocationConstraint\": \"us-west-2\"}\n )\n except Exception as e:\n # OK if bucket already exists, but want to raise other exceptions.\n # The exception raised by `create_bucket` is made using a factory,\n # so we need to check using this method of reading the response rather\n # than just checking the type of the exception.\n response = getattr(e, \"response\", {})\n error_code = response.get(\"Error\", {}).get(\"Code\", \"\")\n if error_code not in (\"BucketAlreadyOwnedByYou\", \"BucketAlreadyExists\"):\n raise\n for _ in range(20):\n # We want to wait until bucket creation is finished.\n if cli.list_buckets()[\"Buckets\"]:\n break\n time.sleep(0.1)\n if not cli.list_buckets()[\"Buckets\"]:\n raise RuntimeError(\"Could not create bucket\")\n\n s3fs.S3FileSystem.clear_instance_cache()\n\n s3 = s3fs.S3FileSystem(client_kwargs={\"endpoint_url\": s3_base})\n\n test_s3_files = [\n (\"modin-bugs/multiple_csv/\", \"modin/tests/pandas/data/multiple_csv/\"),\n (\n \"modin-bugs/test_data_dir.parquet/\",\n \"modin/tests/pandas/data/test_data_dir.parquet/\",\n ),\n (\"modin-bugs/test_data.parquet\", \"modin/tests/pandas/data/test_data.parquet\"),\n (\"modin-bugs/test_data.json\", \"modin/tests/pandas/data/test_data.json\"),\n (\"modin-bugs/test_data.fwf\", \"modin/tests/pandas/data/test_data.fwf\"),\n (\"modin-bugs/test_data.feather\", \"modin/tests/pandas/data/test_data.feather\"),\n (\"modin-bugs/issue5159.parquet/\", \"modin/tests/pandas/data/issue5159.parquet/\"),\n ]\n for s3_key, file_name in test_s3_files:\n s3.put(file_name, f\"{bucket}/{s3_key}\", recursive=s3_key.endswith(\"/\"))\n\n yield conn\n\n s3.rm(bucket, recursive=True)\n for _ in range(20):\n # We want to wait until the deletion finishes.\n if not cli.list_buckets()[\"Buckets\"]:\n break\n time.sleep(0.1)\n\n\n@pytest.fixture\ndef modify_config(request):\n values = request.param\n old_values = {}\n\n for key, value in values.items():\n old_values[key] = key.get()\n key.put(value)\n\n yield # waiting for the test to be completed\n # restoring old parameters\n for key, value in old_values.items():\n try:\n key.put(value)\n except ValueError as e:\n # sometimes bool env variables have 'None' as a default value, which\n # causes a ValueError when we try to set this value back, as technically,\n # only bool values are allowed (and 'None' is not a bool), in this case\n # we try to set 'False' instead\n if key.type == bool and value is None:\n key.put(False)\n else:\n raise e\n\n\n@contextmanager\ndef copy_and_restore(\n dicts: Iterable[defaultdict],\n) -> None:\n \"\"\"\n Make deep copies of defaultdicts and restore them upon exiting this context.\n\n Ideally this function would be a fixture, but we want to pass it parameters\n and use it in other fixtures, and it does not seem to be possible to pass\n parameters from one fixture to another.\n\n Parameters\n ----------\n dicts : Iterable[defaultdict]\n The dicts to copy and restore.\n \"\"\"\n try:\n # Use a tuples of tuples instead of a dict mapping each original dict\n # to its copy, because the original dict is not hashable.\n original_dict_to_copy = tuple(\n (original_dict, copy.deepcopy(original_dict)) for original_dict in dicts\n )\n yield\n finally:\n for original_dict, dict_copy in original_dict_to_copy:\n original_dict.clear()\n original_dict.update(dict_copy)\n\n\n@pytest.fixture(autouse=True)\ndef clean_up_extensions():\n\n with copy_and_restore(\n (\n pd.dataframe.DataFrame._extensions,\n pd.Series._extensions,\n pd.base.BasePandasDataset._extensions,\n _GENERAL_EXTENSIONS,\n pd.groupby.DataFrameGroupBy._extensions,\n pd.groupby.SeriesGroupBy._extensions,\n )\n ):\n yield\n\n from modin.pandas.api.extensions.extensions import _attrs_to_delete_on_test\n\n for k, v in _attrs_to_delete_on_test.items():\n for obj in v:\n delattr(k, obj)\n _attrs_to_delete_on_test.clear()\n\n\n@pytest.fixture(autouse=True)\ndef clean_up_auto_backend_switching():\n\n with copy_and_restore(\n (\n _CLASS_AND_BACKEND_TO_POST_OP_SWITCH_METHODS,\n _CLASS_AND_BACKEND_TO_PRE_OP_SWITCH_METHODS,\n )\n ):\n yield\n\n\n@pytest.fixture(autouse=True)\ndef assert_no_root_logging(caplog):\n try:\n import xgboost\n except ImportError:\n xgboost_path = None\n else:\n xgboost_path = os.path.dirname(xgboost.__file__)\n root_logger = logging.getLogger()\n # Capture logs at any level, i.e. at level >= logging.NOTSET.\n with caplog.at_level(logging.NOTSET):\n yield\n # Note that because this code is in a fixture, we have to use\n # caplog.get_records(when=\"call\") instead of caplog.records:\n # https://github.com/pytest-dev/pytest/issues/4033\n assert not any(\n record.name == root_logger.name\n # Allow xgboost to log to root.\n # TODO(https://github.com/modin-project/modin/issues/5194): Check\n # whether we can remove this exception once we use a newer version of\n # xgboost.\n and not (xgboost_path is not None and record.pathname.startswith(xgboost_path))\n for record in caplog.get_records(when=\"call\")\n )\n"
},
{
"path": "modin/core/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's core functionality.\"\"\"\n"
},
{
"path": "modin/core/computation/__init__.py",
"content": ""
},
{
"path": "modin/core/computation/align.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nCore eval alignment algorithms. Forked from pandas.core.computation.align\n\"\"\"\n\nfrom __future__ import annotations\n\nimport warnings\nfrom collections.abc import Sequence\nfrom functools import (\n partial,\n wraps,\n)\nfrom typing import (\n Callable,\n)\n\nimport numpy as np\nimport pandas\nimport pandas.core.common as com\nfrom pandas._typing import F\nfrom pandas.core.base import PandasObject\nfrom pandas.errors import PerformanceWarning\n\nfrom modin.core.computation.common import result_type_many\nfrom modin.pandas import DataFrame, Series\nfrom modin.pandas.base import BasePandasDataset\n\n\ndef _align_core_single_unary_op(\n term,\n) -> tuple[partial | type[BasePandasDataset], dict[str, pandas.Index] | None]:\n typ: partial | type[BasePandasDataset]\n axes: dict[str, pandas.Index] | None = None\n\n if isinstance(term.value, np.ndarray):\n typ = partial(np.asanyarray, dtype=term.value.dtype)\n else:\n typ = type(term.value)\n if hasattr(term.value, \"axes\"):\n axes = _zip_axes_from_type(typ, term.value.axes)\n\n return typ, axes\n\n\ndef _zip_axes_from_type(\n typ: type[BasePandasDataset], new_axes: Sequence[pandas.Index]\n) -> dict[str, pandas.Index]:\n return {name: new_axes[i] for i, name in enumerate(typ._AXIS_ORDERS)}\n\n\ndef _any_pandas_objects(terms) -> bool:\n \"\"\"\n Check a sequence of terms for instances of PandasObject.\n \"\"\"\n return any(isinstance(term.value, PandasObject) for term in terms)\n\n\ndef _filter_special_cases(f) -> Callable[[F], F]:\n @wraps(f)\n def wrapper(terms):\n # single unary operand\n if len(terms) == 1:\n return _align_core_single_unary_op(terms[0])\n\n term_values = (term.value for term in terms)\n\n # we don't have any pandas objects\n if not _any_pandas_objects(terms):\n return result_type_many(*term_values), None\n\n return f(terms)\n\n return wrapper\n\n\n@_filter_special_cases\ndef _align_core(terms):\n term_index = [i for i, term in enumerate(terms) if hasattr(term.value, \"axes\")]\n term_dims = [terms[i].value.ndim for i in term_index]\n\n ndims = pandas.Series(dict(zip(term_index, term_dims)))\n\n # initial axes are the axes of the largest-axis'd term\n biggest = terms[ndims.idxmax()].value\n typ = biggest._constructor\n axes = biggest.axes\n naxes = len(axes)\n gt_than_one_axis = naxes > 1\n\n for value in (terms[i].value for i in term_index):\n is_series = isinstance(value, Series)\n is_series_and_gt_one_axis = is_series and gt_than_one_axis\n\n for axis, items in enumerate(value.axes):\n if is_series_and_gt_one_axis:\n ax, itm = naxes - 1, value.index\n else:\n ax, itm = axis, items\n\n if not axes[ax].is_(itm):\n axes[ax] = axes[ax].union(itm)\n\n for i, ndim in ndims.items():\n for axis, items in zip(range(ndim), axes):\n ti = terms[i].value\n\n if hasattr(ti, \"reindex\"):\n transpose = isinstance(ti, Series) and naxes > 1\n reindexer = axes[naxes - 1] if transpose else items\n\n term_axis_size = len(ti.axes[axis])\n reindexer_size = len(reindexer)\n\n ordm = np.log10(max(1, abs(reindexer_size - term_axis_size)))\n if ordm >= 1 and reindexer_size >= 10000:\n w = (\n f\"Alignment difference on axis {axis} is larger \"\n + f\"than an order of magnitude on term {repr(terms[i].name)}, \"\n + f\"by more than {ordm:.4g}; performance may suffer.\"\n )\n warnings.warn(w, category=PerformanceWarning)\n\n obj = ti.reindex(reindexer, axis=axis, copy=False)\n terms[i].update(obj)\n\n terms[i].update(terms[i].value.values)\n\n return typ, _zip_axes_from_type(typ, axes)\n\n\ndef align_terms(terms):\n \"\"\"\n Align a set of terms.\n \"\"\"\n try:\n # flatten the parse tree (a nested list, really)\n terms = list(com.flatten(terms))\n except TypeError:\n # can't iterate so it must just be a constant or single variable\n if isinstance(terms.value, (Series, DataFrame)):\n typ = type(terms.value)\n return typ, _zip_axes_from_type(typ, terms.value.axes)\n return np.result_type(terms.type), None\n\n # if all resolved variables are numeric scalars\n if all(term.is_scalar for term in terms):\n return result_type_many(*(term.value for term in terms)).type, None\n\n # perform the main alignment\n typ, axes = _align_core(terms)\n return typ, axes\n\n\ndef reconstruct_object(typ, obj, axes, dtype):\n \"\"\"\n Reconstruct an object given its type, raw value, and possibly empty\n (None) axes.\n\n Parameters\n ----------\n typ : object\n A type\n obj : object\n The value to use in the type constructor\n axes : dict\n The axes to use to construct the resulting pandas object\n\n Returns\n -------\n ret : typ\n An object of type ``typ`` with the value `obj` and possible axes\n `axes`.\n \"\"\"\n try:\n typ = typ.type\n except AttributeError:\n pass\n\n res_t = np.result_type(obj.dtype, dtype)\n\n if not isinstance(typ, partial) and issubclass(typ, PandasObject):\n return typ(obj, dtype=res_t, **axes)\n\n # special case for pathological things like ~True/~False\n if hasattr(res_t, \"type\") and typ == np.bool_ and res_t != np.bool_:\n ret_value = res_t.type(obj)\n else:\n ret_value = typ(obj).astype(res_t)\n # The condition is to distinguish 0-dim array (returned in case of\n # scalar) and 1 element array\n # e.g. np.array(0) and np.array([0])\n if (\n len(obj.shape) == 1\n and len(obj) == 1\n and not isinstance(ret_value, np.ndarray)\n ):\n ret_value = np.array([ret_value]).astype(res_t)\n\n return ret_value\n"
},
{
"path": "modin/core/computation/check.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nForked from pandas.core.computation.check\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom pandas.compat._optional import import_optional_dependency\n\nne = import_optional_dependency(\"numexpr\", errors=\"warn\")\nNUMEXPR_INSTALLED = ne is not None\n\n__all__ = [\"NUMEXPR_INSTALLED\"]\n"
},
{
"path": "modin/core/computation/common.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nForked from pandas.core.computation.common\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom functools import reduce\n\nimport numpy as np\nfrom pandas._config import get_option\nfrom pandas.core.dtypes.cast import find_common_type\nfrom pandas.core.dtypes.common import is_extension_array_dtype\n\n\ndef ensure_decoded(s) -> str:\n \"\"\"\n If we have bytes, decode them to unicode.\n \"\"\"\n if isinstance(s, (np.bytes_, bytes)):\n s = s.decode(get_option(\"display.encoding\"))\n return s\n\n\ndef result_type_many(*arrays_and_dtypes):\n \"\"\"\n Wrapper around numpy.result_type which overcomes the NPY_MAXARGS (32)\n argument limit.\n \"\"\"\n try:\n return np.result_type(*arrays_and_dtypes)\n except ValueError:\n # we have > NPY_MAXARGS terms in our expression\n return reduce(np.result_type, arrays_and_dtypes)\n except TypeError:\n arr_and_dtypes = list(arrays_and_dtypes)\n ea_dtypes, non_ea_dtypes = [], []\n for arr_or_dtype in arr_and_dtypes:\n if is_extension_array_dtype(arr_or_dtype):\n ea_dtypes.append(arr_or_dtype)\n else:\n non_ea_dtypes.append(arr_or_dtype)\n\n if non_ea_dtypes:\n try:\n np_dtype = np.result_type(*non_ea_dtypes)\n except ValueError:\n np_dtype = reduce(np.result_type, arrays_and_dtypes)\n return find_common_type(ea_dtypes + [np_dtype])\n\n return find_common_type(ea_dtypes)\n"
},
{
"path": "modin/core/computation/engines.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nEngine classes for :func:`~pandas.eval`. Forked from pandas.core.computation.engines\n\"\"\"\n\nfrom __future__ import annotations\n\nimport abc\nfrom typing import TYPE_CHECKING\n\nfrom pandas.errors import NumExprClobberingError\nfrom pandas.io.formats import printing\n\nfrom modin.core.computation.align import (\n align_terms,\n reconstruct_object,\n)\nfrom modin.core.computation.ops import (\n MATHOPS,\n REDUCTIONS,\n)\n\nif TYPE_CHECKING:\n from modin.core.computation.expr import Expr\n\n_ne_builtins = frozenset(MATHOPS + REDUCTIONS)\n\n\ndef _check_ne_builtin_clash(expr: Expr) -> None:\n \"\"\"\n Attempt to prevent foot-shooting in a helpful way.\n\n Parameters\n ----------\n expr : Expr\n Terms can contain\n \"\"\"\n names = expr.names\n overlap = names & _ne_builtins\n\n if overlap:\n s = \", \".join([repr(x) for x in overlap])\n raise NumExprClobberingError(\n f'Variables in expression \"{expr}\" overlap with builtins: ({s})'\n )\n\n\nclass AbstractEngine(metaclass=abc.ABCMeta):\n \"\"\"Object serving as a base class for all engines.\"\"\"\n\n has_neg_frac = False\n\n def __init__(self, expr) -> None:\n self.expr = expr\n self.aligned_axes = None\n self.result_type = None\n\n def convert(self) -> str:\n \"\"\"\n Convert an expression for evaluation.\n\n Defaults to return the expression as a string.\n \"\"\"\n return printing.pprint_thing(self.expr)\n\n def evaluate(self) -> object:\n \"\"\"\n Run the engine on the expression.\n\n This method performs alignment which is necessary no matter what engine\n is being used, thus its implementation is in the base class.\n\n Returns\n -------\n object\n The result of the passed expression.\n \"\"\"\n if not self._is_aligned:\n self.result_type, self.aligned_axes = align_terms(self.expr.terms)\n\n # make sure no names in resolvers and locals/globals clash\n res = self._evaluate()\n return reconstruct_object(\n self.result_type, res, self.aligned_axes, self.expr.terms.return_type\n )\n\n @property\n def _is_aligned(self) -> bool:\n return self.aligned_axes is not None and self.result_type is not None\n\n @abc.abstractmethod\n def _evaluate(self):\n \"\"\"\n Return an evaluated expression.\n\n Parameters\n ----------\n env : Scope\n The local and global environment in which to evaluate an\n expression.\n\n Notes\n -----\n Must be implemented by subclasses.\n \"\"\"\n\n\nclass NumExprEngine(AbstractEngine):\n \"\"\"NumExpr engine class\"\"\"\n\n has_neg_frac = True\n\n def _evaluate(self):\n import numexpr as ne\n\n # convert the expression to a valid numexpr expression\n s = self.convert()\n\n env = self.expr.env\n scope = env.full_scope\n _check_ne_builtin_clash(self.expr)\n return ne.evaluate(s, local_dict=scope)\n\n\nclass PythonEngine(AbstractEngine):\n \"\"\"\n Evaluate an expression in Python space.\n\n Mostly for testing purposes.\n \"\"\"\n\n has_neg_frac = False\n\n def evaluate(self):\n return self.expr()\n\n def _evaluate(self) -> None:\n pass\n\n\nENGINES: dict[str, type[AbstractEngine]] = {\n \"numexpr\": NumExprEngine,\n \"python\": PythonEngine,\n}\n"
},
{
"path": "modin/core/computation/eval.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nTop level ``eval`` module. Forked from pandas.core.computation.eval\n\"\"\"\n\nfrom __future__ import annotations\n\nimport tokenize\nimport warnings\n\nfrom pandas.core.dtypes.common import is_extension_array_dtype\nfrom pandas.io.formats.printing import pprint_thing\nfrom pandas.util._validators import validate_bool_kwarg\n\nfrom modin.core.computation.check import NUMEXPR_INSTALLED\nfrom modin.core.computation.engines import ENGINES\nfrom modin.core.computation.expr import (\n PARSERS,\n Expr,\n)\nfrom modin.core.computation.ops import BinOp\nfrom modin.core.computation.parsing import tokenize_string\nfrom modin.core.computation.scope import ensure_scope\nfrom modin.pandas.base import BasePandasDataset\n\n\ndef _check_engine(engine: str | None) -> str:\n \"\"\"\n Make sure a valid engine is passed.\n\n Parameters\n ----------\n engine : str\n String to validate.\n\n Raises\n ------\n KeyError\n * If an invalid engine is passed.\n ImportError\n * If numexpr was requested but doesn't exist.\n\n Returns\n -------\n str\n Engine name.\n \"\"\"\n\n if engine is None:\n engine = \"numexpr\" if NUMEXPR_INSTALLED else \"python\"\n\n if engine not in ENGINES:\n valid_engines = list(ENGINES.keys())\n raise KeyError(\n f\"Invalid engine '{engine}' passed, valid engines are {valid_engines}\"\n )\n\n # TODO: validate this in a more general way (thinking of future engines\n # that won't necessarily be import-able)\n # Could potentially be done on engine instantiation\n if engine == \"numexpr\" and not NUMEXPR_INSTALLED:\n raise ImportError(\n \"'numexpr' is not installed or an unsupported version. Cannot use \"\n + \"engine='numexpr' for query/eval if 'numexpr' is not installed\"\n )\n\n return engine\n\n\ndef _check_parser(parser: str):\n \"\"\"\n Make sure a valid parser is passed.\n\n Parameters\n ----------\n parser : str\n\n Raises\n ------\n KeyError\n * If an invalid parser is passed\n \"\"\"\n if parser not in PARSERS:\n raise KeyError(\n f\"Invalid parser '{parser}' passed, valid parsers are {PARSERS.keys()}\"\n )\n\n\ndef _check_resolvers(resolvers):\n if resolvers is not None:\n for resolver in resolvers:\n if not hasattr(resolver, \"__getitem__\"):\n name = type(resolver).__name__\n raise TypeError(\n f\"Resolver of type '{name}' does not \"\n + \"implement the __getitem__ method\"\n )\n\n\ndef _check_expression(expr):\n \"\"\"\n Make sure an expression is not an empty string\n\n Parameters\n ----------\n expr : object\n An object that can be converted to a string\n\n Raises\n ------\n ValueError\n * If expr is an empty string\n \"\"\"\n if not expr:\n raise ValueError(\"expr cannot be an empty string\")\n\n\ndef _convert_expression(expr) -> str:\n \"\"\"\n Convert an object to an expression.\n\n This function converts an object to an expression (a unicode string) and\n checks to make sure it isn't empty after conversion. This is used to\n convert operators to their string representation for recursive calls to\n :func:`~pandas.eval`.\n\n Parameters\n ----------\n expr : object\n The object to be converted to a string.\n\n Returns\n -------\n str\n The string representation of an object.\n\n Raises\n ------\n ValueError\n * If the expression is empty.\n \"\"\"\n s = pprint_thing(expr)\n _check_expression(s)\n return s\n\n\ndef _check_for_locals(expr: str, stack_level: int, parser: str):\n at_top_of_stack = stack_level == 0\n not_pandas_parser = parser != \"pandas\"\n\n if not_pandas_parser:\n msg = \"The '@' prefix is only supported by the pandas parser\"\n elif at_top_of_stack:\n msg = (\n \"The '@' prefix is not allowed in top-level eval calls.\\n\"\n + \"please refer to your variables by name without the '@' prefix.\"\n )\n\n if at_top_of_stack or not_pandas_parser:\n for toknum, tokval in tokenize_string(expr):\n if toknum == tokenize.OP and tokval == \"@\":\n raise SyntaxError(msg)\n\n\ndef eval(\n expr: str | BinOp, # we leave BinOp out of the docstr bc it isn't for users\n parser: str = \"pandas\",\n engine: str | None = None,\n local_dict=None,\n global_dict=None,\n resolvers=(),\n level: int = 0,\n target=None,\n inplace: bool = False,\n):\n \"\"\"\n Evaluate a Python expression as a string using various backends.\n\n The following arithmetic operations are supported: ``+``, ``-``, ``*``,\n ``/``, ``**``, ``%``, ``//`` (python engine only) along with the following\n boolean operations: ``|`` (or), ``&`` (and), and ``~`` (not).\n Additionally, the ``'pandas'`` parser allows the use of :keyword:`and`,\n :keyword:`or`, and :keyword:`not` with the same semantics as the\n corresponding bitwise operators. :class:`~pandas.Series` and\n :class:`~pandas.DataFrame` objects are supported and behave as they would\n with plain ol' Python evaluation.\n\n Parameters\n ----------\n expr : str\n The expression to evaluate. This string cannot contain any Python\n `statements\n `__,\n only Python `expressions\n `__.\n parser : {'pandas', 'python'}, default 'pandas'\n The parser to use to construct the syntax tree from the expression. The\n default of ``'pandas'`` parses code slightly different than standard\n Python. Alternatively, you can parse an expression using the\n ``'python'`` parser to retain strict Python semantics. See the\n :ref:`enhancing performance ` documentation for\n more details.\n engine : {'python', 'numexpr'}, default 'numexpr'\n\n The engine used to evaluate the expression. Supported engines are\n\n - None : tries to use ``numexpr``, falls back to ``python``\n - ``'numexpr'`` : This default engine evaluates pandas objects using\n numexpr for large speed ups in complex expressions with large frames.\n - ``'python'`` : Performs operations as if you had ``eval``'d in top\n level python. This engine is generally not that useful.\n\n More backends may be available in the future.\n local_dict : dict or None, optional\n A dictionary of local variables, taken from locals() by default.\n global_dict : dict or None, optional\n A dictionary of global variables, taken from globals() by default.\n resolvers : list of dict-like or None, optional\n A list of objects implementing the ``__getitem__`` special method that\n you can use to inject an additional collection of namespaces to use for\n variable lookup. For example, this is used in the\n :meth:`~DataFrame.query` method to inject the\n ``DataFrame.index`` and ``DataFrame.columns``\n variables that refer to their respective :class:`~pandas.DataFrame`\n instance attributes.\n level : int, optional\n The number of prior stack frames to traverse and add to the current\n scope. Most users will **not** need to change this parameter.\n target : object, optional, default None\n This is the target object for assignment. It is used when there is\n variable assignment in the expression. If so, then `target` must\n support item assignment with string keys, and if a copy is being\n returned, it must also support `.copy()`.\n inplace : bool, default False\n If `target` is provided, and the expression mutates `target`, whether\n to modify `target` inplace. Otherwise, return a copy of `target` with\n the mutation.\n\n Returns\n -------\n ndarray, numeric scalar, DataFrame, Series, or None\n The completion value of evaluating the given code or None if ``inplace=True``.\n\n Raises\n ------\n ValueError\n There are many instances where such an error can be raised:\n\n - `target=None`, but the expression is multiline.\n - The expression is multiline, but not all them have item assignment.\n An example of such an arrangement is this:\n\n a = b + 1\n a + 2\n\n Here, there are expressions on different lines, making it multiline,\n but the last line has no variable assigned to the output of `a + 2`.\n - `inplace=True`, but the expression is missing item assignment.\n - Item assignment is provided, but the `target` does not support\n string item assignment.\n - Item assignment is provided and `inplace=False`, but the `target`\n does not support the `.copy()` method\n\n See Also\n --------\n DataFrame.query : Evaluates a boolean expression to query the columns\n of a frame.\n DataFrame.eval : Evaluate a string describing operations on\n DataFrame columns.\n\n Notes\n -----\n The ``dtype`` of any objects involved in an arithmetic ``%`` operation are\n recursively cast to ``float64``.\n\n See the :ref:`enhancing performance ` documentation for\n more details.\n\n Examples\n --------\n >>> df = pd.DataFrame({\"animal\": [\"dog\", \"pig\"], \"age\": [10, 20]})\n >>> df\n animal age\n 0 dog 10\n 1 pig 20\n\n We can add a new column using ``pd.eval``:\n\n >>> pd.eval(\"double_age = df.age * 2\", target=df)\n animal age double_age\n 0 dog 10 20\n 1 pig 20 40\n \"\"\"\n inplace = validate_bool_kwarg(inplace, \"inplace\")\n\n exprs: list[str | BinOp]\n if isinstance(expr, str):\n _check_expression(expr)\n exprs = [e.strip() for e in expr.splitlines() if e.strip() != \"\"]\n else:\n # ops.BinOp; for internal compat, not intended to be passed by users\n exprs = [expr]\n multi_line = len(exprs) > 1\n\n if multi_line and target is None:\n raise ValueError(\n \"multi-line expressions are only valid in the \"\n + \"context of data, use DataFrame.eval\"\n )\n engine = _check_engine(engine)\n _check_parser(parser)\n _check_resolvers(resolvers)\n\n ret = None\n first_expr = True\n target_modified = False\n\n for expr in exprs:\n expr = _convert_expression(expr)\n _check_for_locals(expr, level, parser)\n\n # get our (possibly passed-in) scope\n env = ensure_scope(\n level + 1,\n global_dict=global_dict,\n local_dict=local_dict,\n resolvers=resolvers,\n target=target,\n )\n\n parsed_expr = Expr(expr, engine=engine, parser=parser, env=env)\n\n if engine == \"numexpr\" and (\n is_extension_array_dtype(parsed_expr.terms.return_type)\n or getattr(parsed_expr.terms, \"operand_types\", None) is not None\n and any(\n is_extension_array_dtype(elem)\n for elem in parsed_expr.terms.operand_types\n )\n ):\n warnings.warn(\n \"Engine has switched to 'python' because numexpr does not support \"\n + \"extension array dtypes. Please set your engine to python manually.\",\n RuntimeWarning,\n )\n engine = \"python\"\n\n # construct the engine and evaluate the parsed expression\n eng = ENGINES[engine]\n eng_inst = eng(parsed_expr)\n ret = eng_inst.evaluate()\n\n if parsed_expr.assigner is None:\n if multi_line:\n raise ValueError(\n \"Multi-line expressions are only valid \"\n + \"if all expressions contain an assignment\"\n )\n if inplace:\n raise ValueError(\"Cannot operate inplace if there is no assignment\")\n\n # assign if needed\n assigner = parsed_expr.assigner\n if env.target is not None and assigner is not None:\n target_modified = True\n\n # if returning a copy, copy only on the first assignment\n if not inplace and first_expr:\n try:\n target = env.target\n if isinstance(target, BasePandasDataset):\n target = target.copy(deep=True)\n else:\n target = target.copy()\n except AttributeError as err:\n raise ValueError(\"Cannot return a copy of the target\") from err\n else:\n target = env.target\n\n # TypeError is most commonly raised (e.g. int, list), but you\n # get IndexError if you try to do this assignment on np.ndarray.\n # we will ignore numpy warnings here; e.g. if trying\n # to use a non-numeric indexer\n try:\n if inplace and isinstance(target, BasePandasDataset):\n target.loc[:, assigner] = ret\n else:\n target[assigner] = ret # pyright: ignore[reportGeneralTypeIssues]\n except (TypeError, IndexError) as err:\n raise ValueError(\"Cannot assign expression output to target\") from err\n\n if not resolvers:\n resolvers = ({assigner: ret},)\n else:\n # existing resolver needs updated to handle\n # case of mutating existing column in copy\n for resolver in resolvers:\n if assigner in resolver:\n resolver[assigner] = ret\n break\n else:\n resolvers += ({assigner: ret},)\n\n ret = None\n first_expr = False\n\n # We want to exclude `inplace=None` as being False.\n return (target if target_modified else ret) if inplace is False else None\n"
},
{
"path": "modin/core/computation/expr.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\n:func:`~pandas.eval` parsers.\n\nForked from pandas.core.computation.expr\n\"\"\"\n\nfrom __future__ import annotations\n\nimport ast\nimport tokenize\nfrom functools import (\n partial,\n reduce,\n)\nfrom keyword import iskeyword\nfrom typing import (\n Callable,\n ClassVar,\n TypeVar,\n)\n\nimport numpy as np\nimport pandas.core.common as com\nfrom pandas.errors import UndefinedVariableError\nfrom pandas.io.formats import printing\n\nfrom modin.core.computation.ops import (\n ARITH_OPS_SYMS,\n BOOL_OPS_SYMS,\n CMP_OPS_SYMS,\n LOCAL_TAG,\n UNARY_OPS_SYMS,\n BinOp,\n Constant,\n FuncNode,\n Op,\n Term,\n UnaryOp,\n is_term,\n)\nfrom modin.core.computation.parsing import (\n clean_backtick_quoted_toks,\n tokenize_string,\n)\nfrom modin.core.computation.scope import Scope\n\n\ndef _rewrite_assign(tok: tuple[int, str]) -> tuple[int, str]:\n \"\"\"\n Rewrite the assignment operator for PyTables expressions that use ``=``\n as a substitute for ``==``.\n\n Parameters\n ----------\n tok : tuple of int, str\n ints correspond to the all caps constants in the tokenize module\n\n Returns\n -------\n tuple of int, str\n Either the input or token or the replacement values\n \"\"\"\n toknum, tokval = tok\n return toknum, \"==\" if tokval == \"=\" else tokval\n\n\ndef _replace_booleans(tok: tuple[int, str]) -> tuple[int, str]:\n \"\"\"\n Replace ``&`` with ``and`` and ``|`` with ``or`` so that bitwise\n precedence is changed to boolean precedence.\n\n Parameters\n ----------\n tok : tuple of int, str\n ints correspond to the all caps constants in the tokenize module\n\n Returns\n -------\n tuple of int, str\n Either the input or token or the replacement values\n \"\"\"\n toknum, tokval = tok\n if toknum == tokenize.OP:\n if tokval == \"&\":\n return tokenize.NAME, \"and\"\n elif tokval == \"|\":\n return tokenize.NAME, \"or\"\n return toknum, tokval\n return toknum, tokval\n\n\ndef _replace_locals(tok: tuple[int, str]) -> tuple[int, str]:\n \"\"\"\n Replace local variables with a syntactically valid name.\n\n Parameters\n ----------\n tok : tuple of int, str\n ints correspond to the all caps constants in the tokenize module\n\n Returns\n -------\n tuple of int, str\n Either the input or token or the replacement values\n\n Notes\n -----\n This is somewhat of a hack in that we rewrite a string such as ``'@a'`` as\n ``'__pd_eval_local_a'`` by telling the tokenizer that ``__pd_eval_local_``\n is a ``tokenize.OP`` and to replace the ``'@'`` symbol with it.\n \"\"\"\n toknum, tokval = tok\n if toknum == tokenize.OP and tokval == \"@\":\n return tokenize.OP, LOCAL_TAG\n return toknum, tokval\n\n\ndef _compose2(f, g):\n \"\"\"\n Compose 2 callables.\n \"\"\"\n return lambda *args, **kwargs: f(g(*args, **kwargs))\n\n\ndef _compose(*funcs):\n \"\"\"\n Compose 2 or more callables.\n \"\"\"\n assert len(funcs) > 1, \"At least 2 callables must be passed to compose\"\n return reduce(_compose2, funcs)\n\n\ndef _preparse(\n source: str,\n f=_compose(\n _replace_locals, _replace_booleans, _rewrite_assign, clean_backtick_quoted_toks\n ),\n) -> str:\n \"\"\"\n Compose a collection of tokenization functions.\n\n Parameters\n ----------\n source : str\n A Python source code string\n f : callable\n This takes a tuple of (toknum, tokval) as its argument and returns a\n tuple with the same structure but possibly different elements. Defaults\n to the composition of ``_rewrite_assign``, ``_replace_booleans``, and\n ``_replace_locals``.\n\n Returns\n -------\n str\n Valid Python source code\n\n Notes\n -----\n The `f` parameter can be any callable that takes *and* returns input of the\n form ``(toknum, tokval)``, where ``toknum`` is one of the constants from\n the ``tokenize`` module and ``tokval`` is a string.\n \"\"\"\n assert callable(f), \"f must be callable\"\n return tokenize.untokenize(f(x) for x in tokenize_string(source))\n\n\ndef _is_type(t):\n \"\"\"\n Factory for a type checking function of type ``t`` or tuple of types.\n \"\"\"\n return lambda x: isinstance(x.value, t)\n\n\n_is_list = _is_type(list)\n_is_str = _is_type(str)\n\n\n# partition all AST nodes\n_all_nodes = frozenset(\n node\n for node in (getattr(ast, name) for name in dir(ast))\n if isinstance(node, type) and issubclass(node, ast.AST)\n)\n\n\ndef _filter_nodes(superclass, all_nodes=_all_nodes):\n \"\"\"\n Filter out AST nodes that are subclasses of ``superclass``.\n \"\"\"\n node_names = (node.__name__ for node in all_nodes if issubclass(node, superclass))\n return frozenset(node_names)\n\n\n_all_node_names = frozenset(x.__name__ for x in _all_nodes)\n_mod_nodes = _filter_nodes(ast.mod)\n_stmt_nodes = _filter_nodes(ast.stmt)\n_expr_context_nodes = _filter_nodes(ast.expr_context)\n_boolop_nodes = _filter_nodes(ast.boolop)\n_handler_nodes = _filter_nodes(ast.excepthandler)\n_arguments_nodes = _filter_nodes(ast.arguments)\n_keyword_nodes = _filter_nodes(ast.keyword)\n_alias_nodes = _filter_nodes(ast.alias)\n\n\n# nodes that we don't support directly but are needed for parsing\n_hacked_nodes = frozenset([\"Assign\", \"Module\", \"Expr\"])\n\n\n_unsupported_expr_nodes = frozenset(\n [\n \"Yield\",\n \"GeneratorExp\",\n \"IfExp\",\n \"DictComp\",\n \"SetComp\",\n \"Repr\",\n \"Lambda\",\n \"Set\",\n \"AST\",\n \"Is\",\n \"IsNot\",\n ]\n)\n\n# these nodes are low priority or won't ever be supported (e.g., AST)\n_unsupported_nodes = (\n _stmt_nodes\n | _mod_nodes\n | _handler_nodes\n | _arguments_nodes\n | _keyword_nodes\n | _alias_nodes\n | _expr_context_nodes\n | _unsupported_expr_nodes\n) - _hacked_nodes\n\n# we're adding a different assignment in some cases to be equality comparison\n# and we don't want `stmt` and friends in their so get only the class whose\n# names are capitalized\n_base_supported_nodes = (_all_node_names - _unsupported_nodes) | _hacked_nodes\nintersection = _unsupported_nodes & _base_supported_nodes\n_msg = f\"cannot both support and not support {intersection}\"\nassert not intersection, _msg\n\n\ndef _node_not_implemented(node_name: str) -> Callable[..., None]:\n \"\"\"\n Return a function that raises a NotImplementedError with a passed node name.\n \"\"\"\n\n def f(self, *args, **kwargs):\n raise NotImplementedError(f\"'{node_name}' nodes are not implemented\")\n\n return f\n\n\n_T = TypeVar(\"_T\")\n\n\ndef disallow(nodes: set[str]) -> Callable[[type[_T]], type[_T]]:\n \"\"\"\n Decorator to disallow certain nodes from parsing. Raises a\n NotImplementedError instead.\n\n Returns\n -------\n callable\n \"\"\"\n\n def disallowed(cls: type[_T]) -> type[_T]:\n # error: \"Type[_T]\" has no attribute \"unsupported_nodes\"\n cls.unsupported_nodes = () # type: ignore[attr-defined]\n for node in nodes:\n new_method = _node_not_implemented(node)\n name = f\"visit_{node}\"\n # error: \"Type[_T]\" has no attribute \"unsupported_nodes\"\n cls.unsupported_nodes += (name,) # type: ignore[attr-defined]\n setattr(cls, name, new_method)\n return cls\n\n return disallowed\n\n\ndef _op_maker(op_class, op_symbol):\n \"\"\"\n Return a function to create an op class with its symbol already passed.\n\n Returns\n -------\n callable\n \"\"\"\n\n def f(self, node, *args, **kwargs):\n \"\"\"\n Return a partial function with an Op subclass with an operator already passed.\n\n Returns\n -------\n callable\n \"\"\"\n return partial(op_class, op_symbol, *args, **kwargs)\n\n return f\n\n\n_op_classes = {\"binary\": BinOp, \"unary\": UnaryOp}\n\n\ndef add_ops(op_classes):\n \"\"\"\n Decorator to add default implementation of ops.\n \"\"\"\n\n def f(cls):\n for op_attr_name, op_class in op_classes.items():\n ops = getattr(cls, f\"{op_attr_name}_ops\")\n ops_map = getattr(cls, f\"{op_attr_name}_op_nodes_map\")\n for op in ops:\n op_node = ops_map[op]\n if op_node is not None:\n made_op = _op_maker(op_class, op)\n setattr(cls, f\"visit_{op_node}\", made_op)\n return cls\n\n return f\n\n\n@disallow(_unsupported_nodes)\n@add_ops(_op_classes)\nclass BaseExprVisitor(ast.NodeVisitor):\n \"\"\"\n Custom ast walker. Parsers of other engines should subclass this class\n if necessary.\n\n Parameters\n ----------\n env : Scope\n engine : str\n parser : str\n preparser : callable\n \"\"\"\n\n const_type: ClassVar[type[Term]] = Constant\n term_type: ClassVar[type[Term]] = Term\n\n binary_ops = CMP_OPS_SYMS + BOOL_OPS_SYMS + ARITH_OPS_SYMS\n binary_op_nodes = (\n \"Gt\",\n \"Lt\",\n \"GtE\",\n \"LtE\",\n \"Eq\",\n \"NotEq\",\n \"In\",\n \"NotIn\",\n \"BitAnd\",\n \"BitOr\",\n \"And\",\n \"Or\",\n \"Add\",\n \"Sub\",\n \"Mult\",\n \"Div\",\n \"Pow\",\n \"FloorDiv\",\n \"Mod\",\n )\n binary_op_nodes_map = dict(zip(binary_ops, binary_op_nodes))\n\n unary_ops = UNARY_OPS_SYMS\n unary_op_nodes = \"UAdd\", \"USub\", \"Invert\", \"Not\"\n unary_op_nodes_map = dict(zip(unary_ops, unary_op_nodes))\n\n rewrite_map = {\n ast.Eq: ast.In,\n ast.NotEq: ast.NotIn,\n ast.In: ast.In,\n ast.NotIn: ast.NotIn,\n }\n\n unsupported_nodes: tuple[str, ...]\n\n def __init__(self, env, engine, parser, preparser=_preparse) -> None:\n self.env = env\n self.engine = engine\n self.parser = parser\n self.preparser = preparser\n self.assigner = None\n\n def visit(self, node, **kwargs):\n if isinstance(node, str):\n clean = self.preparser(node)\n try:\n node = ast.fix_missing_locations(ast.parse(clean))\n except SyntaxError as e:\n if any(iskeyword(x) for x in clean.split()):\n e.msg = \"Python keyword not valid identifier in numexpr query\"\n raise e\n\n method = f\"visit_{type(node).__name__}\"\n visitor = getattr(self, method)\n return visitor(node, **kwargs)\n\n def visit_Module(self, node, **kwargs):\n if len(node.body) != 1:\n raise SyntaxError(\"only a single expression is allowed\")\n expr = node.body[0]\n return self.visit(expr, **kwargs)\n\n def visit_Expr(self, node, **kwargs):\n return self.visit(node.value, **kwargs)\n\n def _rewrite_membership_op(self, node, left, right):\n # the kind of the operator (is actually an instance)\n op_instance = node.op\n op_type = type(op_instance)\n\n # must be two terms and the comparison operator must be ==/!=/in/not in\n if is_term(left) and is_term(right) and op_type in self.rewrite_map:\n left_list, right_list = map(_is_list, (left, right))\n left_str, right_str = map(_is_str, (left, right))\n\n # if there are any strings or lists in the expression\n if left_list or right_list or left_str or right_str:\n op_instance = self.rewrite_map[op_type]()\n\n # pop the string variable out of locals and replace it with a list\n # of one string, kind of a hack\n if right_str:\n name = self.env.add_tmp([right.value])\n right = self.term_type(name, self.env)\n\n if left_str:\n name = self.env.add_tmp([left.value])\n left = self.term_type(name, self.env)\n\n op = self.visit(op_instance)\n return op, op_instance, left, right\n\n def _maybe_transform_eq_ne(self, node, left=None, right=None):\n if left is None:\n left = self.visit(node.left, side=\"left\")\n if right is None:\n right = self.visit(node.right, side=\"right\")\n op, op_class, left, right = self._rewrite_membership_op(node, left, right)\n return op, op_class, left, right\n\n def _maybe_downcast_constants(self, left, right):\n f32 = np.dtype(np.float32)\n if (\n left.is_scalar\n and hasattr(left, \"value\")\n and not right.is_scalar\n and right.return_type == f32\n ):\n # right is a float32 array, left is a scalar\n name = self.env.add_tmp(np.float32(left.value))\n left = self.term_type(name, self.env)\n if (\n right.is_scalar\n and hasattr(right, \"value\")\n and not left.is_scalar\n and left.return_type == f32\n ):\n # left is a float32 array, right is a scalar\n name = self.env.add_tmp(np.float32(right.value))\n right = self.term_type(name, self.env)\n\n return left, right\n\n def _maybe_eval(self, binop, eval_in_python):\n # eval `in` and `not in` (for now) in \"partial\" python space\n # things that can be evaluated in \"eval\" space will be turned into\n # temporary variables. for example,\n # [1,2] in a + 2 * b\n # in that case a + 2 * b will be evaluated using numexpr, and the \"in\"\n # call will be evaluated using isin (in python space)\n return binop.evaluate(\n self.env, self.engine, self.parser, self.term_type, eval_in_python\n )\n\n def _maybe_evaluate_binop(\n self,\n op,\n op_class,\n lhs,\n rhs,\n eval_in_python=(\"in\", \"not in\"),\n maybe_eval_in_python=(\"==\", \"!=\", \"<\", \">\", \"<=\", \">=\"),\n ):\n res = op(lhs, rhs)\n\n if res.has_invalid_return_type:\n raise TypeError(\n f\"unsupported operand type(s) for {res.op}: \"\n + f\"'{lhs.type}' and '{rhs.type}'\"\n )\n\n if self.engine != \"pytables\" and (\n res.op in CMP_OPS_SYMS\n and getattr(lhs, \"is_datetime\", False)\n or getattr(rhs, \"is_datetime\", False)\n ):\n # all date ops must be done in python bc numexpr doesn't work\n # well with NaT\n return self._maybe_eval(res, self.binary_ops)\n\n if res.op in eval_in_python:\n # \"in\"/\"not in\" ops are always evaluated in python\n return self._maybe_eval(res, eval_in_python)\n elif self.engine != \"pytables\":\n if (\n getattr(lhs, \"return_type\", None) == object\n or getattr(rhs, \"return_type\", None) == object\n ):\n # evaluate \"==\" and \"!=\" in python if either of our operands\n # has an object return type\n return self._maybe_eval(res, eval_in_python + maybe_eval_in_python)\n return res\n\n def visit_BinOp(self, node, **kwargs):\n op, op_class, left, right = self._maybe_transform_eq_ne(node)\n left, right = self._maybe_downcast_constants(left, right)\n return self._maybe_evaluate_binop(op, op_class, left, right)\n\n def visit_UnaryOp(self, node, **kwargs):\n op = self.visit(node.op)\n operand = self.visit(node.operand)\n return op(operand)\n\n def visit_Name(self, node, **kwargs) -> Term:\n return self.term_type(node.id, self.env, **kwargs)\n\n # TODO(py314): deprecated since Python 3.8. Remove after Python 3.14 is min\n def visit_NameConstant(self, node, **kwargs) -> Term:\n return self.const_type(node.value, self.env)\n\n # TODO(py314): deprecated since Python 3.8. Remove after Python 3.14 is min\n def visit_Num(self, node, **kwargs) -> Term:\n return self.const_type(node.value, self.env)\n\n def visit_Constant(self, node, **kwargs) -> Term:\n return self.const_type(node.value, self.env)\n\n # TODO(py314): deprecated since Python 3.8. Remove after Python 3.14 is min\n def visit_Str(self, node, **kwargs) -> Term:\n name = self.env.add_tmp(node.s)\n return self.term_type(name, self.env)\n\n def visit_List(self, node, **kwargs) -> Term:\n name = self.env.add_tmp([self.visit(e)(self.env) for e in node.elts])\n return self.term_type(name, self.env)\n\n visit_Tuple = visit_List\n\n def visit_Index(self, node, **kwargs):\n \"\"\"df.index[4]\"\"\"\n return self.visit(node.value)\n\n def visit_Subscript(self, node, **kwargs) -> Term:\n from modin.core.computation.eval import eval as pd_eval\n\n value = self.visit(node.value)\n slobj = self.visit(node.slice)\n result = pd_eval(\n slobj, local_dict=self.env, engine=self.engine, parser=self.parser\n )\n try:\n # a Term instance\n v = value.value[result]\n except AttributeError:\n # an Op instance\n lhs = pd_eval(\n value, local_dict=self.env, engine=self.engine, parser=self.parser\n )\n v = lhs[result]\n name = self.env.add_tmp(v)\n return self.term_type(name, env=self.env)\n\n def visit_Slice(self, node, **kwargs) -> slice:\n \"\"\"df.index[slice(4,6)]\"\"\"\n lower = node.lower\n if lower is not None:\n lower = self.visit(lower).value\n upper = node.upper\n if upper is not None:\n upper = self.visit(upper).value\n step = node.step\n if step is not None:\n step = self.visit(step).value\n\n return slice(lower, upper, step)\n\n def visit_Assign(self, node, **kwargs):\n \"\"\"\n support a single assignment node, like\n\n c = a + b\n\n set the assigner at the top level, must be a Name node which\n might or might not exist in the resolvers\n\n \"\"\"\n if len(node.targets) != 1:\n raise SyntaxError(\"can only assign a single expression\")\n if not isinstance(node.targets[0], ast.Name):\n raise SyntaxError(\"left hand side of an assignment must be a single name\")\n if self.env.target is None:\n raise ValueError(\"cannot assign without a target object\")\n\n try:\n assigner = self.visit(node.targets[0], **kwargs)\n except UndefinedVariableError:\n assigner = node.targets[0].id\n\n self.assigner = getattr(assigner, \"name\", assigner)\n if self.assigner is None:\n raise SyntaxError(\n \"left hand side of an assignment must be a single resolvable name\"\n )\n\n return self.visit(node.value, **kwargs)\n\n def visit_Attribute(self, node, **kwargs):\n attr = node.attr\n value = node.value\n\n ctx = node.ctx\n if isinstance(ctx, ast.Load):\n # resolve the value\n resolved = self.visit(value).value\n try:\n v = getattr(resolved, attr)\n name = self.env.add_tmp(v)\n return self.term_type(name, self.env)\n except AttributeError:\n # something like datetime.datetime where scope is overridden\n if isinstance(value, ast.Name) and value.id == attr:\n return resolved\n raise\n\n raise ValueError(f\"Invalid Attribute context {type(ctx).__name__}\")\n\n def visit_Call(self, node, side=None, **kwargs):\n if isinstance(node.func, ast.Attribute) and node.func.attr != \"__call__\":\n res = self.visit_Attribute(node.func)\n elif not isinstance(node.func, ast.Name):\n raise TypeError(\"Only named functions are supported\")\n else:\n try:\n res = self.visit(node.func)\n except UndefinedVariableError:\n # Check if this is a supported function name\n try:\n res = FuncNode(node.func.id)\n except ValueError:\n # Raise original error\n raise\n\n if res is None:\n # error: \"expr\" has no attribute \"id\"\n raise ValueError(\n f\"Invalid function call {node.func.id}\" # type: ignore[attr-defined]\n )\n if hasattr(res, \"value\"):\n res = res.value\n\n if isinstance(res, FuncNode):\n new_args = [self.visit(arg) for arg in node.args]\n\n if node.keywords:\n raise TypeError(\n f'Function \"{res.name}\" does not support keyword arguments'\n )\n\n return res(*new_args)\n\n else:\n new_args = [self.visit(arg)(self.env) for arg in node.args]\n\n for key in node.keywords:\n if not isinstance(key, ast.keyword):\n # error: \"expr\" has no attribute \"id\"\n raise ValueError(\n \"keyword error in function call \"\n + f\"'{node.func.id}'\" # type: ignore[attr-defined]\n )\n\n if key.arg:\n kwargs[key.arg] = self.visit(key.value)(self.env)\n\n name = self.env.add_tmp(res(*new_args, **kwargs))\n return self.term_type(name=name, env=self.env)\n\n def translate_In(self, op):\n return op\n\n def visit_Compare(self, node, **kwargs):\n ops = node.ops\n comps = node.comparators\n\n # base case: we have something like a CMP b\n if len(comps) == 1:\n op = self.translate_In(ops[0])\n binop = ast.BinOp(op=op, left=node.left, right=comps[0])\n return self.visit(binop)\n\n # recursive case: we have a chained comparison, a CMP b CMP c, etc.\n left = node.left\n values = []\n for op, comp in zip(ops, comps):\n new_node = self.visit(\n ast.Compare(comparators=[comp], left=left, ops=[self.translate_In(op)])\n )\n left = comp\n values.append(new_node)\n return self.visit(ast.BoolOp(op=ast.And(), values=values))\n\n def _try_visit_binop(self, bop):\n if isinstance(bop, (Op, Term)):\n return bop\n return self.visit(bop)\n\n def visit_BoolOp(self, node, **kwargs):\n def visitor(x, y):\n lhs = self._try_visit_binop(x)\n rhs = self._try_visit_binop(y)\n\n op, op_class, lhs, rhs = self._maybe_transform_eq_ne(node, lhs, rhs)\n return self._maybe_evaluate_binop(op, node.op, lhs, rhs)\n\n operands = node.values\n return reduce(visitor, operands)\n\n\n_python_not_supported = frozenset([\"Dict\", \"BoolOp\", \"In\", \"NotIn\"])\n\n\n@disallow(\n (_unsupported_nodes | _python_not_supported)\n - (_boolop_nodes | frozenset([\"BoolOp\", \"Attribute\", \"In\", \"NotIn\", \"Tuple\"]))\n)\nclass PandasExprVisitor(BaseExprVisitor):\n def __init__(\n self,\n env,\n engine,\n parser,\n preparser=partial(\n _preparse,\n f=_compose(_replace_locals, _replace_booleans, clean_backtick_quoted_toks),\n ),\n ) -> None:\n super().__init__(env, engine, parser, preparser)\n\n\n@disallow(_unsupported_nodes | _python_not_supported | frozenset([\"Not\"]))\nclass PythonExprVisitor(BaseExprVisitor):\n def __init__(\n self, env, engine, parser, preparser=lambda source, f=None: source\n ) -> None:\n super().__init__(env, engine, parser, preparser=preparser)\n\n\nclass Expr:\n \"\"\"\n Object encapsulating an expression.\n\n Parameters\n ----------\n expr : str\n engine : str, optional, default 'numexpr'\n parser : str, optional, default 'pandas'\n env : Scope, optional, default None\n level : int, optional, default 2\n \"\"\"\n\n env: Scope\n engine: str\n parser: str\n\n def __init__(\n self,\n expr,\n engine: str = \"numexpr\",\n parser: str = \"pandas\",\n env: Scope | None = None,\n level: int = 0,\n ) -> None:\n self.expr = expr\n self.env = env or Scope(level=level + 1)\n self.engine = engine\n self.parser = parser\n self._visitor = PARSERS[parser](self.env, self.engine, self.parser)\n self.terms = self.parse()\n\n @property\n def assigner(self):\n return getattr(self._visitor, \"assigner\", None)\n\n def __call__(self):\n return self.terms(self.env)\n\n def __repr__(self) -> str:\n return printing.pprint_thing(self.terms)\n\n def __len__(self) -> int:\n return len(self.expr)\n\n def parse(self):\n \"\"\"\n Parse an expression.\n \"\"\"\n return self._visitor.visit(self.expr)\n\n @property\n def names(self):\n \"\"\"\n Get the names in an expression.\n \"\"\"\n if is_term(self.terms):\n return frozenset([self.terms.name])\n return frozenset(term.name for term in com.flatten(self.terms))\n\n\nPARSERS = {\"python\": PythonExprVisitor, \"pandas\": PandasExprVisitor}\n"
},
{
"path": "modin/core/computation/ops.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nOperator classes for eval. Forked from pandas.core.computation.ops\n\"\"\"\n\nfrom __future__ import annotations\n\nimport operator\nfrom datetime import datetime\nfrom functools import partial\nfrom typing import (\n TYPE_CHECKING,\n Callable,\n Literal,\n)\n\nimport numpy as np\nimport pandas\nimport pandas.core.common as com\nfrom pandas.core.dtypes.cast import find_common_type\nfrom pandas.core.dtypes.common import (\n is_list_like,\n is_scalar,\n)\nfrom pandas.io.formats.printing import (\n pprint_thing,\n pprint_thing_encoded,\n)\n\nfrom modin.core.computation.common import (\n ensure_decoded,\n result_type_many,\n)\nfrom modin.core.computation.scope import DEFAULT_GLOBALS\n\nif TYPE_CHECKING:\n from collections.abc import (\n Iterable,\n Iterator,\n )\n\nREDUCTIONS = (\"sum\", \"prod\", \"min\", \"max\")\n\n_unary_math_ops = (\n \"sin\",\n \"cos\",\n \"exp\",\n \"log\",\n \"expm1\",\n \"log1p\",\n \"sqrt\",\n \"sinh\",\n \"cosh\",\n \"tanh\",\n \"arcsin\",\n \"arccos\",\n \"arctan\",\n \"arccosh\",\n \"arcsinh\",\n \"arctanh\",\n \"abs\",\n \"log10\",\n \"floor\",\n \"ceil\",\n)\n_binary_math_ops = (\"arctan2\",)\n\nMATHOPS = _unary_math_ops + _binary_math_ops\n\n\nLOCAL_TAG = \"__pd_eval_local_\"\n\n\nclass Term:\n def __new__(cls, name, env, side=None, encoding=None):\n klass = Constant if not isinstance(name, str) else cls\n # error: Argument 2 for \"super\" not an instance of argument 1\n supr_new = super(Term, klass).__new__ # type: ignore[misc]\n return supr_new(klass)\n\n is_local: bool\n\n def __init__(self, name, env, side=None, encoding=None) -> None:\n # name is a str for Term, but may be something else for subclasses\n self._name = name\n self.env = env\n self.side = side\n tname = str(name)\n self.is_local = tname.startswith(LOCAL_TAG) or tname in DEFAULT_GLOBALS\n self._value = self._resolve_name()\n self.encoding = encoding\n\n @property\n def local_name(self) -> str:\n return self.name.replace(LOCAL_TAG, \"\")\n\n def __repr__(self) -> str:\n return pprint_thing(self.name)\n\n def __call__(self, *args, **kwargs):\n return self.value\n\n def evaluate(self, *args, **kwargs) -> Term:\n return self\n\n def _resolve_name(self):\n local_name = str(self.local_name)\n is_local = self.is_local\n if local_name in self.env.scope and isinstance(\n self.env.scope[local_name], type\n ):\n is_local = False\n\n res = self.env.resolve(local_name, is_local=is_local)\n self.update(res)\n\n if hasattr(res, \"ndim\") and res.ndim > 2:\n raise NotImplementedError(\n \"N-dimensional objects, where N > 2, are not supported with eval\"\n )\n return res\n\n def update(self, value) -> None:\n \"\"\"\n search order for local (i.e., @variable) variables:\n\n scope, key_variable\n [('locals', 'local_name'),\n ('globals', 'local_name'),\n ('locals', 'key'),\n ('globals', 'key')]\n \"\"\"\n key = self.name\n\n # if it's a variable name (otherwise a constant)\n if isinstance(key, str):\n self.env.swapkey(self.local_name, key, new_value=value)\n\n self.value = value\n\n @property\n def is_scalar(self) -> bool:\n return is_scalar(self._value)\n\n @property\n def type(self):\n try:\n # potentially very slow for large, mixed dtype frames\n return find_common_type(self._value.dtypes.values)\n except AttributeError:\n try:\n # ndarray\n return self._value.dtype\n except AttributeError:\n # scalar\n return type(self._value)\n\n return_type = type\n\n @property\n def raw(self) -> str:\n return f\"{type(self).__name__}(name={repr(self.name)}, type={self.type})\"\n\n @property\n def is_datetime(self) -> bool:\n try:\n t = self.type.type\n except AttributeError:\n t = self.type\n\n return issubclass(t, (datetime, np.datetime64))\n\n @property\n def value(self):\n return self._value\n\n @value.setter\n def value(self, new_value) -> None:\n self._value = new_value\n\n @property\n def name(self):\n return self._name\n\n @property\n def ndim(self) -> int:\n return self._value.ndim\n\n\nclass Constant(Term):\n def _resolve_name(self):\n return self._name\n\n @property\n def name(self):\n return self.value\n\n def __repr__(self) -> str:\n # in python 2 str() of float\n # can truncate shorter than repr()\n return repr(self.name)\n\n\n_bool_op_map = {\"not\": \"~\", \"and\": \"&\", \"or\": \"|\"}\n\n\nclass Op:\n \"\"\"\n Hold an operator of arbitrary arity.\n \"\"\"\n\n op: str\n\n def __init__(self, op: str, operands: Iterable[Term | Op], encoding=None) -> None:\n self.op = _bool_op_map.get(op, op)\n self.operands = operands\n self.encoding = encoding\n\n def __iter__(self) -> Iterator:\n return iter(self.operands)\n\n def __repr__(self) -> str:\n \"\"\"\n Print a generic n-ary operator and its operands using infix notation.\n \"\"\"\n # recurse over the operands\n parened = (f\"({pprint_thing(opr)})\" for opr in self.operands)\n return pprint_thing(f\" {self.op} \".join(parened))\n\n @property\n def return_type(self):\n # clobber types to bool if the op is a boolean operator\n if self.op in (CMP_OPS_SYMS + BOOL_OPS_SYMS):\n return np.bool_\n return result_type_many(*(term.type for term in com.flatten(self)))\n\n @property\n def has_invalid_return_type(self) -> bool:\n types = self.operand_types\n obj_dtype_set = frozenset([np.dtype(\"object\")])\n return self.return_type == object and types - obj_dtype_set\n\n @property\n def operand_types(self):\n return frozenset(term.type for term in com.flatten(self))\n\n @property\n def is_scalar(self) -> bool:\n return all(operand.is_scalar for operand in self.operands)\n\n @property\n def is_datetime(self) -> bool:\n try:\n t = self.return_type.type\n except AttributeError:\n t = self.return_type\n\n return issubclass(t, (datetime, np.datetime64))\n\n\ndef _in(x, y):\n \"\"\"\n Compute the vectorized membership of ``x in y`` if possible, otherwise\n use Python.\n \"\"\"\n try:\n return x.isin(y)\n except AttributeError:\n if is_list_like(x):\n try:\n return y.isin(x)\n except AttributeError:\n pass\n return x in y\n\n\ndef _not_in(x, y):\n \"\"\"\n Compute the vectorized membership of ``x not in y`` if possible,\n otherwise use Python.\n \"\"\"\n try:\n return ~x.isin(y)\n except AttributeError:\n if is_list_like(x):\n try:\n return ~y.isin(x)\n except AttributeError:\n pass\n return x not in y\n\n\nCMP_OPS_SYMS = (\">\", \"<\", \">=\", \"<=\", \"==\", \"!=\", \"in\", \"not in\")\n_cmp_ops_funcs = (\n operator.gt,\n operator.lt,\n operator.ge,\n operator.le,\n operator.eq,\n operator.ne,\n _in,\n _not_in,\n)\n_cmp_ops_dict = dict(zip(CMP_OPS_SYMS, _cmp_ops_funcs))\n\nBOOL_OPS_SYMS = (\"&\", \"|\", \"and\", \"or\")\n_bool_ops_funcs = (operator.and_, operator.or_, operator.and_, operator.or_)\n_bool_ops_dict = dict(zip(BOOL_OPS_SYMS, _bool_ops_funcs))\n\nARITH_OPS_SYMS = (\"+\", \"-\", \"*\", \"/\", \"**\", \"//\", \"%\")\n_arith_ops_funcs = (\n operator.add,\n operator.sub,\n operator.mul,\n operator.truediv,\n operator.pow,\n operator.floordiv,\n operator.mod,\n)\n_arith_ops_dict = dict(zip(ARITH_OPS_SYMS, _arith_ops_funcs))\n\nSPECIAL_CASE_ARITH_OPS_SYMS = (\"**\", \"//\", \"%\")\n\n\n_binary_ops_dict = {}\n\nfor d in (_cmp_ops_dict, _bool_ops_dict, _arith_ops_dict):\n _binary_ops_dict.update(d)\n\n\ndef is_term(obj) -> bool:\n return isinstance(obj, Term)\n\n\nclass BinOp(Op):\n \"\"\"\n Hold a binary operator and its operands.\n\n Parameters\n ----------\n op : str\n lhs : Term or Op\n rhs : Term or Op\n \"\"\"\n\n def __init__(self, op: str, lhs, rhs) -> None:\n super().__init__(op, (lhs, rhs))\n self.lhs = lhs\n self.rhs = rhs\n\n self._disallow_scalar_only_bool_ops()\n\n self.convert_values()\n\n try:\n self.func = _binary_ops_dict[op]\n except KeyError as err:\n # has to be made a list for python3\n keys = list(_binary_ops_dict.keys())\n raise ValueError(\n f\"Invalid binary operator {repr(op)}, valid operators are {keys}\"\n ) from err\n\n def __call__(self, env):\n \"\"\"\n Recursively evaluate an expression in Python space.\n\n Parameters\n ----------\n env : Scope\n\n Returns\n -------\n object\n The result of an evaluated expression.\n \"\"\"\n # recurse over the left/right nodes\n left = self.lhs(env)\n right = self.rhs(env)\n\n return self.func(left, right)\n\n def evaluate(self, env, engine: str, parser, term_type, eval_in_python):\n \"\"\"\n Evaluate a binary operation *before* being passed to the engine.\n\n Parameters\n ----------\n env : Scope\n engine : str\n parser : str\n term_type : type\n eval_in_python : list\n\n Returns\n -------\n term_type\n The \"pre-evaluated\" expression as an instance of ``term_type``\n \"\"\"\n if engine == \"python\":\n res = self(env)\n else:\n # recurse over the left/right nodes\n\n left = self.lhs.evaluate(\n env,\n engine=engine,\n parser=parser,\n term_type=term_type,\n eval_in_python=eval_in_python,\n )\n\n right = self.rhs.evaluate(\n env,\n engine=engine,\n parser=parser,\n term_type=term_type,\n eval_in_python=eval_in_python,\n )\n\n # base cases\n if self.op in eval_in_python:\n res = self.func(left.value, right.value)\n else:\n from modin.core.computation.eval import eval\n\n res = eval(self, local_dict=env, engine=engine, parser=parser)\n\n name = env.add_tmp(res)\n return term_type(name, env=env)\n\n def convert_values(self) -> None:\n \"\"\"\n Convert datetimes to a comparable value in an expression.\n \"\"\"\n\n def stringify(value):\n encoder: Callable\n if self.encoding is not None:\n encoder = partial(pprint_thing_encoded, encoding=self.encoding)\n else:\n encoder = pprint_thing\n return encoder(value)\n\n lhs, rhs = self.lhs, self.rhs\n\n if is_term(lhs) and lhs.is_datetime and is_term(rhs) and rhs.is_scalar:\n v = rhs.value\n if isinstance(v, (int, float)):\n v = stringify(v)\n v = pandas.Timestamp(ensure_decoded(v))\n if v.tz is not None:\n v = v.tz_convert(\"UTC\")\n self.rhs.update(v)\n\n if is_term(rhs) and rhs.is_datetime and is_term(lhs) and lhs.is_scalar:\n v = lhs.value\n if isinstance(v, (int, float)):\n v = stringify(v)\n v = pandas.Timestamp(ensure_decoded(v))\n if v.tz is not None:\n v = v.tz_convert(\"UTC\")\n self.lhs.update(v)\n\n def _disallow_scalar_only_bool_ops(self):\n rhs = self.rhs\n lhs = self.lhs\n\n # GH#24883 unwrap dtype if necessary to ensure we have a type object\n rhs_rt = rhs.return_type\n rhs_rt = getattr(rhs_rt, \"type\", rhs_rt)\n lhs_rt = lhs.return_type\n lhs_rt = getattr(lhs_rt, \"type\", lhs_rt)\n if (\n (lhs.is_scalar or rhs.is_scalar)\n and self.op in _bool_ops_dict\n and (\n not (\n issubclass(rhs_rt, (bool, np.bool_))\n and issubclass(lhs_rt, (bool, np.bool_))\n )\n )\n ):\n raise NotImplementedError(\"cannot evaluate scalar only bool ops\")\n\n\ndef isnumeric(dtype) -> bool:\n return issubclass(np.dtype(dtype).type, np.number)\n\n\nUNARY_OPS_SYMS = (\"+\", \"-\", \"~\", \"not\")\n_unary_ops_funcs = (operator.pos, operator.neg, operator.invert, operator.invert)\n_unary_ops_dict = dict(zip(UNARY_OPS_SYMS, _unary_ops_funcs))\n\n\nclass UnaryOp(Op):\n \"\"\"\n Hold a unary operator and its operands.\n\n Parameters\n ----------\n op : str\n The token used to represent the operator.\n operand : Term or Op\n The Term or Op operand to the operator.\n\n Raises\n ------\n ValueError\n * If no function associated with the passed operator token is found.\n \"\"\"\n\n def __init__(self, op: Literal[\"+\", \"-\", \"~\", \"not\"], operand) -> None:\n super().__init__(op, (operand,))\n self.operand = operand\n\n try:\n self.func = _unary_ops_dict[op]\n except KeyError as err:\n raise ValueError(\n f\"Invalid unary operator {repr(op)}, valid operators are {UNARY_OPS_SYMS}\"\n ) from err\n\n def __call__(self, env) -> MathCall:\n operand = self.operand(env)\n # error: Cannot call function of unknown type\n return self.func(operand) # type: ignore[operator]\n\n def __repr__(self) -> str:\n return pprint_thing(f\"{self.op}({self.operand})\")\n\n @property\n def return_type(self) -> np.dtype:\n operand = self.operand\n if operand.return_type == np.dtype(\"bool\"):\n return np.dtype(\"bool\")\n if isinstance(operand, Op) and (\n operand.op in _cmp_ops_dict or operand.op in _bool_ops_dict\n ):\n return np.dtype(\"bool\")\n return np.dtype(\"int\")\n\n\nclass MathCall(Op):\n def __init__(self, func, args) -> None:\n super().__init__(func.name, args)\n self.func = func\n\n def __call__(self, env):\n # error: \"Op\" not callable\n operands = [op(env) for op in self.operands] # type: ignore[operator]\n return self.func.func(*operands)\n\n def __repr__(self) -> str:\n operands = map(str, self.operands)\n return pprint_thing(f\"{self.op}({','.join(operands)})\")\n\n\nclass FuncNode:\n def __init__(self, name: str) -> None:\n if name not in MATHOPS:\n raise ValueError(f'\"{name}\" is not a supported function')\n self.name = name\n self.func = getattr(np, name)\n\n def __call__(self, *args) -> MathCall:\n return MathCall(self, args)\n"
},
{
"path": "modin/core/computation/parsing.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\n:func:`~pandas.eval` source string parsing functions. Forked from pandas.core.computation.parsing\n\"\"\"\n\nfrom __future__ import annotations\n\nimport token\nimport tokenize\nfrom io import StringIO\nfrom keyword import iskeyword\nfrom typing import TYPE_CHECKING\n\nif TYPE_CHECKING:\n from collections.abc import (\n Hashable,\n Iterator,\n )\n\n# A token value Python's tokenizer probably will never use.\nBACKTICK_QUOTED_STRING = 100\n\n\ndef create_valid_python_identifier(name: str) -> str:\n \"\"\"\n Create valid Python identifiers from any string.\n\n Check if name contains any special characters. If it contains any\n special characters, the special characters will be replaced by\n a special string and a prefix is added.\n\n Raises\n ------\n SyntaxError\n If the returned name is not a Python valid identifier, raise an exception.\n This can happen if there is a hashtag in the name, as the tokenizer will\n than terminate and not find the backtick.\n But also for characters that fall out of the range of (U+0001..U+007F).\n \"\"\"\n if name.isidentifier() and not iskeyword(name):\n return name\n\n # Create a dict with the special characters and their replacement string.\n # EXACT_TOKEN_TYPES contains these special characters\n # token.tok_name contains a readable description of the replacement string.\n special_characters_replacements = {\n char: f\"_{token.tok_name[tokval]}_\"\n for char, tokval in (tokenize.EXACT_TOKEN_TYPES.items())\n }\n special_characters_replacements.update(\n {\n \" \": \"_\",\n \"?\": \"_QUESTIONMARK_\",\n \"!\": \"_EXCLAMATIONMARK_\",\n \"$\": \"_DOLLARSIGN_\",\n \"€\": \"_EUROSIGN_\",\n \"°\": \"_DEGREESIGN_\",\n # Including quotes works, but there are exceptions.\n \"'\": \"_SINGLEQUOTE_\",\n '\"': \"_DOUBLEQUOTE_\",\n # Currently not possible. Terminates parser and won't find backtick.\n # \"#\": \"_HASH_\",\n }\n )\n\n name = \"\".join([special_characters_replacements.get(char, char) for char in name])\n name = f\"BACKTICK_QUOTED_STRING_{name}\"\n\n if not name.isidentifier():\n raise SyntaxError(f\"Could not convert '{name}' to a valid Python identifier.\")\n\n return name\n\n\ndef clean_backtick_quoted_toks(tok: tuple[int, str]) -> tuple[int, str]:\n \"\"\"\n Clean up a column name if surrounded by backticks.\n\n Backtick quoted string are indicated by a certain tokval value. If a string\n is a backtick quoted token it will processed by\n :func:`_create_valid_python_identifier` so that the parser can find this\n string when the query is executed.\n In this case the tok will get the NAME tokval.\n\n Parameters\n ----------\n tok : tuple of int, str\n ints correspond to the all caps constants in the tokenize module\n\n Returns\n -------\n tok : Tuple[int, str]\n Either the input or token or the replacement values\n \"\"\"\n toknum, tokval = tok\n if toknum == BACKTICK_QUOTED_STRING:\n return tokenize.NAME, create_valid_python_identifier(tokval)\n return toknum, tokval\n\n\ndef clean_column_name(name: Hashable) -> Hashable:\n \"\"\"\n Function to emulate the cleaning of a backtick quoted name.\n\n The purpose for this function is to see what happens to the name of\n identifier if it goes to the process of being parsed a Python code\n inside a backtick quoted string and than being cleaned\n (removed of any special characters).\n\n Parameters\n ----------\n name : hashable\n Name to be cleaned.\n\n Returns\n -------\n name : hashable\n Returns the name after tokenizing and cleaning.\n\n Notes\n -----\n For some cases, a name cannot be converted to a valid Python identifier.\n In that case :func:`tokenize_string` raises a SyntaxError.\n In that case, we just return the name unmodified.\n\n If this name was used in the query string (this makes the query call impossible)\n an error will be raised by :func:`tokenize_backtick_quoted_string` instead,\n which is not caught and propagates to the user level.\n \"\"\"\n try:\n tokenized = tokenize_string(f\"`{name}`\")\n tokval = next(tokenized)[1]\n return create_valid_python_identifier(tokval)\n except SyntaxError:\n return name\n\n\ndef tokenize_backtick_quoted_string(\n token_generator: Iterator[tokenize.TokenInfo], source: str, string_start: int\n) -> tuple[int, str]:\n \"\"\"\n Creates a token from a backtick quoted string.\n\n Moves the token_generator forwards till right after the next backtick.\n\n Parameters\n ----------\n token_generator : Iterator[tokenize.TokenInfo]\n The generator that yields the tokens of the source string (Tuple[int, str]).\n The generator is at the first token after the backtick (`)\n\n source : str\n The Python source code string.\n\n string_start : int\n This is the start of backtick quoted string inside the source string.\n\n Returns\n -------\n tok: Tuple[int, str]\n The token that represents the backtick quoted string.\n The integer is equal to BACKTICK_QUOTED_STRING (100).\n \"\"\"\n string_end = None\n for _, tokval, start, _, _ in token_generator:\n if tokval == \"`\":\n string_end = start[1]\n break\n\n assert string_end is not None\n return BACKTICK_QUOTED_STRING, source[string_start:string_end]\n\n\ndef tokenize_string(source: str) -> Iterator[tuple[int, str]]:\n \"\"\"\n Tokenize a Python source code string.\n\n Parameters\n ----------\n source : str\n The Python source code string.\n\n Returns\n -------\n tok_generator : Iterator[Tuple[int, str]]\n An iterator yielding all tokens with only toknum and tokval (Tuple[ing, str]).\n \"\"\"\n line_reader = StringIO(source).readline\n token_generator = tokenize.generate_tokens(line_reader)\n\n # Loop over all tokens till a backtick (`) is found.\n # Then, take all tokens till the next backtick to form a backtick quoted string\n for toknum, tokval, start, _, _ in token_generator:\n if tokval == \"`\":\n try:\n yield tokenize_backtick_quoted_string(\n token_generator, source, string_start=start[1] + 1\n )\n except Exception as err:\n raise SyntaxError(f\"Failed to parse backticks in '{source}'.\") from err\n else:\n yield toknum, tokval\n"
},
{
"path": "modin/core/computation/scope.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nModule for scope operations. Forked from pandas.core.computation.scope\n\"\"\"\n\nfrom __future__ import annotations\n\nimport datetime\nimport inspect\nimport itertools\nimport pprint\nimport struct\nimport sys\nfrom collections import ChainMap\nfrom io import StringIO\nfrom typing import TypeVar\n\nimport numpy as np\nimport pandas\nfrom pandas.errors import UndefinedVariableError\n\n_KT = TypeVar(\"_KT\")\n_VT = TypeVar(\"_VT\")\n\n\n# https://docs.python.org/3/library/collections.html#chainmap-examples-and-recipes\nclass DeepChainMap(ChainMap[_KT, _VT]):\n \"\"\"\n Variant of ChainMap that allows direct updates to inner scopes.\n\n Only works when all passed mapping are mutable.\n \"\"\"\n\n def __setitem__(self, key: _KT, value: _VT) -> None:\n for mapping in self.maps:\n if key in mapping:\n mapping[key] = value\n return\n self.maps[0][key] = value\n\n def __delitem__(self, key: _KT) -> None:\n \"\"\"\n Raises\n ------\n KeyError\n If `key` doesn't exist.\n \"\"\"\n for mapping in self.maps:\n if key in mapping:\n del mapping[key]\n return\n raise KeyError(key)\n\n\ndef ensure_scope(\n level: int, global_dict=None, local_dict=None, resolvers=(), target=None\n) -> Scope:\n \"\"\"Ensure that we are grabbing the correct scope.\"\"\"\n return Scope(\n level + 1,\n global_dict=global_dict,\n local_dict=local_dict,\n resolvers=resolvers,\n target=target,\n )\n\n\ndef _replacer(x) -> str:\n \"\"\"\n Replace a number with its hexadecimal representation. Used to tag\n temporary variables with their calling scope's id.\n \"\"\"\n # get the hex repr of the binary char and remove 0x and pad by pad_size\n # zeros\n try:\n hexin = ord(x)\n except TypeError:\n # bytes literals masquerade as ints when iterating in py3\n hexin = x\n\n return hex(hexin)\n\n\ndef _raw_hex_id(obj) -> str:\n \"\"\"Return the padded hexadecimal id of ``obj``.\"\"\"\n # interpret as a pointer since that's what really what id returns\n packed = struct.pack(\"@P\", id(obj))\n return \"\".join([_replacer(x) for x in packed])\n\n\nDEFAULT_GLOBALS = {\n \"Timestamp\": pandas.Timestamp,\n \"datetime\": datetime.datetime,\n \"True\": True,\n \"False\": False,\n \"list\": list,\n \"tuple\": tuple,\n \"inf\": np.inf,\n \"Inf\": np.inf,\n}\n\n\ndef _get_pretty_string(obj) -> str:\n \"\"\"\n Return a prettier version of obj.\n\n Parameters\n ----------\n obj : object\n Object to pretty print\n\n Returns\n -------\n str\n Pretty print object repr\n \"\"\"\n sio = StringIO()\n pprint.pprint(obj, stream=sio) # noqa: T203\n return sio.getvalue()\n\n\nclass Scope:\n \"\"\"\n Object to hold scope, with a few bells to deal with some custom syntax\n and contexts added by pandas.\n\n Parameters\n ----------\n level : int\n global_dict : dict or None, optional, default None\n local_dict : dict or Scope or None, optional, default None\n resolvers : list-like or None, optional, default None\n target : object\n\n Attributes\n ----------\n level : int\n scope : DeepChainMap\n target : object\n temps : dict\n \"\"\"\n\n __slots__ = [\"level\", \"scope\", \"target\", \"resolvers\", \"temps\"]\n level: int\n scope: DeepChainMap\n resolvers: DeepChainMap\n temps: dict\n\n def __init__(\n self, level: int, global_dict=None, local_dict=None, resolvers=(), target=None\n ) -> None:\n self.level = level + 1\n\n # shallow copy because we don't want to keep filling this up with what\n # was there before if there are multiple calls to Scope/_ensure_scope\n self.scope = DeepChainMap(DEFAULT_GLOBALS.copy())\n self.target = target\n\n if isinstance(local_dict, Scope):\n self.scope.update(local_dict.scope)\n if local_dict.target is not None:\n self.target = local_dict.target\n self._update(local_dict.level)\n\n frame = sys._getframe(self.level)\n\n try:\n # shallow copy here because we don't want to replace what's in\n # scope when we align terms (alignment accesses the underlying\n # numpy array of pandas objects)\n scope_global = self.scope.new_child(\n (global_dict if global_dict is not None else frame.f_globals).copy()\n )\n self.scope = DeepChainMap(scope_global)\n if not isinstance(local_dict, Scope):\n scope_local = self.scope.new_child(\n (local_dict if local_dict is not None else frame.f_locals).copy()\n )\n self.scope = DeepChainMap(scope_local)\n finally:\n del frame\n\n # assumes that resolvers are going from outermost scope to inner\n if isinstance(local_dict, Scope):\n resolvers += tuple(local_dict.resolvers.maps)\n self.resolvers = DeepChainMap(*resolvers)\n self.temps = {}\n\n def __repr__(self) -> str:\n scope_keys = _get_pretty_string(list(self.scope.keys()))\n res_keys = _get_pretty_string(list(self.resolvers.keys()))\n return f\"{type(self).__name__}(scope={scope_keys}, resolvers={res_keys})\"\n\n @property\n def has_resolvers(self) -> bool:\n \"\"\"\n Return whether we have any extra scope.\n\n For example, DataFrames pass Their columns as resolvers during calls to\n ``DataFrame.eval()`` and ``DataFrame.query()``.\n\n Returns\n -------\n hr : bool\n \"\"\"\n return bool(len(self.resolvers))\n\n def resolve(self, key: str, is_local: bool):\n \"\"\"\n Resolve a variable name in a possibly local context.\n\n Parameters\n ----------\n key : str\n A variable name\n is_local : bool\n Flag indicating whether the variable is local or not (prefixed with\n the '@' symbol)\n\n Returns\n -------\n value : object\n The value of a particular variable\n \"\"\"\n try:\n # only look for locals in outer scope\n if is_local:\n return self.scope[key]\n\n # not a local variable so check in resolvers if we have them\n if self.has_resolvers:\n return self.resolvers[key]\n\n # if we're here that means that we have no locals and we also have\n # no resolvers\n assert not is_local and not self.has_resolvers\n return self.scope[key]\n except KeyError:\n try:\n # last ditch effort we look in temporaries\n # these are created when parsing indexing expressions\n # e.g., df[df > 0]\n return self.temps[key]\n except KeyError as err:\n raise UndefinedVariableError(key, is_local) from err\n\n def swapkey(self, old_key: str, new_key: str, new_value=None) -> None:\n \"\"\"\n Replace a variable name, with a potentially new value.\n\n Parameters\n ----------\n old_key : str\n Current variable name to replace\n new_key : str\n New variable name to replace `old_key` with\n new_value : object\n Value to be replaced along with the possible renaming\n \"\"\"\n if self.has_resolvers:\n maps = self.resolvers.maps + self.scope.maps\n else:\n maps = self.scope.maps\n\n maps.append(self.temps)\n\n for mapping in maps:\n if old_key in mapping:\n mapping[new_key] = new_value\n return\n\n def _get_vars(self, stack, scopes: list[str]) -> None:\n \"\"\"\n Get specifically scoped variables from a list of stack frames.\n\n Parameters\n ----------\n stack : list\n A list of stack frames as returned by ``inspect.stack()``\n scopes : sequence of strings\n A sequence containing valid stack frame attribute names that\n evaluate to a dictionary. For example, ('locals', 'globals')\n \"\"\"\n variables = itertools.product(scopes, stack)\n for scope, (frame, _, _, _, _, _) in variables:\n try:\n d = getattr(frame, f\"f_{scope}\")\n self.scope = DeepChainMap(self.scope.new_child(d))\n finally:\n # won't remove it, but DECREF it\n # in Py3 this probably isn't necessary since frame won't be\n # scope after the loop\n del frame\n\n def _update(self, level: int) -> None:\n \"\"\"\n Update the current scope by going back `level` levels.\n\n Parameters\n ----------\n level : int\n \"\"\"\n sl = level + 1\n\n # add sl frames to the scope starting with the\n # most distant and overwriting with more current\n # makes sure that we can capture variable scope\n stack = inspect.stack()\n\n try:\n self._get_vars(stack[:sl], scopes=[\"locals\"])\n finally:\n # explcitly delete the stack according to the advice here:\n # https://docs.python.org/3/library/inspect.html#inspect.Traceback\n del stack[:], stack\n\n def add_tmp(self, value) -> str:\n \"\"\"\n Add a temporary variable to the scope.\n\n Parameters\n ----------\n value : object\n An arbitrary object to be assigned to a temporary variable.\n\n Returns\n -------\n str\n The name of the temporary variable created.\n \"\"\"\n name = f\"{type(value).__name__}_{self.ntemps}_{_raw_hex_id(self)}\"\n\n # add to inner most scope\n assert name not in self.temps\n self.temps[name] = value\n assert name in self.temps\n\n # only increment if the variable gets put in the scope\n return name\n\n @property\n def ntemps(self) -> int:\n \"\"\"The number of temporary variables in this scope\"\"\"\n return len(self.temps)\n\n @property\n def full_scope(self) -> DeepChainMap:\n \"\"\"\n Return the full scope for use with passing to engines transparently\n as a mapping.\n\n Returns\n -------\n vars : DeepChainMap\n All variables in this scope.\n \"\"\"\n maps = [self.temps] + self.resolvers.maps + self.scope.maps\n return DeepChainMap(*maps)\n"
},
{
"path": "modin/core/dataframe/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe functionality.\"\"\"\n"
},
{
"path": "modin/core/dataframe/algebra/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin Dataframe algebra (core operators).\"\"\"\n\nfrom .binary import Binary\nfrom .fold import Fold\nfrom .groupby import GroupByReduce\nfrom .map import Map\nfrom .operator import Operator\nfrom .reduce import Reduce\nfrom .tree_reduce import TreeReduce\n\n__all__ = [\n \"Operator\",\n \"Map\",\n \"TreeReduce\",\n \"Reduce\",\n \"Fold\",\n \"Binary\",\n \"GroupByReduce\",\n]\n"
},
{
"path": "modin/core/dataframe/algebra/binary.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses builder class for Binary operator.\"\"\"\n\nfrom __future__ import annotations\n\nimport warnings\nfrom typing import TYPE_CHECKING, Any, Callable, Optional, Union\n\nimport numpy as np\nimport pandas\nfrom pandas.api.types import is_bool_dtype, is_scalar\n\nfrom modin.error_message import ErrorMessage\n\nfrom .operator import Operator\n\nif TYPE_CHECKING:\n from pandas._typing import DtypeObj\n\n from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n\n\ndef maybe_compute_dtypes_common_cast(\n first: PandasQueryCompiler,\n second: Union[PandasQueryCompiler, dict, list, tuple, np.ndarray, str, DtypeObj],\n trigger_computations: bool = False,\n axis: int = 0,\n func: Optional[\n Callable[[pandas.DataFrame, pandas.DataFrame], pandas.DataFrame]\n ] = None,\n) -> Optional[pandas.Series]:\n \"\"\"\n Precompute data types for binary operations by finding common type between operands.\n\n Parameters\n ----------\n first : PandasQueryCompiler\n First operand for which the binary operation would be performed later.\n second : PandasQueryCompiler, dict, list, tuple, np.ndarray, str or DtypeObj\n Second operand for which the binary operation would be performed later.\n trigger_computations : bool, default: False\n Whether to trigger computation of the lazy metadata for `first` and `second`.\n If False is specified this method will return None if any of the operands doesn't\n have materialized dtypes.\n axis : int, default: 0\n Axis to perform the binary operation along.\n func : callable(pandas.DataFrame, pandas.DataFrame) -> pandas.DataFrame, optional\n If specified, will use this function to perform the \"try_sample\" method\n (see ``Binary.register()`` docs for more details).\n\n Returns\n -------\n pandas.Series\n The pandas series with precomputed dtypes or None if there's not enough metadata to compute it.\n\n Notes\n -----\n The dtypes of the operands are supposed to be known.\n \"\"\"\n if not trigger_computations:\n if not first.frame_has_materialized_dtypes:\n return None\n\n if isinstance(second, type(first)) and not second.frame_has_materialized_dtypes:\n return None\n\n dtypes_first = first.dtypes.to_dict()\n if isinstance(second, type(first)):\n dtypes_second = second.dtypes.to_dict()\n columns_first = set(first.columns)\n columns_second = set(second.columns)\n common_columns = columns_first.intersection(columns_second)\n # Here we want to XOR the sets in order to find the columns that do not\n # belong to the intersection, these will be NaN columns in the result\n mismatch_columns = columns_first ^ columns_second\n elif isinstance(second, dict):\n dtypes_second = {\n key: pandas.api.types.pandas_dtype(type(value))\n for key, value in second.items()\n }\n columns_first = set(first.columns)\n columns_second = set(second.keys())\n common_columns = columns_first.intersection(columns_second)\n # Here we want to find the difference between the sets in order to find columns\n # that are missing in the dictionary, this will be NaN columns in the result\n mismatch_columns = columns_first.difference(columns_second)\n else:\n if isinstance(second, (list, tuple)):\n second_dtypes_list = (\n [pandas.api.types.pandas_dtype(type(value)) for value in second]\n if axis == 1\n # Here we've been given a column so it has only one dtype,\n # Infering the dtype using `np.array`, TODO: maybe there's more efficient way?\n else [np.array(second).dtype] * len(dtypes_first)\n )\n elif is_scalar(second) or isinstance(second, np.ndarray):\n try:\n dtype = getattr(second, \"dtype\", None) or pandas.api.types.pandas_dtype(\n type(second)\n )\n except TypeError:\n # For example, dtype '' not understood\n dtype = pandas.Series(second).dtype\n second_dtypes_list = [dtype] * len(dtypes_first)\n else:\n raise NotImplementedError(\n f\"Can't compute common type for {type(first)} and {type(second)}.\"\n )\n # We verify operands shapes at the front-end, invalid operands shouldn't be\n # propagated to the query compiler level\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=len(second_dtypes_list) != len(dtypes_first),\n extra_log=\"Shapes of the operands of a binary operation don't match\",\n )\n dtypes_second = {\n key: second_dtypes_list[idx] for idx, key in enumerate(dtypes_first.keys())\n }\n common_columns = first.columns\n mismatch_columns = []\n\n # If at least one column doesn't match, the result of the non matching column would be nan.\n nan_dtype = pandas.api.types.pandas_dtype(type(np.nan))\n dtypes = None\n if func is not None:\n try:\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\")\n df1 = pandas.DataFrame([[1] * len(common_columns)]).astype(\n {i: dtypes_first[col] for i, col in enumerate(common_columns)}\n )\n df2 = pandas.DataFrame([[1] * len(common_columns)]).astype(\n {i: dtypes_second[col] for i, col in enumerate(common_columns)}\n )\n dtypes = func(df1, df2).dtypes.set_axis(common_columns)\n # it sometimes doesn't work correctly with strings, so falling back to\n # the \"common_cast\" method in this case\n except TypeError:\n pass\n if dtypes is None:\n dtypes = pandas.Series(\n [\n pandas.core.dtypes.cast.find_common_type(\n [\n dtypes_first[x],\n dtypes_second[x],\n ]\n )\n for x in common_columns\n ],\n index=common_columns,\n )\n dtypes: pandas.Series = pandas.concat(\n [\n dtypes,\n pandas.Series(\n [nan_dtype] * (len(mismatch_columns)),\n index=mismatch_columns,\n ),\n ]\n )\n return dtypes\n\n\ndef maybe_build_dtypes_series(\n first: PandasQueryCompiler,\n second: Union[PandasQueryCompiler, Any],\n dtype: DtypeObj,\n trigger_computations: bool = False,\n) -> Optional[pandas.Series]:\n \"\"\"\n Build a ``pandas.Series`` describing dtypes of the result of a binary operation.\n\n Parameters\n ----------\n first : PandasQueryCompiler\n First operand for which the binary operation would be performed later.\n second : PandasQueryCompiler, list-like or scalar\n Second operand for which the binary operation would be performed later.\n dtype : DtypeObj\n Dtype of the result.\n trigger_computations : bool, default: False\n Whether to trigger computation of the lazy metadata for `first` and `second`.\n If False is specified this method will return None if any of the operands doesn't\n have materialized columns.\n\n Returns\n -------\n pandas.Series or None\n The pandas series with precomputed dtypes or None if there's not enough metadata to compute it.\n\n Notes\n -----\n Finds a union of columns and finds dtypes for all these columns.\n \"\"\"\n if not trigger_computations:\n if not first.frame_has_columns_cache:\n return None\n\n if isinstance(second, type(first)) and not second.frame_has_columns_cache:\n return None\n\n columns_first = set(first.columns)\n if isinstance(second, type(first)):\n columns_second = set(second.columns)\n columns_union = columns_first.union(columns_second)\n else:\n columns_union = columns_first\n\n dtypes = pandas.Series([dtype] * len(columns_union), index=columns_union)\n return dtypes\n\n\ndef try_compute_new_dtypes(\n first: PandasQueryCompiler,\n second: Union[PandasQueryCompiler, Any],\n infer_dtypes: Optional[str] = None,\n result_dtype: Optional[Union[DtypeObj, str]] = None,\n axis: int = 0,\n func: Optional[\n Callable[[pandas.DataFrame, pandas.DataFrame], pandas.DataFrame]\n ] = None,\n) -> Optional[pandas.Series]:\n \"\"\"\n Precompute resulting dtypes of the binary operation if possible.\n\n The dtypes won't be precomputed if any of the operands doesn't have their dtypes materialized\n or if the second operand type is not supported. Supported types: PandasQueryCompiler, list,\n dict, tuple, np.ndarray.\n\n Parameters\n ----------\n first : PandasQueryCompiler\n First operand of the binary operation.\n second : PandasQueryCompiler, list-like or scalar\n Second operand of the binary operation.\n infer_dtypes : {\"common_cast\", \"try_sample\", \"bool\", None}, default: None\n How dtypes should be infered (see ``Binary.register`` doc for more info).\n result_dtype : np.dtype, optional\n NumPy dtype of the result. If not specified it will be inferred from the `infer_dtypes` parameter.\n axis : int, default: 0\n Axis to perform the binary operation along.\n func : callable(pandas.DataFrame, pandas.DataFrame) -> pandas.DataFrame, optional\n A callable to be used for the \"try_sample\" method.\n\n Returns\n -------\n pandas.Series or None\n \"\"\"\n if infer_dtypes is None and result_dtype is None:\n return None\n\n try:\n if infer_dtypes == \"bool\" or is_bool_dtype(result_dtype):\n dtypes = maybe_build_dtypes_series(\n first, second, dtype=pandas.api.types.pandas_dtype(bool)\n )\n elif infer_dtypes == \"common_cast\":\n dtypes = maybe_compute_dtypes_common_cast(\n first, second, axis=axis, func=None\n )\n elif infer_dtypes == \"try_sample\":\n if func is None:\n raise ValueError(\n \"'func' must be specified if dtypes infering method is 'try_sample'\"\n )\n dtypes = maybe_compute_dtypes_common_cast(\n first, second, axis=axis, func=func\n )\n else:\n # For now we only know how to handle `result_dtype == bool` as that's\n # the only value that is being passed here right now, it's unclear\n # how we should behave in case of an arbitrary dtype, so let's wait\n # for at least one case to appear for this regard.\n dtypes = None\n except NotImplementedError:\n dtypes = None\n\n return dtypes\n\n\nclass Binary(Operator):\n \"\"\"Builder class for Binary operator.\"\"\"\n\n @classmethod\n def register(\n cls,\n func: Callable[..., pandas.DataFrame],\n join_type: str = \"outer\",\n sort: bool = None,\n labels: str = \"replace\",\n infer_dtypes: Optional[str] = None,\n ) -> Callable[..., PandasQueryCompiler]:\n \"\"\"\n Build template binary operator.\n\n Parameters\n ----------\n func : callable(pandas.DataFrame, [pandas.DataFrame, list-like, scalar]) -> pandas.DataFrame\n Binary function to execute. Have to be able to accept at least two arguments.\n join_type : {'left', 'right', 'outer', 'inner', None}, default: 'outer'\n Type of join that will be used if indices of operands are not aligned.\n sort : bool, default: None\n Whether to sort index and columns or not.\n labels : {\"keep\", \"replace\", \"drop\"}, default: \"replace\"\n Whether keep labels from left Modin DataFrame, replace them with labels\n from joined DataFrame or drop altogether to make them be computed lazily later.\n infer_dtypes : {\"common_cast\", \"try_sample\", \"bool\", None}, default: None\n How dtypes should be inferred.\n * If \"common_cast\", casts to common dtype of operand columns.\n * If \"try_sample\", creates small pandas DataFrames with dtypes of operands and\n runs the `func` on them to determine output dtypes. If a ``TypeError`` is raised\n during this process, fallback to \"common_cast\" method.\n * If \"bool\", dtypes would be a boolean series with same size as that of operands.\n * If ``None``, do not infer new dtypes (they will be computed manually once accessed).\n\n Returns\n -------\n callable\n Function that takes query compiler and executes binary operation.\n \"\"\"\n\n def caller(\n query_compiler: PandasQueryCompiler,\n other: Union[PandasQueryCompiler, Any],\n broadcast: bool = False,\n *args: tuple,\n dtypes: Optional[Union[DtypeObj, str]] = None,\n **kwargs: dict,\n ) -> PandasQueryCompiler:\n \"\"\"\n Apply binary `func` to passed operands.\n\n Parameters\n ----------\n query_compiler : PandasQueryCompiler\n Left operand of `func`.\n other : PandasQueryCompiler, list-like object or scalar\n Right operand of `func`.\n broadcast : bool, default: False\n If `other` is a one-column query compiler, indicates whether it is a Series or not.\n Frames and Series have to be processed differently, however we can't distinguish them\n at the query compiler level, so this parameter is a hint that passed from a high level API.\n *args : tuple,\n Arguments that will be passed to `func`.\n dtypes : \"copy\", scalar dtype or None, default: None\n Dtypes of the result. \"copy\" to keep old dtypes and None to compute them on demand.\n **kwargs : dict,\n Arguments that will be passed to `func`.\n\n Returns\n -------\n PandasQueryCompiler\n Result of binary function.\n \"\"\"\n axis: int = kwargs.get(\"axis\", 0)\n if isinstance(other, type(query_compiler)) and broadcast:\n assert (\n len(other.columns) == 1\n ), \"Invalid broadcast argument for `broadcast_apply`, too many columns: {}\".format(\n len(other.columns)\n )\n # Transpose on `axis=1` because we always represent an individual\n # column or row as a single-column Modin DataFrame\n if axis == 1:\n other = other.transpose()\n if dtypes != \"copy\":\n dtypes = try_compute_new_dtypes(\n query_compiler, other, infer_dtypes, dtypes, axis, func\n )\n\n shape_hint = None\n if isinstance(other, type(query_compiler)):\n if broadcast:\n if (\n query_compiler.frame_has_materialized_columns\n and other.frame_has_materialized_columns\n ):\n if (\n len(query_compiler.columns) == 1\n and len(other.columns) == 1\n and query_compiler.columns.equals(other.columns)\n ):\n shape_hint = \"column\"\n return query_compiler.__constructor__(\n query_compiler._modin_frame.broadcast_apply(\n axis,\n lambda left, right: func(\n left, right.squeeze(), *args, **kwargs\n ),\n other._modin_frame,\n join_type=join_type,\n labels=labels,\n dtypes=dtypes,\n ),\n shape_hint=shape_hint,\n )\n else:\n if (\n query_compiler.frame_has_materialized_columns\n and other.frame_has_materialized_columns\n ):\n if (\n len(query_compiler.columns) == 1\n and len(other.columns) == 1\n and query_compiler.columns.equals(other.columns)\n ):\n shape_hint = \"column\"\n return query_compiler.__constructor__(\n query_compiler._modin_frame.n_ary_op(\n lambda x, y: func(x, y, *args, **kwargs),\n [other._modin_frame],\n join_type=join_type,\n sort=sort,\n labels=labels,\n dtypes=dtypes,\n ),\n shape_hint=shape_hint,\n )\n else:\n # TODO: it's possible to chunk the `other` and broadcast them to partitions\n # accordingly, in that way we will be able to use more efficient `._modin_frame.map()`\n if isinstance(other, (dict, list, np.ndarray, pandas.Series)):\n new_modin_frame = query_compiler._modin_frame.apply_full_axis(\n axis,\n lambda df: func(df, other, *args, **kwargs),\n new_index=query_compiler.index,\n new_columns=query_compiler.columns,\n dtypes=dtypes,\n )\n else:\n if (\n query_compiler.frame_has_materialized_columns\n and len(query_compiler._modin_frame.columns) == 1\n and is_scalar(other)\n ):\n shape_hint = \"column\"\n new_modin_frame = query_compiler._modin_frame.map(\n func,\n func_args=(other, *args),\n func_kwargs=kwargs,\n dtypes=dtypes,\n lazy=True,\n )\n return query_compiler.__constructor__(\n new_modin_frame, shape_hint=shape_hint\n )\n\n return caller\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module default2pandas provides templates for a query compiler default-to-pandas methods.\"\"\"\n\nfrom .binary import BinaryDefault\nfrom .cat import CatDefault\nfrom .dataframe import DataFrameDefault\nfrom .datetime import DateTimeDefault\nfrom .default import DefaultMethod\nfrom .groupby import GroupByDefault, SeriesGroupByDefault\nfrom .list import ListDefault\nfrom .resample import ResampleDefault\nfrom .rolling import ExpandingDefault, RollingDefault\nfrom .series import SeriesDefault\nfrom .str import StrDefault\nfrom .struct import StructDefault\n\n__all__ = [\n \"DataFrameDefault\",\n \"DateTimeDefault\",\n \"SeriesDefault\",\n \"StrDefault\",\n \"BinaryDefault\",\n \"ResampleDefault\",\n \"RollingDefault\",\n \"ExpandingDefault\",\n \"DefaultMethod\",\n \"CatDefault\",\n \"GroupByDefault\",\n \"SeriesGroupByDefault\",\n \"ListDefault\",\n \"StructDefault\",\n]\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/binary.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default binary functions builder class.\"\"\"\n\nimport pandas\nfrom pandas.core.dtypes.common import is_list_like\n\nfrom .default import DefaultMethod\n\n\nclass BinaryDefault(DefaultMethod):\n \"\"\"Build default-to-pandas methods which executes binary functions.\"\"\"\n\n @classmethod\n def build_default_to_pandas(cls, fn, fn_name):\n \"\"\"\n Build function that do fallback to pandas for passed binary `fn`.\n\n Parameters\n ----------\n fn : callable\n Binary function to apply to the casted to pandas frame and other operand.\n fn_name : str\n Function name which will be shown in default-to-pandas warning message.\n\n Returns\n -------\n callable\n Function that takes query compiler, does fallback to pandas and applies binary `fn`\n to the casted to pandas frame.\n \"\"\"\n\n def bin_ops_wrapper(df, other, *args, **kwargs):\n \"\"\"Apply specified binary function to the passed operands.\"\"\"\n squeeze_other = kwargs.pop(\"broadcast\", False) or kwargs.pop(\n \"squeeze_other\", False\n )\n squeeze_self = kwargs.pop(\"squeeze_self\", False)\n\n if squeeze_other:\n other = other.squeeze(axis=1)\n\n if squeeze_self:\n df = df.squeeze(axis=1)\n\n result = fn(df, other, *args, **kwargs)\n if (\n not isinstance(result, pandas.Series)\n and not isinstance(result, pandas.DataFrame)\n and is_list_like(result)\n ):\n result = pandas.DataFrame(result)\n return result\n\n return super().build_default_to_pandas(bin_ops_wrapper, fn_name)\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/cat.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default applied-on-category functions builder class.\"\"\"\n\nfrom .series import SeriesDefault\n\n\nclass CatDefault(SeriesDefault):\n \"\"\"Builder for default-to-pandas methods which is executed under category accessor.\"\"\"\n\n @classmethod\n def frame_wrapper(cls, df):\n \"\"\"\n Get category accessor of the passed frame.\n\n Parameters\n ----------\n df : pandas.DataFrame\n\n Returns\n -------\n pandas.core.arrays.categorical.CategoricalAccessor\n \"\"\"\n return df.squeeze(axis=1).cat\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/dataframe.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default DataFrame functions builder class.\"\"\"\n\nimport pandas\n\nfrom modin.utils import _inherit_docstrings\n\nfrom .default import DefaultMethod\n\n\n@_inherit_docstrings(DefaultMethod)\nclass DataFrameDefault(DefaultMethod):\n DEFAULT_OBJECT_TYPE = pandas.DataFrame\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/datetime.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default applied-on-datetime functions builder class.\"\"\"\n\nfrom .series import SeriesDefault\n\n\nclass DateTimeDefault(SeriesDefault):\n \"\"\"Builder for default-to-pandas methods which is executed under datetime accessor.\"\"\"\n\n @classmethod\n def frame_wrapper(cls, df):\n \"\"\"\n Get datetime accessor of the passed frame.\n\n Parameters\n ----------\n df : pandas.DataFrame\n\n Returns\n -------\n pandas.core.indexes.accessors.DatetimeProperties\n \"\"\"\n return df.squeeze(axis=1).dt\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/default.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default functions builder class.\"\"\"\n\nimport pandas\nfrom pandas.core.dtypes.common import is_list_like\n\nfrom modin.core.dataframe.algebra.operator import Operator\nfrom modin.utils import MODIN_UNNAMED_SERIES_LABEL, try_cast_to_pandas\n\n\nclass ObjTypeDeterminer:\n \"\"\"\n Class that routes work to the frame.\n\n Provides an instance which forwards all of the `__getattribute__` calls\n to an object under which `key` function is applied.\n \"\"\"\n\n def __getattr__(self, key):\n \"\"\"\n Build function that executes `key` function over passed frame.\n\n Parameters\n ----------\n key : str\n\n Returns\n -------\n callable\n Function that takes DataFrame and executes `key` function on it.\n \"\"\"\n\n def func(df, *args, **kwargs):\n \"\"\"Access specified attribute of the passed object and call it if it's callable.\"\"\"\n prop = getattr(df, key)\n if callable(prop):\n return prop(*args, **kwargs)\n else:\n return prop\n\n return func\n\n\nclass DefaultMethod(Operator):\n \"\"\"\n Builder for default-to-pandas methods.\n\n Attributes\n ----------\n OBJECT_TYPE : str\n Object type name that will be shown in default-to-pandas warning message.\n DEFAULT_OBJECT_TYPE : object\n Default place to search for a function.\n \"\"\"\n\n OBJECT_TYPE = \"DataFrame\"\n DEFAULT_OBJECT_TYPE = ObjTypeDeterminer\n\n @classmethod\n def register(cls, func, obj_type=None, inplace=None, fn_name=None):\n \"\"\"\n Build function that do fallback to default pandas implementation for passed `func`.\n\n Parameters\n ----------\n func : callable or str,\n Function to apply to the casted to pandas frame or its property accesed\n by ``cls.frame_wrapper``.\n obj_type : object, optional\n If `func` is a string with a function name then `obj_type` provides an\n object to search function in.\n inplace : bool, optional\n If True return an object to which `func` was applied, otherwise return\n the result of `func`.\n fn_name : str, optional\n Function name which will be shown in default-to-pandas warning message.\n If not specified, name will be deducted from `func`.\n\n Returns\n -------\n callable\n Function that takes query compiler, does fallback to pandas and applies `func`\n to the casted to pandas frame or its property accesed by ``cls.frame_wrapper``.\n \"\"\"\n if isinstance(func, str):\n if obj_type is None:\n obj_type = cls.DEFAULT_OBJECT_TYPE\n fn = getattr(obj_type, func)\n else:\n fn = func\n\n if type(fn) is property:\n if fn_name is None and hasattr(fn, \"fget\"):\n # When `fn` is a property, `str(fn)` will be something like\n # \"\". We instead check its `fget` method to get\n # the name of the property.\n # Note that this method is still imperfect because we cannot get the class name\n # of the property. For example, we can only get \"hour\" from `Series.dt.hour`.\n fn_name = f\"\"\n fn = cls.build_property_wrapper(fn)\n else:\n fn_name = getattr(fn, \"__name__\", str(fn)) if fn_name is None else fn_name\n\n def applyier(df, *args, **kwargs):\n \"\"\"\n Apply target function to the casted to pandas frame.\n\n This function is directly applied to the casted to pandas frame, executes target\n function under it and processes result so it is possible to create a valid\n query compiler from it.\n \"\"\"\n # pandas default implementation doesn't know how to handle `dtypes` keyword argument\n kwargs.pop(\"dtypes\", None)\n df = cls.frame_wrapper(df)\n result = fn(df, *args, **kwargs)\n\n if (\n not isinstance(result, pandas.Series)\n and not isinstance(result, pandas.DataFrame)\n and func not in (\"to_numpy\", pandas.DataFrame.to_numpy)\n and func not in (\"align\", pandas.DataFrame.align)\n and func not in (\"divmod\", pandas.Series.divmod)\n and func not in (\"rdivmod\", pandas.Series.rdivmod)\n and func not in (\"to_list\", pandas.Series.to_list)\n and func not in (\"corr\", pandas.Series.corr)\n and func not in (\"to_dict\", pandas.Series.to_dict)\n and func not in (\"mean\", pandas.DataFrame.mean)\n and func not in (\"median\", pandas.DataFrame.median)\n and func not in (\"skew\", pandas.DataFrame.skew)\n and func not in (\"kurt\", pandas.DataFrame.kurt)\n ):\n # When applying a DatetimeProperties or TimedeltaProperties function,\n # if we don't specify the dtype for the DataFrame, the frame might\n # get the wrong dtype, e.g. for to_pydatetime in\n # https://github.com/modin-project/modin/issues/4436\n astype_kwargs = {}\n dtype = getattr(result, \"dtype\", None)\n if dtype and isinstance(\n df,\n (\n pandas.core.indexes.accessors.DatetimeProperties,\n pandas.core.indexes.accessors.TimedeltaProperties,\n ),\n ):\n astype_kwargs[\"dtype\"] = dtype\n result = (\n pandas.DataFrame(result, **astype_kwargs)\n if is_list_like(result)\n else pandas.DataFrame([result], **astype_kwargs)\n )\n if isinstance(result, pandas.Series):\n if result.name is None:\n result.name = MODIN_UNNAMED_SERIES_LABEL\n result = result.to_frame()\n\n inplace_method = kwargs.get(\"inplace\", False)\n if inplace is not None:\n inplace_method = inplace\n return result if not inplace_method else df\n\n return cls.build_wrapper(applyier, fn_name)\n\n @classmethod\n # FIXME: this method is almost a duplicate of `cls.build_default_to_pandas`.\n # Those two methods should be merged into a single one.\n def build_wrapper(cls, fn, fn_name):\n \"\"\"\n Build function that do fallback to pandas for passed `fn`.\n\n In comparison with ``cls.build_default_to_pandas`` this method also\n casts function arguments to pandas before doing fallback.\n\n Parameters\n ----------\n fn : callable\n Function to apply to the defaulted frame.\n fn_name : str\n Function name which will be shown in default-to-pandas warning message.\n\n Returns\n -------\n callable\n Method that does fallback to pandas and applies `fn` to the pandas frame.\n \"\"\"\n wrapper = cls.build_default_to_pandas(fn, fn_name)\n\n def args_cast(self, *args, **kwargs):\n \"\"\"\n Preprocess `default_to_pandas` function arguments and apply default function.\n\n Cast all Modin objects that function arguments contain to its pandas representation.\n \"\"\"\n args = try_cast_to_pandas(args)\n kwargs = try_cast_to_pandas(kwargs)\n return wrapper(self, *args, **kwargs)\n\n return args_cast\n\n @classmethod\n def build_property_wrapper(cls, prop):\n \"\"\"\n Build function that accesses specified property of the frame.\n\n Parameters\n ----------\n prop : str\n Property name to access.\n\n Returns\n -------\n callable\n Function that takes DataFrame and returns its value of `prop` property.\n \"\"\"\n\n def property_wrapper(df):\n \"\"\"Get specified property of the passed object.\"\"\"\n return prop.fget(df)\n\n return property_wrapper\n\n @classmethod\n def build_default_to_pandas(cls, fn, fn_name):\n \"\"\"\n Build function that do fallback to pandas for passed `fn`.\n\n Parameters\n ----------\n fn : callable\n Function to apply to the defaulted frame.\n fn_name : str\n Function name which will be shown in default-to-pandas warning message.\n\n Returns\n -------\n callable\n Method that does fallback to pandas and applies `fn` to the pandas frame.\n \"\"\"\n fn.__name__ = f\"\"\n\n def wrapper(self, *args, **kwargs):\n \"\"\"Do fallback to pandas for the specified function.\"\"\"\n return self.default_to_pandas(fn, *args, **kwargs)\n\n return wrapper\n\n @classmethod\n def frame_wrapper(cls, df):\n \"\"\"\n Extract frame property to apply function on.\n\n This method is executed under casted to pandas frame right before applying\n a function passed to `register`, which gives an ability to transform frame somehow\n or access its properties, by overriding this method in a child class.\n\n Parameters\n ----------\n df : pandas.DataFrame\n\n Returns\n -------\n pandas.DataFrame\n\n Notes\n -----\n Being a base implementation, this particular method does nothing with passed frame.\n \"\"\"\n return df\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/groupby.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default GroupBy functions builder class.\"\"\"\n\nimport warnings\nfrom typing import Any\n\nimport pandas\nfrom pandas.core.dtypes.common import is_list_like\n\n# Defines a set of string names of functions that are executed in a transform-way in groupby\nfrom pandas.core.groupby.base import transformation_kernels\n\nfrom modin.utils import MODIN_UNNAMED_SERIES_LABEL, hashable\n\nfrom .default import DefaultMethod\n\n\n# FIXME: there is no sence of keeping `GroupBy` and `GroupByDefault` logic in a different\n# classes. They should be combined.\nclass GroupBy:\n \"\"\"Builder for GroupBy aggregation functions.\"\"\"\n\n agg_aliases = [\n \"agg\",\n \"dict_agg\",\n pandas.core.groupby.DataFrameGroupBy.agg,\n pandas.core.groupby.DataFrameGroupBy.aggregate,\n ]\n\n @staticmethod\n def is_transformation_kernel(agg_func: Any) -> bool:\n \"\"\"\n Check whether a passed aggregation function is a transformation.\n\n Transformation means that the result of the function will be broadcasted\n to the frame's original shape.\n\n Parameters\n ----------\n agg_func : Any\n\n Returns\n -------\n bool\n \"\"\"\n return hashable(agg_func) and agg_func in transformation_kernels.union(\n # these methods are also producing transpose-like result in a sense we understand it\n # (they're non-aggregative functions), however are missing in the pandas dictionary\n {\"nth\", \"head\", \"tail\"}\n )\n\n @classmethod\n def _call_groupby(cls, df, *args, **kwargs): # noqa: PR01\n \"\"\"Call .groupby() on passed `df`.\"\"\"\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\", category=FutureWarning)\n return df.groupby(*args, **kwargs)\n\n @classmethod\n def validate_by(cls, by):\n \"\"\"\n Build valid `by` parameter for `pandas.DataFrame.groupby`.\n\n Cast all DataFrames in `by` parameter to Series or list of Series in case\n of multi-column frame.\n\n Parameters\n ----------\n by : DateFrame, Series, index label or list of such\n Object which indicates groups for GroupBy.\n\n Returns\n -------\n Series, index label or list of such\n By parameter with all DataFrames casted to Series.\n \"\"\"\n\n def try_cast_series(df):\n \"\"\"Cast one-column frame to Series.\"\"\"\n if isinstance(df, pandas.DataFrame):\n df = df.squeeze(axis=1)\n if not isinstance(df, pandas.Series):\n return df\n if df.name == MODIN_UNNAMED_SERIES_LABEL:\n df.name = None\n return df\n\n if isinstance(by, pandas.DataFrame):\n by = [try_cast_series(column) for _, column in by.items()]\n elif isinstance(by, pandas.Series):\n by = [try_cast_series(by)]\n elif isinstance(by, list):\n by = [try_cast_series(o) for o in by]\n return by\n\n @classmethod\n def inplace_applyier_builder(cls, key, func=None):\n \"\"\"\n Bind actual aggregation function to the GroupBy aggregation method.\n\n Parameters\n ----------\n key : callable\n Function that takes GroupBy object and evaluates passed aggregation function.\n func : callable or str, optional\n Function that takes DataFrame and aggregate its data. Will be applied\n to each group at the grouped frame.\n\n Returns\n -------\n callable,\n Function that executes aggregation under GroupBy object.\n \"\"\"\n inplace_args = [] if func is None else [func]\n\n def inplace_applyier(grp, *func_args, **func_kwargs):\n return key(grp, *inplace_args, *func_args, **func_kwargs)\n\n return inplace_applyier\n\n @classmethod\n def get_func(cls, key, **kwargs):\n \"\"\"\n Extract aggregation function from groupby arguments.\n\n Parameters\n ----------\n key : callable or str\n Default aggregation function. If aggregation function is not specified\n via groupby arguments, then `key` function is used.\n **kwargs : dict\n GroupBy arguments that may contain aggregation function.\n\n Returns\n -------\n callable\n Aggregation function.\n\n Notes\n -----\n There are two ways of how groupby aggregation can be invoked:\n 1. Explicitly with query compiler method: `qc.groupby_sum()`.\n 2. By passing aggregation function as an argument: `qc.groupby_agg(\"sum\")`.\n Both are going to produce the same result, however in the first case actual aggregation\n function can be extracted from the method name, while for the second only from the method arguments.\n \"\"\"\n if \"agg_func\" in kwargs:\n return cls.inplace_applyier_builder(key, kwargs[\"agg_func\"])\n elif \"func_dict\" in kwargs:\n return cls.inplace_applyier_builder(key, kwargs[\"func_dict\"])\n else:\n return cls.inplace_applyier_builder(key)\n\n @classmethod\n def build_aggregate_method(cls, key):\n \"\"\"\n Build function for `QueryCompiler.groupby_agg` that can be executed as default-to-pandas.\n\n Parameters\n ----------\n key : callable or str\n Default aggregation function. If aggregation function is not specified\n via groupby arguments, then `key` function is used.\n\n Returns\n -------\n callable\n Function that executes groupby aggregation.\n \"\"\"\n\n def fn(\n df,\n by,\n axis,\n groupby_kwargs,\n agg_args,\n agg_kwargs,\n drop=False,\n **kwargs,\n ):\n \"\"\"Group DataFrame and apply aggregation function to each group.\"\"\"\n by = cls.validate_by(by)\n\n grp = cls._call_groupby(df, by, axis=axis, **groupby_kwargs)\n agg_func = cls.get_func(key, **kwargs)\n result = agg_func(grp, *agg_args, **agg_kwargs)\n\n return result\n\n return fn\n\n @classmethod\n def build_groupby_reduce_method(cls, agg_func):\n \"\"\"\n Build function for `QueryCompiler.groupby_*` that can be executed as default-to-pandas.\n\n Parameters\n ----------\n agg_func : callable or str\n Default aggregation function. If aggregation function is not specified\n via groupby arguments, then `agg_func` function is used.\n\n Returns\n -------\n callable\n Function that executes groupby aggregation.\n \"\"\"\n\n def fn(\n df, by, axis, groupby_kwargs, agg_args, agg_kwargs, drop=False, **kwargs\n ):\n \"\"\"Group DataFrame and apply aggregation function to each group.\"\"\"\n if not isinstance(by, (pandas.Series, pandas.DataFrame)):\n by = cls.validate_by(by)\n grp = cls._call_groupby(df, by, axis=axis, **groupby_kwargs)\n grp_agg_func = cls.get_func(agg_func, **kwargs)\n return grp_agg_func(\n grp,\n *agg_args,\n **agg_kwargs,\n )\n\n if isinstance(by, pandas.DataFrame):\n by = by.squeeze(axis=1)\n if (\n drop\n and isinstance(by, pandas.Series)\n and by.name in df\n and df[by.name].equals(by)\n ):\n by = [by.name]\n if isinstance(by, pandas.DataFrame):\n df = pandas.concat([df] + [by[[o for o in by if o not in df]]], axis=1)\n by = list(by.columns)\n\n groupby_kwargs = groupby_kwargs.copy()\n as_index = groupby_kwargs.pop(\"as_index\", True)\n groupby_kwargs[\"as_index\"] = True\n\n grp = cls._call_groupby(df, by, axis=axis, **groupby_kwargs)\n func = cls.get_func(agg_func, **kwargs)\n result = func(grp, *agg_args, **agg_kwargs)\n method = kwargs.get(\"method\")\n\n if isinstance(result, pandas.Series):\n result = result.to_frame(\n MODIN_UNNAMED_SERIES_LABEL if result.name is None else result.name\n )\n\n if not as_index:\n if isinstance(by, pandas.Series):\n # 1. If `drop` is True then 'by' Series represents a column from the\n # source frame and so the 'by' is internal.\n # 2. If method is 'size' then any 'by' is considered to be internal.\n # This is a hacky legacy from the ``groupby_size`` implementation:\n # https://github.com/modin-project/modin/issues/3739\n internal_by = (by.name,) if drop or method == \"size\" else tuple()\n else:\n internal_by = by\n\n cls.handle_as_index_for_dataframe(\n result,\n internal_by,\n by_cols_dtypes=(\n df.index.dtypes.values\n if isinstance(df.index, pandas.MultiIndex)\n else (df.index.dtype,)\n ),\n by_length=len(by),\n drop=drop,\n method=method,\n inplace=True,\n )\n\n if result.index.name == MODIN_UNNAMED_SERIES_LABEL:\n result.index.name = None\n\n return result\n\n return fn\n\n @classmethod\n def is_aggregate(cls, key): # noqa: PR01\n \"\"\"Check whether `key` is an alias for pandas.GroupBy.aggregation method.\"\"\"\n return key in cls.agg_aliases\n\n @classmethod\n def build_groupby(cls, func):\n \"\"\"\n Build function that groups DataFrame and applies aggregation function to the every group.\n\n Parameters\n ----------\n func : callable or str\n Default aggregation function. If aggregation function is not specified\n via groupby arguments, then `func` function is used.\n\n Returns\n -------\n callable\n Function that takes pandas DataFrame and does GroupBy aggregation.\n \"\"\"\n if cls.is_aggregate(func):\n return cls.build_aggregate_method(func)\n return cls.build_groupby_reduce_method(func)\n\n @classmethod\n def handle_as_index_for_dataframe(\n cls,\n result,\n internal_by_cols,\n by_cols_dtypes=None,\n by_length=None,\n selection=None,\n partition_idx=0,\n drop=True,\n method=None,\n inplace=False,\n ):\n \"\"\"\n Handle `as_index=False` parameter for the passed GroupBy aggregation result.\n\n Parameters\n ----------\n result : DataFrame\n Frame containing GroupBy aggregation result computed with `as_index=True`\n parameter (group names are located at the frame's index).\n internal_by_cols : list-like\n Internal 'by' columns.\n by_cols_dtypes : list-like, optional\n Data types of the internal 'by' columns. Required to do special casing\n in case of categorical 'by'. If not specified, assume that there is no\n categorical data in 'by'.\n by_length : int, optional\n Amount of keys to group on (including frame columns and external objects like list, Series, etc.)\n If not specified, consider `by_length` to be equal ``len(internal_by_cols)``.\n selection : label or list of labels, optional\n Set of columns that were explicitly selected for aggregation (for example\n via dict-aggregation). If not specified assuming that aggregation was\n applied to all of the available columns.\n partition_idx : int, default: 0\n Positional index of the current partition.\n drop : bool, default: True\n Indicates whether or not any of the `by` data came from the same frame.\n method : str, optional\n Name of the groupby function. This is a hint to be able to do special casing.\n Note: this parameter is a legacy from the ``groupby_size`` implementation,\n it's a hacky one and probably will be removed in the future: https://github.com/modin-project/modin/issues/3739.\n inplace : bool, default: False\n Modify the DataFrame in place (do not create a new object).\n\n Returns\n -------\n DataFrame\n GroupBy aggregation result with the considered `as_index=False` parameter.\n \"\"\"\n if not inplace:\n result = result.copy()\n\n reset_index, drop, lvls_to_drop, cols_to_drop = cls.handle_as_index(\n result_cols=result.columns,\n result_index_names=result.index.names,\n internal_by_cols=internal_by_cols,\n by_cols_dtypes=by_cols_dtypes,\n by_length=by_length,\n selection=selection,\n partition_idx=partition_idx,\n drop=drop,\n method=method,\n )\n\n if len(lvls_to_drop) > 0:\n result.index = result.index.droplevel(lvls_to_drop)\n if len(cols_to_drop) > 0:\n result.drop(columns=cols_to_drop, inplace=True)\n if reset_index:\n result.reset_index(drop=drop, inplace=True)\n return result\n\n @staticmethod\n def handle_as_index(\n result_cols,\n result_index_names,\n internal_by_cols,\n by_cols_dtypes=None,\n by_length=None,\n selection=None,\n partition_idx=0,\n drop=True,\n method=None,\n ):\n \"\"\"\n Compute hints to process ``as_index=False`` parameter for the GroupBy result.\n\n This function resolves naming conflicts of the index levels to insert and the column labels\n for the GroupBy result. The logic of this function assumes that the initial GroupBy result\n was computed as ``as_index=True``.\n\n Parameters\n ----------\n result_cols : pandas.Index\n Columns of the GroupBy result.\n result_index_names : list-like\n Index names of the GroupBy result.\n internal_by_cols : list-like\n Internal 'by' columns.\n by_cols_dtypes : list-like, optional\n Data types of the internal 'by' columns. Required to do special casing\n in case of categorical 'by'. If not specified, assume that there is no\n categorical data in 'by'.\n by_length : int, optional\n Amount of keys to group on (including frame columns and external objects like list, Series, etc.)\n If not specified, consider `by_length` to be equal ``len(internal_by_cols)``.\n selection : label or list of labels, optional\n Set of columns that were explicitly selected for aggregation (for example\n via dict-aggregation). If not specified assuming that aggregation was\n applied to all of the available columns.\n partition_idx : int, default: 0\n Positional index of the current partition.\n drop : bool, default: True\n Indicates whether or not any of the `by` data came from the same frame.\n method : str, optional\n Name of the groupby function. This is a hint to be able to do special casing.\n Note: this parameter is a legacy from the ``groupby_size`` implementation,\n it's a hacky one and probably will be removed in the future: https://github.com/modin-project/modin/issues/3739.\n\n Returns\n -------\n reset_index : bool\n Indicates whether to reset index to the default one (0, 1, 2 ... n) at this partition.\n drop_index : bool\n If `reset_index` is True, indicates whether to drop all index levels (True) or insert them into the\n resulting columns (False).\n lvls_to_drop : list of ints\n Contains numeric indices of the levels of the result index to drop as intersected.\n cols_to_drop : list of labels\n Contains labels of the columns to drop from the result as intersected.\n\n Examples\n --------\n >>> groupby_result = compute_groupby_without_processing_as_index_parameter()\n >>> if not as_index:\n >>> reset_index, drop, lvls_to_drop, cols_to_drop = handle_as_index(**extract_required_params(groupby_result))\n >>> if len(lvls_to_drop) > 0:\n >>> groupby_result.index = groupby_result.index.droplevel(lvls_to_drop)\n >>> if len(cols_to_drop) > 0:\n >>> groupby_result = groupby_result.drop(columns=cols_to_drop)\n >>> if reset_index:\n >>> groupby_result_with_processed_as_index_parameter = groupby_result.reset_index(drop=drop)\n >>> else:\n >>> groupby_result_with_processed_as_index_parameter = groupby_result\n \"\"\"\n if by_length is None:\n by_length = len(internal_by_cols)\n\n reset_index = method != \"transform\" and (by_length > 0 or selection is not None)\n\n # If the method is \"size\" then the result contains only one unique named column\n # and we don't have to worry about any naming conflicts, so inserting all of\n # the \"by\" into the result (just a fast-path)\n if method == \"size\":\n return reset_index, False, [], []\n\n # Pandas logic of resolving naming conflicts is the following:\n # 1. If any categorical is in 'by' and 'by' is multi-column, then the categorical\n # index is prioritized: drop intersected columns and insert all of the 'by' index\n # levels to the frame as columns.\n # 2. Otherwise, aggregation result is prioritized: drop intersected index levels and\n # insert the filtered ones to the frame as columns.\n if by_cols_dtypes is not None:\n keep_index_levels = (\n by_length > 1\n and selection is None\n and any(isinstance(x, pandas.CategoricalDtype) for x in by_cols_dtypes)\n )\n else:\n keep_index_levels = False\n\n # 1. We insert 'by'-columns to the result at the beginning of the frame and so only to the\n # first partition, if partition_idx != 0 we just drop the index. If there are no columns\n # that are required to drop (keep_index_levels is True) then we can exit here.\n # 2. We don't insert 'by'-columns to the result if 'by'-data came from a different\n # frame (drop is False), there's only one exception for this rule: if the `method` is \"size\",\n # so if (drop is False) and method is not \"size\" we just drop the index and so can exit here.\n if (not keep_index_levels and partition_idx != 0) or (\n not drop and method != \"size\"\n ):\n return reset_index, True, [], []\n\n if not isinstance(internal_by_cols, pandas.Index):\n if not is_list_like(internal_by_cols):\n internal_by_cols = [internal_by_cols]\n internal_by_cols = pandas.Index(internal_by_cols)\n\n internal_by_cols = (\n internal_by_cols[\n ~internal_by_cols.str.startswith(MODIN_UNNAMED_SERIES_LABEL, na=False)\n ]\n if hasattr(internal_by_cols, \"str\")\n else internal_by_cols\n )\n\n if selection is not None and not isinstance(selection, pandas.Index):\n selection = pandas.Index(selection)\n\n lvls_to_drop = []\n cols_to_drop = []\n\n if not keep_index_levels:\n # We want to insert only these internal-by-cols that are not presented\n # in the result in order to not create naming conflicts\n if selection is None:\n cols_to_insert = frozenset(internal_by_cols) - frozenset(result_cols)\n else:\n cols_to_insert = frozenset(\n # We have to use explicit 'not in' check and not just difference\n # of sets because of specific '__contains__' operator in case of\n # scalar 'col' and MultiIndex 'selection'.\n col\n for col in internal_by_cols\n if col not in selection\n )\n else:\n cols_to_insert = internal_by_cols\n # We want to drop such internal-by-cols that are presented\n # in the result in order to not create naming conflicts\n cols_to_drop = frozenset(internal_by_cols) & frozenset(result_cols)\n\n if partition_idx == 0:\n lvls_to_drop = [\n i\n for i, name in enumerate(result_index_names)\n if name not in cols_to_insert\n ]\n else:\n lvls_to_drop = result_index_names\n\n drop = False\n if len(lvls_to_drop) == len(result_index_names):\n drop = True\n lvls_to_drop = []\n\n return reset_index, drop, lvls_to_drop, cols_to_drop\n\n\nclass SeriesGroupBy(GroupBy):\n \"\"\"Builder for GroupBy aggregation functions for Series.\"\"\"\n\n @classmethod\n def _call_groupby(cls, df, *args, **kwargs): # noqa: PR01\n \"\"\"Call .groupby() on passed `df` squeezed to Series.\"\"\"\n # We can end up here by two means - either by \"true\" call\n # like Series().groupby() or by df.groupby()[item].\n\n if len(df.columns) == 1:\n # Series().groupby() case\n return df.squeeze(axis=1).groupby(*args, **kwargs)\n # In second case surrounding logic will supplement grouping columns,\n # so we need to drop them after grouping is over; our originally\n # selected column is always the first, so use it\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\", category=FutureWarning)\n return df.groupby(*args, **kwargs)[df.columns[0]]\n\n\nclass GroupByDefault(DefaultMethod):\n \"\"\"Builder for default-to-pandas GroupBy aggregation functions.\"\"\"\n\n _groupby_cls = GroupBy\n\n OBJECT_TYPE = \"GroupBy\"\n\n @classmethod\n def register(cls, func, **kwargs):\n \"\"\"\n Build default-to-pandas GroupBy aggregation function.\n\n Parameters\n ----------\n func : callable or str\n Default aggregation function. If aggregation function is not specified\n via groupby arguments, then `func` function is used.\n **kwargs : kwargs\n Additional arguments that will be passed to function builder.\n\n Returns\n -------\n callable\n Functiom that takes query compiler and defaults to pandas to do GroupBy\n aggregation.\n \"\"\"\n return super().register(\n cls._groupby_cls.build_groupby(func), fn_name=func.__name__, **kwargs\n )\n\n # This specifies a `pandas.DataFrameGroupBy` method to pass the `agg_func` to,\n # it's based on `how` to apply it. Going by pandas documentation:\n # 1. `.aggregate(func)` applies func row/column wise.\n # 2. `.apply(func)` applies func to a DataFrames, holding a whole group (group-wise).\n # 3. `.transform(func)` is the same as `.apply()` but also broadcast the `func`\n # result to the group's original shape.\n # 4. 'direct' mode means that the passed `func` has to be applied directly\n # to the `pandas.DataFrameGroupBy` object.\n _aggregation_methods_dict = {\n \"axis_wise\": pandas.core.groupby.DataFrameGroupBy.aggregate,\n \"group_wise\": pandas.core.groupby.DataFrameGroupBy.apply,\n \"transform\": pandas.core.groupby.DataFrameGroupBy.transform,\n \"direct\": lambda grp, func, *args, **kwargs: func(grp, *args, **kwargs),\n }\n\n @classmethod\n def get_aggregation_method(cls, how):\n \"\"\"\n Return `pandas.DataFrameGroupBy` method that implements the passed `how` UDF applying strategy.\n\n Parameters\n ----------\n how : {\"axis_wise\", \"group_wise\", \"transform\"}\n `how` parameter of the ``BaseQueryCompiler.groupby_agg``.\n\n Returns\n -------\n callable(pandas.DataFrameGroupBy, callable, *args, **kwargs) -> [pandas.DataFrame | pandas.Series]\n\n Notes\n -----\n Visit ``BaseQueryCompiler.groupby_agg`` doc-string for more information about `how` parameter.\n \"\"\"\n return cls._aggregation_methods_dict[how]\n\n\nclass SeriesGroupByDefault(GroupByDefault):\n \"\"\"Builder for default-to-pandas GroupBy aggregation functions for Series.\"\"\"\n\n _groupby_cls = SeriesGroupBy\n\n _aggregation_methods_dict = {\n \"axis_wise\": pandas.core.groupby.SeriesGroupBy.aggregate,\n \"group_wise\": pandas.core.groupby.SeriesGroupBy.apply,\n \"transform\": pandas.core.groupby.SeriesGroupBy.transform,\n \"direct\": lambda grp, func, *args, **kwargs: func(grp, *args, **kwargs),\n }\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/list.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default applied-on-list accessor functions builder class.\"\"\"\n\nfrom .series import SeriesDefault\n\n\nclass ListDefault(SeriesDefault):\n \"\"\"Builder for default-to-pandas methods which is executed under list accessor.\"\"\"\n\n @classmethod\n def frame_wrapper(cls, df):\n \"\"\"\n Get list accessor of the passed frame.\n\n Parameters\n ----------\n df : pandas.DataFrame\n\n Returns\n -------\n pandas.core.arrays.arrow.ListAccessor\n \"\"\"\n return df.squeeze(axis=1).list\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/resample.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default Resamle functions builder class.\"\"\"\n\nfrom .default import DefaultMethod\n\n\n# FIXME: there is no sence of keeping `Resampler` and `ResampleDefault` logic in a different\n# classes. They should be combined.\nclass Resampler:\n \"\"\"Builder class for resampled aggregation functions.\"\"\"\n\n @classmethod\n def build_resample(cls, func, squeeze_self):\n \"\"\"\n Build function that resamples time-series data and does aggregation.\n\n Parameters\n ----------\n func : callable\n Aggregation function to execute under resampled frame.\n squeeze_self : bool\n Whether or not to squeeze frame before resampling.\n\n Returns\n -------\n callable\n Function that takes pandas DataFrame and applies aggregation\n to resampled time-series data.\n \"\"\"\n\n def fn(df, resample_kwargs, *args, **kwargs):\n \"\"\"Resample time-series data of the passed frame and apply specified aggregation.\"\"\"\n if squeeze_self:\n df = df.squeeze(axis=1)\n resampler = df.resample(**resample_kwargs)\n\n if type(func) is property:\n return func.fget(resampler)\n\n return func(resampler, *args, **kwargs)\n\n return fn\n\n\nclass ResampleDefault(DefaultMethod):\n \"\"\"Builder for default-to-pandas resampled aggregation functions.\"\"\"\n\n OBJECT_TYPE = \"Resampler\"\n\n @classmethod\n def register(cls, func, squeeze_self=False, **kwargs):\n \"\"\"\n Build function that do fallback to pandas and aggregate resampled data.\n\n Parameters\n ----------\n func : callable\n Aggregation function to execute under resampled frame.\n squeeze_self : bool, default: False\n Whether or not to squeeze frame before resampling.\n **kwargs : kwargs\n Additional arguments that will be passed to function builder.\n\n Returns\n -------\n callable\n Function that takes query compiler and does fallback to pandas to resample\n time-series data and apply aggregation on it.\n \"\"\"\n return super().register(\n Resampler.build_resample(func, squeeze_self),\n fn_name=func.__name__,\n **kwargs\n )\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/rolling.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default Rolling functions builder class.\"\"\"\n\nfrom .default import DefaultMethod\n\n\nclass RollingDefault(DefaultMethod):\n \"\"\"Builder for default-to-pandas aggregation on a rolling window functions.\"\"\"\n\n OBJECT_TYPE = \"Rolling\"\n\n @classmethod\n def _build_rolling(cls, func):\n \"\"\"\n Build function that creates a rolling window and executes `func` on it.\n\n Parameters\n ----------\n func : callable\n Function to execute on a rolling window.\n\n Returns\n -------\n callable\n Function that takes pandas DataFrame and applies `func` on a rolling window.\n \"\"\"\n\n def fn(df, rolling_kwargs, *args, **kwargs):\n \"\"\"Create rolling window for the passed frame and execute specified `func` on it.\"\"\"\n roller = df.rolling(**rolling_kwargs)\n\n if type(func) is property:\n return func.fget(roller)\n\n return func(roller, *args, **kwargs)\n\n return fn\n\n @classmethod\n def register(cls, func, **kwargs):\n \"\"\"\n Build function that do fallback to pandas to apply `func` on a rolling window.\n\n Parameters\n ----------\n func : callable\n Function to execute on a rolling window.\n **kwargs : kwargs\n Additional arguments that will be passed to function builder.\n\n Returns\n -------\n callable\n Function that takes query compiler and defaults to pandas to apply aggregation\n `func` on a rolling window.\n \"\"\"\n return super().register(\n cls._build_rolling(func), fn_name=func.__name__, **kwargs\n )\n\n\nclass ExpandingDefault(DefaultMethod):\n \"\"\"Builder for default-to-pandas aggregation on an expanding window functions.\"\"\"\n\n OBJECT_TYPE = \"Expanding\"\n\n @classmethod\n def _build_expanding(cls, func, squeeze_self):\n \"\"\"\n Build function that creates an expanding window and executes `func` on it.\n\n Parameters\n ----------\n func : callable\n Function to execute on a expanding window.\n squeeze_self : bool\n Whether or not to squeeze frame before executing the window function.\n\n Returns\n -------\n callable\n Function that takes pandas DataFrame and applies `func` on a expanding window.\n \"\"\"\n\n def fn(df, rolling_args, *args, **kwargs):\n \"\"\"Create rolling window for the passed frame and execute specified `func` on it.\"\"\"\n if squeeze_self:\n df = df.squeeze(axis=1)\n roller = df.expanding(*rolling_args)\n\n if type(func) is property:\n return func.fget(roller)\n\n return func(roller, *args, **kwargs)\n\n return fn\n\n @classmethod\n def register(cls, func, squeeze_self=False, **kwargs):\n \"\"\"\n Build function that do fallback to pandas to apply `func` on a expanding window.\n\n Parameters\n ----------\n func : callable\n Function to execute on an expanding window.\n squeeze_self : bool, default: False\n Whether or not to squeeze frame before executing the window function.\n **kwargs : kwargs\n Additional arguments that will be passed to function builder.\n\n Returns\n -------\n callable\n Function that takes query compiler and defaults to pandas to apply aggregation\n `func` on an expanding window.\n \"\"\"\n return super().register(\n cls._build_expanding(func, squeeze_self=squeeze_self),\n fn_name=func.__name__,\n **kwargs\n )\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/series.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default Series functions builder class.\"\"\"\n\nfrom .default import DefaultMethod\n\n\nclass SeriesDefault(DefaultMethod):\n \"\"\"Builder for default-to-pandas methods which is executed under Series.\"\"\"\n\n OBJECT_TYPE = \"Series\"\n\n @classmethod\n def frame_wrapper(cls, df):\n \"\"\"\n Squeeze passed DataFrame to be able to process Series-specific functions on it.\n\n Parameters\n ----------\n df : pandas.DataFrame\n One-column DataFrame to squeeze.\n\n Returns\n -------\n pandas.Series\n \"\"\"\n return df.squeeze(axis=1)\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/str.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default applied-on-str functions builder class.\"\"\"\n\nfrom .series import SeriesDefault\n\n\nclass StrDefault(SeriesDefault):\n \"\"\"Builder for default-to-pandas methods which is executed under `str` accessor.\"\"\"\n\n @classmethod\n def frame_wrapper(cls, df):\n \"\"\"\n Get `str` accessor of the passed frame.\n\n Parameters\n ----------\n df : pandas.DataFrame\n\n Returns\n -------\n pandas.core.strings.accessor.StringMethods\n \"\"\"\n return df.squeeze(axis=1).str\n"
},
{
"path": "modin/core/dataframe/algebra/default2pandas/struct.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses default applied-on-struct accessor functions builder class.\"\"\"\n\nfrom .series import SeriesDefault\n\n\nclass StructDefault(SeriesDefault):\n \"\"\"Builder for default-to-pandas methods which is executed under struct accessor.\"\"\"\n\n @classmethod\n def frame_wrapper(cls, df):\n \"\"\"\n Get struct accessor of the passed frame.\n\n Parameters\n ----------\n df : pandas.DataFrame\n\n Returns\n -------\n pandas.core.arrays.arrow.StructAccessor\n \"\"\"\n return df.squeeze(axis=1).struct\n"
},
{
"path": "modin/core/dataframe/algebra/fold.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses builder class for Fold operator.\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import TYPE_CHECKING, Callable, Optional\n\nfrom .operator import Operator\n\nif TYPE_CHECKING:\n import pandas\n\n from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n\n\nclass Fold(Operator):\n \"\"\"Builder class for Fold functions.\"\"\"\n\n @classmethod\n def register(\n cls, fold_function: Callable[..., pandas.DataFrame], shape_preserved=False\n ) -> Callable[..., PandasQueryCompiler]:\n \"\"\"\n Build Fold operator that will be performed across rows/columns.\n\n Parameters\n ----------\n fold_function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.DataFrame\n Function to apply across rows/columns.\n shape_preserved : bool, default: False\n Whether the shape of the dataframe is preserved or not\n after applying a function.\n\n Returns\n -------\n callable\n Function that takes query compiler and executes Fold function.\n \"\"\"\n\n def caller(\n query_compiler: PandasQueryCompiler,\n fold_axis: Optional[int] = None,\n *args: tuple,\n new_index=None,\n new_columns=None,\n **kwargs: dict,\n ) -> PandasQueryCompiler:\n \"\"\"\n Execute Fold function against passed query compiler.\n\n Parameters\n ----------\n query_compiler : PandasQueryCompiler\n The query compiler to execute the function on.\n fold_axis : int, optional\n 0 or None means apply across full column partitions. 1 means\n apply across full row partitions.\n *args : tuple\n Additional arguments passed to `fold_function`.\n new_index : list-like, optional\n The index of the result.\n new_columns : list-like, optional\n The columns of the result.\n **kwargs: dict\n Additional keyword arguments passed to `fold_function`.\n\n Returns\n -------\n PandasQueryCompiler\n A new query compiler representing the result of executing the\n function.\n \"\"\"\n return query_compiler.__constructor__(\n query_compiler._modin_frame.fold(\n cls.validate_axis(fold_axis),\n lambda x: fold_function(x, *args, **kwargs),\n new_index=new_index,\n new_columns=new_columns,\n shape_preserved=shape_preserved,\n )\n )\n\n return caller\n"
},
{
"path": "modin/core/dataframe/algebra/groupby.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses builder class for GroupByReduce operator.\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import TYPE_CHECKING, Callable, Optional, Union\n\nimport pandas\n\nfrom modin.core.dataframe.pandas.metadata import ModinIndex\nfrom modin.error_message import ErrorMessage\nfrom modin.utils import MODIN_UNNAMED_SERIES_LABEL, hashable\n\nfrom .default2pandas.groupby import GroupBy, GroupByDefault\nfrom .tree_reduce import TreeReduce\n\nif TYPE_CHECKING:\n from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n\n\nclass GroupByReduce(TreeReduce):\n \"\"\"\n Builder class for GroupBy aggregation functions.\n\n Attributes\n ----------\n ID_LEVEL_NAME : str\n It's supposed that implementations may produce multiple temporary\n columns per one source column in an intermediate phase. In order\n for these columns to be processed accordingly at the Reduce phase,\n an implementation must store unique names for such temporary\n columns in the ``ID_LEVEL_NAME`` level. Duplicated names are not allowed.\n _GROUPBY_REDUCE_IMPL_FLAG : str\n Attribute indicating that a callable should be treated as an\n implementation for one of the TreeReduce phases rather than an\n arbitrary aggregation. Note: this attribute should be considered private.\n \"\"\"\n\n ID_LEVEL_NAME: str = \"__ID_LEVEL_NAME__\"\n _GROUPBY_REDUCE_IMPL_FLAG: str = \"__groupby_reduce_impl_func__\"\n\n @classmethod\n def register(\n cls,\n map_func: Union[str, dict, Callable[..., pandas.DataFrame]],\n reduce_func: Optional[Union[str, dict, Callable[..., pandas.DataFrame]]] = None,\n **call_kwds: dict,\n ) -> Callable[..., PandasQueryCompiler]:\n \"\"\"\n Build template GroupBy aggregation function.\n\n Resulted function is applied in parallel via TreeReduce algorithm.\n\n Parameters\n ----------\n map_func : str, dict or callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame\n Function to apply to the `GroupByObject` at the map phase. If ``str`` was passed it will\n be treated as a DataFrameGroupBy's method name.\n reduce_func : str, dict or callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame, optional\n Function to apply to the ``DataFrameGroupBy`` at the reduce phase. If not specified\n will be set the same as 'map_func'.\n **call_kwds : dict\n Kwargs that will be passed to the returned function.\n\n Returns\n -------\n callable\n Function that takes query compiler and executes GroupBy aggregation\n with TreeReduce algorithm.\n \"\"\"\n if reduce_func is None:\n reduce_func = map_func\n\n def build_fn(name):\n return lambda df, *args, **kwargs: getattr(df, name)(*args, **kwargs)\n\n if isinstance(map_func, str):\n map_func = build_fn(map_func)\n if isinstance(reduce_func, str):\n reduce_func = build_fn(reduce_func)\n\n assert not (\n isinstance(map_func, dict) ^ isinstance(reduce_func, dict)\n ) and not (\n callable(map_func) ^ callable(reduce_func)\n ), \"Map and reduce functions must be either both dict or both callable.\"\n\n return lambda *args, **kwargs: cls.caller(\n *args, map_func=map_func, reduce_func=reduce_func, **kwargs, **call_kwds\n )\n\n @classmethod\n def register_implementation(\n cls,\n map_func: Callable[..., pandas.DataFrame],\n reduce_func: Callable[..., pandas.DataFrame],\n ) -> None:\n \"\"\"\n Register callables to be recognized as an implementations of tree-reduce phases.\n\n Parameters\n ----------\n map_func : callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame\n Callable to register.\n reduce_func : callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame\n Callable to register.\n \"\"\"\n setattr(map_func, cls._GROUPBY_REDUCE_IMPL_FLAG, True)\n setattr(reduce_func, cls._GROUPBY_REDUCE_IMPL_FLAG, True)\n\n @classmethod\n def map(\n cls,\n df: pandas.DataFrame,\n map_func: Callable[..., pandas.DataFrame],\n axis: int,\n groupby_kwargs: dict,\n agg_args: list,\n agg_kwargs: dict,\n other: Optional[pandas.DataFrame] = None,\n by=None,\n drop: bool = False,\n ) -> pandas.DataFrame:\n \"\"\"\n Execute Map phase of GroupByReduce.\n\n Groups DataFrame and applies map function. Groups will be\n preserved in the results index for the following reduce phase.\n\n Parameters\n ----------\n df : pandas.DataFrame\n Serialized frame to group.\n map_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame\n Function to apply to the `GroupByObject`.\n axis : {0, 1}\n Axis to group and apply aggregation function along. 0 means index axis\n when 1 means column axis.\n groupby_kwargs : dict\n Dictionary which carries arguments for `pandas.DataFrame.groupby`.\n agg_args : list-like\n Positional arguments to pass to the aggregation functions.\n agg_kwargs : dict\n Keyword arguments to pass to the aggregation functions.\n other : pandas.DataFrame, optional\n Serialized frame, whose columns are used to determine the groups.\n If not specified, `by` parameter is used.\n by : level index name or list of such labels, optional\n Index levels, that is used to determine groups.\n If not specified, `other` parameter is used.\n drop : bool, default: False\n Indicates whether or not by-data came from the `self` frame.\n\n Returns\n -------\n pandas.DataFrame\n GroupBy aggregation result for one particular partition.\n \"\"\"\n # Set `as_index` to True to track the metadata of the grouping object\n # It is used to make sure that between phases we are constructing the\n # right index and placing columns in the correct order.\n groupby_kwargs[\"as_index\"] = True\n groupby_kwargs[\"observed\"] = True\n # We have to filter func-dict BEFORE inserting broadcasted 'by' columns\n # to avoid multiple aggregation results for 'by' cols in case they're\n # present in the func-dict:\n apply_func = cls.get_callable(\n map_func,\n df,\n # We won't be able to preserve the order as the Map phase would likely\n # produce some temporary columns that won't fit into the original\n # aggregation order. It doesn't matter much as we restore the original\n # order at the Reduce phase.\n preserve_aggregation_order=False,\n )\n if other is not None:\n # Other is a broadcasted partition that represents 'by' data to group on.\n # If 'drop' then the 'by' data came from the 'self' frame, thus\n # inserting missed columns to the partition to group on them.\n if drop or isinstance(\n other := other.squeeze(axis=axis ^ 1), pandas.DataFrame\n ):\n df = pandas.concat(\n [df] + [other[[o for o in other if o not in df]]],\n axis=1,\n )\n other = list(other.columns)\n by_part = other\n else:\n by_part = by\n\n result = apply_func(\n df.groupby(by=by_part, axis=axis, **groupby_kwargs), *agg_args, **agg_kwargs\n )\n # Result could not always be a frame, so wrapping it into DataFrame\n return pandas.DataFrame(result)\n\n @classmethod\n def reduce(\n cls,\n df: pandas.DataFrame,\n reduce_func: Union[dict, Callable[..., pandas.DataFrame]],\n axis: int,\n groupby_kwargs: dict,\n agg_args: list,\n agg_kwargs: dict,\n partition_idx: int = 0,\n drop: bool = False,\n method: Optional[str] = None,\n finalizer_fn: Optional[Callable[[pandas.DataFrame], pandas.DataFrame]] = None,\n ) -> pandas.DataFrame:\n \"\"\"\n Execute Reduce phase of GroupByReduce.\n\n Combines groups from the Map phase and applies reduce function.\n\n Parameters\n ----------\n df : pandas.DataFrame\n Serialized frame which contain groups to combine.\n reduce_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame\n Function to apply to the `GroupByObject`.\n axis : {0, 1}\n Axis to group and apply aggregation function along. 0 means index axis\n when 1 means column axis.\n groupby_kwargs : dict\n Dictionary which carries arguments for `pandas.DataFrame.groupby`.\n agg_args : list-like\n Positional arguments to pass to the aggregation functions.\n agg_kwargs : dict\n Keyword arguments to pass to the aggregation functions.\n partition_idx : int, default: 0\n Internal index of column partition to which this function is applied.\n drop : bool, default: False\n Indicates whether or not by-data came from the `self` frame.\n method : str, optional\n Name of the groupby function. This is a hint to be able to do special casing.\n finalizer_fn : callable(pandas.DataFrame) -> pandas.DataFrame, optional\n A callable to execute at the end a groupby kernel against groupby result.\n\n Returns\n -------\n pandas.DataFrame\n GroupBy aggregation result.\n \"\"\"\n # Wrapping names into an Index should be unnecessary, however\n # there is a bug in pandas with intersection that forces us to do so:\n # https://github.com/pandas-dev/pandas/issues/39699\n by_part = pandas.Index(df.index.names)\n\n groupby_kwargs = groupby_kwargs.copy()\n as_index = groupby_kwargs.get(\"as_index\", True)\n\n # Set `as_index` to True to track the metadata of the grouping object\n groupby_kwargs[\"as_index\"] = True\n\n # since now index levels contain out 'by', in the reduce phace\n # we want to group on these levels\n groupby_kwargs[\"level\"] = list(range(len(df.index.names)))\n\n apply_func = cls.get_callable(reduce_func, df)\n result = apply_func(\n df.groupby(axis=axis, **groupby_kwargs), *agg_args, **agg_kwargs\n )\n\n if not as_index:\n idx = df.index\n GroupBy.handle_as_index_for_dataframe(\n result,\n by_part,\n by_cols_dtypes=(\n idx.dtypes.values\n if isinstance(idx, pandas.MultiIndex) and hasattr(idx, \"dtypes\")\n else (idx.dtype,)\n ),\n by_length=len(by_part),\n selection=reduce_func.keys() if isinstance(reduce_func, dict) else None,\n partition_idx=partition_idx,\n drop=drop,\n method=method,\n inplace=True,\n )\n # Result could not always be a frame, so wrapping it into DataFrame\n result = pandas.DataFrame(result)\n if result.index.name == MODIN_UNNAMED_SERIES_LABEL:\n result.index.name = None\n\n return result if finalizer_fn is None else finalizer_fn(result)\n\n @classmethod\n def caller(\n cls,\n query_compiler: PandasQueryCompiler,\n by,\n map_func: Union[dict, Callable[..., pandas.DataFrame]],\n reduce_func: Union[dict, Callable[..., pandas.DataFrame]],\n axis: int,\n groupby_kwargs: dict,\n agg_args: list,\n agg_kwargs: dict,\n drop: bool = False,\n method: Optional[str] = None,\n default_to_pandas_func: Optional[Callable[..., pandas.DataFrame]] = None,\n finalizer_fn: Optional[Callable[[pandas.DataFrame], pandas.DataFrame]] = None,\n ) -> PandasQueryCompiler:\n \"\"\"\n Execute GroupBy aggregation with TreeReduce approach.\n\n Parameters\n ----------\n query_compiler : PandasQueryCompiler\n Frame to group.\n by : PandasQueryCompiler, column or index label, Grouper or list of such\n Object that determine groups.\n map_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame\n Function to apply to the `GroupByObject` at the Map phase.\n reduce_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame\n Function to apply to the `GroupByObject` at the Reduce phase.\n axis : {0, 1}\n Axis to group and apply aggregation function along. 0 means index axis\n when 1 means column axis.\n groupby_kwargs : dict\n Dictionary which carries arguments for pandas.DataFrame.groupby.\n agg_args : list-like\n Positional arguments to pass to the aggregation functions.\n agg_kwargs : dict\n Keyword arguments to pass to the aggregation functions.\n drop : bool, default: False\n Indicates whether or not by-data came from the `self` frame.\n method : str, optional\n Name of the GroupBy aggregation function. This is a hint to be able to do special casing.\n default_to_pandas_func : callable(pandas.DataFrameGroupBy) -> pandas.DataFrame, optional\n The pandas aggregation function equivalent to the `map_func + reduce_func`.\n Used in case of defaulting to pandas. If not specified `map_func` is used.\n finalizer_fn : callable(pandas.DataFrame) -> pandas.DataFrame, optional\n A callable to execute at the end a groupby kernel against groupby result.\n\n Returns\n -------\n PandasQueryCompiler\n QueryCompiler which carries the result of GroupBy aggregation.\n \"\"\"\n is_unsupported_axis = axis != 0\n # Defaulting to pandas in case of an empty frame as we can't process it properly.\n # Higher API level won't pass empty data here unless the frame has delayed\n # computations. So we apparently lose some laziness here (due to index access)\n # because of the inability to process empty groupby natively.\n is_empty_data = (\n len(query_compiler.columns) == 0 or len(query_compiler.index) == 0\n )\n is_grouping_using_by_arg = (\n groupby_kwargs.get(\"level\", None) is None and by is not None\n )\n is_unsupported_by_arg = isinstance(by, pandas.Grouper) or (\n not hashable(by) and not isinstance(by, type(query_compiler))\n )\n\n if (\n is_unsupported_axis\n or is_empty_data\n or (is_grouping_using_by_arg and is_unsupported_by_arg)\n ):\n if default_to_pandas_func is None:\n default_to_pandas_func = (\n (lambda grp: grp.agg(map_func))\n if isinstance(map_func, dict)\n else map_func\n )\n default_to_pandas_func = GroupByDefault.register(default_to_pandas_func)\n return default_to_pandas_func(\n query_compiler,\n by=by,\n axis=axis,\n groupby_kwargs=groupby_kwargs,\n agg_args=agg_args,\n agg_kwargs=agg_kwargs,\n drop=drop,\n )\n\n # The bug only occurs in the case of Categorical 'by', so we might want to check whether any of\n # the 'by' dtypes is Categorical before going into this branch, however triggering 'dtypes'\n # computation if they're not computed may take time, so we don't do it\n if not groupby_kwargs.get(\"sort\", True) and isinstance(\n by, type(query_compiler)\n ):\n ErrorMessage.mismatch_with_pandas(\n operation=\"df.groupby(categorical_by, sort=False)\",\n message=(\n \"the groupby keys will be sorted anyway, although the 'sort=False' was passed. \"\n + \"See the following issue for more details: \"\n + \"https://github.com/modin-project/modin/issues/3571\"\n ),\n )\n groupby_kwargs = groupby_kwargs.copy()\n groupby_kwargs[\"sort\"] = True\n\n map_fn, reduce_fn = cls.build_map_reduce_functions(\n by=by,\n axis=axis,\n groupby_kwargs=groupby_kwargs,\n map_func=map_func,\n reduce_func=reduce_func,\n agg_args=agg_args,\n agg_kwargs=agg_kwargs,\n drop=drop,\n method=method,\n finalizer_fn=finalizer_fn,\n )\n\n # If `by` is a ModinFrame, then its partitions will be broadcasted to every\n # `self` partition in a way determined by engine (modin_frame.groupby_reduce)\n # Otherwise `by` was already bound to the Map function in `build_map_reduce_functions`.\n broadcastable_by = getattr(by, \"_modin_frame\", None)\n apply_indices = list(map_func.keys()) if isinstance(map_func, dict) else None\n if (\n broadcastable_by is not None\n and groupby_kwargs.get(\"as_index\", True)\n and broadcastable_by.has_materialized_dtypes\n ):\n new_index = ModinIndex(\n # actual value will be assigned on a parent update\n value=None,\n axis=0,\n dtypes=broadcastable_by.dtypes,\n )\n else:\n new_index = None\n new_modin_frame = query_compiler._modin_frame.groupby_reduce(\n axis,\n broadcastable_by,\n map_fn,\n reduce_fn,\n apply_indices=apply_indices,\n new_index=new_index,\n )\n\n result = query_compiler.__constructor__(new_modin_frame)\n return result\n\n @classmethod\n def get_callable(\n cls,\n agg_func: Union[dict, Callable[..., pandas.DataFrame]],\n df: pandas.DataFrame,\n preserve_aggregation_order: bool = True,\n ) -> Callable[..., pandas.DataFrame]:\n \"\"\"\n Build aggregation function to apply to each group at this particular partition.\n\n If it's dictionary aggregation — filters aggregation dictionary for keys which\n this particular partition contains, otherwise do nothing with passed function.\n\n Parameters\n ----------\n agg_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame\n Aggregation function.\n df : pandas.DataFrame\n Serialized partition which contains available columns.\n preserve_aggregation_order : bool, default: True\n Whether to manually restore the order of columns for the result specified\n by the `agg_func` keys (only makes sense when `agg_func` is a dictionary).\n\n Returns\n -------\n callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame\n Aggregation function that can be safely applied to this particular partition.\n \"\"\"\n if not isinstance(agg_func, dict):\n return agg_func\n\n grp_has_id_level = df.columns.names[0] == cls.ID_LEVEL_NAME\n # The 'id' level prevents us from a lookup for the original\n # partition's columns. So dropping the level.\n partition_columns = frozenset(\n df.columns.droplevel(0) if grp_has_id_level else df.columns\n )\n\n partition_dict = {k: v for k, v in agg_func.items() if k in partition_columns}\n return cls._build_callable_for_dict(\n partition_dict, preserve_aggregation_order, grp_has_id_level\n )\n\n @classmethod\n def _build_callable_for_dict(\n cls,\n agg_dict: dict,\n preserve_aggregation_order: bool = True,\n grp_has_id_level: bool = False,\n ) -> Callable[..., pandas.DataFrame]:\n \"\"\"\n Build callable for an aggregation dictionary.\n\n Parameters\n ----------\n agg_dict : dict\n Aggregation dictionary.\n preserve_aggregation_order : bool, default: True\n Whether to manually restore the order of columns for the result specified\n by the `agg_func` keys (only makes sense when `agg_func` is a dictionary).\n grp_has_id_level : bool, default: False\n Whether the frame we're grouping on has intermediate columns\n (see ``cls.ID_LEVEL_NAME``).\n\n Returns\n -------\n callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame\n \"\"\"\n # We have to keep this import away from the module level to avoid circular import\n from modin.pandas.utils import walk_aggregation_dict\n\n # We now filter aggregation functions into those that could be applied natively\n # using pandas (pandas_grp_obj.agg(**native_aggs)) and those that require\n # special treatment (custom_aggs).\n custom_aggs = {}\n native_aggs = {}\n\n result_columns = []\n for col, func, func_name, col_renaming_required in walk_aggregation_dict(\n agg_dict\n ):\n # Filter dictionary\n dict_to_add = (\n custom_aggs if cls.is_registered_implementation(func) else native_aggs\n )\n\n new_value = func if func_name is None else (func_name, func)\n old_value = dict_to_add.get(col, None)\n\n if old_value is not None:\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=not isinstance(old_value, list),\n extra_log=\"Expected for all aggregation values to be a list when at least \"\n + f\"one column has multiple aggregations. Got: {old_value} {type(old_value)}\",\n )\n old_value.append(new_value)\n else:\n # Pandas knows that it has to modify the resulting columns if it meets\n # a function wrapped into a list. Renaming is required if either a new\n # column name was explicitly specified, or multiple functions were\n # specified per one column, or if any other column in the aggregation\n # is going to be renamed.\n dict_to_add[col] = [new_value] if col_renaming_required else new_value\n\n # Construct resulting columns\n if col_renaming_required:\n func_name = str(func) if func_name is None else func_name\n result_columns.append(\n (*(col if isinstance(col, tuple) else (col,)), func_name)\n )\n else:\n result_columns.append(col)\n\n result_columns = pandas.Index(result_columns)\n\n def aggregate_on_dict(grp_obj, *args, **kwargs):\n \"\"\"Aggregate the passed groupby object.\"\"\"\n if len(native_aggs) == 0:\n native_agg_res = None\n elif grp_has_id_level:\n # Adding the 'id' level to the aggregation keys so they match `grp_obj` columns\n native_aggs_modified = {\n (\n cls.ID_LEVEL_NAME,\n *(key if isinstance(key, tuple) else (key,)),\n ): value\n for key, value in native_aggs.items()\n }\n native_agg_res = grp_obj.agg(native_aggs_modified)\n # Dropping the 'id' level from the resulted frame\n native_agg_res.columns = native_agg_res.columns.droplevel(0)\n else:\n native_agg_res = grp_obj.agg(native_aggs)\n\n custom_results = []\n insert_id_levels = False\n\n for col, func, func_name, col_renaming_required in walk_aggregation_dict(\n custom_aggs\n ):\n if grp_has_id_level:\n cols_without_ids = grp_obj.obj.columns.droplevel(0)\n if isinstance(cols_without_ids, pandas.MultiIndex):\n # We may have multiple columns matching the `col` in\n # a MultiIndex case, that's why use `.get_locs` here\n col_pos = cols_without_ids.get_locs(col)\n else:\n # `pandas.Index` doesn't have `.get_locs` method\n col_pos = cols_without_ids.get_loc(col)\n agg_key = grp_obj.obj.columns[col_pos]\n else:\n agg_key = [col]\n\n result = func(grp_obj[agg_key])\n # The `func` may have discarded an ID-level if there were any.\n # So checking for this again.\n result_has_id_level = result.columns.names[0] == cls.ID_LEVEL_NAME\n insert_id_levels |= result_has_id_level\n\n if col_renaming_required:\n func_name = str(func) if func_name is None else func_name\n if result_has_id_level:\n result.columns = pandas.MultiIndex.from_tuples(\n [\n # `old_col[0]` stores values from the 'id'\n # level, the ones we want to preserve here\n (old_col[0], col, func_name)\n for old_col in result.columns\n ],\n names=[\n result.columns.names[0],\n result.columns.names[1],\n None,\n ],\n )\n else:\n result.columns = pandas.MultiIndex.from_tuples(\n [(col, func_name)] * len(result.columns),\n names=[result.columns.names[0], None],\n )\n\n custom_results.append(result)\n\n if insert_id_levels:\n # As long as any `result` has an id-level we have to insert the level\n # into every `result` so the number of levels matches\n for idx, ext_result in enumerate(custom_results):\n if ext_result.columns.names[0] != cls.ID_LEVEL_NAME:\n custom_results[idx] = pandas.concat(\n [ext_result],\n keys=[cls.ID_LEVEL_NAME],\n names=[cls.ID_LEVEL_NAME],\n axis=1,\n copy=False,\n )\n\n if native_agg_res is not None:\n native_agg_res = pandas.concat(\n [native_agg_res],\n keys=[cls.ID_LEVEL_NAME],\n names=[cls.ID_LEVEL_NAME],\n axis=1,\n copy=False,\n )\n\n native_res_part = [] if native_agg_res is None else [native_agg_res]\n parts = [*native_res_part, *custom_results]\n if parts:\n result = pandas.concat(parts, axis=1, copy=False)\n else:\n result = pandas.DataFrame(columns=result_columns)\n\n # The order is naturally preserved if there's no custom aggregations\n if preserve_aggregation_order and len(custom_aggs):\n result = result.reindex(result_columns, axis=1)\n return result\n\n return aggregate_on_dict\n\n @classmethod\n def is_registered_implementation(cls, func: Callable) -> bool:\n \"\"\"\n Check whether the passed `func` was registered as a TreeReduce implementation.\n\n Parameters\n ----------\n func : callable\n\n Returns\n -------\n bool\n \"\"\"\n return callable(func) and hasattr(func, cls._GROUPBY_REDUCE_IMPL_FLAG)\n\n @classmethod\n def build_map_reduce_functions(\n cls,\n by,\n axis: int,\n groupby_kwargs: dict,\n map_func: Union[dict, Callable[..., pandas.DataFrame]],\n reduce_func: Union[dict, Callable[..., pandas.DataFrame]],\n agg_args: list,\n agg_kwargs: dict,\n drop: bool = False,\n method: Optional[str] = None,\n finalizer_fn: Callable[[pandas.DataFrame], pandas.DataFrame] = None,\n ) -> tuple[Callable, Callable]:\n \"\"\"\n Bind appropriate arguments to map and reduce functions.\n\n Parameters\n ----------\n by : BaseQueryCompiler, column or index label, Grouper or list of such\n Object that determine groups.\n axis : {0, 1}\n Axis to group and apply aggregation function along. 0 means index axis\n when 1 means column axis.\n groupby_kwargs : dict\n Dictionary which carries arguments for pandas.DataFrame.groupby.\n map_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame\n Function to apply to the `GroupByObject` at the Map phase.\n reduce_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame\n Function to apply to the `GroupByObject` at the Reduce phase.\n agg_args : list-like\n Positional arguments to pass to the aggregation functions.\n agg_kwargs : dict\n Keyword arguments to pass to the aggregation functions.\n drop : bool, default: False\n Indicates whether or not by-data came from the `self` frame.\n method : str, optional\n Name of the GroupBy aggregation function. This is a hint to be able to do special casing.\n finalizer_fn : callable(pandas.DataFrame) -> pandas.DataFrame, optional\n A callable to execute at the end a groupby kernel against groupby result.\n\n Returns\n -------\n Tuple of callable\n Tuple of map and reduce functions with bound arguments.\n \"\"\"\n # if by is a query compiler, then it will be broadcasted explicit via\n # groupby_reduce method of the modin frame and so we don't want secondary\n # implicit broadcastion via passing it as an function argument.\n if hasattr(by, \"_modin_frame\"):\n by = None\n\n def _map(\n df: pandas.DataFrame,\n other: Optional[pandas.DataFrame] = None,\n **kwargs: dict,\n ) -> pandas.DataFrame:\n def wrapper(\n df: pandas.DataFrame, other: Optional[pandas.DataFrame] = None\n ) -> pandas.DataFrame:\n return cls.map(\n df,\n other=other,\n axis=axis,\n by=by,\n groupby_kwargs=groupby_kwargs.copy(),\n map_func=map_func,\n agg_args=agg_args,\n agg_kwargs=agg_kwargs,\n drop=drop,\n **kwargs,\n )\n\n try:\n result = wrapper(df, other)\n # This will happen with Arrow buffer read-only errors. We don't want to copy\n # all the time, so this will try to fast-path the code first.\n except ValueError:\n result = wrapper(df.copy(), other if other is None else other.copy())\n return result\n\n def _reduce(df: pandas.DataFrame, **call_kwargs: dict) -> pandas.DataFrame:\n def wrapper(df: pandas.DataFrame):\n return cls.reduce(\n df,\n axis=axis,\n groupby_kwargs=groupby_kwargs,\n reduce_func=reduce_func,\n agg_args=agg_args,\n agg_kwargs=agg_kwargs,\n drop=drop,\n method=method,\n finalizer_fn=finalizer_fn,\n **call_kwargs,\n )\n\n try:\n result = wrapper(df)\n # This will happen with Arrow buffer read-only errors. We don't want to copy\n # all the time, so this will try to fast-path the code first.\n except ValueError:\n result = wrapper(df.copy())\n return result\n\n return _map, _reduce\n"
},
{
"path": "modin/core/dataframe/algebra/map.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses builder class for Map operator.\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import TYPE_CHECKING, Callable\n\nfrom .operator import Operator\n\nif TYPE_CHECKING:\n import pandas\n\n from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n\n\nclass Map(Operator):\n \"\"\"Builder class for Map operator.\"\"\"\n\n @classmethod\n def register(\n cls,\n function: Callable[..., pandas.DataFrame],\n *call_args: tuple,\n **call_kwds: dict,\n ) -> Callable[..., PandasQueryCompiler]:\n \"\"\"\n Build Map operator that will be performed across each partition.\n\n Parameters\n ----------\n function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.DataFrame\n Function that will be applied to the each partition.\n Function takes `pandas.DataFrame` and returns `pandas.DataFrame`\n of the same shape.\n *call_args : tuple\n Args that will be passed to the returned function.\n **call_kwds : dict\n Kwargs that will be passed to the returned function.\n\n Returns\n -------\n callable\n Function that takes query compiler and executes map function.\n \"\"\"\n\n def caller(\n query_compiler: PandasQueryCompiler, *args: tuple, **kwargs: dict\n ) -> PandasQueryCompiler:\n \"\"\"Execute Map function against passed query compiler.\"\"\"\n shape_hint = call_kwds.pop(\"shape_hint\", None) or query_compiler._shape_hint\n return query_compiler.__constructor__(\n query_compiler._modin_frame.map(\n lambda x: function(x, *args, **kwargs), *call_args, **call_kwds\n ),\n shape_hint=shape_hint,\n )\n\n return caller\n"
},
{
"path": "modin/core/dataframe/algebra/operator.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module contains an interface for operator builder classes.\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import Callable, Optional\n\n\nclass Operator(object):\n \"\"\"Interface for building operators that can execute in parallel across partitions.\"\"\"\n\n def __init__(self) -> None:\n raise ValueError(\n \"Please use {}.register instead of the constructor\".format(\n type(self).__name__\n )\n )\n\n @classmethod\n def register(cls, func: Callable, **kwargs: dict):\n \"\"\"\n Build operator that applies source function across the entire dataset.\n\n Parameters\n ----------\n func : callable\n Source function.\n **kwargs : dict\n Kwargs that will be passed to the builder function.\n\n Returns\n -------\n callable\n \"\"\"\n raise NotImplementedError(\"Please implement in child class\")\n\n @classmethod\n def validate_axis(cls, axis: Optional[int]) -> int:\n \"\"\"\n Ensure that axis to apply function on has valid value.\n\n Parameters\n ----------\n axis : int, optional\n 0 or None means apply on index, 1 means apply on columns.\n\n Returns\n -------\n int\n Integer representation of given axis.\n \"\"\"\n return 0 if axis is None else axis\n"
},
{
"path": "modin/core/dataframe/algebra/reduce.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses builder class for Reduce operator.\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import TYPE_CHECKING, Callable, Optional\n\nfrom .operator import Operator\n\nif TYPE_CHECKING:\n import pandas\n\n from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n\n\nclass Reduce(Operator):\n \"\"\"Builder class for Reduce operator.\"\"\"\n\n @classmethod\n def register(\n cls,\n reduce_function: Callable[..., pandas.Series],\n axis: Optional[int] = None,\n shape_hint: Optional[str] = None,\n ) -> Callable[..., PandasQueryCompiler]:\n \"\"\"\n Build Reduce operator that will be performed across rows/columns.\n\n It's used if `func` reduces the dimension of partitions in contrast to `Fold`.\n\n Parameters\n ----------\n reduce_function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.Series\n Source function.\n axis : int, optional\n Axis to apply function along.\n shape_hint : {\"row\", \"column\", None}, default: None\n Shape hint for the results known to be a column or a row, otherwise None.\n\n Returns\n -------\n callable\n Function that takes query compiler and executes Reduce function.\n \"\"\"\n\n def caller(\n query_compiler: PandasQueryCompiler, *args: tuple, **kwargs: dict\n ) -> PandasQueryCompiler:\n \"\"\"Execute Reduce function against passed query compiler.\"\"\"\n _axis = kwargs.get(\"axis\") if axis is None else axis\n return query_compiler.__constructor__(\n query_compiler._modin_frame.reduce(\n cls.validate_axis(_axis),\n lambda x: reduce_function(x, *args, **kwargs),\n ),\n shape_hint=shape_hint,\n )\n\n return caller\n"
},
{
"path": "modin/core/dataframe/algebra/tree_reduce.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses builder class for TreeReduce operator.\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import TYPE_CHECKING, Callable, Optional\n\nfrom .operator import Operator\n\nif TYPE_CHECKING:\n import pandas\n from pandas._typing import DtypeObj\n\n from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n\n\nclass TreeReduce(Operator):\n \"\"\"Builder class for TreeReduce operator.\"\"\"\n\n @classmethod\n def register(\n cls,\n map_function: Optional[Callable[..., pandas.DataFrame]],\n reduce_function: Optional[Callable[..., pandas.Series]] = None,\n axis: Optional[int] = None,\n compute_dtypes: Optional[Callable[..., DtypeObj]] = None,\n ) -> Callable[..., PandasQueryCompiler]:\n \"\"\"\n Build TreeReduce operator.\n\n Parameters\n ----------\n map_function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.DataFrame\n Source map function.\n reduce_function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.Series, optional\n Source reduce function.\n axis : int, optional\n Specifies axis to apply function along.\n compute_dtypes : callable(pandas.Series, *func_args, **func_kwargs) -> DtypeObj, optional\n Callable for computing dtypes.\n\n Returns\n -------\n callable\n Function that takes query compiler and executes passed functions\n with TreeReduce algorithm.\n \"\"\"\n if reduce_function is None:\n reduce_function = map_function\n\n def caller(\n query_compiler: PandasQueryCompiler, *args: tuple, **kwargs: dict\n ) -> PandasQueryCompiler:\n \"\"\"Execute TreeReduce function against passed query compiler.\"\"\"\n _axis = kwargs.get(\"axis\") if axis is None else axis\n\n new_dtypes = None\n if compute_dtypes and query_compiler.frame_has_materialized_dtypes:\n new_dtypes = str(compute_dtypes(query_compiler.dtypes, *args, **kwargs))\n\n return query_compiler.__constructor__(\n query_compiler._modin_frame.tree_reduce(\n cls.validate_axis(_axis),\n lambda x: map_function(x, *args, **kwargs),\n lambda y: reduce_function(y, *args, **kwargs),\n dtypes=new_dtypes,\n )\n )\n\n return caller\n"
},
{
"path": "modin/core/dataframe/base/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe classes.\"\"\"\n"
},
{
"path": "modin/core/dataframe/base/dataframe/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe class and Axis and JoinType Enums.\"\"\"\n"
},
{
"path": "modin/core/dataframe/base/dataframe/dataframe.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nModule contains class ModinDataframe.\n\nModinDataframe is a parent abstract class for any dataframe class.\n\"\"\"\n\nfrom abc import ABC, abstractmethod\nfrom typing import Callable, Dict, Hashable, List, Optional, Union\n\nfrom modin.core.dataframe.base.dataframe.utils import Axis, JoinType\n\n\nclass ModinDataframe(ABC):\n \"\"\"\n An abstract class that represents the Parent class for any Dataframe class.\n\n This class is intended to specify the behaviors that a Dataframe must implement.\n\n For more details about how these methods were chosen, please refer to this\n (https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf) paper, which specifies\n a Dataframe algebra that this class exposes.\n \"\"\"\n\n @abstractmethod\n def take_2d_labels_or_positional(\n self,\n row_labels: Optional[List[Hashable]] = None,\n row_positions: Optional[List[int]] = None,\n col_labels: Optional[List[Hashable]] = None,\n col_positions: Optional[List[int]] = None,\n ) -> \"ModinDataframe\":\n \"\"\"\n Mask rows and columns in the dataframe.\n\n Allow users to perform selection and projection on the row and column labels (named notation),\n in addition to the row and column number (positional notation).\n\n Parameters\n ----------\n row_labels : list of hashable, optional\n The row labels to extract.\n row_positions : list of int, optional\n The row positions to extract.\n col_labels : list of hashable, optional\n The column labels to extract.\n col_positions : list of int, optional\n The column positions to extract.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe from the mask provided.\n\n Notes\n -----\n If both `row_labels` and `row_positions` are provided, a ValueError is raised.\n The same rule applies for `col_labels` and `col_positions`.\n \"\"\"\n pass\n\n @abstractmethod\n def filter_by_types(self, types: List[Hashable]) -> \"ModinDataframe\":\n \"\"\"\n Allow the user to specify a type or set of types by which to filter the columns.\n\n Parameters\n ----------\n types : list of hashables\n The types to filter columns by.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with only the columns whose dtypes appear in `types`.\n \"\"\"\n pass\n\n @abstractmethod\n def map(\n self,\n function: Callable,\n axis: Optional[Union[int, Axis]] = None,\n dtypes: Optional[str] = None,\n new_columns: Optional[List[Hashable]] = None,\n ) -> \"ModinDataframe\":\n \"\"\"\n Apply a user-defined function row-wise if `axis`=0, column-wise if `axis`=1, and cell-wise if `axis` is None.\n\n Parameters\n ----------\n function : callable(row|col|cell) -> row|col|cell\n The function to map across the dataframe.\n axis : int or modin.core.dataframe.base.utils.Axis, optional\n The axis to map over.\n dtypes : str, optional\n The data types for the result. This is an optimization\n because there are functions that always result in a particular data\n type, and this allows us to avoid (re)computing it.\n new_columns : List[Hashable], optional\n New column labels of the result, its length has to be identical\n to the older columns. If not specified, old column labels are preserved.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the map applied.\n\n Notes\n -----\n This does not change the shape of the dataframe.\n \"\"\"\n pass\n\n @abstractmethod\n def filter(self, axis: Union[int, Axis], condition: Callable) -> \"ModinDataframe\":\n \"\"\"\n Filter data based on the function provided along the specified axis.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to filter over.\n condition : callable(row|col) -> bool\n The function to use for the filter. This function should filter the\n data itself. It accepts either a row or column (depending on the axis argument) and\n returns True to keep the row/col, and False to drop it.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe filtered by content according to the filter provided by condition.\n \"\"\"\n pass\n\n @abstractmethod\n def explode(\n self,\n axis: Union[int, Axis],\n function: Callable,\n result_schema: Optional[Dict[Hashable, type]] = None,\n ) -> \"ModinDataframe\":\n \"\"\"\n Explode data based on the function provided along the specified axis.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to expand over.\n function : callable\n The function to use to expand the data. This function should accept one\n row/column, and return multiple.\n result_schema : dictionary, optional\n Mapping from column labels to data types that represents the types of the output dataframe.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the specified axis expanded.\n\n Notes\n -----\n Only one axis can be expanded at a time.\n\n The user-defined function may increase the number of rows (columns if axis=1),\n but it should not remove or drop rows.\n \"\"\"\n pass\n\n @abstractmethod\n def window(\n self,\n axis: Union[int, Axis],\n reduce_fn: Callable,\n window_size: int,\n result_schema: Optional[Dict[Hashable, type]] = None,\n ) -> \"ModinDataframe\":\n \"\"\"\n Apply a sliding window operator that acts as a GROUPBY on each window, reducing each window to a single row (column).\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to slide over.\n reduce_fn : callable(rowgroup|colgroup) -> row|col\n The reduce function to apply over the data.\n window_size : int\n The number of row/columns to pass to the function.\n (The size of the sliding window).\n result_schema : dictionary, optional\n Mapping from column labels to data types that represents the types of the output dataframe.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the reduce function applied over windows of the specified\n axis.\n\n Notes\n -----\n The user-defined reduce function must reduce each window's column\n (row if axis=1) down to a single value.\n \"\"\"\n pass\n\n @abstractmethod\n def groupby(\n self,\n axis: Union[int, Axis],\n by: Union[str, List[str]],\n operator: Callable,\n result_schema: Optional[Dict[Hashable, type]] = None,\n ) -> \"ModinDataframe\":\n \"\"\"\n Generate groups based on values in the input column(s) and perform the specified operation on each.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to apply the grouping over.\n by : string or list of strings\n One or more column labels to use for grouping.\n operator : callable\n The operation to carry out on each of the groups. The operator is another\n algebraic operator with its own user-defined function parameter, depending\n on the output desired by the user.\n result_schema : dictionary, optional\n Mapping from column labels to data types that represents the types of the output dataframe.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe containing the groupings specified, with the operator\n applied to each group.\n\n Notes\n -----\n No communication between groups is allowed in this algebra implementation.\n\n The number of rows (columns if axis=1) returned by the user-defined function\n passed to the groupby may be at most the number of rows in the group, and\n may be as small as a single row.\n\n Unlike the pandas API, an intermediate \"GROUP BY\" object is not present in this\n algebra implementation.\n \"\"\"\n pass\n\n @abstractmethod\n def reduce(\n self,\n axis: Union[int, Axis],\n function: Callable,\n dtypes: Optional[str] = None,\n ) -> \"ModinDataframe\":\n \"\"\"\n Perform a user-defined aggregation on the specified axis, where the axis reduces down to a singleton.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to perform the reduce over.\n function : callable(row|col) -> single value\n The reduce function to apply to each column.\n dtypes : str, optional\n The data types for the result. This is an optimization\n because there are functions that always result in a particular data\n type, and this allows us to avoid (re)computing it.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the same columns as the previous, with only a single row.\n\n Notes\n -----\n The user-defined function must reduce to a single value.\n \"\"\"\n pass\n\n @abstractmethod\n def tree_reduce(\n self,\n axis: Union[int, Axis],\n map_func: Callable,\n reduce_func: Optional[Callable] = None,\n dtypes: Optional[str] = None,\n ) -> \"ModinDataframe\":\n \"\"\"\n Perform a user-defined aggregation on the specified axis, where the axis reduces down to a singleton using a tree-reduce computation pattern.\n\n The map function is applied first over multiple partitions of a column, and then the reduce\n function (if specified, otherwise the map function is applied again) is applied to the\n results to produce a single value.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to perform the tree reduce over.\n map_func : callable(row|col) -> row|col|single value\n The map function to apply to each column.\n reduce_func : callable(row|col) -> single value, optional\n The reduce function to apply to the results of the map function.\n dtypes : str, optional\n The data types for the result. This is an optimization\n because there are functions that always result in a particular data\n type, and this allows us to avoid (re)computing it.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the same columns as the previous, with only a single row.\n\n Notes\n -----\n The user-defined function must reduce to a single value.\n\n If the user-defined function requires access to the entire column, please use reduce instead.\n \"\"\"\n pass\n\n @abstractmethod\n def infer_types(self, columns_list: List[str]) -> \"ModinDataframe\":\n \"\"\"\n Determine the compatible type shared by all values in the specified columns, and coerce them to that type.\n\n Parameters\n ----------\n columns_list : list of strings\n List of column labels to infer and induce types over.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the inferred schema.\n \"\"\"\n pass\n\n @abstractmethod\n def join(\n self,\n axis: Union[int, Axis],\n condition: Callable,\n other: \"ModinDataframe\",\n join_type: Union[str, JoinType],\n ) -> \"ModinDataframe\":\n \"\"\"\n Join this dataframe with the other.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to perform the join on.\n condition : callable\n Function that determines which rows should be joined. The condition can be a\n simple equality, e.g. \"left.col1 == right.col1\" or can be arbitrarily complex.\n other : ModinDataframe\n The other data to join with, i.e. the right dataframe.\n join_type : string {\"inner\", \"left\", \"right\", \"outer\"} or modin.core.dataframe.base.utils.JoinType\n The type of join to perform.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe that is the result of applying the specified join over the two\n dataframes.\n\n Notes\n -----\n During the join, this dataframe is considered the left, while the other is\n treated as the right.\n\n Only inner joins, left outer, right outer, and full outer joins are currently supported.\n Support for other join types (e.g. natural join) may be implemented in the future.\n \"\"\"\n pass\n\n @abstractmethod\n def concat(\n self,\n axis: Union[int, Axis],\n others: Union[\"ModinDataframe\", List[\"ModinDataframe\"]],\n ) -> \"ModinDataframe\":\n \"\"\"\n Append rows/columns along the specified axis from multiple dataframes.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis on which to perform the concatenation.\n others : ModinDataframe or list of ModinDataframes\n The other ModinDataframe(s) to concatenate.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe that is the result of concatenating the dataframes over the\n specified axis.\n\n Notes\n -----\n The concat operator incurs fixed overheads, and so this algebra places no\n limit to the number of dataframes that may be concatenated in this way.\n \"\"\"\n pass\n\n @abstractmethod\n def transpose(self) -> \"ModinDataframe\":\n \"\"\"\n Swap the row and column axes.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the row and column axes swapped.\n\n Notes\n -----\n Transposing a dataframe is expensive, and so it is performed lazily. The axes are swapped\n logically immediately, but the physical swap does not occur until absolutely necessary,\n which helps motivate the axis argument to the other operators in this algebra.\n \"\"\"\n pass\n\n @abstractmethod\n def to_labels(self, column_labels: Union[str, List[str]]) -> \"ModinDataframe\":\n \"\"\"\n Replace the row labels with one or more columns of data.\n\n Parameters\n ----------\n column_labels : string or list of strings\n Column label(s) to use as the new row labels.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the row labels replaced by the specified columns.\n\n Notes\n -----\n When multiple column labels are specified, a hierarchical set of labels is created, ordered by the ordering\n of labels in the input.\n \"\"\"\n pass\n\n @abstractmethod\n def from_labels(self) -> \"ModinDataframe\":\n \"\"\"\n Move the row labels into the data at position 0, and sets the row labels to the positional notation.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the row labels moved into the data.\n\n Notes\n -----\n In the case that the dataframe has hierarchical labels, all label \"levels\" are inserted into the dataframe\n in the order they occur in the labels, with the outermost being in position 0.\n \"\"\"\n pass\n\n @abstractmethod\n def rename(\n self,\n new_row_labels: Optional[Union[Dict[Hashable, Hashable], Callable]] = None,\n new_col_labels: Optional[Union[Dict[Hashable, Hashable], Callable]] = None,\n ) -> \"ModinDataframe\":\n \"\"\"\n Replace the row and column labels with the specified new labels.\n\n Parameters\n ----------\n new_row_labels : dictionary or callable, optional\n Mapping or callable that relates old row labels to new labels.\n new_col_labels : dictionary or callable, optional\n Mapping or callable that relates old col labels to new labels.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe with the new row and column labels.\n \"\"\"\n pass\n\n @abstractmethod\n def sort_by(\n self,\n axis: Union[int, Axis],\n labels: Union[str, List[str]],\n ascending: bool = True,\n ) -> \"ModinDataframe\":\n \"\"\"\n Logically reorder rows (columns if axis=1) lexicographically by the data in a column or set of columns.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to perform the sort over.\n labels : string or list of strings\n Column (row if axis=1) label(s) to use to determine lexicographical ordering. If multiple\n columns (rows if axis=1) are provided, the sort is performed on the first column (row if axis=1),\n with ties broken by the other columns (rows if axis=1) provided.\n ascending : boolean, default: True\n Whether to sort in ascending or descending order.\n\n Returns\n -------\n ModinDataframe\n A new ModinDataframe sorted into lexicographical order by the specified column(s).\n \"\"\"\n pass\n"
},
{
"path": "modin/core/dataframe/base/dataframe/utils.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nModule contains useful enums for Modin.\n\nAxis is an enum that represents the `axis` argument for dataframe operations.\nJoinType is an enum that represents the `join_type` or `how` argument for the join algebra operator.\n\"\"\"\n\nfrom enum import Enum\nfrom typing import Dict, List, Sequence, Tuple, cast\n\nimport pandas\nfrom pandas._typing import IndexLabel\nfrom pandas.api.types import is_scalar\nfrom pandas.core.dtypes.common import is_integer_dtype\n\n\nclass Axis(Enum): # noqa: PR01\n \"\"\"\n An enum that represents the `axis` argument provided to the algebra operators.\n\n The enum has 3 values - ROW_WISE to represent the row axis, COL_WISE to represent the\n column axis, and CELL_WISE to represent no axis. ROW_WISE operations iterate over the rows\n COL_WISE operations over the columns, and CELL_WISE operations over any of the partitioning\n schemes that are supported in Modin (row-wise, column-wise, or block-wise).\n \"\"\"\n\n ROW_WISE = 0\n COL_WISE = 1\n CELL_WISE = None\n\n\nclass JoinType(Enum): # noqa: PR01\n \"\"\"\n An enum that represents the `join_type` argument provided to the algebra operators.\n\n The enum has 4 values - INNER to represent inner joins, LEFT to represent left joins, RIGHT to\n represent right joins, and OUTER to represent outer joins.\n \"\"\"\n\n INNER = \"inner\"\n LEFT = \"left\"\n RIGHT = \"right\"\n OUTER = \"outer\"\n\n\ndef join_columns(\n left: pandas.Index,\n right: pandas.Index,\n left_on: IndexLabel,\n right_on: IndexLabel,\n suffixes: Tuple[str, str],\n) -> Tuple[pandas.Index, Dict[IndexLabel, IndexLabel], Dict[IndexLabel, IndexLabel]]:\n \"\"\"\n Compute resulting columns for the two dataframes being merged.\n\n Parameters\n ----------\n left : pandas.Index\n Columns of the left frame to join.\n right : pandas.Index\n Columns of the right frame to join.\n left_on : list-like or scalar\n Column names on which the frames are joined in the left DataFrame.\n right_on : list-like or scalar\n Column names on which the frames are joined in the right DataFrame.\n suffixes : tuple[str, str]\n A 2-length sequence containing suffixes to append to the intersected columns.\n\n Returns\n -------\n pandas.Index, dict[IndexLabel -> IndexLabel], dict[IndexLabel -> IndexLabel]\n Returns columns for the resulting frame and mappings of old to new column\n names for `left` and `right` accordingly.\n\n Raises\n ------\n NotImplementedError\n Raised when one of the keys to join is an index level, pandas behaviour is really\n complicated in this case, so we're not supporting this case for now.\n \"\"\"\n # using `cast` to make `mypy` acknowledged that the variable now ensured to be `Sequence[IndexLabel]`\n left_on = cast(Sequence[IndexLabel], [left_on] if is_scalar(left_on) else left_on)\n right_on = cast(\n Sequence[IndexLabel], [right_on] if is_scalar(right_on) else right_on\n )\n\n # handling a simple case of merging on one column and when the column is located in an index\n if len(left_on) == 1 and len(right_on) == 1 and left_on[0] == right_on[0]:\n if left_on[0] not in left and right_on[0] not in right:\n # in this case the 'on' column will stay in the index, so we can simply\n # drop the 'left/right_on' values and proceed as normal\n left_on = []\n right_on = []\n # in other cases, we can simply add the index name to columns and proceed as normal\n # on python 3.9 with pandas-stubs 2.2, these lines will warn about insert being an untyped call,\n # but this error is no longer present on higher versions\n elif left_on[0] not in left:\n left = left.insert(loc=0, item=left_on[0]) # type: ignore[no-untyped-call, unused-ignore]\n elif right_on[0] not in right:\n right = right.insert(loc=0, item=right_on[0]) # type: ignore[no-untyped-call, unused-ignore]\n\n if any(col not in left for col in left_on) or any(\n col not in right for col in right_on\n ):\n raise NotImplementedError(\n \"Cases, where one of the keys to join is an index level, are not yet supported.\"\n )\n\n left_conflicts = set(left) & (set(right) - set(right_on))\n right_conflicts = set(right) & (set(left) - set(left_on))\n conflicting_cols = left_conflicts | right_conflicts\n\n def _get_new_name(col: IndexLabel, suffix: str) -> IndexLabel:\n if col in conflicting_cols:\n return (\n (f\"{col[0]}{suffix}\", *col[1:])\n if isinstance(col, tuple)\n else f\"{col}{suffix}\"\n )\n else:\n return col\n\n left_renamer: Dict[IndexLabel, IndexLabel] = {}\n right_renamer: Dict[IndexLabel, IndexLabel] = {}\n new_left: List = []\n new_right: List = []\n\n for col in left:\n new_name = _get_new_name(col, suffixes[0])\n new_left.append(new_name)\n left_renamer[col] = new_name\n\n for col in right:\n # If we're joining on the column that exists in both frames then it was already\n # taken from the 'left', don't want to take it again from the 'right'.\n if not (col in left_on and col in right_on):\n new_name = _get_new_name(col, suffixes[1])\n new_right.append(new_name)\n right_renamer[col] = new_name\n\n new_columns = pandas.Index(new_left + new_right)\n return new_columns, left_renamer, right_renamer\n\n\ndef is_trivial_index(index: pandas.Index) -> bool:\n \"\"\"\n Check if the index is a trivial index, i.e. a sequence [0..n].\n\n Parameters\n ----------\n index : pandas.Index\n An index to check.\n\n Returns\n -------\n bool\n \"\"\"\n if len(index) == 0:\n return True\n if isinstance(index, pandas.RangeIndex):\n return index.start == 0 and index.step == 1\n if not (isinstance(index, pandas.Index) and is_integer_dtype(index)):\n return False\n return (\n index.is_monotonic_increasing\n and index.is_unique\n and index.min() == 0\n and index.max() == len(index) - 1\n )\n"
},
{
"path": "modin/core/dataframe/base/interchange/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe functionality related to data exchange protocols.\"\"\"\n"
},
{
"path": "modin/core/dataframe/base/interchange/dataframe_protocol/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nBase Modin Dataframe functionality related to the dataframe exchange protocol.\n\nSee more in https://data-apis.org/dataframe-protocol/latest/index.html.\n\"\"\"\n"
},
{
"path": "modin/core/dataframe/base/interchange/dataframe_protocol/dataframe.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nDataframe exchange protocol implementation.\n\nSee more in https://data-apis.org/dataframe-protocol/latest/index.html.\n\"\"\"\n\nfrom abc import ABC, abstractmethod\nfrom typing import Any, Dict, Iterable, Optional, Sequence, Tuple, TypedDict\n\nfrom .utils import ColumnNullType, DlpackDeviceType, DTypeKind\n\n\nclass ColumnBuffers(TypedDict): # noqa: GL08\n # first element is a buffer containing the column data;\n # second element is the data buffer's associated dtype\n data: Tuple[\"ProtocolBuffer\", Any]\n\n # first element is a buffer containing mask values indicating missing data;\n # second element is the mask value buffer's associated dtype.\n # None if the null representation is not a bit or byte mask\n validity: Optional[Tuple[\"ProtocolBuffer\", Any]]\n\n # first element is a buffer containing the offset values for\n # variable-size binary data (e.g., variable-length strings);\n # second element is the offsets buffer's associated dtype.\n # None if the data buffer does not have an associated offsets buffer\n offsets: Optional[Tuple[\"ProtocolBuffer\", Any]]\n\n\nclass CategoricalDescription(TypedDict): # noqa: GL08\n # whether the ordering of dictionary indices is semantically meaningful\n is_ordered: bool\n # whether a column-style mapping of categorical values to other objects exists\n is_dictionary: bool\n # None if not a column-style categorical.\n categories: Optional[\"ProtocolColumn\"]\n\n\nclass ProtocolBuffer(ABC):\n \"\"\"\n Data in the buffer is guaranteed to be contiguous in memory.\n\n Note that there is no dtype attribute present, a buffer can be thought of\n as simply a block of memory. However, if the column that the buffer is\n attached to has a dtype that's supported by DLPack and ``__dlpack__`` is\n implemented, then that dtype information will be contained in the return\n value from ``__dlpack__``.\n\n This distinction is useful to support both (a) data exchange via DLPack on a\n buffer and (b) dtypes like variable-length strings which do not have a\n fixed number of bytes per element.\n \"\"\"\n\n @property\n @abstractmethod\n def bufsize(self) -> int:\n \"\"\"\n Buffer size in bytes.\n\n Returns\n -------\n int\n \"\"\"\n pass\n\n @property\n @abstractmethod\n def ptr(self) -> int:\n \"\"\"\n Pointer to start of the buffer as an integer.\n\n Returns\n -------\n int\n \"\"\"\n pass\n\n @abstractmethod\n def __dlpack__(self) -> Any:\n \"\"\"\n Produce DLPack capsule (see array API standard).\n\n DLPack not implemented in NumPy yet, so leave it out here.\n\n Raises\n ------\n ``TypeError`` if the buffer contains unsupported dtypes.\n ``NotImplementedError`` if DLPack support is not implemented.\n\n Notes\n -----\n Useful to have to connect to array libraries. Support optional because\n it's not completely trivial to implement for a Python-only library.\n \"\"\"\n pass\n\n @abstractmethod\n def __dlpack_device__(self) -> Tuple[DlpackDeviceType, Optional[int]]:\n \"\"\"\n Device type and device ID for where the data in the buffer resides.\n\n Uses device type codes matching DLPack. Enum members are:\n - CPU = 1\n - CUDA = 2\n - CPU_PINNED = 3\n - OPENCL = 4\n - VULKAN = 7\n - METAL = 8\n - VPI = 9\n - ROCM = 10\n\n Returns\n -------\n tuple\n Device type and device ID.\n\n Notes\n -----\n Must be implemented even if ``__dlpack__`` is not.\n \"\"\"\n pass\n\n\nclass ProtocolColumn(ABC):\n \"\"\"\n A column object, with only the methods and properties required by the interchange protocol defined.\n\n A column can contain one or more chunks. Each chunk can contain up to three\n buffers - a data buffer, a mask buffer (depending on null representation),\n and an offsets buffer (if variable-size binary; e.g., variable-length strings).\n\n TBD: Arrow has a separate \"null\" dtype, and has no separate mask concept.\n Instead, it seems to use \"children\" for both columns with a bit mask,\n and for nested dtypes. Unclear whether this is elegant or confusing.\n This design requires checking the null representation explicitly.\n The Arrow design requires checking:\n 1. the ARROW_FLAG_NULLABLE (for sentinel values)\n 2. if a column has two children, combined with one of those children\n having a null dtype.\n Making the mask concept explicit seems useful. One null dtype would\n not be enough to cover both bit and byte masks, so that would mean\n even more checking if we did it the Arrow way.\n TBD: there's also the \"chunk\" concept here, which is implicit in Arrow as\n multiple buffers per array (= column here). Semantically it may make\n sense to have both: chunks were meant for example for lazy evaluation\n of data which doesn't fit in memory, while multiple buffers per column\n could also come from doing a selection operation on a single\n contiguous buffer.\n Given these concepts, one would expect chunks to be all of the same\n size (say a 10,000 row dataframe could have 10 chunks of 1,000 rows),\n while multiple buffers could have data-dependent lengths. Not an issue\n in pandas if one column is backed by a single NumPy array, but in\n Arrow it seems possible.\n Are multiple chunks *and* multiple buffers per column necessary for\n the purposes of this interchange protocol, or must producers either\n reuse the chunk concept for this or copy the data?\n\n Notes\n -----\n This ProtocolColumn object can only be produced by ``__dataframe__``,\n so doesn't need its own version or ``__column__`` protocol.\n \"\"\"\n\n @abstractmethod\n def size(self) -> int:\n \"\"\"\n Size of the column, in elements.\n\n Corresponds to `DataFrame.num_rows()` if column is a single chunk;\n equal to size of this current chunk otherwise.\n\n Is a method rather than a property because it may cause a (potentially\n expensive) computation for some dataframe implementations.\n\n Returns\n -------\n int\n Size of the column, in elements.\n \"\"\"\n pass\n\n @property\n @abstractmethod\n def offset(self) -> int:\n \"\"\"\n Get the offset of first element.\n\n May be > 0 if using chunks; for example for a column\n with N chunks of equal size M (only the last chunk may be shorter),\n ``offset = n * M``, ``n = 0 .. N-1``.\n\n Returns\n -------\n int\n The offset of first element.\n \"\"\"\n pass\n\n @property\n @abstractmethod\n def dtype(self) -> Tuple[DTypeKind, int, str, str]:\n \"\"\"\n Dtype description as a tuple ``(kind, bit-width, format string, endianness)``.\n\n * Kind : DTypeKind\n * Bit-width : the number of bits as an integer\n * Format string : data type description format string in Apache Arrow C\n Data Interface format.\n * Endianness : current only native endianness (``=``) is supported\n\n Returns\n -------\n tuple\n ``(kind, bit-width, format string, endianness)``.\n\n Notes\n -----\n - Kind specifiers are aligned with DLPack where possible\n (hence the jump to 20, leave enough room for future extension).\n - Masks must be specified as boolean with either bit width 1 (for bit masks)\n or 8 (for byte masks).\n - Dtype width in bits was preferred over bytes\n - Endianness isn't too useful, but included now in case in the future\n we need to support non-native endianness\n - Went with Apache Arrow format strings over NumPy format strings\n because they're more complete from a dataframe perspective\n - Format strings are mostly useful for datetime specification, and for categoricals.\n - For categoricals, the format string describes the type of the categorical\n in the data buffer. In case of a separate encoding of the categorical\n (e.g. an integer to string mapping), this can be derived from ``self.describe_categorical``.\n - Data types not included: complex, Arrow-style null, binary, decimal,\n and nested (list, struct, map, union) dtypes.\n \"\"\"\n pass\n\n @property\n @abstractmethod\n def describe_categorical(self) -> CategoricalDescription:\n \"\"\"\n If the dtype is categorical, there are two options.\n\n - There are only values in the data buffer.\n - There is a separate non-categorical Column encoding categorical values.\n\n TBD: are there any other in-memory representations that are needed?\n\n Returns\n -------\n dict\n Content of returned dict:\n - \"is_ordered\" : bool, whether the ordering of dictionary indices is\n semantically meaningful.\n - \"is_dictionary\" : bool, whether a mapping of\n categorical values to other objects exists\n - \"categories\" : Column representing the (implicit) mapping of indices to\n category values (e.g. an array of cat1, cat2, ...).\n None if not a dictionary-style categorical.\n\n Raises\n ------\n ``TypeError`` if the dtype is not categorical.\n \"\"\"\n pass\n\n @property\n @abstractmethod\n def describe_null(self) -> Tuple[ColumnNullType, Any]:\n \"\"\"\n Return the missing value (or \"null\") representation the column dtype uses.\n\n Return as a tuple ``(kind, value)``.\n * Kind: ColumnNullType\n * Value : if kind is \"sentinel value\", the actual value. If kind is a bit\n mask or a byte mask, the value (0 or 1) indicating a missing value. None\n otherwise.\n\n Returns\n -------\n tuple\n ``(kind, value)``.\n \"\"\"\n pass\n\n @property\n @abstractmethod\n def null_count(self) -> int:\n \"\"\"\n Get number of null elements, if known.\n\n Returns\n -------\n int\n\n Notes\n -----\n Arrow uses -1 to indicate \"unknown\", but None seems cleaner.\n \"\"\"\n pass\n\n @property\n @abstractmethod\n def metadata(self) -> Dict[str, Any]:\n \"\"\"\n Get the metadata for the column.\n\n See `DataFrame.metadata` for more details.\n\n Returns\n -------\n dict\n \"\"\"\n pass\n\n @abstractmethod\n def num_chunks(self) -> int:\n \"\"\"\n Return the number of chunks the column consists of.\n\n Returns\n -------\n int\n The number of chunks the column consists of.\n \"\"\"\n pass\n\n @abstractmethod\n def get_chunks(self, n_chunks: Optional[int] = None) -> Iterable[\"ProtocolColumn\"]:\n \"\"\"\n Return an iterator yielding the chunks.\n\n By default ``n_chunks=None``, yields the chunks that the data is stored as by the producer.\n If given, ``n_chunks`` must be a multiple of ``self.num_chunks()``,\n meaning the producer must subdivide each chunk before yielding it.\n\n Parameters\n ----------\n n_chunks : int, optional\n Number of chunks to yield.\n\n Yields\n ------\n DataFrame\n A ``DataFrame`` object(s).\n\n Raises\n ------\n ``RuntimeError`` if ``n_chunks`` is not a multiple of ``self.num_chunks()``.\n \"\"\"\n pass\n\n @abstractmethod\n def get_buffers(self) -> ColumnBuffers:\n \"\"\"\n Return a dictionary containing the underlying buffers.\n\n Returns\n -------\n dict\n - \"data\": a two-element tuple whose first element is a buffer\n containing the data and whose second element is the data buffer's associated dtype.\n - \"validity\": a two-element tuple whose first element is a buffer\n containing mask values indicating missing data and\n whose second element is the mask value buffer's\n associated dtype. None if the null representation is not a bit or byte mask.\n - \"offsets\": a two-element tuple whose first element is a buffer\n containing the offset values for variable-size binary data\n (e.g., variable-length strings) and whose second element is the offsets\n buffer's associated dtype. None if the data buffer does not have\n an associated offsets buffer.\n \"\"\"\n pass\n\n\nclass ProtocolDataframe(ABC):\n \"\"\"\n A data frame class, with only the methods required by the interchange protocol defined.\n\n Instances of this (private) class are returned from\n ``modin.core.dataframe.base.dataframe.dataframe.ModinDataframe.__dataframe__``\n as objects with the methods and attributes defined on this class.\n\n A \"data frame\" represents an ordered collection of named columns.\n A column's \"name\" must be a unique string. Columns may be accessed by name or by position.\n This could be a public data frame class, or an object with the methods and\n attributes defined on this ProtocolDataframe class could be returned from the\n ``__dataframe__`` method of a public data frame class in a library adhering\n to the dataframe interchange protocol specification.\n \"\"\"\n\n version = 0 # version of the protocol\n\n @abstractmethod\n def __dataframe__(\n self, nan_as_null: bool = False, allow_copy: bool = True\n ) -> \"ProtocolDataframe\":\n \"\"\"\n Construct a new dataframe interchange object, potentially changing the parameters.\n\n See more about the protocol in https://data-apis.org/dataframe-protocol/latest/index.html.\n\n Parameters\n ----------\n nan_as_null : bool, default: False\n A keyword intended for the consumer to tell the producer\n to overwrite null values in the data with ``NaN``.\n This currently has no effect; once support for nullable extension\n dtypes is added, this value should be propagated to columns.\n allow_copy : bool, default: True\n A keyword that defines whether or not the library is allowed\n to make a copy of the data. For example, copying data would be necessary\n if a library supports strided buffers, given that this protocol\n specifies contiguous buffers. Currently, if the flag is set to ``False``\n and a copy is needed, a ``RuntimeError`` will be raised.\n\n Returns\n -------\n ProtocolDataframe\n \"\"\"\n pass\n\n @property\n @abstractmethod\n def metadata(self) -> Dict[str, Any]:\n \"\"\"\n Get the metadata for the data frame, as a dictionary with string keys.\n\n The contents of `metadata` may be anything, they are meant for a library\n to store information that it needs to, e.g., roundtrip losslessly or\n for two implementations to share data that is not (yet) part of the\n interchange protocol specification. For avoiding collisions with other\n entries, please add name the keys with the name of the library\n followed by a period and the desired name, e.g, ``pandas.indexcol``.\n\n Returns\n -------\n dict\n \"\"\"\n pass\n\n @abstractmethod\n def num_columns(self) -> int:\n \"\"\"\n Return the number of columns in the ProtocolDataframe.\n\n Returns\n -------\n int\n The number of columns in the ProtocolDataframe.\n \"\"\"\n pass\n\n @abstractmethod\n def num_rows(self) -> Optional[int]:\n \"\"\"\n Return the number of rows in the ProtocolDataframe, if available.\n\n Returns\n -------\n int\n The number of rows in the ProtocolDataframe.\n \"\"\"\n pass\n\n @abstractmethod\n def num_chunks(self) -> int:\n \"\"\"\n Return the number of chunks the ProtocolDataframe consists of.\n\n Returns\n -------\n int\n The number of chunks the ProtocolDataframe consists of.\n \"\"\"\n pass\n\n @abstractmethod\n def column_names(self) -> Iterable[str]:\n \"\"\"\n Return an iterator yielding the column names.\n\n Yields\n ------\n str\n The name of the column(s).\n \"\"\"\n pass\n\n @abstractmethod\n def get_column(self, i: int) -> ProtocolColumn:\n \"\"\"\n Return the column at the indicated position.\n\n Parameters\n ----------\n i : int\n Positional index of the column to be returned.\n\n Returns\n -------\n Column\n The column at the indicated position.\n \"\"\"\n pass\n\n @abstractmethod\n def get_column_by_name(self, name: str) -> ProtocolColumn:\n \"\"\"\n Return the column whose name is the indicated name.\n\n Parameters\n ----------\n name : str\n String label of the column to be returned.\n\n Returns\n -------\n Column\n The column whose name is the indicated name.\n \"\"\"\n pass\n\n @abstractmethod\n def get_columns(self) -> Iterable[ProtocolColumn]:\n \"\"\"\n Return an iterator yielding the columns.\n\n Yields\n ------\n Column\n The ``Column`` object(s).\n \"\"\"\n pass\n\n @abstractmethod\n def select_columns(self, indices: Sequence[int]) -> \"ProtocolDataframe\":\n \"\"\"\n Create a new ProtocolDataframe by selecting a subset of columns by index.\n\n Parameters\n ----------\n indices : Sequence[int]\n Column indices to be selected out of the ProtocolDataframe.\n\n Returns\n -------\n ProtocolDataframe\n A new ProtocolDataframe with selected a subset of columns by index.\n \"\"\"\n pass\n\n @abstractmethod\n def select_columns_by_name(self, names: Sequence[str]) -> \"ProtocolDataframe\":\n \"\"\"\n Create a new ProtocolDataframe by selecting a subset of columns by name.\n\n Parameters\n ----------\n names : Sequence[str]\n Column names to be selected out of the ProtocolDataframe.\n\n Returns\n -------\n ProtocolDataframe\n A new ProtocolDataframe with selected a subset of columns by name.\n \"\"\"\n pass\n\n @abstractmethod\n def get_chunks(\n self, n_chunks: Optional[int] = None\n ) -> Iterable[\"ProtocolDataframe\"]:\n \"\"\"\n Return an iterator yielding the chunks.\n\n By default `n_chunks=None`, yields the chunks that the data is stored as by the producer.\n If given, `n_chunks` must be a multiple of `self.num_chunks()`,\n meaning the producer must subdivide each chunk before yielding it.\n\n Parameters\n ----------\n n_chunks : int, optional\n Number of chunks to yield.\n\n Yields\n ------\n ProtocolDataframe\n A ``ProtocolDataframe`` object(s).\n\n Raises\n ------\n ``RuntimeError`` if ``n_chunks`` is not a multiple of ``self.num_chunks()``.\n \"\"\"\n pass\n"
},
{
"path": "modin/core/dataframe/base/interchange/dataframe_protocol/utils.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nDataframe exchange protocol implementation.\n\nSee more in https://data-apis.org/dataframe-protocol/latest/index.html.\n\"\"\"\n\nimport enum\nimport re\nfrom typing import Optional, Union\n\nimport numpy as np\nimport pandas\nfrom pandas.api.types import is_datetime64_dtype\n\n\nclass DTypeKind(enum.IntEnum): # noqa PR01\n \"\"\"\n Integer enum for data types.\n Attributes\n ----------\n INT : int\n Matches to signed integer data type.\n UINT : int\n Matches to unsigned integer data type.\n FLOAT : int\n Matches to floating point data type.\n BOOL : int\n Matches to boolean data type.\n STRING : int\n Matches to string data type (UTF-8 encoded).\n DATETIME : int\n Matches to datetime data type.\n CATEGORICAL : int\n Matches to categorical data type.\n \"\"\"\n\n INT = 0\n UINT = 1\n FLOAT = 2\n BOOL = 20\n STRING = 21 # UTF-8\n DATETIME = 22\n CATEGORICAL = 23\n\n\nclass ColumnNullType(enum.IntEnum): # noqa PR01\n \"\"\"\n Integer enum for null type representation.\n Attributes\n ----------\n NON_NULLABLE : int\n Non-nullable column.\n USE_NAN : int\n Use explicit float NaN value.\n USE_SENTINEL : int\n Sentinel value besides NaN.\n USE_BITMASK : int\n The bit is set/unset representing a null on a certain position.\n USE_BYTEMASK : int\n The byte is set/unset representing a null on a certain position.\n \"\"\"\n\n NON_NULLABLE = 0\n USE_NAN = 1\n USE_SENTINEL = 2\n USE_BITMASK = 3\n USE_BYTEMASK = 4\n\n\nclass DlpackDeviceType(enum.IntEnum): # noqa PR01\n \"\"\"Integer enum for device type codes matching DLPack.\"\"\"\n\n CPU = 1\n CUDA = 2\n CPU_PINNED = 3\n OPENCL = 4\n VULKAN = 7\n METAL = 8\n VPI = 9\n ROCM = 10\n\n\nclass ArrowCTypes:\n \"\"\"\n Enum for Apache Arrow C type format strings.\n\n The Arrow C data interface:\n https://arrow.apache.org/docs/format/CDataInterface.html#data-type-description-format-strings\n \"\"\"\n\n NULL = \"n\"\n BOOL = \"b\"\n INT8 = \"c\"\n UINT8 = \"C\"\n INT16 = \"s\"\n UINT16 = \"S\"\n INT32 = \"i\"\n UINT32 = \"I\"\n INT64 = \"l\"\n UINT64 = \"L\"\n FLOAT16 = \"e\"\n FLOAT32 = \"f\"\n FLOAT64 = \"g\"\n STRING = \"u\" # utf-8\n DATE32 = \"tdD\"\n DATE64 = \"tdm\"\n # Resoulution:\n # - seconds -> 's'\n # - miliseconds -> 'm'\n # - microseconds -> 'u'\n # - nanoseconds -> 'n'\n TIMESTAMP = \"ts{resolution}:{tz}\"\n TIME = \"tt{resolution}\"\n\n\nclass Endianness:\n \"\"\"Enum indicating the byte-order of a data-type.\"\"\"\n\n LITTLE = \"<\"\n BIG = \">\"\n NATIVE = \"=\"\n NA = \"|\"\n\n\ndef pandas_dtype_to_arrow_c(dtype: Union[np.dtype, pandas.CategoricalDtype]) -> str:\n \"\"\"\n Represent pandas `dtype` as a format string in Apache Arrow C notation.\n\n Parameters\n ----------\n dtype : np.dtype\n Datatype of pandas DataFrame to represent.\n\n Returns\n -------\n str\n Format string in Apache Arrow C notation of the given `dtype`.\n \"\"\"\n if isinstance(dtype, pandas.CategoricalDtype):\n return ArrowCTypes.INT64\n elif dtype == pandas.api.types.pandas_dtype(\"O\"):\n return ArrowCTypes.STRING\n\n format_str = getattr(ArrowCTypes, dtype.name.upper(), None)\n if format_str is not None:\n return format_str\n\n if is_datetime64_dtype(dtype):\n # Selecting the first char of resolution string:\n # dtype.str -> ' None:\n \"\"\"\n Raise a ``RuntimeError`` mentioning that there's a copy required.\n\n Parameters\n ----------\n copy_reason : str, optional\n The reason of making a copy. Should fit to the following format:\n 'The copy occurred due to {copy_reason}.'.\n \"\"\"\n msg = \"Copy required but 'allow_copy=False' is set.\"\n if copy_reason:\n msg += f\" The copy occurred due to {copy_reason}.\"\n raise RuntimeError(msg)\n"
},
{
"path": "modin/core/dataframe/base/partitioning/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe classes related to its partitioning.\"\"\"\n"
},
{
"path": "modin/core/dataframe/base/partitioning/axis_partition.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base class of an axis partition for a Modin Dataframe.\"\"\"\n\nfrom abc import ABC, abstractmethod\nfrom typing import Any, Callable, Iterable, Optional, Tuple, Type, Union\n\nfrom modin.logging import ClassLogger\nfrom modin.logging.config import LogLevel\n\n\nclass BaseDataframeAxisPartition(\n ABC, ClassLogger, modin_layer=\"VIRTUAL-PARTITION\", log_level=LogLevel.DEBUG\n): # pragma: no cover\n \"\"\"\n An abstract class that represents the parent class for any axis partition class.\n\n This class is intended to simplify the way that operations are performed.\n\n Attributes\n ----------\n _PARTITIONS_METADATA_LEN : int\n The number of metadata values that the object of `partition_type` consumes.\n \"\"\"\n\n @property\n @abstractmethod\n def list_of_blocks(self) -> list:\n \"\"\"Get the list of physical partition objects that compose this partition.\"\"\"\n pass\n\n def apply(\n self,\n func: Callable,\n *args: Iterable,\n num_splits: Optional[int] = None,\n other_axis_partition: Optional[\"BaseDataframeAxisPartition\"] = None,\n maintain_partitioning: bool = True,\n lengths: Optional[Iterable] = None,\n manual_partition: bool = False,\n **kwargs: dict,\n ) -> Any:\n \"\"\"\n Apply a function to this axis partition along full axis.\n\n Parameters\n ----------\n func : callable\n The function to apply. This will be preprocessed according to\n the corresponding `BaseDataframePartition` objects.\n *args : iterable\n Positional arguments to pass to `func`.\n num_splits : int, default: None\n The number of times to split the result object.\n other_axis_partition : BaseDataframeAxisPartition, default: None\n Another `BaseDataframeAxisPartition` object to be applied\n to func. This is for operations that are between two data sets.\n maintain_partitioning : bool, default: True\n Whether to keep the partitioning in the same\n orientation as it was previously or not. This is important because we may be\n operating on an individual axis partition and not touching the rest.\n In this case, we have to return the partitioning to its previous\n orientation (the lengths will remain the same). This is ignored between\n two axis partitions.\n lengths : iterable, default: None\n The list of lengths to shuffle the partition into.\n manual_partition : bool, default: False\n If True, partition the result with `lengths`.\n **kwargs : dict\n Additional keywords arguments to be passed in `func`.\n\n Returns\n -------\n list\n A list of `BaseDataframePartition` objects.\n\n Notes\n -----\n The procedures that invoke this method assume full axis\n knowledge. Implement this method accordingly.\n\n You must return a list of `BaseDataframePartition` objects from this method.\n \"\"\"\n pass\n\n # Child classes must have these in order to correctly subclass.\n partition_type: Type\n _PARTITIONS_METADATA_LEN = 0\n\n def _wrap_partitions(\n self, partitions: list, extract_metadata: Optional[bool] = None\n ) -> list:\n \"\"\"\n Wrap remote partition objects with `BaseDataframePartition` class.\n\n Parameters\n ----------\n partitions : list\n List of remotes partition objects to be wrapped with `BaseDataframePartition` class.\n extract_metadata : bool, optional\n Whether the partitions list contains information about partition's metadata.\n If `None` was passed will take the argument's value from the value of `cls._PARTITIONS_METADATA_LEN`.\n\n Returns\n -------\n list\n List of wrapped remote partition objects.\n \"\"\"\n assert self.partition_type is not None\n\n if extract_metadata is None:\n # If `_PARTITIONS_METADATA_LEN == 0` then the execution doesn't support metadata\n # and thus we should never try extracting it, otherwise assuming that the common\n # approach of always passing the metadata is used.\n extract_metadata = bool(self._PARTITIONS_METADATA_LEN)\n\n if extract_metadata:\n # Here we recieve a 1D array of futures describing partitions and their metadata as:\n # [object_id{partition_idx}, metadata{partition_idx}_{metadata_idx}, ...]\n # Here's an example of such array:\n # [\n # object_id1, metadata1_1, metadata1_2, ..., metadata1_PARTITIONS_METADATA_LEN,\n # object_id2, metadata2_1, ..., metadata2_PARTITIONS_METADATA_LEN,\n # ...\n # object_idN, metadataN_1, ..., metadataN_PARTITIONS_METADATA_LEN,\n # ]\n return [\n self.partition_type(*init_args)\n for init_args in zip(\n # `partition_type` consumes `(object_id, *metadata)`, thus adding `+1`\n *[iter(partitions)]\n * (1 + self._PARTITIONS_METADATA_LEN)\n )\n ]\n else:\n return [self.partition_type(object_id) for object_id in partitions]\n\n def force_materialization(\n self, get_ip: bool = False\n ) -> \"BaseDataframeAxisPartition\":\n \"\"\"\n Materialize axis partitions into a single partition.\n\n Parameters\n ----------\n get_ip : bool, default: False\n Whether to get node ip address to a single partition or not.\n\n Returns\n -------\n BaseDataframeAxisPartition\n An axis partition containing only a single materialized partition.\n \"\"\"\n materialized = self.apply(\n lambda x: x, num_splits=1, maintain_partitioning=False\n )\n return type(self)(materialized, get_ip=get_ip) # type: ignore[call-arg]\n\n def unwrap(\n self, squeeze: bool = False, get_ip: bool = False\n ) -> Union[list, Tuple[list, list]]:\n \"\"\"\n Unwrap partitions from this axis partition.\n\n Parameters\n ----------\n squeeze : bool, default: False\n Flag used to unwrap only one partition.\n get_ip : bool, default: False\n Whether to get node ip address to each partition or not.\n\n Returns\n -------\n list\n List of partitions from this axis partition.\n\n Notes\n -----\n If `get_ip=True`, a tuple of lists of Ray.ObjectRef/Dask.Future to node ip addresses and\n unwrapped partitions, respectively, is returned if Ray/Dask is used as an engine\n (i.e. [(Ray.ObjectRef/Dask.Future, Ray.ObjectRef/Dask.Future), ...]).\n \"\"\"\n if squeeze and len(self.list_of_blocks) == 1:\n if get_ip:\n # TODO(https://github.com/modin-project/modin/issues/5176): Stop ignoring the list_of_ips\n # check once we know that we're not calling list_of_ips on python axis partitions\n return self.list_of_ips[0], self.list_of_blocks[0] # type: ignore[attr-defined]\n else:\n return self.list_of_blocks[0]\n else:\n if get_ip:\n return list(zip(self.list_of_ips, self.list_of_blocks)) # type: ignore[attr-defined]\n else:\n return self.list_of_blocks\n"
},
{
"path": "modin/core/dataframe/pandas/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe classes optimized for pandas storage format.\"\"\"\n"
},
{
"path": "modin/core/dataframe/pandas/dataframe/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe class optimized for pandas storage format.\"\"\"\n"
},
{
"path": "modin/core/dataframe/pandas/dataframe/dataframe.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nModule contains class PandasDataframe.\n\nPandasDataframe is a parent abstract class for any dataframe class\nfor pandas storage format.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport datetime\nimport re\nfrom abc import ABC, abstractmethod\nfrom functools import cached_property\nfrom typing import TYPE_CHECKING, Callable, Dict, Hashable, List, Optional, Union\n\nimport numpy as np\nimport pandas\nfrom pandas._libs.lib import no_default\nfrom pandas.api.types import is_object_dtype\nfrom pandas.core.dtypes.common import is_dtype_equal, is_list_like, is_numeric_dtype\nfrom pandas.core.indexes.api import Index, RangeIndex\n\nfrom modin.config import (\n IsRayCluster,\n MinColumnPartitionSize,\n MinRowPartitionSize,\n NPartitions,\n)\nfrom modin.core.dataframe.base.dataframe.dataframe import ModinDataframe\nfrom modin.core.dataframe.base.dataframe.utils import Axis, JoinType, is_trivial_index\nfrom modin.core.dataframe.pandas.dataframe.utils import (\n ShuffleSortFunctions,\n add_missing_categories_to_groupby,\n lazy_metadata_decorator,\n)\nfrom modin.core.dataframe.pandas.metadata import (\n DtypesDescriptor,\n LazyProxyCategoricalDtype,\n ModinDtypes,\n ModinIndex,\n)\nfrom modin.core.storage_formats.pandas.parsers import (\n find_common_type_cat as find_common_type,\n)\nfrom modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\nfrom modin.core.storage_formats.pandas.utils import get_length_list\nfrom modin.error_message import ErrorMessage\nfrom modin.logging import ClassLogger\nfrom modin.logging.config import LogLevel\nfrom modin.pandas.indexing import is_range_like\nfrom modin.pandas.utils import (\n check_both_not_none,\n get_pandas_backend,\n is_full_grab_slice,\n)\nfrom modin.utils import MODIN_UNNAMED_SERIES_LABEL\n\nif TYPE_CHECKING:\n from pandas._typing import npt\n\n from modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (\n ProtocolDataframe,\n )\n from modin.core.dataframe.pandas.partitioning.partition_manager import (\n PandasDataframePartitionManager,\n )\n\n\nclass PandasDataframe(\n ABC, ClassLogger, modin_layer=\"CORE-DATAFRAME\", log_level=LogLevel.DEBUG\n):\n \"\"\"\n An abstract class that represents the parent class for any pandas storage format dataframe class.\n\n This class provides interfaces to run operations on dataframe partitions.\n\n Parameters\n ----------\n partitions : np.ndarray\n A 2D NumPy array of partitions.\n index : sequence or callable, optional\n The index for the dataframe. Converted to a ``pandas.Index``.\n Is computed from partitions on demand if not specified.\n If ``callable() -> (pandas.Index, list of row lengths or None)`` type,\n then the calculation will be delayed until `self.index` is called.\n columns : sequence, optional\n The columns object for the dataframe. Converted to a ``pandas.Index``.\n Is computed from partitions on demand if not specified.\n row_lengths : list, optional\n The length of each partition in the rows. The \"height\" of\n each of the block partitions. Is computed if not provided.\n column_widths : list, optional\n The width of each partition in the columns. The \"width\" of\n each of the block partitions. Is computed if not provided.\n dtypes : pandas.Series or callable, optional\n The data types for the dataframe columns.\n pandas_backend : {\"pyarrow\", None}, optional\n Backend used by pandas.\n \"\"\"\n\n _partition_mgr_cls: PandasDataframePartitionManager\n _query_compiler_cls = PandasQueryCompiler\n # These properties flag whether or not we are deferring the metadata synchronization\n _deferred_index: bool = False\n _deferred_column: bool = False\n\n _index_cache: ModinIndex = None\n _columns_cache: ModinIndex = None\n _dtypes: Optional[ModinDtypes] = None\n _pandas_backend: Optional[str] = None\n\n @property\n def storage_format(self) -> str:\n \"\"\"\n The storage format for this frame's data.\n\n Returns\n -------\n str\n The storage format.\n \"\"\"\n return \"Pandas\"\n\n @property\n @abstractmethod\n def engine(self) -> str:\n \"\"\"\n The engine for this frame.\n\n Returns\n -------\n str\n The engine.\n \"\"\"\n pass\n\n @cached_property\n def __constructor__(self) -> type[PandasDataframe]:\n \"\"\"\n Create a new instance of this object.\n\n Returns\n -------\n callable\n \"\"\"\n return type(self)\n\n def __init__(\n self,\n partitions,\n index=None,\n columns=None,\n row_lengths=None,\n column_widths=None,\n dtypes: Optional[Union[pandas.Series, ModinDtypes, Callable]] = None,\n pandas_backend: Optional[str] = None,\n ):\n self._partitions = partitions\n self.set_index_cache(index)\n self.set_columns_cache(columns)\n self._row_lengths_cache = row_lengths\n self._column_widths_cache = column_widths\n self._pandas_backend = pandas_backend\n if pandas_backend != \"pyarrow\" or len(partitions) == 0:\n # If the backend is pyarrow and there are no partitions, the computed dtype otherwise becomes NaN,\n # which means we lost the dtype, so actually set it in that case\n self.set_dtypes_cache(dtypes)\n else:\n # In this case, the type precomputation may be incorrect; we need\n # to know the type algebra precisely. Considering the number of operations\n # and different combinations of backends, the best solution would be to\n # introduce optimizations gradually, with a large number of tests.\n self.set_dtypes_cache(None)\n\n self._validate_axes_lengths()\n self._filter_empties(compute_metadata=False)\n\n def _validate_axes_lengths(self):\n \"\"\"Validate that labels are split correctly if split is known.\"\"\"\n if (\n self._row_lengths_cache is not None\n and self.has_materialized_index\n and len(self.index) > 0\n ):\n # An empty frame can have 0 rows but a nonempty index. If the frame\n # does have rows, the number of rows must equal the size of the\n # index.\n num_rows = sum(self._row_lengths_cache)\n if num_rows > 0:\n ErrorMessage.catch_bugs_and_request_email(\n num_rows != len(self.index),\n f\"Row lengths: {num_rows} != {len(self.index)}\",\n )\n ErrorMessage.catch_bugs_and_request_email(\n any(val < 0 for val in self._row_lengths_cache),\n f\"Row lengths cannot be negative: {self._row_lengths_cache}\",\n )\n if (\n self._column_widths_cache is not None\n and self.has_materialized_columns\n and len(self.columns) > 0\n ):\n # An empty frame can have 0 column but a nonempty column index. If\n # the frame does have columns, the number of columns must equal the\n # size of the columns.\n num_columns = sum(self._column_widths_cache)\n if num_columns > 0:\n ErrorMessage.catch_bugs_and_request_email(\n num_columns != len(self.columns),\n f\"Column widths: {num_columns} != {len(self.columns)}\",\n )\n ErrorMessage.catch_bugs_and_request_email(\n any(val < 0 for val in self._column_widths_cache),\n f\"Column widths cannot be negative: {self._column_widths_cache}\",\n )\n\n @property\n def num_parts(self) -> int:\n \"\"\"\n Get the total number of partitions for this frame.\n\n Returns\n -------\n int\n \"\"\"\n return np.prod(self._partitions.shape)\n\n @property\n def row_lengths(self):\n \"\"\"\n Compute the row partitions lengths if they are not cached.\n\n Returns\n -------\n list\n A list of row partitions lengths.\n \"\"\"\n if self._row_lengths_cache is None:\n if len(self._partitions.T) > 0:\n row_parts = self._partitions.T[0]\n self._row_lengths_cache = self._get_lengths(row_parts, Axis.ROW_WISE)\n else:\n self._row_lengths_cache = []\n return self._row_lengths_cache\n\n @classmethod\n def _get_lengths(cls, parts, axis):\n \"\"\"\n Get list of dimensions for all the provided parts.\n\n Parameters\n ----------\n parts : list\n List of parttions.\n axis : {0, 1}\n The axis along which to get the lengths (0 - length across rows or, 1 - width across columns).\n\n Returns\n -------\n list\n \"\"\"\n if axis == Axis.ROW_WISE:\n return [part.length() for part in parts]\n else:\n return [part.width() for part in parts]\n\n def __len__(self) -> int:\n \"\"\"\n Return length of index axis.\n\n Returns\n -------\n int\n \"\"\"\n if self.has_materialized_index:\n _len = len(self.index)\n else:\n _len = sum(self.row_lengths)\n return _len\n\n @property\n def column_widths(self):\n \"\"\"\n Compute the column partitions widths if they are not cached.\n\n Returns\n -------\n list\n A list of column partitions widths.\n \"\"\"\n if self._column_widths_cache is None:\n if len(self._partitions) > 0:\n col_parts = self._partitions[0]\n self._column_widths_cache = self._get_lengths(col_parts, Axis.COL_WISE)\n else:\n self._column_widths_cache = []\n return self._column_widths_cache\n\n def _set_axis_lengths_cache(self, value, axis=0):\n \"\"\"\n Set the row/column lengths cache for the specified axis.\n\n Parameters\n ----------\n value : list of ints\n axis : int, default: 0\n 0 for row lengths and 1 for column widths.\n \"\"\"\n if axis == 0:\n self._row_lengths_cache = value\n else:\n self._column_widths_cache = value\n\n def _get_axis_lengths_cache(self, axis=0):\n \"\"\"\n Get partition's shape caches along the specified axis if avaliable.\n\n Parameters\n ----------\n axis : int, default: 0\n 0 - get row lengths cache, 1 - get column widths cache.\n\n Returns\n -------\n list of ints or None\n If the cache is computed return a list of ints, ``None`` otherwise.\n \"\"\"\n return self._row_lengths_cache if axis == 0 else self._column_widths_cache\n\n def _get_axis_lengths(self, axis: int = 0) -> List[int]:\n \"\"\"\n Get row lengths/column widths.\n\n Parameters\n ----------\n axis : int, default: 0\n\n Returns\n -------\n list of ints\n \"\"\"\n return self.row_lengths if axis == 0 else self.column_widths\n\n @property\n def has_dtypes_cache(self) -> bool:\n \"\"\"\n Check if the dtypes cache exists.\n\n Returns\n -------\n bool\n \"\"\"\n return self._dtypes is not None\n\n @property\n def has_materialized_dtypes(self) -> bool:\n \"\"\"\n Check if dataframe has materialized index cache.\n\n Returns\n -------\n bool\n \"\"\"\n return self.has_dtypes_cache and self._dtypes.is_materialized\n\n def copy_dtypes_cache(self):\n \"\"\"\n Copy the dtypes cache.\n\n Returns\n -------\n pandas.Series, callable or None\n If there is an pandas.Series in the cache, then copying occurs.\n \"\"\"\n dtypes_cache = None\n if self.has_dtypes_cache:\n dtypes_cache = self._dtypes.copy()\n return dtypes_cache\n\n def _maybe_update_proxies(self, dtypes, new_parent=None):\n \"\"\"\n Update lazy proxies stored inside of `dtypes` with a new parent inplace.\n\n Parameters\n ----------\n dtypes : pandas.Series, ModinDtypes or callable\n new_parent : object, optional\n A new parent to link the proxies to. If not specified\n will consider the `self` to be a new parent.\n\n Returns\n -------\n pandas.Series, ModinDtypes or callable\n \"\"\"\n new_parent = new_parent or self\n if isinstance(dtypes, ModinDtypes):\n dtypes = dtypes.maybe_specify_new_frame_ref(new_parent)\n if isinstance(dtypes, pandas.Series):\n LazyProxyCategoricalDtype.update_dtypes(dtypes, new_parent)\n return dtypes\n\n def set_dtypes_cache(self, dtypes):\n \"\"\"\n Set dtypes cache.\n\n Parameters\n ----------\n dtypes : pandas.Series, ModinDtypes, callable or None\n \"\"\"\n dtypes = self._maybe_update_proxies(dtypes)\n if dtypes is None and self.has_materialized_columns:\n # try to set a descriptor instead of 'None' to be more flexible in\n # dtypes computing\n try:\n self._dtypes = ModinDtypes(\n DtypesDescriptor(\n cols_with_unknown_dtypes=self.columns.tolist(), parent_df=self\n )\n )\n except NotImplementedError:\n self._dtypes = None\n elif isinstance(dtypes, ModinDtypes) or dtypes is None:\n self._dtypes = dtypes\n else:\n self._dtypes = ModinDtypes(dtypes)\n\n @property\n def dtypes(self):\n \"\"\"\n Compute the data types if they are not cached.\n\n Returns\n -------\n pandas.Series\n A pandas Series containing the data types for this dataframe.\n \"\"\"\n if self.has_dtypes_cache:\n dtypes = self._dtypes.get()\n else:\n dtypes = self._compute_dtypes()\n self.set_dtypes_cache(dtypes)\n # During materialization, we can find out the backend and, if it\n # is suitable, use the ability to pre-calculate types.\n self._pandas_backend = get_pandas_backend(dtypes)\n return dtypes\n\n def get_dtypes_set(self):\n \"\"\"\n Get a set of dtypes that are in this dataframe.\n\n Returns\n -------\n set\n \"\"\"\n if isinstance(self._dtypes, ModinDtypes):\n return self._dtypes.get_dtypes_set()\n return set(self.dtypes.values)\n\n def _compute_dtypes(self, columns=None) -> pandas.Series:\n \"\"\"\n Compute the data types via TreeReduce pattern for the specified columns.\n\n Parameters\n ----------\n columns : list-like, optional\n Columns to compute dtypes for. If not specified compute dtypes\n for all the columns in the dataframe.\n\n Returns\n -------\n pandas.Series\n A pandas Series containing the data types for this dataframe.\n \"\"\"\n\n def dtype_builder(df):\n return df.apply(lambda col: find_common_type(col.values), axis=0)\n\n if columns is not None:\n # Sorting positions to request columns in the order they're stored (it's more efficient)\n numeric_indices = sorted(self.columns.get_indexer_for(columns))\n if any(pos < 0 for pos in numeric_indices):\n raise KeyError(\n f\"Some of the columns are not in index: subset={columns}; columns={self.columns}\"\n )\n obj = self.take_2d_labels_or_positional(\n col_labels=self.columns[numeric_indices].tolist()\n )\n else:\n obj = self\n\n # For now we will use a pandas Series for the dtypes.\n if len(obj.columns) > 0:\n dtypes = (\n obj.tree_reduce(0, lambda df: df.dtypes, dtype_builder)\n .to_pandas()\n .iloc[0]\n )\n else:\n dtypes = pandas.Series([])\n # reset name to None because we use MODIN_UNNAMED_SERIES_LABEL internally\n dtypes.name = None\n return dtypes\n\n def set_index_cache(self, index):\n \"\"\"\n Set index cache.\n\n Parameters\n ----------\n index : sequence, callable or None\n \"\"\"\n if index is None:\n self._index_cache = ModinIndex(self, axis=0)\n elif isinstance(index, ModinIndex):\n # update reference with the new frame to not pollute memory\n self._index_cache = index.maybe_specify_new_frame_ref(self, axis=0)\n else:\n self._index_cache = ModinIndex(index)\n\n def set_columns_cache(self, columns):\n \"\"\"\n Set columns cache.\n\n Parameters\n ----------\n columns : sequence, callable or None\n \"\"\"\n if columns is None:\n self._columns_cache = ModinIndex(self, axis=1)\n elif isinstance(columns, ModinIndex):\n # update reference with the new frame to not pollute memory\n self._columns_cache = columns.maybe_specify_new_frame_ref(self, axis=1)\n else:\n self._columns_cache = ModinIndex(columns)\n\n def set_axis_cache(self, value, axis=0):\n \"\"\"\n Set cache for the specified axis (index or columns).\n\n Parameters\n ----------\n value : sequence, callable or None\n axis : int, default: 0\n \"\"\"\n if axis == 0:\n self.set_index_cache(value)\n else:\n self.set_columns_cache(value)\n\n def has_axis_cache(self, axis=0) -> bool:\n \"\"\"\n Check if the cache for the specified axis exists.\n\n Parameters\n ----------\n axis : int, default: 0\n\n Returns\n -------\n bool\n \"\"\"\n return self.has_index_cache if axis == 0 else self.has_columns_cache\n\n @property\n def has_index_cache(self):\n \"\"\"\n Check if the index cache exists.\n\n Returns\n -------\n bool\n \"\"\"\n return self._index_cache is not None\n\n def copy_index_cache(self, copy_lengths=False):\n \"\"\"\n Copy the index cache.\n\n Parameters\n ----------\n copy_lengths : bool, default: False\n Whether to copy the stored partition lengths to the\n new index object.\n\n Returns\n -------\n pandas.Index, callable or ModinIndex\n If there is an pandas.Index in the cache, then copying occurs.\n \"\"\"\n idx_cache = self._index_cache\n if self.has_index_cache:\n idx_cache = self._index_cache.copy(copy_lengths)\n return idx_cache\n\n def _get_axis_cache(self, axis=0) -> ModinIndex:\n \"\"\"\n Get axis cache for the specified axis if available.\n\n Parameters\n ----------\n axis : int, default: 0\n\n Returns\n -------\n ModinIndex\n \"\"\"\n return self._index_cache if axis == 0 else self._columns_cache\n\n @property\n def has_columns_cache(self):\n \"\"\"\n Check if the columns cache exists.\n\n Returns\n -------\n bool\n \"\"\"\n return self._columns_cache is not None\n\n def copy_columns_cache(self, copy_lengths=False):\n \"\"\"\n Copy the columns cache.\n\n Parameters\n ----------\n copy_lengths : bool, default: False\n Whether to copy the stored partition lengths to the\n new index object.\n\n Returns\n -------\n pandas.Index or None\n If there is an pandas.Index in the cache, then copying occurs.\n \"\"\"\n columns_cache = self._columns_cache\n if columns_cache is not None:\n columns_cache = columns_cache.copy(copy_lengths)\n return columns_cache\n\n def copy_axis_cache(self, axis=0, copy_lengths=False):\n \"\"\"\n Copy the axis cache (index or columns).\n\n Parameters\n ----------\n axis : int, default: 0\n copy_lengths : bool, default: False\n Whether to copy the stored partition lengths to the\n new index object.\n\n Returns\n -------\n pandas.Index, callable or None\n If there is an pandas.Index in the cache, then copying occurs.\n \"\"\"\n if axis == 0:\n return self.copy_index_cache(copy_lengths)\n else:\n return self.copy_columns_cache(copy_lengths)\n\n @property\n def has_materialized_index(self):\n \"\"\"\n Check if dataframe has materialized index cache.\n\n Returns\n -------\n bool\n \"\"\"\n return self.has_index_cache and self._index_cache.is_materialized\n\n @property\n def has_materialized_columns(self):\n \"\"\"\n Check if dataframe has materialized columns cache.\n\n Returns\n -------\n bool\n \"\"\"\n return self.has_columns_cache and self._columns_cache.is_materialized\n\n def _validate_set_axis(self, new_labels, old_labels):\n \"\"\"\n Validate the possibility of replacement of old labels with the new labels.\n\n Parameters\n ----------\n new_labels : list-like\n The labels to replace with.\n old_labels : list-like\n The labels to replace.\n\n Returns\n -------\n list-like\n The validated labels.\n \"\"\"\n new_labels = (\n ModinIndex(new_labels)\n if not isinstance(new_labels, ModinIndex)\n else new_labels\n )\n old_len = len(old_labels)\n new_len = len(new_labels)\n if old_len != new_len:\n raise ValueError(\n f\"Length mismatch: Expected axis has {old_len} elements, \"\n + f\"new values have {new_len} elements\"\n )\n return new_labels\n\n def _get_index(self):\n \"\"\"\n Get the index from the cache object.\n\n Returns\n -------\n pandas.Index\n An index object containing the row labels.\n \"\"\"\n if self.has_index_cache:\n index, row_lengths = self._index_cache.get(return_lengths=True)\n else:\n index, row_lengths = self._compute_axis_labels_and_lengths(0)\n self.set_index_cache(index)\n if self._row_lengths_cache is None:\n self._row_lengths_cache = row_lengths\n return index\n\n def _get_columns(self):\n \"\"\"\n Get the columns from the cache object.\n\n Returns\n -------\n pandas.Index\n An index object containing the column labels.\n \"\"\"\n if self.has_columns_cache:\n columns, column_widths = self._columns_cache.get(return_lengths=True)\n else:\n columns, column_widths = self._compute_axis_labels_and_lengths(1)\n self.set_columns_cache(columns)\n if self._column_widths_cache is None:\n self._column_widths_cache = column_widths\n return columns\n\n def _set_index(self, new_index):\n \"\"\"\n Replace the current row labels with new labels.\n\n Parameters\n ----------\n new_index : list-like\n The new row labels.\n \"\"\"\n if self.has_materialized_index:\n new_index = self._validate_set_axis(new_index, self._index_cache)\n self.set_index_cache(new_index)\n self.synchronize_labels(axis=0)\n\n def _set_columns(self, new_columns):\n \"\"\"\n Replace the current column labels with new labels.\n\n Parameters\n ----------\n new_columns : list-like\n The new column labels.\n \"\"\"\n if self.has_materialized_columns:\n # do not set new columns if they're identical to the previous ones\n if (\n isinstance(new_columns, pandas.Index)\n and self.columns.identical(new_columns)\n ) or (\n not isinstance(new_columns, pandas.Index)\n and np.array_equal(self.columns.values, new_columns)\n ):\n return\n new_columns = self._validate_set_axis(new_columns, self._columns_cache)\n if isinstance(self._dtypes, ModinDtypes):\n try:\n new_dtypes = self._dtypes.set_index(new_columns)\n except NotImplementedError:\n # can raise on duplicated labels\n new_dtypes = None\n elif isinstance(self._dtypes, pandas.Series):\n new_dtypes = self.dtypes.set_axis(new_columns)\n else:\n new_dtypes = None\n self.set_columns_cache(new_columns)\n # we have to set new dtypes cache after columns,\n # so the 'self.columns' and 'new_dtypes.index' indices would match\n self.set_dtypes_cache(new_dtypes)\n self.synchronize_labels(axis=1)\n\n columns = property(_get_columns, _set_columns)\n index = property(_get_index, _set_index)\n\n @property\n def axes(self):\n \"\"\"\n Get index and columns that can be accessed with an `axis` integer.\n\n Returns\n -------\n list\n List with two values: index and columns.\n \"\"\"\n return [self.index, self.columns]\n\n def get_axis(self, axis: int = 0) -> pandas.Index:\n \"\"\"\n Get index object for the requested axis.\n\n Parameters\n ----------\n axis : {0, 1}, default: 0\n\n Returns\n -------\n pandas.Index\n \"\"\"\n return self.index if axis == 0 else self.columns\n\n def _compute_axis_labels_and_lengths(self, axis: int, partitions=None):\n \"\"\"\n Compute the labels for specific `axis`.\n\n Parameters\n ----------\n axis : int\n Axis to compute labels along.\n partitions : np.ndarray, optional\n A 2D NumPy array of partitions from which labels will be grabbed.\n If not specified, partitions will be taken from `self._partitions`.\n\n Returns\n -------\n pandas.Index\n Labels for the specified `axis`.\n List of int\n Size of partitions alongside specified `axis`.\n \"\"\"\n if partitions is None:\n partitions = self._partitions\n new_index, internal_idx = self._partition_mgr_cls.get_indices(axis, partitions)\n return new_index, list(map(len, internal_idx))\n\n def _filter_empties(self, compute_metadata=True):\n \"\"\"\n Remove empty partitions from `self._partitions` to avoid triggering excess computation.\n\n Parameters\n ----------\n compute_metadata : bool, default: True\n Trigger the computations for partition sizes and labels if they're not done already.\n \"\"\"\n if not compute_metadata and (\n self._row_lengths_cache is None or self._column_widths_cache is None\n ):\n # do not trigger the computations\n return\n\n if (\n self.has_materialized_index\n and len(self.index) == 0\n or self.has_materialized_columns\n and len(self.columns) == 0\n or sum(self.row_lengths) == 0\n or sum(self.column_widths) == 0\n ):\n # This is the case for an empty frame. We don't want to completely remove\n # all metadata and partitions so for the moment, we won't prune if the frame\n # is empty.\n # TODO: Handle empty dataframes better\n return\n self._partitions = np.array(\n [\n [\n self._partitions[i][j]\n for j in range(len(self._partitions[i]))\n if j < len(self.column_widths) and self.column_widths[j] != 0\n ]\n for i in range(len(self._partitions))\n if i < len(self.row_lengths) and self.row_lengths[i] != 0\n ]\n )\n new_col_widths = [w for w in self.column_widths if w != 0]\n new_row_lengths = [r for r in self.row_lengths if r != 0]\n\n # check whether an axis partitioning was modified and if we should reset the lengths id for 'ModinIndex'\n if new_col_widths != self.column_widths:\n self.set_columns_cache(self.copy_columns_cache(copy_lengths=False))\n if new_row_lengths != self.row_lengths:\n self.set_index_cache(self.copy_index_cache(copy_lengths=False))\n\n self._column_widths_cache = new_col_widths\n self._row_lengths_cache = new_row_lengths\n\n def synchronize_labels(self, axis=None):\n \"\"\"\n Set the deferred axes variables for the ``PandasDataframe``.\n\n Parameters\n ----------\n axis : int, optional\n The deferred axis.\n 0 for the index, 1 for the columns.\n \"\"\"\n if axis is None:\n self._deferred_index = True\n self._deferred_column = True\n elif axis == 0:\n self._deferred_index = True\n else:\n self._deferred_column = True\n\n def _propagate_index_objs(self, axis=None) -> None:\n \"\"\"\n Synchronize labels by applying the index object for specific `axis` to the `self._partitions` lazily.\n\n Adds `set_axis` function to call-queue of each partition from `self._partitions`\n to apply new axis.\n\n Parameters\n ----------\n axis : int, optional\n The axis to apply to. If it's None applies to both axes.\n \"\"\"\n self._filter_empties(compute_metadata=False)\n if axis is None or axis == 0:\n cum_row_lengths = np.cumsum([0] + self.row_lengths)\n if axis is None or axis == 1:\n cum_col_widths = np.cumsum([0] + self.column_widths)\n\n if axis is None:\n\n def apply_idx_objs(df, idx, cols):\n # We should make at least one copy to avoid the data modification problem\n # that may arise when sharing buffers from distributed storage\n # (zero-copy pickling).\n return df.set_axis(idx, axis=\"index\").set_axis(\n cols, axis=\"columns\", copy=False\n )\n\n self._partitions = np.array(\n [\n [\n self._partitions[i][j].add_to_apply_calls(\n apply_idx_objs,\n idx=self.index[\n slice(cum_row_lengths[i], cum_row_lengths[i + 1])\n ],\n cols=self.columns[\n slice(cum_col_widths[j], cum_col_widths[j + 1])\n ],\n length=self.row_lengths[i],\n width=self.column_widths[j],\n )\n for j in range(len(self._partitions[i]))\n ]\n for i in range(len(self._partitions))\n ]\n )\n self._deferred_index = False\n self._deferred_column = False\n elif axis == 0:\n\n def apply_idx_objs(df, idx):\n return df.set_axis(idx, axis=\"index\")\n\n self._partitions = np.array(\n [\n [\n self._partitions[i][j].add_to_apply_calls(\n apply_idx_objs,\n idx=self.index[\n slice(cum_row_lengths[i], cum_row_lengths[i + 1])\n ],\n length=self.row_lengths[i],\n width=(\n self.column_widths[j]\n if self._column_widths_cache is not None\n else None\n ),\n )\n for j in range(len(self._partitions[i]))\n ]\n for i in range(len(self._partitions))\n ]\n )\n self._deferred_index = False\n elif axis == 1:\n\n def apply_idx_objs(df, cols):\n return df.set_axis(cols, axis=\"columns\")\n\n self._partitions = np.array(\n [\n [\n self._partitions[i][j].add_to_apply_calls(\n apply_idx_objs,\n cols=self.columns[\n slice(cum_col_widths[j], cum_col_widths[j + 1])\n ],\n length=(\n self.row_lengths[i]\n if self._row_lengths_cache is not None\n else None\n ),\n width=self.column_widths[j],\n )\n for j in range(len(self._partitions[i]))\n ]\n for i in range(len(self._partitions))\n ]\n )\n self._deferred_column = False\n else:\n ErrorMessage.catch_bugs_and_request_email(\n axis is not None and axis not in [0, 1]\n )\n\n @lazy_metadata_decorator(apply_axis=None)\n def take_2d_labels_or_positional(\n self,\n row_labels: Optional[List[Hashable]] = None,\n row_positions: Optional[List[int]] = None,\n col_labels: Optional[List[Hashable]] = None,\n col_positions: Optional[List[int]] = None,\n ) -> PandasDataframe:\n \"\"\"\n Lazily select columns or rows from given indices.\n\n Parameters\n ----------\n row_labels : list of hashable, optional\n The row labels to extract.\n row_positions : list-like of ints, optional\n The row positions to extract.\n col_labels : list of hashable, optional\n The column labels to extract.\n col_positions : list-like of ints, optional\n The column positions to extract.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe from the mask provided.\n\n Notes\n -----\n If both `row_labels` and `row_positions` are provided, a ValueError is raised.\n The same rule applies for `col_labels` and `col_positions`.\n \"\"\"\n if check_both_not_none(row_labels, row_positions):\n raise ValueError(\n \"Both row_labels and row_positions were provided - \"\n + \"please provide only one of row_labels and row_positions.\"\n )\n if check_both_not_none(col_labels, col_positions):\n raise ValueError(\n \"Both col_labels and col_positions were provided - \"\n + \"please provide only one of col_labels and col_positions.\"\n )\n\n if row_labels is not None:\n # Get numpy array of positions of values from `row_labels`\n if isinstance(self.index, pandas.MultiIndex):\n row_positions = np.zeros(len(row_labels), dtype=\"int64\")\n # we can't use .get_locs(row_labels) because the function\n # requires a different format for row_labels\n for idx, label in enumerate(row_labels):\n if isinstance(label, str):\n label = [label]\n # get_loc can return slice that _take_2d_positional can't handle\n row_positions[idx] = self.index.get_locs(label)[0]\n else:\n row_positions = self.index.get_indexer_for(row_labels)\n\n if col_labels is not None:\n # Get numpy array of positions of values from `col_labels`\n if isinstance(self.columns, pandas.MultiIndex):\n col_positions = np.zeros(len(col_labels), dtype=\"int64\")\n # we can't use .get_locs(col_labels) because the function\n # requires a different format for row_labels\n for idx, label in enumerate(col_labels):\n if isinstance(label, str):\n label = [label]\n # get_loc can return slice that _take_2d_positional can't handle\n col_positions[idx] = self.columns.get_locs(label)[0]\n else:\n col_positions = self.columns.get_indexer_for(col_labels)\n\n return self._take_2d_positional(row_positions, col_positions)\n\n def _get_sorted_positions(self, positions):\n \"\"\"\n Sort positions if necessary.\n\n Parameters\n ----------\n positions : Sequence[int]\n\n Returns\n -------\n Sequence[int]\n \"\"\"\n # Helper for take_2d_positional\n if is_range_like(positions) and positions.step > 0:\n sorted_positions = positions\n else:\n sorted_positions = np.sort(positions)\n return sorted_positions\n\n def _get_new_lengths(self, partitions_dict, *, axis: int) -> List[int]:\n \"\"\"\n Find lengths of new partitions.\n\n Parameters\n ----------\n partitions_dict : dict\n axis : int\n\n Returns\n -------\n list[int]\n \"\"\"\n # Helper for take_2d_positional\n if axis == 0:\n axis_lengths = self.row_lengths\n else:\n axis_lengths = self.column_widths\n\n new_lengths = [\n len(\n # Row lengths for slice are calculated as the length of the slice\n # on the partition. Often this will be the same length as the current\n # length, but sometimes it is different, thus the extra calculation.\n range(*part_indexer.indices(axis_lengths[part_idx]))\n if isinstance(part_indexer, slice)\n else part_indexer\n )\n for part_idx, part_indexer in partitions_dict.items()\n ]\n return new_lengths\n\n def _get_new_index_obj(\n self, positions, sorted_positions, axis: int\n ) -> tuple[pandas.Index, slice | npt.NDArray[np.intp]]:\n \"\"\"\n Find the new Index object for take_2d_positional result.\n\n Parameters\n ----------\n positions : Sequence[int]\n sorted_positions : Sequence[int]\n axis : int\n\n Returns\n -------\n pandas.Index\n slice or Sequence[int]\n \"\"\"\n # Helper for take_2d_positional\n # Use the slice to calculate the new columns\n if axis == 0:\n idx = self.index\n else:\n idx = self.columns\n\n # TODO: Support fast processing of negative-step ranges\n if is_range_like(positions) and positions.step > 0:\n # pandas Index is more likely to preserve its metadata if the indexer\n # is slice\n monotonic_idx = slice(positions.start, positions.stop, positions.step)\n else:\n monotonic_idx = np.asarray(sorted_positions, dtype=np.intp)\n\n new_idx = idx[monotonic_idx]\n return new_idx, monotonic_idx\n\n def _take_2d_positional(\n self,\n row_positions: Optional[List[int]] = None,\n col_positions: Optional[List[int]] = None,\n ) -> PandasDataframe:\n \"\"\"\n Lazily select columns or rows from given indices.\n\n Parameters\n ----------\n row_positions : list-like of ints, optional\n The row positions to extract.\n col_positions : list-like of ints, optional\n The column positions to extract.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe from the mask provided.\n \"\"\"\n indexers = []\n for axis, indexer in enumerate((row_positions, col_positions)):\n if is_range_like(indexer):\n if indexer.step == 1 and len(indexer) == len(self.get_axis(axis)):\n # By this function semantics, `None` indexer is a full-axis access\n indexer = None\n elif indexer is not None and not isinstance(indexer, pandas.RangeIndex):\n # Pure python's range is not fully compatible with a list of ints,\n # converting it to ``pandas.RangeIndex``` that is compatible.\n indexer = pandas.RangeIndex(\n indexer.start, indexer.stop, indexer.step\n )\n else:\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=not (indexer is None or is_list_like(indexer)),\n extra_log=\"Mask takes only list-like numeric indexers, \"\n + f\"received: {type(indexer)}\",\n )\n if isinstance(indexer, list):\n indexer = np.array(indexer, dtype=np.int64)\n indexers.append(indexer)\n row_positions, col_positions = indexers\n\n if col_positions is None and row_positions is None:\n return self.copy()\n\n # quite fast check that allows skip sorting\n must_sort_row_pos = row_positions is not None and not np.all(\n row_positions[1:] >= row_positions[:-1]\n )\n must_sort_col_pos = col_positions is not None and not np.all(\n col_positions[1:] >= col_positions[:-1]\n )\n\n if col_positions is None and row_positions is not None:\n # Check if the optimization that first takes part of the data using the mask\n # operation so that later less data is concatenated into a whole column is useful.\n # In the case when only a small portion of the data is discarded, the overhead of the\n # engine (for putting data in and out of storage) can exceed the resulting speedup.\n all_rows = None\n if self.has_materialized_index:\n all_rows = len(self.index)\n elif self._row_lengths_cache or must_sort_row_pos:\n all_rows = sum(self.row_lengths)\n\n # 'base_num_cols' specifies the number of columns that the dataframe should have\n # in order to jump to 'reordered_labels' in case of len(row_positions) / len(self) >= base_ratio;\n # these variables may be a subject to change in order to tune performance more accurately\n base_num_cols = 10\n base_ratio = 0.2\n # Example:\n # len(self.columns): 10 == base_num_cols -> min ratio to jump to reorder_labels: 0.2 == base_ratio\n # len(self.columns): 15 -> min ratio to jump to reorder_labels: 0.3\n # len(self.columns): 20 -> min ratio to jump to reorder_labels: 0.4\n # ...\n # len(self.columns): 49 -> min ratio to jump to reorder_labels: 0.98\n # len(self.columns): 50 -> min ratio to jump to reorder_labels: 1.0\n # len(self.columns): 55 -> min ratio to jump to reorder_labels: 1.0\n # ...\n if (all_rows and len(row_positions) > 0.9 * all_rows) or (\n must_sort_row_pos\n and len(row_positions) * base_num_cols\n >= min(\n all_rows * len(self.columns) * base_ratio,\n len(row_positions) * base_num_cols,\n )\n ):\n return self._reorder_labels(\n row_positions=row_positions, col_positions=col_positions\n )\n sorted_row_positions = sorted_col_positions = None\n if row_positions is not None:\n if must_sort_row_pos:\n sorted_row_positions = self._get_sorted_positions(row_positions)\n else:\n sorted_row_positions = row_positions\n # Get dict of row_parts as {row_index: row_internal_indices}\n row_partitions_dict = self._get_dict_of_block_index(\n 0, sorted_row_positions, are_indices_sorted=True\n )\n new_row_lengths = self._get_new_lengths(row_partitions_dict, axis=0)\n new_index, _ = self._get_new_index_obj(\n row_positions, sorted_row_positions, axis=0\n )\n else:\n row_partitions_dict = {i: slice(None) for i in range(len(self._partitions))}\n new_row_lengths = self._row_lengths_cache\n new_index = self.copy_index_cache(copy_lengths=True)\n\n if col_positions is not None:\n if must_sort_col_pos:\n sorted_col_positions = self._get_sorted_positions(col_positions)\n else:\n sorted_col_positions = col_positions\n # Get dict of col_parts as {col_index: col_internal_indices}\n col_partitions_dict = self._get_dict_of_block_index(\n 1, sorted_col_positions, are_indices_sorted=True\n )\n new_col_widths = self._get_new_lengths(col_partitions_dict, axis=1)\n new_columns, monotonic_col_idx = self._get_new_index_obj(\n col_positions, sorted_col_positions, axis=1\n )\n\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=sum(new_col_widths) != len(new_columns),\n extra_log=f\"{sum(new_col_widths)} != {len(new_columns)}.\\n\"\n + f\"{col_positions}\\n{self.column_widths}\\n{col_partitions_dict}\",\n )\n\n if self.has_materialized_dtypes:\n new_dtypes = self.dtypes.iloc[monotonic_col_idx]\n elif isinstance(self._dtypes, ModinDtypes):\n try:\n supported_monotonic_col_idx = monotonic_col_idx\n if isinstance(monotonic_col_idx, slice):\n supported_monotonic_col_idx = pandas.RangeIndex(\n monotonic_col_idx.start,\n monotonic_col_idx.stop,\n monotonic_col_idx.step,\n ).to_list()\n new_dtypes = self._dtypes.lazy_get(\n supported_monotonic_col_idx, numeric_index=True\n )\n # can raise either on missing cache or on duplicated labels\n except (ValueError, NotImplementedError):\n new_dtypes = None\n else:\n new_dtypes = None\n else:\n col_partitions_dict = {\n i: slice(None) for i in range(len(self._partitions.T))\n }\n new_col_widths = self._column_widths_cache\n new_columns = self.copy_columns_cache(copy_lengths=True)\n new_dtypes = self.copy_dtypes_cache()\n\n new_partitions = np.array(\n [\n [\n self._partitions[row_idx][col_idx].mask(\n row_internal_indices, col_internal_indices\n )\n for col_idx, col_internal_indices in col_partitions_dict.items()\n ]\n for row_idx, row_internal_indices in row_partitions_dict.items()\n ]\n )\n intermediate = self.__constructor__(\n new_partitions,\n new_index,\n new_columns,\n new_row_lengths,\n new_col_widths,\n new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n return self._maybe_reorder_labels(\n intermediate,\n row_positions,\n col_positions,\n )\n\n def _maybe_reorder_labels(\n self,\n intermediate: PandasDataframe,\n row_positions,\n col_positions,\n ) -> PandasDataframe:\n \"\"\"\n Call re-order labels on take_2d_labels_or_positional result if necessary.\n\n Parameters\n ----------\n intermediate : PandasDataFrame\n row_positions : list-like of ints, optional\n The row positions to extract.\n col_positions : list-like of ints, optional\n The column positions to extract.\n\n Returns\n -------\n PandasDataframe\n \"\"\"\n # Check if monotonically increasing, return if it is. Fast track code path for\n # common case to keep it fast.\n if (\n row_positions is None\n # Fast range processing of non-positive-step ranges is not yet supported\n or (is_range_like(row_positions) and row_positions.step > 0)\n or len(row_positions) == 1\n or np.all(row_positions[1:] >= row_positions[:-1])\n ) and (\n col_positions is None\n # Fast range processing of non-positive-step ranges is not yet supported\n or (is_range_like(col_positions) and col_positions.step > 0)\n or len(col_positions) == 1\n or np.all(col_positions[1:] >= col_positions[:-1])\n ):\n return intermediate\n\n # The new labels are often smaller than the old labels, so we can't reuse the\n # original order values because those were mapped to the original data. We have\n # to reorder here based on the expected order from within the data.\n # To do so, we \"unsort\" the indices by using np.argsort() twice, as inspired by\n # https://stackoverflow.com/questions/2483696/undo-or-reverse-argsort-python,\n # meaning that `new_row_order` must be so `np.sort(row_positions)[new_row_order] == row_positions`\n # This is achieved by first calculating the indices which would sort `row_positions`,\n # and then by calculating new indices that would sort \"sorting indices\" themselves.\n # First argsort brings us to the proper \"index space\" (according to smaller labels count),\n # and the second re-orders them to match the original data.\n new_row_order, new_col_order = None, None\n if is_range_like(row_positions):\n if row_positions.step < 0:\n # do not need to re-order positive-step-ranges\n new_row_order = pandas.RangeIndex(len(row_positions) - 1, -1, -1)\n elif row_positions is not None:\n new_row_order = np.argsort(\n np.argsort(np.asarray(row_positions, dtype=np.intp))\n )\n if is_range_like(col_positions):\n if col_positions.step < 0:\n new_col_order = pandas.RangeIndex(len(col_positions) - 1, -1, -1)\n elif col_positions is not None:\n new_col_order = np.argsort(\n np.argsort(np.asarray(col_positions, dtype=np.intp))\n )\n return intermediate._reorder_labels(\n row_positions=new_row_order, col_positions=new_col_order\n )\n\n @lazy_metadata_decorator(apply_axis=\"rows\")\n def from_labels(self) -> PandasDataframe:\n \"\"\"\n Convert the row labels to a column of data, inserted at the first position.\n\n Gives result by similar way as `pandas.DataFrame.reset_index`. Each level\n of `self.index` will be added as separate column of data.\n\n Returns\n -------\n PandasDataframe\n A PandasDataframe with new columns from index labels.\n \"\"\"\n new_row_labels = pandas.RangeIndex(len(self.index))\n if self.index.nlevels > 1:\n level_names = [\n (\n self.index.names[i]\n if self.index.names[i] is not None\n else \"level_{}\".format(i)\n )\n for i in range(self.index.nlevels)\n ]\n else:\n level_names = [\n (\n self.index.names[0]\n if self.index.names[0] is not None\n else (\n \"index\" if \"index\" not in self.columns else \"level_{}\".format(0)\n )\n )\n ]\n names = tuple(level_names) if len(level_names) > 1 else level_names[0]\n new_dtypes = self.index.to_frame(name=names).dtypes\n try:\n new_dtypes = ModinDtypes.concat([new_dtypes, self._dtypes])\n except NotImplementedError:\n # can raise on duplicated labels\n new_dtypes = None\n\n # We will also use the `new_column_names` in the calculation of the internal metadata, so this is a\n # lightweight way of ensuring the metadata matches.\n if self.columns.nlevels > 1:\n # Column labels are different for multilevel index.\n new_column_names = pandas.MultiIndex.from_tuples(\n # Set level names on the 1st columns level and fill up empty level names with empty string.\n # Expand tuples in level names. This is how reset_index works when col_level col_fill are not specified.\n [\n tuple(\n list(level) + [\"\"] * (self.columns.nlevels - len(level))\n if isinstance(level, tuple)\n else [level] + [\"\"] * (self.columns.nlevels - 1)\n )\n for level in level_names\n ],\n names=self.columns.names,\n )\n else:\n new_column_names = pandas.Index(level_names, tupleize_cols=False)\n new_columns = new_column_names.append(self.columns)\n\n def from_labels_executor(\n df: pandas.DataFrame, **kwargs\n ) -> pandas.DataFrame: # pragma: no cover\n # Setting the names here ensures that external and internal metadata always match.\n df.index.names = new_column_names\n\n # Handling of a case when columns have the same name as one of index levels names.\n # In this case `df.reset_index` provides errors related to columns duplication.\n # This case is possible because columns metadata updating is deferred. To workaround\n # `df.reset_index` error we allow columns duplication in \"if\" branch via `concat`.\n if any(name_level in df.columns for name_level in df.index.names):\n columns_to_add = df.index.to_frame()\n columns_to_add.reset_index(drop=True, inplace=True)\n df = df.reset_index(drop=True)\n result = pandas.concat([columns_to_add, df], axis=1, copy=False)\n else:\n result = df.reset_index()\n # Put the index back to the original due to GH#4394\n result.index = df.index\n return result\n\n new_parts = self._partition_mgr_cls.apply_func_to_select_indices(\n 0,\n self._partitions,\n from_labels_executor,\n [0],\n keep_remaining=True,\n )\n new_column_widths = [\n self.index.nlevels + self.column_widths[0]\n ] + self.column_widths[1:]\n result = self.__constructor__(\n new_parts,\n new_row_labels,\n new_columns,\n row_lengths=self._row_lengths_cache,\n column_widths=new_column_widths,\n dtypes=new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n # Set flag for propagating deferred row labels across dataframe partitions\n result.synchronize_labels(axis=0)\n return result\n\n def to_labels(self, column_list: List[Hashable]) -> PandasDataframe:\n \"\"\"\n Move one or more columns into the row labels. Previous labels are dropped.\n\n Parameters\n ----------\n column_list : list of hashable\n The list of column names to place as the new row labels.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe that has the updated labels.\n \"\"\"\n extracted_columns = self.take_2d_labels_or_positional(\n col_labels=column_list\n ).to_pandas()\n\n if len(column_list) == 1:\n new_labels = pandas.Index(\n extracted_columns.squeeze(axis=1), name=column_list[0]\n )\n else:\n new_labels = pandas.MultiIndex.from_frame(\n extracted_columns, names=column_list\n )\n result = self.take_2d_labels_or_positional(\n col_labels=[i for i in self.columns if i not in extracted_columns.columns]\n )\n result.index = new_labels\n return result\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def _reorder_labels(self, row_positions=None, col_positions=None):\n \"\"\"\n Reorder the column and or rows in this DataFrame.\n\n Parameters\n ----------\n row_positions : list of int, optional\n The ordered list of new row orders such that each position within the list\n indicates the new position.\n col_positions : list of int, optional\n The ordered list of new column orders such that each position within the\n list indicates the new position.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe with reordered columns and/or rows.\n \"\"\"\n new_dtypes = self.copy_dtypes_cache()\n if row_positions is not None:\n # We want to preserve the frame's partitioning so passing in ``keep_partitioning=True``\n # in order to use the cached `row_lengths` values for the new frame.\n # If the frame's is re-partitioned using the \"standard\" partitioning,\n # then knowing that, we can compute new row lengths.\n ordered_rows = self._partition_mgr_cls.map_axis_partitions(\n 0,\n self._partitions,\n lambda df: df.iloc[row_positions],\n keep_partitioning=True,\n )\n row_idx = self.index[row_positions]\n\n if len(row_idx) != len(self.index):\n # The frame was re-partitioned along the 0 axis during reordering using\n # the \"standard\" partitioning. Knowing the standard partitioning scheme\n # we are able to compute new row lengths.\n new_lengths = get_length_list(\n axis_len=len(row_idx),\n num_splits=ordered_rows.shape[0],\n min_block_size=MinRowPartitionSize.get(),\n )\n else:\n # If the frame's partitioning was preserved then\n # we can use previous row lengths cache\n new_lengths = self._row_lengths_cache\n else:\n ordered_rows = self._partitions\n row_idx = self.copy_index_cache(copy_lengths=True)\n new_lengths = self._row_lengths_cache\n if col_positions is not None:\n # We want to preserve the frame's partitioning so passing in ``keep_partitioning=True``\n # in order to use the cached `column_widths` values for the new frame.\n # If the frame's is re-partitioned using the \"standard\" partitioning,\n # then knowing that, we can compute new column widths.\n ordered_cols = self._partition_mgr_cls.map_axis_partitions(\n 1,\n ordered_rows,\n lambda df: df.iloc[:, col_positions],\n keep_partitioning=True,\n )\n col_idx = self.columns[col_positions]\n if self.has_materialized_dtypes:\n new_dtypes = self.dtypes.iloc[col_positions]\n elif isinstance(self._dtypes, ModinDtypes):\n try:\n new_dtypes = self._dtypes.lazy_get(col_idx)\n # can raise on duplicated labels\n except NotImplementedError:\n new_dtypes = None\n\n if len(col_idx) != len(self.columns):\n # The frame was re-partitioned along the 1 axis during reordering using\n # the \"standard\" partitioning. Knowing the standard partitioning scheme\n # we are able to compute new column widths.\n new_widths = get_length_list(\n axis_len=len(col_idx),\n num_splits=ordered_cols.shape[1],\n min_block_size=MinColumnPartitionSize.get(),\n )\n else:\n # If the frame's partitioning was preserved then\n # we can use previous column widths cache\n new_widths = self._column_widths_cache\n else:\n ordered_cols = ordered_rows\n col_idx = self.copy_columns_cache(copy_lengths=True)\n new_widths = self._column_widths_cache\n return self.__constructor__(\n ordered_cols,\n row_idx,\n col_idx,\n new_lengths,\n new_widths,\n new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n @lazy_metadata_decorator(apply_axis=None)\n def copy(self):\n \"\"\"\n Copy this object.\n\n Returns\n -------\n PandasDataframe\n A copied version of this object.\n \"\"\"\n return self.__constructor__(\n self._partitions,\n self.copy_index_cache(copy_lengths=True),\n self.copy_columns_cache(copy_lengths=True),\n self._row_lengths_cache,\n self._column_widths_cache,\n self.copy_dtypes_cache(),\n pandas_backend=self._pandas_backend,\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def astype(self, col_dtypes, errors: str = \"raise\"):\n \"\"\"\n Convert the columns dtypes to given dtypes.\n\n Parameters\n ----------\n col_dtypes : dictionary of {col: dtype,...} or str\n Where col is the column name and dtype is a NumPy dtype.\n errors : {'raise', 'ignore'}, default: 'raise'\n Control raising of exceptions on invalid data for provided dtype.\n\n Returns\n -------\n BaseDataFrame\n Dataframe with updated dtypes.\n \"\"\"\n new_dtypes = None\n self_dtypes = self.dtypes\n # When casting to \"category\" we have to make up the whole axis partition\n # to get the properly encoded table of categories. Every block partition\n # will store the encoded table. That can lead to higher memory footprint.\n # TODO: Revisit if this hurts users.\n use_full_axis_cast = False\n if isinstance(col_dtypes, dict):\n for column, dtype in col_dtypes.items():\n if not is_dtype_equal(dtype, self_dtypes[column]):\n if new_dtypes is None:\n new_dtypes = self_dtypes.copy()\n # Update the new dtype series to the proper pandas dtype\n new_dtype = pandas.api.types.pandas_dtype(dtype)\n if self.engine == \"Dask\" and hasattr(dtype, \"_is_materialized\"):\n # FIXME: https://github.com/dask/distributed/issues/8585\n _ = dtype._materialize_categories()\n\n # We cannot infer without computing the dtype if new dtype is categorical\n if isinstance(new_dtype, pandas.CategoricalDtype):\n new_dtypes[column] = LazyProxyCategoricalDtype._build_proxy(\n # Actual parent will substitute `None` at `.set_dtypes_cache`\n parent=None,\n column_name=column,\n materializer=lambda parent, column: parent._compute_dtypes(\n columns=[column]\n )[column],\n )\n use_full_axis_cast = True\n else:\n new_dtypes[column] = new_dtype\n\n def astype_builder(df):\n \"\"\"Compute new partition frame with dtypes updated.\"\"\"\n return df.astype(\n {k: v for k, v in col_dtypes.items() if k in df}, errors=errors\n )\n\n else:\n # Assume that the dtype is a scalar.\n if not (self_dtypes == col_dtypes).all():\n new_dtypes = self_dtypes.copy()\n new_dtype = pandas.api.types.pandas_dtype(col_dtypes)\n if self.engine == \"Dask\" and hasattr(new_dtype, \"_is_materialized\"):\n # FIXME: https://github.com/dask/distributed/issues/8585\n _ = new_dtype._materialize_categories()\n if isinstance(new_dtype, pandas.CategoricalDtype):\n new_dtypes[:] = new_dtypes.to_frame().apply(\n lambda column: LazyProxyCategoricalDtype._build_proxy(\n # Actual parent will substitute `None` at `.set_dtypes_cache`\n parent=None,\n column_name=column.index[0],\n materializer=lambda parent, column: parent._compute_dtypes(\n columns=[column]\n )[column],\n )\n )[0]\n use_full_axis_cast = True\n else:\n new_dtypes[:] = new_dtype\n\n def astype_builder(df):\n \"\"\"Compute new partition frame with dtypes updated.\"\"\"\n return df.astype(col_dtypes, errors=errors)\n\n if new_dtypes is None:\n return self.copy()\n if use_full_axis_cast:\n new_frame = self._partition_mgr_cls.map_axis_partitions(\n 0, self._partitions, astype_builder, keep_partitioning=True\n )\n else:\n new_frame = self._partition_mgr_cls.lazy_map_partitions(\n self._partitions, astype_builder\n )\n return self.__constructor__(\n new_frame,\n self.copy_index_cache(copy_lengths=True),\n self.copy_columns_cache(copy_lengths=True),\n self._row_lengths_cache,\n self._column_widths_cache,\n new_dtypes,\n pandas_backend=get_pandas_backend(new_dtypes),\n )\n\n def numeric_columns(self, include_bool=True):\n \"\"\"\n Return the names of numeric columns in the frame.\n\n Parameters\n ----------\n include_bool : bool, default: True\n Whether to consider boolean columns as numeric.\n\n Returns\n -------\n list\n List of column names.\n \"\"\"\n columns = []\n for col, dtype in zip(self.columns, self.dtypes):\n if is_numeric_dtype(dtype) and (\n include_bool or (not include_bool and dtype != np.bool_)\n ):\n columns.append(col)\n return columns\n\n def _get_dict_of_block_index(self, axis, indices, are_indices_sorted=False):\n \"\"\"\n Convert indices to an ordered dict mapping partition (or block) index to internal indices in said partition.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis along which to get the indices (0 - rows, 1 - columns).\n indices : list of int, slice\n A list of global indices to convert.\n are_indices_sorted : bool, default: False\n Flag indicating whether the `indices` sequence is sorted by ascending or not.\n Note: the internal algorithm requires for the `indices` to be sorted, this\n flag is used for optimization in order to not sort already sorted data.\n Be careful when passing ``True`` for this flag, if the data appears to be unsorted\n with the flag set to ``True`` this would lead to undefined behavior.\n\n Returns\n -------\n dict\n A mapping from partition index to list of internal indices which correspond to `indices` in each\n partition.\n \"\"\"\n # TODO: Support handling of slices with specified 'step'. For now, converting them into a range\n if isinstance(indices, slice) and (\n indices.step is not None and indices.step != 1\n ):\n indices = range(*indices.indices(len(self.get_axis(axis))))\n # Fasttrack slices\n if isinstance(indices, slice) or (is_range_like(indices) and indices.step == 1):\n # Converting range-like indexer to slice\n indices = slice(indices.start, indices.stop, indices.step)\n if is_full_grab_slice(indices, sequence_len=len(self.get_axis(axis))):\n return dict(\n zip(\n range(self._partitions.shape[axis]),\n [slice(None)] * self._partitions.shape[axis],\n )\n )\n # Empty selection case\n if indices.start == indices.stop and indices.start is not None:\n return dict()\n if indices.start is None or indices.start == 0:\n last_part, last_idx = list(\n self._get_dict_of_block_index(axis, [indices.stop]).items()\n )[0]\n dict_of_slices = dict(zip(range(last_part), [slice(None)] * last_part))\n dict_of_slices.update({last_part: slice(last_idx[0])})\n return dict_of_slices\n elif indices.stop is None or indices.stop >= len(self.get_axis(axis)):\n first_part, first_idx = list(\n self._get_dict_of_block_index(axis, [indices.start]).items()\n )[0]\n dict_of_slices = dict({first_part: slice(first_idx[0], None)})\n num_partitions = np.size(self._partitions, axis=axis)\n part_list = range(first_part + 1, num_partitions)\n dict_of_slices.update(\n dict(zip(part_list, [slice(None)] * len(part_list)))\n )\n return dict_of_slices\n else:\n first_part, first_idx = list(\n self._get_dict_of_block_index(axis, [indices.start]).items()\n )[0]\n last_part, last_idx = list(\n self._get_dict_of_block_index(axis, [indices.stop]).items()\n )[0]\n if first_part == last_part:\n return dict({first_part: slice(first_idx[0], last_idx[0])})\n else:\n if last_part - first_part == 1:\n return dict(\n # FIXME: this dictionary creation feels wrong - it might not maintain the order\n {\n first_part: slice(first_idx[0], None),\n last_part: slice(None, last_idx[0]),\n }\n )\n else:\n dict_of_slices = dict({first_part: slice(first_idx[0], None)})\n part_list = range(first_part + 1, last_part)\n dict_of_slices.update(\n dict(zip(part_list, [slice(None)] * len(part_list)))\n )\n dict_of_slices.update({last_part: slice(None, last_idx[0])})\n return dict_of_slices\n if isinstance(indices, list):\n # Converting python list to numpy for faster processing\n indices = np.array(indices, dtype=np.int64)\n # Fasttrack empty numpy array\n if isinstance(indices, np.ndarray) and indices.size == 0:\n # This will help preserve metadata stored in empty dataframes (indexes and dtypes)\n # Otherwise, we will get an empty `new_partitions` array, from which it will\n # no longer be possible to obtain metadata\n return dict([(0, np.array([], dtype=np.int64))])\n negative_mask = np.less(indices, 0)\n has_negative = np.any(negative_mask)\n if has_negative:\n # We're going to modify 'indices' inplace in a numpy way, so doing a copy/converting indices to numpy.\n indices = (\n indices.copy()\n if isinstance(indices, np.ndarray)\n else np.array(indices, dtype=np.int64)\n )\n indices[negative_mask] = indices[negative_mask] % len(self.get_axis(axis))\n # If the `indices` array was modified because of the negative indices conversion\n # then the original order was broken and so we have to sort anyway:\n if has_negative or not are_indices_sorted:\n indices = np.sort(indices)\n if axis == 0:\n bins = np.array(self.row_lengths)\n else:\n bins = np.array(self.column_widths)\n # INT_MAX to make sure we don't try to compute on partitions that don't exist.\n cumulative = np.append(bins[:-1].cumsum(), np.iinfo(bins.dtype).max)\n\n def internal(block_idx: int, global_index):\n \"\"\"Transform global index to internal one for given block (identified by its index).\"\"\"\n return (\n global_index\n if not block_idx\n else np.subtract(\n global_index, cumulative[min(block_idx, len(cumulative) - 1) - 1]\n )\n )\n\n partition_ids = np.digitize(indices, cumulative)\n count_for_each_partition = np.array(\n [(partition_ids == i).sum() for i in range(len(cumulative))]\n ).cumsum()\n # Compute the internal indices and pair those with the partition index.\n # If the first partition has any values we need to return, compute those\n # first to make the list comprehension easier. Otherwise, just append the\n # rest of the values to an empty list.\n if count_for_each_partition[0] > 0:\n first_partition_indices = [\n (0, internal(0, indices[slice(count_for_each_partition[0])]))\n ]\n else:\n first_partition_indices = []\n partition_ids_with_indices = first_partition_indices + [\n (\n i,\n internal(\n i,\n indices[\n slice(\n count_for_each_partition[i - 1],\n count_for_each_partition[i],\n )\n ],\n ),\n )\n for i in range(1, len(count_for_each_partition))\n if count_for_each_partition[i] > count_for_each_partition[i - 1]\n ]\n return dict(partition_ids_with_indices)\n\n @staticmethod\n def _join_index_objects(axis, indexes, how, sort, fill_value=None):\n \"\"\"\n Join the pair of index objects (columns or rows) by a given strategy.\n\n Unlike Index.join() in pandas, if `axis` is 1, `sort` is False,\n and `how` is \"outer\", the result will _not_ be sorted.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis index object to join (0 - rows, 1 - columns).\n indexes : list(Index)\n The indexes to join on.\n how : {'left', 'right', 'inner', 'outer', None}\n The type of join to join to make. If `None` then joined index\n considered to be the first index in the `indexes` list.\n sort : boolean\n Whether or not to sort the joined index.\n fill_value : any, optional\n Value to use for missing values.\n\n Returns\n -------\n (Index, func)\n Joined index with make_reindexer func.\n \"\"\"\n assert isinstance(indexes, list)\n\n # define helper functions\n def merge(left_index, right_index):\n \"\"\"Combine a pair of indices depending on `axis`, `how` and `sort` from outside.\"\"\"\n if axis == 1 and how == \"outer\" and not sort:\n return left_index.union(right_index, sort=False)\n else:\n return left_index.join(right_index, how=how, sort=sort)\n\n # define condition for joining indexes\n all_indices_equal = all(indexes[0].equals(index) for index in indexes[1:])\n do_join_index = how is not None and not all_indices_equal\n\n # define condition for joining indexes with getting indexers\n need_indexers = (\n axis == 0\n and not all_indices_equal\n and any(not index.is_unique for index in indexes)\n )\n indexers = None\n\n # perform joining indexes\n if do_join_index:\n if len(indexes) == 2 and need_indexers:\n # in case of count of indexes > 2 we should perform joining all indexes\n # after that get indexers\n # in the fast path we can obtain joined_index and indexers in one call\n indexers = [None, None]\n joined_index, indexers[0], indexers[1] = indexes[0].join(\n indexes[1], how=how, sort=sort, return_indexers=True\n )\n else:\n joined_index = indexes[0]\n # TODO: revisit for performance\n for index in indexes[1:]:\n joined_index = merge(joined_index, index)\n else:\n joined_index = indexes[0].copy()\n\n if need_indexers and indexers is None:\n indexers = [index.get_indexer_for(joined_index) for index in indexes]\n\n def make_reindexer(do_reindex: bool, frame_idx: int):\n \"\"\"Create callback that reindexes the dataframe using newly computed index.\"\"\"\n # the order of the frames must match the order of the indexes\n if not do_reindex:\n return lambda df: df\n\n if need_indexers:\n assert indexers is not None\n\n return lambda df: df._reindex_with_indexers(\n {0: [joined_index, indexers[frame_idx]]},\n copy=True,\n allow_dups=True,\n fill_value=fill_value,\n )\n return lambda df: df.reindex(joined_index, axis=axis, fill_value=fill_value)\n\n return joined_index, make_reindexer\n\n # Internal methods\n # These methods are for building the correct answer in a modular way.\n # Please be careful when changing these!\n\n def _build_treereduce_func(self, axis, func):\n \"\"\"\n Properly formats a TreeReduce result so that the partitioning is correct.\n\n Parameters\n ----------\n axis : int\n The axis along which to apply the function.\n func : callable\n The function to apply.\n\n Returns\n -------\n callable\n A function to be shipped to the partitions to be executed.\n\n Notes\n -----\n This should be used for any TreeReduce style operation that results in a\n reduced data dimensionality (dataframe -> series).\n \"\"\"\n\n def _tree_reduce_func(df, *args, **kwargs):\n \"\"\"Tree-reducer function itself executing `func`, presenting the resulting pandas.Series as pandas.DataFrame.\"\"\"\n series_result = func(df, *args, **kwargs)\n if axis == 0 and isinstance(series_result, pandas.Series):\n # In the case of axis=0, we need to keep the shape of the data\n # consistent with what we have done. In the case of a reduce, the\n # data for axis=0 should be a single value for each column. By\n # transposing the data after we convert to a DataFrame, we ensure that\n # the columns of the result line up with the columns from the data.\n # axis=1 does not have this requirement because the index already will\n # line up with the index of the data based on how pandas creates a\n # DataFrame from a Series.\n result = pandas.DataFrame(series_result).T\n result.index = [MODIN_UNNAMED_SERIES_LABEL]\n else:\n result = pandas.DataFrame(series_result)\n if isinstance(series_result, pandas.Series):\n result.columns = [MODIN_UNNAMED_SERIES_LABEL]\n return result\n\n return _tree_reduce_func\n\n def _compute_tree_reduce_metadata(self, axis, new_parts, dtypes=None):\n \"\"\"\n Compute the metadata for the result of reduce function.\n\n Parameters\n ----------\n axis : int\n The axis on which reduce function was applied.\n new_parts : NumPy 2D array\n Partitions with the result of applied function.\n dtypes : str, optional\n The data types for the result. This is an optimization\n because there are functions that always result in a particular data\n type, and this allows us to avoid (re)computing it.\n\n Returns\n -------\n PandasDataframe\n Modin series (1xN frame) containing the reduced data.\n \"\"\"\n new_axes, new_axes_lengths = [0, 0], [0, 0]\n\n new_axes[axis] = [MODIN_UNNAMED_SERIES_LABEL]\n new_axes[axis ^ 1] = self.get_axis(axis ^ 1)\n\n new_axes_lengths[axis] = [1]\n new_axes_lengths[axis ^ 1] = self._get_axis_lengths(axis ^ 1)\n\n if dtypes == \"copy\":\n dtypes = self.copy_dtypes_cache()\n elif dtypes is not None:\n dtypes = pandas.Series(\n [pandas.api.types.pandas_dtype(dtypes)] * len(new_axes[1]),\n index=new_axes[1],\n )\n\n result = self.__constructor__(\n new_parts,\n *new_axes,\n *new_axes_lengths,\n dtypes,\n pandas_backend=self._pandas_backend,\n )\n return result\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def reduce(\n self,\n axis: Union[int, Axis],\n function: Callable,\n dtypes: Optional[str] = None,\n ) -> PandasDataframe:\n \"\"\"\n Perform a user-defined aggregation on the specified axis, where the axis reduces down to a singleton. Requires knowledge of the full axis for the reduction.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to perform the reduce over.\n function : callable(row|col) -> single value\n The reduce function to apply to each column.\n dtypes : str, optional\n The data types for the result. This is an optimization\n because there are functions that always result in a particular data\n type, and this allows us to avoid (re)computing it.\n\n Returns\n -------\n PandasDataframe\n Modin series (1xN frame) containing the reduced data.\n\n Notes\n -----\n The user-defined function must reduce to a single value.\n \"\"\"\n axis = Axis(axis)\n function = self._build_treereduce_func(axis.value, function)\n new_parts = self._partition_mgr_cls.map_axis_partitions(\n axis.value, self._partitions, function\n )\n return self._compute_tree_reduce_metadata(axis.value, new_parts, dtypes=dtypes)\n\n @lazy_metadata_decorator(apply_axis=\"opposite\", axis_arg=0)\n def tree_reduce(\n self,\n axis: Union[int, Axis],\n map_func: Callable,\n reduce_func: Optional[Callable] = None,\n dtypes: Optional[str] = None,\n ) -> PandasDataframe:\n \"\"\"\n Apply function that will reduce the data to a pandas Series.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to perform the tree reduce over.\n map_func : callable(row|col) -> row|col\n Callable function to map the dataframe.\n reduce_func : callable(row|col) -> single value, optional\n Callable function to reduce the dataframe.\n If none, then apply map_func twice.\n dtypes : str, optional\n The data types for the result. This is an optimization\n because there are functions that always result in a particular data\n type, and this allows us to avoid (re)computing it.\n\n Returns\n -------\n PandasDataframe\n A new dataframe.\n \"\"\"\n axis = Axis(axis)\n map_func = self._build_treereduce_func(axis.value, map_func)\n if reduce_func is None:\n reduce_func = map_func\n else:\n reduce_func = self._build_treereduce_func(axis.value, reduce_func)\n\n map_parts = self._partition_mgr_cls.map_partitions(self._partitions, map_func)\n reduce_parts = self._partition_mgr_cls.map_axis_partitions(\n axis.value, map_parts, reduce_func\n )\n return self._compute_tree_reduce_metadata(\n axis.value, reduce_parts, dtypes=dtypes\n )\n\n @lazy_metadata_decorator(apply_axis=None)\n def map(\n self,\n func: Callable,\n dtypes: Optional[str] = None,\n new_columns: Optional[pandas.Index] = None,\n func_args=None,\n func_kwargs=None,\n lazy=False,\n ) -> PandasDataframe:\n \"\"\"\n Perform a function that maps across the entire dataset.\n\n Parameters\n ----------\n func : callable(row|col|cell) -> row|col|cell\n The function to apply.\n dtypes : dtypes of the result, optional\n The data types for the result. This is an optimization\n because there are functions that always result in a particular data\n type, and this allows us to avoid (re)computing it.\n new_columns : pandas.Index, optional\n New column labels of the result, its length has to be identical\n to the older columns. If not specified, old column labels are preserved.\n func_args : iterable, optional\n Positional arguments for the 'func' callable.\n func_kwargs : dict, optional\n Keyword arguments for the 'func' callable.\n lazy : bool, default: False\n Whether to prefer lazy execution or not.\n\n Returns\n -------\n PandasDataframe\n A new dataframe.\n \"\"\"\n map_fn = (\n self._partition_mgr_cls.lazy_map_partitions\n if lazy\n else self._partition_mgr_cls.map_partitions\n )\n new_partitions = map_fn(self._partitions, func, func_args, func_kwargs)\n\n if new_columns is not None and self.has_materialized_columns:\n assert len(new_columns) == len(\n self.columns\n ), \"New column's length must be identical to the previous columns\"\n elif new_columns is None:\n new_columns = self.copy_columns_cache(copy_lengths=True)\n if isinstance(dtypes, str) and dtypes == \"copy\":\n dtypes = self.copy_dtypes_cache()\n elif dtypes is not None and not isinstance(dtypes, pandas.Series):\n if isinstance(new_columns, ModinIndex):\n # Materializing lazy columns in order to build dtype's index\n new_columns = new_columns.get(return_lengths=False)\n dtypes = pandas.Series(\n [pandas.api.types.pandas_dtype(dtypes)] * len(new_columns),\n index=new_columns,\n )\n return self.__constructor__(\n new_partitions,\n self.copy_index_cache(copy_lengths=True),\n new_columns,\n self._row_lengths_cache,\n self._column_widths_cache,\n dtypes=dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n def window(\n self,\n axis: Union[int, Axis],\n reduce_fn: Callable,\n window_size: int,\n result_schema: Optional[Dict[Hashable, type]] = None,\n ) -> PandasDataframe:\n \"\"\"\n Apply a sliding window operator that acts as a GROUPBY on each window, and reduces down to a single row (column) per window.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to slide over.\n reduce_fn : callable(rowgroup|colgroup) -> row|col\n The reduce function to apply over the data.\n window_size : int\n The number of row/columns to pass to the function.\n (The size of the sliding window).\n result_schema : dict, optional\n Mapping from column labels to data types that represents the types of the output dataframe.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe with the reduce function applied over windows of the specified\n axis.\n\n Notes\n -----\n The user-defined reduce function must reduce each window’s column\n (row if axis=1) down to a single value.\n \"\"\"\n pass\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def fold(self, axis, func, new_index=None, new_columns=None, shape_preserved=False):\n \"\"\"\n Perform a function across an entire axis.\n\n Parameters\n ----------\n axis : int\n The axis to apply over.\n func : callable\n The function to apply.\n new_index : list-like, optional\n The index of the result.\n new_columns : list-like, optional\n The columns of the result.\n shape_preserved : bool, default: False\n Whether the shape of the dataframe is preserved or not\n after applying a function.\n\n Returns\n -------\n PandasDataframe\n A new dataframe.\n \"\"\"\n new_row_lengths = None\n new_column_widths = None\n if shape_preserved:\n if new_index is None:\n new_index = self.copy_index_cache(copy_lengths=True)\n if new_columns is None:\n new_columns = self.copy_columns_cache(copy_lengths=True)\n new_row_lengths = self._row_lengths_cache\n new_column_widths = self._column_widths_cache\n\n new_partitions = self._partition_mgr_cls.map_axis_partitions(\n axis, self._partitions, func, keep_partitioning=True\n )\n return self.__constructor__(\n new_partitions,\n new_index,\n new_columns,\n row_lengths=new_row_lengths,\n column_widths=new_column_widths,\n pandas_backend=self._pandas_backend,\n )\n\n def infer_objects(self) -> PandasDataframe:\n \"\"\"\n Attempt to infer better dtypes for object columns.\n\n Attempts soft conversion of object-dtyped columns, leaving non-object and unconvertible\n columns unchanged. The inference rules are the same as during normal Series/DataFrame\n construction.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe with the inferred schema.\n \"\"\"\n obj_cols = [\n col for col, dtype in enumerate(self.dtypes) if is_object_dtype(dtype)\n ]\n return self.infer_types(obj_cols)\n\n def infer_types(self, col_labels: List[str]) -> PandasDataframe:\n \"\"\"\n Determine the compatible type shared by all values in the specified columns, and coerce them to that type.\n\n Parameters\n ----------\n col_labels : list\n List of column labels to infer and induce types over.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe with the inferred schema.\n \"\"\"\n # Compute dtypes on the specified columns, and then set those dtypes on a new frame\n new_cols = self.take_2d_labels_or_positional(col_labels=col_labels)\n new_cols_dtypes = new_cols.tree_reduce(0, pandas.DataFrame.infer_objects).dtypes\n new_dtypes = self.dtypes.copy()\n new_dtypes[col_labels] = new_cols_dtypes\n return self.__constructor__(\n self._partitions,\n self.copy_index_cache(copy_lengths=True),\n self.copy_columns_cache(copy_lengths=True),\n self._row_lengths_cache,\n self._column_widths_cache,\n new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n def join(\n self,\n axis: Union[int, Axis],\n condition: Callable,\n other: ModinDataframe,\n join_type: Union[str, JoinType],\n ) -> PandasDataframe:\n \"\"\"\n Join this dataframe with the other.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to perform the join on.\n condition : callable\n Function that determines which rows should be joined. The condition can be a\n simple equality, e.g. \"left.col1 == right.col1\" or can be arbitrarily complex.\n other : ModinDataframe\n The other data to join with, i.e. the right dataframe.\n join_type : string {\"inner\", \"left\", \"right\", \"outer\"} or modin.core.dataframe.base.utils.JoinType\n The type of join to perform.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe that is the result of applying the specified join over the two\n dataframes.\n\n Notes\n -----\n During the join, this dataframe is considered the left, while the other is\n treated as the right.\n\n Only inner joins, left outer, right outer, and full outer joins are currently supported.\n Support for other join types (e.g. natural join) may be implemented in the future.\n \"\"\"\n pass\n\n def rename(\n self,\n new_row_labels: Optional[Union[Dict[Hashable, Hashable], Callable]] = None,\n new_col_labels: Optional[Union[Dict[Hashable, Hashable], Callable]] = None,\n ) -> PandasDataframe:\n \"\"\"\n Replace the row and column labels with the specified new labels.\n\n Parameters\n ----------\n new_row_labels : dictionary or callable, optional\n Mapping or callable that relates old row labels to new labels.\n new_col_labels : dictionary or callable, optional\n Mapping or callable that relates old col labels to new labels.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe with the new row and column labels.\n \"\"\"\n result = self.copy()\n if new_row_labels is not None:\n if callable(new_row_labels):\n new_row_labels = result.index.map(new_row_labels)\n result.index = new_row_labels\n if new_col_labels is not None:\n if callable(new_col_labels):\n new_col_labels = result.columns.map(new_col_labels)\n result.columns = new_col_labels\n return result\n\n def combine_and_apply(\n self, func, new_index=None, new_columns=None, new_dtypes=None\n ):\n \"\"\"\n Combine all partitions into a single big one and apply the passed function to it.\n\n Use this method with care as it collects all the data on the same worker,\n it's only recommended to use this method on small or reduced datasets.\n\n Parameters\n ----------\n func : callable(pandas.DataFrame) -> pandas.DataFrame\n A function to apply to the combined partition.\n new_index : sequence, optional\n Index of the result.\n new_columns : sequence, optional\n Columns of the result.\n new_dtypes : dict-like, optional\n Dtypes of the result.\n\n Returns\n -------\n PandasDataframe\n \"\"\"\n if self._partitions.shape[1] > 1:\n new_partitions = self._partition_mgr_cls.row_partitions(self._partitions)\n new_partitions = np.array([[partition] for partition in new_partitions])\n modin_frame = self.__constructor__(\n new_partitions,\n self.copy_index_cache(copy_lengths=True),\n self.copy_columns_cache(),\n self._row_lengths_cache,\n [len(self.columns)] if self.has_materialized_columns else None,\n self.copy_dtypes_cache(),\n pandas_backend=self._pandas_backend,\n )\n else:\n modin_frame = self\n return modin_frame.apply_full_axis(\n axis=0,\n func=func,\n new_index=new_index,\n new_columns=new_columns,\n dtypes=new_dtypes,\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def _apply_func_to_range_partitioning(\n self,\n key_columns,\n func,\n ascending=True,\n preserve_columns=False,\n data=None,\n data_key_columns=None,\n level=None,\n shuffle_func_cls=ShuffleSortFunctions,\n **kwargs,\n ):\n \"\"\"\n Reshuffle data so it would be range partitioned and then apply the passed function row-wise.\n\n Parameters\n ----------\n key_columns : list of hashables\n Columns to build the range partitioning for. Can't be specified along with `level`.\n func : callable(pandas.DataFrame) -> pandas.DataFrame\n Function to apply against partitions.\n ascending : bool, default: True\n Whether the range should be built in ascending or descending order.\n preserve_columns : bool, default: False\n If the columns cache should be preserved (specify this flag if `func` doesn't change column labels).\n data : PandasDataframe, optional\n Dataframe to range-partition along with the `self` frame. If specified, the `func` will recieve\n a dataframe with an additional MultiIndex level in columns that separates `self` and `data`:\n ``df[\"grouper\"] # self`` and ``df[\"data\"] # data``.\n data_key_columns : list of hashables, optional\n Additional key columns from `data`. Will be combined with `key_columns`.\n level : list of ints or labels, optional\n Index level(s) to build the range partitioning for. Can't be specified along with `key_columns`.\n shuffle_func_cls : cls, default: ShuffleSortFunctions\n A class implementing ``modin.core.dataframe.pandas.utils.ShuffleFunctions`` to be used\n as a shuffle function.\n **kwargs : dict\n Additional arguments to forward to the range builder function.\n\n Returns\n -------\n PandasDataframe\n A new dataframe.\n \"\"\"\n if data is not None:\n # adding an extra MultiIndex level in order to separate `self grouper` from the `data`\n # after concatenation\n new_grouper_cols = pandas.MultiIndex.from_tuples(\n [\n (\"grouper\", *col) if isinstance(col, tuple) else (\"grouper\", col)\n for col in self.columns\n ]\n )\n grouper = self.copy()\n grouper.columns = new_grouper_cols\n\n new_data_cols = pandas.MultiIndex.from_tuples(\n [\n (\"data\", *col) if isinstance(col, tuple) else (\"data\", col)\n for col in data.columns\n ]\n )\n data = data.copy()\n data.columns = new_data_cols\n\n grouper = grouper.concat(axis=1, others=[data], how=\"right\", sort=False)\n\n # since original column names were modified, have to modify 'key_columns' as well\n key_columns = [\n (\"grouper\", *col) if isinstance(col, tuple) else (\"grouper\", col)\n for col in key_columns\n ]\n if data_key_columns is None:\n data_key_columns = []\n else:\n data_key_columns = [\n (\"data\", *col) if isinstance(col, tuple) else (\"data\", col)\n for col in data_key_columns\n ]\n key_columns += data_key_columns\n else:\n grouper = self\n\n # If there's only one row partition can simply apply the function row-wise without the need to reshuffle\n if grouper._partitions.shape[0] == 1:\n result = grouper.apply_full_axis(\n axis=1,\n func=func,\n new_columns=grouper.copy_columns_cache() if preserve_columns else None,\n )\n if preserve_columns:\n result._set_axis_lengths_cache(grouper._column_widths_cache, axis=1)\n return result\n\n # don't want to inherit over-partitioning so doing this 'min' check\n ideal_num_new_partitions = min(len(grouper._partitions), NPartitions.get())\n m = len(grouper) / ideal_num_new_partitions\n sampling_probability = (1 / m) * np.log(ideal_num_new_partitions * len(grouper))\n # If this df is overpartitioned, we try to sample each partition with probability\n # greater than 1, which leads to an error. In this case, we can do one of the following\n # two things. If there is only enough rows for one partition, and we have only 1 column\n # partition, we can just combine the overpartitioned df into one partition, and sort that\n # partition. If there is enough data for more than one partition, we can tell the sorting\n # algorithm how many partitions we want to end up with, so it samples and finds pivots\n # according to that.\n if sampling_probability >= 1:\n from modin.config import MinRowPartitionSize\n\n ideal_num_new_partitions = round(len(grouper) / MinRowPartitionSize.get())\n if len(grouper) < MinRowPartitionSize.get() or ideal_num_new_partitions < 2:\n # If the data is too small, we shouldn't try reshuffling/repartitioning but rather\n # simply combine all partitions and apply the sorting to the whole dataframe\n return grouper.combine_and_apply(func=func)\n\n if ideal_num_new_partitions < len(grouper._partitions):\n if len(grouper._partitions) % ideal_num_new_partitions == 0:\n joining_partitions = np.split(\n grouper._partitions, ideal_num_new_partitions\n )\n else:\n step = round(len(grouper._partitions) / ideal_num_new_partitions)\n joining_partitions = np.split(\n grouper._partitions,\n range(step, len(grouper._partitions), step),\n )\n\n new_partitions = np.array(\n [\n grouper._partition_mgr_cls.column_partitions(\n ptn_grp, full_axis=False\n )\n for ptn_grp in joining_partitions\n ]\n )\n else:\n new_partitions = grouper._partitions\n else:\n new_partitions = grouper._partitions\n\n shuffling_functions = shuffle_func_cls(\n grouper,\n key_columns,\n ascending[0] if is_list_like(ascending) else ascending,\n ideal_num_new_partitions,\n level=level,\n **kwargs,\n )\n\n if key_columns:\n # here we want to get indices of those partitions that hold the key columns\n key_indices = grouper.columns.get_indexer_for(key_columns)\n partition_indices = np.unique(\n np.digitize(key_indices, np.cumsum(grouper.column_widths))\n )\n elif level is not None:\n # each partition contains an index, so taking the first one\n partition_indices = [0]\n else:\n raise ValueError(\"Must specify either 'level' or 'key_columns'\")\n\n new_partitions = grouper._partition_mgr_cls.shuffle_partitions(\n new_partitions,\n partition_indices,\n shuffling_functions,\n func,\n )\n\n result = grouper.__constructor__(new_partitions)\n if preserve_columns:\n result.set_columns_cache(grouper.copy_columns_cache())\n # We perform the final steps of the sort on full axis partitions, so we know that the\n # length of each partition is the full length of the dataframe.\n if grouper.has_materialized_columns:\n result._set_axis_lengths_cache([len(grouper.columns)], axis=1)\n return result\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def sort_by(\n self,\n axis: Union[int, Axis],\n columns: Union[str, List[str]],\n ascending: bool = True,\n **kwargs,\n ) -> PandasDataframe:\n \"\"\"\n Logically reorder rows (columns if axis=1) lexicographically by the data in a column or set of columns.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to perform the sort over.\n columns : string or list\n Column label(s) to use to determine lexicographical ordering.\n ascending : boolean, default: True\n Whether to sort in ascending or descending order.\n **kwargs : dict\n Keyword arguments to pass when sorting partitions.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe sorted into lexicographical order by the specified column(s).\n \"\"\"\n if not isinstance(columns, list):\n columns = [columns]\n\n def sort_function(df): # pragma: no cover\n # When we do a sort on the result of Series.value_counts, we don't rename the index until\n # after everything is done, which causes an error when sorting the partitions, since the\n # index and the column share the same name, when in actuality, the index's name should be\n # None. This fixes the indexes name beforehand in that case, so that the sort works.\n index_renaming = None\n if any(name in df.columns for name in df.index.names):\n index_renaming = df.index.names\n df.index = df.index.set_names([None] * len(df.index.names))\n df = df.sort_values(by=columns, ascending=ascending, **kwargs)\n if index_renaming is not None:\n df.index = df.index.set_names(index_renaming)\n return df\n\n # If this df is empty, we don't want to try and shuffle or sort.\n if len(self.get_axis(1)) == 0 or len(self) == 0:\n return self.copy()\n\n axis = Axis(axis)\n if axis != Axis.ROW_WISE:\n raise NotImplementedError(\n f\"Algebra sort only implemented row-wise. {axis.name} sort not implemented yet!\"\n )\n\n result = self._apply_func_to_range_partitioning(\n key_columns=[columns[0]],\n func=sort_function,\n ascending=ascending,\n preserve_columns=True,\n **kwargs,\n )\n result.set_dtypes_cache(self.copy_dtypes_cache())\n\n if kwargs.get(\"ignore_index\", False):\n result.index = RangeIndex(len(self.get_axis(axis.value)))\n\n # Since the strategy to pick our pivots involves random sampling\n # we could end up picking poor pivots, leading to skew in our partitions.\n # We should add a fix to check if there is skew in the partitions and rebalance\n # them if necessary. Calling `rebalance_partitions` won't do this, since it only\n # resolves the case where there isn't the right amount of partitions - not where\n # there is skew across the lengths of partitions.\n return result\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def filter(self, axis: Union[Axis, int], condition: Callable) -> PandasDataframe:\n \"\"\"\n Filter data based on the function provided along an entire axis.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to filter over.\n condition : callable(row|col) -> bool\n The function to use for the filter. This function should filter the\n data itself.\n\n Returns\n -------\n PandasDataframe\n A new filtered dataframe.\n \"\"\"\n axis = Axis(axis)\n assert axis in (\n Axis.ROW_WISE,\n Axis.COL_WISE,\n ), \"Axis argument to filter operator must be 0 (rows) or 1 (columns)\"\n\n new_partitions = self._partition_mgr_cls.map_axis_partitions(\n axis.value, self._partitions, condition, keep_partitioning=True\n )\n\n new_axes, new_lengths = [0, 0], [0, 0]\n\n new_axes[axis.value] = self.copy_axis_cache(axis.value, copy_lengths=True)\n new_lengths[axis.value] = (\n self._row_lengths_cache if axis.value == 0 else self._column_widths_cache\n )\n new_axes[axis.value ^ 1], new_lengths[axis.value ^ 1] = None, None\n\n return self.__constructor__(\n new_partitions,\n *new_axes,\n *new_lengths,\n self.copy_dtypes_cache() if axis == Axis.COL_WISE else None,\n pandas_backend=self._pandas_backend,\n )\n\n def filter_by_types(self, types: List[Hashable]) -> PandasDataframe:\n \"\"\"\n Allow the user to specify a type or set of types by which to filter the columns.\n\n Parameters\n ----------\n types : list\n The types to filter columns by.\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe from the filter provided.\n \"\"\"\n return self.take_2d_labels_or_positional(\n col_positions=[i for i, dtype in enumerate(self.dtypes) if dtype in types]\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def explode(self, axis: Union[int, Axis], func: Callable) -> PandasDataframe:\n \"\"\"\n Explode list-like entries along an entire axis.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis specifying how to explode. If axis=1, explode according\n to columns.\n func : callable\n The function to use to explode a single element.\n\n Returns\n -------\n PandasFrame\n A new filtered dataframe.\n \"\"\"\n axis = Axis(axis)\n partitions = self._partition_mgr_cls.map_axis_partitions(\n axis.value, self._partitions, func, keep_partitioning=True\n )\n if axis == Axis.COL_WISE:\n new_index, row_lengths = self._compute_axis_labels_and_lengths(\n 0, partitions\n )\n new_columns, column_widths = self.columns, self._column_widths_cache\n else:\n new_index, row_lengths = self.index, self._row_lengths_cache\n new_columns, column_widths = self._compute_axis_labels_and_lengths(\n 1, partitions\n )\n return self.__constructor__(\n partitions,\n new_index,\n new_columns,\n row_lengths,\n column_widths,\n pandas_backend=self._pandas_backend,\n )\n\n def combine(self) -> PandasDataframe:\n \"\"\"\n Create a single partition PandasDataframe from the partitions of the current dataframe.\n\n Returns\n -------\n PandasDataframe\n A single partition PandasDataframe.\n \"\"\"\n new_index = None\n new_columns = None\n if self._deferred_index:\n new_index = self.index\n if self._deferred_column:\n new_columns = self.columns\n partitions = self._partition_mgr_cls.combine(\n self._partitions, new_index, new_columns\n )\n result = self.__constructor__(\n partitions,\n index=self.copy_index_cache(),\n columns=self.copy_columns_cache(),\n row_lengths=(\n [sum(self._row_lengths_cache)]\n if self._row_lengths_cache is not None\n else None\n ),\n column_widths=(\n [sum(self._column_widths_cache)]\n if self._column_widths_cache is not None\n else None\n ),\n dtypes=self.copy_dtypes_cache(),\n pandas_backend=self._pandas_backend,\n )\n return result\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def apply_full_axis(\n self,\n axis,\n func,\n new_index=None,\n new_columns=None,\n apply_indices=None,\n enumerate_partitions: bool = False,\n dtypes=None,\n keep_partitioning=True,\n num_splits=None,\n sync_labels=True,\n pass_axis_lengths_to_partitions=False,\n ) -> PandasDataframe:\n \"\"\"\n Perform a function across an entire axis.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis to apply over (0 - rows, 1 - columns).\n func : callable\n The function to apply.\n new_index : list-like, optional\n The index of the result. We may know this in advance,\n and if not provided it must be computed.\n new_columns : list-like, optional\n The columns of the result. We may know this in\n advance, and if not provided it must be computed.\n apply_indices : list-like, optional\n Indices of `axis ^ 1` to apply function over.\n enumerate_partitions : bool, default: False\n Whether pass partition index into applied `func` or not.\n Note that `func` must be able to obtain `partition_idx` kwarg.\n dtypes : list-like or scalar, optional\n The data types of the result. This is an optimization\n because there are functions that always result in a particular data\n type, and allows us to avoid (re)computing it.\n keep_partitioning : boolean, default: True\n The flag to keep partition boundaries for Modin Frame if possible.\n Setting it to True disables shuffling data from one partition to another in case the resulting\n number of splits is equal to the initial number of splits.\n num_splits : int, optional\n The number of partitions to split the result into across the `axis`. If None, then the number\n of splits will be infered automatically. If `num_splits` is None and `keep_partitioning=True`\n then the number of splits is preserved.\n sync_labels : boolean, default: True\n Synchronize external indexes (`new_index`, `new_columns`) with internal indexes.\n This could be used when you're certain that the indices in partitions are equal to\n the provided hints in order to save time on syncing them.\n pass_axis_lengths_to_partitions : bool, default: False\n Whether pass partition lengths along `axis ^ 1` to the kernel `func`.\n Note that `func` must be able to obtain `df, *axis_lengths`.\n\n Returns\n -------\n PandasDataframe\n A new dataframe.\n\n Notes\n -----\n The data shape may change as a result of the function.\n \"\"\"\n return self.broadcast_apply_full_axis(\n axis=axis,\n func=func,\n new_index=new_index,\n new_columns=new_columns,\n apply_indices=apply_indices,\n enumerate_partitions=enumerate_partitions,\n dtypes=dtypes,\n other=None,\n keep_partitioning=keep_partitioning,\n num_splits=num_splits,\n sync_labels=sync_labels,\n pass_axis_lengths_to_partitions=pass_axis_lengths_to_partitions,\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def apply_full_axis_select_indices(\n self,\n axis,\n func,\n apply_indices=None,\n numeric_indices=None,\n new_index=None,\n new_columns=None,\n keep_remaining=False,\n new_dtypes: Optional[Union[pandas.Series, ModinDtypes]] = None,\n ):\n \"\"\"\n Apply a function across an entire axis for a subset of the data.\n\n Parameters\n ----------\n axis : int\n The axis to apply over.\n func : callable\n The function to apply.\n apply_indices : list-like, optional\n The labels to apply over.\n numeric_indices : list-like, optional\n The indices to apply over.\n new_index : list-like, optional\n The index of the result. We may know this in advance,\n and if not provided it must be computed.\n new_columns : list-like, optional\n The columns of the result. We may know this in\n advance, and if not provided it must be computed.\n keep_remaining : boolean, default: False\n Whether or not to drop the data that is not computed over.\n new_dtypes : ModinDtypes or pandas.Series, optional\n The data types of the result. This is an optimization\n because there are functions that always result in a particular data\n type, and allows us to avoid (re)computing it.\n\n Returns\n -------\n PandasDataframe\n A new dataframe.\n \"\"\"\n assert apply_indices is not None or numeric_indices is not None\n # Convert indices to numeric indices\n old_index = self.index if axis else self.columns\n if apply_indices is not None:\n numeric_indices = old_index.get_indexer_for(apply_indices)\n # Get the indices for the axis being applied to (it is the opposite of axis\n # being applied over)\n dict_indices = self._get_dict_of_block_index(axis ^ 1, numeric_indices)\n new_partitions = (\n self._partition_mgr_cls.apply_func_to_select_indices_along_full_axis(\n axis,\n self._partitions,\n func,\n dict_indices,\n keep_remaining=keep_remaining,\n )\n )\n # TODO Infer columns and index from `keep_remaining` and `apply_indices`\n if new_index is None:\n new_index = self.index if axis == 1 else None\n if new_columns is None:\n new_columns = self.columns if axis == 0 else None\n return self.__constructor__(\n new_partitions,\n new_index,\n new_columns,\n None,\n None,\n dtypes=new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def apply_select_indices(\n self,\n axis,\n func,\n apply_indices=None,\n row_labels=None,\n col_labels=None,\n new_index=None,\n new_columns=None,\n new_dtypes: Optional[pandas.Series] = None,\n keep_remaining=False,\n item_to_distribute=no_default,\n ) -> PandasDataframe:\n \"\"\"\n Apply a function for a subset of the data.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis to apply over.\n func : callable\n The function to apply.\n apply_indices : list-like, optional\n The labels to apply over. Must be given if axis is provided.\n row_labels : list-like, optional\n The row labels to apply over. Must be provided with\n `col_labels` to apply over both axes.\n col_labels : list-like, optional\n The column labels to apply over. Must be provided\n with `row_labels` to apply over both axes.\n new_index : list-like, optional\n The index of the result, if known in advance.\n new_columns : list-like, optional\n The columns of the result, if known in advance.\n new_dtypes : pandas.Series, optional\n The dtypes of the result, if known in advance.\n keep_remaining : boolean, default: False\n Whether or not to drop the data that is not computed over.\n item_to_distribute : np.ndarray or scalar, default: no_default\n The item to split up so it can be applied over both axes.\n\n Returns\n -------\n PandasDataframe\n A new dataframe.\n \"\"\"\n # TODO Infer columns and index from `keep_remaining` and `apply_indices`\n if new_index is None:\n new_index = self.index if axis == 1 else None\n if new_columns is None:\n new_columns = self.columns if axis == 0 else None\n if new_columns is not None and isinstance(new_dtypes, pandas.Series):\n assert new_dtypes.index.equals(\n new_columns\n ), f\"{new_dtypes=} doesn't have the same columns as in {new_columns=}\"\n\n if axis is not None:\n assert apply_indices is not None\n # Convert indices to numeric indices\n old_index = self.index if axis else self.columns\n numeric_indices = old_index.get_indexer_for(apply_indices)\n # Get indices being applied to (opposite of indices being applied over)\n dict_indices = self._get_dict_of_block_index(axis ^ 1, numeric_indices)\n new_partitions = self._partition_mgr_cls.apply_func_to_select_indices(\n axis,\n self._partitions,\n func,\n dict_indices,\n keep_remaining=keep_remaining,\n )\n # Length objects for new object creation. This is shorter than if..else\n # This object determines the lengths and widths based on the given\n # parameters and builds a dictionary used in the constructor below. 0 gives\n # the row lengths and 1 gives the column widths. Since the dimension of\n # `axis` given may have changed, we currently just recompute it.\n # TODO Determine lengths from current lengths if `keep_remaining=False`\n lengths_objs = {\n axis: (\n [len(apply_indices)]\n if not keep_remaining\n else [self.row_lengths, self.column_widths][axis]\n ),\n axis ^ 1: [self.row_lengths, self.column_widths][axis ^ 1],\n }\n return self.__constructor__(\n new_partitions,\n new_index,\n new_columns,\n lengths_objs[0],\n lengths_objs[1],\n new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n else:\n # We are applying over both axes here, so make sure we have all the right\n # variables set.\n assert row_labels is not None and col_labels is not None\n assert keep_remaining\n assert item_to_distribute is not no_default\n row_partitions_list = self._get_dict_of_block_index(0, row_labels).items()\n col_partitions_list = self._get_dict_of_block_index(1, col_labels).items()\n new_partitions = self._partition_mgr_cls.apply_func_to_indices_both_axis(\n self._partitions,\n func,\n row_partitions_list,\n col_partitions_list,\n item_to_distribute,\n # Passing caches instead of values in order to not trigger shapes recomputation\n # if they are not used inside this function.\n self._row_lengths_cache,\n self._column_widths_cache,\n )\n return self.__constructor__(\n new_partitions,\n new_index,\n new_columns,\n self._row_lengths_cache,\n self._column_widths_cache,\n new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def broadcast_apply(\n self,\n axis,\n func,\n other,\n join_type=\"left\",\n copartition=True,\n labels=\"keep\",\n dtypes=None,\n ):\n \"\"\"\n Broadcast axis partitions of `other` to partitions of `self` and apply a function.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to broadcast over.\n func : callable\n Function to apply.\n other : PandasDataframe\n Modin DataFrame to broadcast.\n join_type : str, default: \"left\"\n Type of join to apply.\n copartition : bool, default: True\n Whether to align indices/partitioning of the `self` and `other` frame.\n Disabling this may save some time, however, you have to be 100% sure that\n the indexing and partitioning are identical along the broadcasting axis,\n this might be the case for example if `other` is a projection of the `self`\n or vice-versa. If copartitioning is disabled and partitioning/indexing are\n incompatible then you may end up with undefined behavior.\n labels : {\"keep\", \"replace\", \"drop\"}, default: \"keep\"\n Whether keep labels from `self` Modin DataFrame, replace them with labels\n from joined DataFrame or drop altogether to make them be computed lazily later.\n dtypes : \"copy\", pandas.Series or None, optional\n Dtypes of the result. \"copy\" to keep old dtypes and None to compute them on demand.\n\n Returns\n -------\n PandasDataframe\n New Modin DataFrame.\n \"\"\"\n if copartition:\n # Only sort the indices if they do not match\n (\n left_parts,\n right_parts,\n joined_index,\n partition_sizes_along_axis,\n ) = self._copartition(\n axis,\n other,\n join_type,\n )\n # unwrap list returned by `copartition`.\n right_parts = right_parts[0]\n else:\n left_parts = self._partitions\n right_parts = other._partitions\n partition_sizes_along_axis, joined_index = self._get_axis_lengths_cache(\n axis\n ), self.copy_axis_cache(axis)\n\n new_frame = self._partition_mgr_cls.broadcast_apply(\n axis, func, left_parts, right_parts\n )\n if isinstance(dtypes, str) and dtypes == \"copy\":\n dtypes = self.copy_dtypes_cache()\n\n def _pick_axis(get_axis, sizes_cache):\n if labels == \"keep\":\n return get_axis(), sizes_cache\n if labels == \"replace\":\n return joined_index, partition_sizes_along_axis\n assert labels == \"drop\", f\"Unexpected `labels`: {labels}\"\n return None, None\n\n if axis == 0:\n # Pass shape caches instead of values in order to not trigger shape computation.\n new_index, new_row_lengths = _pick_axis(\n self.copy_index_cache, self._row_lengths_cache\n )\n new_columns, new_column_widths = (\n self.copy_columns_cache(),\n self._column_widths_cache,\n )\n else:\n new_index, new_row_lengths = (\n self.copy_index_cache(),\n self._row_lengths_cache,\n )\n new_columns, new_column_widths = _pick_axis(\n self.copy_columns_cache, self._column_widths_cache\n )\n\n return self.__constructor__(\n new_frame,\n new_index,\n new_columns,\n new_row_lengths,\n new_column_widths,\n dtypes=dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n def _prepare_frame_to_broadcast(self, axis, indices, broadcast_all):\n \"\"\"\n Compute the indices to broadcast `self` considering `indices`.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to broadcast along.\n indices : dict\n Dict of indices and internal indices of partitions where `self` must\n be broadcasted.\n broadcast_all : bool\n Whether broadcast the whole axis of `self` frame or just a subset of it.\n\n Returns\n -------\n dict\n Dictionary with indices of partitions to broadcast.\n\n Notes\n -----\n New dictionary of indices of `self` partitions represents that\n you want to broadcast `self` at specified another partition named `other`. For example,\n Dictionary {key: {key1: [0, 1], key2: [5]}} means, that in `other`[key] you want to\n broadcast [self[key1], self[key2]] partitions and internal indices for `self` must be [[0, 1], [5]]\n \"\"\"\n if broadcast_all:\n sizes = self.row_lengths if axis else self.column_widths\n return {key: dict(enumerate(sizes)) for key in indices.keys()}\n passed_len = 0\n result_dict = {}\n for part_num, internal in indices.items():\n result_dict[part_num] = self._get_dict_of_block_index(\n axis ^ 1, np.arange(passed_len, passed_len + len(internal))\n )\n passed_len += len(internal)\n return result_dict\n\n def _extract_partitions(self):\n \"\"\"\n Extract partitions if partitions are present.\n\n If partitions are empty return a dummy partition with empty data but\n index and columns of current dataframe.\n\n Returns\n -------\n np.ndarray\n NumPy array with extracted partitions.\n \"\"\"\n if self._partitions.size > 0:\n return self._partitions\n else:\n dtypes = None\n if self.has_materialized_dtypes:\n dtypes = self.dtypes\n return self._partition_mgr_cls.create_partition_from_metadata(\n index=self.index, columns=self.columns, dtypes=dtypes\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def broadcast_apply_select_indices(\n self,\n axis,\n func,\n other: PandasDataframe,\n apply_indices=None,\n numeric_indices=None,\n keep_remaining=False,\n broadcast_all=True,\n new_index=None,\n new_columns=None,\n ) -> PandasDataframe:\n \"\"\"\n Apply a function to select indices at specified axis and broadcast partitions of `other` Modin DataFrame.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to apply function along.\n func : callable\n Function to apply.\n other : PandasDataframe\n Partitions of which should be broadcasted.\n apply_indices : list, optional\n List of labels to apply (if `numeric_indices` are not specified).\n numeric_indices : list, optional\n Numeric indices to apply (if `apply_indices` are not specified).\n keep_remaining : bool, default: False\n Whether drop the data that is not computed over or not.\n broadcast_all : bool, default: True\n Whether broadcast the whole axis of right frame to every\n partition or just a subset of it.\n new_index : pandas.Index, optional\n Index of the result. We may know this in advance,\n and if not provided it must be computed.\n new_columns : pandas.Index, optional\n Columns of the result. We may know this in advance,\n and if not provided it must be computed.\n\n Returns\n -------\n PandasDataframe\n New Modin DataFrame.\n \"\"\"\n assert (\n apply_indices is not None or numeric_indices is not None\n ), \"Indices to apply must be specified!\"\n\n if other is None:\n if apply_indices is None:\n apply_indices = self.get_axis(axis)[numeric_indices]\n return self.apply_select_indices(\n axis=axis,\n func=func,\n apply_indices=apply_indices,\n keep_remaining=keep_remaining,\n new_index=new_index,\n new_columns=new_columns,\n )\n\n if numeric_indices is None:\n old_index = self.index if axis else self.columns\n numeric_indices = old_index.get_indexer_for(apply_indices)\n\n dict_indices = self._get_dict_of_block_index(axis ^ 1, numeric_indices)\n broadcasted_dict = other._prepare_frame_to_broadcast(\n axis, dict_indices, broadcast_all=broadcast_all\n )\n new_partitions = self._partition_mgr_cls.broadcast_apply_select_indices(\n axis,\n func,\n self._partitions,\n other._partitions,\n dict_indices,\n broadcasted_dict,\n keep_remaining,\n )\n return self.__constructor__(\n new_partitions,\n index=new_index,\n columns=new_columns,\n pandas_backend=self._pandas_backend,\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def broadcast_apply_full_axis(\n self,\n axis,\n func,\n other,\n new_index=None,\n new_columns=None,\n apply_indices=None,\n enumerate_partitions=False,\n dtypes=None,\n keep_partitioning=True,\n num_splits=None,\n sync_labels=True,\n pass_axis_lengths_to_partitions=False,\n ):\n \"\"\"\n Broadcast partitions of `other` Modin DataFrame and apply a function along full axis.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to apply over (0 - rows, 1 - columns).\n func : callable\n Function to apply.\n other : PandasDataframe or list\n Modin DataFrame(s) to broadcast.\n new_index : list-like, optional\n Index of the result. We may know this in advance,\n and if not provided it must be computed.\n new_columns : list-like, optional\n Columns of the result. We may know this in\n advance, and if not provided it must be computed.\n apply_indices : list-like, optional\n Indices of `axis ^ 1` to apply function over.\n enumerate_partitions : bool, default: False\n Whether pass partition index into applied `func` or not.\n Note that `func` must be able to obtain `partition_idx` kwarg.\n dtypes : list-like or scalar, optional\n Data types of the result. This is an optimization\n because there are functions that always result in a particular data\n type, and allows us to avoid (re)computing it.\n keep_partitioning : boolean, default: True\n The flag to keep partition boundaries for Modin Frame if possible.\n Setting it to True disables shuffling data from one partition to another in case the resulting\n number of splits is equal to the initial number of splits.\n num_splits : int, optional\n The number of partitions to split the result into across the `axis`. If None, then the number\n of splits will be infered automatically. If `num_splits` is None and `keep_partitioning=True`\n then the number of splits is preserved.\n sync_labels : boolean, default: True\n Synchronize external indexes (`new_index`, `new_columns`) with internal indexes.\n This could be used when you're certain that the indices in partitions are equal to\n the provided hints in order to save time on syncing them.\n pass_axis_lengths_to_partitions : bool, default: False\n Whether pass partition lengths along `axis ^ 1` to the kernel `func`.\n Note that `func` must be able to obtain `df, *axis_lengths`.\n\n Returns\n -------\n PandasDataframe\n New Modin DataFrame.\n \"\"\"\n if other is not None:\n if not isinstance(other, list):\n other = [other]\n other = [o._extract_partitions() for o in other] if len(other) else None\n\n if apply_indices is not None:\n numeric_indices = self.get_axis(axis ^ 1).get_indexer_for(apply_indices)\n apply_indices = self._get_dict_of_block_index(\n axis ^ 1, numeric_indices\n ).keys()\n\n apply_func_args = None\n if pass_axis_lengths_to_partitions:\n if axis == 0:\n apply_func_args = (\n self._column_widths_cache\n if self._column_widths_cache is not None\n else [part.width(materialize=False) for part in self._partitions[0]]\n )\n else:\n apply_func_args = (\n self._row_lengths_cache\n if self._row_lengths_cache is not None\n else [\n part.length(materialize=False) for part in self._partitions.T[0]\n ]\n )\n\n new_partitions = self._partition_mgr_cls.broadcast_axis_partitions(\n axis=axis,\n left=self._partitions,\n right=other,\n apply_func=self._build_treereduce_func(axis, func),\n apply_indices=apply_indices,\n enumerate_partitions=enumerate_partitions,\n keep_partitioning=keep_partitioning,\n num_splits=num_splits,\n apply_func_args=apply_func_args,\n )\n kw = {\"row_lengths\": None, \"column_widths\": None}\n if isinstance(dtypes, str) and dtypes == \"copy\":\n kw[\"dtypes\"] = self.copy_dtypes_cache()\n elif isinstance(dtypes, DtypesDescriptor):\n kw[\"dtypes\"] = ModinDtypes(dtypes)\n elif dtypes is not None:\n if isinstance(dtypes, (pandas.Series, ModinDtypes)):\n kw[\"dtypes\"] = dtypes.copy()\n else:\n if new_columns is None:\n assert not is_list_like(dtypes)\n dtype = pandas.api.types.pandas_dtype(dtypes)\n kw[\"dtypes\"] = ModinDtypes(DtypesDescriptor(remaining_dtype=dtype))\n else:\n kw[\"dtypes\"] = (\n pandas.Series(dtypes, index=new_columns)\n if is_list_like(dtypes)\n else pandas.Series(\n [pandas.api.types.pandas_dtype(dtypes)] * len(new_columns),\n index=new_columns,\n )\n )\n is_index_materialized = ModinIndex.is_materialized_index(new_index)\n is_columns_materialized = ModinIndex.is_materialized_index(new_columns)\n if axis == 0:\n if (\n is_columns_materialized\n and len(new_partitions.shape) > 1\n and new_partitions.shape[1] == 1\n ):\n kw[\"column_widths\"] = [len(new_columns)]\n elif axis == 1:\n if is_index_materialized and new_partitions.shape[0] == 1:\n kw[\"row_lengths\"] = [len(new_index)]\n if not keep_partitioning:\n if kw[\"row_lengths\"] is None and is_index_materialized:\n if axis == 0:\n kw[\"row_lengths\"] = get_length_list(\n axis_len=len(new_index),\n num_splits=new_partitions.shape[0],\n min_block_size=MinRowPartitionSize.get(),\n )\n elif axis == 1:\n if self._row_lengths_cache is not None and len(new_index) == sum(\n self._row_lengths_cache\n ):\n kw[\"row_lengths\"] = self._row_lengths_cache\n if kw[\"column_widths\"] is None and is_columns_materialized:\n if axis == 1:\n kw[\"column_widths\"] = get_length_list(\n axis_len=len(new_columns),\n num_splits=new_partitions.shape[1],\n min_block_size=MinColumnPartitionSize.get(),\n )\n elif axis == 0:\n if self._column_widths_cache is not None and len(\n new_columns\n ) == sum(self._column_widths_cache):\n kw[\"column_widths\"] = self._column_widths_cache\n else:\n if axis == 0:\n if (\n kw[\"row_lengths\"] is None\n and self._row_lengths_cache is not None\n and is_index_materialized\n and len(new_index) == sum(self._row_lengths_cache)\n # to avoid problems that may arise when filtering empty dataframes\n and all(r != 0 for r in self._row_lengths_cache)\n ):\n kw[\"row_lengths\"] = self._row_lengths_cache\n elif axis == 1:\n if (\n kw[\"column_widths\"] is None\n and self._column_widths_cache is not None\n and is_columns_materialized\n and len(new_columns) == sum(self._column_widths_cache)\n # to avoid problems that may arise when filtering empty dataframes\n and all(w != 0 for w in self._column_widths_cache)\n ):\n kw[\"column_widths\"] = self._column_widths_cache\n\n result = self.__constructor__(\n new_partitions,\n index=new_index,\n columns=new_columns,\n **kw,\n pandas_backend=self._pandas_backend,\n )\n if sync_labels and new_index is not None:\n result.synchronize_labels(axis=0)\n if sync_labels and new_columns is not None:\n result.synchronize_labels(axis=1)\n return result\n\n def _check_if_axes_identical(self, other: PandasDataframe, axis: int = 0) -> bool:\n \"\"\"\n Check whether indices/partitioning along the specified `axis` are identical when compared with `other`.\n\n Parameters\n ----------\n other : PandasDataframe\n Dataframe to compare indices/partitioning with.\n axis : int, default: 0\n\n Returns\n -------\n bool\n \"\"\"\n if self.has_axis_cache(axis) and other.has_axis_cache(axis):\n self_cache, other_cache = self._get_axis_cache(axis), other._get_axis_cache(\n axis\n )\n equal_indices = self_cache.equals(other_cache)\n if equal_indices:\n equal_lengths = self_cache.compare_partition_lengths_if_possible(\n other_cache\n )\n if isinstance(equal_lengths, bool):\n return equal_lengths\n return self._get_axis_lengths(axis) == other._get_axis_lengths(axis)\n return False\n return self.get_axis(axis).equals(\n other.get_axis(axis)\n ) and self._get_axis_lengths(axis) == other._get_axis_lengths(axis)\n\n def _copartition(\n self, axis, other, how, sort=None, force_repartition=False, fill_value=None\n ):\n \"\"\"\n Copartition two Modin DataFrames.\n\n Perform aligning of partitions, index and partition blocks.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to copartition along (0 - rows, 1 - columns).\n other : PandasDataframe\n Other Modin DataFrame(s) to copartition against.\n how : str\n How to manage joining the index object (\"left\", \"right\", etc.).\n sort : bool, default: None\n Whether sort the joined index or not.\n If ``None``, sort is defined in depend on labels equality along the axis.\n force_repartition : bool, default: False\n Whether force the repartitioning or not. By default,\n this method will skip repartitioning if it is possible. This is because\n reindexing is extremely inefficient. Because this method is used to\n `join` or `append`, it is vital that the internal indices match.\n fill_value : any, optional\n Value to use for missing values.\n\n Returns\n -------\n tuple\n Tuple containing:\n 1) 2-d NumPy array of aligned left partitions\n 2) list of 2-d NumPy arrays of aligned right partitions\n 3) joined index along ``axis``, may be ``ModinIndex`` if not materialized\n 4) If materialized, list with sizes of partitions along axis that partitioning\n was done on, otherwise ``None``. This list will be empty if and only if all\n the frames are empty.\n \"\"\"\n if isinstance(other, type(self)):\n other = [other]\n\n if not force_repartition and all(\n o._check_if_axes_identical(self, axis) for o in other\n ):\n return (\n self._partitions,\n [o._partitions for o in other],\n self.copy_axis_cache(axis, copy_lengths=True),\n self._get_axis_lengths_cache(axis),\n )\n\n if sort is None:\n sort = not all(self.get_axis(axis).equals(o.get_axis(axis)) for o in other)\n\n self_index = self.get_axis(axis)\n others_index = [o.get_axis(axis) for o in other]\n joined_index, make_reindexer = self._join_index_objects(\n axis, [self_index] + others_index, how, sort, fill_value\n )\n\n frames = [self] + other\n non_empty_frames_idx = [\n i for i, o in enumerate(frames) if o._partitions.size != 0\n ]\n\n # If all frames are empty\n if len(non_empty_frames_idx) == 0:\n return (\n self._partitions,\n [o._partitions for o in other],\n joined_index,\n # There are no partition sizes because the resulting dataframe\n # has no partitions.\n [],\n )\n\n base_frame_idx = non_empty_frames_idx[0]\n other_frames = frames[base_frame_idx + 1 :]\n\n # Picking first non-empty frame\n base_frame = frames[non_empty_frames_idx[0]]\n base_index = base_frame.get_axis(axis)\n\n # define conditions for reindexing and repartitioning `self` frame\n do_reindex_base = not base_index.equals(joined_index)\n do_repartition_base = force_repartition or do_reindex_base\n\n # Perform repartitioning and reindexing for `base_frame` if needed.\n # Also define length of base and frames. We will need to know the\n # lengths for alignment.\n if do_repartition_base:\n reindexed_base = base_frame._partition_mgr_cls.map_axis_partitions(\n axis,\n base_frame._partitions,\n make_reindexer(do_reindex_base, base_frame_idx),\n )\n if axis:\n base_lengths = [obj.width() for obj in reindexed_base[0]]\n else:\n base_lengths = [obj.length() for obj in reindexed_base.T[0]]\n else:\n reindexed_base = base_frame._partitions\n base_lengths = base_frame.column_widths if axis else base_frame.row_lengths\n\n others_lengths = [o._get_axis_lengths(axis) for o in other_frames]\n\n # define conditions for reindexing and repartitioning `other` frames\n do_reindex_others = [\n not o.get_axis(axis).equals(joined_index) for o in other_frames\n ]\n\n do_repartition_others = [None] * len(other_frames)\n for i in range(len(other_frames)):\n do_repartition_others[i] = (\n force_repartition\n or do_reindex_others[i]\n or others_lengths[i] != base_lengths\n )\n\n # perform repartitioning and reindexing for `other_frames` if needed\n reindexed_other_list = [None] * len(other_frames)\n for i in range(len(other_frames)):\n if do_repartition_others[i]:\n # indices of others frame start from `base_frame_idx` + 1\n reindexed_other_list[i] = other_frames[\n i\n ]._partition_mgr_cls.map_axis_partitions(\n axis,\n other_frames[i]._partitions,\n make_reindexer(do_repartition_others[i], base_frame_idx + 1 + i),\n lengths=base_lengths,\n )\n else:\n reindexed_other_list[i] = other_frames[i]._partitions\n reindexed_frames = (\n [frames[i]._partitions for i in range(base_frame_idx)]\n + [reindexed_base]\n + reindexed_other_list\n )\n return (reindexed_frames[0], reindexed_frames[1:], joined_index, base_lengths)\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def n_ary_op(\n self,\n op,\n right_frames: list[PandasDataframe],\n join_type=\"outer\",\n sort=None,\n copartition_along_columns=True,\n labels=\"replace\",\n dtypes: Optional[pandas.Series] = None,\n ) -> PandasDataframe:\n \"\"\"\n Perform an n-opary operation by joining with other Modin DataFrame(s).\n\n Parameters\n ----------\n op : callable\n Function to apply after the join.\n right_frames : list of PandasDataframe\n Modin DataFrames to join with.\n join_type : str, default: \"outer\"\n Type of join to apply.\n sort : bool, default: None\n Whether to sort index and columns or not.\n copartition_along_columns : bool, default: True\n Whether to perform copartitioning along columns or not.\n For some ops this isn't needed (e.g., `fillna`).\n labels : {\"replace\", \"drop\"}, default: \"replace\"\n Whether use labels from joined DataFrame or drop altogether to make\n them be computed lazily later.\n dtypes : pandas.Series, optional\n Dtypes of the resultant dataframe, this argument will be\n received if the resultant dtypes of n-opary operation is precomputed.\n\n Returns\n -------\n PandasDataframe\n New Modin DataFrame.\n \"\"\"\n left_parts, list_of_right_parts, joined_index, row_lengths = self._copartition(\n 0,\n right_frames,\n join_type,\n sort=sort,\n )\n if copartition_along_columns:\n new_left_frame = self.__constructor__(\n left_parts,\n joined_index,\n self.copy_columns_cache(copy_lengths=True),\n row_lengths,\n self._column_widths_cache,\n pandas_backend=self._pandas_backend,\n )\n new_right_frames = [\n self.__constructor__(\n right_parts,\n joined_index,\n right_frame.copy_columns_cache(copy_lengths=True),\n row_lengths,\n right_frame._column_widths_cache,\n pandas_backend=self._pandas_backend,\n )\n for right_parts, right_frame in zip(list_of_right_parts, right_frames)\n ]\n\n (\n left_parts,\n list_of_right_parts,\n joined_columns,\n column_widths,\n ) = new_left_frame._copartition(\n 1,\n new_right_frames,\n join_type,\n sort=sort,\n )\n else:\n joined_columns = self.copy_columns_cache(copy_lengths=True)\n column_widths = self._column_widths_cache\n\n new_frame = (\n np.array([])\n if len(left_parts) == 0\n or any(len(right_parts) == 0 for right_parts in list_of_right_parts)\n else self._partition_mgr_cls.n_ary_operation(\n left_parts, op, list_of_right_parts\n )\n )\n if labels == \"drop\":\n joined_index = joined_columns = row_lengths = column_widths = None\n\n return self.__constructor__(\n new_frame,\n joined_index,\n joined_columns,\n row_lengths,\n column_widths,\n dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def concat(\n self,\n axis: Union[int, Axis],\n others: Union[PandasDataframe, List[PandasDataframe]],\n how,\n sort,\n ) -> PandasDataframe:\n \"\"\"\n Concatenate `self` with one or more other Modin DataFrames.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n Axis to concatenate over.\n others : list\n List of Modin DataFrames to concatenate with.\n how : str\n Type of join to use for the axis.\n sort : bool\n Whether sort the result or not.\n\n Returns\n -------\n PandasDataframe\n New Modin DataFrame.\n \"\"\"\n axis = Axis(axis)\n new_widths = None\n new_lengths = None\n\n def _compute_new_widths():\n widths = None\n if self._column_widths_cache is not None and all(\n o._column_widths_cache is not None for o in others\n ):\n widths = self._column_widths_cache + [\n width for o in others for width in o._column_widths_cache\n ]\n return widths\n\n # Fast path for equivalent columns and partitioning\n if axis == Axis.ROW_WISE and all(\n o._check_if_axes_identical(self, axis=1) for o in others\n ):\n joined_index = self.copy_columns_cache(copy_lengths=True)\n left_parts = self._partitions\n right_parts = [o._partitions for o in others]\n new_widths = self._column_widths_cache\n elif axis == Axis.COL_WISE and all(\n o._check_if_axes_identical(self, axis=0) for o in others\n ):\n joined_index = self.copy_index_cache(copy_lengths=True)\n left_parts = self._partitions\n right_parts = [o._partitions for o in others]\n new_lengths = self._row_lengths_cache\n # we can only do this for COL_WISE because `concat` might rebalance partitions for ROW_WISE\n new_widths = _compute_new_widths()\n else:\n (\n left_parts,\n right_parts,\n joined_index,\n partition_sizes_along_axis,\n ) = self._copartition(\n axis.value ^ 1, others, how, sort=sort, force_repartition=False\n )\n if axis == Axis.COL_WISE:\n new_lengths = partition_sizes_along_axis\n new_widths = _compute_new_widths()\n else:\n new_widths = partition_sizes_along_axis\n new_partitions, new_lengths2 = self._partition_mgr_cls.concat(\n axis.value, left_parts, right_parts\n )\n if new_lengths is None:\n new_lengths = new_lengths2\n new_dtypes = None\n new_index = None\n new_columns = None\n if axis == Axis.ROW_WISE:\n if all(obj.has_materialized_index for obj in (self, *others)):\n new_index = self.index.append([other.index for other in others])\n new_columns = joined_index\n frames = [self] + others\n # TODO: should we wrap all `concat` call into \"try except\" block?\n # `ModinDtypes.concat` can throw exception in case of duplicate values\n new_dtypes = ModinDtypes.concat([frame._dtypes for frame in frames], axis=1)\n # If we have already cached the length of each row in at least one\n # of the row's partitions, we can build new_lengths for the new\n # frame. Typically, if we know the length for any partition in a\n # row, we know the length for the first partition in the row. So\n # just check the lengths of the first column of partitions.\n if not new_lengths:\n new_lengths = []\n if new_partitions.size > 0:\n if all(\n part._length_cache is not None for part in new_partitions.T[0]\n ):\n new_lengths = self._get_lengths(new_partitions.T[0], axis)\n else:\n new_lengths = None\n else:\n if all(obj.has_materialized_columns for obj in (self, *others)):\n new_columns = self.columns.append([other.columns for other in others])\n new_index = joined_index\n try:\n new_dtypes = ModinDtypes.concat(\n [self.copy_dtypes_cache()] + [o.copy_dtypes_cache() for o in others]\n )\n except NotImplementedError:\n new_dtypes = None\n # If we have already cached the width of each column in at least one\n # of the column's partitions, we can build new_widths for the new\n # frame. Typically, if we know the width for any partition in a\n # column, we know the width for the first partition in the column.\n # So just check the widths of the first row of partitions.\n if not new_widths:\n new_widths = []\n if new_partitions.size > 0:\n if all(part._width_cache is not None for part in new_partitions[0]):\n new_widths = self._get_lengths(new_partitions[0], axis)\n else:\n new_widths = None\n\n return self.__constructor__(\n new_partitions,\n new_index,\n new_columns,\n new_lengths,\n new_widths,\n new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n def _apply_func_to_range_partitioning_broadcast(\n self,\n right,\n func,\n key,\n new_index=None,\n new_columns=None,\n new_dtypes: Optional[Union[ModinDtypes, pandas.Series]] = None,\n ):\n \"\"\"\n Apply `func` against two dataframes using range-partitioning implementation.\n\n The method first builds range-partitioning for both dataframes using the data from\n `self[key]`, after that, it applies `func` row-wise to `self` frame and\n broadcasts row-parts of `right` to `self`.\n\n Parameters\n ----------\n right : PandasDataframe\n func : callable(left : pandas.DataFrame, right : pandas.DataFrame) -> pandas.DataFrame\n key : list of labels\n Columns to use to build range-partitioning. Must present in both dataframes.\n new_index : pandas.Index, optional\n Index values to write to the result's cache.\n new_columns : pandas.Index, optional\n Column values to write to the result's cache.\n new_dtypes : pandas.Series or ModinDtypes, optional\n Dtype values to write to the result's cache.\n\n Returns\n -------\n PandasDataframe\n \"\"\"\n if self._partitions.shape[0] == 1:\n result = self.broadcast_apply_full_axis(\n axis=1,\n func=func,\n new_columns=new_columns,\n dtypes=new_dtypes,\n other=right,\n )\n return result\n\n if not isinstance(key, list):\n key = [key]\n\n shuffling_functions = ShuffleSortFunctions(\n self,\n key,\n ascending=True,\n ideal_num_new_partitions=self._partitions.shape[0],\n )\n\n # here we want to get indices of those partitions that hold the key columns\n key_indices = self.columns.get_indexer_for(key)\n partition_indices = np.unique(\n np.digitize(key_indices, np.cumsum(self.column_widths))\n )\n\n new_partitions = self._partition_mgr_cls.shuffle_partitions(\n self._partitions,\n partition_indices,\n shuffling_functions,\n func,\n right_partitions=right._partitions,\n )\n\n return self.__constructor__(\n new_partitions,\n index=new_index,\n columns=new_columns,\n dtypes=new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def groupby(\n self,\n axis: Union[int, Axis],\n internal_by: List[str],\n external_by: List[PandasDataframe],\n by_positions: List[int],\n operator: Callable,\n result_schema: Optional[Dict[Hashable, type]] = None,\n align_result_columns: bool = False,\n series_groupby: bool = False,\n add_missing_cats: bool = False,\n **kwargs: dict,\n ) -> PandasDataframe:\n \"\"\"\n Generate groups based on values in the input column(s) and perform the specified operation on each.\n\n Parameters\n ----------\n axis : int or modin.core.dataframe.base.utils.Axis\n The axis to apply the grouping over.\n internal_by : list of strings\n One or more column labels from the `self` dataframe to use for grouping.\n external_by : list of PandasDataframes\n PandasDataframes to group by (may be specified along with or without `internal_by`).\n by_positions : list of ints\n Specifies the order of grouping by `internal_by` and `external_by` columns.\n Each element in `by_positions` specifies an index from either `external_by` or `internal_by`.\n Indices for `external_by` are positive and start from 0. Indices for `internal_by` are negative\n and start from -1 (so in order to convert them to a valid indices one should do ``-idx - 1``).\n '''\n by_positions = [0, -1, 1, -2, 2, 3]\n internal_by = [\"col1\", \"col2\"]\n external_by = [sr1, sr2, sr3, sr4]\n\n df.groupby([sr1, \"col1\", sr2, \"col2\", sr3, sr4])\n '''.\n operator : callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame\n The operation to carry out on each of the groups. The operator is another\n algebraic operator with its own user-defined function parameter, depending\n on the output desired by the user.\n result_schema : dict, optional\n Mapping from column labels to data types that represents the types of the output dataframe.\n align_result_columns : bool, default: False\n Whether to manually align columns between all the resulted row partitions.\n This flag is helpful when dealing with UDFs as they can change the partition's shape\n and labeling unpredictably, resulting in an invalid dataframe.\n series_groupby : bool, default: False\n Whether to convert a one-column DataFrame to a Series before performing groupby.\n add_missing_cats : bool, default: False\n Whether to add missing categories from `by` columns to the result.\n **kwargs : dict\n Additional arguments to pass to the ``df.groupby`` method (besides the 'by' argument).\n\n Returns\n -------\n PandasDataframe\n A new PandasDataframe containing the groupings specified, with the operator\n applied to each group.\n\n Notes\n -----\n No communication between groups is allowed in this algebra implementation.\n\n The number of rows (columns if axis=1) returned by the user-defined function\n passed to the groupby may be at most the number of rows in the group, and\n may be as small as a single row.\n\n Unlike the pandas API, an intermediate \"GROUP BY\" object is not present in this\n algebra implementation.\n \"\"\"\n axis = Axis(axis)\n if axis != Axis.ROW_WISE:\n raise NotImplementedError(\n f\"Algebra groupby only implemented row-wise. {axis.name} axis groupby not implemented yet!\"\n )\n\n has_external_grouper = len(external_by) > 0\n skip_on_aligning_flag = \"__skip_me_on_aligning__\"\n duplicated_suffix = \"__duplicated_suffix__\"\n duplicated_pattern = r\"_[\\d]*__duplicated_suffix__\"\n kwargs[\"observed\"] = True\n level = kwargs.get(\"level\")\n\n if level is not None and not isinstance(level, list):\n level = [level]\n\n def apply_func(df): # pragma: no cover\n if has_external_grouper:\n external_grouper = df[\"grouper\"]\n external_grouper = [\n # `df.groupby()` can only take a list of Series'es, so splitting\n # the df into a list of individual Series'es\n external_grouper.iloc[:, i]\n for i in range(len(external_grouper.columns))\n ]\n\n # renaming 'None' and duplicated names back to their original names\n for obj in external_grouper:\n if not isinstance(obj, pandas.Series):\n continue\n name = obj.name\n if isinstance(name, str):\n if name.startswith(MODIN_UNNAMED_SERIES_LABEL):\n name = None\n elif name.endswith(duplicated_suffix):\n name = re.sub(duplicated_pattern, \"\", name)\n elif isinstance(name, tuple):\n if name[-1].endswith(duplicated_suffix):\n name = (\n *name[:-1],\n re.sub(duplicated_pattern, \"\", name[-1]),\n )\n obj.name = name\n\n df = df[\"data\"]\n else:\n external_grouper = []\n\n by = []\n # restoring original order of 'by' columns\n for idx in by_positions:\n if idx >= 0:\n by.append(external_grouper[idx])\n else:\n by.append(internal_by[-idx - 1])\n\n if series_groupby:\n df = df.squeeze(axis=1)\n\n if kwargs.get(\"level\") is not None:\n assert len(by) == 0\n # passing an empty list triggers an error\n by = None\n\n result = operator(df.groupby(by, **kwargs))\n\n if align_result_columns and df.empty and result.empty:\n # We want to align columns only of those frames that actually performed\n # some groupby aggregation, if an empty frame was originally passed\n # (an empty bin on reshuffling was created) then there were no groupby\n # executed over this partition and so it has incorrect columns\n # that shouldn't be considered on the aligning phase\n result.attrs[skip_on_aligning_flag] = True\n return result\n\n if has_external_grouper:\n grouper = (\n external_by[0]\n if len(external_by) == 1\n else external_by[0].concat(\n axis=1, others=external_by[1:], how=\"left\", sort=False\n )\n )\n\n new_grouper_cols = []\n columns_were_changed = False\n same_columns = {}\n # duplicated names break range-partitioning mechanism, so renaming them.\n # original names will be reverted in the actual groupby kernel\n for col in grouper.columns:\n suffix = same_columns.get(col)\n if suffix is None:\n same_columns[col] = 0\n else:\n same_columns[col] += 1\n col = (\n (*col[:-1], f\"{col[-1]}_{suffix}{duplicated_suffix}\")\n if isinstance(col, tuple)\n else f\"{col}_{suffix}{duplicated_suffix}\"\n )\n columns_were_changed = True\n new_grouper_cols.append(col)\n\n if columns_were_changed:\n grouper.columns = pandas.Index(new_grouper_cols)\n grouper_key_columns = grouper.columns\n data = self\n data_key_columns = internal_by\n else:\n grouper = self\n grouper_key_columns = internal_by\n data, data_key_columns = None, None\n\n result = grouper._apply_func_to_range_partitioning(\n key_columns=grouper_key_columns,\n func=apply_func,\n data=data,\n data_key_columns=data_key_columns,\n level=level,\n )\n # no need aligning columns if there's only one row partition\n if add_missing_cats or align_result_columns and result._partitions.shape[0] > 1:\n # FIXME: the current reshuffling implementation guarantees us that there's only one column\n # partition in the result, so we should never hit this exception for now, however\n # in the future, we might want to make this implementation more broader\n if result._partitions.shape[1] > 1:\n raise NotImplementedError(\n \"Aligning columns is not yet implemented for multiple column partitions.\"\n )\n\n # There're two implementations:\n # 1. The first one work faster, but may stress the network a lot in cluster mode since\n # it gathers all the dataframes in a single ray-kernel.\n # 2. The second one works slower, but only gathers light pandas.Index objects,\n # so there should be less stress on the network.\n if add_missing_cats or not IsRayCluster.get():\n if self.has_materialized_dtypes:\n original_dtypes = pandas.Series(\n {\n # lazy proxies hold a reference to another modin's DataFrame which can be\n # a problem during serialization, in this scenario we don't need actual\n # categorical values, so a \"category\" string will be enough\n name: (\n \"category\"\n if isinstance(dtype, LazyProxyCategoricalDtype)\n else dtype\n )\n for name, dtype in self.dtypes.items()\n }\n )\n else:\n original_dtypes = None\n\n def compute_aligned_columns(*dfs, initial_columns=None, by=None):\n \"\"\"Take row partitions, filter empty ones, and return joined columns for them.\"\"\"\n if align_result_columns:\n valid_dfs = [\n df\n for df in dfs\n if not df.attrs.get(skip_on_aligning_flag, False)\n ]\n\n if len(valid_dfs) == 0 and len(dfs) != 0:\n valid_dfs = dfs\n\n # Using '.concat()' on empty-slices instead of 'Index.join()'\n # in order to get identical behavior to pandas when it joins\n # results of different groups\n combined_cols = pandas.concat(\n [df.iloc[:0] for df in valid_dfs], axis=0, join=\"outer\"\n ).columns\n else:\n combined_cols = dfs[0].columns\n\n masks = None\n if add_missing_cats:\n masks, combined_cols = add_missing_categories_to_groupby(\n dfs,\n by,\n operator,\n initial_columns,\n combined_cols,\n is_udf_agg=align_result_columns,\n kwargs=kwargs.copy(),\n initial_dtypes=original_dtypes,\n )\n return (\n (combined_cols, masks)\n if align_result_columns\n else (None, masks)\n )\n\n external_by_cols = [\n None if col.startswith(MODIN_UNNAMED_SERIES_LABEL) else col\n for obj in external_by\n for col in obj.columns\n ]\n by = []\n # restoring original order of 'by' columns\n for idx in by_positions:\n if idx >= 0:\n by.append(external_by_cols[idx])\n else:\n by.append(internal_by[-idx - 1])\n\n # Passing all partitions to the 'compute_aligned_columns' kernel to get\n # aligned columns\n parts = result._partitions.flatten()\n aligned_columns = parts[0].apply(\n compute_aligned_columns,\n *[part._data for part in parts[1:]],\n initial_columns=pandas.Index(external_by_cols).append(self.columns),\n by=by,\n )\n\n def apply_aligned(df, args, partition_idx):\n combined_cols, mask = args\n if mask is not None and mask.get(partition_idx) is not None:\n values = mask[partition_idx]\n\n original_names = df.index.names\n # TODO: inserting 'values' based on 'searchsorted' result might be more efficient\n # in cases of small amount of 'values'\n df = pandas.concat([df, values])\n if kwargs[\"sort\"]:\n df = df.sort_index(axis=0)\n df.index.names = original_names\n if combined_cols is not None:\n df = df.reindex(columns=combined_cols)\n return df\n\n # Lazily applying aligned columns to partitions\n new_partitions = self._partition_mgr_cls.lazy_map_partitions(\n result._partitions,\n apply_aligned,\n func_args=(aligned_columns._data,),\n enumerate_partitions=True,\n )\n else:\n\n def join_cols(df, *cols):\n \"\"\"Join `cols` and apply the joined columns to `df`.\"\"\"\n valid_cols = [\n pandas.DataFrame(columns=col) for col in cols if col is not None\n ]\n if len(valid_cols) == 0:\n return df\n # Using '.concat()' on empty-slices instead of 'Index.join()'\n # in order to get identical behavior to pandas when it joins\n # results of different groups\n result_col = pandas.concat(valid_cols, axis=0, join=\"outer\").columns\n return df.reindex(columns=result_col)\n\n # Getting futures for columns of non-empty partitions\n cols = [\n part.apply(\n lambda df: (\n None\n if df.attrs.get(skip_on_aligning_flag, False)\n else df.columns\n )\n )._data\n for part in result._partitions.flatten()\n ]\n\n # Lazily joining and applying the aligned columns\n new_partitions = self._partition_mgr_cls.lazy_map_partitions(\n result._partitions,\n join_cols,\n func_args=cols,\n )\n result = self.__constructor__(\n new_partitions,\n index=result.copy_index_cache(),\n row_lengths=result._row_lengths_cache,\n pandas_backend=self._pandas_backend,\n )\n\n if (\n not result.has_materialized_index\n and not has_external_grouper\n and level is None\n ):\n by_dtypes = ModinDtypes(self._dtypes).lazy_get(internal_by)\n if by_dtypes.is_materialized:\n new_index = ModinIndex(value=result, axis=0, dtypes=by_dtypes)\n result.set_index_cache(new_index)\n\n if result_schema is not None:\n new_dtypes = pandas.Series(result_schema)\n\n result.set_dtypes_cache(new_dtypes)\n result.set_columns_cache(new_dtypes.index)\n\n return result\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def groupby_reduce(\n self,\n axis,\n by,\n map_func,\n reduce_func,\n new_index=None,\n new_columns=None,\n apply_indices=None,\n ):\n \"\"\"\n Groupby another Modin DataFrame dataframe and aggregate the result.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to groupby and aggregate over.\n by : PandasDataframe or None\n A Modin DataFrame to group by.\n map_func : callable\n Map component of the aggregation.\n reduce_func : callable\n Reduce component of the aggregation.\n new_index : pandas.Index, optional\n Index of the result. We may know this in advance,\n and if not provided it must be computed.\n new_columns : pandas.Index, optional\n Columns of the result. We may know this in advance,\n and if not provided it must be computed.\n apply_indices : list-like, optional\n Indices of `axis ^ 1` to apply groupby over.\n\n Returns\n -------\n PandasDataframe\n New Modin DataFrame.\n \"\"\"\n by_parts = by if by is None else by._partitions\n if by is None:\n self._propagate_index_objs(axis=0)\n\n if apply_indices is not None:\n numeric_indices = self.get_axis(axis ^ 1).get_indexer_for(apply_indices)\n apply_indices = list(\n self._get_dict_of_block_index(axis ^ 1, numeric_indices).keys()\n )\n\n if by_parts is not None:\n # inplace operation\n if by_parts.shape[axis] != self._partitions.shape[axis]:\n self._filter_empties(compute_metadata=False)\n new_partitions = self._partition_mgr_cls.groupby_reduce(\n axis, self._partitions, by_parts, map_func, reduce_func, apply_indices\n )\n return self.__constructor__(\n new_partitions,\n index=new_index,\n columns=new_columns,\n pandas_backend=self._pandas_backend,\n )\n\n @classmethod\n def from_pandas(cls, df):\n \"\"\"\n Create a Modin DataFrame from a pandas DataFrame.\n\n Parameters\n ----------\n df : pandas.DataFrame\n A pandas DataFrame.\n\n Returns\n -------\n PandasDataframe\n New Modin DataFrame.\n \"\"\"\n new_index = df.index\n new_columns = df.columns\n new_dtypes = df.dtypes\n new_frame, pandas_backend, new_lengths, new_widths = (\n cls._partition_mgr_cls.from_pandas(df, True)\n )\n return cls(\n new_frame,\n new_index,\n new_columns,\n new_lengths,\n new_widths,\n dtypes=new_dtypes,\n pandas_backend=pandas_backend,\n )\n\n @classmethod\n def from_arrow(cls, at):\n \"\"\"\n Create a Modin DataFrame from an Arrow Table.\n\n Parameters\n ----------\n at : pyarrow.table\n Arrow Table.\n\n Returns\n -------\n PandasDataframe\n New Modin DataFrame.\n \"\"\"\n new_frame, pandas_backend, new_lengths, new_widths = (\n cls._partition_mgr_cls.from_arrow(at, return_dims=True)\n )\n new_columns = Index.__new__(Index, data=at.column_names, dtype=\"O\")\n new_index = Index.__new__(RangeIndex, data=range(at.num_rows))\n new_dtypes = pandas.Series(\n [cls._arrow_type_to_dtype(col.type) for col in at.columns],\n index=at.column_names,\n )\n return cls(\n partitions=new_frame,\n index=new_index,\n columns=new_columns,\n row_lengths=new_lengths,\n column_widths=new_widths,\n dtypes=new_dtypes,\n pandas_backend=pandas_backend,\n )\n\n @classmethod\n def _arrow_type_to_dtype(cls, arrow_type):\n \"\"\"\n Convert an arrow data type to a pandas data type.\n\n Parameters\n ----------\n arrow_type : arrow dtype\n Arrow data type to be converted to a pandas data type.\n\n Returns\n -------\n object\n Any dtype compatible with pandas.\n \"\"\"\n import pyarrow\n\n try:\n # TODO: should we map arrow types to pyarrow-backed pandas types?\n # It seems like this might help avoid the expense of transferring\n # data between backends (numpy and pyarrow), but we need to be sure\n # how this fits into the type inference system in pandas.\n res = arrow_type.to_pandas_dtype()\n # Conversion to pandas is not implemented for some arrow types,\n # perform manual conversion for them:\n except NotImplementedError:\n if pyarrow.types.is_time(arrow_type):\n res = np.dtype(datetime.time)\n else:\n raise\n\n if not isinstance(res, (np.dtype, str)):\n return np.dtype(res)\n return res\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def to_pandas(self):\n \"\"\"\n Convert this Modin DataFrame to a pandas DataFrame.\n\n Returns\n -------\n pandas.DataFrame\n \"\"\"\n df = self._partition_mgr_cls.to_pandas(self._partitions)\n if df.empty:\n df = pandas.DataFrame(columns=self.columns, index=self.index)\n if len(df.columns) and self.has_materialized_dtypes:\n df = df.astype(self.dtypes)\n else:\n for axis, has_external_index in enumerate(\n [\"has_materialized_index\", \"has_materialized_columns\"]\n ):\n # no need to check external and internal axes since in that case\n # external axes will be computed from internal partitions\n if getattr(self, has_external_index):\n external_index = self.columns if axis else self.index\n ErrorMessage.catch_bugs_and_request_email(\n not df.axes[axis].equals(external_index),\n f\"Internal and external indices on axis {axis} do not match.\",\n )\n # have to do this in order to assign some potentially missing metadata,\n # the ones that were set to the external index but were never propagated\n # into the internal ones\n df = df.set_axis(axis=axis, labels=external_index, copy=False)\n\n return df\n\n def to_numpy(self, **kwargs):\n \"\"\"\n Convert this Modin DataFrame to a NumPy array.\n\n Parameters\n ----------\n **kwargs : dict\n Additional keyword arguments to be passed in `to_numpy`.\n\n Returns\n -------\n np.ndarray\n \"\"\"\n arr = self._partition_mgr_cls.to_numpy(self._partitions, **kwargs)\n ErrorMessage.catch_bugs_and_request_email(\n self.has_materialized_index\n and len(arr) != len(self.index)\n or self.has_materialized_columns\n and len(arr[0]) != len(self.columns)\n )\n return arr\n\n @lazy_metadata_decorator(apply_axis=None, transpose=True)\n def transpose(self):\n \"\"\"\n Transpose the index and columns of this Modin DataFrame.\n\n Reflect this Modin DataFrame over its main diagonal\n by writing rows as columns and vice-versa.\n\n Returns\n -------\n PandasDataframe\n New Modin DataFrame.\n \"\"\"\n new_partitions = self._partition_mgr_cls.lazy_map_partitions(\n self._partitions, lambda df: df.T\n ).T\n if self.has_materialized_dtypes:\n new_dtypes = pandas.Series(\n np.full(len(self.index), find_common_type(self.dtypes.values)),\n index=self.index,\n )\n else:\n new_dtypes = None\n return self.__constructor__(\n new_partitions,\n self.copy_columns_cache(copy_lengths=True),\n self.copy_index_cache(copy_lengths=True),\n self._column_widths_cache,\n self._row_lengths_cache,\n dtypes=new_dtypes,\n pandas_backend=self._pandas_backend,\n )\n\n @lazy_metadata_decorator(apply_axis=\"both\")\n def finalize(self):\n \"\"\"\n Perform all deferred calls on partitions.\n\n This makes `self` Modin Dataframe independent of a history of queries\n that were used to build it.\n \"\"\"\n self._partition_mgr_cls.finalize(self._partitions)\n\n def wait_computations(self):\n \"\"\"Wait for all computations to complete without materializing data.\"\"\"\n self._partition_mgr_cls.wait_partitions(self._partitions.flatten())\n\n def support_materialization_in_worker_process(self) -> bool:\n \"\"\"\n Whether it's possible to call function `to_pandas` during the pickling process, at the moment of recreating the object.\n\n Returns\n -------\n bool\n \"\"\"\n return True\n\n def __dataframe__(self, nan_as_null: bool = False, allow_copy: bool = True):\n \"\"\"\n Get a Modin DataFrame that implements the dataframe exchange protocol.\n\n See more about the protocol in https://data-apis.org/dataframe-protocol/latest/index.html.\n\n Parameters\n ----------\n nan_as_null : bool, default: False\n A keyword intended for the consumer to tell the producer\n to overwrite null values in the data with ``NaN`` (or ``NaT``).\n This currently has no effect; once support for nullable extension\n dtypes is added, this value should be propagated to columns.\n allow_copy : bool, default: True\n A keyword that defines whether or not the library is allowed\n to make a copy of the data. For example, copying data would be necessary\n if a library supports strided buffers, given that this protocol\n specifies contiguous buffers. Currently, if the flag is set to ``False``\n and a copy is needed, a ``RuntimeError`` will be raised.\n\n Returns\n -------\n ProtocolDataframe\n A dataframe object following the dataframe protocol specification.\n \"\"\"\n from modin.core.dataframe.pandas.interchange.dataframe_protocol.dataframe import (\n PandasProtocolDataframe,\n )\n\n return PandasProtocolDataframe(\n self, nan_as_null=nan_as_null, allow_copy=allow_copy\n )\n\n @classmethod\n def from_interchange_dataframe(cls, df: ProtocolDataframe) -> PandasDataframe:\n \"\"\"\n Convert a DataFrame implementing the dataframe exchange protocol to a Core Modin Dataframe.\n\n See more about the protocol in https://data-apis.org/dataframe-protocol/latest/index.html.\n\n Parameters\n ----------\n df : ProtocolDataframe\n The DataFrame object supporting the dataframe exchange protocol.\n\n Returns\n -------\n PandasDataframe\n A new Core Modin Dataframe object.\n \"\"\"\n if type(df) is cls:\n return df\n\n if not hasattr(df, \"__dataframe__\"):\n raise ValueError(\n \"`df` does not support DataFrame exchange protocol, i.e. `__dataframe__` method\"\n )\n\n from modin.core.dataframe.pandas.interchange.dataframe_protocol.from_dataframe import (\n from_dataframe_to_pandas,\n )\n\n ErrorMessage.default_to_pandas(message=\"`from_dataframe`\")\n pandas_df = from_dataframe_to_pandas(df)\n return cls.from_pandas(pandas_df)\n\n def case_when(self, caselist):\n \"\"\"\n Replace values where the conditions are True.\n\n This is Series.case_when() implementation and, thus, it's designed to work\n only with single-column DataFrames.\n\n Parameters\n ----------\n caselist : list of tuples\n\n Returns\n -------\n PandasDataframe\n \"\"\"\n # The import is here to avoid an incorrect module initialization when running tests.\n # This module is loaded before `pytest_configure()` is called. If `pytest_configure()`\n # changes the engine, the `remote_function` decorator will not be valid.\n from modin.core.execution.utils import remote_function\n\n @remote_function\n def remote_fn(df, name, caselist): # pragma: no cover\n caselist = [\n tuple(\n (\n data.squeeze(axis=1)\n if isinstance(data, pandas.DataFrame)\n else data\n )\n for data in case_tuple\n )\n for case_tuple in caselist\n ]\n return pandas.DataFrame({name: df.squeeze(axis=1).case_when(caselist)})\n\n cls = type(self)\n use_map = True\n is_trivial_idx = None\n name = self.columns[0]\n # Lists of modin frames: first for conditions, second for replacements\n modin_lists = [[], []]\n # Fill values for conditions and replacements respectively\n fill_values = [True, None]\n new_caselist = []\n for case_tuple in caselist:\n new_case = []\n for data, modin_list, fill_value in zip(\n case_tuple, modin_lists, fill_values\n ):\n if isinstance(data, cls):\n modin_list.append(data)\n elif callable(data):\n data = remote_function(data)\n elif isinstance(data, pandas.Series):\n use_map = False\n if is_trivial_idx is None:\n self_idx = self.index\n length = len(self_idx)\n is_trivial_idx = is_trivial_index(self_idx)\n if is_trivial_idx and is_trivial_index(data.index):\n data = data[:length]\n diff = length - len(data)\n if diff > 0:\n data = pandas.concat(\n [data, pandas.Series([fill_value] * diff)],\n ignore_index=True,\n )\n else:\n data = data.reindex(self_idx, fill_value=fill_value)\n elif use_map and is_list_like(data):\n use_map = False\n new_case.append(data)\n new_caselist.append(tuple(new_case))\n\n if modin_lists[0] or modin_lists[1]:\n # Copartition modin frames\n use_map = False\n columns = self.columns\n column_widths = [1]\n for modin_list, fill_value in zip(modin_lists, fill_values):\n _, list_of_right_parts, joined_index, row_lengths = self._copartition(\n Axis.ROW_WISE.value,\n modin_list,\n how=\"left\",\n sort=False,\n fill_value=fill_value,\n )\n modin_list.clear()\n modin_list.extend(\n self.__constructor__(\n part,\n joined_index,\n columns,\n row_lengths,\n column_widths,\n pandas_backend=self._pandas_backend,\n )\n for part in list_of_right_parts\n )\n\n # Replace modin frames with copartitioned\n caselist = new_caselist\n new_caselist = []\n for i in range(2):\n modin_lists[i] = iter(modin_lists[i])\n for case_tuple in caselist:\n new_case = tuple(\n next(modin_list) if isinstance(data, cls) else data\n for data, modin_list in zip(case_tuple, modin_lists)\n )\n new_caselist.append(new_case)\n\n # If all the conditions are callable and the replacements are either\n # callable or scalar, use map().\n if use_map:\n return self.map(func=remote_fn, func_args=[name, new_caselist], lazy=True)\n\n # Get the chunk of data corresponding the the specified partition\n def map_data(\n part_idx,\n part_len,\n data,\n data_offset,\n fill_value,\n ):\n if isinstance(data, cls):\n return data._partitions[part_idx][0]._data\n if isinstance(data, pandas.Series):\n return data[data_offset : data_offset + part_len]\n return (\n data[data_offset : data_offset + part_len]\n if is_list_like(data)\n else data\n )\n\n parts = [p[0] for p in self._partitions]\n lengths = self.row_lengths\n new_parts = []\n data_offset = 0\n\n # Split the data and apply the remote function to each partition\n # with the corresponding chunk of data\n for i, part, part_len in zip(range(len(parts)), parts, lengths):\n cases = [\n tuple(\n map_data(i, part_len, data, data_offset, fill_value)\n for data, fill_value in zip(c, (True, None))\n )\n for c in new_caselist\n ]\n new_parts.append(\n part.add_to_apply_calls(\n remote_fn,\n name,\n cases,\n length=part_len,\n width=1,\n )\n )\n data_offset += part_len\n new_parts = np.array([[p] for p in new_parts])\n return self.__constructor__(\n new_parts,\n columns=self.columns,\n index=self.index,\n row_lengths=lengths,\n column_widths=[1],\n pandas_backend=self._pandas_backend,\n )\n"
},
{
"path": "modin/core/dataframe/pandas/dataframe/utils.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Collection of algebra utility functions, used to shuffle data across partitions.\"\"\"\n\nimport abc\nfrom collections import namedtuple\nfrom typing import TYPE_CHECKING, Callable, Optional, Union\n\nimport numpy as np\nimport pandas\nfrom pandas._libs.tslibs import to_offset\nfrom pandas.core.dtypes.common import is_list_like, is_numeric_dtype\nfrom pandas.core.resample import _get_timestamp_range_edges\n\nfrom modin.error_message import ErrorMessage\nfrom modin.utils import _inherit_docstrings\n\nif TYPE_CHECKING:\n from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe\n\nColumnInfo = namedtuple(\"ColumnInfo\", [\"name\", \"pivots\", \"is_numeric\"])\n\n\nclass ShuffleFunctions:\n \"\"\"\n Defines an interface to perform the sampling, quantiles picking, and the splitting stages for the range-partitioning building.\n\n Parameters\n ----------\n modin_frame : PandasDataframe\n The frame to build the range-partitioning for.\n columns : str or list of strings\n The column/columns to use as a key.\n ascending : bool\n Whether the ranges should be in ascending or descending order.\n ideal_num_new_partitions : int\n The ideal number of new partitions.\n **kwargs : dict\n Additional keyword arguments.\n \"\"\"\n\n def __init__(\n self, modin_frame, columns, ascending, ideal_num_new_partitions, **kwargs\n ):\n pass\n\n @abc.abstractmethod\n def sample_fn(self, partition: pandas.DataFrame) -> pandas.DataFrame:\n \"\"\"\n Pick samples over the given partition.\n\n Parameters\n ----------\n partition : pandas.DataFrame\n\n Returns\n -------\n pandas.DataFrame:\n The samples for the partition.\n \"\"\"\n pass\n\n @abc.abstractmethod\n def pivot_fn(self, samples: \"list[pandas.DataFrame]\") -> int:\n \"\"\"\n Determine quantiles from the given samples and save it for the future ``.split_fn()`` calls.\n\n Parameters\n ----------\n samples : list of pandas.DataFrames\n\n Returns\n -------\n int\n The number of bins the ``.split_fn()`` will return.\n \"\"\"\n pass\n\n @abc.abstractmethod\n def split_fn(self, partition: pandas.DataFrame) -> \"tuple[pandas.DataFrame, ...]\":\n \"\"\"\n Split the given dataframe into the range-partitions defined by the preceding call of the ``.pivot_fn()``.\n\n Parameters\n ----------\n partition : pandas.DataFrame\n\n Returns\n -------\n tuple of pandas.DataFrames\n\n Notes\n -----\n In order to call this method you must call the ``.pivot_fn()`` first.\n \"\"\"\n pass\n\n\n@_inherit_docstrings(ShuffleFunctions)\nclass ShuffleSortFunctions(ShuffleFunctions):\n \"\"\"\n Perform the sampling, quantiles picking, and the splitting stages for the range-partitioning building.\n\n Parameters\n ----------\n modin_frame : PandasDataframe\n The frame to build the range-partitioning for.\n columns : str, list of strings or None\n The column/columns to use as a key. Can't be specified along with `level`.\n ascending : bool\n Whether the ranges should be in ascending or descending order.\n ideal_num_new_partitions : int\n The ideal number of new partitions.\n level : list of strings or ints, or None\n Index level(s) to use as a key. Can't be specified along with `columns`.\n closed_on_right : bool, default: False\n Whether to include the right limit in range-partitioning.\n True: bins[i - 1] < x <= bins[i]\n False: bins[i - 1] <= x < bins[i]\n **kwargs : dict\n Additional keyword arguments.\n \"\"\"\n\n def __init__(\n self,\n modin_frame: \"PandasDataframe\",\n columns: Optional[Union[str, list]],\n ascending: Union[list, bool],\n ideal_num_new_partitions: int,\n level: Optional[list[Union[str, int]]] = None,\n closed_on_right: bool = False,\n **kwargs: dict,\n ):\n self.frame_len = len(modin_frame)\n self.ideal_num_new_partitions = ideal_num_new_partitions\n self.columns = columns if is_list_like(columns) else [columns]\n self.ascending = ascending\n self.kwargs = kwargs.copy()\n self.level = level\n self.columns_info = None\n self.closed_on_right = closed_on_right\n\n def sample_fn(self, partition: pandas.DataFrame) -> pandas.DataFrame:\n if self.level is not None:\n partition = self._index_to_df_zero_copy(partition, self.level)\n else:\n partition = partition[self.columns]\n return self.pick_samples_for_quantiles(\n partition, self.ideal_num_new_partitions, self.frame_len\n )\n\n def pivot_fn(self, samples: \"list[pandas.DataFrame]\") -> int:\n key = self.kwargs.get(\"key\", None)\n samples = pandas.concat(samples, axis=0, copy=False)\n\n columns_info: \"list[ColumnInfo]\" = []\n number_of_groups = 1\n cols = []\n for i, col in enumerate(samples.columns):\n num_pivots = int(self.ideal_num_new_partitions / number_of_groups)\n if num_pivots < 2 and len(columns_info):\n break\n column_val = samples[col]\n cols.append(col)\n is_numeric = is_numeric_dtype(column_val.dtype)\n\n # When we are not sorting numbers, we need our quantiles to not do arithmetic on the values\n method = \"linear\" if is_numeric else \"inverted_cdf\"\n pivots = self.pick_pivots_from_samples_for_sort(\n column_val, num_pivots, method, key\n )\n columns_info.append(\n ColumnInfo(\n self.level[i] if self.level is not None else col,\n pivots,\n is_numeric,\n )\n )\n number_of_groups *= len(pivots) + 1\n self.columns_info = columns_info\n return number_of_groups\n\n def split_fn(\n self,\n partition: pandas.DataFrame,\n ) -> \"tuple[pandas.DataFrame, ...]\":\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=self.columns_info is None,\n extra_log=\"The 'split_fn' doesn't have proper metadata, the probable reason is that it was called before 'pivot_fn'\",\n )\n return self.split_partitions_using_pivots_for_sort(\n partition,\n self.columns_info,\n self.ascending,\n keys_are_index_levels=self.level is not None,\n closed_on_right=self.closed_on_right,\n **self.kwargs,\n )\n\n @staticmethod\n def _find_quantiles(\n df: Union[pandas.DataFrame, pandas.Series], quantiles: list, method: str\n ) -> np.ndarray:\n \"\"\"\n Find quantiles of a given dataframe using the specified method.\n\n We use this method to provide backwards compatibility with NumPy versions < 1.23 (e.g. when\n the user is using Modin in compat mode). This is basically a wrapper around `np.quantile` that\n ensures we provide the correct `method` argument - i.e. if we are dealing with objects (which\n may or may not support algebra), we do not want to use a method to find quantiles that will\n involve algebra operations (e.g. mean) between the objects, since that may fail.\n\n Parameters\n ----------\n df : pandas.DataFrame or pandas.Series\n The data to pick quantiles from.\n quantiles : list[float]\n The quantiles to compute.\n method : str\n The method to use. `linear` if dealing with numeric types, otherwise `inverted_cdf`.\n\n Returns\n -------\n np.ndarray\n A NumPy array with the quantiles of the data.\n \"\"\"\n if method == \"linear\":\n # This is the default method for finding quantiles, so it does not need to be specified,\n # which keeps backwards compatibility with older versions of NumPy that do not have a\n # `method` keyword argument in np.quantile.\n return np.unique(np.quantile(df, quantiles))\n else:\n try:\n return np.unique(np.quantile(df, quantiles, method=method))\n except Exception:\n # In this case, we're dealing with an array of objects, but the current version of\n # NumPy does not have a `method` kwarg. We need to use the older kwarg, `interpolation`\n # instead.\n return np.unique(np.quantile(df, quantiles, interpolation=\"lower\"))\n\n @staticmethod\n def pick_samples_for_quantiles(\n df: pandas.DataFrame,\n num_partitions: int,\n length: int,\n ) -> pandas.DataFrame:\n \"\"\"\n Pick samples over the given partition.\n\n This function picks samples from the given partition using the TeraSort algorithm - each\n value is sampled with probability 1 / m * ln(n * t) where m = total_length / num_partitions,\n t = num_partitions, and n = total_length.\n\n Parameters\n ----------\n df : pandas.Dataframe\n The masked dataframe to pick samples from.\n num_partitions : int\n The number of partitions.\n length : int\n The total length.\n\n Returns\n -------\n pandas.DataFrame:\n The samples for the partition.\n\n Notes\n -----\n This sampling algorithm is inspired by TeraSort. You can find more information about TeraSort\n and the sampling algorithm at https://www.cse.cuhk.edu.hk/~taoyf/paper/sigmod13-mr.pdf.\n \"\"\"\n m = length / num_partitions\n probability = (1 / m) * np.log(num_partitions * length)\n return df.sample(frac=probability)\n\n def pick_pivots_from_samples_for_sort(\n self,\n samples: pandas.Series,\n ideal_num_new_partitions: int,\n method: str = \"linear\",\n key: Optional[Callable] = None,\n ) -> np.ndarray:\n \"\"\"\n Determine quantiles from the given samples.\n\n This function takes as input the quantiles calculated over all partitions from\n `sample_func` defined above, and determines a final NPartitions.get() quantiles\n to use to roughly sort the entire dataframe. It does so by collating all the samples\n and computing NPartitions.get() quantiles for the overall set.\n\n Parameters\n ----------\n samples : pandas.Series\n The samples computed by ``get_partition_quantiles_for_sort``.\n ideal_num_new_partitions : int\n The ideal number of new partitions.\n method : str, default: linear\n The method to use when picking quantiles.\n key : Callable, default: None\n The key to use on the samples when picking pivots.\n\n Returns\n -------\n np.ndarray\n A list of overall quantiles.\n \"\"\"\n samples = samples.to_numpy()\n # We don't call `np.unique` on the samples, since if a quantile shows up in multiple\n # partition's samples, this is probably an indicator of skew in the dataset, and we\n # want our final partitions to take this into account.\n if key is not None:\n samples = key(samples)\n # We don't want to pick very many quantiles if we have a very small dataframe.\n num_quantiles = ideal_num_new_partitions\n quantiles = [i / num_quantiles for i in range(1, num_quantiles)]\n # If we only desire 1 partition, we need to ensure that we're not trying to find quantiles\n # from an empty list of pivots.\n if len(quantiles) > 0:\n return self._find_quantiles(samples, quantiles, method)\n return np.array([])\n\n @staticmethod\n def split_partitions_using_pivots_for_sort(\n df: pandas.DataFrame,\n columns_info: \"list[ColumnInfo]\",\n ascending: bool,\n keys_are_index_levels: bool = False,\n closed_on_right: bool = False,\n **kwargs: dict,\n ) -> \"tuple[pandas.DataFrame, ...]\":\n \"\"\"\n Split the given dataframe into the partitions specified by `pivots` in `columns_info`.\n\n This function takes as input a row-axis partition, as well as the quantiles determined\n by the `pivot_func` defined above. It then splits the input dataframe into NPartitions.get()\n dataframes, with the elements in the i-th split belonging to the i-th partition, as determined\n by the quantiles we're using.\n\n Parameters\n ----------\n df : pandas.Dataframe\n The partition to split.\n columns_info : list of ColumnInfo\n Information regarding keys and pivots for range partitioning.\n ascending : bool\n The ascending flag.\n keys_are_index_levels : bool, default: False\n Whether `columns_info` describes index levels or actual columns from `df`.\n closed_on_right : bool, default: False\n Whether to include the right limit in range-partitioning.\n True: bins[i - 1] < x <= bins[i]\n False: bins[i - 1] <= x < bins[i]\n **kwargs : dict\n Additional keyword arguments.\n\n Returns\n -------\n tuple[pandas.DataFrame]\n A tuple of the splits from this partition.\n \"\"\"\n if len(columns_info) == 0:\n # We can return the dataframe with zero changes if there were no pivots passed\n return (df,)\n\n key_data = (\n ShuffleSortFunctions._index_to_df_zero_copy(\n df, [col_info.name for col_info in columns_info]\n )\n if keys_are_index_levels\n else df[[col_info.name for col_info in columns_info]]\n )\n na_index = key_data.isna().squeeze(axis=1)\n if na_index.ndim == 2:\n na_index = na_index.any(axis=1)\n na_rows = df[na_index]\n non_na_rows = df[~na_index]\n\n def get_group(grp, key, df):\n \"\"\"Get a group with the `key` from the `grp`, if it doesn't exist return an empty slice of `df`.\"\"\"\n try:\n return grp.get_group(key)\n except KeyError:\n return pandas.DataFrame(index=df.index[:0], columns=df.columns).astype(\n df.dtypes\n )\n\n groupby_codes = []\n group_keys = []\n for col_info in columns_info:\n pivots = col_info.pivots\n if len(pivots) == 0:\n continue\n # If `ascending=False` and we are dealing with a numeric dtype, we can pass in a reversed list\n # of pivots, and `np.digitize` will work correctly. For object dtypes, we use `np.searchsorted`\n # which breaks when we reverse the pivots.\n if not ascending and col_info.is_numeric:\n # `key` is already applied to `pivots` in the `pick_pivots_from_samples_for_sort` function.\n pivots = pivots[::-1]\n group_keys.append(range(len(pivots) + 1))\n key = kwargs.pop(\"key\", None)\n cols_to_digitize = (\n non_na_rows.index.get_level_values(col_info.name)\n if keys_are_index_levels\n else non_na_rows[col_info.name]\n )\n if key is not None:\n cols_to_digitize = key(cols_to_digitize)\n\n if cols_to_digitize.ndim == 2:\n cols_to_digitize = cols_to_digitize.squeeze()\n\n if col_info.is_numeric:\n groupby_col = np.digitize(\n cols_to_digitize, pivots, right=closed_on_right\n )\n # `np.digitize` returns results based off of the sort order of the pivots it is passed.\n # When we only have one unique value in our pivots, `np.digitize` assumes that the pivots\n # are sorted in ascending order, and gives us results based off of that assumption - so if\n # we actually want to sort in descending order, we need to swap the new indices.\n if not ascending and len(np.unique(pivots)) == 1:\n groupby_col = len(pivots) - groupby_col\n else:\n groupby_col = np.searchsorted(\n pivots,\n cols_to_digitize,\n side=\"left\" if closed_on_right else \"right\",\n )\n # Since np.searchsorted requires the pivots to be in ascending order, if we want to sort\n # in descending order, we need to swap the new indices.\n if not ascending:\n groupby_col = len(pivots) - groupby_col\n groupby_codes.append(groupby_col)\n\n if len(group_keys) == 0:\n # We can return the dataframe with zero changes if there were no pivots passed\n return (df,)\n elif len(group_keys) == 1:\n group_keys = group_keys[0]\n else:\n group_keys = pandas.MultiIndex.from_product(group_keys)\n\n if len(non_na_rows) == 1:\n groups = [\n # taking an empty slice for an index's metadata\n (\n pandas.DataFrame(index=df.index[:0], columns=df.columns).astype(\n df.dtypes\n )\n if key != groupby_codes[0]\n else non_na_rows\n )\n for key in group_keys\n ]\n else:\n grouped = non_na_rows.groupby(groupby_codes)\n groups = [get_group(grouped, key, df) for key in group_keys]\n index_to_insert_na_vals = (\n -1 if kwargs.get(\"na_position\", \"last\") == \"last\" else 0\n )\n groups[index_to_insert_na_vals] = pandas.concat(\n [groups[index_to_insert_na_vals], na_rows]\n ).astype(df.dtypes)\n return tuple(groups)\n\n @staticmethod\n def _index_to_df_zero_copy(\n df: pandas.DataFrame, levels: list[Union[str, int]]\n ) -> pandas.DataFrame:\n \"\"\"\n Convert index `level` of `df` to a ``pandas.DataFrame``.\n\n Parameters\n ----------\n df : pandas.DataFrame\n levels : list of labels or ints\n Index level to convert to a dataframe.\n\n Returns\n -------\n pandas.DataFrame\n The columns in the resulting dataframe use the same data arrays as the index levels\n in the original `df`, so no copies.\n \"\"\"\n # calling 'df.index.to_frame()' creates a copy of the index, so doing the conversion manually\n # to avoid the copy\n data = {\n (\n df.index.names[lvl] if isinstance(lvl, int) else lvl\n ): df.index.get_level_values(lvl)\n for lvl in levels\n }\n index_data = pandas.DataFrame(data, index=df.index, copy=False)\n return index_data\n\n\n@_inherit_docstrings(ShuffleSortFunctions)\nclass ShuffleResample(ShuffleSortFunctions):\n def __init__(\n self,\n modin_frame: \"PandasDataframe\",\n columns: Union[str, list],\n ascending: Union[list, bool],\n ideal_num_new_partitions: int,\n resample_kwargs: dict,\n **kwargs: dict,\n ):\n resample_kwargs = resample_kwargs.copy()\n rule = resample_kwargs.pop(\"rule\")\n\n if resample_kwargs[\"closed\"] is None:\n # this rule regarding the default value of 'closed' is inherited\n # from pandas documentation for 'pandas.DataFrame.resample'\n if rule in (\"ME\", \"YE\", \"QE\", \"BME\", \"BA\", \"BQE\", \"W\"):\n resample_kwargs[\"closed\"] = \"right\"\n else:\n resample_kwargs[\"closed\"] = \"left\"\n\n super().__init__(\n modin_frame,\n columns,\n ascending,\n ideal_num_new_partitions,\n closed_on_right=resample_kwargs[\"closed\"] == \"right\",\n **kwargs,\n )\n\n resample_kwargs[\"freq\"] = to_offset(rule)\n self.resample_kwargs = resample_kwargs\n\n @staticmethod\n def pick_samples_for_quantiles(\n df: pandas.DataFrame,\n num_partitions: int,\n length: int,\n ) -> pandas.DataFrame:\n # to build proper bins we need min and max timestamp of the whole DatetimeIndex,\n # so computing it in each partition\n return pandas.concat([df.min().to_frame().T, df.max().to_frame().T])\n\n def pick_pivots_from_samples_for_sort(\n self,\n samples: np.ndarray,\n ideal_num_new_partitions: int,\n method: str = \"linear\",\n key: Optional[Callable] = None,\n ) -> np.ndarray:\n if key is not None:\n raise NotImplementedError(key)\n\n max_value = samples.max()\n\n first, last = _get_timestamp_range_edges(\n samples.min(),\n max_value,\n self.resample_kwargs[\"freq\"],\n unit=samples.dt.unit,\n closed=self.resample_kwargs[\"closed\"],\n origin=self.resample_kwargs[\"origin\"],\n offset=self.resample_kwargs[\"offset\"],\n )\n\n all_bins = pandas.date_range(\n start=first,\n end=last,\n freq=self.resample_kwargs[\"freq\"],\n ambiguous=True,\n nonexistent=\"shift_forward\",\n unit=samples.dt.unit,\n )\n\n all_bins = self._adjust_bin_edges(\n all_bins,\n max_value,\n freq=self.resample_kwargs[\"freq\"],\n closed=self.resample_kwargs[\"closed\"],\n )\n\n # take pivot values with an even interval\n step = 1 / ideal_num_new_partitions\n bins = [\n all_bins[int(len(all_bins) * i * step)]\n for i in range(1, ideal_num_new_partitions)\n ]\n return bins\n\n def _adjust_bin_edges(\n self,\n binner: pandas.DatetimeIndex,\n end_timestamp,\n freq,\n closed,\n ) -> pandas.DatetimeIndex:\n \"\"\"\n Adjust bin edges.\n\n This function was copied & simplified from ``pandas.core.resample.TimeGrouper._adjuct_bin_edges()``.\n\n Parameters\n ----------\n binner : pandas.DatetimeIndex\n end_timestamp : pandas.Timestamp\n freq : str\n closed : bool\n\n Returns\n -------\n pandas.DatetimeIndex\n \"\"\"\n # Some hacks for > daily data, see pandas-dev/pandas#1471, pandas-dev/pandas#1458, pandas-dev/pandas#1483\n\n if freq.name not in (\"BME\", \"ME\", \"W\") and freq.name.split(\"-\")[0] not in (\n \"BQE\",\n \"BYE\",\n \"QE\",\n \"YE\",\n \"W\",\n ):\n return binner\n\n # If the right end-point is on the last day of the month, roll forwards\n # until the last moment of that day. Note that we only do this for offsets\n # which correspond to the end of a super-daily period - \"month start\", for\n # example, is excluded.\n if closed == \"right\":\n # GH 21459, GH 9119: Adjust the bins relative to the wall time\n edges_dti = binner.tz_localize(None)\n edges_dti = (\n edges_dti\n + pandas.Timedelta(days=1, unit=edges_dti.unit).as_unit(edges_dti.unit)\n - pandas.Timedelta(1, unit=edges_dti.unit).as_unit(edges_dti.unit)\n )\n binner = edges_dti.tz_localize(binner.tz)\n\n # intraday values on last day\n if binner[-2] > end_timestamp:\n binner = binner[:-1]\n return binner\n\n @staticmethod\n def split_partitions_using_pivots_for_sort(\n df: pandas.DataFrame,\n columns_info: \"list[ColumnInfo]\",\n ascending: bool,\n closed_on_right: bool = True,\n **kwargs: dict,\n ) -> \"tuple[pandas.DataFrame, ...]\":\n def add_attr(df, timestamp):\n if \"bin_bounds\" in df.attrs:\n df.attrs[\"bin_bounds\"] = (*df.attrs[\"bin_bounds\"], timestamp)\n else:\n df.attrs[\"bin_bounds\"] = (timestamp,)\n return df\n\n result = ShuffleSortFunctions.split_partitions_using_pivots_for_sort(\n df, columns_info, ascending, **kwargs\n )\n # it's required for each bin to know its bounds in order for resampling to work\n # properly when down-sampling occurs. Reach here for an example:\n # https://github.com/modin-project/modin/pull/7140#discussion_r1549246505\n # We're writing the bounds as 'attrs' to avoid duplications in the final partition\n for i, pivot in enumerate(columns_info[0].pivots):\n add_attr(result[i], pivot - pandas.Timedelta(1, unit=\"ns\"))\n if i + 1 <= len(result):\n add_attr(result[i + 1], pivot + pandas.Timedelta(1, unit=\"ns\"))\n return result\n\n\ndef lazy_metadata_decorator(apply_axis=None, axis_arg=-1, transpose=False):\n \"\"\"\n Lazily propagate metadata for the ``PandasDataframe``.\n\n This decorator first adds the minimum required reindexing operations\n to each partition's queue of functions to be lazily applied for\n each PandasDataframe in the arguments by applying the function\n run_f_on_minimally_updated_metadata. The decorator also sets the\n flags for deferred metadata synchronization on the function result\n if necessary.\n\n Parameters\n ----------\n apply_axis : str, default: None\n The axes on which to apply the reindexing operations to the `self._partitions` lazily.\n Case None: No lazy metadata propagation.\n Case \"both\": Add reindexing operations on both axes to partition queue.\n Case \"opposite\": Add reindexing operations complementary to given axis.\n Case \"rows\": Add reindexing operations on row axis to partition queue.\n axis_arg : int, default: -1\n The index or column axis.\n transpose : bool, default: False\n Boolean for if a transpose operation is being used.\n\n Returns\n -------\n Wrapped Function.\n \"\"\"\n\n def decorator(f):\n from functools import wraps\n\n @wraps(f)\n def run_f_on_minimally_updated_metadata(self, *args, **kwargs):\n from .dataframe import PandasDataframe\n\n for obj in (\n [self]\n + [o for o in args if isinstance(o, PandasDataframe)]\n + [v for v in kwargs.values() if isinstance(v, PandasDataframe)]\n + [\n d\n for o in args\n if isinstance(o, list)\n for d in o\n if isinstance(d, PandasDataframe)\n ]\n + [\n d\n for _, o in kwargs.items()\n if isinstance(o, list)\n for d in o\n if isinstance(d, PandasDataframe)\n ]\n ):\n if apply_axis == \"both\":\n if obj._deferred_index and obj._deferred_column:\n obj._propagate_index_objs(axis=None)\n elif obj._deferred_index:\n obj._propagate_index_objs(axis=0)\n elif obj._deferred_column:\n obj._propagate_index_objs(axis=1)\n elif apply_axis == \"opposite\":\n if \"axis\" not in kwargs:\n axis = args[axis_arg]\n else:\n axis = kwargs[\"axis\"]\n if axis == 0 and obj._deferred_column:\n obj._propagate_index_objs(axis=1)\n elif axis == 1 and obj._deferred_index:\n obj._propagate_index_objs(axis=0)\n elif apply_axis == \"rows\":\n obj._propagate_index_objs(axis=0)\n result = f(self, *args, **kwargs)\n if apply_axis is None and not transpose:\n result._deferred_index = self._deferred_index\n result._deferred_column = self._deferred_column\n elif apply_axis is None and transpose:\n result._deferred_index = self._deferred_column\n result._deferred_column = self._deferred_index\n elif apply_axis == \"opposite\":\n if axis == 0:\n result._deferred_index = self._deferred_index\n else:\n result._deferred_column = self._deferred_column\n elif apply_axis == \"rows\":\n result._deferred_column = self._deferred_column\n return result\n\n return run_f_on_minimally_updated_metadata\n\n return decorator\n\n\ndef add_missing_categories_to_groupby(\n dfs,\n by,\n operator,\n initial_columns,\n combined_cols,\n is_udf_agg,\n kwargs,\n initial_dtypes=None,\n):\n \"\"\"\n Generate values for missing categorical values to be inserted into groupby result.\n\n This function is used to emulate behavior of ``groupby(observed=False)`` parameter,\n it takes groupby result that was computed using ``groupby(observed=True)``\n and computes results for categorical values that are not presented in `dfs`.\n\n Parameters\n ----------\n dfs : list of pandas.DataFrames\n Row partitions containing groupby results.\n by : list of hashable\n Column labels that were used to perform groupby.\n operator : callable\n Aggregation function that was used during groupby.\n initial_columns : pandas.Index\n Column labels of the original dataframe.\n combined_cols : pandas.Index\n Column labels of the groupby result.\n is_udf_agg : bool\n Whether ``operator`` is a UDF.\n kwargs : dict\n Parameters that were passed to ``groupby(by, **kwargs)``.\n initial_dtypes : pandas.Series, optional\n Dtypes of the original dataframe. If not specified, assume it's ``int64``.\n\n Returns\n -------\n masks : dict[int, pandas.DataFrame]\n Mapping between partition idx and a dataframe with results for missing categorical values\n to insert to this partition.\n new_combined_cols : pandas.Index\n New column labels of the groupby result. If ``is_udf_agg is True``, then ``operator``\n may change the resulted columns.\n \"\"\"\n kwargs[\"observed\"] = False\n new_combined_cols = combined_cols\n\n ### At first we need to compute missing categorical values\n indices = [df.index for df in dfs]\n # total_index contains all categorical values that resided in the result,\n # missing values are computed differently depending on whether we're grouping\n # on multiple groupers or not\n total_index = indices[0].append(indices[1:])\n if isinstance(total_index, pandas.MultiIndex):\n if all(\n not isinstance(level, pandas.CategoricalIndex)\n for level in total_index.levels\n ):\n return {}, new_combined_cols\n missing_cats_dtype = {\n name: (\n level.dtype\n if isinstance(level.dtype, pandas.CategoricalDtype)\n # it's a bit confusing but we have to convert the remaining 'by' columns to categoricals\n # in order to compute a proper fill value later in the code\n else pandas.CategoricalDtype(level)\n )\n for level, name in zip(total_index.levels, total_index.names)\n }\n # if we're grouping on multiple groupers, then the missing categorical values is a\n # carthesian product of (actual_missing_categorical_values X all_values_of_another_groupers)\n complete_index = pandas.MultiIndex.from_product(\n [\n value.categories.astype(total_level.dtype)\n for total_level, value in zip(\n total_index.levels, missing_cats_dtype.values()\n )\n ],\n names=by,\n )\n missing_index = complete_index[~complete_index.isin(total_index)]\n else:\n if not isinstance(total_index, pandas.CategoricalIndex):\n return {}, new_combined_cols\n # if we're grouping on a single grouper then we simply compute the difference\n # between categorical values in the result and the values defined in categorical dtype\n missing_index = total_index.categories.difference(total_index.values)\n missing_cats_dtype = {by[0]: pandas.CategoricalDtype(missing_index)}\n missing_index.names = by\n\n if len(missing_index) == 0:\n return {}, new_combined_cols\n\n ### At this stage we want to get a fill_value for missing categorical values\n if is_udf_agg and isinstance(total_index, pandas.MultiIndex):\n # if grouping on multiple columns and aggregating with an UDF, then the\n # fill value is always `np.nan`\n missing_values = pandas.DataFrame({0: [np.nan]})\n else:\n # In case of a udf aggregation we're forced to run the operator against each\n # missing category, as in theory it can return different results for each\n # empty group. In other cases it's enough to run the operator against a single\n # missing categorical and then broadcast the fill value to each missing value\n if not is_udf_agg:\n missing_cats_dtype = {\n key: pandas.CategoricalDtype(value.categories[:1])\n for key, value in missing_cats_dtype.items()\n }\n\n empty_df = pandas.DataFrame(columns=initial_columns)\n # HACK: default 'object' dtype doesn't fit our needs, as most of the aggregations\n # fail on a non-numeric columns, ideally, we need dtypes of the original dataframe,\n # however, 'int64' also works fine here if the original schema is not available\n empty_df = empty_df.astype(\n \"int64\" if initial_dtypes is None else initial_dtypes\n )\n empty_df = empty_df.astype(missing_cats_dtype)\n missing_values = operator(empty_df.groupby(by, **kwargs))\n\n if is_udf_agg and not isinstance(total_index, pandas.MultiIndex):\n missing_values = missing_values.drop(columns=by, errors=\"ignore\")\n new_combined_cols = pandas.concat(\n [\n pandas.DataFrame(columns=combined_cols),\n missing_values.iloc[:0],\n ],\n axis=0,\n join=\"outer\",\n ).columns\n else:\n # HACK: If the aggregation has failed, the result would be empty. Assuming the\n # fill value to be `np.nan` here (this may not always be correct!!!)\n fill_value = np.nan if len(missing_values) == 0 else missing_values.iloc[0, 0]\n missing_values = pandas.DataFrame(\n fill_value, index=missing_index, columns=combined_cols\n )\n\n # restoring original categorical dtypes for the indices (MultiIndex already have proper dtypes)\n if not isinstance(missing_values.index, pandas.MultiIndex):\n missing_values.index = missing_values.index.astype(total_index.dtype)\n\n ### Then we decide to which missing categorical values should go to which partition\n if not kwargs[\"sort\"]:\n # If the result is allowed to be unsorted, simply insert all the missing\n # categories to the last partition\n mask = {len(indices) - 1: missing_values}\n return mask, new_combined_cols\n\n # If the result has to be sorted, we have to assign missing categoricals to proper partitions.\n # For that purpose we define bins with corner values of each partition and then using either\n # np.digitize or np.searchsorted find correct bins for each missing categorical value.\n # Example: part0-> [0, 1, 2]; part1-> [3, 4, 10, 12]; part2-> [15, 17, 20, 100]\n # bins -> [2, 12] # took last values of each partition excluding the last partition\n # (every value that's matching 'x > part[-2][-1]' should go to the\n # last partition, meaning that including the last value of the last\n # partitions doesn't make sense)\n # missing_cats -> [-2, 5, 6, 14, 21, 120]\n # np.digitize(missing_cats, bins) -> [ 0, 1, 1, 2, 2, 2]\n # ^-- mapping between values and partition idx to insert\n bins = []\n old_bins_to_new = {}\n offset = 0\n # building bins by taking last values of each partition excluding the last partition\n for idx in indices[:-1]:\n if len(idx) == 0:\n # if a partition is empty, we can't use its values to define a bin, thus we simply\n # skip it and remember the number of skipped partitions as an 'offset'\n offset += 1\n continue\n # remember the number of skipped partitions before this bin, in order to restore original\n # indexing at the end\n old_bins_to_new[len(bins)] = offset\n # for MultiIndices we always use the very first level for bins as using multiple levels\n # doesn't affect the result\n bins.append(idx[-1][0] if isinstance(idx, pandas.MultiIndex) else idx[-1])\n old_bins_to_new[len(bins)] = offset\n\n if len(bins) == 0:\n # insert values to the first non-empty partition\n return {old_bins_to_new.get(0, 0): missing_values}, new_combined_cols\n\n # we used the very first level of MultiIndex to build bins, meaning that we also have\n # to use values of the first index's level for 'digitize'\n lvl_zero = (\n missing_values.index.levels[0]\n if isinstance(missing_values.index, pandas.MultiIndex)\n else missing_values.index\n )\n if pandas.api.types.is_any_real_numeric_dtype(lvl_zero):\n part_idx = np.digitize(lvl_zero, bins, right=True)\n else:\n part_idx = np.searchsorted(bins, lvl_zero)\n\n ### In the end we build a dictionary mapping partition index to a dataframe with missing categoricals\n ### to be inserted into this partition\n masks = {}\n if isinstance(total_index, pandas.MultiIndex):\n for idx, values in pandas.RangeIndex(len(lvl_zero)).groupby(part_idx).items():\n masks[idx] = missing_values[\n pandas.Index(missing_values.index.codes[0]).isin(values)\n ]\n else:\n frame_idx = missing_values.index.to_frame()\n for idx, values in lvl_zero.groupby(part_idx).items():\n masks[idx] = missing_values[frame_idx.iloc[:, 0].isin(values)]\n\n # Restore the original indexing by adding the amount of skipped missing partitions\n masks = {key + old_bins_to_new[key]: value for key, value in masks.items()}\n return masks, new_combined_cols\n"
},
{
"path": "modin/core/dataframe/pandas/interchange/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe functionality related to data exchange protocols and optimized for pandas storage format.\"\"\"\n"
},
{
"path": "modin/core/dataframe/pandas/interchange/dataframe_protocol/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nBase Modin Dataframe functionality related to the dataframe exchange protocol and optimized for pandas storage format.\n\nSee more in https://data-apis.org/dataframe-protocol/latest/index.html.\n\"\"\"\n"
},
{
"path": "modin/core/dataframe/pandas/interchange/dataframe_protocol/buffer.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nDataframe exchange protocol implementation.\n\nSee more in https://data-apis.org/dataframe-protocol/latest/index.html.\n\nNotes\n-----\n- Interpreting a raw pointer (as in ``Buffer.ptr``) is annoying and unsafe to\n do in pure Python. It's more general but definitely less friendly than having\n ``to_arrow`` and ``to_numpy`` methods. So for the buffers which lack\n ``__dlpack__`` (e.g., because the column dtype isn't supported by DLPack),\n this is worth looking at again.\n\"\"\"\n\nimport enum\nfrom typing import Tuple\n\nimport numpy as np\n\nfrom modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (\n ProtocolBuffer,\n)\nfrom modin.utils import _inherit_docstrings\n\n\n@_inherit_docstrings(ProtocolBuffer)\nclass PandasProtocolBuffer(ProtocolBuffer):\n \"\"\"\n Data in the buffer is guaranteed to be contiguous in memory.\n\n Note that there is no dtype attribute present, a buffer can be thought of\n as simply a block of memory. However, if the column that the buffer is\n attached to has a dtype that's supported by DLPack and ``__dlpack__`` is\n implemented, then that dtype information will be contained in the return\n value from ``__dlpack__``.\n\n This distinction is useful to support both (a) data exchange via DLPack on a\n buffer and (b) dtypes like variable-length strings which do not have a\n fixed number of bytes per element.\n\n Parameters\n ----------\n x : np.ndarray\n Data to be held by ``Buffer``.\n allow_copy : bool, default: True\n A keyword that defines whether or not the library is allowed\n to make a copy of the data. For example, copying data would be necessary\n if a library supports strided buffers, given that this protocol\n specifies contiguous buffers. Currently, if the flag is set to ``False``\n and a copy is needed, a ``RuntimeError`` will be raised.\n \"\"\"\n\n def __init__(self, x: np.ndarray, allow_copy: bool = True) -> None:\n if not x.strides == (x.dtype.itemsize,):\n # The protocol does not support strided buffers, so a copy is\n # necessary. If that's not allowed, we need to raise an exception.\n if allow_copy:\n x = x.copy()\n else:\n raise RuntimeError(\n \"Exports cannot be zero-copy in the case \"\n + \"of a non-contiguous buffer\"\n )\n\n # Store the numpy array in which the data resides as a private\n # attribute, so we can use it to retrieve the public attributes\n self._x = x\n\n @property\n def bufsize(self) -> int:\n return self._x.size * self._x.dtype.itemsize\n\n @property\n def ptr(self) -> int:\n return self._x.__array_interface__[\"data\"][0]\n\n def __dlpack__(self):\n raise NotImplementedError(\"__dlpack__\")\n\n def __dlpack_device__(self) -> Tuple[enum.IntEnum, int]:\n class Device(enum.IntEnum):\n CPU = 1\n\n return (Device.CPU, None)\n\n def __repr__(self) -> str:\n \"\"\"\n Return a string representation for a particular ``PandasProtocolBuffer``.\n\n Returns\n -------\n str\n \"\"\"\n return (\n \"Buffer(\"\n + str(\n {\n \"bufsize\": self.bufsize,\n \"ptr\": self.ptr,\n \"device\": self.__dlpack_device__()[0].name,\n }\n )\n + \")\"\n )\n"
},
{
"path": "modin/core/dataframe/pandas/interchange/dataframe_protocol/column.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nDataframe exchange protocol implementation.\n\nSee more in https://data-apis.org/dataframe-protocol/latest/index.html.\n\nNotes\n-----\n- Interpreting a raw pointer (as in ``Buffer.ptr``) is annoying and unsafe to\n do in pure Python. It's more general but definitely less friendly than having\n ``to_arrow`` and ``to_numpy`` methods. So for the buffers which lack\n ``__dlpack__`` (e.g., because the column dtype isn't supported by DLPack),\n this is worth looking at again.\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom functools import cached_property\nfrom typing import Any, Dict, Iterable, Optional, Tuple\n\nimport numpy as np\nimport pandas\n\nfrom modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (\n CategoricalDescription,\n ProtocolColumn,\n)\nfrom modin.core.dataframe.base.interchange.dataframe_protocol.utils import (\n ColumnNullType,\n DTypeKind,\n pandas_dtype_to_arrow_c,\n)\nfrom modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe\nfrom modin.utils import _inherit_docstrings\n\nfrom .buffer import PandasProtocolBuffer\nfrom .exception import NoOffsetsBuffer, NoValidityBuffer\n\n_NO_VALIDITY_BUFFER = {\n ColumnNullType.NON_NULLABLE: \"This column is non-nullable so does not have a mask\",\n ColumnNullType.USE_NAN: \"This column uses NaN as null so does not have a separate mask\",\n ColumnNullType.USE_SENTINEL: \"This column uses a sentinel value so does not have a mask\",\n}\n\n\n@_inherit_docstrings(ProtocolColumn)\nclass PandasProtocolColumn(ProtocolColumn):\n \"\"\"\n A column object, with only the methods and properties required by the interchange protocol defined.\n\n A column can contain one or more chunks. Each chunk can contain up to three\n buffers - a data buffer, a mask buffer (depending on null representation),\n and an offsets buffer (if variable-size binary; e.g., variable-length strings).\n\n TBD: Arrow has a separate \"null\" dtype, and has no separate mask concept.\n Instead, it seems to use \"children\" for both columns with a bit mask,\n and for nested dtypes. Unclear whether this is elegant or confusing.\n This design requires checking the null representation explicitly.\n The Arrow design requires checking:\n 1. the ARROW_FLAG_NULLABLE (for sentinel values)\n 2. if a column has two children, combined with one of those children\n having a null dtype.\n Making the mask concept explicit seems useful. One null dtype would\n not be enough to cover both bit and byte masks, so that would mean\n even more checking if we did it the Arrow way.\n TBD: there's also the \"chunk\" concept here, which is implicit in Arrow as\n multiple buffers per array (= column here). Semantically it may make\n sense to have both: chunks were meant for example for lazy evaluation\n of data which doesn't fit in memory, while multiple buffers per column\n could also come from doing a selection operation on a single\n contiguous buffer.\n Given these concepts, one would expect chunks to be all of the same\n size (say a 10,000 row dataframe could have 10 chunks of 1,000 rows),\n while multiple buffers could have data-dependent lengths. Not an issue\n in pandas if one column is backed by a single NumPy array, but in\n Arrow it seems possible.\n Are multiple chunks *and* multiple buffers per column necessary for\n the purposes of this interchange protocol, or must producers either\n reuse the chunk concept for this or copy the data?\n\n Parameters\n ----------\n column : PandasDataframe\n A ``PandasDataframe`` object.\n allow_copy : bool, default: True\n A keyword that defines whether or not the library is allowed\n to make a copy of the data. For example, copying data would be necessary\n if a library supports strided buffers, given that this protocol\n specifies contiguous buffers. Currently, if the flag is set to ``False``\n and a copy is needed, a ``RuntimeError`` will be raised.\n\n Notes\n -----\n This Column object can only be produced by ``__dataframe__``,\n so doesn't need its own version or ``__column__`` protocol.\n \"\"\"\n\n def __init__(self, column: PandasDataframe, allow_copy: bool = True) -> None:\n if not isinstance(column, PandasDataframe):\n raise NotImplementedError(f\"Columns of type {type(column)} not handled yet\")\n\n self._col = column\n self._allow_copy = allow_copy\n\n def size(self) -> int:\n return len(self._col.index)\n\n @property\n def offset(self) -> int:\n return 0\n\n @cached_property\n def dtype(self) -> Tuple[DTypeKind, int, str, str]:\n dtype = self._col.dtypes.iloc[0]\n\n if isinstance(dtype, pandas.CategoricalDtype):\n pandas_series = self._col.to_pandas().squeeze(axis=1)\n codes = pandas_series.values.codes\n (\n _,\n bitwidth,\n c_arrow_dtype_f_str,\n _,\n ) = self._dtype_from_primitive_pandas_dtype(codes.dtype)\n dtype_cache = (\n DTypeKind.CATEGORICAL,\n bitwidth,\n c_arrow_dtype_f_str,\n \"=\",\n )\n elif pandas.api.types.is_string_dtype(dtype):\n dtype_cache = (DTypeKind.STRING, 8, pandas_dtype_to_arrow_c(dtype), \"=\")\n else:\n dtype_cache = self._dtype_from_primitive_pandas_dtype(dtype)\n\n return dtype_cache\n\n def _dtype_from_primitive_pandas_dtype(\n self, dtype\n ) -> Tuple[DTypeKind, int, str, str]:\n \"\"\"\n Deduce dtype specific for the protocol from pandas dtype.\n\n See `self.dtype` for details.\n\n Parameters\n ----------\n dtype : any\n A pandas dtype.\n\n Returns\n -------\n tuple\n \"\"\"\n _np_kinds = {\n \"i\": DTypeKind.INT,\n \"u\": DTypeKind.UINT,\n \"f\": DTypeKind.FLOAT,\n \"b\": DTypeKind.BOOL,\n \"M\": DTypeKind.DATETIME,\n }\n kind = _np_kinds.get(dtype.kind, None)\n if kind is None:\n raise NotImplementedError(\n f\"Data type {dtype} not supported by the dataframe exchange protocol\"\n )\n return (\n kind,\n dtype.itemsize * 8,\n pandas_dtype_to_arrow_c(dtype),\n dtype.byteorder,\n )\n\n @property\n def describe_categorical(self) -> CategoricalDescription:\n if self.dtype[0] != DTypeKind.CATEGORICAL:\n raise TypeError(\n \"`describe_categorical only works on a column with \"\n + \"categorical dtype!\"\n )\n\n pandas_series = self._col.to_pandas().squeeze(axis=1)\n cat_frame = type(self._col).from_pandas(\n pandas.DataFrame({\"cat\": pandas_series.cat.categories})\n )\n return {\n \"is_ordered\": pandas_series.cat.ordered,\n \"is_dictionary\": True,\n \"categories\": PandasProtocolColumn(cat_frame, self._allow_copy),\n }\n\n @property\n def describe_null(self) -> Tuple[int, Any]:\n nulls = {\n DTypeKind.FLOAT: (ColumnNullType.USE_NAN, None),\n DTypeKind.DATETIME: (ColumnNullType.USE_NAN, None),\n DTypeKind.INT: (ColumnNullType.NON_NULLABLE, None),\n DTypeKind.UINT: (ColumnNullType.NON_NULLABLE, None),\n DTypeKind.BOOL: (ColumnNullType.NON_NULLABLE, None),\n # Null values for categoricals are stored as `-1` sentinel values\n # in the category date (e.g., `col.values.codes` is int8 np.ndarray)\n DTypeKind.CATEGORICAL: (ColumnNullType.USE_SENTINEL, -1),\n # follow Arrow in using 1 as valid value and 0 for missing/null value\n DTypeKind.STRING: (ColumnNullType.USE_BYTEMASK, 0),\n }\n\n kind = self.dtype[0]\n try:\n null, value = nulls[kind]\n except KeyError:\n raise NotImplementedError(f\"Data type {kind} not yet supported\")\n\n return null, value\n\n @cached_property\n def null_count(self) -> int:\n\n def map_func(df):\n return df.isna()\n\n def reduce_func(df):\n return pandas.DataFrame(df.sum())\n\n intermediate_df = self._col.tree_reduce(0, map_func, reduce_func)\n # Set ``pandas.RangeIndex(1)`` to index and column labels because\n # 1) We internally use `MODIN_UNNAMED_SERIES_LABEL` for labels of a reduced axis\n # 2) The return value of `reduce_func` is a pandas DataFrame with\n # index and column labels set to ``pandas.RangeIndex(1)``\n # 3) We further use `to_pandas().squeeze()` to get an integer value of the null count.\n # Otherwise, we get mismatching internal and external indices for both axes\n intermediate_df.index = pandas.RangeIndex(1)\n intermediate_df.columns = pandas.RangeIndex(1)\n return intermediate_df.to_pandas().squeeze(axis=1).item()\n\n @property\n def metadata(self) -> Dict[str, Any]:\n return {\"modin.index\": self._col.index}\n\n def num_chunks(self) -> int:\n return self._col._partitions.shape[0]\n\n def get_chunks(\n self, n_chunks: Optional[int] = None\n ) -> Iterable[\"PandasProtocolColumn\"]:\n cur_n_chunks = self.num_chunks()\n n_rows = self.size()\n if n_chunks is None or n_chunks == cur_n_chunks:\n cum_row_lengths = np.cumsum([0] + self._col.row_lengths)\n for i in range(len(cum_row_lengths) - 1):\n yield PandasProtocolColumn(\n self._col.take_2d_labels_or_positional(\n row_positions=range(cum_row_lengths[i], cum_row_lengths[i + 1]),\n col_positions=None,\n ),\n allow_copy=self._col._allow_copy,\n )\n return\n\n if n_chunks % cur_n_chunks != 0:\n raise RuntimeError(\n \"The passed `n_chunks` must be a multiple of `self.num_chunks()`.\"\n )\n\n if n_chunks > n_rows:\n raise RuntimeError(\n \"The passed `n_chunks` value is bigger than `self.num_rows()`.\"\n )\n\n chunksize = n_rows // n_chunks\n new_lengths = [chunksize] * n_chunks\n new_lengths[-1] = n_rows % n_chunks + new_lengths[-1]\n\n new_partitions = self._col._partition_mgr_cls.map_axis_partitions(\n 0,\n self._col._partitions,\n lambda df: df,\n keep_partitioning=False,\n lengths=new_lengths,\n )\n new_df = self._col.__constructor__(\n new_partitions,\n self._col.index,\n self._col.columns,\n new_lengths,\n self._col.column_widths,\n )\n cum_row_lengths = np.cumsum([0] + new_df.row_lengths)\n for i in range(len(cum_row_lengths) - 1):\n yield PandasProtocolColumn(\n new_df.take_2d_labels_or_positional(\n row_positions=range(cum_row_lengths[i], cum_row_lengths[i + 1]),\n col_positions=None,\n ),\n allow_copy=self._allow_copy,\n )\n\n def get_buffers(self) -> Dict[str, Any]:\n buffers = {}\n buffers[\"data\"] = self._get_data_buffer()\n try:\n buffers[\"validity\"] = self._get_validity_buffer()\n except NoValidityBuffer:\n buffers[\"validity\"] = None\n\n try:\n buffers[\"offsets\"] = self._get_offsets_buffer()\n except NoOffsetsBuffer:\n buffers[\"offsets\"] = None\n\n return buffers\n\n _data_buffer_cache = None\n\n def _get_data_buffer(\n self,\n ) -> Tuple[PandasProtocolBuffer, Any]: # Any is for self.dtype tuple\n \"\"\"\n Return the buffer containing the data and the buffer's associated dtype.\n\n Returns\n -------\n tuple\n The data buffer.\n \"\"\"\n if self._data_buffer_cache is not None:\n return self._data_buffer_cache\n\n dtype = self.dtype\n if dtype[0] in (\n DTypeKind.INT,\n DTypeKind.UINT,\n DTypeKind.FLOAT,\n DTypeKind.BOOL,\n DTypeKind.DATETIME,\n ):\n buffer = PandasProtocolBuffer(\n self._col.to_numpy().flatten(), allow_copy=self._allow_copy\n )\n elif dtype[0] == DTypeKind.CATEGORICAL:\n pandas_series = self._col.to_pandas().squeeze(axis=1)\n codes = pandas_series.values.codes\n buffer = PandasProtocolBuffer(codes, allow_copy=self._allow_copy)\n dtype = self._dtype_from_primitive_pandas_dtype(codes.dtype)\n elif dtype[0] == DTypeKind.STRING:\n # Marshal the strings from a NumPy object array into a byte array\n buf = self._col.to_numpy().flatten()\n b = bytearray()\n\n # TODO: this for-loop is slow; can be implemented in Cython/C/C++ later\n for i in range(buf.size):\n if type(buf[i]) is str:\n b.extend(buf[i].encode(encoding=\"utf-8\"))\n\n # Convert the byte array to a pandas \"buffer\" using a NumPy array as the backing store\n buffer = PandasProtocolBuffer(np.frombuffer(b, dtype=\"uint8\"))\n\n # Define the dtype for the returned buffer\n dtype = (\n DTypeKind.STRING,\n 8,\n \"u\",\n \"=\",\n ) # note: currently only support native endianness\n else:\n raise NotImplementedError(f\"Data type {self._col.dtype[0]} not handled yet\")\n\n self._data_buffer_cache = (buffer, dtype)\n return self._data_buffer_cache\n\n _validity_buffer_cache = None\n\n def _get_validity_buffer(self) -> Tuple[PandasProtocolBuffer, Any]:\n \"\"\"\n Get the validity buffer.\n\n The buffer contains the mask values indicating\n missing data and the buffer's associated dtype.\n\n Returns\n -------\n tuple\n The validity buffer.\n\n Raises\n ------\n ``NoValidityBuffer`` if null representation is not a bit or byte mask.\n \"\"\"\n if self._validity_buffer_cache is not None:\n return self._validity_buffer_cache\n\n null, invalid = self.describe_null\n\n if self.dtype[0] == DTypeKind.STRING:\n # For now, have the mask array be comprised of bytes, rather than a bit array\n buf = self._col.to_numpy().flatten()\n\n # Determine the encoding for valid values\n valid = invalid == 0\n invalid = not valid\n\n mask = np.empty(shape=(len(buf),), dtype=np.bool_)\n for i, obj in enumerate(buf):\n mask[i] = valid if isinstance(obj, str) else invalid\n\n # Convert the mask array to a Pandas \"buffer\" using a NumPy array as the backing store\n buffer = PandasProtocolBuffer(mask)\n\n # Define the dtype of the returned buffer\n dtype = (DTypeKind.BOOL, 8, \"b\", \"=\")\n\n self._validity_buffer_cache = (buffer, dtype)\n return self._validity_buffer_cache\n\n try:\n msg = _NO_VALIDITY_BUFFER[null]\n except KeyError:\n raise NotImplementedError(\"See self.describe_null\")\n\n raise NoValidityBuffer(msg)\n\n _offsets_buffer_cache = None\n\n def _get_offsets_buffer(self) -> Tuple[PandasProtocolBuffer, Any]:\n \"\"\"\n Get the offsets buffer.\n\n The buffer contains the offset values for variable-size binary data\n (e.g., variable-length strings) and the buffer's associated dtype.\n\n Returns\n -------\n tuple\n The offsets buffer.\n\n Raises\n ------\n ``NoOffsetsBuffer`` if the data buffer does not have an associated offsets buffer.\n \"\"\"\n if self._offsets_buffer_cache is not None:\n return self._offsets_buffer_cache\n\n if self.dtype[0] == DTypeKind.STRING:\n # For each string, we need to manually determine the next offset\n values = self._col.to_numpy().flatten()\n ptr = 0\n offsets = [ptr] + [None] * len(values)\n for i, v in enumerate(values):\n # For missing values (in this case, `np.nan` values), we don't increment the pointer)\n if type(v) is str:\n b = v.encode(encoding=\"utf-8\")\n ptr += len(b)\n\n offsets[i + 1] = ptr\n\n # Convert the list of offsets to a NumPy array of signed 64-bit integers (note: Arrow allows the offsets array to be either `int32` or `int64`; here, we default to the latter)\n buf = np.asarray(offsets, dtype=\"int64\")\n\n # Convert the offsets to a Pandas \"buffer\" using the NumPy array as the backing store\n buffer = PandasProtocolBuffer(buf)\n\n # Assemble the buffer dtype info\n dtype = (\n DTypeKind.INT,\n 64,\n \"l\",\n \"=\",\n ) # note: currently only support native endianness\n else:\n raise NoOffsetsBuffer(\n \"This column has a fixed-length dtype so does not have an offsets buffer\"\n )\n\n self._offsets_buffer_cache = (buffer, dtype)\n return self._offsets_buffer_cache\n"
},
{
"path": "modin/core/dataframe/pandas/interchange/dataframe_protocol/dataframe.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nDataframe exchange protocol implementation.\n\nSee more in https://data-apis.org/dataframe-protocol/latest/index.html.\n\nNotes\n-----\n- Interpreting a raw pointer (as in ``Buffer.ptr``) is annoying and unsafe to\n do in pure Python. It's more general but definitely less friendly than having\n ``to_arrow`` and ``to_numpy`` methods. So for the buffers which lack\n ``__dlpack__`` (e.g., because the column dtype isn't supported by DLPack),\n this is worth looking at again.\n\"\"\"\n\nimport collections\nfrom typing import Any, Dict, Iterable, Optional, Sequence\n\nimport numpy as np\n\nfrom modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (\n ProtocolDataframe,\n)\nfrom modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe\nfrom modin.utils import _inherit_docstrings\n\nfrom .column import PandasProtocolColumn\n\n\n@_inherit_docstrings(ProtocolDataframe)\nclass PandasProtocolDataframe(ProtocolDataframe):\n \"\"\"\n A data frame class, with only the methods required by the interchange protocol defined.\n\n Instances of this (private) class are returned from ``modin.pandas.DataFrame.__dataframe__``\n as objects with the methods and attributes defined on this class.\n\n A \"data frame\" represents an ordered collection of named columns.\n A column's \"name\" must be a unique string. Columns may be accessed by name or by position.\n This could be a public data frame class, or an object with the methods and\n attributes defined on this DataFrame class could be returned from the\n ``__dataframe__`` method of a public data frame class in a library adhering\n to the dataframe interchange protocol specification.\n\n Parameters\n ----------\n df : PandasDataframe\n A ``PandasDataframe`` object.\n nan_as_null : bool, default:False\n A keyword intended for the consumer to tell the producer\n to overwrite null values in the data with ``NaN`` (or ``NaT``).\n This currently has no effect; once support for nullable extension\n dtypes is added, this value should be propagated to columns.\n allow_copy : bool, default: True\n A keyword that defines whether or not the library is allowed\n to make a copy of the data. For example, copying data would be necessary\n if a library supports strided buffers, given that this protocol\n specifies contiguous buffers. Currently, if the flag is set to ``False``\n and a copy is needed, a ``RuntimeError`` will be raised.\n \"\"\"\n\n def __init__(\n self,\n df: PandasDataframe,\n nan_as_null: bool = False,\n allow_copy: bool = True,\n ) -> None:\n self._df = df\n self._nan_as_null = nan_as_null\n self._allow_copy = allow_copy\n\n def __dataframe__(self, nan_as_null: bool = False, allow_copy: bool = True):\n return PandasProtocolDataframe(\n self._df, nan_as_null=nan_as_null, allow_copy=allow_copy\n )\n\n @property\n def metadata(self) -> Dict[str, Any]:\n return {\"modin.index\": self._df.index}\n\n def num_columns(self) -> int:\n return len(self._df.columns)\n\n def num_rows(self) -> int:\n return len(self._df.index)\n\n def num_chunks(self) -> int:\n return self._df._partitions.shape[0]\n\n def column_names(self) -> Iterable[str]:\n for col in self._df.columns:\n yield col\n\n def get_column(self, i: int) -> PandasProtocolColumn:\n return PandasProtocolColumn(\n self._df.take_2d_labels_or_positional(\n row_positions=None, col_positions=[i]\n ),\n allow_copy=self._allow_copy,\n )\n\n def get_column_by_name(self, name: str) -> PandasProtocolColumn:\n return PandasProtocolColumn(\n self._df.take_2d_labels_or_positional(\n row_positions=None, col_labels=[name]\n ),\n allow_copy=self._allow_copy,\n )\n\n def get_columns(self) -> Iterable[PandasProtocolColumn]:\n for name in self._df.columns:\n yield PandasProtocolColumn(\n self._df.take_2d_labels_or_positional(\n row_positions=None, col_labels=[name]\n ),\n allow_copy=self._allow_copy,\n )\n\n def select_columns(self, indices: Sequence[int]) -> \"PandasProtocolDataframe\":\n if not isinstance(indices, collections.abc.Sequence):\n raise ValueError(\"`indices` is not a sequence\")\n\n return PandasProtocolDataframe(\n self._df.take_2d_labels_or_positional(\n row_positions=None, col_positions=indices\n ),\n allow_copy=self._allow_copy,\n )\n\n def select_columns_by_name(self, names: Sequence[str]) -> \"PandasProtocolDataframe\":\n if not isinstance(names, collections.abc.Sequence):\n raise ValueError(\"`names` is not a sequence\")\n\n return PandasProtocolDataframe(\n self._df.take_2d_labels_or_positional(row_positions=None, col_labels=names),\n allow_copy=self._allow_copy,\n )\n\n def get_chunks(\n self, n_chunks: Optional[int] = None\n ) -> Iterable[\"PandasProtocolDataframe\"]:\n cur_n_chunks = self.num_chunks()\n n_rows = self.num_rows()\n if n_chunks is None or n_chunks == cur_n_chunks:\n cum_row_lengths = np.cumsum([0] + self._df.row_lengths)\n for i in range(len(cum_row_lengths) - 1):\n yield PandasProtocolDataframe(\n self._df.take_2d_labels_or_positional(\n row_positions=range(cum_row_lengths[i], cum_row_lengths[i + 1]),\n col_positions=None,\n ),\n allow_copy=self._allow_copy,\n )\n return\n if n_chunks % cur_n_chunks != 0:\n raise RuntimeError(\n \"The passed `n_chunks` must be a multiple of `self.num_chunks()`.\"\n )\n\n if n_chunks > n_rows:\n raise RuntimeError(\n \"The passed `n_chunks` value is bigger than `self.num_rows()`.\"\n )\n\n chunksize = n_rows // n_chunks\n new_lengths = [chunksize] * n_chunks\n new_lengths[-1] = n_rows % n_chunks + new_lengths[-1]\n\n new_partitions = self._df._partition_mgr_cls.map_axis_partitions(\n 0,\n self._df._partitions,\n lambda df: df,\n keep_partitioning=False,\n lengths=new_lengths,\n )\n new_df = self._df.__constructor__(\n new_partitions,\n self._df.index,\n self._df.columns,\n new_lengths,\n self._df.column_widths,\n )\n cum_row_lengths = np.cumsum([0] + new_df.row_lengths)\n for i in range(len(cum_row_lengths) - 1):\n yield PandasProtocolDataframe(\n new_df.take_2d_labels_or_positional(\n row_positions=range(cum_row_lengths[i], cum_row_lengths[i + 1]),\n col_positions=None,\n ),\n allow_copy=self._allow_copy,\n )\n"
},
{
"path": "modin/core/dataframe/pandas/interchange/dataframe_protocol/exception.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Exceptions that can be caught by dataframe exchange protocol implementation for pandas storage format.\"\"\"\n\n\nclass NoValidityBuffer(Exception):\n \"\"\"Exception to be raised if there is no validity buffer for ``PandasProtocolColumn``.\"\"\"\n\n pass\n\n\nclass NoOffsetsBuffer(Exception):\n \"\"\"Exception to be raised if there is no offsets buffer for ``PandasProtocolColumn``.\"\"\"\n\n pass\n"
},
{
"path": "modin/core/dataframe/pandas/interchange/dataframe_protocol/from_dataframe.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses the functions building a ``pandas.DataFrame`` from a DataFrame exchange protocol object.\"\"\"\n\nimport ctypes\nimport re\nfrom typing import Any, Optional, Tuple, Union\n\nimport numpy as np\nimport pandas\n\nfrom modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (\n ProtocolBuffer,\n ProtocolColumn,\n ProtocolDataframe,\n)\nfrom modin.core.dataframe.base.interchange.dataframe_protocol.utils import (\n ArrowCTypes,\n ColumnNullType,\n DTypeKind,\n Endianness,\n)\n\nnp_types_map = {\n DTypeKind.INT: {8: np.int8, 16: np.int16, 32: np.int32, 64: np.int64},\n DTypeKind.UINT: {8: np.uint8, 16: np.uint16, 32: np.uint32, 64: np.uint64},\n DTypeKind.FLOAT: {32: np.float32, 64: np.float64},\n # Consider bitmask to be a uint8 dtype to parse the bits later\n DTypeKind.BOOL: {1: np.uint8, 8: bool},\n}\n\n\ndef from_dataframe_to_pandas(df: ProtocolDataframe, n_chunks: Optional[int] = None):\n \"\"\"\n Build a ``pandas.DataFrame`` from an object supporting the DataFrame exchange protocol, i.e. `__dataframe__` method.\n\n Parameters\n ----------\n df : ProtocolDataframe\n Object supporting the exchange protocol, i.e. `__dataframe__` method.\n n_chunks : int, optional\n Number of chunks to split `df`.\n\n Returns\n -------\n pandas.DataFrame\n \"\"\"\n if not hasattr(df, \"__dataframe__\"):\n raise ValueError(\"`df` does not support __dataframe__\")\n\n df = df.__dataframe__()\n if isinstance(df, dict):\n df = df[\"dataframe\"]\n\n pandas_dfs = []\n for chunk in df.get_chunks(n_chunks):\n pandas_df = protocol_df_chunk_to_pandas(chunk)\n pandas_dfs.append(pandas_df)\n\n pandas_df = pandas.concat(pandas_dfs, axis=0, ignore_index=True)\n\n index_obj = df.metadata.get(\"modin.index\", df.metadata.get(\"pandas.index\", None))\n if index_obj is not None:\n pandas_df.index = index_obj\n\n return pandas_df\n\n\ndef protocol_df_chunk_to_pandas(df):\n \"\"\"\n Convert exchange protocol chunk to ``pandas.DataFrame``.\n\n Parameters\n ----------\n df : ProtocolDataframe\n\n Returns\n -------\n pandas.DataFrame\n \"\"\"\n # We need a dict of columns here, with each column being a NumPy array (at\n # least for now, deal with non-NumPy dtypes later).\n columns = dict()\n buffers = [] # hold on to buffers, keeps memory alive\n for name in df.column_names():\n if not isinstance(name, str):\n raise ValueError(f\"Column {name} is not a string\")\n if name in columns:\n raise ValueError(f\"Column {name} is not unique\")\n col = df.get_column_by_name(name)\n columns[name], buf = unpack_protocol_column(col)\n buffers.append(buf)\n\n pandas_df = pandas.DataFrame(columns)\n pandas_df._buffers = buffers\n return pandas_df\n\n\ndef unpack_protocol_column(\n col: ProtocolColumn,\n) -> Tuple[Union[np.ndarray, pandas.Series], Any]:\n \"\"\"\n Unpack an interchange protocol column to a pandas-ready column.\n\n Parameters\n ----------\n col : ProtocolColumn\n Column to unpack.\n\n Returns\n -------\n tuple\n Tuple of resulting column (either an ndarray or a series) and the object\n which keeps memory referenced by the column alive.\n \"\"\"\n dtype = col.dtype[0]\n if dtype in (\n DTypeKind.INT,\n DTypeKind.UINT,\n DTypeKind.FLOAT,\n DTypeKind.BOOL,\n ):\n return primitive_column_to_ndarray(col)\n elif dtype == DTypeKind.CATEGORICAL:\n return categorical_column_to_series(col)\n elif dtype == DTypeKind.STRING:\n return string_column_to_ndarray(col)\n elif dtype == DTypeKind.DATETIME:\n return datetime_column_to_ndarray(col)\n else:\n raise NotImplementedError(f\"Data type {dtype} not handled yet\")\n\n\ndef primitive_column_to_ndarray(col: ProtocolColumn) -> Tuple[np.ndarray, Any]:\n \"\"\"\n Convert a column holding one of the primitive dtypes (int, uint, float or bool) to a NumPy array.\n\n Parameters\n ----------\n col : ProtocolColumn\n\n Returns\n -------\n tuple\n Tuple of np.ndarray holding the data and the memory owner object that keeps the memory alive.\n \"\"\"\n buffers = col.get_buffers()\n\n data_buff, data_dtype = buffers[\"data\"]\n data = buffer_to_ndarray(data_buff, data_dtype, col.offset, col.size())\n\n data = set_nulls(data, col, buffers[\"validity\"])\n return data, buffers\n\n\ndef categorical_column_to_series(col: ProtocolColumn) -> Tuple[pandas.Series, Any]:\n \"\"\"\n Convert a column holding categorical data to a pandas Series.\n\n Parameters\n ----------\n col : ProtocolColumn\n\n Returns\n -------\n tuple\n Tuple of pandas.Series holding the data and the memory owner object that keeps the memory alive.\n \"\"\"\n cat_descr = col.describe_categorical\n ordered, is_dict, categories = (\n cat_descr[\"is_ordered\"],\n cat_descr[\"is_dictionary\"],\n cat_descr[\"categories\"],\n )\n\n if not is_dict or categories is None:\n raise NotImplementedError(\"Non-dictionary categoricals not supported yet\")\n\n buffers = col.get_buffers()\n\n codes_buff, codes_dtype = buffers[\"data\"]\n codes = buffer_to_ndarray(codes_buff, codes_dtype, col.offset, col.size())\n\n # Doing module in order to not get ``IndexError`` for out-of-bounds sentinel values in `codes`\n cat_values, categories_buf = unpack_protocol_column(categories)\n values = cat_values[codes % len(cat_values)]\n\n cat = pandas.Categorical(values, categories=cat_values, ordered=ordered)\n data = pandas.Series(cat)\n\n data = set_nulls(data, col, buffers[\"validity\"])\n return data, [buffers, categories_buf]\n\n\ndef _inverse_null_buf(buf: np.ndarray, null_kind: ColumnNullType) -> np.ndarray:\n \"\"\"\n Inverse the boolean value of buffer storing either bit- or bytemask.\n\n Parameters\n ----------\n buf : np.ndarray\n Buffer to inverse the boolean value for.\n null_kind : {ColumnNullType.USE_BYTEMASK, ColumnNullType.USE_BITMASK}\n How to treat the buffer.\n\n Returns\n -------\n np.ndarray\n Logically inversed buffer.\n \"\"\"\n if null_kind == ColumnNullType.USE_BITMASK:\n return ~buf\n assert (\n null_kind == ColumnNullType.USE_BYTEMASK\n ), f\"Unexpected null kind: {null_kind}\"\n # bytemasks use 0 for `False` and anything else for `True`, so convert to bool\n # by direct comparison instead of bitwise reversal like we do for bitmasks\n return buf == 0\n\n\ndef string_column_to_ndarray(col: ProtocolColumn) -> Tuple[np.ndarray, Any]:\n \"\"\"\n Convert a column holding string data to a NumPy array.\n\n Parameters\n ----------\n col : ProtocolColumn\n\n Returns\n -------\n tuple\n Tuple of np.ndarray holding the data and the memory owner object that keeps the memory alive.\n \"\"\"\n null_kind, sentinel_val = col.describe_null\n\n if null_kind not in (\n ColumnNullType.NON_NULLABLE,\n ColumnNullType.USE_BITMASK,\n ColumnNullType.USE_BYTEMASK,\n ):\n raise NotImplementedError(\n f\"{null_kind} null kind is not yet supported for string columns.\"\n )\n\n buffers = col.get_buffers()\n\n # Retrieve the data buffer containing the UTF-8 code units\n data_buff, protocol_data_dtype = buffers[\"data\"]\n # We're going to reinterpret the buffer as uint8, so making sure we can do it safely\n assert protocol_data_dtype[1] == 8 # bitwidth == 8\n assert protocol_data_dtype[2] == ArrowCTypes.STRING # format_str == utf-8\n # Convert the buffers to NumPy arrays, in order to go from STRING to an equivalent ndarray,\n # we claim that the buffer is uint8 (i.e., a byte array)\n data_dtype = (\n DTypeKind.UINT,\n 8,\n ArrowCTypes.UINT8,\n Endianness.NATIVE,\n )\n # Specify zero offset as we don't want to chunk the string data\n data = buffer_to_ndarray(data_buff, data_dtype, offset=0, length=col.size())\n\n # Retrieve the offsets buffer containing the index offsets demarcating the beginning and end of each string\n offset_buff, offset_dtype = buffers[\"offsets\"]\n # Offsets buffer contains start-stop positions of strings in the data buffer,\n # meaning that it has more elements than in the data buffer, do `col.size() + 1` here\n # to pass a proper offsets buffer size\n offsets = buffer_to_ndarray(\n offset_buff, offset_dtype, col.offset, length=col.size() + 1\n )\n\n null_pos = None\n if null_kind in (ColumnNullType.USE_BITMASK, ColumnNullType.USE_BYTEMASK):\n valid_buff, valid_dtype = buffers[\"validity\"]\n null_pos = buffer_to_ndarray(valid_buff, valid_dtype, col.offset, col.size())\n if sentinel_val == 0:\n null_pos = _inverse_null_buf(null_pos, null_kind)\n\n # Assemble the strings from the code units\n str_list = [None] * col.size()\n for i in range(col.size()):\n # Check for missing values\n if null_pos is not None and null_pos[i]:\n str_list[i] = np.nan\n continue\n\n # Extract a range of code units\n units = data[offsets[i] : offsets[i + 1]]\n\n # Convert the list of code units to bytes\n str_bytes = bytes(units)\n\n # Create the string\n string = str_bytes.decode(encoding=\"utf-8\")\n\n # Add to our list of strings\n str_list[i] = string\n\n # Convert the string list to a NumPy array\n return np.asarray(str_list, dtype=\"object\"), buffers\n\n\ndef datetime_column_to_ndarray(col: ProtocolColumn) -> Tuple[np.ndarray, Any]:\n \"\"\"\n Convert a column holding DateTime data to a NumPy array.\n\n Parameters\n ----------\n col : ProtocolColumn\n\n Returns\n -------\n tuple\n Tuple of np.ndarray holding the data and the memory owner object that keeps the memory alive.\n \"\"\"\n buffers = col.get_buffers()\n\n _, _, format_str, _ = col.dtype\n dbuf, dtype = buffers[\"data\"]\n # Consider dtype being `uint` to get number of units passed since the 01.01.1970\n data = buffer_to_ndarray(\n dbuf,\n (\n DTypeKind.UINT,\n dtype[1],\n getattr(ArrowCTypes, f\"UINT{dtype[1]}\"),\n Endianness.NATIVE,\n ),\n col.offset,\n col.size(),\n )\n\n def parse_format_str(format_str, data):\n \"\"\"Parse datetime `format_str` to interpret the `data`.\"\"\"\n # timestamp 'ts{unit}:tz'\n timestamp_meta = re.match(r\"ts([smun]):(.*)\", format_str)\n if timestamp_meta:\n unit, tz = timestamp_meta.group(1), timestamp_meta.group(2)\n if tz != \"\":\n raise NotImplementedError(\"Timezones are not supported yet\")\n if unit != \"s\":\n # the format string describes only a first letter of the unit, add one extra\n # letter to make the unit in numpy-style: 'm' -> 'ms', 'u' -> 'us', 'n' -> 'ns'\n unit += \"s\"\n data = data.astype(f\"datetime64[{unit}]\")\n return data\n\n # date 'td{Days/Ms}'\n date_meta = re.match(r\"td([Dm])\", format_str)\n if date_meta:\n unit = date_meta.group(1)\n if unit == \"D\":\n # NumPy doesn't support DAY unit, so converting days to seconds\n # (converting to uint64 to avoid overflow)\n data = (data.astype(np.uint64) * (24 * 60 * 60)).astype(\"datetime64[s]\")\n elif unit == \"m\":\n data = data.astype(\"datetime64[ms]\")\n else:\n raise NotImplementedError(f\"Date unit is not supported: {unit}\")\n return data\n\n raise NotImplementedError(f\"DateTime kind is not supported: {format_str}\")\n\n data = parse_format_str(format_str, data)\n data = set_nulls(data, col, buffers[\"validity\"])\n return data, buffers\n\n\ndef buffer_to_ndarray(\n buffer: ProtocolBuffer,\n dtype: Tuple[DTypeKind, int, str, str],\n offset: int = 0,\n length: Optional[int] = None,\n) -> np.ndarray:\n \"\"\"\n Build a NumPy array from the passed buffer.\n\n Parameters\n ----------\n buffer : ProtocolBuffer\n Buffer to build a NumPy array from.\n dtype : tuple\n Data type of the buffer conforming protocol dtypes format.\n offset : int, default: 0\n Number of elements to offset from the start of the buffer.\n length : int, optional\n If the buffer is a bit-mask, specifies a number of bits to read\n from the buffer. Has no effect otherwise.\n\n Returns\n -------\n np.ndarray\n\n Notes\n -----\n The returned array doesn't own the memory. A user of the function must keep the memory\n owner object alive as long as the returned NumPy array is being used.\n \"\"\"\n kind, bit_width, _, _ = dtype\n\n column_dtype = np_types_map.get(kind, {}).get(bit_width, None)\n if column_dtype is None:\n raise NotImplementedError(f\"Convertion for {dtype} is not yet supported.\")\n\n # TODO: No DLPack yet, so need to construct a new ndarray from the data pointer\n # and size in the buffer plus the dtype on the column. Use DLPack as NumPy supports\n # it since https://github.com/numpy/numpy/pull/19083\n ctypes_type = np.ctypeslib.as_ctypes_type(column_dtype)\n data_pointer = ctypes.cast(\n buffer.ptr + (offset * bit_width // 8), ctypes.POINTER(ctypes_type)\n )\n\n if bit_width == 1:\n assert length is not None, \"`length` must be specified for a bit-mask buffer.\"\n arr = np.ctypeslib.as_array(data_pointer, shape=(buffer.bufsize,))\n return bitmask_to_bool_ndarray(arr, length, first_byte_offset=offset % 8)\n else:\n return np.ctypeslib.as_array(\n data_pointer, shape=(buffer.bufsize // (bit_width // 8),)\n )\n\n\ndef bitmask_to_bool_ndarray(\n bitmask: np.ndarray, mask_length: int, first_byte_offset: int = 0\n) -> np.ndarray:\n \"\"\"\n Convert bit-mask to a boolean NumPy array.\n\n Parameters\n ----------\n bitmask : np.ndarray[uint8]\n NumPy array of uint8 dtype representing the bitmask.\n mask_length : int\n Number of elements in the mask to interpret.\n first_byte_offset : int, default: 0\n Number of elements to offset from the start of the first byte.\n\n Returns\n -------\n np.ndarray[bool]\n \"\"\"\n bytes_to_skip = first_byte_offset // 8\n bitmask = bitmask[bytes_to_skip:]\n first_byte_offset %= 8\n\n bool_mask = np.zeros(mask_length, dtype=bool)\n\n # Proccessing the first byte separately as it has its own offset\n val = bitmask[0]\n mask_idx = 0\n bits_in_first_byte = min(8 - first_byte_offset, mask_length)\n for j in range(bits_in_first_byte):\n if val & (1 << (j + first_byte_offset)):\n bool_mask[mask_idx] = True\n mask_idx += 1\n\n # `mask_length // 8` describes how many full bytes to process\n for i in range((mask_length - bits_in_first_byte) // 8):\n # doing `+ 1` as we already processed the first byte\n val = bitmask[i + 1]\n for j in range(8):\n if val & (1 << j):\n bool_mask[mask_idx] = True\n mask_idx += 1\n\n if len(bitmask) > 1:\n # Processing reminder of last byte\n val = bitmask[-1]\n for j in range(len(bool_mask) - mask_idx):\n if val & (1 << j):\n bool_mask[mask_idx] = True\n mask_idx += 1\n\n return bool_mask\n\n\ndef set_nulls(\n data: Union[np.ndarray, pandas.Series],\n col: ProtocolColumn,\n validity: Tuple[ProtocolBuffer, Tuple[DTypeKind, int, str, str]],\n allow_modify_inplace: bool = True,\n):\n \"\"\"\n Set null values for the data according to the column null kind.\n\n Parameters\n ----------\n data : np.ndarray or pandas.Series\n Data to set nulls in.\n col : ProtocolColumn\n Column object that describes the `data`.\n validity : tuple(ProtocolBuffer, dtype) or None\n The return value of ``col.buffers()``. We do not access the ``col.buffers()``\n here to not take the ownership of the memory of buffer objects.\n allow_modify_inplace : bool, default: True\n Whether to modify the `data` inplace when zero-copy is possible (True) or always\n modify a copy of the `data` (False).\n\n Returns\n -------\n np.ndarray or pandas.Series\n Data with the nulls being set.\n \"\"\"\n null_kind, sentinel_val = col.describe_null\n null_pos = None\n\n if null_kind == ColumnNullType.USE_SENTINEL:\n null_pos = data == sentinel_val\n elif null_kind in (ColumnNullType.USE_BITMASK, ColumnNullType.USE_BYTEMASK):\n valid_buff, valid_dtype = validity\n null_pos = buffer_to_ndarray(valid_buff, valid_dtype, col.offset, col.size())\n if sentinel_val == 0:\n null_pos = _inverse_null_buf(null_pos, null_kind)\n elif null_kind in (ColumnNullType.NON_NULLABLE, ColumnNullType.USE_NAN):\n pass\n else:\n raise NotImplementedError(f\"Null kind {null_kind} is not yet supported.\")\n\n if null_pos is not None and np.any(null_pos):\n if not allow_modify_inplace:\n data = data.copy()\n try:\n data[null_pos] = None\n except TypeError:\n # TypeError happens if the `data` dtype appears to be non-nullable in numpy notation\n # (bool, int, uint), if such happens, cast the `data` to nullable float dtype.\n data = data.astype(float)\n data[null_pos] = None\n\n return data\n"
},
{
"path": "modin/core/dataframe/pandas/metadata/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Utilities and classes to handle work with metadata.\"\"\"\n\nfrom .dtypes import (\n DtypesDescriptor,\n LazyProxyCategoricalDtype,\n ModinDtypes,\n extract_dtype,\n)\nfrom .index import ModinIndex\n\n__all__ = [\n \"ModinDtypes\",\n \"ModinIndex\",\n \"LazyProxyCategoricalDtype\",\n \"DtypesDescriptor\",\n \"extract_dtype\",\n]\n"
},
{
"path": "modin/core/dataframe/pandas/metadata/dtypes.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module contains class ``ModinDtypes``.\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import TYPE_CHECKING, Callable, Optional, Union\n\nimport pandas\nfrom pandas._typing import DtypeObj, IndexLabel\nfrom pandas.core.dtypes.cast import find_common_type\n\nif TYPE_CHECKING:\n from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe\n from .index import ModinIndex\n\nfrom modin.error_message import ErrorMessage\n\n\nclass DtypesDescriptor:\n \"\"\"\n Describes partial dtypes.\n\n Parameters\n ----------\n known_dtypes : dict[IndexLabel, DtypeObj] or pandas.Series, optional\n Columns that we know dtypes for.\n cols_with_unknown_dtypes : list[IndexLabel], optional\n Column names that have unknown dtypes. If specified together with `remaining_dtype`, must describe all\n columns with unknown dtypes, otherwise, the missing columns will be assigned to `remaining_dtype`.\n If `cols_with_unknown_dtypes` is incomplete, you must specify `know_all_names=False`.\n remaining_dtype : DtypeObj, optional\n Dtype for columns that are not present neither in `known_dtypes` nor in `cols_with_unknown_dtypes`.\n This parameter is intended to describe columns that we known dtypes for, but don't know their\n names yet. Note, that this parameter DOESN'T describe dtypes for columns from `cols_with_unknown_dtypes`.\n parent_df : PandasDataframe, optional\n Dataframe object for which we describe dtypes. This dataframe will be used to compute\n missing dtypes on ``.materialize()``.\n columns_order : dict[int, IndexLabel], optional\n Order of columns in the dataframe. If specified, must describe all the columns of the dataframe.\n know_all_names : bool, default: True\n Whether `known_dtypes` and `cols_with_unknown_dtypes` contain all column names for this dataframe besides those,\n that are being described by `remaining_dtype`.\n One can't pass `know_all_names=False` together with `remaining_dtype` as this creates ambiguity\n on how to interpret missing columns (whether they belong to `remaining_dtype` or not).\n _schema_is_known : bool, optional\n Whether `known_dtypes` describe all columns in the dataframe. This parameter intended mostly\n for internal use.\n \"\"\"\n\n def __init__(\n self,\n known_dtypes: Optional[Union[dict[IndexLabel, DtypeObj], pandas.Series]] = None,\n cols_with_unknown_dtypes: Optional[list[IndexLabel]] = None,\n remaining_dtype: Optional[DtypeObj] = None,\n parent_df: Optional[PandasDataframe] = None,\n columns_order: Optional[dict[int, IndexLabel]] = None,\n know_all_names: bool = True,\n _schema_is_known: Optional[bool] = None,\n ):\n if not know_all_names and remaining_dtype is not None:\n raise ValueError(\n \"It's not allowed to pass 'remaining_dtype' and 'know_all_names=False' at the same time.\"\n )\n # columns with known dtypes\n self._known_dtypes: dict[IndexLabel, DtypeObj] = (\n {} if known_dtypes is None else dict(known_dtypes)\n )\n if known_dtypes is not None and len(self._known_dtypes) != len(known_dtypes):\n raise NotImplementedError(\n \"Duplicated column names are not yet supported by DtypesDescriptor\"\n )\n # columns with unknown dtypes (they're not described by 'remaining_dtype')\n if cols_with_unknown_dtypes is not None and len(\n set(cols_with_unknown_dtypes)\n ) != len(cols_with_unknown_dtypes):\n raise NotImplementedError(\n \"Duplicated column names are not yet supported by DtypesDescriptor\"\n )\n self._cols_with_unknown_dtypes: list[IndexLabel] = (\n [] if cols_with_unknown_dtypes is None else cols_with_unknown_dtypes\n )\n # whether 'known_dtypes' describe all columns in the dataframe\n self._schema_is_known: Optional[bool] = _schema_is_known\n if self._schema_is_known is None:\n self._schema_is_known = False\n if (\n # if 'cols_with_unknown_dtypes' was explicitly specified as an empty list and\n # we don't have any 'remaining_dtype', then we assume that 'known_dtypes' are complete\n cols_with_unknown_dtypes is not None\n and know_all_names\n and remaining_dtype is None\n and len(self._known_dtypes) > 0\n ):\n self._schema_is_known = len(cols_with_unknown_dtypes) == 0\n\n self._know_all_names: bool = know_all_names\n # a common dtype for columns that are not present in 'known_dtypes' nor in 'cols_with_unknown_dtypes'\n self._remaining_dtype: Optional[DtypeObj] = remaining_dtype\n self._parent_df: Optional[PandasDataframe] = parent_df\n if columns_order is None:\n self._columns_order: Optional[dict[int, IndexLabel]] = None\n # try to compute '._columns_order' using 'parent_df'\n self.columns_order\n else:\n if remaining_dtype is not None:\n raise ValueError(\n \"Passing 'columns_order' and 'remaining_dtype' is ambiguous. You have to manually \"\n + \"complete 'known_dtypes' using the information from 'columns_order' and 'remaining_dtype'.\"\n )\n elif not self._know_all_names:\n raise ValueError(\n \"Passing 'columns_order' and 'know_all_names=False' is ambiguous. You have to manually \"\n + \"complete 'cols_with_unknown_dtypes' using the information from 'columns_order' \"\n + \"and pass 'know_all_names=True'.\"\n )\n elif len(columns_order) != (\n len(self._cols_with_unknown_dtypes) + len(self._known_dtypes)\n ):\n raise ValueError(\n \"The length of 'columns_order' doesn't match to 'known_dtypes' and 'cols_with_unknown_dtypes'\"\n )\n self._columns_order: Optional[dict[int, IndexLabel]] = columns_order\n\n def update_parent(self, new_parent: PandasDataframe):\n \"\"\"\n Set new parent dataframe.\n\n Parameters\n ----------\n new_parent : PandasDataframe\n \"\"\"\n self._parent_df = new_parent\n LazyProxyCategoricalDtype.update_dtypes(self._known_dtypes, new_parent)\n # try to compute '._columns_order' using 'new_parent'\n self.columns_order\n\n @property\n def columns_order(self) -> Optional[dict[int, IndexLabel]]:\n \"\"\"\n Get order of columns for the described dataframe if available.\n\n Returns\n -------\n dict[int, IndexLabel] or None\n \"\"\"\n if self._columns_order is not None:\n return self._columns_order\n if self._parent_df is None or not self._parent_df.has_materialized_columns:\n return None\n\n actual_columns = self._parent_df.columns\n self._normalize_self_levels(actual_columns)\n\n self._columns_order = {i: col for i, col in enumerate(actual_columns)}\n # we got information about new columns and thus can potentially\n # extend our knowledge about missing dtypes\n if len(self._columns_order) > (\n len(self._known_dtypes) + len(self._cols_with_unknown_dtypes)\n ):\n new_cols = [\n col\n for col in self._columns_order.values()\n if col not in self._known_dtypes\n and col not in self._cols_with_unknown_dtypes\n ]\n if self._remaining_dtype is not None:\n self._known_dtypes.update(\n {col: self._remaining_dtype for col in new_cols}\n )\n self._remaining_dtype = None\n if len(self._cols_with_unknown_dtypes) == 0:\n self._schema_is_known = True\n else:\n self._cols_with_unknown_dtypes.extend(new_cols)\n self._know_all_names = True\n return self._columns_order\n\n def __repr__(self): # noqa: GL08\n return (\n f\"DtypesDescriptor:\\n\\tknown dtypes: {self._known_dtypes};\\n\\t\"\n + f\"remaining dtype: {self._remaining_dtype};\\n\\t\"\n + f\"cols with unknown dtypes: {self._cols_with_unknown_dtypes};\\n\\t\"\n + f\"schema is known: {self._schema_is_known};\\n\\t\"\n + f\"has parent df: {self._parent_df is not None};\\n\\t\"\n + f\"columns order: {self._columns_order};\\n\\t\"\n + f\"know all names: {self._know_all_names}\"\n )\n\n def __str__(self): # noqa: GL08\n return self.__repr__()\n\n def lazy_get(\n self, ids: list[Union[IndexLabel, int]], numeric_index: bool = False\n ) -> DtypesDescriptor:\n \"\"\"\n Get dtypes descriptor for a subset of columns without triggering any computations.\n\n Parameters\n ----------\n ids : list of index labels or positional indexers\n Columns for the subset.\n numeric_index : bool, default: False\n Whether `ids` are positional indixes or column labels to take.\n\n Returns\n -------\n DtypesDescriptor\n Descriptor that describes dtypes for columns specified in `ids`.\n \"\"\"\n if len(set(ids)) != len(ids):\n raise NotImplementedError(\n \"Duplicated column names are not yet supported by DtypesDescriptor\"\n )\n\n if numeric_index:\n if self.columns_order is not None:\n ids = [self.columns_order[i] for i in ids]\n else:\n raise ValueError(\n \"Can't lazily get columns by positional indixers if the columns order is unknown\"\n )\n\n result = {}\n unknown_cols = []\n columns_order = {}\n for i, col in enumerate(ids):\n columns_order[i] = col\n if col in self._cols_with_unknown_dtypes:\n unknown_cols.append(col)\n continue\n dtype = self._known_dtypes.get(col)\n if dtype is None and self._remaining_dtype is None:\n unknown_cols.append(col)\n elif dtype is None and self._remaining_dtype is not None:\n result[col] = self._remaining_dtype\n else:\n result[col] = dtype\n remaining_dtype = self._remaining_dtype if len(unknown_cols) != 0 else None\n return DtypesDescriptor(\n result,\n unknown_cols,\n remaining_dtype,\n self._parent_df,\n columns_order=columns_order,\n )\n\n def copy(self) -> DtypesDescriptor:\n \"\"\"\n Get a copy of this descriptor.\n\n Returns\n -------\n DtypesDescriptor\n \"\"\"\n return type(self)(\n # should access '.columns_order' first, as it may compute columns order\n # and complete the metadata for 'self'\n columns_order=(\n None if self.columns_order is None else self.columns_order.copy()\n ),\n known_dtypes=self._known_dtypes.copy(),\n cols_with_unknown_dtypes=self._cols_with_unknown_dtypes.copy(),\n remaining_dtype=self._remaining_dtype,\n parent_df=self._parent_df,\n know_all_names=self._know_all_names,\n _schema_is_known=self._schema_is_known,\n )\n\n def set_index(self, new_index: Union[pandas.Index, ModinIndex]) -> DtypesDescriptor:\n \"\"\"\n Set new column names for this descriptor.\n\n Parameters\n ----------\n new_index : pandas.Index or ModinIndex\n\n Returns\n -------\n DtypesDescriptor\n New descriptor with updated column names.\n\n Notes\n -----\n Calling this method on a descriptor that returns ``None`` for ``.columns_order``\n will result into information lose.\n \"\"\"\n if len(new_index) != len(set(new_index)):\n raise NotImplementedError(\n \"Duplicated column names are not yet supported by DtypesDescriptor\"\n )\n\n if self.columns_order is None:\n # we can't map new columns to old columns and lost all dtypes :(\n return DtypesDescriptor(\n cols_with_unknown_dtypes=new_index,\n columns_order={i: col for i, col in enumerate(new_index)},\n parent_df=self._parent_df,\n know_all_names=True,\n )\n\n new_self = self.copy()\n renamer = {old_c: new_index[i] for i, old_c in new_self.columns_order.items()}\n new_self._known_dtypes = {\n renamer[old_col]: value for old_col, value in new_self._known_dtypes.items()\n }\n new_self._cols_with_unknown_dtypes = [\n renamer[old_col] for old_col in new_self._cols_with_unknown_dtypes\n ]\n new_self._columns_order = {\n i: renamer[old_col] for i, old_col in new_self._columns_order.items()\n }\n return new_self\n\n def equals(self, other: DtypesDescriptor) -> bool:\n \"\"\"\n Compare two descriptors for equality.\n\n Parameters\n ----------\n other : DtypesDescriptor\n\n Returns\n -------\n bool\n \"\"\"\n return (\n self._known_dtypes == other._known_dtypes\n and set(self._cols_with_unknown_dtypes)\n == set(other._cols_with_unknown_dtypes)\n and self._remaining_dtype == other._remaining_dtype\n and self._schema_is_known == other._schema_is_known\n and self.columns_order == other.columns_order\n and self._know_all_names == other._know_all_names\n )\n\n @property\n def is_materialized(self) -> bool:\n \"\"\"\n Whether this descriptor contains information about all dtypes in the dataframe.\n\n Returns\n -------\n bool\n \"\"\"\n return self._schema_is_known\n\n def _materialize_all_names(self):\n \"\"\"Materialize missing column names.\"\"\"\n if self._know_all_names:\n return\n\n all_cols = self._parent_df.columns\n self._normalize_self_levels(all_cols)\n for col in all_cols:\n if (\n col not in self._known_dtypes\n and col not in self._cols_with_unknown_dtypes\n ):\n self._cols_with_unknown_dtypes.append(col)\n\n self._know_all_names = True\n\n def _materialize_cols_with_unknown_dtypes(self):\n \"\"\"Compute dtypes for cols specified in `._cols_with_unknown_dtypes`.\"\"\"\n if (\n len(self._known_dtypes) == 0\n and len(self._cols_with_unknown_dtypes) == 0\n and not self._know_all_names\n ):\n # here we have to compute dtypes for all columns in the dataframe,\n # so avoiding columns materialization by setting 'subset=None'\n subset = None\n else:\n if not self._know_all_names:\n self._materialize_all_names()\n subset = self._cols_with_unknown_dtypes\n\n if subset is None or len(subset) > 0:\n self._known_dtypes.update(self._parent_df._compute_dtypes(subset))\n\n self._know_all_names = True\n self._cols_with_unknown_dtypes = []\n\n def materialize(self):\n \"\"\"Complete information about dtypes.\"\"\"\n if self.is_materialized:\n return\n if self._parent_df is None:\n raise RuntimeError(\n \"It's not allowed to call '.materialize()' before '._parent_df' is specified.\"\n )\n\n self._materialize_cols_with_unknown_dtypes()\n\n if self._remaining_dtype is not None:\n cols = self._parent_df.columns\n self._normalize_self_levels(cols)\n self._known_dtypes.update(\n {\n col: self._remaining_dtype\n for col in cols\n if col not in self._known_dtypes\n }\n )\n\n # we currently not guarantee for dtypes to be in a proper order:\n # https://github.com/modin-project/modin/blob/8a332c1597c54d36f7ccbbd544e186b689f9ceb1/modin/pandas/test/utils.py#L644-L646\n # so restoring the order only if it's possible\n if self.columns_order is not None:\n assert len(self.columns_order) == len(self._known_dtypes)\n self._known_dtypes = {\n self.columns_order[i]: self._known_dtypes[self.columns_order[i]]\n for i in range(len(self.columns_order))\n }\n\n self._schema_is_known = True\n self._remaining_dtype = None\n self._parent_df = None\n\n def to_series(self) -> pandas.Series:\n \"\"\"\n Convert descriptor to a pandas Series.\n\n Returns\n -------\n pandas.Series\n \"\"\"\n self.materialize()\n return pandas.Series(self._known_dtypes)\n\n def get_dtypes_set(self) -> set[DtypeObj]:\n \"\"\"\n Get a set of dtypes from the descriptor.\n\n Returns\n -------\n set[DtypeObj]\n \"\"\"\n if len(self._cols_with_unknown_dtypes) > 0 or not self._know_all_names:\n self._materialize_cols_with_unknown_dtypes()\n known_dtypes: set[DtypeObj] = set(self._known_dtypes.values())\n if self._remaining_dtype is not None:\n known_dtypes.add(self._remaining_dtype)\n return known_dtypes\n\n @classmethod\n def _merge_dtypes(\n cls, values: list[Union[DtypesDescriptor, pandas.Series, None]]\n ) -> DtypesDescriptor:\n \"\"\"\n Union columns described by ``values`` and compute common dtypes for them.\n\n Parameters\n ----------\n values : list of DtypesDescriptors, pandas.Series or Nones\n\n Returns\n -------\n DtypesDescriptor\n \"\"\"\n known_dtypes = {}\n cols_with_unknown_dtypes = []\n know_all_names = True\n dtypes_are_unknown = False\n\n # index - joined column names, columns - dtypes taken from 'values'\n # 0 1 2 3\n # col1 int bool float int\n # col2 int int int int\n # colN bool bool bool int\n dtypes_matrix = pandas.DataFrame()\n\n for i, val in enumerate(values):\n if isinstance(val, cls):\n know_all_names &= val._know_all_names\n dtypes = val._known_dtypes.copy()\n dtypes.update({col: \"unknown\" for col in val._cols_with_unknown_dtypes})\n if val._remaining_dtype is not None:\n # we can't process remaining dtypes, so just discarding them\n know_all_names = False\n\n # setting a custom name to the Series to prevent duplicated names\n # in the 'dtypes_matrix'\n series = pandas.Series(dtypes, name=i)\n dtypes_matrix = pandas.concat([dtypes_matrix, series], axis=1)\n if not (val._know_all_names and val._remaining_dtype is None):\n dtypes_matrix.fillna(\n value={\n # If we encountered a 'NaN' while 'val' describes all the columns, then\n # it means, that the missing columns for this instance will be filled with NaNs (floats),\n # otherwise, it may indicate missing columns that this 'val' has no info about,\n # meaning that we shouldn't try computing a new dtype for this column,\n # so marking it as 'unknown'\n i: \"unknown\",\n },\n inplace=True,\n )\n elif isinstance(val, pandas.Series):\n dtypes_matrix = pandas.concat([dtypes_matrix, val], axis=1)\n elif val is None:\n # one of the 'dtypes' is None, meaning that we wouldn't been infer a valid result dtype,\n # however, we're continuing our loop so we would at least know the columns we're missing\n # dtypes for\n dtypes_are_unknown = True\n know_all_names = False\n else:\n raise NotImplementedError(type(val))\n\n if dtypes_are_unknown:\n return DtypesDescriptor(\n cols_with_unknown_dtypes=dtypes_matrix.index.tolist(),\n know_all_names=know_all_names,\n )\n\n def combine_dtypes(row):\n if (row == \"unknown\").any():\n return \"unknown\"\n row = row.fillna(pandas.api.types.pandas_dtype(\"float\"))\n return find_common_type(list(row.values))\n\n dtypes = dtypes_matrix.apply(combine_dtypes, axis=1)\n\n for col, dtype in dtypes.items():\n if dtype == \"unknown\":\n cols_with_unknown_dtypes.append(col)\n else:\n known_dtypes[col] = dtype\n\n return DtypesDescriptor(\n known_dtypes,\n cols_with_unknown_dtypes,\n remaining_dtype=None,\n know_all_names=know_all_names,\n )\n\n @classmethod\n def concat(\n cls, values: list[Union[DtypesDescriptor, pandas.Series, None]], axis: int = 0\n ) -> DtypesDescriptor:\n \"\"\"\n Concatenate dtypes descriptors into a single descriptor.\n\n Parameters\n ----------\n values : list of DtypesDescriptors and pandas.Series\n axis : int, default: 0\n If ``axis == 0``: concatenate column names. This implements the logic of\n how dtypes are combined on ``pd.concat([df1, df2], axis=1)``.\n If ``axis == 1``: perform a union join for the column names described by\n `values` and then find common dtypes for the columns appeared to be in\n an intersection. This implements the logic of how dtypes are combined on\n ``pd.concat([df1, df2], axis=0).dtypes``.\n\n Returns\n -------\n DtypesDescriptor\n \"\"\"\n if axis == 1:\n return cls._merge_dtypes(values)\n known_dtypes = {}\n cols_with_unknown_dtypes = []\n schema_is_known = True\n # some default value to not mix it with 'None'\n remaining_dtype = \"default\"\n know_all_names = True\n\n for val in values:\n if isinstance(val, cls):\n all_new_cols = (\n list(val._known_dtypes.keys()) + val._cols_with_unknown_dtypes\n )\n if any(\n col in known_dtypes or col in cols_with_unknown_dtypes\n for col in all_new_cols\n ):\n raise NotImplementedError(\n \"Duplicated column names are not yet supported by DtypesDescriptor\"\n )\n know_all_names &= val._know_all_names\n known_dtypes.update(val._known_dtypes)\n cols_with_unknown_dtypes.extend(val._cols_with_unknown_dtypes)\n if know_all_names:\n if (\n remaining_dtype == \"default\"\n and val._remaining_dtype is not None\n ):\n remaining_dtype = val._remaining_dtype\n elif (\n remaining_dtype != \"default\"\n and val._remaining_dtype is not None\n and remaining_dtype != val._remaining_dtype\n ):\n remaining_dtype = None\n know_all_names = False\n else:\n remaining_dtype = None\n schema_is_known &= val._schema_is_known\n elif isinstance(val, pandas.Series):\n if any(\n col in known_dtypes or col in cols_with_unknown_dtypes\n for col in val.index\n ):\n raise NotImplementedError(\n \"Duplicated column names are not yet supported by DtypesDescriptor\"\n )\n known_dtypes.update(val)\n elif val is None:\n remaining_dtype = None\n schema_is_known = False\n know_all_names = False\n else:\n raise NotImplementedError(type(val))\n return cls(\n known_dtypes,\n cols_with_unknown_dtypes,\n None if remaining_dtype == \"default\" else remaining_dtype,\n parent_df=None,\n _schema_is_known=schema_is_known,\n know_all_names=know_all_names,\n )\n\n @staticmethod\n def _normalize_levels(columns, reference=None):\n \"\"\"\n Normalize levels of MultiIndex column names.\n\n The function fills missing levels with empty strings as pandas do:\n '''\n >>> columns = [\"a\", (\"l1\", \"l2\"), (\"l1a\", \"l2a\", \"l3a\")]\n >>> _normalize_levels(columns)\n [(\"a\", \"\", \"\"), (\"l1\", \"l2\", \"\"), (\"l1a\", \"l2a\", \"l3a\")]\n >>> # with a reference\n >>> idx = pandas.MultiIndex(...)\n >>> idx.nlevels\n 4\n >>> _normalize_levels(columns, reference=idx)\n [(\"a\", \"\", \"\", \"\"), (\"l1\", \"l2\", \"\", \"\"), (\"l1a\", \"l2a\", \"l3a\", \"\")]\n '''\n\n Parameters\n ----------\n columns : sequence\n Labels to normalize. If dictionary, will replace keys with normalized columns.\n reference : pandas.Index, optional\n An index to match the number of levels with. If reference is a MultiIndex, then the reference number\n of levels should not be greater than the maximum number of levels in `columns`. If not specified,\n the `columns` themselves become a `reference`.\n\n Returns\n -------\n sequence\n Column values with normalized levels.\n dict[hashable, hashable]\n Mapping from old column names to new names, only contains column names that\n were changed.\n\n Raises\n ------\n ValueError\n When the reference number of levels is greater than the maximum number of levels\n in `columns`.\n \"\"\"\n if reference is None:\n reference = columns\n\n if isinstance(reference, pandas.Index):\n max_nlevels = reference.nlevels\n else:\n max_nlevels = 1\n for col in reference:\n if isinstance(col, tuple):\n max_nlevels = max(max_nlevels, len(col))\n\n # if the reference is a regular flat index, then no actions are required (the result will be\n # a flat index containing tuples of different lengths, this behavior fully matches pandas).\n # Yes, this shortcut skips the 'if max_columns_nlevels > max_nlevels' below check on purpose.\n if max_nlevels == 1:\n return columns, {}\n\n max_columns_nlevels = 1\n for col in columns:\n if isinstance(col, tuple):\n max_columns_nlevels = max(max_columns_nlevels, len(col))\n\n if max_columns_nlevels > max_nlevels:\n raise ValueError(\n f\"The reference number of levels is greater than the maximum number of levels in columns: {max_columns_nlevels} > {max_nlevels}\"\n )\n\n new_columns = []\n old_to_new_mapping = {}\n for col in columns:\n old_col = col\n if not isinstance(col, tuple):\n col = (col,)\n col = col + (\"\",) * (max_nlevels - len(col))\n new_columns.append(col)\n if old_col != col:\n old_to_new_mapping[old_col] = col\n\n return new_columns, old_to_new_mapping\n\n def _normalize_self_levels(self, reference=None):\n \"\"\"\n Call ``self._normalize_levels()`` for known and unknown dtypes of this object.\n\n Parameters\n ----------\n reference : pandas.Index, optional\n \"\"\"\n _, old_to_new_mapping = self._normalize_levels(\n self._known_dtypes.keys(), reference\n )\n for old_col, new_col in old_to_new_mapping.items():\n value = self._known_dtypes.pop(old_col)\n self._known_dtypes[new_col] = value\n self._cols_with_unknown_dtypes, _ = self._normalize_levels(\n self._cols_with_unknown_dtypes, reference\n )\n\n\nclass ModinDtypes:\n \"\"\"\n A class that hides the various implementations of the dtypes needed for optimization.\n\n Parameters\n ----------\n value : pandas.Series, callable, DtypesDescriptor or ModinDtypes, optional\n \"\"\"\n\n def __init__(\n self,\n value: Optional[Union[Callable, pandas.Series, DtypesDescriptor, ModinDtypes]],\n ):\n if callable(value) or isinstance(value, pandas.Series):\n self._value = value\n elif isinstance(value, DtypesDescriptor):\n self._value = value.to_series() if value.is_materialized else value\n elif isinstance(value, type(self)):\n self._value = value.copy()._value\n elif isinstance(value, None):\n self._value = DtypesDescriptor()\n else:\n raise ValueError(f\"ModinDtypes doesn't work with '{value}'\")\n\n def __repr__(self): # noqa: GL08\n return f\"ModinDtypes:\\n\\tvalue type: {type(self._value)};\\n\\tvalue:\\n\\t{self._value}\"\n\n def __str__(self): # noqa: GL08\n return self.__repr__()\n\n @property\n def is_materialized(self) -> bool:\n \"\"\"\n Check if the internal representation is materialized.\n\n Returns\n -------\n bool\n \"\"\"\n return isinstance(self._value, pandas.Series)\n\n def get_dtypes_set(self) -> set[DtypeObj]:\n \"\"\"\n Get a set of dtypes from the descriptor.\n\n Returns\n -------\n set[DtypeObj]\n \"\"\"\n if isinstance(self._value, DtypesDescriptor):\n return self._value.get_dtypes_set()\n if not self.is_materialized:\n self.get()\n return set(self._value.values)\n\n def maybe_specify_new_frame_ref(self, new_parent: PandasDataframe) -> ModinDtypes:\n \"\"\"\n Set a new parent for the stored value if needed.\n\n Parameters\n ----------\n new_parent : PandasDataframe\n\n Returns\n -------\n ModinDtypes\n A copy of ``ModinDtypes`` with updated parent.\n \"\"\"\n new_self = self.copy()\n if new_self.is_materialized:\n LazyProxyCategoricalDtype.update_dtypes(new_self._value, new_parent)\n return new_self\n if isinstance(self._value, DtypesDescriptor):\n new_self._value.update_parent(new_parent)\n return new_self\n return new_self\n\n def lazy_get(self, ids: list, numeric_index: bool = False) -> ModinDtypes:\n \"\"\"\n Get new ``ModinDtypes`` for a subset of columns without triggering any computations.\n\n Parameters\n ----------\n ids : list of index labels or positional indexers\n Columns for the subset.\n numeric_index : bool, default: False\n Whether `ids` are positional indixes or column labels to take.\n\n Returns\n -------\n ModinDtypes\n ``ModinDtypes`` that describes dtypes for columns specified in `ids`.\n \"\"\"\n if isinstance(self._value, DtypesDescriptor):\n res = self._value.lazy_get(ids, numeric_index)\n return ModinDtypes(res)\n elif callable(self._value):\n new_self = self.copy()\n old_value = new_self._value\n new_self._value = lambda: (\n old_value().iloc[ids] if numeric_index else old_value()[ids]\n )\n return new_self\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=not self.is_materialized\n )\n return ModinDtypes(self._value.iloc[ids] if numeric_index else self._value[ids])\n\n @classmethod\n def concat(cls, values: list, axis: int = 0) -> ModinDtypes:\n \"\"\"\n Concatenate dtypes.\n\n Parameters\n ----------\n values : list of DtypesDescriptors, pandas.Series, ModinDtypes and Nones\n axis : int, default: 0\n If ``axis == 0``: concatenate column names. This implements the logic of\n how dtypes are combined on ``pd.concat([df1, df2], axis=1)``.\n If ``axis == 1``: perform a union join for the column names described by\n `values` and then find common dtypes for the columns appeared to be in\n an intersection. This implements the logic of how dtypes are combined on\n ``pd.concat([df1, df2], axis=0).dtypes``.\n\n Returns\n -------\n ModinDtypes\n \"\"\"\n preprocessed_vals = []\n for val in values:\n if isinstance(val, cls):\n val = val._value\n if isinstance(val, (DtypesDescriptor, pandas.Series)) or val is None:\n preprocessed_vals.append(val)\n else:\n raise NotImplementedError(type(val))\n\n try:\n desc = DtypesDescriptor.concat(preprocessed_vals, axis=axis)\n except NotImplementedError as e:\n # 'DtypesDescriptor' doesn't support duplicated labels, however, if all values are pandas Series,\n # we still can perform concatenation using pure pandas\n if (\n # 'pd.concat(axis=1)' fails on duplicated labels anyway, so doing this logic\n # only in case 'axis=0'\n axis == 0\n and \"duplicated\" not in e.args[0].lower()\n or not all(isinstance(val, pandas.Series) for val in values)\n ):\n raise e\n desc = pandas.concat(values)\n return ModinDtypes(desc)\n\n def set_index(self, new_index: Union[pandas.Index, ModinIndex]) -> ModinDtypes:\n \"\"\"\n Set new column names for stored dtypes.\n\n Parameters\n ----------\n new_index : pandas.Index or ModinIndex\n\n Returns\n -------\n ModinDtypes\n New ``ModinDtypes`` with updated column names.\n \"\"\"\n new_self = self.copy()\n if self.is_materialized:\n new_self._value.index = new_index\n return new_self\n elif callable(self._value):\n old_val = new_self._value\n new_self._value = lambda: old_val().set_axis(new_index)\n return new_self\n elif isinstance(new_self._value, DtypesDescriptor):\n new_self._value = new_self._value.set_index(new_index)\n return new_self\n else:\n raise NotImplementedError()\n\n def get(self) -> pandas.Series:\n \"\"\"\n Get the materialized internal representation.\n\n Returns\n -------\n pandas.Series\n \"\"\"\n if not self.is_materialized:\n if callable(self._value):\n self._value = self._value()\n if self._value is None:\n self._value = pandas.Series([])\n elif isinstance(self._value, DtypesDescriptor):\n self._value = self._value.to_series()\n else:\n raise NotImplementedError(type(self._value))\n return self._value\n\n def __len__(self):\n \"\"\"\n Redirect the 'len' request to the internal representation.\n\n Returns\n -------\n int\n\n Notes\n -----\n Executing this function materializes the data.\n \"\"\"\n if not self.is_materialized:\n self.get()\n return len(self._value)\n\n def __reduce__(self):\n \"\"\"\n Serialize an object of this class.\n\n Returns\n -------\n tuple\n\n Notes\n -----\n The default implementation generates a recursion error. In a short:\n during the construction of the object, `__getattr__` function is called, which\n is not intended to be used in situations where the object is not initialized.\n \"\"\"\n return (self.__class__, (self._value,))\n\n def __getattr__(self, name):\n \"\"\"\n Redirect access to non-existent attributes to the internal representation.\n\n This is necessary so that objects of this class in most cases mimic the behavior\n of the ``pandas.Series``. The main limitations of the current approach are type\n checking and the use of this object where pandas dtypes are supposed to be used.\n\n Parameters\n ----------\n name : str\n Attribute name.\n\n Returns\n -------\n object\n Attribute.\n\n Notes\n -----\n Executing this function materializes the data.\n \"\"\"\n if not self.is_materialized:\n self.get()\n return self._value.__getattribute__(name)\n\n def copy(self) -> ModinDtypes:\n \"\"\"\n Copy an object without materializing the internal representation.\n\n Returns\n -------\n ModinDtypes\n \"\"\"\n idx_cache = self._value\n if not callable(idx_cache):\n idx_cache = idx_cache.copy()\n return ModinDtypes(idx_cache)\n\n def __getitem__(self, key): # noqa: GL08\n if not self.is_materialized:\n self.get()\n return self._value.__getitem__(key)\n\n def __setitem__(self, key, item): # noqa: GL08\n if not self.is_materialized:\n self.get()\n self._value.__setitem__(key, item)\n\n def __iter__(self): # noqa: GL08\n if not self.is_materialized:\n self.get()\n return iter(self._value)\n\n def __contains__(self, key): # noqa: GL08\n if not self.is_materialized:\n self.get()\n return key in self._value\n\n\nclass LazyProxyCategoricalDtype(pandas.CategoricalDtype):\n \"\"\"\n A lazy proxy representing ``pandas.CategoricalDtype``.\n\n Parameters\n ----------\n categories : list-like, optional\n ordered : bool, default: False\n\n Notes\n -----\n Important note! One shouldn't use the class' constructor to instantiate a proxy instance,\n it's intended only for compatibility purposes! In order to create a new proxy instance\n use the appropriate class method `._build_proxy(...)`.\n \"\"\"\n\n def __init__(self, categories=None, ordered=False):\n # These will be initialized later inside of the `._build_proxy()` method\n self._parent, self._column_name, self._categories_val, self._materializer = (\n None,\n None,\n None,\n None,\n )\n super().__init__(categories, ordered)\n\n @staticmethod\n def update_dtypes(dtypes, new_parent):\n \"\"\"\n Update a parent for categorical proxies in a dtype object.\n\n Parameters\n ----------\n dtypes : dict-like\n A dict-like object describing dtypes. The method will walk through every dtype\n an update parents for categorical proxies inplace.\n new_parent : object\n \"\"\"\n for key, value in dtypes.items():\n if isinstance(value, LazyProxyCategoricalDtype):\n dtypes[key] = value._update_proxy(new_parent, column_name=key)\n\n def _update_proxy(self, parent, column_name):\n \"\"\"\n Create a new proxy, if either parent or column name are different.\n\n Parameters\n ----------\n parent : object\n Source object to extract categories on demand.\n column_name : str\n Column name of the categorical column in the source object.\n\n Returns\n -------\n pandas.CategoricalDtype or LazyProxyCategoricalDtype\n \"\"\"\n if self._is_materialized:\n # The parent has been materialized, we don't need a proxy anymore.\n return pandas.CategoricalDtype(self.categories, ordered=self._ordered)\n elif parent is self._parent and column_name == self._column_name:\n return self\n else:\n return self._build_proxy(parent, column_name, self._materializer)\n\n @classmethod\n def _build_proxy(cls, parent, column_name, materializer, dtype=None):\n \"\"\"\n Construct a lazy proxy.\n\n Parameters\n ----------\n parent : object\n Source object to extract categories on demand.\n column_name : str\n Column name of the categorical column in the source object.\n materializer : callable(parent, column_name) -> pandas.CategoricalDtype\n A function to call in order to extract categorical values.\n dtype : dtype, optional\n The categories dtype.\n\n Returns\n -------\n LazyProxyCategoricalDtype\n \"\"\"\n result = cls()\n result._parent = parent\n result._column_name = column_name\n result._materializer = materializer\n result._dtype = dtype\n return result\n\n def _get_dtype(self):\n \"\"\"\n Get the categories dtype.\n\n Returns\n -------\n dtype\n \"\"\"\n if self._dtype is None:\n self._dtype = self.categories.dtype\n return self._dtype\n\n def __reduce__(self):\n \"\"\"\n Serialize an object of this class.\n\n Returns\n -------\n tuple\n\n Notes\n -----\n This object is serialized into a ``pandas.CategoricalDtype`` as an actual proxy can't be\n properly serialized because of the references it stores for its potentially distributed parent.\n \"\"\"\n return (pandas.CategoricalDtype, (self.categories, self.ordered))\n\n @property\n def _categories(self):\n \"\"\"\n Get materialized categorical values.\n\n Returns\n -------\n pandas.Index\n \"\"\"\n if not self._is_materialized:\n self._materialize_categories()\n return self._categories_val\n\n @_categories.setter\n def _categories(self, categories):\n \"\"\"\n Set new categorical values.\n\n Parameters\n ----------\n categories : list-like\n \"\"\"\n self._categories_val = categories\n self._parent = None # The parent is not required any more\n self._materializer = None\n self._dtype = None\n\n @property\n def _is_materialized(self) -> bool:\n \"\"\"\n Check whether categorical values were already materialized.\n\n Returns\n -------\n bool\n \"\"\"\n return self._categories_val is not None\n\n def _materialize_categories(self):\n \"\"\"Materialize actual categorical values.\"\"\"\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=self._parent is None,\n extra_log=\"attempted to materialize categories with parent being 'None'\",\n )\n categoricals = self._materializer(self._parent, self._column_name)\n self._categories = categoricals.categories\n self._ordered = categoricals.ordered\n\n\ndef get_categories_dtype(\n cdt: Union[LazyProxyCategoricalDtype, pandas.CategoricalDtype],\n) -> DtypeObj:\n \"\"\"\n Get the categories dtype.\n\n Parameters\n ----------\n cdt : LazyProxyCategoricalDtype or pandas.CategoricalDtype\n\n Returns\n -------\n dtype\n \"\"\"\n return (\n cdt._get_dtype()\n if isinstance(cdt, LazyProxyCategoricalDtype)\n else cdt.categories.dtype\n )\n\n\ndef extract_dtype(value) -> DtypeObj | pandas.Series:\n \"\"\"\n Extract dtype(s) from the passed `value`.\n\n Parameters\n ----------\n value : object\n\n Returns\n -------\n DtypeObj or pandas.Series of DtypeObj\n \"\"\"\n try:\n dtype = pandas.api.types.pandas_dtype(value)\n except (TypeError, ValueError):\n dtype = pandas.Series(value).dtype\n\n return dtype\n"
},
{
"path": "modin/core/dataframe/pandas/metadata/index.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module contains class ModinIndex.\"\"\"\n\nimport uuid\nfrom typing import Optional\n\nimport pandas\nfrom pandas.core.dtypes.common import is_list_like\nfrom pandas.core.indexes.api import ensure_index\n\n\nclass ModinIndex:\n \"\"\"\n A class that hides the various implementations of the index needed for optimization.\n\n Parameters\n ----------\n value : sequence, PandasDataframe or callable() -> (pandas.Index, list of ints), optional\n If a sequence passed this will be considered as the index values.\n If a ``PandasDataframe`` passed then it will be used to lazily extract indices\n when required, note that the `axis` parameter must be passed in this case.\n If a callable passed then it's expected to return a pandas Index and a list of\n partition lengths along the index axis.\n If ``None`` was passed, the index will be considered an incomplete and will raise\n a ``RuntimeError`` on an attempt of materialization. To complete the index object\n you have to use ``.maybe_specify_new_frame_ref()`` method.\n\n axis : int, optional\n Specifies an axis the object represents, serves as an optional hint. This parameter\n must be passed in case value is a ``PandasDataframe``.\n dtypes : pandas.Series, optional\n Materialized dtypes of index levels.\n \"\"\"\n\n def __init__(self, value=None, axis=None, dtypes: Optional[pandas.Series] = None):\n from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe\n\n self._is_default_callable = False\n self._axis = axis\n self._dtypes = dtypes\n\n if callable(value):\n self._value = value\n elif isinstance(value, PandasDataframe):\n assert axis is not None\n self._value = self._get_default_callable(value, axis)\n self._is_default_callable = True\n elif value is None:\n assert axis is not None\n self._value = value\n else:\n self._value = ensure_index(value)\n\n self._lengths_cache = None\n # index/lengths ID's for faster comparison between other ModinIndex objects,\n # these should be propagated to the copies of the index\n self._index_id = uuid.uuid4()\n self._lengths_id = uuid.uuid4()\n\n def maybe_get_dtypes(self) -> Optional[pandas.Series]:\n \"\"\"\n Get index dtypes if available.\n\n Returns\n -------\n pandas.Series or None\n \"\"\"\n if self._dtypes is not None:\n return self._dtypes\n if self.is_materialized:\n self._dtypes = (\n self._value.dtypes\n if isinstance(self._value, pandas.MultiIndex)\n else pandas.Series([self._value.dtype], index=[self._value.name])\n )\n return self._dtypes\n return None\n\n @staticmethod\n def _get_default_callable(dataframe_obj, axis):\n \"\"\"\n Build a callable extracting index labels and partitions lengths for the specified axis.\n\n Parameters\n ----------\n dataframe_obj : PandasDataframe\n axis : int\n 0 - extract indices, 1 - extract columns.\n\n Returns\n -------\n callable() -> tuple(pandas.Index, list[ints])\n \"\"\"\n return lambda: dataframe_obj._compute_axis_labels_and_lengths(axis)\n\n def maybe_specify_new_frame_ref(self, value, axis) -> \"ModinIndex\":\n \"\"\"\n Set a new reference for a frame used to lazily extract index labels if it's needed.\n\n The method sets a new reference only if the indices are not yet materialized and\n if a PandasDataframe was originally passed to construct this index (so the ModinIndex\n object holds a reference to it). The reason the reference should be updated is that\n we don't want to hold in memory those frames that are already not needed. Once the\n reference is updated, the old frame will be garbage collected if there are no\n more references to it.\n\n Parameters\n ----------\n value : PandasDataframe\n New dataframe to reference.\n axis : int\n Axis to extract labels from.\n\n Returns\n -------\n ModinIndex\n New ModinIndex with the reference updated.\n \"\"\"\n if self._value is not None and (\n not callable(self._value) or not self._is_default_callable\n ):\n return self\n\n new_index = self.copy(copy_lengths=True)\n new_index._axis = axis\n new_index._value = self._get_default_callable(value, new_index._axis)\n # if the '._value' was 'None' initially, then the '_is_default_callable' flag was\n # also being set to 'False', since now the '._value' is a default callable,\n # so we want to ensure that the flag is set to 'True'\n new_index._is_default_callable = True\n return new_index\n\n @property\n def is_materialized(self) -> bool:\n \"\"\"\n Check if the internal representation is materialized.\n\n Returns\n -------\n bool\n \"\"\"\n return self.is_materialized_index(self)\n\n @classmethod\n def is_materialized_index(cls, index) -> bool:\n \"\"\"\n Check if the passed object represents a materialized index.\n\n Parameters\n ----------\n index : object\n An object to check.\n\n Returns\n -------\n bool\n \"\"\"\n # importing here to avoid circular import issue\n from modin.pandas.indexing import is_range_like\n\n if isinstance(index, cls):\n index = index._value\n return is_list_like(index) or is_range_like(index) or isinstance(index, slice)\n\n def get(self, return_lengths=False) -> pandas.Index:\n \"\"\"\n Get the materialized internal representation.\n\n Parameters\n ----------\n return_lengths : bool, default: False\n In some cases, during the index calculation, it's possible to get\n the lengths of the partitions. This flag allows this data to be used\n for optimization.\n\n Returns\n -------\n pandas.Index\n \"\"\"\n if not self.is_materialized:\n if callable(self._value):\n index, self._lengths_cache = self._value()\n self._value = ensure_index(index)\n elif self._value is None:\n raise RuntimeError(\n \"It's not allowed to call '.materialize()' before '._value' is specified.\"\n )\n else:\n raise NotImplementedError(type(self._value))\n if return_lengths:\n return self._value, self._lengths_cache\n else:\n return self._value\n\n def equals(self, other: \"ModinIndex\") -> bool:\n \"\"\"\n Check equality of the index values.\n\n Parameters\n ----------\n other : ModinIndex\n\n Returns\n -------\n bool\n The result of the comparison.\n \"\"\"\n if self._index_id == other._index_id:\n return True\n\n if not self.is_materialized:\n self.get()\n\n if not other.is_materialized:\n other.get()\n\n return self._value.equals(other._value)\n\n def compare_partition_lengths_if_possible(self, other: \"ModinIndex\"):\n \"\"\"\n Compare the partition lengths cache for the index being stored if possible.\n\n The ``ModinIndex`` object may sometimes store the information about partition\n lengths along the axis the index belongs to. If both `self` and `other` have\n this information or it can be inferred from them, the method returns\n a boolean - the result of the comparison, otherwise it returns ``None``\n as an indication that the comparison cannot be made.\n\n Parameters\n ----------\n other : ModinIndex\n\n Returns\n -------\n bool or None\n The result of the comparison if both `self` and `other` contain\n the lengths data, ``None`` otherwise.\n \"\"\"\n if self._lengths_id == other._lengths_id:\n return True\n\n can_extract_lengths_from_self = self._lengths_cache is not None or callable(\n self._value\n )\n can_extract_lengths_from_other = other._lengths_cache is not None or callable(\n other._value\n )\n if can_extract_lengths_from_self and can_extract_lengths_from_other:\n return self.get(return_lengths=True)[1] == other.get(return_lengths=True)[1]\n return None\n\n def __len__(self):\n \"\"\"\n Redirect the 'len' request to the internal representation.\n\n Returns\n -------\n int\n\n Notes\n -----\n Executing this function materializes the data.\n \"\"\"\n if not self.is_materialized:\n self.get()\n return len(self._value)\n\n def __reduce__(self):\n \"\"\"\n Serialize an object of this class.\n\n Returns\n -------\n tuple\n\n Notes\n -----\n The default implementation generates a recursion error. In a short:\n during the construction of the object, `__getattr__` function is called, which\n is not intended to be used in situations where the object is not initialized.\n \"\"\"\n return (\n self.__class__,\n (self._value, self._axis),\n {\n \"_lengths_cache\": self._lengths_cache,\n \"_index_id\": self._index_id,\n \"_lengths_id\": self._lengths_id,\n \"_is_default_callable\": self._is_default_callable,\n },\n )\n\n def __getitem__(self, key):\n \"\"\"\n Get an index value at the position of `key`.\n\n Parameters\n ----------\n key : int\n\n Returns\n -------\n label\n \"\"\"\n if not self.is_materialized:\n self.get()\n return self._value[key]\n\n def __getattr__(self, name):\n \"\"\"\n Redirect access to non-existent attributes to the internal representation.\n\n This is necessary so that objects of this class in most cases mimic the behavior\n of the ``pandas.Index``. The main limitations of the current approach are type\n checking and the use of this object where pandas indexes are supposed to be used.\n\n Parameters\n ----------\n name : str\n Attribute name.\n\n Returns\n -------\n object\n Attribute.\n\n Notes\n -----\n Executing this function materializes the data.\n \"\"\"\n if not self.is_materialized:\n self.get()\n return self._value.__getattribute__(name)\n\n def copy(self, copy_lengths=False) -> \"ModinIndex\":\n \"\"\"\n Copy an object without materializing the internal representation.\n\n Parameters\n ----------\n copy_lengths : bool, default: False\n Whether to copy the stored partition lengths to the\n new index object.\n\n Returns\n -------\n ModinIndex\n \"\"\"\n idx_cache = self._value\n if idx_cache is not None and not callable(idx_cache):\n idx_cache = idx_cache.copy()\n result = ModinIndex(idx_cache, axis=self._axis, dtypes=self._dtypes)\n result._index_id = self._index_id\n result._is_default_callable = self._is_default_callable\n if copy_lengths:\n result._lengths_cache = self._lengths_cache\n result._lengths_id = self._lengths_id\n return result\n"
},
{
"path": "modin/core/dataframe/pandas/partitioning/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe classes related to its partitioning and optimized for pandas storage format.\"\"\"\n"
},
{
"path": "modin/core/dataframe/pandas/partitioning/axis_partition.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"The module defines base interface for an axis partition of a Modin DataFrame.\"\"\"\n\nimport warnings\n\nimport numpy as np\nimport pandas\n\nfrom modin.config import MinColumnPartitionSize, MinRowPartitionSize\nfrom modin.core.dataframe.base.partitioning.axis_partition import (\n BaseDataframeAxisPartition,\n)\nfrom modin.core.storage_formats.pandas.utils import (\n generate_result_of_axis_func_pandas,\n split_result_of_axis_func_pandas,\n)\n\nfrom .partition import PandasDataframePartition\n\n\nclass PandasDataframeAxisPartition(BaseDataframeAxisPartition):\n \"\"\"\n An abstract class is created to simplify and consolidate the code for axis partition that run pandas.\n\n Because much of the code is similar, this allows us to reuse this code.\n\n Parameters\n ----------\n list_of_partitions : Union[list, PandasDataframePartition]\n List of ``PandasDataframePartition`` and\n ``PandasDataframeAxisPartition`` objects, or a single\n ``PandasDataframePartition``.\n get_ip : bool, default: False\n Whether to get node IP addresses to conforming partitions or not.\n full_axis : bool, default: True\n Whether or not the axis partition encompasses the whole axis.\n call_queue : list, optional\n A list of tuples (callable, args, kwargs) that contains deferred calls.\n length : the future's type or int, optional\n Length, or reference to length, of wrapped ``pandas.DataFrame``.\n width : the future's type or int, optional\n Width, or reference to width, of wrapped ``pandas.DataFrame``.\n \"\"\"\n\n def __init__(\n self,\n list_of_partitions,\n get_ip=False,\n full_axis=True,\n call_queue=None,\n length=None,\n width=None,\n ):\n if isinstance(list_of_partitions, PandasDataframePartition):\n list_of_partitions = [list_of_partitions]\n self.full_axis = full_axis\n self.call_queue = call_queue or []\n self._length_cache = length\n self._width_cache = width\n # Check that all axis partition axes are the same in `list_of_partitions`\n # We should never have mismatching axis in the current implementation. We add this\n # defensive assertion to ensure that undefined behavior does not happen.\n assert (\n len(\n set(\n obj.axis\n for obj in list_of_partitions\n if isinstance(obj, PandasDataframeAxisPartition)\n )\n )\n <= 1\n )\n self._list_of_constituent_partitions = list_of_partitions\n # Defer computing _list_of_block_partitions because we might need to\n # drain call queues for that.\n self._list_of_block_partitions = None\n\n @property\n def list_of_blocks(self):\n \"\"\"\n Get the list of physical partition objects that compose this partition.\n\n Returns\n -------\n list\n A list of physical partition objects (``ray.ObjectRef``, ``distributed.Future`` e.g.).\n \"\"\"\n # Defer draining call queue (which is hidden in `partition.list_of_blocks` call) until we get the partitions.\n # TODO Look into draining call queue at the same time as the task\n return [\n partition.list_of_blocks[0] for partition in self.list_of_block_partitions\n ]\n\n @property\n def list_of_block_partitions(self) -> list:\n \"\"\"\n Get the list of block partitions that compose this partition.\n\n Returns\n -------\n List\n A list of ``PandasDataframePartition``.\n \"\"\"\n if self._list_of_block_partitions is not None:\n return self._list_of_block_partitions\n self._list_of_block_partitions = []\n # Extract block partitions from the block and axis partitions that\n # constitute this partition.\n for partition in self._list_of_constituent_partitions:\n if isinstance(partition, PandasDataframeAxisPartition):\n if partition.axis == self.axis:\n # We are building an axis partition out of another\n # axis partition `partition` that contains its own list\n # of block partitions, partition.list_of_block_partitions.\n # `partition` may have its own call queue, which has to be\n # applied to the entire `partition` before we execute any\n # further operations on its block parittions.\n partition.drain_call_queue()\n self._list_of_block_partitions.extend(\n partition.list_of_block_partitions\n )\n else:\n # If this axis partition is made of axis partitions\n # for the other axes, squeeze such partitions into a single\n # block so that this partition only holds a one-dimensional\n # list of blocks. We could change this implementation to\n # hold a 2-d list of blocks, but that would complicate the\n # code quite a bit.\n self._list_of_block_partitions.append(\n partition.force_materialization().list_of_block_partitions[0]\n )\n else:\n self._list_of_block_partitions.append(partition)\n return self._list_of_block_partitions\n\n @classmethod\n def _get_drain_func(cls): # noqa: GL08\n return PandasDataframeAxisPartition.drain\n\n def drain_call_queue(self, num_splits=None):\n \"\"\"\n Execute all operations stored in this partition's call queue.\n\n Parameters\n ----------\n num_splits : int, default: None\n The number of times to split the result object.\n \"\"\"\n if len(self.call_queue) == 0:\n # this implicitly calls `drain_call_queue` for block partitions,\n # which might have deferred call queues\n _ = self.list_of_blocks\n return\n call_queue = self.call_queue\n try:\n # Clearing the queue before calling `.apply()` so it won't try to drain it repeatedly\n self.call_queue = []\n drained = self.apply(\n self._get_drain_func(), num_splits=num_splits, call_queue=call_queue\n )\n except Exception:\n # Restoring the call queue in case of an exception as it most likely wasn't drained\n self.call_queue = call_queue\n raise\n if not isinstance(drained, list):\n drained = [drained]\n self._list_of_block_partitions = drained\n\n def force_materialization(self, get_ip=False):\n \"\"\"\n Materialize partitions into a single partition.\n\n Parameters\n ----------\n get_ip : bool, default: False\n Whether to get node ip address to a single partition or not.\n\n Returns\n -------\n PandasDataframeAxisPartition\n An axis partition containing only a single materialized partition.\n \"\"\"\n materialized = super().force_materialization(get_ip=get_ip)\n self._list_of_block_partitions = materialized.list_of_block_partitions\n return materialized\n\n def apply(\n self,\n func,\n *args,\n num_splits=None,\n other_axis_partition=None,\n maintain_partitioning=True,\n lengths=None,\n manual_partition=False,\n **kwargs,\n ):\n \"\"\"\n Apply a function to this axis partition along full axis.\n\n Parameters\n ----------\n func : callable\n The function to apply.\n *args : iterable\n Positional arguments to pass to `func`.\n num_splits : int, default: None\n The number of times to split the result object.\n other_axis_partition : PandasDataframeAxisPartition, default: None\n Another `PandasDataframeAxisPartition` object to be applied\n to func. This is for operations that are between two data sets.\n maintain_partitioning : bool, default: True\n Whether to keep the partitioning in the same\n orientation as it was previously or not. This is important because we may be\n operating on an individual AxisPartition and not touching the rest.\n In this case, we have to return the partitioning to its previous\n orientation (the lengths will remain the same). This is ignored between\n two axis partitions.\n lengths : iterable, default: None\n The list of lengths to shuffle the object.\n manual_partition : bool, default: False\n If True, partition the result with `lengths`.\n **kwargs : dict\n Additional keywords arguments to be passed in `func`.\n\n Returns\n -------\n list\n A list of `PandasDataframePartition` objects.\n \"\"\"\n if not self.full_axis:\n # If this is not a full axis partition, it already contains a subset of\n # the full axis, so we shouldn't split the result further.\n num_splits = 1\n if len(self.call_queue) > 0:\n self.drain_call_queue()\n\n if num_splits is None:\n num_splits = len(self.list_of_blocks)\n\n if other_axis_partition is not None:\n if not isinstance(other_axis_partition, list):\n other_axis_partition = [other_axis_partition]\n\n # (other_shape[i-1], other_shape[i]) will indicate slice\n # to restore i-1 axis partition\n other_shape = np.cumsum(\n [0] + [len(o.list_of_blocks) for o in other_axis_partition]\n )\n\n return self._wrap_partitions(\n self.deploy_func_between_two_axis_partitions(\n self.axis,\n func,\n args,\n kwargs,\n num_splits,\n len(self.list_of_blocks),\n other_shape,\n *tuple(\n self.list_of_blocks\n + [\n part\n for axis_partition in other_axis_partition\n for part in axis_partition.list_of_blocks\n ]\n ),\n min_block_size=(\n MinRowPartitionSize.get()\n if self.axis == 0\n else MinColumnPartitionSize.get()\n ),\n )\n )\n result = self._wrap_partitions(\n self.deploy_axis_func(\n self.axis,\n func,\n args,\n kwargs,\n num_splits,\n maintain_partitioning,\n *self.list_of_blocks,\n min_block_size=(\n MinRowPartitionSize.get()\n if self.axis == 0\n else MinColumnPartitionSize.get()\n ),\n lengths=lengths,\n manual_partition=manual_partition,\n )\n )\n if self.full_axis:\n return result\n else:\n # If this is not a full axis partition, just take out the single split in the result.\n return result[0]\n\n def split(\n self, split_func, num_splits, f_args=None, f_kwargs=None, extract_metadata=False\n ):\n \"\"\"\n Split axis partition into multiple partitions using the `split_func`.\n\n Parameters\n ----------\n split_func : callable(pandas.DataFrame) -> list[pandas.DataFrame]\n A function that takes partition's content and split it into multiple chunks.\n num_splits : int\n The number of splits the `split_func` return.\n f_args : iterable, optional\n Positional arguments to pass to the `split_func`.\n f_kwargs : dict, optional\n Keyword arguments to pass to the `split_func`.\n extract_metadata : bool, default: False\n Whether to return metadata (length, width, ip) of the result. Passing `False` may relax\n the load on object storage as the remote function would return X times fewer futures\n (where X is the number of metadata values). Passing `False` makes sense for temporary\n results where you know for sure that the metadata will never be requested.\n\n Returns\n -------\n list\n List of wrapped remote partition objects.\n \"\"\"\n f_args = tuple() if f_args is None else f_args\n f_kwargs = {} if f_kwargs is None else f_kwargs\n return self._wrap_partitions(\n self.deploy_splitting_func(\n self.axis,\n split_func,\n f_args,\n f_kwargs,\n num_splits,\n *self.list_of_blocks,\n extract_metadata=extract_metadata,\n ),\n extract_metadata=extract_metadata,\n )\n\n @classmethod\n def deploy_splitting_func(\n cls,\n axis,\n split_func,\n f_args,\n f_kwargs,\n num_splits,\n *partitions,\n extract_metadata=False,\n ):\n \"\"\"\n Deploy a splitting function along a full axis.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis to perform the function along.\n split_func : callable(pandas.DataFrame) -> list[pandas.DataFrame]\n The function to perform.\n f_args : list or tuple\n Positional arguments to pass to `split_func`.\n f_kwargs : dict\n Keyword arguments to pass to `split_func`.\n num_splits : int\n The number of splits the `split_func` return.\n *partitions : iterable\n All partitions that make up the full axis (row or column).\n extract_metadata : bool, default: False\n Whether to return metadata (length, width, ip) of the result. Note that `True` value\n is not supported in `PandasDataframeAxisPartition` class.\n\n Returns\n -------\n list\n A list of pandas DataFrames.\n \"\"\"\n dataframe = pandas.concat(list(partitions), axis=axis, copy=False)\n # to reduce peak memory consumption\n del partitions\n return split_func(dataframe, *f_args, **f_kwargs)\n\n @classmethod\n def deploy_axis_func(\n cls,\n axis,\n func,\n f_args,\n f_kwargs,\n num_splits,\n maintain_partitioning,\n *partitions,\n min_block_size,\n lengths=None,\n manual_partition=False,\n return_generator=False,\n ):\n \"\"\"\n Deploy a function along a full axis.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis to perform the function along.\n func : callable\n The function to perform.\n f_args : list or tuple\n Positional arguments to pass to ``func``.\n f_kwargs : dict\n Keyword arguments to pass to ``func``.\n num_splits : int\n The number of splits to return (see `split_result_of_axis_func_pandas`).\n maintain_partitioning : bool\n If True, keep the old partitioning if possible.\n If False, create a new partition layout.\n *partitions : iterable\n All partitions that make up the full axis (row or column).\n min_block_size : int\n Minimum number of rows/columns in a single split.\n lengths : list, optional\n The list of lengths to shuffle the object.\n manual_partition : bool, default: False\n If True, partition the result with `lengths`.\n return_generator : bool, default: False\n Return a generator from the function, set to `True` for Ray backend\n as Ray remote functions can return Generators.\n\n Returns\n -------\n list | Generator\n A list or generator of pandas DataFrames.\n \"\"\"\n len_partitions = len(partitions)\n lengths_partitions = [len(part) for part in partitions]\n widths_partitions = [len(part.columns) for part in partitions]\n\n dataframe = pandas.concat(list(partitions), axis=axis, copy=False)\n\n # to reduce peak memory consumption\n del partitions\n\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\", category=FutureWarning)\n try:\n result = func(dataframe, *f_args, **f_kwargs)\n except ValueError as err:\n if \"assignment destination is read-only\" in str(err):\n result = func(dataframe.copy(), *f_args, **f_kwargs)\n else:\n raise err\n\n # to reduce peak memory consumption\n del dataframe\n\n if num_splits == 1:\n # If we're not going to split the result, we don't need to specify\n # split lengths.\n lengths = None\n elif manual_partition:\n # The split function is expecting a list\n lengths = list(lengths)\n # We set lengths to None so we don't use the old lengths for the resulting partition\n # layout. This is done if the number of splits is changing or we are told not to\n # keep the old partitioning.\n elif num_splits != len_partitions or not maintain_partitioning:\n lengths = None\n else:\n if axis == 0:\n lengths = lengths_partitions\n if sum(lengths) != len(result):\n lengths = None\n else:\n lengths = widths_partitions\n if sum(lengths) != len(result.columns):\n lengths = None\n if return_generator:\n return generate_result_of_axis_func_pandas(\n axis,\n num_splits,\n result,\n min_block_size,\n lengths,\n )\n else:\n return split_result_of_axis_func_pandas(\n axis, num_splits, result, min_block_size, lengths\n )\n\n @classmethod\n def deploy_func_between_two_axis_partitions(\n cls,\n axis,\n func,\n f_args,\n f_kwargs,\n num_splits,\n len_of_left,\n other_shape,\n *partitions,\n min_block_size,\n return_generator=False,\n ):\n \"\"\"\n Deploy a function along a full axis between two data sets.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis to perform the function along.\n func : callable\n The function to perform.\n f_args : list or tuple\n Positional arguments to pass to ``func``.\n f_kwargs : dict\n Keyword arguments to pass to ``func``.\n num_splits : int\n The number of splits to return (see `split_result_of_axis_func_pandas`).\n len_of_left : int\n The number of values in `partitions` that belong to the left data set.\n other_shape : np.ndarray\n The shape of right frame in terms of partitions, i.e.\n (other_shape[i-1], other_shape[i]) will indicate slice to restore i-1 axis partition.\n *partitions : iterable\n All partitions that make up the full axis (row or column) for both data sets.\n min_block_size : int\n Minimum number of rows/columns in a single split.\n return_generator : bool, default: False\n Return a generator from the function, set to `True` for Ray backend\n as Ray remote functions can return Generators.\n\n Returns\n -------\n list | Generator\n A list or generator of pandas DataFrames.\n \"\"\"\n lt_frame = pandas.concat(partitions[:len_of_left], axis=axis, copy=False)\n\n rt_parts = partitions[len_of_left:]\n\n # to reduce peak memory consumption\n del partitions\n\n # reshaping flattened `rt_parts` array into a frame with shape `other_shape`\n combined_axis = [\n pandas.concat(\n rt_parts[other_shape[i - 1] : other_shape[i]],\n axis=axis,\n copy=False,\n )\n for i in range(1, len(other_shape))\n ]\n\n # to reduce peak memory consumption\n del rt_parts\n\n rt_frame = pandas.concat(combined_axis, axis=axis ^ 1, copy=False)\n\n # to reduce peak memory consumption\n del combined_axis\n\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\", category=FutureWarning)\n result = func(lt_frame, rt_frame, *f_args, **f_kwargs)\n\n # to reduce peak memory consumption\n del lt_frame, rt_frame\n\n if return_generator:\n return generate_result_of_axis_func_pandas(\n axis,\n num_splits,\n result,\n min_block_size,\n )\n else:\n return split_result_of_axis_func_pandas(\n axis,\n num_splits,\n result,\n min_block_size,\n )\n\n @classmethod\n def drain(cls, df: pandas.DataFrame, call_queue: list):\n \"\"\"\n Execute all operations stored in the call queue on the pandas object (helper function).\n\n Parameters\n ----------\n df : pandas.DataFrame\n call_queue : list\n Call queue that needs to be executed on pandas DataFrame.\n\n Returns\n -------\n pandas.DataFrame\n \"\"\"\n for func, args, kwargs in call_queue:\n df = func(df, *args, **kwargs)\n return df\n\n def mask(self, row_indices, col_indices):\n \"\"\"\n Create (synchronously) a mask that extracts the indices provided.\n\n Parameters\n ----------\n row_indices : list-like, slice or label\n The row labels for the rows to extract.\n col_indices : list-like, slice or label\n The column labels for the columns to extract.\n\n Returns\n -------\n PandasDataframeAxisPartition\n A new ``PandasDataframeAxisPartition`` object, materialized.\n \"\"\"\n return (\n self.force_materialization()\n .list_of_block_partitions[0]\n .mask(row_indices, col_indices)\n )\n\n def to_pandas(self):\n \"\"\"\n Convert the data in this partition to a ``pandas.DataFrame``.\n\n Returns\n -------\n pandas DataFrame.\n \"\"\"\n return self.force_materialization().list_of_block_partitions[0].to_pandas()\n\n def to_numpy(self):\n \"\"\"\n Convert the data in this partition to a ``numpy.array``.\n\n Returns\n -------\n NumPy array.\n \"\"\"\n return self.force_materialization().list_of_block_partitions[0].to_numpy()\n\n _length_cache = None\n\n def length(self, materialize=True):\n \"\"\"\n Get the length of this partition.\n\n Parameters\n ----------\n materialize : bool, default: True\n Whether to forcibly materialize the result into an integer. If ``False``\n was specified, may return a future of the result if it hasn't been\n materialized yet.\n\n Returns\n -------\n int\n The length of the partition.\n \"\"\"\n if self._length_cache is None:\n if self.axis == 0:\n self._length_cache = sum(\n obj.length() for obj in self.list_of_block_partitions\n )\n else:\n self._length_cache = self.list_of_block_partitions[0].length(\n materialize\n )\n return self._length_cache\n\n _width_cache = None\n\n def width(self, materialize=True):\n \"\"\"\n Get the width of this partition.\n\n Parameters\n ----------\n materialize : bool, default: True\n Whether to forcibly materialize the result into an integer. If ``False``\n was specified, may return a future of the result if it hasn't been\n materialized yet.\n\n Returns\n -------\n int\n The width of the partition.\n \"\"\"\n if self._width_cache is None:\n if self.axis == 1:\n self._width_cache = sum(\n obj.width() for obj in self.list_of_block_partitions\n )\n else:\n self._width_cache = self.list_of_block_partitions[0].width(materialize)\n return self._width_cache\n\n def wait(self):\n \"\"\"Wait completing computations on the object wrapped by the partition.\"\"\"\n pass\n\n def add_to_apply_calls(self, func, *args, length=None, width=None, **kwargs):\n \"\"\"\n Add a function to the call queue.\n\n Parameters\n ----------\n func : callable or a future type\n Function to be added to the call queue.\n *args : iterable\n Additional positional arguments to be passed in `func`.\n length : A future type or int, optional\n Length, or reference to it, of wrapped ``pandas.DataFrame``.\n width : A future type or int, optional\n Width, or reference to it, of wrapped ``pandas.DataFrame``.\n **kwargs : dict\n Additional keyword arguments to be passed in `func`.\n\n Returns\n -------\n PandasDataframeAxisPartition\n A new ``PandasDataframeAxisPartition`` object.\n \"\"\"\n return type(self)(\n self.list_of_block_partitions,\n full_axis=self.full_axis,\n call_queue=self.call_queue + [[func, args, kwargs]],\n length=length,\n width=width,\n )\n"
},
{
"path": "modin/core/dataframe/pandas/partitioning/partition.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"The module defines base interface for a partition of a Modin DataFrame.\"\"\"\n\nfrom __future__ import annotations\n\nimport logging\nimport uuid\nfrom abc import ABC\nfrom copy import copy\nfrom functools import cached_property\n\nimport pandas\nfrom pandas.api.types import is_scalar\n\nfrom modin.core.storage_formats.pandas.utils import length_fn_pandas, width_fn_pandas\nfrom modin.logging import ClassLogger, get_logger\nfrom modin.logging.config import LogLevel\nfrom modin.pandas.indexing import compute_sliced_len\n\n\nclass PandasDataframePartition(\n ABC, ClassLogger, modin_layer=\"BLOCK-PARTITION\", log_level=LogLevel.DEBUG\n): # pragma: no cover\n \"\"\"\n An abstract class that is base for any partition class of ``pandas`` storage format.\n\n The class providing an API that has to be overridden by child classes.\n \"\"\"\n\n _length_cache = None\n _width_cache = None\n _identity_cache = None\n _data = None\n execution_wrapper = None\n\n # these variables are intentionally initialized at runtime\n # so as not to initialize the engine during import\n _iloc_func = None\n\n def __init__(self):\n if type(self)._iloc_func is None:\n # Places `_iloc` function into the storage to speed up\n # remote function calls and caches the result.\n # It also postpones engine initialization, which happens\n # implicitly when `execution_wrapper.put` is called.\n if self.execution_wrapper is not None:\n type(self)._iloc_func = staticmethod(\n self.execution_wrapper.put(self._iloc)\n )\n else:\n type(self)._iloc_func = staticmethod(self._iloc)\n\n @cached_property\n def __constructor__(self) -> type[PandasDataframePartition]:\n \"\"\"\n Create a new instance of this object.\n\n Returns\n -------\n PandasDataframePartition\n New instance of pandas partition.\n \"\"\"\n return type(self)\n\n def get(self):\n \"\"\"\n Get the object wrapped by this partition.\n\n Returns\n -------\n object\n The object that was wrapped by this partition.\n\n Notes\n -----\n This is the opposite of the classmethod `put`.\n E.g. if you assign `x = PandasDataframePartition.put(1)`, `x.get()` should\n always return 1.\n \"\"\"\n log = get_logger()\n self._is_debug(log) and log.debug(f\"ENTER::Partition.get::{self._identity}\")\n self.drain_call_queue()\n result = self.execution_wrapper.materialize(self._data)\n self._is_debug(log) and log.debug(f\"EXIT::Partition.get::{self._identity}\")\n return result\n\n @property\n def list_of_blocks(self):\n \"\"\"\n Get the list of physical partition objects that compose this partition.\n\n Returns\n -------\n list\n A list of physical partition objects (``ray.ObjectRef``, ``distributed.Future`` e.g.).\n \"\"\"\n # Defer draining call queue until we get the partitions.\n # TODO Look into draining call queue at the same time as the task\n self.drain_call_queue()\n return [self._data]\n\n def apply(self, func, *args, **kwargs):\n \"\"\"\n Apply a function to the object wrapped by this partition.\n\n Parameters\n ----------\n func : callable\n Function to apply.\n *args : iterable\n Additional positional arguments to be passed in `func`.\n **kwargs : dict\n Additional keyword arguments to be passed in `func`.\n\n Returns\n -------\n PandasDataframePartition\n New `PandasDataframePartition` object.\n\n Notes\n -----\n It is up to the implementation how `kwargs` are handled. They are\n an important part of many implementations. As of right now, they\n are not serialized.\n \"\"\"\n pass\n\n def add_to_apply_calls(self, func, *args, length=None, width=None, **kwargs):\n \"\"\"\n Add a function to the call queue.\n\n Parameters\n ----------\n func : callable\n Function to be added to the call queue.\n *args : iterable\n Additional positional arguments to be passed in `func`.\n length : reference or int, optional\n Length, or reference to length, of wrapped ``pandas.DataFrame``.\n width : reference or int, optional\n Width, or reference to width, of wrapped ``pandas.DataFrame``.\n **kwargs : dict\n Additional keyword arguments to be passed in `func`.\n\n Returns\n -------\n PandasDataframePartition\n New `PandasDataframePartition` object with the function added to the call queue.\n\n Notes\n -----\n This function will be executed when `apply` is called. It will be executed\n in the order inserted; apply's func operates the last and return.\n \"\"\"\n return self.__constructor__(\n self._data,\n call_queue=self.call_queue + [[func, args, kwargs]],\n length=length,\n width=width,\n )\n\n def drain_call_queue(self):\n \"\"\"Execute all operations stored in the call queue on the object wrapped by this partition.\"\"\"\n pass\n\n def wait(self):\n \"\"\"Wait for completion of computations on the object wrapped by the partition.\"\"\"\n pass\n\n def to_pandas(self):\n \"\"\"\n Convert the object wrapped by this partition to a ``pandas.DataFrame``.\n\n Returns\n -------\n pandas.DataFrame\n\n Notes\n -----\n If the underlying object is a pandas DataFrame, this will likely\n only need to call `get`.\n \"\"\"\n dataframe = self.get()\n assert isinstance(dataframe, (pandas.DataFrame, pandas.Series))\n return dataframe\n\n def to_numpy(self, **kwargs):\n \"\"\"\n Convert the object wrapped by this partition to a NumPy array.\n\n Parameters\n ----------\n **kwargs : dict\n Additional keyword arguments to be passed in ``to_numpy``.\n\n Returns\n -------\n np.ndarray\n\n Notes\n -----\n If the underlying object is a pandas DataFrame, this will return\n a 2D NumPy array.\n \"\"\"\n return self.apply(lambda df: df.to_numpy(**kwargs)).get()\n\n @staticmethod\n def _iloc(df, row_labels, col_labels): # noqa: RT01, PR01\n \"\"\"Perform `iloc` on dataframes wrapped in partitions (helper function).\"\"\"\n return df.iloc[row_labels, col_labels]\n\n def mask(self, row_labels, col_labels):\n \"\"\"\n Lazily create a mask that extracts the indices provided.\n\n Parameters\n ----------\n row_labels : list-like, slice or label\n The row labels for the rows to extract.\n col_labels : list-like, slice or label\n The column labels for the columns to extract.\n\n Returns\n -------\n PandasDataframePartition\n New `PandasDataframePartition` object.\n \"\"\"\n\n def is_full_axis_mask(index, axis_length):\n \"\"\"Check whether `index` mask grabs `axis_length` amount of elements.\"\"\"\n if isinstance(index, slice):\n return index == slice(None) or (\n isinstance(axis_length, int)\n and compute_sliced_len(index, axis_length) == axis_length\n )\n return (\n hasattr(index, \"__len__\")\n and isinstance(axis_length, int)\n and len(index) == axis_length\n )\n\n row_labels = [row_labels] if is_scalar(row_labels) else row_labels\n col_labels = [col_labels] if is_scalar(col_labels) else col_labels\n\n if is_full_axis_mask(row_labels, self._length_cache) and is_full_axis_mask(\n col_labels, self._width_cache\n ):\n return copy(self)\n\n new_obj = self.add_to_apply_calls(self._iloc_func, row_labels, col_labels)\n\n def try_recompute_cache(indices, previous_cache):\n \"\"\"Compute new axis-length cache for the masked frame based on its previous cache.\"\"\"\n if not isinstance(indices, slice):\n return len(indices)\n if not isinstance(previous_cache, int):\n return None\n return compute_sliced_len(indices, previous_cache)\n\n new_obj._length_cache = try_recompute_cache(row_labels, self._length_cache)\n new_obj._width_cache = try_recompute_cache(col_labels, self._width_cache)\n return new_obj\n\n @classmethod\n def put(cls, obj):\n \"\"\"\n Put an object into a store and wrap it with partition object.\n\n Parameters\n ----------\n obj : object\n An object to be put.\n\n Returns\n -------\n PandasDataframePartition\n New `PandasDataframePartition` object.\n \"\"\"\n pass\n\n @classmethod\n def preprocess_func(cls, func):\n \"\"\"\n Preprocess a function before an `apply` call.\n\n Parameters\n ----------\n func : callable\n Function to preprocess.\n\n Returns\n -------\n callable\n An object that can be accepted by `apply`.\n\n Notes\n -----\n This is a classmethod because the definition of how to preprocess\n should be class-wide. Also, we may want to use this before we\n deploy a preprocessed function to multiple `PandasDataframePartition`\n objects.\n \"\"\"\n pass\n\n @classmethod\n def _length_extraction_fn(cls):\n \"\"\"\n Return the function that computes the length of the object wrapped by this partition.\n\n Returns\n -------\n callable\n The function that computes the length of the object wrapped by this partition.\n \"\"\"\n return length_fn_pandas\n\n @classmethod\n def _width_extraction_fn(cls):\n \"\"\"\n Return the function that computes the width of the object wrapped by this partition.\n\n Returns\n -------\n callable\n The function that computes the width of the object wrapped by this partition.\n \"\"\"\n return width_fn_pandas\n\n def length(self, materialize=True):\n \"\"\"\n Get the length of the object wrapped by this partition.\n\n Parameters\n ----------\n materialize : bool, default: True\n Whether to forcibly materialize the result into an integer. If ``False``\n was specified, may return a future of the result if it hasn't been\n materialized yet.\n\n Returns\n -------\n int or its Future\n The length of the object.\n \"\"\"\n if self._length_cache is None:\n self._length_cache = self.apply(self._length_extraction_fn()).get()\n return self._length_cache\n\n def width(self, materialize=True):\n \"\"\"\n Get the width of the object wrapped by the partition.\n\n Parameters\n ----------\n materialize : bool, default: True\n Whether to forcibly materialize the result into an integer. If ``False``\n was specified, may return a future of the result if it hasn't been\n materialized yet.\n\n Returns\n -------\n int or its Future\n The width of the object.\n \"\"\"\n if self._width_cache is None:\n self._width_cache = self.apply(self._width_extraction_fn()).get()\n return self._width_cache\n\n @property\n def _identity(self):\n \"\"\"\n Calculate identifier on request for debug logging mode.\n\n Returns\n -------\n str\n \"\"\"\n if self._identity_cache is None:\n self._identity_cache = uuid.uuid4().hex\n return self._identity_cache\n\n def split(self, split_func, num_splits, *args):\n \"\"\"\n Split the object wrapped by the partition into multiple partitions.\n\n Parameters\n ----------\n split_func : Callable[pandas.DataFrame, List[Any]] -> List[pandas.DataFrame]\n The function that will split this partition into multiple partitions. The list contains\n pivots to split by, and will have the same dtype as the major column we are shuffling on.\n num_splits : int\n The number of resulting partitions (may be empty).\n *args : List[Any]\n Arguments to pass to ``split_func``.\n\n Returns\n -------\n list\n A list of partitions.\n \"\"\"\n log = get_logger()\n self._is_debug(log) and log.debug(f\"ENTER::Partition.split::{self._identity}\")\n\n self._is_debug(log) and log.debug(f\"SUBMIT::_split_df::{self._identity}\")\n outputs = self.execution_wrapper.deploy(\n split_func, [self._data] + list(args), num_returns=num_splits\n )\n self._is_debug(log) and log.debug(f\"EXIT::Partition.split::{self._identity}\")\n return [self.__constructor__(output) for output in outputs]\n\n @classmethod\n def empty(cls):\n \"\"\"\n Create a new partition that wraps an empty pandas DataFrame.\n\n Returns\n -------\n PandasDataframePartition\n New `PandasDataframePartition` object.\n \"\"\"\n return cls.put(pandas.DataFrame(), 0, 0)\n\n def _is_debug(self, logger=None):\n \"\"\"\n Check that the logger is set to debug mode.\n\n Parameters\n ----------\n logger : logging.logger, optional\n Logger obtained from Modin's `get_logger` utility.\n Explicit transmission of this parameter can be used in the case\n when within the context of `_is_debug` call there was already\n `get_logger` call. This is an optimization.\n\n Returns\n -------\n bool\n \"\"\"\n if logger is None:\n logger = get_logger()\n return logger.isEnabledFor(logging.DEBUG)\n"
},
{
"path": "modin/core/dataframe/pandas/partitioning/partition_manager.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nModule holding base PartitionManager class - the thing that tracks partitions across the distribution.\n\nThe manager also allows manipulating the data - running functions at each partition, shuffle over the distribution, etc.\n\"\"\"\n\nimport os\nimport warnings\nfrom abc import ABC\nfrom functools import wraps\nfrom typing import TYPE_CHECKING, Optional\n\nimport numpy as np\nimport pandas\nfrom pandas._libs.lib import no_default\n\nfrom modin.config import (\n BenchmarkMode,\n CpuCount,\n DynamicPartitioning,\n Engine,\n MinColumnPartitionSize,\n MinRowPartitionSize,\n NPartitions,\n PersistentPickle,\n ProgressBar,\n)\nfrom modin.core.dataframe.pandas.utils import create_pandas_df_from_partitions\nfrom modin.core.storage_formats.pandas.utils import compute_chunksize\nfrom modin.error_message import ErrorMessage\nfrom modin.logging import ClassLogger\nfrom modin.logging.config import LogLevel\nfrom modin.pandas.utils import get_pandas_backend\n\nif TYPE_CHECKING:\n from modin.core.dataframe.pandas.dataframe.utils import ShuffleFunctions\n\n\ndef wait_computations_if_benchmark_mode(func):\n \"\"\"\n Make sure a `func` finished its computations in benchmark mode.\n\n Parameters\n ----------\n func : callable\n A function that should be performed in syncronous mode.\n\n Returns\n -------\n callable\n Wrapped function that executes eagerly (if benchmark mode) or original `func`.\n\n Notes\n -----\n `func` should return NumPy array with partitions.\n \"\"\"\n\n @wraps(func)\n def wait(cls, *args, **kwargs):\n \"\"\"Wait for computation results.\"\"\"\n result = func(cls, *args, **kwargs)\n if BenchmarkMode.get():\n if isinstance(result, tuple):\n partitions = result[0]\n else:\n partitions = result\n # When partitions have a deferred call queue, calling\n # partition.wait() on each partition serially will serially kick\n # off each deferred computation and wait for each partition to\n # finish before kicking off the next one. Instead, we want to\n # serially kick off all the deferred computations so that they can\n # all run asynchronously, then wait on all the results.\n cls.finalize(partitions)\n # The partition manager invokes the relevant .wait() method under\n # the hood, which should wait in parallel for all computations to finish\n cls.wait_partitions(partitions.flatten())\n return result\n\n return wait\n\n\nclass PandasDataframePartitionManager(\n ClassLogger, ABC, modin_layer=\"PARTITION-MANAGER\", log_level=LogLevel.DEBUG\n):\n \"\"\"\n Base class for managing the dataframe data layout and operators across the distribution of partitions.\n\n Partition class is the class to use for storing each partition.\n Each partition must extend the `PandasDataframePartition` class.\n \"\"\"\n\n _partition_class = None\n # Column partitions class is the class to use to create the column partitions.\n _column_partitions_class = None\n # Row partitions class is the class to use to create the row partitions.\n _row_partition_class = None\n _execution_wrapper = None\n\n @classmethod\n def materialize_futures(cls, input_list):\n \"\"\"\n Materialize all futures in the input list.\n\n Parameters\n ----------\n input_list : list\n The list that has to be manipulated.\n\n Returns\n -------\n list\n A new list with materialized objects.\n \"\"\"\n # Do nothing if input_list is None or [].\n if input_list is None:\n return None\n filtered_list = []\n filtered_idx = []\n for idx, item in enumerate(input_list):\n if cls._execution_wrapper.is_future(item):\n filtered_idx.append(idx)\n filtered_list.append(item)\n filtered_list = cls._execution_wrapper.materialize(filtered_list)\n result = input_list.copy()\n for idx, item in zip(filtered_idx, filtered_list):\n result[idx] = item\n return result\n\n @classmethod\n def preprocess_func(cls, map_func):\n \"\"\"\n Preprocess a function to be applied to `PandasDataframePartition` objects.\n\n Parameters\n ----------\n map_func : callable\n The function to be preprocessed.\n\n Returns\n -------\n callable\n The preprocessed version of the `map_func` provided.\n\n Notes\n -----\n Preprocessing does not require any specific format, only that the\n `PandasDataframePartition.apply` method will recognize it (for the subclass\n being used).\n\n If your `PandasDataframePartition` objects assume that a function provided\n is serialized or wrapped or in some other format, this is the place\n to add that logic. It is possible that this can also just return\n `map_func` if the `apply` method of the `PandasDataframePartition` object\n you are using does not require any modification to a given function.\n \"\"\"\n if cls._execution_wrapper.is_future(map_func):\n return map_func # Has already been preprocessed\n\n old_value = PersistentPickle.get()\n # When performing a function with Modin objects, it is more profitable to\n # do the conversion to pandas once on the main process than several times\n # on worker processes. Details: https://github.com/modin-project/modin/pull/6673/files#r1391086755\n # For Dask, otherwise there may be an error: `coroutine 'Client._gather' was never awaited`\n need_update = not PersistentPickle.get() and Engine.get() != \"Dask\"\n if need_update:\n PersistentPickle.put(True)\n try:\n result = cls._partition_class.preprocess_func(map_func)\n finally:\n if need_update:\n PersistentPickle.put(old_value)\n return result\n\n # END Abstract Methods\n\n @classmethod\n def create_partition_from_metadata(\n cls, dtypes: Optional[pandas.Series] = None, **metadata\n ):\n \"\"\"\n Create NumPy array of partitions that holds an empty dataframe with given metadata.\n\n Parameters\n ----------\n dtypes : pandas.Series, optional\n Column dtypes.\n Upon creating a pandas DataFrame from `metadata` we call `astype` since\n pandas doesn't allow to pass a list of dtypes directly in the constructor.\n **metadata : dict\n Metadata that has to be wrapped in a partition.\n\n Returns\n -------\n np.ndarray\n A NumPy 2D array of a single partition which contains the data.\n \"\"\"\n metadata_dataframe = pandas.DataFrame(**metadata)\n if dtypes is not None:\n metadata_dataframe = metadata_dataframe.astype(dtypes)\n return np.array([[cls._partition_class.put(metadata_dataframe)]])\n\n @classmethod\n def column_partitions(cls, partitions, full_axis=True):\n \"\"\"\n Get the list of `BaseDataframeAxisPartition` objects representing column-wise partitions.\n\n Parameters\n ----------\n partitions : list-like\n List of (smaller) partitions to be combined to column-wise partitions.\n full_axis : bool, default: True\n Whether or not this partition contains the entire column axis.\n\n Returns\n -------\n list\n A list of `BaseDataframeAxisPartition` objects.\n\n Notes\n -----\n Each value in this list will be an `BaseDataframeAxisPartition` object.\n `BaseDataframeAxisPartition` is located in `axis_partition.py`.\n \"\"\"\n if not isinstance(partitions, list):\n partitions = [partitions]\n return [\n cls._column_partitions_class(col, full_axis=full_axis)\n for frame in partitions\n for col in frame.T\n ]\n\n @classmethod\n def row_partitions(cls, partitions):\n \"\"\"\n List of `BaseDataframeAxisPartition` objects representing row-wise partitions.\n\n Parameters\n ----------\n partitions : list-like\n List of (smaller) partitions to be combined to row-wise partitions.\n\n Returns\n -------\n list\n A list of `BaseDataframeAxisPartition` objects.\n\n Notes\n -----\n Each value in this list will an `BaseDataframeAxisPartition` object.\n `BaseDataframeAxisPartition` is located in `axis_partition.py`.\n \"\"\"\n if not isinstance(partitions, list):\n partitions = [partitions]\n return [cls._row_partition_class(row) for frame in partitions for row in frame]\n\n @classmethod\n def axis_partition(cls, partitions, axis, full_axis: bool = True):\n \"\"\"\n Logically partition along given axis (columns or rows).\n\n Parameters\n ----------\n partitions : list-like\n List of partitions to be combined.\n axis : {0, 1}\n 0 for column partitions, 1 for row partitions.\n full_axis : bool, default: True\n Whether or not this partition contains the entire column axis.\n\n Returns\n -------\n list\n A list of `BaseDataframeAxisPartition` objects.\n \"\"\"\n make_column_partitions = axis == 0\n if not full_axis and not make_column_partitions:\n raise NotImplementedError(\n (\n \"Row partitions must contain the entire axis. We don't \"\n + \"support virtual partitioning for row partitions yet.\"\n )\n )\n return (\n cls.column_partitions(partitions)\n if make_column_partitions\n else cls.row_partitions(partitions)\n )\n\n @classmethod\n def groupby_reduce(\n cls, axis, partitions, by, map_func, reduce_func, apply_indices=None\n ):\n \"\"\"\n Groupby data using the `map_func` provided along the `axis` over the `partitions` then reduce using `reduce_func`.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to groupby over.\n partitions : NumPy 2D array\n Partitions of the ModinFrame to groupby.\n by : NumPy 2D array\n Partitions of 'by' to broadcast.\n map_func : callable\n Map function.\n reduce_func : callable,\n Reduce function.\n apply_indices : list of ints, default: None\n Indices of `axis ^ 1` to apply function over.\n\n Returns\n -------\n NumPy array\n Partitions with applied groupby.\n \"\"\"\n if apply_indices is not None:\n partitions = (\n partitions[apply_indices] if axis else partitions[:, apply_indices]\n )\n\n if by is not None:\n # need to make sure that the partitioning of the following objects\n # coincides in the required axis, because `partition_manager.broadcast_apply`\n # doesn't call `_copartition` unlike `modin_frame.broadcast_apply`\n assert partitions.shape[axis] == by.shape[axis], (\n f\"the number of partitions along {axis=} is not equal: \"\n + f\"{partitions.shape[axis]} != {by.shape[axis]}\"\n )\n mapped_partitions = cls.broadcast_apply(\n axis, map_func, left=partitions, right=by\n )\n else:\n mapped_partitions = cls.map_partitions(partitions, map_func)\n\n # Assuming, that the output will not be larger than the input,\n # keep the current number of partitions.\n num_splits = min(len(partitions), NPartitions.get())\n return cls.map_axis_partitions(\n axis,\n mapped_partitions,\n reduce_func,\n enumerate_partitions=True,\n num_splits=num_splits,\n )\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def broadcast_apply_select_indices(\n cls,\n axis,\n apply_func,\n left,\n right,\n left_indices,\n right_indices,\n keep_remaining=False,\n ):\n \"\"\"\n Broadcast the `right` partitions to `left` and apply `apply_func` to selected indices.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to apply and broadcast over.\n apply_func : callable\n Function to apply.\n left : NumPy 2D array\n Left partitions.\n right : NumPy 2D array\n Right partitions.\n left_indices : list-like\n Indices to apply function to.\n right_indices : dictionary of indices of right partitions\n Indices that you want to bring at specified left partition, for example\n dict {key: {key1: [0, 1], key2: [5]}} means that in left[key] you want to\n broadcast [right[key1], right[key2]] partitions and internal indices\n for `right` must be [[0, 1], [5]].\n keep_remaining : bool, default: False\n Whether or not to keep the other partitions.\n Some operations may want to drop the remaining partitions and\n keep only the results.\n\n Returns\n -------\n NumPy array\n An array of partition objects.\n\n Notes\n -----\n Your internal function must take these kwargs:\n [`internal_indices`, `other`, `internal_other_indices`] to work correctly!\n \"\"\"\n if not axis:\n partitions_for_apply = left.T\n right = right.T\n else:\n partitions_for_apply = left\n\n [obj.drain_call_queue() for row in right for obj in row]\n\n def get_partitions(index):\n \"\"\"Grab required partitions and indices from `right` and `right_indices`.\"\"\"\n must_grab = right_indices[index]\n partitions_list = np.array([right[i] for i in must_grab.keys()])\n indices_list = list(must_grab.values())\n return {\"other\": partitions_list, \"internal_other_indices\": indices_list}\n\n new_partitions = np.array(\n [\n (\n partitions_for_apply[i]\n if i not in left_indices\n else cls._apply_func_to_list_of_partitions_broadcast(\n apply_func,\n partitions_for_apply[i],\n internal_indices=left_indices[i],\n **get_partitions(i),\n )\n )\n for i in range(len(partitions_for_apply))\n if i in left_indices or keep_remaining\n ]\n )\n if not axis:\n new_partitions = new_partitions.T\n return new_partitions\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def base_broadcast_apply(cls, axis, apply_func, left, right):\n \"\"\"\n Broadcast the `right` partitions to `left` and apply `apply_func` function.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to apply and broadcast over.\n apply_func : callable\n Function to apply.\n left : np.ndarray\n NumPy array of left partitions.\n right : np.ndarray\n NumPy array of right partitions.\n\n Returns\n -------\n np.ndarray\n NumPy array of result partition objects.\n\n Notes\n -----\n This will often be overridden by implementations. It materializes the\n entire partitions of the right and applies them to the left through `apply`.\n \"\"\"\n\n def map_func(df, *others):\n other = (\n pandas.concat(others, axis=axis ^ 1) if len(others) > 1 else others[0]\n )\n # to reduce peak memory consumption\n del others\n return apply_func(df, other)\n\n map_func = cls.preprocess_func(map_func)\n rt_axis_parts = cls.axis_partition(right, axis ^ 1)\n return np.array(\n [\n [\n part.apply(\n map_func,\n *(\n rt_axis_parts[col_idx].list_of_blocks\n if axis\n else rt_axis_parts[row_idx].list_of_blocks\n ),\n )\n for col_idx, part in enumerate(left[row_idx])\n ]\n for row_idx in range(len(left))\n ]\n )\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def broadcast_axis_partitions(\n cls,\n axis,\n apply_func,\n left,\n right,\n keep_partitioning=False,\n num_splits=None,\n apply_indices=None,\n broadcast_all=True,\n enumerate_partitions=False,\n lengths=None,\n apply_func_args=None,\n **kwargs,\n ):\n \"\"\"\n Broadcast the `right` partitions to `left` and apply `apply_func` along full `axis`.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to apply and broadcast over.\n apply_func : callable\n Function to apply.\n left : NumPy 2D array\n Left partitions.\n right : NumPy 2D array\n Right partitions.\n keep_partitioning : boolean, default: False\n The flag to keep partition boundaries for Modin Frame if possible.\n Setting it to True disables shuffling data from one partition to another in case the resulting\n number of splits is equal to the initial number of splits.\n num_splits : int, optional\n The number of partitions to split the result into across the `axis`. If None, then the number\n of splits will be infered automatically. If `num_splits` is None and `keep_partitioning=True`\n then the number of splits is preserved.\n apply_indices : list of ints, default: None\n Indices of `axis ^ 1` to apply function over.\n broadcast_all : bool, default: True\n Whether or not to pass all right axis partitions to each of the left axis partitions.\n enumerate_partitions : bool, default: False\n Whether or not to pass partition index into `apply_func`.\n Note that `apply_func` must be able to accept `partition_idx` kwarg.\n lengths : list of ints, default: None\n The list of lengths to shuffle the object. Note:\n 1. Passing `lengths` omits the `num_splits` parameter as the number of splits\n will now be inferred from the number of integers present in `lengths`.\n 2. When passing lengths you must explicitly specify `keep_partitioning=False`.\n apply_func_args : list-like, optional\n Positional arguments to pass to the `func`.\n **kwargs : dict\n Additional options that could be used by different engines.\n\n Returns\n -------\n NumPy array\n An array of partition objects.\n \"\"\"\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=keep_partitioning and lengths is not None,\n extra_log=f\"`keep_partitioning` must be set to `False` when passing `lengths`. Got: {keep_partitioning=} | {lengths=}\",\n )\n\n # Since we are already splitting the DataFrame back up after an\n # operation, we will just use this time to compute the number of\n # partitions as best we can right now.\n if keep_partitioning and num_splits is None:\n num_splits = len(left) if axis == 0 else len(left.T)\n elif lengths:\n num_splits = len(lengths)\n elif num_splits is None:\n num_splits = NPartitions.get()\n else:\n ErrorMessage.catch_bugs_and_request_email(\n failure_condition=not isinstance(num_splits, int),\n extra_log=f\"Expected `num_splits` to be an integer, got: {type(num_splits)} | {num_splits=}\",\n )\n preprocessed_map_func = cls.preprocess_func(apply_func)\n left_partitions = cls.axis_partition(left, axis)\n right_partitions = None if right is None else cls.axis_partition(right, axis)\n # For mapping across the entire axis, we don't maintain partitioning because we\n # may want to line to partitioning up with another BlockPartitions object. Since\n # we don't need to maintain the partitioning, this gives us the opportunity to\n # load-balance the data as well.\n kw = {\n \"num_splits\": num_splits,\n \"maintain_partitioning\": keep_partitioning,\n }\n if lengths:\n kw[\"lengths\"] = lengths\n kw[\"manual_partition\"] = True\n\n if apply_indices is None:\n apply_indices = np.arange(len(left_partitions))\n\n result_blocks = np.array(\n [\n left_partitions[i].apply(\n preprocessed_map_func,\n *(apply_func_args if apply_func_args else []),\n other_axis_partition=(\n right_partitions if broadcast_all else right_partitions[i]\n ),\n **kw,\n **({\"partition_idx\": idx} if enumerate_partitions else {}),\n **kwargs,\n )\n for idx, i in enumerate(apply_indices)\n ]\n )\n # If we are mapping over columns, they are returned to use the same as\n # rows, so we need to transpose the returned 2D NumPy array to return\n # the structure to the correct order.\n return result_blocks.T if not axis else result_blocks\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def base_map_partitions(\n cls,\n partitions,\n map_func,\n func_args=None,\n func_kwargs=None,\n ):\n \"\"\"\n Apply `map_func` to every partition in `partitions`.\n\n Parameters\n ----------\n partitions : NumPy 2D array\n Partitions housing the data of Modin Frame.\n map_func : callable\n Function to apply.\n func_args : iterable, optional\n Positional arguments for the 'map_func'.\n func_kwargs : dict, optional\n Keyword arguments for the 'map_func'.\n\n Returns\n -------\n NumPy array\n An array of partitions\n \"\"\"\n preprocessed_map_func = cls.preprocess_func(map_func)\n return np.array(\n [\n [\n part.apply(\n preprocessed_map_func,\n *func_args if func_args is not None else (),\n **func_kwargs if func_kwargs is not None else {},\n )\n for part in row_of_parts\n ]\n for row_of_parts in partitions\n ]\n )\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def broadcast_apply(\n cls,\n axis,\n apply_func,\n left,\n right,\n ):\n \"\"\"\n Broadcast the `right` partitions to `left` and apply `apply_func` function using different approaches to achieve the best performance.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to apply and broadcast over.\n apply_func : callable\n Function to apply.\n left : np.ndarray\n NumPy array of left partitions.\n right : np.ndarray\n NumPy array of right partitions.\n\n Returns\n -------\n np.ndarray\n NumPy array of result partition objects.\n \"\"\"\n if not DynamicPartitioning.get():\n # block-wise broadcast\n new_partitions = cls.base_broadcast_apply(\n axis,\n apply_func,\n left,\n right,\n )\n else:\n # The dynamic partitioning behavior of `broadcast_apply` differs from that of `map_partitions`,\n # since the columnar approach for `broadcast_apply` results in slowdown.\n # axis-wise broadcast\n new_partitions = cls.broadcast_axis_partitions(\n axis=axis ^ 1,\n left=left,\n right=right,\n apply_func=apply_func,\n broadcast_all=False,\n keep_partitioning=True,\n )\n return new_partitions\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def map_partitions(\n cls,\n partitions,\n map_func,\n func_args=None,\n func_kwargs=None,\n ):\n \"\"\"\n Apply `map_func` to `partitions` using different approaches to achieve the best performance.\n\n Parameters\n ----------\n partitions : NumPy 2D array\n Partitions housing the data of Modin Frame.\n map_func : callable\n Function to apply.\n func_args : iterable, optional\n Positional arguments for the 'map_func'.\n func_kwargs : dict, optional\n Keyword arguments for the 'map_func'.\n\n Returns\n -------\n NumPy array\n An array of partitions\n \"\"\"\n if not DynamicPartitioning.get():\n # block-wise map\n new_partitions = cls.base_map_partitions(\n partitions, map_func, func_args, func_kwargs\n )\n else:\n # axis-wise map\n # we choose an axis for a combination of partitions\n # whose size is closer to the number of CPUs\n if abs(partitions.shape[0] - CpuCount.get()) < abs(\n partitions.shape[1] - CpuCount.get()\n ):\n axis = 1\n else:\n axis = 0\n\n column_splits = CpuCount.get() // partitions.shape[1]\n\n if axis == 0 and column_splits > 1:\n # splitting by parts of columnar partitions\n new_partitions = cls.map_partitions_joined_by_column(\n partitions, column_splits, map_func, func_args, func_kwargs\n )\n else:\n # splitting by full axis partitions\n new_partitions = cls.map_axis_partitions(\n axis,\n partitions,\n lambda df: map_func(\n df,\n *(func_args if func_args is not None else ()),\n **(func_kwargs if func_kwargs is not None else {}),\n ),\n keep_partitioning=True,\n )\n return new_partitions\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def lazy_map_partitions(\n cls,\n partitions,\n map_func,\n func_args=None,\n func_kwargs=None,\n enumerate_partitions=False,\n ):\n \"\"\"\n Apply `map_func` to every partition in `partitions` *lazily*.\n\n Parameters\n ----------\n partitions : NumPy 2D array\n Partitions of Modin Frame.\n map_func : callable\n Function to apply.\n func_args : iterable, optional\n Positional arguments for the 'map_func'.\n func_kwargs : dict, optional\n Keyword arguments for the 'map_func'.\n enumerate_partitions : bool, default: False\n\n Returns\n -------\n NumPy array\n An array of partitions\n \"\"\"\n preprocessed_map_func = cls.preprocess_func(map_func)\n return np.array(\n [\n [\n part.add_to_apply_calls(\n preprocessed_map_func,\n *(tuple() if func_args is None else func_args),\n **func_kwargs if func_kwargs is not None else {},\n **({\"partition_idx\": i} if enumerate_partitions else {}),\n )\n for part in row\n ]\n for i, row in enumerate(partitions)\n ]\n )\n\n @classmethod\n def map_axis_partitions(\n cls,\n axis,\n partitions,\n map_func,\n keep_partitioning=False,\n num_splits=None,\n lengths=None,\n enumerate_partitions=False,\n **kwargs,\n ):\n \"\"\"\n Apply `map_func` to every partition in `partitions` along given `axis`.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to perform the map across (0 - index, 1 - columns).\n partitions : NumPy 2D array\n Partitions of Modin Frame.\n map_func : callable\n Function to apply.\n keep_partitioning : boolean, default: False\n The flag to keep partition boundaries for Modin Frame if possible.\n Setting it to True disables shuffling data from one partition to another in case the resulting\n number of splits is equal to the initial number of splits.\n num_splits : int, optional\n The number of partitions to split the result into across the `axis`. If None, then the number\n of splits will be infered automatically. If `num_splits` is None and `keep_partitioning=True`\n then the number of splits is preserved.\n lengths : list of ints, default: None\n The list of lengths to shuffle the object. Note:\n 1. Passing `lengths` omits the `num_splits` parameter as the number of splits\n will now be inferred from the number of integers present in `lengths`.\n 2. When passing lengths you must explicitly specify `keep_partitioning=False`.\n enumerate_partitions : bool, default: False\n Whether or not to pass partition index into `map_func`.\n Note that `map_func` must be able to accept `partition_idx` kwarg.\n **kwargs : dict\n Additional options that could be used by different engines.\n\n Returns\n -------\n NumPy array\n An array of new partitions for Modin Frame.\n\n Notes\n -----\n This method should be used in the case when `map_func` relies on\n some global information about the axis.\n \"\"\"\n return cls.broadcast_axis_partitions(\n axis=axis,\n left=partitions,\n apply_func=map_func,\n keep_partitioning=keep_partitioning,\n num_splits=num_splits,\n right=None,\n lengths=lengths,\n enumerate_partitions=enumerate_partitions,\n **kwargs,\n )\n\n @classmethod\n def map_partitions_joined_by_column(\n cls,\n partitions,\n column_splits,\n map_func,\n map_func_args=None,\n map_func_kwargs=None,\n ):\n \"\"\"\n Combine several blocks by column into one virtual partition and apply \"map_func\" to them.\n\n Parameters\n ----------\n partitions : NumPy 2D array\n Partitions of Modin Frame.\n column_splits : int\n The number of splits by column.\n map_func : callable\n Function to apply.\n map_func_args : iterable, optional\n Positional arguments for the 'map_func'.\n map_func_kwargs : dict, optional\n Keyword arguments for the 'map_func'.\n\n Returns\n -------\n NumPy array\n An array of new partitions for Modin Frame.\n \"\"\"\n if column_splits < 1:\n raise ValueError(\n \"The value of columns_splits must be greater than or equal to 1.\"\n )\n # step cannot be less than 1\n step = max(partitions.shape[0] // column_splits, 1)\n preprocessed_map_func = cls.preprocess_func(map_func)\n result = np.empty(partitions.shape, dtype=object)\n for i in range(\n 0,\n partitions.shape[0],\n step,\n ):\n partitions_subset = partitions[i : i + step]\n # This is necessary when ``partitions.shape[0]`` is not divisible\n # by `column_splits` without a remainder.\n actual_step = len(partitions_subset)\n kw = {\n \"num_splits\": actual_step,\n }\n joined_column_partitions = cls.column_partitions(partitions_subset)\n for j in range(partitions.shape[1]):\n result[i : i + actual_step, j] = joined_column_partitions[j].apply(\n preprocessed_map_func,\n *map_func_args if map_func_args is not None else (),\n **kw,\n **map_func_kwargs if map_func_kwargs is not None else {},\n )\n\n return result\n\n @classmethod\n def concat(cls, axis, left_parts, right_parts):\n \"\"\"\n Concatenate the blocks of partitions with another set of blocks.\n\n Parameters\n ----------\n axis : int\n The axis to concatenate to.\n left_parts : np.ndarray\n NumPy array of partitions to concatenate with.\n right_parts : np.ndarray or list\n NumPy array of partitions to be concatenated.\n\n Returns\n -------\n np.ndarray\n A new NumPy array with concatenated partitions.\n list[int] or None\n Row lengths if possible to compute it.\n\n Notes\n -----\n Assumes that the blocks are already the same shape on the\n dimension being concatenated. A ValueError will be thrown if this\n condition is not met.\n \"\"\"\n # TODO: Possible change is `isinstance(right_parts, list)`\n if type(right_parts) is list:\n # `np.array` with partitions of empty ModinFrame has a shape (0,)\n # but `np.concatenate` can concatenate arrays only if its shapes at\n # specified axis are equals, so filtering empty frames to avoid concat error\n right_parts = [o for o in right_parts if o.size != 0]\n to_concat = (\n [left_parts] + right_parts if left_parts.size != 0 else right_parts\n )\n result = (\n np.concatenate(to_concat, axis=axis) if len(to_concat) else left_parts\n )\n else:\n result = np.append(left_parts, right_parts, axis=axis)\n if axis == 0:\n return cls.rebalance_partitions(result)\n else:\n return result, None\n\n @classmethod\n def to_pandas(cls, partitions):\n \"\"\"\n Convert NumPy array of PandasDataframePartition to pandas DataFrame.\n\n Parameters\n ----------\n partitions : np.ndarray\n NumPy array of PandasDataframePartition.\n\n Returns\n -------\n pandas.DataFrame\n A pandas DataFrame\n \"\"\"\n return create_pandas_df_from_partitions(\n cls.get_objects_from_partitions(partitions.flatten()), partitions.shape\n )\n\n @classmethod\n def to_numpy(cls, partitions, **kwargs):\n \"\"\"\n Convert NumPy array of PandasDataframePartition to NumPy array of data stored within `partitions`.\n\n Parameters\n ----------\n partitions : np.ndarray\n NumPy array of PandasDataframePartition.\n **kwargs : dict\n Keyword arguments for PandasDataframePartition.to_numpy function.\n\n Returns\n -------\n np.ndarray\n A NumPy array.\n \"\"\"\n return np.block(\n [[block.to_numpy(**kwargs) for block in row] for row in partitions]\n )\n\n @classmethod\n def split_pandas_df_into_partitions(\n cls, df, row_chunksize, col_chunksize, update_bar\n ):\n \"\"\"\n Split given pandas DataFrame according to the row/column chunk sizes into distributed partitions.\n\n Parameters\n ----------\n df : pandas.DataFrame\n row_chunksize : int\n col_chunksize : int\n update_bar : callable(x) -> x\n Function that updates a progress bar.\n\n Returns\n -------\n 2D np.ndarray[PandasDataframePartition]\n \"\"\"\n put_func = cls._partition_class.put\n # even a full-axis slice can cost something (https://github.com/pandas-dev/pandas/issues/55202)\n # so we try not to do it if unnecessary.\n if col_chunksize >= len(df.columns):\n col_parts = [df]\n else:\n col_parts = [\n df.iloc[:, i : i + col_chunksize]\n for i in range(0, len(df.columns), col_chunksize)\n ]\n parts = [\n [\n update_bar(\n put_func(col_part.iloc[i : i + row_chunksize]),\n )\n for col_part in col_parts\n ]\n for i in range(0, len(df), row_chunksize)\n ]\n return np.array(parts)\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def from_pandas(cls, df, return_dims=False):\n \"\"\"\n Return the partitions from pandas.DataFrame.\n\n Parameters\n ----------\n df : pandas.DataFrame\n A pandas.DataFrame.\n return_dims : bool, default: False\n If it's True, return as (np.ndarray, row_lengths, col_widths),\n else np.ndarray.\n\n Returns\n -------\n (np.ndarray, backend) or (np.ndarray, backend, row_lengths, col_widths)\n A NumPy array with partitions (with dimensions or not).\n \"\"\"\n num_splits = NPartitions.get()\n min_row_block_size = MinRowPartitionSize.get()\n min_column_block_size = MinColumnPartitionSize.get()\n row_chunksize = compute_chunksize(df.shape[0], num_splits, min_row_block_size)\n col_chunksize = compute_chunksize(\n df.shape[1], num_splits, min_column_block_size\n )\n\n bar_format = (\n \"{l_bar}{bar}{r_bar}\"\n if os.environ.get(\"DEBUG_PROGRESS_BAR\", \"False\") == \"True\"\n else \"{desc}: {percentage:3.0f}%{bar} Elapsed time: {elapsed}, estimated remaining time: {remaining}\"\n )\n if ProgressBar.get():\n with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\")\n try:\n from tqdm.autonotebook import tqdm as tqdm_notebook\n except ImportError:\n raise ImportError(\"Please pip install tqdm to use the progress bar\")\n\n rows = max(1, round(len(df) / row_chunksize))\n cols = max(1, round(len(df.columns) / col_chunksize))\n update_count = rows * cols\n pbar = tqdm_notebook(\n total=round(update_count),\n desc=\"Distributing Dataframe\",\n bar_format=bar_format,\n )\n else:\n pbar = None\n\n def update_bar(f):\n if ProgressBar.get():\n pbar.update(1)\n return f\n\n parts = cls.split_pandas_df_into_partitions(\n df, row_chunksize, col_chunksize, update_bar\n )\n backend = get_pandas_backend(df.dtypes)\n if ProgressBar.get():\n pbar.close()\n if not return_dims:\n return parts, backend\n else:\n row_lengths = [\n (\n row_chunksize\n if i + row_chunksize < len(df)\n else len(df) % row_chunksize or row_chunksize\n )\n for i in range(0, len(df), row_chunksize)\n ]\n col_widths = [\n (\n col_chunksize\n if i + col_chunksize < len(df.columns)\n else len(df.columns) % col_chunksize or col_chunksize\n )\n for i in range(0, len(df.columns), col_chunksize)\n ]\n return parts, backend, row_lengths, col_widths\n\n @classmethod\n def from_arrow(cls, at, return_dims=False):\n \"\"\"\n Return the partitions from Apache Arrow (PyArrow).\n\n Parameters\n ----------\n at : pyarrow.table\n Arrow Table.\n return_dims : bool, default: False\n If it's True, return as (np.ndarray, row_lengths, col_widths),\n else np.ndarray.\n\n Returns\n -------\n (np.ndarray, backend) or (np.ndarray, backend, row_lengths, col_widths)\n A NumPy array with partitions (with dimensions or not).\n \"\"\"\n return cls.from_pandas(at.to_pandas(), return_dims=return_dims)\n\n @classmethod\n def get_objects_from_partitions(cls, partitions):\n \"\"\"\n Get the objects wrapped by `partitions` (in parallel if supported).\n\n Parameters\n ----------\n partitions : np.ndarray\n NumPy array with ``PandasDataframePartition``-s.\n\n Returns\n -------\n list\n The objects wrapped by `partitions`.\n \"\"\"\n if hasattr(cls, \"_execution_wrapper\"):\n # more efficient parallel implementation\n for idx, part in enumerate(partitions):\n if hasattr(part, \"force_materialization\"):\n partitions[idx] = part.force_materialization()\n assert all(\n [len(partition.list_of_blocks) == 1 for partition in partitions]\n ), \"Implementation assumes that each partition contains a single block.\"\n return cls._execution_wrapper.materialize(\n [partition.list_of_blocks[0] for partition in partitions]\n )\n return [partition.get() for partition in partitions]\n\n @classmethod\n def wait_partitions(cls, partitions):\n \"\"\"\n Wait on the objects wrapped by `partitions`, without materializing them.\n\n This method will block until all computations in the list have completed.\n\n Parameters\n ----------\n partitions : np.ndarray\n NumPy array with ``PandasDataframePartition``-s.\n\n Notes\n -----\n This method should be implemented in a more efficient way for engines that supports\n waiting on objects in parallel.\n \"\"\"\n for partition in partitions:\n partition.wait()\n\n @classmethod\n def get_indices(cls, axis, partitions, index_func=None):\n \"\"\"\n Get the internal indices stored in the partitions.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to extract the labels over.\n partitions : np.ndarray\n NumPy array with PandasDataframePartition's.\n index_func : callable, default: None\n The function to be used to extract the indices.\n\n Returns\n -------\n pandas.Index\n A pandas Index object.\n list of pandas.Index\n The list of internal indices for each partition.\n\n Notes\n -----\n These are the global indices of the object. This is mostly useful\n when you have deleted rows/columns internally, but do not know\n which ones were deleted.\n \"\"\"\n if index_func is None:\n index_func = lambda df: df.axes[axis] # noqa: E731\n ErrorMessage.catch_bugs_and_request_email(not callable(index_func))\n func = cls.preprocess_func(index_func)\n target = partitions.T if axis == 0 else partitions\n if len(target):\n new_idx = [idx.apply(func) for idx in target[0]]\n new_idx = cls.get_objects_from_partitions(new_idx)\n else:\n new_idx = [pandas.Index([])]\n\n # filter empty indexes in case there are multiple partitions\n total_idx = list(filter(len, new_idx))\n if len(total_idx) > 0:\n # TODO FIX INFORMATION LEAK!!!!1!!1!!\n total_idx = total_idx[0].append(total_idx[1:])\n else:\n # Meaning that all partitions returned a zero-length index,\n # in this case, we return an index of any partition to preserve\n # the index's metadata\n total_idx = new_idx[0]\n return total_idx, new_idx\n\n @classmethod\n def _apply_func_to_list_of_partitions_broadcast(\n cls, func, partitions, other, **kwargs\n ):\n \"\"\"\n Apply a function to a list of remote partitions.\n\n `other` partitions will be broadcasted to `partitions`\n and `func` will be applied.\n\n Parameters\n ----------\n func : callable\n The func to apply.\n partitions : np.ndarray\n The partitions to which the `func` will apply.\n other : np.ndarray\n The partitions to be broadcasted to `partitions`.\n **kwargs : dict\n Keyword arguments for PandasDataframePartition.apply function.\n\n Returns\n -------\n list\n A list of PandasDataframePartition objects.\n \"\"\"\n preprocessed_func = cls.preprocess_func(func)\n return [\n obj.apply(preprocessed_func, other=[o.get() for o in broadcasted], **kwargs)\n for obj, broadcasted in zip(partitions, other.T)\n ]\n\n @classmethod\n def _apply_func_to_list_of_partitions(cls, func, partitions, **kwargs):\n \"\"\"\n Apply a function to a list of remote partitions.\n\n Parameters\n ----------\n func : callable\n The func to apply.\n partitions : np.ndarray\n The partitions to which the `func` will apply.\n **kwargs : dict\n Keyword arguments for PandasDataframePartition.apply function.\n\n Returns\n -------\n list\n A list of PandasDataframePartition objects.\n\n Notes\n -----\n This preprocesses the `func` first before applying it to the partitions.\n \"\"\"\n preprocessed_func = cls.preprocess_func(func)\n return [obj.apply(preprocessed_func, **kwargs) for obj in partitions]\n\n @classmethod\n def combine(cls, partitions, new_index=None, new_columns=None):\n \"\"\"\n Convert a NumPy 2D array of partitions to a NumPy 2D array of a single partition.\n\n Parameters\n ----------\n partitions : np.ndarray\n The partitions which have to be converted to a single partition.\n new_index : pandas.Index, optional\n Index for propagation into internal partitions.\n Optimization allowing to do this in one remote kernel.\n new_columns : pandas.Index, optional\n Columns for propagation into internal partitions.\n Optimization allowing to do this in one remote kernel.\n\n Returns\n -------\n np.ndarray\n A NumPy 2D array of a single partition.\n \"\"\"\n if partitions.size <= 1 and new_index is None and new_columns is None:\n return partitions\n\n def to_pandas_remote(df, partition_shape, *dfs):\n \"\"\"Copy of ``cls.to_pandas()`` method adapted for a remote function.\"\"\"\n return create_pandas_df_from_partitions(\n (df,) + dfs,\n partition_shape,\n called_from_remote=True,\n new_index=new_index,\n new_columns=new_columns,\n )\n\n preprocessed_func = cls.preprocess_func(to_pandas_remote)\n partition_shape = partitions.shape\n partitions_flattened = partitions.flatten()\n for idx, part in enumerate(partitions_flattened):\n if hasattr(part, \"force_materialization\"):\n partitions_flattened[idx] = part.force_materialization()\n partition_refs = [\n partition.list_of_blocks[0] for partition in partitions_flattened[1:]\n ]\n combined_partition = partitions.flat[0].apply(\n preprocessed_func, partition_shape, *partition_refs\n )\n return np.array([combined_partition]).reshape(1, -1)\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def apply_func_to_select_indices(\n cls, axis, partitions, func, indices, keep_remaining=False\n ):\n \"\"\"\n Apply a function to select indices.\n\n Parameters\n ----------\n axis : {0, 1}\n Axis to apply the `func` over.\n partitions : np.ndarray\n The partitions to which the `func` will apply.\n func : callable\n The function to apply to these indices of partitions.\n indices : dict\n The indices to apply the function to.\n keep_remaining : bool, default: False\n Whether or not to keep the other partitions. Some operations\n may want to drop the remaining partitions and keep\n only the results.\n\n Returns\n -------\n np.ndarray\n A NumPy array with partitions.\n\n Notes\n -----\n Your internal function must take a kwarg `internal_indices` for\n this to work correctly. This prevents information leakage of the\n internal index to the external representation.\n \"\"\"\n if partitions.size == 0:\n return np.array([[]])\n # Handling dictionaries has to be done differently, but we still want\n # to figure out the partitions that need to be applied to, so we will\n # store the dictionary in a separate variable and assign `indices` to\n # the keys to handle it the same as we normally would.\n if isinstance(func, dict):\n dict_func = func\n else:\n dict_func = None\n if not axis:\n partitions_for_apply = partitions.T\n else:\n partitions_for_apply = partitions\n # We may have a command to perform different functions on different\n # columns at the same time. We attempt to handle this as efficiently as\n # possible here. Functions that use this in the dictionary format must\n # accept a keyword argument `func_dict`.\n if dict_func is not None:\n if not keep_remaining:\n result = np.array(\n [\n cls._apply_func_to_list_of_partitions(\n func,\n partitions_for_apply[o_idx],\n func_dict={\n i_idx: dict_func[i_idx]\n for i_idx in list_to_apply\n if i_idx >= 0\n },\n )\n for o_idx, list_to_apply in indices.items()\n ]\n )\n else:\n result = np.array(\n [\n (\n partitions_for_apply[i]\n if i not in indices\n else cls._apply_func_to_list_of_partitions(\n func,\n partitions_for_apply[i],\n func_dict={\n idx: dict_func[idx]\n for idx in indices[i]\n if idx >= 0\n },\n )\n )\n for i in range(len(partitions_for_apply))\n ]\n )\n else:\n if not keep_remaining:\n # We are passing internal indices in here. In order for func to\n # actually be able to use this information, it must be able to take in\n # the internal indices. This might mean an iloc in the case of Pandas\n # or some other way to index into the internal representation.\n result = np.array(\n [\n cls._apply_func_to_list_of_partitions(\n func,\n partitions_for_apply[idx],\n internal_indices=list_to_apply,\n )\n for idx, list_to_apply in indices.items()\n ]\n )\n else:\n # The difference here is that we modify a subset and return the\n # remaining (non-updated) blocks in their original position.\n result = np.array(\n [\n (\n partitions_for_apply[i]\n if i not in indices\n else cls._apply_func_to_list_of_partitions(\n func,\n partitions_for_apply[i],\n internal_indices=indices[i],\n )\n )\n for i in range(len(partitions_for_apply))\n ]\n )\n return result.T if not axis else result\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def apply_func_to_select_indices_along_full_axis(\n cls, axis, partitions, func, indices, keep_remaining=False\n ):\n \"\"\"\n Apply a function to a select subset of full columns/rows.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis to apply the function over.\n partitions : np.ndarray\n The partitions to which the `func` will apply.\n func : callable\n The function to apply.\n indices : list-like\n The global indices to apply the func to.\n keep_remaining : bool, default: False\n Whether or not to keep the other partitions.\n Some operations may want to drop the remaining partitions and\n keep only the results.\n\n Returns\n -------\n np.ndarray\n A NumPy array with partitions.\n\n Notes\n -----\n This should be used when you need to apply a function that relies\n on some global information for the entire column/row, but only need\n to apply a function to a subset.\n For your func to operate directly on the indices provided,\n it must use `internal_indices` as a keyword argument.\n \"\"\"\n if partitions.size == 0:\n return np.array([[]])\n # Handling dictionaries has to be done differently, but we still want\n # to figure out the partitions that need to be applied to, so we will\n # store the dictionary in a separate variable and assign `indices` to\n # the keys to handle it the same as we normally would.\n if isinstance(func, dict):\n dict_func = func\n else:\n dict_func = None\n preprocessed_func = cls.preprocess_func(func)\n # Since we might be keeping the remaining blocks that are not modified,\n # we have to also keep the block_partitions object in the correct\n # direction (transpose for columns).\n if not keep_remaining:\n selected_partitions = partitions.T if not axis else partitions\n selected_partitions = np.array([selected_partitions[i] for i in indices])\n selected_partitions = (\n selected_partitions.T if not axis else selected_partitions\n )\n else:\n selected_partitions = partitions\n if not axis:\n partitions_for_apply = cls.column_partitions(selected_partitions)\n partitions_for_remaining = partitions.T\n else:\n partitions_for_apply = cls.row_partitions(selected_partitions)\n partitions_for_remaining = partitions\n # We may have a command to perform different functions on different\n # columns at the same time. We attempt to handle this as efficiently as\n # possible here. Functions that use this in the dictionary format must\n # accept a keyword argument `func_dict`.\n if dict_func is not None:\n if not keep_remaining:\n result = np.array(\n [\n part.apply(\n preprocessed_func,\n func_dict={idx: dict_func[idx] for idx in indices[i]},\n )\n for i, part in zip(indices, partitions_for_apply)\n ]\n )\n else:\n result = np.array(\n [\n (\n partitions_for_remaining[i]\n if i not in indices\n else cls._apply_func_to_list_of_partitions(\n preprocessed_func,\n partitions_for_apply[i],\n func_dict={idx: dict_func[idx] for idx in indices[i]},\n )\n )\n for i in range(len(partitions_for_apply))\n ]\n )\n else:\n if not keep_remaining:\n # See notes in `apply_func_to_select_indices`\n result = np.array(\n [\n part.apply(preprocessed_func, internal_indices=indices[i])\n for i, part in zip(indices, partitions_for_apply)\n ]\n )\n else:\n # See notes in `apply_func_to_select_indices`\n result = np.array(\n [\n (\n partitions_for_remaining[i]\n if i not in indices\n else partitions_for_apply[i].apply(\n preprocessed_func, internal_indices=indices[i]\n )\n )\n for i in range(len(partitions_for_remaining))\n ]\n )\n return result.T if not axis else result\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def apply_func_to_indices_both_axis(\n cls,\n partitions,\n func,\n row_partitions_list,\n col_partitions_list,\n item_to_distribute=no_default,\n row_lengths=None,\n col_widths=None,\n ):\n \"\"\"\n Apply a function along both axes.\n\n Parameters\n ----------\n partitions : np.ndarray\n The partitions to which the `func` will apply.\n func : callable\n The function to apply.\n row_partitions_list : iterable of tuples\n Iterable of tuples, containing 2 values:\n 1. Integer row partition index.\n 2. Internal row indexer of this partition.\n col_partitions_list : iterable of tuples\n Iterable of tuples, containing 2 values:\n 1. Integer column partition index.\n 2. Internal column indexer of this partition.\n item_to_distribute : np.ndarray or scalar, default: no_default\n The item to split up so it can be applied over both axes.\n row_lengths : list of ints, optional\n Lengths of partitions for every row. If not specified this information\n is extracted from partitions itself.\n col_widths : list of ints, optional\n Widths of partitions for every column. If not specified this information\n is extracted from partitions itself.\n\n Returns\n -------\n np.ndarray\n A NumPy array with partitions.\n\n Notes\n -----\n For your func to operate directly on the indices provided,\n it must use `row_internal_indices`, `col_internal_indices` as keyword\n arguments.\n \"\"\"\n partition_copy = partitions.copy()\n row_position_counter = 0\n\n if row_lengths is None:\n row_lengths = [None] * len(row_partitions_list)\n if col_widths is None:\n col_widths = [None] * len(col_partitions_list)\n\n def compute_part_size(indexer, remote_part, part_idx, axis):\n \"\"\"Compute indexer length along the specified axis for the passed partition.\"\"\"\n if isinstance(indexer, slice):\n shapes_container = row_lengths if axis == 0 else col_widths\n part_size = shapes_container[part_idx]\n if part_size is None:\n part_size = (\n remote_part.length() if axis == 0 else remote_part.width()\n )\n shapes_container[part_idx] = part_size\n indexer = range(*indexer.indices(part_size))\n return len(indexer)\n\n for row_idx, row_values in enumerate(row_partitions_list):\n row_blk_idx, row_internal_idx = row_values\n col_position_counter = 0\n row_offset = 0\n for col_idx, col_values in enumerate(col_partitions_list):\n col_blk_idx, col_internal_idx = col_values\n remote_part = partition_copy[row_blk_idx, col_blk_idx]\n\n row_offset = compute_part_size(\n row_internal_idx, remote_part, row_idx, axis=0\n )\n col_offset = compute_part_size(\n col_internal_idx, remote_part, col_idx, axis=1\n )\n\n if item_to_distribute is not no_default:\n if isinstance(item_to_distribute, np.ndarray):\n item = item_to_distribute[\n row_position_counter : row_position_counter + row_offset,\n col_position_counter : col_position_counter + col_offset,\n ]\n else:\n item = item_to_distribute\n item = {\"item\": item}\n else:\n item = {}\n block_result = remote_part.add_to_apply_calls(\n func,\n row_internal_indices=row_internal_idx,\n col_internal_indices=col_internal_idx,\n **item,\n )\n partition_copy[row_blk_idx, col_blk_idx] = block_result\n col_position_counter += col_offset\n row_position_counter += row_offset\n return partition_copy\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def n_ary_operation(cls, left, func, right: list):\n r\"\"\"\n Apply an n-ary operation to multiple ``PandasDataframe`` objects.\n\n This method assumes that all the partitions of the dataframes in left\n and right have the same dimensions. For each position i, j in each\n dataframe's partitions, the result has a partition at (i, j) whose data\n is func(left_partitions[i,j], \\*each_right_partitions[i,j]).\n\n Parameters\n ----------\n left : np.ndarray\n The partitions of left ``PandasDataframe``.\n func : callable\n The function to apply.\n right : list of np.ndarray\n The list of partitions of other ``PandasDataframe``.\n\n Returns\n -------\n np.ndarray\n A NumPy array with new partitions.\n \"\"\"\n func = cls.preprocess_func(func)\n\n def get_right_block(right_partitions, row_idx, col_idx):\n partition = right_partitions[row_idx][col_idx]\n blocks = partition.list_of_blocks\n \"\"\"\n NOTE:\n Currently we do one remote call per right virtual partition to\n materialize the partitions' blocks, then another remote call to do\n the n_ary operation. we could get better performance if we\n assembled the other partition within the remote `apply` call, by\n passing the partition in as `other_axis_partition`. However,\n passing `other_axis_partition` requires some extra care that would\n complicate the code quite a bit:\n - block partitions don't know how to deal with `other_axis_partition`\n - the right axis partition's axis could be different from the axis\n of the corresponding left partition\n - there can be multiple other_axis_partition because this is an n-ary\n operation and n can be > 2.\n So for now just do the materialization in a separate remote step.\n \"\"\"\n if len(blocks) > 1:\n partition.force_materialization()\n assert len(partition.list_of_blocks) == 1\n return partition.list_of_blocks[0]\n\n return np.array(\n [\n [\n part.apply(\n func,\n *(\n get_right_block(right_partitions, row_idx, col_idx)\n for right_partitions in right\n ),\n )\n for col_idx, part in enumerate(left[row_idx])\n ]\n for row_idx in range(len(left))\n ]\n )\n\n @classmethod\n def finalize(cls, partitions):\n \"\"\"\n Perform all deferred calls on partitions.\n\n Parameters\n ----------\n partitions : np.ndarray\n Partitions of Modin Dataframe on which all deferred calls should be performed.\n \"\"\"\n [part.drain_call_queue() for row in partitions for part in row]\n\n @classmethod\n def rebalance_partitions(cls, partitions):\n \"\"\"\n Rebalance a 2-d array of partitions if we are using ``PandasOnRay`` or ``PandasOnDask`` executions.\n\n For all other executions, the partitions are returned unchanged.\n\n Rebalance the partitions by building a new array\n of partitions out of the original ones so that:\n\n - If all partitions have a length, each new partition has roughly the same number of rows.\n - Otherwise, each new partition spans roughly the same number of old partitions.\n\n Parameters\n ----------\n partitions : np.ndarray\n The 2-d array of partitions to rebalance.\n\n Returns\n -------\n np.ndarray\n A NumPy array with the same; or new, rebalanced, partitions, depending on the execution\n engine and storage format.\n list[int] or None\n Row lengths if possible to compute it.\n \"\"\"\n # We rebalance when the ratio of the number of existing partitions to\n # the ideal number of partitions is larger than this threshold. The\n # threshold is a heuristic that may need to be tuned for performance.\n max_excess_of_num_partitions = 1.5\n num_existing_partitions = partitions.shape[0]\n ideal_num_new_partitions = NPartitions.get()\n if (\n num_existing_partitions\n <= ideal_num_new_partitions * max_excess_of_num_partitions\n ):\n return partitions, None\n # If any partition has an unknown length, give each axis partition\n # roughly the same number of row partitions. We use `_length_cache` here\n # to avoid materializing any unmaterialized lengths.\n if any(\n partition._length_cache is None for row in partitions for partition in row\n ):\n # We need each partition to go into an axis partition, but the\n # number of axis partitions may not evenly divide the number of\n # partitions.\n chunk_size = compute_chunksize(\n num_existing_partitions, ideal_num_new_partitions, min_block_size=1\n )\n new_partitions = np.array(\n [\n cls.column_partitions(\n partitions[i : i + chunk_size],\n full_axis=False,\n )\n for i in range(\n 0,\n num_existing_partitions,\n chunk_size,\n )\n ]\n )\n return new_partitions, None\n\n # If we know the number of rows in every partition, then we should try\n # instead to give each new partition roughly the same number of rows.\n new_partitions = []\n # `start` is the index of the first existing partition that we want to\n # put into the current new partition.\n start = 0\n total_rows = sum(part.length() for part in partitions[:, 0])\n ideal_partition_size = compute_chunksize(\n total_rows, ideal_num_new_partitions, min_block_size=1\n )\n for _ in range(ideal_num_new_partitions):\n # We might pick up old partitions too quickly and exhaust all of them.\n if start >= len(partitions):\n break\n # `stop` is the index of the last existing partition so far that we\n # want to put into the current new partition.\n stop = start\n partition_size = partitions[start][0].length()\n # Add existing partitions into the current new partition until the\n # number of rows in the new partition hits `ideal_partition_size`.\n while stop < len(partitions) and partition_size < ideal_partition_size:\n stop += 1\n if stop < len(partitions):\n partition_size += partitions[stop][0].length()\n # If the new partition is larger than we want, split the last\n # current partition that it contains into two partitions, where\n # the first partition has just enough rows to make the current\n # new partition have length `ideal_partition_size`, and the second\n # partition has the remainder.\n if partition_size > ideal_partition_size * max_excess_of_num_partitions:\n prev_length = sum(row[0].length() for row in partitions[start:stop])\n new_last_partition_size = ideal_partition_size - prev_length\n partitions = np.insert(\n partitions,\n stop + 1,\n [\n obj.mask(slice(new_last_partition_size, None), slice(None))\n for obj in partitions[stop]\n ],\n 0,\n )\n # TODO: explicit `_length_cache` computing may be avoided after #4903 is merged\n for obj in partitions[stop + 1]:\n obj._length_cache = partition_size - (\n prev_length + new_last_partition_size\n )\n\n partitions[stop, :] = [\n obj.mask(slice(None, new_last_partition_size), slice(None))\n for obj in partitions[stop]\n ]\n # TODO: explicit `_length_cache` computing may be avoided after #4903 is merged\n for obj in partitions[stop]:\n obj._length_cache = new_last_partition_size\n\n # The new virtual partitions are not `full_axis`, even if they\n # happen to span all rows in the dataframe, because they are\n # meant to be the final partitions of the dataframe. They've\n # already been split up correctly along axis 0, but using the\n # default full_axis=True would cause partition.apply() to split\n # its result along axis 0.\n new_partitions.append(\n cls.column_partitions(partitions[start : stop + 1], full_axis=False)\n )\n start = stop + 1\n new_partitions = np.array(new_partitions)\n lengths = [part.length() for part in new_partitions[:, 0]]\n return new_partitions, lengths\n\n @classmethod\n @wait_computations_if_benchmark_mode\n def shuffle_partitions(\n cls,\n partitions,\n index,\n shuffle_functions: \"ShuffleFunctions\",\n final_shuffle_func,\n right_partitions=None,\n ):\n \"\"\"\n Return shuffled partitions.\n\n Parameters\n ----------\n partitions : np.ndarray\n The 2-d array of partitions to shuffle.\n index : int or list of ints\n The index(es) of the column partitions corresponding to the partitions that contain the column to sample.\n shuffle_functions : ShuffleFunctions\n An object implementing the functions that we will be using to perform this shuffle.\n final_shuffle_func : Callable(pandas.DataFrame) -> pandas.DataFrame\n Function that shuffles the data within each new partition.\n right_partitions : np.ndarray, optional\n Partitions to broadcast to `self` partitions. If specified, the method builds range-partitioning\n for `right_partitions` basing on bins calculated for `partitions`, then performs broadcasting.\n\n Returns\n -------\n np.ndarray\n A list of row-partitions that have been shuffled.\n \"\"\"\n # Mask the partition that contains the column that will be sampled.\n masked_partitions = partitions[:, index]\n # Sample each partition\n sample_func = cls.preprocess_func(shuffle_functions.sample_fn)\n if masked_partitions.ndim == 1:\n samples = [partition.apply(sample_func) for partition in masked_partitions]\n else:\n samples = [\n cls._row_partition_class(row_part, full_axis=False).apply(sample_func)\n for row_part in masked_partitions\n ]\n # Get each sample to pass in to the pivot function\n samples = cls.get_objects_from_partitions(samples)\n num_bins = shuffle_functions.pivot_fn(samples)\n # Convert our list of block partitions to row partitions. We need to create full-axis\n # row partitions since we need to send the whole partition to the split step as otherwise\n # we wouldn't know how to split the block partitions that don't contain the shuffling key.\n row_partitions = cls.row_partitions(partitions)\n if num_bins > 1:\n # Gather together all of the sub-partitions\n split_row_partitions = np.array(\n [\n partition.split(\n shuffle_functions.split_fn,\n num_splits=num_bins,\n # The partition's metadata will never be accessed for the split partitions,\n # thus no need to compute it.\n extract_metadata=False,\n )\n for partition in row_partitions\n ]\n ).T\n\n if right_partitions is None:\n # We need to convert every partition that came from the splits into a column partition.\n return np.array(\n [\n [\n cls._column_partitions_class(\n row_partition, full_axis=False\n ).apply(final_shuffle_func)\n ]\n for row_partition in split_row_partitions\n ]\n )\n\n right_row_parts = cls.row_partitions(right_partitions)\n right_split_row_partitions = np.array(\n [\n partition.split(\n shuffle_functions.split_fn,\n num_splits=num_bins,\n extract_metadata=False,\n )\n for partition in right_row_parts\n ]\n ).T\n return np.array(\n [\n cls._column_partitions_class(row_partition, full_axis=False).apply(\n final_shuffle_func,\n other_axis_partition=cls._column_partitions_class(\n right_row_partitions\n ),\n )\n for right_row_partitions, row_partition in zip(\n right_split_row_partitions, split_row_partitions\n )\n ]\n )\n\n else:\n # If there are not pivots we can simply apply the function row-wise\n if right_partitions is None:\n return np.array(\n [row_part.apply(final_shuffle_func) for row_part in row_partitions]\n )\n right_row_parts = cls.row_partitions(right_partitions)\n return np.array(\n [\n row_part.apply(\n final_shuffle_func, other_axis_partition=right_row_part\n )\n for right_row_part, row_part in zip(right_row_parts, row_partitions)\n ]\n )\n"
},
{
"path": "modin/core/dataframe/pandas/utils.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\n\"\"\"Collection of utility functions for the PandasDataFrame.\"\"\"\n\nimport pandas\nfrom pandas.api.types import union_categoricals\n\nfrom modin.error_message import ErrorMessage\n\n\ndef concatenate(dfs, copy=True):\n \"\"\"\n Concatenate pandas DataFrames with saving 'category' dtype.\n\n All dataframes' columns must be equal to each other.\n\n Parameters\n ----------\n dfs : list\n List of pandas DataFrames to concatenate.\n copy : bool, default: True\n Make explicit copy when creating dataframe.\n\n Returns\n -------\n pandas.DataFrame\n A pandas DataFrame.\n \"\"\"\n for df in dfs:\n assert df.columns.equals(dfs[0].columns)\n for i in dfs[0].columns.get_indexer_for(dfs[0].select_dtypes(\"category\").columns):\n columns = [df.iloc[:, i] for df in dfs]\n all_categorical_parts_are_empty = None\n has_non_categorical_parts = False\n for col in columns:\n if isinstance(col.dtype, pandas.CategoricalDtype):\n if all_categorical_parts_are_empty is None:\n all_categorical_parts_are_empty = len(col) == 0\n continue\n all_categorical_parts_are_empty &= len(col) == 0\n else:\n has_non_categorical_parts = True\n # 'union_categoricals' raises an error if some of the passed values don't have categorical dtype,\n # if it happens, we only want to continue when all parts with categorical dtypes are actually empty.\n # This can happen if there were an aggregation that discards categorical dtypes and that aggregation\n # doesn't properly do so for empty partitions\n if has_non_categorical_parts and all_categorical_parts_are_empty:\n continue\n union = union_categoricals(columns)\n for df in dfs:\n df.isetitem(\n i, pandas.Categorical(df.iloc[:, i], categories=union.categories)\n )\n # `ValueError: buffer source array is read-only` if copy==False\n if len(dfs) == 1 and copy:\n # concat doesn't make a copy if len(dfs) == 1,\n # so do it explicitly\n return dfs[0].copy()\n return pandas.concat(dfs, copy=copy)\n\n\ndef create_pandas_df_from_partitions(\n partition_data,\n partition_shape,\n called_from_remote=False,\n new_index=None,\n new_columns=None,\n):\n \"\"\"\n Convert partition data of multiple dataframes to a single dataframe.\n\n Parameters\n ----------\n partition_data : list\n List of pandas DataFrames or list of Object references holding pandas DataFrames.\n partition_shape : int or tuple\n Shape of the partitions NumPy array.\n called_from_remote : bool, default: False\n Flag used to check if explicit copy should be done in concat.\n new_index : pandas.Index, optional\n Index for propagation into internal partitions.\n Optimization allowing to do this in one remote kernel.\n new_columns : pandas.Index, optional\n Columns for propagation into internal partitions.\n Optimization allowing to do this in one remote kernel.\n\n Returns\n -------\n pandas.DataFrame\n A pandas DataFrame.\n \"\"\"\n if all(\n isinstance(obj, (pandas.DataFrame, pandas.Series)) for obj in partition_data\n ):\n height, width, *_ = tuple(partition_shape) + (0,)\n # restore 2d array\n objs = iter(partition_data)\n partition_data = [[next(objs) for _ in range(width)] for __ in range(height)]\n else:\n # Partitions do not always contain pandas objects.\n # This implementation comes from the fact that calling `partition.get`\n # function is not always equivalent to `partition.to_pandas`.\n partition_data = [[obj.to_pandas() for obj in part] for part in partition_data]\n if all(isinstance(part, pandas.Series) for row in partition_data for part in row):\n axis = 0\n elif all(\n isinstance(part, pandas.DataFrame) for row in partition_data for part in row\n ):\n axis = 1\n else:\n ErrorMessage.catch_bugs_and_request_email(True)\n\n def is_part_empty(part):\n return part.empty and (\n not isinstance(part, pandas.DataFrame) or (len(part.columns) == 0)\n )\n\n df_rows = [\n pandas.concat([part for part in row], axis=axis, copy=False)\n for row in partition_data\n if not all(is_part_empty(part) for part in row)\n ]\n\n # to reduce peak memory consumption\n del partition_data\n\n if len(df_rows) == 0:\n res = pandas.DataFrame()\n else:\n res = concatenate(df_rows, copy=not called_from_remote)\n\n if new_index is not None:\n res.index = new_index\n if new_columns is not None:\n res.columns = new_columns\n\n return res\n"
},
{
"path": "modin/core/execution/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to execution engines supported.\"\"\"\n"
},
{
"path": "modin/core/execution/dask/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to Dask execution engine.\"\"\"\n"
},
{
"path": "modin/core/execution/dask/common/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Common utilities for Dask execution engine.\"\"\"\n\nfrom .engine_wrapper import DaskWrapper\nfrom .utils import initialize_dask\n\n__all__ = [\n \"initialize_dask\",\n \"DaskWrapper\",\n]\n"
},
{
"path": "modin/core/execution/dask/common/engine_wrapper.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses class responsible for execution of remote operations.\"\"\"\n\nfrom collections import UserDict\n\nimport pandas\nfrom dask.distributed import wait\nfrom distributed import Future\nfrom distributed.client import default_client\nfrom distributed.worker import get_worker\n\n\ndef get_dask_client():\n \"\"\"\n Get the Dask client, reusing the worker's client if execution is on a Dask worker.\n\n Returns\n -------\n distributed.Client\n The Dask client.\n \"\"\"\n try:\n client = default_client()\n except ValueError:\n # We ought to be in a worker process\n worker = get_worker()\n client = worker.client\n return client\n\n\ndef _deploy_dask_func(func, *args, return_pandas_df=None, **kwargs): # pragma: no cover\n \"\"\"\n Wrap `func` to ease calling it remotely.\n\n Parameters\n ----------\n func : callable\n A local function that we want to call remotely.\n *args : iterable\n Positional arguments to pass to `func` when calling remotely.\n return_pandas_df : bool, optional\n Whether to convert the result of `func` to a pandas DataFrame or not.\n **kwargs : dict\n Keyword arguments to pass to `func` when calling remotely.\n\n Returns\n -------\n distributed.Future or list\n Dask identifier of the result being put into distributed memory.\n \"\"\"\n result = func(*args, **kwargs)\n if return_pandas_df and not isinstance(result, pandas.DataFrame):\n result = pandas.DataFrame(result)\n return result\n\n\nclass DaskWrapper:\n \"\"\"The class responsible for execution of remote operations.\"\"\"\n\n @classmethod\n def deploy(\n cls,\n func,\n f_args=None,\n f_kwargs=None,\n return_pandas_df=None,\n num_returns=1,\n pure=True,\n ):\n \"\"\"\n Deploy a function in a worker process.\n\n Parameters\n ----------\n func : callable or distributed.Future\n Function to be deployed in a worker process.\n f_args : list or tuple, optional\n Positional arguments to pass to ``func``.\n f_kwargs : dict, optional\n Keyword arguments to pass to ``func``.\n return_pandas_df : bool, optional\n Whether to convert the result of `func` to a pandas DataFrame or not.\n num_returns : int, default: 1\n The number of returned objects.\n pure : bool, default: True\n Whether or not `func` is pure. See `Client.submit` for details.\n\n Returns\n -------\n list\n The result of ``func`` split into parts in accordance with ``num_returns``.\n \"\"\"\n client = get_dask_client()\n args = [] if f_args is None else f_args\n kwargs = {} if f_kwargs is None else f_kwargs\n if callable(func):\n remote_task_future = client.submit(func, *args, pure=pure, **kwargs)\n else:\n # for the case where type(func) is distributed.Future\n remote_task_future = client.submit(\n _deploy_dask_func,\n func,\n *args,\n pure=pure,\n return_pandas_df=return_pandas_df,\n **kwargs,\n )\n if num_returns != 1:\n return [\n client.submit(lambda tup, i: tup[i], remote_task_future, i)\n for i in range(num_returns)\n ]\n return remote_task_future\n\n @classmethod\n def is_future(cls, item):\n \"\"\"\n Check if the item is a Future.\n\n Parameters\n ----------\n item : distributed.Future or object\n Future or object to check.\n\n Returns\n -------\n boolean\n If the value is a future.\n \"\"\"\n return isinstance(item, Future)\n\n @classmethod\n def materialize(cls, future):\n \"\"\"\n Materialize data matching `future` object.\n\n Parameters\n ----------\n future : distributed.Future or list\n Future object of list of future objects whereby data needs to be materialized.\n\n Returns\n -------\n Any\n An object(s) from the distributed memory.\n \"\"\"\n client = get_dask_client()\n return client.gather(future)\n\n @classmethod\n def put(cls, data, **kwargs):\n \"\"\"\n Put data into distributed memory.\n\n Parameters\n ----------\n data : list, dict, or object\n Data to scatter out to workers. Output type matches input type.\n **kwargs : dict\n Additional keyword arguments to be passed in `Client.scatter`.\n\n Returns\n -------\n List, dict, iterator, or queue of futures matching the type of input.\n \"\"\"\n if isinstance(data, dict):\n # there is a bug that looks similar to https://github.com/dask/distributed/issues/3965;\n # to avoid this we could change behaviour for serialization:\n # \n # vs\n # {'sep': , \\\n # 'delimiter': ...\n data = UserDict(data)\n client = get_dask_client()\n return client.scatter(data, **kwargs)\n\n @classmethod\n def wait(cls, obj_ids, num_returns=None):\n \"\"\"\n Wait on the objects without materializing them (blocking operation).\n\n Parameters\n ----------\n obj_ids : list, scalar\n num_returns : int, optional\n \"\"\"\n if not isinstance(obj_ids, list):\n obj_ids = [obj_ids]\n if num_returns is None:\n num_returns = len(obj_ids)\n if num_returns == len(obj_ids):\n wait(obj_ids, return_when=\"ALL_COMPLETED\")\n else:\n # Dask doesn't natively support `num_returns` as int.\n # `wait` function doesn't always return only one finished future,\n # so a simple loop is not enough here\n done, not_done = wait(obj_ids, return_when=\"FIRST_COMPLETED\")\n while len(done) < num_returns and (i := 0 < num_returns):\n extra_done, not_done = wait(not_done, return_when=\"FIRST_COMPLETED\")\n done.update(extra_done)\n i += 1\n"
},
{
"path": "modin/core/execution/dask/common/utils.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses utility function to initialize Dask environment.\"\"\"\n\nimport os\n\nfrom modin.config import (\n CIAWSAccessKeyID,\n CIAWSSecretAccessKey,\n CpuCount,\n DaskThreadsPerWorker,\n GithubCI,\n Memory,\n NPartitions,\n)\nfrom modin.core.execution.utils import set_env\n\n\ndef initialize_dask():\n \"\"\"Initialize Dask environment.\"\"\"\n from distributed.client import default_client\n from distributed.worker import get_worker\n\n try:\n # Check if running within a Dask worker process\n get_worker()\n # If the above line does not raise an error, we are in a worker process\n # and should not create a new client\n return\n except ValueError:\n # Not in a Dask worker, proceed to check for or create a client\n pass\n\n try:\n client = default_client()\n\n def _disable_warnings():\n import warnings\n\n warnings.simplefilter(\"ignore\", category=FutureWarning)\n\n client.run(_disable_warnings)\n\n except ValueError:\n from distributed import Client\n\n num_cpus = CpuCount.get()\n threads_per_worker = DaskThreadsPerWorker.get()\n memory_limit = Memory.get()\n worker_memory_limit = memory_limit // num_cpus if memory_limit else \"auto\"\n\n # when the client is initialized, environment variables are inherited\n with set_env(PYTHONWARNINGS=\"ignore::FutureWarning\"):\n client = Client(\n n_workers=num_cpus,\n threads_per_worker=threads_per_worker,\n memory_limit=worker_memory_limit,\n )\n\n if GithubCI.get():\n # set these keys to run tests that write to the mock s3 service. this seems\n # to be the way to pass environment variables to the workers:\n # https://jacobtomlinson.dev/posts/2021/bio-for-2021/\n access_key = CIAWSAccessKeyID.get()\n aws_secret = CIAWSSecretAccessKey.get()\n client.run(\n lambda: os.environ.update(\n {\n \"AWS_ACCESS_KEY_ID\": access_key,\n \"AWS_SECRET_ACCESS_KEY\": aws_secret,\n }\n )\n )\n\n num_cpus = len(client.ncores())\n NPartitions._put(num_cpus)\n CpuCount._put(num_cpus)\n"
},
{
"path": "modin/core/execution/dask/implementations/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to Dask execution engine and optimized for specific storage formats.\"\"\"\n"
},
{
"path": "modin/core/execution/dask/implementations/pandas_on_dask/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to Dask execution engine and optimized for pandas storage format.\"\"\"\n"
},
{
"path": "modin/core/execution/dask/implementations/pandas_on_dask/dataframe/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe class optimized for pandas on Dask execution.\"\"\"\n\nfrom .dataframe import PandasOnDaskDataframe\n\n__all__ = [\"PandasOnDaskDataframe\"]\n"
},
{
"path": "modin/core/execution/dask/implementations/pandas_on_dask/dataframe/dataframe.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses class that implements ``PandasDataframe``.\"\"\"\n\nfrom modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe\nfrom modin.utils import _inherit_docstrings\n\nfrom ..partitioning.partition_manager import PandasOnDaskDataframePartitionManager\n\n\nclass PandasOnDaskDataframe(PandasDataframe):\n \"\"\"\n The class implements the interface in ``PandasDataframe``.\n\n Parameters\n ----------\n partitions : np.ndarray\n A 2D NumPy array of partitions.\n index : sequence\n The index for the dataframe. Converted to a pandas.Index.\n columns : sequence\n The columns object for the dataframe. Converted to a pandas.Index.\n row_lengths : list, optional\n The length of each partition in the rows. The \"height\" of\n each of the block partitions. Is computed if not provided.\n column_widths : list, optional\n The width of each partition in the columns. The \"width\" of\n each of the block partitions. Is computed if not provided.\n dtypes : pandas.Series, optional\n The data types for the dataframe columns.\n pandas_backend : {\"pyarrow\", None}, optional\n Backend used by pandas. None - means default NumPy backend.\n \"\"\"\n\n _partition_mgr_cls = PandasOnDaskDataframePartitionManager\n\n @classmethod\n def reconnect(cls, address, attributes): # noqa: GL08\n # The main goal is to configure the client for the worker process\n # using the address passed by the custom `__reduce__` function\n try:\n from distributed import default_client\n\n default_client()\n except ValueError:\n from distributed import Client\n\n # setup `default_client` for worker process\n _ = Client(address)\n obj = cls.__new__(cls)\n obj.__dict__.update(attributes)\n return obj\n\n def __reduce__(self): # noqa: GL08\n from distributed import default_client\n\n address = default_client().scheduler_info()[\"address\"]\n return self.reconnect, (address, self.__dict__)\n\n @property\n @_inherit_docstrings(PandasDataframe.engine)\n def engine(self) -> str:\n return \"Dask\"\n"
},
{
"path": "modin/core/execution/dask/implementations/pandas_on_dask/io/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base IO classes optimized for pandas on Dask execution.\"\"\"\n\nfrom .io import PandasOnDaskIO\n\n__all__ = [\n \"PandasOnDaskIO\",\n]\n"
},
{
"path": "modin/core/execution/dask/implementations/pandas_on_dask/io/io.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses class that implements ``BaseIO`` using Dask as an execution engine.\"\"\"\n\nimport numpy as np\nfrom distributed.client import default_client\n\nfrom modin.core.execution.dask.common import DaskWrapper\nfrom modin.core.execution.dask.implementations.pandas_on_dask.dataframe import (\n PandasOnDaskDataframe,\n)\nfrom modin.core.execution.dask.implementations.pandas_on_dask.partitioning import (\n PandasOnDaskDataframePartition,\n)\nfrom modin.core.io import (\n BaseIO,\n CSVDispatcher,\n ExcelDispatcher,\n FeatherDispatcher,\n FWFDispatcher,\n JSONDispatcher,\n ParquetDispatcher,\n SQLDispatcher,\n)\nfrom modin.core.storage_formats.pandas.parsers import (\n PandasCSVParser,\n PandasExcelParser,\n PandasFeatherParser,\n PandasFWFParser,\n PandasJSONParser,\n PandasParquetParser,\n PandasSQLParser,\n)\nfrom modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\nfrom modin.distributed.dataframe.pandas.partitions import (\n from_partitions,\n unwrap_partitions,\n)\nfrom modin.experimental.core.io import (\n ExperimentalCSVGlobDispatcher,\n ExperimentalCustomTextDispatcher,\n ExperimentalGlobDispatcher,\n ExperimentalSQLDispatcher,\n)\nfrom modin.experimental.core.storage_formats.pandas.parsers import (\n ExperimentalCustomTextParser,\n ExperimentalPandasCSVGlobParser,\n ExperimentalPandasJsonParser,\n ExperimentalPandasParquetParser,\n ExperimentalPandasPickleParser,\n ExperimentalPandasXmlParser,\n)\nfrom modin.pandas.series import Series\nfrom modin.utils import MODIN_UNNAMED_SERIES_LABEL\n\n\nclass PandasOnDaskIO(BaseIO):\n \"\"\"The class implements interface in ``BaseIO`` using Dask as an execution engine.\"\"\"\n\n frame_cls = PandasOnDaskDataframe\n frame_partition_cls = PandasOnDaskDataframePartition\n query_compiler_cls = PandasQueryCompiler\n build_args = dict(\n frame_cls=PandasOnDaskDataframe,\n frame_partition_cls=PandasOnDaskDataframePartition,\n query_compiler_cls=PandasQueryCompiler,\n base_io=BaseIO,\n )\n\n def __make_read(*classes, build_args=build_args):\n # used to reduce code duplication\n return type(\"\", (DaskWrapper, *classes), build_args).read\n\n def __make_write(*classes, build_args=build_args):\n # used to reduce code duplication\n return type(\"\", (DaskWrapper, *classes), build_args).write\n\n read_csv = __make_read(PandasCSVParser, CSVDispatcher)\n read_fwf = __make_read(PandasFWFParser, FWFDispatcher)\n read_json = __make_read(PandasJSONParser, JSONDispatcher)\n read_parquet = __make_read(PandasParquetParser, ParquetDispatcher)\n to_parquet = __make_write(ParquetDispatcher)\n # Blocked on pandas-dev/pandas#12236. It is faster to default to pandas.\n # read_hdf = __make_read(PandasHDFParser, HDFReader)\n read_feather = __make_read(PandasFeatherParser, FeatherDispatcher)\n read_sql = __make_read(PandasSQLParser, SQLDispatcher)\n to_sql = __make_write(SQLDispatcher)\n read_excel = __make_read(PandasExcelParser, ExcelDispatcher)\n\n # experimental methods that don't exist in pandas\n read_csv_glob = __make_read(\n ExperimentalPandasCSVGlobParser, ExperimentalCSVGlobDispatcher\n )\n read_parquet_glob = __make_read(\n ExperimentalPandasParquetParser, ExperimentalGlobDispatcher\n )\n to_parquet_glob = __make_write(\n ExperimentalGlobDispatcher,\n build_args={**build_args, \"base_write\": BaseIO.to_parquet},\n )\n read_json_glob = __make_read(\n ExperimentalPandasJsonParser, ExperimentalGlobDispatcher\n )\n to_json_glob = __make_write(\n ExperimentalGlobDispatcher,\n build_args={**build_args, \"base_write\": BaseIO.to_json},\n )\n read_xml_glob = __make_read(ExperimentalPandasXmlParser, ExperimentalGlobDispatcher)\n to_xml_glob = __make_write(\n ExperimentalGlobDispatcher,\n build_args={**build_args, \"base_write\": BaseIO.to_xml},\n )\n read_pickle_glob = __make_read(\n ExperimentalPandasPickleParser, ExperimentalGlobDispatcher\n )\n to_pickle_glob = __make_write(\n ExperimentalGlobDispatcher,\n build_args={**build_args, \"base_write\": BaseIO.to_pickle},\n )\n read_custom_text = __make_read(\n ExperimentalCustomTextParser, ExperimentalCustomTextDispatcher\n )\n read_sql_distributed = __make_read(\n ExperimentalSQLDispatcher, build_args={**build_args, \"base_read\": read_sql}\n )\n\n del __make_read # to not pollute class namespace\n del __make_write # to not pollute class namespace\n\n @classmethod\n def from_dask(cls, dask_obj):\n \"\"\"\n Create a Modin `query_compiler` from a Dask DataFrame.\n\n Parameters\n ----------\n dask_obj : dask.dataframe.DataFrame\n The Dask DataFrame to convert from.\n\n Returns\n -------\n BaseQueryCompiler\n QueryCompiler containing data from the Dask DataFrame.\n \"\"\"\n client = default_client()\n dask_fututures = client.compute(dask_obj.to_delayed())\n modin_df = from_partitions(dask_fututures, axis=0)._query_compiler\n return modin_df\n\n @classmethod\n def to_dask(cls, modin_obj):\n \"\"\"\n Convert a Modin DataFrame/Series to a Dask DataFrame/Series.\n\n Parameters\n ----------\n modin_obj : modin.pandas.DataFrame, modin.pandas.Series\n The Modin DataFrame/Series to convert.\n\n Returns\n -------\n dask.dataframe.DataFrame or dask.dataframe.Series\n Converted object with type depending on input.\n \"\"\"\n from dask.dataframe import from_delayed\n\n partitions = unwrap_partitions(modin_obj, axis=0)\n\n # partiotions must be converted to pandas Series\n if isinstance(modin_obj, Series):\n client = default_client()\n\n def df_to_series(df):\n series = df[df.columns[0]]\n if df.columns[0] == MODIN_UNNAMED_SERIES_LABEL:\n series.name = None\n return series\n\n partitions = [client.submit(df_to_series, part) for part in partitions]\n\n return from_delayed(partitions)\n\n @classmethod\n def from_map(cls, func, iterable, *args, **kwargs):\n \"\"\"\n Create a Modin `query_compiler` from a map function.\n\n This method will construct a Modin `query_compiler` split by row partitions.\n The number of row partitions matches the number of elements in the iterable object.\n\n Parameters\n ----------\n func : callable\n Function to map across the iterable object.\n iterable : Iterable\n An iterable object.\n *args : tuple\n Positional arguments to pass in `func`.\n **kwargs : dict\n Keyword arguments to pass in `func`.\n\n Returns\n -------\n BaseQueryCompiler\n QueryCompiler containing data returned by map function.\n \"\"\"\n func = cls.frame_cls._partition_mgr_cls.preprocess_func(func)\n partitions = np.array(\n [\n [\n cls.frame_partition_cls(\n DaskWrapper.deploy(\n func,\n f_args=(obj,) + args,\n f_kwargs=kwargs,\n return_pandas_df=True,\n )\n )\n ]\n for obj in iterable\n ]\n )\n return cls.query_compiler_cls(cls.frame_cls(partitions))\n"
},
{
"path": "modin/core/execution/dask/implementations/pandas_on_dask/partitioning/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe classes related to its partitioning and optimized for pandas on Dask execution.\"\"\"\n\nfrom .partition import PandasOnDaskDataframePartition\nfrom .partition_manager import PandasOnDaskDataframePartitionManager\nfrom .virtual_partition import (\n PandasOnDaskDataframeColumnPartition,\n PandasOnDaskDataframeRowPartition,\n PandasOnDaskDataframeVirtualPartition,\n)\n\n__all__ = [\n \"PandasOnDaskDataframePartition\",\n \"PandasOnDaskDataframePartitionManager\",\n \"PandasOnDaskDataframeVirtualPartition\",\n \"PandasOnDaskDataframeColumnPartition\",\n \"PandasOnDaskDataframeRowPartition\",\n]\n"
},
{
"path": "modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses class that wraps data (block partition) and its metadata.\"\"\"\n\nimport pandas\nfrom distributed import Future\nfrom distributed.utils import get_ip\n\nfrom modin.core.dataframe.pandas.partitioning.partition import PandasDataframePartition\nfrom modin.core.execution.dask.common import DaskWrapper\nfrom modin.logging import get_logger\nfrom modin.pandas.indexing import compute_sliced_len\n\n\nclass PandasOnDaskDataframePartition(PandasDataframePartition):\n \"\"\"\n The class implements the interface in ``PandasDataframePartition``.\n\n Parameters\n ----------\n data : distributed.Future\n A reference to pandas DataFrame that need to be wrapped with this class.\n length : distributed.Future or int, optional\n Length or reference to it of wrapped pandas DataFrame.\n width : distributed.Future or int, optional\n Width or reference to it of wrapped pandas DataFrame.\n ip : distributed.Future or str, optional\n Node IP address or reference to it that holds wrapped pandas DataFrame.\n call_queue : list, optional\n Call queue that needs to be executed on wrapped pandas DataFrame.\n \"\"\"\n\n execution_wrapper = DaskWrapper\n\n def __init__(self, data, length=None, width=None, ip=None, call_queue=None):\n super().__init__()\n assert isinstance(data, Future)\n self._data = data\n if call_queue is None:\n call_queue = []\n self.call_queue = call_queue\n self._length_cache = length\n self._width_cache = width\n self._ip_cache = ip\n\n log = get_logger()\n self._is_debug(log) and log.debug(\n \"Partition ID: {}, Height: {}, Width: {}, Node IP: {}\".format(\n self._identity,\n str(self._length_cache),\n str(self._width_cache),\n str(self._ip_cache),\n )\n )\n\n def apply(self, func, *args, **kwargs):\n \"\"\"\n Apply a function to the object wrapped by this partition.\n\n Parameters\n ----------\n func : callable or distributed.Future\n A function to apply.\n *args : iterable\n Additional positional arguments to be passed in `func`.\n **kwargs : dict\n Additional keyword arguments to be passed in `func`.\n\n Returns\n -------\n PandasOnDaskDataframePartition\n A new ``PandasOnDaskDataframePartition`` object.\n\n Notes\n -----\n The keyword arguments are sent as a dictionary.\n \"\"\"\n log = get_logger()\n self._is_debug(log) and log.debug(f\"ENTER::Partition.apply::{self._identity}\")\n call_queue = self.call_queue + [[func, args, kwargs]]\n if len(call_queue) > 1:\n self._is_debug(log) and log.debug(\n f\"SUBMIT::_apply_list_of_funcs::{self._identity}\"\n )\n futures = self.execution_wrapper.deploy(\n func=apply_list_of_funcs,\n f_args=(call_queue, self._data),\n num_returns=2,\n pure=False,\n )\n else:\n # We handle `len(call_queue) == 1` in a different way because\n # this improves performance a bit.\n func, f_args, f_kwargs = call_queue[0]\n futures = self.execution_wrapper.deploy(\n func=apply_func,\n f_args=(self._data, func, *f_args),\n f_kwargs=f_kwargs,\n num_returns=2,\n pure=False,\n )\n self._is_debug(log) and log.debug(f\"SUBMIT::_apply_func::{self._identity}\")\n self._is_debug(log) and log.debug(f\"EXIT::Partition.apply::{self._identity}\")\n return self.__constructor__(futures[0], ip=futures[1])\n\n def drain_call_queue(self):\n \"\"\"Execute all operations stored in the call queue on the object wrapped by this partition.\"\"\"\n log = get_logger()\n self._is_debug(log) and log.debug(\n f\"ENTER::Partition.drain_call_queue::{self._identity}\"\n )\n if len(self.call_queue) == 0:\n return\n call_queue = self.call_queue\n if len(call_queue) > 1:\n self._is_debug(log) and log.debug(\n f\"SUBMIT::_apply_list_of_funcs::{self._identity}\"\n )\n futures = self.execution_wrapper.deploy(\n func=apply_list_of_funcs,\n f_args=(call_queue, self._data),\n num_returns=2,\n pure=False,\n )\n else:\n # We handle `len(call_queue) == 1` in a different way because\n # this improves performance a bit.\n func, f_args, f_kwargs = call_queue[0]\n self._is_debug(log) and log.debug(f\"SUBMIT::_apply_func::{self._identity}\")\n futures = self.execution_wrapper.deploy(\n func=apply_func,\n f_args=(self._data, func, *f_args),\n f_kwargs=f_kwargs,\n num_returns=2,\n pure=False,\n )\n self._data = futures[0]\n self._ip_cache = futures[1]\n self._is_debug(log) and log.debug(\n f\"EXIT::Partition.drain_call_queue::{self._identity}\"\n )\n self.call_queue = []\n\n def wait(self):\n \"\"\"Wait completing computations on the object wrapped by the partition.\"\"\"\n self.drain_call_queue()\n self.execution_wrapper.wait(self._data)\n\n def mask(self, row_labels, col_labels):\n \"\"\"\n Lazily create a mask that extracts the indices provided.\n\n Parameters\n ----------\n row_labels : list-like, slice or label\n The row labels for the rows to extract.\n col_labels : list-like, slice or label\n The column labels for the columns to extract.\n\n Returns\n -------\n PandasOnDaskDataframePartition\n A new ``PandasOnDaskDataframePartition`` object.\n \"\"\"\n log = get_logger()\n self._is_debug(log) and log.debug(f\"ENTER::Partition.mask::{self._identity}\")\n new_obj = super().mask(row_labels, col_labels)\n if isinstance(row_labels, slice) and isinstance(self._length_cache, Future):\n if row_labels == slice(None):\n # fast path - full axis take\n new_obj._length_cache = self._length_cache\n else:\n new_obj._length_cache = self.execution_wrapper.deploy(\n func=compute_sliced_len, f_args=(row_labels, self._length_cache)\n )\n if isinstance(col_labels, slice) and isinstance(self._width_cache, Future):\n if col_labels == slice(None):\n # fast path - full axis take\n new_obj._width_cache = self._width_cache\n else:\n new_obj._width_cache = self.execution_wrapper.deploy(\n func=compute_sliced_len, f_args=(col_labels, self._width_cache)\n )\n self._is_debug(log) and log.debug(f\"EXIT::Partition.mask::{self._identity}\")\n return new_obj\n\n def __copy__(self):\n \"\"\"\n Create a copy of this partition.\n\n Returns\n -------\n PandasOnDaskDataframePartition\n A copy of this partition.\n \"\"\"\n return self.__constructor__(\n self._data,\n length=self._length_cache,\n width=self._width_cache,\n ip=self._ip_cache,\n call_queue=self.call_queue,\n )\n\n @classmethod\n def put(cls, obj):\n \"\"\"\n Put an object into distributed memory and wrap it with partition object.\n\n Parameters\n ----------\n obj : any\n An object to be put.\n\n Returns\n -------\n PandasOnDaskDataframePartition\n A new ``PandasOnDaskDataframePartition`` object.\n \"\"\"\n return cls(\n cls.execution_wrapper.put(obj, hash=False),\n len(obj.index),\n len(obj.columns),\n )\n\n @classmethod\n def preprocess_func(cls, func):\n \"\"\"\n Preprocess a function before an ``apply`` call.\n\n Parameters\n ----------\n func : callable\n The function to preprocess.\n\n Returns\n -------\n callable\n An object that can be accepted by ``apply``.\n \"\"\"\n return cls.execution_wrapper.put(func, hash=False, broadcast=True)\n\n def length(self, materialize=True):\n \"\"\"\n Get the length of the object wrapped by this partition.\n\n Parameters\n ----------\n materialize : bool, default: True\n Whether to forcibly materialize the result into an integer. If ``False``\n was specified, may return a future of the result if it hasn't been\n materialized yet.\n\n Returns\n -------\n int or distributed.Future\n The length of the object.\n \"\"\"\n if self._length_cache is None:\n self._length_cache = self.apply(len)._data\n if isinstance(self._length_cache, Future) and materialize:\n self._length_cache = self.execution_wrapper.materialize(self._length_cache)\n return self._length_cache\n\n def width(self, materialize=True):\n \"\"\"\n Get the width of the object wrapped by the partition.\n\n Parameters\n ----------\n materialize : bool, default: True\n Whether to forcibly materialize the result into an integer. If ``False``\n was specified, may return a future of the result if it hasn't been\n materialized yet.\n\n Returns\n -------\n int or distributed.Future\n The width of the object.\n \"\"\"\n if self._width_cache is None:\n self._width_cache = self.apply(lambda df: len(df.columns))._data\n if isinstance(self._width_cache, Future) and materialize:\n self._width_cache = self.execution_wrapper.materialize(self._width_cache)\n return self._width_cache\n\n def ip(self, materialize=True):\n \"\"\"\n Get the node IP address of the object wrapped by this partition.\n\n Parameters\n ----------\n materialize : bool, default: True\n Whether to forcibly materialize the result into an integer. If ``False``\n was specified, may return a future of the result if it hasn't been\n materialized yet.\n\n Returns\n -------\n str\n IP address of the node that holds the data.\n \"\"\"\n if self._ip_cache is None:\n self._ip_cache = self.apply(lambda df: pandas.DataFrame([]))._ip_cache\n if materialize and isinstance(self._ip_cache, Future):\n self._ip_cache = self.execution_wrapper.materialize(self._ip_cache)\n return self._ip_cache\n\n\ndef apply_func(partition, func, *args, **kwargs):\n \"\"\"\n Execute a function on the partition in a worker process.\n\n Parameters\n ----------\n partition : pandas.DataFrame\n A pandas DataFrame the function needs to be executed on.\n func : callable\n The function to perform.\n *args : list\n Positional arguments to pass to ``func``.\n **kwargs : dict\n Keyword arguments to pass to ``func``.\n\n Returns\n -------\n pandas.DataFrame\n The resulting pandas DataFrame.\n str\n The node IP address of the worker process.\n\n Notes\n -----\n Directly passing a call queue entry (i.e. a list of [func, args, kwargs]) instead of\n destructuring it causes a performance penalty.\n \"\"\"\n result = func(partition, *args, **kwargs)\n return result, get_ip()\n\n\ndef apply_list_of_funcs(call_queue, partition):\n \"\"\"\n Execute all operations stored in the call queue on the partition in a worker process.\n\n Parameters\n ----------\n call_queue : list\n A call queue of ``[func, args, kwargs]`` triples that needs to be executed on the partition.\n partition : pandas.DataFrame\n A pandas DataFrame the call queue needs to be executed on.\n\n Returns\n -------\n pandas.DataFrame\n The resulting pandas DataFrame.\n str\n The node IP address of the worker process.\n \"\"\"\n for func, f_args, f_kwargs in call_queue:\n partition = func(partition, *f_args, **f_kwargs)\n return partition, get_ip()\n"
},
{
"path": "modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition_manager.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses class that implements ``PandasDataframePartitionManager``.\"\"\"\n\nfrom modin.core.dataframe.pandas.partitioning.partition_manager import (\n PandasDataframePartitionManager,\n)\nfrom modin.core.execution.dask.common import DaskWrapper\n\nfrom .partition import PandasOnDaskDataframePartition\nfrom .virtual_partition import (\n PandasOnDaskDataframeColumnPartition,\n PandasOnDaskDataframeRowPartition,\n)\n\n\nclass PandasOnDaskDataframePartitionManager(PandasDataframePartitionManager):\n \"\"\"The class implements the interface in `PandasDataframePartitionManager`.\"\"\"\n\n # This object uses PandasOnDaskDataframePartition objects as the underlying store.\n _partition_class = PandasOnDaskDataframePartition\n _column_partitions_class = PandasOnDaskDataframeColumnPartition\n _row_partition_class = PandasOnDaskDataframeRowPartition\n _execution_wrapper = DaskWrapper\n\n @classmethod\n def wait_partitions(cls, partitions):\n \"\"\"\n Wait on the objects wrapped by `partitions` in parallel, without materializing them.\n\n This method will block until all computations in the list have completed.\n\n Parameters\n ----------\n partitions : np.ndarray\n NumPy array with ``PandasDataframePartition``-s.\n \"\"\"\n cls._execution_wrapper.wait(\n [block for partition in partitions for block in partition.list_of_blocks]\n )\n"
},
{
"path": "modin/core/execution/dask/implementations/pandas_on_dask/partitioning/virtual_partition.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses classes responsible for storing a virtual partition and applying a function to it.\"\"\"\n\nimport pandas\nfrom distributed.utils import get_ip\n\nfrom modin.core.dataframe.pandas.partitioning.axis_partition import (\n PandasDataframeAxisPartition,\n)\nfrom modin.core.execution.dask.common import DaskWrapper\nfrom modin.utils import _inherit_docstrings\n\nfrom .partition import PandasOnDaskDataframePartition\n\n\nclass PandasOnDaskDataframeVirtualPartition(PandasDataframeAxisPartition):\n \"\"\"\n The class implements the interface in ``PandasDataframeAxisPartition``.\n\n Parameters\n ----------\n list_of_partitions : Union[list, PandasOnDaskDataframePartition]\n List of ``PandasOnDaskDataframePartition`` and\n ``PandasOnDaskDataframeVirtualPartition`` objects, or a single\n ``PandasOnDaskDataframePartition``.\n get_ip : bool, default: False\n Whether to get node IP addresses of conforming partitions or not.\n full_axis : bool, default: True\n Whether or not the virtual partition encompasses the whole axis.\n call_queue : list, optional\n A list of tuples (callable, args, kwargs) that contains deferred calls.\n length : distributed.Future or int, optional\n Length, or reference to length, of wrapped ``pandas.DataFrame``.\n width : distributed.Future or int, optional\n Width, or reference to width, of wrapped ``pandas.DataFrame``.\n \"\"\"\n\n axis = None\n _PARTITIONS_METADATA_LEN = 3 # (length, width, ip)\n partition_type = PandasOnDaskDataframePartition\n\n @property\n def list_of_ips(self):\n \"\"\"\n Get the IPs holding the physical objects composing this partition.\n\n Returns\n -------\n List\n A list of IPs as ``distributed.Future`` or str.\n \"\"\"\n # Defer draining call queue until we get the ip address\n result = [None] * len(self.list_of_block_partitions)\n for idx, partition in enumerate(self.list_of_block_partitions):\n partition.drain_call_queue()\n result[idx] = partition.ip(materialize=False)\n return result\n\n @classmethod\n @_inherit_docstrings(PandasDataframeAxisPartition.deploy_splitting_func)\n def deploy_splitting_func(\n cls,\n axis,\n func,\n f_args,\n f_kwargs,\n num_splits,\n *partitions,\n extract_metadata=False,\n ):\n return DaskWrapper.deploy(\n func=_deploy_dask_func,\n f_args=(\n PandasDataframeAxisPartition.deploy_splitting_func,\n axis,\n func,\n f_args,\n f_kwargs,\n num_splits,\n *partitions,\n ),\n f_kwargs={\"extract_metadata\": extract_metadata},\n num_returns=(\n num_splits * (1 + cls._PARTITIONS_METADATA_LEN)\n if extract_metadata\n else num_splits\n ),\n pure=False,\n )\n\n @classmethod\n def deploy_axis_func(\n cls,\n axis,\n func,\n f_args,\n f_kwargs,\n num_splits,\n maintain_partitioning,\n *partitions,\n min_block_size,\n lengths=None,\n manual_partition=False,\n ):\n \"\"\"\n Deploy a function along a full axis.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis to perform the function along.\n func : callable\n The function to perform.\n f_args : list or tuple\n Positional arguments to pass to ``func``.\n f_kwargs : dict\n Keyword arguments to pass to ``func``.\n num_splits : int\n The number of splits to return (see `split_result_of_axis_func_pandas`).\n maintain_partitioning : bool\n If True, keep the old partitioning if possible.\n If False, create a new partition layout.\n *partitions : iterable\n All partitions that make up the full axis (row or column).\n min_block_size : int\n Minimum number of rows/columns in a single split.\n lengths : iterable, default: None\n The list of lengths to shuffle the partition into.\n manual_partition : bool, default: False\n If True, partition the result with `lengths`.\n\n Returns\n -------\n list\n A list of distributed.Future.\n \"\"\"\n result_num_splits = len(lengths) if lengths else num_splits\n return DaskWrapper.deploy(\n func=_deploy_dask_func,\n f_args=(\n PandasDataframeAxisPartition.deploy_axis_func,\n axis,\n func,\n f_args,\n f_kwargs,\n num_splits,\n maintain_partitioning,\n *partitions,\n ),\n f_kwargs={\n \"min_block_size\": min_block_size,\n \"lengths\": lengths,\n \"manual_partition\": manual_partition,\n },\n num_returns=result_num_splits * (1 + cls._PARTITIONS_METADATA_LEN),\n pure=False,\n )\n\n @classmethod\n def deploy_func_between_two_axis_partitions(\n cls,\n axis,\n func,\n f_args,\n f_kwargs,\n num_splits,\n len_of_left,\n other_shape,\n *partitions,\n min_block_size,\n ):\n \"\"\"\n Deploy a function along a full axis between two data sets.\n\n Parameters\n ----------\n axis : {0, 1}\n The axis to perform the function along.\n func : callable\n The function to perform.\n f_args : list or tuple\n Positional arguments to pass to ``func``.\n f_kwargs : dict\n Keyword arguments to pass to ``func``.\n num_splits : int\n The number of splits to return (see `split_result_of_axis_func_pandas`).\n len_of_left : int\n The number of values in `partitions` that belong to the left data set.\n other_shape : np.ndarray\n The shape of right frame in terms of partitions, i.e.\n (other_shape[i-1], other_shape[i]) will indicate slice to restore i-1 axis partition.\n *partitions : iterable\n All partitions that make up the full axis (row or column) for both data sets.\n min_block_size : int\n Minimum number of rows/columns in a single split.\n\n Returns\n -------\n list\n A list of distributed.Future.\n \"\"\"\n return DaskWrapper.deploy(\n func=_deploy_dask_func,\n f_args=(\n PandasDataframeAxisPartition.deploy_func_between_two_axis_partitions,\n axis,\n func,\n f_args,\n f_kwargs,\n num_splits,\n len_of_left,\n other_shape,\n *partitions,\n ),\n f_kwargs={\n \"min_block_size\": min_block_size,\n },\n num_returns=num_splits * (1 + cls._PARTITIONS_METADATA_LEN),\n pure=False,\n )\n\n def wait(self):\n \"\"\"Wait completing computations on the object wrapped by the partition.\"\"\"\n self.drain_call_queue()\n DaskWrapper.wait(self.list_of_blocks)\n\n\n@_inherit_docstrings(PandasOnDaskDataframeVirtualPartition)\nclass PandasOnDaskDataframeColumnPartition(PandasOnDaskDataframeVirtualPartition):\n axis = 0\n\n\n@_inherit_docstrings(PandasOnDaskDataframeVirtualPartition)\nclass PandasOnDaskDataframeRowPartition(PandasOnDaskDataframeVirtualPartition):\n axis = 1\n\n\ndef _deploy_dask_func(\n deployer,\n axis,\n f_to_deploy,\n f_args,\n f_kwargs,\n *args,\n extract_metadata=True,\n **kwargs,\n):\n \"\"\"\n Execute a function on an axis partition in a worker process.\n\n This is ALWAYS called on either ``PandasDataframeAxisPartition.deploy_axis_func``\n or ``PandasDataframeAxisPartition.deploy_func_between_two_axis_partitions``, which both\n serve to deploy another dataframe function on a Dask worker process.\n\n Parameters\n ----------\n deployer : callable\n A `PandasDataFrameAxisPartition.deploy_*` method that will call `deploy_f`.\n axis : {0, 1}\n The axis to perform the function along.\n f_to_deploy : callable or RayObjectID\n The function to deploy.\n f_args : list or tuple\n Positional arguments to pass to ``f_to_deploy``.\n f_kwargs : dict\n Keyword arguments to pass to ``f_to_deploy``.\n *args : list\n Positional arguments to pass to ``func``.\n extract_metadata : bool, default: True\n Whether to return metadata (length, width, ip) of the result. Passing `False` may relax\n the load on object storage as the remote function would return 4 times fewer futures.\n Passing `False` makes sense for temporary results where you know for sure that the\n metadata will never be requested.\n **kwargs : dict\n Keyword arguments to pass to ``func``.\n\n Returns\n -------\n list\n The result of the function ``func`` and metadata for it.\n \"\"\"\n result = deployer(axis, f_to_deploy, f_args, f_kwargs, *args, **kwargs)\n if not extract_metadata:\n return result\n ip = get_ip()\n if isinstance(result, pandas.DataFrame):\n return result, len(result), len(result.columns), ip\n elif all(isinstance(r, pandas.DataFrame) for r in result):\n return [i for r in result for i in [r, len(r), len(r.columns), ip]]\n else:\n return [i for r in result for i in [r, None, None, ip]]\n"
},
{
"path": "modin/core/execution/dispatching/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to dispatching to specific execution.\"\"\"\n"
},
{
"path": "modin/core/execution/dispatching/factories/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Factories responsible for dispatching to specific execution.\"\"\"\n"
},
{
"path": "modin/core/execution/dispatching/factories/dispatcher.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nContain IO dispatcher class.\n\nDispatcher routes the work to execution-specific functions.\n\"\"\"\n\nfrom typing import Union\n\nfrom pandas._libs.lib import NoDefault, no_default\n\nfrom modin.config import Backend, Engine, IsExperimental, StorageFormat\nfrom modin.core.execution.dispatching.factories import factories\nfrom modin.core.storage_formats.base import BaseQueryCompiler\nfrom modin.utils import _inherit_docstrings\n\n\nclass FactoryNotFoundError(AttributeError):\n \"\"\"\n ``FactoryNotFound`` exception class.\n\n Raise when no matching factory could be found.\n \"\"\"\n\n pass\n\n\nclass StubIoEngine(object):\n \"\"\"\n IO-Engine that does nothing more than raise NotImplementedError when any method is called.\n\n Parameters\n ----------\n factory_name : str\n Factory name, which will be reflected in error messages.\n\n Notes\n -----\n Used for testing purposes.\n \"\"\"\n\n def __init__(self, factory_name=\"\"):\n self.factory_name = factory_name or \"Unknown\"\n\n def __getattr__(self, name):\n \"\"\"\n Return a function that raises `NotImplementedError` for the `name` method.\n\n Parameters\n ----------\n name : str\n Method name to indicate in `NotImplementedError`.\n\n Returns\n -------\n callable\n \"\"\"\n\n def stub(*args, **kw):\n raise NotImplementedError(\n f\"Method {self.factory_name}.{name} is not implemented\"\n )\n\n return stub\n\n\nclass StubFactory(factories.BaseFactory):\n \"\"\"\n Factory that does nothing more than raise NotImplementedError when any method is called.\n\n Notes\n -----\n Used for testing purposes.\n \"\"\"\n\n io_cls = StubIoEngine()\n\n @classmethod\n def set_failing_name(cls, factory_name):\n \"\"\"\n Fill in `.io_cls` class attribute with ``StubIoEngine`` engine.\n\n Parameters\n ----------\n factory_name : str\n Name to pass to the ``StubIoEngine`` constructor.\n \"\"\"\n cls.io_cls = StubIoEngine(factory_name)\n return cls\n\n\nclass FactoryDispatcher(object):\n \"\"\"\n Class that routes IO-work to the factories.\n\n This class is responsible for keeping selected factory up-to-date and dispatching\n calls of IO-functions to its actual execution-specific implementations.\n \"\"\"\n\n __factory: factories.BaseFactory = None\n\n @classmethod\n def get_factory(cls) -> factories.BaseFactory:\n \"\"\"Get current factory.\"\"\"\n if cls.__factory is None:\n\n from modin.pandas import _initialize_engine\n\n Engine.subscribe(\n lambda engine_parameter: _initialize_engine(engine_parameter.get())\n )\n Backend.subscribe(cls._update_factory)\n return_value = cls.__factory\n return return_value\n\n @classmethod\n def _get_prepared_factory_for_backend(cls, backend) -> factories.BaseFactory:\n \"\"\"\n Get factory for the specified backend.\n\n Parameters\n ----------\n backend : str\n Backend name.\n\n Returns\n -------\n factories.BaseFactory\n Factory for the specified backend.\n \"\"\"\n execution = Backend.get_execution_for_backend(backend)\n from modin.pandas import _initialize_engine\n\n _initialize_engine(execution.engine)\n factory_name = f\"{execution.storage_format}On{execution.engine}Factory\"\n experimental_factory_name = \"Experimental\" + factory_name\n try:\n factory = getattr(factories, factory_name, None) or getattr(\n factories, experimental_factory_name\n )\n except AttributeError:\n if not IsExperimental.get():\n # allow missing factories in experimental mode only\n msg = (\n \"Cannot find neither factory {} nor experimental factory {}. \"\n + \"Potential reason might be incorrect environment variable value for \"\n + f\"{StorageFormat.varname} or {Engine.varname}\"\n )\n raise FactoryNotFoundError(\n msg.format(factory_name, experimental_factory_name)\n )\n factory = StubFactory.set_failing_name(factory_name)\n else:\n try:\n factory.prepare()\n except ModuleNotFoundError as err:\n raise ModuleNotFoundError(\n f\"Make sure all required packages are installed: {str(err)}\"\n ) from err\n return factory\n\n @classmethod\n def _update_factory(cls, *args):\n \"\"\"\n Update and prepare factory with a new one specified via Modin config.\n\n Parameters\n ----------\n *args : iterable\n This parameters serves the compatibility purpose.\n Does not affect the result.\n \"\"\"\n cls.__factory = cls._get_prepared_factory_for_backend(Backend.get())\n\n @classmethod\n def from_pandas(\n cls, df, backend: Union[str, NoDefault] = no_default\n ) -> BaseQueryCompiler:\n \"\"\"\n Create a Modin query compiler from a pandas DataFrame.\n\n Parameters\n ----------\n df : pandas.DataFrame\n The pandas DataFrame to convert.\n backend : str or NoDefault, default: NoDefault\n The backend to use for the resulting query compiler. If NoDefault,\n use the current global default ``Backend`` from the Modin config.\n\n Returns\n -------\n BaseQueryCompiler\n A Modin query compiler that wraps the input pandas DataFrame.\n \"\"\"\n return (\n cls.get_factory()\n if backend is no_default\n else cls._get_prepared_factory_for_backend(backend)\n )._from_pandas(df)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._from_arrow)\n def from_arrow(cls, at):\n return cls.get_factory()._from_arrow(at)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._from_non_pandas)\n def from_non_pandas(cls, *args, **kwargs):\n return cls.get_factory()._from_non_pandas(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._from_interchange_dataframe)\n def from_interchange_dataframe(cls, *args, **kwargs):\n return cls.get_factory()._from_interchange_dataframe(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._from_ray)\n def from_ray(cls, ray_obj):\n return cls.get_factory()._from_ray(ray_obj)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._from_dask)\n def from_dask(cls, dask_obj):\n return cls.get_factory()._from_dask(dask_obj)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._from_map)\n def from_map(cls, func, iterable, *args, **kwargs):\n return cls.get_factory()._from_map(func, iterable, *args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_parquet)\n def read_parquet(cls, **kwargs):\n return cls.get_factory()._read_parquet(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_csv)\n def read_csv(cls, **kwargs):\n return cls.get_factory()._read_csv(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._read_csv_glob)\n def read_csv_glob(cls, **kwargs):\n return cls.get_factory()._read_csv_glob(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._read_pickle_glob)\n def read_pickle_glob(cls, **kwargs):\n return cls.get_factory()._read_pickle_glob(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_json)\n def read_json(cls, **kwargs):\n return cls.get_factory()._read_json(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_gbq)\n def read_gbq(cls, **kwargs):\n return cls.get_factory()._read_gbq(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_html)\n def read_html(cls, **kwargs):\n return cls.get_factory()._read_html(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_clipboard)\n def read_clipboard(cls, **kwargs):\n return cls.get_factory()._read_clipboard(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_excel)\n def read_excel(cls, **kwargs):\n return cls.get_factory()._read_excel(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_hdf)\n def read_hdf(cls, **kwargs):\n return cls.get_factory()._read_hdf(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_feather)\n def read_feather(cls, **kwargs):\n return cls.get_factory()._read_feather(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_stata)\n def read_stata(cls, **kwargs):\n return cls.get_factory()._read_stata(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_sas)\n def read_sas(cls, **kwargs): # pragma: no cover\n return cls.get_factory()._read_sas(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_pickle)\n def read_pickle(cls, **kwargs):\n return cls.get_factory()._read_pickle(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_sql)\n def read_sql(cls, **kwargs):\n return cls.get_factory()._read_sql(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._read_sql_distributed)\n def read_sql_distributed(cls, **kwargs):\n return cls.get_factory()._read_sql_distributed(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_fwf)\n def read_fwf(cls, **kwargs):\n return cls.get_factory()._read_fwf(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_sql_table)\n def read_sql_table(cls, **kwargs):\n return cls.get_factory()._read_sql_table(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_sql_query)\n def read_sql_query(cls, **kwargs):\n return cls.get_factory()._read_sql_query(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._read_spss)\n def read_spss(cls, **kwargs):\n return cls.get_factory()._read_spss(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._to_sql)\n def to_sql(cls, *args, **kwargs):\n return cls.get_factory()._to_sql(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._to_pickle)\n def to_pickle(cls, *args, **kwargs):\n return cls.get_factory()._to_pickle(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._to_pickle_glob)\n def to_pickle_glob(cls, *args, **kwargs):\n return cls.get_factory()._to_pickle_glob(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._read_parquet_glob)\n def read_parquet_glob(cls, *args, **kwargs):\n return cls.get_factory()._read_parquet_glob(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._to_parquet_glob)\n def to_parquet_glob(cls, *args, **kwargs):\n return cls.get_factory()._to_parquet_glob(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._read_json_glob)\n def read_json_glob(cls, *args, **kwargs):\n return cls.get_factory()._read_json_glob(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._to_json_glob)\n def to_json_glob(cls, *args, **kwargs):\n return cls.get_factory()._to_json_glob(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._read_xml_glob)\n def read_xml_glob(cls, *args, **kwargs):\n return cls.get_factory()._read_xml_glob(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._to_xml_glob)\n def to_xml_glob(cls, *args, **kwargs):\n return cls.get_factory()._to_xml_glob(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.PandasOnRayFactory._read_custom_text)\n def read_custom_text(cls, **kwargs):\n return cls.get_factory()._read_custom_text(**kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._to_csv)\n def to_csv(cls, *args, **kwargs):\n return cls.get_factory()._to_csv(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._to_json)\n def to_json(cls, *args, **kwargs):\n return cls.get_factory()._to_json(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._to_json)\n def to_json_series(cls, *args, **kwargs):\n return cls.get_factory()._to_json_series(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._to_xml)\n def to_xml(cls, *args, **kwargs):\n return cls.get_factory()._to_xml(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._to_parquet)\n def to_parquet(cls, *args, **kwargs):\n return cls.get_factory()._to_parquet(*args, **kwargs)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._to_ray)\n def to_ray(cls, modin_obj):\n return cls.get_factory()._to_ray(modin_obj)\n\n @classmethod\n @_inherit_docstrings(factories.BaseFactory._to_dask)\n def to_dask(cls, modin_obj):\n return cls.get_factory()._to_dask(modin_obj)\n"
},
{
"path": "modin/core/execution/dispatching/factories/factories.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nModule contains Factories for all of the supported Modin executions.\n\nFactory is a bridge between calls of IO function from high-level API and its\nactual implementation in the execution, bound to that factory. Each execution is represented\nwith a Factory class.\n\"\"\"\n\nimport re\nimport typing\nimport warnings\n\nimport pandas\nfrom pandas.util._decorators import doc\n\nfrom modin.core.io import BaseIO\nfrom modin.core.storage_formats.pandas.native_query_compiler import NativeQueryCompiler\nfrom modin.utils import get_current_execution\n\n_doc_abstract_factory_class = \"\"\"\nAbstract {role} factory which allows to override the IO module easily.\n\nThis class is responsible for dispatching calls of IO-functions to its\nactual execution-specific implementations.\n\nAttributes\n----------\nio_cls : BaseIO\n IO module class of the underlying execution. The place to dispatch calls to.\n\"\"\"\n\n_doc_factory_class = \"\"\"\nFactory of {execution_name} execution.\n\nThis class is responsible for dispatching calls of IO-functions to its\nactual execution-specific implementations.\n\nAttributes\n----------\nio_cls : {execution_name}IO\n IO module class of the underlying execution. The place to dispatch calls to.\n\"\"\"\n\n_doc_factory_prepare_method = \"\"\"\nInitialize Factory.\n\nFills in `.io_cls` class attribute with {io_module_name} lazily.\n\"\"\"\n\n_doc_io_method_raw_template = \"\"\"\nBuild query compiler from {source}.\n\nParameters\n----------\n{params}\n\nReturns\n-------\nQueryCompiler\n Query compiler of the selected storage format.\n\"\"\"\n\n_doc_io_method_template = (\n _doc_io_method_raw_template\n + \"\"\"\nSee Also\n--------\nmodin.pandas.{method}\n\"\"\"\n)\n\n_doc_io_method_all_params = \"\"\"*args : args\n Arguments to pass to the QueryCompiler builder method.\n**kwargs : kwargs\n Arguments to pass to the QueryCompiler builder method.\"\"\"\n\n_doc_io_method_kwargs_params = \"\"\"**kwargs : kwargs\n Arguments to pass to the QueryCompiler builder method.\"\"\"\n\n\ntypes_dictionary = {\"pandas\": {\"category\": pandas.CategoricalDtype}}\n\nsupported_executions = (\n \"PandasOnRay\",\n \"PandasOnUnidist\",\n \"PandasOnDask\",\n)\n\n\nclass FactoryInfo(typing.NamedTuple):\n \"\"\"\n Structure that stores information about factory.\n\n Parameters\n ----------\n engine : str\n Name of underlying execution engine.\n partition : str\n Name of the partition format.\n experimental : bool\n Whether underlying engine is experimental-only.\n \"\"\"\n\n engine: str\n partition: str\n experimental: bool\n\n\nclass NotRealFactory(Exception):\n \"\"\"\n ``NotRealFactory`` exception class.\n\n Raise when no matching factory could be found.\n \"\"\"\n\n pass\n\n\n@doc(_doc_abstract_factory_class, role=\"\")\nclass BaseFactory(object):\n io_cls: typing.Type[BaseIO] = None # The module where the I/O functionality exists.\n\n @classmethod\n def get_info(cls) -> FactoryInfo:\n \"\"\"\n Get information about current factory.\n\n Notes\n -----\n It parses factory name, so it must be conformant with how ``FactoryDispatcher``\n class constructs factory names.\n \"\"\"\n try:\n experimental, partition, engine = re.match(\n r\"^(Experimental)?(.*)On(.*)Factory$\", cls.__name__\n ).groups()\n except AttributeError:\n raise NotRealFactory()\n return FactoryInfo(\n engine=engine, partition=partition, experimental=bool(experimental)\n )\n\n @classmethod\n @doc(\n _doc_factory_prepare_method,\n io_module_name=\"an underlying execution's IO-module\",\n )\n def prepare(cls):\n raise NotImplementedError(\"Subclasses of BaseFactory must implement prepare\")\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"pandas DataFrame\",\n params=\"df : pandas.DataFrame\",\n method=\"io.from_pandas\",\n )\n def _from_pandas(cls, df):\n return cls.io_cls.from_pandas(df)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"Arrow Table\",\n params=\"at : pyarrow.Table\",\n method=\"io.from_arrow\",\n )\n def _from_arrow(cls, at):\n return cls.io_cls.from_arrow(at)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a non-pandas object (dict, list, np.array etc...)\",\n params=_doc_io_method_all_params,\n method=\"io.from_non_pandas\",\n )\n def _from_non_pandas(cls, *args, **kwargs):\n return cls.io_cls.from_non_pandas(*args, **kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a DataFrame object supporting exchange protocol `__dataframe__()`\",\n params=_doc_io_method_all_params,\n method=\"io.from_interchange_dataframe\",\n )\n def _from_interchange_dataframe(cls, *args, **kwargs):\n return cls.io_cls.from_interchange_dataframe(*args, **kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a Ray Dataset\",\n params=\"ray_obj : ray.data.Dataset\",\n method=\"modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO.from_ray\",\n )\n def _from_ray(cls, ray_obj):\n return cls.io_cls.from_ray(ray_obj)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a Dask DataFrame\",\n params=\"dask_obj : dask.dataframe.DataFrame\",\n method=\"modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO.from_dask\",\n )\n def _from_dask(cls, dask_obj):\n return cls.io_cls.from_dask(dask_obj)\n\n @classmethod\n def _from_map(cls, func, iterable, *args, **kwargs):\n \"\"\"\n Create a Modin `query_compiler` from a map function.\n\n This method will construct a Modin `query_compiler` split by row partitions.\n The number of row partitions matches the number of elements in the iterable object.\n\n Parameters\n ----------\n func : callable\n Function to map across the iterable object.\n iterable : Iterable\n An iterable object.\n *args : tuple\n Positional arguments to pass in `func`.\n **kwargs : dict\n Keyword arguments to pass in `func`.\n\n Returns\n -------\n BaseQueryCompiler\n QueryCompiler containing data returned by map function.\n \"\"\"\n return cls.io_cls.from_map(func, iterable, *args, **kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a Parquet file\",\n params=_doc_io_method_kwargs_params,\n method=\"read_parquet\",\n )\n def _read_parquet(cls, **kwargs):\n return cls.io_cls.read_parquet(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a CSV file\",\n params=_doc_io_method_kwargs_params,\n method=\"read_csv\",\n )\n def _read_csv(cls, **kwargs):\n return cls.io_cls.read_csv(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a JSON file\",\n params=_doc_io_method_kwargs_params,\n method=\"read_json\",\n )\n def _read_json(cls, **kwargs):\n return cls.io_cls.read_json(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a Google BigQuery\",\n params=_doc_io_method_kwargs_params,\n method=\"read_gbq\",\n )\n def _read_gbq(cls, **kwargs):\n return cls.io_cls.read_gbq(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"an HTML document\",\n params=_doc_io_method_kwargs_params,\n method=\"read_html\",\n )\n def _read_html(cls, **kwargs):\n return cls.io_cls.read_html(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"clipboard\",\n params=_doc_io_method_kwargs_params,\n method=\"read_clipboard\",\n )\n def _read_clipboard(cls, **kwargs): # pragma: no cover\n return cls.io_cls.read_clipboard(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"an Excel file\",\n params=_doc_io_method_kwargs_params,\n method=\"read_excel\",\n )\n def _read_excel(cls, **kwargs):\n return cls.io_cls.read_excel(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"an HDFStore\",\n params=_doc_io_method_kwargs_params,\n method=\"read_hdf\",\n )\n def _read_hdf(cls, **kwargs):\n return cls.io_cls.read_hdf(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a feather-format object\",\n params=_doc_io_method_kwargs_params,\n method=\"read_feather\",\n )\n def _read_feather(cls, **kwargs):\n return cls.io_cls.read_feather(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a Stata file\",\n params=_doc_io_method_kwargs_params,\n method=\"read_stata\",\n )\n def _read_stata(cls, **kwargs):\n return cls.io_cls.read_stata(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a SAS file\",\n params=_doc_io_method_kwargs_params,\n method=\"read_sas\",\n )\n def _read_sas(cls, **kwargs): # pragma: no cover\n return cls.io_cls.read_sas(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a pickled Modin or pandas DataFrame\",\n params=_doc_io_method_kwargs_params,\n method=\"read_pickle\",\n )\n def _read_pickle(cls, **kwargs):\n return cls.io_cls.read_pickle(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a SQL query or database table\",\n params=_doc_io_method_kwargs_params,\n method=\"read_sql\",\n )\n def _read_sql(cls, **kwargs):\n return cls.io_cls.read_sql(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a table of fixed-width formatted lines\",\n params=_doc_io_method_kwargs_params,\n method=\"read_fwf\",\n )\n def _read_fwf(cls, **kwargs):\n return cls.io_cls.read_fwf(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a SQL database table\",\n params=_doc_io_method_kwargs_params,\n method=\"read_sql_table\",\n )\n def _read_sql_table(cls, **kwargs):\n return cls.io_cls.read_sql_table(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"a SQL query\",\n params=_doc_io_method_kwargs_params,\n method=\"read_sql_query\",\n )\n def _read_sql_query(cls, **kwargs):\n return cls.io_cls.read_sql_query(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_template,\n source=\"an SPSS file\",\n params=_doc_io_method_kwargs_params,\n method=\"read_spss\",\n )\n def _read_spss(cls, **kwargs):\n return cls.io_cls.read_spss(**kwargs)\n\n @classmethod\n def _to_sql(cls, *args, **kwargs):\n \"\"\"\n Write query compiler content to a SQL database.\n\n Parameters\n ----------\n *args : args\n Arguments to the writer method.\n **kwargs : kwargs\n Arguments to the writer method.\n \"\"\"\n return cls.io_cls.to_sql(*args, **kwargs)\n\n @classmethod\n def _to_pickle(cls, *args, **kwargs):\n \"\"\"\n Pickle query compiler object.\n\n Parameters\n ----------\n *args : args\n Arguments to the writer method.\n **kwargs : kwargs\n Arguments to the writer method.\n \"\"\"\n return cls.io_cls.to_pickle(*args, **kwargs)\n\n @classmethod\n def _to_csv(cls, *args, **kwargs):\n \"\"\"\n Write query compiler content to a CSV file.\n\n Parameters\n ----------\n *args : args\n Arguments to pass to the writer method.\n **kwargs : kwargs\n Arguments to pass to the writer method.\n \"\"\"\n return cls.io_cls.to_csv(*args, **kwargs)\n\n @classmethod\n def _to_json(cls, *args, **kwargs):\n \"\"\"\n Write query compiler content to a JSON file.\n\n Parameters\n ----------\n *args : args\n Arguments to pass to the writer method.\n **kwargs : kwargs\n Arguments to pass to the writer method.\n \"\"\"\n return cls.io_cls.to_json(*args, **kwargs)\n\n @classmethod\n def _to_json_series(cls, *args, **kwargs):\n \"\"\"\n Write query compiler content of a Series to a JSON file.\n\n Parameters\n ----------\n *args : args\n Arguments to pass to the writer method.\n **kwargs : kwargs\n Arguments to pass to the writer method.\n \"\"\"\n return cls.io_cls.to_json_series(*args, **kwargs)\n\n @classmethod\n def _to_xml(cls, *args, **kwargs):\n \"\"\"\n Write query compiler content to a XML file.\n\n Parameters\n ----------\n *args : args\n Arguments to pass to the writer method.\n **kwargs : kwargs\n Arguments to pass to the writer method.\n \"\"\"\n return cls.io_cls.to_xml(*args, **kwargs)\n\n @classmethod\n def _to_parquet(cls, *args, **kwargs):\n \"\"\"\n Write query compiler content to a parquet file.\n\n Parameters\n ----------\n *args : args\n Arguments to pass to the writer method.\n **kwargs : kwargs\n Arguments to pass to the writer method.\n \"\"\"\n return cls.io_cls.to_parquet(*args, **kwargs)\n\n @classmethod\n def _to_ray(cls, modin_obj):\n \"\"\"\n Write query compiler content to a Ray Dataset.\n\n Parameters\n ----------\n modin_obj : modin.pandas.DataFrame, modin.pandas.Series\n The Modin DataFrame/Series to write.\n\n Returns\n -------\n ray.data.Dataset\n A Ray Dataset object.\n\n Notes\n -----\n Modin DataFrame/Series can only be converted to a Ray Dataset if Modin uses a Ray engine.\n \"\"\"\n return cls.io_cls.to_ray(modin_obj)\n\n @classmethod\n def _to_dask(cls, modin_obj):\n \"\"\"\n Write query compiler content to a Dask DataFrame/Series.\n\n Parameters\n ----------\n modin_obj : modin.pandas.DataFrame, modin.pandas.Series\n The Modin DataFrame/Series to write.\n\n Returns\n -------\n dask.dataframe.DataFrame or dask.dataframe.Series\n A Dask DataFrame/Series object.\n\n Notes\n -----\n Modin DataFrame/Series can only be converted to a Dask DataFrame/Series if Modin uses a Dask engine.\n \"\"\"\n return cls.io_cls.to_dask(modin_obj)\n\n # experimental methods that don't exist in pandas\n @classmethod\n @doc(\n _doc_io_method_raw_template,\n source=\"CSV files\",\n params=_doc_io_method_kwargs_params,\n )\n def _read_csv_glob(cls, **kwargs):\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_read_csv_glob()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.read_csv_glob(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_raw_template,\n source=\"Pickle files\",\n params=_doc_io_method_kwargs_params,\n )\n def _read_pickle_glob(cls, **kwargs):\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_read_pickle_glob()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.read_pickle_glob(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_raw_template,\n source=\"SQL files\",\n params=_doc_io_method_kwargs_params,\n )\n def _read_sql_distributed(cls, **kwargs):\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n extra_parameters = (\n \"partition_column\",\n \"lower_bound\",\n \"upper_bound\",\n \"max_sessions\",\n )\n if any(\n param in kwargs and kwargs[param] is not None\n for param in extra_parameters\n ):\n warnings.warn(\n f\"Distributed read_sql() was only implemented for {', '.join(supported_executions)} executions.\"\n )\n for param in extra_parameters:\n del kwargs[param]\n return cls.io_cls.read_sql(**kwargs)\n return cls.io_cls.read_sql_distributed(**kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_raw_template,\n source=\"Custom text files\",\n params=_doc_io_method_kwargs_params,\n )\n def _read_custom_text(cls, **kwargs):\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_read_custom_text()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.read_custom_text(**kwargs)\n\n @classmethod\n def _to_pickle_glob(cls, *args, **kwargs):\n \"\"\"\n Distributed pickle query compiler object.\n\n Parameters\n ----------\n *args : args\n Arguments to the writer method.\n **kwargs : kwargs\n Arguments to the writer method.\n \"\"\"\n # TODO(https://github.com/modin-project/modin/issues/7429): Use\n # frame-level execution instead of the global, default execution.\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_to_pickle_glob()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.to_pickle_glob(*args, **kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_raw_template,\n source=\"Parquet files\",\n params=_doc_io_method_kwargs_params,\n )\n def _read_parquet_glob(cls, **kwargs):\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_read_parquet_glob()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.read_parquet_glob(**kwargs)\n\n @classmethod\n def _to_parquet_glob(cls, *args, **kwargs):\n \"\"\"\n Write query compiler content to several parquet files.\n\n Parameters\n ----------\n *args : args\n Arguments to pass to the writer method.\n **kwargs : kwargs\n Arguments to pass to the writer method.\n \"\"\"\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_to_parquet_glob()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.to_parquet_glob(*args, **kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_raw_template,\n source=\"Json files\",\n params=_doc_io_method_kwargs_params,\n )\n def _read_json_glob(cls, **kwargs):\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_read_json_glob()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.read_json_glob(**kwargs)\n\n @classmethod\n def _to_json_glob(cls, *args, **kwargs):\n \"\"\"\n Write query compiler content to several json files.\n\n Parameters\n ----------\n *args : args\n Arguments to pass to the writer method.\n **kwargs : kwargs\n Arguments to pass to the writer method.\n \"\"\"\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_to_json_glob()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.to_json_glob(*args, **kwargs)\n\n @classmethod\n @doc(\n _doc_io_method_raw_template,\n source=\"XML files\",\n params=_doc_io_method_kwargs_params,\n )\n def _read_xml_glob(cls, **kwargs):\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_read_xml_glob()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.read_xml_glob(**kwargs)\n\n @classmethod\n def _to_xml_glob(cls, *args, **kwargs):\n \"\"\"\n Write query compiler content to several XML files.\n\n Parameters\n ----------\n *args : args\n Arguments to pass to the writer method.\n **kwargs : kwargs\n Arguments to pass to the writer method.\n \"\"\"\n current_execution = get_current_execution()\n if current_execution not in supported_executions:\n raise NotImplementedError(\n f\"`_to_xml_glob()` is not implemented for {current_execution} execution.\"\n )\n return cls.io_cls.to_xml_glob(*args, **kwargs)\n\n\n@doc(_doc_factory_class, execution_name=\"PandasOnRay\")\nclass PandasOnRayFactory(BaseFactory):\n @classmethod\n @doc(_doc_factory_prepare_method, io_module_name=\"``PandasOnRayIO``\")\n def prepare(cls):\n from modin.core.execution.ray.implementations.pandas_on_ray.io import (\n PandasOnRayIO,\n )\n\n cls.io_cls = PandasOnRayIO\n\n\n@doc(_doc_factory_class, execution_name=\"PandasOnPython\")\nclass PandasOnPythonFactory(BaseFactory):\n @classmethod\n @doc(_doc_factory_prepare_method, io_module_name=\"``PandasOnPythonIO``\")\n def prepare(cls):\n from modin.core.execution.python.implementations.pandas_on_python.io import (\n PandasOnPythonIO,\n )\n\n cls.io_cls = PandasOnPythonIO\n\n\n@doc(_doc_factory_class, execution_name=\"PandasOnDask\")\nclass PandasOnDaskFactory(BaseFactory):\n @classmethod\n @doc(_doc_factory_prepare_method, io_module_name=\"``PandasOnDaskIO``\")\n def prepare(cls):\n from modin.core.execution.dask.implementations.pandas_on_dask.io import (\n PandasOnDaskIO,\n )\n\n cls.io_cls = PandasOnDaskIO\n\n\n@doc(_doc_factory_class, execution_name=\"PandasOnUnidist\")\nclass PandasOnUnidistFactory(BaseFactory):\n @classmethod\n @doc(_doc_factory_prepare_method, io_module_name=\"``PandasOnUnidistIO``\")\n def prepare(cls):\n from modin.core.execution.unidist.implementations.pandas_on_unidist.io import (\n PandasOnUnidistIO,\n )\n\n cls.io_cls = PandasOnUnidistIO\n\n\nclass NativeIO(BaseIO):\n \"\"\"\n I/O class for native pandas execution.\n\n This class inherits the default function implementations from the\n ``BaseIO`` parent class.\n \"\"\"\n\n _should_warn_on_default_to_pandas: bool = False\n query_compiler_cls = NativeQueryCompiler\n\n\n@doc(_doc_factory_class, execution_name=\"NativeOnNative\")\nclass NativeOnNativeFactory(BaseFactory):\n\n @classmethod\n @doc(_doc_factory_prepare_method, io_module_name=\"`NativeIO`\")\n def prepare(cls):\n cls.io_cls = NativeIO\n"
},
{
"path": "modin/core/execution/modin_aqp.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nThe module for working with displaying progress bars for Modin execution engines.\n\nModin Automatic Query Progress (AQP).\n\"\"\"\n\nimport inspect\nimport os\nimport threading\nimport time\nimport warnings\n\nfrom modin.config import Engine, ProgressBar\n\nprogress_bars = {}\nbar_lock = threading.Lock()\n\n\ndef call_progress_bar(result_parts, line_no):\n \"\"\"\n Attach a progress bar to given `result_parts`.\n\n The progress bar is expected to be shown in a Jupyter Notebook cell.\n\n Parameters\n ----------\n result_parts : list of list of object refs (futures)\n Objects which are being computed for which progress is requested.\n line_no : int\n Line number in the call stack which we're displaying progress for.\n \"\"\"\n with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\")\n try:\n from tqdm.autonotebook import tqdm as tqdm_notebook\n except ImportError:\n raise ImportError(\"Please pip install tqdm to use the progress bar\")\n from IPython import get_ipython\n\n try:\n cell_no = get_ipython().execution_count\n # This happens if we are not in ipython or jupyter.\n # No progress bar is supported in that case.\n except AttributeError:\n return\n pbar_id = f\"{cell_no}-{line_no}\"\n futures = [\n block\n for row in result_parts\n for partition in row\n for block in partition.list_of_blocks\n ]\n bar_format = (\n \"{l_bar}{bar}{r_bar}\"\n if \"DEBUG_PROGRESS_BAR\" in os.environ\n and os.environ[\"DEBUG_PROGRESS_BAR\"] == \"True\"\n else \"{desc}: {percentage:3.0f}%{bar} Elapsed time: {elapsed}, estimated remaining time: {remaining}\"\n )\n bar_lock.acquire()\n if pbar_id in progress_bars:\n if hasattr(progress_bars[pbar_id], \"container\"):\n if hasattr(progress_bars[pbar_id].container.children[0], \"max\"):\n index = 0\n else:\n index = 1\n progress_bars[pbar_id].container.children[index].max = progress_bars[\n pbar_id\n ].container.children[index].max + len(futures)\n progress_bars[pbar_id].total = progress_bars[pbar_id].total + len(futures)\n progress_bars[pbar_id].refresh()\n else:\n progress_bars[pbar_id] = tqdm_notebook(\n total=len(futures),\n desc=\"Estimated completion of line \" + str(line_no),\n bar_format=bar_format,\n )\n bar_lock.release()\n\n threading.Thread(target=_show_time_updates, args=(progress_bars[pbar_id],)).start()\n\n # TODO(https://github.com/modin-project/modin/issues/7429): Use\n # frame-level engine config.\n modin_engine = Engine.get()\n engine_wrapper = None\n if modin_engine == \"Ray\":\n from modin.core.execution.ray.common.engine_wrapper import RayWrapper\n\n engine_wrapper = RayWrapper\n elif modin_engine == \"Unidist\":\n from modin.core.execution.unidist.common.engine_wrapper import UnidistWrapper\n\n engine_wrapper = UnidistWrapper\n else:\n raise NotImplementedError(\n f\"ProgressBar feature is not supported for {modin_engine} engine.\"\n )\n\n for i in range(1, len(futures) + 1):\n engine_wrapper.wait(futures, num_returns=i)\n progress_bars[pbar_id].update(1)\n progress_bars[pbar_id].refresh()\n if progress_bars[pbar_id].n == progress_bars[pbar_id].total:\n progress_bars[pbar_id].close()\n\n\ndef display_time_updates(bar):\n \"\"\"\n Start displaying the progress `bar` in a notebook.\n\n Parameters\n ----------\n bar : tqdm.tqdm\n The progress bar wrapper to display in a notebook cell.\n \"\"\"\n threading.Thread(target=_show_time_updates, args=(bar,)).start()\n\n\ndef _show_time_updates(p_bar):\n \"\"\"\n Refresh displayed progress bar `p_bar` periodically until it is complete.\n\n Parameters\n ----------\n p_bar : tqdm.tqdm\n The progress bar wrapper being displayed to refresh.\n \"\"\"\n while p_bar.total > p_bar.n:\n time.sleep(1)\n if p_bar.total > p_bar.n:\n p_bar.refresh()\n\n\ndef progress_bar_wrapper(f):\n \"\"\"\n Wrap computation function inside a progress bar.\n\n Spawns another thread which displays a progress bar showing\n estimated completion time.\n\n Parameters\n ----------\n f : callable\n The name of the function to be wrapped.\n\n Returns\n -------\n callable\n Decorated version of `f` which reports progress.\n \"\"\"\n from functools import wraps\n\n @wraps(f)\n def magic(*args, **kwargs):\n result_parts = f(*args, **kwargs)\n if ProgressBar.get():\n current_frame = inspect.currentframe()\n function_name = None\n while function_name != \"\":\n (\n filename,\n line_number,\n function_name,\n lines,\n index,\n ) = inspect.getframeinfo(current_frame)\n current_frame = current_frame.f_back\n t = threading.Thread(\n target=call_progress_bar,\n args=(result_parts, line_number),\n )\n t.start()\n # We need to know whether or not we are in a jupyter notebook\n from IPython import get_ipython\n\n try:\n ipy_str = str(type(get_ipython()))\n if \"zmqshell\" not in ipy_str:\n t.join()\n except Exception:\n pass\n return result_parts\n\n return magic\n"
},
{
"path": "modin/core/execution/python/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to Python execution engine.\"\"\"\n"
},
{
"path": "modin/core/execution/python/common/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to Python execution engine.\"\"\"\n\nfrom .engine_wrapper import PythonWrapper\n\n__all__ = [\"PythonWrapper\"]\n"
},
{
"path": "modin/core/execution/python/common/engine_wrapper.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to Python execution engine.\"\"\"\n\n\nclass PythonWrapper:\n \"\"\"Python engine wrapper serving for the compatibility purpose with other engines.\"\"\"\n\n @classmethod\n def deploy(cls, func, f_args=None, f_kwargs=None, num_returns=1):\n \"\"\"\n Run the passed function.\n\n Parameters\n ----------\n func : callable\n f_args : sequence, optional\n Positional arguments to pass to the `func`.\n f_kwargs : dict, optional\n Keyword arguments to pass to the `func`.\n num_returns : int, default: 1\n Number of return values from the `func`.\n\n Returns\n -------\n object\n Returns the result of the `func`.\n \"\"\"\n args = [] if f_args is None else f_args\n kwargs = {} if f_kwargs is None else f_kwargs\n return func(*args, **kwargs)\n\n @classmethod\n def is_future(cls, item):\n \"\"\"\n Check if the item is a Future.\n\n Parameters\n ----------\n item : object\n\n Returns\n -------\n boolean\n Always return false.\n \"\"\"\n return False\n\n @classmethod\n def materialize(cls, obj_id):\n \"\"\"\n Get the data from the data storage.\n\n The method only serves for the compatibility purpose, what it actually\n does is just return the passed value as is.\n\n Parameters\n ----------\n obj_id : object\n\n Returns\n -------\n object\n The passed `obj_id` itself.\n \"\"\"\n return obj_id\n\n @classmethod\n def put(cls, data, **kwargs):\n \"\"\"\n Put data into the data storage.\n\n The method only serves for the compatibility purpose, what it actually\n does is just return the passed value as is.\n\n Parameters\n ----------\n data : object\n **kwargs : dict\n\n Returns\n -------\n object\n The passed `data` itself.\n \"\"\"\n return data\n"
},
{
"path": "modin/core/execution/python/implementations/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to Python execution engine and optimized for specific storage formats.\"\"\"\n"
},
{
"path": "modin/core/execution/python/implementations/pandas_on_python/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to Python execution engine and optimized for pandas storage format.\"\"\"\n"
},
{
"path": "modin/core/execution/python/implementations/pandas_on_python/dataframe/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe class optimized for pandas on Python execution.\"\"\"\n"
},
{
"path": "modin/core/execution/python/implementations/pandas_on_python/dataframe/dataframe.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nModule contains class ``PandasOnPythonDataframe``.\n\n``PandasOnPythonDataframe`` is dataframe class with pandas storage format and Python engine.\n\"\"\"\n\nfrom modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe\nfrom modin.utils import _inherit_docstrings\n\nfrom ..partitioning.partition_manager import PandasOnPythonDataframePartitionManager\n\n\nclass PandasOnPythonDataframe(PandasDataframe):\n \"\"\"\n Class for dataframes with pandas storage format and Python engine.\n\n ``PandasOnPythonDataframe`` doesn't implement any specific interfaces,\n all functionality is inherited from the ``PandasDataframe`` class.\n\n Parameters\n ----------\n partitions : np.ndarray\n A 2D NumPy array of partitions.\n index : sequence\n The index for the dataframe. Converted to a ``pandas.Index``.\n columns : sequence\n The columns object for the dataframe. Converted to a ``pandas.Index``.\n row_lengths : list, optional\n The length of each partition in the rows. The \"height\" of\n each of the block partitions. Is computed if not provided.\n column_widths : list, optional\n The width of each partition in the columns. The \"width\" of\n each of the block partitions. Is computed if not provided.\n dtypes : pandas.Series, optional\n The data types for the dataframe columns.\n pandas_backend : {\"pyarrow\", None}, optional\n Backend used by pandas. None - means default NumPy backend.\n \"\"\"\n\n _partition_mgr_cls = PandasOnPythonDataframePartitionManager\n\n @property\n @_inherit_docstrings(PandasDataframe.engine)\n def engine(self) -> str:\n return \"Python\"\n"
},
{
"path": "modin/core/execution/python/implementations/pandas_on_python/io/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base IO classes optimized for pandas on Python execution.\"\"\"\n\nfrom .io import PandasOnPythonIO\n\n__all__ = [\n \"PandasOnPythonIO\",\n]\n"
},
{
"path": "modin/core/execution/python/implementations/pandas_on_python/io/io.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module for housing IO classes with pandas storage format and Python engine.\"\"\"\n\nfrom modin.core.execution.python.implementations.pandas_on_python.dataframe.dataframe import (\n PandasOnPythonDataframe,\n)\nfrom modin.core.io import BaseIO\nfrom modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n\n\nclass PandasOnPythonIO(BaseIO):\n \"\"\"\n Class for storing IO functions operating on pandas storage format and Python engine.\n\n Inherits default function implementations from ``BaseIO`` parent class.\n \"\"\"\n\n frame_cls = PandasOnPythonDataframe\n query_compiler_cls = PandasQueryCompiler\n"
},
{
"path": "modin/core/execution/python/implementations/pandas_on_python/partitioning/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Base Modin Dataframe classes related to its partitioning and optimized for pandas on Python execution.\"\"\"\n\nfrom .partition import PandasOnPythonDataframePartition\nfrom .partition_manager import PandasOnPythonDataframePartitionManager\nfrom .virtual_partition import (\n PandasOnPythonDataframeAxisPartition,\n PandasOnPythonDataframeColumnPartition,\n PandasOnPythonDataframeRowPartition,\n)\n\n__all__ = [\n \"PandasOnPythonDataframePartition\",\n \"PandasOnPythonDataframePartitionManager\",\n \"PandasOnPythonDataframeAxisPartition\",\n \"PandasOnPythonDataframeColumnPartition\",\n \"PandasOnPythonDataframeRowPartition\",\n]\n"
},
{
"path": "modin/core/execution/python/implementations/pandas_on_python/partitioning/partition.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"The module defines interface for a partition with pandas storage format and Python engine.\"\"\"\n\nimport warnings\n\nfrom modin.core.dataframe.pandas.partitioning.partition import PandasDataframePartition\nfrom modin.core.execution.python.common import PythonWrapper\n\n\nclass PandasOnPythonDataframePartition(PandasDataframePartition):\n \"\"\"\n Partition class with interface for pandas storage format and Python engine.\n\n Class holds the data and metadata for a single partition and implements\n methods of parent abstract class ``PandasDataframePartition``.\n\n Parameters\n ----------\n data : pandas.DataFrame\n ``pandas.DataFrame`` that should be wrapped with this class.\n length : int, optional\n Length of `data` (number of rows in the input dataframe).\n width : int, optional\n Width of `data` (number of columns in the input dataframe).\n call_queue : list, optional\n Call queue of the partition (list with entities that should be called\n before partition materialization).\n\n Notes\n -----\n Objects of this class are treated as immutable by partition manager\n subclasses. There is no logic for updating in-place.\n \"\"\"\n\n execution_wrapper = PythonWrapper\n\n def __init__(self, data, length=None, width=None, call_queue=None):\n super().__init__()\n if hasattr(data, \"copy\"):\n data = data.copy()\n self._data = data\n if call_queue is None:\n call_queue = []\n self.call_queue = call_queue\n self._length_cache = length\n self._width_cache = width\n\n def get(self):\n \"\"\"\n Flush the `call_queue` and return copy of the data.\n\n Returns\n -------\n pandas.DataFrame\n Copy of DataFrame that was wrapped by this partition.\n\n Notes\n -----\n Since this object is a simple wrapper, just return the copy of data.\n \"\"\"\n self.drain_call_queue()\n return self._data.copy() if hasattr(self._data, \"copy\") else self._data\n\n def apply(self, func, *args, **kwargs):\n \"\"\"\n Apply a function to the object wrapped by this partition.\n\n Parameters\n ----------\n func : callable\n Function to apply.\n *args : iterable\n Additional positional arguments to be passed in `func`.\n **kwargs : dict\n Additional keyword arguments to be passed in `func`.\n\n Returns\n -------\n PandasOnPythonDataframePartition\n New ``PandasOnPythonDataframePartition`` object.\n \"\"\"\n\n def call_queue_closure(data, call_queue):\n \"\"\"\n Apply callables from `call_queue` on copy of the `data` and return the result.\n\n Parameters\n ----------\n data : pandas.DataFrame or pandas.Series\n Data to use for computations.\n call_queue : array-like\n Array with callables and it's kwargs to be applied to the `data`.\n\n Returns\n -------\n pandas.DataFrame or pandas.Series\n \"\"\"\n result = data.copy()\n for func, f_args, f_kwargs in call_queue:\n try:\n result = func(result, *f_args, **f_kwargs)\n except Exception as err:\n self.call_queue = []\n raise err\n return result\n\n self._data = call_queue_closure(self._data, self.call_queue)\n self.call_queue = []\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\", category=FutureWarning)\n return self.__constructor__(func(self._data.copy(), *args, **kwargs))\n\n def drain_call_queue(self):\n \"\"\"Execute all operations stored in the call queue on the object wrapped by this partition.\"\"\"\n if len(self.call_queue) == 0:\n return\n self.apply(lambda x: x)\n\n def wait(self):\n \"\"\"\n Wait for completion of computations on the object wrapped by the partition.\n\n Internally will be done by flushing the call queue.\n \"\"\"\n self.drain_call_queue()\n\n @classmethod\n def put(cls, obj):\n \"\"\"\n Create partition containing `obj`.\n\n Parameters\n ----------\n obj : pandas.DataFrame\n DataFrame to be put into the new partition.\n\n Returns\n -------\n PandasOnPythonDataframePartition\n New ``PandasOnPythonDataframePartition`` object.\n \"\"\"\n return cls(obj.copy(), len(obj.index), len(obj.columns))\n\n @classmethod\n def preprocess_func(cls, func):\n \"\"\"\n Preprocess a function before an ``apply`` call.\n\n Parameters\n ----------\n func : callable\n Function to preprocess.\n\n Returns\n -------\n callable\n An object that can be accepted by ``apply``.\n\n Notes\n -----\n No special preprocessing action is required, so unmodified\n `func` will be returned.\n \"\"\"\n return func\n"
},
{
"path": "modin/core/execution/python/implementations/pandas_on_python/partitioning/partition_manager.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module houses class for managing partitions with pandas storage format and Python engine.\"\"\"\n\nfrom modin.core.dataframe.pandas.partitioning.partition_manager import (\n PandasDataframePartitionManager,\n)\nfrom modin.core.execution.python.common import PythonWrapper\n\nfrom .partition import PandasOnPythonDataframePartition\nfrom .virtual_partition import (\n PandasOnPythonDataframeColumnPartition,\n PandasOnPythonDataframeRowPartition,\n)\n\n\nclass PandasOnPythonDataframePartitionManager(PandasDataframePartitionManager):\n \"\"\"\n Class for managing partitions with pandas storage format and Python engine.\n\n Inherits all functionality from ``PandasDataframePartitionManager`` base class.\n \"\"\"\n\n _partition_class = PandasOnPythonDataframePartition\n _column_partitions_class = PandasOnPythonDataframeColumnPartition\n _row_partition_class = PandasOnPythonDataframeRowPartition\n _execution_wrapper = PythonWrapper\n"
},
{
"path": "modin/core/execution/python/implementations/pandas_on_python/partitioning/virtual_partition.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"The module defines interface for a virtual partition with pandas storage format and python engine.\"\"\"\n\nfrom modin.core.dataframe.pandas.partitioning.axis_partition import (\n PandasDataframeAxisPartition,\n)\nfrom modin.utils import _inherit_docstrings\n\nfrom .partition import PandasOnPythonDataframePartition\n\n\nclass PandasOnPythonDataframeAxisPartition(PandasDataframeAxisPartition):\n \"\"\"\n Class defines axis partition interface with pandas storage format and Python engine.\n\n Inherits functionality from ``PandasDataframeAxisPartition`` class.\n\n Parameters\n ----------\n list_of_partitions : Union[list, PandasOnPythonDataframePartition]\n List of ``PandasOnPythonDataframePartition`` and\n ``PandasOnPythonDataframeVirtualPartition`` objects, or a single\n ``PandasOnPythonDataframePartition``.\n get_ip : bool, default: False\n Whether to get node IP addresses to conforming partitions or not.\n full_axis : bool, default: True\n Whether or not the virtual partition encompasses the whole axis.\n call_queue : list, optional\n A list of tuples (callable, args, kwargs) that contains deferred calls.\n length : int, optional\n Length, or reference to length, of wrapped ``pandas.DataFrame``.\n width : int, optional\n Width, or reference to width, of wrapped ``pandas.DataFrame``.\n \"\"\"\n\n partition_type = PandasOnPythonDataframePartition\n\n\n@_inherit_docstrings(PandasOnPythonDataframeAxisPartition)\nclass PandasOnPythonDataframeColumnPartition(PandasOnPythonDataframeAxisPartition):\n axis = 0\n\n\n@_inherit_docstrings(PandasOnPythonDataframeAxisPartition)\nclass PandasOnPythonDataframeRowPartition(PandasOnPythonDataframeAxisPartition):\n axis = 1\n"
},
{
"path": "modin/core/execution/ray/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin's functionality related to Ray execution engine.\"\"\"\n"
},
{
"path": "modin/core/execution/ray/common/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Common utilities for Ray execution engine.\"\"\"\n\nfrom .engine_wrapper import MaterializationHook, RayWrapper, SignalActor\nfrom .utils import initialize_ray\n\n__all__ = [\n \"initialize_ray\",\n \"RayWrapper\",\n \"MaterializationHook\",\n \"SignalActor\",\n]\n"
},
{
"path": "modin/core/execution/ray/common/deferred_execution.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Module with classes and utilities for deferred remote execution in Ray workers.\"\"\"\n\nfrom enum import Enum\nfrom itertools import islice\nfrom typing import (\n Any,\n Callable,\n Dict,\n Generator,\n Iterable,\n List,\n Optional,\n Tuple,\n Union,\n)\n\nimport pandas\nimport ray\nfrom ray._private.services import get_node_ip_address\n\nfrom modin.config import RayTaskCustomResources\nfrom modin.core.execution.ray.common import MaterializationHook, RayWrapper\nfrom modin.logging import get_logger\n\nObjectRefType = Union[ray.ObjectRef, None]\nObjectRefOrListType = Union[ObjectRefType, List[ObjectRefType]]\nListOrTuple = (list, tuple)\n\n\nclass DeferredExecution:\n \"\"\"\n Deferred execution task.\n\n This class represents a single node in the execution tree. The input is either\n an object reference or another node on which this node depends.\n The output is calculated by the specified Callable.\n\n If the input is a DeferredExecution node, it is executed first and the execution\n output is used as the input for this one. All the executions are performed in a\n single batch (i.e. using a single remote call) and the results are saved in all\n the nodes that have multiple subscribers.\n\n Parameters\n ----------\n data : ObjectRefType or DeferredExecution\n The execution input.\n func : callable or ObjectRefType\n A function to be executed.\n args : list or tuple\n Additional positional arguments to be passed in `func`.\n kwargs : dict\n Additional keyword arguments to be passed in `func`.\n num_returns : int, optional\n The number of the return values.\n\n Attributes\n ----------\n data : ObjectRefType or DeferredExecution\n The execution input.\n func : callable or ObjectRefType\n A function to be executed.\n args : list or tuple\n Additional positional arguments to be passed in `func`.\n kwargs : dict\n Additional keyword arguments to be passed in `func`.\n num_returns : int\n The number of the return values.\n flat_args : bool\n True means that there are no lists or DeferredExecution objects in `args`.\n In this case, no arguments processing is performed and `args` is passed\n to the remote method as is.\n flat_kwargs : bool\n The same as `flat_args` but for the `kwargs` values.\n \"\"\"\n\n def __init__(\n self,\n data: Union[\n ObjectRefType,\n \"DeferredExecution\",\n List[Union[ObjectRefType, \"DeferredExecution\"]],\n ],\n func: Union[Callable, ObjectRefType],\n args: Union[List[Any], Tuple[Any]],\n kwargs: Dict[str, Any],\n num_returns=1,\n ):\n if isinstance(data, DeferredExecution):\n data.subscribe()\n self.data = data\n self.func = func\n self.args = args\n self.kwargs = kwargs\n self.num_returns = num_returns\n self.flat_args = self._flat_args(args)\n self.flat_kwargs = self._flat_args(kwargs.values())\n self.subscribers = 0\n\n @classmethod\n def _flat_args(cls, args: Iterable):\n \"\"\"\n Check if the arguments list is flat and subscribe to all `DeferredExecution` objects.\n\n Parameters\n ----------\n args : Iterable\n\n Returns\n -------\n bool\n \"\"\"\n flat = True\n for arg in args:\n if isinstance(arg, DeferredExecution):\n flat = False\n arg.subscribe()\n elif isinstance(arg, ListOrTuple):\n flat = False\n cls._flat_args(arg)\n return flat\n\n def exec(\n self,\n ) -> Tuple[ObjectRefOrListType, Union[\"MetaList\", List], Union[int, List[int]]]:\n \"\"\"\n Execute this task, if required.\n\n Returns\n -------\n tuple\n The execution result, MetaList, containing the length, width and\n the worker's ip address (the last value in the list) and the values\n offset in the list. I.e. length = meta_list[offset],\n width = meta_list[offset + 1], ip = meta_list[-1].\n \"\"\"\n if self.has_result:\n return self.data, self.meta, self.meta_offset\n\n if (\n not isinstance(self.data, DeferredExecution)\n and self.flat_args\n and self.flat_kwargs\n and self.num_returns == 1\n ):\n result, length, width, ip = remote_exec_func.options(\n resources=RayTaskCustomResources.get()\n ).remote(self.func, self.data, *self.args, **self.kwargs)\n meta = MetaList([length, width, ip])\n self._set_result(result, meta, 0)\n return result, meta, 0\n\n # If there are no subscribers, we still need the result here. We don't need to decrement\n # it back. After the execution, the result is saved and the counter has no effect.\n self.subscribers += 2\n consumers, output = self._deconstruct()\n # The last result is the MetaList, so adding +1 here.\n num_returns = sum(c.num_returns for c in consumers) + 1\n results = self._remote_exec_chain(num_returns, *output)\n meta = MetaList(results.pop())\n meta_offset = 0\n results = iter(results)\n for de in consumers:\n if de.num_returns == 1:\n de._set_result(next(results), meta, meta_offset)\n meta_offset += 2\n else:\n res = list(islice(results, num_returns))\n offsets = list(range(0, 2 * num_returns, 2))\n de._set_result(res, meta, offsets)\n meta_offset += 2 * num_returns\n return self.data, self.meta, self.meta_offset\n\n @property\n def has_result(self):\n \"\"\"\n Return true if this task has already been executed and the result is set.\n\n Returns\n -------\n bool\n \"\"\"\n return not hasattr(self, \"func\")\n\n def subscribe(self):\n \"\"\"\n Increment the `subscribers` counter.\n\n Subscriber is any instance that could trigger the execution of this task.\n In case of a multiple subscribers, the execution could be triggerred multiple\n times. To prevent the multiple executions, the execution result is returned\n from the worker and saved in this instance. Subsequent calls to `execute()`\n return the previously saved result.\n \"\"\"\n self.subscribers += 1\n\n def unsubscribe(self):\n \"\"\"Decrement the `subscribers` counter.\"\"\"\n self.subscribers -= 1\n assert self.subscribers >= 0\n\n def _deconstruct(self) -> Tuple[List[\"DeferredExecution\"], List[Any]]:\n \"\"\"\n Convert the specified execution tree to a flat list.\n\n This is required for the automatic Ray object references\n materialization before passing the list to a Ray worker.\n\n The format of the list is the following:\n sequence<
![\"\"](\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/MODIN_ver2_hrz.png?raw=true\")
![\"Twitter](\"https://img.shields.io/twitter/follow/modin_project?style=social\"){kind=small}
| ![](\"https://static.pepy.tech/personalized-badge/modin?period=total&units=international_system&left_color=black&right_color=blue&left_text=Downloads\"){kind=icon}
![](\"https://codecov.io/gh/modin-project/modin/branch/main/graph/badge.svg\"){kind=icon}
\n![](\"https://github.com/modin-project/modin/actions/workflows/push-to-main.yml/badge.svg?branch=main\"){kind=icon}
\n![](\"https://github.com/modin-project/modin/actions/workflows/ci.yml/badge.svg?branch=main\"){kind=icon}
\n![\"PyPI](\"https://badge.fury.io/py/modin.svg\"){kind=icon}
\n![](\"https://img.shields.io/badge/benchmarked%20by-asv-blue.svg\"){kind=icon}
\n![](\"https://github.com/modin-project/modin/raw/main/docs/img/Import.gif\")
\n\nIn the GIFs below, Modin (left) and pandas (right) perform *the same pandas operations* on a 2GB dataset. The only difference between the two notebook examples is the import statement. \n\n![](\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/MODIN_ver2_hrz.png?raw=True\")
![](\"https://pandas.pydata.org/static/img/pandas.svg\")
![](\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Modin.gif\")
![](\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Pandas.gif\")
![](\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Modin_Speedup.svg\")
\n\n### Installation\n\n#### From PyPI\n\nModin can be installed with `pip` on Linux, Windows and MacOS:\n\n```bash\npip install \"modin[all]\" # (Recommended) Install Modin with Ray and Dask engines.\n```\n\nIf you want to install Modin with a specific engine, we recommend:\n\n```bash\npip install \"modin[ray]\" # Install Modin dependencies and Ray.\npip install \"modin[dask]\" # Install Modin dependencies and Dask.\npip install \"modin[mpi]\" # Install Modin dependencies and MPI through unidist.\n```\n\nTo get Modin on MPI through unidist (as of unidist 0.5.0) fully working\nit is required to have a working MPI implementation installed beforehand.\nOtherwise, installation of `modin[mpi]` may fail. Refer to\n[Installing with pip](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-pip)\nsection of the unidist documentation for more details about installation.\n\n**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: `ray` instead of `ray[default]`.\nThis means that the dashboard and cluster launcher are no longer installed by default.\nIf you need those, consider installing `ray[default]` along with `modin[ray]`.\n\nModin automatically detects which engine(s) you have installed and uses that for scheduling computation.\n\n#### From conda-forge\n\nInstalling from [conda forge](https://github.com/conda-forge/modin-feedstock) using `modin-all`\nwill install Modin and three engines: [Ray](https://github.com/ray-project/ray), [Dask](https://github.com/dask/dask) and\n[MPI through unidist](https://github.com/modin-project/unidist).\n\n```bash\nconda install -c conda-forge modin-all\n```\n\nEach engine can also be installed individually (and also as a combination of several engines):\n\n```bash\nconda install -c conda-forge modin-ray # Install Modin dependencies and Ray.\nconda install -c conda-forge modin-dask # Install Modin dependencies and Dask.\nconda install -c conda-forge modin-mpi # Install Modin dependencies and MPI through unidist.\n```\n\n**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: `ray-core` instead of `ray-default`.\nThis means that the dashboard and cluster launcher are no longer installed by default.\nIf you need those, consider installing `ray-default` along with `modin-ray`.\n\nRefer to\n[Installing with conda](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-conda)\nsection of the unidist documentation for more details on how to install a specific MPI implementation to run on.\n\nTo speed up conda installation we recommend using libmamba solver. To do this install it in a base environment:\n\n```bash\nconda install -n base conda-libmamba-solver\n```\n\nand then use it during istallation either like:\n\n```bash\nconda install -c conda-forge modin-ray --experimental-solver=libmamba\n```\n\nor starting from conda 22.11 and libmamba solver 22.12 versions:\n\n```bash\nconda install -c conda-forge modin-ray --solver=libmamba\n```\n\n#### Choosing a Compute Engine\n\nIf you want to choose a specific compute engine to run on, you can set the environment\nvariable `MODIN_ENGINE` and Modin will do computation with that engine:\n\n```bash\nexport MODIN_ENGINE=ray # Modin will use Ray\nexport MODIN_ENGINE=dask # Modin will use Dask\nexport MODIN_ENGINE=unidist # Modin will use Unidist\n```\n\nIf you want to choose the Unidist engine, you should set the additional environment \nvariable ``UNIDIST_BACKEND``. Currently, Modin only supports MPI through unidist:\n\n```bash\nexport UNIDIST_BACKEND=mpi # Unidist will use MPI backend\n```\n\nThis can also be done within a notebook/interpreter before you import Modin:\n\n```python\nimport modin.config as modin_cfg\nimport unidist.config as unidist_cfg\n\nmodin_cfg.Engine.put(\"ray\") # Modin will use Ray\nmodin_cfg.Engine.put(\"dask\") # Modin will use Dask\n\nmodin_cfg.Engine.put('unidist') # Modin will use Unidist\nunidist_cfg.Backend.put('mpi') # Unidist will use MPI backend\n```\n\n_Note: You should not change the engine after your first operation with Modin as it will result in undefined behavior._\n\n#### Which engine should I use?\n\nOn Linux, MacOS, and Windows you can install and use either Ray, Dask or MPI through unidist. There is no knowledge required\nto use either of these engines as Modin abstracts away all of the complexity, so feel\nfree to pick either!\n\n### Pandas API Coverage\n\n
| 
| ![](\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/read_csv_benchmark.png?raw=true\")
\n\nThe `modin.pandas` DataFrame is an extremely light-weight parallel DataFrame.\nModin transparently distributes the data and computation so that you can continue using the same pandas API\nwhile working with more data faster. Because it is so light-weight,\nModin provides speed-ups of up to 4x on a laptop with 4 physical cores.\n\nIn pandas, you are only able to use one core at a time when you are doing computation of\nany kind. With Modin, you are able to use all of the CPU cores on your machine. Even with a\ntraditionally synchronous task like `read_csv`, we see large speedups by efficiently\ndistributing the work across your entire machine.\n\n```python\nimport modin.pandas as pd\n\ndf = pd.read_csv(\"my_dataset.csv\")\n```\n\n#### Modin can handle the datasets that pandas can't \n\nOften data scientists have to switch between different tools\nfor operating on datasets of different sizes. Processing large dataframes with pandas\nis slow, and pandas does not support working with dataframes that are too large to fit\ninto the available memory. As a result, pandas workflows that work well\nfor prototyping on a few MBs of data do not scale to tens or hundreds of GBs (depending on the size\nof your machine). Modin supports operating on data that does not fit in memory, so that you can comfortably\nwork with hundreds of GBs without worrying about substantial slowdown or memory errors.\nWith [cluster](https://modin.readthedocs.io/en/latest/getting_started/using_modin/using_modin_cluster.html)\nand [out of core](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html)\nsupport, Modin is a DataFrame library with both great single-node performance and high\nscalability in a cluster.\n\n#### Modin Architecture\n\nWe designed [Modin's architecture](https://modin.readthedocs.io/en/latest/development/architecture.html)\nto be modular so we can plug in different components as they develop and improve:\n\n![\"Modin's](\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/modin_architecture.png\")
\n\n### Other Resources\n\n#### Getting Started with Modin\n\n- [Documentation](https://modin.readthedocs.io/en/latest/)\n- [10-min Quickstart Guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html)\n- [Examples and Tutorials](https://modin.readthedocs.io/en/latest/getting_started/examples.html)\n- [Videos and Blogposts](https://modin.readthedocs.io/en/latest/getting_started/examples.html#talks-podcasts)\n- [Benchmarking Modin](https://modin.readthedocs.io/en/latest/usage_guide/benchmarking.html)\n\n#### Modin Community\n\n- [Slack](https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA)\n- [Twitter](https://twitter.com/modin_project)\n- [Mailing List](https://groups.google.com/g/modin-dev)\n- [GitHub Issues](https://github.com/modin-project/modin/issues)\n- [StackOverflow](https://stackoverflow.com/questions/tagged/modin)\n\n#### Learn More about Modin\n\n- [Frequently Asked Questions (FAQs)](https://modin.readthedocs.io/en/latest/getting_started/faq.html)\n- [Troubleshooting Guide](https://modin.readthedocs.io/en/latest/getting_started/troubleshooting.html)\n- [Development Guide](https://modin.readthedocs.io/en/latest/development/index.html)\n- Modin is built on many years of research and development at UC Berkeley. Check out these selected papers to learn more about how Modin works:\n - [Flexible Rule-Based Decomposition and Metadata Independence in Modin](https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf) (VLDB 2021)\n - [Dataframe Systems: Theory, Architecture, and Implementation](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-193.pdf) (PhD Dissertation 2021)\n - [Towards Scalable Dataframe Systems](https://arxiv.org/pdf/2001.00888.pdf) (VLDB 2020)\n\n#### Getting Involved\n\n***`modin.pandas` is currently under active development. Requests and contributions are welcome!***\n\nFor more information on how to contribute to Modin, check out the\n[Modin Contribution Guide](https://modin.readthedocs.io/en/latest/development/contributing.html).\n\n### License\n\n[Apache License 2.0](LICENSE)\n"
},
{
"path": "asv_bench/README.md",
"content": "# Modin ASV benchmarks\n\n## Here are some scenarios in which [ASV](https://asv.readthedocs.io/en/stable/index.html) can be used:\n\n* Check the impact of the new patch on the performance of a certain set of operations:\n\n `asv continuous -f 1.05 src/main HEAD -b TimeGroupBy --launch-method=spawn`\n\n* Check for presence of errors inside of benchmarks after changing them or writing new ones:\n\n `asv run --quick --show-stderr --python=same --launch-method=spawn`\n\n* Run entire benchmark suite to get the current times:\n\n `asv run --launch-method=spawn`\n\n* Check the range of commits for performance degradation:\n\n ```\n asv run [start_hash]..[end_hash] --launch-method=spawn\n asv publish\n asv preview\n ```\n\nFor more consistent results, you may need to use the following parameters which\ndescription is in [ASV docs](https://asv.readthedocs.io/en/stable/benchmarks.html?highlight=sample_time#timing-benchmarks):\n\n* `-a sample_time=1`\n* `-a warmup_time=1`\n\n### Notes about using Modin on Ray with Asv:\n\n* `--launch-method=forkserver` is not working;\n* Each set of parameters for each test is launched in its own process, which brings\n a large overhead, since for each process redis server and other necessary processes\n from ray initialization are started and destroyed.\n\n## Adding new benchmark\n\nBasic information on writing benchmarks is present [in ASV documentation](https://asv.readthedocs.io/en/stable/writing_benchmarks.html)\n\nBenchmarks from `benchmarks/benchmarks.py`, `benchmarks/scalability/scalability_benchmarks.py` or `benchmarks/io/csv.py`\ncould be used as a starting point.\n\nRequirements:\n* the benchmark should be able to run both on Modin and on Pandas when the appropriate value\nof the environment variable `MODIN_ASV_USE_IMPL` is selected.\n* the size of the benchmark dataset should depend on the environment variable `MODIN_TEST_DATASET_SIZE`.\n\n## Changing existing benchmark\n\nIt should be remembered that the hash calculated from the benchmark source code is used to display the results.\nWhen changing the benchmark, the old results will no longer be displayed in the dashboard. In general, this is the correct\nbehavior so as not to get a situation when incomparable numbers are displayed in the dashboard.\nBut it should be noted that there could be changes in the source code when it is still correct to compare\nthe \"before\" and \"after\" versions, for example, name of a variable changed, comment added, etc.\nIn this case you must either run a new version of the benchmark for all the commits ever accounted for or manually change\nthe hash in the corresponding result files.\n\n## Pipeline for displaying results in a dashboard\n\nStep 1: checking benchmarks for validity, runs in PRs CI.\n During the test, the benchmarks are run once on small data.\n The implementation can be found in `test-asv-benchmarks` job of [ci.yml](https://github.com/modin-project/modin/blob/main/.github/workflows/ci.yml)\n\nStep 2: running benchmarks with saving the results in [modin-bench@master](https://github.com/modin-project/modin-bench).\n The launch takes place on internal server using specific TeamCity configuration.\n The description of the server can be found in the [\"Benchmark list\"](https://modin.org/modin-bench/#summarylist?sort=0&dir=asc) tab,\n on the left when you hover the mouse over the machine name. \n This step starts as scheduled (now every half hour), subject to the presence of new commits in the Modin `main` branch.\n Command to run benchmarks: `asv run HASHFILE:hashfile.txt --show-stderr --machine xeon-e5 --launch-method=spawn`.\n In the file `hashfile.txt` is the last modin commit hash.\n Writing to a `modin-bench@master` triggers 3 step of the pipeline.\n\nStep 3: converting the results to html representation, which is saved in [modin-bench@gh-pages](https://github.com/modin-project/modin-bench)\n The implementation can be found in `deploy-gh-pages` job of [push.yml](https://github.com/modin-project/modin-bench/blob/master/.github/workflows/push.yml)\n\nBasic actions for step 2:\n* setup environment variable:\n * export MODIN_TEST_DATASET=Big\n * export MODIN_CPUS=44\n* setup git client\n* prepare json file with machine description\n * This file should be placed in the user's home directory.\n * ASV does not always automatically create the file with the description of the machine correctly (e.g. due to being run in a container).\n It is recommended to create a file using [asv machine](https://asv.readthedocs.io/en/stable/commands.html?highlight=machine%20description#asv-machine) command, and manually check the result.\n [Example](https://github.com/modin-project/modin-bench/blob/master/results/xeon-e5/machine.json)\n* copy old result to folder where new result will appear\n (conflict resolution will be performed by ASV itself instead of git)\n* push performance result to modin-bench repository\n"
},
{
"path": "asv_bench/asv.conf.dask.json",
"content": "{\n // The version of the config file format. Do not change, unless\n // you know what you are doing.\n \"version\": 1,\n\n // The name of the project being benchmarked\n \"project\": \"modin\",\n\n // The project's homepage\n \"project_url\": \"https://modin.readthedocs.io/\",\n\n // The URL or local path of the source code repository for the\n // project being benchmarked\n \"repo\": \"..\",\n\n // List of branches to benchmark. If not provided, defaults to \"master\"\n // (for git) or \"default\" (for mercurial).\n \"branches\": [\"main\"],\n\n // Customizable commands for building, installing, and\n // uninstalling the project. See asv.conf.json documentation.\n \"install_command\": [\"in-dir={env_dir} python -mpip install {wheel_file}[dask]\"],\n\n // The tool to use to create environments. May be \"conda\",\n // \"virtualenv\" or other value depending on the plugins in use.\n // If missing or the empty string, the tool will be automatically\n // determined by looking for tools on the PATH environment\n // variable.\n \"environment_type\": \"conda\",\n\n // timeout in seconds for installing any dependencies in environment\n // defaults to 10 min\n \"install_timeout\": 6000,\n\n // the base URL to show a commit for the project.\n \"show_commit_url\": \"https://github.com/modin-project/modin/commit/\",\n\n // The Pythons you'd like to test against. If not provided, defaults\n // to the current version of Python used to run `asv`.\n \"pythons\": [\"3.9\"],\n\n // The list of conda channel names to be searched for benchmark\n // dependency packages in the specified order\n \"conda_channels\": [\"conda-forge\", \"defaults\"],\n\n // The directory (relative to the current directory) to cache the Python\n // environments in. If not provided, defaults to \"env\"\n \"env_dir\": \".asv/env\",\n\n // The directory (relative to the current directory) that raw benchmark\n // results are stored in. If not provided, defaults to \"results\".\n \"results_dir\": \".asv/results\",\n\n // The directory (relative to the current directory) that the html tree\n // should be written to. If not provided, defaults to \"html\".\n \"html_dir\": \".asv/html\",\n}\n"
},
{
"path": "asv_bench/asv.conf.json",
"content": "{\n // The version of the config file format. Do not change, unless\n // you know what you are doing.\n \"version\": 1,\n\n // The name of the project being benchmarked\n \"project\": \"modin\",\n\n // The project's homepage\n \"project_url\": \"https://modin.readthedocs.io/\",\n\n // The URL or local path of the source code repository for the\n // project being benchmarked\n \"repo\": \"..\",\n\n // List of branches to benchmark. If not provided, defaults to \"master\"\n // (for git) or \"default\" (for mercurial).\n \"branches\": [\"main\"],\n\n // Customizable commands for building, installing, and\n // uninstalling the project. See asv.conf.json documentation.\n \"install_command\": [\"in-dir={env_dir} python -mpip install {wheel_file}[ray]\"],\n\n // The tool to use to create environments. May be \"conda\",\n // \"virtualenv\" or other value depending on the plugins in use.\n // If missing or the empty string, the tool will be automatically\n // determined by looking for tools on the PATH environment\n // variable.\n \"environment_type\": \"conda\",\n\n // timeout in seconds for installing any dependencies in environment\n // defaults to 10 min\n \"install_timeout\": 6000,\n\n // the base URL to show a commit for the project.\n \"show_commit_url\": \"https://github.com/modin-project/modin/commit/\",\n\n // The Pythons you'd like to test against. If not provided, defaults\n // to the current version of Python used to run `asv`.\n \"pythons\": [\"3.9\"],\n\n // The list of conda channel names to be searched for benchmark\n // dependency packages in the specified order\n \"conda_channels\": [\"conda-forge\", \"defaults\"],\n\n // The directory (relative to the current directory) to cache the Python\n // environments in. If not provided, defaults to \"env\"\n \"env_dir\": \".asv/env\",\n\n // The directory (relative to the current directory) that raw benchmark\n // results are stored in. If not provided, defaults to \"results\".\n \"results_dir\": \".asv/results\",\n\n // The directory (relative to the current directory) that the html tree\n // should be written to. If not provided, defaults to \"html\".\n \"html_dir\": \".asv/html\",\n}\n"
},
{
"path": "asv_bench/asv.conf.unidist.json",
"content": "{\n // The version of the config file format. Do not change, unless\n // you know what you are doing.\n \"version\": 1,\n\n // The name of the project being benchmarked\n \"project\": \"modin\",\n\n // The project's homepage\n \"project_url\": \"https://modin.readthedocs.io/\",\n\n // The URL or local path of the source code repository for the\n // project being benchmarked\n \"repo\": \"..\",\n\n // List of branches to benchmark. If not provided, defaults to \"master\"\n // (for git) or \"default\" (for mercurial).\n \"branches\": [\"main\"],\n\n // Customizable commands for building, installing, and\n // uninstalling the project. See asv.conf.json documentation.\n \"install_command\": [\"in-dir={env_dir} python -mpip install {wheel_file}[unidist]\"],\n\n // The tool to use to create environments. May be \"conda\",\n // \"virtualenv\" or other value depending on the plugins in use.\n // If missing or the empty string, the tool will be automatically\n // determined by looking for tools on the PATH environment\n // variable.\n \"environment_type\": \"conda\",\n\n // timeout in seconds for installing any dependencies in environment\n // defaults to 10 min\n \"install_timeout\": 6000,\n\n // the base URL to show a commit for the project.\n \"show_commit_url\": \"https://github.com/modin-project/modin/commit/\",\n\n // The Pythons you'd like to test against. If not provided, defaults\n // to the current version of Python used to run `asv`.\n \"pythons\": [\"3.9\"],\n\n // The list of conda channel names to be searched for benchmark\n // dependency packages in the specified order\n \"conda_channels\": [\"conda-forge\", \"defaults\"],\n\n // The directory (relative to the current directory) to cache the Python\n // environments in. If not provided, defaults to \"env\"\n \"env_dir\": \".asv/env\",\n\n // The directory (relative to the current directory) that raw benchmark\n // results are stored in. If not provided, defaults to \"results\".\n \"results_dir\": \".asv/results\",\n\n // The directory (relative to the current directory) that the html tree\n // should be written to. If not provided, defaults to \"html\".\n \"html_dir\": \".asv/html\",\n}\n"
},
{
"path": "asv_bench/benchmarks/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin benchmarks.\"\"\"\n"
},
{
"path": "asv_bench/benchmarks/benchmarks.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"General Modin benchmarks.\"\"\"\n\n# define `MODIN_CPUS` env var to control the number of partitions\n# it should be defined before modin.pandas import (in case of using os.environ)\n\n# define `MODIN_ASV_USE_IMPL` env var to choose library for using in performance\n# measurements\n\nimport math\n\nimport numpy as np\n\nfrom .utils import (\n GROUPBY_NGROUPS,\n IMPL,\n RAND_HIGH,\n RAND_LOW,\n execute,\n gen_nan_data,\n generate_dataframe,\n get_benchmark_shapes,\n random_booleans,\n random_columns,\n random_string,\n translator_groupby_ngroups,\n)\n\n\nclass BaseTimeGroupBy:\n def setup(self, shape, ngroups=5, groupby_ncols=1):\n ngroups = translator_groupby_ngroups(ngroups, shape)\n self.df, self.groupby_columns = generate_dataframe(\n \"int\",\n *shape,\n RAND_LOW,\n RAND_HIGH,\n groupby_ncols,\n count_groups=ngroups,\n )\n\n\nclass TimeGroupByMultiColumn(BaseTimeGroupBy):\n param_names = [\"shape\", \"ngroups\", \"groupby_ncols\"]\n params = [\n get_benchmark_shapes(\"TimeGroupByMultiColumn\"),\n GROUPBY_NGROUPS,\n [6],\n ]\n\n def time_groupby_agg_quan(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).agg(\"quantile\"))\n\n def time_groupby_agg_mean(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).apply(lambda df: df.mean()))\n\n\nclass TimeGroupByDefaultAggregations(BaseTimeGroupBy):\n param_names = [\"shape\", \"ngroups\"]\n params = [\n get_benchmark_shapes(\"TimeGroupByDefaultAggregations\"),\n GROUPBY_NGROUPS,\n ]\n\n def time_groupby_count(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).count())\n\n def time_groupby_size(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).size())\n\n def time_groupby_sum(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).sum())\n\n def time_groupby_mean(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).mean())\n\n\nclass TimeGroupByDictionaryAggregation(BaseTimeGroupBy):\n param_names = [\"shape\", \"ngroups\", \"operation_type\"]\n params = [\n get_benchmark_shapes(\"TimeGroupByDictionaryAggregation\"),\n GROUPBY_NGROUPS,\n [\"reduce\", \"aggregation\"],\n ]\n operations = {\n \"reduce\": [\"sum\", \"count\", \"prod\"],\n \"aggregation\": [\"quantile\", \"std\", \"median\"],\n }\n\n def setup(self, shape, ngroups, operation_type):\n super().setup(shape, ngroups)\n self.cols_to_agg = self.df.columns[1:4]\n operations = self.operations[operation_type]\n self.agg_dict = {\n c: operations[i % len(operations)] for i, c in enumerate(self.cols_to_agg)\n }\n\n def time_groupby_dict_agg(self, *args, **kwargs):\n execute(self.df.groupby(by=self.groupby_columns).agg(self.agg_dict))\n\n\nclass TimeJoin:\n param_names = [\"shapes\", \"how\", \"sort\"]\n params = [\n get_benchmark_shapes(\"TimeJoin\"),\n [\"left\", \"inner\"],\n [False],\n ]\n\n def setup(self, shapes, how, sort):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n\n def time_join(self, shapes, how, sort):\n # join dataframes on index to get the predictable shape\n execute(self.df1.join(self.df2, how=how, lsuffix=\"left_\", sort=sort))\n\n\nclass TimeJoinStringIndex:\n param_names = [\"shapes\", \"sort\"]\n params = [\n get_benchmark_shapes(\"TimeJoinStringIndex\"),\n [True, False],\n ]\n\n def setup(self, shapes, sort):\n assert shapes[0] % 100 == 0, \"implementation restriction\"\n level1 = IMPL.Index([f\"i-{i}\" for i in range(10)], dtype=object).values\n level2 = IMPL.Index(\n [f\"i-{i}\" for i in range(shapes[0] // 100)], dtype=object\n ).values\n codes1 = np.arange(10).repeat(shapes[0] // 100)\n codes2 = np.tile(np.arange(shapes[0] // 100), 10)\n index2 = IMPL.MultiIndex(levels=[level1, level2], codes=[codes1, codes2])\n self.df_multi = IMPL.DataFrame(\n np.random.randn(len(index2), 4), index=index2, columns=[\"A\", \"B\", \"C\", \"D\"]\n )\n\n self.key1 = np.tile(level1.take(codes1), 10)\n self.key2 = np.tile(level2.take(codes2), 10)\n self.df = generate_dataframe(\"int\", *shapes, RAND_LOW, RAND_HIGH)\n # just to keep source shape\n self.df = self.df.drop(columns=self.df.columns[-2:])\n self.df[\"key1\"] = self.key1\n self.df[\"key2\"] = self.key2\n execute(self.df)\n\n self.df_key1 = IMPL.DataFrame(\n np.random.randn(len(level1), 4), index=level1, columns=[\"A\", \"B\", \"C\", \"D\"]\n )\n self.df_key2 = IMPL.DataFrame(\n np.random.randn(len(level2), 4), index=level2, columns=[\"A\", \"B\", \"C\", \"D\"]\n )\n\n def time_join_dataframe_index_multi(self, shapes, sort):\n execute(self.df.join(self.df_multi, on=[\"key1\", \"key2\"], sort=sort))\n\n def time_join_dataframe_index_single_key_bigger(self, shapes, sort):\n execute(self.df.join(self.df_key2, on=\"key2\", sort=sort))\n\n def time_join_dataframe_index_single_key_small(self, shapes, sort):\n execute(self.df.join(self.df_key1, on=\"key1\", sort=sort))\n\n\nclass TimeMergeDefault:\n param_names = [\"shapes\", \"how\", \"sort\"]\n params = [\n get_benchmark_shapes(\"TimeMergeDefault\"),\n [\"left\", \"inner\"],\n [True, False],\n ]\n\n def setup(self, shapes, how, sort):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n\n def time_merge(self, shapes, how, sort):\n execute(IMPL.merge(self.df1, self.df2, how=how, sort=sort))\n\n\nclass TimeMerge:\n param_names = [\"shapes\", \"how\", \"sort\"]\n params = [\n get_benchmark_shapes(\"TimeMerge\"),\n [\"left\", \"inner\"],\n [True, False],\n ]\n\n def setup(self, shapes, how, sort):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n\n def time_merge(self, shapes, how, sort):\n # merge dataframes by index to get the predictable shape\n execute(\n self.df1.merge(\n self.df2, left_index=True, right_index=True, how=how, sort=sort\n )\n )\n\n def time_merge_dataframe_empty_right(self, shapes, how, sort):\n # Getting an empty dataframe using `iloc` should be very fast,\n # so the impact on the time of the merge operation should be negligible.\n execute(IMPL.merge(self.df1, self.df2.iloc[:0], how=how, sort=sort))\n\n def time_merge_dataframe_empty_left(self, shapes, how, sort):\n # Getting an empty dataframe using `iloc` should be very fast,\n # so the impact on the time of the merge operation should be negligible.\n execute(IMPL.merge(self.df1.iloc[:0], self.df2, how=how, sort=sort))\n\n\nclass TimeMergeCategoricals:\n param_names = [\"shapes\", \"data_type\"]\n params = [\n get_benchmark_shapes(\"MergeCategoricals\"),\n [\"object\", \"category\"],\n ]\n\n def setup(self, shapes, data_type):\n assert len(shapes) == 2\n assert shapes[1] == 2\n size = (shapes[0],)\n self.left = IMPL.DataFrame(\n {\n \"X\": np.random.choice(range(0, 10), size=size),\n \"Y\": np.random.choice([\"one\", \"two\", \"three\"], size=size),\n }\n )\n\n self.right = IMPL.DataFrame(\n {\n \"X\": np.random.choice(range(0, 10), size=size),\n \"Z\": np.random.choice([\"jjj\", \"kkk\", \"sss\"], size=size),\n }\n )\n\n if data_type == \"category\":\n self.left = self.left.assign(Y=self.left[\"Y\"].astype(\"category\"))\n execute(self.left)\n self.right = self.right.assign(Z=self.right[\"Z\"].astype(\"category\"))\n execute(self.right)\n\n def time_merge_categoricals(self, shapes, data_type):\n execute(IMPL.merge(self.left, self.right, on=\"X\"))\n\n\nclass TimeConcat:\n param_names = [\"shapes\", \"how\", \"axis\", \"ignore_index\"]\n params = [\n get_benchmark_shapes(\"TimeConcat\"),\n [\"inner\", \"outer\"],\n [0, 1],\n [True, False],\n ]\n\n def setup(self, shapes, how, axis, ignore_index):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n\n def time_concat(self, shapes, how, axis, ignore_index):\n execute(\n IMPL.concat(\n [self.df1, self.df2], axis=axis, join=how, ignore_index=ignore_index\n )\n )\n\n\nclass TimeBinaryOp:\n param_names = [\"shapes\", \"binary_op\", \"axis\"]\n params = [\n get_benchmark_shapes(\"TimeBinaryOp\"),\n [\"mul\"],\n [0, 1],\n ]\n\n def setup(self, shapes, binary_op, axis):\n self.df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n self.df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n self.op = getattr(self.df1, binary_op)\n\n def time_binary_op(self, shapes, binary_op, axis):\n execute(self.op(self.df2, axis=axis))\n\n\nclass TimeBinaryOpSeries:\n param_names = [\"shapes\", \"binary_op\"]\n params = [\n get_benchmark_shapes(\"TimeBinaryOpSeries\"),\n [\"mul\"],\n ]\n\n def setup(self, shapes, binary_op):\n df1 = generate_dataframe(\"int\", *shapes[0], RAND_LOW, RAND_HIGH)\n df2 = generate_dataframe(\"int\", *shapes[1], RAND_LOW, RAND_HIGH)\n self.series1 = df1[df1.columns[0]]\n self.series2 = df2[df2.columns[0]]\n self.op = getattr(self.series1, binary_op)\n execute(self.series1)\n execute(self.series2)\n\n def time_binary_op_series(self, shapes, binary_op):\n execute(self.op(self.series2))\n\n\nclass BaseTimeSetItem:\n param_names = [\"shape\", \"item_length\", \"loc\", \"is_equal_indices\"]\n\n @staticmethod\n def get_loc(df, loc, axis, item_length):\n locs_dict = {\n \"zero\": 0,\n \"middle\": len(df.axes[axis]) // 2,\n \"last\": len(df.axes[axis]) - 1,\n }\n base_loc = locs_dict[loc]\n range_based_loc = np.arange(\n base_loc, min(len(df.axes[axis]), base_loc + item_length)\n )\n return (\n (df.axes[axis][base_loc], base_loc)\n if len(range_based_loc) == 1\n else (df.axes[axis][range_based_loc], range_based_loc)\n )\n\n def setup(self, shape, item_length, loc, is_equal_indices):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH).copy()\n self.loc, self.iloc = self.get_loc(\n self.df, loc, item_length=item_length, axis=1\n )\n\n self.item = self.df[self.loc] + 1\n self.item_raw = self.item.to_numpy()\n if not is_equal_indices:\n self.item.index = reversed(self.item.index)\n\n\nclass TimeSetItem(BaseTimeSetItem):\n params = [\n get_benchmark_shapes(\"TimeSetItem\"),\n [1],\n [\"zero\", \"middle\", \"last\"],\n [True, False],\n ]\n\n def time_setitem_qc(self, *args, **kwargs):\n self.df[self.loc] = self.item\n execute(self.df)\n\n def time_setitem_raw(self, *args, **kwargs):\n self.df[self.loc] = self.item_raw\n execute(self.df)\n\n\nclass TimeInsert(BaseTimeSetItem):\n params = [\n get_benchmark_shapes(\"TimeInsert\"),\n [1],\n [\"zero\", \"middle\", \"last\"],\n [True, False],\n ]\n\n def time_insert_qc(self, *args, **kwargs):\n self.df.insert(loc=self.iloc, column=random_string(), value=self.item)\n execute(self.df)\n\n def time_insert_raw(self, *args, **kwargs):\n self.df.insert(loc=self.iloc, column=random_string(), value=self.item_raw)\n execute(self.df)\n\n\nclass TimeArithmetic:\n param_names = [\"shape\", \"axis\"]\n params = [\n get_benchmark_shapes(\"TimeArithmetic\"),\n [0, 1],\n ]\n\n def setup(self, shape, axis):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n def time_sum(self, shape, axis):\n execute(self.df.sum(axis=axis))\n\n def time_count(self, shape, axis):\n execute(self.df.count(axis=axis))\n\n def time_median(self, shape, axis):\n execute(self.df.median(axis=axis))\n\n def time_nunique(self, shape, axis):\n execute(self.df.nunique(axis=axis))\n\n def time_apply(self, shape, axis):\n execute(self.df.apply(lambda df: df.sum(), axis=axis))\n\n def time_mean(self, shape, axis):\n execute(self.df.mean(axis=axis))\n\n def time_mode(self, shape, axis):\n execute(self.df.mode(axis=axis))\n\n def time_add(self, shape, axis):\n execute(self.df.add(2, axis=axis))\n\n def time_mul(self, shape, axis):\n execute(self.df.mul(2, axis=axis))\n\n def time_mod(self, shape, axis):\n execute(self.df.mod(2, axis=axis))\n\n def time_abs(self, shape, axis):\n execute(self.df.abs())\n\n def time_aggregate(self, shape, axis):\n execute(self.df.aggregate(lambda df: df.sum(), axis=axis))\n\n def time_is_in(self, shape, axis):\n execute(self.df.isin([0, 2]))\n\n def time_transpose(self, shape, axis):\n execute(self.df.transpose())\n\n\nclass TimeSortValues:\n param_names = [\"shape\", \"columns_number\", \"ascending_list\"]\n params = [\n get_benchmark_shapes(\"TimeSortValues\"),\n [1, 2, 10, 100],\n [False, True],\n ]\n\n def setup(self, shape, columns_number, ascending_list):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n self.columns = random_columns(self.df.columns, columns_number)\n self.ascending = (\n random_booleans(columns_number)\n if ascending_list\n else bool(random_booleans(1)[0])\n )\n\n def time_sort_values(self, shape, columns_number, ascending_list):\n execute(self.df.sort_values(self.columns, ascending=self.ascending))\n\n\nclass TimeDrop:\n param_names = [\"shape\", \"axis\", \"drop_ncols\"]\n params = [\n get_benchmark_shapes(\"TimeDrop\"),\n [0, 1],\n [1, 0.8],\n ]\n\n def setup(self, shape, axis, drop_ncols):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n drop_count = (\n int(len(self.df.axes[axis]) * drop_ncols)\n if isinstance(drop_ncols, float)\n else drop_ncols\n )\n self.labels = self.df.axes[axis][:drop_count]\n\n def time_drop(self, shape, axis, drop_ncols):\n execute(self.df.drop(self.labels, axis=axis))\n\n\nclass TimeHead:\n param_names = [\"shape\", \"head_count\"]\n params = [\n get_benchmark_shapes(\"TimeHead\"),\n [5, 0.8],\n ]\n\n def setup(self, shape, head_count):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n self.head_count = (\n int(head_count * len(self.df.index))\n if isinstance(head_count, float)\n else head_count\n )\n\n def time_head(self, shape, head_count):\n execute(self.df.head(self.head_count))\n\n\nclass TimeTail:\n param_names = [\"shape\", \"tail_count\"]\n params = [\n get_benchmark_shapes(\"TimeTail\"),\n [5, 0.8],\n ]\n\n def setup(self, shape, tail_count):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n self.tail_count = (\n int(tail_count * len(self.df.index))\n if isinstance(tail_count, float)\n else tail_count\n )\n\n def time_tail(self, shape, tail_count):\n execute(self.df.tail(self.tail_count))\n\n\nclass TimeExplode:\n param_names = [\"shape\"]\n params = [\n get_benchmark_shapes(\"TimeExplode\"),\n ]\n\n def setup(self, shape):\n self.df = generate_dataframe(\n \"int\", *shape, RAND_LOW, RAND_HIGH, gen_unique_key=True\n )\n\n def time_explode(self, shape):\n execute(self.df.explode(\"col1\"))\n\n\nclass TimeFillnaSeries:\n param_names = [\"value_type\", \"shape\", \"limit\"]\n params = [\n [\"scalar\", \"dict\", \"Series\"],\n get_benchmark_shapes(\"TimeFillnaSeries\"),\n [None, 0.8],\n ]\n\n def setup(self, value_type, shape, limit):\n self.series = gen_nan_data(*shape)\n\n if value_type == \"scalar\":\n self.value = 18.19\n elif value_type == \"dict\":\n self.value = {k: k * 1.23 for k in range(shape[0])}\n elif value_type == \"Series\":\n self.value = IMPL.Series(\n [k * 1.23 for k in range(shape[0])], index=IMPL.RangeIndex(shape[0])\n )\n else:\n assert False\n limit = int(limit * shape[0]) if limit else None\n self.kw = {\"value\": self.value, \"limit\": limit}\n\n def time_fillna(self, value_type, shape, limit):\n execute(self.series.fillna(**self.kw))\n\n def time_fillna_inplace(self, value_type, shape, limit):\n self.series.fillna(inplace=True, **self.kw)\n execute(self.series)\n\n\nclass TimeFillnaDataFrame:\n param_names = [\"value_type\", \"shape\", \"limit\"]\n params = [\n [\"scalar\", \"dict\", \"DataFrame\", \"Series\"],\n get_benchmark_shapes(\"TimeFillnaDataFrame\"),\n [None, 0.8],\n ]\n\n def setup(self, value_type, shape, limit):\n self.df = gen_nan_data(*shape)\n columns = self.df.columns\n\n if value_type == \"scalar\":\n self.value = 18.19\n elif value_type == \"dict\":\n self.value = {k: i * 1.23 for i, k in enumerate(columns)}\n elif value_type == \"Series\":\n self.value = IMPL.Series(\n [i * 1.23 for i in range(len(columns))], index=columns\n )\n elif value_type == \"DataFrame\":\n self.value = IMPL.DataFrame(\n {\n k: [i + j * 1.23 for j in range(shape[0])]\n for i, k in enumerate(columns)\n },\n index=IMPL.RangeIndex(shape[0]),\n columns=columns,\n )\n else:\n assert False\n limit = int(limit * shape[0]) if limit else None\n self.kw = {\"value\": self.value, \"limit\": limit}\n\n def time_fillna(self, value_type, shape, limit):\n execute(self.df.fillna(**self.kw))\n\n def time_fillna_inplace(self, value_type, shape, limit):\n self.df.fillna(inplace=True, **self.kw)\n execute(self.df)\n\n\nclass BaseTimeValueCounts:\n def setup(self, shape, ngroups=5, subset=1):\n ngroups = translator_groupby_ngroups(ngroups, shape)\n self.df, self.subset = generate_dataframe(\n \"int\",\n *shape,\n RAND_LOW,\n RAND_HIGH,\n groupby_ncols=subset,\n count_groups=ngroups,\n )\n\n\nclass TimeValueCountsFrame(BaseTimeValueCounts):\n param_names = [\"shape\", \"ngroups\", \"subset\"]\n params = [\n get_benchmark_shapes(\"TimeValueCountsFrame\"),\n GROUPBY_NGROUPS,\n [2, 10],\n ]\n\n def time_value_counts(self, *args, **kwargs):\n execute(self.df.value_counts(subset=self.subset))\n\n\nclass TimeValueCountsSeries(BaseTimeValueCounts):\n param_names = [\"shape\", \"ngroups\", \"bins\"]\n params = [\n get_benchmark_shapes(\"TimeValueCountsSeries\"),\n GROUPBY_NGROUPS,\n [None, 3],\n ]\n\n def setup(self, shape, ngroups, bins):\n super().setup(ngroups=ngroups, shape=shape)\n self.df = self.df[self.subset[0]]\n\n def time_value_counts(self, shape, ngroups, bins):\n execute(self.df.value_counts(bins=bins))\n\n\nclass TimeIndexing:\n param_names = [\"shape\", \"indexer_type\"]\n params = [\n get_benchmark_shapes(\"TimeIndexing\"),\n [\n \"bool_array\",\n \"bool_series\",\n \"scalar\",\n \"slice\",\n \"continuous_slice\",\n \"numpy_array_take_all_values\",\n \"python_list_take_10_values\",\n \"function\",\n ],\n ]\n\n indexer_getters = {\n \"bool_array\": lambda df: np.array([False, True] * (len(df) // 2)),\n # This boolean-Series is a projection of the source frame, it shouldn't\n # be reimported or triggered to execute:\n \"bool_series\": lambda df: df.iloc[:, 0] > 50,\n \"scalar\": lambda df: len(df) // 2,\n \"slice\": lambda df: slice(0, len(df), 2),\n \"continuous_slice\": lambda df: slice(len(df) // 2),\n \"numpy_array_take_all_values\": lambda df: np.arange(len(df)),\n \"python_list_take_10_values\": lambda df: list(range(min(10, len(df)))),\n \"function\": lambda df: (lambda df: df.index[::-2]),\n }\n\n def setup(self, shape, indexer_type):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n self.indexer = self.indexer_getters[indexer_type](self.df)\n if isinstance(self.indexer, (IMPL.Series, IMPL.DataFrame)):\n # HACK: Triggering `dtypes` meta-data computation in advance,\n # so it won't affect the `loc/iloc` time:\n self.indexer.dtypes\n\n def time_iloc(self, shape, indexer_type):\n # Pandas doesn't implement `df.iloc[series boolean_mask]` and raises an exception on it.\n # Replacing this with the semantically equivalent construction:\n if indexer_type != \"bool_series\":\n execute(self.df.iloc[self.indexer])\n else:\n execute(self.df[self.indexer])\n\n def time_loc(self, shape, indexer_type):\n execute(self.df.loc[self.indexer])\n\n\nclass TimeIndexingColumns:\n param_names = [\"shape\"]\n params = [get_benchmark_shapes(\"TimeIndexing\")]\n\n def setup(self, shape):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n self.numeric_indexer = [0, 1]\n self.labels_indexer = self.df.columns[self.numeric_indexer].tolist()\n\n def time_iloc(self, shape):\n execute(self.df.iloc[:, self.numeric_indexer])\n\n def time_loc(self, shape):\n execute(self.df.loc[:, self.labels_indexer])\n\n def time___getitem__(self, shape):\n execute(self.df[self.labels_indexer])\n\n\nclass TimeMultiIndexing:\n param_names = [\"shape\"]\n params = [get_benchmark_shapes(\"TimeMultiIndexing\")]\n\n def setup(self, shape):\n df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n index = IMPL.MultiIndex.from_product(\n [df.index[: shape[0] // 2], [\"bar\", \"foo\"]]\n )\n columns = IMPL.MultiIndex.from_product(\n [df.columns[: shape[1] // 2], [\"buz\", \"fuz\"]]\n )\n\n df.index = index\n df.columns = columns\n\n self.df = df.sort_index(axis=1)\n\n def time_multiindex_loc(self, shape):\n execute(\n self.df.loc[\n self.df.index[2] : self.df.index[-2],\n self.df.columns[2] : self.df.columns[-2],\n ]\n )\n\n\nclass TimeResetIndex:\n param_names = [\"shape\", \"drop\", \"level\"]\n params = [\n get_benchmark_shapes(\"TimeResetIndex\"),\n [False, True],\n [None, \"level_1\"],\n ]\n\n def setup(self, shape, drop, level):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n if level:\n index = IMPL.MultiIndex.from_product(\n [self.df.index[: shape[0] // 2], [\"bar\", \"foo\"]],\n names=[\"level_1\", \"level_2\"],\n )\n self.df.index = index\n\n def time_reset_index(self, shape, drop, level):\n execute(self.df.reset_index(drop=drop, level=level))\n\n\nclass TimeAstype:\n param_names = [\"shape\", \"dtype\", \"astype_ncolumns\"]\n params = [\n get_benchmark_shapes(\"TimeAstype\"),\n [\"float64\", \"category\"],\n [\"one\", \"all\"],\n ]\n\n def setup(self, shape, dtype, astype_ncolumns):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n if astype_ncolumns == \"all\":\n self.astype_arg = dtype\n elif astype_ncolumns == \"one\":\n self.astype_arg = {\"col1\": dtype}\n else:\n raise ValueError(f\"astype_ncolumns: {astype_ncolumns} isn't supported\")\n\n def time_astype(self, shape, dtype, astype_ncolumns):\n execute(self.df.astype(self.astype_arg))\n\n\nclass TimeDescribe:\n param_names = [\"shape\"]\n params = [\n get_benchmark_shapes(\"TimeDescribe\"),\n ]\n\n def setup(self, shape):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n def time_describe(self, shape):\n execute(self.df.describe())\n\n\nclass TimeProperties:\n param_names = [\"shape\"]\n params = [\n get_benchmark_shapes(\"TimeProperties\"),\n ]\n\n def setup(self, shape):\n self.df = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH)\n\n def time_shape(self, shape):\n return self.df.shape\n\n def time_columns(self, shape):\n return self.df.columns\n\n def time_index(self, shape):\n return self.df.index\n\n\nclass TimeIndexingNumericSeries:\n param_names = [\"shape\", \"dtype\", \"index_structure\"]\n params = [\n get_benchmark_shapes(\"TimeIndexingNumericSeries\"),\n (np.int64, np.uint64, np.float64),\n (\"unique_monotonic_inc\", \"nonunique_monotonic_inc\"),\n ]\n\n def setup(self, shape, dtype, index_structure):\n N = shape[0]\n indices = {\n \"unique_monotonic_inc\": IMPL.Index(range(N), dtype=dtype),\n \"nonunique_monotonic_inc\": IMPL.Index(\n list(range(N // 100)) + [(N // 100) - 1] + list(range(N // 100, N - 1)),\n dtype=dtype,\n ),\n }\n self.data = IMPL.Series(np.random.rand(N), index=indices[index_structure])\n self.array = np.arange(N // 2)\n self.index_to_query = N // 2\n self.array_list = self.array.tolist()\n execute(self.data)\n\n def time_getitem_scalar(self, shape, index, index_structure):\n # not calling execute as execute function fails for scalar\n self.data[self.index_to_query]\n\n def time_getitem_slice(self, shape, index, index_structure):\n execute(self.data[: self.index_to_query])\n\n def time_getitem_list_like(self, shape, index, index_structure):\n execute(self.data[[self.index_to_query]])\n\n def time_getitem_array(self, shape, index, index_structure):\n execute(self.data[self.array])\n\n def time_getitem_lists(self, shape, index, index_structure):\n execute(self.data[self.array_list])\n\n def time_iloc_array(self, shape, index, index_structure):\n execute(self.data.iloc[self.array])\n\n def time_iloc_list_like(self, shape, index, index_structure):\n execute(self.data.iloc[[self.index_to_query]])\n\n def time_iloc_scalar(self, shape, index, index_structure):\n # not calling execute as execute function fails for scalar\n self.data.iloc[self.index_to_query]\n\n def time_iloc_slice(self, shape, index, index_structure):\n execute(self.data.iloc[: self.index_to_query])\n\n def time_loc_array(self, shape, index, index_structure):\n execute(self.data.loc[self.array])\n\n def time_loc_list_like(self, shape, index, index_structure):\n execute(self.data.loc[[self.index_to_query]])\n\n def time_loc_scalar(self, shape, index, index_structure):\n self.data.loc[self.index_to_query]\n\n def time_loc_slice(self, shape, index, index_structure):\n execute(self.data.loc[: self.index_to_query])\n\n\nclass TimeReindex:\n param_names = [\"shape\"]\n params = [get_benchmark_shapes(\"TimeReindex\")]\n\n def setup(self, shape):\n rows, cols = shape\n rng = IMPL.date_range(start=\"1/1/1970\", periods=rows, freq=\"1min\")\n self.df = IMPL.DataFrame(\n np.random.rand(rows, cols), index=rng, columns=range(cols)\n )\n self.df[\"foo\"] = \"bar\"\n self.rng_subset = IMPL.Index(rng[::2])\n self.df2 = IMPL.DataFrame(\n index=range(rows), data=np.random.rand(rows, cols), columns=range(cols)\n )\n level1 = IMPL.Index(\n [f\"i-{i}\" for i in range(rows // 10)], dtype=object\n ).values.repeat(10)\n level2 = np.tile(\n IMPL.Index([f\"i-{i}\" for i in range(10)], dtype=object).values, rows // 10\n )\n index = IMPL.MultiIndex.from_arrays([level1, level2])\n self.s = IMPL.Series(np.random.randn(rows), index=index)\n self.s_subset = self.s[::2]\n self.s_subset_no_cache = self.s[::2].copy()\n\n mi = IMPL.MultiIndex.from_product([rng[: len(rng) // 10], range(10)])\n self.s2 = IMPL.Series(np.random.randn(len(mi)), index=mi)\n self.s2_subset = self.s2[::2].copy()\n execute(self.df), execute(self.df2)\n execute(self.s), execute(self.s_subset)\n execute(self.s2), execute(self.s2_subset)\n execute(self.s_subset_no_cache)\n\n def time_reindex_dates(self, shape):\n execute(self.df.reindex(self.rng_subset))\n\n def time_reindex_columns(self, shape):\n execute(self.df2.reindex(columns=self.df.columns[1:5]))\n\n def time_reindex_multiindex_with_cache(self, shape):\n # MultiIndex._values gets cached (pandas specific)\n execute(self.s.reindex(self.s_subset.index))\n\n def time_reindex_multiindex_no_cache(self, shape):\n # Copy to avoid MultiIndex._values getting cached (pandas specific)\n execute(self.s.reindex(self.s_subset_no_cache.index.copy()))\n\n def time_reindex_multiindex_no_cache_dates(self, shape):\n # Copy to avoid MultiIndex._values getting cached (pandas specific)\n execute(self.s2_subset.reindex(self.s2.index.copy()))\n\n\nclass TimeReindexMethod:\n params = [\n get_benchmark_shapes(\"TimeReindexMethod\"),\n [\"pad\", \"backfill\"],\n [IMPL.date_range, IMPL.period_range],\n ]\n param_names = [\"shape\", \"method\", \"constructor\"]\n\n def setup(self, shape, method, constructor):\n N = shape[0]\n self.idx = constructor(\"1/1/2000\", periods=N, freq=\"1min\")\n self.ts = IMPL.Series(np.random.randn(N), index=self.idx)[::2]\n execute(self.ts)\n\n def time_reindex_method(self, shape, method, constructor):\n execute(self.ts.reindex(self.idx, method=method))\n\n\nclass TimeFillnaMethodSeries:\n params = [get_benchmark_shapes(\"TimeFillnaMethodSeries\"), [\"pad\", \"backfill\"]]\n param_names = [\"shape\", \"method\"]\n\n def setup(self, shape, method):\n N = shape[0]\n self.idx = IMPL.date_range(\"1/1/2000\", periods=N, freq=\"1min\")\n ts = IMPL.Series(np.random.randn(N), index=self.idx)[::2]\n self.ts_reindexed = ts.reindex(self.idx)\n self.ts_float32 = self.ts_reindexed.astype(\"float32\")\n execute(self.ts_reindexed), execute(self.ts_float32)\n\n def time_reindexed(self, shape, method):\n execute(self.ts_reindexed.fillna(method=method))\n\n def time_float_32(self, shape, method):\n execute(self.ts_float32.fillna(method=method))\n\n\nclass TimeFillnaMethodDataframe:\n params = [get_benchmark_shapes(\"TimeFillnaMethodDataframe\"), [\"pad\", \"backfill\"]]\n param_names = [\"shape\", \"method\"]\n\n def setup(self, shape, method):\n self.idx = IMPL.date_range(\"1/1/2000\", periods=shape[0], freq=\"1min\")\n df_ts = IMPL.DataFrame(np.random.randn(*shape), index=self.idx)[::2]\n self.df_ts_reindexed = df_ts.reindex(self.idx)\n self.df_ts_float32 = self.df_ts_reindexed.astype(\"float32\")\n execute(self.df_ts_reindexed), execute(self.df_ts_float32)\n\n def time_reindexed(self, shape, method):\n execute(self.df_ts_reindexed.fillna(method=method))\n\n def time_float_32(self, shape, method):\n execute(self.df_ts_float32.fillna(method=method))\n\n\nclass TimeLevelAlign:\n params = [get_benchmark_shapes(\"TimeLevelAlign\")]\n param_names = [\"shapes\"]\n\n def setup(self, shapes):\n rows, cols = shapes[0]\n rows_sqrt = round(math.sqrt(rows))\n # the new number of rows may differ from the requested (slightly, so ok)\n rows = rows_sqrt * rows_sqrt\n self.index = IMPL.MultiIndex(\n levels=[np.arange(10), np.arange(rows_sqrt), np.arange(rows_sqrt)],\n codes=[\n np.arange(10).repeat(rows),\n np.tile(np.arange(rows_sqrt).repeat(rows_sqrt), 10),\n np.tile(np.tile(np.arange(rows_sqrt), rows_sqrt), 10),\n ],\n )\n self.df1 = IMPL.DataFrame(\n np.random.randn(len(self.index), cols), index=self.index\n )\n self.df2 = IMPL.DataFrame(np.random.randn(*shapes[1]))\n execute(self.df1), execute(self.df2)\n\n def time_align_level(self, shapes):\n left, right = self.df1.align(self.df2, level=1, copy=False)\n execute(left), execute(right)\n\n def time_reindex_level(self, shapes):\n # `reindex` returns the same result here as `align`.\n # Approximately the same performance is expected.\n execute(self.df2.reindex(self.index, level=1))\n\n\nclass TimeDropDuplicatesDataframe:\n params = [get_benchmark_shapes(\"TimeDropDuplicatesDataframe\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n rows, cols = shape\n N = rows // 10\n K = 10\n data = {}\n # dataframe would have cols-1 keys(strings) and one value(int) column\n for col in range(cols - 1):\n data[\"key\" + str(col + 1)] = IMPL.Index(\n [f\"i-{i}\" for i in range(N)], dtype=object\n ).values.repeat(K)\n data[\"value\"] = np.random.randn(N * K)\n self.df = IMPL.DataFrame(data)\n execute(self.df)\n\n def time_drop_dups(self, shape):\n execute(self.df.drop_duplicates(self.df.columns[:-1]))\n\n def time_drop_dups_inplace(self, shape):\n self.df.drop_duplicates(self.df.columns[:-1], inplace=True)\n execute(self.df)\n\n\nclass TimeDropDuplicatesSeries:\n params = [get_benchmark_shapes(\"TimeDropDuplicatesSeries\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n rows = shape[0]\n self.series = IMPL.Series(\n np.tile(\n IMPL.Index([f\"i-{i}\" for i in range(rows // 10)], dtype=object).values,\n 10,\n )\n )\n execute(self.series)\n\n def time_drop_dups(self, shape):\n execute(self.series.drop_duplicates())\n\n def time_drop_dups_string(self, shape):\n self.series.drop_duplicates(inplace=True)\n execute(self.series)\n\n\nclass TimeDatetimeAccessor:\n params = [get_benchmark_shapes(\"TimeDatetimeAccessor\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.series = IMPL.Series(\n IMPL.timedelta_range(\"1 days\", periods=shape[0], freq=\"h\")\n )\n execute(self.series)\n\n def time_dt_accessor(self, shape):\n execute(self.series.dt)\n\n def time_timedelta_days(self, shape):\n execute(self.series.dt.days)\n\n def time_timedelta_seconds(self, shape):\n execute(self.series.dt.seconds)\n\n\nclass BaseCategories:\n def setup(self, shape):\n rows = shape[0]\n arr = [f\"s{i:04d}\" for i in np.random.randint(0, rows // 10, size=rows)]\n self.ts = IMPL.Series(arr).astype(\"category\")\n execute(self.ts)\n\n\nclass TimeSetCategories(BaseCategories):\n params = [get_benchmark_shapes(\"TimeSetCategories\")]\n param_names = [\"shape\"]\n\n def time_set_categories(self, shape):\n execute(self.ts.cat.set_categories(self.ts.cat.categories[::2]))\n\n\nclass TimeRemoveCategories(BaseCategories):\n params = [get_benchmark_shapes(\"TimeRemoveCategories\")]\n param_names = [\"shape\"]\n\n def time_remove_categories(self, shape):\n execute(self.ts.cat.remove_categories(self.ts.cat.categories[::2]))\n\n\nclass BaseReshape:\n def setup(self, shape):\n rows, cols = shape\n k = 10\n arrays = [\n np.arange(rows // k).repeat(k),\n np.roll(np.tile(np.arange(rows // k), k), 25),\n ]\n index = IMPL.MultiIndex.from_arrays(arrays)\n self.df = IMPL.DataFrame(np.random.randn(rows, cols), index=index)\n execute(self.df)\n\n\nclass TimeStack(BaseReshape):\n params = [get_benchmark_shapes(\"TimeStack\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n super().setup(shape)\n self.udf = self.df.unstack(1)\n execute(self.udf)\n\n def time_stack(self, shape):\n execute(self.udf.stack())\n\n\nclass TimeUnstack(BaseReshape):\n params = [get_benchmark_shapes(\"TimeUnstack\")]\n param_names = [\"shape\"]\n\n def time_unstack(self, shape):\n execute(self.df.unstack(1))\n\n\nclass TimeReplace:\n params = [get_benchmark_shapes(\"TimeReplace\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n rows, cols = shape\n self.to_replace = {i: getattr(IMPL, \"Timestamp\")(i) for i in range(rows)}\n self.df = IMPL.DataFrame(np.random.randint(rows, size=(rows, cols)))\n execute(self.df)\n\n def time_replace(self, shape):\n execute(self.df.replace(self.to_replace))\n\n\nclass TimeGroups:\n params = [get_benchmark_shapes(\"TimeGroups\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.series = IMPL.Series(np.random.randint(0, 100, size=shape[0]))\n execute(self.series)\n\n # returns a pretty dict thus not calling execute\n def time_series_groups(self, shape):\n self.series.groupby(self.series).groups\n\n # returns a dict thus not calling execute\n def time_series_indices(self, shape):\n self.series.groupby(self.series).indices\n\n\nclass TimeRepr:\n params = [get_benchmark_shapes(\"TimeRepr\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.df = IMPL.DataFrame(np.random.randn(*shape))\n execute(self.df)\n\n # returns a string thus not calling execute\n def time_repr(self, shape):\n repr(self.df)\n\n\nclass TimeMaskBool:\n params = [get_benchmark_shapes(\"TimeMaskBool\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.df = IMPL.DataFrame(np.random.randn(*shape))\n self.mask = self.df < 0\n execute(self.df), execute(self.mask)\n\n def time_frame_mask(self, shape):\n execute(self.df.mask(self.mask))\n\n\nclass TimeIsnull:\n params = [get_benchmark_shapes(\"TimeIsnull\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n sample = np.array([np.nan, 1.0])\n data = np.random.choice(sample, (shape[0], shape[1]))\n self.df = IMPL.DataFrame(data)\n execute(self.df)\n\n def time_isnull(self, shape):\n execute(IMPL.isnull(self.df))\n\n\nclass TimeDropna:\n params = ([\"all\", \"any\"], [0, 1], get_benchmark_shapes(\"TimeDropna\"))\n param_names = [\"how\", \"axis\", \"shape\"]\n\n def setup(self, how, axis, shape):\n row, col = shape\n self.df = IMPL.DataFrame(np.random.randn(row, col))\n self.df.iloc[row // 20 : row // 10, col // 3 : col // 2] = np.nan\n self.df[\"foo\"] = \"bar\"\n execute(self.df)\n\n def time_dropna(self, how, axis, shape):\n execute(self.df.dropna(how=how, axis=axis))\n\n\nclass TimeEquals:\n params = [get_benchmark_shapes(\"TimeEquals\")]\n param_names = [\"shape\"]\n\n def setup(self, shape):\n self.df = IMPL.DataFrame(np.random.randn(*shape))\n self.df.iloc[-1, -1] = np.nan\n execute(self.df)\n\n # returns a boolean thus not calling execute\n def time_frame_float_equal(self, shape):\n self.df.equals(self.df)\n\n\nfrom .utils import setup # noqa: E402, F401\n"
},
{
"path": "asv_bench/benchmarks/io/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"IO Modin benchmarks.\"\"\"\n"
},
{
"path": "asv_bench/benchmarks/io/csv.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nimport numpy as np\n\nfrom ..utils import (\n ASV_USE_IMPL,\n IMPL,\n RAND_HIGH,\n RAND_LOW,\n execute,\n generate_dataframe,\n get_benchmark_shapes,\n get_shape_id,\n prepare_io_data,\n)\n\n\nclass BaseReadCsv:\n # test data file should be created only once\n def setup_cache(self, test_filename=\"io_test_file\"):\n test_filenames = prepare_io_data(\n test_filename, self.data_type, get_benchmark_shapes(self.__class__.__name__)\n )\n return test_filenames\n\n def setup(self, test_filenames, shape, *args, **kwargs):\n # ray init\n if ASV_USE_IMPL == \"modin\":\n IMPL.DataFrame([])\n self.shape_id = get_shape_id(shape)\n\n\nclass TimeReadCsvSkiprows(BaseReadCsv):\n shapes = get_benchmark_shapes(\"TimeReadCsvSkiprows\")\n skiprows_mapping = {\n \"lambda_even_rows\": lambda x: x % 2,\n \"range_uniform\": np.arange(1, shapes[0][0] // 10),\n \"range_step2\": np.arange(1, shapes[0][0], 2),\n }\n data_type = \"str_int\"\n\n param_names = [\"shape\", \"skiprows\"]\n params = [\n shapes,\n [None, \"lambda_even_rows\", \"range_uniform\", \"range_step2\"],\n ]\n\n def setup(self, test_filenames, shape, skiprows):\n super().setup(test_filenames, shape, skiprows)\n self.skiprows = self.skiprows_mapping[skiprows] if skiprows else None\n\n def time_skiprows(self, test_filenames, shape, skiprows):\n execute(IMPL.read_csv(test_filenames[self.shape_id], skiprows=self.skiprows))\n\n\nclass TimeReadCsvTrueFalseValues(BaseReadCsv):\n data_type = \"true_false_int\"\n\n param_names = [\"shape\"]\n params = [get_benchmark_shapes(\"TimeReadCsvTrueFalseValues\")]\n\n def time_true_false_values(self, test_filenames, shape):\n execute(\n IMPL.read_csv(\n test_filenames[self.shape_id],\n true_values=[\"Yes\", \"true\"],\n false_values=[\"No\", \"false\"],\n ),\n )\n\n\nclass TimeReadCsvNamesDtype:\n shapes = get_benchmark_shapes(\"TimeReadCsvNamesDtype\")\n _dtypes_params = [\"Int64\", \"Int64_Timestamp\"]\n _timestamp_columns = [\"col1\", \"col2\"]\n\n param_names = [\"shape\", \"names\", \"dtype\"]\n params = [\n shapes,\n [\"array-like\"],\n _dtypes_params,\n ]\n\n def _get_file_id(self, shape, dtype):\n return get_shape_id(shape) + dtype\n\n def _add_timestamp_columns(self, df):\n df = df.copy()\n date_column = IMPL.date_range(\"2000\", periods=df.shape[0], freq=\"ms\")\n for col in self._timestamp_columns:\n df[col] = date_column\n return df\n\n def setup_cache(self, test_filename=\"io_test_file_csv_names_dtype\"):\n # filenames with a metadata of saved dataframes\n cache = {}\n for shape in self.shapes:\n for dtype in self._dtypes_params:\n df = generate_dataframe(\n \"int\", *shape, RAND_LOW, RAND_HIGH, impl=\"pandas\"\n )\n if dtype == \"Int64_Timestamp\":\n df = self._add_timestamp_columns(df)\n\n file_id = self._get_file_id(shape, dtype)\n cache[file_id] = (\n f\"{test_filename}_{file_id}.csv\",\n df.columns.to_list(),\n df.dtypes.to_dict(),\n )\n df.to_csv(cache[file_id][0], index=False)\n return cache\n\n def setup(self, cache, shape, names, dtype):\n # ray init\n if ASV_USE_IMPL == \"modin\":\n IMPL.DataFrame([])\n file_id = self._get_file_id(shape, dtype)\n self.filename, self.names, self.dtype = cache[file_id]\n\n self.parse_dates = None\n if dtype == \"Int64_Timestamp\":\n # cached version of dtype should not change\n self.dtype = self.dtype.copy()\n for col in self._timestamp_columns:\n del self.dtype[col]\n self.parse_dates = self._timestamp_columns\n\n def time_read_csv_names_dtype(self, cache, shape, names, dtype):\n execute(\n IMPL.read_csv(\n self.filename,\n names=self.names,\n header=0,\n dtype=self.dtype,\n parse_dates=self.parse_dates,\n )\n )\n\n\nfrom ..utils import setup # noqa: E402, F401\n"
},
{
"path": "asv_bench/benchmarks/io/parquet.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nfrom ..utils import (\n ASV_USE_IMPL,\n IMPL,\n execute,\n get_benchmark_shapes,\n get_shape_id,\n prepare_io_data_parquet,\n)\n\n\nclass TimeReadParquet:\n shapes = get_benchmark_shapes(\"TimeReadParquet\")\n data_type = \"str_int\"\n\n param_names = [\"shape\"]\n params = [\n shapes,\n ]\n\n # test data file should be created only once\n def setup_cache(self, test_filename=\"io_test_file\"):\n test_filenames = prepare_io_data_parquet(\n test_filename, self.data_type, get_benchmark_shapes(self.__class__.__name__)\n )\n return test_filenames\n\n def setup(self, test_filenames, shape):\n # ray init\n if ASV_USE_IMPL == \"modin\":\n IMPL.DataFrame([])\n self.shape_id = get_shape_id(shape)\n\n def time_read_parquet(self, test_filenames, shape):\n execute(\n IMPL.read_parquet(\n test_filenames[self.shape_id],\n )\n )\n\n\nfrom ..utils import setup # noqa: E402, F401\n"
},
{
"path": "asv_bench/benchmarks/scalability/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Benchmarks measuring how Modin performance scales when MODIN_CPUS are changed.\"\"\"\n"
},
{
"path": "asv_bench/benchmarks/scalability/scalability_benchmarks.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"These benchmarks are supposed to be run only for modin, since they do not make sense for pandas.\"\"\"\n\nimport modin.pandas as pd\n\ntry:\n from modin.pandas.io import from_pandas\nexcept ImportError:\n from modin.pandas.utils import from_pandas\n\ntry:\n from modin.pandas.io import to_numpy, to_pandas\nexcept ImportError:\n try:\n from modin.utils import to_numpy, to_pandas\n except ImportError:\n # This provides compatibility with older versions of the Modin, allowing us to test old commits.\n from modin.pandas.utils import to_pandas\n\nimport pandas\n\nfrom ..utils import (\n RAND_HIGH,\n RAND_LOW,\n execute,\n gen_data,\n generate_dataframe,\n get_benchmark_shapes,\n)\n\n\nclass TimeFromPandas:\n param_names = [\"shape\", \"cpus\"]\n params = [\n get_benchmark_shapes(\"TimeFromPandas\"),\n [4, 16, 32],\n ]\n\n def setup(self, shape, cpus):\n self.data = pandas.DataFrame(gen_data(\"int\", *shape, RAND_LOW, RAND_HIGH))\n from modin.config import NPartitions\n\n NPartitions.get = lambda: cpus\n # trigger ray init\n pd.DataFrame([])\n\n def time_from_pandas(self, shape, cpus):\n execute(from_pandas(self.data))\n\n\nclass TimeToPandas:\n param_names = [\"shape\", \"cpus\"]\n params = [\n get_benchmark_shapes(\"TimeToPandas\"),\n [4, 16, 32],\n ]\n\n def setup(self, shape, cpus):\n from modin.config import NPartitions\n\n NPartitions.get = lambda: cpus\n self.data = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH, impl=\"modin\")\n\n def time_to_pandas(self, shape, cpus):\n # to_pandas is already synchronous\n to_pandas(self.data)\n\n\nclass TimeToNumPy:\n param_names = [\"shape\", \"cpus\"]\n params = [\n get_benchmark_shapes(\"TimeToNumPy\"),\n [4, 16, 32],\n ]\n\n def setup(self, shape, cpus):\n from modin.config import NPartitions\n\n NPartitions.get = lambda: cpus\n self.data = generate_dataframe(\"int\", *shape, RAND_LOW, RAND_HIGH, impl=\"modin\")\n\n def time_to_numpy(self, shape, cpus):\n # to_numpy is already synchronous\n to_numpy(self.data)\n\n\nfrom ..utils import setup # noqa: E402, F401\n"
},
{
"path": "asv_bench/benchmarks/utils/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Modin benchmarks utils.\"\"\"\n\nfrom .common import (\n IMPL,\n execute,\n gen_data,\n gen_nan_data,\n generate_dataframe,\n get_shape_id,\n prepare_io_data,\n prepare_io_data_parquet,\n random_booleans,\n random_columns,\n random_string,\n setup,\n translator_groupby_ngroups,\n)\nfrom .compatibility import ASV_USE_IMPL, ASV_USE_STORAGE_FORMAT\nfrom .data_shapes import GROUPBY_NGROUPS, RAND_HIGH, RAND_LOW, get_benchmark_shapes\n\n__all__ = [\n \"ASV_USE_IMPL\",\n \"ASV_USE_STORAGE_FORMAT\",\n \"RAND_LOW\",\n \"RAND_HIGH\",\n \"GROUPBY_NGROUPS\",\n \"get_benchmark_shapes\",\n \"IMPL\",\n \"execute\",\n \"get_shape_id\",\n \"gen_data\",\n \"gen_nan_data\",\n \"generate_dataframe\",\n \"prepare_io_data\",\n \"prepare_io_data_parquet\",\n \"random_string\",\n \"random_columns\",\n \"random_booleans\",\n \"translator_groupby_ngroups\",\n \"setup\",\n]\n"
},
{
"path": "asv_bench/benchmarks/utils/common.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"\nThe module contains the functionality that is used when benchmarking Modin commits.\n\nIn the case of using utilities from the main Modin code, there is a chance that when\nbenchmarking old commits, the utilities changed, which in turn can unexpectedly affect\nthe performance results, hence some utility functions are duplicated here.\n\"\"\"\n\nimport logging\nimport uuid\nfrom typing import Optional, Union\n\nimport numpy as np\nimport pandas\n\nimport modin.pandas\n\nfrom .compatibility import ASV_DATASET_SIZE, ASV_USE_ENGINE, ASV_USE_IMPL\nfrom .data_shapes import RAND_HIGH, RAND_LOW\n\nPOSSIBLE_IMPL = {\n \"modin\": modin.pandas,\n \"pandas\": pandas,\n}\nIMPL = POSSIBLE_IMPL[ASV_USE_IMPL]\n\n\ndef translator_groupby_ngroups(groupby_ngroups: Union[str, int], shape: tuple) -> int:\n \"\"\"\n Translate a string representation of the number of groups, into a number.\n\n Parameters\n ----------\n groupby_ngroups : str or int\n Number of groups that will be used in `groupby` operation.\n shape : tuple\n Same as pandas.Dataframe.shape.\n\n Returns\n -------\n int\n \"\"\"\n if ASV_DATASET_SIZE == \"big\":\n if groupby_ngroups == \"huge_amount_groups\":\n return min(shape[0] // 2, 5000)\n return groupby_ngroups\n else:\n return groupby_ngroups\n\n\nclass weakdict(dict): # noqa: GL08\n __slots__ = (\"__weakref__\",)\n\n\ndata_cache = dict()\ndataframes_cache = dict()\n\n\ndef gen_nan_data(nrows: int, ncols: int) -> dict:\n \"\"\"\n Generate nan data with caching.\n\n The generated data are saved in the dictionary and on a subsequent call,\n if the keys match, saved data will be returned. Therefore, we need\n to carefully monitor the changing of saved data and make its copy if needed.\n\n Parameters\n ----------\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n\n Returns\n -------\n modin.pandas.DataFrame or pandas.DataFrame or modin.pandas.Series or pandas.Series\n DataFrame or Series with shape (nrows, ncols) or (nrows,), respectively.\n \"\"\"\n cache_key = (ASV_USE_IMPL, nrows, ncols)\n if cache_key in data_cache:\n return data_cache[cache_key]\n\n logging.info(\"Generating nan data {} rows and {} columns\".format(nrows, ncols))\n\n if ncols > 1:\n columns = [f\"col{x}\" for x in range(ncols)]\n data = IMPL.DataFrame(np.nan, index=IMPL.RangeIndex(nrows), columns=columns)\n elif ncols == 1:\n data = IMPL.Series(np.nan, index=IMPL.RangeIndex(nrows))\n else:\n assert False, \"Number of columns (ncols) should be >= 1\"\n\n data_cache[cache_key] = data\n return data\n\n\ndef gen_int_data(nrows: int, ncols: int, rand_low: int, rand_high: int) -> dict:\n \"\"\"\n Generate int data.\n\n Parameters\n ----------\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n\n Returns\n -------\n dict\n Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.\n \"\"\"\n data = {\n \"col{}\".format(i): np.random.randint(rand_low, rand_high, size=(nrows))\n for i in range(ncols)\n }\n return data\n\n\ndef gen_str_int_data(nrows: int, ncols: int, rand_low: int, rand_high: int) -> dict:\n \"\"\"\n Generate int data and string data.\n\n Parameters\n ----------\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n\n Returns\n -------\n dict\n Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.\n One of the columns with string values.\n \"\"\"\n data = gen_int_data(nrows, ncols, rand_low, rand_high).copy()\n # convert values in arbitary column to string type\n key = list(data.keys())[0]\n data[key] = [f\"str_{x}\" for x in data[key]]\n return data\n\n\ndef gen_true_false_int_data(nrows, ncols, rand_low, rand_high):\n \"\"\"\n Generate int data and string data \"true\" and \"false\" values.\n\n Parameters\n ----------\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n\n Returns\n -------\n dict\n Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.\n One half of the columns with integer values, another half - with \"true\" and\n \"false\" string values.\n \"\"\"\n data = gen_int_data(nrows // 2, ncols // 2, rand_low, rand_high)\n\n data_true_false = {\n \"tf_col{}\".format(i): np.random.choice(\n [\"Yes\", \"true\", \"No\", \"false\"], size=(nrows - nrows // 2)\n )\n for i in range(ncols - ncols // 2)\n }\n data.update(data_true_false)\n return data\n\n\ndef gen_data(\n data_type: str,\n nrows: int,\n ncols: int,\n rand_low: int,\n rand_high: int,\n) -> dict:\n \"\"\"\n Generate data with caching.\n\n The generated data are saved in the dictionary and on a subsequent call,\n if the keys match, saved data will be returned. Therefore, we need\n to carefully monitor the changing of saved data and make its copy if needed.\n\n Parameters\n ----------\n data_type : {\"int\", \"str_int\", \"true_false_int\"}\n Type of data generation.\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n\n Returns\n -------\n dict\n Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.\n\n Notes\n -----\n Returned data type depends on the `data_type` parameter in the next way:\n - `data_type`==\"int\" - all columns will be contain only integer values;\n - `data_type`==\"str_int\" some of the columns will be of string type;\n - `data_type`==\"true_false_int\" half of the columns will be filled with\n string values representing \"true\" and \"false\" values and another half - with\n integers.\n \"\"\"\n type_to_generator = {\n \"int\": gen_int_data,\n \"str_int\": gen_str_int_data,\n \"true_false_int\": gen_true_false_int_data,\n }\n cache_key = (data_type, nrows, ncols, rand_low, rand_high)\n if cache_key in data_cache:\n return data_cache[cache_key]\n\n logging.info(\n \"Generating {} data {} rows and {} columns [{}-{}]\".format(\n data_type, nrows, ncols, rand_low, rand_high\n )\n )\n assert data_type in type_to_generator\n data_generator = type_to_generator[data_type]\n\n data = data_generator(nrows, ncols, rand_low, rand_high)\n data_cache[cache_key] = weakdict(data)\n\n return data\n\n\ndef generate_dataframe(\n data_type: str,\n nrows: int,\n ncols: int,\n rand_low: int,\n rand_high: int,\n groupby_ncols: Optional[int] = None,\n count_groups: Optional[int] = None,\n gen_unique_key: bool = False,\n cache_prefix: str = None,\n impl: str = None,\n) -> Union[modin.pandas.DataFrame, pandas.DataFrame]:\n \"\"\"\n Generate DataFrame with caching.\n\n The generated dataframes are saved in the dictionary and on a subsequent call,\n if the keys match, one of the saved dataframes will be returned. Therefore, we need\n to carefully monitor that operations that change the dataframe work with its copy.\n\n Parameters\n ----------\n data_type : str\n Type of data generation;\n supported types: {\"int\", \"str_int\"}.\n nrows : int\n Number of rows.\n ncols : int\n Number of columns.\n rand_low : int\n Low bound for random generator.\n rand_high : int\n High bound for random generator.\n groupby_ncols : int, default: None\n Number of columns for which `groupby` will be called in the future;\n to get more stable performance results, we need to have the same number of values\n in each group every benchmarking time.\n count_groups : int, default: None\n Count of groups in groupby columns.\n gen_unique_key : bool, default: False\n Generate `col1` column where all elements are unique.\n cache_prefix : str, optional\n Prefix to add to the cache key of the requested frame.\n impl : str, optional\n Implementation used to create the dataframe;\n supported implemetations: {\"modin\", \"pandas\"}.\n\n Returns\n -------\n modin.pandas.DataFrame or pandas.DataFrame [and list]\n\n Notes\n -----\n The list of groupby columns names returns when groupby columns are generated.\n \"\"\"\n assert not (\n (groupby_ncols is None) ^ (count_groups is None)\n ), \"You must either specify both parameters 'groupby_ncols' and 'count_groups' or none of them.\"\n\n if groupby_ncols and count_groups:\n ncols -= groupby_ncols\n\n if impl is None:\n impl = ASV_USE_IMPL\n\n cache_key = (\n impl,\n data_type,\n nrows,\n ncols,\n rand_low,\n rand_high,\n groupby_ncols,\n count_groups,\n gen_unique_key,\n )\n\n if cache_prefix is not None:\n cache_key = (cache_prefix, *cache_key)\n\n if cache_key in dataframes_cache:\n return dataframes_cache[cache_key]\n\n logging.info(\n \"Allocating {} DataFrame {}: {} rows and {} columns [{}-{}]\".format(\n impl, data_type, nrows, ncols, rand_low, rand_high\n )\n )\n data = gen_data(data_type, nrows, ncols, rand_low, rand_high)\n\n if groupby_ncols and count_groups:\n groupby_columns = [f\"groupby_col{x}\" for x in range(groupby_ncols)]\n for groupby_col in groupby_columns:\n data[groupby_col] = np.tile(np.arange(count_groups), nrows // count_groups)\n\n if gen_unique_key:\n data[\"col1\"] = np.arange(nrows)\n\n df = POSSIBLE_IMPL[impl].DataFrame(data)\n\n if groupby_ncols and count_groups:\n dataframes_cache[cache_key] = df, groupby_columns\n return df, groupby_columns\n\n dataframes_cache[cache_key] = df\n return df\n\n\ndef random_string() -> str:\n \"\"\"\n Create a 36-character random string.\n\n Returns\n -------\n str\n \"\"\"\n return str(uuid.uuid4())\n\n\ndef random_columns(df_columns: list, columns_number: int) -> list:\n \"\"\"\n Pick sublist of random columns from a given sequence.\n\n Parameters\n ----------\n df_columns : list\n Columns to choose from.\n columns_number : int\n How many columns to pick.\n\n Returns\n -------\n list\n \"\"\"\n return list(np.random.choice(df_columns, size=columns_number))\n\n\ndef random_booleans(number: int) -> list:\n \"\"\"\n Create random list of booleans with `number` elements.\n\n Parameters\n ----------\n number : int\n Count of booleans in result list.\n\n Returns\n -------\n list\n \"\"\"\n return list(np.random.choice([True, False], size=number))\n\n\ndef execute(df: Union[modin.pandas.DataFrame, pandas.DataFrame]):\n \"\"\"\n Make sure the calculations are finished.\n\n Parameters\n ----------\n df : modin.pandas.DataFrame or pandas.Datarame\n DataFrame to be executed.\n \"\"\"\n if ASV_USE_IMPL == \"modin\":\n partitions = df._query_compiler._modin_frame._partitions.flatten()\n mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls\n if len(partitions) and hasattr(mgr_cls, \"wait_partitions\"):\n mgr_cls.wait_partitions(partitions)\n return\n\n # compatibility with old Modin versions\n all(\n map(\n lambda partition: partition.drain_call_queue() or True,\n partitions,\n )\n )\n if ASV_USE_ENGINE == \"ray\":\n from ray import wait\n\n all(map(lambda partition: wait([partition._data]), partitions))\n elif ASV_USE_ENGINE == \"dask\":\n from dask.distributed import wait\n\n all(map(lambda partition: wait(partition._data), partitions))\n elif ASV_USE_ENGINE == \"python\":\n pass\n\n elif ASV_USE_IMPL == \"pandas\":\n pass\n\n\ndef get_shape_id(shape: tuple) -> str:\n \"\"\"\n Join shape numbers into a string with `_` delimiters.\n\n Parameters\n ----------\n shape : tuple\n Same as pandas.Dataframe.shape.\n\n Returns\n -------\n str\n \"\"\"\n return \"_\".join([str(element) for element in shape])\n\n\ndef prepare_io_data(test_filename: str, data_type: str, shapes: list):\n \"\"\"\n Prepare data for IO tests with caching.\n\n Parameters\n ----------\n test_filename : str\n Unique file identifier that is used to distinguish data\n for different tests.\n data_type : {\"int\", \"str_int\", \"true_false_int\"}\n Type of data generation.\n shapes : list\n Data shapes to prepare.\n\n Returns\n -------\n test_filenames : dict\n Dictionary that maps dataset shape to the file on disk.\n \"\"\"\n test_filenames = {}\n for shape in shapes:\n shape_id = get_shape_id(shape)\n test_filenames[shape_id] = f\"{test_filename}_{shape_id}_{data_type}.csv\"\n df = generate_dataframe(data_type, *shape, RAND_LOW, RAND_HIGH, impl=\"pandas\")\n df.to_csv(test_filenames[shape_id], index=False)\n\n return test_filenames\n\n\ndef prepare_io_data_parquet(test_filename: str, data_type: str, shapes: list):\n \"\"\"\n Prepare data for IO tests with caching.\n\n Parameters\n ----------\n test_filename : str\n Unique file identifier that is used to distinguish data\n for different tests.\n data_type : \"str_int\"\n Type of data generation.\n shapes : list\n Data shapes to prepare.\n\n Returns\n -------\n test_filenames : dict\n Dictionary that maps dataset shape to the file on disk.\n \"\"\"\n test_filenames = {}\n for shape in shapes:\n shape_id = get_shape_id(shape)\n test_filenames[shape_id] = f\"{test_filename}_{shape_id}_{data_type}.parquet\"\n df = generate_dataframe(data_type, *shape, RAND_LOW, RAND_HIGH, impl=\"pandas\")\n df.to_parquet(test_filenames[shape_id], index=False)\n\n return test_filenames\n\n\ndef setup(*args, **kwargs): # noqa: GL08\n # This function just needs to be imported into each benchmark file to\n # set up the random seed before each function. ASV run it automatically.\n # https://asv.readthedocs.io/en/latest/writing_benchmarks.html\n np.random.seed(42)\n"
},
{
"path": "asv_bench/benchmarks/utils/compatibility.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Compatibility layer for parameters used by ASV.\"\"\"\n\nimport os\n\nimport modin.pandas as pd\n\ntry:\n from modin.config import NPartitions\n\n NPARTITIONS = NPartitions.get()\nexcept ImportError:\n NPARTITIONS = pd.DEFAULT_NPARTITIONS\n\ntry:\n from modin.config import AsvImplementation, Engine, StorageFormat, TestDatasetSize\n\n ASV_USE_IMPL = AsvImplementation.get()\n ASV_DATASET_SIZE = TestDatasetSize.get() or \"Small\"\n ASV_USE_ENGINE = Engine.get()\n ASV_USE_STORAGE_FORMAT = StorageFormat.get()\nexcept ImportError:\n # The same benchmarking code can be run for different versions of Modin, so in\n # case of an error importing important variables, we'll just use predefined values\n ASV_USE_IMPL = os.environ.get(\"MODIN_ASV_USE_IMPL\", \"modin\")\n ASV_DATASET_SIZE = os.environ.get(\"MODIN_TEST_DATASET_SIZE\", \"Small\")\n ASV_USE_ENGINE = os.environ.get(\"MODIN_ENGINE\", \"Ray\")\n ASV_USE_STORAGE_FORMAT = os.environ.get(\"MODIN_STORAGE_FORMAT\", \"Pandas\")\n\nASV_USE_IMPL = ASV_USE_IMPL.lower()\nASV_DATASET_SIZE = ASV_DATASET_SIZE.lower()\nASV_USE_ENGINE = ASV_USE_ENGINE.lower()\nASV_USE_STORAGE_FORMAT = ASV_USE_STORAGE_FORMAT.lower()\n\nassert ASV_USE_IMPL in (\"modin\", \"pandas\")\nassert ASV_DATASET_SIZE in (\"big\", \"small\")\nassert ASV_USE_ENGINE in (\"ray\", \"dask\", \"python\", \"unidist\")\nassert ASV_USE_STORAGE_FORMAT in (\"pandas\")\n"
},
{
"path": "asv_bench/benchmarks/utils/data_shapes.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\n\"\"\"Define data shapes.\"\"\"\n\nimport json\nimport os\n\nfrom .compatibility import ASV_DATASET_SIZE\n\nRAND_LOW = 0\nRAND_HIGH = 100\n\nBINARY_OP_DATA_SIZE = {\n \"big\": [\n [[5000, 5000], [5000, 5000]],\n # the case extremely inefficient\n # [[20, 500_000], [10, 1_000_000]],\n [[500_000, 20], [1_000_000, 10]],\n ],\n \"small\": [[[250, 250], [250, 250]], [[10_000, 20], [25_000, 10]]],\n}\nUNARY_OP_DATA_SIZE = {\n \"big\": [\n [5000, 5000],\n # the case extremely inefficient\n # [10, 1_000_000],\n [1_000_000, 10],\n ],\n \"small\": [[250, 250], [10_000, 10]],\n}\nSERIES_DATA_SIZE = {\n \"big\": [[100_000, 1]],\n \"small\": [[10_000, 1]],\n}\nBINARY_OP_SERIES_DATA_SIZE = {\n \"big\": [\n [[500_000, 1], [1_000_000, 1]],\n [[500_000, 1], [500_000, 1]],\n ],\n \"small\": [[[5_000, 1], [10_000, 1]]],\n}\n\n\nDEFAULT_GROUPBY_NGROUPS = {\n \"big\": [100, \"huge_amount_groups\"],\n \"small\": [5],\n}\nGROUPBY_NGROUPS = DEFAULT_GROUPBY_NGROUPS[ASV_DATASET_SIZE]\n\n_DEFAULT_CONFIG_T = [\n (\n UNARY_OP_DATA_SIZE[ASV_DATASET_SIZE],\n [\n # Pandas storage format benchmarks\n \"TimeGroupByMultiColumn\",\n \"TimeGroupByDefaultAggregations\",\n \"TimeGroupByDictionaryAggregation\",\n \"TimeSetItem\",\n \"TimeInsert\",\n \"TimeArithmetic\",\n \"TimeSortValues\",\n \"TimeDrop\",\n \"TimeHead\",\n \"TimeTail\",\n \"TimeExplode\",\n \"TimeFillna\",\n \"TimeFillnaDataFrame\",\n \"TimeValueCountsFrame\",\n \"TimeValueCountsSeries\",\n \"TimeIndexing\",\n \"TimeMultiIndexing\",\n \"TimeResetIndex\",\n \"TimeAstype\",\n \"TimeDescribe\",\n \"TimeProperties\",\n \"TimeReindex\",\n \"TimeReindexMethod\",\n \"TimeFillnaMethodDataframe\",\n \"TimeDropDuplicatesDataframe\",\n \"TimeStack\",\n \"TimeUnstack\",\n \"TimeRepr\",\n \"TimeMaskBool\",\n \"TimeIsnull\",\n \"TimeDropna\",\n \"TimeEquals\",\n # IO benchmarks\n \"TimeReadCsvSkiprows\",\n \"TimeReadCsvTrueFalseValues\",\n \"TimeReadCsvNamesDtype\",\n \"TimeReadParquet\",\n # Scalability benchmarks\n \"TimeFromPandas\",\n \"TimeToPandas\",\n \"TimeToNumPy\",\n ],\n ),\n (\n BINARY_OP_DATA_SIZE[ASV_DATASET_SIZE],\n [\n # Pandas storage format benchmarks\n \"TimeJoin\",\n \"TimeMerge\",\n \"TimeMergeDefault\",\n \"TimeConcat\",\n \"TimeAppend\",\n \"TimeBinaryOp\",\n \"TimeLevelAlign\",\n ],\n ),\n (\n SERIES_DATA_SIZE[ASV_DATASET_SIZE],\n [\n # Pandas storage format benchmarks\n \"TimeFillnaSeries\",\n \"TimeGroups\",\n \"TimeIndexingNumericSeries\",\n \"TimeFillnaMethodSeries\",\n \"TimeDatetimeAccessor\",\n \"TimeSetCategories\",\n \"TimeRemoveCategories\",\n \"TimeDropDuplicatesSeries\",\n ],\n ),\n (\n BINARY_OP_SERIES_DATA_SIZE[ASV_DATASET_SIZE],\n [\n # Pandas storage format benchmarks\n \"TimeBinaryOpSeries\",\n ],\n ),\n]\n\nDEFAULT_CONFIG = {}\nDEFAULT_CONFIG[\"MergeCategoricals\"] = (\n [[10_000, 2]] if ASV_DATASET_SIZE == \"big\" else [[1_000, 2]]\n)\nDEFAULT_CONFIG[\"TimeJoinStringIndex\"] = (\n [[100_000, 64]] if ASV_DATASET_SIZE == \"big\" else [[1_000, 4]]\n)\nDEFAULT_CONFIG[\"TimeReplace\"] = (\n [[10_000, 2]] if ASV_DATASET_SIZE == \"big\" else [[1_000, 2]]\n)\nfor config in (_DEFAULT_CONFIG_T,):\n for _shape, _names in config:\n DEFAULT_CONFIG.update({_name: _shape for _name in _names})\n\n# Correct shapes in the case when the operation ended with a timeout error\nif ASV_DATASET_SIZE == \"big\":\n DEFAULT_CONFIG[\"TimeMergeDefault\"] = [\n [[1000, 1000], [1000, 1000]],\n [[500_000, 20], [1_000_000, 10]],\n ]\n DEFAULT_CONFIG[\"TimeLevelAlign\"] = [\n [[2500, 2500], [2500, 2500]],\n [[250_000, 20], [500_000, 10]],\n ]\n DEFAULT_CONFIG[\"TimeStack\"] = [\n [1500, 1500],\n [100_000, 10],\n ]\n DEFAULT_CONFIG[\"TimeUnstack\"] = DEFAULT_CONFIG[\"TimeStack\"]\n\nCONFIG_FROM_FILE = None\n\n\ndef get_benchmark_shapes(bench_id: str):\n \"\"\"\n Get custom benchmark shapes from a json file stored in MODIN_ASV_DATASIZE_CONFIG.\n\n If `bench_id` benchmark is not found in the file, then the default value will\n be used.\n\n Parameters\n ----------\n bench_id : str\n Unique benchmark identifier that is used to get shapes.\n\n Returns\n -------\n list\n Benchmark shapes.\n \"\"\"\n global CONFIG_FROM_FILE\n if not CONFIG_FROM_FILE:\n try:\n from modin.config import AsvDataSizeConfig\n\n filename = AsvDataSizeConfig.get()\n except ImportError:\n filename = os.environ.get(\"MODIN_ASV_DATASIZE_CONFIG\", None)\n if filename:\n # should be json\n with open(filename) as _f:\n CONFIG_FROM_FILE = json.load(_f)\n\n if CONFIG_FROM_FILE and bench_id in CONFIG_FROM_FILE:\n return CONFIG_FROM_FILE[bench_id]\n return DEFAULT_CONFIG[bench_id]\n"
},
{
"path": "asv_bench/test/__init__.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n"
},
{
"path": "asv_bench/test/test_utils.py",
"content": "# Licensed to Modin Development Team under one or more contributor license agreements.\n# See the NOTICE file distributed with this work for additional information regarding\n# copyright ownership. The Modin Development Team licenses this file to you under the\n# Apache License, Version 2.0 (the \"License\"); you may not use this file except in\n# compliance with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software distributed under\n# the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\n# ANY KIND, either express or implied. See the License for the specific language\n# governing permissions and limitations under the License.\n\nfrom unittest.mock import Mock, mock_open, patch\n\nimport numpy as np\nimport pytest\nfrom benchmarks.utils import data_shapes, execute, get_benchmark_shapes\n\nimport modin.pandas as pd\nfrom modin.config import AsvDataSizeConfig\n\n\n@pytest.mark.parametrize(\n \"asv_config_content, result\",\n [\n (\n '{\"TimeJoin\": [[[10, 10], [15, 15]], [[11, 11], [13, 13]]], \\\n \"TimeGroupBy\": [[11, 11], [13, 13]]}',\n [\n [\n # binary shapes\n [[10, 10], [15, 15]],\n [[11, 11], [13, 13]],\n ],\n [\n # unary shapes\n [11, 11],\n [13, 13],\n ],\n ],\n ),\n ],\n)\n@patch.object(data_shapes, \"CONFIG_FROM_FILE\", new=None)\ndef test_get_benchmark_shapes(asv_config_content, result):\n AsvDataSizeConfig.put(\"mock_filename\")\n with patch(\"builtins.open\", mock_open(read_data=asv_config_content)):\n assert result[0] == get_benchmark_shapes(\"TimeJoin\")\n assert result[1] == get_benchmark_shapes(\"TimeGroupBy\")\n\n\n@pytest.mark.parametrize(\n \"asv_config_content, result\",\n [\n (\n '{\"TimeJoin\": [[[10, 10], [15, 15]]]',\n [[100, 100]],\n ),\n ],\n)\n@patch.object(data_shapes, \"CONFIG_FROM_FILE\", new=None)\ndef test_get_benchmark_shapes_default(asv_config_content, result):\n AsvDataSizeConfig.put(None)\n with patch.object(data_shapes, \"DEFAULT_CONFIG\", new={\"TimeJoin\": result}):\n assert result == get_benchmark_shapes(\"TimeJoin\")\n\n\ndef test_execute():\n df = pd.DataFrame(np.random.rand(100, 64))\n partitions = df._query_compiler._modin_frame._partitions.flatten()\n mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls\n with patch.object(mgr_cls, \"wait_partitions\", new=Mock()):\n execute(df)\n mgr_cls.wait_partitions.assert_called_once()\n assert (mgr_cls.wait_partitions.call_args[0] == partitions).all()\n"
},
{
"path": "ci/teamcity/Dockerfile.teamcity-ci",
"content": "# Create images from this container like this (in modin repo root):\n#\n# git rev-parse HEAD > ci/teamcity/git-rev\n#\n# tar cf ci/teamcity/modin.tar .\n#\n# docker build --build-arg ENVIRONMENT=environment-dev.yml -t modin-project/teamcity-ci:${BUILD_NUMBER} -f ci/teamcity/Dockerfile.teamcity-ci ci/teamcity\n\nFROM rayproject/ray:latest\n\nARG ENVIRONMENT=environment-dev.yml\n\nADD modin.tar /modin\nADD git-rev /modin/git-rev\n\nWORKDIR /modin\nRUN sudo chown -R ray /modin\n\n# Make RUN commands use `bash --login`:\nSHELL [\"/bin/bash\", \"--login\", \"-c\"]\n\n# Initialize conda in bash config files:\nRUN conda init bash\nENV PATH /home/ray/anaconda3/envs/modin/bin:$PATH\n\nRUN conda config --set channel_priority strict\nRUN conda update python -y\nRUN conda env create -f ${ENVIRONMENT}\nRUN conda install curl PyGithub\n\n# Activate the environment, and make sure it's activated:\n# The following line also removed conda initialization from\n# ~/.bashrc so conda starts complaining that it should be\n# initialized for bash. But it is necessary to do it because\n# activation is not always executed when \"docker exec\" is used\n# and then conda initialization overwrites PATH with its base\n# environment where python doesn't have any packages installed.\nRUN echo \"conda activate modin\" > ~/.bashrc\nRUN echo \"Make sure environment is activated\"\nRUN conda list -n modin\n"
},
{
"path": "ci/teamcity/build-docker.py",
"content": "import os\nimport sys\n\n\ndef execute_command(cmd):\n status = os.system(cmd)\n ec = os.WEXITSTATUS(status)\n if ec != 0:\n raise SystemExit('Command \"{}\" failed'.format(cmd))\n\n\nif sys.platform.startswith(\"linux\"):\n execute_command(\"git rev-parse HEAD > git-rev\")\n execute_command(\n \"(cd ../.. && git archive -o ci/teamcity/modin.tar $(cat ci/teamcity/git-rev))\"\n )\n base_image = \"ray-project/deploy\"\n requirements = \"requirements-dev.txt\"\n execute_command(\n \"docker build -f Dockerfile.modin-base --build-arg BASE_IMAGE={} -t modin-project/modin-base .\".format(\n base_image\n )\n )\nelse:\n raise SystemExit(\n \"TeamCity CI in Docker containers is supported only on Linux at the moment.\"\n )\n\nexecute_command(\n \"docker build -f Dockerfile.teamcity-ci --build-arg REQUIREMENTS={} -t modin-project/teamcity-ci .\".format(\n requirements\n )\n)\n\nif sys.platform.startswith(\"linux\"):\n execute_command(\"rm ./modin.tar ./git-rev\")\n"
},
{
"path": "ci/teamcity/comment_on_pr.py",
"content": "\"\"\"\nPost the comment like the following to the PR:\n```\n:robot: TeamCity test results bot :robot:\n\n![\"\"](\"https://upload.wikimedia.org/wikipedia/commons/thumb/8/86/Teamcity_Logo.png/600px-Teamcity_Logo.png\"){kind=logo}
\n TeamCity {} test results bot![](\\\"../../../img/pandas_multicore.png\\\")
![](\\\"../../../img/modin_multicore.png\\\")
\\n\",\n \"